DM Performance: Decile Lift Analysis

1
DM Performance Decile (Lift) Analysis
2
Decile Maximization(DMAX)

• Objective
• Find model f(x) (predictor variables x)
• such that performance in upper deciles
  (specified depth-of-file) is maximized
• Explicitly manages resource constraint
• mailings to particular depths-of file
• Performance at different mailing depths
• models optimized for different mailing depths

3
DMAX Illustrative Example
4
Case II 2 Response RateCum Response Lift
Comparison
5
Response Model Experimental Study

• Two aspects to fitness
• decile performance, overall fit to data
• Modeling Response
• Model 1/(1exp(-wx))
• Fitness fw1D w2C
• Decile performance (responders in top d deciles)
  
• Fit-to-data (Hosmer-Lemeshow goodness-of-fit)
• Bhattacharyya, S., Direct Marketing Response
  Models using Genetic Algorithms,
• KDD-98 Proceedings.

6
Top decile (DMAX 10)
2nd decile (DMAX 20)
3rd decile (DMAX 30)
7th decile (DMAX 70)
7
Learning with Resampling

• Fitness as average of performance over multiple
  sub-samples
• cross-validation
• high variance in performance
• sampling with replacement
• member-wise, generation-wise, run-wise

DMAX
Logit
8
Modeling on Multiple Objectives

• Model y1,..,yk f (x)
• simultaneously optimize on multiple objectives
• Some common DM modeling desirables
• response and high purchase revenues
• likely churners with high usage of services
• high tenure and usage
• purchase and non-return
• cross-selling, etc.
• or CPR (Combined Profit and Response) Models

9
Multiple objectives

• Traditional approaches
• multiple single-objective models, and combine
• weighted average of objectives
• conflicting objectives
• different levels of tradeoffs
• frontier of non-dominated solutions
• choice of final model based on diverse
  decision-maker objectives, can also be subjective

10
Pareto Frontier

• Non-dominated solutions
• multiple objectives ?i, f a(x) better than f
  b(x) if
• Single GA run obtains
• tradeoff frontier of
• non-dominated solutions f k(x)

?2
non-dominated models
dominated models
?1
11
Experimental Study Data

• Cellular-phone provider seeking to identify
  potential high-value churners
• two dependent variables
• binary Churn variable
• continuous variable measuring revenue ()
• predictors minutes-of-use (peak and off-peak),
  average charges, and payment information, etc.
• obtained after EDA, normalized to 0 mean 1 s.d
• 50,000 sample 25,000 for training, 25,000 for
  test set

12
Multiple Objectives Performance

• Churn lift
• model capturing more churners in top deciles is
  better
• -Lift
• model giving high revenue customers in upper
  deciles is better
• overall modeling objective
• maximize expected revenue saved through
  identification of high-value churners
• Churn-Lift -Lift

13
Experimental StudyNon-dominated models Decile 1
(Training)
Decile 1 (trg)
400
350
GP
300
GA
250
-Lift
Logistic
200
OLS
150
100
50
0
0
100
200
300
400
500
600
Churn-Lift
5 independent GA runs, aggregate the sets of
non-dominated solutions
14
Experimental StudyNon-dominated models Decile 1
(Test)
Decile 1 (Test)
400
350
300
GP
250
GA
-Lift
200
Logistic
150
OLS
100
50
0
0
100
200
300
400
500
Churn-Lift
15
Experimental StudyNon-dominated models Decile 2
(Test)
Decile 2 (Test)
300
250
GP
200
GA
-Lift
Logistic
150
OLS
100
50
0
0
50
100
150
200
250
300
350
400
450
Churn-Lift
16
Experimental StudyNon-dominated models Decile 3
(Test)
Decile 3 (Test)
250
GP
200
GA
Logistic
150
-Lift
OLS
100
50
0
0
50
100
150
200
250
300
350
Churn-Lift
17
Experimental StudyNon-dominated models Decile 7
(Test)
GP
GA
Logistic
OLS
18
Experimental StudyPerformance Summary
19
Multi-Objective ModelsElitism and Population
Size

• Elitism
• preserves non-dominated solutions in next
  generation

• Elitism particularly helpful when using smaller
  
• populations

20
General Optimization of Lifts

• Fitness function
• Seeks a general maximization of lifts at all
  deciles
• n of observations, nR of responders
• dependent-B value of ith obs.
• dependent-C value of ith obs. (obs. sorted
  on model scores)

(binary dependent var.)
(continuous dependent var.)
21
Specific vs. General Lift Opt
Table Best Prod-Lifts in Deciles
22
Specific vs. General Lift Opt.
Table Best -Lift and Churn-Lifts in Deciles