Look-ahead Linear Regression Trees (LLRT)

1
Look-ahead Linear Regression Trees (LLRT)

• David Vogel, A.I. Insight
• Ognian Asparouhov, A.I. Insight
• Tobias Scheffer, Max Planck Institute for
  Computer Science

2
What is a Linear Regression Tree?
3
Optimizing a Model Tree
4
Simulated Example 1
5
Simulated Example 2
Regression CART split on X2 LLRT split on X1 True Model
Training 1.338 0.440 0.104 0.103
Validation 1.491 0.490 0.101 0.101
6
LLRT Idea

• Brute Force Try every possible split on each
  splitting variable
• For each possible split, try every possible model
  in right and left partitions to achieve maximum
  accuracy
• Use massive amounts of optimization to make this
  possible

7
Related Citations

• 1992, RETIS (Karalic) optimizes the overall RSS
• RETIS optimization cited as intractable or
  non-scalable as recent as 2005 (Machine
  Learning)

8
Optimizations in LLRT

• Quick calculations for evaluating multiple leaf
  models from common sufficient statistics
• Matrix Solutions G.E. versus SVD
• Forward Stepwise model selection shortcuts
• Limit 10-20 possible splits per variable

9
Optimizing Regression Analysis

10
S-fold validation to avoid over-fitting
11
Stopping Rule

• Proposed split fails to improve result with two
  different samplings

12
M5 Run-time
13
(No Transcript)
14
(No Transcript)
15
Reviewer Criticisms

• Limitation of 10-20 possible split points
• Experimental results do not include comparisons
  to many model tree algorithms
• LLRT is slower than other model tree algorithms