Title: Meta Optimization
1Meta Optimization
- Improving Compiler Heuristics with Machine
Learning
Mark Stephenson, Una-May OReilly, Martin Martin,
and Saman Amarasinghe MIT Computer Architecture
Group
2Motivation
- Compiler writers are faced with many challenges
- Many compiler problems are NP-hard
- Modern architectures are inextricably complex
- Simple models cant capture architecture
intricacies - Micro-architectures change quickly
3Motivation
- Heuristics alleviate complexity woes
- Find good approximate solutions for a large class
of applications - Find solutions quickly
- Unfortunately
- They require a lot of trial-and-error tweaking to
achieve suitable performance
4Priority Functions
- A heuristics Achilles heel
- A single priority or cost function often dictates
the efficacy of a heuristic - Priority functions rank the options available to
a compiler heuristic - Graph coloring register allocation (selecting
nodes to spill) - List scheduling (identifying instructions in
worklist to schedule first) - Hyperblock formation (selecting paths to include)
5Machine Learning
- We propose using machine learning techniques to
automatically search the priority function space - Search space is feasible
- Make use of spare computer cycles
6Case Study I Hyperblock Formation
- Find predicatable regions of control flow
- Enumerate paths of control in region
- Exponential, but in practice its okay
- Prioritize paths based on several characteristics
- The priority function we want to optimize
- Add paths to hyperblock in priority order
7Case Study I IMPACTs Function
8Hyperblock Formation
- What are the important characteristic of a
hyperblock formation priority function? - IMPACT uses four characteristics
- Extract all the characteristics you can think of
and have a machine learning algorithm find the
priority function
9Hyperblock Formation
x1 Maximum ops over paths x2 Dependence height
x3 Number of paths x4 Number of operations
x5 Does path have subroutine calls? x6 Number of branches
x7 Does path have unsafe calls? x8 Path execution ratio
x9 Does path have pointer derefs? x10 Average ops executed in path
x11 Issue width of processor x12 Average predictability of branches in path
xN Predictability product of branches in path
10Genetic Programming
- GPs representation is a directly executable
expression - Basically a lisp expression (or an AST)
- In our case, GP variables are interesting
characteristics of the program
11Genetic Programming
- Searching algorithm analogous to natural
selection - Maintain a population of expressions
- Selection
- The fittest expressions in the population are
more likely to reproduce - Sexual reproduction
- Crossing over subexpressions of two expressions
- Mutation
12Genetic Programming
Create initial population (initial solutions)
- Most expressions in initial population are
randomly generated - It also seeded with the compiler writers best
guesses
Evaluation
Generation of variants (mutation and crossover)
Selection
Generations lt Limit?
END
13Genetic Programming
- Each expression is evaluated by compiling and
running benchmark(s) - Fitness is the relative speedup over the baseline
on benchmark(s)
Create initial population (initial solutions)
Evaluation
Generation of variants (mutation and crossover)
Selection
Generations lt Limit?
END
14Genetic Programming
- Just as with Natural Selection, the fittest
individuals are more likely to survive and
reproduce.
Create initial population (initial solutions)
Evaluation
Generation of variants (mutation and crossover)
Selection
Generations lt Limit?
END
15Genetic Programming
Create initial population (initial solutions)
Evaluation
Generation of variants (mutation and crossover)
Selection
Generations lt Limit?
END
16Genetic Programming
- Use crossover and mutation to generate new
expressions
Create initial population (initial solutions)
Evaluation
Generation of variants (mutation and crossover)
Selection
Generations lt Limit?
END
17Hyperblock ResultsCompiler Specialization
3.5
Train data set
Alternate data set
3
(add (sub (cmul (gt (cmul b0 0.8982 d17)d7))
(cmul b0 0.6183 d28)))
2.5
(add (div d20 d5) (tern b2 d0 d9))
2
Speedup
1.5
1.54
1.23
1
0.5
0
toast
Average
huff_enc
huff_dec
mpeg2dec
rawcaudio
rawdaudio
g721encode
g721decode
129.compress
18Hyperblock ResultsA General Purpose Priority
Function
19Cross ValidationTesting General Purpose
Applicability
20Case Study II Register AllocationA General
Purpose Priority Function
21Register Allocation ResultsCross Validation
22Conclusion
- Machine learning techniques can identify
effective priority functions - Proof of concept by evolving two well known
priority functions - Human cycles v. computer cycles
23GP Hyperblock SolutionsGeneral Purpose
- (add
- (sub (mul exec_ratio_mean 0.8720) 0.9400)
- (mul 0.4762
- (cmul (not has_pointer_deref)
- (mul 0.6727 num_paths)
- (mul 1.1609
- (add (sub
- (mul (div num_ops dependence_height)
10.8240) - exec_ratio)
- (sub (mul (cmul has_unsafe_jsr
predict_product_mean 0.9838) - (sub 1.1039 num_ops_max))
- (sub (mul dependence_height_mean
num_branches_max) num_paths)))))))
Intron that doesnt affect solution
24GP Hyperblock SolutionsGeneral Purpose
- (add
- (sub (mul exec_ratio_mean 0.8720) 0.9400)
- (mul 0.4762
- (cmul (not has_pointer_deref)
- (mul 0.6727 num_paths)
- (mul 1.1609
- (add (sub
- (mul (div num_ops dependence_height)
10.8240) - exec_ratio)
- (sub (mul (cmul has_unsafe_jsr
predict_product_mean 0.9838) - (sub 1.1039 num_ops_max))
- (sub (mul dependence_height_mean
num_branches_max) num_paths)))))))
Favor paths that dont have pointer dereferences
25GP Hyperblock SolutionsGeneral Purpose
- (add
- (sub (mul exec_ratio_mean 0.8720) 0.9400)
- (mul 0.4762
- (cmul (not has_pointer_deref)
- (mul 0.6727 num_paths)
- (mul 1.1609
- (add (sub
- (mul (div num_ops dependence_height)
10.8240) - exec_ratio)
- (sub (mul (cmul has_unsafe_jsr
predict_product_mean 0.9838) - (sub 1.1039 num_ops_max))
- (sub (mul dependence_height_mean
num_branches_max) num_paths)))))))
26GP Hyperblock SolutionsGeneral Purpose
- (add
- (sub (mul exec_ratio_mean 0.8720) 0.9400)
- (mul 0.4762
- (cmul (not has_pointer_deref)
- (mul 0.6727 num_paths)
- (mul 1.1609
- (add (sub
- (mul (div num_ops dependence_height)
10.8240) - exec_ratio)
- (sub (mul (cmul has_unsafe_jsr
predict_product_mean 0.9838) - (sub 1.1039 num_ops_max))
- (sub (mul dependence_height_mean
num_branches_max) num_paths)))))))
If a path calls a subroutine that may have side
effects, penalize it
27Case Study I IMPACTs Algorithm
A
4k
24k
Path exec haz ops dep pr
A-B-D-F-G 0 1.0 13 4 0
A-B-F-G 0.14 1.0 10 4 0.21
A-C-F-G 0.79 1.0 9 2 1.44
A-C-E-F-G 0.07 0.25 13 5 0.02
A-C-E-G 0 0.25 11 3 0
B
C
4k
22k
2k
10
E
D
2k
25
10
F
28k
G
28k
28Case Study I IMPACTs Algorithm
A
4k
24k
Path exec haz ops dep pr
A-B-D-F-G 0 1.0 13 4 0
A-B-F-G 0.14 1.0 10 4 0.21
A-C-F-G 0.79 1.0 9 2 1.44
A-C-E-F-G 0.07 0.25 13 5 0.02
A-C-E-G 0 0.25 11 3 0
B
C
4k
22k
2k
10
E
D
2k
25
10
F
28k
G
28k