Slightly beyond Turings computability for studying Genetic Programming

1
Slightly beyond Turings computability for
studying Genetic Programming

• Olivier Teytaud, Tao, Inria, Lri, UMR CNRS 8623,
  Univ. Paris-Sud, Pascal, Digiteo

2
Outline

• What is genetic programming
• Formal analysis of Genetic Programming
• Why is there nothing else than Genetic
  Programming ?
• Computability point of view
• Complexity point of view

3
What is Genetic Programming (GP)

• GP mining Turing-equivalent spaces of functions
• Typical example symbolic regression.
• Inputs
• x1,x2,x3,,xN in 0,1
• y1,y2,y3,,yN in 0,1
  yif(xi)
• (xi,yi) assumed independently identically
  distributed (unknown distribution of probability)
• Goal
• Finding g such that
• Eg(x)-y C E Time(g,x)
• as small as possible

4
How does GP works ?

• GP evolutionary algorithm.
• Evolutionary algorithm
• P initial population
• While (my favorite criterion)
• Selection best functions in P according to some
  score
• Mutations random perturbations of progs in the
  Selection
• Cross-over merging of programs in the Selection
• P Selection Mutations Cross-over

5
How does GP works ?

• GP evolutionary algorithm.
• Evolutionary algorithm
• P initial population
• While (my favorite criterion)
• Selection best functions in P according to some
  score
• Mutations random perturbations of progs in the
  Selection
• Cross-over merging of programs in the Selection
• P Selection Mutations Cross-over

Does it work ?
6
How does GP works ?

• GP evolutionary algorithm.
• Evolutionary algorithm
• P initial population
• While (my favorite criterion)
• Selection best functions in P according to some
  score
• Mutations random perturbations of progs in the
  Selection
• Cross-over merging of programs in the Selection
• P Selection Mutations Cross-over

Does it work ?
Definitely, yes for robust and multimodal optimiza
tion in complex domains (trees, bitstrings,).
7
How does GP works ?

• GP evolutionary algorithm.
• Evolutionary algorithm
• P initial population
• While (my favorite criterion)
• Selection best functions in P according to some
  score
• Mutations random perturbations of progs in the
  Selection
• Cross-over merging of programs in the Selection
• P Selection Mutations Cross-over

Does it work ?
8
How does GP works ?

• GP evolutionary algorithm.
• Evolutionary algorithm
• P initial population
• While (my favorite criterion)
• Selection best functions in P according to some
  score
• Mutations random perturbations of progs in the
  Selection
• Cross-over merging of programs in the Selection
• P Selection Mutations Cross-over

Which score ? A nice question for mathematicians
9
Why studying GP ?

• GP is studied by many people
• 5440 articles in the GP bibliography 5
• More than 880 authors
• GP seemingly works
• Human-competitive results http//www.genetic-progr
  amming.com/humancompetitive.html
• Nothing else for mining Turing-equivalent spaces
  of programs
• Probably better than random search
• Not so many mathematical fundations in GP
• Not so many open problems in computability, in
  particular with applications

10
Outline

• What is genetic programming
• Formal analysis of Genetic Programming
• Why is there nothing else than Genetic
  Programming ?
• Computability point of view
• Complexity point of view

11
Formalization of GP

• What is typically GP ?
• No halting criterion. We stop when time is
  exhausted.
• No use of prior knowledge no use of f, whenever
  you know it.
• People (often) do not like GP because
• It is slow and has no halting criterion
• It uses the yif(xi) and not f (different from
  automatic code generation)
• ? Are these two elements necessary ?

12
Iterative algorithms
13
Black-box ?
14
Formalization of GP

• Summary
• GP uses only the f(xi) and the Time(f,xi).
• GP never halts O1, O2, O3, .
• Can we do better ?

15
Outline

• What is genetic programming
• Formal analysis of Genetic Programming
• Why is there nothing else than Genetic
  Programming ?
• Computability point of view
• Complexity point of view

16
Known results

• Whenever f is available (and not only the f(xi)
  ), computing O such that
• Of
• O optimal for size (or speed, or space )
• is not possible.
• (i.e. theres no Turing machine performing that
  task for all f)

17
A first (easy) good reason for GP.

• Whenever f is available (and not only the f(xi)
  ), computing O1, O2, , such that
• Op f for p sufficiently large
• Lim size(Op) optimal
• is possible, with proved convergence rates, e.g.
  by bloat penalization
• - while (true) - select the best program P for a
  compromise
• relevance on the n first examples
• penalization of size,
• e.g. Sum P(xi)-yi C( P , n )
• i lt n
• - nn1
• (see details of the proof and of the algorithm
  in the paper)

18
A first (easy) good reason for GP.

• Whenever f is not available (and not only the
  f(xi) ), computing O1, O2, , such that
• Op f for p sufficiently large
• Lim size(Op) optimal
• is possible, with proved convergence rates, e.g.
  by bloat penalization
• - consider a population of programs set n1
• - while (true) - select the best program P for a
  compromise
• relevance on the n first examples
• penalization of size,
• e.g. Sum P(xi)-yi C( P , n )
• i lt n
• - nn1
• (see details of the proof and of the algorithm
  in the paper)

19
A first (easy) good reason for GP.

• ? Asymptotically (only!), finding an optimal
• function O f is possible.
• ? No halting criterion is possible
• (avoids the use of an oracle in 0)

20
Outline

• What is genetic programming
• Formal analysis of Genetic Programming
• Why is there nothing else than Genetic
  Programming ?
• Computability point of view
• Complexity point of view

21
Outline

• What is genetic programming
• Formal analysis of Genetic Programming
• Why is there nothing else than Genetic
  Programming ?
• Computability point of view
• Complexity point of view
• Kolmogorovs complexity with bounded time
• Application to genetic programming

22
Kolmogorovs complexity

• Kolmogorovs complexity of x
• Minimum size of a program generating x
• Kolmogorovs complexity of x with time at most T
  
• Minimum size of a program generating x
• in time at most T.
• Kolmogorovs complexity in bounded time
• computable.

23
Outline

• What is genetic programming
• Formal analysis of Genetic Programming
• Why is there nothing else than Genetic
  Programming ?
• Computability point of view
• Complexity point of view
• Kolmogorovs complexity with bounded time
• Application to genetic programming

24
Kolmogorovs complexity and genetic programming

• GP uses expensive simulations of programs
• Can we get rid of the simulation time ? e.g. by
  using f not only as a black box ?
• Essentially, no
• Example of GP problem finding O as small as
  possible with
• ETime(O,x)ltTn,
• OltSn
• O(x)y
• If Tn O(2n) and some Sn O(log(n)), this
  requires time at least Tn/polynomial(n)
• Just simulating all programs shorter than Sn and
   faster  than Tn is possible in time
  polynomial(n)Tn

25
Outline

• What is genetic programming
• Formal analysis of Genetic Programming
• Why is there nothing else than Genetic
  Programming ?
• Computability point of view
• Complexity point of view
• Kolmogorovs complexity with bounded time
• Application to genetic programming
• Conclusion

26
Conclusion

• Summary
• GP is typically solving approximately problems in
  0
• A lot of work about approximating NP-complete
  problems, but not a lot about 0
• We provide a theoretical analysis of GP
• Conclusions
• GP uses expensive simulations, but the simulation
  cost can anyway not be removed.
• GP has no halting criterion, but no halting
  criterion can be found.
• Also,  bloat  penalization ensures consistency
  ? this point proposes a parametrization of the
  usual algorithms.

27
Conclusion

• Summary
• GP is typically solving approximately problems in
  0
• A lot of work about approximating NP-complete
  problems, but not a lot about 0
• We provide a theoretical analysis of GP
• Conclusions
• GP uses expensive simulations, but the simulation
  cost can anyway not be removed.
• GP has no halting criterion, but no halting
  criterion can be found.
• Also,  bloat  penalization ensures consistency
  ? this point proposes a parametrization of the
  usual algorithms.

28
Conclusion

• Summary
• GP is typically solving approximately problems in
  0
• A lot of work about approximating NP-complete
  problems, but not a lot about 0
• We provide a mathematical analysis of GP
• Conclusions
• GP uses expensive simulations, but the simulation
  cost can anyway not be removed.
• GP has no halting criterion, but no halting
  criterion can be found.
• Also,  bloat  penalization ensures consistency
  ? this point proposes a parametrization of the
  usual algorithms.