Continuations and Stackless Python

1
Continuations and Stackless Python

• Where Do You Want To Jump Today?

2
Stacklessssshhh Python ?
... And the clueless implementor ...
3
Applications of Stackless Python

• FAU Florida Atlantic University )
• Soft real time control system for autonomous
  underwater vehicles.
• Concurrent hierarchy of hierarchical state
  machines that can be loaded dynamically. Language
  Python.
• Real thread implementation is expensive and
  operates at 1 Hz Resolution. Microthreads with
  Stackless Python supposed to run at 12 Hz or
  faster, also supporting multiple simultaneous
  behaviors. My current project ?
• Gordon McMillan Highly parallelized FTP-Server
  (not open Source yet)
• Sam Rushing Medusa on top of coroutines ?)
• MMPOG Eve! Massively Multi-Player Online Game.
  ?)
• ) sponsored work ?) may become sponsored, soon

4
Moving Targets

• Early version (1999)
• Try to minimize changes in order to get into the
  core. Guido reejcts it but accepts the paper for
  IPC8.
• New version (2000)
• Still only a small number of changes
• Massive changes to ceval.c for speed
• Gave up on integration
• Make it attractive even without stackless
  features
• This is hard to maintain! Long delay for SLP 2.0.
• Version 2.0 (2000/2001) The Kiel Edition
• Maintainability was re-gained
• Minimized code-reorderings, all systematic
  optimization done by script

5
Stackless Python 2.2? No!

• Integration into Standard
• New implementation
• Massive changes to core
• Avoid any recursion if possible.
• Coroutines, Generatrors, Microthreads
• Not to be done by me (!!reasons Gordo does it)
• No continuations at all!
• Splitting paths?
• Stackless python will
• do all of the above
• Continue to support continuations
• Educational purposes
• Studying expense
• Try to be faster

6
How Does It Work?

• Stackful Python
• ...
• Interpreter interprets opcodes
• CALL_FUNCTION (Python function)
• Builds new frame
• Calls new interpreter, waiting
• ...
• Interpreter interprets opcodes
• ...
• Continues with result

7
How Does It Work? Trampoline Style

• Stackless Python
• Dispatcher
• Starts interpreter with topmost frame result
• Interpreter interprets opcodes
• CALL_FUNCTION (Python function)
• Builds new frame
• Returns to Dispatcher with unwind flag
• Dispatcher
• Starts interpreter with topmost frame result
• Interpreter interprets opcodes
• ... and so on

8
Why Continuations?

• Appears to be the simplest possible, most
  powerful structure
• Can model any kind of control flow
• (also any control flaw)
• Can build Generators, Coroutines and Microthreads
  in Python
• Can do much more! (But is it needed?)
• Scheme does it, so Python too?
• Can build very fast class-method like functions
• (see example)

9
Why Continuations - Example

• Def very_fast_with_state(init_params)
• do complicated initialization
• prepare local state. Locals are persisting
• args continuation.return_current()
• do the cheap calculation
• Return result
• fast_func very_fast_with_state(Hi, 4)
• fast_func(1) use it many times

10
Why Not Continuations?

• Generators, Coroutines and Microthreads can be
  implemented directly in C.
• Will be much faster.
• Safe encapsulation of internal continuations
• Full continuation support is cheap today
• But may become expensive in the future (Stack
  optimization, native compiler)
• Can build very fast class-method like functions
  with generators as well
• (see example)

11
Why Not Continuations - Example

• Def very_fast_no_continuation(init_params)
• do complicated initialization
• prepare local state. Locals are persisting
• args generator.initialize()
• While 1
• do the cheap calculation
• Args generator.suspend(result)
• fast_func very_fast_no_continuation(Hi, 4)
• fast_func(1) use it many times

12
Generator vs. Continuation

• Continuation is immutable
• Generator is mutable
• Continuation takes control over order of
  execution
• Generator does not introduce new control
  structure. Frame jumps from alone by opcode.

13
First Class vs. One-Shot

• Continuations are everywhere
• Program counter / return stack.
• Every todays program consists of a series of
  one-shot continuations.
• One-shot
• The continuation does not survive its execution
• First-class continuation
• Does survive execution, can be executed infinite
  times
• State of frame can always be reset
• First class is most powerful
• One-Shot is sufficient for Microthreads,
  Coroutines and Generators.

14
New Project Cheetah

• Stackless from the ground
• First Quick system-independent extension
  language
• Later Fast system-specific extension language
• Forth related/inspired engine
• Parsed from restricted Python code
• Able to extend regular Python with new primitives
• Able to replace the whole Python implementation
• Sponsors are most welcome