The New Dyninst Event Model

1
The New Dyninst Event Model

• James Waskiewicz

2
Motivation the series

• Several years in the making
• Paradyn separation requires
• DyninstAPI support for threads
• Among other things
• Supporting threads (a la Paradyn) requires
• Asynchronous event handling, which requires
• Solving the multi-event-source problem
• Eg. Listen for and handle events coming from
• Socket / file descriptor
• System call ( wait() )
• Solution use threads
• Just a can of worms
• Or Pandoras box?

3
Overall Design Concerns

• Keep thread stuff out of the way
• Dont want to pollute dyninst code with
  synchronization primitives
• Dyninst is already hard to fix when broken
• Keep threading issues modular
• Try to keep existing code thread-context free
• In other words, most dyninst code should be able
  to run successfully no matter what thread
  executes it.
• There can be some tricky situations here
  (inferior RPCs for example)
• Obvious exceptions in signal handling

4
The Plan The 3 thread model

• 3 threads
• (1) User Interface Thread
• API users mutator
• (1) Synchronous Event Thread
• Eg. Loop on waitpid()
• (1) Asynchronous Event Thread
• Eg. Loop on select()
• And a mailbox
• Hide internal threads by forcing non UI threads
  to mail events to the UI thread.
• Eg. ensure that user-supplied callbacks are only
  executed on the UI thread.

5
The Plan The 3 thread model
UI Thread
Mutatee 1
Synchronous Event Thread
waitpid()
poll()
Mutatee 2
Mailbox
Asynchronous Event Thread
select()
Mutatee N
6
But theres a problem (linux)

• Ptrace() may only be called by one thread,
  period.
• But we really want all threads to be able to
  issue ptrace() commands
• Processpause()
• Write/readDataSpace()
• Solution Add another thread
• Solely responsible for accepting and handling
  requests for calling ptrace()
• And sometimes waitpid() and fork()

7
The Plan The 4 thread model
Ptracer (DBI) Thread
UI Thread
Mutatee 1
Synchronous Event Thread
waitpid()
poll()
Mutatee 2
Mailbox
Asynchronous Event Thread
select()
Mutatee N
8
But theres another problem (windows)

• Restrictive OS level debugging API
• The thread that starts the mutatee process is the
  only thread that is allowed to call
  WaitNextDebugEvent()
• Solution add N threads !
• N number of mutatees
• This is actually a good idea
• One listener thread per process
• Elegant in terms of code modularity
• A nice simple concept a thread that starts a
  process and receives events from it until
  termination/detach
• Handles events as they occur.

9
The Plan The (n3) thread model
Ptracer (DBI) Thread
Synchronous Event Thread 1
Mutatee 1
UI Thread
waitpid() Poll()
Synchronous Event Thread 2
Mutatee 2
Mailbox
waitpid() Poll()
Synchronous Event Thread N
Mutatee N
waitpid() Poll()
Asynchronous Event Thread
select()
10
But theres another problem recursion

• Dyninst is full of recursive event patterns
• ie. An event that causes another event, and
  needs to wait for its completion
• Best solution will involve encapsulating this
  behavior into a natural structure
• Either that or re-architect many functional
  elements of dyninst
• And dont forget add more threads!
• Solution decouple event handling concept into
• Event generation
• One dedicated thread (per process)
• Event handling
• Multiple possible threads (thread pool)
• Number of threads depends on recursion depth
• This is a bit nasty
• Could probably be done using the mailboxs
  recursive callback features, but
• Theyre complicated too, and need more thought

11
The Plan The (2n3x) thread model
Ptracer (DBI) Thread
Signal Generator Thread 1
Mutatee 1
UI Thread
SH
SH
SH
SH
SH
SH
Signal Generator Thread 1
Mutatee 2
Mailbox
SH
SH
SH
SH
SH
SH
Signal Generator Thread 1
Mutatee N
SH
SH
SH
SH
SH
SH
Asynchronous Event Thread
select()
12
A simple example full stop

• UI Thread issues stopExecution()
• Sends SIGSTOP to mutatee
• Blocks inside waitForStop()

kill (SIGSTOP)
UI Thread
Mutatee
Wait for event evtProcessStop
Signal GeneratorThread
block
Event Gate
Signal Handler Thread
13
A simple example full stop

• Mutatee Stops
• Notifes parent process of SIGSTOP
• This is received by SignalGenerator, which is
  listening for mutatee events

kill (SIGSTOP)
UI Thread
Mutatee
Wait for event evtProcessStop
Signal GeneratorThread
block
SIGSTOP
Event Gate
Signal Handler Thread
14
A simple example full stop

• Signal Generator decodes SIGSTOP and assigns
  event
• Creates an Event with platform independent type
  evtProcessStop
• Assigns Event to available SignalHandler Thread

kill (SIGSTOP)
UI Thread
Mutatee
Wait for event evtProcessStop
Signal GeneratorThread
block
SIGSTOP
evtProcessStop
Event Gate
Signal Handler Thread
15
A simple example full stop

• Signal Hander handles evtProcessStop
• Platform independent code for handling
  evtProcessStop
• Signal waiting EventGates that evtProcessStop
  happened

kill (SIGSTOP)
UI Thread
Mutatee
Wait for event evtProcessStop
Signal GeneratorThread
block
SIGSTOP
evtProcessStop
Event Gate
Signal Handler Thread
Signal event evtProcessStop
16
A simple example full stop

• Event Gate receives target event
• Release UI Thread from block

kill (SIGSTOP)
UI Thread
Mutatee
Wait for event evtProcessStop
Signal GeneratorThread
block
SIGSTOP
evtProcessStop
Event Gate
Signal Handler Thread
resume
Signal event evtProcessStop
17
A (slightly) more complex example

• User registers a DynLibrary Callback
• When a library is loaded, insert some
  instrumentation

UI Thread
CB Manager
Mutatee
register callback
Signal GeneratorThread
Mailbox
SH - 0
SH - 1
18
A (slightly) more complex example

• Continue mutatee execution
• Wait for it to exit
• Mutatee runs

kill (SIGCONT)
UI Thread
CB Manager
Mutatee
register callback
Signal GeneratorThread
Continue execution
Mailbox
Block until exit event
Event Gate
SH - 0
SH - 1
19
A (slightly) more complex example

• Eventually, mutatee executes dlopen()
• Stops, sending SIGTRAP to mutator

kill (SIGCONT)
UI Thread
CB Manager
Mutatee
register callback
Signal GeneratorThread
SIGTRAP
dlopen()
Continue execution
Mailbox
Block until exit event
Event Gate
SH - 0
SH - 1
20
A (slightly) more complex example

• SignalGenerator Decodes SIGTRAP
• Analyzes runtime link maps, finds new library
• Sends an evtLoadLibrary Event to a Signal Handler

kill (SIGCONT)
UI Thread
CB Manager
Mutatee
register callback
Signal GeneratorThread
SIGTRAP
dlopen()
Continue execution
Mailbox
Block until exit event
evtLoadLibrary
Event Gate
SH - 0
SH - 1
21
A (slightly) more complex example

• SignalHandler handles evtLoadLibrary
• Adjusts Dyninst Internal State (platform indep)
• Obtains relevant Callback from CBManager
• Executes loadLibrary callback (zoom in to
  mailbox)

UI Thread
CB Manager
Mutatee
register callback
Signal GeneratorThread
SIGTRAP
dlopen()
Continue execution
Mailbox
Block until exit event
evtLoadLibrary
Event Gate
SH - 0
SH - 1
22
A (slightly) more complex example

• Callback Execution (zoom in)
• Registers instance of callback with Mailbox
  (target UI Thread)
• Signals Event Gates that evtLoadLibrary happened
• Blocks until completion of callback

UI Thread
SH - 0
Mailbox
Block until exit event
CB
Register CB
Event Gate
Signal evtLoadLibrary
Block until CB complete
23
A (slightly) more complex example

• UI Thread, signalled, wakes up
• Checks mailbox and finds callback with
  target_thread UIThread
• But the Event does not match target
• Executes callback (zoom back out )

UI Thread
SH - 0
Mailbox
Block until exit event
CB
Register CB
Event Gate
Signal evtLoadLibrary
Wake up, Execute Callbacks
Block until CB complete
CB
24
A (slightly) more complex example

• But the callback triggers another event
• InsertSnippet needs to allocate memory before
  insertion
• Issues inferiorMalloc RPC and blocks for its
  completion

UI Thread
CB Manager
Mutatee
register callback
Signal GeneratorThread
dlopen()
Continue execution
Mailbox
Block until exit event
Event Gate
Insert
Inferior Malloc RPC
SH - 0
SH - 1
block
block
resume
Event Gate
25
A (slightly) more complex example

• Mutatee issues malloc() and traps
• SignalGenerator receives SIGTRAP
• Discovers that we just completed a RPC

UI Thread
CB Manager
Mutatee
register callback
Signal GeneratorThread
dlopen()
Continue execution
Mailbox
SIGTRAP
malloc()
Block until exit event
Event Gate
Insert
Inferior Malloc RPC
SH - 0
SH - 1
block
block
resume
Event Gate
26
A (slightly) more complex example

• SignalGenerator sees that SH-0 is busy (blocked)
• Assigns evtRPCSignal event to SH-1

UI Thread
CB Manager
Mutatee
register callback
Signal GeneratorThread
dlopen()
Continue execution
Mailbox
SIGTRAP
malloc()
Block until exit event
evtRPCSignal
Event Gate
Insert
Inferior Malloc RPC
SH - 0
SH - 1
block
block
resume
Event Gate
27
A (slightly) more complex example

• SH -1 handles RPC completion event
• Manages Dyninst internal state (RPC Manager)
• Signals Event Gates that evtRPCSignal happened

UI Thread
CB Manager
Mutatee
register callback
Signal GeneratorThread
dlopen()
Continue execution
Mailbox
SIGTRAP
malloc()
Block until exit event
evtRPCSignal
Event Gate
Insert
Inferior Malloc RPC
SH - 0
SH - 1
block
block
resume
Event Gate
evtRPCSignal
28
A (slightly) more complex example

• EventGate waiting for evtRPCSignal wakes up
• Releases UI thread from RPC block

UI Thread
CB Manager
Mutatee
register callback
Signal GeneratorThread
dlopen()
Continue execution
Mailbox
SIGTRAP
malloc()
Block until exit event
evtRPCSignal
Event Gate
Insert
Inferior Malloc RPC
SH - 0
SH - 1
block
block
resume
Event Gate
evtRPCSignal
29
A (slightly) more complex example

• Callback finishes executing
• Inserts instrumentation in newly malloc()d space
• SH-0 is released from blocking state

UI Thread
CB Manager
Mutatee
register callback
Signal GeneratorThread
dlopen()
Continue execution
Mailbox
SIGTRAP
malloc()
Block until exit event
evtRPCSignal
Event Gate
Insert
Inferior Malloc RPC
SH - 0
SH - 1
block
resume
resume
Event Gate
evtRPCSignal
30
A (slightly) more complex example

• UI Thread resumes blocking for process exit
• Inserts instrumentation in newly malloc()d space
• SH-0 is released from blocking state

UI Thread
CB Manager
Mutatee
register callback
Signal GeneratorThread
dlopen()
Continue execution
Mailbox
malloc()
Block until exit event
Event Gate
Insert
Inferior Malloc RPC
SH - 0
SH - 1
block
resume
31
A (slightly) more complex example

• Finally, Mutatee Exits
• evtProcessExit propagates through Event Handling
• UI Thread is released from block

UI Thread
CB Manager
Mutatee
register callback
Signal GeneratorThread
dlopen()
Continue execution
Mailbox
malloc()
Block until exit event
evtProcessExit
Event Gate
SH - 0
SH - 1
resume
exit()
Signal event evtProcessExit
32
Thread Inflation aside

• Good things have emerged too
• Unified Dyninst event concept
• All event oriented subsystems operate on the same
  generic event concept
• Well, its really just a new c class
• Unified Dyninst event handling
• Event handling framework is now platform
  independent
• Dyninst takes uniform actions when specific
  events occur, across platforms
• More or less
• But more than before

33
Implications (future features)

• Finer grained Callback APIs
• We currently have callbacks for events like
• LoadLibrary
• Process Exit
• Thread Creation
• Now possible (and easy) to expose generic event
  callback to user
• Eg. Execute func() when mutatee receives signal S
• Eg. Execte func() when mutatee issues dlclose()

34
Implications (future features)

• Spin-off Debugger API library
• Separate out code for process control, signal
  handling, other debugger-like operations into
  shared library
• Requires solving some complex problems
• Eg. How to deal with RPCs
• How much gets spun off?
• This is a hard question to answer
• Related RPC and shared-object subsytems are
  heavily intertwined.
• Where to draw the line?