GILK: A Dynamic Instrumentation Tool for the Linux Kernel

1
GILK A Dynamic Instrumentation Tool for the
Linux Kernel

• David J. Pearce, Paul H.J. Kelly, Tony Field and
  Uli Harder
• d.pearce_at_doc.ic.ac.uk

Based on paper presented at 12th International
Conference on Computer Performance Evaluation
(TOOLS02), London, April 2002 (Springer LNCS
2324) See http//www.doc.ic.ac.uk/djp1/gilk.html
2
Motivation

• Instrumentation Do programmers really know what
  is going on?
• Need to visualise program behaviour
• Build, debug/validate performance models
• Understand and fix performance problems
• Exploration users need to explore system
  behaviour interactively

• Auditing How can we monitor what is going on?
• Relies on understanding of program behaviour
• Minimise impact on system
• Need flexible programmable tool
• Operating at lowest possible level to circumvent
  countermeasures

3
Introduction instrumenting the Linux kernel

• GILK is a dynamic instrumentation tool for the
  Linux Kernel
• Run-time patching of the OS kernels code
• On the fly no need to reboot, no need to restart
  servers
• Instruments can be added and removed via a
  simple GUI
• Or programmatically adding and removing an
  instrument takes milliseconds
• No need to recompile the kernel
• No need to interrupt the web server etc

4
Measure, audit, intercept

• Instruments
• We provide a simple set of instruments for
  performance measurement, selectable from the GUI
• User-definable instruments
• API allows instruments to be defined by the user
  in C
• Intercept any kernel function
• Log parameters, message contents etc
• Instruments can also check parameters against
  security policy, enforce rules, etc

5
GILK the GUI

• GUI starts with Linux kernels symbol table
• Lists all kernel functions
• System calls
• Network protocol stack
• File system
• Device drivers
• Inter-process communication
• Memory management

6
GILK An Introduction

• Click on a kernel function
• Dialogue offers to attach instrument to function
• Various instruments
• Before, after or both
• Or browse control flow graph to instrument
  internal blocks

7
GILK An Introduction

• An instrumentation experiment runs
  automatically
• Each instrument has its own launch and splashdown
  time
• So that performance impact of instrumentation can
  be staggered
• All instruments automatically removed on
  completion/exit
• Instrument data is logged to experiment output
  file

8
Example validate TCPdump

• Uli Harder is researching statistical
  self-similarity properties of network traffic
• Using standard TCPDUMP tool to obtain network
  timing information
• Produced fair amount of data
• But occasional zero or negative inter-arrival
  time was spotted
• Are results so far wasted?
• Is TCPDUMP useable or not?
• Uli used GILK for comparison with TCPDUMP
• GILK provides high resolution packet arrival
  times
• TCPDUMP does generate erroneous results
• But previous results valid statistical
  properties unchanged

9
What We Did

• Browse Linux source code and call graph
• Identify high-level socket interface
• Add rdtsc timer instrument
• Is socket interface timing what we want?
• Find ethernet device driver
• Instrument it to time actual packet transfer

Linux Network Stack

BSD Socket Layer
sock_recvmsg
RDTSC

Ethernet Driver
RDTSC
speedo_rx
10
Where was the Success?

• Ulis not a kernel programmer
• Didnt know beforehand what was needed
• Time wasted trying different instrumentation
• GILK is fast, and thats important
• Instruments can be moved, adjusted, relaunched
  interactively while the system is under the test
  load
• Allows us to try many different instrumentations
• Fits exploratory nature of debugging and
  modelling system behaviour

11
How it Works

• GILK uses code splicing technology
• Allows instrumentation of active kernel
• Instrumentation performed at machine code level
• Works on unmodified kernel image
• No patching, restarting, recompiling etc
• Instruments written in C
• Implemented as Kernel Modules
• Access to full language features
• Access to kernel functionality
• System Requirements
• Only Intel x86 Architecture supported
• Kernel versions 2.0 2.2 supported
• Extension to 2.4 is trivial

12
Instruction Boundaries
Before
After
splice
unused
instrumentation
Relocated sequence
13
Conclusions

• GILK quickly instruments Linux Kernel
• Runtime code splicing useful for instrumentation
• Some development work needed ease of use,
  visualisation of instrument output
• Intriguing potential.
• Automatic instrumentation bottleneck
  identification strategies
• Security applications
• Intercept, monitor for unusual behaviour
• Characterise processes which initiate unexpected
  behaviour
• Insert authorisation policies
• check client process id, application image id, IP
  addresses, message contents etc
• Future work
• Simplify GUI, develop instrumentation strategies,
  visualization
• Automate extraction of dynamic call graph, search
  for bottlenecks
• Explore security policy ideas context-dependent
  quality-of-service?
• Later linux kernels (2.4 should be easy), SMPs,
  non-Intel platforms (for Solaris see Paradyn
  project at Wisconsin)
• User-Space programs
• Our Research group is now doing this for Java