Intrusion Detection via Static Analysis

1
Intrusion Detection via Static Analysis

• David Wagner
• Drew Dean

2
Motivation

• New security problems are discovered every day
• Majority of security problems are buffer
  overflows (foreign code is executed) and not
  logic errors (native application code continues
  to execute)
• Try to solve this by monitoring applications for
  any abnormal behavior

3
Framework

• Assumption A compromised application cannot
  cause much harm unless it interacts with the
  underlying operating system, and those
  interactions may be readily monitored.
• In most of the cases, the only way to interact
  with an OS is via system calls.
• Solution Monitor applications system call trace
  for any unexpected interaction with an OS.

4
Trivial Model

• Create the set of system calls that the
  application can ever make
• If a system call outside of the allowed set is
  executed, terminate the application
• Pluses simple, easy to implement, efficient
• Minuses Fails to detect many attacks (i.e. ones
  that use only system calls from the allowed set),
  too coarse-grained (certain system calls can
  cause a lot of damage, i.e. open() )

5
Callgraph Model

• Improves the trivial model by reintroducing the
  ordering of the system calls
• Represent the system call trace as a
  non-deterministic finite automaton (NDFA)
• Monitor the application by simulating the
  operation of the NDFA on the observed system call
  trace
• Pluses more precise than the trivial model, does
  not introduce any false positive alarms
• Minuses harder to implement, not efficient,
  includes impossible paths due to function call
  treatment and presents certain risks due to
  non-determinism of the model

6
Abstract Stack Model

• Improves the callgraph model by eliminating
  impossible paths (by characterizing more
  precisely the set of possible syscall traces)
• Represent the system call trace as a
  non-deterministic pushdown automaton (NDPDA)
• Monitor the application by simulating the
  operation of the NDPDA and comparing the
  application call stack with a list of all valid
  stacks
• Pluses eliminates impossible paths
• Minuses much harder to monitor the application
  efficiently

7
Digraph Model

• Combines some of the advantages of the callgraph
  model in a simpler formulation
• Model consists of a list of possible k-sequences
  of consecutive system calls (k2 for simplicity)
• Monitor the application by checking the executed
  system calls vs. a precomputed list of the
  allowed k-sequences
• Pluses much more efficient than the callgraph or
  abstract stack models
• Minuses less precise than the callgraph or
  abstract stack models

8
Implementation Issues

• Non-standard control
• Function pointers
• Signals
• Setjmp()
• Other modeling challenges
• Libraries
• Dynamic linking
• Threads

9
Optimizations

• Irrelevant systems calls
• Not monitoring harmless but frequently executed
  system calls such as brk() can greatly improve
  the performance
• System call arguments
• Monitoring the arguments at runtime improves both
  precision and performance

10
Evaluation

• Performance
• Precise callgraph and abstract stack models
  introduce too much overhead
• Mimicry attacks
• Require high precision models to detect (poor
  performance)

11
Unaddressed issues

• Applying static analysis to binaries when source
  code is not available
• Explaining how runtime monitoring agent works and
  how it can be used to improve the security and
  the model precision (run as a separate SMT thread
  with 0 overhead?)

12
Questions

• What are the ramifications of the callgraph model
  non-determinism?
• What are the risks of having k2 in the
  k-sequences model?