Intrusion Detection via Static Analysis

1
Intrusion Detection via Static Analysis

• David Wagner, Drew Dean

IEEE Security Privacy 2001
2
Introduction

• Anomaly detection
• Develop a model of normal program behavior
• Flag any deviations from the model as potential
  intrusions
• Approach
• Constrains the system call trace of a programs
  execution to be consistent with the programs
  source code

3
Related Work

• IDS methodologies
• Generate a model of a programs or systems
  behavior from observing its behavior on known
  inputs
• Require the generation of a rule base
• Current IDS problem
• From statistical data, whether the system is
  valid or not
• The false alarm rate of systems is a major
  problem in practice
• Challenge
• Find models that reduce the false alarm rate

4
Framework

• Assumption A compromised application cannot
  cause much harm unless it interacts with the
  underlying operating system, and those
  interactions may be readily monitored.
• In most of the cases, the only way to interact
  with an OS is via system calls.
• Solution Monitor applications system call trace
  for any unexpected interaction with an OS.

5
Trivial Model

• Create the set of system calls that the
  application can ever make
• If a system call outside of the allowed set is
  executed, terminate the application
• Pluses simple, easy to implement, efficient
• Minuses Fails to detect many attacks (i.e. ones
  that use only system calls from the allowed set),
  too coarse-grained (certain system calls can
  cause a lot of damage, i.e. open() )

6
Callgraph Model

• Improves the trivial model by reintroducing the
  ordering of the system calls
• Represent the system call trace as a
  non-deterministic finite automaton (NDFA)
• Monitor the application by simulating the
  operation of the NDFA on the observed system call
  trace
• Pluses more precise than the trivial model, does
  not introduce any false positive alarms
• Minuses harder to implement, not efficient,
  includes impossible paths due to function call
  treatment and presents certain risks due to
  non-determinism of the model

7
Callgraph Model
8
Abstract Stack Model

• Improves the callgraph model by eliminating
  impossible paths (by characterizing more
  precisely the set of possible syscall traces)
• Represent the system call trace as a
  non-deterministic pushdown automaton (NDPDA)
• Monitor the application by simulating the
  operation of the NDPDA and comparing the
  application call stack with a list of all valid
  stacks
• Pluses eliminates impossible paths
• Minuses much harder to monitor the application
  efficiently

9
Abstract Stack Model
10
Digraph Model

• Combines some of the advantages of the callgraph
  model in a simpler formulation
• Model consists of a list of possible k-sequences
  of consecutive system calls (k2 for simplicity)
• Monitor the application by checking the executed
  system calls vs. a precomputed list of the
  allowed k-sequences
• Pluses much more efficient than the callgraph or
  abstract stack models
• Minuses less precise than the callgraph or
  abstract stack models

11
Implementation Issues

• Non-standard control
• Function pointers
• Signals
• setjmp(), longjmp()
• Other modeling challenges
• Libraries
• Dynamic linking
• Threads

12
Optimizations

• Irrelevant systems calls
• Not monitoring harmless but frequently executed
  system calls such as brk() can greatly improve
  the performance
• System call arguments
• Monitoring the arguments at runtime improves both
  precision and performance

13
Evaluation

• Performance
• Precise callgraph and abstract stack models
  introduce too much overhead
• Mimicry attacks
• Require high precision models to detect (poor
  performance)

14
(No Transcript)
15
(No Transcript)
16
Conclusion

• New models for characterizing program behavior
• Static analysis lets you constrain program
  behavior to be consistent with the source code