Enforcing Security Policies Using Machine Virtualization Techniques

1
Enforcing Security Policies Using Machine
Virtualization Techniques
Prashanth P. Bungale Division of Engineering and
Applied Sciences (DEAS)Harvard
University CS253r Project
2
Reference Monitor

• Observes execution of a target system
• Halts the system whenever upon violation of a
  security policy
• Must be protected from subversion by target system

3
Security Automaton

• Involves
• A set of states
• An input alphabet
• A transition relation
• Transition relation defines a next state for the
  automaton given its current state and an input
  symbol
• If no transition can be made, then the security
  automaton rejects its input
• Security automata can be regarded as defining
  reference monitors
• Input alphabet ? Events seen by reference monitor
• Transition relation encodes a security policy

4
SASI(Erlingsson and Schneider 2)

• A technique that modifies object code for a
  target system to include reference monitors that
  enforce security policies described by security
  automata
• Encode the automaton state transitions
  immediately before each security-relevant
  instruction, halting the code whenever the
  security automaton rejects an input
• Ultimately, the authors found that
• Even though their mechanism offered high
  flexibility
• It suffered from extreme performance overheads
  because of the enormous amount of extra code
  inserted by their mechanism

5
Machine Virtualization Techniques

• Machine-level dynamic binary translation and
  protection from subversion by target
• Have been used in wide range of applications
• Debugging
• Performance analysis
• Dynamic optimization
• Full machine virtualization
• Program Shepherding

6
Program Shepherding (Kiriansky et al. 2002 3)

• Method for Monitoring control flow transfers
  during program execution to enforce security
  policies
• Three techniques
• Restricts execution privileges on the basis of
  code origins
• Restricts control transfers based on instruction
  class, source, and target
• Guarantees that sandboxing checks placed around
  any type of program operation will never be
  bypassed
• Minimal or no performance penalties

7
VDebug(Bungale et al. 2004 1)

• x86 to x86 dynamic translation system
• Designed to achieve least complexity rather than
  maximal optimization
• Resulting implementation exposes overheads
  intrinsic to all binary translation mechanisms
• Yields performance comparable to more aggressive
  translation strategies that use intermediate code
  gen.

8
Our Approach SA-VDebug

• Use machine virtualization techniques to enforce
  Schneiders security automata 4
• Batch state transitions into transition summaries
  for an entire basic-block (instead of once every
  instruction)
• Maintain cache of already-computed transition
  summaries and re-use them
• Use techniques provided by underlying VDebug
  system to protect against subversion by target
  system

9
Key Advantages

• General framework for extremely flexible
  enforcement of security policies
• Reuses protection mechanisms already implemented
  by underlying virtualization system
• While, at the same time, incurring reasonable
  performance overhead at run-time
• Can easily be enhanced to pin-point the trace
  that resulted in a security automaton violation
• Provides the security guarantee without making
  any assumptions about the target program

10
Key Limitations

• Currently supports only the subset of security
  automata not involving input symbols that depend
  on state outside the security automaton
• Since VDebug is not an optimizing dynamic
  translator
• Multi-threaded programs not supported yet
• Security automaton not enforced across Interrupts
  / Exceptions / Signals
• Only enforced within program execution context

Note None of these are inherent limitations of
our approach
11
Example

PROLOGUE
BASIC-BLOCK
12
Optimizations

• Ignore the self-transitions
• If there are no transitions
• Omit the basic-block prologue
• If it is guaranteed that there cannot be a
  transition into the bad state
• Omit the bound instruction
• If a transition into the bad state is guaranteed
  in all cases
• Omit the basic-block itself and directly halt

13
Performance Evaluation
14
Performance Evaluation (contd)
15
Observations

• Translator overhead is negligible Translatee
  overhead is the significant factor
• Size (i.e., number of states, transitions, etc.)
  of the automaton doesn't really matter
• But, the number and the (popularity) class of
  input symbols does matter
• The optimizations that we introduced did matter

16
References

• Prashanth P. Bungale, Swaroop Sridhar and
  Jonathan S. Shapiro. Low-Complexity Dynamic
  Translation in VDebug, Masters Project Report ,
  The Johns Hopkins University, Baltimore, MD
  21218, USA, May 2004.
• Ulfar Erlingsson and Fred B. Schneider. SASI
  Enforcement of Security Policies A
  Retrospective, Proceedings of the New Security
  Paradigm Workshop, Ontario, Canada, September
  1999, pp. 87-95.
• Vladimir Kiriansky, Derek Bruening, and Saman
  Amarasinghe. Secure Execution via Program
  Shepherding, Proceedings of the 11th USENIX
  Security Symposium, San Francisco, CA, August
  2002.