Software Security Monitors: Theory

1
Software Security MonitorsTheory Practice

• David Walker
• Princeton University
• (joint work with Lujo Bauer and Jay Ligatti)

2
Extensible Systems
Web Browser
Mail Script
Plug-in
Mail Server
Active Router
Servlet
Web Server
protocol
3
Securing Extensible Systems

• Compile-time/link-time security
• policies memory safety, type safety
• tools type systems, proof-carrying code, model
  checking
• Run-time security
• policies access control, resource bounds
• tools access control lists, capabilities, stack
  inspection

4
Run-time Security

• In general, run-time security properties are
  enforced by program monitors
• Abstractly, a program monitor is a process that
  runs in parallel with an untrusted application
• monitors examine application actions
• decide to allow/disallow application actions
• may terminate an application, log application
  actions, reinterpret application actions, etc.
• monitors detect, prevent, and recover from
  erroneous or malicious behavior at run time
• generalizes specific enforcement mechanisms such
  as access control lists, stack inspection, etc.

5
Securing Extensible Systems

• Many questions
• Our application requires property X. Can we
  enforce it precisely or will we have to get by
  with an approximation?
• How do we write down our policy succinctly and
  unambiguously?
• What specific mechanism will we need to enforce
  our property?
• How do we implement the mechanism?

6
Talk Outline

• Theory
• What is a software security monitor?
• What is a security policy and what does it mean
  to enforce one?
• What policies can software security monitors
  enforce?
• Language design
• Programming simple policies
• Programming complex policies
• Summary, related work and conclusions

7
What is a program monitor?

• Monitors analyze transform untrusted
  application actions

Monitor
Input Stream
Output Stream
a3
a1
a2
a2
a4
a2
a2
a1


Application generates actions to be input into
monitor
Machine executes actions output by monitor
8
Possible Monitor Actions

• Accept the action
• Halt the application
• Suppress (skip) the operation
• Insert some computation
• Some combination of these

9
Formalizing security monitors

• Program monitors gt formal automata that
  transform a stream of program actions
• Given a set of possible program actions A
• Monitors are deterministic state machines (Q,
  q0, T) where
• Q state set
• q0 start state
• T transition function

10
Operational Semantics

• Single step (determined by T)
• (Sin, q) ? (Sin, q)
• Multi-step (reflexive, transitive closure of T)
• (Sin, q) ? (Sin, q)
• Output sequence is observable
• Input sequences are not observable

So
So
11
A Hierarchy of Security Monitors
We classify monitors based on their
transformational abilities (ie based on T).
Insert Suppress OK
Halt Truncation Suppression Insertion Edit
?
?
?
?
?
?
?
?
?
?
?
?
12
An Example E-Banana.com

• Set of application actions A take(n),
  // take n bananas pay(n), // pay for n
  bananas browse, // browse for bananas
  receipt // commit
• Edit Automaton

take(n)
pay(n)
pn
browse
browse
pay(n)
take(n)
start
tpn
init
tn
receipt
pay(n)take(n)receipt
13
Edit Automata

• Definition (Q,q0,T)
• where T (t,e,i)
• State transition function t
• t action x state ? state
• Emission function e
• e action x state ? ,-
• Insertion function i
• i action x state ? action sequence x state

14
Edit Automata

• Operational Semantics
• (S, q) ? (S, q)if SaS and t(a,q)q and
  e(a,q)
• (S, q) ? (S, q)if SaS and t(a,q)q and
  e(a,q) -
• (S, q) ? (S, q)if SaS and i(a,q)(Sins, q)
• (S, q) ? (empty, q)otherwise

a
(E-Accept)
(E-Suppress)
Sins
(E-Insert)
(E-Halt)
15
Talk Outline

• Theory
• What is a software security monitor?
• What is a security policy and what does it mean
  to enforce one?
• What policies can software security monitors
  enforce?
• Language design
• Programming simple policies
• Programming complex policies
• Summary, related work and conclusions

16
Security Policies

• A Security Policy is a predicate P over sequences
  of actions.
• Example Policies
• In any program execution, bananas taken equal
  bananas paid for
• Access control, resource bounds policies are
  properties
• Non-policies (for our purposes)
• Cryptographic uniformity property The set of
  all possible outputs of the cryptographic key
  generation algorithm forms a uniform distribution
  over the integers
• Information-flow policies

17
What does it mean to enforce a policy?

• Principle of Soundness
• All observable outputs obey the policy ?
  sequences Sin . ? state q . ? sequence So
• 1. (Sin, q0) ? (empty, q)
• 2. P(So)
• Principle of Transparency
• Semantics of executions that already obey policy
  must be preserved 3. P(Sin)?? (Sin ??So)

So
18
Some Useful Equivalences

• Remove/Insert unnecessary actions
• fclose(f)fclose(f)?? fclose(f)
• Replace a sequence with equivalent actions
• socket(S)send(S,m)?? socketSend(S,m)
• Permute independent actions
• fopen(f)fopen(g)?? fopen(g)fopen(f)
• Necessary properties
• reflexive, symmetic transitive
• S?? S ?? P(S)?? P(S)

19
E-Banana.com

• Equivalence Rules

1) (browse S) ? S 2) (S1 take(n) pay(n)
S2) ? (S1 pay(n) take(n) S2)
20
Conservative Enforcement

• Enforcer satisfies Soundness but not necessarily
  Transparency
• ? properties P . (? sequence S . P(S)) ? P can
  be conservatively enforced

Conservative
21
Effective Enforcement

• Enforcer satisfies Soundness and Transparency
• Valid sequences can be altered

Effective
Conservative
22
Precise Enforcement

• Motivation
• In practice, some operations cannot be delayed
• Definition
• Enforcer satisfies Soundness and Transparency
• Enforcer must output actions in lock-step with
  application

Precise
Effective
Conservative
23
Talk Outline

• Theory
• What is a software security monitor?
• What is a security policy and what does it mean
  to enforce one?
• What policies can software security monitors
  enforce?
• Language design
• Programming simple policies
• Programming complex policies
• Summary, related work and conclusions

24
What properties can be enforced?

• The enforceable properties depend upon
• the definition of enforcement (conservative,
  effective, precise)
• the class of automaton (truncation, suppression,
  insertion, edit)
• the space of possible input programs
• static program analysis (type systems
  proof-carrying code) constrain program execution
• if the monitor can assume certain bad
  executions do not occur, it can enforce more
  properties

25
Effective Enforcement

• An E-Banana.com policy
• Our edit automaton is an effective enforcer
• It satisfies Soundness
• It satisfies Transparency
• Proofs are by induction over the possible inputs
• Less powerful automata (truncation, suppression
  and insertion) cannot enforce the E-Banana
  property
• Proof by contradiction shows either Soundness or
  Transparency will be violated

browse ((take(n)pay(n) pay(n)take(n))
receipt)
26
A Simple Theorem

• Theorem Any decideable predicate P on
  executions is a property that can be effectively
  enforced by some edit automaton
• Proof construct a transactional edit automaton
  that suppresses and logs program actions when
  P(S) and commits (outputs) when P(S)

27
Effectively Enforceable Properties
Editing Properties
Insertion Properties
Suppression Properties
Trunc. Prop.
28
Talk Outline

• Theory
• What is a software security monitor?
• What is a security policy and what does it mean
  to enforce one?
• What policies can software security monitors
  enforce?
• Language design
• Programming simple policies
• Programming complex policies
• Summary, related work and conclusions

29
Polymer, the Language

• Polymer
• A domain-specific language for programming
  security monitors (edit automata)
• Java a couple of simple extensions
• simple policy definitions containing
• a set of security-relevant actions
• security state
• decision procedure that produces security
  suggestions (halt, suppress action, insert
  action, etc)
• complex policy definitions involving
• higher-order policy combinators

30
Securing Untrusted Applications
untrusted code
describes security-relevant program points
Java application
policy interface
instrumented application
separately compiled from policy
contains hooks to call monitor
31
Securing Untrusted Applications
Java application
implements dynamic security policy
policy interface
policy implementation
instrumented application
combines application and policy
secure application
32
Talk Outline

• Theory
• What is a software security monitor?
• What is a security policy?
• What does it mean to enforce a policy?
• What policies can software security monitors
  enforce?
• Language design
• Programming simple policies
• Programming complex policies
• Summary, related work and conclusions

33
A Simple Polymer Policy
new policy definition extends policy class
class limitFiles extends Policy private int
openFiles 0 private int maxOpen 0
limitFiles(int max) maxOpen max
....
private policy state
policy constructor
34
A Simple Polymer Policy Continued
class limitFiles extends Policy private int
openFiles ... private int maxOpen ...
public ActionPattern actions new
ActionPattern ltFile
fileOpen(String)gt, ltvoid fileClose(File)gt
....
set of policy- relevant methods
35
A Simple Polymer Policy Continued
class limitFiles extends Policy private int
openFiles ... private int maxOpen ...
public ActionPattern actions ...
Suggestion before(Action a) aswitch (a)
case fileOpen(String s) if
(openFiles lt maxOpen) return
Suggestion.OK() else
return Suggestion.Halt() case
fileClose(File f) ...
policy behavior
36
A Simple Polymer Policy Continued
class limitFiles extends Policy private int
openFiles ... private int maxOpen ...
public ActionPattern actions ...
Suggestion before(Action a) aswitch (a)
case fileOpen(String s) if
(openFiles lt maxOpen) return
Suggestion.OK() else
return Suggestion.Halt() case
fileClose(File f) ...
37
A Simple Polymer Policy Continued
class limitFiles extends Policy public
ActionPattern actions ... private int
openFiles ... private int maxOpen ...
Suggestion before(Action a) aswitch (a)
case fileOpen(String s) if
(openFiles lt maxOpen) return
Suggestion.OK() else
return Suggestion.Halt() case
fileClose(File f) ...
38
Talk Outline

• Theory
• What is a software security monitor?
• What is a security policy?
• What does it mean to enforce a policy?
• What policies can software security monitors
  enforce?
• Language design
• Programming simple policies
• Programming complex policies
• Summary, related work and conclusions

39
Complex Monitors

• Combine simple policies defined over a variety of
  different resources
• eg sample applet policy
• file system access control
• bounds on bytes written and number of files
  opened
• restricted network access
• no network access after local file is read
• communication with applet source only

40
Policy Combinators

• Programmers may write parameterized policy
  combinators
• And, Or, Forall, Exists, Chinese wall,...

P1
P2
AndPolicy
?
s2
s1
s
41
Policy Combinators

• class AndPolicy extends Policy
• private Policy p1
• private Policy p2
• AndPolicy(Policy pol1, Policy pol2)
• p1 pol1
• p2 pol2
• ...

first-class policies
42
Policy Combinators

• class AndPolicy extends Policy
• ...
• Suggestion before(Action a)
• Suggestion s1 p1.before(a)
• Suggestion s2 p2.before(a)
• if (s1.isOK() s2.isOK())
• return Suggestion.OK()
• else ...

using suggestions
system interprets suggestions at the top level
43
Talk Outline

• Theory
• What is a software security monitor?
• What is a security policy?
• What does it mean to enforce a policy?
• What policies can software security monitors
  enforce?
• Language design
• Programming simple policies
• Programming complex policies
• Summary, related work and conclusions

44
Future Work

• Theory
• infinite sequences gt coinductive proof
  techniques
• resource-bounded programs monitors
• time, space and randomness
• Practice
• complete Polymer 1.0 (end of summer)
• Polymer evaluation
• next up transactional policies

45
Related Work

• Enforceable security policies
• Schneider 00 HMS 02
• Monitoring languages
• Naccio ET 99 Poet and Pslang ES 99, ES 00
  others
• New polymer features first-class policies
  policy combinators, suggestions, abstract
  actions, formal semantics
• Aspect-oriented Programming
• AspectJ HyperJ
• New polymer features as above
• With Dan Dantas, we are developing AspectML

46
Summary

• A general framework for formal reasoning about
  security monitors
• defined a hierarchy of security monitors
• gave meaning to the word enforceable
• developed rigorous proofs concerning enforceable
  properties
• Polymer A programming language for composing
  security monitors
• techniques for modular monitor design
  composition
• formal semantics as an extension of FeatherWeight
  Java

47
Conclusions

• Technology for securing extensible systems is in
  high demand
• Software security monitors are one part of the
  solution
• For more information, see
• Edit Automata Enforcement Mechanisms for
  Run-time Security Policies. IJIS 2003.
• Types and effects for non-interfering program
  monitors.  ISSS 2002 LNCS 2609.
• More Enforceable Security Policies. FCS 2002.
• www.cs.princeton.edu/sip/projects/polymer/

48
End
49
Realistic Monitors

• Protect complex system interfaces
• interfaces replicate functionality in many
  different places
• method parameters communicate information in
  different forms
• eg Java file system interface
• 9 different methods to open files
• 4 different methods to close files
• filename strings, file objects, self used to
  identify files

50
Abstract Action Definitions
java.lang.io
FileReader(String fileName) FileReader(File
file) RandomAccessFile(...) ... FileReader.clos
e() RandomAccessFile.close() ...
fileOpen(String n) fileClose()
51
Abstract Action Definitions
class fileOpen extends ActionSig boolean
canMatch(Action a) aswitch (a)
case FileReader(_) return true case
RandomAccessFile () return true ...
String parameter1(Action a) ....

52
Abstract Action Pattern Matching
class limitFiles extends Policy ...
Suggestion step(Action a) aswitch (a)
case fileOpen(String s) ...
case fileClose() ...
fileOpen.parameter1(a)
fileOpen.canMatch(a)
53
Taxonomy of Precisely Enforceable Properties
54
Secure Application
Untrusted application
Host System (Java)
Program Monitor Definition
Polymer language extensions
Java core
55
Policy Architecture Simple Policies
system interface
Simple Policy Def.
Host System (Java)
Polymer language extensions
Java core
56
Policy Architecture Abstract Actions
abstract system interface
Host System (Java)
Simple Policy Def.
Abstract Action Def.
Polymer language extensions
concrete system interface
Java core
57
Policy ArchitectureComplex Policies
Complex, System-specific Policy
abstract system interface
Simple Policy Def.
Policy Comb. Def.
Abstract Action Def.
Host System (Java)
Polymer language extensions
concrete system interface
Java core