A Binary Rewriting Defense Against Stack-based Buffer Overflow Attacks

1
A Binary Rewriting Defense Against Stack-based
Buffer Overflow Attacks

• Manish Prasad, Tzi-cker Chiueh
• SUNY Stony Brook

2
Roadmap

• Binary Translation
• Buffer Overflow and Return Address Defense (RAD)
• Static BT and RAD
• Experimental Results
• Conclusion

3
Binary Translation

• Motivation
• To do things without access to source
• Traditional Applications
• Legacy Code Migration (HP Aries, UQBT)
• Program Optimization (Etch, IBM BOA)
• Approaches (and their limitations)
• Static Lacks transparency, accuracy
• Dynamic Less efficient

4
Binary Translation and Software Security

• Legacy applications from outside vendor

5
Binary Translation and Software Security
Application A Vendor X

• NO SOURCE CODE

CERT Advisory Application A Vendor
X VULNERABILITY !!!
6
Binary Translation and Software Security
CERT Advisory Application A Vendor
X VULNERABILITY !!!
7
Has it been done yet ??

• Dynamic Translation
• DynamoRIO MIT
• LibVerify Bell-Labs
• Hardware Support UIUC

8
Why Static Binary Translation (SBT) ?

• Inferring legacy program behavior
• Application-specific security policies
• Use Static Analysis if you can
• Butler Lampson, Hints for Computer System Design

9
Goals

• How far can we go with pure static BT ?
• Buffer Overflow protection
• Widespread
• Simple yet covers most SBT issues
• Foundation for general Win32/PE instrumentation
  framework

10
Contributions

• Comprehensive Treatment of Static BT
• Why and where it fails ?
• Prototype implementation incorporating static BT
  state-of-the-art
• High disassembly precision
• Exhaustive experiments with several commercial
  grade Windows applications

11
Buffer Overflow Attack
Buffer Overflow and RAD
0
Stack Growth
4G
Stack Evolution on a Function Call
12
Buffer Overflow Attack
Buffer Overflow and RAD
0

Stack Growth
Function Arguments
caller
4G
Stack Evolution on a Function Call
13
Buffer Overflow Attack
Buffer Overflow and RAD
0
Stack Growth
CALL
Return Address
Function Arguments
4G
Stack Evolution on a Function Call
14
Buffer Overflow Attack
Buffer Overflow and RAD
0
Other Local Variables
Local Buffer
callee
Local Variables
Old Frame Pointer
Stack Growth
Return Address
Function Arguments
4G
Stack Evolution on a Function Call
15
Buffer Overflow Attack
Buffer Overflow and RAD
0
Other Local Variables
target
Local Buffer
Local Variables
Old Frame Pointer
Stack Growth
Return Address
Function Arguments
4G
Stack Evolution on a Function Call
16
Buffer Overflow Attack
Buffer Overflow and RAD
0
A addr of exploit Exp Exploit Code
Attack Buffer
Other Local Variables
Local Buffer
A
Local Variables
A
Old Frame Pointer
A
Stack Growth
Return Address
A
Function Arguments
Exp
4G
Unbounded Buffer Copy
17
Buffer Overflow Attack
Buffer Overflow and RAD
0
Other Local Variables
Address of Exploit
Address of Exploit
Address of Exploit
Stack Growth
Address of Exploit
Exploit Code
4G
Overflow Aftermath
18
Return Address Defense (RAD)
Buffer Overflow and RAD
Stack
Return Address Repository (RAR)
Ret Addr Copy
19
Return Address Defense (RAD)
Buffer Overflow and RAD
Stack
Return Address Repository
Local Variable Space
callee
Return Addr

Ret Addr Copy
Arguments
20
Return Address Defense (RAD)
Buffer Overflow and RAD
Stack
Return Address Repository
Address Of Exploit Code
Unsafe Buffer Copy
Ret addr corrupted

Ret Addr Copy
21
Return Address Defense (RAD)
Buffer Overflow and RAD
Stack
Return Address Repository
Ret addr corrupted

Ret Addr Copy
22
Return Address Defense (RAD)
Buffer Overflow and RAD
Stack
Return Address Repository
Attack !!
Ret addr corrupted
Ret Addr Copy
23
Static BT RAD Central Issues
Static BT and RAD

• Disassembly
• Code Instrumentation

Binary File
Disassembler
Instrumentation
24
Disassembly
Static BT and RAD

• Core component for static analysis of binaries
• Principal Approaches
• Linear Sweep
• Recursive Traversal

25
Disassembly
Static BT and RAD

• Core component for static analysis of binaries
• Principal Approaches
• Linear Sweep
• Recursive Traversal

valid inst
First Byte
valid inst
valid inst
valid inst
invalid inst
26
Disassembly
Static BT and RAD

• Core component for static analysis of binaries
• Principal Approaches
• Linear Sweep
• Recursive Traversal

Entry Point
CALL fn
fn
RET
27
Disassembly
Static BT and RAD

• Core component for static analysis of binaries
• Principal Approaches
• Linear Sweep
• Recursive Traversal

RET
Entry Point
CALL fn
fn
RET
28
Disassembly
Static BT and RAD

• Core component for static analysis of binaries
• Principal Approaches
• Recursive Traversal
• Linear Sweep

RET
Entry Point
CALL fn
fn
RET
29
Disassembly Impediments
Static BT and RAD

• Code/Data distinction
• Variable x86 instruction size
• Indirect Branches
• Functions without explicit CALL
• PIC

30
Disassembly Impediments
Static BT and RAD
0x0F 0x85 0xC0 0x0F 0x85 .
0x0F 0x85 0xC0 0x0F 0x85
0x0F
0x0F 0x85 ..
JNE offset
0x0F // data TEST eax, eax JNE offset

• Code/Data distinction
• Variable x86 instruction size
• Indirect Branches
• Functions without explicit CALL
• PIC

31
Disassembly Impediments
Static BT and RAD
0x0F 0x85 0xC0 0x0F 0x85 .
0x0F 0x85 0xC0 0x0F 0x85
0x0F
0x0F 0x85 ..
JNE offset
0x0F // data TEST eax, eax JNE offset

• Code/Data distinction
• Variable x86 instruction size
• Indirect Branches
• Functions without explicit CALL
• PIC

32
Disassembly Impediments
Static BT and RAD

• Code/Data distinction
• Variable x86 instruction size
• Indirect Branches
• Functions without explicit CALL
• PIC

33
Disassembler Approach
Static BT and RAD

• Recursive Traversal
• Accurately code/data identification
• Linear Sweep
• Reach areas not covered by recursive traversal
• Compiler-independent heuristics
• Recover from errors

34
Disassembly Accuracy
Static BT and RAD
35
Code Instrumentation
Static BT and RAD

• Add desired functionality
• Preserve original program semantics

36
Code Instrumentation RAD
Static BT and RAD

• Add buffer overflow protection
• Preserve original program semantics

37
What to instrument ?
Static BT and RAD

• Function boundary identification
• Prologue and epilogue must be both
  instrumented/uninstrumented
• False alarms
• Pattern matching (for interesting functions)
• Stack frame allocation and deallocation

38
What to instrument ?
Prototype
Binary File
Disassembler
Core Binary Rewrite Engine
RAD
If interesting function
Each Instruction
39
Inserting Checking Code
Static BT and RAD
Prologue
JMP (save ret addr)
Epilogue
JMP (check ret addr)

• JMP at prologue and epilogue to RAD code
• Replace 5 byte worth instructions
• Should not disturb branch targets
• Stack frame allocation (at prologue) gt 5 bytes
• Stack frame deallocation possible in 2 4 bytes
• Return address check in INT 3 handler

40
Inserting Checking Code
Static BT and RAD
ADD ESP, x // 3-6 bytes POP EBP // 1 byte RET //
1 byte
PUSH EBP // 1 byte MOV EBP, ESP // 2 byte SUB
ESP, x // 3-6 bytes

• JMP at prologue and epilogue to RAD code
• Replace 5 byte worth instructions
• Should not disturb branch targets
• Stack frame allocation (at prologue) gt 5 bytes
• Stack frame deallocation possible in 2 4 bytes
• Return address check in INT 3 handler

MOV ESP, EBP // 2 bytes POP EBP // 1 byte RET //
1 byte
LEAVE // 1 byte RET // 1 byte
INT 3 // 1 byte
41
INT 3 Statistics
Static BT and RAD
42
Experimental Evaluation
Experimental Evaluation

• Goals of experiments
• Effect on program correctness
• Resilience to buffer overflow attacks
• Performance and space overhead

43
Space Overhead
Experimental Evaluation
Micro-Benchmark
Macro-Benchmark
44
Execution Time Overhead
Experimental Evaluation
Overhead Execution Time with RAD Execution
Time without RAD Execution Time without RAD
45
Resilience to Buffer Overflow Attack
Experimental Evaluation

• Windows Help (Winhlp32.exe)
• Windows NT 4.0 with Service Pack 4.0
• Content file (.CNT) with long heading string
• Published exploit code

Resists Attack !!
46
Known Limitations

• Disassembly Limitations
• Hand-crafted Assembly
• RAD Limitations
• Multi-Threaded Applications
• Self-Modifying Code

47
Known Limitations

• Disassembly Limitations
• Indirect branches
• Hand-crafted Assembly
• RAD Limitations
• Multi-Threaded Applications
• Self-Modifying Code

48
Indirect Branches and Control Flow Analysis
49
Known Limitations

• Disassembly Limitations
• Hand-crafted Assembly
• Inter-procedural jumps, multiple function entry
  points
• RAD Limitations
• Multi-Threaded Applications
• Self-Modifying Code

50
Known Limitations

• Disassembly Limitations
• Hand-crafted Assembly
• RAD Limitations
• Memory Pointer corruption
• Multi-Threaded Applications
• Self-Modifying Code

51
Known Limitations

• Disassembly Limitations
• Hand-crafted Assembly
• RAD Limitations
• Multi-Threaded Applications
• Per-thread RAR needed
• Self-Modifying Code

52
Known Limitations

• Disassembly Limitations
• Hand-crafted Assembly
• RAD Limitations
• Multi-Threaded Applications
• Self-Modifying Code
• Beyond static analysis

53
Conclusions

• First study to implement Static BT
  state-of-the-art into a working system
• High disassembly accuracy
• Demonstrated effectiveness on commercial grade
  Windows applications
• Qualified success as a security tool
• Basis for a general Win32/PE Instrumentation
  framework

54
Looking Ahead

• Support for DLLs
• Dynamic translation to ameliorate static BT
  deficiencies
• Automatic application-specific security policy
  generation
• Copy and Tamper resistant software

55
Project Page

• http//www.ecsl.cs.sunysb.edu/brew.htm
• Disassembler Sources
• http//www.ecsl.cs.sunysb.edu/disassembler.tgz

56
Questions ??
57
and Im looking for work !!!
58
Disassembler Passes
Static BT and RAD
0x400100
DATA


CALL 0x40011C
JMP 0x400110

0x400110
RET
0x40011C
MOV EAX, 0x40012E PUSH 0x400100
CALL EAX
RET
0x40012E
RET
MOV EAX, ECX
CALL 0X400140
DATA
DATA
59
(No Transcript)
60
Disassembler Passes
Static BT and RAD
0x400100
DATA
Entry Point


CALL 0x40011C
JMP 0x400110

0x400110
Inter-Procedural Call Graph
RET
0x40011C
MOV EAX, 0x40012E PUSH 0x400100
CALL EAX
RET
0x40012E
RET
MOV EAX, ECX
Pass 1 Recursive Traversal
CALL 0X400140
DATA
DATA
61
Disassembler Passes
Static BT and RAD
0x400100
DATA
Entry Point


Function Control Flow Graph
CALL 0x40011C
JMP 0x400110

0x400110
Inter-Procedural Call Graph
RET
0x40011C
Backward Edge
MOV EAX, 0x40012E PUSH 0x400100
CALL EAX
RET
0x40012E
RET
MOV EAX, ECX
Pass 1 Recursive Traversal
CALL 0X400140
DATA
Function Body
DATA
62
Disassembler Passes
Static BT and RAD
0x400100
DATA
Entry Point


CALL 0x40011C
JMP 0x400110

0x400110
RET
0x40011C
MOV EAX, 0x40012E PUSH 0x400100
CALL EAX
RET
0x40012E
RET
MOV EAX, ECX
Pass 1 Recursive Traversal
CALL 0X400140
DATA
Function Body
DATA
63
Disassembler Passes
Static BT and RAD
0x400100
DATA
Pass 2 Linear Sweep


CALL 0x40011C
JMP 0x400110

0x400110
RET
0x40011C
MOV EAX, 0x40012E PUSH 0x400100
CALL EAX
RET
0x40012E
RET
Legal inst. Bytes as code
MOV EAX, ECX
CALL 0X400140
DATA
Function Body
DATA
64
Disassembler Passes
Static BT and RAD
0x400100
DATA
Pass 3 Error Recovery Heuristics


CALL 0x40011C
JMP 0x400110

0x400110
RET
0x40011C
MOV EAX, 0x40012E PUSH 0x400100
CALL EAX
RET
0x40012E
RET
MOV EAX, ECX
Code sequence not ending with JMP/RET gt not code
CALL 0X400140
DATA
Function Body
DATA
65
Execution Time Micro-benchmark
Experimental Evaluation
Penalty Additional RAD Overhead Original Run
Time
66
WattProbe

• Linux-based experimental platform for low-power
  computing
• Power vs. Performance measurements
• Fine Grained
• High Resolution
• Across Multiple Resources
• Power characterization of interactive
  applications
• Application-specific low power modes

67
Caching Infrastructure for Stackable File Systems
(FAST 03 WIP)

• Cache coherence support
• Transparent to native and stackable file systems
• Minimal changes to the page cache code
• Hooks into a cache manager
• Prototype Linux implementation

68
Communication and Membership Management for Linux
Clusters

• Low-latency link-level semi-reliable transport
  layer (Springer LNCS, HiPC 02)
• No hardware support
• Membership Management
• Cluster-wide consistent membership view
• Scalable failure detection

69
Thanks
70
(No Transcript)
71
(No Transcript)
72
Disassembly Limitations
Prototype

• False Negatives (Functions Missed)
• False Positives (Falsely Identified Functions)

73
Disassembly Limitations
Prototype

• False Negatives (Functions Missed)
• Partly/fully misidentified as data
• Identified fully as code
• False Positives (Falsely Identified Functions

Prologue
Data
Prologue
Instrumented
Function Body
Security Hole
RAR Overflow
Epilogue
Uninstrumented
Identified as Data
74
Disassembly Limitations
Prototype

• False Negatives (Functions Missed)
• Partly/fully misidentified as data
• Identified fully as code
• False Positives (Falsely Identified Functions

Prologue
Instrumented
Fn1
RAR overflow
Identified as Data
Fn2
Security Hole
Epilogue
Uninstrumented
75
Disassembly Limitations
Prototype

• False Negatives (Functions Missed)
• Partly/fully misidentified as data (pop up)
• Identified fully as code
• False Positives (Falsely Identified Functions)

Data
Identified as Code
Identified as Code
Security Hole
Interesting Prolog
Function Body
76
Disassembly Limitations
Prototype

• False Negatives (Functions Missed)
• False Positives (Falsely Identified Functions)
• Code following data/unconditional branch
• Missed attacks
• RAR overflow

Indirect Jump
Data
Jump Target
Function Entry Pt
77
Indirect Branches and Control Flow Analysis
Prototype
78
RAD Limitations
Prototype

• Attacks resisted
• Return Address
• Frame Pointer
• Does not protect
• Memory pointer attacks
• Function pointer corruption
• Import table overwrites

79
Multi-Threaded Applications
Prototype

• RAR shared between threads
• Possible Solution
• Thread Information Block (TIB) access using FS
  register
• Thread Local Storage (TLS) slots in TIB
• Separate RAR for each thread
• Thread-specific RAR addresses in TLS slots

80
Hand-Crafted Assembly Quirks
Prototype
No interesting prolog
Fn1 entered Return address not saved
Fn 1
JNE label
False Alarm
No interesting epilog
Interesting prolog
Fn 2
label
Exit from Fn2 Return address check made
Interesting epilog
Inter-procedural jump in MSAccess
81
Software Architecture Flow of Control
Prototype
Binary File
Instrumentation
Disassembler
Core Binary Rewrite Engine
RAD
PE-specific component
New PE Section
Set up RAD mine zones