Stealthy Malware Detection Through VMM-based

1
Stealthy Malware Detection Through VMM-based
Out-of-the-Box Semantic View Reconstruction
CCS07, Alexandria, VA, Oct 29 Nov 2, 2007
Xuxian Jiang, Xinyuan Wang, Dongyan Xu
George Mason University Purdue University
2
Motivation

• Internet malware remains a top threat
• Malware viruses, worms, rootkits, spyware, bots

3
Motivation

• Recent Trend on Rootkits

Viruses/worms/bots, PUPs,
700 growth
400 growth
Q1 of 2005
Source McAfee Avert Lab Report (April 2006)
4
Existing Defenses (e.g., Anti-Virus Software)

• Running inside the monitored system
• Advantages
• They can see everything (e.g., files,
  processes,)
• Disadvantages
• Once compromised by advanced stealthy malware,
  they may not see anything!

VirusScan
Firefox
IE

OS Kernel
5
Existing Defenses

• Key observation
• Both anti-virus software and vulnerable software
  are running inside the same system
• Hard to guarantee tamper-resistance
• Solution Out-of-the-box defense

?
Virtual Machine Monitor (VMM)
6
The Semantic-Gap Challenge
Semantic Gap
Guest OS
Virtual Machine Monitor (e.g., VMware, Xen, QEMU)

• What we can observe?
• Low-level states
• Memory pages, disk blocks,
• Low-level events
• Privileged instructions,
• Interrupts, I/O access,

• What we want to observe?
• High-level states w/ semantic info.
• Files, processes,
• high-level events w/ semantic info.
• System calls, context switches,

7
Main Contribution

• VMwatcher A systematic approach to bridge the
  semantic gap
• Reconstructing semantic objects and events from
  low-level VMM observations

Capability I Out-of-the-box execution of
commodity anti-malware software
Firefox
IE

VMwatcher
OS Kernel
Capability II View comparison-based stealthy
malware detection
Virtual Machine Monitor (VMM)
8
VMwatcher Bridging the Semantic Gap

• Step 1 Procuring low-level VM states and events
• Disk blocks, memory pages, registers,
• Traps, interrupts,
• Step 2 Reconstructing high-level semantic view
• Files, directories, processes, and kernel
  modules,
• System calls, context switches,

VM Introspection
Guest View Casting
9
Step 1 VM Introspection
Raw VMM Observations Virtual Machines (VMs) Virtual Machines (VMs) Virtual Machines (VMs) Virtual Machines (VMs)
Raw VMM Observations




VM Disk Image
VM Physical Memory
VM Hardware State (e.g., registers)
VM-related low-level events (e.g., interrupts)
VMware Academic Program
10
Step 2 Guest View Casting
Cross-view
Semantic Gap
Guest OS
Virtual Machine Monitor (VMM)
Key observation The guest OS already contains
all necessary semantic definitions of data
structures as well as functionalities to
construct the semantic view
11
Guest View Casting
Raw VMM Observations Casted Guest Functions Data Structures Reconstructed Semantic View




Device drivers, file system drivers
VM Disk Image
Memory translation, task_struct, mm_struct
VM Physical Memory
VM Hardware State (e.g., registers)
CR3, MSR_SYSENTER_CS, MSR_SYSENTER_EIP/ESP
Event-specific arguments
Syscalls, Context switches, ....
Event semantics
VM-related low-level events (e.g., interrupts)
Demo clip (3.5mins) http//www.ise.gmu.edu/xjian
g/
12
Guest View Casting on Memory State (Linux)
Process List
Process Memory Layout
13
Guest Memory Addressing

• Traditional memory addressing
• Given a VA, MMU translates VA to PA
• OSes used to map with known PA
• Linux VA 0xc0000000 PA 0x0
• Windows VA 0x80000000 PA 0x0
• VM complicates the translation
• Guest virtual -gt guest physical
• Guest physical -gt host physical

Emulated Address Translation
VM Introspection
Reverse Address Translation
14
Evaluation

• Effectiveness
• Cross-view malware detection
• Exp. I Cross-view detection on volatile state
• Exp. II Cross-view detection on persistent state
• Exp. III Cross-view detection on both volatile
  and persistent state
• Out-of-the-box execution of commodity
  anti-malware software
• Exp. IV Symantec AntiVirus
• Exp. V Windows Defender
• Performance
• Difference between internal scanning external
  scanning

15
Exp. I Cross-view detection on volatile memory
state

• Experiment Setup
• Guest VM Windows XP (SP2)
• Windows Fu Rootkit
• Host OS Scientific Linux 4.4
• VMM VMware Server 1.0.1

Diff
VMwatcher view
Inside-the-box view
16
Exp. II Cross-view detection on persistent disk
state

• Experiment Setup
• Guest VM A Redhat 7.2-based honeypot
• Linux SHv4 rootkit
• Host OS Windows XP (SP2)
• VMM VMware Server 1.0.1

Diff
VMwatcher view
Inside-the-box view
17
(No Transcript)
18
Experiment (IV)

• Experiment Setup
• Both guest OS and host OS run Windows XP (SP2)
• VMM VMware Server 1.0.1
• Running Symantec AntiVirus Twice
• Outside
• Inside

Hacker Defender
NTRootkit
19
Internal Scanning Result
Diff
External Scanning Result
20
Performance

• Internal scanning time vs. external scanning time

Internal scanning takes longer to complete !
21
Related Work

• Enhancing security with virtualization
  (LivewireGarfinkel03, IntroVirtJoshi05,
  HyperSpectorKourai05)
• Focusing on targeted attacks with specialized
  IDSes
• Cross-view detection (Strider GhostBusterWang05,
  RootkitRevealer/ Blacklight/IceSword/)
• Either destroying the volatile state or obtaining
  two internal views
• Secure monitors
• CoPilotPetroni04, TerraGarfinkel03,
  sHypeSailer05, SecVisorPerrig07,TRANGO,

22
Conclusions

• VMwatcher A systematic approach that bridges
  the semantic gap and enables two unique malware
  detection capabilities
• Cross-view malware detection
• Out-of-the-box execution of commodity
  anti-malware software

23
Thank you!
For more information Email xjiang_at_ise.gmu.edu
URL http//www.ise.gmu.edu/xjiang