J'B' Speed School of Engineering

1
Methods for Detecting Kernel Rootkits Dissertatio
n Defense Doug Wampler
J.B. Speed School of Engineering University of
Louisville Louisville, KY 40292 502.852.6100
2
Outline

• Introduction
• Background Literature Survey
• Contributions
• Approach
• Studies
• Experimental Results

3
Outline, continued

• Conclusions
• Future Research Directions
• Contributions

4
Introduction

• Rootkits are stealthy malicious software
• Deployed after break-in
• Conceal activity dispose of evidence
• Provide backdoors
• A primary method for maintaining access
• How modify operating system calls

5
Introduction, continued

• Importance of rootkit detection
• Operating system can be made to lie to system
  administrator
• Plethora of sophisticated backdoors to maintain
  access
• Increasing rootkit sophistication

6
Background

• Binary rootkits
• Library rootkits
• Linux Kernel Module (LKM) rootkits
• Runtime kernel patching (RKP) rootkits
• Future atypical rootkits
• Contemporary LKM patching rootkits

7
Background, continued

• Simple detection methods
• Ad hoc rootkit detection
• Host based intrusion detection
• Rootkit detection applications
• Hardware based rootkit detection
• Anomaly Signature Based

8
LKM Rootkit Functionality
9
RKP Rootkit Functionality
10
Contributions

• A framework for real time kernel analysis
• A greatly reduced requirement for a priori
  knowledge for use in rootkit detection
• Key observations on the general and statistical
  properties of several classes of operating
  systems
• A more rigorous, more formal, more mathematically
  based approach to rootkit detection

11
Research Goals

• Reduce need for specific a priori knowledge in
  rootkit detection
• Provide more formality/rigor than existing
  rootkit detection methods

12
Approach

• Outlier analysis used for rootkit detection
• Understand underlying distribution of system call
  memory addresses in OS
• Understand location of outliers
• Need a measure of outlyingness
• Anderson-Darling goodness of fit score

13
Approach, continued

• Memory model determines probable location of
  outliers
• System call addresses determine underlying
  distribution of memory addresses
• Location of outliers and the underlying
  distribution of memory addresses constitute
  reduced a priori knowledge

14
General Preparations

• Obtained rootkits on the internet from various
  sources
• Quality, functionality, and availability vary
  wildly
• Six rootkits evaluated

15
Operating System Preparations

• Kernel modifications
• Stripped kernels
• Support for some tools used in analysis

16
Operating System Preparations, Continued

• Data Gathering Toolset
• The GNU Debugger, gdb
• Perl

17
Operating System Preparations, continued

• Analysis Toolset
• Minitab

18
Preliminary Work

• Recompile kernel need non-stripped version
• Install data gathering tools gcc, gdb, and Perl
• Download, compile, and install rootkits
• Obtain memory addresses of system calls using
  data gathering tools

19
Preliminary Work

• Analyze memory addresses data using Minitab
• Typically, remove outliers from right side of
  distribution until goodness of fit score for
  underlying distribution returns to close to
  original score
• These outliers (system calls) are targets
  affected by rootkit

20
Studies

• Detecting system call table modification attacks
  using general distribution models
• Detecting system call table modification attacks
  using normal distribution models
• Detecting system call target modification attacks
  using general distribution models

21
Studies, continued

• Detecting system call target modification attacks
  using normal distribution models
• Detecting Windows rootkits

22
Experimental Results

• Detecting system call table modification attacks
  using general distribution models

23
Experimental Results, continued
Results from Knark 2.4.3 Experiment
24
Experimental Results, continued

• Detecting system call table modification attacks
  using normal distribution models
• Successful with Linux 2.4.27 kernel on Intel
  architecture
• Unsuccessful with other OS/architecture pairs
• Suffers from false positives, natural outliers,
  no way to exclude them

25
Experimental Results, continued

• Detecting system call target modification attacks
  using general distribution models
• Much larger amount of data why?
• Approach does not scale well
• Detection is possible, but with a very narrow
  margin for error
• Disassembly doesnt work well for SPARC

26
Experimental Results, continued

• Detecting system call target modification attacks
  using normal distribution models
• There are natural outliers in this dataset
• Data happens to be multivariate. How?
• The order in which system calls are loaded into
  memory matters
• A two phase detection technique may be used

27
Experimental Results, continued

• First discordancy test simple z-score test
• Second discordancy test ratio based on order of
  appearance multiplied by z-score
• Individuals must pass both discordancy tests to
  be considered outliers

28
Experimental Results, continued
29
Experimental Results, continued

• Windows rootkit detection
• SSDT similar to system call table
• Best fitting distribution 3-parameter Weibull
• However, many fit extremely well including
  Normal
• Webwatcher rootkit

30
Experimental Results, continued

• Weibull 3.029 before infection, 8454.206 after
  infection, 2.920 when cleaned.
• Before infection, all individuals have z-scores lt
  3. After infection, all three attack locations
  have z-scores gt 9.
• Both general and normal distribution models are
  effective in detecting this attack.

31
Conclusions

• Detecting system call table modification attacks
  using general distribution models is effective,
  multi platform, and results in complete detection
• Detecting system call table modification attacks
  using normal distribution models yields false
  positives, but may someday prove useful

32
Conclusions, continued

• Detecting system call target modification attacks
  using general distribution models does not scale
  well, has a narrow margin for error.
• Detecting system call target modification attacks
  using normal distribution models is effective,
  taking into account new observations about the
  kernel

33
Conclusions, continued

• Surprisingly, Windows rootkit detection using
  these models is effective as well.
• Windows 2000 SSDT is very normally distributed.
  This is unexpected.
• Both normal and general distribution models are
  successful against SSDT attacks on Windows.

34
Future Research Directions

• Virtualized rootkits Blue Pill on Windows using
  AMD Pacifica hardware virtualization.
• Test more Windows rootkits, including
  virtualization rootkit(s).
• Generalizing a priori knowledge
• Database of operating system properties.
• Compilation options

35
Contributions

• A framework for real time kernel analysis
• A greatly reduced requirement for a priori
  knowledge for use in rootkit detection
• Key observations on the general and statistical
  properties of several classes of operating
  systems
• A more rigorous, more formal, more mathematically
  based approach to rootkit detection

36
Questions?