Normalizing Metamorphic Malware Using Term Rewriting

1
Normalizing Metamorphic Malware Using Term
Rewriting

• A. Walenstein, R. Mathur, M. R. Chouchane, and A.
  Lakhotia
• Software Research Laboratory
• The University of Louisiana at Lafayette
• Sixth IEEE International Workshop on Source Code
  Analysis and Manipulation
• 27th-29th September 2006
• Philadelphia, PA, USA

2
About this Work

• The core of the paper's work formed the Master's
  thesis of Rachit Mathur.
• He has since graduated and is now working at
  McAfee.

3
Malware Identification
Malware are malicious programs such as viruses,
worms, and Trojans.
Virus
Form - A
Antivirus scanners use extracted patterns, or
signatures to identify known malware.
Anti-Virus
Signature
Signature
4
Metamorphic Malware
Virus
Virus
Virus
M
M
Form - A
Form - B
Form - C

• Metamorphic malware change as they propagate
• They create multiple variants of themselves

5
Metamorphic Malware Challenge
Virus
Virus
Virus
M
M
Form - A
Form - B
Form - C
Using different signatures for most variants
cannot scale.
Anti-Virus
Signature
Too many signatures challenge the AV Scanner
6
Proposed approach normalizer
N
N
N
Normalizer Construction ProblemReduce the
number of signatures needed to detect all
variants.
Virus
NormalForm
Anti-Virus
Signature
7
Inspiration undo transformations
push ecx mov ecx, ebp 10 mov ecx, ebp push
eax add eax, 2342 mov eax, 33 add ecx, eax pop
eax mov eax, esi push eax mov esi, ecx push
edx xor edx, 778f mov edx, 34 sub esi, edx pop
edx mov esi-2, eax pop esi pop ecx
push ecx mov ecx, ebp push eax mov eax, 33 add
ecx, eax pop eax push esi mov esi, ecx push
edx mov edx, 34 sub esi, edx pop edx mov esi -
2, eax pop esi pop ecx
push ecx mov ecx,ebp add ecx,33 push esi mov
esi,ecx sub esi,34 mov esi-2,eax pop esi pop ecx
push ecx mov ecx,ebp add ecx,33 mov
ecx-36,eax pop ecx
mov ebp - 3, eax
8
Problem 1 naïve undo is naïve
push 0x04 mov eax, 0x04
mov eax, 0x04 push eax mov eax, 0x04
mov eax, 0x04 push eax
push 0x04
9
Problem 2 conditional transformations
mov edi, 0x04
push ecx mov ecx, 0x04 mov edi, ecx
pop ecx
unconditional
push eax mov eax, 0x04
push eax
eax not live
mov eax, 0x04 push eax
push 0x04
eax not live
Q how to reorient rules while guaranteeing
termination?
10
Term rewriting approach

• Adopted term-rewriting framework
• Model the metamorphic engine as TRS
• Modify it to create normalizing rule set and
  engine
• apply completion procedure, which reorients rules
• Can guarantee needed properties (termination,
  confluence)

11
Completion procedure sketch
push 0x04 mov eax, 0x04
mov eax, 0x04 push eax mov eax, 0x04
Critical Pairs
mov eax, 0x04 push eax
push 0x04
12
Completion procedure sketch
push 0x04 mov eax, 0x04
mov eax, 0x04 push eax mov eax, 0x04
Reorient
New Rule
mov eax, 0x04 push eax
push 0x04
13
What to do when completion procedure fails?

• Successful completion guarantees a unique normal
  form for all variants
• The perfect normalizer
• but
• Completion procedure may not terminate!
• Number of rules in the normalizer may be too high
  to be practical
• Does not take into account conditions
• ? Need alternative scheme

14
Priority Scheme

• Simple
• No Need for costly/imprecise condition evaluation
• Improved through Ad-hoc completion

If possible, Apply a rule from NC
Y
yes
no
NU Unconditional rules NC Conditional rules
HALT
15
Question condition checking required?

• Conditional rules require checking of conditions
• Can be expensive, or impossible
• What is the practical penalty of incorrectly
  checking conditions?
• e.g., ignoring conditions completely?

16
Case Study W32.Evol

• Virus can generate huge number of variants
• Tested the normalization schemes on 26 variants
  over 6 generations
• Manually Extracted rules used by W32.Evol
• 55 rules
• 84 overlaps
• TXL implementations
• Ordinary and priority-based evaluation

17
Results
18
Contributions

• Applications for assisting malware scanners
• Initial exploration of possibility of perfect
  normalization
• Indications of usefulness of heuristic
  alternatives (priority scheme and ignoring
  conditions)

19
Future Work

• Expanded scope and empirical study
• Extensions for semantics-non-preserving
  metamorphic engines?
• Localized normalization using term rewriting
• M. Chouchane and A. Lakhotia Using Engine
  Signature to Detect Metamorphic Malware,
  Workshop on Rapid Malcode, Fairfax, VA, Nov. 2006
  (to appear)
• More at www.cacs.louisiana.edu/labs/SRL

20

• Alumni
• Nitin Jyoti, Avertlabs
• Aditya Kapoor, McAfee
• Erik Uday Kumar, Authentium
• Rachit Mathur,
• McAfee
• Moinuddin Mohammed,MicrosoftPrashant Pathak,
  Symantec
• Prabhat Singh,SymantecFunded by Louisiana
  Governors IT Initiative

Software Research Lab Center for Advanced
Computer Studies University of Louisiana at
Lafayette Arun Lakhotia Director Andrew
Walenstein Research Scientist Michael
Venable Software Engineer and Alumnus Ph.D.
StudentsMohamed R. ChouchaneMd Enamul Karim
M.S. Students Christopher Thompson Matthew
Hayes