Backtracking Algorithmic Complexity Attacks Against a NIDS

1
Backtracking Algorithmic Complexity Attacks
Against a NIDS

• Randy Smith, Cristian Estan, Somesh Jha
• University of WisconsinMadison

2
Algorithmic Complexity Attacks

• Vulnerable algorithm algorithm whose worst case
  differs from typical case. The larger the
  difference, the more vulnerable the algorithm.
• Examples

Algorithm Average Worst
Quicksort O(n log n) O(n2)
Hash lookup constant O(n)
3
Algorithmic Complexity Attacks

• Algorithmic Complexity Attack an attacker
  induces worst-case behavior in a vulnerable
  algorithm.
• Common observable effect is denial of service.
• Crosby and Wallach induced worst-case behavior
  in hash function implementations.
• Algorithms are now part of the attack surface
  (Crosby and Wallach, 2003)

4
Are NIDS vulnerable?

• NIDS and IPS are ubiquitous, but
• Do they contain vulnerable algorithms? Can they
  be exploited?
• YES! Only need 1 packet every 3 seconds.

5
Evading a NIDS

• Attackers Goal Evade NIDS
• Two attack vectors in an evasion attempt
• 1stalg. complexity attack targeting the NIDS
• 2ndtrue attack targeting the network
• Effect of an algorithmic complexity attack
• (NIDS) Packets enter network unexamined
• (fail-closed IPS) Packets are dropped

6
Main results

• In Snort, vulnerability in rule-matching
• worst-case vs. typical case 6 orders of
  magnitude.
• Backtracking Attack
• Easily exploitable through packet payloads
• Improved rule-matching algorithm limits running
  time differences to within 1 order of magnitude.

7
Outline

• Snort rule matching
• Inducing backtracking attacks
• Countermeasures
• Measurement results
• Conclusion

8
Snort Rule Matching
9
Snort Rule Matching
alert tcp EXT_NET any -gt HOME_NET 99
(msgAudioPlayer jukebox exploit
contentfmt //P1
pcre/(mp3ogg)/,relative //P2
contentplayer //P3
pcre/.exe.com/,relative //P4
contentoverflow,relative //P5
sid5678)
Rule matches!






fmtacc playerdefault fmtmp3 rate14kbps
playercmd.exe?overflow_at_!
10
Matching the packet
P1
alert tcp EXT_NET any -gt HOME_NET 99
(msgAudioPlayer jukebox exploit
contentfmt //P1
pcre/(mp3ogg)/,relative //P2
contentplayer //P3
pcre/.exe.com/,relative //P4
contentoverflow,relative //P5
sid5678)






fmtacc playerdefault fmtmp3 rate14kbps
playercmd.exe?overflow_at_!
11
Inducing Backtracking attacks

• P1,P2,P3,P4 match in 3 positions each
• P5 never matches

alert tcp EXT_NET any -gt HOME_NET 99
(msgReelAudio jukebox exploit
contentfmt //P1
pcre/(mp3ogg)/,relative //P2
contentplayer //P3
pcre/.exe.com/,relative //P4
contentoverflow,relative //P5
sid5678)

• Leads to excessive packet traversals!

fmtmp3fmtmp3fmtmp3playerplayerplayer.exe.exe
.exe
fmtacc playerdefault fmtmp3 rate14kbps
playercmd.exe?overflow_at_!
12
Matching the malicious packet
P1
P2










fmtmp3fmtmp3fmtmp3playerplayerplayer.exe.exe
.exe
13
Are real rules vulnerable?
Rule number Processing (s/GB) Slowdown Slowdown
Rule number Processing (s/GB) Same proto All traffic
3682 (SMTP) 30,933,874 232,936X 1,501,644X
2611 (Oracle) 6,220,768 56,296X 301,979X
1382 (IRC) 1,956,858 134,031X 94,993X
2403 (NetBIOS) 357,777 490X 17,368X
1755 (IMAP) 89,181 444X 4,329X
14
Safer backtracking

• Memoization maintain a table of subproblem
  answers never evaluate a predicate twice at
  the same starting payload offset

alert tcp EXT_NET any -gt HOME_NET 99
(msgAudioPlayer jukebox exploit
contentfmt //P1
pcre/(mp3ogg)/,relative //P2
contentplayer //P3
pcre/.exe.com/,relative //P4
contentoverflow,relative //P5
sid5678)

• Identify constrained predicate sequences
• Monotone memoization dont re-evaluate monotone
  predicates that have been evaluated at lower
  offsets

15
Reductions in processing cost
P1
P2
P3
P4
P4
P5
P5
P5
P5
P5
P5
fmtmp3fmtmp3fmtmp3playerplayerplayer.exe.exe
.exe
16
Outline

• Snort rule matching
• Inducing backtracking attacks
• Protecting against backtracking attacks
• Measurement results
• Conclusion

17
Measurement results
Rule number Slowdown factor w.r.t. same protocol Slowdown factor w.r.t. same protocol
Rule number Before w/ Memo
3682 (SMTP) 232,936X 0.95X
2611 (Oracle) 56,296X 1.57X
1382 (IRC) 134,031X 6.00X
2403 (NetBIOS) 490X 0.17X
1755 (IMAP) 444X 0.46X
18
Live experiment topology
Background Traffic
AC Attack
True Attack
19
Live experiment

• Background Traffic _at_ 10Mbps
• AC Attack
• Targets Snort SMTP rule 3682
• Directed at sendmail server
• True Attack NIMDA
• 300 exploit attempts, sent 1 byte per second.
• New exploit started every second.

20
Live experiment results
Attack Description Exploits Detected Required Rate (kbps)
Control (No attack) 300/300 --
2 packets every 60 s. 220/300 0.4
1 packet every 5 s. 4/300 2.4
1 packet every 3 s. 0/300 4.0
20 packets initially 0/300 0.8
1 packet every 3 s. 300/300 --
20 packets initially 300/300 --
21
Conclusions

• NIDS operation is complex. Many opportunities
  for vulnerable algorithms.
• In Snort, rule-matching is vulnerable and can be
  exploited by an attacker.
• Memoization, along with other semantics-preserving
  operations, significantly reduces vulnerability.
• Other vulnerable algoritms exist.

22
Backtracking Algorithmic Complexity Attacks
Against a NIDS

• Thank you.