Immunix

1
Immunix DefconDefending Vulnerable Code From
Intense Attack

• Crispin Cowan, Ph.D
• Seth Arnold, Steve Beattie, Chris Wright
• WireX
• and
• John Viega, Secure Software

2
Talk Outline

• About WireX and Immunix
• Secure Linux Systems
• The Defcon Challenge
• Defend vulnerable code against massive attack
• Technology Transfer
• Commercial products built on this technology

3
Software Security

• Software security is really simple
• Make sure you only run perfect software
• Uh-oh -)
• Intrusion Prevention
• Systems that detect attack attempts in real time,
  and reject them
• When bugs occur, they are not exploitable
• Attacker cannot exploit the bug to gain
  unintended privileges

4
Immunix Security Technologies

• Shipping
• StackGuard stops buffer overflows
• FormatGuard stops printf format bugs
• RaceGuard stops temp file races
• SubDomain contains vulnerable programs

5
StackGuard

• Stack Smashing Problem
• Weak bounds checking on inputs in C programs
• Attacker overflows input buffer, corrupting
  adjacent state to gain control of the program
• Most common target function return address on
  the stack

• StackGuard
• C compiler enhancement
• Ornaments call stack to detect corruption
• Very low performance overhead
• WireX has been shipping fully StackGuardd system
  since 1999

6
FormatGuard

• Format String Problem
• Sudden discovery in June 2000
• Vulnerability in WU-FTPD
• Followed by hundreds of similar vulnerabilities
• Basis arcane n printf format string directive
• Treat corresponding argument as an int
• Write back number of items formatted so far
• Problem programs that pass un-filtered user
  input strings direct to printf

• FormatGuard
• Similar to StackGuard
• compiled defense against printf format string
  vulnerabilities
• CPP macro
• Counts arguments at the call site at compile time
• Compares that number to the format string
  presented at run time

7
RaceGuard

• Temporary File Race Problem
• Portable procedure for temporary file creation is
  non-atomic
• If attacker gets in the middle, can redirect
  temporary file creation by privileged programs to
  corrupt the system

• RaceGuard
• Kernel enhancement to detect race attacks mid-way
  through
• Abstract method detect changes between stat()
  and open() accesses to the same file name

8
Containment

• If your software is vulnerable anyway, you need
  to contain it so that it runs with the least
  privilege necessary to perform designated
  function
• Chroot basic isolation for vulnerable programs
• Immunix SubDomain flexible confinement for
  vulnerable programs

9
Containment WithChroot

• Change root makes some subdirectory appear to
  be the root (/) directory for the calling
  process and its children
• Available as both a shell command and a system
  call
• Effect chrootd programs cannot affect anything
  outside the chroot jail
• Limits impact of bugs in program, e.g. chroot BIND

• Benefits
• Standard Comes with most UNIXs
• Compatible several current programs have been
  modified to work within a chroot jail
• Fast no performance degradation
• Limitations
• Work must move copies of everything a jailed
  program needs into the jail
• Isolation jailed program cannot interact at all
  with the rest of the system

10
Containment WithImmunix SubDomain

• Part of Immunix Kernel Extension
• Specify the list of files that a SubDomained
  program may access
• Effect SubDomained programs cannot affect
  anything they dont explicitly need access to
• Limits impact of bugs in program, e.g. SubDomain
  CGI scripts

• Benefits
• Flexible SubDomained programs can have
  controlled interaction with the rest of the
  system
• Compatible SubDomain can confine binary programs
  without modifications
• Fast 1 or less performance overhead
• Limitations
• Work must specify shape of SubDomain

11
Containing PHF

• PHF infamous vulnerable CGI script
• legitimate function database lookup of user
  information
• sloppy parsing of CGI input
• can get PHF to start an xterm on an arbitrary
  display
• To SubDomain PHF
• Specify all the files that PHF needs
• Effect
• access to all other files is denied
• Including xterm -)

• Place this file in /etc/subdomain.conf/phf
• /home/httpd/cgi-bin/phf
• /bin/sh x ,
• /etc/ld.so.cache r ,
• /etc/nsswitch.conf r ,
• /lib/ld-linux.so.2 r ,
• /lib/libc.so.6 r ,
• /lib/libtermcap.so.2 r ,
• /usr/local/bin/ph ix ,

12
WireX SystemsImmunix Secure OS

• Linux system similar to Red Hat Linux
• RPM based
• All source-available programs compiled with
  StackGuard and FormatGuard
• PointGuard in future releases
• Kernel equipped with SubDomain and RaceGuard
• All network-accessible daemons SubDomain-profiled

13
Experimentation ...

• Some real-world red teaming
• Play an Immunix server in the Defcon Capture the
  Flag (CtF) games
• Almost no holds barred
• No flooding
• No physical attacks
• New gaming rig designed by the Ghettohackers

14
Basic Defcon CtF Rules
Player Nodes
15
Basic Defcon CtF Rules
If all services found ...
Scorebot Polls player nodes, Looking for req.
services
Player Nodes
16
Basic Defcon CtF Rules
If all services found, Score one point for
the Flag currently on that node
Scorebot Polls player nodes, Looking for req.
services
Player Nodes
17
Basic Defcon CtF Rules
If all services found, Score one point for
the Flag currently on that node
Scorebot Polls player nodes, Looking for req.
services
while each team tries to replace others flags
Player Nodes
18
No Flooding

• DoS attacks are not interesting
• Explicit rule against flooding attacks
• Game masters will make you stop if you are caught
  at it
• Goal ensure that all teams are actually able to
  play
• Penalties
• Kicked out for overt DoS attacks
• Pay for bandwidth with a point penalty

19
Area View
20
Sporting Event
Immunix was white, hence Weiss Labs

• There was an official bookie -)
• Score broadcast on hotel cable

Score obfuscated
Teams named funky colors
21
The Catch

• The required services are secret
• Only a few clues
• They supply us with a VMWare/Linux image
  reference distribution that provides all required
  services
• It is also riddled with vulnerabilities
• The scorebot polls for the required services
• But the scorebot stops its poll if it finds
  something it doesnt like

22
The Reference Distribution

• Red Hat 6.2, unpatched
• nmap shows nearly everything open
• finger, POP, IMAP, SMTP, SNMP, Webmin ...
• Apache running as root
• CGIs for adduser and deleteuser
• Anonymous can create a user login on your node
• As any user number, including zero

23
Example Services the Scorebot Wanted

• Create a user
• Send that user mail
• Finger the user
• POP in to fetch the mail
• Delete the user
• Note no crypto protocols
• No proper authentication of the scorebot
• Must heuristically distinguish scorebot from
  attacks using behavior signatures

24
Interesting Challenge

• Not just survive severe attack, but also
• Protect bad code
• A lot of it
• Vague functional specification
• Rapid deployment
• Great new game infrastructure from Ghettohackers
• Interesting challenge
• Engaging scoreboard

25
Captains Meeting

• Explain the rules in detail
• Hand us the reference distribution

26
Setting Up
27
The Popular Strategy Human Intrusion Detection

• Launch the reference Linux distribution
• Ad hoc patch as stuff happens
• Defend
• look for logins, I.e. non-scorebot behavior
• kill them off ASAP
• very labor-intensive

28
The Immunix Strategy Protect Bad Code with
Immunix Tools

• Port all plausible services to Immunix 7
  distribution
• Use our own fingerd, httpd, etc., up-to-date and
  compiled with StackGuard and FormatGuard
• Run on an Immunix kernel with SubDomain and
  RaceGuard
• Wrap vulnerable services CGIs with SubDomain
  profiles to limit access to least privilege
  necessary
• Launch only when we were reasonably confident
  that the Immunix machine was configured securely

29
Dealing with Logins the SubDomain Shim

• Change adduser CGI to use a special default
  shell /bin/fubush
• /bin/fubush is just a hard link to /bin/bash
• Restrict /bin/fubush to only the operations
  needed by the scorebot
• Attackers can go ahead and create a login with
  uid 0 and it still wont do them any good
• They get a root shell, stuck in a tiny sandbox

30
Immunix Team
31
Immunix Team
Me
Seth Arnold
Steve Beattie
Chris Wright

• Plus 15 volunteers

32
From Our Corner
33
From Our Corner
John Viega
Me
Steve Beattie
Seth Arnold
Chris Wright
34
Mental Stress

• This is a tough game to play
• Head-to-head competition with a lot of very smart
  people
• Real-time, continuous
• The intensity of qualifying exams
• That go on for 22 hours in a 48 hour period
• set in the middle of a rave
• Hydrate or die -)

35
Rave

• Loud music
• Smoking
• Gawkers
• Social engineering
• Periodic news breaks

36
Our Strategic Error

• What We Did
• For first 4 hours
• No server at all
• Porting services to Immunix ASAP, based largely
  on nmap and source inspection
• Next 4 hours
• Launch Immunix server
• Its secure, but is not making the scorebot
  happy
• Cost us massive points
• Too focused on the science of can we defend
  Immunix? and not enough on the game rules

• What We Should Have Done
• Launch reference system immediately
• Defend ad hoc like everyone else
• Run network sniffer to determine what the
  scorebot wants
• Would have
• Put us over the top on points
• Learned what scorebot wants much faster
• We eventually did this

37
Immunix Server Not Up Yet
6th place
38
Once Immunix Server Up in the Scorebots
Opinion )

• Our score quickly rose

2nd place
39
Once Immunix Server Up in the Scorebots
Opinion )
Close 2nd place
40
Once Immunix Server Up in the Scorebots
Opinion )
1st place

• Stayed there most of Saturday

41
Late SaturdayNew Service Requirement

• With 4 hours of play to go, the scorebot
  changed now it wanted Webmin
• Open source web-GUI for Linux administration
• Competitor to WireXs commercial server appliance
  software
• Rather famously vulnerable )
• Took us 2 hours Sunday morning to make the
  scorebot happy again
• Lost our lead

42
Some of Our Creative Attacks

• Lock Out the Owner
• Once we root the machine, install a back door
• Also replace roots login shell with /sbin/halt
• Owner cant log in to their own machine
• But we can

• Spambot
• Add user to their server
• User sends spam mail to all the other teams
• Costs them penalty points
• Penalties are per connection
• Spambot sends 1-byte e-mails

43
Final Score 2nd Place
44
Lesson Symmetric Red Teaming Solves Rules Issues

• Everyone is both an attacker and defender
• Bad everyone needs to learn how to attack
• Good
• Everyone should learn how attacks are done -)
• Rule fussing about how hard or easy it is for the
  attacker apply to all parties -gt less fussing
• Ghettohackers have designed a great game
• Looking for technology transfer to Government

45
Lesson Mandatory Access Control is Not Enough

• telnetd was a required service
• WireX never bothered to patch a vulnerability in
  telnetd for Immunix
• Only idiots run telnetd -)
• Someone hacked our telnetd
• Didnt get out of the SubDomain sandbox
• Did make our telnetd stop working
• Cost us a point that round
• General case MAC protects your system, but not
  your individual services

46
Lesson Resource Management is a Security
Attribute

• SubDomain confined attacker logins to only run
  prescribed code
• Including PERL
• Attacker launched a PERL fork bomb
• Consumed all of real and virtual memory
• While our machine is thrashing, the scorebot
  passes us by
• Costs us a point that round

47
Lesson Redundancy Helps When You Are Vulnerable

• Penetration attacks take a long time to recover
• Must clean up state, find fix vulnerability
• DoS attacks take a long time to recover
• If machine crashes, must fsck file system can
  take 10 minutes
• Hot spare can be on-line in seconds
• Heterogeneous hot spare keeps attacker from
  immediately deploying the same attack

48
Lesson Redundancy is Resource-Constrained

• Must have humans on watch to clean up the
  compromised machine
• The hot spare will not protect you for long
• Presumption that hot spare prevents repeat
  attacks assumes resource limit at the attackers
  end
• If attacker has lots of exploits/resources, they
  will hack your heterogeneous server just as
  quickly
• We had a hot spare, but not enough of them

49
Lesson Immunix was Impenetrable, but not
Incorruptible

• No one ever flagged the Immunix server
• Others did plant enemy flags on our reference
  server (as expected)
• But they did hit the Immunix server hard enough
  to compromise availability
• Take out one required service, and the scorebot
  doesnt award a point
• We missed first place by less than 4 points out
  of 55

50
Immunix SecureServer Appliances

• Base Immunix Secure OS
• Appliances that defend themselves
• Turn-key installation
• WireX installer turns PC hardware into server
  appliance in 5 minutes
• No technical skills required
• Graphical ease of use
• WireX Secure Server Manager Web-GUI management
  system

51
(No Transcript)
52
Immunix SecureServer Solutions

• Partner with HP
• First product Trend Micro Antivirus Mail scanner
  http//h18000.www1.hp.com/products/servers/solutio
  ns/iis
• Websense content filter
• Secure Webmail Appliance
• Customized for USTRANSCOM
• Provides secure webmail access to Microsoft
  Exchange
• Supports Microsoft Exchange/Outlook calendar
  events
• Partner with Secure Computing
• All being demonstrated in our booth

53
Summary

• WireX Immunix
• Secure Linux system with low overhead and
  administrative hassle
• Defcon fun
• Couldnt crack Immunix
• Integrated undocumented code into Immunix in 12
  hours
• Available as
• Secure Linux system
• Turn-key secure server solutions