Experiences with Countering Internet Attacks

1
Experiences with Countering Internet Attacks

• Vern Paxson
• International Computer Science Institute
  andLawrence Berkeley National Laboratory
• Berkeley, California USA
• vern_at_icsi.berkeley.edu, vern_at_ee.lbl.gov
• December 6, 2006

2
Detecting Blocking Internet AttacksBy
Monitoring Network Activity

• Opportunities and styles of network intrusion
  detection
• Architecture of the Bro system
• The fundamental problem of evasion
• The fundamental problem of background radiation
• Building a large honeyfarm
• Why the problem is becoming much more worrisome
• Context for the talk
• A decade conducting network security research at
  USA Lawrence Berkeley National Laboratory
• for which the main system (Bro) operates 24x7
  since 1996

3
Network Intrusion Detection

• Idea tap a network link, analyze whats going
  on, look for trouble
• Appealing because its cheap (broad coverage)
• Can monitor activity of many hosts using just one
  NIDS (network intrusion detection system)
• Though this gets harder as traffic speed/volume
  increases
• Rather than passive (just watching), can operate
  in-line and actively block undesired activity
• An intrusion prevention system
• Greatly raises the bar in terms of performance
  reliability
• Can also provide insight into a sites general
  network use (potentially a huge benefit!)

4
Styles of network intrusion detection
Signature-based

• Core idea look for specific, known attacks.
• Example (from the Snort IDS)
• alert tcp EXTERNAL_NET any -gt HOME_NET 139
  flowto_server,established
• content"eb2f 5feb 4a5e 89fb 893e 89f2"
• msg"EXPLOIT x86 linux samba overflow"
• referencebugtraq,1816
• referencecve,CVE-1999-0811
• classtypeattempted-admin

5
Signature-based, cont

• Can be at different semantic layers, e.g. IP/TCP
  header fields packet payload URLs.
• Pro good attack libraries, easy to understand
  results.
• Con unable to detect new attacks, or even just
  variants.

6
Styles of network intrusion detection
Anomaly-detection

• Core idea attacks are peculiar.
• Approach build/infer a profile of normal use,
  flag deviations.
• Example user joe only logs in from host A,
  usually at night.
• Note works best for narrowly-defined entities.
• Pro potentially detects wide range of attacks
• including novel.
• Con potentially misses wide range of attacks
• including known.
• Con can potentially be trained to accept
  attacks as normal.

7
Styles of network intrusion detection
Specification-based

• Core idea look for patterns of activity that
  deviate from asites policy.
• Example user joe is only allowed to log in from
  host A.
• Pro potentially detect wide range of attacks,
  including novel
• Pro framework can accommodate signatures,
  anomalies
• Con policies/specifications require significant
  development maintenance.
• Con harder to construct attack libraries.
• Note Bro is well-suited to this approach.

8
A look at Bro design goals constraints

• High-speed, large volume monitoring
• FDDI (1996) GigEther (2000) 10Gig (2006)
• Real-time notification
• Mechanism separate from policy
• Extensible
• Avoid simple mistakes ? specialized policy
  language
• Makes Bro an environment for analyzing network
  traffic, especially at the application layer
• The monitor will be attacked

9
How Bro Works

• Taps GigEther fiber link passively, sends up a
  copy of all network traffic.

Network
10
How Bro Works
Filtered Packet Stream
Tcpdump Filter

• Kernel filters down high-volume stream via
  standard libpcap packet capture library.

libpcap
Packet Stream
Network
11
How Bro Works
Event Stream
Event Control

• Event engine distills filtered stream into
  high-level, policy-neutral events reflecting
  underlying network activity
• E.g., connection_attempt, http_reply,
  user_logged_in
• These span a range of semantic levels
• Currently about 300 different types

Event Engine
Filtered Packet Stream
Tcpdump Filter
libpcap
Packet Stream
Network
12
How Bro Works
Real-time Notification Record To Disk
Policy Script

• Policy script processes event stream,
  incorporates
• Context from past events
• Sites particular policies

Policy Script Interpreter
Event Stream
Event Control
Event Engine
Filtered Packet Stream
Tcpdump Filter
libpcap
Packet Stream
Network
13
How Bro Works
Real-time Notification Record To Disk
Policy Script

• Policy script processes event stream,
  incorporates
• Context from past events
• Sites particular policies

Policy Script Interpreter
Event Stream
Event Control
Event Engine
and takes action Records to disk - extensive
logs Generates alerts via syslog or paging Sends
events to other Bros Executes programs as a form
of response
Filtered Packet Stream
Tcpdump Filter
libpcap
Packet Stream
Network
14
Experiences With the Bro System

• Operational at LBL continuously since 1996
• Also at a number of other sites
• Used as an intrusion prevention system
• Automatically install blocks of malicious hosts
• (100s to 1000s of these every day!)
• Tears down TCP connections by injecting RST
  packets
• Provides extensive logs (27 billion recorded TCP
  connections)
• Invaluable for forensics site traffic analysis
• Zillions of incidents one felony conviction
• 135K lines of C, 23K lines of policy scripts
• www.bro-ids.org
• Runs on commodity Unix PCs
• But this is getting very challenging!

15
How Bro Works
Filtered Packet Stream
Tcpdump Filter

• Kernel filters down high-volume stream via
  standard libpcap packet capture library.

libpcap
Packet Stream

• Originally 100X gain
• Recently 10X gain
• Must analyze more applications in traffic
• Today no gain
• Must analyze traffic that is trying to hide by
  using other ports
• E.g., Skype
• E.g., botnet command-and-control over IRC

Network
16
Some general considerations about the problem
space

• Security is about policy.
• The goal is risk management, not bulletproof
  protection.
• Much of the effort concerns raising the bar and
  trading off resources
• Threat model what you are defending against
• E.g., federal laboratory embarrassing
  newspaper articles ? DC
• E.g., California university SB1386 personal
  identity information disclosure

17
Some general considerations about the problem
space, cont

• All intrusion detection systems suffer from the
  twin problems of false positives and false
  negatives.
• These are not minor, but an Achilles heel.
• Scaling works against us as the volume of
  monitored traffic grows, so does its diversity.
• One-in-a-million false positives happen every day
• NIDS research in the lab is far removed from
  operational reality.

18
The Problem of Evasion

• Presence of adversary raises fundamental problems
• Network traffic seen from within a network is
  inherently ambiguous
• Analyzing network traffic at a high semantic
  level requires extensive state which an
  adversary can target.
• Consider detecting occurrences of the string
  root inside a network connection (Lets
  disregard the wholly separate issue of
  false positives whether this is a good signature)

19
Detecting root Attempt 1

• Method scan each packet for r, o, o, t
• Perhaps using Boyer-Moore, Aho-Corasick, Bloom
  filters

But TCP doesnt preserve text boundaries
20
Detecting root Attempt 2

• Method remember match from end of previous
  packet

- Now were managing state -(
21
Detecting root Attempt 3

• Method reassemble entire byte stream
• Keep track of full TCP connection state -( -(
• This is still evadable!

22
Full TCP Reassembly is Not Enough
Packet discarded in transit due to TTL hop count
expiring
Sender / Attacker
Receiver
????
r???
ro??
roo?
root
rice? roce? rict? roct? riot? root? rioe? rooe?
nice? noce? nict? noct? niot? noot? nioe? nooe?
r???? n????
ri??? ni???
ri??? ro??? ni??? no???
ric?? roc?? rio?? roo?? nic?? noc?? nio?? noo??
IDS
r???
????
23
The Problem of Evasion

• Okay, cant you then generate an alarm when you
  see an inconsistent TCP retransmission?
• Or, more generally, on any ambiguous or strange
  traffic?

24
Crud Seen on a Network Access Link

• Storms of 140,000,000 FIN packets, due to TCP
  bugs.
• Storms due to foggy days.
• Private (unroutable!) addresses leaking out.
• Legitimate tiny fragments.
• Fragments with DF (Dont Fragment) set.
• Overlapping fragments.
• TCPs that acknowledge data that was never sent
  (!).
• TCPs that retransmit different data than sent the
  first time (!).
• Many evasions have benign counterparts that are
  rare but do occur

25
Evasion At Higher Semantic Levels

• Consider the following attack URL
• http//./c/winnt/system32/cmd.exe?/cdir
• Easy enough to scan for (e.g., cmd.exe), right?
• But what about
• http//./c/winnt/system32/cm64.exe?/cdir
• Okay, we need to handle escapes. (64d)
• But what about
• http//./c/winnt/system32/cm255452.exe?/cdir
• Oops. Will server double-expand escapes or
  not?
• 25 546 524

26
The Problem of Evasion, cont

• There are many such ambiguities
• At the network layer will this packet arrive at
  the receiver?
• At the transport layer for this inconsistent
  retransmission, will the receiver take the first
  version or the second?
• At the application layer how will this
  corner-case in the spec be interpreted? Will the
  spec be honored?
• Problem is fundamentally hard
• Cant reliably alarm on presence of ambiguity due
  to prevalence of crud in real traffic
• Most promising approach normalization
• Rewrite traffic inline to scrub out ambiguities
• But raises very thorny issues of forwarding
  performance
• and state management

27
The Lay of the Land Changes
28
55 growth/year
29
(No Transcript)
30
What is All That Junk?

• Malice.
• Internet background radiation entire network
  probed 24x7
• Depending where you live, each (unfiltered)
  Internet address receives a probe every 90 sec -
  15 min
• Misconfigurations (a little)
• Backscatter from remote attacks (a little)
• Automated scanning looking for weakness (a lot)
• Worms and bots searching for new victims
• Indiscriminant probing of random addresses/blocks

31
Background RadiationOpportunities and Challenges

• Opportunity many attacks preceded by blind
  probing of Internet addresses
• Includes both worms bots
• Therefore if we monitor a large number of
  addresses, they will come to us

32
Background RadiationOpportunities and Challenges

• Challenge much of the probing is boring
• Corresponds to endemic worms
• E.g., we still see Nimda worm (released Sept.
  2001)!
• or scanning for very well-known vulnerabilities
• How do we tell when were seeing something new
  and/or interesting?
• Its not enough to look at the service being
  scanned
• We must interact with the prober to elicit their
  intent

33
Honeypots

• Honeypot a machine whose only function is to
  attract attackers in order to infer their intent
• Any traffic to it is immediately suspect
• though need to be careful regarding
  mistakes/misconfigurations
• Span a range of fidelity
• Low-fidelity interaction is completely
  fake/scripted
• Appealing since can be done cheaply
• High-fidelity use an actual, compromisable
  machine
• Some types of attacks require high-fidelity
  interaction to discern intent/originality
• E.g., attacker injects code that phones home to
  download bot executable. Code must actually
  execute.

34
GQ Building a Large-Scale Honeyfarm

• Honeyfarm use a network telescope to route scan
  traffic to a set of honeypots
• Goal scale to 100,000s of monitored addresses
• at high fidelity

Note architecture shared with UCSDs Potemkin
35
GQ Building a Large-Scale Honeyfarm

• Honeyfarm use a network telescope to route scan
  traffic to a set of honeypots
• Goal scale to 100,000s of monitored addresses
• at high fidelity

Dark space blocks of otherwise unallocated
addresses
36
GQ Building a Large-Scale Honeyfarm

• Honeyfarm use a network telescope to route scan
  traffic to a set of honeypots
• Goal scale to 100,000s of monitored addresses
• at high fidelity

Physical Honeyfarm Servers
Global Internet
GRE Tunnelsor direct routing
Advertised Dark Space
MGMT
Gateway
VM
VM
VM
VM
VM
VM
Routers send dark space traffic either via
tunnels or direct attachment
VM
VM
VM
37
GQ Building a Large-Scale Honeyfarm

• Honeyfarm use a network telescope to route scan
  traffic to a set of honeypots
• Goal scale to 100,000s of monitored addresses
• at high fidelity

Physical Honeyfarm Servers
Global Internet
GRE Tunnelsor direct routing
Advertised Dark Space
MGMT
Gateway
VM
VM
VM
VM
VM
VM
Gateway applies filtering to reduce load,
allocates honeypot and mediates communication
VM
VM
VM
38
GQ Building a Large-Scale Honeyfarm

• Honeyfarm use a network telescope to route scan
  traffic to a set of honeypots
• Goal scale to 100,000s of monitored addresses
• at high fidelity

Physical Honeyfarm Servers
Global Internet
GRE Tunnelsor direct routing
Advertised Dark Space
MGMT
Gateway
VM
VM
VM
VM
VM
VM
Outbound communication attempted by a honeypot
can be redirected back to another honeypot
VM
VM
VM
39
GQ Building a Large-Scale Honeyfarm

• Honeyfarm use a network telescope to route scan
  traffic to a set of honeypots
• Goal scale to 100,000s of monitored addresses
• at high fidelity

Physical Honeyfarm Servers
Global Internet
GRE Tunnelsor direct routing
Advertised Dark Space
MGMT
Gateway
VM
VM
VM
VM
VM
VM
If redirected traffic again tries to communicate
outbound, then we have found a worm
VM
VM
VM
40
GQ Architecture

• Controller VM independent
• Aggressive filtering
• Containment and redirection
• Mapping and NAT link incoming traffic to
  selected VM
• Honeypot Manager VM dependent

41
Efficacy of GQs Scan Filter
Raw scans/min
Filtered scans/min
42
Experiences With GQ

• Began operation in late 2005
• 28 honeypots on 4 VMware ESX servers
• Soon to be expanded to 7 servers
• Can run 10 different system images
• Primarily use 3 which cover wide range of
  vulnerablities
• Unpatched Windows XP Professional
• Unpatched Windows 2000 Server
• Fully-patched Windows XP Professional with
  insecure configuration and weak password
• Can capture worms exploiting Windows
  vulnerabilities on 80/tcp, 135/tcp, 139/tcp, and
  445/tcp

43
Experiences With GQ, cont

• Network telescopes
• One /14 (262,144 addresses) - actually two nearby
  /15s
• One special hot /23 block
• gt 1,000 times more active than /14, per address!
• Automatically captured 717 worms of 66 distinct
  types (14 different families) during 4 months of
  operation
• Not only buffer-overflow worms but also those
  exploiting weak passwords
• All required multiple connections to complete
• As many as 72 for W32.Mumu.C !

44
The Lay of the Land Changes Again
45
(No Transcript)
46
(No Transcript)
47
The Underground Marketplace

• Economies drive specialization
• Markets enable buyers and sellers to find one
  another
• Commercialization of Malware markets arise
• E.g., ShadowCrew
• Shadowcrew about 4,000 members in 2004
  established the standard for cybercrime forums
  -- set up on well-designed, interactive Web
  pages and run much like a well-organized co-op.
  Communication takes place methodically, via the
  exchange of messages posted in topic areas.
  Members can also exchange private messages.
• recent move of the forum's host computer server
  to Iran, putting it far beyond the reach of U.S.
  authorities. He described Iran as "possibly the
  most politically distant country to the united
  states in the world today."

48
Roles Lingo in the Underground Economy

• Seller one who sells goods or services
• E.g., stolen credit cards, botd hosts, spamming
• E-Gold popular gold-backed currency
• Ripper one who scams others in the community
• Cashier specialist in extracting money from
  compromised bank accounts

49
Roles Lingo, cont

• Drop one who takes delivery of stolen goods or
  funds
• Cardable web site that doesnt validate credit
  cards used to purchase from it
• Dump ATM card info. May or may not include
  PINs. Track 2 refers to additional info only
  available w/ physical card.

50
Summary

• Security is not about bullet-proof it's about
  policies and tradeoffs informed by threat model
• Network analysis can detect all sorts of
  undesirable activity
• but there are significant problems with evasion
• At multiple semantic levels
• Traffic contains much more diversity/junk than
  you'd think, including incessant scanning for
  vulnerabilities (background radiation)

51
Summary, cont

• We can leverage indiscriminate scanning to engage
  attackers using honeypots fed by network
  telescopes (a honeyfarm)
• but requires a great deal of thought regarding
  filtering to reduce traffic to tractable levels
• The most worrisome development for the future is
  the criminalization of malware
• leading to the emergence of an economy of
  specialization
• Threatens to accelerate attacker innovation
• Attackers will bring greater resources to bear
• Changes the pace of the arms race