Characteristics of Network Traffic Flow Anomalies

1
Characteristics of Network Traffic Flow Anomalies

• Paul Barford and David Plonka
• University of Wisconsin Madison
• SIGCOMM IMW, 2001

2
Motivation

• Traffic anomalies are a fact of life in computer
  networks
• Anomaly detection and identification is
  challenging
• Operators typically monitor by eye using SNMP or
  IP flows
• Simple thresholding is ineffective
• Some anomalies are obvious, other are not
• Characteristics of anomalous behavior in IP flows
  have not been established
• Do same types of anomalies have same
  characteristics?
• Can characteristics be effectively used in
  detection systems?

3
Related Work

• Network traffic characterization
• Eg. Caceres89, Leland93, Paxson97, Zhang01
• Focus on typical behavior
• Fault and anomaly detection techniques
• Eg. Feather93, Brutlag00
• Focus on thresholds and time series models
• Eg. Paxson99
• Rule based tool for intrusion detection
• Eg. Moore01
• Backscatter technique can be used to identify DoS
  attacks
• No work which identifies anomaly characteristics

4
Our Approach to Data Gathering

• Consider anomalies in IP flow data
• Collected at UW border router - 5 minute
  intervals
• Archive of two years worth of data (packets,
  bytes, flows)
• Includes identification of anomalies
  (after-the-fact analysis)
• Group anomalies into three categories
• Network operation anomalies
• Steep drop offs in service followed by quick
  return to normal behavior
• Flash crowd anomalies
• Steep increase in service followed by slow return
  to normal behavior
• Network abuse anomalies
• Steep increase in flows in one direction followed
  by quick return to normal behavior

5
IP Flows

• An IP Flow is defined as a unidirectional series
  of packets between source/dest IP/port pair over
  a period of time
• Exported by Lightweight Flow Accounting Protocol
  (LFAP) enabled routers (Ciscos NetFlow)
• We use FlowScan Plonka00 to collect and process
  Netflow data
• Combines flow collection engine, database,
  visulaization tool
• Provides a near real-time visualization of
  network traffic
• Breaks down traffic into well known service or
  application

SRC_IP/Port,DST_IP/Port,Pkts,Bytes,Start/End
Time,TCP Flags,IP Prot
6
Characteristics of Normal traffic
7
Our Approach to Analysis

• Analyze examples of each type of anomaly via
  statistics, time series and wavelets (our initial
  focus)
• Wavelets provide a means for describing time
  series data that considers both frequency and
  scale
• Particularly useful for characterizing data with
  sharp spikes and discontinuities
• More robust than Fourier analysis which only
  shows what frequencies exist in a signal
• Tricky to determine which wavelets provide best
  resolution of signals in data
• We use tools developed at UW Wavelet IDR center
• First step Identify which filters isolate
  anomalies

8
First Look at Analysis of Normal Traffic

• Wavelets easily localize familiar daily/weekly
  signals

9
First Look Analysis of Attacks

• DoS sharp increase in flows and/or packets in
  one direction
• Linear splines seem to be a good filter to
  distinguish DoS attacks

10
Characteristics of Flash Crowds

• Sharp increase in packets/bytes/flows followed by
  slow return to normal behavior eg. Linux releases
• Leading edge not significantly different from DoS
  signal so next step is to look within the spikes

11
Characteristics of Network Anomalies

• Typically a steep drop off in packets/bytes/flows
  followed a short time later by restoration

12
Conclusion and Next Steps

• Project to characterize network traffic flow
  anomalies
• Based on flow data collected at UW border router
• Anomalies have been grouped into three categories
• Analysis approach statistical, time series,
  wavelet
• Initial results
• Good indications that we can isolate signals
• Future
• Continue analysis of anomaly data
• Analysis of data from other sites
• Application of results in (distributed) detection
  systems

13
Acknowledgements

• Somesh Jha
• Jeff Kline
• Amos Ron