Title: New Directions in Traffic Measurement and Accounting
1New Directions in Traffic Measurement and
Accounting
- Focusing on the Elephants, Ignoring the Mice
Cristian Estan and George Varghese University of
California, San Diego
2Talk outline
- Problem definition
- Sample and hold
- Multistage filters
- Validation, measurements
- Conclusions
3Traffic analysis today
Workstation
Router
Concise analysis results
Collection and analysis software
Large raw data
Measurement module
Sampled packets
Offline analysis
Fast link
4Our research agenda
Router
Concise analysis results
Real-time analysis
Measurement module
- Is it doable?
- Is it better?
Fast link
5What is traffic analysis used for?
- Network planning need to know traffic between
pairs of networks (traffic matrix) - Accounting usage based billing
- Detecting DoS attacks flood attacks
- Application characterization breaking up the
traffic based on port numbers
6Common abstractions
- Packets are grouped together into streams based
on header fields - Traffic matrix by source and destination AS
- DoS attacks by destination IP address
- Measuring large streams (this paper)
- Estimating the number of active streams (poster)
7Why is measuring streams hard?
- Cheap memories (DRAM) are too slow to count all
packets - Fast memories (SRAM) are too small to keep
counters for all streams - Opportunity elephants matter, mice dont
- Problem usually we dont know in advance which
streams are large
8Problem definition
- Given a fixed definition for streams, measure
large streams accurately - Large above 1 of link capacity over a 1 minute
interval - Assumptions
- Mice dont matter
- Accuracy of results important
9Talk outline
- Problem definition
- Sample and hold
- Multistage filters
- Validation, measurements
- Conclusions
10How does sample and hold work?
stream memory
Sample
Insert
stream1 1
11How does sample and hold work?
stream memory
Update
stream1 1
stream1 2
12How does sample and hold work?
stream memory
Sample
stream1 2
Insert
stream2 1
13Why is sample hold better?
Sample and hold
Ordinary sampling
14How much better is it?
- Comparing the relative error of the estimate for
a stream at 1/F of the link bandwidth - Memory limited to M entries
15Talk outline
- Problem definition
- Sample and hold
- Multistage filters
- Validation, measurements
- Conclusions
16Multistage filters
- Characteristics
- No large stream is ever omitted
- Very few entries are used by small streams
- Better performance but implementation and tuning
is more complex
17How do multistage filters work?
stream memory
Array of counters
Hash(Pink)
18How do multistage filters work?
stream memory
Array of counters
Hash(Green)
19How do multistage filters work?
stream memory
Array of counters
Hash(Green)
20How do multistage filters work?
stream memory
21How do multistage filters work?
stream memory
Collisions are OK
22How do multistage filters work?
Reached threshold
stream memory
stream1 1
Insert
23How do multistage filters work?
stream memory
stream1 1
24How do multistage filters work?
stream memory
stream1 1
stream2 1
25How do multistage filters work?
stream memory
Stage 1
stream1 1
26Conservative update
Gray all prior packets
27Conservative update
28Conservative update
29Talk outline
- Problem definition
- Sample and hold
- Multistage filters
- Validation, measurements
- Conclusions
30Validation
- Analytical evaluation
- Comparison of analytical results to measured
performance - Comparison of full measurement devices using
different algorithms
31On traces, algorithms much better than analysis
predicts
Percentage of small streams passing filter (log
scale)
Theory Zipf Actual
Conservative update
Number of stages
32Measurement results
- Setup OC48 trace, 100,000 TCP flows, 5 second
intervals, ordinary sampling - unlimited memory,
sampling 1 in 16 our algorithms - 1Mbit, adapting
parameters to keep it around 90 full - Large streams (above 0.1) ordinary sampling has
an error of 9 sample and hold 0.075, multistage
filter 0.037
33Talk outline
- Problem definition
- Sample and hold
- Multistage filters
- Validation, measurements
- Conclusions
34Our contributions
- Abstraction
- Real-time packet analysis abstractions can help
systematize router implementations. - While the notion of elephants and mice is
inherent in earlier work, we abstracted
measurement of large streams - it can be used by
many applications.
35Our contributions (2)
- Algorithms
- Sample and hold is a simple and efficient
algorithm for identifying and measuring large
streams. - Multistage filters with conservative update
perform better but are more complex. - Both can be used for real-time as well as offline
analysis.
36Our contributions (3)
- Validation
- Theoretical results that make no assumptions on
traffic distribution - Simulations on traces are orders of magnitude
better - Preliminary hardware design (John Huber)
indicates feasibility at OC192 speeds
37Thank you!