New Directions in Traffic Measurement and Accounting

1
New Directions in TrafficMeasurement and
Accounting

• Cristian Estan
• (joint work with George Varghese)

2
The Problem

• Measurement and monitoring of network traffic
  required for Internet backbones.
• Useful for short-term monitoring (e.g., DOS
  attacks), traffic engineering (e.g., rerouting),
  and accounting (e.g., usage based pricing)
• How can we do so without tracking millions of
  ants to track a few elephants?

3
State of the art Cisco NetFlow

• Sample packets at high speeds
• Per flow information based on samples
• Aggregate (based on ASes, prefixes, ports) at the
  router
• Problems inaccurate (due to sampling and loss),
  memory-intensive, slow (needs DRAM).

4
Towards a NetFlow Alternative

• Small Percentage of flows (elephants) account for
  large percentage of traffic.
• Top 9 of flows account for 90 of AS pair
  traffic in backbones (Fang-Peterson).
• Can we directly track flows that send say over 1
  of link bandwidth without keeping track of all
  flows?

5
How to identify large flows?
We introduce two new methods for this purpose

• Sample-Counting uses sampling only to decide
  which flows to watch exhaustively.
• Multistage filter uses multiple hash tables
  allowing large flows to be identified while only
  allowing a small number of small flows (false
  positives) to pass through filter.

6
Identify large flows by sampling
7
Multistage filters
8
Operation of Sampled NetFlow
How accurate is Sampled NetFlow? 1 gigabyte/100
megabytes of data Sampling one in 100 packets
9
Operation of our algorithms
Sampling 1/1000
Filter error 0
10
Comparison
11
High Speed Implementation?

• John Huber of MMC Networks did a design of a chip
  doing filter counting.
• 450,000 transistors, under 1 watt of power, runs
  at OC-192 rates, 32 nsec per packet
• Seems easily feasible to implement sample
  counting with similar complexity.

12
Potential Application scalable threshold
accounting

• Measure flows sending over x of link bandwidth
  using sample/filter counting.
• Bill using flat fee per byte charge for flows
  over x
• Track aggregates directly to avoid evasion using
  several flows, each lt x
• Generalizes usage based (x 0) and duration
  based (x 100) pricing.

13
Conclusions

• Paradigm shift for measurement by concentrating
  only on heavy flows
• Two new techniques (sample and filter counting)
  with small memory footprints and provable
  performance.
• Techniques make threshold accounting feasible,
  generalizing usage and duration based pricing.