Flowtools Tutorial

1
Flow-tools Tutorial

• Mark Fullmer
• maf_at_oar.net

2
Agenda

• Deployment motivation.
• Network flows.
• Cisco / Juniper implementation NetFlow.
• Cisco / Juniper Configuration.
• flow-tools programs overview and examples from
  Abilene and Ohio-Gigapop.

3
Motivations

• Where your campus exchanges traffic with by IP
  address, IP Prefix, or ASN.
• What type and how much traffic (SMTP, WEB, File
  Sharing, etc).
• What services running on campus.
• Department level traffic summaries.
• Track network based viruses back to hosts.

4
Motivations

• Track DoS attacks to the source(s), ie the 100
  Solaris servers flooding whitehouse.gov that have
  been 0wn3d.
• Find busy hosts on campus.
• How many destinations each campus host exchanges
  traffic with.
• Campus host counts by service, ie how many active
  web servers.

5
Network Flows

• Packets or frames that have a common attribute.
• Creation and expiration policy what conditions
  start and stop a flow.
• Counters packets,bytes,time.
• Routing information AS, network mask,
  interfaces.

6
Network Flows

• Unidirectional or bidirectional.
• Bidirectional flows can contain other information
  such as round trip time, TCP behavior.
• Application flows look past the headers to
  classify packets by their contents.
• Aggregated flows flows of flows.

7
ssh session
faith.splintered ssh eng4.oar.net w 1012PM
up 476 days, 2106, 7 users, load averages 0.00,
0.00, 0.00 USER TTY FROM
LOGIN_at_ IDLE WHAT maf p0
dhcp9578217.colu 30Sep02 2days ssh maf
p1 dhcp9578217.colu Wed12AM 12 -zsh
(zsh) maf p2 dhcp9578217.colu
Wed07AM 303 -zsh (zsh) maf p7
login.enss.net 22Mar02 199days - maf
p8 login.enss.net 12Apr02 47days - maf
p9 login.enss.net 24Jul02
75days - maf pa login.enss.net
25Jul02 47days -
8
ssh session
faith.splintered/usr/home/maf tcpdump -q -i
sis0 -n ip and host eng4.oar.net tcpdump
listening on sis0 222746.565664
24.95.78.217.1065 gt 192.148.244.24.22 tcp 0
(DF) 222746.603058 192.148.244.24.22 gt
24.95.78.217.1065 tcp 0 (DF) 222746.603523
24.95.78.217.1065 gt 192.148.244.24.22 tcp 0
(DF) 222746.717333 192.148.244.24.22 gt
24.95.78.217.1065 tcp 15 (DF) tos 0x10
222746.717706 24.95.78.217.1065 gt
192.148.244.24.22 tcp 20 (DF) 222746.830214
192.148.244.24.22 gt 24.95.78.217.1065 tcp 276
(DF) tos 0x10 222746.846743
24.95.78.217.1065 gt 192.148.244.24.22 tcp 156
(DF) 222746.967105 192.148.244.24.22 gt
24.95.78.217.1065 tcp 12 (DF) tos 0x10
222746.967292 24.95.78.217.1065 gt
192.148.244.24.22 tcp 20 (DF) 222747.062173
192.148.244.24.22 gt 24.95.78.217.1065 tcp 60
(DF) tos 0x10 222747.062239
192.148.244.24.22 gt 24.95.78.217.1065 tcp 12
(DF) tos 0x10 222747.062433
24.95.78.217.1065 gt 192.148.244.24.22 tcp 0
(DF) 222747.062636 24.95.78.217.1065 gt
192.148.244.24.22 tcp 140 (DF) 222747.196829
192.148.244.24.22 gt 24.95.78.217.1065 tcp 140
(DF) tos 0x10 222747.204546
24.95.78.217.1065 gt 192.148.244.24.22 tcp 28 (DF)
28 packets, 728 byte IP/TCP overhead.
9
Unidirectional Flow with Source/Destination IP Key
ssh 10.0.0.2 w
reply
10.0.0.1
10.0.0.2
Active Flows
Flow Source IP Destination IP

• 10.0.0.1 10.0.0.2
• 10.0.0.2 10.0.0.1

10
Unidirectional Flow with Source/Destination IP Key
telnet 10.0.0.2
ping 10.0.0.2
login
10.0.0.1
10.0.0.2
ICMP echo reply
Active Flows
Flow Source IP Destination IP

• 10.0.0.1 10.0.0.2
• 10.0.0.2 10.0.0.1

11
Unidirectional Flow with IP, Port,Protocol Key
telnet 10.0.0.2
ping 10.0.0.2
login
10.0.0.1
10.0.0.2
ICMP echo reply
Active Flows
Flow Source IP Destination
IP prot srcPort dstPort

• 10.0.0.1 10.0.0.2 TCP 32000 23
  
• 10.0.0.2 10.0.0.1 TCP 23
  32000
• 10.0.0.1 10.0.0.2 ICMP 0 0
• 10.0.0.2 10.0.0.1 ICMP 0 0

12
Bidirectional Flow with IP, Port,Protocol Key
telnet 10.0.0.2
ping 10.0.0.2
login
10.0.0.1
10.0.0.2
ICMP echo reply
Active Flows
Flow Source IP Destination
IP prot srcPort dstPort

• 10.0.0.1 10.0.0.2 TCP 32000 23
  
• 10.0.0.1 10.0.0.2 ICMP 0 0

13
Application Flow
Web server on Port 9090
netscape http//10.0.0.2/9090
10.0.0.1
10.0.0.2
Content-type
Active Flows
Flow Source IP Destination
IP Application

• 10.0.0.1 10.0.0.2 HTTP

14
Aggregated Flow
Main Active flow table
Flow Source IP Destination
IP prot srcPort dstPort

• 10.0.0.1 10.0.0.2 TCP 32000 23
  
• 10.0.0.2 10.0.0.1 TCP 23
  32000
• 10.0.0.1 10.0.0.2 ICMP 0 0
• 10.0.0.2 10.0.0.1 ICMP 0 0

Source/Destination IP Aggregate
Flow Source IP Destination IP

• 10.0.0.1 10.0.0.2
• 10.0.0.2 10.0.0.1

15
NetFlow data reduction
Look at 1 day of flows exports from krc4 Flows
111182160 Octets
2450050798277 Packets 4057574675 Flow
size 48 bytes Flows/packet 30
(typical) Packet overhead 24 bytes Octets in a
flow 1464 bytes
16
NetFlow data reduction
Look at 1 day of flows exports from
krc4 Packets/Flow 37 Octets/Flow
22036 Octets/Packet 603 Packets/Flow
export 1095 Octets/Flow export
661092 Octets/Octets in a flow export 452

17
Packets/Flow distribution
18
Octets/Flow distribution
19
Flow Descriptors

• A Key with more elements will generate more
  flows.
• Greater number of flows leads to more post
  processing time to generate reports, more memory
  and CPU requirements for device generating flows.
• Depends on application. Traffic engineering vs.
  intrusion detection.

20
Flow Accounting

• Accounting information accumulated with flows.
• Packets, Bytes, Start Time, End Time.
• Network routing information masks and
  autonomous system number.

21
Flow Collection

• Passive monitor.
• Router.
• Other existing network device.

22
Passive Monitor Collection
Workstation A
Workstation B
Flow probe connected to switch port in traffic
mirror mode
23
Router Collection
Flow collector stores exported flows from router.
24
Passive Monitor

• Directly connected to a LAN segment via a
  switch/router port in mirror mode, optical
  splitter, or repeated segment.
• Generate flows for all local LAN traffic.
• Must have an interface or monitor deployed on
  each LAN segment.
• Support for more detailed flows bidirectional
  and application (external probe has more
  resources).

25
Router Collection

• Router will generate flows for traffic that
  traverses the router.
• Flows are not generated for local LAN traffic
  (not a sniffer).
• Limited to simple flow criteria (packet
  headers).
• Generally easier to deploy no new equipment.

26
Cisco NetFlow

• Unidirectional flows.
• IPv4 unicast and multicast.
• Aggregated (v8) and unaggregated (v1,5,6,7).
• Flows exported via UDP.
• Supported on IOS and CatIOS platforms.
• Catalyst NetFlow is different implementation.

27
Cisco NetFlow Versions

• 4 Unaggregated types (1,5,6,7).
• 14 Aggregated types (8.x).
• Each version has its own packet format.
• Version 1 does not have sequence numbers no way
  to detect lost flows.
• The version defines what type of data is in the
  flow.
• Some versions specific to Catalyst platform.

28
Cisco NetFlow Versions

• v9 in development. More flexible packet format
  so new fields can be added without creating yet
  another version while still maintaining compact
  encoding.

29
NetFlow v1

• Key fields Source/Destination IP,
  Source/Destination Port, IP Protocol, ToS, Input
  interface.
• Accounting Packets, Octets, Start/End time,
  Output interface.
• Other Bitwise OR of TCP flags.
• Historical dont use. No sequence s.

30
NetFlow v5

• Key fields Source/Destination IP,
  Source/Destination Port, IP Protocol, ToS, Input
  interface.
• Accounting Packets, Octets, Start/End time,
  Output interface.
• Other Bitwise OR of TCP flags,
  Source/Destination AS and IP Mask.
• Packet format adds sequence numbers for detecting
  lost exports.

31
NetFlow v8

• Aggregated v5 flows.
• 3 Catalyst 65xx specific that correspond to the
  configurable flow mask.
• Much less data to post process, but lose fine
  granularity of v5 no IP addresses.

32
NetFlow v8

• AS
• Protocol/Port
• Source Prefix
• Destination Prefix
• Prefix
• Destination (Catalyst 65xx)
• Source/Destination (Catalyst 65xx)
• Full Flow (Catalyst 65xx)

33
NetFlow v8

• ToS/AS
• ToS/Protocol/Port
• ToS/Source Prefix
• ToS/Destination Prefix
• Tos/Source/Destination Prefix
• ToS/Prefix/Port

34
NetFlow Packet Format

• Common header among export versions.
• All but v1 have a sequence number.
• Version specific data field where N records of
  data type are exported.
• N is determined by the size of the flow
  definition. Packet size is kept under 1480
  bytes. No fragmentation on Ethernet. No PMTU
  detection.

35
NetFlow v5 Packet Example
IP/UDP packet
NetFlow v5 header
v5 record


v5 record
36
NetFlow v5 Packet (Header)
struct ftpdu_v5 / 24 byte header /
u_int16 version / 5 / u_int16 count
/ The number of records in the PDU /
u_int32 sysUpTime / Current time in
millisecs since router booted / u_int32
unix_secs / Current seconds since 0000 UTC
1970 / u_int32 unix_nsecs / Residual
nanoseconds since 0000 UTC 1970 / u_int32
flow_sequence/ Seq counter of total flows seen
/ u_int8 engine_type / Type of flow
switching engine (RP,VIP,etc.) / u_int8
engine_id / Slot number of the flow
switching engine / u_int16 reserved
37
NetFlow v5 Packet (Records)
/ 48 byte payload / struct ftrec_v5
u_int32 srcaddr / Source IP Address /
u_int32 dstaddr / Destination IP Address /
u_int32 nexthop / Next hop router's IP
Address / u_int16 input / Input
interface index / u_int16 output /
Output interface index / u_int32 dPkts
/ Packets sent in Duration / u_int32
dOctets / Octets sent in Duration. /
u_int32 First / SysUptime at start of flow
/ u_int32 Last / and of last packet
of flow / u_int16 srcport / TCP/UDP
source port number or equivalent / u_int16
dstport / TCP/UDP destination port number or
equiv / u_int8 pad u_int8 tcp_flags
/ Cumulative OR of tcp flags / u_int8
prot / IP protocol, e.g., 6TCP, 17UDP,
... / u_int8 tos / IP
Type-of-Service / u_int16 src_as /
originating AS of source address / u_int16
dst_as / originating AS of destination
address / u_int8 src_mask / source
address prefix mask bits / u_int8 dst_mask
/ destination address prefix mask bits /
u_int16 drops recordsFT_PDU_V5_MAXFLOWS
38
NetFlow v8 Packet Example(AS Aggregation)
IP/UDP packet
NetFlow v8 header
v8 record


v8 record
39
NetFlow v8 AS agg. Packet
struct ftpdu_v8_1 / 28 byte header /
u_int16 version / 8 / u_int16 count
/ The number of records in the PDU /
u_int32 sysUpTime / Current time in
millisecs since router booted / u_int32
unix_secs / Current seconds since 0000 UTC
1970 / u_int32 unix_nsecs / Residual
nanoseconds since 0000 UTC 1970 / u_int32
flow_sequence / Seq counter of total flows seen
/ u_int8 engine_type / Type of flow
switching engine (RP,VIP,etc.) / u_int8
engine_id / Slot number of the flow
switching engine / u_int8 aggregation /
Aggregation method being used / u_int8
agg_version / Version of the aggregation
export / u_int32 reserved / 28 byte
payload / struct ftrec_v8_1 u_int32
dFlows / Number of flows / u_int32
dPkts / Packets sent in duration /
u_int32 dOctets / Octets sent in duration
/ u_int32 First / SysUpTime at start
of flow / u_int32 Last / and of last
packet of flow / u_int16 src_as /
originating AS of source address / u_int16
dst_as / originating AS of destination
address / u_int16 input / input
interface index / u_int16 output /
output interface index / recordsFT_PDU_V8_1_
MAXFLOWS
40
Cisco IOS Configuration

• Configured on each input interface.
• Define the version.
• Define the IP address of the collector (where to
  send the flows).
• Optionally enable aggregation tables.
• Optionally configure flow timeout and main (v5)
  flow table size.
• Optionally configure sample rate.

41
Cisco IOS Configuration
interface FastEthernet0/0/0 ip address 10.0.0.1
255.255.255.0 no ip directed-broadcast ip
route-cache flow interface ATM1/0/0 no ip
address no ip directed-broadcast ip
route-cache flow interface Loopback0 ip address
10.10.10.10 255.255.255.255 no ip
directed-broadcast ip flow-export version 5
origin-as ip flow-export destination 10.0.0.10
5004 ip flow-export source loopback 0 ip
flow-aggregation cache prefix export destination
10.0.0.10 5555 enabled
42
Cisco IOS Configuration
krc4sh ip flow export Flow export is enabled
Exporting flows to 10.0.0.10 (5004) Exporting
using source IP address 10.10.10.10 Version 5
flow records, origin-as Cache for prefix
aggregation Exporting flows to 10.0.0.10
(5555) Exporting using source IP address
10.10.10.10 3176848179 flows exported in
105898459 udp datagrams 0 flows failed due to
lack of export packet 45 export packets were
sent up to process level 0 export packets were
punted to the RP 5 export packets were dropped
due to no fib 31 export packets were dropped
due to adjacency issues 0 export packets were
dropped due to fragmentation failures 0 export
packets were dropped due to encapsulation fixup
failures 0 export packets were dropped
enqueuing for the RP 0 export packets were
dropped due to IPC rate limiting 0 export
packets were dropped due to output drops
43
Cisco IOS Configuration
krc4sho ip ca fl IP packet size distribution
(106519M total packets) 1-32 64 96 128
160 192 224 256 288 320 352 384 416 448
480 .002 .405 .076 .017 .011 .010 .007 .005
.004 .005 .004 .004 .003 .002 .002 512 544
576 1024 1536 2048 2560 3072 3584 4096 4608
.002 .006 .024 .032 .368 .000 .000 .000 .000 .000
.000 IP Flow Switching Cache, 4456704 bytes
36418 active, 29118 inactive, 3141073565 added
3132256745 ager polls, 0 flow alloc failures
Active flows timeout in 30 minutes Inactive
flows timeout in 15 seconds last clearing of
statistics never Protocol Total Flows
Packets Bytes Packets Active(Sec)
Idle(Sec) -------- Flows /Sec
/Flow /Pkt /Sec /Flow
/Flow TCP-Telnet 2951815 0.6 61
216 42.2 26.6 21.4 TCP-FTP
24128311 5.6 71 748 402.3
15.0 26.3 TCP-FTPD 2865416 0.6
916 843 611.6 34.7 19.8 TCP-WWW
467748914 108.9 15 566 1675.8
4.9 21.6 TCP-SMTP 46697428
10.8 14 370 159.6 4.0
20.1 TCP-X 521071 0.1 203
608 24.7 24.5 24.2 TCP-BGP
2835505 0.6 5 94 3.3
16.2 20.7
44
Cisco IOS Configuration
krc4sho ip ca fl TCP-other 1620253066
377.2 47 631 18001.6 27.3
23.4 UDP-DNS 125622144 29.2 2
78 82.5 4.6 24.7 UDP-NTP
67332976 15.6 1 76 22.0
2.7 23.4 UDP-TFTP 37173 0.0
2 76 0.0 4.1 24.6 UDP-Frag
68421 0.0 474 900 7.5
111.7 21.6 UDP-other 493337764 114.8
17 479 1990.3 3.8 20.2 ICMP
243659509 56.7 3 166
179.7 3.3 23.3 IGMP 18601
0.0 96 35 0.4 941.4
8.1 IPINIP 12246 0.0 69
52 0.1 548.4 15.2 GRE
125763 0.0 235 156 6.9
50.3 21.1 IP-other 75976755 17.6
2 78 45.4 3.9 22.8 Total
3176854246 739.6 33 619 24797.4
16.2 22.6 SrcIf
SrcIPaddress DstIf DstIPaddress Pr
SrcP DstP Pkts AT5/0/0.4 206.21.162.150
AT1/0/0.1 141.219.73.45 06 0E4B A029 507
AT4/0/0.10 132.235.174.9 AT1/0/0.1
137.99.166.126 06 04BE 074C 3 AT4/0/0.12
131.123.59.33 AT1/0/0.1 137.229.58.168 06
04BE 09BB 646 AT1/0/0.1 137.99.166.126
AT4/0/0.10 132.235.174.9 06 074C 04BE 3
45
Cisco CatIOS Configuration
set mls flow full set mls nde version 7 set mls
nde 10.0.0.10 9110 set mls agingtime 32
46
Cisco CatIOS Configuration
swlap1gt sh mls Total packets switched
5116997156 Total bytes switched
2289120109999 Total routes 2283 IP statistics
flows aging time 32 seconds IP statistics flows
fast aging time 0 seconds, packet threshold
0 IP Current flow mask is Full flow Netflow Data
Export version 8 Netflow Data Export
disabled Netflow Data Export configured for port
9110 on host 10.0.0.10 Total packets exported
6545 IPX statistics flows aging time 256
seconds IPX flow mask is Destination flow IPX max
hop is 255 Module 15 Physical MAC-Address
00-04-9b-78-bb-fc
47
Cisco CatIOS Configuration
swlap1gt sh mls stat entry
Last Used Destination IP Source IP
Prot DstPrt SrcPrt Stat-Pkts
Stat-Bytes ---------------- --------------- -----
------ ------ ---------- ---------
64.219.177.137 206.21.217.6 TCP 2206 WWW
4 1113 207.254.193.44
198.30.37.194 TCP 56403 WWW 9
10567 198.30.37.19 66.183.100.164 TCP
WWW 1604 4 731
192.88.193.144 199.218.4.130 UDP DNS
61449 1 71 192.88.193.144
199.218.5.131 TCP DNS 2157 5
235 199.218.4.3 208.249.92.145 TCP
WWW 37519 6 506
12.42.50.51 198.30.37.19 TCP 11200 WWW
5 494 199.218.5.130
142.22.16.54 TCP WWW 54658 6
745 207.254.193.44 198.30.37.194 TCP
56391 WWW 40 50762
206.21.0.139 24.178.12.64 TCP 110 1085
8 484 217.10.162.50
199.218.5.130 TCP 46983 WWW 53
73775 65.224.146.242 198.30.37.19 TCP
2282 WWW 4 390
199.218.4.135 65.31.16.245 TCP WWW 1196
6 650 24.150.19.19
205.133.123.10 TCP 58043 WWW 4
176
48
Cisco Catalyst Native IOS Configuration
mls flow ip destination-source mls nde flow
include mls nde src_address 10.0.0.9 version 7 ip
flow-export source Loopback0 ip flow-export
version 5 peer-as ip flow-export destination
10.0.0.5 5555
49
Cisco Issues

• Cat 65K Sup2/MSFC2 NetFlow implementation does
  not fill in important fields like input/output
  interface. Resolved?

Bug Id CSCdt21216 Headline Netflow records
source / dst interface index missing Product
cat6000 Model
x6k-sup2 Component earl
Duplicate of Severity 3
Status C Version Found 6.1(1)
Fixed-in Version 6.2(1)
6.1(3) 6.3(1)PAN Release Notes On Sup2, the
destination and source Interface indices would be
reported as zero in netflow exports. This is a
hardware limitation.
50
Juniper Configration

• Sample packets with firewall filter and forward
  to routing engine.
• Sampling rate is limited to 7000pps (addressed
  with future PIC).
• Fine for traffic engineering, but restrictive
  for DoS and intrusion detection.
• Juniper calls NetFlow cflowd.

51
Juniper Configration
Firewall filter Enable sampling / flows
forwarding-options sampling input
family inet rate
100 output
cflowd 10.0.0.16 port
2055 version 5

firewall filter all term all
then sample
accept
52
Juniper Configration
Apply firewall filter to each interface.
interfaces ge-0/3/0 unit 0
family inet filter
input all
output all
address 192.148.244.1/24

53
Flow-tools

• Collection of programs to collect and post
  process Cisco NetFlow compatible flows.
• Written in C, designed to be fast (scales to
  large installations).
• Includes library (ftlib) for custom applications.
• Installation with configuremakemake install on
  most platforms.
• Distributed design for large installations.

54
flow-capture

• Collect NetFlow export packets and store to disk.
• Built in compression.
• Manages disk space by expiring older flow files
  at configurable limits (dataset size or number of
  files).
• Pre-filtering and Pre-tagging.

55
flow-capture

• Instrumentation for flows/second, packets/second,
  and dropped packets.
• Server for TCP based flow-client.
• Privacy mask option for removing host bits from
  flows.

56
flow-capture flows/second
57
flow-capture packets/second
58
flow-capture flow drops/second
59
flow-fanout

• Replicate NetFlow UDP streams from one source to
  many destinations.
• Destination may be a multicast address.
• Same instrumentation as flow-capture.
• Can translate NetFlow packet format on output.
• Privacy mask support.

60
flow-expire

• Expire (remove) old flow files based on dataset
  size or number of files.
• Same functionality built in to flow-capture.
• Used when managing disk space in a distributed
  environment.

61
Abilene Configuration

• Collect and process flows for Abilene routers.
• Use sampled NetFlow on Cisco GSRs.
• Distribute flows to Asta and Arbor Networks.
• Nightly usage reports.
• Archive of raw anonymized flows.

62
Abilene Configuration

• Dsitribute post processed data to internet2.edu.
• Moving to sampled data from Juniper T640 and new
  architecture.

63
Current Abilene Configuration
Server running flow-fanout _at_ IU
NetFlow exports (UDP)
12 Abilene core routers Configured with sampled
NetFlow
Post Processed NetFlow (ssh)
64
Current Abilene Configuration
NetFlow exports
Pull compressed flow files with rsync from
collectors.
4 servers running flow-capture (data anonymized
at collection)
Server with 2 Terabyte RAID5 Array web
serverfor nightly reports.
flow-expire to manage disk space
65
New Abilene Configuration
Server running flow-fanout _at_ IU
POP router with local directly connected server
running flow-capture and flow-fanout.
NetFlow exports (UDP)
Compressed files (rsync via TCP)
Post Processed NetFlow (ssh)
66
New Abilene Configuration
Ohio ITEC
Pull compressed flow files with rsync from
collectors.
flow-expire to manage disk space flow-report,
flow-nfilter, flow-tag for reports.
Server with 2 Terabyte RAID5 Array web
serverfor nightly reports.
67
Collector Placement and configuration

• NetFlow is UDP so the collector should ideally be
  directly connected to the router to minimize
  packet loss and IP spoofing risks.
• No flow control. Undersized collector will drop
  flows. Monitor netstat s grep buf and
  configure syslog so dropped flows will be logged.

68
flow-print

• Formatted output of flow files.

eng1 flow-print lt ft-v05.2002-01-21.093345-0500
head -15 srcIP dstIP prot
srcPort dstPort octets packets 131.238.205.199
194.210.13.1 6 6346 40355 221
5 192.5.110.20 128.195.186.5
17 57040 33468 40 1
128.146.1.7 194.85.127.69 17 53
53 64 1 193.170.62.114
132.235.156.242 6 1453 1214 192
4 134.243.5.160 192.129.25.10 6
80 3360 654 7
132.235.156.242 193.170.62.114 6 1214
1453 160 4 130.206.43.51
130.101.99.107 6 3226 80 96
2 206.244.141.3 128.163.62.17 6
35593 80 739 10
206.244.141.3 128.163.62.17 6 35594
80 577 6 212.33.84.160
132.235.152.47 6 1447 1214 192
4 132.235.157.187 164.58.150.166 6
1214 56938 81 2
129.1.246.97 152.94.20.214 6 4541
6346 912 10 132.235.152.47
212.33.84.160 6 1214 1447 160
4 130.237.131.52 130.101.9.20 6
1246 80 902 15
69
flow-cat

• Concat many flow files or directories of files.

eng1 ls ft-v05.2002-01-21.160001-0500
ft-v05.2002-01-21.170001-0500 ft-v05.2002-01-21.16
1501-0500 ft-v05.2002-01-21.171501-0500 ft-v05.2
002-01-21.163001-0500 ft-v05.2002-01-21.173001-0
500 ft-v05.2002-01-21.164501-0500
tmp-v05.2002-01-21.174501-0500 eng1 flow-cat .
flow-print srcIP dstIP
prot srcPort dstPort octets packets 138.26.220.
46 192.5.110.20 17 62242 33456 40
1 143.105.55.23 18.123.66.15
17 41794 41794 40 1
129.15.134.66 164.107.69.33 6 1214
2222 4500 3 132.235.170.19
152.30.96.188 6 6346 1475 128
3
70
flow-merge

• Flow-merge is similar to flow-cat except it
  maintains relative ordering of flows when
  combining the files.
• Typically used when combining flows from multiple
  collectors.

71
flow-filter

• Filter flows based on port, protocol, ASN, IP
  address, ToS bits, TCP bits, and tags.
  (Historical, use flow-nfilter).

eng1 flow-cat . flow-filter -P119 flow-print
head -10 srcIP dstIP
prot srcPort dstPort octets
packets 155.52.46.50 164.107.115.4 6
33225 119 114 2
128.223.220.29 129.137.4.135 6 52745
119 1438382 1022 155.52.46.50
164.107.115.4 6 33225 119 374
6 164.107.115.4 192.58.107.160 6
60141 119 5147961 8876
128.223.220.29 129.137.4.135 6 52745
119 1356325 965 128.223.220.29
129.137.4.135 6 52714 119 561016
398 130.207.244.18 129.22.8.64 6
36033 119 30194 121
155.52.46.50 164.107.115.4 6 33225
119 130 2 198.108.1.146
129.137.4.135 6 17800 119
210720652 216072
72
flow-nfilter

• Filter flows based any defined fields including
  derived operations such as pps, bbs, and
  duration.
• Configuration file based.
• Supports AND and OR operations.
• Filters and primitives are named for ease of use.
• Use patricia trees, hash tables, and bucket
  lookups where possible for fast processing.

73
flow-nfilter
filter-primitive abilene-interface type
ifindex permit 25 filter-definition
to-abilene match dst-ifindex abilene-interface
filter-definition from-abilene match
src-ifindex abilene-interface filter-primitive
UDPTCP type ip-protocol permit tcp permit
udp filter-definition udptcp match ip-protocol
UDPTCP
74
flow-nfilter
filter-primitive OSU type ip-address-prefix
permit 128.146/16 permit 140.254/16 permit
164.107/16 filter-primitive DNS type ip-port
match 53 filter-primitive WEB type ip-port
match 80,8080,443 filter-definition OSUDNS
match ip-address OSU match ip-protocol UDP
match ip-destination-port DNS
75
flow-nfilter

• IP address lookup is patricia trie, worst case
  performance O(W) (W is length of address).
• IP Protocol lookup is bucket, always O(1).
• IP Port lookup is bucket, always O(1).
• Performance remains relatively constant even if
  loading up prefix list with full internet routing
  table (100,000 entries).

76
flow-nfilter

• Some other lookups use hash tables, for example
  IP address list or tag list. Usually O(1).
• A few filters do require linear lookups but by
  nature list is short, ie pps filter or start time
  filter.
• Short circuit evaluation for AND and OR
  operations.

77
flow-split

• Split flow files into smaller files.
• Split based on time, tags, or number of flows.
• Typically used with flow-stat/flow-report for
  graphing. For example flow-split can produce 5
  minute intervals from a day long dataset for
  time-series graphing.

78
flow-tag

• Adds a tag field to flows based on IP exporter,
  IP prefix, Autonomous System, or next hop.
• Used to manage groups of prefixes or ASNs.
• Example, group IP prefixes by customer ID for
  billing.

79
flow-tag
tag format 0 7 15 23
31 0000 0000 0000 0000 0000 0000 0000
0000 (32 bits) RRRRRRRRRRRRRR TTTT
NNNNNNNNNNNNNNNNNNN
Site name Site
type Reserved BGP community
65000X is site name (X -gt N) BGP community
65001Y is site type (Y -gt T) SITE_NAME_MASK
0x0000FFFF SITE_TYPE_MASK 0x00FF0000
ID Name -----------------------------
---- 0x0001 OSU 0x0002 CWRU
0x0003 BGSU 0x0004 UC 0x0005
UAKRON 0x0006 WRIGHT 0x0007
KENT 0x0008 DAYTON 0x0009
OBERLIN
ID Type ------------------------
0x01 Participant 0x02 SEGP
0x03 Sponsored-Participant 0x04
Gigapop 0x05 MULTICAST
80
flow-tag
tag-action OHIO-GIGAPOP_DST type dst-prefix
OSU match 128.146/16 set-dst 0x010001 match
164.107/16 set-dst 0x010001 match 140.254/16
set-dst 0x010001 match 192.153.26/24 set-dst
0x010001 CWRU match 129.22/16 set-dst
0x010002 match 192.5.110/24 set-dst 0x010002
BGSU match 129.1/16 set-dst 0x010003
UC match 129.137/16 set-dst 0x010004
UAKRON match 130.101/16 set-dst 0x010005
WRIGHT match 130.108/16 set-dst 0x010006
KENT match 131.123/16 set-dst 0x010007
DAYTON match 131.238/16 set-dst 0x020008
OBERLIN match 132.162/16 set-dst 0x020009
tag-action OTHER_DST type src-prefix match 0/0
set-dst 0x0 tag-action OTHER_SRC type
src-prefix match 0/0 set-src 0x0 tag-definition
OHIO-GIGAPOP term input-filter 25 action
OTHER_DST action OHIO-GIGAPOP_DST term
output-filter 25 action OTHER_SRC action
OHIO-GIGAPOP_SRC
81
flow-header

• Display meta information in flow file.

eng1 flow-header lt ft-v05.2002-01-21.093345-0500
mode normal capture
hostname eng1.oar.net exporter IP address
0.0.0.0 capture start Mon Jan 21
093345 2002 capture end Mon Jan 21
094501 2002 capture period 676
seconds compress on byte order
little stream version 3
export version 5 lost flows
0 corrupt packets 0 sequencer resets
0 capture flows 341370
82
flow-stat

• Generates reports from flow files.
• Output is readable and easily imported into
  graphing programs (gnuplot, etc).
• IP Address, IP address pairs, ports, packets,
  bytes, interfaces, next hop, Autonomous System,
  ToS bits, exporter, and tags.
• Historical use flow-report.

83
flow-stat
--- ---- ---- Report Information --- ---
--- Fields Total Symbols Disabled
Sorting None Name Overall Summary
Args flow-stat -f0 Total Flows
111182160 Total Octets
2450050798277 Total Packets
4057574675 Total Time (1/1000 secs) (flows)
2414764456464 Duration of data (realtime)
86409 Duration of data (1/1000 secs)
88281720 Average flow time (1/1000 secs)
21718.0000 Average packet size (octets)
603.0000 Average flow size (octets)
22036.0000 Average packets per flow
36.0000
84
flow-report

• Replacement for flow-stat.
• Configuration file based.
• Multiple reports per data pass.
• Concurrent output per report (file, program,
  sorting options, fields).
• Integrated tagging and filtering for performance
  gain and readability.
• 70 reports currently defined.

85
flow-report
include-filter ./filter include-tag
./gigapop-tags stat-report to-abilene-by-customer
type source-tag tag-mask 0xFF 0xFF filter
to-abilene output path out/to-abilene-by-custo
mer options header,names fields
-pps,-bps,-duration sort octets stat-report
from-abilene-by-customer type destination-tag
tag-mask 0xFF 0xFF filter from-abilene output
path out/from-abilene-by-customer options
header,names fields -pps,-bps,-duration
sort octets
stat-report from-abilene-by-protocol type
ip-protocol filter from-abilene output path
out/from-abilene-by-protocol options
header,names fields -pps,-bps,-duration
sort octets stat-report to-abilene-tcp type
ip-destination-port filter tcp-to-abilene
output path out/to-abilene-tcp options
header,names fields -pps,-bps,-duration
sort octets output path out/to-abilene-tcp.p
options header,names,percent-total fields
-pps,-bps,-duration sort octets
86
flow-report
stat-report to-abilene-host-count type
ip-source-address-destination-count filter
to-abilene output path out/to-abilene-host-coun
t options header,names fields
-pps,-bps,-duration sort count stat-report
to-abilene-multicast type ip-source/destination-
address filter mcast-to-abilene output
path flow-rptfmt Y-d-m options header
fields -pps,-bps,-duration sort octets
stat-definition all tag OHIO-GIGAPOP report
to-abilene-by-customer report from-abilene-by-cus
tomer report to-abilene-by-group report
from-abilene-by-group report to-abilene-by-protoc
ol report from-abilene-by-protocol report
to-abilene-tcp report from-abilene-tcp report
to-abilene-udp report from-abilene-udp
87
Flow-report

• The following examples are all for flows received
  on Oct 2, 2002.
• The output has been formatted with flow-rptfmt.

88
flow-reportTo abilene by customer
source-tag flows octets
packets OHIOU
12468213 346780284114
474584209 OSU 10083752
216222538921 456970704
MIAMI 11555447 85209430257
211139319 UC
3408919 69337541299
150235362 CWRU 2146398
53967968301 71557236
KENT 3548222 35856830763
98904064 UAKRON
680448 30197772896
76634728 WRIGHT 695999
29646284289 57189398
DAYTON 2439731 18139794910
59435826 BGSU
2572766 16310992145
40206277 OBERLIN 802277
10831055817 21659836
CSU 542017 9751230620
17011505 0
578005 8311955805
17920369 XAVIER 509438
8303822859 19977123
DENISON 1335965 4930300208
25778299 CENTRAL_STATE
12113 3587783969 2946397
OHIO-GIGAPOP 145075
3409588817 4352398
OTTERBEIN 393828 3337857516
6427175 (27 records total)
89
flow-reportFrom abilene by customer
destination-tag flows octets
packets OSU
8206984 404410296660
493518138 OHIOU 10632410
378662580090 553882360
UC 3039419
146987191906 166540812 MIAMI
10203261 71462550082
182178469 WRIGHT 625040
64552082817 63606132
KENT 3571049 55469667804
100115387 CWRU
1809929 47298827158
66748219 UAKRON 1203008
42302272842 76792866
BGSU 2556832 27454028852
38833117 DAYTON
4039081 27083462962
64213411 OHIO-ITEC 24500
23197891069 15634326
DENISON 1771432 22412975083
28021003 OSC
74247 21575406055
15617253 XAVIER 539444
17389557985 21496360
OBERLIN 976343 16896684646
21949892 0
495554 14545105602
18824353 BWC 1132527
9667335610 15154831
MULTICAST 37725 9408996691
102650979 CSU
569675 9202402106
16078735 (28 records total)
90
flow-reportTo/From abilene by group
source-tag flows octets
packets PART
36146734 808071443348
1443293483 SEGP 19074659
140069521118 376283762
0 578005 8311955805
17920369 GIGAPOP
253853 7210265828
16180995 SPART 154001
685371077 4022173
destination-tag flows octets
packets PART
32214346 1176339350235
1576115766 SEGP 19779445
182946067604 357284495
GIGAPOP 129062 48388568451
35179510 0
495554 14545105602
18824353 MULTICAST 37725
9408996691 102650979
SPART 144483 5741808731
5270298
91
flow-reportTo/From Abilene by Protocol
ip-protocol flows octets
packets tcp 38903962
949344938829 1780524499
udp 14733318 14280174521
70130455 icmp 2054787
578664366 5964561
gre 1104 81503144
230466 169 510943
58796208 816614
esp 671 3200640
20259 ipv6 2467
1279468 13928 ip-protocol
flows octets packets
tcp 36737595
1389381631832 1905052404 udp
13734354 47439394244
184078339 icmp 1862340
379697738 5177470 gre
906 110143341
225355 169 462826
54259200 753600 esp
646 1672800
12308 ipv6 1731
1577881 11379
ax.25 93 882162
3288 ipencap 92
590868 11124
igmp 12 44296
86 narp 18
2576 40
pup 2 376
8
92
flow-reportTo Abilene TCP Dest Port
ip-destination-port flows octets
packets FastTrack
24.835828 6.603882
11.541223 Gnutella
7.692790 2.308477
4.933651 Napster
0.159431 1.214922
1.358084 7999
0.003208 0.607430
0.240277 http
4.813186 0.491013
2.402686 59
0.000496 0.398512
0.193378 innosys
0.033706 0.396740
0.303003 ssh
0.031968 0.386647
0.218494 smtp
0.360454 0.371606
0.239597 eDonkey-2000
0.064587 0.353453
0.253181 nntp
0.014585 0.336791
0.247589 2234
0.070181 0.335752
0.190099 2702
0.015261 0.314940
0.135489 10021
0.000673 0.302725
0.114831 ftp-data
0.012528 0.293089
1.122457 telnet
0.030663 0.291930
0.143617 1107
0.011328 0.259371
0.107312 7668
0.003095 0.253869
0.100611 Gnutella
0.972104 0.246086
0.593843 Neomodus-Direct-Connect
0.003958 0.204204
0.204218
93
flow-reportFrom Abilene TCP Dest Port
ip-destination-port flows octets
packets FastTrack
13.538197 5.509694
9.777353 msg-icp
0.000133 0.934877
0.456198 Neomodus-Direct-Connect
0.007053 0.526346
0.383283 Gnutella
6.074029 0.466143
1.583139 ftp
0.063839 0.394695
0.224846 nntp
0.027901 0.365369
0.303404 4422
0.013477 0.339689
0.179834 http
4.857253 0.319698
1.131411 aol
0.012902 0.257276
0.351811 3819
0.012325 0.221330
0.159259 47399
0.000180 0.217029
0.123149 Napster
0.031839 0.215064
0.369720 smtp
0.357062 0.201978
0.202899 innosys
0.021909 0.181785
0.135780 1976
0.013640 0.173067
0.113857 1097
0.020655 0.166452
0.088196 2925
0.018736 0.165947
0.088380 7776
0.000452 0.158655
0.082350 ftp-data
0.009565 0.154542
0.395363 1677
0.021615 0.148936
0.258940
94
flow-reportTo Abilene UDP Dest Port
ip-destination-port flows octets
packets 2055
0.006312 17.053153
2.329062 commplex-link
0.000387 12.342609
1.677807 41170
25.035270 10.970910
32.354782 49156
0.000217 5.389666
1.138699 10000
0.002640 3.168155
0.496024 49606
0.000339 2.922246
1.292035 FastTrack
13.620897 2.795093
3.861790 27015
0.351659 2.651897
7.886515 5012
0.000068 2.514314
0.582493 49176
0.000081 2.157357
0.650475 27005
0.001317 2.108271
2.622176 5016
0.000041 2.087801
0.468828 6970
0.000543 2.067873
0.665665 radius
2.717311 1.561363
3.717907 domain
7.108589 1.528589
4.462461 12203
0.000842 1.458771
1.551721 49198
0.000034 1.287165
0.283817 8888
0.000964 1.228599
0.167672 49154
0.001140 1.196136
0.468568 1235
0.002213 0.996790
0.385473
95
flow-reportFrom Abilene UDP Dest Port
ip-destination-port flows octets
packets 56464
0.049555 17.677823
51.282764 4252
0.000692 11.352055
1.960886 4247
0.000612 7.053652
1.220310 4244
0.000655 6.386842
1.103243 4250
0.000619 5.128119
0.885821 6970
0.002621 4.618467
1.651840 4245
0.000604 4.574414
0.790175 4242
0.000619 4.147176
0.716384 afs3-prserver
0.026030 3.593472
0.627750 4254
0.000714 2.819961
0.487140 41170
30.282116 2.635842
9.693955 4243
0.000728 2.179707
0.376556 4248
0.001318 1.769443
0.307173 59818
0.001427 1.730026
4.210950 4246
0.000670 1.627574
0.283242 27005
0.003007 1.580597
2.507883 49606
0.000218 1.134879
0.370680 radius
3.873011 1.073899
3.489514 4249
0.000699 1.040643
0.181706 afs3-callback
0.032007 1.015535
0.179987
96
flow-reportTo Abilene Multicast S,G
ip-source-address ip-destination-address
flows octets packets
128.146.112.93 224.2.0.1 52
3644792 22848
164.107.73.49 224.2.133.133 49
2104830 8190
164.107.73.50 224.2.133.133 48
2042136 8136
128.146.111.36 224.2.0.1 57
1732352 22988
164.107.73.50 224.2.133.134 48
1502592 8256
164.107.73.49 224.2.133.134 48
1490216 8188
193.166.0.41 224.2.127.254
286 237666 286
171.68.122.14 224.2.127.254
275 224675 275
128.107.150.34 224.2.127.254
199 197010 199
129.217.131.30 224.2.127.254
280 180040 280
192.148.244.23 224.5.5.5 3
149304 102
64.65.127.133 224.2.127.254
287 110495 287
130.83.47.123 224.2.127.254
183 91317 183
171.69.248.71 224.2.127.254
163 61832 163
130.83.126.22 224.2.127.254
168 61824 168
212.219.151.116 224.2.127.254 67
29390 67
205.189.33.76 224.2.127.254 22
27214 22
139.133.204.110 224.2.127.254 50
27000 50
207.75.164.44 224.2.127.254 41
23042 41
97
flow-reportFrom Abilene Multicast S,G
ip-source-address ip-destination-address
flows octets packets
155.101.21.38 224.2.127.254 64
14115449 49187
131.252.80.100 224.2.127.254
580 11091500 38113
150.29.224.28 224.2.177.155 54
8569074 9722
131.252.176.50 224.2.127.254
284 6283704 18984
171.69.248.71 224.2.145.19 18
5535907 6313
128.3.10.50 224.2.177.155 60
5488196 7028
131.193.77.102 224.2.177.155 50
4987121 6273
134.174.178.254 224.2.127.254 49
4559324 17402
134.174.178.253 224.2.127.254 49
4557228 17394
128.223.83.33 224.2.127.254
252 4197456 3839
129.105.153.48 224.2.127.254
121 4122602 13059
129.105.153.49 224.2.127.254
116 3628240 11935
129.105.12.35 224.2.177.155 23
3188598 3723
141.225.215.61 224.2.127.254 66
2843808 9416
63.105.122.14 224.2.127.254
206 2719485 4495
131.247.105.10 224.2.133.133 55
2169700 8345
128.227.8.136 224.2.133.133 49
2158428 8366
140.221.8.53 224.2.177.155 23
2154591 2722
98
flow-reportTo Abilene by source Address
ip-source-address flows
octets packets
128.146.11.117 0.001658
1.198353 0.436777
64.247.109.217 0.001745
0.827703 0.323890
164.107.77.104 0.001007
0.683243 0.350049
131.187.254.2 0.553731
0.622047 0.615308
164.107.77.100 0.000203
0.490449 0.412104
64.247.105.200 0.010792
0.472982 0.172323
129.22.39.145 0.002991
0.472295 0.164211
132.235.196.5 0.000059
0.443100 0.177759
64.247.65.121 0.011723
0.398491 0.150519
164.107.35.58 0.000751
0.394327 0.154084
128.146.135.184 0.000151
0.393943 0.145393
129.137.150.144 0.006720
0.388005 0.178204
129.22.164.158 0.000557
0.371489 0.137157
64.247.94.142 0.012429
0.371459 0.132373
64.247.110.33 0.002179
0.363492 0.205407
164.107.210.142 0.001585
0.363146 0.197966
137.148.203.177 0.012794
0.351391 0.139850
206.21.71.10 0.687776
0.340878 0.340017
99
flow-reportFrom Abilene by dst Address
ip-destination-address flows
octets packets
192.148.248.24 0.000335
0.931192 0.426093
164.107.77.100 0.000206
0.908344 0.494996
64.247.108.177 0.088410
0.694466 0.339623
131.187.254.2 0.526903
0.601483 0.536259
233.2.171.1 0.012888
0.583444 4.505289
129.137.164.187 0.021733
0.529286 0.263854
164.107.242.71 0.019257
0.523264 0.286042
129.137.150.89 0.012727
0.493833 0.252883
64.247.85.174 0.001750
0.486535 0.226529
164.107.3.40 0.019604
0.475143 0.279502
129.137.155.167 0.003318
0.418701 0.248909
206.21.71.10 0.703418
0.417402 0.315859
129.22.130.35 0.023909
0.406043 0.188727
129.22.34.18 0.005377
0.379816 0.204850
192.148.251.54 0.000104
0.374664 0.172263
129.137.240.228 0.002119
0.370352 0.180073
128.146.135.106 0.000263
0.361298 0.197680
164.107.220.83 0.013447
0.361096 0.180690
64.247.65.205 0.001045
0.360547 0.175456
140.254.232.142 0.000386
0.349813 0.160018
100
flow-reportTo Abilene SMTP / Customer
source-tag flows octets
packets OSU
25.239783 20.947484
21.210771 MIAMI 13.119068
10.971892 17.199629
OHIOU 3.949198 9.196418
7.206689 DENISON
0.432857 9.156816
5.579017 CWRU 7.459834
8.121457 6.653185
UC 5.421768 8.035289
8.601174 DAYTON
1.192318 5.946226
3.953595 KENT 15.641335
4.911743 6.972634
OBERLIN 1.494677 3.988554
2.602640 WRIGHT
1.337793 3.655101
3.306141 0 0.456390
2.449019 1.528096
BGSU 4.155287 2.430585
3.172927 UAKRON
2.397473 2.240855
1.994214 BWC 0.472078
1.331747 1.045406
OSC 12.000200 1.294894
4.188048 CEDARVILLE
0.479922 1.276318
0.854342 CHMCC 0.704552
1.221819 0.934251
CSU 1.298572 1.101268
1.252083 MCO
0.250301 0.854932
0.566654 FINDLAY 0.084860
0.285458 0.242798 (26
records total)
101
flow-reportTo Abilene SMTP / Address
ip-source-address flows
octets packets
140.141.31.28 0.054196
7.621240 4.219975
131.238.75.33 0.482775
4.601938 2.969837
132.162.1.220 0.733789
3.700020 2.225292
130.108.128.60 0.759461
3.048794 2.735314
134.53.253.21 2.367522
3.025292 4.628053
129.137.2.198 0.800822
2.7598