Developed by Cisco Systems in 1996

1
Netflow Overview

• Developed by Cisco Systems in 1996
• The value of information in the cache was a
  secondary discovery
• Initially designed as a switching path
• NetFlow is now the primary network accounting
  technology in the industry
• Answers questions regarding IP traffic
• who, what, where, when, and how
• NetFlow version 9 an IETF standard

2
Traffic Analysis

• What we needs
• application performance
• application-based accounting
• network security
• Network behavior, application recognition
• debug ip packet in router?
• IP Sniffing in shared LAN (or using switch to do
  so)
• Port Span in switch (how about port span in
  router?)
• Circuit Sniffing
• Netflow
• What we prefer in backbone
• Embeded
• Fixed length partial packet export
• Real-time filtered packet export

3
Addressing The Needs with Netflow
4
Netflow Possible Applications

• Network Monitoring
• Network planning
• Security Analysis
• Application Monitoring
• User Monitoring
• Traffic Engineering
• Peering Agreement
• Usage-base Billing
• Destination sensitive billing

5
What is a flow?

• Defined by seven unique keys
• Source IP address
• Destination IP address
• Source port
• Destination port
• Layer 3 protocol
• TOS byte (DSCP)
• Input interface (ifIndex)

Exported Data
A Flow is Unidirectional!
6
NetFlow Sequence

1. Create and update flows in NetFlow Cache
2. Expiration
3. Aggregation?
4. Export Version
5. Transport Protocol

7
NetFlow Sequence (continued)
step1

• Inactive timer expired (15 sec is default)
• Active timer expired (30 min (1800 sec) is
  default)
• NetFlow cache is full (oldest flows are expired)
• RST or FIN TCP Flag

step2
step3
Yes
No
e.g. Protocol-Port Aggregation Scheme becomes
step4
Aggregated Flows export Version 8 or 9
Non-Aggregated Flows export Version 5 or 9
Export Packet
step5
Payload (flows)
Header
8
Netflow Processing Order
Pre- Processing
Features And Services
Post Processing
Packet Sampling Filtering
IP Multicast MPLS IPv6
Aggregation schemes Non-key fields
lookup Export
9
Creating Export Packets
Enable NetFlow
Traffic
Core Network (IP, MPLS)
PE

• Export Packets
• Approximately 1500 bytes
• Typically contain 20-50 flow records
• Sent more frequently if traffic increases on
  NetFlow-enabled interfaces

UDP NetFlow Export Packets
Application Performance Billing Security
Collector (Solaris, HP-UX, or Linux)
10
NetFlow Principles

• Inbound traffic only (with some exceptions)
• Unidirectional flow
• Accounts for both transit traffic and traffic
  destined for the router
• Works with Cisco Express Forwarding (CEF) or fast
  switching
• Almost supported on all interfaces and Cisco IOS
  Software platforms
• Provides the sub-interface information in the
  flow records
• 6500/7600 enables Netflow on all interfaces by
  default

11
Comprehensive Platform Support
GSR 12000
ESR 10000
Catalyst 5000/6500/7600
Catalyst 4500
7200/7500/
AS5300/5800
4500/4700
3700
3600
2500/2600
1400/ 1600/ 1700
12
NetFlow Versions
13
Version 5 - Flow Format
From/to

• Source IP Address
• Destination IP Address

• Packet Count
• Byte Count

Usage
Time of Day

• Source TCP/UDP Port
• Destination TCP/UDP Port

• Start sysUpTime
• End sysUpTime

Application

• Input ifIndex
• Output ifIndex

• Next Hop Address
• Source AS Number
• Dest. AS Number
• Source Prefix Mask
• Dest. Prefix Mask

Routing and Peering

• Type of Service
• TCP Flags
• Protocol

QoS
Blue Key Field (7) Red - Lookup Field
(5) Black- Value Field (6)
14
Netflow Configuration Commands

• ip flow-export version ltversiongt origin-as
  peer-as bgp-nexthop
• e.g. ip flow-export version 5
• ip flow-export destination ltaddressgt ltportgt
• e.g. ip flow-export destination 10.0.0.1 65001
• ip flow export source ltinterfacegt
• default is interface with best route to
  collector. Recommendation configure loopback
  interface.
• ip flow-aggregation cache ltname of aggregation
  schemegt
• select the aggregation cache
• ip flow-cache timeout inactive ltsecondsgt
• sets the seconds an inactive flow will remain in
  the cache before expiration. 15 seconds is
  default
• ip flow-cache timeout active ltmintuesgt
• sets the minutes an active flow will remain in
  the cache bvefore expiration. 30 minutes is
  default
• ip flow-cache entries ltnumbergt
• sets the maximum number of flow entries in the
  cache. The default varies dependent on platform.

15
Netflow Show Commands

• show ip cache verbose flow
• shows Netflow statistics
• show cache flow aggregation ltname of aggregation
  schemegt
• shows netflow statistics for the configured
  aggregation scheme
• show ip flow export
• shows export statistics
• clear ip cache flow
• clears netflow statistics
• clear ip flow stats
• clears export statistics

16
Show ip cache flow

• IP packet size distribution (2175M total
  packets)
• 1-32 64 96 128 160 192 224 256 288
  320 352 384 416 448 480
• .001 .440 .139 .014 .008 .000 .000 .000 .000
  .000 .000 .000 .011 .000 .000
• 512 544 576 1024 1536 2048 2560 3072 3584
  4096 4608
• .000 .000 .000 .002 .377 .000 .000 .000 .000
  .000 .000
• IP Flow Switching Cache, 4456704 bytes
• 550 active, 64986 inactive, 509378135 added
• 3145787062 ager polls, 0 flow alloc failures
• Active flows timeout in 30 minutes
• Inactive flows timeout in 15 seconds
• Protocol Total Flows Packets Bytes
  Packets Active(Sec) Idle(Sec)
• -------- Flows /Sec /Flow /Pkt
  /Sec /Flow /Flow
• TCP-WWW 10431912 2.4 10 181
  25.7 7.1 20.4
• TCP-SMTP 773843 0.1 6 98
  1.1 8.3 16.7
• .
• Total 509377507 118.5 4 567
  506.4 1.7 15.9

17
Show ip flow export

• Routergt sh ip flow export
• Flow export v5 is enabled for main cache
• Exporting flows to 192.168.1.2 (2055)
  192.168.2.3 (2054)
• Exporting using source interface Loopback0
• Version 5 flow records, origin-as
• 998016649 flows exported in 33267252 udp
  datagrams
• 0 flows failed due to lack of export packet
• 0 export packets were sent up to process level
• 0 export packets were dropped due to no fib
• 0 export packets were dropped due to adjacency
  issues
• 0 export packets were dropped due to
  fragmentation failures
• 0 export packets were dropped due to
  encapsulation fixup failures
• 0 export packets were dropped enqueuing for the
  RP
• 0 export packets were dropped due to IPC rate
  limiting

18
Version 7

• Adds NetFlow switching support for
• Cisco Catalyst 5000 Series Switches with an RSM
• Cisco Catalyst 5000 Series Switches with an MSFC
• Uses MultiLayer Switching (MLS) or CEF with Cisco
  Catalyst 6000 Series Switches with SUP2
• IP unicast only
• No multicast or IPX, even if MLS can do all three
• MLS cache is the equivalent of the NetFlow cache

19
Version 8

• Router-based aggregation
• Enables router to summarize NetFlow data
• Reduces NetFlow Export data volume
• Decreases NetFlow Export bandwidth requirements
• Currently 11 aggregation schemes
• Five original schemes
• Six new schemes with the TOS byte field
• Several aggregations can be enabled simultaneously

20
Version 9

• Fixed formats (versions 1, 5, 7, and 8) are not
  flexible and adaptable
• Cisco needed to build a new version each time a
  customer wanted to export new fields
• When new versions are created, partners need to
  reengineer to support the new export format

Solution Build a flexible and extensible export
format!
21
Netflow v9 Principles

• Version 9 is an export format
• Still a push model
• Sent the template regularly (configurable)
• Independent of the underlying protocol, it is
  ready for any reliable protocol (ie TCP, SCTP)
• Advantage we can add new technologies and data
  types quickly
• E.g. MPLS, IPv6, BGP Next Hop, Multicast

22
Netflow V9 Template

• NetFlow Version 9 Export format is template
  based. Version 9 record format consists of a
  packet header followed by at least one or more
  template or data FlowSets. A template FlowSet
  (collection of one or more template) provides a
  description of the fields that will be present in
  future data FlowSets. Templates provide an
  extensible design to the record format, a feature
  that should allow future enhancements to NetFlow
  services without requiring concurrent changes to
  the basic flow-record format.
• template composed of type and length
• flow records composed of template ID and value
• sent the template regularly (configurable),
  because of UDP

23
Netflow Version 9 Scenario
24
Netflow v9 Example for Template Definition
25
Netflow Version9 Export Packet
26
Netflow v9 Example for 1 Export Packet
27
NetFlow v9 Export Packet
To support technologies such as MPLS or
Multicast, this export format can be leveraged to
easily insert new fields
Flows from Interface A
Flows from Interface B
Option Template FlowSet
Data FlowSet
Template FlowSet

• Header

Data FlowSet
Option Data FlowSet FlowSet ID
FlowSet ID 1
FlowSet ID 2
Template Record Template ID 1 (specific Field
types and lengths)
Template Record Template ID 2 (specific Field
types and lengths)
(version, packets, sequence , Source ID)
Data Record (Field values)
Data Record (Field values)
Option Data Record (Field values)
Option Data Record (Field values)
Template ID (specific Field types and lengths)
Data Record (Field values)

• Matching ID s is the way to associate Template
  to the Data Records
• The Header follows the same format as prior
  NetFlow versions so Collectors will be backward
  compatible
• Each Data Record represents one flow
• If exported flows have the same fields then they
  can be contained in the same Template Record e.g.
  unicast traffic can be combined with multicast
  records
• If exported flows have different fields then they
  cant be contained in the same Template Record
  e.g. BGP next-hop cant be combined with MPLS
  Aware NetFlow records

28
NetFlow v9 Export

• test(config) ip flow-export version ?
• 1
• 5
• 9
• test(config) ip flow-export version 9
  .

Configuring Version 9 export
Export versions available for standard NetFlow
flows
Configuring Version 9 export for an aggregation
scheme
test(config) ip flow-aggregation cache as
test(config-flow-cache) enabled
test(config-flow-cache) export ? destination
Specify the Destination IP address version
configure aggregation cache export version
test(config-flow-cache) export version ? 8
Version 8 export format 9 Version 9 export
format test(config-flow-cache) export version 9
Export versions available for aggregated NetFlow
flows
29
IETF IP Flow information Export(IPFIX) Working
Group

• IPFIX is an effort to
• Define the notion of a "standard IP flow"
• Devise data encoding for IP flows
• Consider the notion of IP flow information export
  based upon packet sampling
• Identify and address any security privacy
  concerns affecting flow data
• Specify the transport mapping for carrying IP
  flow information(IETF approved congestion-aware
  transport protocol)
• Netflow version 9 has been selected as a basis
  for the IPFIX protocol

30
IETF Packet Sampling WG(PSAMP)

• PSAMP agreed to use IPFIX(Netflow version9) for
  export
• PSAMP is an effort to
• specify a set of selection operations by which
  packets are sampled
• describe protocols by which information on
  sampled packets is reported to applicatons
• http//www.ietf.org/html.charters/psamp-charter.ht
  ml
• Note Netflow is already using some sampling
  mechanisms

31
NetFlow Infrastructure
32
NetFlow Uses
Access
Distribution
Distribution
Access
Core
Network Layer

• Attack Mitigation
• User (IP) monitoring
• Application monitoring

• Billing
• Chargeback
• AS Peer Monitoring

• Traffic Engineering
• Traffic Analysis

• Attack Mitigation
• User (IP) monitoring
• Application monitoring

• Billing
• Chargeback
• AS Peer Monitoring

Applications

• Aggregation Schemes (v8)
• show ip cache flow command
• Arbor Networks

• NetFlow MPLS Egress Accounting
• BGP Next-hop (v9)
• Multicast NetFlow (v9)

• MPLS Aware NetFlow (v9)
• BGP Next-hop (v9)
• Sampled NetFlow

• NetFlow MPLS Egress Accounting
• BGP Next-hop (v9)
• Multicast NetFlow (v9)

• Aggregation Schemes (v8)
• show ip cache flow command
• Arbor Networks

NetFlow Features
33
Netflow Collector(NFC) 5.0
34
Netflow on the Network Analysis Module (NAM)
35
Netflow Partners
36
Billing

• Flat-rate billing does not necessarily scale
• Competitive pricing models can be created with
  usage-based billing
• Usage-based billing considerations
• Time of day
• Within or outside of the network
• Application
• Distance-based
• Quality of Service (QoS) / Class of Service (CoS)
• Bandwidth usage
• Transit or peer
• Data transferred
• Traffic class

37
Tracking Users

• Who are my top N talkers, and what percentage of
  traffic do they represent?
• How many users are on the network at a given
  time?
• When will upgrades affect the least number of
  users?
• How long do users spend connected to the network?
• Where Internet sites do they use?
• What is a typical pattern of usage between sites?
• Are users staying within an acceptable usage
  policy (AUP)?
• Alarm DOS attacks like smurf, fraggle, and SYN
  flood
• Will watch for these attack, regardless of
  source / destination

38
Principle Netflow Benefits
Service Provider
Enterprise

• Internet access monitoring (protocol
  distribution, where traffic is going/coming)
• User Monitoring
• Application Monitoring
• Charge Back billing for departments
• Security Monitoring

• Peering arrangements
• Network Planning
• Traffic Engineering
• Accounting and billing
• Security Monitoring

39
NetFlow Charge Back Billing
Account per network (rather that per IP addresses)
Example charge the department for the cost of
the Internet link
Internet
Finance
RD
HR
40
NetFlow Peering Agreement
Account per BGP AS, to Review Peering Agreements
ISP
41
NetFlow Peering Agreement
Public Routers 1, 2, 3 Month of
SeptemberOutbound Traffic
4
1
2
1
6
1
1
1
1
1
1
8
1
8
10
32
20
42
MPLS Aware NetFlow (v9)
IP Fields Source and destination IP address
IP Fields Input and output sub-interfaces
IP Fields Transport layer protocol
IP Fields Source and destination application port numbers
IP Fields 8 bit IP Type of Service (ToS)
IP Fields TCP Flags (accumulation from all packets in the flow)
MPLS Fields Up to three incoming MPLS labels with experimental (EXP) bits and end-of-stack (S) bit
MPLS Fields Position of each of the three labels
MPLS Fields Type of the top label
MPLS Fields IP address associated with the top label
Traditional NetFlow Fields Number of packets
Traditional NetFlow Fields Number of bytes (count either IP or MPLS header / payload)
Traditional NetFlow Fields Time-stamps of first and last packets in the flow
43
MPLS
Traditional NetFlow for IP to MPLS traffic
Egress MPLS NetFlow Accounting for MPLS to IP
traffic
MPLS Aware NetFlow (version 9)
IP
MPLS
IP
Traffic Flow

• Egress MPLS NetFlow Accounting
• IP information only
• Ideal for billing
• Current availability Cisco IOS Software Releases
  12.0(10)ST and 12.1(5)T
• MPLS Aware NetFlow (version 9)
• Exports up to three MPLS labels, and IP packet
  information
• Ideal for Traffic Engineering
• Will be available in Cisco IOS Software Releases
  12.0(24)S, 12.2S, and 12.3

44
Autonomous System

• Origin-AS
• Specifies that export statistics include the
  origin autonomous system (AS) for the source and
  destination
• Peer-AS
• Specifies that export statistics include the peer
  AS for the source and destination

• 3600-4(config) ip flow-export version 5 ?
• origin-as record origin AS
• peer-as record peer AS
• ltcrgt
• 3600-4(config)

45
Autonomous System
NetFlow enabled
AS 101
AS 104
AS 102
AS 103

• Configuring Peer-AS
• Source AS AS 103
• Destination AS AS 105

AS 105
Router(config)ip flow-export version 5 peer-as
AS 106
46
Autonomous System
NetFlow enabled
AS 101
AS 104
AS 102
AS 103
AS 105
Router(config)ip flow-export version 5 origin-as

• Configuring Origin-AS
• Source AS AS 101
• Destination AS AS 106

AS 106
47
BGP next-hop

• Supported only in version 9 export
• For traffic engineering/analysis and possible
  billing applications
• Fields that are exported include all those found
  in version 5 export
• Will be supported in Cisco IOS Software Releases
  12.0(26)S, 12.2S, and 12.3

48
BGP next-hop
49
Netflow BGP next-hop
50
BGP next-hop Details

• Supported only in version 9 export
• For traffic engineering/analysis (traffic matrix)
  and possible billing applications. "What is the
  Next hop IP address of my BGP traffic?"
• exported fields include all version 5 fields,
  including IP next hop
• Adds 16 bytes to each Netflow flow record (goes
  from 64 bytes to 80 bytes), while CPU increase is
  negligible
• Edge to Edge traffic matrix for
  engineering/analysis and possible billing
  applications
• Supported in Cisco IOS Software releases
  12.0(26)S, 12.2(18)S, and 12.3(1)

51
BGP next-hop

• pamela(config) ip flow-export version ?
  
• 1
• 5
• 9
• pamela(config) ip flow-export version 9
  .

Configuring Version 9 export
Configuring Version 9 export with BGP next-hop
pamela(config) ip flow-export version 9 ?
bgp-nexthop record BGP NextHop origin-as
record origin AS peer-as record peer AS
ltcrgt pamela(config) ip flow-export version 9
bgp-nexthop
52
Multicast NetFlow

• Three types of NetFlow implementations for
  Multicast traffic
• Traditional NetFlow
• Multicast NetFlow Ingress
• Multicast NetFlow Egress

53
Multicast Traditional NetFlow
(S, G) - (10.0.0.2, 224.10.10.100)
NetFlow Collector server
Traditional NetFlow configuration
Interface Ethernet 0 ip route-cache flow ip
flow-export version 9 ip flow-export
destination 127.0.0.1 9995
10.0.0.2
Eth 0
127.0.0.1
Eth 1
Eth 3
Eth 2
Flow Record Created in NetFlow Cache

• There is only one flow per NetFlow configured
  input interface
• The 7 Key fields that define a unique flow are
  marked in red
• Destination interface is marked as Null
• Bytes and Packets are the incoming values

54
Multicast NetFlow Ingress
(S, G) - (10.0.0.2, 224.10.10.100)
NetFlow Collector server
Multicast NetFlow Ingress configuration
Interface Ethernet 0 ip multicast netflow
ingress ip flow-export version 9 ip
flow-export destination 127.0.0.1 9995
10.0.0.2
Eth 0
127.0.0.1
Eth 1
Eth 3
Eth 2
Flow Record Created in NetFlow Cache

• There is only one flow per NetFlow configured
  input interface
• The 7 Key fields that define a unique flow are
  marked in red
• Destination interface is marked as Null
• Bytes and Packets are the outgoing values

55
Multicast NetFlow Egress
Multicast NetFlow Egress configuration
(S, G) - (10.0.0.2, 224.10.10.100)
NetFlow Collector server
Interface Ethernet 1 ip multicast netflow
egress Interface Ethernet 2 ip multicast netflow
egress Interface Ethernet 3 ip multicast netflow
egress ip flow-export version 9 ip flow-export
destination 127.0.0.1 9995
10.0.0.2
Eth 0
127.0.0.1
Eth 1
Eth 3
Eth 2
Flow Records Created in NetFlow Cache

• There is one flow per Multicast NetFlow Egress
  configured output interface
• One of the 7 Key fields that define a unique flow
  has changed from Source Interface to Destination
  Interface
• Bytes and Packets are the outgoing values

56
Multicast NetFlow Summary

• Supported via NetFlow version 9 export format
• Availability
• Cisco IOS Software Releases 12.0(27)S, 12.2S, and
  12.3
• Not supported in 120000
• Performance Ingress vs. Egress
• Multicast NetFlow Ingress and traditional NetFlow
  will have similar performance numbers
• Multicast NetFlow Egress will have performance
  impact that is proportional to the number of
  interfaces on which it is enabled (include input
  interface)
• Cisco Catalyst 6500/7600 Series Switches
• Do not currently support the tracking of
  multicast traffic via NetFlow due to current ASIC
  limitation
• Will have this support in a future Supervisor

57
How to Identify a Security Attack?

• Suddenly highly-increased overall traffic in the
  network
• Higher CPU and memory utilization of network
  devices
• Unexpectedly large amount of traffic generated by
  individual hosts
• Increased number of accounting records generated
• Multiple accounting records with abnormal
  content, like one packet per flow record (e.g.
  TCP SYN flood)
• A changed mix of traffic applications, e.g. a
  sudden increase of "unknown" applications
• An increase of certain traffic types and
  messages, e.g. TCP resets or ICMP messages
• An increasing number of ACL violations

58
What Does a DOS Attack Look Like?
59
NetFlow Mitigating Attacks

• Cost Saver
• sh ip cache flow command to find top volume
  flows
• Identify source of attack
• Write access-list to block
• Monitor via show ip cache flow Null entry
  in DestIf field to show that it is blocked
• Prefix-port aggregation can be configured, while
  sh ip cache flow aggregation prefix-port is
  used
• Most Effective
• Arbor Networks leverages NetFlow to provide a
  quicker response and more sophisticated solution

60
Security Analysis Best Practices
61
Quality of Service Example
ToS bits
Precedence bits
DS5 DS4 DS3 DS2 DS1 DS0 ECN ECN
128 64 32 16 8 4 2 1
Early Congestion Notification (ECN) bits
DiffServ field AKA IP DSCP markings
62
Quality of Service Example
63
Tracking TOS with NetFlow
7200-3-netflow show ip cache verbose flow SrcIf
SrcIPaddress DstIf
DstIPaddress Pr TOS Flgs Pkts Port Msk AS
Port Msk AS NextHop
B/Pk Active SR6/0 210.210.210.2
PO1/0 200.200.200.2 FF 00 10
21K 0000 /0 0 0000 /0 0
0.0.0.0 1496 665.4 SR6/0
210.210.210.2 PO1/0 200.200.200.2 06
C0 00 21K 0000 /0 0
0000 /0 0 0.0.0.0 1496
666.0 7200-3-netflow show ip cache verbose flow
SrcIf SrcIPaddress DstIf
DstIPaddress Pr TOS Flgs Pkts Port Msk AS
Port Msk AS NextHop
B/Pk Active Et1/1 52.52.52.1 Fd4/0
42.42.42.1 01 55 10 3748 0000
/8 50 0000 /8 40
202.120.130.2 28 17.8 Et1/2
52.52.52.1 Fd4/0 42.42.42.1 01
CC 10 3568 0000 /8 50
0000 /8 40 202.120.130.2 28
17.8 Et1/2 10.1.3.2 Fd4/0
42.42.42.1 01 C0 10 1124 0000 /0 0
0000 /8 40 202.120.130.2
28 17.8
Hex Decimal Binary 55 85 0101 0101 Precedence 2
- Immediate (Class 2), Delay - low, Reliability -
high, Endpoints of transport protocol
ECN-capable C0 192 1100 0000 Precedence 6 -
Internetwork Control (Routing Protocols) CC 204 11
00 1100 Precedence 6 - Internetwork Control
(Routing Protocols), Throughput - high,
Reliability - high
64
Sampled NetFlow

• Deterministic
• Original type
• Cisco 12000 Series Internet Routers
• Cisco Catalyst 6500 Series Switches Release
  12.1(13)E
• Random (recommended per statistical principles)
• Cisco IOS Software Releases 12.0(26)S, 12.2S, and
  12.3
• Cisco 2500, 2600, 3600, 7200, and 7500 Series
  Routers
• Cisco 12000 Series Internet Routers
• Time-based
• Cisco Catalyst 6500 Series Switches Release
  12.1(13)E
• Trajectory (Hash-based)
• in development

65
Sampling configuration

• GSR 12xxx (IOS Version 12.0(31)S2
• R1(config) ip flow-sampling-mode packet-interval
  256
• R1(config-if) ip route-cache flow sampled input
• R1(config-if) ip route-cache flow sampled output
• bj2-bgw(config)ip flow-sampling-mode
  packet-interval ?
• lt10-16382gt Specify the packet interval at
  which to sample
• 7609 (12.2(18)SXD6)
• R1(config) mls flow ip source
• R1(config) mls nde sender version 5
• R1(config) mls sampling time-based 64 //
  641
• R1(config-if) ip route-cache flow
• R1(config-if) mls netflow sampling

66
Cisco Catalyst 6500 and 7600 Series Switches

• Export is centrally via the supervisor and MSFC,
  each line card has its own hardware NetFlow cache
  and forwarding table, i.e. distributed platform

67
Cisco 12000 Series Internet Routers NetFlow

• Engine 0 software support
• Engine 1 software support
• Engine 2 supported in ASICs, but lower priority
  so beware if running many other features
• Engine 3 version 5 support in software, version
  8 support in ASIC
• Engine 4 not supported
• Engine 4 supported in ASICs

68
Cisco 12000 Series Internet Routers Sampled
NetFlow
Engine Full NetFlow Sampled NetFlow
0
1
2
3
4
4
Not supported
Supported
69
Scaling - Memory Utilization
70
Scaling - Sample TrafficDeterministic vs. Random
Sampling
71
Sampled Netflow Details

• Deterministic
• Cisco C6500/7600 Series switches(12.1(13)E)
• Cisco 12000 series internet routers (12.0(11)S
  and 12.0(14)ST)
• Random (select packet to export per statistical
  principles)
• Cisco IOS Software Releases 12.0(26)S, 12.2S(18),
  and 12.3(1)T
• Cisco 800, 1700, 1800, 2600, 2800, 3600, 3700,
  3800, 7200, and 7500 series routers
• Time-based
• Cisco C6500/7600 series Random and Time based
  sampling 12.1(13)E

72
Sampled Netflow CPU Reduction
73
Netflow Multiple Export Destinations
74
Performance Testing Conclusions

• Additional CPU utilization

Number of Active Flows Additional CPU Utilization
10,000 lt4
45,000 lt12
65,000 lt16

• NetFlow Data Export (single/dual)
• No significant impact
• NetFlow v5 versus v8 little or not impact
• NetFlow Feature Acceleration
• gt200 lines of ACLs and/or Policy Based-Routing
  (PBR)
• NetFlow versus Sampled NetFlow on the Cisco 12000
  Series Internet Routers
• 23 versus 3 (65,000 flows, 1100)

75
Performance TestingNetFlow Version 9

• Similar CPU and throughput numbers result from
  configuration of both NetFlow version 5 and 9
• No change in NetFlow performance after the
  addition of version 9
• Cisco IOS Software Releases 12.0(24)S, 12.2S, and
  12.3
• CPU is slightly higher immediately following
  initial boot up or configuration
• Caused by sending Template Flowsets to Collector

76
Reducing Performance Impact

• Reduce CPU and memory impact on the router,
  collector, or network
• Aging timers (router)
• Sampled NetFlow (router)
• Enable NetFlow Feature Acceleration (router)
• Flow Masks (only Cat6000/7600)
• Enable on specific sub-interface (upcoming router
  feature)
• Aggregation schemes (v8 on router or on
  collector)
• Filters (router or collector)
• Data Compression (collector)
• Increase collection bucket sizes (collector)
• Collector and router can be placed on the same
  LAN segment (network)

77
Netflow Deployment Rules of Thumb
78
Netflow Deployment Considerations
79
Cisco Netflow MIB
80
Netflow MIB applications

• Netflow Configuration
• Checking Netflow Configuration
• Monitoring and security
• export statistics
• protocol statistics
• top flows information (top talkers)

81
Netflow Mib Overview

• Defined groups of objects
• 1. cnfCacheInfo
• A group of objects related to cache information
  and configuration stored per cache configuration.
• 2. cnfExportInfo
• A group of objects related to Export
  configuration and information.
• 4. cnfExportStatistics
• Provides export statistics.
• 5. cnfProtocolStatistics
• Provides a summary of NetFlow cache statistics
  per protocol and port.
• 6. cnfExportTemplate
• Provides Template based Version 9 flow export
  information and statistic.
• 7. cnfTopFlows
• Provides top Netflow flows.

82
Netflow MIB Monitoring
83
Egress Netflow Accounting
84
Netflow and IPv6

• Collects IPv6 flow records
• Based on Netflow Version9
• Support or both ingress and egress traffic
• "Full NetFlow" i.e. non-sampled
• Data export is still IPv4
• Available in release 12.3(7)T

85
Netflow Summary

• Netflow is a mature Cisco IOS feature (in Cisco
  IOS since 1996)
• Netflow provides input for Accounting,
  Performance, Fault, Security, and Billing
  Applications
• Cisco has IETF and industry leadership
• Netflow v9 eases the exporting of additional
  fields
• A lot of new features have been added

86
SFlow

• sFlow is an industry standard technology for
  monitoring high speed switched networks,
  Junipers devices support it.
• similar to netflow
• NetStream from Huawei Company

• SFlow Packet
• Packet header (eg MAC,IPv4,IPv6,IPX,AppleTalk,TCP
  ,UDP, ICMP)
• Sample process parameters (rate, pool etc.)
• Input/output ports
• Priority (802.1p and TOS)
• VLAN (802.1Q)
• Source/destination prefix
• Next hop address
• Source AS, Source Peer AS
• Destination AS Path
• Communities, local preference
• User IDs (TACACS/RADIUS) for source/destination
• URL associated with source/destination
• Interface statistics (RFC 1573, RFC 2233, and
  RFC 2358)

87
Tools for Netflow

• Cisco NFC
• Arbor Peakflow
• Flow tools
• Ntop
• http//ww.ntop.org
• Etc.

88
Flow-tools

• Flow-tools is library and a collection of
  programs used to collect, send, process, and
  generate reports from NetFlow data.
• Can be used together on a single server or
  distributed to multiple servers for large
  deployments.
• The flow-tools library provides an API for
  development of custom applications for NetFlow
  export versions 1,5,6 and the 14 currently
  defined version 8 subversions.
• Version 9 is not supported now

89
Flow-tools utilities

• flow-capture - Collect, compress, store, and
  manage disk space for exported flows from a
  router.
• flow-cat - Concatenate flow files. Typically
  flow files will contain a small window of 5 or 15
  minutes of exports. Flow-cat can be used to
  append files for generating reports that span
  longer time periods.
• flow-fanout - Replicate NetFlow datagrams to
  unicast or multicast destinations. Flow-fanout
  is used to facilitate multiple collectors
  attached to a single router.
• flow-report - Generate reports for NetFlow data
  sets. Reports include source/destination IP
  pairs, source/destination AS, and top talkers.
  Over 50 reports are currently supported.
• flow-tag - Tag flows based on IP address or AS
  . Flow-tag is used to group flows by customer
  network. The tags can later be used with
  flow-fanout or flow-report to generate customer
  based traffic reports.
• flow-filter - Filter flows based on any of the
  export fields. Flow-filter is used in-line with
  other programs to generate reports based on flows
  matching filter expressions.
• flow-import - Import data from ASCII or cflowd
  format.
• flow-export - Export data to ASCII or cflowd
  format.

90
Flow-tools utilities( Cont.)

• flow-send - Send data over the network using the
  NetFlow protocol.
• flow-receive - Receive exports using the NetFlow
  protocol without storing to disk like
  flow-capture.
• flow-gen - Generate test data.
• flow-dscan - Simple tool for detecting some types
  of network scanning and Denial of Service
  attacks.
• flow-merge - Merge flow files in chronoligical
  order.
• flow-xlate - Perform translations on some flow
  fields.
• flow-expire - Expire flows using the same policy
  of flow-capture.
• flow-header - Display meta information in flow
  file.
• flow-split - Split flow files into smaller
  files based on size, time, or tags.

91
Configuration in Cisco Router

• R1(config) ip flow-export source Loopback0
• R1(config) ip flow-export version 5 origin-as
• R1(config) ip flow-export destination
  202.112.xx.xx 9800
• R1(config-if) ip route-cache flow

92
flow-capture

• Flow-tools most useful and important command
• flow-capture -w /flows/dat -m 255.255.248.0 -E5G
  0/10.0.0.1/9800
• Receive flows from the exporter at 10.0.0.1
  port 9800. Maintain 5 Gigabytes of flow files in
  /flows/dat. Mask the source and destination IP
  addresses contained in the flow exports with
  255.255.248.0.
• flow-capture -w /flows/dat 0/0/9800 -S5
• Receive flows from any exporter on port 9800.
  Do not perform any flow file space management.
  Store the exports in /flows/dat. Emit a stat
  log message every 5 minutes.

93
Flow-cat
94
Flow-print

• FreeBSD1 flow-print lt ft-v01.2006-09-02.13411408
  00
• srcIP dstIP
  prot sPort dPort octets pkts
• 202.204.79.253 202.204.239.227 6 4414
  1433 48 1
• 202.204.79.253 202.204.239.229 6 4450
  1433 96 2
• 202.204.79.253 202.204.239.240 6 4535
  1433 48 1
• 202.204.79.253 202.204.239.228 6 4443
  1433 48 1
• 202.204.79.253 202.204.239.233 6 4472
  1433 96 2
• 202.204.79.253 202.204.239.231 6 4461
  1433 48 1

95
Flow-stat
96
Flow-stat exam. 1

• flow-cat -p /flows/dat flow-stat
• IP packet size distribution
• 1-32 64 96 128 160 192 224 256 288
  320 352 384 416 448 480
• .000 .906 .029 .004 .002 .009 .001 .001 .004
  .027 .004 .000 .000 .000 .000
• 512 544 576 1024 1536 2048 2560 3072 3584
  4096 4608
• .000 .000 .001 .001 .012 .000 .000 .000 .000
  .000 .000
• Packets per flow distribution
• 1 2 4 8 12 16 20 24 28
  32 36 40 44 48 52
• .812 .157 .010 .013 .006 .001 .000 .000 .000
  .000 .000 .001 .000 .000 .000
• 60 100 200 300 400 500 600 700 800
  900 gt900
• .000 .001 .000 .000 .000 .000 .000 .000 .000
  .000 .000
• Octets per flow distribution
• 32 64 128 256 512 1280 2048 2816 3584
  4352 5120 5888 6656 7424 8192
• .000 .754 .183 .009 .012 .015 .014 .008 .004
  .002 .000 .000 .000 .000 .000

97
formats
98
Flow-stat exam. 2

• flow-cat -p /flows/dat flow-stat -f10 -S4
• Provide a report on top source/destination IP
  pairs sorted by octets
• Fields Total
• Symbols Disabled
• Sorting Descending Field 4
• Name Source/Destination IP
• src IPaddr dst IPaddr flows
  octets packets
• 202.204.192.1 10.20.0.12 1
  3720 12
  
• 202.204.192.1 10.20.0.8 3
  3128 11
  
• 202.204.192.1 10.20.0.9 2
  3269 11
  
• 202.204.193.1 64.84.7.4 1
  390 3
  
• 202.204.204.148 221.137.69.66 3
  144 3
  
• 216.186.143.246 202.204.227.118 1
  144 3
  
• 202.204.79.253 202.204.239.233 1
  96 2

99
Flow-scan
100
Netflow in CERNET-POP Traffic Statistics
101
Netflow in CERNET-POP PPS Statistics
102
Netflow in CERNET-POP Average Packet Size
Statistics
103
Netflow in CERNET-POP Protocol Statistics
104
Thank You!

• Most materials in this PPT is from network,
  thanks goes to the authors
• Any Questions?