Sunday Folayan

1

• NOC Services and Applications
• Sunday Folayan
• Nishal Goburdhan
• Isatou Jah

2
What is Network Management?

• In order to operate a reliable service, the
  network must be managed according to a determined
  discipline, using a coherent structure of
  information management.
• Geoff Huston, ISP Survival Guide

3
What is a NOC?

• Network Operations Centre (NOC)
• Monitors and manages a service providers network
• Information about current, historical and planned
  availability of systems
• Network status and operational statistics
• Fault monitoring and management
• Engineers can coordinate their work through the
  NOC

4
Network Management - Components

• Parts of Network Management
• Configuration/Change management
• Performance/Accounting management
• Fault management
• Security management

5
Configuration Management

• Maintaining information relating to the design of
  the network and its current configuration
• Network State
• Record of network topology
• Static
• what is deployed
• where it is deployed
• how it is attached
• Who is responsible for it
• How do I contact them
• Dynamic
• operational status of the network elements

6
Configuration Management

• inventory management
• database of network elements
• history of changes problems
• directory maintenance
• all hosts applications
• nameserver database
• host and service naming coordination
• "Information is not information if you can't find
  it"

7
Configuration Management

• Operational Control of network
• Start/stop individual components
• Alter configuration of devices
• Load and save config versions
• Hardware/Software upgrades
• Methods of access
• SNMPGet / SNMPSet
• Out-of-Band access

8
RANCID

• RANCID - Really Awesome New Cisco confIg Differ
• Also works for IOS/CatOS/JunOS/...
• Open Source
• Runs on FreeBSD, Linux, OSX, even MS-Windows
• http//www.shrubbery.net/ (lots of other useful
  tools here too!)

9
RANCID

• Collections of scripts that run from cron and
  automate
• logging into routers
• capturing configuration
• highlighting configuration differences
• emailing the diffs to a mail list
• installing diffs into CVS

10
RANCID

• Track config changes
• Normal day-to-day
• Track hardware changes
• Wheres that spare?
• Track (I)OS changes
• Malicious changes ?
• What did your NOC do last night?
• Retrieve dead router configs.
• Track router crashes!!

11
RANCID aka Big Brother

• Announce changes to entire team - everybody
  starts looking out for anyone making random
  changes!
• If its broken, whats changed?
• Make it user friendly - CVSWeb

12
RANCID Sample Output

• !Slot 2/MBUS hvers 1.1
• !Slot 2/MBUS software 01.36 (RAM) (ROM version
  is 01.33)
• !Slot 2/MBUS 128 Mbytes DRAM, 16384 Kbytes
  SDRAM
• !
• - !Slot 6 1 Port Gigabit Ethernet
• - !Slot 6/PCA part 73-3302-03 rev C0 ver 3,
  serial CAB031216OL
• - !Slot 6/PCA hvers 1.1
• - !Slot 6/MBUS part 73-2146-07 rev B0 dev 0,
  serial CAB031112SB
• - !Slot 6/MBUS hvers 1.2
• - !Slot 6/MBUS software 01.36 (RAM) (ROM version
  is 01.33)
• !Slot 7 Route Processor
• !Slot 7/PCA part 73-2170-03 rev B0 ver 3,
  serial CAB024901SI
• !Slot 7/PCA hvers 1.4
• !Slot 7/MBUS part 73-2146-06 rev A0 dev 0,
  serial CAB02060044

13
RANCID Demo

• Demo of live RANCID system

14
RANCID Re-use

• More than configuration management.
• Cheap Asset Tracker/NMS
• UNIX script - easily extendible to other
  applications.
• Re-use login scripts
• Manage configuration changes
• Correlate syslog and RANCID using Simple Event
  Correlator (SEC)
• http//threebit.net/mail-archive/cisco-nsp/msg0005
  3.html

15
RANCID - Even More Uses

• Looking Glass software
• See Joe Abley and Stephen Stuart NANOG
  presentation
• http//www.nanog.org/mtg-0210/abley.html
• Consistency/Audit checks
• Generate DNS zone files
• Create Topographic maps

16
What is SNMP?

• Simple Network Management Protocol
• query - response system
• can obtain status from a device
• standard queries
• enterprise specific
• uses database defined in MIB
• management information base

17
What do we use SNMP for?

• query routers for
• in and out bytes per second
• CPU load
• uptime
• BGP peer session status
• query hosts for
• network status
• Message queues
• Web traffic
• Squid proxy load

18
SNMP Exercise
19
Configuration Management
SNMP driven display
husc6
mghgw
wjh12
harvard
generali
talcott
wjhgw1
harvisr
huelings
geo
pitirium
nngw
nnhvd
oitgw1
sphgw1
lmagw1
dfch
tch
tch
20
Performance Management

• A Consistent level of network performance
• Data collection
• interface stats
• throughput
• error rates
• usage
• percent availability
• Data analysis for performance metrics and trends
• Establishment of performance thresholds
• Capacity planning and deployment

21
Importance of Network Statistics

• Accounting
• Troubleshooting
• Long-term trend analysis
• Capacity Planning
• Two different types
• active measurement
• passive measurement
• Management Tools have statistical functionality

22
MRTG
23
MRTG and MRTG Exercise
24
Netflow

• Cisco developed - 1996
• Initially a mechanism for forwarding packets
• No longer - Now, primarily used for
• Accounting/Billing
• Network planning
• Peering arrangements
• Traffic engineering
• Security monitoring

25
Netflow

• Netflow packet typically contains
• IP SRCDST
• Port SRCDST
• Protocol information
• TOS byte (DSCP)
• Input logical interface (ifIndex)
• Extendible (IOS capable)
• AS / VRF / ...

26
Netflow

• Uses CPU and memory!
• Export Netflow to external collector (or use
  online on router)
• http//www.splintered.net/sw/flow-tools/
• Router summarisation possible
• Netflow V5 is most commonly used
• http//www.cisco.com/go/netflow

27
Netflow

• Only works on inbound traffic
• Unidirectional flow
• Shows transit (traffic through) and to the
  router.
• Enabled by
• ip route-cache flow
• ip flow ingress (new syntax)
• Output seen with
• show ip cache verbose flow

28
Netflow Example

• From your workstation
• ping 196.200.220.1
• On your router
• router conf t
• router(config) int fa0/0
• router(config-if) ip flow ingress
• router show ip cache flow

29
Netflow Example (cont).

• Whats missing?
• (Why are the flows only in 1 direction?)
• How do you fix it ?
• Now repeat the BCP38 packet spoofing exercise,
  but track the bogus packets with Netflow. Pay
  attention to what happens when uRPF is enabled.

30
Netflow examples

• Top ten lists (or top five)

Top 5 AS's based on number of bytes
srcAS dstAS pkts
bytes 6461 237 4473872
3808572766 237 237 22977795
3180337999 3549 237 6457673
2816009078 2548 237 5215912
2457515319
Top 5 Nets based on number of bytes
Net Matrix ---------- number of net
entries 931777 SRCNET/MASK DSTNET/MASK
PKTS BYTES 165.123.0.0/16
35.8.0.0/13 745858 1036296098
207.126.96.0/19 198.108.98.0/24 708205
907577874 206.183.224.0/19 198.108.16.0/22
740218 861538792 35.8.0.0/13
128.32.0.0/16 671980 467274801
Top 10 Ports input
output port packets bytes
packets bytes 119 10863322
2808194019 5712783 427304556 80
36073210 862839291 17312202 1387817094 20
1079075 1100961902 614910
62754268 7648 1146864 419882753
1147081 414663212 25 1532439 97294492
2158042 722584770
31
Accounting Management

• What do you account for?
• Use of the network and the services it provides
• Types of accounting data
• RADIUS/TACACS accounting data from Access servers
• Interface statistics
• Protocol statistics
• Accounting Data affects Business Models
• Bill on usage?
• Flat-rate billing?

32
Fault Management

• Identify the fault
• Regular polling of network elements
• Isolate the fault
• Diagnosis of the network components
• Respond to the fault
• Allocate resources to resolve the fault
• Priority scheduling
• Technical/management escalation
• Resolve the fault
• notification

33
Fault Management - systems

• reporting mechanism
• link to NOC
• notify on-call personnel
• setup control alarm procedures
• repair/recovery procedures
• ticket system

34
Fault Management - Fault Detection

• Who notices a problem with the network?
• Network Operations Center w/ 24x7 operations
  staff
• open trouble ticket to track problem
• preliminary troubleshooting
• Assign engineer to problem or escalate ticket
  status
• Customer call
• Other ISPs

35
Fault Management - Fault Detection (con)

• How can you tell if there is a problem with the
  network?
• Network Monitoring Tools
• common utilities
• ping
• Traceroute
• Ethereal
• Snmp
• Monitoring Systems
• NOCol
• Big Brother
• Nagios
• HP Openview, etc
• Report state or unreachability
• detect node down
• routing problems

36
Fault Management - Ticket System

• Very Important!
• Need mechanism to track
• failures
• current status of outage
• carrier tickets

37
Fault ManagementTicket System

• system provides for
• short term memory communication
• scheduling and work assignment
• referrals and dispatching
• oversight
• statistical analysis
• long term accountability

38
Fault Management - Ticket Usage

• create a ticket on ALL calls
• create a ticket on ALL problems
• create a ticket for ALL scheduled events
• copy of ticket mailed to reporter and mailing
  list(s)
• all milestones in resolution of problem maintain
  the same ticket
• ticket stays "open" until problem resolved
• Ticket reporter determines that ticket should be
  closed.

39
Fault Management - Ticket Example

• Sample opening ticket

Subject Serial Number Fix sshd on E2
instructor machines 6 Area Queue none
afnog-noc Requestors Owner pfs_at_cisco.com
inst Status Last User Contact resolved
Wed May 10 170221 2006 (12 hr ago) Current
Priority Final Priority 1 1 Due No date
assigned Last Action Wed May 10 170221 2003
(12 hr ago) Created Mon May 8 140808 2003 (2
days ago)
40
Exercise Ticket System

• RT is already installed on http//e2-noc.ws.afnog.
  org
• Create tickets to track network occurrences as
  they occur - network failures will be provided -)

41
Fault Management - typical failures

• Node unpingable
• no ip connectivity to router
• possible reasons
• serial link down
• call telco
• router down/hardware problem
• call engineer
• routing problem
• troubleshoot with traceroute
• routeviews machine

42
Security Management Dos Donts

• Dont leave things that are likely to be
  interesting to mice lying on the kitchen table
  overnight
• Plug the holes that mice are using to get into
  the house
• Dont provide places within the house for mice to
  build nests
• Set traps along walls where you often see mice
  out of the corner of your eye
• Check the traps daily to rebait them and to
  dispose of squashed mice. Full traps dont catch
  mice, and they smell
• Avoid using commercial bait-and-kill poisons.
  Traditional snap traps are best.
• Get a cat!

43
Security Management - Tools

• security tools
• cops - host configuration checker (www.cert.org)
• swatch - email reports of activity on machine
• Tcpwrappers log connections, restrict access
• ssh/skey crypto authentication and
  communications
• Tripwire monitor changes to system files
• Keep up to date with security information
• bug reports
• CERT advisories mailing list
• http//www.cert.org./contact_cert/certmaillist.htm
  l
• bug fixes
• intruder alerts

44
Security Management Good Practice

• reporting procedure for security events
• e.g. break-ins
• abuse email address for customers to report
  complaints (abuse_at_your-isp.net)
• control internal and external gateways
• control firewalls (external and internal)
• security log management
• centralized logging host
• Stealth logger, so it cannot be compromised

45
How do I manage my network?

• Which tools should I use? What do I really need?
• Keep it simple!
• Need to consider engineers working remotely
• Dont want to spend too much time maintaining the
  tool (it should be helping you!)
• Different tools for NOC and engineers
• Different tools for statistics
• RELIABILITY!

46
References

• http//www.merit.edu/ipma/docs/isp.html
• http//www.nanog.org
• http//www.caida.org
• http//www.nlanr.net
• http//www.cisco.com
• http//www.amazing.com/internet/
• http//www.isp-resource.com/
• http//www.merit.edu/ipma
• http//www.ripe.net

47
More Tools!

• http//www.caida.org/Tools/
• OC3Mon/Coral
• http//www.merit.edu/ipma
• RouteTracker
• IRRj
• ASExplorer
• http//www.geektools.com/
• http//www.merit.edu/ipma/tools/other.html

48
SNMP Tool references

• MON - http//www.kernel.org/software/mon/
• NOCol - ftp//ftp.navya.com/pub/vikas/nocol.tar.gz
  
• Sysmon - ftp//puck.nether.net/pub/jared
• Rover - http//www.merit.edu/rover
• Concord - http//www.concord.com
• http//www.merit.net/netscarf