CPR and GAMMON Deployment Experiences'

1
CPR and GAMMONDeployment Experiences.

• Warren Matthews
• Georgia Institute of Technology.

2
Georgia Tech

• Campus
• Backbone group maintain 180 buildings, 1700
  switches, 55000 ports.
• Southern Crossroads gigapop (SOX)
• Provides connectivity for 20 Universities
  throughout the South East
• 10Gbps link to Abilene backbone.

3
Motivation

• Measurement Infrastructure typically means WAN
  monitoring
• But problems are LAN and host based
• Network Operations
• Single point of view
• Catastrophic failure is easier to detect
• Little quantitative data to troubleshoot
  performance problems

4
CPR

• Campus-wide Network Performance Monitoring and
  Recovery
• 50 hosts on Campus
• Collocated with switches in data closets
• Multiple views of the network
• Especially the users view
• Regular tests across campus network

5
Deployment
Boggs
ET
EDI
GLC
SOX
Savannah
GTL
Yamacraw
Class
505
TechSq Classroom
Weber
Servernet
Gateway Routers
Savant
Rich133
Savant44
Rich
Rich2
Cherry-Emerson
Daniel
GTRI
MARC
Arch
Admin
EST
Core Routers
811
Couch
845
MiRC
Skiles
SEB
SSC
Neely
NI
SI
MRDC
DMSmith
IBB
Habersham
Ajax
OHR
Sc-class
Lyman
Mason
French-class
FAB
OKeefe
King
GCATT
LAWN
Howey
French
OHR
Lib-class
Lyman
6
GAMMON

• Georgia Measurement and Monitoring
• State-wide initiative
• Distance Learning and Professional Education
  (DLPE)
• Valdosta State University, Armstrong Atlantic
  State University, Barrow County School System.

7
Other Deployments

• Local ISPs
• Major providers (Level3, Qwest, Charter)
• Residential (SpeedFactory, BellSouth, Charter,
  Cox)
• Global collaborations
• International focus in strategic plan
• Metz, Shanghai
• Leverage Global PMP Infrastructure and
  communicate using emerging standards

8
Deployment Experience

• On the whole, very smooth
• Control of Network
• Firewall
• Physical access
• Control of satellite server, DNS
• Warmly welcomed
• Help is on the way
• Pass the buck

9
Deployment Experience

• Donated hardware
• Cheap
• Flakey
• Scavenging of parts before donation
• GAMMON/International deployment faces more of the
  traditional challenges

10
Toolset

• No in-house development of measurement tools.
• Original plan also didnt include much
  visualization.
• Inconvenient to click through numerous graphs

11
Measurements

• Currently
• Smokeping - roundtrip time and graphs.
• Nagios - Services.
• Security - nessus and nmap.
• Also available
• Iperf (bwctl) - TCP throughput only.
• Pathchar, traceroute
• Coming soon
• NDT (central, distributed)
• Test bed for tools under development

12
Measurement Experiences

• Passive Monitoring of border traffic
• Bwctl.sox is outside GT.
• OWAMP
• NTP
• Alleged network problems are often issues with
  applications
• Added application monitoring using Nagios
• Information Security
• CPR as a platform for other groups, research

13
Analysis

• Analysis
• Create base-lines for historical comparison
• Use multiple view to detect location
• Middleware
• Alarm system
• Plateau detector (AMP), RIPE-TT
• How should we react to alarms?
• Troubleshooting guide

14
Analysis Experience

• Initial design was active monitoring
• Integrate with passive tools
• Wireless tools, NETI
• Provide a platform for localized netflow reports
• Initial design was to avoid looking at graphs
• Auto detect and alarms
• MyCPR

15
MyCPR

• Configurable alarms and graphs

16
Case Studies

• CPR has helped solve numerous issues
• Firewall
• Network slowness for file sharing
• Dropped sessions
• Not everything is a network issue

17
Experiences

• Dont assume anything
• If you dont measure, you dont know.
• Dont expect people to do what they should
• When one door closes, another one opens.

18
This is the end

• Contact
• Warren.matthews_at_oit.gatech.edu
• Project WebSite
• http//www.rnoc.gatech.edu/cpr