The GrangeNet NOC

1
The GrangeNet NOC

• Greg Wickham

2
Contents

• Goals
• Design Features
• Architecture
• Implementation
• Future
• Conclusion

3
Contents

• Goals
• Design Features
• Architecture
• Implementation
• Future
• Conclusion

4
Goals

• Central repository where all information about
  the network is available
• Monitoring of the network including availability
  and utilisation
• Access control to ensure that the right people
  have access to the right data
• Leverage existing utilities where practical

5
Contents

• Goals
• Design Features
• Architecture
• Implementation
• Future
• Conclusion

6
Design Features

• Easy to use
• Intuitive
• Web Framework aids rapid development
• Modular Construction
• Extensible
• Security from the ground up
• Unified configuration
• Mimimise replication of data

7
Contents

• Goals
• Design Features
• Architecture
• Implementation
• Future
• Conclusion

8
Architecture

• Log file monitoring
• Security server
• Database backend
• Historical configurations

9
Architecture
10
Architecture
Database contains information on devices
connections interfaces as numbers prefixes
subnets members
db
11
Architecture
devices
logfiles
db
Devices connected to the network syslog to the NOC
12
Architecture
logwatcher
devices
logfiles
db
Logwatcher process monitors log files messages
from devices defined in the db are stored in the
db
13
Architecture
logwatcher
devices
logfiles
pancho
db
Logwatcher notices a configuration change pancho
is used to download the config and store in a RCS
configs
14
Architecture
datacollector
logwatcher
utilisation
devices
logfiles
pancho
db
Datacollector periodically examines SNMP counters
on device stores in RRDTool db
configs
15
Architecture
datacollector
tacasip
logwatcher
utilisation
devices
logfiles
pancho
db
Tacacs authentication is used for all Cisco
devices no shared enable secret
configs
16
Architecture
datacollector
tacasip
logwatcher
utilisation
devices
logfiles
pancho
db
confreport
configs
Configuration change reports are mailed to the
noc-manager nightly
17
Architecture
datacollector
tacasip
logwatcher
utilisation
devices
logfiles
pancho
db
confreport
configs
www
Web server is primary interactive portal
18
Architecture
datacollector
tacasip
logwatcher
utilisation
devices
logfiles
pancho
db
confreport
configs
weathermap
www
Weathermap renders graphic of total network
utilisation
19
Architecture
datacollector
tacasip
logwatcher
utilisation
devices
logfiles
pancho
db
Grapher renders historical graphs
confreport
configs
grapher
weathermap
www
20
Architecture
datacollector
tacasip
logwatcher
utilisation
devices
logfiles
pancho
db
cookie cutter configurations are generated
confreport
configs
grapher
weathermap
www
Config generator
21
Architecture
datacollector
tacasip
logwatcher
utilisation
devices
logfiles
pancho
db
confreport
configs
grapher
weathermap
www
Config generator
22
Contents

• Goals
• Design Features
• Architecture
• Implementation
• Future
• Conclusion

23
Leverage Open Source Software
24
The Perl Libraries

• Unified configuration format
• RCS
• Object Orientated HTML
• Dynamic Web Framework

25
Unified configuration format

• Similar to Apache
• ltNODE apac.routergt
• COORD-RELATIVE edge1.act
• COORD-SYSTEM polar
• polar coords are ( distance, angle )
• COORD 130,-45
• LABEL TEXT apac
• RADIUS 20
• lt/NODEgt

26
RCS

• Based on CPAN RCS
• Added support for -p option
• Diff into array

27
Object Orientated HTML

• Much easier to render dynamic HTML
• Automatic closure
• Inline back patching
• my html new HTMLOO
• html-gttable()-gttr()-gttd(Hello World)
• my a html-gta( href gt link )
• a-gtprint(Click here)

28
Dynamic Web Framework

• All web pages are rendered in real time
• Concept of Modules and Hierarchy
• Integrated cookie based authentication
• Integrated with LDAP
• Simplifies writing new modules, especially when
  the new module
• Is required to integrate with existing modules
• Needs authentication protection
• Can leverage existing library routines

29
The Database

• Uses PostgreSQL
• Contains information describing the network
• Who is connected?
• Where are they connected?
• What devices are connected?
• What interfaces are on the devices?
• Asset management

30
The Data Collector

• Extracts list of devices to monitor from database
• MIBS automatically monitored by defined device
  classes
• Automatically tracks hardware
• For interfaces interface name is used
• For others SNMP index is used (ie cpu etc)

31
The tacacs server

• Tacacs used as authentication protocol
• Backended into AARNet LDAP
• Stateful protocol - No shared enable password
• Accounting (console command logging)
• Supports some NAS features

32
The Logwatcher

• Monitors logged output from network devices
• Logged messages are screen for sanity (bad
  formatting etc.)
• Inserted into the database
• Any configuration change triggers a download of
  the configuration
• Store configuration files
• stripped of secrets
• Stored in RCS

33
The Web Server

• Uses Apache
• Perl embedded interpreter (mod_perl)
• mod_ssl / OpenSSL (Security)
• Mod_perl
• Increases
• Apache memory footprint
• Flexibility (access internal API)
• Performance (db access)
• Decreases
• CGI script startup time

34
The Configuration Reporter

• Every night all registered devices are checked
• Are running-config and startup-config the same?
• Report any changes made in the last 24 hours
• Summary report emailed to noc-manager

35
The weathermapper

• Goal At a glance network status
• Configuration file defines the weathermap
• Nodes Defines nodes and connections
• Map Defines nodes to draw
• Automatically generates
• Full map (all data rendered as requested)
• Quick mapping
• Background pre-rendered
• Foreground drawn over background
• Meta data for ISMAP clicking

36
The weathermapper

• Goal At a glance network status
• Configuration file defines the weathermap
• Nodes Defines nodes and connections
• Map Defines nodes to draw
• Automatically generates
• Full map (all data rendered as requested)
• Quick mapping
• Background pre-rendered
• Foreground drawn over background
• Meta data for ISMAP clicking

37
The weathermapper

• Embedded Nagios status

38
The weathermapper

• Throughput on link
• Percentage utilisation indicated by colour

39
The weathermapper

• Legend for colurs / percentage utilisation

40
The weathermapper

• Links down

41
The weathermapper

• Connection status highlight (In progress)

42
The Support Cast

• Suite of scripts that
• Collect SNMP data
• Check interface names against db
• Analyse Nagios status (scraping)

43
The Result

• Live demo
• http//noc.grangenet.net
• GrangeNet Survey

44
Contents

• Goals
• Design Features
• Architecture
• Implementation
• Future
• Conclusion

45
The Future

• Nagios? Where does it fit
• Automatic config generation (from db)
• Many similar queries to the same device arent
  efficient
• More reporting
• Link statistics
• Bandwidth utilisation
• Shim layer
• Scraping currently used
• Improved efficiency if APIs are well defined
• Ie Nagios

46
Contents

• Goals
• Design Features
• Architecture
• Applications
• Implementation
• Future
• Conclusion

47
Conclusion

• Perl Rapid application development
• Proven to be very useful
• Centralised data
• Describes the entire network
• The one source of information

48
Questions?

• greg.wickham_at_grangenet.net
• http//noc.grangenet.net