MIRnet Administrative Data Analysis System MADAS

1
MIRnet Administrative Data Analysis System (MADAS)

• Greg Cole, Natasha Bulashova
• Friends Partners
• NCSA

2
Description

• System converts netflow data into structured data
  stored in a series of relational database tables
• System provides means of browsing summary
  statistics in graphic and table format
• A work in progress since 1998 first version in
  summer of 1999, second in fall of 2000 (for HPIIS
  review), third in February 2001

FOR MORE INFO...

• http//www.friends-partners.org/madasd/

3
Description
141.142.121.5193.233.46.2131303130UDP-Other55
63492979067306979067523 193.233.46.21141.142.
121.531303130UDP-Other55656929790673069790
67523 198.32.1.116193.233.82.3533271UDP-DNS1
4821979067480979067480 195.208.55.40194.81.150
.1676349980TCP-WWW2961979067547979067550 1
94.226.45.8193.0.72.165335432UDP-DNS263419
79067717979067721 195.208.55.40194.81.150.16863
50080TCP-WWW2961979067547979067550 195.208.
55.40194.81.158.1286149280TCP-WWW296197906
7677979067680 194.226.65.17128.61.81.129512702
1TCP-FTP63603979067720979067781 194.226.65.1
7128.61.81.1295127121TCP-FTP63603979067720
979067781 195.19.10.23818.72.1.202048ICMP11
5001979067753979067753 195.208.55.40194.81.150
.1696350180TCP-WWW2961979067547979067550 1
93.233.46.21141.142.121.531433128TCP-Other51
4861979067620979067620 141.142.121.5193.233.46
.2131283143TCP-Other5104319790676209790676
20 195.208.55.40194.81.158.1296149380TCP-WWW2
961979067677979067680 195.208.55.40194.81.150
.1706350280TCP-WWW2961979067547979067550 2
12.192.244.68193.0.0.193102453UDP-DNS17119
79067714979067714
4
Process

• Aggregate netflow data from Router
• Load into primary database tables
• Update summary tables
• Update heap tables
• Wait 10 minutes (and do it again)

5
Primary IPheaders table
1. row
ip_source
193.233.46.3 ip_destination 152.3.233.71
port_source 40C-45C port_destination 25
protocol TCP-SMTP packets 199
octets 285413 flows 1
timestart 2000-08-28 225021 timeend
1999-09-08 061809 channel BE
periodbegin 1999-09-08 061149
periodduration 600 keyid 2
domain_source 42 domain_dest
28 2. row
ip_source
195.208.220.5 ip_destination 128.148.55.233
port_source 80 port_destination 1K-2K
protocol TCP-WWW packets 11
octets 11128 flows 1
timestart 2000-08-29 183941 timeend
1999-09-08 062052 channel BE
periodbegin 1999-09-08 061149
periodduration 600 keyid 3
domain_source 9 domain_dest 125

• All network flows must meet minimum traffic
  threshold to be included in live database (for
  MIRnet, this is set to 10K)
• Lose 1 of total traffic volume but reduce 96 of
  records
• All data kept in archives
• Currently maintains 17,000,000 network flow
  records (June 1, 2001)

6
Primary DNSdata table
-----------------------------------------------
---------------------------------------
ip_address ip_name
createtime modifytime ip_domain
---------------------------------------------
-----------------------------------------
128.178.16.37 icpmac12.epfl.ch
20010110104036 00000000000000 6203
156.17.180.31 budm31.ar.wroc.pl
20010110104036 00000000000000 3232
62.32.36.134 ip134-tpas-1.ti.net.ge
20010110104032 00000000000000 6131
194.82.81.146 dyn081-146.stanmore.ac.uk
20010110104029 00000000000000 9760
194.83.11.34 gosh-atm.ex.ac.uk
20010110104026 00000000000000 9488
194.81.127.202 194.81.127.202
20010110104025 00000000000000 2
194.81.174.83 194.81.174.83
20010110104025 00000000000000 2
195.25.253.130 195.25.253.130
20010110104024 00000000000000 2
194.80.105.9 paul.cvcp.ac.uk
20010110104024 00000000000000 9456
194.81.127.113 194.81.127.113
20010110104023 00000000000000 2
131.114.187.5 endo1.endoc.med.unipi.it
20010110104023 00000000000000 6214
193.99.163.9 193.99.163.9
20010110104022 00000000000000 2
194.80.104.23 194.80.104.23
20010110104022 00000000000000 2
194.80.104.3 194.80.104.3
20010110104022 00000000000000 2
194.81.33.48 imb.hope.ac.uk
20010110104021 00000000000000 9526
---------------------------------------------
-----------------------------------------
Currently maintains 806,431 DNSdata IP records
(January 10, 2001)
7
Primary Domains table
1. row
domainid 715
domainname anl.gov latitude 41.858
longitude -88.017 domainlabel Argonne Natl Lab
createtime 20010103224037 modifytime
20001227191828 origin US shortlabel
Argonne Natl Lab location pdomainid 715
rdomainid 715 loccity Chicago locstate
IL loccountry United States orgclass US
Government,US Govt DOE worldclass North
America regionclass USA Great Lakes

• Heart and soul of MADAS system
• Adding new intelligence to this database
  enables entirely new classes of analysis
• Currently maintains 11,771 domain records
  (January 10, 2001)

2. row
domainid 948
domainname doe.gov latitude 38.892
longitude -77.017 domainlabel US Department of
Energy createtime 20001227170946 modifytime
20001227170946 origin US shortlabel
US-DOE location Washington, DC pdomainid
948 rdomainid 948 loccity Washington
locstate DC loccountry United States
orgclass US Government,US Govt DOE worldclass
North America regionclass USA Atlantic Central
8
Other Primary Tables
-----------------------------------------------
code country worldclass
-------------------------------------------
---- ?? Unknown
Unclassified AC Ascension Island
Other AD Andorra
Europe AE United Arab Emirates
Middle East AF Afghanistan(Islamic
St.) Middle East AG Antigua and
Barbuda North America AI Anguilla
Other AL Albania
Europe AM
Armenia Middle East AN
Netherland Antilles Other
---------------------------------------------
--

• IP Today (last 24 hours of ipheaders records)
• Country Codes
• Parent domains
• Color mappings

----------------------- parentid
parentname -----------------------
1308 ac.jp 3 ac.ru
959 ac.uk 986 edu.tw
6 free.net 735
nasa.gov 41 nlanr.net
4762 ircache.net 100 ras.ru
-----------------------
---------------- code value
---------------- ?? pink CA
lblue CH purple DE lbrown
DK green EE dgray FI
white FR cyan IL gold
IT lred JP dpink NL
lpurple NO gray Other lyellow
PL orange RU blue SE
lgray TW yellow UK marine
US lgreen ----------------
9
Capabilities

• With these tables (updated every 10 minutes), we
  can provide all sorts of live (and historical)
  traffic analysis between world regions,
  countries, country regions, cities, institutions,
  organizations, network protocols by year, month,
  day, hour, minute, . .

But . .
10
Need to use Indexed Summary Tables

• Database mirsum
• 8 tables updated live every 10 minutes
• 2 Heap (RAM-based) tables used for most live
  queries
• Pre-query optimizer selects best tables for
  current query

• Domain_date_proto
• Domain_date_proto_mm
• Domain_date
• Domain_date_mm
• Country_date_proto
• Country_date_proto_mm
• Country_date
• Country_date_mm
• Heap_domain_date_proto
• Heap_domain_date_proto_mm

11
A word about technologies

• No proprietary software
• Mysql for database
• PHP for query interface
• Web/CGI for stats interface
• Perl for code/CGI base
• DBI for interaction with Mysql
• GDGraph graphics libraries

12
Perl Code (object-oriented)

• Analysis that in original MADAS system took
  400-500 lines of perl code, now looks like

2 chart showing total
volume with breakdown by top countries my
self MADASCountry-new(
database "mirsum", table
"domain_date", variable
"origin_dest", imagemapcgi
"/cgi-bin/madas/printtable.pl",
imagemap 0, percent
1, graphtype "bars",
title1 "Total MIRnet Traffic Flow
by Destination Country", rh_input
\in)
self-set_title2("Period " .
self-get_timebegin . " - " .
self-get_timeend .
"")
self-doit()
13
Demonstration
14
(No Transcript)
15
(No Transcript)
16
(No Transcript)
17
World Regions (by country)
18
Countries (by domain)
19
US Regions
Russian Regions
20
(No Transcript)
21
(No Transcript)
22
(No Transcript)
23
(No Transcript)
24
(No Transcript)
25
(No Transcript)
26
Advantages

• Higher-level analysis of network usage (not just
  for engineers)
• System encourages exploration
• Better understanding of users and their
  applications
• Immediate feedback on traffic problems/issues

27
Future Plans

• Evaluate shared use of Domains and DNSdata tables
  (perhaps via LDAP)
• Standard monthly and quarterly reports of traffic
  utilization
• Monster query
• Project level accounting/analysis

more . . .
28
Future Plans (continued)

• Create always-running server to maintain data,
  provide instant stats, manage web
  site/interface
• Provide statistical analysis routines
• Create database to maintain all global settings
• Port-level analysis (looking for napster, etc.)

more . . .
29
Future Plans (continued)

• Explore integration/sharing with HPIIS projects
  (others?)
• Develop data maintenance applications for Domains
  database
• Develop world-map graphics applications

more . . .
30
Future Plans (continued)

• Develop partnerships analyses (looking at
  domain-domain and machine-machine partnerships)
• Add additional organizational classes (i.e.,
  US Govt DOE, University)
• Add state-level analyses
• Clean-up/refine Domains database

more . . .
31
Future Plans (continued)

• Add science classifiers and project
  identifiers to regular traffic flows
• Integrate this with database describing high
  performance network science applications
• Integrate back-end reporting with front-end
  reservation system

32
Future plans (continued)

• Authentication system for machine-level
  inquiry/analysis
• Device independent display of usage (for
  text-only, email, WAP devices)
• Handle IP address cache expiration problem
• Etc. . . .