Logfile Visualization The Beauty of Graphs BCS 2006, Jakarta

1
Logfile Visualization The Beauty of GraphsBCS
2006, Jakarta

• Raffael Marty, GCIA, CISSPManager Solutions _at_
  ArcSightAugust 30th, 2006

2
Raffael Marty, GCIA, CISSP

• Enterprise Security Management (ESM) specialist
• Strategic Application Solutions _at_ ArcSight, Inc.
• Intrusion Detection Research _at_ IBM Research
• See http//thor.cryptojail.net
• IT Security Consultant _at_ PriceWaterhouse Coopers
• Open Vulnerability and Assessment Language (OVAL)
  board member
• Passion for Visual Security Event Analysis

3
Table Of Contents

• Introduction
• Graphing Basics
• Graph Use Cases
• Visual Analysis Process
• AfterGlow
• Firewall Log Visualization

4
Introduction
5
Disclaimer
IP addresses and host names showingup in event
graphs and descriptions were obfuscated/changed.
The addresses are completely random and any
resemblancewith well-known addresses or host
namesare purely coincidental.
6
A Picture is Worth a Thousand Log Entries
Detect the Expected Discover the Unexpected
Reduce Analysis and Response Times
Make Better Decisions
7
Text or Visuals?

• What would you rather look at?

Jun 17 094230 rmarty ifup Determining IP
information for eth0... Jun 17 094235 rmarty
ifup failed no link present. Check cable? Jun
17 094235 rmarty network Bringing up interface
eth0 failed Jun 17 094238 rmarty sendmail
sendmail shutdown succeeded Jun 17 094238
rmarty sendmail sm-client shutdown succeeded Jun
17 094239 rmarty sendmail sendmail startup
succeeded Jun 17 094239 rmarty sendmail
sm-client startup succeeded Jun 17 094339
rmarty vmnet-dhcpd DHCPINFORM from
172.16.48.128 Jun 17 094542 rmarty last message
repeated 2 times Jun 17 094547 rmarty
vmnet-dhcpd DHCPINFORM from 172.16.48.128 Jun 17
095602 rmarty vmnet-dhcpd DHCPDISCOVER from
000c29b7b247 via vmnet8 Jun 17 095603
rmarty vmnet-dhcpd DHCPOFFER on 172.16.48.128 to
000c29b7b247 via vmnet8 Jun 17 095603
rmarty vmnet-dhcpd DHCPREQUEST for 172.16.48.128
from 000c29b7b247 via vmnet8 Jun 17 095603
rmarty vmnet-dhcpd DHCPACK on 172.16.48.128 to
000c29b7b247 via vmnet8 Jun 17 100003
rmarty crond(pam_unix)30534 session opened for
user root by (uid0) Jun 17 100010 rmarty
crond(pam_unix)30534 session closed for user
root Jun 17 100102 rmarty crond(pam_unix)30551
session opened for user root by (uid0) Jun 17
100107 rmarty crond(pam_unix)30551 session
closed for user root Jun 17 100502 rmarty
crond(pam_unix)30567 session opened for user
idabench by (uid0) Jun 17 100505 rmarty
crond(pam_unix)30567 session closed for user
idabench Jun 17 101305 rmarty portsentry4797
attackalert UDP scan from host
192.168.80.19/192.168.80.19 to UDP port 192 Jun
17 101305 rmarty portsentry4797 attackalert
Host 192.168.80.19/192.168.80.19 is already
blocked Ignoring Jun 17 101409 rmarty
portsentry4797 attackalert UDP scan from
host 192.168.80.8/192.168.80.8 to UDP port
68 Jun 17 101409 rmarty portsentry4797
attackalert Host 192.168.80.8/192.168.80.8 is
already blocked Ignoring Jun 17 101409 rmarty
portsentry4797 attackalert UDP scan from
host 192.168.80.8/192.168.80.8 to UDP port
68 Jun 17 101409 rmarty portsentry4797
attackalert Host 192.168.80.8/192.168.80.8 is
already blocked Ignoring Jun 17 102130 rmarty
portsentry4797 attackalert UDP scan from
host 192.168.80.8/192.168.80.8 to UDP port
68 Jun 17 102130 rmarty portsentry4797
attackalert Host 192.168.80.8/192.168.80.8 is
already blocked Ignoring Jun 17 102840 rmarty
vmnet-dhcpd DHCPDISCOVER from 000c29b7b247
via vmnet8 Jun 17 102841 rmarty vmnet-dhcpd
DHCPOFFER on 172.16.48.128 to 000c29b7b247
via vmnet8 Jun 17 102841 rmarty vmnet-dhcpd
DHCPREQUEST for 172.16.48.128 from
000c29b7b247 via vmnet8 Jun 17 102845
rmarty vmnet-dhcpd DHCPACK on 172.16.48.128 to
000c29b7b247 via vmnet8 Jun 17 103047
rmarty portsentry4797 attackalert UDP scan
from host 192.168.80.8/192.168.80.8 to UDP port
68 Jun 17 103047 rmarty portsentry4797
attackalert Host 192.168.80.8/192.168.80.8 is
already blocked Ignoring Jun 17 103047 rmarty
portsentry4797 attackalert UDP scan from
host 192.168.80.8/192.168.80.8 to UDP port
68 Jun 17 103047 rmarty portsentry4797
attackalert Host 192.168.80.8/192.168.80.8 is
already blocked Ignoring Jun 17 103528 rmarty
vmnet-dhcpd DHCPINFORM from 172.16.48.128 Jun 17
103531 rmarty vmnet-dhcpd DHCPINFORM from
172.16.48.128 Jun 17 103851 rmarty vmnet-dhcpd
DHCPREQUEST for 172.16.48.128 from
000c29b7b247 via vmnet8 Jun 17 103852
rmarty vmnet-dhcpd DHCPACK on 172.16.48.128 to
000c29b7b247 via vmnet8 Jun 17 104235
rmarty vmnet-dhcpd DHCPINFORM from
172.16.48.128 Jun 17 104238 rmarty vmnet-dhcpd
DHCPINFORM from 172.16.48.128
8
Graphing Basics
9
How To Generate A Graph
... Normalization ...
Device
Event Visualizer
Parser
Jun 17 094230 rmarty ifup Determining IP
information for eth0... Jun 17 094235 rmarty
ifup failed no link present. Check cable? Jun
17 094235 rmarty network Bringing up interface
eth0 failed Jun 17 094238 rmarty sendmail
sendmail shutdown succeeded Jun 17 094238
rmarty sendmail sm-client shutdown succeeded Jun
17 094239 rmarty sendmail sendmail startup
succeeded Jun 17 094239 rmarty sendmail
sm-client startup succeeded Jun 17 094339
rmarty vmnet-dhcpd DHCPINFORM from
172.16.48.128 Jun 17 094542 rmarty last message
repeated 2 times Jun 17 094547 rmarty
vmnet-dhcpd DHCPINFORM from 172.16.48.128 Jun 17
095602 rmarty vmnet-dhcpd DHCPDISCOVER from
000c29b7b247 via vmnet8 Jun 17 095603
rmarty vmnet-dhcpd DHCPOFFER on 172.16.48.128 to
000c29b7b247 via vmnet8 NH
Visual
Log File
10
Visual Types
TreeMaps
Link Graphs
AfterGlow 1.x - Perl
AfterGlow 2.0 - JAVA
11
Link Graph Configurations
Raw Event 119232 RPC portmap UDP proxy
attempt Classification Decode of an RPC
Query Priority 2 06/04-155628.219753
192.168.10.9032859 -gt 192.168.10.255111 UDP
TTL64 TOS0x0 ID0 IpLen20 DgmLen148 DF Len
120 Different node configurations
192.168.10.90 RPC portmap 192.168.10.255
192.168.10.90 192.168.10.255 111
192.168.10.90 32859 111
RPC portmap 192.168.10.90 192.168.10.255
12
Tree Maps
All Network Traffic
13
Tree Maps
20
80
Configuration (Hierarchy)
Protocol
14
Tree Maps
Configuration (Hierarchy)
Protocol -gt Service
15
Graph Use Cases
16
Graph Use-Cases
Situational Awareness Dashboard
17
Suspicious Activity?
Graph Use-Cases
18
Network Scan
Graph Use-Cases
19
Port Scan ?
Graph Use-Cases

• Port scan or something else?

20
PortScan
Graph Use-Cases
SIP
DIP
DPort
21
Telecom Malicious Code Propagation
Graph Use-Cases
22
Email Relays
Graph Use-Cases
23
Visual Analysis Process
24
Event Feedback Loop
Visual Analysis Process
Feb 18 133915.598491 rule 71/0(match) pass in
on xl0 195.27.249.139.63263 gt 195.141.69.42.80
S 492525755492525755(0) win 32768 ltmss
1460,nop,wscale 0,nop,nop,timestamp 24053 0gt
(DF) Feb 18 133915.899644 rule 71/0(match)
pass in on xl0 195.27.249.139.63264 gt
195.141.69.42.80 S 875844783875844783(0) win
32768 ltmss 1460,nop,wscale 0,nop,nop,timestamp
24054 0gt (DF)
Device
Normalization
195.27.249.139,195.141.69.42,80 195.27.249.139,195
.141.69.42,80
Filter
195.27.249.139,195.141.69.42,80
Service stopped
Correlation
Visual
25
Event Feedback Loop
Visual Analysis Process
Visual Detection
Real-timeData Processing
Forensic and Historical Analysis
Visual Investigation
Creation of new Filtersand Correlation Components
Assign to Content Author
26
Visual Analysis Process
Visual Detection
Beginning of Analysts shift
27
Visual Analysis Process
Visual Detection
Scanning activity is displayed
Firewall Blocks
Scan Events
28
Visual Analysis Process
Visual Investigation
29
Visual Analysis Process
Defining New Content
3. Open a ticket for Operations to quarantine
and clean infected machines
30
AfterGlow

• http//afterglow.sourceforge.net
• Two Versions
• AfterGlow 1.x Perl for Link Graphs
• AfterGlow 2.0 Java for TreeMaps
• Collection of Parsers
• pf2csv.pl BSD PacketFilter (pf)
• tcpdump2csv.pl tcpdump 3.9
• sendmail2csv.pl Sendmail transaction logs

31
AfterGlow
afterglow.sourceforge.net
32
AfterGlowParsers

• tcpdump2csv.pl
• Takes care of swapping response source and
  targets
• tcpdump -vttttnnelr /tmp/log.tcpdump
  ./tcpdump2csv.pl
• sendmail_parser.pl
• Reassemble email conversations
• pf2csv.pl
• Parsing OpenBSD pf output

"sip dip sport"
Jul 24 210116 rmarty sendmail17072
j6P41Gqt017072 fromltroot_at_localhost.localdomaingt,
size650, class0, nrcpts1, Jul 24 210116
rmarty sendmail17073 j6P41Gqt017072 toram,
ctladdrltroot_at_localhost.localdomaingt (0/0),
delay000000, xdelay000000, mailerlocal,
pri30881, dsn2.0.0, statSent
33
AfterGlow 1.x - Perl

• Supported graphing tools
• GraphViz from ATT (dot, neato, circo, twopi)
  http//www.graphviz.org
• LGL (Large Graph Layout) by Alex
  Adaihttp//bioinformatics.icmb.utexas.edu/lgl/

Parser
AfterGlow
Grapher
Graph LanguageFile
CSV File
34
AfterGlow 1.xFeatures

• Generate Link Graphs
• Filtering Nodes
• Based on name
• Based on number of occurrences
• Fan Out Filtering
• Coloring
• Edges
• Nodes
• Clustering

35
AfterGlow 1.xHello World
Input Data
Command
a,b a,c b,c d,e
cat file ./afterglow c simple.properties t
\ neato Tgif o test.gif
simple.properties
color.sourcegreen if (fields0 ne d)
color.targetblue if (fields1 ne e)
Output
d
color.sourcered
colorgreen
e
a
36
AfterGlow 1.xProperty File Color Definition

• Coloring
• color.sourceeventtargetedge ltperl
  expression returning a color namegt
• Array _at_fields contains input-line, split into
  tokens
• color.eventred if (fields1 /192\..)
• Filter nodes with invisible color
• color.targetinvisible if (fields0 eq IIS
  Action)

37
AfterGlow 1.xProperty File - Clustering

• Clustering
• cluster.sourceeventtarget ltperl expression
  returning a cluster namegt

38
AfterGlow 2.0 - Java

• Command line arguments
• -h help
• -c file property file
• -f file data file

Parser
AfterGlow - Java
CSV File
39
AfterGlow 2.0 Example

• Data
• Launch
• ./afterglow-java.sh c afterglow.properties

AfterGlow - JAVA 2.0 Properties File File
to load file.name/home/ram/afterglow/data/sample.
csv Column Types (default is STRING), start
with 0! Valid values STRING
INTEGER CATEGORICAL column.type.count4 col
umn.type0.column0 column.type0.typeINTEGER c
olumn.type1.column1 column.type1.typeCATEGOR
ICAL column.type2.column2 column.type2.typeC
ATEGORICAL column.type3.column3 column.type3.
typeCATEGORICAL Size Column (default is
0) size.column0 Color Column (default is
0) color.column2
Target System Type,SIP,DIP,User,Outcome Developmen
t,192.168.10.1,10.10.2.1,ram,failure VPN,192.168.1
0.1,10.10.2.1,ram,success Financial
System,192.168.20.1,10.0.3.1,drob,success VPN,192.
168.10.1,10.10.2.1,ram,success VPN,192.168.10.1,10
.10.2.1,jmoe,failure Financial System,192.168.10.1
,10.10.2.1,jmoe,success Financial
System,192.168.10.1,10.10.2.1,jmoe,failure
40
AfterGlow 2.0Output
41
AfterGlow 2.0Interaction

• Left-click
• Zoom in
• Right-click
• Zoom all the way out
• Middle-click
• Change Coloring to currentdepth
• (Hack Use SHIFT for leafs)

42
AfterGlowFirewall Log Analysis Example
Input (pflog)
Feb 18 133915.598491 rule 71/0(match) pass in
on xl0 195.27.249.139.63263 gt 195.141.69.42.80
S 492525755492525755(0) win 32768 ltmss
1460,nop,wscale 0,nop,nop,timestamp 24053 0gt
(DF) Feb 18 133915.899644 rule 71/0(match)
pass in on xl0 195.27.249.139.63264 gt
195.141.69.42.80 S 875844783875844783(0) win
32768 ltmss 1460,nop,wscale 0,nop,nop,timestamp
24054 0gt (DF)
Command
cat pflog pf2csv.pl sip dip dport
Output
195.27.249.139,195.141.69.42,80 195.27.249.139,195
.141.69.42,80
AfterGlow Input
Visualization
cat pflog pf2csv.pl sip dip dport
\ afterglow c properties neato Tgif o foo.gif
43
AfterGlowFirewall Log Analysis Example
Command
cat log grep pass_in ./afterglow c
properties d dot Tgif o foo.gif
44
Summary

• Quickly Visualize Log Files
• Understand Relationships
• Find Outliers
• Spot suspicious activity
• Visual Data Analysis Process
• AfterGlow
• Firewall Log File Analysis

Dont Read Log Files Visualize Them!!
45
THANKS!raffy_at_arcsight.com
Raffael Marty
45
DefCon 2006 Las Vegas