Sniffing network traffic in Python

1
Sniffing network traffic in Python

• Jose Nazario, Ph.D. ltjose_at_monkey.orggt

2
Why Python?

• Interpreted language
• Bound to be slower than C
• Rapid development
• Easy data structure use
• Fewer LoC per tool
• Easy to manipulate strings
• http//www.python.org/

3
Marrying Python and Sniffing

• Librares in C
• Often SWIGged, exported to Python
• pcap, dnet, nids
• Modules
• pypcap/pcappy pcap for python
• dpkt packet deconstruction library
• libdnet packet construction library (has python
  bindings in the distribution)
• pynids connection reassembly tool

4
libnids reassemble IP streams
NIDS E box (event generation box) Userland
TCP/IP stack Based on Linux 2.0.36 IP stack Uses
libpcap, libnet internally IP fragment reassembly
5
Userland
Kernel
IP stack
6
Userland
Kernel
IP stack
Libnids
IP stack
7
libnids Basics

• Initialize
• nids_init()
• Register callbacks
• nids_register_tcp()
• nids_regster_ip()
• nids_regiser_udp()
• Run!
• nids_run()
• React
• nids_kill_tcp()

8
nids_run()
UDP callback
TCP callback
IP callback
TCP stream object - TCP state - client data -
server data - source IP, port - dest IP, port
- seq, ack, etc
UDP packet - source IP, port - dest IP, port
- UDP payload
IP packet - struct IP packet - contains upper
layers
9
libnids TCP states

• NIDS_JUST_ESTABLISHED
• New TCP connected state (3WHS)
• Must set stream-gtclient,server.collect1 to get
  stream payload collected
• NIDS_DATA
• Data within a known, established TCP connection
• NIDS_RESET, NIDS_CLOSE, NIDS_TIMED_OUT
• TCP connection is reset, closed gracefully, or
  was lost

libnids doesnt expose SYN_SENT, FIN_WAIT, etc
10
pynids Basics

• Event driven interface (nids_run(), nids_next())
• TCP stream reassembly
• TCP state exposure
• Creates a TCP object
• Holds addresses, data, etc
• UDP and IP packet reassembly

11
Basic pynids Steps

• Initialize
• nids_init()
• Establish parameters
• nids.param(attribute, value)
• Register callbacks
• nids.register_tcp(handleTcp)
• def handleTcp(tcp)
• Go!
• nids_run()
• while 1 nids_next()

12
pynids Order of Operations

• Packets come in
• TCP?
• State exist? Create state or reuse state
• Append data
• Process based on state in callback
• UDP or IP?
• Use handler, pass packet in
• You process in callback

13
Code Example (Python)

• import nids
• lthandleTcpStreamgt
• def main()
• nids.param("scan_num_hosts", 0)
• if not nids.init()
• print "error -", nids.errbuf()
• sys.exit(1)
• nids.register_tcp(handleTcpStream)
• try nids.run() loop forever
• except KeyboardInterrupt
• sys.exit(1)

14
Code Example (Python) cont

• def handleTcpStream(tcp)
• if tcp.nids_state nids.NIDS_JUST_EST
• if dport in (80, 8000, 8080)
• tcp.client.collect 1
• tcp.server.collect 1
• elif tcp.nids_state nids.NIDS_DATA
• tcp.discard(0)
• elif tcp.nids_state in end_states
• print "addr", tcp.addr
• may be binary
• print "To server, tcp.server.data
• print "To client, tcp.client.data

15
Code Example (C)

• int main(int argv, char argv)
• if (nids_init() 0)
• err(1, error, s, nids_errbuf)
• nids_register_tcp(handleTcp)
• nids_run()
• exit(0)

16
Code Example (C), cont

• int handleTcp(struct tcp_stream tcp)
• switch (tcp-gtnids_state)
• case NIDS_JUST_EST
• if ((tcp-gtaddr.dest 80)
• (tcp-gtaddr.dest 8000)
• (tcp-gtaddr.dest 8080)
• tcp.server.collect 1
• tcp.client.collect 1
• break
• case NIDS_DATA
• nids_discard(tcp, 0)
• break
• case NIDS_CLOSE
• case NIDS_RESET
• case NIDS_TIMED_OUT
• printf(((s, d), (s, d))\n,
  inet_ntoa(tcp-gtsaddr), tcp.srce,
• inet_ntoa(tcp-gtdaddr), tcp.dest)

About the same LoC, until we start string
manipulation
17
VersionDetect

• Small python tool
• Reports on headers
• Fully passive
• Support for SSH (client, server), WWW (client,
  server), and SMTP clients
• Motivation coordinate data collection with TCP
  stack fingerprinting

63.236.16.161 SymbianOS 6048 (on Nokia
7650?) www 80/tcp
63.236.16.161 80 Microsoft-IIS/6.0
18
VersionDetect Output

• 192.168.1.7 22 SSH-2.0-OpenSSH_3.5
• 192.168.1.101http Mozilla/5.0 (X11 U
  OpenBSD i386 en-
• US rv1.5a) Gecko/20031030 Mozilla
  Firebird/0.6.1
• 168.75.65.85 80 Microsoft-IIS/5.0
• 165.1.76.60 80 Netscape-Enterprise/3.6
  SP2
• 168.75.65.69 80 Microsoft-IIS/5.0
• 168.75.65.87 80 Microsoft-IIS/5.0
• 69.28.159.7 80 ZEDO 3G
• 198.65.148.234 80 Apache/1.3.29 (Unix)
  PHP/4.3.3
• 216.150.209.231 80 Apache/1.3.31 (Unix)
• 212.187.153.30 80 Apache/1.3.31 (Unix)
• 212.187.153.37 80 Apache/1.3.31 (Unix)
• 212.187.153.32 80 thttpd/2.25b 29dec2003
• 64.209.232.207 80 Apache/1.3.27 (Unix)
• mod_perl/1.27
• 216.239.39.99 80 CAFE/1.0

19
http-graph

• Small, passive python tool
• Examines HTTP request header
• GET /blog/styles-site.css HTTP/1.1
• Host www.jackcheng.com
• User-Agent Mozilla/5.0 (X11 U OpenBSD i386
  en-US rv1.5a) Gecko/20031030 Mozilla
  Firebird/0.6.1Accept text/css,/q0.1
• Referer http//www.jackcheng.com/blog/archives/20
  04/12/ipod_rumors.html

20
http-graph

• Directed graph history of browsing
• Reconstructs graph from referrer and URL in the
  header
• Referrer Request
• Lets you view your history as you took it
• Shows natural hubs of information
• See also http//www.uiweb.com.nyud.net8090/issue
  s/issue37.htm

21
Displaying http-graph Output

• Writes a small dot file
• dot part of graphviz tool
• Use neato to graph
• Output formats SVG, PS, PDF, image map
• Can make fully interactive!

22
Example http-graph Output
23
Grabbing Data with pynids

• tcp.server, client.data and just strings
• Any string operations will work
• Searching
• if HTTP/1.0 in tcp.client.data
• Regular Expression searches
• if re.search(HTTP/1.10, tcp.client.data)
• Rewriting
• string.replace(req, GET HTTP/1.0, , 1)

24
More Fun!

• Privacy invasion
• Snarf mail
• Log conversations
• IRC, AIM, etc
• Steal files
• FTP, P2P apps, HTTP downloads
• Disrupt sessions
• tcp.kill()

New dsniff is written in Python
25
flowgrep

• Marries sniffing with regular expressions
• A lot like ngrep, tcpkill, and dsniff
• Logs the whole connection, not just a packet
• Look for data in streams using regular
  expressions
• Log or kill selected streams
• Dirt cheap IDS or IPS
• Under 400 lines of code

26
Resources

• http//www.tcpdump.org/
• http//www.packetfactory.net/projects/libnids/
• http//monkey.org/provos/libevent/
• http//monkey.org/dugsong/dpkt, pycap
• http//oss.coresecurity.com/projects/pcapy.html
• http//monkey.org/jose/software/flowgrep/
• http//pilcrow.madison.wi.us/pynids/

27
Additional Resources

• Stevens, TCP/IP Illustrated vols 1 and 2
• Schiffman, Building Open Source Network Security
  Tools
• RFCs from the IETF