Security Monitoring Thru Log Analysis

1
Security Monitoring Thru Log Analysis

• Sam.NG_at_PISA

2
Once upon a time An email server of my company
was HACKED! A user tmp with uid 0 and no
password was created
3
What the hacker did

• Startup script was modified, a line is added to
  the rc.local
• ./usr/bin/t0x5mm (I cant remember the exact
  name)
• ls the file, nothing showed
• I tried to remove the file by rm, rm ran
  without any error
• Many months later, I knew thats something call
  rootkit

4
Log was gone!

• Some of the log files are truncated
• I wanted to know how the hacker got in, and I
  redirected syslog to /dev/lp0 and ran for a
  couple of weeks
• Several inches of paper printed, but there is
  more than I could handle
• I cant remember how, but finally I knew the
  hacker got in by exploiting an IMAP4 vulnerability

5
Lesson Learnt

• You will need the log in sometime
• And better still, store it in a safe place
• Write once read many (WORM) is good, but
  searching capability is even more important
• And the most important you have to prepare it
  beforehand

6
But how can I use the log for security monitoring?
7
The Challenge

• I dont have the staff to handle 140,000 alerts.
  I dont even have the staff to handle 12,000
  alerts David MacLeod, Ph.D., CISSPThe Regence
  Group CISO

8
(No Transcript)
9
Block Diagram
Phase I
Phase III
Phase IV
Phase II
Log Collection
Noise Reduction
Data Mining
Alert/Ticket Management
10
(No Transcript)
11
Log Collectors

• Send your log to DATABASE
• Can be direct (open database connection from the
  client) or in-direct (e.g. post data to a web
  application)
• Each different log may need a different log
  collector
• Many logs have built-in support to store data in
  database

12
Unix syslog

• syslog can be redirected
• Can be redirected to printer! (/dev/lp0 in
  Linux)
• Can be redirected to remote syslog host
• But redirect to program (pipe) seems not
  working!!??
• syslog-ng support logging directly to database
• Win32 solutions to accept remote syslog records
  and log to database
• Plain text format, easy to write a program to
  watch the data and then send to database

13
Demo

• A smiple tail base perl script to monitor new
  records in syslog and send it to remote database

14
Windows Eventlog

• Microsoft logparser can log to database
• WMI interface allows you to query remote eventlog
• Many third party solutions support logging to
  database

15
Demo

• A smiple C program to monitor new records in
  Eventlog and send it to remote database

16
Quiz 1
Which of the following Windows Server will pass
Microsoft Baseline Analyzer (MSBA) 2.0 auditing
requirements with default install?

• Windows NT
• Windows 2000
• Windows 2003
• None of the above

17
DO YOU KNOW?

• You can pass MSBA 2.0 if you enabled Success and
  Failure auditing for the Account Logon Events
• But indeed, it recommends you to enable Success
  and Failure auditing Account Logon Events,
  Account Management, Policy Change, System
  EventsFailure auditing Directory Service
  Access, Object Access
• Data volume may be quite high, a server config in
  this way may generate a eventlog for every 2
  seconds (actual volume depends on your server)

18
Microsoft IIS Web Server

• Native support log to ODBC (but seems to be only
  available on server platform only)

19
Microsoft IIS Web Server (2)

• Microsoft does not recommend logging to database
  if the IIS is busy (Q245243)
• But nowadays most web pages are generated with
  SQL queries
• Test ODBC logging on your own environment
• PrepWebLog (Q296093) convert lIS log to SQL
  insertion statements in plain text format, but
  still have to run these SQL insertion by some
  other means
• Still, the best would be realtime logging to
  database
• Writing a tail -f similar program in IIS is
  difficult

20
DO YOU KOWN?

• IIS log file is updated in 64k chunks. On servers
  that do not have a high usage rate, the
  statistics will not be up-to-date because of the
  delayed write (Q142557)
• When it flush, may be only first 20k of the chunk
  contains data, the rest 44k data is \0 and will
  be filled in next flush
• Obviously a mmapped file, may be due to
  performance consideration but

21
Suggestions to programmer

• In C/C, STDERR, cerr are not buffered
• In fact, error log should be send out
  immediately, should not be cached/buffered

man stderr CONSIDERATIONS The stream
stderr is unbuffered. The stream stdout is line-
buffered when it points to a terminal. Partial
lines will not appear until fflush(3) or
exit(3) is called, or a newline is printed.
This can produce unexpected results, especially
with debugging output.
22
J2EE Application Server

• Most application server support log4j or
  java.util.logging (JDK 1.4)
• log4j natively support logging to database thru
  JDBCAppender

23
My Experience

• I have a program developed since JDK1.3
• At that time, Java dont have java.util.logging
• And I dont know much about log4j
• I wrote my own log handling routine (similar to
  syslog) to insert my own records into database
• But is not configurable/adaptable, and is not
  consolidated with the Application Servers log
• Since logging is widely spread all over the
  codes, it is very difficult for me to change the
  program to use these new technology
• Log architecture should be planned in DESIGN PHASE

24
Snort IDS

• Comes with database output plugin to send alerts
  to database
• The packet analysis thread is also responsible
  for database insertion

25
My Experience

• I have experience in using Snort to monitor a
  20M Internet link, with database output plugin,
  default rules
• CPU is just about 30, seems good ?
• When I changed to log to local file, CPU sharp
  jump to 100
• Barnyard seems to be a solution but it does not
  support database!!

26
And many others

• Checkpoint FW1 thru LEA enabled product
  http//www.opsec.com/
• Microsoft ISA default support ODBC logging
  (KB838710)
• Microsoft Exchange same as Eventlog
• Apache httpd mod_log_sql

27
Noise Reduction

• Very important
• Noise will kill the system
• And to improve overall performance
• Can be done on several layers
• ignore certain message at the log collection
  sender
• ignore certain message at the log collection
  receiver end
• delete (mark as ignore) at the database
• fine tune your IDS rule, firewall logging policy
  etc..
• exception cases for data mining phase

28
Data Mining

• Data processing using sophisticated data search
  capabilities and statistical algorithms to
  discover patterns and correlations in large
  preexisting databases a way to discover new
  meaning in data
• From http//www.tfd.com

29
Data Mining (2)

• The heart of Security Monitoring
• A board term, a general concept
• Utilize database queries to get the information
  you want
• Can be an external program, can be a SQL server
  schedule job

30
Data Mining Techniques
31
Attack Definition
IIS Unicode Directory Traversal (cmd.exe) attack
strings

• GET /..c0af../winnt/system32/cmd.exe?/cdirc\
• GET /MSADC/root.exe?/cdir
• GET /_mem_bin/..255c../..255c../..255c../winnt/
  system32/cmd.exe?/cdir
• GET /_vti_bin/.252e/.252e/.252e/.252e/winnt/sy
  stem32/cmd.exe?/cdirc\
• GET /_vti_bin/..255c../..255c../..255c../winnt/
  system32/cmd.exe?/cdir
• GET /_vti_bin/..c0af..c0af..c0af..c0af..c
  0af../winnt/system32/cmd.exe?/cdirc\
• GET /_vti_bin/..c0af../..c0af../..c0af../win
  nt/system32/cmd.exe?/cdirc\
• GET /_vti_cnf/..c0af..c0af..c0af..c0af..c
  0af../winnt/system32/cmd.exe?/cdirc\
• GET /adsamples/..c0af..c0af..c0af..c0af..
  c0af../winnt/system32/cmd.exe?/cdirc\
• GET /cgi-bin/..255c../..255c../..255c../winnt/s
  ystem32/cmd.exe?/cdirc/
• GET /cgi-bin/..c0af..c0af..c0af..c0af..c0
  af../winnt/system32/cmd.exe?/cdirc/
• GET /cgi-bin/..c0af..c0af..c0af..c0af..c0
  af../winnt/system32/cmd.exe?/cdirc\
• GET /cgi-bin/..f08080af../winnt/system32/cmd.e
  xe?/cdirc/
• GET /iisadmpwd/..c0af..c0af..c0af..c0af..
  c0af../winnt/system32/cmd.exe?/cdirc\
• GET /iisadmpwd/..c0af../..c0af../..c0af../wi
  nnt/system32/cmd.exe?/cdirc\
• GET /msadc/.252e/.252e/.252e/.252e/winnt/syste
  m32/cmd.exe?/cdirc\
• GET /msadc/..c0af../..c0af../winnt/system32/cm
  d.exe?/cdirc\
• GET /msdac/root.exe?/cdirc\
• GET /scripts/..e080af../winnt/system32/cmd.exe?
  /cdirc\

32
Vulnerability characteristic ?attack
characteristic

• CodeRed I /default.idq?NNNNN
• CodeRed II /default.idq?XXXXX
• In fact, the vulnerability can be exploited if
  the variable name is around 240 bytes
• Regular Expression \/default\.idq\?.240,
• Difficult to write an effective and accurate
  definition
• And not all attack leave audit trail

33
Event Correlation

• E.g. 1000 login failures followed by ONE
  successful login from the same IP
• E.g.IF http_response_code 500 THEN
  find_all_other_url_accessed()ENDIF

34
Anomaly Detection
But what is NORMAL??
35
Normal Definition

• You define what is normal and then monitor it
• E.g. Operators login time should be corresponding
  to their shift duty
• E.g. All server services should not be restarted
  unless necessary (ignore service start within
  3minutes of system startup)

36
Statistical Analysis

• E.g. on average a event occurs 10 time a day,
  with a standard deviation of 2.3. But today we
  have 2000 records.
• E.g. anything happens more than 200 times in the
  past 30 minutes
• E.g. a event never seen in the past 7 days
• E.g. TOP 10 events/users/hosts, etc

37
Alert/Ticket Management

• Works like Bug Tracker
• Save the alert as a Ticket
• A ticket is a something like an outstanding job
• Assign the ticket to a staff to follow up
• Escalate it if remain unresolved for some time

• And dont forget People Management(Time and
  Skill)

38
Alert/Ticket Management (2)

• According to my experience, at least 90 of the
  alerts generated by data mining are still FALSE
  ALARMS
• People will get use to it and tend to think EVERY
  alerts are false alarms
• If possible, fine tune the system to eliminate
  the false alarm from occurring again

Noise Can Kill
39
Couterpane 2003 Results
40
DO YOU KNOW?

• TSL providing Email to pager service for about
  HK80/mth

41
How should I start?

• Do it step by step, phase by phase, EventLog,
  syslog are easy to start with
• A group brain storming section would give you at
  least 10 such data mining rules, and is a good
  starting point
• Security Monitoring is a (long term) process, do
  not regard it as a single one-shot install and
  forget project

42
Q A
Contact samngms_at_yahoo.com