Processing Intelligence Feeds with Open Source Software

1
Processing Intelligence Feeds with Open Source
Software

• Chris Horsley, SC Leung, Tomas Lima, L. Aaron
  Kaplan, Raphael Vinot

2
Overview

• Current topics in automatic incident handling for
  CERTs
• IFAS
• HKCERT , IFAS and use-cases
• IHAP project
• ContactDB project
• Current RD

3
IFAS

• Information Feed Analysis System

4
Knowing whats going on
5
How do national CSIRTs know whats happening?

• National CSIRTs need visibility on network in
  their economy
• However, many national CSIRTs dont operate
  networks themselves, and normally dont have
  global (or any) direct visibility
• How does the CSIRT know whats going on in their
  country?

6
The kindness of strangers

• Luckily, there are a lot of network operators,
  research teams, vendors, and other CSIRTs out
  there that collect information, and will share it
  with national CSIRTs.
• And here comes the but...

7
So much data, so many formats

• There are many feeds, all with their own data
  formats and mediums
• Formats CSV, JSON, XML, STIX, IODEF
• Mediums HTML, RSS, email, HTTP APIs
• While there are efforts to standardise data
  formats, this will take a long time, and will
  likely never cover 100 of feeds
• We cant change the format of remote feeds - we
  can only change what we do with the data.

8
The need for standards

• Different feeds use many terms to mean the same
  thing
• ip, source_ip, src_ip, endpoint, attacker_ip,
  cnc_ip...
• If we receive events from many feeds, we need to
  normalise so we can compare them together.

9
The need for storage

• As a national CSIRT, were concerned with the
  health of national networks which means
  measurement.
• We can only measure longterm if we store events,
  enabling us to analyse them.
• We also want to search through events, like
• CC servers in domestic networks in last week
• Bots infected with Trojan.abc on BigISP
• Defaced web sites targeting gov.zz

10
Need for automation

• Theres way too much network event data out there
  to manually process
• Options
• a) use lots of analyst time doing tedious log
  processing
• b) write lots of small, independent scripts
• c) ignore inbound logs completely
• d) use an automated processing system

11
So what do we need?

• We need something which automatically
• Gathers many different types of feeds
• Normalises the data in those feeds
• Stores that data somewhere
• Allows search and performs statistical analysis

12
IFAS

• IFAS Information Feed Analysis System
• Project sponsored by HKCERT and developed by
  HKCERT and CSIRT Foundry
• An integration of open source tools, released as
  open source for CSIRTs

13
Architecture
14
Architecture

• Abusehelper gather, process, and enrich feeds,
  generate events
• Logstash process and normalise feeds
• Elasticsearch store events in schema-free index
  server
• Kibana search through events
• IFAS Reporter get overall statistics, build
  realtime dashboards

15
(No Transcript)
16
Kibana event searches
17
Freeform statistical reporting
18
Nesting, filtering, deduplication
19
IFAS Dashboard

• Visualize information

20
Realtime dashboards
21
What you need to start
22
Software

• Open source under Apache 2.0 License
• Only possible with the hard work released under
  open source licenses from Abusehelper and
  Elasticsearch teams
• Contributions, bug reports, feature requests most
  welcome!

23
Hardware

• Production 8-16GB memory machine
• Dev 4GB possible
• Multi-core machine (4 ideal)
• Runs in a VM no problem

24
Out of the box feeds

• Other developed Plugins
• Malc0de
• Malicious Domain List
• Arbor SRF
• Shadowserver
• Zone-H
• Future
• more, and your own

• Out of Box Feed Plugins
• (4 publicly available)
• Abuse.ch
• CleanMX
• Millersmiles
• Phishtank

25
Where to get it

• Currently under closed pilot to trusted CSIRTs
• Eventually public release
• Please contact contact_at_ifas.io for details

26
Demos
27
(No Transcript)
28
(No Transcript)
29
(No Transcript)
30
(No Transcript)
31
(No Transcript)
32
IFAS and Use Cases

• SC Leung, HKCERT

33
IFAS

• IFAS Information Feed Analysis System
• Project sponsored by HKCERT and developed by
  HKCERT and CSIRT Foundry
• An integration of open source tools, released as
  open source for CSIRTs

34
IFAS
(D3.js)
35
Architecture

• Abusehelper gather, process, and enrich feeds,
  generate events
• Logstash process and normalise feeds
• Elasticsearch store events in schema-free index
  server
• Kibana search through events
• IFAS Reporter get overall statistics, build
  realtime dashboards charts based on D3.js

36
Information Feed Analysis System (IFAS)

• Collaborate of global intelligences
• No sensor installation required
• Assimilate different input formats / standards
• Normalize field names
• Collect and normalize information 24x7
• Situational Awareness
• Alerts
• Public Awareness
• Tracking Trends

37

• Actionable
• Proactive Discovery
• Quality information for incident handling
• Automate Incident Response
• Engagement Tool
• Engage top ISPs to cooperate
• Cooperate on large scale clean up (bots)
• Standardization
• Standardize as measurement

38

• Total governance on data
• Built on open source tools that has good
  community
• Abusehelper, Elasticsearch, D3.js
• Low hurdle to use and customize
• Open source under Apache 2.0 License
• Free of charge
• UTF Support
• Support JSON/CSV import export
• Modular
• add plug-ins for new feeds
• add output chart plug-ins using D3.js

39
All you need is.

1. Machine with 8GB Multi-core CPU
2. Ubuntu Server (can run in VM)
3. Bitbucket Account (code repository)
4. 30 minutes of installation and configuration

40
Give a sense of Todays Events
41
IFAS - Log Search

• Powerful search on all the information collected

42
IFAS - Reporter

• Statistical analysis-Trends Distributions
• Free form statistical reports

3.
5.
4.
2.
1.
6.
43
Nesting, filtering, deduplication
Number of phishings in .AU in each ASN by brand
44
IFAS - Alert

• Set tracking criteria get notify ASAP
• domain .gov.hk
• Alert lists educational institutions (hkedu),
  NGOs (hkorg)

!
45
Dashboard
Real-time situational awareness for CERT
management
46
Public Situational Awarenesson Compromised
Servers / PCs
47
Hong Kong Security Watch Report
48
(No Transcript)
49
(No Transcript)
50
Analysis of Trend with Events

• Correlate Cryptolocker 2013-Oct with Zeus

51
Engage ISPs for large scale incident handling
ISP

• Data do help HKCERT engaging ISPs (their sales
  team)
• Data do help a server hosting SP understand their
  customers security problems

52
Converting security events into incident reports

• Defacement
• Phishing
• ? Export to CSV for batch processing, with some
  other scripts
• Malware hosting a bit difficult
• Large volume of incidents need prioritisation

53
Future of IFAS - a collaboration platform

• All you can use
• All you can contribute
• Add input filters for new feeds
• Add new plug-in modules
• Add new chart and visualization
• Integrate with other systems, e.g. RTIR
• Standard language STIX, taxonomy of ENISA

54
DSMS (Decision Support Monitoring System)

• An ongoing project that turn security events into
  Actionable Data
• Set Priority, Choose Monitors, Consolidate
  Results

Decision Support Sub-system

Interfaces to Monitors

IFAS
Request to monitor
Incident Mgmt
Output
Consolidated Results
55
IHAP

• Incident handling automation project

56
IHAP

• Very similar to IFAS, developed in parallel by
  CERT.pt, CERT.at
• Also uses Logstash, Elastic Search and
  Abusehelper
• Less work on the Webinterface, more work on
  Ontology, Data harmonisation document

57
IHAP - History

• Discussions about CERT.AT developments/documents
• Discussions about cooperation between CERTs
• ENISA support

58
IHAP - Goals

• Open Source
• Maintainable
• Flexible and Modular - must be possible to
  integrate existing software and modules
  (Pastemon, AbuseHelper, etc..)
• Reusable
• Easily Extendable - should require little
  knowledge and basic programming skills
• Easily Deployable
• Easily Updatable easy to share new
  developments with other CERTs and update the
  system with that new code
• Easily Configurable - config files that can be
  easily modified to fit CERTs needs
• Documented - must be well documented

59
Links Code

• http//www.enisa.europa.eu/activities/cert/support
  /incident-handling-automation

60
Common field names for AH

• https//bitbucket.org/clarifiednetworks/abusehelpe
  r/wiki/Data20Harmonization20Ontology
• A standard set of well defined field names within
  Abusehelper (AH)
• Allows CERTs to
• Write bots which are interoperable within AH
• Measure in identical ways
• Easier to parse different feeds (generic
  santizer bot) you just have to define the
  mappings

61
contactDB
62
Background/ problem

• abuse_at_ lookups suck (IRT object not in use, no
  standard Just now RIPE DB is changing with
  abuse-c)
• Getting the right lookup is non-trivial, complex
• Many (national) CERTs create their own abuse
  contact lookup DBs.
• National CERT DB, TI directory, FIRST data can
  not be looked up automatically via scripts.

63
Idea

• A caching contact database with more specific
  internal data
• Some of this data (tel nos, etc) will never be in
  the public whois
• Unify with TI, FIRST etc data
• Make it query-able by scripts

64
Abuse contact lookup - flow

• What databases exist? What can we query?

65
(No Transcript)
66
What exists now?

• Public code repo -)
• Whois server (thx Mauro)
• RESTful API (Mauro, Rafiot)
• Some scripts to import TI data (Aaron, David)
• Still some bugs -)

67
Code document with RIPE

• Document (WIP)
• https//github.com/certtools/contactdb/blob/master
  /doc/contact-databases-for-abuse-handling.mkd
• Codebasehttps//github.com/certtools/contactdb
• (thx Rafiot, David, Mauro!)

68
News from the trenches
69
Architecture revisited IntelMQ
70
Architecture revisited IntelMQ

• AH Architecture showed the way but its
  complex
• Current experiments underway with RabbitMQ,
  Redis, logstash Elastic Search
• Promising results so far, easy to use, more
  testing needed
• Problem re-implement all parsers??

71
Summary
72
Summary

• The CERT community has limited ressources for
  development
• We re-implement the same thing all the time
• Lets share code or at least exchange ideas on
  how to automate incident handling!
• Lets share on how to measure success
• Thanks HKCERT, ENISA, CERT.at, CERT.pt, CIRCL,
  etc..
• Mailinglist https//tiss.trusted-introducer.org/m
  ailman/listinfo/ihap

73
Thanks!