STIF

1
STIF Security Tools Integration
Framework STIF-WARE EVOLUTION Fyodor
Yarochkin Meder Kydyraliev fyodor_at_o0o.nu meder_at_o
0o.nu
HackInTheBox, Kuala Lumpur - 2005
2
Agenda(best question gets an Industry Slave
HITB T-shirt)

• Introduction to STIF-ware concepts
• First generation of STIF (automation,
  integration, unification)
• Demonstration
• Problems with the first generation of STIF
• STIF2 wider coverage of knowledge
  representation format, functionality decoupling,
  distributed multi-agent system, open system
  architecture
• STIF2 prototype

3
Introduction
Security Tools Integration Framework (STIF) is
aimed to provide a unified environment and data
exchange platform for automated security
assessments in heterogeneous environments. In
simple words it is a platform for hacking
automation, where STIF emulates the brain of a
security analyst to perform repetitive tasks.
4
Why automation?

• machine-based knowledge processing
• automate routine tasks, spend more time on tasks
  that require brain power
• create intrusion scenarios, and let machine
  probe them (nIDS testing)
• human error mitigation
• reduce human labor involvement in modern
  corporate pen-testing sweatshop

5
Why integration?

• Various security tools, written in different
  languages, are available, but no unified format
  for data exchange and representation
• No machine data analysis, aggregation and
  correlation possibilities
• Handling large-scale assessments w/
  disintegrated tools is a nightmare
• No possibilities to automate distributed attacks

6
Typical scenario for security analyst
7
Want to see what happened to Joe the analyst
after one month?
8
Poor Joe
Look what repetitive and boring hacking has
done to him
9
Why not let machine do the boring part???
10
Of course, you can ...

• script it ls al /code/scripts/
• (ab)use security scanners (nessus)
• (ab)use exploit toolkits (e.g metasploit)
• hire a full room of pen-testing monkeys, that
  will do the boring part (sweatshop production)

11
Scanners vs. STIF

• Problems with scanners
• hardcoded sequence of execution
• vendor-specific integration (e.g. NASL, plug-in
  APIs), requires rewrite or code hacking
• vendor-specific data representation/storage
  (hard to integrate into existing solutions, e.g.
  custom DBs)

12
STIF solution
STIF is designed to solve the problems outlined
earlier, by introducing the common format for
data representation and by providing a platform
for data exchange among tools.
13
First generation STIF provides

• Highly customizable rule-based inference engine,
  which enables analyst to script out ANY scenario
  based on the data that was returned by tools
• Unified data exchange and representation format
• Generic database publishing module (save data
  from tools in DB w/ any scheme)
• IRC BOT interface data publisher and importer

14
STIF Features (continued)

• Distributed architecture
• ready to use DB schema
• STIF is written in Java
• the reason for that decision is simple quicker
  development cycle, cross-platform compatibility

15
Data representation unification
STIF encapsulates data in a set of XML messages
(STIF-Message) Input data, provided in XML
format, converted by Exec module into the form,
which could be understood by the tool The results
of tools execution are converted to STIF and are
fed back into the Inference Engine.
16
STIF-Message
Sample STIF-Message ltSTIF-Message
created"2004-09-02T1503016"gt ltPort
number"80" state"open" protocol"tcp"gt ltAddres
s type"ipv4-addr"gt192.168.1.1lt/Addressgt ltProtoc
olgt HTTP ltApplicationgt Apache/1.3.27
(Unix) PHP/4.3.1 lt/Applicationgt lt/Protocolgt
lt/Portgt lt/STIF-Messagegt
17
Inference engine

• responsible for data interflow between various
  tools
• makes decisions on which tools to be executed,
  when new data appears
• provides data aggregation and correlation
  facilities (including regular expressions based
  matching to the
• knowledge base facts)
• maintains execution flow using rule-based
  scenarios

18
Data Publishing facility
Publishing in STIF environment means providing
the Publisher with newly arrived facts
(STIF-Messages from tools). STIF is able to
execute several data/fact publishing modules
simultaneously (e.g. database publishing, IRC
publishing).
19
SQL Publisher
STIF comes with SQL publishing module, which can
publish/store data received from tools in a form
of a STIF-Message, in databases of arbitrary
scheme. ltmessage type"Target"gt ltquerygt INSERT
INTO ip_address VALUES(NULL,'h') lt/querygt
lt/messagegt ltmessage type"Port"gt ltquerygt SELECT
id FROM ip_address WHERE ip_address'h'
lt/querygt ltquerygt INSERT INTO port VALUES(NULL,
1, 'n', 'P', 'S', 'p', 'a')
lt/querygt lt/messagegt
20
IRC Importer/Publisher
STIF supports command input over IRC and can
publish new facts to an IRC channel or using
private messages. Other software tools can act
as STIF nodes embedding the IRC
importer/publisher functionality
21
Your favorite tools integration to support STIF?

• STIF provides several means to import data into
  STIF inference engine
• Generic2STIFConverter, extracts data from output
  using regular expressions to form STIF-Message
• Tool-specific wrappers

22
Integration using STIF Generic2STIF Converter
Define rules in parser.xml lt?xml
version"1.0"?gt ltConfiggt ltTool
name"nmap-syn-version"gt ltGroup
name"target address"gt
ltDelimetergtInterestinglt/Delimetergt
ltRegex name"address" required"true"gt
.ports on .\((\d\.)\).
lt/Regexgt ltGroup
name"port" generate"port"gt
ltDelimetergt
newline lt/Delimetergt
ltRegex name"portNumber" required"true"gt
(\d)/(?tcpudp).
lt/Regexgt
23

• ltRegex name"portProtocol" required"true"gt
• \d/(tcpudp).
• lt/Regexgt
• ltRegex name"portState"
  required"true"gt
• 
  \d/(?tcpudp)\s(openclosedfiltered).
• lt/Regexgt
• ltRegex name"portService"
  required"true"gt
• \d/(?tcpudp)\s(?openclo
  sedfiltered)\s(\w-).
• lt/Regexgt
• ltRegex name"portApplication"
  required"false"gt
• \d/(?tcpudp)\s(?openclo
  sedfiltered)\s\w-\s(.)
• lt/Regexgt
• lt/Groupgt
• lt/Groupgt
• lt/Toolgt
• lt/Configgt

24
How can you help?

• You can do several things to contribute
• to our efforts
• Try it!!!
• Ask your favorite tools author to become
  STIF-compliant
• Write regular expressions to parse output for
  Generic2STIFConverter
• Patch you favorite tools to be STIF-compliant
• or.. wait until STIF2 is out

25
First generation STIF Demonstation
26
Problems with current STIF implementation

• Complexity massive coupled piece of code
• Centralized system limited support for task
  distribution
• Non-dynamic (fixed at startup) inference engine
  rules
• Knowledge interchange format needs to be extended

27
STIF2 Concepts

• Functionality decoupling

28
STIF2 Concepts

• Platform independent
• Composed of independent agents
• Agents communicate with each other using
  messaging protocol
• Agent capability service exists to provide agent
  capability lookup and matching facility

29
STIF2 Multi-Agent Architecture

• Multi-agent architecture
• Tool wrapper Agents
• Scanning, connection forwarding, attack launching
• Logic Execution Agents
• User Interface Agents
• And more

30
Message Exchange Framework

• Provides facilities for agent communication
• Provides facilities for communication channel
  selection (covert channels, tunneling,
  stenography)

31
Goal-Driven execution

• Goal-driven execution flow
• Each agent describes its functionality with a set
  of capabilities. Each capability can be executed
  on certain type of data object (network, host,
  user, URL)
• Each agent is given task to execute the
  capability, which becomes agent goal. Agent may
  have different plans to execute the same
  capability. Plans are scored based on execution
  success rate

32
Goal Driven execution

• Each also plan may be assigned with qualifiers
• Stealth-ness
• Latency
• Which can be matched to current environment
  settings

33
Event-driven execution

• Event-driven execution flow
• Each agent may subscribe to interests,
  expressing its interest to certain types of data
  objects, which agent is interested in (network,
  host, open port, URL, a valid user)
• When an agent discovers a new data. The
  interests list is queried for the list of
  interested agents. The agent is responsible to
  forward the data to interested partners.

34
Agent Data Cache (beliefs)

• Agent caches data locally (local data Cache,
  beliefs)
• Agent may query other agents or KB for missing
  data

35
Current Implementation prototype

• Based on Java/JADE framework
• The communication protocol in progress
• The knowledge interchange format reviewing
  current standards (KIF, DAML)
• Once the communication framework is finalized,
  JADE messaging framework to be replaced with
  home-brewed implementations (ports for different
  languages)

36
Questions (remember we give out T-shirt for best
question)? Suggestions ?
fyodor_at_o0o.nu meder_at_o0o.nu http//o0o.nu/sec/STIF/
37
Thanks!!!!