Cyber Security KTN

1
Cyber Security KTN Metrics SIG 15th September
2006 Chaired by Jeremy Ward
2
Introduction

• Purpose of meeting is to consider members
  contributions
• Agenda
• Contributions from
• John Murdoch (York University)
• John Leach (John Leach Information Security)
• Sadie Creese (QinetiQ)
• Summary and conclusions
• Next Steps

3
Cyber Security KTN Metrics SIG Contribution
from John Murdoch (York University)
4
Cyber Security KTN / Metrics SIG Observations

• Metrics SIG recently established within the Cyber
  Security KTN
• Need for projects that foster collaborative
  development of practical
• measurement approaches and tools
• A project has been proposed to develop means to
  assess actual internet security risks and effects
  of mitigations, to support decision making. A
  Test Bed has been proposed to trial and assess
  measurement proposals
• This project provides a valuable learning vehicle
  for the SIG
• The project will be most effective if
• We are as informed as possible about existing
  measurements and methods of measurement
  development
• We integrate/ generalize findings from the
  project for wider application
• We define those cyber security information /
  decision support needs addressed by the project,
  and scope the issues not addressed

5
Comments

• Support proposed project by applying the best of
  what we currently know
• Review whats out there in security
  measurement that is applicable to cyber security
  measurements, measurement methods tools
• Integrate findings from project into something
  more general, applicable to other situations
  e.g. Cyber Security Measurement Guidance
• Based on KTN and SIG objectives, develop a
  strategy that fosters application of measurement
  in this domain and its transition to use what
  can/should the SIG be doing in terms of
• ongoing monitoring of measurement field,
  research, transition, education, training,
  professional development, standards, project
  support,
• defining and exploring relationships with other
  groups/ related efforts
• considering additional projects/ case studies to
  cover other identified areas of information need/
  decision support and integrate similarly

6
(No Transcript)
7
SIG Activities

• Activities best done collaboratively by
  interested parties, but with named leads
• SIG Objectives and Strategy
• Develop a strategy for cyber security
  measurement, matched to the goals of the KTN and
  needs of participants
• SIG Inputs Initial Review
• Review current metrics work in information
  security, network security and system devlpmt
  select and focus onto the scope of the SIG,
  propose means to provide continued monitoring and
  dissemination
• SIG Outputs Initial Definition
• Identification and planning, propose means for
  ongoing integration and dissemination
• Project Cyber Security Risk and Mitigation
  Assessment
• (or preferred name for proposed project)
• Identify information needs, develop measurable
  concepts, review existing measures, develop
  measurement constructs, implement measures,
  collect data, analysis, support decision-making/
  use of data

8
Proposal

• Measurement Definition for Cyber Security Risk
  and Mitigation Assessment Project
• Support project by applying systematic
  measurement methodology/ measurement principles,
  based on experience with related standards,
  measurement practices, security research
• Proposed method apply ISO/IEC 15939 principles,
  currently being used as basis for draft ISO/IEC
  27004
• Supported by information security, statistical
  and math skills at York
• Combine with domain knowledge of project
  partners
• Explore and propose measurable concepts in
  collaboration - two workshops
• Input
• Objectives of project information needs served,
  decisions to be supported
• Measurable concepts, current measurements
• Systems, software and technology context of
  planned and potential measurements
• Output
• In collaboration with project partners, proposed
  set of measurements that serve defined
  information needs and that can be implemented,
  demonstrated on a test bed. Possibly support data
  analysis and reporting, depending on interests of
  other participants

9
Comments on Security Measurement (1)

• Need clarity about the information needs we are
  serving/ decisions/ who. e.g. for a user
• prospectively, what is the business case for
  investing in security protection XYZ? What is the
  cost and predicted benefit, including risk
  reduction? Opportunity costs.
• retrospectively, having spent the money, how can
  I tell what benefits I actually gain?
• Separate issues as much as possible Pr(attack),
  Pr(detectionattack), Pr(damageattack),
  Pr(damagedetection), size of damage, costs of
  false positives use CC type thinking with PPs
• Measurement is really comparison need to set up
  measurement of input and output so as to isolate
  areas of interest. Enable comparison like for like

10
Comments on security Measurement (2)

• Definition of instrumentation what entities
  and attributes are observable? Currently used,
  feasible but not used, new? Costs of implementing
  these? Uncertainty, assurance?
• Develop indicators define constructs that link
  the information needs to sets of base measures
  (15939/PSM)
• Measurable concepts depends on models /
  cause-effect theory in domain establish
  sufficient models and define measurements with
  respect to these. Then be prepared to improve
  them
• Distinguish between measurement and prediction
  not everything that is measurable in retrospect
  is predictable (c.f. stock market), Can we
  separate issues predictable from the chaotic?
  Consider leaving aspects to judgment of informed
  decision-makers, other aspects supported formally
• Careful, systematic approach to measurement will
  be useful and repeatable

11
Cyber Security KTN Metrics SIG Contribution
from John Leach (John Leach Information Security)
12
The Challenge

• We need to be clear about what we want to measure
• We cant measure the threat or risk using a
  localised test bed
• We can measure the number of attacks and
  incidents using one
• We can measure the effectiveness of a given
  countermeasure using one

create
Threats
Risks
Global
Local
Attacks
Incidents
Countermeasure
13
Proposal

• Decide which technological countermeasure we want
  to study
• Decide which countermeasure parameters and threat
  parameters we want to study
• Measure the threat, profiling it against the
  desired parameter
• Use the test bed to measure the local attack
  profile.
• Use the test bed to measure the local incidents
  as a function of the local attack profile and
  different settings of our countermeasure
• Example
• Countermeasure AV software threat e-mail
  viruses
• Then
• C/m parameter the frequency of update of virus
  signatures
• Threat parameter the age of the virus carried
  by the e-mail

14
Suggestion

• If the countermeasure we want to study is
  software patching
• Measure the current malware threat
• Use the test bed to measure and profile the local
  attacks, and measure the local incidents as a
  function of various countermeasure parameter
  settings.

15
Cyber Security KTN Metrics SIG Contribution
from Sadie Creese (QinetiQ)
16
Objective of a SIG - assumed

• To develop practical metrics which the approach
  is valid and which can be validated using fit
  into and will form the basis for a broader
  strategy and which prove some form of test-bed.

17
Possible Approach

• Top down and bottom up together (Daves pincer
  movement)
• We use a bottom-up consideration of
  practicalities to guide our choice of targets
  for measurement this includes considering how
  to achieve validation, who needs to be involved
  etc
• We use a top-down approach to developing the
  broader solution measurement concept, then
  assessing community needs to select the subset of
  candidates for measurement which, if we are
  successful, will demonstrate to the community the
  value of our work (e.g. take our stakeholders
  with us).
• - If we dont do this then we simply make it
  harder for ourselves to communicate the value of
  our results (assuming that we do conclude that
  there are metrics to be had..)

18
Possible Approach

• Develop a generic model of security solutions
• - E.g. things which we might want to measure and
  why
• Generate list of target model components to be
  measured by
• - Top down consideration of impact (community
  and stakeholders)
• - Bottom up consideration of validation
  practicalities
• Develop metrics for targets
• Draft an approach to aggregating component scores
  to give overall system security metric
• Develop validation architecture

19
Generic model

• Develop a generic model of security solutions
• - Is this a taxonomy no more informal.
• - Needs to include the human process bits and the
  real edges
• - Needs accompanying explanation of why we might
  wish to measure components
• OR how they fit together
• - Graphical please.
• Probably models already existing in community
  so relatively low effort to develop
• - Need validation mechanism
• QQ FSEL have worked on aggregation metrics
  before can feed this into the approach

20
Metrics

• PSM (York) White Paper
• - Focus on whole system view, top down
• - Security Measurement Map driven from a systems
  development perspective
• - Describes generic strategy for developing
  security measures
• we could use this for a general approach
• Practical selection
• - Suggest software technologies such as virus
  protection, firewalls, IDS sensors etc

21
Test bed validation strategy

• Use statistics from last discussion to guide
  choice of technologies
• - Pervasiveness of use will heighten impact
• Could we use honey monkey approach where we
  supply the box with a particular config which is
  also a tar pit, in that it cannot be used to
  compromise the wider network
• - But allow experiment partners control over it
• Would this invalidate the results?
• No use logs to check whether changes were made
  then take the candidate out of the experiment if
  they were
• Alternative approach invite black hat community
  to take part in the experiment so we dont need
  to worry about them knowing it is not a real
  box treat them like security researchers as
  MS have been doing.
• - They might be willing to help.
• - Risk vendors wont want to play.

22
Conclusions

• Summary and conclusions
• Academic (research and development)
• Practical (test-bed build and structure)
• Synthesis
• All three approaches are valid and can be
  combined
• Need to focus on a simple deliverable for this
  SIG
• Paper outlining approach, suggested content
• Phase 1 Scope
• Phase 2 Research
• Phase 3 Test-bed design
• Phase 3 Test-bed build
• Phase 4 Test-bed operation
• Phase 5 Publication/dissemination of results
• Phase 6 Ongoing use and development

23
Next Steps

• Agree on deliverable
• Decide how deliverable will be produced by whom
• Agree timetable for deliverable
• Recommendation roundtable brainstorming
  session for next meeting. Suggest wb 6th
  November.