Jason C' Richards

1
Security Metrics What Matters MostPresented to
VASCAN Conference 2008

• Jason C. Richards
• Chief Information Security Officer
• Virginia Community College System
• jrichards_at_vccs.edu

2
Disclaimer

• Information presented is taken from various
  sources and credited wherever possible or is my
  personal opinion.

3
Dogberts Take on Metrics
4
What is a Metric?

• Derived through analysis applied to measurements
• Provide quantitative data about a target process
  or asset in order to achieve an explicit purpose
• Truly useful metrics provide the incite needed to
  make better decisions
• Defines what, where, and how risk is occurring

5
SMART Metrics

• Specific
• Measurable
• Attainable
• Relevant
• Timely
• Winning With Quality Applying Quality Principles
  in Product Development, by John Wesner et al.

6
Why Do We Need Metrics?

• Measure the security posture of the organization
• Show return on security investment (ROSI) for
  security projects
• Understand risks to the organization
• Project planning and roadmap
• Identify areas that need attention/resources
• Raise the profile of the security team

7
What Matters?

• Mike Rothman Author of The Pragmatic CSO
• Tons of operational metrics but few are relevant
  to the folks that run your business.
• Use the language of businessalign metrics with
  the business.
• You need to figure out what is going to resonate
  with your management team, and the only way to do
  that is to talk to them.

8
Where to Start

• Figure out who the metrics will be reported to
• Security Officer
• CIO
• Chancellor/President
• Oversight Board
• Understand what is relevant to your audience
• Technical Vs. Management
• Technical 100 viruses blocked, 2 missed 98
  success rate
• Management Business impact 2 missed viruses led
  to 20 man-hours to control, and may have exposed
  sensitive data.

9
Where to Start contd

• Metrics Basics
• Know what is important
• Risk analysis is a must
• Identify incident trends that matter to key
  senior managers
• Develop a few value indicators that provide
  reliable information that you can track
• Set up a security council
• Track changes
• Use metrics in planning
• Check your numbers

10
Who Needs What?

• Security Officer
• Operational Metrics
• Helps to see the big picture
• Compliance Metrics
• Measured against your regulations/requirements/sta
  ndards
• Where the issues are at
• Project Metrics
• Show Return On Security Investment (ROSI)
• Derived from Operational Metrics
• Need metrics to inform and advise senior
  management in managing risk. Manage and improve
  security processes.

11
Who Needs What? contd

• CIO
• Project Metrics
• Compliance Metrics
• Maybe Operational Metrics
• Especially as they may relate to the other IT
  functions
• Risk
• May not want to know specific metrics just what
  the risks are
• Security Posture
• Time Metrics
• Operational efficiency, reduced cycle time
• Financial Metrics
• Increased productivity, lower costs, lower
  headcount
• Ask what they want and what concerns them
• Translate into how the security program can
  support it and other priorities or the
  organization

12
Who Needs What? contd

• Chancellor/President/Board
• Risk
• What is the overall security posture of the
  organization
• What risks exist that would impact our brand
• Compliance Metrics
• Confidence in external reporting of regulatory
  compliance, enterprise risk management status,
  how do we compare to peers?
• Ask what they want and what concerns them
• Translate into how the security program can
  support it and other priorities of the
  organization

13
Good Metrics

• Baseline Defenses Coverage (AV, FW, etc)
• Measurement of how well you are protecting your
  enterprise against the most basic information
  security threats.
• 94 to 98 less than 90 cause for concern
• Patch Latency
• Time between a patchs release and your
  successful deployment of that patch.
• Express as averages and criticality
• Platform Security Scores
• Measures your hardening guidelines
• Compliance
• Measure schools/departments against security
  standards

14
Tools and Reporting

• Automated tools
• ClearPoint Metrics for dashboard
• Network scanning tools
• AV console
• Dashboards
• Graphs
• Charts
• Visualize the data (especially for executives)

15
Tools and Reporting contd

• Good Visualization of Metrics
• Dont oversimplify
• Dont be overly ornate
• Do use a consistent scale
• Do include a benchmark

16
Conclusion

• Be able to answer the following question We are
  implementing a security metrics program because
• Adhere to measurement best practices
• Measure what?
• Why measure it?
• Measure it for whom?
• Interview key stakeholders to determine what they
  want
• Refine into measurable items
• Start with a manageable, useful set of metrics
• Dont forget to set a baseline.
• Must have to show improvement

17
References

• www.securitymetrics.org
• Metricon
• Security Metrics Replacing Fear, Uncertainty,
  and Doubt. Andrew Jaquith
• Security Metrics Standards ISO 27004 standard on
  Information Security Management Measurements
• KoreLogic Security Metrics Presentation to
  Central Virginia ISSA