IT Security Metrics

1
IT Security Metrics A Practical Approach to
Measuring Information Security Measuring
Security at the System Level
2
Introduction
3
IT Security Metrics Training

• Audience Federal IT security personnel with
  GISRA reporting responsibilities
• Goal To train Federal IT security personnel how
  to develop metrics that they can use immediately
  to assist with GISRA reporting
• Duration 3 hours

4
Objectives

• After completing this workshop, you will be able
  to
• Identify why metrics are important for IT
  security
• Identify the relationship among GISRA, NIST SP
  800-26, and IT Security Metrics
• Describe IT Security Metrics
• Describe metrics development process
• Apply metrics development process by completing a
  Metrics Form for one of the OMB GISRA reporting
  requirements for FY02
• Identify metrics-related Roles and
  Responsibilities
• Describe how to implement a Metrics Program

5
Metrics Development
6
In this section, you will

• Learn the definition and characteristics of IT
  Security Metrics
• Identify the difference between Performance Goals
  and IT Security Metrics
• Learn the seven-step IT Security Metrics
  Development Process
• Discover the types of information and insights
  that can be gained from IT Security Metrics
• Complete three examples of IT Security Metrics

7
What are IT Security Metrics?

• IT Security Metrics are tools that facilitate
  decision making and accountability through
  collection, analysis, and reporting of relevant
  performance data.
• Based on IT security performance goals and
  objectives
• Quantifiable
• Obtainable/feasible to measure
• Repeatable
• Provide relevant performance trends over time
• Useful in tracking performance and directing
  resources

8
Why Measure IT Security?
Requirements
Benefits
Categories

• Government Information Security Reform Act
  (GISRA)
• Clinger-Cohen Act
• Government Paperwork Reduction Act (GPRA)

• Satisfy regulatory requirements

Regulatory

• Measure successes and failures ofpast and
  current security investments
• Justify future investments

• Enable investment targeting to identified
  areas in need
• Ensure best value from security

Financial

• Build confidence in leadership
• Demonstrate improvement to stakeholders
• Play key role in initiating improvementactions
  based on performance trends
• Enable relevant, realistic, appropriatesecurity
  procedure modification

• Improve accountability to stakeholders
• Ensure appropriate level of mission support
• Determine IT security program effectiveness
• Improve customer confidence

Organizational
9
IT Security Metrics should support IT security
goals and objectives

• IT Security Performance Goals identify desired
  results of system security program
  implementation.
• IT Security Performance Objectives enable
  accomplishment of goals by
• Identifying strategic practices, defined by
  security policies, procedures, and controls
• Directing consistent implementation of policies
  and procedures across the organization
• IT Security Metrics monitor accomplishment of
  goals and objectives by
• Quantifying the level of implementation of
  security control objectives and techniques for a
  system and the effectiveness and efficiency of
  the controls within the organization
• Using analysis of collected IT Security Metrics
  to determine adequacy of security activities and
  make appropriate business decisions

10
Exercise Performance Goal or IT Security Metric?
IT Security Metric
Statement
Performance Goal
Program Officials understand the risk to systems
undertheir control and determine the acceptable
level of risk.
.
Percentage of system security plans that
areupdated annually.
.
Duties are separated to ensure least
privilegeand individuals accountability.
Percentage of systems with automated virus
updating.
Data integrity and validation controls are used
to provide assurance that the information has
not beenaltered and the system functions as
intended.
11
Metrics development is a seven step process
12
The focus of the metrics program depends on IT
security program maturity
13
Stakeholders and Interests

• Anyone within an organization is an IT security
  stakeholder, though some functions have a greater
  stake than others
• CIO
• Program Manager/System Owner
• Security Program Manager
• Resource Manager
• Training/Human Resources Personnel
• Each stakeholder needs a set of metrics that
  provides a view of the organizations IT security
  performance within their needs, for a total of no
  more than 10-20 metrics per stakeholder
• Many IT Security Metrics can be created to
  measure each aspect of the organizations IT
  security. Selecting the most critical elements
  of the organizations IT security program during
  metrics prioritization will make the program
  manageable and successful

14
IT Security Performance Goals and Objectives

• IT security performance goals and objectives are
  expressed in the form of high level policies and
  requirements in many laws, regulations, policies,
  and guidance that describe the dimensions of an
  effective IT security program
• Clinger Cohen Act
• Presidential Decision Directives 63
• Government Information Security Reform Act
  (GISRA)
• OMB Circular A-130, Appendix III
• Critical Elements within NIST Special
  Publication 800-26
• Federal Information Security Compliance Audit
  Manual (FISCAM)

15
IT Security Policies, Guidance, and Procedures

• Some Federal guidance and agency-specific
  policies and procedures provide more detailed
  information specific to the agency
• NIST SP 800-12, 800-14
• Agency-specific policy and guidance
• Subordinate questions within NIST Special
  Publication 800-26

16
System Security Program Implementation

• System Security implementation includes
• Processes and procedures in place
• Existing capabilities
• Areas for improvement
• Existing metrics
• Existing data sources that can be used to derive
  metrics data
• These may be documented in the following sources
• System Security Plans
• OMB Plan of Actions and Milestones (POAM)
  reports
• Latest GAO and IG findings
• Tracking of security-related activities
• Risk assessments and penetration testing results

17
Metrics can describe three aspects of IT security
program operations and management
Business Impactcan be measured through
correlation analysis oncean organizations
processes are self-regenerating andmeasurement
data gathering is transparent.
7
Level of Implementation Most organizations are
new to measuring IT security with performance
metrics. They will begin by measuring level of
implementation of its security policies and
procedures. Instituting a metrics program is
the first step to process maturity.
Security Program Effectiveness and EfficiencyAs
an organizations process maturity increases and
performance data becomes more readily available,
metrics will focus on program efficiency and
effectiveness.
18
It is important to record the specifics of each
metric for the purposes of data analysis and
possible metric reuse
Describes the overall functionality obtained by
collecting the metric
Metric
Purpose
Defines the metric by describing the quantitative
measurement(s) provided by the metric
Purpose
Metric
ImplementationEvidence
Specific questions that will need to be answered
via survey or through automatic data gathering to
be able to calculate the metric
Survey Question(s)
Proposes the periods for collection of data to be
used for measuring changes over time. Suggested
time periods are based on likely updates
occurring in the applicable process
Frequency
Describes the calculation to be performed that
results in a numeric expression of a metric
Formula
Frequency
Formula
Data Source
Lists the location of the data to be used in
calculating the metric
Target
Provides information about the meaning of the
metric and its performance trend. Proposes
possible causes of trends, identified through
measurement, and points at possible solutions to
correct observed shortcomings
Indicators
19
Metrics can help identify causes of poor
performance, including
Areas
Examples

• Insufficient human, monetary, or other resources
  can be causing negative performance trends
• Lack of appropriate training for the personnel
  installing, administering, maintaining, or using
  the systems
• Security patches that have been removed during
  the operating system upgrades
• New or upgraded systems that are not configured
  with all required security settings and patches
• Security patches or upgrades that are
  incompatible with software applications supported
  by the system
• Lack of management awareness and/or commitment
  to security
• Lack of policies and procedures that are required
  to ensure existence, use, and audit of required
  security functions
• Poor system and security architectures that make
  systems vulnerable
• Inefficient planning processes that influence the
  metrics (including communication processes
  necessary to direct organizational actions)

Resources
Training
System Upgrades
Configuration Management Practices
Software Compatibility
Awareness and Commitment
Policies and Procedures
Architectures
Inefficient Processes
20
How does NIST SP 800-26 relate to metrics?
Critical Element
PerformanceGoal
13.1. Have employees received adequatetraining
to fulfill their security responsibilities?
Subordinate Questions
13.1.3 Is there a mandatory annual refresher
training?
13.1.1 Have employees received a copy ofthe
Rules of Behavior?
PerformanceObjectives
13.1.4 Are methods employed to make employees
aware ofsecurity, i.e., posters, booklets?
13.1.5 Have employeesreceived a copy ofor have
easy access toagency security procedures and
policies?
Implementation Evidence
of employees who signed employee agreements
of employees who received annual refresher
training
Metrics
of new employees who underwent security
awareness training
. . .
21
Example 1 Security Awareness, Training, and
Education (Implementation)
Critical Element 13.1 Have employees received
adequate training to fulfill their security
responsibilities? Subordinate Questions 13.1.1
Have employees received a copy of Rules of
Behavior? 13.1.5 Have employees received a copy
of or have easy access to agency security
procedures and policies?
Metric
Percentage employees who underwent initial IT
security awareness training
To determine the number of new employees who
underwent required IT security awareness
training, including receiving a copy of Rules of
Behavior and security policies and procedures.
Purpose
1. Is security awareness training required for
new employees?
Yes
No
ImplementationEvidence
2. Do employees receive a copy of Rules of
Behavior as a part of their training?
Yes
No
3. Do employees receive a copy of policies and
procedures or a summary with references as a part
of their training?
Yes
No
4. How many new employees joined your
organization during current past reporting
period? Fill in the blank____________ How many
of those took the training? Fill in the
blank______________
Semi-Annually
Frequency
Formula
If response to 1-3 was Yes ( who took
training) / (Total of new employees)
Data Source
Security awareness training tracking by the
responsible office, or manual or automated system.
Security awareness training is effective when it
includes specific information on rules of
behavior and information on security policies and
procedures. This metric validates the content of
training and determines the percentage of
employees who took the training. High numbers
close to 100 are highly desirable. Having
employees who do not understand the security
implications of their actions significantly
increases the number of security incidents caused
by insiders. This may also increase the
opportunities for unauthorized access to systems
caused by staff not following security policies
and procedures.
Indicators
22
Example 2 Incident Response Capability
(Efficiency)
Critical Element 14.1 Is there a capability to
provide help to users when a security incident
occurs in the system? Subordinate Question
14.1.3 Are incidents monitored and tracked until
resolved?
Elapsed time between when an incident is reported
and when affected partiesare notified of a
problem.
Metric
To determine how long it takes to notify users of
incidents
Purpose
What is an average number of hours that it takes
to notify users that may be affected by a
reported incident?
ImplementationEvidence
Under 1
Under 24
Under 12
Over 24
Monthly
Frequency
At the program, level average time weighted
average among all responses
Formula
Incident Response Database
Data Source
Users should be notified of a problem as quickly
as possible. A delay in notification time may
point to a lack of incident monitoring and
tracking until resolution. Higher number points
to inefficiency in (or lack of) a notification
process. This metric can also be calculated as
percentage of all incidents that are communicated
to affected parties within 1 hour, 12 hours, 24
hours, and over 24 hours.
Indicators
23
Example 3 Hardware and System Software
Maintenance (Implementation)
Critical Element 10.3. Are systems managed to
reduce vulnerabilities? Subordinate Question
10.3.2 Are systems periodically reviewed for
known vulnerabilities and software patches
promptly installed?
Metric
Percentage of systems with latest patches
installed
To quantify the level of risk exposure caused by
the lack of current security patch implementation
Purpose

1. Is regular vulnerability scanning conducted?

ImplementationEvidence
Yes
No
2. How many components are scanned every time?
Fill in the blank ________ 3. How many of those
had up-to-date patches during the last scanning
cycle? Fill in the blank ________
4. If your answer to question 1 was no, what
was the reason?
Insufficient staff
Insufficient funding
Superseding other priorities
Frequency
Monthly
( components with up-to-date patches)/(Total
of components)
Formula
Data Source
Regular Vulnerability Scanning
This metric monitors compliance with applicable
patches and provides useful information about the
level of risk exposure at a system level. The
goal in this case is 100. The desired trend for
this metric is downward. Question 3 identifies
why a patch compliance validation process may be
lacking and points at specific corrective actions
to facilitate establishment of such process.
Indicators
24
Metrics Development Criteria What is a Good
Metric?

• ? Based on IT security performance goals and
  objectives NIST SP 800-26 Critical Elements and
  Subordinate Questions are used to derive
  performance goals and objectives
• ? Quantifiable Metrics should yield
  quantitative rather than qualitative information
  to increase the objectivity and validity of data
• ? Obtainable/Feasible to measure Metrics data
  should be available or easily collected through
  interviewing or by accessing data repositories.
  If a metric requires significant modification of
  agency processes or implementing a new tool, data
  collection may not be feasible at this time
• ? Repeatable Measurements should be able to be
  repeated in a standard way at predetermined
  intervals to identify trends or identify if
  positive changes have occurred as a result of
  corrective actions
• ? Provide relevant performance trends over
  time Repeated measurements reveal change in a
  timely manner
• ? Useful in tracking performance and directing
  resources Metrics should be useful to
  stakeholders and should yield information that is
  important in financial decision making

25
Breakout Session
26
Breakout Session

• Goal To complete a Metric Form for one of the
  metrics that is required for GISRA reporting for
  FY 2002. This includes identifying the NIST SP
  800-26 Critical Element and Subordinate Question
  that map to the specific GISRA question from OMB
  guidance.
• Duration 30 minutes
• Method
• Read the metric your Breakout Group is assigned
• Select the NIST SP 800-26 Critical Element that
  includes your metric
• Select the Subordinate Question within the
  Critical Element that maps to your metric.
  Remember, a single metric can use more than one
  Subordinate Question
• Complete the Metric Forms sections, giving
  particular attention to what implementation
  evidence may exist that corresponds to your
  Subordinate Question
• Follow up Each Group will have five minutes to
  brief their Form to the other groups. This
  brief should include
• The metric your Breakout Group was assigned
• The Critical Element and Subordinate Question
  that maps to your metric.
• The completed Metric Form, including
  implementation evidence and indicators. A list
  of possible sources of the data you need to
  uncover for your metric

27
Critical Element Subordinate Question
Metric
Purpose
ImplementationEvidence
Frequency
Formula
Data Source
Indicators
28
Metrics Program Implementation
29
In this section, you will

• Receive an introduction to the IT Security
  Metrics-related roles and responsibilities
• Learn the steps involved in IT Security Metrics
  program implementation by learning the process
  and following an example through the process

30
Multiple success factors can influence quality
and sophistication of IT Security Metrics (slide
1 of 2)

• Ensure that IT Security Metrics Program is
  manageable
• Use no more than 10-20 metrics at a time, based
  on current priorities
• Phase old metrics out and phase new metrics in
  when performance targets are reached or when
  requirements change
• Ensure acceptable quality of data
• Data collection methods and data repositories
  should be standardized
• Events must be reported in a standard manner
  throughout the organization and the results of
  such reports need to be stored in the data
  repository

31
Multiple success factors can influence quality
and sophistication of IT Security Metrics (slide
2 of 2)

• Obtain organizational acceptance
• Metrics need to be validated with organizations
  stakeholders within headquarters and in the field
• Metrics should be vetted through appropriate
  approval channels
• Ensure that metrics are useful and relevant
• Useful data should be collected
• Not all data are useful

32
Metrics-related roles and responsibilities are
dispersed throughout an organization
33
Each organization will implement a metrics
program specific to its needs

• Tailor to organization and business processes
• Identify IT Security Metrics-related stakeholder
  roles and responsibilities
• Lay out required infrastructure changes, such as
  creation of web-based data collection tools and
  of new data repositories
• Identify required modifications of the current
  data sources
• Define data reporting formats

34
Output from standard security activities can be
used to quantify IT security performance

• Incident Handling
• Testing
• Network Management
• Audit Logs
• Network and System Billing
• Configuration Management
• Contingency Planning
• Training
• Certification and Accreditation

IT Security Metrics data collection must be as
transparent and non-intrusive as possible.
35
IT Security Metrics Program Implementation Process

• Analyze collected data
• Conduct gap analysis - Identify gaps between
  actual and desired performance
• Identify reasons for undesired results
• Identify areas requiringimprovement

• Determine range of corrective actions
• Select most appropriate corrective actions
• Prioritize corrective actions based on overall
  risk mitigation goals

• Develop cost model - Project cost for
  each corrective action
• Perform sensitivity analysis
• Develop business case
• Prepare budget submission

• Identify stakeholders
• Determine goals / objectives
• Review existing metrics
• Develop new metrics
• Identify data collection methods and tools
• Collect metrics

• Budget allocated
• Available resources prioritized
• Resources assigned

• Management
• Operational
• Technical

• Track progress
• Report as required

36
Process Implementation Example
Lack of IT security refreshertraining may be
causing weakpasswords, identified bya password
cracker that isrun regularly.
Employees should be required to take annual IT
security refresher training as part of their
annual review process.
Since annual refresher training has ceased, the
number of weak passwords has increased by 50.
Only 5 of employees receive annual IT security
refresher training.

• A budget submission detailingmetrics findings
  related to annual IT security refresher training
  was submitted, and funding received.

Annual refresher training,an operational
control,is instituted.
Since the training was re-instituted, the
percentageof weak passwords hasdecreased by 40
37
Summary

• Discussed why Metrics are important for IT
  security
• Obtained understanding of the relationship
  between GISRA, NIST SP 800-26, and IT Security
  Metrics
• Described IT Security Metrics
• Described the Metrics Development Process
• Created metrics to be implemented at a system
  level through applying metrics development
  process
• Discussed metrics-related Roles and
  Responsibilities
• Described how to implement a Metrics Program

38
Next Steps

• You can immediately use what you have learned
  today to propose some metrics within your agency
• Notes of the workshop will be published in two
  weeks
• You can use the three metrics presented during
  the workshop and those that we developed together
  for your GISRA submission
• Metrics Guidance first draft will be published by
  September 30, 2002
• Please contact Marianne Swanson if you have any
  questions at marianne.swanson_at_nist.gov,
  301-975-3293