Security Measures and Metrics

1
Security Measures and Metrics

• Pete Lindstrom
• Research Director
• Spire Security

2
Agenda

• Elements of metrics
• Interlude Four disciplines
• Back to metrics
• ROI/ROSI

3
Status of security

• Difficult to define good security
• Minimal difference between security and lucky
• We dont know how to measure success.
• One incident doesnt necessarily mean failure

4
Key elements of security metrics
5
Key elements of security metrics
Activities Four Disciplines
People Admins by Department
Time Hr/Day Month/Yr
Costs Salaries, Consulting HW, SW, Maint.
Resources User accts, systems, apps
Building Blocks
Lets put them together
6
Agenda

• Elements of metrics
• Interlude Four disciplines
• Back to metrics
• ROI/ROSI

7
Process Effectiveness Metrics
Process Effectiveness
a.k.a. doing things right

• Elements
• Activities
• errors

error rates

• For example
• Accts per person
• Vulns per person
• Patches per person

8
Security reference model
4. Monitor/detect inappropriate and/or malicious
activity
2. Control sources (users/others)

• Harden the
• Infrastructure

3. Harden the Process/data
9
Four disciplines of security management
Threat MANAGEMENT
Identity MANAGEMENT
Identity Validation Account Management Password
Management
Threat Identification Security Monitoring Incident
Management
INLINE
Authentication User Access Control
Intrusion Prevention
Encryption Integrity
System Access Control
Policy Management Security Arch. Design Ticket
Management
Vulnerability Assessments Patch
Management Software Security
Trust MANAGEMENT
Vulnerability MANAGEMENT
10
Identity management

• Functions
• Identify users
• Assign accounts/rights
• Maintain identity (passwords)
• Validate sessions
• Authorize access

11
Vulnerability management

• Functions
• Scan for exposures
• Eliminate vulnerabilities
• Remediate vulnerabilities
• Mitigate vulnerabilities
• Manage compliance

12
Trust management

• Functions
• Write policies
• Design security
• Ensure confidentiality
• Ensure integrity

13
Threat management

• Functions
• Analyze traffic
• Analyze logs
• Manage incidents
• Conduct forensics

14
Agenda

• Elements of metrics
• Interlude Four disciplines
• Back to metrics
• ROI/ROSI

15
Process Effectiveness Metrics
Process Effectiveness
a.k.a. doing things right

• Elements
• Activities
• errors

error rates

• For example
• Accts per person
• Vulns per person
• Patches per person

16
Process effectiveness

• Error rates
• Identity management
• Request errors
• Vulnerability management
• Vulnerabilities remaining
• Threat management
• Incident response
• Trust management
• Policy violations

17
Staff Productivity Metrics
Staff productivity
a.k.a. people doing things better

• Elements
• People
• Activities

• For example
• Accts per person
• Vulns per person
• Patches per person

18
Staff productivity

• Productivity and workload for all manual
  activities (activities/people)
• Identity management
• Requests per administrator
• Account disablements per admin
• Password resets per admin
• Vulnerability management
• Vulnerabilities resolved per administrator
• Threat management
• Incidents per person
• Trust management
• Policy changes per person

19
Cycle Time Metrics
Cycle Time
a.k.a. avg time to perform activity x

• Elements
• Time
• Activities

• For example
• Accts per month
• Vulns fixed per month
• Patches per month

20
Process efficiency (cycle time)

• Time/activities
• Identity management
• Request time
• Vulnerability management
• Remediation time
• Threat management
• Incident response time
• Trust management
• Policy creation time

21
Efficiency Metrics
Admins by Department

• Elements
• People
• Activities
• Time

Efficiency
a.k.a. people doing things quicker
2000 Hours per FTE

• For example
• Accts/person/hr
• Vulns/person/hr
• Patches/person/hr

22
Cost Effectiveness Metrics
Admins by Department
Cost effectiveness

• Elements
• People
• Activities
• Costs

a.k.a. people doing things cheaper
Salaries, Consulting Fees

• For example
• Cost per acct
• Cost per vuln fixed
• Cost per patch

23
Cost effectiveness

• Dollars/activities dollars/resources
  dollars/demographics
• Identity management
• Cost per request
• Cost per password reset
• Vulnerability management
• Cost per vulnerability
• Cost per system setting
• Threat management
• Cost per incident
• Trust management
• Cost per policy
• Cost per project

24
When to use metrics

• Process effectiveness
• Six Sigma
• Staff productivity
• ROI / promotions
• Cycle time
• Balanced scorecard
• Efficiency
• ROI
• Cost effectiveness
• Activity-based costing
• ROI/TCO

25
Business uses of security

• Benchmarking (Balanced scorecard)
• Baselining (Six Sigma)
• Activity-based costing/Mgt
• ROI
• Risk management (ROSI)

26
Missing Element RISK!

• Elements
• Activities
• Resources

Risk Management
a.k.a. people doing things more securely!

• Four Disciplines
• Identity Mgt
• Vuln Mgt
• Trust Mgt
• Threat Mgt

27
Risk metrics

• Resources/resources resources/demographics
• Identity management
• User accounts per application
• Vulnerability management
• Vulnerabilities per resource
• Threat management
• Incidents per resource
• Trust management
• Policies per resource

28
Risk effectiveness

• Activities/activities (automated)
• Identity management
• Failed logins/total logins
• Vulnerability management
• Access denied/total access
• Threat management
• Incidents/events
• Trust management

29
Agenda

• Elements of metrics
• Interlude Four disciplines
• Back to metrics
• ROI/ROSI

30

• Examples
• Return on Investment (ROI) Return on Security
  Investment (ROSI)

31
The elements of value (Loss)

• ROI
• IT productivity (time)
• User productivity (time)
• these also have ROSI value
• ROSI
• Legal/regulatory costs (fees/fines)
• Direct revenue
• Stored asset value (intellectual property,
  financial assets)

32
Lets talk ROI

• Keyword is efficiency
• Reduced Capital Expenditures (CapEx)
• Lower h/w, s/w costs
• Scalability, manageability, performance
• Reduced Operating Expenditures (OpEx)
• Lower IT, end-user costs
• (higher productivity)

33
Productivity

• Where users and IT spend their time.
• Time-is-money philosophy.
• Often the only aspect of loss we quantify.
• Basic source of ROI.
• Hourly rate x hours of effort.
• In order to determine the value of activities,
  you first have to determine what activities are
  performed.

34
Identity management ROI

• Provisioning
• New employee productivity
• Automated account management
• Password management
• Reduced help desk time
• Employee productivity
• Web access control
• Developer efficiency (build vs. buy)

35
Trust management ROI

• Public Key Infrastructure
• Managing certificates
• Virtual Private Networks
• Leased lines
• SSL Acceleration
• Hardware efficiency

36
Vulnerability management ROI

• Firewalls
• Reduce ACL management
• Vulnerability assess/remediate
• Reduce manual efforts
• Patch management
• Automate patching
• Software quality
• Reduce bug fixes

37
Threat management ROI

• Antivirus
• Recovery of systems
• Network IDS
• Reduce manual detection/forensics
• Host IDS
• Manual log efforts
• Security Event Management
• Aggregation/prioritization of work

38
Getting to ROI

• Identify amount of labor allocated to individual
  security activities.
• Identify solution and its corresponding
  activities.
• Identify labor difference with and without
  solution.

39
The roots of ROSI

• Our overall objective is to reduce risk.
• We are relatively new to spending on solutions.
• We often didnt really do anything that was
  considered a recurring expense (I am guessing a
  bit here).
• But, the Internet has changed all that (or at
  least made it apparent).

40
Return on Security Investment

• Keyword Effectiveness
• Effectiveness Reduced risk
• Protecting Value and Loss
• Legal/regulatory costs (fees/fines)
• Direct revenue
• Stored asset value (intellectual property,
  financial assets)

41
Legal/regulatory costs

• Lawsuits
• Privacy suits
• Downstream liability
• Legal fees
• Regulatory issues
• Regulatory fines
• Remediation costs

42
Direct revenue

• E-Commerce systems
• Level of materiality
• Seasons, cycles, forecasts drive expected losses
• Some benchmarks shrinkage materiality (internal
  controls)

43
Stored asset value

• Stored Value (financial assets)
• Stored Knowledge (intellectual property)
• Market Cap (or equivalent) Book Value
  Goodwill (intangible assets)
• Some of this Goodwill is attributable to
  information assets.
• Professional services higher percentage
• Contract manufacturing or retail - lower

44
Determining loss

• No physical goods
• Ubiquitous supply
• Full asset value is not necessarily lost
• Look at loss in other ways
• Type of loss
• For each application/system

45
Types of losses

• How much value would be lost under the following
  conditions (for each app/dataset)?
• Information-centric loss
• Modified data (Integrity)
• Copied data (Confidentiality)
• Deleted data (Availability)
• System/App-centric loss
• Resource availability (Productivity)
• Resource misuse (Liability)

46
Loss potential
Read Modify Delete Avail Misuse
Asset Value H M M L L
Revenue M H H H L
Fines M/H H L L ?
IT Prod. L H M L M
EU Prod. L L M H M
47
Calculating potential loss
Annual Loss Expectancy Probability x Value ALE
P x A (Insurance Industry)

• Level One Calculate overall loss potential in 5
  categories.
• ALE P x L(Assets, Revenue, Fines, IT Prod, EU
  Prod)
• Level Two Take above and factor in types of
  losses.
• ALE P x (C(A,R,F,I,E) I(A,R,F,I,E)
  A(A,R,F,I,E))
• Level Three Perform above for all
  applications/data.
• ALE P x App1(C(A,R,F,I,E) I(A,R,F,I,E)
  A(A,R,F,I,E)) Appn(C(A,R,F,I,E) I(A,R,F,I,E)
  A(A,R,F,I,E))

48
Getting to ROSI

• Determines cost effectiveness of proposed
  solution.
• Calculate losses with and without solution.
• Compare the difference.

49
Agree? Disagree?
Pete Lindstrom petelind_at_spiresecurity.com www.spir
esecurity.com