Russ Ryan, Vice President

1
The Importance of Biometric Testing

• Russ Ryan, Vice President
• National Biometric Security Project

2
National Biometric Security Project

• Biometrics for National Security (BiNS)
• National Signatures Project
• National Energy Technology Lab (NETL)
• NIST
• International Organization for Migration
  (IOM)
• Office of Presidential Affairs (UAE)
• International Labour Organization (ILO)
• BioAPI Consortium
• State of West Virginia

NBSP
3
Biometric Applications
?

• Robust biometric passports
• Financial and medical services authorizations
• Border and travel services
• Drivers licenses
• Physical and Logical access

GAO
HSPD-24
4
Understanding Biometric Performance

• Increasing reliance on biometrics to secure
  access, transactions identity
• Equally increasing demand for accurate,
  unbiased evaluations
• Testing can provide accurate metrics on how
  the technology will perform in the real world
  
• Alleviating unfounded concerns about
  operational performance

5
Attributes of an Ideal Biometric

• Universal
• Unique
• Permanence
• Collectable
• Performance
• Acceptance
• Spoof Resistance

6
Biometric Testing Today

• Performance of biometric systems is a function
  of
• strength of the underlying biometric.
• quality and information content of the input
• configuration and architecture of the system
• the relationship of accuracy and throughput
• error rates, the nature of failures and their
  cost,
• and system vulnerabilities which contribute
  to an
• overall assessment of system performance
• Increasingly, biometric devices are components
  of larger systems imposing external variables
  that impact biometric system performance in the
  field

7
Biometric Testing Today

• Three major considerations in testing biometric
  products
• dependence of measured error rates on the
  application
• need for a large test population
• necessity for a time delay between enrollment and
  testing

8
Comparison of Testing Types

• Technology Testing
• Goal Produce a repeatable and scalable
  assessment of an algorithm/sensor using offline
  data processing
• Scenario Testing
• Goal Determine overall system performance (both
  algorithmic human factors performance measures)
• Operational Testing
• Goal Determine biometric system performance in a
  specific environment with a specific target
  population
• Best Practices in Testing and Reporting
  Performance of Biometric Devices, by A. J.
  Mansfield, National Physical Laboratory and J. L.
  Wayman, San Jose State University. Published 2002
  by The Centre for Mathematics and Scientific
  Computing ,National Physical Laboratory, Queens
  Road, 88, Middlesex, England.

9
Technology Testing

• Understand/compare software techniques used to
  acquire, process and compare biometric data
• Main focus is on the pattern matching
  technique used to compare biometric data
• Evaluates different classification and matching
  methods on efficiency, speed and performance
• Offline processing of data carried out in
  laboratory
• Evaluation compares competing algorithms
• from a single type of technology
• carried out on a standardized database
• collected by a universal sensor
• results determine the relative
• effectiveness of the tested algorithms

10
Scenario Testing

• Evaluates performance across biometric devices
• Each system has its own acquisition sensor and
  receives different data inputs than those tested
  in technology (algorithm) evaluation
• Data collected for all tested systems must come
  from same environment and same population
• Test results are only considered repeatable
  under identical control variables environment
• Scenario evaluation helps an end user decide
  which biometric device has the potential to work
  best for his/her needs

11
Operational Testing

• Determine performance of a biometric system in
  a real application environment
• Population and environment are not controlled
• System vulnerability can also be performed

Helps determine how system as a whole will
perform by testing a live system in its native
environment for its intended application
12
Conformance Testing

• Determines conformance with relevant
  published ISO/IEC standards
• Utilizes conformance test suites designed for
  specific standards
• Evaluations will expand to include additional
  standards as the software modules are written
  and field tested

Standards Evaluated Target Value
INCITS 377-2004 Pass/Fail
INCITS 378-2004 Pass/Fail
ISO 19794-2-2005 Pass/Fail
INCITS 379-2004 Pass/Fail
INCITS 381-2004 Pass/Fail
INCITS 385-2004 Pass/Fail
ISO 19794-2-2005 Pass/Fail
INCITS 396-2005 Pass/Fail
INCITS 395-2005 Pass/Fail
ILO SID Pass/Fail
ICAO LDS 1.7 Pass/Fail
BioAPI Pass/Fail
13
Vulnerability Testing

• Impersonation attempts (disguises) or spoofing
  (artifact substitution for live feature)
• Database attacks (exchanging or corrupting
  references)
• Tampering with threshold settings
• Network-based attacks

Product vulnerabilities must be defined in the
context of the operating environment and proper
usage within the design parameters of the product
14
Interoperability Testing

• Multi-modal systems demand acceleration of
  biometric interoperability
• Interoperability testing assesses
• ability to exchange and use information on a
  single system in a multi-modal environment
• interface of the biometric component with
  the holistic security program

15
Interoperability Trade-offs
Standard
Proprietary
Lowers complexity of the application -   Re-use -   Future Proofing -   Vendor independence -   Upgrade path -   Simplifies CM -   Simplified integration Product optimization Better performance Lower level control More sophistication Can be faster to market (due to standards development time)
May incur additional overhead -   May not be able to take advantage of vendor unique capabilities -   Interfaces are generic and consensus based, so may not be optimized for a particular use Custom interfaces for each proprietary product to be interfaced Increased cost/complexity Added CM -  Product changes affect application -  Can result in vendor dependence
Advantages
Disadvantages
Courtesy of Cathy Tilton, VP Standards
Technology, Daon
16
Usability Testing

• Intuitiveness of the system interface with the
  user community
• Is the transaction an inviting and positive
  experience?
• Is consistent instruction and feedback built into
  the process?
• Is the performance reliable for operational staff
  as well as users?

17
Qualified Product List Testing

• First initiated and commercialized by NBSP
• Utilizes comprehensive scenario test capability
• Initially used to identify products that
  successfully passed common performance thresholds
  
• Increasingly tailored to the application

18
QPL Testing Benefits

• Catalog of commercially available products that
  meets minimum standards for a specific
  application
• Significant reduction in duplicative pilot tests
• Acceleration of acquisition process by
  identifying
• a field of suitable products
• Opportunity for vendors to demonstrate
• general or specified performance capabilities

19
Factors Affecting Biometric Performance

• Variations in
• biometric pattern
• the way users present the biometric
• the way the sensor reads the biometric
• System scalability
• the transmission process (including noise
  introduced by compression expansion)
• User acceptance/application-specific limitations

20
Additional Measurement Parameters

• Reliability, availability, scalability,
  maintainability
• Security, including vulnerability to spoofing
• Human factors, including user acceptance
• Cost/benefit in comparison to existing security
  processes and systems
• Privacy regulation compliance

21
Laboratory Certification

• BSI awarded ISO/IEC 17025 Accreditation
• specifies requirements for competency to conduct
  biometric tests
• covers testing performed using standard methods,
  non- standard methods and laboratory-developed
  method
• laboratory customers, regulatory authorities
  and accreditation bodies use it to confirm the
  competency of laboratories.
• NIST
• NIST Handbook 150-25 with technical requirements
  and guidance for accreditation of laboratories
  under the NVLAP Biometrics Testing program
  released Sept. 2009

22
The Importance of Biometric Testing

• Russ Ryan, rryan_at_nationalbiometric.org
• 703-201-8179
• www.nationalbiometric.org