NGA Justice Information Sharing Regional Training

1

NGA Justice Information Sharing Regional
Training Achieving Implementation Measuring
Progress and Outcomes Performance Measurement
Tools for Information Technology Projects Funded
by the Bureau of Justice Assistance Grant
2002LD-BX K002 August 8, 2003 Denise L. Baer,
Ph.D. Consultant Center for Society, Law and
Justice University of New Orleans
2
CSLJ PROJECT PURPOSES

• Support DOJ Accountability
• 1993 Government Results Performance Act (GPRA)
• 1996 Information Technology Management Reform
  Act (ITMRA)
• The Presidents Management Agenda
• OMBs Program Assessment Rating Tool
  (PART)
• 2. Support DOJ IT Initiative Which Seeks To
• Promote implementation of integrated IT
  systems
• Facilitate the exchange of appropriate
  information
• Build consensus on models, approaches, and
  standards.
• 3. Provide Reporting Tools to States and Local
  Units of Government That
• Are easy to use reduce Federal reporting
  burdens
• Mesh with State and local-level evaluation and
  policy needs
• Support performance-based management
  benchmarking

3
PROJECT METHODS

• 12 Month Project to Build Tools
• Field-driven Approach
• Work with Project Leaders/Real Projects 26 NGA
  Projects
• Build Consensus on What to Measure
• Incorporate Existing Data Measures
• Address Field-Driven Criteria and Needs
• Identify Meta Data Issues
• Project Staff
• Dr. Peter Scharf, CSLJ Executive Director
• Dr. Denise Baer, Project Consultant
• Dr. Michael Geerken, Project Consultant
• Dr. Heidi Unter, CSLJ Program Coordinator

4
CSLJ PM PROJECT TOOLS

• Business Case for PM
• BRIEFING PAPER Benefits/Case for Performance
  Measurement
• 2. Plug Play Resources for States
• Inventory of Performance Measures
• Logic Models for Criminal History
• 3. Facilitation Tools
• Steps in Facilitating a Logic Mode/Theory of
  Change Team
• Exercise on Developing Performance Measures
• Exercise on Identifying a Business Process for
  Justice Integration
• Performance Path Measurement Matrix Tool
  Worksheet
• 4. Capacity-Building
• Definition of Attributes of Justice Performance
  Measurement
• Identification of Analytical and Methodological
  Issues

5
INVENTORY OF PERFORMANCE MEASURES
MISSION VISION STRATEGY
PROJECT IMPLEMENTATION MEASURES IT Project
Milestones Project Management
IT SYSTEM OUTPUT MEASURES Data
Report Outputs Number Served Usability of
System Data Quality Quality Assurance
Functional Requirements Compliance
Availability of System Resources System Capacity
IT SYSTEM CAPACITY MEASURES
INPUTS?PROCESS MEASURES?O U T P U T M E A S U R
E S? O U T C O M E M E A S U R E S
6
LESSONS LEARNED

• Demand is High for PM Tools
• Need for Tools and Training Paramount
• Performance Measurement Increasingly Required
• Tight Budgets Competing Demands
• 2. Integration Projects Not Where We Thought
• Some Legislature Imposed Measures
• Existing Measures -- Only Simple Measures
• Most are NOT Outcome Oriented
• Many Justice Professionals Unfamiliar with PM
• 3. Justice Arena Unique Challenges for PM
• Lack Advantages of Health Care Field
• State/Local Role and Split-Sovereignty

7
PUBLIC PERFORMANCE MEASURES

• Six Key Characteristics
• Target End Outcomes
• Population-Based (i.e., geographic defined,
  accountable entities)
• Drawn from Routinely Collected Data
• Track Accountability for Managing as well as
  Planning and Evaluation
• Should be Derived From Core/Priority Strategies
• Produces Intended Result When Tied to
• Measurement of a Stable, Managed Business
  Process

8
USE LOGIC MODEL APPROACH
INPUTS are your resources, while OUTPUTS are the
product of an activity OUTCOMES are not what the
program or initiative itself does, but the
consequences of the program/initiative efforts --
changes in conditions, attitudes or behavior of
individuals or outcomes for agencies and
communities. The STRATEGIC PLAN and THEORY OF
CHANGE links these together
9
DEFINING PERFORMANCE MEASURES
SAMPLE EXERCISE Identifying Performance Measures
10
DEFINING INTEGRATION OUTCOMES
ULTIMATE GOAL Enterprise-Wide Justice
Integration Information that is More
Accurate More Timely More Complete Less Expensive
BETTER JUSTICE OUTCOMES Public
Safety Accountability Less Recidivism Reduced
Crime
MATURE SYSTEMS
AUTOMATED PROCESSES Cross-System
Integration Reduced Cycle Time Outcome-Based Perfo
rmance
MORE BENEFITS More Efficient More
Effective Better Evaluation Data Better Targeting
of Programs Services
By Creating
To Result in
To Produce
but what are the PROCESSES that
define integration?
11
META-DATA ISSUE 1 .From the
Perspective of Process Management
Factor Improvement Reengineering
LEVEL OF CHANGE Incremental Radical
STARTING POINT Existing Process Clean Slate
ORIGINS Bottom-Up Top-Down
SCOPE Narrow Functional Broad Cross-Functional
BENEFITS Small Large
FOCUS Cycle Time - Efficiency Effectiveness
RISK Low to Moderate Moderate to High
PRIMARY ENABLER Statistical Control Information Technology
12
META-DATA ISSUE 2
..From the Perspective of Research Metrics
IMPLEMENTING MANAGING MONITORING
BEFORE
AFTER
?
ROI Return on Investment Business Case
Cost-Benefit Analysis Program Evaluation
Performance Measurement is a Metric that Links
All 3
13
META-DATA ISSUE CHALLENGES
Integration is a KEY ENABLER PM is Measurement
of a PROCESS PM is a Hybrid (NIST Baldrige
Criteria) Performance Measurement
Identifying the PROCESSES that define
integration is CRITICAL for PM
14
DEFINING A BUSINESS PROCESS

1. Start, End and Purpose
2. Crosses Functional Units/Org Boundaries
3. Five Levels ad hoc to optimized
4. Produces Normative Outcome (value)
5. Event or Activity Driven
6. Defined Processes Defined Inputs Resources
7. Subprocess smaller value chains
8. Processes are often Invisible Until Mapped or
   Defined Strategically
9. Manage Through Measurement
10. Efficiency Reduces CYCLE TIME
11. PROCESS REENGINEERING Dramatic Improvements

SAMPLE EXERCISE What is the Business Process
for Integration?
15
IS IT THE STAGE OF INTEGRATION ?
ULTIMATE GOAL Enterprise- Wide Justice
Integration
16
MATURE PROJECTS ARE MORE CAPABLE
ENTERPRISE-WIDE INTEGRATION 2
SYSTEMS SHARED PRACTICE
DISCRETE PROCESS
INTEGRATION AS-NEEDED
CRITICAL INFORMATION SHARING
IT SYSTEM PERFORMANCE
LONGTERM
INTERMEDIATE
MATURE INTEGRATION PROJECTS
SHORT-TERM
CRITICAL THRESHOLD
SYSTEM CAPACITY BUILDING
EXPANDING ELECTRONIC ACCESS
ABILITY to ACHIEVE LONGTERM OUTCOMES
IMMATURE INTEGRATION PROJECTS
AD HOC INFORMATION SHARING
PROCESS MEASURES
PAPER AND PENCIL SYSTEMS
17
IS IT THE EXCHANGE OF INFORMATION ?
OPEN CASE FILE
COURT DISPOSITION
SEAL RECORDS
INCIDENT
ARREST
CUSTODY
INCARCERATION
1
C O R R E C T I O N S
INVESTIGATION
DETENTION
2
PRE-DISPOSITION SUPERVISION
3
PRE-DISPOSITION COURT
4
POST-DISPOSITION COURT
5
POST-DISPOSITION SUPERVISION
6
7
INCARCERATION
8
POST-SUPERVISION
VISUAL MODELING OF JIEM INFORMATION
EXCHANGE PROCESSES (SEARCH)
18
DEPENDS ON WHAT TYPE OF PROCESS 3
TYPES Which is Integration?

• A Management Process
• 2. A Support Process
• 3. A Core Mission Process

19
CORE MISSION PROCESSES
Expected Organizational Outcomes
Development and
Alignment diverge/converge goals, objectives
and measures
Mission Statement
Strategic Goal
Vital Few Performance Goals
Communication, Collection, Analysis, Use
Vital Few Measures
Management And Staff Performance Expectations
20
PERFORMANCE MEASUREMENT SYSTEMS

• Key Checklist Questions
• Is the program aligned with public policy
  purposes?
• Is the program managed to maximize achievement of
  these purposes and goals?
• Are reasonable performance targets and indicators
  identified that track progress towards achieving
  these purposes and goals?
• Are the inferences about data indicators
  justified?
• Are the performance indicators tracked and
  reported on a regular basis?
• Are the performance data used to revise policy
  and
• program goals and management?

SAMPLE TOOL Draft Checklist for Developing
Performance Measures
21
META-DATA TOOL

• Importance/Priority of Outcome Area
• a. High priority for maximizing public order
  and safety of persons
• or populations.
• b. Financially important
• c. Demonstrated variation in services
  and/or potential for improvement
• 2. Usefulness in improving Public Safety or
  Order
• a. Based on established scientific or
  practice recommendations
• b. Potentially actionable by the user
• c. Meaningful and interpretable to the user
• 3. Measure Design
• a. Well defined specifications
• b. Documented reliability
• c. Documented validity
• d. Allowance for risk

SAMPLE TOOL Desirable Attributes for Justice
Performance Measurement
22
META-DATA DILEMMA FOUR DIFFERENT
LANGUAGES CRITICAL TO PERFORMANCE MEASUREMENT

• Policy Strategies
• Program Strategies
• Management Strategies
• Accountability Strategies

23
FACILITATION TOOLS

• Invest in Performance Measurement
• 2. Form a Team That Includes all 4 Languages
• Identify the Business Process
• Specify the Logic Model using the Theory of
  Change Method
• Decide the Audience / Level of the Measures
• Keep in Mind that the Model Should be Plausible,
  Doable and Testable
• Aim for Compound Measures Using Attributes
• List These Strategies, Activities, and Outcomes
  on the Performance Path Matrix Tool and Worksheet
• Revise as Appropriate

24
FACILITATION TOOLS

• Assumptions Factors to Consider
• There will be different TYPES of Logic Models
  (e.g., access vs. capacity-building vs. discrete
  outcomes)
• Within these types, there may be different LOGIC
  MODEL GOALS.
• Each will need to be mapped to Program
  COMPONENTS, PROCESS MEASURES AND OUTPUTS.
• Each Logic Model will have Stages of
  Time-Sequenced Initial/Short-term, Intermediate
  and Longterm OUTCOMES.
• For each Stage, there will be appropriate EVENTS
  and MEASURES (Indicators)

SAMPLE TOOL Steps in Facilitating a PM Logic
Model/ Theory of Change Team
25
DEVELOPING A THEORY OF CHANGE

• Process for Surfacing
• Articulating a Theory of Change
• Use Focus Group Format
• Start with Longterm Outcomes
• Work Backward Towards Initial Activities
• Map Required Existing Resources
• Reconcile Multiple Theories of Change
• Adapted from J.P. Connell, J.L. A.C. Kubish, L.B.
  Schorr C.H. Weiss, 1995, New Approaches to
  Evaluating Community Initiatives, Aspen
  Institute.

26

S O C I A L I N D I C A T O R S

ACCOUNTABILITY STRATEGIES
Outcome Measure
Outcome Measure
Outcome Measure
Outcome Measure
System Change Measure
POLICY STRATEGIES
Resources Authority Programs
Outcome
Outcome
Outcome
Final Outcome
CRIMINAL JUSTICE PROGRAM STRATEGIES
Activities
Activities
Activities
Activities
Start-Up Activities
Output Measure
Output Measure
Output Measure
Output Measure
Formative Measure
MANAGEMENT STRATEGIES
W O
R K L O A D I N D I C A T O R S

SAMPLE TOOL PERFORMANCE PATH MEASUREMENT
MATRIX MODEL
27

PLEASE EVALUATE THE TOOLS
Performance Measurement Tools for IT
Projects www.cslj.net CONTACT Dr. Peter
Scharf Executive Director Center for Society, Law
and Justice University of New Orleans 3330 North
Causeway Boulevard, Suite 413 Metairie, LA
70002 [phone] Dr. Denise L. Baer Project
Lead [phone] src_dlbaer_at_hotmail.com dbaer_at_
gwu.edu