Decision Support Building Analytical Skills

1
Decision SupportBuilding Analytical Skills

• ASMC PDI
• San Antonio, TX
• May 28, 2009
• Ron Maccaroni
• rmaccaroni_at_cox.net

2
Decision Support

• Definition
• Enabling decision makers to make informed
  decisions that wisely and better utilize
  resources, and improve program and operational
  effectiveness.
• Why is this important?

3
Why Should We Care About Better Decision Support?

• Huge annual deficits contributing to an 11
  trillion debt.
• Trillions in unfunded liabilities.
• Medicare, Medicaid, and Social Security cist will
  increase dramatically.
• Uncertain world climate can lead to future
  unexpected events that increase financial burden.
• Growing financial burden on the States and
  Localities.

4
Objectives of Presentation

• Emphasize the need for relevant, data-driven
  analysis to support decision making.
• Demonstrate the need and benefits of using a
  structured approach to analysis.
• Discuss a 5-step analytical approach.
• Show that while analysis can be challenging and
  complex, using the 5-step approach is not
  difficult.

5
The New Watchwords

• Accountability and Transparency
• GAO changed its name from General Accounting
  Office to Government Accountability Office.
• Accounting and auditing professions are now
  referred to as the accountability professions.
• Many politicians, academicians, private sector
  leaders, and citizen intermediaries refer to
  both terms.
• Laws passed that call for either or both.

6
The New Playing Field Whats Different Today?

• Focus on achieving and measuring
• results!
• Agencies must demonstrate they are getting
  results when preparing budgets and reporting
  annually.
• Citizens are more interested in seeing results of
  public expenditures and meeting their
  expectations.
• Greater use of the Internet for public access to
  information on government operations.

7

• In every program, and in every agency, we are
  measuring success not by good intentions or by
  dollars spent, but rather by results achieved.
• The Budget Message of the President
• February 2006

8
Drivers Of Stronger Analysis To Support
Decision-making
9
Management Reform Legislation

• Numerous laws have revolutionized federal
  management and accountability.
• Mandates for planning, evaluating and reporting
  results.
• Key legislation on reform and accountability
• 1950 Accounting and Auditing Procedures Act
• 1970 Legislative Reform Act
• 1978 Inspector General Act
• 1982 Federal Managers Financial Integrity Act
• 1990 Chief Financial Officers Act
• 1993 Government Performance and Results Act
• 1994 Government Management and Reform Act
• 1996 Federal Financial Management Improvement
  Act
• 2002 Sarbanes Oxley Act

10
Presidential Initiatives

• Virtually each President in recent history has
  implemented initiatives to improve federal
  government management.
• Johnson Planning, Programming, and Budgeting
  System.
• Nixon Management by Objective.
• Carter Zero-Based Budgeting.
• Clinton National Performance Review.
• Bush Presidents Management Agenda.

11
Federal Management Process Has Changed
12
Where is Analysis Needed to Support Decisions

• Problem/Issue
• Budgetary Resources
• Financial Condition
• Business Processes
• Cost/Benefit
• Policy/Program Delivery Alternatives
• Program Operations and Impact

13
How Was the Analysis Done?

• A decision-maker should always know how an
  analysis was done in order to
• Know how solid it is.
• Be comfortable with and be able to defend any
  decision made based on the analysis.

14
So . . . Whats in YOUR Analysis?

• A question on which your future may depend
• In conducting analysis, how do you know how
  reliable, verifiable, relevant, sufficient, and
  consistent the data you depend on are?
• Are you at risk if you dont know?

15
Keys to Decision Support

• Critical Thinking
• Creativity
• Broad Perspective
• Analysis
• Qualitative
• Quantitative
• Communications Skills

16
Guidelines

• Ask questions be willing to wonder.
• Define the problem/concern/question/need.
• Examine the evidence.
• Analyze assumptions and biases.
• Avoid emotional reasoning.
• Dont use either/or thinking or over generalize.
• Consider other interpretations.
• Tolerate uncertainty/questions.

17
What it Means to YOU!

• Today there are challenges in all areas of
  government.
• But!
• There are also opportunities for those who are
  prepared.

18
Remember . . .

• African Proverb
• Every morning in Africa, a gazelle wakes up.
  It knows it must run faster than the fastest lion
  or it will be killed. Every morning a lion wakes
  up. It knows it must outrun the slowest gazelle,
  or it will starve to death. It doesnt matter
  whether you are a lion or a gazelle. When the
  sun comes up, youd better start running!

19
Structured Approach to Conducting Analysis

• Objective and systematic way of collecting,
  examining and reporting information to support
  decision-making.
• Conceptual way of thinking.
• Applies to all aspects of managing programs,
  finances, and operations.
• A logical, comprehensive, structured approach
  contributes to completeness and effectiveness of
  analysis.

20
Structured Approach to Conducting Analysis
(contd)

• To be effective, analysts must
• Understand organizations business.
• Understand how analytic need relates to the
  business.
• Apply sound, thorough analytical methods
  specifically tied to the need.
• Provide useful information to support
  decision-making.
• Analysis for decision-making
• leads to continual line of questioning.

21
Five Step Structured Analytic Approach
Step 1 - Define Questions
Step 2 - Identify Data Sources
Step 3 - Collect Data
Step 4 Conduct Analysis
Step 5 Present Results
22
Step 1Defining Questions

• Question will drive the entire analysis.
• Analysis must zero in on users need.
• Critical considerations
• The users business and role in it.
• What the user needs to know and why.
• How the results will be used for decision-making.
• Thoroughness requires asking 6 key questions
  what, where, when, who, how, and why

23
Getting the Question Right

• How effective is the highway patrol at increasing
  safety?

24
Refining the Question(Getting at the real
question)

• How effective is the highway patrol . . .
• Do resources used give adequate security?
• Are fatalities per 100,000 miles going down?
• Is ratio of convictions/violations going up?
• . . . at increasing safety?
• Is the number o accidents going down?
• Are there fewer safety-related incidents?
• Is there less crime on the roads?

25
Step 2Identifying Data Required

• Key question What data do you need to answer
  the question?
• Data collected from
• Primary sources you collect
• Secondary sources others collected
• Sources and quantity of data gather depend on
• The questions to be answered.
• The availability of data.
• The time allowed for the analysis.
• If data doesnt exist, revisit questions.

26
Consider Three Data Categories

• Budgetary
• Prospective
• Retrospective
• Program
• Strategic and annual plans
• Program design, operation, and results
• Financial
• Cost of operation and financial position (assets,
  liabilities, net worth)
• Cost data by office, program, function, output,
  or outcome

27
Assessing Data Quality

• Assumption All decision are data driven.
• The Quality Axiom
• The quality of the decision
• can be no better than
• the quality of the data supporting the decision!

28
Step 3Collecting the Data

• (Sounds like the easy step?)
• Key questions in deciding how to collect data
• Where are you going to get the data?
• How will you use it to answer the questions?
• How are you going to get it?
• Common data sources
• Records reports, manual/automated files.
• People within and outside the organization.
• Time available key to source and amount.
• Verification important reliability,
  completeness, accuracy, relevancy.

29
Data Collection Hierarchy

• TIME REQUIREMENT
• Standard Agency Reports Low
• Automated Systems
• Manual Files
• Personal Interviews
• Focus Groups
• Surveys High

30
Step 4Analyzing the Data

• Analytic method steps taken to understand,
  display, or interpret data.
• Methodology dependent on question(s).
• Methods numerous and varied.
• Important to know purpose/function of each and
  when to use them.
• Review results of analysis
• Did it address question(s)?
• How sensitive is it to internal/external
  influences?

31
Step 4 Axiom

• The rigor of the analysis must reflect the risk
  of the potential impact of the decision!

32
Conduct Analysis Methods Screening

• There Categories - Which fits question(s)?
• Descriptive what exists
• How was the high-tech training program
  implemented?
• Normative what exists compared to what was
  expected
• Were job-placement goals of the high-tech
  training program met?
• Impact did the initiative have achieve the
  intended results
• Did the high-tech training program have an effect
  on job-placement rates?

33
Determining the Type of Analysis

• Primary Data Analysis
• You collect, assemble, and analyze data.
• Secondary Data Analysis
• You use data gathered and in some cases analyzed
  prior to your own analysis.
• Evaluation Synthesis
• You use the combined results from two or more
  previous analysis.

34
Analytical Methods and Their Uses
35
Document Analytic Design

• Written plan can serve as guide and control steps
  in the structured analytic approach.
• Several factors drive need to document plan
• Complexity of analysis.
• Number of analysts involved.
• Requirements of user/organization.
• An analytic design matrix could help.

36
Step 5Reporting Analytic Results

• Important phase of process -- impact
  decision-making bring about change.
• Reporting methods have different advantages
• Written report
• Communicate results to user as well as others.
• Results less susceptible to misunderstanding.
• Facilitate follow-up.
• Oral report
• Timely support to user.
• Reduce misunderstanding.
• Help reach agreement.
• Goal convince audience that work done and
  results reported is reasonable, appropriate, and
  actionable.

37
Reporting Analysis Results (contd)

• Remember what drove the analysis
• Who is the user/customer and what is their role?
• In what subject/issue is the user interested?
• What does the user want to know?
• What does the user plan to do with the results?
• Be timely, complete, accurate, objective,
  convincing, clear, and concise.
• Ensure analytic results are sufficient, relevant,
  and competent.
• Answering analytic objective, i.e. question, is
  most critical goal.

38
Structured Approach Steps Interactive and
Overlapping

• Each step dependent upon the successful
  completion of previous step.
• Recheck step plans for and results of each step
  to ensure synchronization with previous steps.
• As complexity/comprehensiveness of one step
  grows, so grows complexity/comprehensiveness of
  succeeding steps, and time required.

39
Structured Approach Steps Interactive and
Overlapping (contd)
40
Revalidating Questions(Continuous throughout
analysis!)

• Regularly re-check and validate questions.
• Questions may need to be modified, dropped,
  redefined or added.

Analysis Feedback Loop
Define Questions
Identify Data Sources
Collect Data
Conduct Analysis
Report Results
41
Key Points to Remember

• Accountability expectations drive need for
  critical analysis.
• Structured approach objective/systematic way of
  collecting, examining, and reporting information.
• Five step structured approach
• Identify question(s)
• Identify data
• Collect data
• Analyze data
• Report results
• Steps highly interrelated drives need to
  revisit results of each step throughout process.

42
Summary

• Analysis
• Growing need support difficult decisions.
• Growing importance support performance and
  financial accountability.
• Best if
• Follows a structured approach.
• Based on users/customers questions/need.
• Impacts decision-making/results.