Risk Assessment for Performance Auditing

1
Risk Assessment for Performance Auditing

• Association of Government Accountants
• 2007 Professional Development Conference
• Nashville, TN
• June 25, 2007

Beth L. Ashcroft, CIA - Director Office of
Program Evaluation and Government
Accountability Maine State Legislature
2
Outline

• What is Performance Auditing?
• What is Risk Assessment?
• Risk Assessment at an Enterprise Level
• Risk Assessment on Individual Audits

3
What is Performance Auditing?
4
Definition of Performance Auditing

• Fair and impartial assessment
• Provides objective information about performance
  of programs, activities and functions
• Provides specifics about where improvements can
  be made and likely impact of improvements

5
Purpose

• Assess and report on extent to which entity is
  faithfully, economically, efficiently and
  effectively carrying out the programs and
  activities for which it is responsible.

6
What is Risk Assessment?
7
A Definition of Enterprise-wide Risk Assessment
A systematic approach to identifying, and
assessing the significance of, existing or
potential situations that either threaten
(negative consequences) or could enhance (missed
opportunities) the achievement of an
organizations objectives across the entire
enterprise or within a specific audit area.
8
Purpose of Enterprise-wide Risk Assessment
To make determinations about strategies that
should be employed and resources that should be
applied to control activities in order to
mitigate risks of negative consequences or missed
opportunities. (ERM) To determine where we
should audit.
9
Risk Assessment at an Enterprise Level
10
Designing the Risk Assessment

• Auditable activities
• Risk factors
• Data collection
• Analyzing data
• Presentation of results

11
I. Defining Auditable Activities

• Auditable activities are subjects, units or
  systems capable of being defined and evaluated

12
I. Defining Auditable Activities

• Possible auditable activities
• Organizational units/entities
• Programs
• Functions
• Policies, procedures and practices
• Processes
• Laws and regulations
• Information systems

13
I. Defining Auditable Activities

• Considerations
• Amount of credible information readily available
• Breadth of topic desired for individual audits

14
II. Determining Risk Factors

• Risk factors are criteria or indicators used to
  assess the relative significance of, and
  likelihood that, conditions and/or events may
  occur that could adversely affect the
  organization

15
II. Determining Risk Factors

• Potential risk factors
• Ethical climate and pressure to perform
• Quality of objectives and performance measures
• Complexity of operations/services
• Degree of organizational, management,
  operational, technological or economic changes
• Geographic organization centralized vs
  decentralized

16
II. Determining Risk Factors

• Potential risk factors (cont.)
• Degree of financial impact
• Financial changes
• Degree of public impact
• Competence and adequacy of personnel
• Degree of automation/computerized information
• Changes in laws and regulations
• Results of previous audits

17
II. Determining Risk Factors

• Potential risk factors (cont.)
• Volume of transactions or customers
• Maturity level (brand new vs ancient)
• Extent of judgment
• Customer, employee or vendor complaints
• Employee morale/satisfaction
• Internal control environment (inc. tone at the
  top)

18
II. Determining Risk Factors

• Considerations
• Most relevant to audit organizations objectives
  (inc. oversight body)
• Existence, availability and accessibility of
  information
• Relevant to each auditable activity

19
III. Collecting data on risk factors

• Potential collection methods
• Interviews
• Surveys and questionnaires
• Document review
• Internet research

20
III. Collecting data on risk factors

• Considerations
• Degree of precision desired
• Quality of the data
• Comparable data for all auditable activities
• Ease of collection

21
IV. Analyzing data

• Using risk matrix
• Choose risk factors to be scored
• Choose scoring scale
• Assign a score for each risk factor
• Weight each risk factor
• Multiply score by weight to get final numerical
  result

22
IV. Analyzing data

• Considerations
• Different risk factors or weightings for
  different auditable activities
• Weighting based on impact
• Weighting to reflect areas of interest
• Weighting to reflect quality of data
• May score and weight in different groupings or at
  multiple levels

23
V. Presenting results

• Presentation ideas
• Show numerical detail, i.e. rankings, weights and
  final results for each risk factor
• Color coding to quickly identify high, medium or
  low risk
• Plot results on a graph or scale
• List auditable activities that had highest results

24
V. Presenting results

• Considerations
• Audience
• Purpose of presentation
• Ease of understanding
• Level of detail desired

25
Example 1 - OPEGA
26
Example 1 OPEGA (cont.)
27
Example 2 - CMP

• Auditable Activities
• Organizational Units from Top Down
• Risk Factors to be Considered
• Degree of Risk related to Information
• Degree of Risk related to Competition
• Degree of Legal Risk
• Degree of Organizational Risk
• Degree of Financial Risk
• Work Environment
• Control Environment

28
Example 2 - CMP

• Data Collection
• Survey of Department Heads
• Follow-up Interviews
• Interview Senior Officers
• Audit Committee Concerns
• Internal Audit Past Experience Judgment

29
Example 2 - CMP

• Survey questions Information Risk
• To what degree does your organization rely on
  information and data from sources outside your
  immediate organization for planning,
  decision-making or daily operations critical to
  your organizations success?
• Not reliant --------------------------------------
  --------------Very reliant
• How comfortable are you that the information and
  data received from outside sources are valid,
  accurate and complete?
• Not comfortable-----------------------------------
  --------Very comfortable

30
Example 2 - CMP

• Survey questions Information Risk (cont.)
• How often does your organization provide
  information and data to individuals in other
  organizations or top management for their use in
  strategy-setting, planning, decision-making and
  daily operations?
• Never ________ Frequently _______
• Sometimes ______ All the time ______
• Does your organization possess, maintain or have
  access to information that should be kept
  confidential from other parts of the corporation
  or from outside parties?
• Yes _______ No ________

31
Example 2 - CMP

• Survey questions Legal Risk
• To what degree is your organization affected by
  federal or state laws and regulations?
• Not affected -------------------------------------
  --------------Very affected
• How comfortable are you that members of your
  organization adequately understand applicable
  aspects of the laws and regulations that affect
  your operations?
• Not comfortable-----------------------------------
  --------Very comfortable

32
Example 2 - CMP

• Survey questions Legal Risk (cont.)
• How much of your organizations business is
  related to specific contractual agreements that
  you are initiating, entering into, fulfilling or
  operating under?
• None --------------------------------------------
  -----------------All
• What is the potential that your organizations
  operations could result in harm to public health,
  safety and welfare or damage to public or private
  property?
• No potential ------------------------------------
  -High potential

33
Example 2 - CMP

• Analyzing Data
• Subjectively assigned score of 1-5 to each risk
  area (information, competition, legal,
  organizational and financial) based on survey
  responses, interviews, past experience
• Added or subtracted points (1 to -1) for Work
  and Control Environments based on same

34
Example 2 - CMP

• Analyzing Data (cont.)
• Identified hot spots of definite or possible
  problem areas in each organizational unit based
  on survey responses, for example
• Comfort with validity, accuracy and completeness
  of data received
• Understanding of applicable laws and regs
• Comfort that have right people with necessary
  knowledge and skills
• Morale
• Control activities
• Understanding of ethical policies

35
Example 2 - CMP

• Presenting results
• Used color coded organizational chart to show
• organizational units with High and Med/High
  overall risk
• risk areas that were high or med/high within each
  organizational unit and
• hot spots identified within each organizational
  unit

36
Example 2 - CMP
37
Risk Assessment on Individual Audits
38
Determining Audit ObjectivesService Delivery
Model As Basis
Expanded Service Delivery Model can be used as a
basis in defining objectives
39
Considering Performance Aspects
40
Evaluating Risks and Controls
Threats to Achievement or Missed Opportunities
Objectives (Audit, Business, Program)
Established Controls
Is it Acceptable?
Residual Risk
41
Sources of Risk

• Commercial/Legal
• Product/Service Liability
• Finance/Economic
• Environmental Liability
• Fraud/Corruption
• Equipment/Technology
• Human Behavior
• Public Perception

42
Sources of Risk (cont.)

• Political Influences
• Natural Events
• Competition
• Employees
• Suppliers
• Customers
• Missed Objectives
• Control Design

43
Core Risk Concepts

• Risks are relevant to the extent they impact
  current and future objectives.
• Some risks create and add to the universe of
  objectives necessary to succeed, i.e. mitigating
  certain risks should be included as an objective.
• Deficiencies in control design can be a
  significant source of risk.

44
Core Risk Concepts (cont.)

• Considering risks without making explicit
  linkages to objectives can lead to sub-optimal
  allocation of resources.
• Risk status is dynamic as the various risk and
  control elements interact
• Mgts focus should be on defining objectives and
  analyzing residual risk. Auditors should focus
  on whether there is an effective framework to do
  this.

45
Applying Risk Assessment in Focusing the Audit

• Identify the key objectives
• Determine individual threats
• Assess inherent risk associated with each threat
  (likelihood and impact)
• Determine controls in place to mitigate threats
  esp. those with moderate to high inherent risk
• Assess strength of control system for each threat

46
Applying Risk Assessment in Focusing the Audit
(cont.)

• Assess residual risk (inherent risk x strength of
  controls)
• Focus further work on
• Validating control strength for threats with
  moderate/high inherent risk and acceptable
  residual risk
• Determining degree of exposure for threats with
  moderate/high inherent risk and unacceptable
  residual risk
• Assessing whether too many resources are being
  spent on threats with low inherent risk

47
Examples

• CMP Outage Hours Per Customer
• State of Maine Bureau of Rehabilitation Services

48
Resources

• Performance Auditing A Measurement Approach -
  the Institute of Internal Auditors at
  www.theiia.org
• National Legislative Program Evaluation Society
  (NLPES) at www.ncsl.org/programs/nlpes
• Federal Government Accountability Office at
  www.gao.gov
• Adding Value Seven Roads to Success the
  Institute of Internal Auditors Research
  Foundation

49
Contact Info

• Beth Ashcroft, Director
• Office of Program Evaluation Government
  Accountability
• Maine State Legislature
• 82 SHS
• Augusta, ME 04333-0082
• [phone]
• beth.ashcroft_at_legislature.maine.gov
• www.maine.gov/legis/opega