Experimental Research in Auditing

1
CAR Ph.D. Consortium Niagara-on-the Lake, Ontario
Experimental Research in Auditing
Ira Solomon R. C. Evans Endowed Chair in
Business and Head of the Department of
Accountancy University of Illinois at
Urbana-Champaign
November 4, 2005
2
Thanks to my Collaborators
Tim Bell, Director KPMG LLP Mark Peecher,
Associate Professor University of Illinois,
Urbana-Champaign
3
Session Takeaways

• Judgment is critical to the external audit-- the
  public company audit largely consists of
  judgments.
• Risk assessment activities, including their
  attendant antecedents comprise a large part of
  contemporary public company auditing.
• Researchers have a fundamental role to play in
  terms of discovering how how well and how
  should public company auditors make judgments and
  decisions-- experiments can be a most effective
  way of developing answers to these types of
  questions.
• Time is of the essence!

4
The Audit Context
WorldCom, Enron, HealthSouth, Parmalot . . .
Public-Company Auditing is now a regulated
activity in the U.S. and Canada and in many
other jurisdictions. Societies are now placing a
premium on audit quality.
5
Audit Quality
What are the elements of a public company audit,
especially those elements central to achieving
audit quality? How is audit quality
controlled/established on an audit engagement?
6
How is Audit Quality Achieved?
Traditional answer-- Compliance with GAAS
(authoritative pronocements) Compliance with
Quality Control Standards More recent
answer-- An Integrated Audit
7
Overview of the Evolution of the Risk-Assessment
Orientation in Auditing
Equity-Markets Era
State of the Business (Entity Business States)
Owner-Manager Information Needs Measure costs
prevent detect bookkeeper/employee theft
Investor Information Needs Information on
financial outcomes other related entity
business states
f ( Value Proposition, Strategy,
Business Execution )
Audit Objectives Approach Verify internal
consistency of bookkeeping detect employee
theft Approach Exhaustive detailed audit
primarily within the books records
Audit Objectives Approach Opine on veracity of
management business representations of selected
entity business states Approach Risk assessment,
including via selective testing
Knows much about
Does not know much about
Nature of Audit Risk Failure to detect
mechanical bookkeeping errors Failure to detect
bookkeeper fraud
Nature of Audit Risk Failure to detect
unintentional material errors Failure to detect
intentional material misstatements Non-sampling
sampling risks
Information That Faithfully Represents Decision-Re
levant Entity Business States
Nature of Audit Evidence Limited primarily to
internal records
Nature of Audit Evidence All information used by
auditors in arriving at final risk assessments
Increasing complexity of business organizations,
relationships activities Technological
changes Increasing role of judgment in accounting
policy choices auditing Instances of management
fraud increasing auditors responsibility to
detect it Changes in the nature of the audit
objectives, techniques evidence
Time
8
Evolution of the Public Company Audit

• Early auditing in America
• Owner-managers hire auditors to detect prevent
  bookkeeper fraud errors
• Auditors perform the exhaustive detailed audit in
  search of bookkeeping errors and
  bookkeeper/employee fraud
• Procedures for checking the internal consistency
  of the records came to be called substantive
  tests of details

As the only benefit to be derived from frauds in
accounts arises from a desire to embezzle the
cash, the greater part of the trouble in
detecting errors may be obviated by examining the
Cash account only. Auditors Guide Being a
Complete Exposition of Bookkeepers Fraud
Mettenheimer 1869
In some audits, and not only small ones, we
verified every footing and every posting. .
. . The auditor of fifty years ago . . .
was little recognized because the matters which
were referred to him were relatively unimportant
and this unimportance tended to reduce him to the
level of a clerk. Fifty Years of Accountancy,
R. H. Montgomery 1939
9
Evolution of the Public Company Audit

• Increasing size complexity of business
  organizations necessitates selective audit
  testing, which the historical record shows was
  introduced prior to 1900
• A pre-1900 New York CPA exam question asks In an
  audit where an exhaustive detailed examination .
  . . is not stipulated or practicable, what
  examination is necessary to assure . . .
  general correctness?

With the rapid growth of American business
following the Spanish-American war, the increase
in the size of many enterprises and the auditing
of larger concerns, there developed the necessity
for making an audit one of selected tests of the
accounts rather than an endeavor to examine all
of the transactions of the period. Auditing
Developments During the Present Century, Walter
A. Staub Harvard University Press, 1942
10
Evolution of the Public Company Audit

• Auditors recognize relationship between strength
  of internal checking and extent of external
  audit test work

The first true recognition of internal controls
as a foundation for deciding on the amount of
detailed verification to be done appeared in the
American version of Dicksees Auditing 1905
. . . A proper system of internal check
will frequently obviate the necessity of a
detailed audit. Changing Audit Objectives
and Techniques, The Accounting Review, R. G.
Brown 1962
11
Evolution of the Public Company Audit

• Exhaustive detailed audit transforms to auditing
  as risk assessment (whenever uncertainty exists
  the best one can do is to assess)
• Professional judgment becomes the keystone of the
  audit
• Risk assessment ( a particular kind of judgment),
  therefore, has been present in and central to
  financial statement auditing for at least 100
  years

Keystone That one of a number of associated
parts or things that supports or holds together
the others. Websters New World Dictionary.
12
More on the Evolution of the Public Company Audit

• The rise of absentee ownership changes audit
  objectives
• Detection of bookkeeper fraud is relegated to a
  minor audit objective
• Auditors responsibility for detection of
  intentional misstatement by management is opaque
  or simply does not exist

• In what might be called the formative days of
  auditing, students were taught that the chief
  objects of the audit were
• detection and prevention of fraud
• detection and prevention of errors
• In recent years there has been a decided change
  in demand and service. Present-day purposes are
• to ascertain actual financial condition and
  earnings of an enterprise
• i.e., selected entity business states
• detection of fraud and errors, but this is a
  minor objective.
• Auditing Theory and Practice, R. H. Montgomery
  1912

13
More on the Evolution of the Public Company Audit

• Thought leaders recognize the role and import of
  judgment in income determination and, by
  implication, in auditing . . .

. . . the accounts of a modern business are
not entirely statements of fact, but are, to a
large extent, expressions of opinion based partly
on accounting conventions, partly on assumptions,
explicit or implicit, and partly on
judgment. The ascertainment of profit is in
every case necessarily a matter of estimate and
opinion. . . . the inferences drawn from
facts and the opinions based on them are usually
more important than the bare facts
themselves. George Oliver May in Memorandum
Regarding Securities Bill HR 4314 1933 in
Twenty-Five Years of Accounting Responsibility,
1911 1936 (Scholars Book Company, 1936)
14
More on the Evolution of the Public Company Audit

• . . . but he evolving literature (and
  authoritative guidance) focuses heavily on the
  auditors determination of the extent of testing,
  with less emphasis on how to improve judgment,
  inference and opinion formulation
• With selective testing the question arises how
  much testing?
• Publications dealing with statistical audit
  sampling can be found as early as 1933 (See
  Auditing Practice, Research, and Education A
  Productive Collaboration, edited by T. B. Bell
  A. M. Wright AICPA 1995 for an historical
  overview of developments in audit sampling)

Roughly speaking, statistical sampling helps
answer one of the auditors three key questions
concerning the nature, extent, and timing of his
audit procedures. The auditor can determine the
extent of testing more objectively when using
statistical sampling in tests of details rather
than judgmental samples. Statistical Auditing,
D. M. Roberts AICPA, 1978
15
More on the Evolution of the Public Company Audit

• Auditors develop the Audit Risk Model (ARM) to
  formalize some definitions and
  interrelationships for different types of audit
  risks, primarily to aid the planning of sample
  sizes for tests of details
• AUR RMM X DR
• Where
• Audit Risk (AUR) the risk that the auditor
  expresses an inappropriate audit opinion when the
  financial statements are materiality misstated
• Risk of Material Misstatement (RMM) the risk
  that the financial statements are materially
  misstated prior to the audit
• Detection Risk (DR) the risk that the auditor
  will not detect misstatements that could be
  material individually or when aggregated with
  other misstatements
• Materiality the magnitude of an omission or
  misstatement of accounting information that, in
  light of surrounding circumstances, makes it
  probable that the judgment of a reasonable person
  relying on the information would have been
  changed or influenced by the omission or
  misstatement

The ARM first appeared in equation form in 1972
in Appendix B of AICPA Statement on Auditing
Procedure No. 54--The Auditors Study and
Evaluation of Internal Control
16
More on the Evolution of the Public Company Audit

• More on the ARM
• The ARM further decomposes RMM into two separate
  types of risk at the class of transactions or
  account balances level
• Inherent Risk (IR) the susceptibility of an
  assertion to a material misstatement, assuming
  that there were no related internal controls
• Control Risk (CR) the risk that a misstatement
  that could occur in an assertion and that could
  be material, individually or when aggregated with
  other misstatements, will not be prevented or
  detected and corrected on a timely basis by the
  entitys internal controls

17
More on the Evolution of the Public Company Audit

• More on the ARM
• The ARM further decomposes DR into two separate
  types of risk, based on the nature of audit
  procedures
• Analytical Procedures Risk (AP) the auditors
  assessment of the risk that analytical procedures
  and other relevant substantive tests would fail
  to detect misstatements that could occur in an
  assertion equal to tolerable misstatement . .
  .
• Test of Details Risk the allowable risk of
  incorrect acceptance (of an assertion) for the
  substantive test of details . . .

18
More on the Evolution of the Public Company Audit

• Emphasis on planning sample sizes SAS No. 39
  (U.S.) on Audit Sampling
• An auditor might use this model to obtain an
  understanding of an appropriate risk of incorrect
  acceptance (of an assertion) for a substantive
  test of details as follows
• TD AUR/(IR x CR x AP)
• The auditor planning a statistical sample can
  use the relationship to assist in planning his
  allowable risk of incorrect acceptance for a
  specific substantive test of details.
• The multiplicative form of the model implies a
  compensatory relationship among the different
  sources of risk, e.g., higher inherent and
  control risks necessitate lower planned detection
  risk, and vice versa
• Note The auditor needs a quantitative
  probability measure of TD (i.e., the allowable
  risk of incorrect acceptance) to compute a sample
  size using formulae based on the laws of
  probability

19
More on the Evolution of the Public Company Audit

• More on the ARM
• One consequence of the co-development of
  statistical sampling and the audit risk model was
  that the profession partitioned DR into
• Sampling Risk (SR) the possibility that an
  auditors conclusion, based on a sample, may be
  different from the conclusion reached if the
  entire population were subjected to the same
  audit procedure, and
• Non-Sampling Risk (NSR) detection risk that
  arises from factors that cause the auditor to
  reach an erroneous conclusion for any reason not
  related to the size of the sample e.g., poor
  assessments of RMM made during planning
• Non-sampling risk can be reduced to a negligible
  level through such factors as adequate planning
  and supervision . . . (SAS No. 39)

20
More on the Evolution of the Public Company Audit

• More on the ARM
• Controlling this non-sampling risk is very
  important and should be carefully considered by
  the auditor in determining the nature and timing
  and extent of the auditing procedures.
• An inappropriate decision on the part of the
  auditor to limit the size of a sample due to low
  assessed RMM is an example of a non-sampling risk

Controlling this non-sampling risk is very
important and should be carefully considered by
the auditor in determining the nature and timing
of the auditing procedures. . . . the
auditor can never reduce audit risk to a lower
level than the non-sampling risk. Consequently,
unless the audit procedures have a non-sampling
risk well below a tolerable level of audit risk,
neither statistical nor nonstatistical sampling
will be particularly helpful. Statistical
Auditing, D. M. Roberts AICPA, 1978
21
More on the Evolution of the Public Company Audit

• By the 1980s, non-sampling risk is considered a
  major source of detection risk (especially the
  risk that the auditor will fail to detect
  financial statement fraud)

22
More on the Evolution of the Public Company Audit

• During the 1980s the U.S. economy experiences a
  Savings Loan (SL) crisis
• Auditors failure to effectively manage and
  control non-sampling risk arising from poor
  understanding of business situations and risks is
  considered a primary driver of detection risk for
  audits of the SLs
• The bailout of Lincoln Savings Loan (LSL)
  becomes the most costly bailout by the Federal
  Government in U.S. history at that time

23
More on the Evolution of the Public Company Audit

• Non-sampling risk appears to be the culprit on
  the LSL audit
• The problematic transactions were included in the
  sample of transactions subjected to testing,
  but factors unrelated to sample size point to
  management representations of entity business
  states that appeared too good to be true

The main conclusion from our analysis is that
the most significant shortcoming in the LSL audit
was the auditors failure to obtain and use
knowledge of LSLs business, the industry in
which it operated, and the economic forces that
influenced this industry/business. . . .
The auditors evaluated the compliance of each
material real estate transactions form with the
mechanical aspects of SFAS No. 66 Accounting
for Sales of Real Estate (e.g., the down payments
appeared to meet the requirements of SFAS No.
66). . . . Applying knowledge of LSLs
business, the real estate industry, and economic
trends in that industry would have been the most
effective audit procedures available to LSLs
auditors. In cases of management fraud, auditors
are unlikely to receive reliable evidence from a
client. . . . The procedures we recommend
should be used routinely by auditors because they
are effective both in the presence and absence of
fraud. They are, therefore, more effective than
transactions-based procedures, which, as the
evidence presented in this study suggests, are
simply not effective in dealing with management
fraud. Erickson et. al. 2000
24
More on the Evolution of the Public Company Audit

• More recent evidence on non-sampling risk
  SOX Section 704 Report (SEC) on causes of fraud
  alleged audit failures
• Non-sampling risk appears to be the primary
  culprit
• Examples of specific charges against auditors
• Failed to obtain sufficient knowledge about the
  issuers business
• Failed to obtain sufficient competent evidence to
  corroborate management representations
  explanations
• Failed to conduct adequate risk assessments
• Failed to respond to information suggesting
  assets were overvalued

25
More on the Evolution of the Public Company Audit

• More recent evidence on non-sampling risk SOX
  Section 704 Report (SEC) on causes of fraud
  alleged audit failures
• Non-sampling risk appears to be the primary
  culprit
• Examples of specific charges against auditors
• Truncating analytical and substantive procedures
• Issued unqualified opinion despite being aware of
  many accounting improprieties disclosure
  failures
• Failed to exercise professional skepticism on
  unusual, last minute, or related party
  transactions

26
Still More on the Evolution of the Public Company
Audit

• Frameworks emerge whose purpose is to begin to
  help auditors think about how to improve
  judgment, inference, and opinion formulation
  (i.e., improve non-sampling risk control)

Business modeling process analysis frameworks
27
Still More on the Evolution of the Public Company
Audit

• Consider a simple example
• A client company offers subscriptions to an
  Internet portal (e.g., AOL). Successful
  execution of the companys strategy rests on its
  success at attracting and retaining subscribers.
  The company incurs significant costs to acquire
  and retain subscribers, e.g., marketing costs,
  development costs for services provided via the
  portal. A critical accounting policy decision is
  to determine what amount, if any, of these costs
  should be capitalized, indicating they are
  investments by the company expected to produce a
  significant amount of future revenue. Management
  has decided to capitalize 75 of these customer
  acquisition costs and amortize over a period of
  24 months.
• What types of evidence should the auditor obtain
  to make his/her independent inference about the
  judgment made by management?

28
Still More on the Evolution of the Public Company
Audit

• Some possibilities you rate their relative
  importance to the auditors inference

Very
Somewhat
Not
Type of Information

Imp
ortant

Important

Relevant




A. Relevant accounting standards and literature




B. Practices of other firms in the industry




C. The companys historical patterns of customer

retention




D. Evolving competitive landscape in the industry




E. Data on pattern of customer usage of the
companys services




F. Data on customer satisfaction with the
services offered by the

company




G. Technological changes in the industry




H. Effectiveness of the companys customer
acquisition

retention
strategies and
process
es




I. Information on the financial statement
impact of various

alternative accounting choices


29
Recap Commentary Evolution of the Audit

• Some factors that have changed the nature of the
  audit
• increasing size complexity of business
  organizations, relationships activities
• rise of widely-dispersed absentee ownership and
  societys demand for market liquidity
• societys demand for reasonable (which is a high
  level of) assurance that financial statements are
  not materially misstated due to fraud
• role and import of judgment in the determination
  of income and other disclosures about
  decision-relevant entity business states

30
Recap Commentary Evolution of the Audit

• Non-sampling risk is a significant determinant of
  auditors failure to detect material
  misstatements, especially those caused by fraud
• Models and techniques have been advanced to help
  the auditor to think about how to improve control
  of sampling risk
• Models have begun to emerge to help auditors
  improve judgment, inference and opinion
  formulation (i.e., non-sampling risk control)

31
Core Concepts for21st Century Public Company
Auditing

• Beliefs, Knowledge, the Audit Evidence
  Threshold

32
Core Concepts for21st Century Public Company
Auditing

• 21st century public company auditing is best
  characterized as evidence-driven, belief-based
  risk assessment
• Audit quality is determined by the extent to
  which auditors risk assessments rest on
  sufficiently well-justified beliefs, and auditors
  must be able to demonstrate to others why and how
  such beliefs are sufficiently well-justified
• Risk assessment is a recursive process involving
  repeated risk assessments and responses until the
  auditors goal developing sufficiently
  well-justified beliefs is reached

33
Core Concepts for21st Century Public Company
Auditing

• What is recursive, evidence-driven, belief-based
  risk assessment?

34
Core Concepts for21st Century Public Company
Auditing

• All audit procedures regardless of their nature
  or timingproduce evidence to improve the
  accuracy of the auditors assessments of RMW and
  RMM, and assessment and management of DR

RMW Risk of Material Weaknesses in Internal
Control Over Financial Reporting (ICOFR) RMM
Risk of Material Misstatement (I.e., Inherent
Risk (IR) Control Risk (CR) DR Detection
Risk, especially non-sampling risk related to the
auditors assessment of RMM due to
fraud
35
Core Concepts for21st Century Public Company
Auditing

• Professional judgment is the keystone of 21st
  century public company auditing, and the
  connective mortar that links audit objectives,
  evidence, beliefs, and risk assessments
• The subjective risk assessment process is fraught
  with non-sampling risks which can feed forward
  throughout the process

36
Core Concepts for21st Century Public Company
Auditing

• A prudent strategy for dealing effectively with
  non-sampling risks inherent in the exercise of
  professional judgment is to identify and assess
  audit risks from multiple perspectives, using
  multiple sources of evidence (what we call
  triangulation)
• Independent evidence of and from entity business
  states (EBS) is particularly important for
  assessing and addressing RMW, RMM and DR due to
  management fraud

37
Core Concepts for21st Century Public Company
Auditing

• The Financial Reporting Process

38
Core Concepts for21st Century Public Company
Auditing

• Late 19th-century auditors already had seen the
  need to obtain evidence on selected entity
  business states from outside of the company
  records

Proof should be sought outside the books in the
statements of debtors and creditors themselves
for comparison with the books . . .
. Science of Accounts, G. P. Greer 1882
Methods adopted for verification of transactions
by securing evidence outside the records of the
client implies that auditors were finding it
desirable and necessary to consider more than
mere clerical accuracy and detection of
bookkeeper fraud. Early Developments in
American Auditing, The Accounting Review, C. A.
Moyer 1951
39
Core Concepts for21st Century Public Company
Auditing

• Documentation on the McKesson and Robbins case
  recognizes the need for more direct and
  independent evidence on entity business states

For many years accountants have in regularly
applied procedures gone outside the records to
establish the actual existence of assets and
liabilities by physical inspection or independent
confirmation. There are many ways in which this
can be extended. Particularly, it is our opinion
that . . . Inspection of inventories and
confirmation of receivables, which, prior to our
hearings, had been considered optional steps,
should . . . be accepted as normal auditing
procedures. . . . SECs Original
Investigation, as discussed in Brief 1982
Until the 1930s, it was customary to limit the
audit work for inventories to an examination of
records only. . . . This situation
changed with the McKesson and Robbins, Inc.
investigation in the United States. . . .
Because of that investigation, the public
accounting profession was faced with the
necessity of accepting responsibility for
verifying the physical existence of
inventoriesor being challenged that its audit
function offered no real protection to investors
or other users of financial statements. Audits
of Inventories, AICPA 1986
40
Core Concepts for21st Century Public Company
Auditing

• Prudent auditors compensate for
  difficult-to-control non-sampling risks by way of
  evidence triangulation to develop sufficiently
  well-justified beliefs

41
Core Concepts for21st Century Public Company
Auditing

• Triangulation is not a completely new concept

42
Core Concepts for21st Century Public Company
Auditing

• For the 21st century public company audit,
  consistent with the concept of triangulation, the
  evidence acquisition frame is becoming
• How can the auditor best lever all three
  fundamental sources of evidence (EBS, MII and
  MBR) to develop sufficiently well-justified
  beliefs and risk assessments?
• The former perspective implied by the audit risk
  model was a compensatory perspective on evidence
  acquisition (How can the auditor trade off one
  type of evidence for another?), while this is a
  complementary perspective

43
Core Concepts for21st Century Public Company
Auditing

• Triangulation also involves assessing the risk of
  material weakness (RMW), risk of material
  misstatement (RMM), and detection risk (DR) from
  both the top-down and bottom-up perspectives

44
Core Concepts for21st Century Public Company
Auditing
45
Implications for Change

• Whats needed to continue to improve audit
  quality?
• Improvement of models, frameworks and aids for
  assisting auditors to make the risk assessment
  and other judgments that are central to audit
  quality, e.g., that
• represent the true nature of the 21st century
  public company audit as recursive,
  evidence-driven, belief-based risk assessment
• embrace the complexity of the integrated audit
• help auditors to improve their control of
  non-sampling risks (I.e., the risks due to
  judgment errors)

46
Some Definitions

• Scientific research is systematic, controlled,
  empirical and critical investigation of phenomena
  guided by theory and hypotheses about the
  presumed relations among such phenomena
  (Kerlinger, F.N. Foundations of Behavioral
  Research).

47
Definitions

• Theory-- a set of interrelated constructs
  (concepts), definitions, and propositions that
  present a systematic view of phenomena by
  specifying relations among variables with the
  purpose of explaining and predicting phenomena
  (Kerlinger).

48
Research and Theory

Degree of Understanding
Focus of Research
49
Experiments

• If the objective is to test or refine theory,
  experiments are particularly effective because
  experiments allow for strong causal inferences.

50
On the Other Hand . . .

• If the objective is to identify real-world
  degrees of association and effect sizes
  (irrespective of underlying causal factors),
  other methods likely dominate experiments.

51
Experiments

• Experiments allow one to distinguish the
  influence of extraneous factors from that of
  factors of theoretical interest.
• In experiments, internal validity is central and
  external validity is primarily instrumental

52
Experiment- A Definition

• Phenomena (an individual or social process) is
  reproduced in controlled situations, and then
  various measurements are made of the phenomena,
  often measurements that could not be collected in
  natural setting.
• Reynolds, P.D., A Primer in Theory
  Construction, 1987, p. 156.

53
Features of Experiments

• Manipulation of independent variables
• Ensure manipulations effectiveness
• Manipulate construct of interest
• Control extraneous variables
• random assignment (less self-selection)
• hold specific factors constant
• measure specific factors (e.g., covariates)

54
More Features of Experiments

• Researcher makes several choices about
  independent variables
• Number and nature of independent
  variables Theory-based interactions
• Between vs. within participants
• How to implement (operationalize)
• Manipulation checks

55
Still More Features of Experiments

• Researcher makes several choices about dependent
  variables
• Operationalization
• Number
• Response mode
• Categorical vs. continuous
• Judgment vs. choice

56
Some Features of experiments
57
Summary

• Researchers frequently make TRADEOFFS when
  designing experiments.
• But, some tradeoffs are better than others.
• NEVER sacrifice internal validity.
• If you do, youre not conducting a valid
  experiment and cannot test theory.

58
What is the status of Experimental Research Today?
59
Auditing Research in Journals 1976- 2000
60
Published J/DM Audit Experiments
61
Published Audit Research in CAR 2001-2005
62
Potential Topics for Experimental Studies

• Professional Skepticism
• Reasonable (High) Assurance
• Evidentiary Triangulation
• Risk Assessment

63
Session Takeaways

• Judgment is critical to the external audit-- the
  public company audit largely consists of
  judgments.
• Risk assessment activities, including their
  attendant antecedents comprise a large part of
  contemporary public company auditing.
• Researchers have a fundamental role to play in
  terms of discovering how how well and how
  should public company auditors make judgments and
  decisions-- experiments can be a most effective
  way of developing answers to these types of
  questions.
• Time is of the essence!