Producing Good Research:

1
Producing Good Research Behavioral and
Otherwise

Jane Jollineau Kennedy University of Washington
CAAA Doctoral/New Faculty Consortium
Montreal November 8, 2007
2
Todays Outline

• What is behavioral research and why do we care?
• Why do experiments?
• What is good research (experimental and
  otherwise)?
• Finding a topic in (Behavioral) research
• Common Pitfalls

3
What is behavioral research in accounting?

• Study of how individuals use financial
  information to make judgments and decisions, and
• How those judgments and decisions can be improved
• Usually guided by some psychological theory

4
We are not unbiased information processors

• Tend to make systematic judgment errors
• Often a by-product of generally functional
  heuristics or shortcuts that we take
• Not all due to lack of effort (Arkes 1991)
• Some not easily reduced or fixed gt address the
  cause of the error
• Use psych. theory to predict where we are likely
  to have a problem in judgment decision making
  and how to remedy that problem

5
Reference Points

• Imagine you are lying on the beach on a hot day.
  All you have to drink is ice water. For the
  last hour you have been thinking about how much
  you would enjoy a nice cold bottle of your
  favorite brand of beer. A companion gets up to
  go make a phone call and offers to bring back a
  beer from the only nearby place where beer is
  sold, a fancy resort hotel (a small, run-down
  grocery store). He says the beer might be
  expensive so asks how much you are willing to pay
  for the beer. He will buy the beer if it costs
  as much or less than the price you state. But if
  it costs more than the price you state he will
  not buy it. You trust your friend and there is
  no possibility of bargaining with the bar tender.
  What price do you tell him?

6
(No Transcript)
7
Framing

•      Imagine that you have just learned that the
  sole supplier of a crucial component is going to
  raise prices. The price increase is expected to
  cost the company 6,000,000. Two alternative
  plans have been formulated to counter the effect
  of the price increase. The anticipated
  consequences of these plans are as follows
  (Check one)
• _____ If Plan A is adopted, the company will save
  2,000,000 (lose 4,000,000).
• _____ If plan B is adopted, there is a 1/3
  probability that 6,000,000 will be saved (there
  will be no loss) and 2/3 probability that nothing
  will be saved (the company will lose 6,000,000).

8
(No Transcript)
9
Conjunction Fallacy

•   Linda is 31 years old, single, outspoken and
  very bright. She majored in philosophy. As a
  student, she was deeply concerned with issues of
  discrimination and social justice, and also
  participated in antinuclear demonstrations.
  Check the most likely alternative. 
• _____ Linda is a bank teller.
• _____ Linda is a bank teller and is active in the
  
  feminist movement.

10
(No Transcript)
11
Bayesian Updating

• Suppose you are a physician who has just
  examined a patient for a specific type of cancer.
  The patient has a lump and based on your many
  years of experience, you estimate the odds that
  the lump is malignant as 1 in 100. Just to be
  safe, though, you order a test. The test
  accurately classifies roughly 80 of malignant
  tumors and 90 of benign tumors. The test report
  comes back indicating malignant. Given your
  prior view that the chances of a malignancy were
  only 1, and given test results that are 80 or 90
  reliable, what would you say the overall
  chances of malignancy are now?
• _____

12
(No Transcript)
13
Appreciation of Sample Size

• A certain town is served by two hospitals.
  In the larger hospital about 45 babies are born
  each day, and in the smaller hospital about 15
  babies are born each day. As you know, about 50
  of all babies are boys. However, the exact
  percentage varies from day to day. Sometimes it
  may be higher than 50, sometimes lower. For a
  period of one year, each hospital recorded the
  days on which more than 60 of the babies born
  were boys. Which hospital do you think recorded
  more such days?
• _____ The larger hospital
• _____ The smaller hospital
• _____ About the same (i.e., within 5 of each
  other)

14
(No Transcript)
15
Takeaways from our brief experiment.

• You are human!
• You are likely influenced by frames, format,
  order of evidence, and your statistical intuition
  is sometimes flawed
• Most of the time this is not a problem but it can
  lead to predictable errors in judgment
• When the judgments are important, we care about
  how people make judgments and how to improve them

16
Accounting-related judgments decisions of

• Managers (e.g., Post retirement health care
  benefits, pension structure and benefits, stock
  options, bonus plan metrics, reporting choices)
• Investors (e.g., response to good/bad news,
  recognition vs. disclosure, format of reports)
• Analysts (e.g., forecasts, recommendations)
• Creditors credit rating agencies (risk
  assessment)
• Auditors (risk of misstatement, going concern)
• Other stakeholders (e.g., employees, customers,
  suppliers)

17
Behavioral Research Methods

• Archival data analysis (e.g., Burgstahler
  Dichev 1997 behavioral finance)
• Controlled experiments (my emphasis today)
• Surveys (Nelson, Elliott Tarpley 2002)
• Case studies (rare)

18
Burgstahler Dichev (JAE 1997)Earnings
management to avoid earnings decreases and losses

• Behaviorally informed archival research
• Used prospect theory
• Gains and losses relative to a reference point
• Concave value function for gains, convex for
  losses
• Loss aversion (steeper disutility for losses than
  utility for gains)

19
(No Transcript)
20
Burgstahler Dichev (1997)

• Sharp discontinuity around zero for levels of
  earnings and changes in earnings (compared to
  last year).
• Firms are more likely to make .01/share than to
  lose .01/share or exceed last years earnings by
  a penny than to miss by a penny
• Prospect theory predicts loss aversion

21
Behavioral Research Methods

• Archival data analysis (Burgstahler Dichev
  1997 behavioral finance)
• Controlled experiments (my emphasis today)
• Surveys (Nelson, Elliott Tarpley 2002)
• Case studies (rare)

22
II. Why use experiments?

• Manipulate the variables of interest and control
  or randomize on others
• Causal theory can be tested (not just
  association)
• Can get at process not just outcomes
• Ex ante research is possible (FASB SEC
  interested e.g., Hopkins 1996)
• Some conditions that do not exist in natural
  settings can be created in the lab

23
III. What is good research?

• Good topic
• Good design
• Good analysis

24
A. What is a good research topic?

• Problem is described clearly, comprehensively
• Important constituents care about it!
• Problem is challenging but not impossible
• Possible to evaluate solutions or interpret the
  direction of results
• The results will add to our knowledge about the
  problem and have theoretical and/or practical
  ramifications

25
Tip Kinneys 3 Paragraphs (footnote 18 in his
1986 TAR article)

• For every research idea/paper write 3 paragraphs
  answering the following
• What problem or issue will be addressed?
• Why is the problem or issue important?
• How you will address the problem
• (and what do you expect to find)?

26
My Example Paper for Today

• Frank Hodge, TAR, October 2001
• Hyperlinking unaudited information to audited
  financial statements effects on investor
  judgments
• Test of Kinneys 3 questions

27
Q1 Hodge 2001

• This study investigates whether firms can
  influence investors perceptions of their
  financial reports by hyperlinking unaudited
  information to information in the audited
  financial statements.

28
Hodge 2001 Q1 continued

• Specifically, does hyperlinking optimistic,
  unaudited information about a firms prospects to
  audited financial statements lead investors to
• Misclassify unaudited info. as audited,
• Inflate the credibility of unaudited info.,
• Judge the firms earnings potential to be higher,
  relative to viewing the same info. in hardcopy
  format?
• Will a warning help?

29
Q2 Who cares about this problem?

• Management
• SEC
• FASB
• AICPA
• Accounting/auditing researchers
• System designers(?)

30
Q3 Method and results?

• Method
• Used an experiment.
• Results
• Hyperlinked info was perceived as more credible
• Firm judged to have greater earnings potential
• Simple labels (Un)audited reduced the effects

31
Q3 Hodge 2001

• Method
• Used an experiment. Why?
• Data not observable from investors investment
  decisions.
• Can control info. and extraneous factors re.
  people, tasks, and environment.

32
B. What is good Research design?

• Theory
• Answers why
• Construct Validity
• Internal Validity
• External Validity
• Giving yourself a decent chance of testing your
  theory and finding results

33
Model Definitions (Kinney 86)
Y f ( X, Vs, Zs )
Y phenomenon to be explained
X your (new) theory about a cause of Y
Vs prior causes of Y
Zs contemporaneous causes of Y
34
A simple model
35
Building a Research Model
36
For X to cause Y

• X and Y are correlated
• Alternative explanations are ruled out by design,
  including
• Y causes X
• X, Y are caused by an omitted V or Z
• Reason to believe that operational X and Y
  represent X and Y
• Reason to believe that X Y relation generalizes
  to other persons, times, and settings.

37
. . .threats to Predictive Validity

• Statistical Conclusion Validity (5)
• Internal Validity (4)
• Construct Validity (2 and 3)
• External Validity (1)

38
Threats to Validity (contd)
1 Ext. Val.
5 Stat. Val.
4 Internal Validity
39
Research Model for Hodge 2001
Independent
Dependent
1
Financial Decisions
Assoc. of unaudited Information with Audited F/S
Conceptual Operational Control
40
To achieve Internal Validity

• account for Vs and Zs by
• random assignment of participants to
  treatment/control
• estimate statistically and remove (covariate
  analysis, regression)
• match on Vs
• hold constant by design/selection
• (special case of matching)

41
The Researchers Problem (Kinney 1986) or Giving
Yourself a Chance to find Results
? risk that data incorrectly accepts new
theory
? risk that data incorrectly rejects new
theory
? true effect of X on Y
? residual variation given research design
(i.e., after effects of Vs and Zs)
n available sample size
All five are related through a single, simple
formula
42
Researchers Problem (continued)
ß is the risk of failing to detect true
phenomenon (you want to minimize)
? is fixed at .05 or .10 by convention
?? f(X)
?? f(Vs, Zs)
n is semi-fixed by data availability or cost
43
Graphically . . . (small d, small s b okay)
Y
0
44
Graphically (large d, large s b okay)
_ Y HA, n, s
_ Y H0, n, s
a
Y
d
0
45
Graphically (small d, large s
_ Y HA, n, s
_ Y H0, n, s
a
Y
d
0
46
What to do? Need Power!

• Increase sample size (not always possible)
• Improve research design
• Theory
• Have you identified and measured all the major
  Vs and Zs?
• Have you manipulated the Xs at appropriate
  levels that allow you to find effects?
• Manipulation checks?
• Upfront decisions in experiments ? no second
  chances!!

47
(Difficult but) Necessary Choices in Experiments

• Professional participants?
• Monetary incentives?
• Between- or within-subjects design (see
  Schepanski, Tubbs, and Grimlund 1992)?
• Level of analysis, individual or market (see
  Libby, Bloomfield Nelson 2002)?

48
1. Choice of participants?

• Theory should dictate this choice
• Negative externalities of using professionals
  when not needed limited resource (Libby,
  Bloomfield Nelson 2002)
• Selection bias is stronger with professionals
  than with more available groups like students
  (Peecher Solomon 2001)
• Hodge 2001 used MBA students as investors
• Elliott, Hodge, Kennedy Pronk (2007)

49
2. Monetary incentives?

• Dictated by theory (effort sensitive task?)
• Want participants to take the task seriously
• Many results driven more by ability than
  incentives (running a 4-minute mile)
• Incentives do not necessarily mitigate judgment
  bias depends on type of bias (Arkes 1991
  Camerer Hogarth 1999)
• Reduce variance but mean unchanged
• Incentives are not always possible/practical in
  real life either!

50
Camerer Hogarth (1999)

• Empirically misguided and counterproductive to
  say incentives always help or never help
• Bargaining games and risky gambles
• Reduces risk preferring and generous behavior
• JDM
• Depends on task. Can help or hurt performance
• Multiplying stakes does not produce similar
  boosts in performance (none to low has most
  effect)
• Little effect on mean but can reduce variance

51
3. Within (vs. Between)

• Enhanced statistical power, subjects are their
  own control less variance
• Increases salience of treatment effects
• Vulnerable to carry-over effects
• Can help demonstrate the (un)intentional nature
  of cognition
• Requires proper analysis
• Hodge 2001 used a between-subject design
• Our experiment was between subjects

52
Hodge 2001 Experimental Design

• Three condtions
• Control Hardcopy of unaudited letter and
  audited financial statements
• Treatment 1 Audited financial statements on
  computer with hyperlinked unaudited letter
• Treatment 2 same as 1. Above, except that when
  the hyperlink was activated a notice popped up
  saying (Un)audited.

53
4. Individuals or Markets?

• Theory dictates
• Markets are expensive
• Markets can have less observations, therefore
  less power (unless repeated measures)
• Research shows that biases are not driven out by
  markets (may be reduced though)
• Hodge 2001 studies individual judgments but
  ultimately he wants to say something about
  behavior in the market systematic individual
  results aggregate to market behavior?

54
IV. Finding a research topic?

• Dont be method driven (let the topic determine
  the method not vice versa)
• Be behaviorally informed-- Archival data with
  behavioral theory?
• Be archivally informed gt what has been done on
  this topic?
• Many financial reporting issues are particularly
  well-suited to experiments
• E.g., ex ante questions, no data
• Think broadly!
• You want to get the big potatoes!

55
Finding a research topic (contd)

• Read, read, read
• Collaborate with others who may not use the same
  methods as you
• Collaborate with others in strategy, mgmt,
  finance, psychology, economics
• Be creative and curious
• It is a long season
• Do what you find interesting

56
V. Common Pitfalls

• Topic fails Yawn Test
• Predicting the null hypothesis
• Accepting the null hypothesis
• Low power tests
• Poor motivation small incremental contribution
• Running experiments too early (design issues)
• Not getting adequate feedback before submitting
  (poorly written or organized)

57
Top 5 Takeaways

• Have fun!
• Seek lots of input at each stage
• Use a well-thought out design
• Choose the research method that best informs the
  question
• Choose an interesting topic

58
Q A?
59
Helpful Papers on Research

• Arkes, H. R. Psychological Bulletin, Nov. 1991.
• Bonner, S., Accounting Horizons, Dec. 1999
• Camerer, C, R. Hogarth. JRU, 1999.
• Kinney, W., The Accounting Review, Apr. 1986.
• Libby, R. M. Lipe, JAR, 30, 1992.
• Libby, R., Bloomfield, R., M. Nelson. AOS,
  Nov., 2002.
• Maines, L., BRIA, Supplement 1994.
• Peecher, M. I. Solomon, IJA, 2001.

60
During the Breakout Session

• Each group thinks of a simple research idea to
  examine experimentally
• Design an experiment with at least one
  independent variable and one dependent variable
• Prediction(s)?
• Who will be your participants?
• Between or within- subjects design?

61
When we return from break

• Each group quickly presents idea, independent
  variable, dependent variable, prediction, choice
  of participants, and design
• We will decide on the best idea with an applause
  meter!
• Be creative!

62
(No Transcript)
63
What about Efficient Markets?

• Systematically biased traders (dont cancel)
• Limits on arbitrage (cant fully exploit)
• Some empirical results inconsistent with market
  efficiency
• Post earnings announcement drift (Nichols
  Whalen 2004 Bernard Thomas 1989)
• Accrual anomalies

64
How Do Managers Value Stock Options and
Restricted Stock?
http//papers.ssrn.com/sol3/papers.cfm?abstract_id
677438