Information Aggregation: Experiments and Industrial Applications

1
Information AggregationExperiments and
Industrial Applications

• Kay-Yut Chen
• HP Labs

2
Agenda

• Lessons from HP Information Markets
• (Chen and Plott 2002)
• Scoring Rules and Identification of Experts
• (Chen, Fine and Huberman 2004)
• (Chen and Hogg 2004)
• Public Information
• (Chen, Fine and Huberman 2004)

3
HP Information Markets (Chen and Plott)

• Summary of Events
• 12 events, from 1996 to 1999
• 11 events sales related
• 8 events had official forecasts
• Methodology Procedures
• Contingent state asset (i.e. winning ticket pays
  1, others 0)
• Sales amount (unit/revenue) divided into (8-10)
  finite intervals
• Web-based real time double-auction
• 15-20 min phone training for EVERY subject
• Market open for one week at restricted time of
  the day (typically lunch and after hours)
• Market size 10-25 people

4
(No Transcript)
5
Results
6
Business Constraints and Research Issues

• Not allowed to bet players own money -gt stakes
  limited to an average of 50 per person
• Time horizon constraints -gt 3 months to be useful
• Recruit the right people
• Asset design affects the results (How to set the
  intervals?)
• Thin markets (sum of price 1.11 to 1.31 over
  the dollar)
• Few players
• Not enough participation

7
Reporting with Scoring Rule
Outcome
A
B
C
Reports of Probability Distribution
p1
p2
p3
Pays C1C2Log(p3)
8
Information Aggregation Function
If reports are independent, Bayes Law applies
9
Two Complications

• Non-Risk Neutral Behavior
• Public Information

10
Dealing with Risks AttitudesTwo-Stage Mechanism
Stage 1 Information Market Call Market to
Solicit Risk Attitudes
Event 1
Time
Event 2
Event 3
Event 4
Stage 2 Probability Reporting
Aggregation Individual Report of Probability
Distribution Nonlinear Aggregated Function
Event 5
Event 6
Event 7
Event 8
11
Second Stage Aggregation Function
Bayes Law with Behavioral Correction
Normalizing constant for individual risks
ir(V i / i)c
Holding value/Risk - measure relative risk of
individuals
market risk sum of prices/winning payoff
12
ExperimentsInducing Diverse Information
Outcome
A
B
C
Random Draws Provide Info
In actual experiments, there are TEN states
13
Comparison To All Information Probability
Kullback-Leibler 1.453
Experiment 4, Period 17 No Information
14
Kullback-Leibler Measure

• Relative entropy
• Always gt0
• 0 if two distributions are identical
• Addictive for independent events

15
Comparison To All Information Probability
Kullback-Leibler 1.337
Experiment 4, Period 17 1 Player
16
Comparison To All Information Probability
Kullback-Leibler 1.448
Experiment 4, Period 17 2 Players Aggregated
17
Comparison To All Information Probability
Kullback-Leibler 1.606
Experiment 4, Period 17 3 Players Aggregated
18
Comparison To All Information Probability
Kullback-Leibler 1.362
Experiment 4, Period 17 4 Players Aggregated
19
Comparison To All Information Probability
Kullback-Leibler 0.905
Experiment 4, Period 17 5 Players Aggregated
20
Comparison To All Information Probability
Kullback-Leibler 1.042
Experiment 4, Period 17 6 Players Aggregated
21
Comparison To All Information Probability
Kullback-Leibler 0.550
Experiment 4, Period 17 7 Players Aggregated
22
Comparison To All Information Probability
Kullback-Leibler 0.120
Experiment 4, Period 17 8 Players Aggregated
23
Comparison To All Information Probability
Kullback-Leibler 0.133
Experiment 4, Period 17 9 Players Aggregated
24
Comparison To All Information Probability
Experiment 4, Period 17
25
  KL Measures for Private Info Experiments  
 
 
26
Group Size Performance
27
Did the Markets Pick out Experts?

• KL measure of all query data
• Pick groups of 3

28
Did Previous Queries Pick out Experts?

• KL measure of second half of query data
• Pick groups of 3

29
Public Information

• Information observed by more than one
• Double counting problem

30
Information Aggregation with Public
Information Kullback-Leibler 2.591
Public Info Experiment 3, Period 9 11 Players
Aggregated
31
Dealing with Public InformationAdd a Game to
the Second Stage
Stage 1 Information Market Call Market to
Solicit Risk Attitudes
Event 1
Time
Event 2
Event 3
Event 4
Stage 2 Probability Reporting
Aggregation Individual Report of Probability
Distribution Matching Game to Recover Public
Information Modified Nonlinear Aggregated Function
Event 5
Event 6
Event 7
Event 8
32
Assumptions

• Individuals know their public information
• Private Public Info Independent
• Structure of Public Info Arbitrary

33
Matching Game
Outcome
A
B
C
Reports of Probability Distribution
q11
q12
q13
Player 1 q1
q21
q22
q23
Player 2 q2
q31
q32
q33
Player 3 q3
. . .
. . .
. . .
. . .
34
Matching Game

• Any match function f(q1,q2) with property
• Max when q1q2
• Multiple Equilibria
• Payoff increases as entropy decreases
• Hopefully, individuals report public information

35
Aggregation Function withPublic Information
Correction
Bayes Law with a) Behavioral Correction b)
Public Info Correction
Normalizing constant for individual risks
ir(V i / i)c
Holding value/Risk - measure relative risk of
individuals
market risk sum of prices/winning payoff
36
Public Information Experiments

• 5 Experiments
• Various Information Structures
• All subject received 2 private draws 2 public
  draws
• All subject received 3 private draws 1 public
  draws
• All subject received 3 private draws half of
  the subjects receive 1 public draws
• All subject received 3 private draws 1 public
  draws. 2 groups of independent public
  information.
• 9 to 11 participants in each experiments

37
Correcting for Public Information
Kullback-Leibler 0.291
Public Info Experiment 3, Period 9 11 Players
Aggregated
38
  KL Measures for Public Info Experiments  
39
Summary

• IAM with public info correction did better than
  best person.
• IAM with public info correction did better than
  markets in 4 out of 5 cases.
• IAM corrected with true public info did
  significant better than all other methods.

40
(No Transcript)
41
(No Transcript)
42
Supplementary
43
Previous Research

• Academic Studies
• Information Aggregation in Markets
• Plott, Sunder, Camerer, Forsythe, Lundholm,
  Weber,
• Pari-mutuel Betting Markets
• Plott, Wit Yang
• Real World Applications
• Iowa Electronic Markets
• Hollywood Stock Exchange
• HP Information Markets
• Newsfuture
• Tradesport.com

44
Risk Attitudes
45
Dealing with Risks AttitudesTwo-Stage Mechanism
Stage 1 Information Market Call Market to
Solicit Risk Attitudes
Event 1
Time
Event 2
Event 3
Event 4
Stage 2 Probability Reporting
Aggregation Individual Report of Probability
Distribution Nonlinear Aggregated Function
Event 5
Event 6
Event 7
Event 8
46
Probability Reporting
Outcome
A
B
C
Reports of Probability Distribution
p1
p2
p3
Pays C1C2Log(p3)
47
Second Stage Aggregation Function
Bayes Law with Behavioral Correction
Normalizing constant for individual risks
ir(V i / i)c
Holding value/Risk - measure relative risk of
individuals
market risk sum of prices/winning payoff
48
Private Information Experiments

• 5 Experiments
• Various Information Conditions
• All subject received 3 draws
• Half received 5 draws, half received 1 draw
• Half received 3 draws, half received random
  number of draws
• 8 to 13 participants in each experiments

49
Next Step

• Field Test (Fine and Huberman)