The Information Pump: A New Qualitative Research Methodology

1
The Information PumpA New Qualitative Research
Methodology

• October, 2003

Presented by
V2.0
2
The Information Pump
3
Overview

• Product development requires in-depth,
  representative, and timely information about
  customer needs.
• Research respondents who provide this crucial
  information are typically paid for their time but
  not for the quality of the information itself.
• The Information Pump provides respondent
  incentives for high-quality as well as quantity
  of feedback relevant to the product design
  process.

4
Key Strengths

• Most current methods, such as interviews, focus
  groups, and Voice of the Customer, presume a
  good-faith effort on the part of the participant.
  
• Information Pump incentives deliver at least
  three advantages over traditional qualitative
  methods for generating feedback on product
  concepts.
• Promotes a higher level of motivation and effort,
  which is especially important if the task is
  difficult or fatiguing.
• Communicates clearly what kinds of information
  are being solicited.
• Provides a way of identifying superior
  respondents who can then be retained for
  subsequent studies.

5
Background

• First developed by as part of the Virtual
  Customer Initiative at MITs Sloan School of
  Management, by Professor Drazen Prelec.
• Using a game theory approach, the Information
  Pump "bootstraps" respondent incentives by
  comparing the information provided by one
  respondent against that provided by other
  respondents at the same time.
• Professor Prelec, in his technical paper on the
  Information Pump, proves that when respondents
  play the game to maximize their own incentive
  payments, optimal new insights are gained for the
  product concept.

6
Structure of the Information Pump game

• A respondent receives a rich description of a new
  product concept and then provides concept
  statements, which are sent to other respondents.
• The discussion is structured so that instead of
  simply voicing opinions, participants vie to
  produce the most insightful observations about
  the new product.
• The flow has some resemblance to a parlor game in
  that there is an element of "challenge" and each
  participant accumulates a dollar amount for
  superior performance.
• In essence, one gets credit for each statement
  that is non-redundant and that is recognized as
  relevant (an "a-ha").
• The key feature of the protocol is that it
  provides optimal incentives to respondents to
  provide this information.
• The Information Pump game can be deployed in a
  variety of settings
• Within focus groups
• Through direct mail surveys exchanged through a
  clearing house
• In an online chat room
• Through use of an interactive Web site

7
How the Information Pump game operates

• The game is set up between pairs of players. An
  unlimited number of pairs is possible.
• In a given pair, one player (Encoder) sees the
  product concept, perhaps with graphics, and
  writes three (or more) statements about it.  The
  statements may be true or false, in the view of
  the Encoder.
• In Round One, the other player (Decoder), without
  seeing the concept, guesses at the truth or
  falsity of each of the Encoders statements.
• After making the initial guesses, the Decoder, in
  Round Two, then gets to see the concept.  He/she
  then guesses again whether each of the Encoders
  statements is true or false.
• The Encoder's score is based on the difference
  between the Decoders correct second guesses
  (Round One) and correct first guesses (Round
  Two).
• The Decoder's score, on the other hand, is based
  on the total number of correct guesses in both
  rounds.
• The players are rewarded monetarily on the basis
  of their scores.
• (Note for simplicity in explaining the game, a
  two-player version is detailed in the case study
  below. Multiple-player versions are also
  possible.)

8
Case StudyEncoder and Decoder Instructions and
Product Stimulus
9
Encoder Instructions - Introduction

• You are about to participate in a game against a
  single opponent whom we have chosen for you. It
  should take no more than ten minutes of your
  time, and you have the chance to win up to
  90.00.
• On a page to follow, you will find a description
  of a new snack bar. After reading the
  description, you will be asked to write three
  statements about the product. Any or all of your
  statements can be either TRUE or FALSE, based on
  how you see the product. You decide whether any
  statement is true or false.
• Your statements will be sent to your opponent in
  this game. In the first round of the game, he or
  she will not get to see the product description.
  Instead, he or she will be asked to guess whether
  your statements are TRUE or FALSE, based solely
  on what you have stated and that the product is a
  snack bar.
• In the next round, your opponent will get to see
  the product description. Again, he or she will
  be asked to assess whether your statements were
  TRUE or FALSE.

10
Encoder Instructions - Scoring

• Both you and your opponent get 30.00 at the
  beginning of the game.
• Every time your opponent guesses correctly on one
  of your statements in the first round (without
  seeing the product description), he or she wins
  10.00, and you lose 10.00.
• For every statement that your opponent guesses
  incorrectly in the first round, he or she loses
  10.00, and you win 10.00.
• For example
• If your opponent guesses correctly on all three
  in the first round, he or she will have 60.00
  (the original 30.00 plus 30.00 won from you).
  You will then have 0.00.
• If your opponent guesses incorrectly on all
  three, you end up with 60.00 and he/she ends up
  with 0.00 in the first round.
• Thus, the idea in the first round is for you to
  write statements that will fool your opponent
  into guessing wrong that is, to guess TRUE on
  your FALSE statements, or to guess FALSE on your
  TRUE statements.

11
Encoder Instructions Scoring (continued)

• In the second round, your opponent gets to see
  the product concept and guess again about whether
  your statements are true or false.
• This time, if he or she guesses correctly on one
  of your statements, you get 10.00 AND your
  opponent gets 10.00.
• If he or she guesses incorrectly on a statement,
  neither of you are paid for that statement in
  this round.
• For example
• If your opponent has guessed incorrectly on all
  three of the statements in the first round (you
  then have 60.00 at the end of that round) but
  then guesses correctly on all three in the second
  round, you will end up with 90.00.
• If your opponent guesses correctly on all three
  statements in the first and second round, he or
  she ends up with 90.00 and you will have 30.00
  (since you will have lost 30.00 in the first
  round but gained 30.00 back in the second
  round).
• If your opponent guesses correctly on all three
  statements in the first round, but incorrectly on
  all three in the second round (which is
  unlikely), you end up with your 0.00.

12
Encoder Instructions Scoring (continued)

• Thus, the idea for both rounds is for you to
  write statements that do not appear to be
  obviously true or false on their face, but which
  are indeed obvious once someone sees the product
  description!
• Good Luck!

13
Encoder Instructions Product Stimulus
14
Encoder Instructions True-False Statements

• Now, type three statements that may or may not
  describe this product. For each statement,
  indicate whether you believe it to be TRUE or
  FALSE. Refer to the picture above as needed.
• Statement 1
• _______________________________________________
• _______________________________________________

15
Encoder Instructions True-False Statements
(continued)

• Statement 2
• _______________________________________________
• _______________________________________________

• Statement 3
• _______________________________________________
• _______________________________________________

16
Decoder Instructions - Introduction

• You are about to participate in a game against a
  single opponent whom we have chosen for you. It
  should take no more than ten minutes of your
  time, and you have the chance to win up to
  90.00.
• On a page to follow you will find three
  statements that your opponent has made about a
  new snack bar. Each statement is either TRUE or
  FALSE.
• In the first round of the game, you will be asked
  to tell us whether you think each statement is
  true or false, based solely on the statements
  themselves. It may seem difficult to tell, but
  just use your best judgment about snack bars.
• In the second round of the game, you will be able
  to read a description of the product. Following
  that, you will again be asked to guess whether
  the original three statements are TRUE or FALSE.
  This part is obviously easier than the first.

17
Decoder Instructions - Scoring

• Both you and your opponent get 30.00 at the
  beginning of the game.
• Every time you guess correctly on one of your
  opponents statements in the first round (without
  seeing the product description), you win 10.00,
  and your opponent loses 10.00.
• For every statement that you guess incorrectly in
  the first round, you lose 10.00, and your
  opponent wins 10.00.
• For example
• If you guess correctly on all three in the first
  round, you will have 60.00 (the original 30.00
  plus 30.00 won from your opponent). He or she
  will end up with 0.00.
• If you guess incorrectly on all three, your
  opponent ends up with 60.00 and you end up with
  0.00.
• Thus, the idea in the first round is for you to
  assess how true or false each statement is likely
  to be about a new snack bar.

18
Decoder Instructions Scoring (continued)

• In the second round, you get to see the product
  concept and guess again about whether your
  opponents statements are true or false.
• This time, if you guess correctly on one of his
  or her statements, you get 10.00 AND your
  opponent gets 10.00. If you guess incorrectly
  on a statement, neither of you are paid for that
  statement in this round.
• For example
• If you have guessed incorrectly on all three of
  the statements in the first round (you then have
  0.00 at the end of that round) but then guess
  correctly on all three in the second round, you
  will end up with 30.00.
• If you guess correctly on all three statements in
  the first and second round, you end up with
  90.00 (since you will have won 30.00 in the
  first round, 30.00 in the second round, plus the
  30.00 you began with).
• If for some reason you guess incorrectly on all
  three statements in the first round and
  incorrectly on all three in the second round
  (which is unlikely), you end up with 0.00.

19
Decoder Instructions Scoring (continued)

• Thus, the idea for both rounds is for you to
  guess how true each statement is likely to be,
  both before and after seeing the product
  description!
• Good Luck!

20
Decoder Instructions True-False Statements for
Round One

• Please provide your best guess about whether each
  of these statements is true or false for the new
  snack bar (you will get to see the product
  description after this).

• Encoders statement 1
• _______________________________
• _______________________________

• Encoders statement 2
• _______________________________
• _______________________________

• Encoders statement 3
• _______________________________
• _______________________________

21
Decoder Instructions Product Exposure after
Round One
22
Decoder Instructions True-False Statements for
Round Two

• Now that youve had a chance to see the product,
  please assess again whether you think each
  statement is true or false

• Encoders statement 1
• _______________________________
• _______________________________

• Encoders statement 2
• _______________________________
• _______________________________

• Encoders statement 3
• _______________________________
• _______________________________

23
Analysis of Data Collected from the Information
Pump Game
24
Assessing the Value of Product Concept Statements

• For the purposes of this example
• We assume that 50 encoders have reviewed the
  product description shown in the Encoder
  Instructions.
• Following this, they have each contributed 3
  statements about the product, which they
  designate as either true or false.
• This produces a total of 150 (50 X 3) statements.
  
• We assume that 50 decoders have each evaluated
  one set of encoder statements (the number and
  combination of encoders and decoders, in
  practice, is flexible).
• The decoders have provided
• On the first round, true or false guesses for
  each encoder statement without viewing the
  product description.
• On the second round, another guess, this time
  after having viewed the product.
• The encoder statements are grouped into semantic
  classes. Each class communicates one relatively
  unique message or attribute for the new snack bar
  product.

25
Data structure

• In the table below, True is coded as a 1
  while False is coded as a 0.

• The table shows the data for only the first three
  encoders (id 1, 2 and 3). Each encoder has
  contributed three statements, true or false,
  about the new product.

26
Data structure

• Considering the data in this table, we can infer
  that the decoders get the ideas behind
  statement 2, 3, 5, 8, and 9 after viewing the
  product description, even though those statements
  seemed false (or perhaps implausible) to begin
  with. These are the Ah-Has.
• Statements 1, 4, 6, and 7, on the other hand, are
  obvious in the sense that the decoders guessed
  correctly about them even before seeing the
  product description. These might be attributes
  that pertain, for example, to snack bars in
  general and so are easy to guess at.

27
Analysis of the Encoder-Decoder Data

• Below, we show how the encoder and decoder data
  can be quantified and portrayed graphically to
  help find aspects of the product that are
  ah-has.
• Quantifying the Information Pump data proceeds as
  follows.
• Group the encoder statements into classes.
• As noted above, the 150 statements in our example
  were coded into 10 classes.
• Each class communicates one more or less unified
  message (e.g., tastes salty, good for
  parties, etc.)
• For each class of encoder statements, determine
  the percentages of
• Correct first guesses (that is, in decoder
  agreement with the encoder truth or falsehood of
  the statements in the class), before viewing the
  product description.
• Correct second guesses, after viewing the product
  description.

28
X-Y graphing of first and second guesses

• On an X-Y graph, plot the percentage of incorrect
  first guesses against the percentage of correct
  second guesses for each class of statements.

29
Interpretation of the X-Y graph

• The graph above tells us that, of the 10 classes
  of statements, two were ah-has in the sense
  that the decoders were not able to guess the
  encoders perceptions correctly until after
  viewing the product description.
• The statements are good on scouting trips and
  take on airplanes. These two help pin down
  what may be unique about the Mojo Bar within the
  general snack bar category.
• Perhaps the salty element is seen as a plus for
  hiking trips, while the bar also represents a
  good adult snack for air travel when meals are
  not otherwise available.
• Either or both of these scenarios could be
  potentially useful in advertising or direct
  marketing campaigns.
• The graph also pinpoints some attributes/benefits
  that seem too obvious to highlight the products
  uniqueness.
• Any snack bar might make the claim that it
  tastes good or is inexpensive. Thus, these
  dimensions may not be as fruitful for marketing
  communications strategies.
• In addition, some features represent potentially
  useful benefit dimensions, even though they are
  not strong ah-has and not too obvious.

30
Basic Scoring

• In some situations it is advantageous to assign a
  score to each specific class of statements. This
  quantifies the perceived dimensions and allows
  them to be rank-ordered and prioritized.
• One type of score is to calculate
• ( of correct second guesses following an
  incorrect first guess) minus the ( of correct
  second guesses following a correct first guess)
• This score represents the relative degree of how
  much a class of statements is ah-ha versus too
  obvious.
• Although the graphing technique yields similar
  information, scoring each class of statements
  gives it a single metric that can be used to
  assess the statements effectiveness.

31
Variations on basic scoring

• Some variations on the basic score can increase
  its effectiveness and/or ability to predict the
  success of marketing communications. These
  include
• Applying different weights to the first component
  (second correct, first incorrect) and to the
  second component (second correct, first correct).
• Weighting statements or classes of statements
  differently when they are true rather than false.
  In some cases, true statements offer more value
  to a marketing campaign than do false statements.
• Using different formulas to combine the
  percentages. These could include, for example, a
  log-odds ratio instead of subtraction, with or
  without weighting factors.

32
Statistical Testing

• It can be helpful to apply statistical tests to
  assess how different the classes of statements
  (perceived dimensions) are from one another.
• Even if the scores are rank-ordered, unless their
  differences are statistically significant, we
  cannot be as confident about their meanings.
• Two general types of statistical tests (along
  with others) can be applied to the data
• Fishers exact tests
• Maximum likelihood/asymptotic tests
• These tests can tell us the probability that the
  pattern of percentages (first guess, second
  guess, etc.) may have occurred by chance alone.

33
Profiling of participant background, usage and
preferences

• In most cases, we will have not only the results
  of the Information Pump game but data on the
  participants
• Age, gender, household size, income
• Snack food consumption habits, preferences and
  price sensitivity.
• Thus, we can examine the relationships between
  these background variables and the Information
  Pump data, especially tying them to the types of
  statements generated during the game.
• The profiling results can help identify different
  consumer types (segments) who are attracted to
  different benefit dimensions. This will provide
  additional strategic information when targeting
  market segments in any given campaign or strategy.

34
Profiling of participant background, usage and
preferences

• (Simplified) example of how profiling works with
  the basic scores generated by the Information
  Pump data

• The data above indicate that there may be unique
  targeting opportunities among
• (red) Business travelers who carry snack bars
  with them.
• (blue) Larger families with kids in scouting or
  other outdoor organizations.
• (green) Women 35-44 with young families who
  already consume a large quantity of snack bars,
  looking for a healthy snack for their children.

35
Summary

• How are these analyses different from brand
  preference and focus group findings?
• They can produce a better picture of the unique
  benefits (ah-has) offered by a product than
  typical survey and focus group research.
• They can be used to prioritize messages for
  marketing communications.
• By examining segments consumers who have produced
  higher scores, we can find out not only what they
  like, but what it is about the product that
  appeals uniquely to them within the brands
  competitive context.

36
Contact Information

• Nexus Business Information, Inc.
• Phone [phone]
• FAX [phone]
• Web www.nbii.com
• E-Mail information_at_nbii.com
• 1824 Sandalwood Pkwy., Ste. 110
• Grapevine, TX 76051-7344
• PO Box 613233, D/FW Airport, TX 75261-3233
• Mike Morgan
• Phone [phone]
• Cell 972 816.1603
• Michael.Morgan_at_nbii.com

37
Mike Morgan EVP / Senior Partner

• Experience
• MARC Research VP and Senior Consultant 2
  years
• Sprint Business Database Marketing and Primary
  Research groups - 3 years
• Savitz Research Marketing Science Director 1
  year
• Cornell University Marketing Professor 6
  years.
• Decision Scripts (statistical consulting) - 8
  years.
• Ten years of other management experience.
• Knowledge Areas
• Buyer Judgment/Decision Processes and Segments
• Modeling Choice Behavior and Other Responses to
  Marketing Strategies
• Pricing Theory and Model Development
• New Product Acceptance and Growth Models
• Sales Force Incentive Plan Models and Analysis
• Distribution Channel Analysis and Modeling
• Education
• Ph.D., Marketing, University of Texas at Dallas
• MS, Marketing, University of Texas at Dallas
• BA, University of Texas at Austin

38
Robert New Pres./Managing Partner

• Experience
• Nexus Business Information President and Dir.
  Research - 4 years
• Gartner Group Director Consulting - 3 years
• Nortel Networks Dir. Strategic Marketing and
  Research - 10 years
• Ten years of other management experience with
  GTE, Piper Aircraft and USAF.
• Knowledge Areas
• New Product Acceptance and Growth Models
• Buyer behavior and Cognitive Modeling
• Modeling Choice Behavior and Other Responses to
  Marketing Strategies
• Pricing and promotions
• Advertising response
• Marketing Strategy
• Education
• Ph.D. Marketing, University of Texas at Dallas
  (bda)
• MBA, Marketing/Finance, University of Florida
• BA, University of Texas at Austin