Peekaboom: A Game for Locating Objects in Images

1
Peekaboom A Game for Locating Objects in Images

• Roy Liu
• Carnegie Mellon University
• Lamps of Aladdin Presentation
• Joint work with Luis von Ahn and Manuel Blum

2
Object Location in Images
Given an image, determine what objects there are
and locate them
Woman Man Umbrella Tree Sailboat Dog
3
As Things Stand Now

• No algorithm is known for taking an image and
  determining what objects are in it, let alone
  telling you where they are.
• Fortunately, this talk isnt about developing
  such an algorithm. Lets try a different approach.

4
Harnessing the Power of Human Cycles

• Math is hard. Lets go shopping! Barbie
• On similar line of thinking
• Programming computers to locate objects in images
  is hard, so
• Lets not think about it.
• Instead, lets get humans to do the work for us?

5
Problems

• Wait! Your average human probably wants
• Enjoyment they want to have a good time
• Incentives they want something in return
• How do we address both?

6
A Game

• Have people do the work for us by playing a game.
• Many design issues arise
• What will be the core idea of the game?
• How do we collect data?
• How do we ensure the quality of the data?

7
An Earlier Idea Luis von Ahns ESP Game

• Two players, with no communication, each try to
  guess what the other is thinking about a
  particular image they both see.
• If they agree on a word, the game moves on and
  increases both players scores.

8
A Sample Run

• Player 2 Guesses
• Woman
• Shirt
• Denim
• Girl
• Model

• Player 1 Guesses
• Pants
• Model
• Lady

Server Agreed, Model
9
Why ESP Works

• By agreeing on a word, the players
• Say what it is we call this assigning a label
  to the image.
• Check their own work the fact that two
  strangers agree on a label is a witness of the
  labels quality.

10
The Limitations of ESP

• The ESP Game can label images (and consequently
  tell you whats in them), but it cannot
• Find the objects being labeled.
• Determine the way in which the object appears
  does the label car refer to the text car or
  an actual car in the image?

11
Completing the Image Cycle
12
A New Idea Peekaboom

• Two players are assigned the roles of revealer
  and guesser.
• The revealer sees an image with a label. The
  guesser sees nothing.
• The revealer shows the guesser parts of the
  image. If the guesser guesses correctly, the game
  moves on.

13
Statement of Purpose

• We would like to collect data about images
  systematically and en masse.
• We hope our collection will form the basis for
  data sets that can be used to train computer
  vision algorithms.

14
A (Simplified) Trial Run
15

• The Guesser guesses
• Flower
• Petal
• Butterfly

The Revealer clicks on parts of the image and
shows them to the Guesser.
Server Correct, Butterfly
16
Why Peekaboom Works

• By getting the guesser to guess correctly, the
  revealer locates objects by clicking on the
  relevant parts of the image

17
But Wait, Theres More

• Peekaboom not only locates objects, but also
• Gives the context necessary to identify them.
• Classifies the image as Text, Noun, or Verb
  by way of hints.
• Lets examine how Peekaboom does both.

18
Object Context
The label trunk

• Pings help separate the context of object with
  the object itself.
• They help the guesser distinguish trunk from
  other possibly correct labels like elephant,
  tusk, and ear.

19
Hints
The label car is ambiguous -- this is
car this is also car
The hints help distinguish the manner in which
the label car appears this is the object
car this is the text car
20
The Architecture of Peekaboom
21
Peekaboom is

• fun
• novel
• aesthetically appealing
• networked
• scalable
• widely deployable

22
Conclusions

• Peekaboom will be released to a general audience
  within two months.
• We hope that it will solve difficult AI tasks
  while achieving
• Low costs One game server.
• Quality Accurately locate objects in images.
• Quantity Locate objects in millions of images.

23
Advertisement

• The best way to understand this talk is to try
  the game out for yourselfwww.peekaboom.org
• We look forward to collecting your cycles!