Language Emergence for Activities

1
Language Emergence for Activities

• Manas Agarwal
• Under the guidance of
• Dr. Amitabha Mukerjee

2
The Problem Area

• Language emergence and development.
• How are universal features of language specified?
• How structures in languages emerge?

3
Language Emergence

• Many theories
• Regularities in expressions based on holophrases.
• Biological adaptation.
• Regularities in interaction with natural world.

4
Language Game

• Vogt and Steels 1.
• A model to study language emergence.
• Multi-agent.
• Object identification and naming.

5
Language Game

• One generation transmits their language to the
  next generation.
• Played by two agents
• Speaker
• Hearer
• Speaker from older generation.
• Hearer from a new generation.

6
Language Game

• Agents make contact.
• Agents categorize sensory experience of objects.
• Speaker encodes feature set into expression.
• Hearer decodes the expression from its lexicon
• Hearer compares the decoded features with the
  expected ones. Positive feedback if successful.

7
Language Game
The Naming Game processes
8
The Iterative Learning Model

• Framework to model cultural transmission of
  languages.

9
The Iterative Learning Model

• Novel users learn from adult users.
• After some number of games, adults leave,
  replaced by new users.
• This iteration continues.

10
Whats missing

• Learning for Activities.
• Studies for only noun/object learning in
  languages has been done.

11
Why activities?

• It has been seen that infants build their
  concepts more on dynamic scenes.
• Verbs are essential, provide a meaning to
  sentence.

12
Our model

• On the lines of the naming game.
• Game consists of two distinct objects.
• Scene consists of basic interactions between
  objects.

• Push
• Shove
• Hit
• Harass
• Bounce
• Counter-shove
• Chase

13
Our model

• Feature vector set
• Color of object A
• Color of object B
• Tuples (f1 , f2 , t)
• f1 (xB-xA). (vB-vA)
• f2 (xB-xA). (vBvA)
• t timestamp of the frame

14
Categorizing Activity

• Satish 5 model.
• Unsupervised learning.
• Merge Neural Gas Algorithm.
• Uses f1 , f2 as input signals.

Delauney triangulation of neurons.
Delauney triangulation of high signal activity
region
15
Categorizing Activities
Clustering and Labelling results
16
The syntax

• The speaker will encode the feature set into a 3
  words utterance
• The 2 objects in the scene
• The interaction between them
• The order of the 3 words can be any, 6 ways.

17
Our aim

• To be able to categorize activities.
• Implement the game.
• Which syntax emerges?
• What Lexicon emerges?

18
References

• Steels, L. and Vogt, P. , Grounding adaptive
  language games in robotic agents.
• Vogt, P. , The emergence of compositional
  structures in perceptually grounded language
  games.
• Brighton,H., Kirby,S. and Smith,K. , Situated
  cognition and the role of multi-agent models in
  explaining language structure.
• Cohen,P.R.,Morrison,C.T. and Cannon,E., Maps for
  verbs The relation between interaction dynamics
  and verb use.
• Satish,G. and Mukerjee,A. , Acquiring linguistic
  argument structure from multimodal input using
  attentive focus.

19
Thank You