The Evolution of Cooperation

1
The Evolution of Cooperation
Robert Axelrods
A Computer Game for Political Science

• Kentaro Toyama
• Microsoft Research India
• Indian Institute of Science August 10, 2005

2
Outline

• Prisoners Dilemma
• Two Contests
• Some Analysis
• Real-World Scenarios
• Agent-Based Simulation
• Discussion

3
Robert Axelrod

• Professor of Political Science and Public Policy
  at U Michigan, Ann Arbor
• First paper on cooperation published in 1980.
• Book (left) published in 1984 to wide acclaim.
• Best known for this and related work still
  active in this area and publishing new research.
• http//www-personal.umich.edu/axe/

4
Outline

• Prisoners Dilemma
• Two Contests
• Some Analysis
• Real-World Scenarios
• Agent-Based Simulation
• Discussion

5
The Prisoners Dilemma

• Two-player game
• Non-zero-sum
• Model for many real-world scenarios
• Story based on two criminals caught by police and
  interrogated separately

6
The Prisoners Dilemma
Player B
Cooperate
Defect
Payoff Matrix
-2
-2
0
-5
Cooperate
Player A
0
-5
-4
-4
Defect
Think of payoffs as number of years of life lost,
spent in jail.
7
The Prisoners Dilemma
Player B
Cooperate
Defect
Payoff Matrix
3
3
5
0
Cooperate
Player A
5
0
1
1
Defect
(For ease of thinking, add 5 to each payoff.
The larger the payoff, the better.)
8
The Prisoners Dilemma
Player B
Cooperate
Defect
Payoff Matrix
3
3
5
0
Cooperate
Player A
5
0
1
1
Defect
If Player A cooperates, Player B should defect.
9
The Prisoners Dilemma
Player B
Cooperate
Defect
Payoff Matrix
3
3
5
0
Cooperate
Player A
5
0
1
1
Defect
If Player A cooperates, Player B should defect.
If Player A defects, Player B should defect.
10
The Prisoners Dilemma
Player B
Cooperate
Defect
Payoff Matrix
3
3
5
0
Cooperate
Player A
5
0
1
1
Defect
No matter what the other player does, a rational,
self-interested player will defect. (This is a
Nash equilibrium.)
11
The Prisoners Dilemma
Player B
Cooperate
Defect
Payoff Matrix
3
3
5
0
Cooperate
Player A
5
0
1
1
Defect
No matter what the other player does, a rational,
self-interested player will defect. (This is a
Nash equilibrium.)
The Dilemma There is a joint strategy that
could result in better payoffs for both players.
(The Nash equilibrium is not Pareto-optimal.)
12
Other Kinds of Games
C
D
C
D
1 1
C
1 2
4 4
C
2 3
0 0
0 1
3 2
1 1
D
D
Exploitation
Linked Fates
Swerve
Straight
Heads
Tails
2 2
Swerve
1 3
1 -1
Heads
-1 1
3 1
0 0
-1 1
1 -1
Straight
Tails
Chicken
Matching Coins
13
The Prisoners Dilemma
Player B
Cooperate
Defect
Payoff Matrix
Reward for cooperation
Temptation to defect
3
5
Cooperate
Player A
Punishment for defection
Suckers payoff
0
1
Defect
T gt R gt P gt S
R gt (S T) / 2
14
The Prisoners Dilemma
Player B
Cooperate
Defect
Payoff Matrix
3
30
5
0
Cooperate
Player A
50
0
1
10
Defect
T gt R gt P gt S
R gt (S T) / 2
Payoffs do not have to be symmetrical.
15
PD as a Model for Real-Life Scenarios
16
Iterated Prisoners Dilemma

• Two players
• Prisoners Dilemma played repeatedly
• History of previous interactions remembered by
  each player
• No other outside knowledge

17
Iterated Prisoners Dilemma

• Two-game iteration

No matter what the other player does, a rational,
self-interested player will defect on the second
(last) game.
18
Iterated Prisoners Dilemma

• Two-game iteration

No matter what the other player does, a rational,
self-interested player will defect on the second
(last) game.
Both players know this, so on the first game,
both players will defect, as well.
19
Iterated Prisoners Dilemma

• N-game iteration

No matter what the other player does, a rational,
self-interested player will defect on the second
game.
Both players know this, so on the first game,
both players will defect, as well.
A rational, self-interested player should defect
all N times.
20
Iterated Prisoners Dilemma

• If number of iterations uncertain

?
?
?
?
Best strategy is no longer clear! Unlike, e.g.,
chess, there is no single best strategy it
depends on the strategy of the other player.
21
Outline

• Prisoners Dilemma
• Two Contests
• Some Analysis
• Real-World Scenarios
• Agent-Based Simulation
• Discussion

22
Contest 1

• Call for entries to game theorists
• All entrants told of preliminary experiments
• 15 strategies
• 14 entries 1 RANDOM
• Round-robin tournament against all other players
  and twin
• Each game 200 iterations
• Games run 5 times against each strategy
• Scores averaged over all games

23
And, the winner is

• TIT FOR TAT
• Cooperate on first move, thereafter reciprocate
  opponents previous action
• Shortest program submitted
• By psychologist, Anatol Rapoport

24
Analysis Nice Guys Finish First

• Top 8 strategies never defect first.

25
Analysis To Forgive, Divine

• Top two rules are willing to cooperate even after
  defections, if other player is contrite

DOWNING - Kingmaker - Tries to learn
behavior of other player starts by defecting
twice. - Hurts strategies that are unforgiving.
26
Other Interesting Strategies

• TIT FOR TWO TATS
• Retaliate only if previous two are Ds
• Could have won tournament, if entered
• NICE DOWNING
• Like DOWNING, but start with Cs
• Could have won tournament, if entered
• Variations on TIT FOR TAT
• Did well, but none beat TIT FOR TAT

27
Contest 2

• Same set up as Contest 1, except
• Entries from first-round contestants as well as
  open call in magazine
• 63 strategies
• 62 entries 1 RANDOM
• Each game iterated an uncertain number of
  iterations, with probability 0.00346 of ending

28
And, the winner is

• TIT FOR TAT, again!
• (Again, by Anatol Rapoport)

29
Analysis Contest 1 Lessons Validated

• 14 of top 15 strategies never defect first.
• 14 of bottom 15 strategies were not nice.
• Forgiveness important.

30
Analysis Be Retaliatory

• Some entrees tried to take advantage of nice
  strategies
• TRANQUILIZER cooperate first, if other
  cooperates, too, throw in a few defections.
• TESTER defect first, if other doesnt
  retaliate, cooperate twice, then alternate
  defection and cooperation. If other ever
  defects, do TIT FOR TAT.
• Strategies that were unresponsive to defections
  get taken advantage of.
• Top strategies retaliate quickly.

31
Analysis Sneaki-ness Doesnt Pay

• Entrees that try to take advantage of nice
  strategies, dont gain as much as they lose.
• TRANQUILIZER 27th place in tournament.
• TESTER 46th place in tournament (out of 63).

32
Outline

• Prisoners Dilemma
• Two Contests
• Some Analysis
• Real-World Scenarios
• Agent-Based Simulation
• Discussion

33
Robustness of TIT FOR TAT

• In six variations of Contest 2, TIT FOR TAT took
  first place in five and second place in one.
• In a population simulation with 63 strategies
  (right), TIT FOR TAT emerges as the winner.
• In an genetic algorithm experiment (1987),
  TIT-FOR-TAT-like algorithms prevailed.

34
Stability of TIT FOR TAT

• A population of TIT FOR TAT strategists cannot be
  invaded by a single strategy.
• Nor can a population of ALWAYS DEFECT
  strategists.
• But! A cluster of TIT FOR TATs can invade ALWAYS
  DEFECT, while the converse is not true.

35
Stability of TIT FOR TAT

• A population of TIT FOR TAT strategists cannot be
  invaded by a single strategy.
• Nor can a population of ALWAYS DEFECT
  strategists.
• But! A cluster of TIT FOR TATs can invade ALWAYS
  DEFECT, while the converse is not true.

36
Stability of TIT FOR TAT

• A population of TIT FOR TAT strategists cannot be
  invaded by a single strategy.
• Nor can a population of ALWAYS DEFECT
  strategists.
• But! A cluster of TIT FOR TATs can invade ALWAYS
  DEFECT, while the converse is not true.

Under certain conditions that imply that the
future is sufficiently important for all players.
37
General Lessons

• Dont be envious. (It doesnt matter if others
  win.)
• TIT FOR TAT never scores more than the other
  player.
• Be nice. (Dont defect first.)
• The best way to do well is to cooperate with
  others who are also nice.
• Retaliate swiftly.
• Or, others will take advantage.
• Forgive.
• Feuds are costly. Defections shouldnt prevent
  cooperation later on.
• Dont be too clever.
• Too much cleverness looks RANDOM.

38
Outline

• Prisoners Dilemma
• Two Contests
• Some Analysis
• Real-World Scenarios
• Agent-Based Simulation
• Discussion

39
Trench Warfare

• Common form of battle in World War I
• Armies in deep trenches on either side of battle
  line
• Machine guns and artillery
• Prolonged engagement with same group of enemy
  troops

40
Trench Warfare is an IPD
You
Cooperate
Shoot to Kill
Payoff Matrix
You live and win a medal.
Cooperate
Both live.
Them
You die and they win.
Shoot to Kill
Both die.
For a single round, no matter what the enemy
does, its better to shoot to kill.
But, for an indefinite number of rounds?
41
Trench Warfare

• Cooperation spontaneously evolved
• If the British shelled the Germans, the Germans
  replied, and the damage was equal.
• A British staff officer was astonished to
  observe German soldiers walking about within
  rifle range
• These people did not know there was a war on.
  Both sides believed in live and let live.
• Suddenly a salvo arrived but did no damage.
  Naturally both sides got down and our men started
  swearing at the Germans, when all at once a brave
  German got on to his parapet and shouted out We
  are very sorry about that we hope no one was
  hurt. It is not our fault, it is that damned
  Prussian artillery.

42
Biological Mutualism
43
Fig Tree and Fig Wasp
Wasp
Lay eggs without pollinating
Lay eggs and pollinate
Payoff Matrix
Bear fruit breed good wasps.
No fruit breed bad wasps.
Let fig ripen
Live, and have kids.
Die, but have kids.
Tree
No fruit.
Bear fruit.
Cut off fig
Die, no kids.
Live, no kids.
For a single round, trees should cut off figs,
wasps should lay eggs without pollinating.
But, for an indefinite number of rounds?
44
Fig Tree and Fig Wasp Mutualism
45
Outline

• Prisoners Dilemma
• Two Contests
• Some Analysis
• Real-World Scenarios
• Agent-Based Simulation
• Discussion

46
Agent-Based Modeling

• Simulation as a scientific method
• Simulation allows hypothesis discovery,
  verification, and prediction.

Simulation is particularly valuable for
interactions of many agents and the agents are
expected to adapt.
47
Other Modeled Social Theories

• 1963 Cyert March Behavioral theory of the
  firm
• 1974 Schelling Segregated neighborhoods
• 1980 Axelrod Cooperation
• 2003 Axelrod Ethnocentrism

48
Outline

• Prisoners Dilemma
• Two Contests
• Some Analysis
• Real-World Scenarios
• Agent-Based Simulation
• Discussion

49
Summary

• Iterated Prisoners Dilemma as a model for many
  different types of interaction
• There is no single optimal strategy in an IPD
  game, but TIT FOR TAT is strong, robust, and
  stable.
• In real-world IPD scenarios, TIT-FOR-TAT-like
  strategies naturally evolve, even among
  antagonists and unintelligent players.
• Agent-based modeling is a powerful tool for
  modeling populations in social and biological
  sciences.

50
TIT FOR TAT and Ethics

• Mahabharata (3000 BC)
• One should not behave towards others in a way
  which is disagreeable to oneself. This is the
  essence of morality. All other activities are due
  to selfish desire.
• Hammurabis Code (1750 BC)
• If a man put out the eye of another man, his eye
  shall be put out. If he break another man's
  bone, his bone shall be broken.
• The Golden Rule (30 AD)
• Do unto others as you would have them do unto
  you.
• Kants Categorical Imperative
• Act so that the maxim of action may be capable
  of becoming a universal law.
• Garrett Hardin (The Tragedy of the Commons,
  1968)
• Conscience is self-eliminating.

51
Thank you!