Study of Evolutionary Game Theory

1
Study of Evolutionary Game Theory
2

• Introduction of game theory (GT)
• Nash equilibrium
• Mixed strategy Nash equilibrium
• Evolutionary game theory (EGT)
• Difference between GT and EGT
• Evolutionary stable strategy
• Replicator dynamics
• Formalization of EGT
• Dynamics in a single population
• Dynamics in multiple populations

3
Introduction to Game Theory (GT)

• The game theory is the model to describe
  decision-making in the game, which involves
  players, each of whom performs his behavior to
  gain payoff according to given conditions
• The game theory answers the question, What is
  the best behavior to gain payoff when the result
  of a players decision is affected by other
  players decision involving in the game?
• Behaviors a player can perform is called
  strategies
• The game is denoted as G (N, X, P)
• N is the number of players
• X Xi where Xi S1, , SM is a set of
  strategies for player i
• M is the number of strategies
• P Pi where Pi is a payoff matrix for player i
• A payoff matrix specifies the amount of gain for
  players when they have decided their strategies

4
Type of Strategy

• There are two types of strategies Pure and Mixed
• Pure strategy
• A player selects one of the strategies
• Mixed strategy
• A player selects the strategies according to the
  probabilistic distribution
• X (x, 1-x) indicates mixed strategy, where a
  player selects S1 with probability x and S2 with
  probability 1-x
• Note For different mixed strategy (which has
  different distribution), we can use Y (y, 1-y),
  which means that a player selects S1 with
  probability y and S2 with probability 1-y

5
Expected payoff

• Expected payoff for the strategy
• Pure strategy
• U1(S1, S1) indicates payoff for player 1 when
  player 1 selects S1 and player 2 selects S1,
  which is aA
• Similarly, U2(S1, S1) aB, U1(S1, S2) bA,
  U2(S1, S2) bA,
• Mixed strategy
• Ui(X, Y) indicates payoff for player i when
  player 1 use mixed strategy X and player 2 use
  mixed strategy Y
• x y aA (1-y) bA (1-x) y cA
  (1-y) dA

Go back
6
Nash Equilibrium

• Nash Equilibrium is a strategy pair that no
  player can increase his payoff by changing his
  strategy when the other players have decided
  their strategies
• By using expected payoff
• S is called as Nash Equilibrium if
• For all i (players) and all m (strategies),
  Ui(S) gt Ui(Si, S-i)
• where S is a strategy pair of players
• Si is a strategy for player 1
• S-i is a strategy for other players
• Note Ui(S) is equal to Ui(Si, S-i)
• There is no better strategy to increase payoff
  when the strategies for the other players are
  fixed

7
Example of Nash Equilibrium

• Chicken race game
• 2 players game
• A player drives a car toward a wall
• If a player stops a car closer to the wall, then
  he wins
• Strategy
• S1 keep running until the other player stops
• S2 stop driving a car before the other player
• Player 1 plays
• Player 2 plays
• Payoff matrix shows scores a player can gain

8
Example of Nash Equilibrium

• Now, player 1 wants to compute x, which maximizes
  U1
• We can denote such x as x(y)
• Find x (between 0 and 1)
• which satisfy dU1/dx 0 ? 3/5 y 0 ? y
  3/5
• Similarly, player 2 wants to compute y(x),
  maximizes U2

Nash Equilibria
9
Evolutionary Game Theory (EGT)

• Evolutionary game theory is a model to describe
  the dynamics of strategy change in the repeated
  game
• In EGT, we analyze how a population evolves over
  time
• EGT formalizes the evolution of population by
  using the concept of GT
• A population indicates a set of multiple players
• Each player has its own strategy
• A set of players who play the same strategy is
  called a group
• EGT assumes that the strategy, which a player
  performs, is pre-programmed before the game
• E.g., Player1 plays Sa when timelt10 and Sb at
  timegt10
• Player1 plays a different strategy of
  Player2 at time t-1
• Player plays Sa with probability p and Sb
  with 1-p
• At each play during a game, a player plays
  against another player randomly drawn in a
  population
• The opponent may be from the same group or
  different group
• The winner will be replicated, and the loser will
  be dead (or, we can say the loser change its
  strategy to that of the winner)
• Generally, it is assume that the replication rate
  is proportional to payoffs from the game

10
Differences between GT and EGT

• GT
• Focus on a single game play of a player vs. a
  player
• Observe how a player behaves (micro)
• Evaluate the best rational strategy for a player
  in the game
• Interested in the properties of stable state
• e.g., Nash Equilibria

• EGT
• Focus on population, consisting of multiple
  players, and multiple plays of the repeated game
  among the players
• Observe how population behave (macro)
• Evaluate how the group (a set of players with the
  same strategy) shares the population over time
• Interested in the dynamics of state change of
  population
• e.g., Replicator dynamics

11
General questions in GT and EGT
A player1 (from group A) plays against a player2
(from group B) What is the reasonable strategy
for player 1 to maximize his gain? How about the
strategy of the player 2?
GT
A population consists of three types of players A
player plays against randomly selected player in
a population How a population share by groups
changes over time? Does it converge or not? Which
initial conditions conclude the equilibrium of
all groups?
EGT
1
Ratio of Group A
Ratio of Group B
0.5
Ratio of Group C
0.25
t
12
Key Concepts in EGT

• Two key concepts in EGT
• Evolutionary stable strategy (ESS) and Replicator
  dynamics (RD)
• ESS is a strategy (group) which cannot be invaded
  by any alternative strategy (group) in a
  population over time
• This is the refinement of Nash equilibrium for
  EGT
• Replicator dynamics specifies how population
  state changes over time
• Population state indicates the population shares
  by different groups
• A population state represents population shares
  by groups
• A population state at time T is defined by the
  vector of the size of groups N1(T), N2(T), ,
  Nk(T)

13
Evolutionary Stable Strategy

• An evolutionary stable strategy (ESS) is a
  strategy which cannot be invaded by any
  alternative strategy over time
• Assume there are two groups of players in a
  population
• If the payoff of a group with alternative
  strategy is less than that of another group with
  a particular strategy, then the alternative
  strategy cannot be propagated into a population
  over time
• Such the particular strategy is called ESS

14
Definition of ESS
y
1-y
x

• Lets formalize ESS in the case that
• A group A with strategy appears in a
  population together with a group B with strategy
• The ratio of group A and group B is (1e, e)
• Expected payoff for players of a group A
• Expected payoff for players of a group B
• Strategy is evolutionary stable if

1-x
15
Definition of ESS
y
1-y
x

• Lets formalize ESS in the case that
• A group A with strategy appears in a
  population together with a group B with strategy
• The ratio of group A and group B is (1e, e)
• When e? 0
• When ,
• ?
• ?
• therefore,

1-x
(1)
(2)
16
Replicator Dynamics (RD)

• Replicator dynamics is a model to analyze the
  dynamics of population in a evolutionary game
  mathematically
• A way to formalize how population evolve over
  time
• i.e. how population state changes over time
• Replicator dynamics are represented as a set of
  differential equations
• There are two types of Replicator dynamics in
  terms of interactions between individuals
• Interactions among individuals in a same group
• Interactions between individuals in different
  groups

17
RD in a single group

• Again, a single group RD is a model to analyze
  how the population state (i.e., strategy
  distribution) change over time
• Game involves N players
• Two Strategies
• At time t (generation)
• The total number of players at time t
• where Ni(t) of players with strategy si
• The ratio of group i to the population size
• The population state (i.e. strategy distribution)

18
RD in a single group

• Rule of the game
• of players at time t is proportional to the
  payoff gain at the previous generation at time
  t-1
• At time t1, of players with strategy ei (e1
  S1, e2 S2)
• From

19
RD in a single group
On the other hand
Therefore
20
Example of a single group RD

• Consider how Beetle swarm evolves
• There are two types (classes) of beetle Big and
  Small
• Which Beetle will survive, or stable ?
• Beetles are contesting a resource with a value 12
• Strategy
• S1 having Big body
• S2 having Small body
• Rule
• Beetles with the same strategy share the resource
  (6)
• But, Big v.s. Big ? they get 5 (decline in value)
• When one is S1 and the other is S2
• The Big wins, but get 4 (because of fight)
• Let x be the ratio of Big beetles to total of
  Beetle
• By analyzing RD , Lets see which Beetle
  will survive

21
Example of a single group RD

• Expected payoff
• Replicator equation

22
Example of a single group RD

• Three equilibria point at
• Stability analysis
• x0, (eq 3) -2 lt0 (stable)
• x1, (eq 3) -5 lt0 (stable)
• x2/7, (eq 3) 10/7 gt0 (unstable)
• If Beetle swarm contains Big beetlesmore than
  2/7, then the Big beetles willoccupy swarm over
  time

(3)
23
(No Transcript)
24
Example of ESS

• Hawk-Dove game
• Animals are contesting a resource with value of v
• Strategy
• S1 aggressive strategy (hawk)
• Escalate and continue to fight until injured or
  until the opponent retreats
• S2 accommodating strategy (dove)
• Display and retreat immediately if the opponent
  escalates
• Assumption under the conflict
• When hawk meets, winning probability is 0.5
• When the conflict occurs, the cost c has to be
  paid
• When dove meets, retreating probability is 0.5
• Payoff matrix
• When they fight, they get (v-c)/2
• When both avoid fighting, both get v/2
• When one tries to fight but the other not, the
  winner gets v

25
Example of ESS
Calculation of the expected payoff
26
Example of ESS
Assume a group A with strategy S (v/c, 1-(v/c))
First condition
Expected payoff when an animal meets the animal
with different strategy
Second condition
SEES (v/c, 1-(v/c)) is evolutionary stable
strategy
27
(No Transcript)
28
RD in a single group

• Assume that at each time t (generation), of
  individuals constantly increase with natural
  birth rate B and decrease with natural death rate
  D
• ? Gain by natural birth, and dis loss by natural
  death
• At time t1, of individuals with strategy ei
• From and

29
RD in a single group
30
(No Transcript)
31
(No Transcript)
32
Example of EGT

• Two players two strategies game
• A player has two cards A and B
• Players open one of the cards at the same time
• They gain money written in the matrix according
  to the result of opened cards
• In game theory (consider one time game)
• A player tries to maximize his own gain (selfish
  player)
• If both players select card B, then they both
  gain benefit
• But, there is a risk to be betrayed (i.e. loosing
  money)
• The state (A, A) is Nash equilibrium
• It ends up no gaining
• In evolutionary game theory (consider multiple
  games)
• How can players maximize their benefit?
• They may start to think about cooperation
  (altruistic player)
• They can gain 100 every game if they cooperate
• Is that really good strategy for both players?

33
Example of EGT
q
1-q
p
1-p

• Assumption
• Player 1 selects a card A with probability p
• Player 2 selects a card B with probability q
• Note this assumption is equivalent to Player 1
  selects a card A pN times during N game plays
• Expected value of gain for each strategy
• For player 1, E(1SA) q lu1 (1-q) ru1 4
  (1-q)
• E(1SB) q lb1 (1-q) rb1 1 2q
• For player 2, E(2SA) p lu2 (1-p) lb2 4
  (1-p)
• E(2SB) p ru2 (1-p) rb2 1 2p
• Expected value of gain for each player
• For player 1, E(P1) p E(1SA) (1-p) E(1SB)
• For player 2, E(P2) q E(1SA) (1-q) E(1SB)

34
Example of EGT
q
1-q
p
1-p

• Expected value of gain for each player
• For player 1, E(P1) p E(1SA) (1-p) E(1SB)
• For player 2, E(P2) q E(1SA) (1-q) E(1SB)
• A player tries to maximize his gain to control p
  or q
• For player 1, find p that maximize E(P1)
• -gt d E(P1) / dp 0
• 0 E(1SA) E(1SB) q (lu1 lb1)
  (1-q) (ru1 rb1)
• 0 q 3 (1 q) 1 4q ? q ¼
• For player 2, find p that maximize E(P2)
• -gt d E(P2) / dq 0
• 0 E(2SA) E(2SB) p (lu2 lb2)
  (1-p) (ru2 rb2)
• 0 p 3 (1 p) 1 4p ? p ¼

35
Mixed strategy Nash equilibrium
q
1-q
p

• When players select card A with probability¼
  (card B with probability ¾), they both canget
  more gain than the strategy concludedby Nash
  equilibrium
• E(P1) E(P2) ¼ (¾ 4) ¾ (- ¼ ¾ 1)
  9/8
• This mixed strategy is better than cooperation
  strategy
• Mixed strategy Nash equilibrium is the
  equilibrium where a players mixed strategy
  yields the player as high expected payoff as any
  other mixed strategy of the player, given the
  mixed strategies of the other players.
• We can call state(strategy(p, q)(¼, ¼)) as mixed
  strategy Nash equilibrium
• Note there

1-p
36
(No Transcript)
37
Replicator Dynamics

• Mixed strategy x (x1, x2, , xn) is interpreted
  as population state, where each element xi is the
  share of individuals who play strategy i in a
  population
• Payoff represents fitness, the degree of
  survivability to the next play (e.g., next
  generation) in the game
• Replicators are strategies, which can be copied
  from parent to child

38
The Replicator Dynamics Model

• There are n pure strategies in the whole
  population. Individuals can only be programmed to
  play pure strategies.
• A mixed strategy x (x1,x2,,xn) is interpreted
  as a population state, each component xi is the
  population share of individuals who play pure
  strategy i.
• Payoff represents fitness (the number of
  offsprings), and each offspring inherits its
  single parents strategy.
• Replicators are pure strategies, which can be
  copied without error from parent to child.
• Reproduction takes place continuously over time.

39
Payoff in replicator dynamics
ei pure strategy i. xi population share of
pure strategy ei. (equivalent to a component of
mixed strategy x.) u(ei, x) expected payoff
(fitness) of strategy ei in a random match with a
random player when the population is in state
x(x1, , xn). (equivalent to payoff of strategy
ei against mixed strategy x.) u(x, x)
population expected payoff (fitness) is the
expected payoff to an individual drawn at random
from the population. (equivalent to the payoff of
mixed strategy x against mixed strategy x.)
40
Replicator Dynamics Model

• xi population share of pure strategy i.
• ei pure strategy i.
• u(ei, x) expected payoff (fitness) of strategy
  i at a random match when the population is in
  state x(x1, , xn).
• u(x, x) population expected payoff (fitness) is
  the expected payoff to an individual drawn at
  random from the population

41
Example replicator dynamics for a doubly
symmetric game
Fitness of strategy 1
A
Mixed strategy
Average population fitness
42
Stability Concepts in Nonlinear System
Nonlinear system with state variable
x(t)(x1(t),,xn(t))

• Lyapunov Stability a state x is stable or
  Lyapunov stable if no small perturbation of the
  state induces a movement away from x(x1,,xn).
• no push away from x
• Asymptotical Stability a state x is
  asymptotical stable if it is Lyapunov stable
  and all sufficiently small perturbations of the
  state induce a movement back toward x.

43
ESS and Replicator Dynamics

• ESS x asymptotical stability of population
  state x.
• proved by choosing a Lyapunov function, which is
  a relative-entropy function in this case.
• converse may be not true.

44
Example Rock-Scissors-Paper (RSP) Game

• Unique NE strategy x(1/3, 1/3, 1/3) is NOT ESS !
• How about the Replicator Dynamics?

45
Example Rock-Scissors-Paper (RSP) Game
A is the payoff matrix of one player
Replicator Dynamics
46
Rock-Scissors-Paper (RSP) Game
NE strategy x(1/3, 1/3, 1/3), but not ESS NE
strategy is Lyapunov stable, but not
asymptotically stable
Replicator Dynamics
Start from any initial state, the system moves
forever along a closed curve!
Paper (x3)
Rock (x1)
Scissors (x2)
47
Evolutionary Game Theory and Computer Networking

• An evolutionary game perspective to ALOHA with
  power control.
• E. Altman, N. Bonneau, M. Debbah, and G. Caire.
  Proceedings of the 19th International Teletraffic
  Congress, 2005.
• An evolutionary game-theoretic approach to
  congestion control.
• D.S. Menasch a, D.R. Figueiredob, E. de Souza e
  Silvaa. Performance Evaluation 62 (2005)

48
Example of Replicator Equation

• Zebra and Lions
• Population of zebra and lion x and y
• Birth rate dx / dt Z x and dy / dt L y
• Z b a x R y
• b Natural birth ratio
• a Environmental factor
• R ratio of hosyoku
• L d S x
• d natural birth ratio
• S factor proportional to population of Zebra
• dx/dt bx ax2 R x y
• dy/dt d y S x y

49
(No Transcript)