Value Function Transfer for General Game Playing

1
Value Function Transfer for General Game Playing

• Bikramjit Banerjee
• Gregory Kuhlmann
• Peter Stone
• The University of Texas at Austin

2
General Game Playing

• http//games.stanford.edu
• Game Description Language (GDL)
• AAAI GGP Challenge - 2005, 2006
• Our goal develop a GGP learner that will compete
  repeatedly with a non-adaptive opponent

3
Transfer Learning in GGP
Source
Target
4
Transfer across different games

• Challenge State and action spaces
• Difficult to draw correspondences
• Differing transition dynamics could violate
  manual correspondences
• First-cut approach
• Define correspondence in terms of transition
  dynamics
• Where do we ground the similarity based on
  transition dynamics?

Rewards
5
Handcrafted features
6
Feature Examples Mark to Win
Connect-3
Tic-tac-toe
X
Xs turn
X
Capture-Go
Blacks turn
7
Examples Block Opponent
Connect-3
Tic-tac-toe
X
Xs turn
X
Capture-Go
Blacks turn
Also matches Mark to win
8
Examples Fail to Block Opponent
Connect-3
Tic-tac-toe
Xs turn
Capture-Go
?
Blacks turn
?
9
Examples Create a Fork
Connect-3
Tic-tac-toe
Fork does not materialize until Os next
(forced) move
Xs turn
X
X
Capture-Go
Blacks turn
10
Value function Transfer
11
Scaling Up Using Symmetry

• Same type of game, different sizes
• Idea Speed up learning by taking advantage of
  board symmetries
• Requires only knowledge of types of symmetry, not
  specific symmetry in particular games
• Requires complete solution to discover and verify
  presence of symmetries
• Transfer Solving large problem completely is too
  expensive
• Discover and verify presence of symmetries on
  smaller task
• Transfer knowledge to larger task

12
Scaling Up Using Symmetry

• Example Minichess
• Reflectional Symmetry

13
Scaling Up Using Symmetry

• Rotational Symmetry

14
Symmetry Transfer

• Completely solve 3x3 Minichess
• Use dynamic programming
• Verify presence of reflectional and rotational
  symmetries
• Time is negligible (lt 1 min)
• Learn 5x5 Minichess
• Table-based Q-learning vs. Random opponent
• For each transition, make backups for the (up to
  8) symmetric transitions
• Verifying symmetries directly on 5x5 would
  require too much memory

15
Results
4x4 Connect-3
3x3 CaptureGo
16
Latest Future work

• Automated feature construction
• Opponent-specific values
• More appropriate than simple lookahead search
  with a fixed heuristic (such as minimax)
• Future
• GDL analysis for correspondences
• Optimization under transition constraints