Automatic Construction of Static Evaluation Functions for Computer Game Players

1
Automatic Construction of Static Evaluation
Functions for Computer Game Players

• D1 Miwa Makoto
• Chikayama Taura Lab

2
Outline

• Introduction
• Computer Game Player
• Static Evaluation Functions
• Automatic Construction of Static Evaluation
  Functions
• GLEM
• Static Evaluation Functions from domain theory
• Conclusion Future Work

3
Computer Game Player

• Game
• 2-player, zero-sum, deterministic, and perfect
  information game
• ex. chess, Shogi, Go, etc
• Computer Game Player
• receives a position and returns the next move
• Important Features
• Evaluation Functions
• Game Tree Search ( search strategy)

4
Static Evaluation Functions (SEFs)

• Static ? without search
• Evaluate positions and assign their heuristic
  values
• Probability to win
• Distance to goal

E(p) number of lines open for X
number of lines open for O
E(p) 3 2 1 Evaluation Value
5
Static Evaluation Functions (SEFs)
Evaluation value

• Rank treating positions

WIN
1
Static Evaluation Function
2
1
0
-1
-2
-1
LOSE
6
Game Tree Search
MAX
1
Tic Tac Toe with horizon 2
Static evaluation function
MIN
-1
0
1
Static evaluation fuctions decide the computer
game players behavior
7
Construction of SEFs

• Two requirements
• evaluation accuracy
• speed
• Representation of SEFs
• Combinations of features
• feature
• Number of pieces, Position, etc

trade-off
8
Construction of SEFs

• To write SEFs by hand, developers need ...
• Deep knowledge about the treating game
• To find useful features and essential features
• Much effort to tune
• To avoid oversights
• ex. self-play, play with other players
• What if no experts exist?
• What if the player is stronger than human
  experts?
• Automatic construction of SEFs

9
Automatic Construction of SEFs

• Construction from simple features
• Simple features
• A set of units to represent a position
• ex. Black stone on A1 (Othello)
• 2 kinds of methods
• Direct Method
• Hybrid Method

10
Automatic Construction of SEFs

• Direct Method
• High-level combination of simple features
• Genetic Programming, Neural networks, etc
• Feature
• High expressivity
• Much resources
• Difficult to analyze
• Ex.
• TD-Gammon, The Distributed Chess Project

11
Automatic Construction of SEFs

• Hybrid Method
• Linear combination of high-level features from
  simple features
• Feature (in comparison with direct method)
• Less expressive
• Less resources
• Easy to analyze and optimize constucted SEFs
• ?Fast SEFs
• Ex.
• ZENITH, GLEM

12
GLEM (M. Buro, 1998)

• Generalized Linear Evaluation Model
• Application Othello
• Logistello
• One of the strongest computer Othello players
• won the human champion in 1997

13
Method

• Extraction of atomic features
• Generation of configurations by conjunction of
  features
• Weighting configurations using linear regression

14
Atomic features

• Simple binary features
• ex. Black stone on A1
• Extracted by hand
• ex. Othello
• 264 features - places and their stones
• Can not be expressed by a combination of other
  features.

15
Configurations

• Extracting all conjunctions of atomic features is
  unreasonable
• ex. Othello - 2128 conjunctions
• Extract only frequent conjunction
• (Configurations)
• frequent
• at least N times in the training set

16
Pattern

• Treating the whole position at once costs high
• Extract a part of the position by hand
• Pattern values are pre-computed and preserved in
  a hash table
• ? Fast Evaluation

Patterns in LOGISTELLO
17
Pattern

• Evaluation using hash tables

Patterns in LOGISTELLO
18
Application Othello

• 13 game stages (every 4 discs)
• Sum of 51 pattern values
• 1.4 million positions/sec on Athlon 2000
• 1.5 million weights
• 17 million training positions
• 10x speedup
• compared with the old one which won the human
  champion in 1997

19
SEFs from domain theory(Kaneko et al. 2003)

• Based on ZENITH (Fawcett 1993)
• Application Othello

20
SEFs from domain theory

• Generation of logical features from domain theory
• Extraction of evaluation features from logical
  features and selection
• Weighting evaluation features using linear
  regression

21
Domain Theory

• Rules and goal conditions written by a set of
  Horn Clauses
• ex. Othello

22
Logical Features

• Features generated from domain theory
• Generation Strategy
• Translate domain theory
• Remove expensive features
• Select features by backward elimination method

23
Logical Features

• Translations

24
Evaluation Features

• Evaluation Features
• Boolean value
• Conjunction of facts
• ex. blank(a1) Æ owns(x, a2) Æ owns(o, a3)
• (white can play on square a1)
• fact
• a clause without body
• ex. owns, blank

25
Extraction of Evaluation Features

• Translate logical feature to patterns.
• logical feature
• f(A) - legal_move(A, o).
• unfolding
• unfolded features
• legal_move(a1, o) - blank(a1), owns(x, a2),
  owns(o, a3).
• legal_move(a1, o) - blank(a1), owns(x, b1),
  owns(o, c1). .
• extract conjunction of body
• Evaluation Features
• blank(a1) Æ owns(x, a2) Æ owns(o, a3)
• blank(a1) Æ owns(x, b1) Æ owns(o, c1) .

26
Selection of Evaluation Features

• Frequency
• Reject high- and extreme low-frequency evaluation
  features
• Approximated Forward Selection
• Select high correlation pattern iteratively

27
Fast access to features

• Hasse diagram
• Kernel Extraction

28
Experimental Results

• 11,079 logical features
• 8,592,664 evaluation features without selection
• 800,000 positions for training and 6,000
  positions for testing
• Positions of 55 and 60 discs
• Range of evaluation values -64, 64

29
Accuracy
as well or better than GLEM in accuracy
30
Speed

• Athlon MP 2100
• 30,000 positions/sec (1.4 million in GLEM)

31
Conclusion

• Automatic Construction of Static Evaluation
  Functions
• GLEM
• SEFs from domain theory

32
Future Work

• More sophisticated feature selection
• Automatic extraction of GLEMs pattern
• More expressive combination of simple features