Automated Negotiation Agents

1
Automated Negotiation Agents

• Sarit Kraus
• Dept. of Computer Science
• Bar-Ilan University

2
Negotiation

• A discussion in which interested parties
  exchange information and come to an agreement.
  Davis and Smith, 1977

3
What is an Agent?

• PROPERTY MEANING
• Situated Sense and act in
  dynamic/uncertain
• environments
• Flexible Reactive (responds to changes
  in the environment)
  
• Pro-active (acting
  ahead of time)
• Autonomous Exercises control over its own
  actions
• Goal-oriented Purposeful
• Persistent Continuously running process
• Social Interacts with other
  agents/people
• Learning Adaptive
• Mobile Able to transport itself

4
No Agent is an Island automated agents
negotiate with other automated agents

• Monitoring electricity networks (Jennings)
• Distributed design and engineering (Petrie et
  al.)
• Distributed meeting scheduling (Sen Durfee,
  Tambe)
• Teams of robotic systems acting in hostile
  environments (Balch Arkin, Tambe, Kaminka)
• Electronic commerce (Kraus et al.)
• Collaborative Internet-agents (Etzioni Weld,
  Weiss)
• Collaborative interfaces (Grosz Ortiz, Andre)
• Information agent on the Internet (Klusch, Kraus
  et al.)
• Cooperative transportation scheduling (Fischer)
• Supporting hospital patient scheduling (Decker
  Jin)

5
Agents negotiate with humans

• Training people in negotiations
• Trade agents for the Web
• Elves agents representing people

6
Plan of talk agents negotiate with humans

• Automated agent for bilateral negotiations with
  complete information the fishing dispute
  (collaborators Penina Hoz-Weiss, Jon Wilkenfeld)
• Automated agent for multi-party negotiations the
  Diplomacy game(collaborators Daniel Lehmann and
  Eitan Ephrati)
• On going work learning, incomplete information
  mediation(collaborators Dudi Sarne, Barbara
  Grosz Lin Raz, Michal Halamish)

7
Fishing Dispute

• Negotiators Canada and Spain
• Canadas stock of flatfish decreases over the
  years.
• Spain has fished this same stock of flatfish for
  many years, but outside the Canadian exclusive
  economic zone (EEZ).
• Canada would like Spain to restrict its fishing
  near her EEZ. Spain is dependent on fishing in
  the area outside the EEZ for employment and trade
  purposes.

8
Possible Outcomes

• An agreement on Total Allowable Catch (TAC).
• An agreement on limiting the length of the
  fishing season.
• Canada enforces conservation measures with
  military forces against Spain.
• Spain enforces its right to fish throughout the
  fishery with military force against Canada.
• If the negotiation has not ended prior to the
  deadline, then it terminates with a status quo
  outcome.

9
World State Parameters

• World state parameters are also negotiable and
  affect the utility of players
• Canada subsidizes removal of Spain's ships (0, 5,
  10, 15, 20 ships).
• Spain reduces the amount of pollution caused by
  the fishing fleet (0, 15, 25, 50).
• Canada imposes trade sanctions on Spain.
• Spain imposes trade sanctions on Canada

10
Fishing Dispute
Outcomes
TAC Limit Season Opt Out
Status Quo
World State Parameters
Canada subsidizes Spain reduces Canada
imposes Spain imposes ships
Pollution Trade Sanctions
Trade Sanctions
11
Negotiation Process

• Each of the parties can make requests, threats,
  offers, conditional offers and counteroffers, as
  well as to comment on the negotiation.
• The utility of each ending is affected by the
  period when the negotiation ended.
• Canada loses over time since Spain continues to
  fish while negotiating. Spain gains over time for
  the same reason.
• Spain ? Thule Canada? Ultima

12
Negotiations in the Fishing Dispute
Spain asks that Canada compensate Spain for
Spains restricted fishing practices by replacing
the income of twenty ships.
13
(No Transcript)
14
(No Transcript)
15
(No Transcript)
16
(No Transcript)
17
(No Transcript)
18
(No Transcript)
19
(No Transcript)
20
(No Transcript)
21
Other Negotiations Games

• Team Games (SPIRE) negotiations on coordination
  exchange of information finding solutions is
  complex
• Competitive games when agents can benefit from
  reaching an agreement (also in bilateral games).
• Trade games Monopoly, Traders of Genoa, Kohle,
  Kies Knete,Treasure game
• War games Diplomacy, Risk
• Crisis games Hostage Crisis.
• Semi-cooperative games Color Trail, Majority
  Game

22
Chess

• Programs play chess as well as people
• Programs play chess in a way much different than
  people they mainly search the game tree

23
Search tree for Tic-Tac-Toe
. . .
24
Fishing dispute vs. Chess

• Type of game crisis game vs. war
  game.Coordination game vs. zero sum game
• Number of players 2
• Moves simultaneously negotiations vs.
  sequentially need to reach an agreement.
• Number of pieces to move no pieces vs. one piece
  at a time
• Information Complete information.
• Needed capabilities Negotiation skills vs.
  strategic skills.

25
Playing Techniques

• NEGOTIATIONS
• Game theory techniques formalize the game find
  an equilibrium follow the equilibrium strategy.
• Market techniques. Appropriate for games of many
  players that can exchange similar items.
• Heuristics domain specific advice books
  human like strategies
• Markov Decision Processes.
• Modeling the opponent
• Learning from DB
• Learning from experience

• CHESS
• Heuristic Search

26
The Automated Negotiator Agent (fishing dispute)

• The agent plays the role of one of the countries.
  
• During the negotiation the agent receives
  messages, analyzes them and responds. It also
  initiates a discussion on one or more parameters
  of the agreement.
• It takes actions when needed.

27
Nash Equilibrium

• An action profile is an order set a(a1,,aN) of
  one action for each of the N players in the
  game.
• An action profile a is a Nash Equilibrium (Nash
  53) of a strategic game, if each agent j does
  not have a different action yielding an outcome
  that it prefers to that generated when chooses
  aj, given that every other player i chooses ai.

28
Strategy of Negotiation
Formal strategic negotiation theory The agent
is based on the a bargaining model. By
backward induction the agent builds the
strategy to be reached at each time period
according to the sequential equilibrium (Kraus,
Strategic Negotiation in Multiagent Environments,
MIT Press 2001). When the agent plays against
humans Not Enough

Heuristics
29
Automated agent Using equilibrium strategy when
playing against humans

• Human negotiators do not use equilibrium
  strategies even though game is not complex and
  the automated agent finds equilibrium fast.
• Not surprising Kahneman Tversky showed that
  humans do not use decision theory.
• The agent using the equilibrium did not reach
  beneficial agreements.

30
Heuristics

• Negotiation tactics
• Attributes
• Risk Attitude
• Opting out
• Fine tuning

31
Attributes

• Number of points lower than the equilibrium
  utility value that the agent will agree to.
• The number of fish ton (TAC) the agent will
  increase/decrease in his offer.
• Sending the first message / waiting to receive
  a message.
• Full offer message or not.

32
(No Transcript)
33
Modeling the risk attitude of theopponent

• The agent is always neutral toward risk, but is
  sensitive to the risk level of its human opponent
  and will change its view of the humans utility
  function accordingly.
• Risk attitude influences the agreement an
  opponent is willing to accept.
• The agent begins with the assumption that its
  opponent is risk neutral. It uses a heuristic
  method to decide whether to change the estimation
  of the risk attitude of the opponent.
• When the agent decides that its opponent is
  risk prone, it changes the opponents utility
  function. This leads the agent to a recalculation
  of his strategy.

34
Experiments Results
35
Fishing Dispute Conclusions

• We developed an agent that can play well against
  a human player.
• The agent was tested on students in their third
  year of computer science studies.
• The results of the experiments implied that the
  agent plays well and fair.
• It raised the sum of the utilities in the
  simulation it was involved in.
• The agent played as Spain significantly better
  than a human did, and just as good as a human
  Canada player.

36
Diplomacys Rules

• Each player represents one of seven European
  powers England, Germany, Russia, Turkey,
  Austria-Hungary, Italy and France.

37
Diplomacys Map
38
(No Transcript)
39
Diplomacys Rules (Cont.)

• Winner The power that gains control over the
  majority of the board.
• Beginning 1901 two seasons a year.
• A season consists of a negotiations stage and a
  move stage.
• Moves All players secretly write the orders for
  all of their units simultaneously.
• Negotiations Coalitions and agreements among the
  players reached in the negotiations stage
  significantly affect the course of the game. The
  rules of the game do not bind a player to
  anything she says. Deciding who to trust as
  situations arise is part of the game.

40
Negotiations in Diplomacy
If you will not help me I will attack you
Dont trust Germany
41
Moves in Diplomacy

• Only one unit may be in any space at one time.
• A unit can be ordered to move, support, hold
  (convoy).
• An army or a fleet may support the move of
  another unit of that country or any other
  country in making a move.
• Support can also be given on a defensive basis.
• Opposing units with equal support do not move. An
  advantage of only one support is sufficient to
  win.

42
Moves in Diplomacy
43
The Need for Negotiations in Diplomacy

• Moves require close cooperation between various
  allied powers.
• Incomplete information communications between
  players are done secretly.
• The game is complex 834 possible moves in each
  step of the game (without negotiation moves) .
  Negotiation is used to obtain information about
  the goals of the other players.
• Others negotiate.

44
Diplomacy vs. Chess

• Type of game war games.
• Number of players 7 vs. 2
• Moves simultaneous vs. sequential.
• Number of pieces to move all pieces vs. one
  piece.
• Information uncertainty about messages exchanged
  between other players vs. full information
• Needed capabilities negotiation skills vs.
  strategic skills.

45
Playing Techniques

• NEGOTIATIONS
• Game theory techniques formalize the game find
  an equilibrium follow the equilibrium
  strategy.Impossible in Diplomacy because of
  complexity.
• Market techniques. Appropriate for games of many
  players that can exchange similar items.
• Heuristics domain specific advice books
  human like strategies
• Markov Decision Processes.
• Learning from DB
• Learning from experience

• CHESS
• Heuristic Search

46
Diplomat an Automated Diplomacy player
Previous Agreements
Beliefs on other players
Board Status
Analysis
Others Moves
Analysis Strategies Finder
Analysis Strategies Finder
Moves
Negotiations
Agreements
Detailed plans and their estimated value for
possible coalitions
47
Diplomacy Structure
Secretary
Prime Minister
Front 2
Ministry Of Defense
Front 1
Strategies Finder
Front 3
Foreign Office
Military Headquarters
Intelligence
Analyzer 13
Analyzer 14
Write orders 15
Desk 10
Desk 11
Desk 12
Write orders 15
48
Strategies Finder (SF)

• Front possible enemies and possible allies,
  e.g., Russia and Italy against Austria and
  Germany.
• Diplomats strategy for a given front includes
• A list of orders associated with their purpose.
• The expected average profit from carrying out the
  strategy for each power who participates in the
  strategy and the common expected profit for all
  of the powers.
• A Venice (I) moves to Triests in order to attack
  Triest
• A Vienna (R) supports A Venice to Trieste in
  order to attack Trieste
• Expected outcome Aver 10617 Min 5002 Max
  20862 Russia
  3358 Italy 18117

49
Strategies Finder (SF) (Cont)

• Diplomat identifies possible front based on
  on-going agreements, beliefs about other agents
  and their relations.
• SF finds some strategies for each front using
  domain specific heuristics. The value of each
  strategy is computed by finding strategies for
  the enemies of the front.
• The negotiation is done based on the identified
  strategies.
• Question What is the best strategy?

50
Diplomats negotiation
Exchange information Decide what kind of
agreement to try to achieve. Find common
enemies.
Negotiating about the general purposes of an
agreement spaces on the board to attack, to
defend, to leave or to enter.

• Signing the final
• Agreement Deciding
• if to keep it.

Deciding on the specific movements in order to
achieve the purposes From previous stage
51
Diplomats behavior is not deterministic

• Diplomat has special personality'' traits that
  affect its behavior and may be varied easily from
  game to game.
• Diplomat flips coins'' in the following cases
• To decide whether to pretend to keep an agreement
  or to tell the other partner that it will break
  the agreement (the probability depends on the
  personality traits.)
• To decide whether to give more details about a
  suggestion.
• To decide which opening to use.
• When SF searches for possible strategies. For
  example, to decide which units will participate
  in the attack or defense of a given location and
  to guess which of the enemy's units will
  participate in the battle of that location.

52
Diplomats Evaluation

• Diplomat was evaluated and consistently played
  better than human players.
• It did not play enough games to gain statistical
  results.
• It was hard to evaluate what contributed to its
  success.

53
Conclusions

• It is possible to develop automated negotiators!!

• Is it possible to develop standard methods for
  playing negotiation games (as in Chess)?

• On going work incomplete information Modeling the
  opponents preferences
• Learning to negotiate

54
Learning to negotiate 3-players majority game

• You are one of 3 Players
• You need to divide the rights for a goldmine

Player 1
Player 2
You
55
Simple Game Protocol (cont.)

• Each Game Round one player is selected Randomly
• And he/she gets to make a division proposal

Player 1
Player 2
You
Player 1
Player 2
You
56
Simple Game Protocol (cont.)

• Based on the proposals the players vote
• It takes a majority to make a decision the
  proposer and one other player

Player 1
Player 2
You
57
Simple Game Protocol (cont.)

• Once a majority was reached the game ends each
  player gets his/her share
• Otherwise (no agreement) A new proposer is
  selected and an additional round is being played

Player 2
Player 1
You
58
Simple Game Protocol (cont.)

• However it is not certain that a new round will
  take place!!!
• There is a continuation probability if no
  agreement was reached, there is a possibility
  that the game will suddenly end and all players
  will get zero

No Agreement
P(New Turn)0.9
P(End Game)0.1
59
Agent Design

• Collect and Manage a DB of previous games
• Given a new game find similar situations in DB
• Maximize utility given previous behaviour

60
Color Trail Game

• Co-developer
• Barbara Grosz
• Harvard University

61
(No Transcript)
62
(No Transcript)