Key Points

1
Key Points

• Karl Lieberherr

2
Challengeold high-level description

• Price
• Set of problems

3
Challengenew high-level description

• Challenge(X)
• Price
• A constructive belief involving
• algorithms in domain X and
• a set of problems in X
• A constructive belief (X)
• A claim about algorithms in domain X
• A constructive belief defines a protocol to
  demonstrate that the claim is not supported
• Bryan A challenge represents a constructive
  belief

4
Algorithms in a domain

• The algorithms are about
• Solve problems in the domain
• Hide secrets in problems of the domain
• Find hard problems

5
Market Design

• The market is about constructive beliefs and
  constructive support for those beliefs.
• Constructive means that the objects that are
  claimed to exist must be constructed by an
  algorithm (constructive mathematics). Reference
  to Speckers Theorem.
• Constructive support involves an interactive
  protocol (reference to interactive proofs).
• If the constructive support of a belief fails,
  the belief is called unsupported.

6
Constructive support

• Demonstrations for unsupported claims must be
  efficiently checkable.

7
Unsupported beliefs and counterexamples

• An unsupported belief is not a counter example.
  When a belief is unsupported, it might still be a
  true fact. Alice might have failed to find the
  right f in F.
• When Alice is perfect, she will always be able to
  support her belief if it is a true fact.
• When Bob is perfect, he will always be able to
  show that the belief is unsupported if it is a
  wrong fact.

8
Why beliefs?

• Beliefs are easier to work with than theorems.

9
Project Summary to Slides

• Traditional benchmarks set standards for the
  evaluation of software innovation in many
  disciplines within computer science in both
  industry and academia. Although necessary,
  developing benchmarks is challenging and once
  developed, benchmarks are inflexible,
  representing only a sample of the application
  space, and are difficult to maintain as
  applications typically evolve quickly. The
  objective of this proposal is to address
  benchmark problems by creating a dynamic strategy
  based on \it artificial markets of
  computational challenges. Dynamic market
  evaluation fosters more innovation through
  effective feedback, aiding in the discovery of
  better algorithms and the production of high
  quality software --- ultimately increasing the
  overall algorithmic knowledge of a given domain.

10

• Given a concise description of a computational
  problem domain, X, it will be possible to
  create an \it artificial market around X,
  \marketX. Software responsible for the creation
  of the market will generate a starting \it
  agent and a trust-worthy \it market
  administrator. Teams and individuals competing
  within the problem domain will improve their
  agent by participating in regular, administrator
  run competitions. Throughout the course of a
  competition, each agent offers and solves
  computational \it challenges, and is rewarded
  for both offering hard problems in X, and
  solving other agent's challenges effectively.
  Agents are ranked based on their performance
  within competitions and the corresponding teams
  can use the log of a competition to discover hard
  problems for their agent to offer, and gather
  ideas to improve their agent's algorithms.

11
Additional

• Fundamental to the \market(X) design is the
  concept of a constructive belief and the concept
  of supported or unsupported constructive belief.
  We build on ideas from interactive proofs and
  program checking (a subarea of interactive
  proofs). Each challenge represents a constructive
  belief. Demonstrating that a constructive belief
  is unsupported does not imply that the belief is
  wrong. It only implies that the current holder of
  the belief cannot constructively support the
  belief. Another holder might be able to support
  it.

12
Additional 2

• A constructive belief is specific to a holder. It
  is like I, Alice, believe, theorem z holds and
  then Alice must interact with a verifier (Bob,
  who does not believe theorem z holds) to support
  her belief.

13
Additional 3

• Outcomes
• Z holds
• Supported Alice understands Z and she makes it
  impossible for Bob to attack the belief.
• Unsupported Alice does not understand Z and she
  makes it possible for Bob to attack her belief.
  Alice is buggy.
• Z does not hold
• Supported Bob does not understand why Z does not
  hold and cannot find an attack. Bob is buggy.
• Unsupported Bob understands why Z does not hold
  and he can find an attack.

14
Additional 4

• True statements may be shown to not be supported.
  The holder of the true statement (Alice) is
  buggy. The holder is likely to offer the
  challenge at a price that is too low.
• Wrong statements may be shown to be supported.
  The challenger of the wrong statement (Bob) is
  buggy. The challenger is likely to accept the
  challenge at a price that is too high.

15
Supporting is weaker than proving

• Supporting a belief against a perfect challenger
  is like proving it.
• To show that a belief is unsupported against a
  perfect holder is like finding a counterexample.
• A perfect holder or challenger has unlimited
  resources and the perfect algorithm.
• In reality, holders and challengers are not
  perfect.
• They will become better by playing the game for a
  while.

16
Witnesses

• Supporting a belief is like finding a positive
  witness. Replace for all by an object and show
  that prop holds.
• To show that a belief is unsupported is like
  finding a negative witness.

17
Algorithms
18
From teaching to Research

• Undergraduates need to deal with programming
  errors and conceptual problems.
• Researchers need to deal with finding
  state-of-the-art techniques.

19
Beliefs

• Mathematical
• Must be of the form EAEA
• Secret (time t)
• You cannot find my secret
• You cannot approximate my secret
• Duel
• I solve your problems better than you solve my
  problems

20
Beliefs

• With beliefs we can express claims and show that
  they are supported or unsupported even if we
  dont have enough information or knowledge to
  formally prove the claim.
• A belief, when challenged, is either supported
  or unsupported. When it is supported against a
  strong challenger the belief is more likely to
  hold. When it is unsupported against a strong
  holder, the belief is more likely to be wrong.

21

• \bf Intellectual Merit The main objectives of
  this project are (1) to study the design of
  artificial markets of computational challenges
  with specific objectives (2) show that
  artificial markets of computational challenges
  are a useful dynamic benchmark tool for software
  and (3) show that they drive innovation, helping
  to both find better algorithms and improve
  understanding of their development. The overall
  goal is to help program officers, managers, and
  professors with the evaluation of research
  prototypes, competing software packages, and
  student programs, respectively, and to help
  designers improve their algorithms using feedback
  from the artificial market. The design of
  particular artificial markets will drive
  innovation in specific objective areas,
  ultimately providing targeted feedback. The
  development of complex agents, administrators,
  and game generators will be supported by a number
  of generic programming tools, libraries, and
  design patterns that push the state-of-the-art.
  The resulting languages and methodologies are
  independent of the problem domain and contribute
  to programmer productivity and software quality.

22

• \bf Broader Impacts The impact of this project
  will be through advancing discovery and
  understanding of how to solve computational
  problems and providing an exciting platform for
  interdisciplinary teaching and learning. A
  generic artificial market supports the
  improvement of algorithmic techniques for a wide
  variety of problems through an experiential
  teaching curriculum. Students become engrossed in
  learning the art of software development,
  algorithmic analysis, and even modeling, through
  the refinement of their agent and frequent
  competitions. Distributed competitions will be
  run over the Internet allowing agents from
  different schools, companies, and even countries
  to compete within a common domain. The resulting
  algorithms, improved software, and useful tools,
  libraries and techniques will benefit software
  developers, scientific communities, and the
  general public as they make their way into
  various products and services.

23
Compare belief/theorem

• Belief B

• Theorem T

• Supporting B is short.
• Supporting !B is short.

• Proof of T is long.
• Proof for !T is short.

24
Definition Proof System

• Proof system for a language L
• P-proof x for y P(x,y) for y in L.
• Soundness if y has a P-proof, y is in L.
• Completeness If y in L, y has a P-proof.
• Effectiveness P(x,y) is in P.
• http//citeseer.ist.psu.edu/cache/papers/cs/33367/
  httpzSzzSzwww.math.cas.czzSzkrajicekzSzdehn.pdf/
  dehn-function-and-length.pdf

25
Terminology

• Adversary (instead of verifier)
• Supporter (instead of prover)
• BSS questioner / supporter
• Adversary Disprover Questioner Challenger
• AM agents challenges
• Challenging to disprove
• Challenging beliefs that the challenge represents
• Questioning challenge

26
Assumptions

• All beliefs are constructive beliefs. The
  challenging of a belief is constructive.
• The beliefs are about properties of algorithms
  but could be about other topics.

27
Definition Constructive Belief Support System

• For statements y of the form (EA) in first-order
  predicate calculus.
• Derive interactive protocol from y. x is the
  outcome of the protocol for y.
• P-support x for y P(x,y) . x is the support for
  y.
• P-unsupport x for y Pun(x,y). x shows that y
  has no support.
• Soundness related
• Weak soundness if y has P-support with
  challenger of strength q, y is more likely to
  hold with confidence q.
• Completeness related
• Contra-Weak completeness If y has P-unsupport
  with supporter of strength q, y is more likely to
  be false with confidence q.
• Completeness if y holds, y has P-support x,
  P(x,y), for some x.
• Effectiveness P(x,y) and Pun(x,y) are in P.

28
Properties Constructive Belief Support System

• For statements y of the form (EA) in first-order
  predicate calculus.
• Derive interactive protocol from y. x is the
  outcome of the protocol for y.
• P-support x for y P(x,y) . x is the support for
  y.
• P-unsupport x for y Pun(x,y). x shows that y
  has no support.
• For all y there is an x so that either P(x,y) or
  Pun(x,y) and such an x can be found efficiently
  by following the protocol.
• Soundness related
• Weak soundness if y has P-support with
  challenger of strength q, y is more likely to
  hold with confidence q.
• Perfect soundness If y has P-support with
  perfect challenger, y holds.
• Completeness related
• Contra-Weak completeness If y has P-unsupport
  with supporter of strength q, y is more likely to
  be false with confidence q.
• Contra-Perfect completeness If y has P-unsupport
  with perfect supporter, y is false.
• Completeness if y holds, y has P-support x,
  P(x,y), for some x.
• Effectiveness P(x,y) and Pun(x,y) are in P.

Supporting a belief against a perfect challenger
is like proving it.
29
Why?

• To show that a belief is unsupported against a
  perfect supporter is like finding a
  counterexample.
• Because the perfect supporter will find the
  hardest object. If the challenger can find an
  object that negates the predicate, we must have a
  counterexample.
• Supporting a belief against a perfect challenger
  is like proving it.
• Because the perfect challenger will find the best
  object and if this object does not negate the
  predicate, the supporter must have found the
  object that the statement claims to exist.

30
Definition ConstructiveBelief Support System,
secret form

• For statements y of the secret form You cannot
  recover my secret solution for objects satisfying
  predicate pred.
• Derive interactive protocol from y. x is the
  outcome of the protocol for y.
• P-support x for y P(x,y) . x is the support for
  y.
• P-unsupport x for y Pun(x,y). x shows that y
  has no support.
• Soundness related
• Weak soundness if y has P-support with
  challenger of strength q, y is more likely to
  hold with confidence q.
• Completeness related
• Contra-Weak completeness If y has P-unsupport
  with supporter of strength q, y is more likely to
  be false with confidence q.
• Completeness if y holds, y has P-support x,
  P(x,y), for some x.
• Effectiveness P(x,y) and Pun(x,y) are in P.

31
Definition Constructive Belief Support System,
duel form

• For statements y of the duel form If you give me
  k problems (satisfying pred) and I give you k
  problems satisfying pred, I will solve your
  problems better than you solve mine.
• Derive interactive protocol from y. x is the
  outcome of the protocol for y.
• P-support x for y P(x,y) . x is the support for
  y.
• P-unsupport x for y Pun(x,y). x shows that y
  has no support.
• Soundness related
• Weak soundness if y has P-support with
  challenger of strength q, y is more likely to
  hold with confidence q.
• Completeness related
• Contra-Weak completeness If y has P-unsupport
  with supporter of strength q, y is more likely to
  be false with confidence q.
• Completeness if y holds, y has P-support x,
  P(x,y), for some x.
• Effectiveness P(x,y) and Pun(x,y) are in P.

32
Terminology

• Challenger supports y.
• Challenger cannot find an object negating claim
  y.
• Challenger unsupports y.
• Challenger finds an object negating claim y.
• Finding unsupport. Find an object which negates
  the claim.

33
Properties Constructive Belief Support System,
testing perspective

• For statements y of the form (EA) in first-order
  predicate calculus.
• Derive interactive protocol from y. x is the
  outcome of the protocol for y.
• P-support x for y P(x,y) . x is the support for
  y.
• P-unsupport x for y Pun(x,y). x shows that y
  has no support.
• For all y there is an x so that either P(x,y) or
  Pun(x,y) and such an x can be found efficiently
  by following the protocol.
• Soundness related
• Weak soundness if y has P-support with
  challenger of strength q, y is more likely to
  hold with confidence q.
• Perfect soundness If y has P-support with
  perfect challenger, y holds.
• Completeness related
• Contra-Weak completeness If y has P-unsupport
  with supporter of strength q, y is more likely to
  be false with confidence q.
• Contra-Perfect completeness If y has P-unsupport
  with perfect supporter, y is false.
• Completeness if y holds, y has P-support x,
  P(x,y), for some x.
• Testing related
• If y holds, and challenger unsupports y,
  supporter is buggy.
• If y does not hold, and challenger supports y,
  challenger is buggy.
• Effectiveness P(x,y) and Pun(x,y) are in P.

Supporting a belief against a perfect challenger
is like proving it.
34
Advantages of Belief Support System

• Given a claim, we can quickly find support or
  unsupport.
• The Belief Support system feeds the game
• Beliefs which cannot be supported by the
  supporter, cost money.
• Beliefs must be made and put on the market.
• Beliefs which are challenged must be challenged
  successfully, otherwise they cost money.
• The stronger the agents become, the better the
  beliefs become unchallenged beliefs become
  truths and challenged beliefs turn into
  counterexamples.
• The game acts as a truth or belief maintenance
  system for constructive beliefs about algorithms.
  Bad beliefs are eliminated because agents having
  bad beliefs lose money and drop out from the game.

35
Challenge / Belief

• Mathematical challenge
• Constructive mathematical belief
• Implies protocol for support/unsupport
• Price
• Etc. for other challenges

36
About beliefs

• http//citeseerx.ist.psu.edu/viewdoc/summary?doi1
  0.1.1.40.2822
• http//en.wikipedia.org/wiki/Truth_maintenance_sys
  tem

37
Focus of proposal

• algorithmic competitions for dynamic benchmarks
• The dynamic benchmarks are created by
  mathematical beliefs, secret beliefs and duel
  beliefs.
• Mathematical beliefs, optimization problems, EA
• Specific example CSP
• Secret beliefs, optimization problems
• Reproduce
• Approximate within bound
• Specific example CSP
• Duel beliefs, optimization problems
• Specific example CSP

38
Artificial Market and Constructive Belief Support

• A constructive belief support system is the
  foundation for the artificial market.
• Money is made when a belief is supported
  encourages agents to offer only beliefs they can
  support. Pushes them towards offering correct
  beliefs.
• Money is made when a belief is successfully
  challenged encourages agents to only challenge
  beliefs they can successfully challenge. Pushes
  them towards challenging only wrong beliefs.
• Efficiency (administrator must be efficient)
• It must be easy to express successful support and
  it must be easy to check successful support.
• It must be easy to express successful challenge
  and must be easy to check successful support.

39
Difficult Problems for Dynamic Benchmarks

• The beliefs are about what an algorithm can do.
• To support the belief means to have a good
  algorithm.
• To challenge the belief means to give the
  algorithm difficult problems.

40
Explaining SCG(X)
No prices No game rules Supported
and Unsupported Beliefs
ConstructiveBeliefSupportSystem(X)
ArtificialMarket(X)
Add market rules
Beliefs about optimization problems
ArtificialMarket optimization (X)
SCG math (X)
SCG secret (X)
SCG duel (X)
41
Market Design

• Belief Support Systems
• Artificial Markets

42

• Bryan
• ? \not\exists x . !sup(x,y) gt valid(y)
• ? \forall x . sup(x,y) gt valid(y)
• \exists x . !sup(x,y) gt !valid(y) if
  confidence 1.0
• Karl
• \exists x.sup(x,y) there is support for y

43
For interesting y

• Sup(x,y) x supports claim y
• Even if y is true \exists x !sup(x,y)
• Even if y is false \exists x sup(x,y)

44

• There exists a y so that Even if y is true
  \exists x !sup(x,y)
• There exists a y so that Even if y is false
  \exists x sup(x,y)

45
New terminology

• Constructive Belief Support System
• Belief
• supporter
• questioner adversary
• Artificial Market
• Challenge
• supporter challenger
• questioner adversary acceptor

46
Research on Constructive Belief Support Systems

• Belief support systems for other claims than
  mathematical (EA), secret and duel.
• Formally defining confidence.
• Deriving the protocol from the claim form.
• Further formal properties of belief support
  systems.

47
Research on Constructive Belief Support Systems
and Artificial Markets

• Different ways of embedding belief support
  systems into artificial markets.

48
Research on Applicability Range of SCGWhat is
needed for winning?

• Programming Skills

• Algorithmic Knowledge

• Currently high, although a baby agent with basic
  communication skills is generated. Still solid
  programming skills needed.
• Future would like to make it easier to transfer
  knowledge to agent.

• Good knowledge about successfully supporting and
  questioning beliefs.
• For appropriate beliefs turns into good
  knowledge about how pose hard problems and how to
  solve hard problems

49
Special case Belief form EA q

• Beliefs

• Problems

• Alice supports belief
• Bob questions belief

• Alice poses hard problem
• Bob solves hard problem

50
Interactive Proof SystemConstructive Belief
Support System

• Interactive Proof System

• Constructive Belief Support

• Prover, all-powerful
• Verifier, limited resources
• True statement
• False statement
• Completeness
• If true, verifier will be convinced
• Soundness
• If false, verifier will not be convinced (small
  prob.)

• Supporter, limited resources
• Questioner, limited resources
• Supported statement
• Unsupported statement
• Completeness
• If true, questioning will not succeed
• Soundness
• If false, questioner will succeed

51
How to influence the hearts ofcomputer science
students

• When the students see their agent suffer when
  playing with the agents of their peers, they want
  to help their agent. While doing so they learn a
  lot of computer science skills while transferring
  their knowledge of the subject domain into their
  agent.

52
Relevant reference

• http//citeseerx.ist.psu.edu/viewdoc/summary?doi1
  0.1.1.53.4593
• Reasoning about Beliefs and Actions under
  Computational Resource Constraints (1987) by Eric
  J. Horvitz
• In Proceedings of the 1987 Workshop on
  Uncertainty in Artificial Intelligence

53
Questions based on Reasoning about Beliefs and
Actions under Computational Resource Constraints

• What is the comprehensive value of computation?
• Does more computation help me to create a belief
  that I can support?
• Does more computation help me to successfully
  question this belief.
• Cost/benefit tradeoffs.

54
Value of partial solution

• The parameter p in belief B(p) must be above
  p_low and below p_high.
• There is one right answer.
• How can we refine this result with increasing
  amounts of computation?

55
Dempster-Shafer Theory ???

• Epistemic or subjective uncertainty (lack of
  knowledge about a system)

56
Implications between beliefs

• If x supports belief y1 then x supports belief y2
• E.g. Ef AJ lt 0.6 implies Ef AJ lt 0.7
• A claim that is never successfully questioned is
  true?

57
Measure confidenceAccumulate evidence

• What can we learn from a long game history about
  the confidence in agents and statements?
• How can we measure the confidence of an agent?
  Confidence in agents use something similar to
  the FIDE rating approach for Chess.

58
Difficult Problems for Dynamic Benchmarks

• The beliefs are about what an algorithm can do.
• To support the belief means to have a good
  algorithm.
• To challenge the belief means to give the
  algorithm difficult problems.

THIS IS WRONG SEE NEXT SLIDE
59
Difficult Problems for Dynamic Benchmarks

• The beliefs are about what an algorithm can do.
  Belief is of the form Exists f in F For all J in
  fs(f) q(f,J) lt c
• To support the belief means to find hard problems
  f.
• To challenge the belief means to efficiently
  solve hard problems f.

60
extra
61
Relation numbers

• AND 128
• NAND 127
• OR 254
• NOR 1

62
Alternative names

• Hypothesis management system
• Hypothesis support system

63
Artificial Markets of Computational Challenges

• Constructive beliefs supported or unsupported
• Challenge (belief(p), price p) I believe I can
  support belief belief(p) and you cannot
  successfully question belief(p). If you can, you
  win 2p, if you cannot you get 0. In both cases
  you had to pay p to accept the challenge.

64
Interesting Belief(price,predicate,quality)

• Computational domain.
• Predicate selects set of problems.
  Challenge(Belief(price,predicate,quality), price)
• If I give you problem p satisfying predicate, you
  cannot find a solution of quality. If you can,
  you get 2price, otherwise 0. The cost of
  accepting the challenge is price.
• Belief form Exists problem your algorithm
  cannot do a task

65
Belief support system

• Does not need the notion of a computational
  domain. Just supported and unsupported beliefs.

66
Artificial Markets

• Based on beliefs.
• Creating challenges based on beliefs that you can
  support, makes money.
• Accepting challenges based on beliefs that you
  can successfully question, makes money.
• Supporter (prover) and questioner (verifier) have
  both limited resources.
• If a belief is supported, there is evidence that
  the belief is a theorem. The stronger the
  questioner, the stronger the evidence.
• If a belief is unsupported, there is evidence
  that the belief is not a theorem. The stronger
  the supporter, the stronger the evidence.

67
Constructive Belief math

• Problem domain. Define feasible solutions fs(f)
  for a problem f. Define quality function q(f,J).
• Predicate F defining a subset of problems in
  domain.
• Belief math(F, q(f,J) lt c, c) Alice believes
  that if she chooses f in F and Bob chooses J in
  fs(f), q(f,J) lt c (belief is supported by (f,J)).
• If !(q(f,J)ltc), belief is not supported by (f,J).

68
Example

• Domain MAX CSP, relations of arity 3
• Belief math(F,q(f,J)ltc,c)
• F relation one in three
• Second parameter fsat(f,J) lt c
• c 0.4
• Bob can show that this belief is not supported.
• If c 0.5, Alice can prevent Bob from showing
  that belief is unsupported.
• 0.444 break-even price

69
Constructive Belief secret

• Problem domain. Define feasible solutions fs(f)
  for a problem f.
• Predicate F defining a subset of problems in
  domain.
• secret(f) maps f to a secret feasible solution.
• Belief secret(F,q(f,J,Js)) I believe if Alice
  chooses f in F and secret(f) and Bob chooses J in
  fs(f) in time t, q(f, J,secret(f)) (belief is
  supported by (f,J)).
• If ! q(f, secret(f), J), belief is not supported.

70
Example secret Belief

• Domain MAX CSP, relations of arity 3
• Belief secret(F,q(f,J)ltc,c)
• F relation one in three
• Second parameter fsat(f,J) lt c
• c 0.4
• Bob can show that this belief is not supported.
• If c 0.5, Alice can prevent Bob from showing
  that belief is unsupported.
• 0.444 break-even price

71
Transition to Challenge

• Belief math(F,q(f,J,c),c) is combined with
  profit Alice offers challenge math (belief math
  (F,q(f,J,c),c), price)
• If Bob accepts challenge, he must show that
  belief is not supported.
• If he succeeds, he gains g(price, quality(f,J))
• If he fails, he looses price.

72
Proposed Research

• 4 Design