Programming by Demonstration

1
Programming by Demonstration

• Christopher Cole
• Scott Dial

2
Outline

• Programming by Demonstration
• What is it?
• Prior Art
• CHINLE
• SUPPLE
• Version Spaces
• Implementation

3
Why PBD?

• Often, in computer interfaces, users wind up
  doing the same or similar sequences of operations
  over and over again in different situations. But
  if computers are so good at repetition, why is it
  that the users are the ones who keep repeating
  things?
• (Henry Lierberman)

4
What is PBD?

• Allow user to demonstrate a procedures
• The system records user actions and generalizes a
  program

5
What Makes PDB Hard?

• But there is more to a program than meets the
  recording. Users are quietly and imperceptibly
  making decisions. They are searching by eye. They
  are reading and understanding natural language.
  They are interpreting the interface in terms of
  their goals.
• (Allen Cypher)

6
PBD Techniques

• Conditional random fields (CRFs)
• Hidden Markov Models (HMMs)
• Version space (VS) algebra

7
Other Examples of PBD

• Pygmalion, Smith (1975)
• Stagecast Creator, Cypher, Smith, and Tesler
  (2000)
• SMARTedit, Lau, Wolfman, and Domingos (2001)
• DocWizards, Prabaker, Bergman, and Castelli (2006)

http//acypher.com/wwid/BackMatter/Chronology.html
8
CHINLE Overview
A functional framework for User Interfaces
SUPPLE
A mathematical description of the
possible programs represented by training data
Version Spaces
9
SUPPLE
SUPPLE Gajos Weld, 2004 is a functional
specification for user interfaces and an
implementation of the specification in Java. A
functional specification does not detail which
controls are used, where they are placed, and so
on. Instead it specifies the information an
application requires from a user and seeks to
optimize a usage utility function through a
selection of widgets and their layout constrained
by the available resources (such as screen
size). The authors suggest that user traces may
be useful in optimizing the layout for particular
users.
10
Version Spaces - Overview
Version spaces provide a way of describing all
possible programs consistent with a set of
constraints (for example, program traces). Using
a description of the problem domain (for example,
the user interface specification), one can
generate a Version Space. Version Space is
constructed using a Version Space
Algebra. Version Spaces never suggest
inconsistent programs. They assume constraints
are correct.
11
Version Spaces - Algebra
A Version Space is a set of hypotheses
(input-output pairs). More complicated VS can be
composed of simpler ones using the these
operators Union - Combine all members of two or
more version spaces to form a VS. Join - Form
the cartesian product of two or more version
spaces subject to a consistency predicate
12
Version Spaces - Base Cases
In the CHINLE system, there are three base
version spaces. The complete version space is
just the result of applying the previous
operators to the base cases. These cases
are Constant value (fixed, such as number of
pages per sheet) Constant delta (represents
iteration, such as through a list) Conditional
statement (if condition, then value1, else value2)
Do these really form a complete basis?
13
Version Spaces - Example
Suppose we want to perform two actions. In action
one the user wants to open an excel document. In
the second action the user will use Adobe
distiller to save the file as a landscape format
PDF. A version space might resemble the following
Program
Join
Action 1 (open in Excel)
Action 2 (save as pdf)
Union
Union
Boolean (landscape?)
More
String (filename)
More
14
VS - Choosing a Program
A version space can collapses when it has
contradictory constraints. In this case it cannot
select any valid program. A version space can be
degenerate. The constraints select exactly one
program. Generally, there are not enough
constraints to select exactly one program. CHINLE
applies a probability weight to every hypothesis
and selects the program with the
highest probability (let the hypotheses vote).
15
CHINLE - Choosing a Program
CHINLE first searches through all the possible
hypotheses. If they all yield the same output,
then any program will do. Otherwise, CHINLE as
the hypotheses vote to determine the actual
course of action. CHINLE equips the version
space with a probability distribution over the
space. A hypothesis gets increased weight if it
is unconditional. Among conditional hypotheses,
those conditioned on the value of the most
recently changed variable receive
more weight. For those interested, the paper
gives the exact algorithm (recursive descent
assigning different probability a leaves, joins,
and unions.)
16
CHINLE Automatic VS
For a given functional user interface
specification, the Version Space is a base
version space if the specification is a leaf node
of the interface, otherwise it is the union of
all version spaces of the subtrees (sub
components) of the specification.
This is just a recursive descent of
the specification.
17
Automatic Generation Example
Specification
Version Space by Alg. I
18
CHINLE - Recording
The first time the user records a trace for
CHINLE, it records the decisions the user makes.
It provides no suggestions, but it learns how
many actions the user plans to take on each
component of the user interface. CHINLE provides
suggestions to the user on subsequent
training rounds to help keep traces aligned,
detect training errors, and provide feedback to
the user about how well CHINLE has learned the
macro.
19
CHINLE - Training Example
Using the interface from the running example, one
notices that the suggested next step is
surrounded in a green box. CHINLE is confident
that this is the next step in the macro.
20
CHINLE - Possible Errors
It is possible that the user will make errors
while training CHINLE. Some possible errors
are Step Omission The user did not select a
setting in the recorder (perhaps because the
setting did not change from the default). The
user may discovers this when CHENLI suggests a
step that the user knows is not
next. Inconsistent Traces The user may have
made a mistake in the macro recording. It is also
possible that only a partial program can
handle what the user wants to do. Missing Data
Various faults can lead to missing data.
21
Omission/Missing Data
When the user realizes that a step is missing,
the user may insert this step by demonstrating it
during training. CHINLE adds the step to the
version space and treats information for that
step from its previous traces as selecting a
default or as missing data. Often such
selections lead to contradictions and collapse of
the version space. The authors assert that other
PBD systems fail and require the user to
start over the training. Are there reasons that
this may be a better approach?
22
CHINLE - Selection
The user may monitor the training process. In
particular, the user may select particular
hypotheses or remove previous training data.
23
Partial ProgramsorDealing with VS Collapse
When there is a version space collapse we have
contradictory data. If this is the result of
training error, then the user can use the
selection process to remove bad traces or simply
select one of the contradictory hypotheses. If
it is not due to user error, it means that only a
partial program is possible. For example, a macro
may be designed to process a different file each
time it is used. In this case the macro
really should request this information from the
user each time. Another reason for a partial
program is that the version space is not
sufficiently robust.
24
CHINLE - Partial Program
In the case where the user decides to allow
CHINLE to generate a partial program, it uses its
knowledge of the user interface to construct a
Wizard. This is a strength of using a
functional representation like SUPPLE.
25
Additional References
K. Gajos and D. S. Weld. Supple automatically
generating user interfaces. In Proc. Of IUI04.
Explains SUPPLE and its optimization
algorithm. H. Hirsh. Theoretical underpinnings
of version spaces. In Proc. of IJCAI91.
Specifies the mathematical criteria that
the version space set and constraints must
have. T. Lau, S. Wolfman, P. Domingos, and D. S.
Weld. Programming by demonstration using version
space algebra. Machine Learning, 53(1-2)111-156,
October-November 2003. Applies the version space
technique to generating programs that learn the
function of other programs (reverse engineering).