Computational Discovery of Communicable Knowledge

1
Inducing Process Models from Continuous Data
Pat Langley Institute for the Study of Learning
and Expertise Javier Sanchez CSLI / Stanford
University Ljupco Todorovski Saso Dzeroski Jozef
Stefan Institute
Supported by NTT Communication Science
Laboratories, by Grant NCC 2-1220 from NASA Ames
Research Center, and by EU Grant IST-2000-26469.
2
Exploratory Research in Machine Learning
Dietterich (1990) claims an exploratory research
report should

• define a challenging new problem for machine
  learning
• show that established methods cannot solve the
  problem
• present an initial approach that addresses the
  new task and
• outline an agenda for future research efforts in
  the area.

In this talk, we explore the problem of inducing
process models from continuous data.
3
Inductive Process Modeling
training data
learned knowledge
Induction
background knowledge
4
Inductive Process Modeling
training data
Observed values for a set of continuous
variables as they vary over time or situations
learned model
A specific process model that explains the
observed values and predicts future data
accurately
Induction
Generic processes that characterize causal
relationships among variables in terms
of conditional equations
background knowledge
5
A Process Model of an Ice-Water System
model WaterPhaseChange variables temp, heat,
ice_mass, water_mass observables temp, heat,
ice_mass, water_mass process ice-warming
conditions ice_mass 0, temp dtemp,t heat / (0.00206 ? ice_mass) process
ice-melting conditions ice_mass 0, temp
0 equations dice_mass,t ? (18 ? heat) /
6.02, dwater_mass,t (18 ?
heat) / 6.02 process water-warming conditions
ice_mass 0, water_mass 0,
temp 0, temp heat / (0.004184 ? water_mass)
6
Why Are Process Models Interesting?
Process models are a crucial target for machine
learning because

• they incorporate scientific formalisms rather
  than AI notations
• that are easily communicable to scientists and
  engineers
• they move beyond descriptive generalization to
  explanation
• while retaining the modularity needed to support
  induction.

These reasons point to process models as an ideal
representation for scientific and engineering
knowledge. Process models are an important
alternative to formalisms used currently in
machine learning.
7
Challenges of Inductive Process Modeling
Process model induction differs from typical
learning tasks in that

• process models characterize behavior of dynamical
  systems
• variables are mainly continuous and data are
  unsupervised
• observations are not independently and
  identically distributed
• process models contain unobservable processes and
  variables
• multiple processes can interact to produce
  complex behavior.

Compensating factors include a focus on
deterministic systems and the availability of
background knowledge.
8
Can Existing Methods Induce Process Models?
9
Facets of Inductive Process Modeling
To describe a system that learns process models,
we must specify

• characteristics of the data (observations to be
  explained)
• a representation for background knowledge
  (generic processes)
• a representation for learned knowledge (process
  models)
• a performance element that makes predictions (a
  simulator)
• a learning method that induces process models.

We will use an example from population dynamics
to illustrate an initial approach to inductive
process modeling.
10
Data for an Aquatic Ecosystem
11
Generic Processes for Population Dynamics
process exponential_growth process
exponential_decay variables P population
variables P population equations dP,t
0, 1,? ? P equations dP,t ? 0, 1, ?
? P process logistic_growth variables P
population equations dP,t 0, 1, ? ? P
? (1 ? P / 0, 1, ?) process constant_inflow
variables I inorganic_nutrient equations
dI,t 0, 1, ? process consumption
variables P1 population, P2 population,
nutrient_P2 number equations dP1,t 0,
1, ? ? P1 ? nutrient_P2,
dP2,t ? 0, 1, ? ? P1 ? nutrient_P2 process
no_saturation variables P number,
nutrient_P number equations nutrient_P
P process saturation variables P number,
nutrient_P number equations nutrient_P P /
(P 0, 1, ?)
12
Process Model for an Aquatic Ecosystem
model AquaticEcosystem variables nitro, phyto,
zoo, nutrient_nitro, nutrient_phyto observables
nitro, phyto, zoo process phyto_exponential_growt
h equations dphyto,t 0.1 ? phyto process
zoo_logistic_growth equations dzoo,t 0.1 ?
zoo / (1 ? zoo / 1.5) process phyto_nitro_consump
tion equations dnitro,t ?1 ? phyto ?
nutrient_nitro, dphyto,t 1
? phyto ? nutrient_nitro process
phyto_nitro_no_saturation equations
nutrient_nitro nitro process
zoo_phyto_consumption equations dphyto,t
?1 ? zoo ? nutrient_phyto,
dzoo,t 1 ? zoo ? nutrient_phyto process
zoo_phyto_saturation equations nutrient_phyto
phyto / (phyto 0.5)
13
Making Predictions with Process Models
Specify initial values for input variables and
the size for time steps
On each time step, check conditions to decide
which processes are active
Solve algebraic and differential equations with
known values
Propagate values and recurse to solve other
equations
Add the effects of different processes on each
variable
14
The IPM Method for Process Model Induction
Find all ways to instantiate known generic
processes with specific variables
Combine subsets of instantiated processes into
generic models
Remove candidates that are too complex or not
connected graphs
For each generic model, search for good
parameter values
Return parameterized model with the smallest
error
15
Initial Evaluation of IPM Algorithm
To demonstrate IPM's functionality at inducing
process models, we ran it on synthetic data for a
known system.
1. We used the aquatic ecosystem model to
generate data for 100 time steps, setting
nitrogen 1.0, phyto 0.01, zoo 0.01 2. We
replaced each true value x with x ? (1 r ?
0.05), where r came from a Gaussian distribution
(? 0 and ? 1) 3. We ran IPM on these noisy
data, giving it type constraints and generic
processes as background knowledge.
The IPM algorithm examined a space of 2196
generic models, each with an embedded parameter
optimization.
16
Predictions from IPMs Induced Model
17
Process Model Generated by IPM
model AquaticEcosystem variables nitro, phyto,
zoo, nutrient_nitro_1, nutrient_nitro_2,
nutrient_phyto observables nitro, phyto,
zoo process phyto_exponential_growth
equations dphyto,t 0.089 ? phyto process
zoo_logistic_growth equations dzoo,t 0.013
? zoo / (1 ? zoo / 0.469) process
phyto_nitro_consumption equations dnitro,t
?1.174 ? phyto ? nutrient_nitro_1,
dphyto,t 1.058 ? phyto ?
nutrient_nitro_1 process phyto_nitro_no_saturatio
n equations nutrient_nitro_1 nitro process
zoo_phyto_consumption equations dphyto,t
?0.986 ? zoo ? nutrient_phyto,
dzoo,t 1.089 ? zoo ? nutrient_phyto process
zoo_phyto_saturation equations nutrient_phyto
phyto / (phyto 0.487)
18
Process Model Generated by IPM(continued)
process nitro_constant_inflow equations
dnitro,t 0.067 process zoo_nitro_consumption
equations dnitro,t ?0.470 ? zoo ?
nutrient_nitro_2, dzoo,t
1.089 ? zoo ? nutrient_nitro_2 process
zoo_nitro_saturation equations
nutrient_nitro_2 nitro / (nitro 0.020)
These extra processes complicate the model but
have little effect on its behavior or its
predictive accuracy.
19
A Proposed Research Agenda
Future research on process modeling should
explore methods that

• reduce variance and overfitting (e.g., through
  pruning)
• determine the conditions on processes from
  training data
• associate variables with phyiscal entities to
  constrain search
• use a taxonomy of process types to organize and
  limit search
• use knowledge of dimensions and conservation to
  limit search
• support the induction of qualitative process
  models and
• revise existing process models rather than
  construct them.

This work should draw on traditional induction
methods, which have many relevant ideas.
20
Evaluation of Process Models
Research on this new class of problems should
follow the accepted standards thus, papers
should

• make explicit claims about an induction method's
  abilities
• support these claims with experimental or
  theoretical evidence
• study behavior on natural data sets to ensure
  relevance
• utilize synthetic data sets to vary dimensions of
  interest and
• incorporate ideas from other tasks and utilize
  existing methods whenever sensible.

In addition, the focus on communicability and use
of background knowledge suggests collaborations
with domain experts.
21
Concluding Remarks
In this exploratory research contribution, we
have

• proposed a new problem that involves induction of
  process models from components to explain
  observations
• argued that this task does not lend itself to
  established methods
• proposed a formalism for models and background
  knowledge
• presented an initial system that induces such
  process models
• demonstrated its functionality in a population
  dynamics domain
• outlined an agenda for future research in this
  new area.

Process model induction has great potential to
aid development of models in science and
engineering.
22
In Memoriam
Early last year, computational scientific
discovery lost two of its founding fathers

• Herbert A. Simon (1916 2001)
• Jan M. Zytkow (1945 2001)

Both contributed to the field in many ways
posing new problems, inventing methods, training
students, and organizing meetings. Moreover, both
were interdisciplinary researchers who
contributed to computer science, psychology,
philosophy, and statistics. Herb Simon and Jan
Zytkow were excellent role models that we should
all aim to emulate.
23
(No Transcript)
24
The LaGramge Discovery System
Our approach to inductive process modeling builds
on LaGramge (Todorovski Dzeroski, 1997), a
discovery system that

• specifies a space of abstract numeric equations
  in terms of a context-free grammar
• searches exhaustively through this space, to a
  given depth, to generate candidate abstract
  equations
• calls on established optimization techniques to
  determine the parameters for each equation and
• uses either squared error or minimum description
  length to select its final equations.

LaGramge has rediscovered an impressive class of
differential and algebraic equations from noisy
data.
25
Making Predictions with Process Models
To simulate a given process models behavior over
time, we can

• specify initial values for input variables and
  time step size
• on each time step, determine which processes are
  active
• solve active algebraic/differential equations
  with known values
• propagate values and recursively solve other
  active equations
• when multiple processes influence the same
  variable, assume their effects are additive.

This performance element makes specific
predictions that we can compare to observations.
26
A Method for Process Model Induction
We have implemented IPM, an algorithm that
constructs process models from generic components
in four stages
1. Find all ways to instantiate known generic
processes with specific variables 2.
Combine subsets of instantiated processes into
generic models, each specifying an explanatory
structure 2a. Ensure that each candidate
consists of a connected graph 2b. Limit
the maximum number of processes that can connect
any two variables and the total number
of processes 3. Translate the candidate into a
context-free grammar and invoke LaGramge to
search for good parameter values 4. Return the
model with the least error produced by LaGramge.