Nonlinear estimators and time embedding

1
Nonlinear estimatorsand time embedding
FIAS Frankfurt 7th August 2008
Raul Vicente raulvicente_at_mpih-frankfurt.mpg.de
2
OUTLINE
Introduction to nonlinear systems
Phase space methods
Exponents and dimensions
Interdependence measures
Take-home messages
2
3
OUTLINE
? Introduction to nonlinear systems
Definition Why nonlinear methods? Linear
techniques
Phase space methods
Exponents and dimensions
Interdependence measures
Take-home messages
3
4
INTRODUCTION
Definition
Nonlinear is a very popular word in
(neuro)science but what does it really mean?
A nonlinear system is one whose behavior cant be
expressed as a sum of the behaviors of its parts.
In technical terms, the behavior of nonlinear
systems is not subject to the principle of
superposition.
The brain as a whole is a nonlinear device
Ex our perception can be more than the sum of
responses to individual stimulus
Surface completion
4
5
INTRODUCTION
Definition
Nonlinear is a very popular word in
(neuro)science but what does it really mean?
A nonlinear system is one whose behavior cant be
expressed as a sum of the behaviors of its parts.
In technical terms, the behavior of nonlinear
systems is not subject to the principle of
superposition.
Individual neurons are also nonlinear
Excitable cells with all or none responses
Nonlinear frequency response
Double the input does not mean double the output
4
6
INTRODUCTION
Definition
Nonlinear is a very popular word in
(neuro)science but what does it really mean?
A nonlinear system is one whose behavior cant be
expressed as a sum of the behaviors of its parts.
In technical terms, the behavior of nonlinear
systems is not subject to the principle of
superposition.
Individual neurons are also nonlinear
The brain as a whole is a nonlinear device
Ex our perception can be more than the sum of
responses to individual stimulus
Double the input does not mean double the output
4
7
INTRODUCTION
Why nonlinear methods?
The study of non-linear physics is like the
study of non-elephant biology Unknown
Neuronal activity is highly nonlinear
Nonlinear features will be present in the
recorded neurophysiological data From neuronal
action potentials (spikes) to integrated activity
(EEG, MEG, fMRI) Linear techniques might fail to
capture key information
Nonlinear indices measure complexity of EEG,
monitoring depth of anaesthesia, studies of
epilepsy, detection of interdependence, etc...
5
8
INTRODUCTION
Linear techniques
Linear systems always need irregular inputs to
produce bounded irregular signals
Linear System
...,Sn-1, Sn, Sn1,...
Most simple system which produces nonperiodic
(interesting) signals is a linear stochastic
process
p(s) probability dist.
Measurement of state Sn at time n of such a
process
Information about p(s) can be inferred from the
time series
6
9
INTRODUCTION
Linear techniques
Linear methods interpret all regular structure in
a data set, such as a dominant frequency, as
linear correlations (time or frequency domain)
Autocorrelation at lag n
7
10
INTRODUCTION
Linear techniques
Cross-correlation function measures the linear
correlation between two variables X and Y as a
function of their delay time (t)
Cross-correlation at lag n
rxy 0.63
rxy 0.25
8
11
INTRODUCTION
Linear techniques
Cross-correlation function measures the linear
correlation between two variables X and Y as a
function of their delay time (t)
Cross-correlation at lag n
tendency to have similar values with the same
sign tendency to have similar values with
opposite sign suggest lack of linear
interdependence
t that maximizes this function
estimator delay between signals
8
12
INTRODUCTION
Linear techniques
Coherence measures the linear correlation
between two signals as a function of the frequency
Coherence at frequency f
In forming an estimate of coherence, it is always
essential to simulate ensemble averaging.
EEG and MEG signals are subdivided in epochs or
for event-related data spectra are averaged over
trials
activities of the signals in this frequency are
linearly independent maximum linear correlation
for this frequency
9
13
INTRODUCTION
Linear techniques
Prediction we have a sequence of measuraments
sn, n 1,...,N and we want to predict the
outcome of the following measurement, sN1
Linear prediction
minimising the error
In-sample
Out-of-sample
10
14
INTRODUCTION
Linear techniques
In neurophysiology, a question of great interest
is whether there exists a causal relation
between two brain regions. Inferring causality
from the time delay in the cross-correlation is
not always straighforward
Causality (Nobert Wiener in 1956) for two
simultaneously measured signals, if one can
predict the first signal better by incorporating
the past information from the second signal than
using only information from the first one, then
the second signal can be called causal to the
first one
Predicting the future of X improves when
incorporating the information about the past of
Y ? Y is causal to X
11
15
INTRODUCTION
Linear techniques
Causality (Nobert Wiener in 1956) for two
simultaneously measured signals, if one can
predict the first signal better by incorporating
the past information from the second signal than
using only information from the first one, then
the second signal can be called causal to the
first one
Granger just applied this definition in the
context of linear stochastic models. If X is
influencing Y, then adding the past values of
the first variable to the regression of the
second one will improve its prediction error.
Bivariate fitting
Univariate fitting
Prediction performance is assessed by the
variances of the prediction errors
Granger causality
12
16
OUTLINE
Introduction to nonlinear systems
? Phase space methods
The concept of phase space Attractor
reconstruction Time embedding Application
nonlinear predictor
Exponents and dimensions
Interdependence measures
Take-home messages
13
17
Phase space
The concept
Phase space is a space in which all possible
states of a system are represented, with each
possible state of the system corresponding to one
unique point in the phase space
Ex the Fitzhugh-Nagumo model is a
two-dimensional simplification of the
Hodgkin-Huxley model of spike generation
- membrane potential - recovery variable
V(t)
(V, W)
W(t)
In a phase space, every degree of freedom or
parameter of the system is represented as an axis
of a multidimensional space. Ex dim(HH) 4
For deterministic systems (no noise), the system
state at time t consists of all information
needed to uniquely determine the future system
states for times gt t
14
18
Phase space
Attractor reconstruction
Attractor a set of points in phase space such
that for "many" choices of initial point the
system will evolve towards them. It is a set to
which the system evolves after a long enough time
Attractor set A point A curve A manifold
Behavior Constant Periodic Possibly chaotic
Van der Pol limit cycle attractor
Strange attractors produce chaotic
behavior. Nonlinear systems irregular dynamics
without invoking noise!
Lorenz attractor
15
19
Phase space
Attractor reconstruction
In general (and especially in biological systems)
it is impossible to access all relevant variables
of a system. Ex usually in electrophysiology we
just measure membrane voltage.
These vectors constructed from a single variable
play a role similar to x(t),y(t),z(t)
16
20
Phase space
Time embedding
In general (and especially in biological systems)
it is impossible to access all relevant variables
of a system. Ex usually in electrophysiology we
just measure membrane voltage.
17
21
Phase space
Application nonlinear predictor
Depending on the type of signals the power of
predictability and the best strategy changes
- A signal does not change is easy to predict
take the last observation as a forecast for the
next one
- A periodic system is also easy one observed for
a full cycle
- For independent random numbers the best
prediction is the mean value

• Interesting signals are not periodic but contain
  some kind of structure which can be exploited to
  obtain better
• predictions

If the source of predictability are linear
correlations in time next observations will be
given approximately by a linear combination of
preceding observations
What if I know that my series is nonlinear?
18
22
Phase space
Application nonlinear predictor
For nonlinear deterministic systems all future
states are unambiguosly determined by specifying
its present state. Nonlinear correlations can be
exploited with new techniques
Current state
Neighbors
Next values of
Predicted state
Lorenz method of analogues Prediction for the
future state XN1 Look for recorded states close
to the one we want to predict Predict the average
of the next states of the past neighbors
Better prediction for short time scales than
linear predictors
19
23
OUTLINE
Introduction to nonlinear systems
Phase space methods
? Exponents and dimensions
Sensibility to initial conditions Lyapunov
exp. Self-similarity correlation dimension
Interdependence measures
Take-home messages
20
24
Exponents and dimensions
Sensibility to initial conditions
The most striking feature of chaos is the
long-term unpredictability of the future despite
a deterministic time evolution.
The cause is the inherent instability of the
solutions, reflected in their sensitive dependence
on initial conditions.
Amplification of errors since nearby trajectories
separate exponentially fast. How fast it is
measured by the Lyapunov exponent l.
type of motion maximal Lyap. exp.
stable fixed point l lt 0
limit cycle l 0
chaos 0 lt l lt 8
noise l 8
The inverse of the maximal Lyap. Exp. defines the
time beyond which predictability is impossible.
21
25
Exponents and dimensions
Sensibility to initial conditions
The most striking feature of chaos is the
long-term unpredictability of the future despite
a deterministic time evolution.
The cause is the inherent instability of the
solutions, reflected in their sensitive dependence
on initial conditions.
Maximal Lyapunov exponent from time series
- Delay embedding - Compute the average diverging
rate
- The slope of S(Dn) is an estimate of the
maximal LE
22
26
Exponents and dimensions
Correlation dimension
Strange attractors with fractal dimension are
typical of chaotic systems. Non
integer dimensions are assigned to geometrical
objects which exhibit self-similarity and
structure on all length scales.
Box-counting dimension
Kaplan-Yorke conjecture relates D to Lyapunov
spectra
For time series
- Delay embedding - Compute the correlation sum
23
27
Exponents and dimensions
Applications
Nonlinear statistics such exponents, dimensions,
prediction errors, etc., can be computed to
characterize non-trivial differences in signals
(EEG) between different stages (brain states
sleep/rest, eyes open/closed).
Word of caution such quantities are used to
compare data from similar situations!
Ex ECG series taken during exercise are more
noisy due to sweat of patient skin. The
different noise levels at rest and exercising can
affect the former nonlinear estimators and
erroneusly conclude a higher complexity of the
heart during exercise just because of sweat on
the skin.
24
28
OUTLINE
Introduction to nonlinear systems
Phase space methods
Exponents and dimensions
? Interdependence measures
Synchronization
Causality Transfer entropy
Take-home messages
25
29
Interdependence measures
Synchronization
Synchronization is the dynamical process by which
two or more oscillators adjust their rhythms due
to their weak interaction.

• A universal phenomenon found everywhere
• Mechanical systems (pendula, Londons bridge, )
• Electrical generators (power grids, Josephson
  junctions, )
• Life sciences (biological clocks, firing
  neurons, pacemaker cells, )
• Chemical reactions (Belousov-Zhabotinsky)
• ...

Synchronization refers to the way in which
coupled elements, due to their dynamics,
communicate and exhibit collective behavior. In
large populations of oscillators synchronization
can be understood as a self-organization process.
Neural synchronization is one of the most
promising mechanism to underlay the flexible
formation of cell assemblies and thus bind the
information processed at different areas.
Without a master o leader the individuals
spontaneously tend to oscillate in synchrony.
26
30
Interdependence measures
Synchronization hallmarks
__________________________Frequency
locking___________________________
Before coupling Df
After coupling DF
DFF2-F10
f2
f1
Higher order F2/F1q/p
____________________________Phase
locking______________________________
Phase shift q is fixed

• Phase can be extracted from
• data by several techniques
• Hilbert transform
• Wavelet transform
• Poincare map

q
q0 ? in-phase
qp ? anti-phase
27
31
Interdependence measures
Synchronization solutions
Different types of synchronization capture
different relationships between the signals x1(t)
and x2(t) of two interacting systems
Classical adjustment of rhythms in periodic
oscillators. Identical coincidence of outputs
due to their coupling, x1(t)x2(t). Generalized
captures a more general relationship like
x1(t)F(x2(t)). Phase expresses the regime
where the phase difference between two irregular
oscillators is bounded but their amplitudes are
uncorrelated. Lag accounts for relation
between two systems when compared at different
times such as x1(t)x2(t-t). Noise-induced
synchronization induced by a common noise source.
28
Information theory mutual information,
entropies,...
32
Correlation does not imply causality
In March 1999, Nature
29
33
Interdependence measures
Nonlinear causality measures
Information theory
Transfer entropy
30
34
INTRODUCTION
Interdependence measures
Nonlinear causality measures
Apply transfer entropy concept to determine the
causal/effective connectivity from ERFs (MEG).

• Embedding of the time series from each channel
  (Caos criterium)
• Kozachenko-Leonenko estimator of entropies from
  kernel-based
• probability densities

together M. Wibral, J. Triesch, G. Pipa
31
35
OUTLINE
Introduction to nonlinear systems
Phase space methods
Exponents and dimensions
Interdependence measures
? Take-home messages
32
36
Take-home messages
Linear vs nonlinear
- Linear techniques are much well understood and
rigorous.
- Linear and nonlinear estimates may assess
different characteristics of the signals.
- Complementary approaches to the analysis of
temporal series.
Even though most of systems in Nature are
nonlinear do not underestimate linear methods
33
37
Take-home messages
Phase space methods

• Attractor reconstruction is a powerful technique
  to recover the topological structure
• of an attractor given a scalar time series.

- Useful complexity quantifiers of the signal can
be computed after the reconstruction.
- Word of caution in their use.
34
38
Take-home messages
Do it yourself... with a little help

• TISEAN is a very complete software package for
  nonlinear time series analysis

• Nonlinear time series analysis by Holger Kantz
  and Thomas Schreiber,
• Cambridge University Press.

35