Linear and nonlinear time series analysis

1
Linear and nonlinear time series analysis

• Benvinguts a tothom!
• Bienvenido a todos!
• Welcome everybody!
• Ralph Gregor Andrzejak
• Department of Information and Communication
  Technologies
• Universitat Pompeu Fabra

2

• We will use Matlab and or Octave
• Lectures
• Please participate actively, ask and answer
  questions
• Lecture notes, practices and exemplary source
  codes http//www.cns.upf.edu/ralph/
• Feedback, suggestions?
• Anytime!

3

• Linear and nonlinear signal analysis
• Introduction to theoretical concepts
• Application to different classes of signals

4
MotivationMatlab
motivateexample.m
5
What do we want to learn?
Dynamical system
Characterization
Understanding
Measurement
Interpretation
Time series analysis algorithms
Properties
a 2.3 L 5 d 3.1 g 0.25
6
Definition Dynamical system System described by
d variables that evolve in time.
7
Harmonic oscillatora simple dynamical system
k
x(t)
m
8
Temporal evolution of the Harmonic
oscillatorMatlab
CallHarmonicOsciODEplotHarmonicOsciTSMS
9
Definition Signal Time dependent variable
reflecting the temporal evolution of an
underlying dynamical system.
10
Definitions Measurement, time series
11
Repetition Definition Dynamical system System
described by d variables that evolve in time.
Definitions State space, trajectory Abstract
space where each axis corresponds to one variable
of a dynamical system. A certain state of the
dynamical system is unambiguously represented by
a point in the state space. A temporal sequence
of such states/points forms a trajectory.
12
Brain storming What variables can be used to
construct the state space of the harmonic
oscillator?
k
x(t)
m
13
State space of theHarmonic oscillatorMatlab
(CallHarmonicOsciODE)(plotHarmonicOsciTSMS) plotH
armonicOsci
14
Repetition Definition Dynamical system System
described by d variables that evolve in time.
Definition Mathematical Model system Set of
equations which can be used to generate the
temporal evolution of a set of variables. Often
such models are used to reproduce the temporal
evolution of real dynamical systems.
15
Definition Time derivative
16
Differential equation of the Harmonic
oscillatorWhiteboard
17
Definition Differential equation of linear
harmonic oscillator, parameter The periodic
motion of a linear harmonic oscillator is
described by where x represents the
position and y is proportional to the velocity.
The quantity w represents a parameter of the
dynamics
18
Definition Set of first order ordinary
differential equations Or equivalently,
19
Definition Deterministic dynamical system A
dynamical system is called deterministic if its
temporal evolution is fully described by a set of
differential equations and a set of initial
conditions. If f is linear, the system is linear
(e.g. harmonic oscillator). If f is nonlinear,
the system is nonlinear (e.g. Lorenz).
20
Numerical solution of differential
equationsWhiteboard
21
Algorithm 4-th order Runge-Kutta integration
scheme A set of first order ordinary
differential equations and a set of initial
conditions can be solved under the iterative
application of Runge-Kutta steps
22
Numerical integration harmonic
oscillatorInfluence of stepsizeMatlab
CallHarmonicOsciVarstepkind.m
23
Definition Differential equation of Lorenz
dynamics The motion of a the Lorenz dynamics is
described by
24
Numerical integration Lorenz dynamicsState
spaceMatlab
CallLorenzSense.m
25
Rep Definition Deterministic dynamical
system A dynamical system is called
deterministic if its temporal evolution is fully
described by a set of differential equations and
a set of initial conditions. If f is linear, the
system is linear (e.g. harmonic oscillator). If
f is nonlinear, the system is nonlinear (e.g.
Lorenz).
Important fact For any given initial condition
of a deterministic dynamical system the future
evolution is fully determined. Accordingly, trajec
tories of deterministic dynamical systems cannot
intersect.
26
Definition Attractor Finite subspace of the
state space to which the solution of a dynamical
system, i.e. the trajectory, converges.
Definitions Chaos, sensitive dependence on
initial conditions, strange attractor For
chaotic dynamics two realizations with
infinitesimally different initial conditions
will always follow completely different
trajectories in time. Both trajectories will
approach the same strange attractor but via
completely different paths. Nonlinearity is
necessary but not sufficient for the emergence
of chaos.
27
Definition Stochastic dynamics For stochastic
dynamics the temporal evolution is not
unambiguously determined by rules such as
differential equations. Rather the temporal
evolution is governed by some random process.
Important fact For whatever given initial
condition is the future evolution of a
stochastic dynamical not determined.
Accordingly, trajectories of stochastic
dynamical systems can very well intersect.
28
Important fact Temporally uncorrelated random
numbers with a Gaussian amplitude distribution
are a fundamental stochastic processes.
29
Definition Autoregressive processes An
autoregressive (AR) process of order p is defined
by where hi is Gaussian white noise with
zero mean and unit variance. An autoregressive
process is a simple stochastic process.
30
Autoregressive processstochastic, deterministic
and chaotic dynamicsWhiteboard and Matlab
ar1ts CallHarmonicOsciVarstepkind
CallLorenzSense
31
Definition Statistical moments
32
Important fact Many formulas in both linear and
nonlinear time series simplify for time series
with zero mean and unit variance.
33
Definition Linear correlation coefficient
(Pearson)
34
Simplification of correlation coefficient for
zero mean and unit varianceWhiteboard
illustratepearson.m
35
Definition Autocorrelation function Not
e there are different ways of normalizing the
autocorrelation function.
36
Illustration of autocorrelation functionMatlab
callautocorr.m
37
Definition Decay time of the autocorrelation
function Note that there exist different
definitions of the decay time
Important fact The decay time of the
autocorrelation is a measure of the strength of
the autocorrelation of a time series
38
Comparison of autocorrelation functions for
different dynamicsMatlab
autocorrcompare
39
Important fact The autocorrelation of periodic
signals is periodic, The autocorrelation of
approximately periodic signals is strongly
oscillating (e.g. chaotic Rössler).
40
Definition Stationary dynamics For stationary
dynamics all parameters must not depend on
time. Parameters can for example be coefficients
in differential equations or autoregressive
processes.
41
Illustration of non-stationary Lorenz and AR
dynamicsTest for nonstationarityMatlab
Calllorenznonstat generatenonstatAR
42
Definition Moving window analysis Divide a long
time series in shorter windows and analyze
these short windows separately.
Important fact A simple test for stationarity
consists of extracting certain features in a
moving window fashion.
43
Important advice Notations may differ!
Notations used in different sources will
differ. There exist certain preferences and
conventions, however one always has to check the
applied notation carefully. In writing your own
documents always specify all symbols. E.g.
Three different but equivalent notations for a
time series with N samples
44
Repetition illustration measurementand sampling
time Matlab
CallHarmonicosciODE plotharmonisoscitsms
45
Definition Fourier transform, inverse Fourier
transform
46
Complex Fourier transform,location of positive
and negative frequenciesSymmetry of real and
imaginary partWhiteboard and Matlab
plot(real(fft(x))), fftshift, etc
47
Important fact The Fourier transform of
real-valued signals obeys the following
important symmetry
48
Important fact Parsevals theorem The total
power P in a signal is the same in the time and
frequency domain. The middle term in the above
equation is referred to as mean squared
amplitude.
49
Definition Periodogram
50
Periodograms for differentautoregressive
processes Matlab
illustratearperiodogram
51
Periodograms for superimposedprocesses Matlab
Illustratelinearityoffft illustratenoisydynamics
52
The edge effect, aliasingMatlab
Illustrateedgeeffect, illustratealiasing
53
Important fact The Fourier transform assumes the
time series to be one period of a periodic
signal. Therefore, differences in the amplitudes
and local slopes between the end and the
beginning of the time series have the same
effect as such a discontinuity would have in the
middle of the time series.
54
Important fact If the raw signal has energy at
frequencies higher than the Nyquist frequencies,
these will not be correctly resolved by the
Fourier Transform. Rather one finds spurious
peaks at wrong locations, an effect referred to
as aliasing. Remember that the Nyquist
frequency is ½ of the sampling frequency.
55
Extremely important fact Measures from linear
time series analysis such as the decay of the
autocorrelation function, statistical moments, or
the Periodogram are most appropriate for the
characterization of linear dynamics. They can
also be helpful for the characterization of
nonlinear dynamics. However, they are blind
for certain properties of nonlinear dynamics.
56
Illustration of two time series with identical
power spectra but very different structureMatlab
Illustratesamespectrabutdifferent
57
Repetition Definition Dynamical system System
described by d variables that evolve in time.
Repetition Definitions State space,
trajectory Abstract space where each axis
corresponds to one variable of a dynamical
system. A certain state of the dynamical system
is unambiguously represented by a point in the
state space. A temporal sequence of such
states/points forms a trajectory.
58
Important fact Typically one can not access all
variables of a dynamical system. Therefore, the
complete trajectory in the original state space
of the dynamics cannot be reconstructed.
59
Different state spaces of the harmonic oscillator
Matlab
illuembedding
60
Definition Delay Coordinates
Important fact Delay coordinates can be used to
construct an estimate of the trajectory of the
dynamical system underlying a scalar time
series.
61
Periodic, non-periodic Lorenz, and Rössler
dynamics drawn in delay coordinates Matlab
illuembedding
62
Important fact For the limit of an infinitely
long and noise-free time series measured from a
dynamics of dimension d, a proper reconstruction
of the dynamics can be obtained using an
embedding dimension of m2d1, and (almost) all
time delays t. For real, noisy time series of
finite length measured from a dynamical system
with unknown dimension, a good choice of m and
t is important to get a good reconstruction. In
the following lectures we will learn what
proper and good should mean. Note that
these are very qualitative statements. For a
formal description see for example (Kantz
Schreiber, 1997)
63
Study influence of time delays Matlab
illuembedding
64
Brain storming Which criterion can we use to
distinguish deterministic and stochastic
dynamics?
65
Important fact Neighboring trajectories of
deterministic dynamics are typically aligned,
whereas neighboring trajectories of stochastic
dynamics are typically not aligned.
In the following lectures we will use the
so-called nonlinear prediction error to
quantify this criterion between Stochastic and
deterministic dynamics.
66
Definition Distance matrix
67
Illustration distance matrix fromoriginal and
reconstructed Lorenz Matlab
illudistancematrix
68
Definition Nearest neighbors
69
(No Transcript)
70
Important fact The nonlinear prediction error
should give low values for deterministic
dynamics and higher values for stochastic
dynamics. Values around PE 1.1 are obtained for
white noise.
71
Rep Definition Mathematical Model system Set of
equations which can be used to generate the
temporal evolution of a set of variables. Often
such models are used to reproduce the temporal
evolution of real dynamical systems.
Very important fact Mathematical model systems
can be used to test measures such as the
nonlinear prediction error under controlled
conditions. Only with this experience from model
systems can we interpret results obtained for
experimental time series.
72
Important fact Nonlinear measures such as the
nonlinear prediction error depend strongly on
the choice of parameters. There exists a long
list of recipes for the optimal choice of these
parameters. However, the best thing to do is to
always look at results for a whole range of the
parameters.
73
Illustration nonlinear prediction error Matlab
CallNPEplots
74
Repetition Important fact Many formulas in both
linear and nonlinear time series analysis
simplify for time series with zero mean and unit
variance.
75
Illustration of the necessity of normalization of
the time seriesMatlab
callNPEplots
76
Important fact For the normalization of the
nonlinear prediction error it is necessary and
sufficient to normalize the time series to
zero mean and unit variance.
77
Illustration of the necessity of Theiler
correction for the nearest neighbor searchMatlab
callNPEplots
78
Influence of the length of the Theiler correction
Matlab
callNPEplots
79
Important fact The Theiler correction is
essential to reduce the influence of linear
correlations on nonlinear measures such as the
Nonlinear prediction error. Some high value of
W should be used.
callNPEplots
80
Influence of the length of the prediction
horizonMatlab
callNPEplots
81
Influence correlation strength of autoregressive
model Matlab
callNPEplots
82
Important fact The nonlinear prediction error is
strongly influenced by the strength of linear
correlations of stochastic dynamics.
Important problem Strongly correlated stochastic
dynamics might well have lower values of the
nonlinear prediction error than
(noisy) deterministic dynamics.
83
Important fact Often absolute values of
nonlinear measures are very difficult to
interpret. E.g. The value of the nonlinear
prediction error by itself does not allow one to
distinguish between deterministic and stochastic
dynamics.
84
Brain storming How can we do to distinguish
noisy deterministic from correlated stochastic
dynamics?
85
Results for phase-randomized signalsMatlab
callNPEplots
86
Definition Null hypothesis In our context
Assumption that a time series was measured from a
well-defined type of dynamical system.
Example Null hypothesis The time series was
measured from a stationary correlated Gaussian
stochastic process.
87
Definition Surrogates Random control signals.
Time series that have specified constraints in
common with the time series but are
otherwise random. In particular, surrogates are
constructed to be consistent with a certain null
hypothesis.
Example Surrogates A phase-randomized version of
the original time series can be used to test the
null hypothesis of a stationary correlated
Gaussian stochastic process.
88
Definition Discriminating statistics Some
measure that has to be sensitive to at least one
property not consistent with the null
hypothesis.
Example Discriminating statistics The nonlinear
prediction error can be used to test the null
hypothesis of a stationary correlated Gaussian
stochastic process.
89
Summary Null hypothesis Stationary correlated
Gaussian noise Surrogates Signal produced by
constrained phase randomization preserved power
spectrum, autocorrelation function destroyed the
rest Discriminating statistics e.g. Nonlinear
prediction error
90

• Definition
• Surrogate test
• Specify a null hypothesis
• Construct an ensemble of surrogates accordingly
• Choose a suitable discriminating statistics
• Calculate the discriminating statistics for the
  original signal
• as well as for all surrogates
• If and only if the result for the original signal
  is significantly
• outside the distribution obtained for the
  surrogates can the
• null hypothesis be rejected.

91
Very important fact The rejection of a null
hypothesis means only that the null hypothesis
was not correct, not more! It can not be taken as
any positive evidence such as if the time
series is not stationary Gaussian noise then it
must be deterministic. If a null hypothesis is
accepted that does not prove that it is correct!
It only means that with the applied
discriminating statistics we could not reject
it.
92
Very important fact One should at first always
consider the simplest null hypothesis. Only if
this is rejected more complicated hypothesis
should be considered.
Brain storming We tested the null hypothesis of
stationary correlated Gaussian noise. What
would be a simpler null hypothesis to tested?
93
Summary Null hypothesis Stationary uncorrelated
white noise Surrogates Random shuffle of the
original amplitude values preserved amplitude
distribution, statistic moments destroyed the
rest Discriminating statistics Nonlinear
prediction error, autocorrelation function
94
Illustration random shuffle surrogatesMatlab
callnpeplots
95
NPE of S signalsMatlab
callnpeplots
96
NPE of NSt signalsMatlab
callnpeplots
97
NPE of NonlM signalsMatlab
callnpeplots
98

• Type I
• Linear stationary uncorrelated white noise
• Random shuffle of amplitude values
• Preserved Amplitude distribution, statistical
  moments
• Destroyed Deterministic structure, power
  spectrum,
• Type II
• Linear stationary correlated Gaussian noise
• Phase-randomized signal
• Preserved Power spectrum, correlations
• Destroyed Deterministic structure, amplitude
  distribution,

Brain storming Suppose that the null hypothesis
of a type-II surrogates was rejected. Which
property could be included into those that are
preserved by the surrogates?
99

• Type I
• Linear stationary uncorrelated white noise
• Random shuffle of amplitude values
• Preserved Amplitude distribution, statistical
  moments
• Destroyed Deterministic structure, power
  spectrum,
• Type II
• Linear stationary correlated Gaussian noise
• Phase-randomized signal
• Preserved Power spectrum, correlations
• Destroyed Deterministic structure, amplitude
  distribution,
• Type III-A
• Linear stationary correlated Gaussian noise
  measured with a
• static, nonlinear but monotonic measurement
  function
• Phase-randomized-amplitude-adjusted surrogate
• Preserved Power spectrum (approximately),
  amplitude distribution (perfectly)
• Destroyed Deterministic structure,

100
Definition Static nonlinearity If a linear
stochastic Gaussian (un-)correlated noise is
measured with a nonlinear measurement function,
this will introduce a static nonlinearity.
Very important fact If a time series was
measured with a nonlinear but monotonic and
stationary measurement function, one can try to
account for this using amplitude adjusted
surrogates.
101
Illustration nonlinear measurement
functionMatlab
102
Motivation correlation sumWhiteboard
103
Motivation correlation sumWhiteboard Matlab
104
Correlation sum examples Harmonic
OscillatorMatlab
callCorrPlotOsci
105
Important fact If and only if there is a clear
scaling-region, that is a broad plateau of
almost constant values, in the plot of the
derivative of the correlation sum, can this
plateau be used to derive an estimate of the
correlation dimension of the dynamics. If there
is no such a scaling region, no evidence of any
finite- dimensional structure can be derived.
106
Correlation sum examples Lorenz, influence of
different parametersMatlab
callCorrPlotLorenz
107
Definition Strange attractor The trajectory of
chaotic dynamics, such as the Lorenz dynamics,
form a strange attractor. The sensitive
dependence on the initial conditions manifests
itself in the exponential divergence of nearby
trajectory segments. Furthermore, the space
filled by the attractor is not of an integer
dimension. Rather the dimension of
strange attractor is a non-integer number. E.g.
The Lorenz attractor has a correlation dimension
of approximately DC2.06. Strange attractors are
fractals.
108
Correlation sum examples Lorenz, influence of
embedding dimensionMatlab
Influencem.fig
109
Important fact For the calculation of the
correlation dimension too low values of the
embedding dimension result in projections which
can result in a too low estimate of the
correlation dimension. Too high values of the
embedding dimension can in turn result in too
strong statistical fluctuations and a very
narrow and noisy scaling region. Therefore, some
intermediate range of the embedding dimension
should be taken. Values of m between 10 and 15
can be regarded as a good starting point.
110
Important fact For the calculation of the
correlation dimension the use of a Theiler
correction is very important. If the Theiler
correction is not used this can corrupt a true
scaling region for low- dimensional dynamics, or
result in a pseudo scaling-region for stochastic
dynamics.
111
Important fact Stochastic dynamics have no
finite dimension. Accordingly If delay
coordinates are used to reconstruct the dynamics
with increasing embedding dimension, each new
dimension will be occupied. This is in
difference to deterministic, finite- dimensional
dynamics, which remain confined to a
low- dimensional sub-space, even if embedded in
higher- dimensional spaces.
112
Correlation sum examples AR Matlab
callCorrPlotAr
113
Correlation sum examples Lorenz, influence
noiseMatlab
callCorrPlotLorenznoise
114
Correlation sum and prediction errorOriginal
versus surrogatesMatlab
Prepared Figures
115
(No Transcript)