Stochastic Processes

1
Stochastic Processes

• Shane Whelan
• L527

2
Time Series
3
Univariate Time Series

• A univariate time series is a sequence, ltxngt for
  n1,N, where each observation, xn, is recorded
  at regular time interval t0n?t.
• The order of observations clearly matters it is
  crucial to the analysis.
• e.g., daily stock index values
• e.g., monthly inflation rates
• Time series modellling uses discrete time steps
  and the term time series is used to denote the
  random process ltXngt and a realisation, ltxngt of
  ltXngt.
• Note, in a useful generalisation, N can equal ?
  (a one-sided infinite sequence) and, in fact we
  can have a two-sided infinite sequence
  lt,x-1,x0,x1,gt

4
Deconstructing Univariate Time Series

• A useful decomposition of time series might be
  into
• Trend a one directional movement in the
  observations over the long-term.
• Cyclical component a periodic effect, with
  period considerably less than N. A seasonal
  effect is a cyclical component, the cycle having
  a period of 12 months.
• Random component a non-trending, non-cyclical
  error term representing the influence of
  non-modelled factors on the series.

5
Deconstructing Univariate Time Series

• Also can decompose time series into various
  frequencies
• Trend too long a period to estimate from given
  data (i.e., too low frequency)
• Cyclical as before, a medium frequency
• Random component a very high frequency that
  appears a mix of all frequencies (white noise,
  white light), so appears random or unmodellable.

6
Examples of Time Series

• Random Walk with Normally distributed increments,
  Sn, where
• and Xi are independent and normally distributed.

7
Exercises

• Exercise TS.1 Write down the density function of
  Sn.
• Exercise TS.2 Let in denote the interest rate in
  time period n-1 to n and let (1in) be
  distributed log-normal, parameters ?n, ?n. Assume
  that interest rates each year are indep. of one
  another. At time O the invested wealth is W0.
  What is the distribution of Wt, the accumulated
  wealth at time t?

8
Exercises

• Exercise TS.3 Assume in TS.2 that (1in) is
  distributed log-normal, with, for each n
  parameters ?0.075 and ?0.025. Let W01,000.
  Find the upper and lower quartiles of W5.

9
Examples of Time Series (Cont.)

• Moving Average (MA) Model. Let Yj, be a sequence
  of i.i.d random variables indexed from -? to ?
  and let ?0, ?1 ,,?p be p1 real numbers with ?p
  ?0. Let
• Then Xn is a moving average process of order p,
  denoted MA(p).

10
MA Models

• Often used to smooth datasee filtering later.
• Autocovariance only depends on lag (see later).
• Exercise TS.4 Prove that a moving average
  process has a constant mean.

11
Examples of Time Series (Cont.)

• An autoregressive process (AR) of order p is a
  sequence of random variables Xn (indexed from -?
  to ?) such that
• Where ?p ?0 and ltengt is a zero-mean white noise
  process.
• Denoted AR(p).

12
AR Processes - Remarks

• Mean is generally zero.
• Includes random walk as special case when p1 and
  ?11.

13
Filtering Time Series

• Often we want to detect, isolate and remove
  deterministic components or, in general, other
  features of a time series.
• One common general method is to filter the
  original or input time series (xn) to create a
  filtered or output time series (yn). An important
  class of filters is the linear filter (i.e., a
  linear weighted sum of original)
• The weights, ltajgt, are the filter.

14
Filtering Time Series

• Give example using difference filter.
• Of course, filters generally contain a finite
  number of terms, which are small relative to the
  number of data points in original time series.
  The filtered time series is generally shorter
  than the original one.

15
Examples of Filters

• The Perfect Delay. Consider a time series ltxngt.
  The time series delayed or lagged by j periods is
  the time series yn, where ynxn-j. This is a
  linear filter with ?k1 if kj, otherwise 0.

16
Examples of Filters

• The Centred Moving Average Filter. Define filter
  by
• where -q?k?q
• otherwise
• This filter does not affect linear trend in
  series but reduces the standard deviation of
  noise component by a factor of (2q1)-½ with some
  further assumptions.

17
Examples of Filters

• A linear filter is said to be recursive if the
  filtered series, yn, can be written in terms of
  lagged values of the series to be filtered,xn, or
  itself, i.e.,
• Recursive form is often handy for computation.
• By substitution, rewrite recursive filters in
  conventional format (i.e., output series as a
  function of input series only) before theoretical
  analysis.

18
Example of a Recursive Filter

• The exponential smoothing filter is defined
• ynayn-1(1-a)xn
• If ?a?lt1, then it can be written as an ordinary
  filter,
• This is an example of an MA(?) filter.

19
Filters in the frequency domain - Motivation

• Times series are often studied in the frequency
  domain (a function of w, frequency) as opposed to
  the time domain (a function of t) theoretically
  easier to handle and intuitive (after you make
  the initial leap)similar to the justification
  for using MGF rather than density functions in
  probability.
• Consider
• Then A is the amplitude, w is the frequency or
  angular frequency, ? is the phase and et is some
  (assumed) stationary series.

20
Filters in the frequency domain - Motivation

• But time series will tend to be a blend of many
  different periodic variations, so write
• Recall that eiwtcos(wt)isin(wt)
• So the representation above can be written

21
But how general is the new representation?

• Theorem Let be an absolutely
  summable series (i.e., ) then it can be
  represented in the form

22
So

• Apply Theorem to a filter so we know
  that it can be represented
• Note the slight change in where constant
  placed. A(w) is known as the transfer function
  of (linear) filter

23
Hence

• Given input time series and filter
  then the output time series is given
  by

24
Examples of Using Filters in Frequency Domain

• The transfer function of
• Perfect delay filter by j is given by
• Centred MA filter is given by
• Exponential smoothing filter is given by

25
Why prefer frequency domain?

• Recall that the transfer function X(w) or A(w)
  contains all the information in the time series
  or filter.
• Now filtering requires messy double summationthe
  convolution formula.
• Put filtering series ltxngt with tha ltakgt filter
  is, in the frequency domain, given by X(w).A(w).
• Hence we can apply many filters relatively
  easily.

26
Example

• An idealised filter, A(w)1 if wlt?/2 otherwise
  0, is applied to xn. What is the output? Describe
  its effect.

27
Backshift Operator Characteristic Equations

• The backshift operator, denoted B, when applied
  to series produces series
  where ynxn-1, i.e., B
• Sometimes, in a (slight) abuse of notation, it is
  written
• yn Bxn xn-1
• This is the same as the perfect delay filter with
  lag of 1.

28
Degression on Linear Operators

• B is a linear operator, i.e.,
• B(axby)a(Bx)b(By) for series x,y and constants
  a,b.
• (BC), the product of operators is defined as the
  end result of applying operator C first and then
  B to the result, i.e., BCx(B(Cx)).
• What is B2?
• How would you define B0?

29
The Backward Difference Operator

• The backward difference operator, denoted ?, when
  applied to series produces series
  where ynxn-xn-1
• Sometimes, in a (slight) abuse of notation, it is
  written
• yn ? xn xn - xn-1
• Also, ? (1-B)
• Exercise Prove ? is a linear operator
• Exercise Write down ?3xn in terms of
• (a) xn-k ,and,
• (b) in terms of just xn, constants and B
  operators.

30
Characteristic Equations

• Let be AR(p) process. Then
• xna1xn-1apxn-pen
• Rewrite as
• xn-a1xn-1--apxn-pen
• (1- a1B1--apBp)xn en
• Define the p-degree polynomial
• ?(?)1- a1 ? 1--ap ? p
• ?(?) is called the characteristic polynomial of
  the time series
• So now, ?(B)xn en

31
Characteristic Equations

• ?(?)0 is is called the characteristic equation.
• Let the roots of the characteristic equation be
  ?1,?2,,?p. Then the characteristic polynomial
  can be written in its factor form

32
Characteristic Equations

• Exercise
• Consider the time series
• xn(-1/6)xn-1(2/3)xn-2(-1/6)xn-3en
• Given that 2 is a root of its characteristic
  equation, write the characteristic equation in
  its factor form.

33
Stationary Time Series

• Recall that a stochastic process is weakly
  stationary, covariance stationary or second-order
  stationary, if EXn? and Covxn,xn-t?(t)
• Aside Can you recall what a stationary or
  strictly stationary stochastic process is?
• Generally, calibration techniques (i.e.,
  estimating the parameters of time series models)
  requires stationarity. Hence one tries to
  transform a given time series into a stationary
  one.

34
Autocovariance Autocorrelation

• Now, given stationarity, we have
• Cov(Xs, Xr)EXsXr-EXs2 ?(s-r)
• So ?(0) is the common variance of the terms.
• Properties of the autocovariance function, ?(n)
• It is an even function, i.e., ?(n) ?(-n)
• It is a non-negative definite function,i.e.,
• For all sets of real numbers, ?1,,?n and any set
  of integers k1,,kn the function ?(.) satisfies

35
Example White Noise

• A random process ltengt is termed white noise if
• Een? for all n.
• ?(m)cov(en,enm) ?2 if m0, otherwise 0.
• If ?0 then it is known as zero-mean white noise.
• When ltengt are iid Normal is an important special
  case of a white noise process.

36
Autovariance Autocorrelation

• Define the autocorrelation function (a
  dimensionless number) as
• ?n ?(n) / ?(0) and, so ?n?1.

37
Partial Autocorrelation

• The partial autocorrelation function of the
  series is denoted lt?(k)gt where k ranges from 1 to
  ? and defined as
• ?(k)det Pk/Pk
• Where Pk is the kxk autocorrelation matrix
• that is, Pk?ij where pijpi-j and Pk is Pk
  with the last column replaced by the transpose of
  (?1, ?2, ?3, ?k).

38
Examples of Partial Autocorrelation functions

• An AR(2) process given by Xn?1Xn-1?2Xn-2en has
  acf ?k. Prove that the pacf at lag 2, ?(2) ?2.
• This result holds in general, i.e., if Xn is
  AR(p) so Xn ?1Xn-1?2Xn-2 ?pXn-p en has acf
  ?k. Then the pacf at lag j, ?(j) ?j when j?p.

39
Partial Autocorrelation

• Another way to define and evaluate the pacf
  function ?(k) is recursively.Put ?(k) ?k,k then

40
Modelling Estimation

• The challenge is, given a finite time series (N
  points), to find a model of which the time series
  is a likely realisation.
• If stationary then common mean is often estimated
  using sample mean
• And autocovariance by sample autocovariances
• The autocorrelation function and pacf as
  functions of above so to estimate them just
  replace the true value with its estimate in the
  formula.

41
Some Modelling Tips

• The correlogram is a very useful diagnostic tool
  a graph of the acf as a function of its lag.
• Also graph the pacf against the lag as it
  indicates the order of the AR(p) process (i.e.
  the pacf of an AR(p) process cuts to 0 at lags
  p1 and greater).

42
Time Series 1
43
Correlogram
44
Correlogram Blown Up
45
Time Series 1 After Taking First Differences
46
Correlogram
47
Time Series 2
48
Correlogram Time Series 2
49
Main Linear Models of Time Series

• The 4 main linear models we shall treat are
• AR autoregressive model
• MA moving average model
• ARMA - autoregressive model with moving average
  residuals
• ARIMA - autoregressive integrated model with
  moving average residuals

50
Definition of AR(p) process

• Let B be the backshift operator and let EXn?
  ?n. Then the series ltXngt is AR(p) if ?(x) is a
  polynomial of order p and
• ?(B)(Xn- ?)en
• where en is zero-mean white noise series.
• To completely specify an AR(p) we need p initial
  conditions (x1,..xp) as well as the formula. The
  initial conditions can be real numbers or random
  variables.
• Example AR(1)

51
Theorem on AR Processes

• Let ltXngt be an AR(1) process,i.e.,
• Xn ? ?(Xn-1- ?) en where en is zero-mean white
  noise and given initial condition, Xs-1x ?
• Then,

52
Corollary

• An AR(1) process is not weakly stationary in
  generalboth mean and autocovariance depend on
  initial value of process and/or time lag since
  initiation.
• But non-stationarity decays with time since
  initiation, so for large values of n-s then it
  tends to be very like a stationary process. We
  call this property transient non-stationarity.
• If we assume that the process began infinitely
  long ago then the transient non-stationarity has
  worn off and now it has constant mean, ?, and
  acf, ?k?k.
• The pacf is zero for an AR(1) process at all lags
  greater than 1.

53
Similarly, we can fully characterise an AR(2)
process

• Let ltXngt be AR(2), i.e.,
• Xn?1Xn-1?2Xn-2 en , ngts-1
• Xs-1x1
• Xs-2x0
• And en is iid normal with zero mean and variance
  ?2.
• Then.

54
Theorem AR(2) Process Deconstructed

• Then we have ltXngt for n?s
• where ?1, ?2 are the reciprocals of the roots to
  the characteristic equation.

55
Moving Average Model

• Definition A moving average process of order p,
  denoted, MA(p), is a sequence of random
  variables, ltXigt, such that ?p?0 and
• Xn?en?1en-1?2en-2?pen-p
• Where en is a zero-mean white noise with variance
  ?2
• This can also be written in form
• Xn ??(B)en
• Where ? is a pth degree polynomial in B.

56
Moving Average Model

• An MA(p) process is called invertible if the
  p-roots of the equation ?(x)0 are greater than
  unity in absolute value.
• Consider an MA(1) process. Rewrite it as
• en-?-?1en-1 xn
• Then, on repeated substitution, we arrive at
• In this form, it says that there is a dependency
  on e0 which will only wear away if ?1lt1 (which
  is equivalent to saying the MA process is
  invertible).

57
Theorem on MA Models

• MA(p) is weakly stationary with EXn? and
  autovariance function given by
• when k?p and ?(k)0 if kgtp.
• Proof The proof relies on Eeiej?ij?2.Care is
  needed in handling the subscripts.
• Corollory An MA(p) process is weakly stationary.

58
Exercise

• Consider the MA(p) process given by
• Xtb0Ztb1Zt-1bpZt-p where Zi are zero-mean
  white noise.
• What is..
• The mean of EXt
• The variance of EXt
• Its autocorrelation function at lag k?

59
Exercise

• Classify the process
• Xn(0.25)Xn-2 en
• and, given that it has an infinite history, show
  that it can be written in the form
• Xnen(0.25)en-2(0.25)2en-4

60
Recap AR Processes

• We have shown that
• For the AR(1) case we derived its mean and
  covariance structure. We showed (in exercises)
  that its acf decays geometrically and its pacf at
  lags greater than 1 is zero. In particular we
  noted that it is transient non-stationary.
• These properties extend to the AR(p) case its
  acf decays (with maybe some ripples) and its pacf
  at lags greater than p is 0. It is transient
  non-stationary.
• Another fact (not proved) For an AR process to
  be (long run) stationary then all its
  characteristic roots must have modulus strictly
  greater than 1.

61
Recap MA Processes

• We have shown that
• For the MA(p) case we derived its mean and
  covariance structure. We showed that its acf cuts
  to 0 at lags greater than p. In particular we
  noted that it is stationary.
• It can be shown that the pacf of an MA(p) process
  tends to decay (with maybe some ripples).
• Another fact (not proved) For an MA process to
  be stationary then all its characteristic roots
  must have modulus strictly greater than 1.

62
Bringing it together - ARMA

• An ARMA(p,q) model is given by
• Xn?1Xn-1?2Xn-2?pXn-p enb1en-1
  b2en-2bqen-q
• Where ej is a zero-mean white noise.
• Using the Backshift operator we can write it as
• ?(B)Xn?(B)en.
• We need to specify p initial conditions.

63
Theorem (Proof not on Course)

• Let Xn be an ARMA(1,1) process defined by
• Then

64
Theorem continued (Proof not on Course)
65
Stochastic Processes

• Shane Whelan
• L527