LeastSquares Estimation

1

• Lecture
• Least-Squares Estimation

2
Introduction

• Application of Projection Theorem
• Areas of statistical estimation
• Formulated as equivalent minimum norm problems in
  Hilbert space
• Forms of least-squares estimation
• Classified by the choice of optimality
  criterion and
• required statistical assumptions
• Least-squares
• Means, variances, and covariances
• Maximum Likelihood and Bayesian
• Complete statistical description in terms of
  joint probability distribution function

3
Hilbert Space of Random Variables

• Brief Review of Probability
• For real-valued random variable x
• Probability distribution P
• P(z) Pr(x ? z).
• Expected value of a function g of x.
• Eg(x) ?Wg(z)dP(z)
• Expected value, second moment, and variance
• E(x), E(x2), E(x-Ex)2
• Given a finite collection of r.vs x1,x2,.,xn
• Joint probability distribution P
• P(z1, z2,., zn)Pr(x1 ?z1 ,x2 ? z2 ,.,xn ? zn)
• Expected value of a function g of x
• Eg(x1,x2,.,xn) ?W ?W ?W g(z1, z2,.,
  zn)dP(z1, z2,., zn)

4

• nxn covariance matrix cov(x1,x2,.,xn)
• Exi-Exixj-Exj ijth element
• Uncorrelated
• If Exixj ExiExj , or Exi-Exixj-Exj0
• Hilbert space of r.v.s
• Let y1,y2,.,yn be a finite collection
  of r.v.s with E(yi2) lt ?
• for each i.
• H consisting of all r.v.s which are linear
  combinations
• of yis equipped with inner product (xy)
  Exy
• E(?iaiyi)(?jbjyj), if x?iaiyi, and
  y?jbjyj
• Note finite-dim. Hilbert space with dim.
  equal to at most n.

5

• Properties
• If ExEz0, then x and z are orthogonal if they
  are uncorrelated. (xz)ExEz0.
• An n-dim. vector-valued r.v. x is an ordered
  collection of n scalar valued r.v.s
  xx1,x2,,xnT
• Hilbert space (H) of random vectors
• Suppose y1,y2,,ym is a collection of n-dim.
  random vectors, where each element has n
  components, each of which is a random variable
  with finite variance
• n-dim. random vectors as consisting of all
  vectors whose components are linear combinations
  yis. y?nKiyi, where Kis are real nxn matrices.

6

• Inner product (xz)E(?nxizi)E(xz) for x,z ?
  H
• The Norm llxllTraceE(xx)1/2

7
The Least-Squares Estimate
8
Minimum-Variance Unbiased Estimate

• Gauss-Markov Estimate
• An experiment of m-dim. data vector y of the
  form y Wbe, where W is a known matrix, b is an
  n-dim. unknown vector of parameters, and e an
  m-dim. random vector of zero mean and with
  covariance EeeQ (positive definite)
• Problem
• Find a linear estimate b of the form bKy,
  minimizing the norm of the error
• Ellb-bll2 EllKy-bll2EllKWbKe-bll2
  
• llKWb-bll2Trace(KQK)
• Note K is a function of b.

9

• Problem
• Find the estimate bKy minimizing Ellb-bll2
  while satisfying KWI.
• unbiased estimator E(b)E(Ky)E(KWbKe)KWb. If
  KWI, then E(b)E(b)
• Find the unbiased linear estimator which
  minimizes
• E(llb-bll2)
• Observations
• N separable problems in a Hilbert space

10

• Equivalent deterministic problem minimum norm
  problem in a apce of matrices
• Minimize TraceKQK
• Subject to KWI
• Minimize kiQki
• Subject to kiwjdij, where wj is the jth
  column of W and dij is the Kronecker delta
  function
• With inner product, (xy)QxQy
• Min (kiKi)Q
• Subject to (kiQ-1wi)Qdij
• Solution
• K Q-1W(WQ-1W)-1

11
Gauss-Markov Theorem

• Theorem

12

• Observations
• Minimum variance unbiased rather than minimum
  covariance unbiased
• Minimum variance unbiased estimate of each
  component in the sense of minimizing the sum of
  the individual variances
• If E(ee)I, then the linear minimum-variance
  unbiased estimate is identical with the
  least-squares estimate
• G-M problem is n separable minimum norm problem
• LS estimate is a single elementary minimum norm
  problem

13

• Additional properties
• Theorem 1
• The minimum variance linear estimate of a
  linear function of b, based on the random vector
  y, is equal to the same linear function of the
  minimum-variance estimate of b i.e., given an
  arbitrary pxn matrix T, the best estimate of Tb
  is Tb
• Proof
• by projection theorem
• Note
• Deduce the optimal estimate of a linear function
  of b

14

• Theorem 2
• If bKy is the linear minimum-variance
  estimate of b, then is also the linear estimate
  minimizing E(b-b)P(b-b) for any
  positive-semidefinite nxn matrix P.
• Proof
• Let P1/2 be the unique positive semi-definite
  square root of P. According to Theorem 1, P1/2 b
  is the minimum-variance estimate of P1/2 b and
  hence b minimizes
• Ell P1/2 b- P1/2
  bll2E(b-b)P(b-b)
• Note
• changes when the optimality criterion is more
  general quadratic form

15

• Theorem 2
• Let b be a member of a Hilbert space H of
  r.v.s and let b1 denote its orthogonal
  projection on a closed subspace Y1 of H. Let y2
  be an m-vector of r.v.s generating a subspace Y2
  of H, and let y2 denote the m-dim. vector of the
  projections of the components of onto Y1. Let
  y2y2-y2. Then the projection of b onto the
  subspace Y1Y2, denoted b, is b
  b1E(by2)E(y2 y2)-1 y2. In other words,
  b is b1 plus the best estimate of b in the
  space Y2 generated by y2

16

• Note
• Determines how an estimate is changed if the
  additional measurement data become available
• The updating must be based on the part of the new
  data which is orthogonal to the old data.
• Procedure(Sketchy of Proof)

Y2
Y2
Y1 ?Y2 Y1 Y2
Y1
17
Recursive Estimation

• Estimation Problems
• Prediction
• Estimation of future values of the past from past
  observations
• Filtering
• Estimation of the present value of a random
  process from inexact measurements of the process
  up present time
• Estimation of one random process from the
  observations of a different but related process
• Smoothing
• Defn
• Orthogonal(or white) iff Ex(j)x(k)ajdij

18

• Assumption
• Underlying every observed random process is an
  orthogonal process in the sense that the
  variables of the process are linear combinations
  of past values of the orthogonal process
• Examples
• Moving Average Process

19

• Autoregressive Scheme of Order 1
• Finite-Difference Scheme (AR Scheme of order n)

20
Dynamic model of a random process(n-dim.)

• Defn

21

• Estimation Problem
• Obtain the linear minimum-variance estimate of
  the state vector from the measurements v.
• Only cases when k?j are considered Prediction
• Solution process requires successive application
  of updating procedure

22

• Theorem (Kalman)

23

• Proof

24

• Proof(Cont.)