Multivariate Regression

1
Multivariate Regression- Techniques and
ToolsHeikki Hyötyniemi
2
LESSON 4

• Quick and Dirty

3
Multi-Linear Regression

• Straightforward (pseudoinverse) solution to the
  matching problem
• Applied routinely in all arenas
• Optimality does not guarantee good behavior!

4
SSE Criterion

• Extending the model Y XF to noisy data, so
  that
• the sum of squared errors the criterion to be
  minimized can be expressed as

5
Derivatives of the Criterion

• Gradient of the SSE
• and its Hessian

6
Solution

• Setting derivative to zero,
• the solution (for one output) is found

7
Parameter Sensitivity

• The accuracy of the model parameters can be
  estimated

8
Measures of Fit

• Calculate
• where

9
Multivariate Case

• For m distinct outputs there holds separately
• Combining these, one has the MLR formula

10
Optimal, But ...

• Generality problems caused by different types of
  variables (stochastic deterministic)
• Robustness problems caused by nonoptimal data
  (low number of samples or collinearity)
• Note Both problems solved during Lesson 5!

11
Error In Variables (EIV)
12
Colored Data

• Assuming that all variables are stochastic,
• applying the MLR formula gives
• so that there will be bias error and scale error.

13
Total Least Squares (TLS)

• Explicit search for the minimum-distance plane
  in the data space
• so that the regression (for one output) becomes

14
Distance from Plane

• Defining
• calculating gives the distance from
  the plane determined by the (normalized) vector f

15
Average Distances

• Average distance is given by
• where

16
Criterion of TLS

• Constrained optimization problem is found
• and using Lagrange multipliers the criterion is

17
Solution of TLS

• Minimum fulfills
• resulting in an eigenproblem

18
Analysis of TLS

• Solution to TLS is given by the eigenvector of
  the (extended) data covariance matrix
  corresponding to the smallest eigenvalue
• This eigenvector spans the null space of the
  data, revealing the variable redundancies
• Later, analogous eigenproblems will become very
  familiar!

19
Robustness Problems

• Matrix to be inverted in MLR
• may become badly conditioned if there are
• Too little data, there does not hold k gtgt n
• Collinear variables

20
Collinearity
21
Modeling Complex Processes

• Collinearity emerges if the variables are
  (almost) linearly dependent
• When a system is characterized using too many
  measurements, this inevitably happens
• Problem is not visible in simple well-structured
  systems, but emerges in complex PDE systems
• TLS does not solve the problem

22
Orthogonal Least Squares

• Try to avoid invertibility problems by enhancing
  the numerical data properties
• Data in X is orthogonalized before constructing
  the regression model
• In principle, Gram-Schmidt procedure
  In practice, QR factorization

23
Ridge Regression

• Not only emphasize error, but also parameter
  size
• This gives a better conditioned inversion problem

24
Data As Seen By RR

• Regression formula
• Result