Linear Regression

1
Linear Regression

• Using a linear function to interpolate the
  training set

• The most popular criterion Least squares
  approach

• Given the training set

where
Find a linear function
is determined by solving the minimization problem
2
Linear Regression (Cont.)

• Different measures of loss are possible

• 1-norm loss function

• Hubers regression

• Ridge regression

where
3
Solution of the Least Squares Problem
Some notations
with the samllest
We are going to find the
i.e.,
square loss.
4
The Widrow-Hoff Algorithm (Primal Form)
Initial
Repeat
Until convergence criterion satisfied return

• Minimize the square loss function using gradient
  descent

)

• Dual form exists (i.e.

5
The Normal Equations of LSQ
Letting
we have
the normal equations of LSQ
is inversable then
If
Note The above result is based on the First Order
Optimality Conditions (necessary sufficient for
differentiable convex minimization problems)
is singular ?
What if