Ordinary Least-Squares

1
Regularized Least-Squares
2
Outline

• Why regularization?
• Truncated Singular Value Decomposition
• Damped least-squares
• Quadratic constraints

3
Why regularization?

• We have seen that

4
Why regularization?

• We have seen that
• But what happens if the system is almost
  dependent?
• The solution becomes very sensitive to the data
• Poor conditioning

5
The 1-dimensional case

• The 1-dimensional normal equation

6
The 1-dimensional case

• The 1-dimensional normal equation

7
The 1-dimensional case

• The 1-dimensional normal equation

8
Why regularization

• Contradiction between data and model

9
A more interesting examplescattered data
interpolation
10
True curve
11
Radial basis functions
12
Radial basis functions
13
Rbf are popular

• Modeling
• J. C. Carr, R. K. Beatson, J. B. Cherrie, T. J.
  Mitchell,W. R. Fright, B. C. McCallum, and T. R.
  Evans. Reconstruction and representation of 3d
  objects with radial basis functions. In
  Proceedings of ACM SIGGRAPH 2001, Computer
  Graphics Proceedings, Annual Conference Series,
  pages 6776, August 2001.
• G. Turk and J. F. OBrien. Modelling with
  implicit surfaces that interpolate. ACM
  Transactions on Graphics, 21(4)855873, October
  2002.
• Animation
• J. Noh and U. Neumann. Expression cloning. In
  Proceedings of ACMSIGGRAPH 2001, Computer
  Graphics Proceedings, Annual Conference Series,
  pages 277288, August 2001.
• F. Pighin, J. Hecker, D. Lischinski, R. Szeliski,
  and D. H. Salesin. Synthesizing realistic facial
  expressions from photographs. In Proceedings of
  SIGGRAPH 98, Computer Graphics Proceedings,
  Annual Conference Series, pages 7584, July 1998.

14
Radial basis functions

• At every point

15
Radial basis functions

• At every point
• Solve the least-squares problem

16
Radial basis functions

• At every point
• Solve the least-squares problem

17
Rbf results
18
pi0 close to pi1
19
Radial basis functions

• At every point
• Solve the least-squares problem

20
Radial basis functions

• At every point
• Solve the least-squares problem
• If pi0 close to pi1, A is near singular

21
pi0 close to pi1
22
pi0 close to pi1
23
Rbf results with noise
24
Rbf results with noise
25
The Singular Value Decomposition

• Every matrix A (nxm) can be decomposed into
• where
• U is an nxn orthogonal matrix
• V is an mxm orthogonal matrix
• D is an nxm diagonal matrix

26
The Singular Value Decomposition

• Every matrix A (nxm) can be decomposed into
• where
• U is an nxn orthogonal matrix
• V is an mxm orthogonal matrix
• D is an nxm diagonal matrix

27
Geometric interpretation
28
Solving with the SVD
29
Solving with the SVD
30
Solving with the SVD
31
Solving with the SVD
32
Solving with the SVD
33
A is nearly rank defficient
34
A is nearly rank defficient
35
A is nearly rank defficient
36
A is nearly rank defficient

• A is nearly rank defficient
• gtsome of the are close to 0

37
A is nearly rank defficient

• A is nearly rank defficient
• gtsome of the are close to 0
• gtsome of the are close to

38
A is nearly rank defficient

• A is nearly rank defficient
• gtsome of the are close to 0
• gtsome of the are close to
• Problem with

39
A is nearly rank defficient

• A is nearly rank defficient
• gtsome of the are close to 0
• gtsome of the are close to
• Problem with
• Truncate the SVD

40
pi0 close to pi1
41
Rbf fit with truncated SVD
42
Rbf results with noise
43
Rbf fit with truncated SVD
44
Choosing cutoff value k

• The first k such as

45
Example inverse skinningSkinning Mesh
Animations, James and Twigg, siggraph

• Skinning

46
Example inverse skinningSkinning Mesh
Animations, James and Twigg, siggraph

• Skinning

?
47
Example inverse skinningSkinning Mesh
Animations, James and Twigg, siggraph

• Skinning

48
Example inverse skinningSkinning Mesh
Animations, James and Twigg, siggraph

• Skinning
• Inverse skinning
• Let be a set of pairs of geometry and skeleton
  configurations

49
Example inverse skinningSkinning Mesh
Animations, James and Twigg, siggraph

• Skinning
• Inverse skinning
• Let be a set of pairs of geometry and skeleton
  configurations

50
Example inverse skinningSkinning Mesh
Animations, James and Twigg, siggraph

• Skinning
• Inverse skinning
• Let be a set of pairs of geometry and skeleton
  configurations

51
Example inverse skinningSkinning Mesh
Animations, James and Twigg, siggraph

• Skinning
• Inverse skinning
• Let be a set of pairs of geometry and skeleton
  configurations

52
Skinning Mesh Animations, James and Twigg,
siggraph
53
Problem with the TSVD

• We have to compute the SVD of A, and O() process
• impractical for large marices
• Little control over regularization

54
Damped least-squares

• Replace
• by
• where is a scalar and L is a matrix

55
Damped least-squares

• Replace
• by
• where is a scalar and L is a matrix
• The solution verifies

56
Examples of L
Diagonal
Differential
Limit scale
Enforce smoothness
57
Rbf results with noise
58
(No Transcript)
59
(No Transcript)
60
(No Transcript)
61
Example Least-Squares Meshes, Sorkin and
Cohen-Or, siggaph

• Reconstruct a mesh given
• Control points
• Connectivity (planar mesh)

62
Example Least-Squares Meshes, Sorkin and
Cohen-Or, siggaph

• Reconstruct a mesh given
• Control points
• Connectivity (planar mesh)
• Smooth reconstruction

63
Example Least-Squares Meshes, Sorkin and
Cohen-Or, siggaph

• Reconstruct a mesh given
• Control points
• Connectivity (planar mesh)
• Smooth reconstruction
• In matrix form

64
Reconstruction

• Minimize reconstruction error
• where

65
Least-Squares Meshes, Sorkine and Cohen-Or,
siggraph
66
Quadratic constraints

• Solve
• or

67
Quadratic constraints

• Solve
• or

68
Example
69
Example
70
Example
71
Discussion

• If , there is no
  solution (since there is no x for which
  )

72
Discussion

• If , there is no
  solution (since there is no x for which
  )
• If , the solution exists
  and is unique

73
Discussion

• If , there is no
  solution (since there is no x for which
  )
• If , the solution exists
  and is unique
• Either the solution of
  is in the feasible set

74
Discussion

• If , there is no
  solution (since there is no x for which
  )
• If , the solution exists
  and is unique
• Either the solution of
  is in the feasible set
• or the solution is at the boundary
• Solve

75
Discussion

• Solve
• where is a Lagrange multiplier

76
Conclusion

• TSVD really useful if you need an SVD

77
Conclusion

• TSVD really useful if you need an SVD
• Regularization constrains the solution
• Value, differential operator, other properties
• Soft (damping) or hard constraint (quadratic)
• Linear or non-linear

78
Conclusion

• TSVD really useful if you need an SVD
• Regularization constrains the solution
• Value, differential operator, other properties
• Soft (damping) or hard constraint (quadratic)
• Linear or non-linear
• Danger of over-damping or constraining

79
Example inverse kinematic

• Problem solve for joint angles given
  end-effector positions

80
Example inverse kinematic

• Problem solve for joint angles given
  end-effector positions

?
81
Example inverse kinematic

• Problem solve for joint angles given
  end-effector positions

82
Example inverse kinematic

• Problem solve for joint angles given
  end-effector positions