Quadratic Minimisation Problems in Statistics

1
Quadratic Minimisation Problems in Statistics

• Casper Albers, Frank Critchley John Gower
• Department of Statistics, The Open University

2
Outline

• Introduction to problem (1)
• Statistical examples of problem (1)
• Geometrical insights some easy, some hard
• Concluding remarks

3
The essential problem
(1)

• A and B are square matrices (of the same order
  p)
• A is p.d. or p.s.d.
• B can be anything
• The constraint is consistent

4
Equivalent forms

• Eq. (1) can occur in many other shapes and forms,
  e.g.
• min (x t)'A(x t) subject to (x-s)'B(x-s)
  2g'(x-s) k
• minx Xx y2 subject to x'Bx 2b'x k
• min trace (X T)'A(X T)
• subject to trace (X'BX 2G'X) k
• We present a unified solution to all such
  problems.

5
General canonical form

• After simple affine transformations z T-1 x m
  and s T-1 t m where T is such that,
• 
  , (1) reduces to

6
Applications

• Problem (1) arises, for example, in
• Canonical analysis
• Normal linear models with quadratic constraints
• The fitting of cubic splines to a cloud of points
• Various forms of oblique Procrustes analysis
• Procrustes analysis with missing values
• Bayesian decision theory under quadratic loss
• Minimum distance estimation
• Hardy-Weinberg estimation
• Updating ALSCAL algorithm

7
Application Hardy-Weinberg

• Genotypes AA, BB, AB in proportions p (p1, p2,
  p3)
• Observed proportions q (q1, q2, q3)
• HW equilibrium constraint p32 4 p1 p2
• Additional constraints 1' p 1, p 0
• GCF
• Note linear term

8
Indefinite constrained regression

• Ten Berge (1983) considers for the ALSCAL
  algorithm
• The GCF has eigenvalues
• (1 v2, ½, 1 - v2)

9
Ratios of quadratic forms (1)

• Canonical analysis min x'Wx / x'Bx.
• When W or B is of full rank, we have
• min x'Wx s.t. x'Bx 1, of form (1) with
• Lagrangian Wx ?Bx.
• BUT the ratio form requires only a weak
  constraint while if the Lagrangian is taken as
  fundamental, the constraint becomes strong (see
  Healy Goldstein, 1976, for x'1 1).
• In canonical analysis, multiple solutions are
  standard but seem to have no place in our more
  general problem (1).

10
Ratios of quadratic forms (2)

• When both A and B are of deficient rank
• In the canonical case, the ANOVA T W B
  implies that the null space of T is shared by B
  and W, and a simple modification of the usual
  two-sided eigenvalue solution suffices.
• However, for general matrices A, B things become
  much more complicated.

11
Geometry helps understanding

• The following slides illustrate the problem
  geometrically showing some of the complications
  that have to be covered by the algebra and
  algorithms.

12
PD and indefinite case
B is positive definite
B is indefinite
13
Lower dimensional target space
14
Lower dimensional target space
15
Indefinite constraints
Lower dimensional target space
Full dimensional target space
16
Parabola
17
Projections onto target space
B not canonical
B canonical
18
Fundamental Canonical Form

• (1) boils down to minz z s2 subject to
  z' G z k
• This gives Lagrangian form z s2 ?(z' G z
  k)
• With z (I ? G)-1 s, the constraint becomes
• In general, solutions found by solving this
  Lagrangian
• Feasible region (FR)
• When B is indefinite 1/?1 ? 1/?p
• When B is p.(s.)d. 8 ? 1/?p
• f(?) increases monotonically in the FR
• If s1 or sp are zero, adaptations are necessary

19
Lagrangian forms
B indefinite
B p.(s.)d.
20
Lagrangian forms phantom asymptotes
root
s2 0
s1 0
21
Movement from the origin
22
Movement from the origin
23
Movement from the origin
24
Movement along the major axis
25
Movement along the major axis
26
Conclusions

• Equation (1) subsumes many statistical problems.
• A unified methodology eliminates examination of
  many special cases.
• Geometry helps understanding algebra helps
  detailed analysis and provides essential
  underpinning for a general purpose algorithm.
• By identifying potential pathological situations,
  the algorithm can
• be made robust
• provide warnings.

27
Conclusions (informal)

• The unification is interesting and potentially
  useful.
• Its usefulness largely depends on the
  availability of a general purpose algorithm.
  Coming soon.
• Algorithms depend on detailed algebraic
  underpinning Done.
• Developing the algebra depends on understanding
  the geometry. Done

28
Some references

• C.J. Albers, F. Critchley, J.C. Gower, Quadratic
  Minimisation Problems in Statistics, 21st century
• M.W. Browne, On oblique Procrustes rotation,
  Psychometrika 32, 1967
• J.M.F. ten Berge, A generalization of Verhelsts
  solution for a constrained regression problem in
  ALSCAL and related MDS algorithms, Psychometrika
  48, 1983
• F. Critchley, On the minimisation of a positive
  definite quadratic form under quadratic
  constraints analytical solution and statistical
  applications. Warwick Statistics Research Report,
  1990
• M.J.R. Healy and H. Goldstein, An approach to the
  scaling of categorical attributes, Biometrika 63,
  1976
• J. de Leeuw, Generalized eigenvalue problems with
  psd matrices, Psychometrika 47, 1982
• J.J. Moré, Generalizations of the trust region
  problem, Optimization methods and software, Vol.
  II, 1993
• J.C. Gower G.B. Dijksterhuis, Procrustes
  Problems, Oxford University Press, 2004