central path behavior

1
introduction central path behavior redundant
Klee-Minty cube refined redundant Klee-Minty
cube iteration-complexity sketch of the proof
How good are interior point methods?
A. Deza E. Nematollahi T. Terlaky
October 2005
How good are interior point methods?
Eissa Nematollahi, McMaster University
2
introduction central path behavior redundant
Klee-Minty cube refined redundant Klee-Minty
cube iteration-complexity sketch of the proof
Introduction linear optimization
simplex methods Klee-Minty cubes
interior point methods Central path behavior
redundancy effect how curly the central path
can be Redundant Klee-Minty cubes
redundancy Klee-Minty cubes playing with
numbers (intuition) feasibility and
solution iteration-complexity Refined
redundant Klee-Minty cubes refined
redundancy Klee-Minty cubes playing with
numbers (intuition) feasibility and solution
Iteration-complexity tightening the
iteration-complexity gap closing the
iteration-complexity gap Sketch of the proof
Eissa Nematollahi, McMaster University
3
introduction central path behavior redundant
Klee-Minty cube refined redundant Klee-Minty
cube iteration-complexity sketch of the proof
Simplex method linear optimization Klee-Minty
cube interior point methods
linear optimization, simplex methods (1947)
standard form for linear optimization
problem min subject to where A
has full row rank.
simplex methods

• start from a feasible basis
• use a pivot rule
• find an optimal solution (after finite number of
  iterations)
• most pivot rules variants are known to be
  exponential
• nevertheless very efficient implementations
  exist.

Eissa Nematollahi, McMaster University
4
introduction central path behavior redundant
Klee-Minty cube refined redundant Klee-Minty
cube iteration-complexity sketch of the proof
Simplex method linear optimization Klee-Minty
cube interior point methods
some linear optimization events
Eissa Nematollahi, McMaster University
5
introduction central path behavior redundant
Klee-Minty cube refined redundant Klee-Minty
cube iteration-complexity sketch of the proof
Simplex method linear optimization Klee-Minty
cube interior point methods
Klee-Minty worst-case example for simplex
methods (1972)
simplex methods may take 2n - 1 pivots to reach
the optimum on Klee-Minty cubes (the edge-path
followed by the simplex method visits all the 2n
vertices)
min subject to
for
Klee-Minty 2-cube
Klee-Minty 3-cube
n variables 2n constrains
Eissa Nematollahi, McMaster University
6
introduction central path behavior redundant
Klee-Minty cube refined redundant Klee-Minty
cube iteration-complexity sketch of the proof
Simplex method linear optimization Klee-Minty
cube interior point methods
Klee-Minty worst-case example for simplex
methods (1972)
simplex methods may take 2n - 1 pivots to reach
the optimum on Klee-Minty cubes (the edge-path
followed by the simplex method visits all the 2n
vertices)
min subject to
for
Klee-Minty 2-cube
Klee-Minty 3-cube
n variables 2n constrains
Eissa Nematollahi, McMaster University
7
introduction central path behavior redundant
Klee-Minty cube refined redundant Klee-Minty
cube iteration-complexity sketch of the proof
Simplex method linear optimization Klee-Minty
cube interior point methods
central path following interior point
methods (1985)

• start from the analytic center
• follow the central path
• converge to an optimal solution.
• are polynomial time algorithms for linear
  optimization
• number of iterations
• N number of inequalities
• L input-data bit-length
• ? central path parameter

analytic center
central path
optimal solution
Eissa Nematollahi, McMaster University
8
introduction central path behavior redundant
Klee-Minty cube refined redundant Klee-Minty
cube iteration-complexity sketch of the proof
redundancy effect how curly the central path can
be
how redundancy can affect the central path?
Eissa Nematollahi, McMaster University
9
introduction central path behavior redundant
Klee-Minty cube refined redundant Klee-Minty
cube iteration-complexity sketch of the proof
redundancy effect how curly the central path can
be
how curly can the central path be?
Q Can the central path be bent along the
edge-path followed by the simplex method on
the Klee-Minty cube? (can the central path
visit an arbitrary small neighborhood of all 2n
vertices?)
Starting point
1
0.5
Yes! - if
Optimal point
we carefully add an exponential
number of redundant constrains
0
0
1
0.5
0.5
0
1
Eissa Nematollahi, McMaster University
10
introduction central path behavior redundant
Klee-Minty cube refined redundant Klee-Minty
cube iteration-complexity sketch of the proof
redundant Klee-Minty construction playing with
numbers feasibility solution iteration-complexity
redundant Klee-Minty construction

• ? small positive factor by which the
  Klee-Minty cube is squashed
• ? size of the neighborhood
• h (h1, . . . , hn ) number of redundant
  constraints
• d distance of the redundant constraints

h1
??
?
d
h2
d
we assume that ? ? and ? lt ? n
Eissa Nematollahi, McMaster University
11
introduction central path behavior redundant
Klee-Minty cube refined redundant Klee-Minty
cube iteration-complexity sketch of the proof
redundant Klee-Minty construction playing with
numbers feasibility solution iteration-complexity
redundant Klee-Minty construction
min subject to
for
repeated h1 times repeated h2 times repeated hn
times
n variables 2n h1 h2 hn constrains
Eissa Nematollahi, McMaster University
12
introduction central path behavior redundant
Klee-Minty cube refined redundant Klee-Minty
cube iteration-complexity sketch of the proof
redundant Klee-Minty construction playing with
numbers feasibility solution iteration-complexity
playing with numbers (animations)
Animation 1 effect of d Animation 2
effect of h (h1, . . . , hn ) Animation 3
effect of ? Animation 4 effect of ?
Eissa Nematollahi, McMaster University
13
introduction central path behavior redundant
Klee-Minty cube refined redundant Klee-Minty
cube iteration-complexity sketch of the proof
redundant Klee-Minty construction playing with
numbers feasibility solution iteration-complexity
playing with numbers (animations)
Animation 1 effect of d Animation 2
effect of h (h1, . . . , hn ) Animation 3
effect of ? Animation 4 effect of ?
Eissa Nematollahi, McMaster University
14
introduction central path behavior redundant
Klee-Minty cube refined redundant Klee-Minty
cube iteration-complexity sketch of the proof
redundant Klee-Minty construction playing with
numbers feasibility solution iteration-complexity
playing with numbers (animations)
Animation 1 effect of d Animation 2
effect of h (h1, . . . , hn ) Animation 3
effect of ? Animation 4 effect of ?
Eissa Nematollahi, McMaster University
15
introduction central path behavior redundant
Klee-Minty cube refined redundant Klee-Minty
cube iteration-complexity sketch of the proof
redundant Klee-Minty construction playing with
numbers feasibility solution iteration-complexity
playing with numbers (animations)
Animation 1 effect of d Animation 2
effect of h (h1, . . . , hn ) Animation 3
effect of ? Animation 4 effect of ?
Eissa Nematollahi, McMaster University
16
introduction central path behavior redundant
Klee-Minty cube refined redundant Klee-Minty
cube iteration-complexity sketch of the proof
redundant Klee-Minty construction playing with
numbers feasibility solution iteration-complexity
playing with numbers (animations)
Animation 1 effect of d Animation 2
effect of h (h1, . . . , hn ) Animation 3
effect of ? Animation 4 effect of ?
Eissa Nematollahi, McMaster University
17
introduction central path behavior redundant
Klee-Minty cube refined redundant Klee-Minty
cube iteration-complexity sketch of the proof
redundant Klee-Minty construction playing with
numbers feasibility solution iteration-complexity
playing with numbers (intuition)

• d should be big enough
• hk1 should be big enough vs hk (h1 lt h2 lt . .
  . lt hn )
• hk should be big enough vs ? hk 1 (h1 gt h2?
  gt . . . gt hn?n -1)
• hn (or h1) should be big enough
• the smaller ? and/or ? are, the bigger h has to
  be

Eissa Nematollahi, McMaster University
18
introduction central path behavior redundant
Klee-Minty cube refined redundant Klee-Minty
cube iteration-complexity sketch of the proof
redundant Klee-Minty construction playing with
numbers feasibility solution iteration-complexity
how to choose h and d

• d ? n2n1
• h should satisfy

where
0
0
0
? ? ?
0
0
0
? ? ?
0
0
? ? ?
A
0
0
? ? ?
? ? ?
? ? ?
? ? ?
? ? ?
? ? ?
? ? ?
? ? ?

• the system admits solutions

Eissa Nematollahi, McMaster University
19
introduction central path behavior redundant
Klee-Minty cube refined redundant Klee-Minty
cube iteration-complexity sketch of the proof
redundant Klee-Minty construction playing with
numbers feasibility solution iteration-complexity
what we obtained

• redundant Klee-Minty example for interior point
  methods
• 2n 2 sharp turns ? iterations
• but exponential number of redundant constrains
• d ? n2n1
• number of constraints
• input-data bit-length

Eissa Nematollahi, McMaster University
20
introduction central path behavior redundant
Klee-Minty cube refined redundant Klee-Minty
cube iteration-complexity sketch of the proof
redundant Klee-Minty construction playing with
numbers feasibility solution iteration-complexity
iteration-complexity bounds for redundant
Klee-Minty cubes

• theoretical iteration-complexity upper bound
• redundant Klee-Minty iteration-complexity lower
  bound
• ? iterations ?
• Q. Can we tighten the iteration-complexity
  bounds
• (while keeping ??(2n) lower bound, can
  we reach O(2n) upper bound?)

Eissa Nematollahi, McMaster University
21
introduction central path behavior redundant
Klee-Minty cube refined redundant Klee-Minty
cube iteration-complexity sketch of the proof
refined redundant Klee-Minty construction playing
with numbers feasibility solution
refined redundant Klee-Minty construction

• ? small positive factor by which the
  Klee-Minty cube is squashed
• ? size of the neighborhood
• h (h1, . . . , hn ) number of redundant
  constraints
• d (d1, . . . , dn ) distances of the
  redundant constraints

h1
??
?
d1
h2
d2
Eissa Nematollahi, McMaster University
22
introduction central path behavior redundant
Klee-Minty cube refined redundant Klee-Minty
cube iteration-complexity sketch of the proof
refined redundant Klee-Minty construction playing
with numbers feasibility solution
playing with numbers (intuition)

• ? should be closer to and
• ? not too tiny
• Further
• dn should be big enough and d1 gt d2 gt . . . gt dn
• hk 1 should be big enough vs hk (h1
  lt h2 lt . . . lt hn )
• hk should be big enough vs ? hk 1 (h1 gt
  h2? gt . . . gt hn? n-1)
• hn (or h1) should be big enough

Eissa Nematollahi, McMaster University
23
introduction central path behavior redundant
Klee-Minty cube refined redundant Klee-Minty
cube iteration-complexity sketch of the proof
refined redundant Klee-Minty construction playing
with numbers feasibility solution
how to choose (refined) h and (refined) d

• ? positive factor by which the
  Klee-Minty cube is squashed
• ? ? size of the neighborhood
• h (h1, . . . , hn ) number of redundant
  constraints
• d (d1, . . . , dn ) with dk corresponding to
  (original) d for dimension n k 3

h1
??
?
d1
h2
d2
Eissa Nematollahi, McMaster University
Eissa Nematollahi, McMaster University
24
introduction central path behavior redundant
Klee-Minty cube refined redundant Klee-Minty
cube iteration-complexity sketch of the proof
refined redundant Klee-Minty construction playing
with numbers feasibility solution
how to choose (refined) h and (refined) d

• d n(2n 3,,2n - k 4,,24)
• h should satisfy

where
0
0
0
? ? ?
0
0
? ? ?
0
0
? ? ?
A
? ? ?
? ? ?
? ? ?
? ? ?
? ? ?
? ? ?
0
0
? ? ?
0
0
? ? ?

• the system admits solutions

Eissa Nematollahi, McMaster University
25
introduction central path behavior redundant
Klee-Minty cube refined redundant Klee-Minty
cube iteration-complexity sketch of the proof
refined redundant Klee-Minty construction playing
with numbers feasibility solution
what we obtained (refined)

• refined redundant Klee-Minty example for
  interior point methods
• still 2n 2 sharp turns ?
  iterations
• but fewer (still exponential) redundant
  constrains
• number of constraints
• input-data bit-length

Eissa Nematollahi, McMaster University
Eissa Nematollahi, McMaster University
26
introduction central path behavior redundant
Klee-Minty cube refined redundant Klee-Minty
cube iteration-complexity sketch of the proof
tightening iteration-complexity gap closing the
iteration-complexity gap
refined iteration-complexity bounds for redundant
Klee-Minty cubes (1)

• theoretical iteration-complexity upper bound
• refined redundant Klee-Minty iteration-complexity
  lower bound
• Q. Can we further tighten the iteration-complexit
  y bounds
• (can we get an L-independent
  iteration-complexity?)

? iterations ?
Eissa Nematollahi, McMaster University
27
introduction central path behavior redundant
Klee-Minty cube refined redundant Klee-Minty
cube iteration-complexity sketch of the proof
tightening iteration-complexity gap closing the
iteration-complexity gap
refined iteration-complexity bounds for redundant
Klee-Minty cubes (1)

• theoretical iteration-complexity upper bound
• refined redundant Klee-Minty iteration-complexity
  lower bound
• Q. Can we further tighten the iteration-complexit
  y bounds
• (can we get an L-independent
  iteration-complexity?)
• Yes! as we can set
• central path parameter at stopping point
• central path parameter at starting point
• L-independent iteration-complexity

? iterations ?
Eissa Nematollahi, McMaster University
28
introduction central path behavior redundant
Klee-Minty cube refined redundant Klee-Minty
cube iteration-complexity sketch of the proof
tightening iteration-complexity gap closing the
iteration-complexity gap
refined iteration-complexity bounds for redundant
Klee-Minty cubes (2)

• theoretical iteration-complexity upper bound
• refined redundant Klee-Minty iteration-complexit
  y lower bound
• the gap is essentially closed
• ? iterations ?

Todd-Ye (1996) ?
iterations ?

• additional questions
• how are condition numbers affected by
  redundancy?
• how ? changes along the linear and curly parts
  of the central path?
• can the central path follow any simplex path for
  any polyhedron?

Eissa Nematollahi, McMaster University
29
introduction central path behavior redundant
Klee-Minty cube refined redundant Klee-Minty
cube iteration-complexity sketch of the proof
simplex path tunnel optimality conditions inequali
ties and estimates
sketch of the proof (1)
simplex path tunnel
where
Eissa Nematollahi, McMaster University
30
introduction central path behavior redundant
Klee-Minty cube refined redundant Klee-Minty
cube iteration-complexity sketch of the proof
simplex path tunnel optimality conditions inequali
ties and estimates
sketch of the proof (1)
simplex path tunnel
where
A
2 ?
Eissa Nematollahi, McMaster University
31
introduction central path behavior redundant
Klee-Minty cube refined redundant Klee-Minty
cube iteration-complexity sketch of the proof
simplex path tunnel optimality conditions inequali
ties and estimates
sketch of the proof (1)
simplex path tunnel
where
A
2 ?
A
3 ?
Eissa Nematollahi, McMaster University
32
introduction central path behavior redundant
Klee-Minty cube refined redundant Klee-Minty
cube iteration-complexity sketch of the proof
simplex path tunnel optimality conditions inequali
ties and estimates
sketch of the proof (1)
simplex path tunnel
where

A
2 ?
A
3 ?
P
?
P0 simplex path (? 0)
Eissa Nematollahi, McMaster University
33
introduction central path behavior redundant
Klee-Minty cube refined redundant Klee-Minty
cube iteration-complexity sketch of the proof
simplex path tunnel optimality conditions inequali
ties and estimates
sketch of the proof (2)

with and

• Klee-Minty cube

central path P belongs to simplex path tunnel P?
Eissa Nematollahi, McMaster University
34
introduction central path behavior redundant
Klee-Minty cube refined redundant Klee-Minty
cube iteration-complexity sketch of the proof
simplex path tunnel optimality conditions inequali
ties and estimates
sketch of the proof (3)

• analytic center is solution

• notations

and

• optimality conditions (at optimality gradient
  0)

Eissa Nematollahi, McMaster University
35
introduction central path behavior redundant
Klee-Minty cube refined redundant Klee-Minty
cube iteration-complexity sketch of the proof
simplex path tunnel optimality conditions inequali
ties and estimates
sketch of the proof (4)

• inequalities and estimates

• conditions on d and h for bending the central
  path

?
?
and
Eissa Nematollahi, McMaster University