The Smoothed Analysis of Algorithms

1
The Smoothed Analysis of Algorithms

• Daniel A. Spielman
• MIT

With Shang-Hua Teng (Boston University) John
Dunagan (Microsoft Research) and Arvind Sankar
(Goldman Sachs)
2
Outline
Why?
What?
The Simplex Method
Gaussian Elimination
Other Problems
Conclusion
3
Problem Heuristics that work in practice,
with no sound theoretical explanation
Exponential worst-case complexity, but works
in practice
Polynomial worst-case complexity, but much
faster in practice
Heuristic speeds up code, with poor results in
worst-case
4
Attempted resolution Average-case analysis

• Measure expected performance
• on random inputs

5
Random
is not typical
6
Critique of Average-case Analysis
Random objects have very special properties
with exponentially high probability

Actual inputs might not look random.
7
Smoothed Analysis a hybrid of worst and
average case
worst case average case
8
Smoothed Analysis a hybrid of worst and
average case
worst case average case smoothed
complexity
9
Smoothed Complexity
Interpolates between worst and average
case Considers neighborhood of every input If
low, all bad inputs are unstable
10
Complexity Landscape
worst case
run time
average case
input space
11
Smoothed Complexity Landscape (convolved with
Gaussian)
run time
smoothed complexity
input space
12
(No Transcript)
13
Classical Example Simplex Method for Linear
Programming
max s.t.
Worst-Case exponential Average-Case
polynomial Widely used in practice
14
The Diet Problem
Min s.t.
15
Classical Example Simplex Method for Linear
Programming
max s.t.
Worst-Case exponential Average-Case
polynomial Widely used in practice
16
The Simplex Method
opt
start
17
Smoothed Analysis of Simplex Method
G is Gaussian
Theorem For all A, b, c, simplex method takes
expected time polynomial in
18
Analysis of Simplex Method
Using Shadow-Vertex Pivot Rule
19
Shadow vertex pivot rule
start
objective
20
The Polar of a Polytope
21
Polar Form of Linear Programming
c
max ? ?c Î ConvexHull(a1, a2, ..., am)
22
Shadow vertex pivot rule, in polar
23
(No Transcript)
24
Count facets by discretizingto N directions, N ?8
25
Count pairs in different facets


Different Facets
lt c/N
Pr
So, expect c Facets
26
Unlikely cone has small angle
27
Angle
Distance
28
Isolate on one Simplex
29
Smoothed Analysis of Simplex Method
G is Gaussian
Theorem For all A, b, c, simplex method takes
expected time polynomial in
30
Interior Point Methods for Linear Programming
Analysis Method Iterations

Observation Worst-Case, upper Worst-Case,
lower Average-Case Smoothed, upper
( )
Dunagan-S-Teng, S-Teng
Conjecture
31
Gaussian Elimination for Ax b
gtgt A randn(2) A -0.4326 0.1253
-1.6656 0.2877 gtgt b randn(2,1) b
-1.1465 1.1909
gtgt x A \ b x -5.6821 -28.7583 gtgt
norm(Ax - b) ans 8.0059e-016
32
Gaussian Elimination for Ax b
gtgt A 2eye(70) - tril(ones(70)) gtgt A(,70)
1 gtgt b randn(70,1) gtgt x A \ b gtgt norm(Ax
- b) ans 3.5340e004
Failed!
Perturb A
gtgt Ap A randn(70) / 109 gtgt x Ap \ b gtgt
norm(Apx - b) ans 5.8950e-015
33
Gaussian Elimination for Ax b
34
Gaussian Elimination for Ax b
35
Gaussian Elimination for Ax b
36
Gaussian Elimination with Partial Pivoting
Fast heuristic for maintaining precision, by
trying to keep entries small
37
Gaussian Elimination with Partial Pivoting
Fast heuristic for maintaining precision, by
trying to keep entries small Pivot not just on
zeros, but to move up entry of largest
magnitude
38
Gaussian Elimination with Partial Pivoting
Gaussian elimination with partial pivoting is
utterly stable in practice. In fifty years of
computing, no matrix problems that excite an
explosive instability are know to have arisen
under natural circumstances Matrices with
large growth factors are vanishingly rare in
applications.
Nick Trefethen
39
Gaussian Elimination with Partial Pivoting
Gaussian elimination with partial pivoting is
utterly stable in practice. In fifty years of
computing, no matrix problems that excite an
explosive instability are know to have arisen
under natural circumstances Matrices with
large growth factors are vanishingly rare in
applications.
Nick Trefethen
Theorem
Sankar-S-Teng
40
Mesh Generation
Parallel complexity of Rupperts Delaunay
refinement is O( (log n/s)2)
Spielman-Teng-Üngör
41
Other Smoothed Analyses
Perceptron
Blum-Dunagan Quicksort
Banderier-Beier-Mehlhorn Parallel
connectivity in digraphs Frieze-Flaxman Comple
x Gaussian Elimination Yeung Smoothed
analysis of K(A)
Wschebor On smoothed analysis in dense graphs
and formulas
Krivelevich-Sudakov-Tetali
Smoothed Number of Extreme Points under Uniform
Noise
Damerow-Sohler Typical
Properties of Winners and Losers in Discrete
Optimization Beier-Vöcking
Multi-Level Feedback scheduling
Becchetti-Leonardi-Marchetti-Shäfer-
Vredeveld Smoothed motion complexity Damerow,
Meyer auf der Heide, Räcke, Scheideler, Sohler

42
Future Smoothed Analyses

• Multilevel graph partitioning
• Smoothed Analysis of Chaco and Metis
• Differential Evolution
• and other optimization heuristics
• Computing Nash Equilibria

43
Future Smoothed Analyses
Perturb less! Preserve zeros Preserve
magnitudes of numbers Property-preserving
perturbations More Discrete smoothed
analyses New algorithms

For more, see the Smoothed Analysis Homepage