The Most Important Concept in

1
The Most Important Concept in Optimization
(minimization)

• A point is said to be an optimal solution of a
• unconstrained minimization if there exists no
• decent direction

• A point is said to be an optimal solution of a
• constrained minimization if there exists no
• feasible decent direction

• There might exist decent direction but move
• along this direction will leave out the
  feasible
• region

2
Decent Direction of

• Move alone the decent direction for a certain
• stepsize will decrease the objective function
  value
• i.e.,

3
Feasible Direction of

• Move alone the feasible direction from for a
  
• certain stepsize will not leave the feasible
  region
• i.e.,

4
(No Transcript)
5
Steep Decent with Exact Line Search
Start with any
. Having
stop if
(i) Steep Decent Direction
(ii) Finding Stepsize by Exact Line Search
6
Minimum Principle
be a convex and differentiable function
Let
be the feasible region.
Example
7
(No Transcript)
8
Minimization Problem
vs.
Kuhn-Tucker Stationary-point Problem
9
Lagrangian Function
Let
and
are convex then

• If

is convex.

• For a fixed

, if
then

• Above result is a sufficient condition if

is convex.
10
KTSP with Equality Constraints?
such that
KTSP
Find
11
KTSP with Equality Constraints
Find
such that
KTSP
12
Generalized Lagrangian Function
is convex.
then
13
Lagrangian Dual Problem
subject to
14
Lagrangian Dual Problem
where
15
Weak Duality Theorem
be a feasible solution of the primal
Let
problem and
a feasible solution of the
dual problem. Then
16
Weak Duality Theorem
solve the primal and dual problem respectively.
In this case,
17
Saddle Point of Lagrangian
satisfying
Let
is called
Then
The saddle point of the Lagrangian function
18
Dual Problem of Linear Program
Primal LP
subject to
Dual LP
subject to

• All duality theorems hold and work perfectly!

19
Application of LP Duality (I)
Farkas Lemma
For any matrix
and any vector
either
or
but never both.
20
Application of LP Duality (II)
LSQ-Normal Equation Always Has a Solution
For any matrix
and any vector
consider
Claim
always has a solution.
21
Dual Problem of Strictly Convex Quadratic Program
Primal QP
subject to
With strictly convex assumption, we have
Dual QP