NLP

1
NLP

• KKT Practice and Second Order Conditions from
  Nash and Sofer

2
Unconstrained

• First Order Necessary Condition
• Second Order Necessary
• Second Order Sufficient

3
Easiest Problem

• Linear equality constraints

4
KKT Conditions

• Note for equality multipliers are
  unconstrained
• Complementarity not an issue

5
Null Space Representation

• Let x be a feasible point, Axb.
• Any other feasible point can be written as
  xxp where Ap0
• The feasible region
• x xp p?N(A)
• where N(A) is null space of A

6
Null and Range Spaces
See Section 3.2 of Nash and Sofer for example
7
Orthogonality
8
Null Space Review
9
Constrained to Unconstrained

• You can convert any linear equality constrained
  optimization problem to an equivalent
  unconstrained problem
• Method 1 substitution
• Method 2 using Null space representation and a
  feasible point.

10
Example

• Solve by substitution
• becomes

11
Null Space Method

• x 4 0 0
• xxZv
• becomes

12
General Method

• There exists a Null Space Matrix
• The feasible region is
• Equivalent Reduced Problem

13
Optimality Conditions

• Assume feasible point and convert to null space
  formulation

14
Where is KKT?

• KKT implies null space
• Null Space implies KKT
• Gradient is not in Null(A), thus it must be
  in Range(A)

15
Lemma 14.1 Necessary Conditions

• If x is a local min of f over xAxb, and Z is
  a null matrix
• Or equivalently use KKT Conditions

16
Lemma 14.2 Sufficient Conditions

• If x satisfies (where Z is a basis matrix for
  Null(A))
• then x is a strict local minimizer

17
Lemma 14.2 Sufficient Conditions (KKT form)

• If (x,?) satisfies (where Z is a basis matrix
  for Null(A))
• then x is a strict local minimizer

18
Lagrangian Multiplier

• ? is called the Lagrangian Multiplier
• It represents the sensitivity of solution to
  small perturbations of constraints

19
Optimality conditions

• Consider min (x24y2)/2 s.t. x-y10

20
Optimality conditions

• Find KKT point Check SOSC

21
In Class Practice

• Find a KKT point
• Verify SONC and
• SOSC

22
Linear Equality Constraints - I
23
Linear Equality Constraints - II
24
Linear Equality Constraints - III
so SOSC satisfied, and x is a strict local
minimum Objective is convex, so KKT conditions
are sufficient.
25
Next Easiest Problem

• Linear equality constraints
• Constraints form a polyhedron

26
Close to Equality Case
Equality FONC
a2x b
a2x b
-a2
Polyhedron Axgtb
a3x b
-a1
a4x b
a1x b
contour set of function
unconstrained minimum
Which ?i are 0? What is the sign of ?I?
27
Close to Equality Case
Equality FONC
a2x b
a2x b
-a2
Polyhedron Axgtb
a3x b
-a1
a4x b
a1x b
Which ?i are 0? What is the sign of ?I?
28
Inequality Case
Inequality FONC
a2x b
a2x b
-a2
Polyhedron Axgtb
a3x b
-a1
a4x b
a1x b
Nonnegative Multipliers imply gradient points to
the less than Side of the constraint.
29
Lagrangian Multipliers
30
Lemma 14.3 Necessary Conditions

• If x is a local min of f over xAxb, and Z is
  a null-space matrix for active constraints then
  for some vector ?

31
Lemma 14.5 Sufficient Conditions (KKT form)

• If (x,?) satisfies

32
Lemma 14.5 Sufficient Conditions (KKT form)

• where Z is a basis matrix for Null(A ) and A
  corresponds to nondegenerate active constraints)
• i.e.

33
Sufficient Example

• Find solution and verify SOSC

34
Linear Inequality Constraints - I
35
Linear Inequality Constraints - II
36
Linear Inequality Constraints - III
37
Linear Inequality Constraints - IV
38
Example

• Problem

39
You Try

• Solve the problem using above theorems

40
Why Necessary and Sufficient?

• Sufficient conditions are good for?
• Way to confirm that a candidate point
• is a minimum (local)
• Butnot every min satisifies any given SC
• Necessary tells you
• If necessary conditions dont hold then you
  know you dont have a minimum.
• Under appropriate assumptions, every point that
  is a min satisfies the necessary cond.
• Good stopping criteria
• Algorithms look for points that satisfy Necessary
  conditions

41
General Constraints

42
Lagrangian Function

• Optimality conditions expressed using
• Lagrangian function
• and Jacobian matrix
• were each row is a gradient of a constraint

43
Theorem 14.2 Sufficient Conditions Equality (KKT
form)

• If (x,?) satisfies

44
Theorem 14.4 Sufficient Conditions Inequality
(KKT)

• If (x,?) satisfies

45
Lemma 14.4 Sufficient Conditions (KKT form)

• where Z is a basis matrix for Null(A ) and A
  corresponds to Jacobian of nondegenerate active
  constraints)
• i.e.

46
Sufficient Example

• Find solution and verify SOSC

47
Nonlinear Inequality Constraints - I
48
Nonlinear Inequality Constraints - II
49
Nonlinear Inequality Constraints - III
50
Sufficient Example

• Find solution and verify SOSC

51
Nonlinear Inequality Constraints - V
52
Nonlinear Inequality Constraints - VI
53
Theorem 14.1 Necessary Conditions- Equality

• If x is a local min of f over xg(x)0, Z is a
  null-space matrix of the Jacobian ?g(x), and x
  is a regular point then

54
Theorem 14.3 Necessary Conditions

• If x is a local min of f over xg(x)gt0, Z is
  a null-space matrix of the Jacobian ?g(x), and
  x is a regular point then

55
Regular point

• If x is a regular point with respect to the
  constraints g(x) if the gradient of the active
  constraints are linearly independent.
• For equality constraints, all constraints are
  active so
• should have linearly independent rows.

56
Necessary Example

• Show optimal solution x1,0
• is regular and find KKT point

57
Constraint Qualifications

• Regularity is an example of a constraint
  qualification CQ.
• The KKT conditions are based on linearizations of
  the constraints.
• CQ guarantees that this linearization is not
  getting us into trouble. Problem is
• KKT point might not exist.
• There are many other CQ,e.g., for inequalities
  Slater is there exists g(x)lt0.
• Note CQ not needed for linear constraints.

58
KKT Summary
X is global min
Convex f Convex constraints
X is local min
SOSC
CQ
KKT Satisfied