Constrained%20Optimization

1
Constrained Optimization

• Rong Jin

2
Outline

• Equality constraints
• Inequality constraints
• Linear Programming
• Quadratic Programming

3
Optimization Under Equality Constraints

• Maximum Entropy Model
• English in ? French
• dans (1), en (2), à (3), au cours de (4),
  pendant (5)

4
Reducing variables

• Representing variables using only p1 and p4
• Objective function is changed
• Solution p1 0.2, p2 0.3, p3 0.1, p4 0.2,
  p5 0.2

5
Maximum Entropy Model for Classification

• It is unlikely that we can use the previous
  simple approach to solve such a general
• Solution Lagrangian

6
Equality Constraints Lagrangian

• Introduce a Lagrange multiplier ? for the
  equality constraint
• Construct the Lagrangian
• Necessary condition
• A optimal solution for the original optimization
  problem has to be one of the stationary point of
  the Lagrangian

7
Example

• Introduce a Lagrange multiplier ? for constraint
• Construct the Lagrangian
• Stationary points

8
Lagrange Multipliers

• Introducing a Lagrange multiplier for each
  constraint
• Construct the Lagrangian for the original
  optimization problem

9
Lagrange Multiplier

• We have more variables
• p1, p2, p3, p4, p5 and, ?1, ?2, ?3
• Necessary condition (first order condition)
• A local/global optimum point for the original
  constrained optimization problem ? a stationary
  point of the corresponding Lagrangian

10
Stationary Points for Lagrangian
All probabilities p1, p2, p3, p4, p5 are
expressed as functions of Lagrange multipliers ?s
11
Dual Problem

• p1, p2, p3, p4, p5 are expressed as functions of
  ?s
• We can even remove the variable ?3
• Put together necessary condition
• Still difficult to solve

12
Dual Problem

• p1, p2, p3, p4, p5 are expressed as functions of
  ?s
• We can even remove the variable ?3
• Put together necessary condition
• Still difficult to solve

13
Dual Problem

• Dual problem
• Substitute the expression for ps into the
  Lagrangian
• Find the ?s that MINIMIZE the substituted
  Lagrangian

14
Dual Problem
Original Lagrangian
Finding ?s such that the above objective function
is minimized
15
Dual Problem

• Using dual problem
• Constrained optimization ? unconstrained
  optimization
• Need to change maximization to minimization
• Only valid when the original optimization problem
  is convex/concave (strong duality)

x? When convex/concave
16
Maximum Entropy Model for Classification

• Introduce a Lagrange multiplier for each linear
  constraint

17
Maximum Entropy Model for Classification

• Construct the Lagrangian for the original
  optimization problem

18
Stationary Points
Conditional Exponential Model !
19
Dual Problem
20
Dual Problem
21
Dual Problem
22
Dual Problem
What is wrong?
23
Dual Problem
24
Dual Problem
25
Dual Problem
26
Dual Problem
27
Dual Problem
Minimizing L ? maximizing the log-likelihood
28
Support Vector Machine

• Having many inequality constraints
• Solving the above problem directly could be
  difficult
• Many variables w, b, ?
• Unable to use nonlinear kernel function

29
Inequality Constraints Modified Lagrangian

• Introduce a Lagrange multiplier ? for the
  inequality constraint
• Construct the Lagrangian
• Karush-Kuhn-Tucker (KKT) condition
• A optimal solution for the original optimization
  problem will satisfy the following conditions

30
Example

• Introduce a Lagrange multiplier ? for constraint
• Construct the Lagrangian
• KT conditions

• Expressing objective function using ?
• Solution is ?3

31
Example

• Introduce a Lagrange multiplier ? for constraint
• Construct the Lagrangian
• KT conditions

• Expressing objective function using ?
• Solution is ?3

32
Example

• Introduce a Lagrange multiplier ? for constraint
• Construct the Lagrangian
• KKT conditions

33
SVM Model

• Lagrange multipliers for inequality constraints

34
SVM Model

• Lagrangian for SVM model
• Karush-Kuhn-Tucker condition

35
SVM Model

• Lagrangian for SVM model
• Karush-Kuhn-Tucker condition

36
Dual Problem for SVM

• Express w, b, ? using ? and ?

37
Dual Problem for SVM

• Express w, b, ? using ? and ?
• Finding solution satisfying KKT conditions is
  difficult

38
Dual Problem for SVM

• Rewrite the Lagrangian function using only ? and
  ?
• Simplify using KT conditions

39
Dual Problem for SVM

• Final dual problem

40
Quadratic Programming
Find
Subject to
41
Linear Programming

• Very very useful algorithm
• 1300 papers
• 100 books
• 10 courses
• 100s of companies
• Main methods
• Simplex method
• Interior point method

Most important how to convert a general problem
into the above standard form
42
Example

• Need to change max to min

43
Example

• Need change ? to ?

44
Example

• Need to convert the inequality

45
Example

• Need change x3