June 17 Lecture Outline EnvE 320

1
June 17 Lecture Outline EnvE 320

• Generalized overview of Nonlinear Programming
  (NLP) Methods

NLP

• Generalized for the purposes of this course
  more methods problem categorization

2
Multivariable NLP

• Rely heavily on calculus - a few theorems to
  remember
• In nonlinear problems, the optimal solution can
  occur at
•   at a stationary point (all first partial
  derivatives 0)
•   along the boundary of the feasible region
•   at a singular point (first partial
  derivatives do not exist)
• Solution methods for this course will focus on
  finding stationary points
• Two special subtypes
• unconstrained problems where the only constraints
  are that the decision variables are real numbers
• constrained problems more difficult, generally
  solve by simplifying constraints or converting to
  unconstrained problem

3
Unconstrained NLP

• General solution approach
• Find stationary points which are candidate
  solutions
• Classify each stationary point as a local
  minimum, maximum, neither or unknown based on the
  Hessian matrix.
• If possible, classify the objective function as
  convex or concave ? then it may be possible to
  claim that stationary point is the global
  optimum.
•  A point x having for all i 1,
  2, n is called a stationary point of f.

4
Classifying Stationary Points

• Definition The k th leading principal minor of
  an n x n matrix is the determinant of the k x k
  matrix obtained by deleting the last n-k rows and
  columns of the matrix.
• Examples

5
Classifying Stationary Points

• Let Hk be the k th leading principal minor of the
  Hessian matrix and n be the number of decision
  variables
• Theorems
• If Hk(x0) gt 0, k 1, 2, , n, then a stationary
  point x0 is a local minimum for an unconstrained
  NLP
• If for k 1, 2, , n, Hk(x0) is non-zero and has
  the same sign as (-1)k, then a stationary point
  x0 is a local maximum for an unconstrained NLP
• If Hn(x0) is non-zero and conditions in 1 and 2
  above do not hold, the stationary point x0 is not
  a local extremum (it is called a saddle point)

6
Classifying Stationary Points

• Example 1 Find all local minima, maxima and
  saddle points of

7
Classifying Stationary Points

• Upon classifying the stationary points, we may be
  able to claim an optimal solution was identified
  in an unconstrained problem with objective
  function f
• Theorems
• For a maximization problem, if f is concave, then
  the stationary point is an optimal solution to
  the NLP
• For a minimization problem, if f is convex, then
  the stationary point is an optimal solution to
  the NLP
• The Theorems above require proving concavity or
  convexity (proving this beyond this course)

8
Difficulties Determining Stationary Points

• Derivatives can be complicated
• Simultaneous nonlinear set of equations can be
  impossible to solve algebraically
• Problems can have numerous stationary points,
  e.g.

9
Common Unconstrained Multivariable Search Methods

• Use these when difficulty identifying stationary
  points
• These algorithms are the basis for most NLP
  solvers including excel
• Examples include
• Method of Steepest Descent (or Ascent) Gradient
  Search
• requires first partial derivatives
• Newtons Method
• requires first and second partial derivatives
• Both of the above use analytical functions for
  first and second derivatives. However,
  equivalent approaches using numerical derivatives
  exist (e.g. Quasi-Newton)

10
Nongradient Optimization Methods

• Simple examples you should remember
• Random Search randomly sample each decision
  variable from a uniform distribution between the
  upper and lower bounds. Random samples generated
  as
• xnew xmin U(0,1)xmax-xmin, where U(0,1) is
  a uniform random between 0 and 1 ? Note
  U(0,1) randu() in MS Excel
• Gridded Search evaluate all points on a
  multi-dimensional grid.
• with 10 decision variables (or dimensions), and
  dividing each range into 5 possible values
  requires 5 x 5 x x 5 510 ? 9.8 million
  objective function evaluations!
• Other methods popular in various Engineering
  fields
• Genetic Algorithms, Simulated Annealing etc.
• These methods are also called global optimization
  methods and heuristic optimization

11
Solving NLPs with Excel Solver

• Note that Excel uses the Generalized Reduced
  Gradient Method for nonlinear models
• Initial solution should be feasible not
  absolutely required
• How to find a feasible solution if not obvious?
• Random search
• create a new, easier optimization model to
  determine feasible solution (e.g. minimize the
  sum of squared constraint violations)
• As with all NLP solvers, the excel solver
  searches for a single local optimum close to the
  initial solution
• If multiple local optima exist, then multiple
  solver trials from multiple initial solutions
  required to increase chance of finding global
  optimum
• However, without determining sufficient
  conditions, solution is not gauranteed to be
  globally optimal ? report as best solution
  found

12
Solving NLPs with Excel Solver

• Example 1

13
Solving NLPs with Excel Solver

• Example 1 sensitivity information
• If you needed exact sensitivity information
  simply resolve the model and note the change in
  solution

14
Solving NLPs with Excel Solver

• Example 2
• see the Solver NLP.xls spreadsheet on the course
  website (Example 2)