Optimization

1
Optimization

• KE-42.4520
• Process modeling methods and tools
• Kari I. Keskinen
• 31.3.2008

2
Introduction

• Optimization is one of the keywords in many
  design problems.
• Here we will review
• what is optimization,
• how to formulate an optimization problem,
• how constraints are handled,
• classification of optimization problems,
• selection of the optimization method,
• operation principle of some of the methods,
• problems met in optimization.

3
The optimum

• The optimum is expressed as the minimum value
  (e.g. costs) or the maximum value (e.g.
  profitability) of the function that measures the
  desired goal.
• Mathematically a maximum value of a function is
  obtained with the same independent variable value
  than the minimum value of the same function
  multiplied with -1.

4
The optimum 2
5
The optimum 3

• Variable x is called independent variable, i.e.
  its value can be freely chosen.
• In optimization independent variables are called
  decision variables.
• There are often limits or constraints for the
  independent variable
• Upper and lower limits
• For example a mole fraction can not be less than
  zero or more than one.
• Temperature can not be less than 0 K, etc.

6
The optimum 4

• The independent variable value is to be chosen so
  that the function f(x) obtains its maximum or
  minimum. This value of the independent variable
  is then the optimum value xopt.
• Normally optimization programs look only the
  minimum value of the objective function f(x).
  Maximization of f(x) is then converted to
  minimization of -f(x).

7
The optimum 5
8
The optimum 6

• Often the problem of multiple minimum and/or
  maximum values are met.
• Normally, the target is to find a global optimum
  in the search interval.
• The problem gets even more worse when we have
  many decision variables. An objective function of
  two decision variables can be visualized, but for
  even more decision variables it is often
  difficult to imagine how the multidimensional
  surface will behave.

9
The optimum 7
10
Limits and constraints

• The constraints do not limit the value of the
  objective function.
• The independent variables are bound by limits and
  also often with some constraints.
• Most dependent model variables are bound by
  physical limits.
• Constraints can be
• Equality constraints, where more than one
  independent variable are bound together.
• These are actually additional model equations
  that can be used to eliminate some variables that
  are not anymore independent.

11
Limits and constraints 2

• Constraints are often inequalities
• There are special methods to treat different
  limits and constraints in optimization depending
  on the objective function type.

12
Conditions for a minimum or a maximum value of a
function

• The variable value of x in a minimum or a maximum
  (local) of function f(x) is given by
• The quality of the function value (extrema, i.e.
  minimum/ maximum) is given by the second
  derivative of the function evaluated at the
  stationary points x1

13
Conditions for a minimum or a maximum value of a
function 2

• If the second derivative expression calculated at
  point xi is
• positive, then the point is a minimum
• zero, then the point is an inflection point
• negative, then the point is a maximum

14
(No Transcript)
15
Exercise

• Calculate the minimum and maximum values and
  determine their quality for function

16
Exercise solution
17
Conditions for a minimum or a maximum value of a
function 3

• Correspondingly, for a function f(x) of several
  independent variables x
• Calculate and set it to zero.
• Solve the equation set to get a solution vector
  x.
• Calculate .
• Evaluate it at x.
• Inspect the Hessian matrix
• at point x.

18
Conditions for a minimum or a maximum value of a
function 4
An example of a Hessian.
19
Conditions for a minimum or a maximum value of a
function 5

• The Hessian matrix
• must be evaluated to determine the nature of
  function f(x).
• H is positive definite if and only if xTHx is gt0
  for all x?0.
• H is negative definite if and only if xTHx is lt0
  for all x?0.
• H is indefinite if xTHx lt0 for some x and gt0 for
  other x.

20
(No Transcript)
21
Conditions for a minimum or a maximum value of a
function 6
22
Conditions for a minimum or a maximum value of a
function 7

• The necessary condition (1 and 2 below) and the
  sufficient condition (3) to guarantee that x is
  an extremum
• f(x) is twice differentiable at x
• , that is, a stationary point
  exist at x
• is positive definite for a minimum
  to exist at x, and negative definite for a
  maximum to exist at x.
• Sometimes it is possible to have an extrema
  although, e.g. condition 3. is not satisfied

23
Conditions for a minimum or a maximum value of a
function 8

• Thus for utilizing the derivative method the
  function must be continuous and have continuous
  derivatives.

24
Discontinuities
Cost
25
Discontinuities 2

• Often the discontinuous functions are presented
  as continuous ones (e.g. pipe diameter can obtain
  all floating point values) and after the
  optimizations a final checking calculation can be
  carried out with two closest available values
  (e.g. two nearest standard pipe diameters).
• It is not cheap to order non-standard sizes that
  are not in production. These might be much more
  expensive than standard sizes.

26
Discontinuities 3

• Discontinuities arise from
• Only standard sizes of equipments are
  manufactured
• Temperature limit exceeded for a certain
  material, more expensive material has be used
• Pressure level increases just above the limit of
  certain pipe class leading to use of thicker pipe
  wall
• Small amounts of impurities are allowed for
  process streams, but in case there is e.g. too
  much chlorides then stainless steel can not used
  and more expensive material has to be selected

27
Discontinuities 4

• Maximum size of available equipment exceeded.
  Must have several units in series/parallel.
• Certain unit operations are optimal for their
  characteristic operation range. If we go for too
  small or too large scale then another unit
  operation can be much more economic.
• Sometimes the optimization is limited by a
  customer desire to have 20, 50 or 100 over
  design for future expansion. This is to be taken
  into account in the formulation of the problem.

28
Setting the goal for optimization

• This is the most crucial point in optimization.
  If not done properly the study must be repeated
  with correct starting values.
• Often we have to satisfy several objectives,
  which in many cases are contradictory/conflicting
  
• investment cost
• operation costs
• waste production (i.e. loss of valuable material
  in side-products)
• safety
• flexibility for different raw materials
• reliability
• energy usage
• sustainability
• CO2 production

29
Setting the goal for optimization 2

• Plant location?
• Logistics?
• Integration to existing units?
• Capacity? If too large usage of raw material,
  will it affect the raw material price and
  availability?
• Market studies? How much will sell and with what
  price?
• Technology maturity? Old well established
  technology will not give huge opportunities for
  advantages over competition.
• New technology has risks but give possible
  advantages over competition.

30
Setting the goal for optimization 3

• The target for the optimization must be well
  defined.
• There are uncertainties
• raw material price and availability
• product price and market situation
• politics (new regulations for products can even
  kill production, e.g. MTBE business in USA)
• investment project takes few years and the price
  of the construction materials can increase
  considerably
• new technology can be implemented by competitors
• the time value of money is difficult to predict
  will the inflation or currency rate changes
  affect the project

31
Setting the goal for optimization 4

• Once the above mentioned facts are taken into
  account and the corresponding items are fixed the
  mathematical treatment in the optimization study
  can take place.
• Uncertainties can be utilized on the optimized
  design to check the sensitivity of the result on
  them.
• calculate the objective function for the optimal
  design when a basic data is changed e.g. from -50
  50 (change the basic data one at a time)
• this gives an indication of the fact that is the
  most important for the profitability.

32
Setting the goal for optimization 5

• There are methods to combine contradictory or
  conflicting items together.
• Most common is to combine money from different
  time moments
• Investment costs
• Operation costs
• IRR, ROI, NPV, PBP
• One has to select the project total time (20
  years of operation?), rate of interest (15 ??)
• The selection of the method and its parameters
  affects the optimized result! This is due to the
  fact that we are combining items that are not
  directly comparable!

33
Setting the goal for optimization 6

• Sometimes the combination of multiple objectives
  into one objective function is called political
  objective function.
• There are cases where it is difficult to combine
  multiple objectives into one goal function
• e.g. how do we evaluate safety? (giving price on
  a lethal accident due to saving in investment on
  process safety issues and including that into the
  objective function???)
• we are not going in to more deeply on the
  multiple-criteria optimization

34
Alternatives in optimization
C
A
Operation cost/kg product
D
B
Capacity or investment cost
Process alternatives. The ellipses describe the
area that can be covered by changing process
parameters.
35
(No Transcript)
36
Role of optimization in process design

• The needs define the capacity, raw material,
  product etc.
• Synthesis stage is for creating the process
  concept (what unit operations)
• An initial process flowsheet is created
• Initial values for process parameters are
  obtained from literature, laboratory results etc.
• Decision variables are the process parameters to
  be optimized

37
Role of optimization in process design 2

• Analysis consists of three stages
• Heat and material balances. The flowsheeting
  programs in simulation mode are used for this.
• Sizing and costing follows once material and heat
  balances are done.
• Economic evaluation means the calculation of the
  objective function (investment costs operation
  costs using selected method NPV, IRR, ROI etc.
  and parameters years of operation, rate of
  interest, ).

38
Role of optimization in process design 3

• Once the objective function value is obtained,
  the inner loop Parameter optimization generates
  new values for the independent variable, i.e.
  decision variables, like
• Reactor temperature
• Reactor pressure
• Distillation column reflux ratio
• Etc.
• At the end we have a fixed flowsheet with its
  optimal process parameters.

39
Role of optimization in process design 5

• Then it is possible to change the flowsheet
  structure and carry out the parameter
  optimization for the new structure.
• The outer loop is repeated as many times as new
  promising process alternatives can be created.
  All of these are optimized for suitable process
  decision variables.
• In the structure optimization we do not have
  continuous variables, they are more discrete
  ones. Thus a different optimization method has to
  be used. This leads to so called MINLP Mixed
  Integer Non-Linear Optimization.

40
Classification of model types

• All variables and functions are continuous
• Unconstrained non-linear models
• One variable
• Multivariable problem
• Constrained models
• Linear model
• Quadratic model
• Non-linear model
• Functions of discontinuities require some care,
  special methods available.

41
Classification of model types 2

• Discrete variables
• Integer programming
• Discrete variables and linear model, MILP Mixed
  Integer Linear Programming
• Discrete variables and non-linear model, MINLP
  Mixed Integer Non-Linear Programming

42
Type of optimum

• Local extrema
• Minumum
• Maximum
• Global optimum (in the search interval)
• Global maximum
• Global minimum

43
Global optimum

• When the search interval is defined several
  methods can be used
• Exhaustive search (using local optimum finding
  methods with many different starting points to
  find all minima and keeping the best of them as
  the global minimum, but no guarantee)
• Adaptive random search
• Simulated annealing
• Genetic algorithms
• All of these are more or less so-called brute
  force methods, and computationally intensive.

44
Unconstrained non-linear one-dimensional methods

• Analytical methods already discussed
• In many problems the constraints can be inserted
  into the objective function so that the
  dimensionality of the problem is reduced to one
  variable
• Multivariable unconstrained and constrained
  optimization problems generally involve repeated
  use of a one-dimensional search

45
Unconstrained non-linear one-dimensional methods
2

• Numerical methods are used in practice.
• Scanning and bracketing methods
• Used to find the search interval
• The decision variable value is changed in steps
  and objective function (OF) evaluated until the
  sign of the change in OF happens. We have passed
  over the extremum and know the limits for its
  location. Then region elimination methods can be
  used.

46
Unconstrained non-linear one-dimensional methods
3

• Newton, Quasi-Newton and Secant methods
• Newtons method
• Recall that the primary necessary condition for
  f(x) to have a local minimum is that f(x)0.
  Consequently, you can solve the equation f(x)0
  by Newtons method to get
• making sure on each iteration stage k that
  f(xk1) lt f(xk) for a minimum.
• Locally quadratically convergent to the extremum
  as long as f(x)?0. For a quadratic function,
  minimum is found in one iteration.

47
Unconstrained non-linear one-dimensional methods
4

• Quasi-Newton Method
• Finite difference approximation of Newtons
  method
• Central differences used above, but forward
  difference or any other difference scheme would
  suffice as long as the step size h is selected to
  match the difference formula and computer
  (machine) precision.
• More function evaluation than in Newtons method,
  but no need obtain the tedious derivatives of the
  f(x).

48
Unconstrained non-linear one-dimensional methods
5

• Secant method
• The secant method is applied to solve f(x)0.
• where is the approximation to
  achieved on one iteration. Note that f(x) can
  itself be approximated by a finite difference
  substitute. Points p and q required.

49
Secant method
50
Unconstrained non-linear one-dimensional methods
7

• Region elimination methods
• Calculate the objective function in different
  points and eliminate the region that is limited
  by the points where the OF has highest values.
• Generate new point inside the search region and
  again eliminate the region that is limited by the
  points where the OF has highest values.
• Repeat until required accuracy reached.
• Different schemes for area elimination, two-point
  equal interval and golden section search are
  often used.

51
Unconstrained non-linear one-dimensional methods
8

• Polynomial approximation methods
• Quadratic interpolation
• Cubic interpolation

52
Use of one-dimensional search in multidimensional
search

• The one-dimensional search methods can be used in
  multidimensional search as follows
• Start from an initial point.
• For the multidimensional function determine the
  search direction at the point. This is done with
  negative gradient of the function.
• Reduce the function f(x) value by taking one or
  more steps in that search direction. For this we
  can use the one-dimensional search methods to
  optimize how far we go in the search direction.
• Go back to step 2. Once the gradient is zero and
  step length negligible, we have a solution.

53
Unconstrained multivariable optimization

• Direct methods
• Random search
• Search direction and step selected randomly,
  repeat many times
• Grid search
• Use methods of experimental design (statistics
  methods)
• Univariate search
• Perform search in one direction at a time using
  one-dimensional search methods.
• Go through all variables in a sequence and then
  repeat the sequence.

54
Univariate search
55
Unconstrained multivariable optimization
3

• Direct Methods Simplex method
• Originally fixed length of sides on the
  multidimensional vertex (Spendley, Hext and
  Himsworth, 1962).
• Nelder-Mead Simplex (1965) can expand and
  contract and is more complicated.
• Only function values at different points are
  evaluated, no derivatives/gradients required.
• Works always even with bad initial point, but
  inefficient in near vicinity of the minimum point.

56
Simplex method
57
Unconstrained multivariable optimization
5

• Direct methods
• Conjugate search directions
• Two direction si and sj are said to be conjugated
  with respect to each other if (si)TQ(sj)T0. This
  can be generalized for multiple dimensions and in
  optimization Q is selected to be the Hessian in
  the search point.
• Powells method
• Locates the minimum of a function by sequential
  one-dimensional searches from an initial point
  along a set of conjugate directions generated by
  the algorithm. New search directions are
  introduced as the search progresses.

58
Unconstrained multivariable optimization
6

• Indirect methods first order
• Derivatives are used in determining the search
  direction.
• Gradient method uses directly the function
  gradient at current point to find search
  direction. Steepest ascent for maximization.
  Steepest descent for minimization

59
Unconstrained multivariable optimization
7

• Indirect methods first order
• Conjugate gradient method was devised by Fletcher
  and Reeves in 1964.
• Improvement over steepest descent
• Only marginal increase in computation effort
  compared to steepest descent
• Uses conjugate directions
• Stores only little data during computation

60
Unconstrained multivariable optimization
8

• Indirect methods second order
• Newtons method
• Makes a second order (quadratic) approximation of
  the OF in point xk. Takes into account the
  curvature of the OF and identifies better the
  search directions than can be obtained via the
  gradient method.
• Here is the inverse matrix of the
  Hessian matrix. Can be modified to include the
  step length.

61
Unconstrained multivariable optimization
9

• Indirect methods second order
• Marquardt and Levenberg suggested that the
  Hessian is modified on each stage so that it is
  kept positive definite and well-conditioned.
• Many other researchers have suggested
  improvements in different details of the method.

62
Unconstrained multivariable optimization
10

• Secant methods
• By analogy with secant methods for functions of a
  single variable.
• Uses only function values and gradients.
• The Hessian matrix is approximated from the
  combination of the function values and gradient
  values.
• Also, the gradients can be approximated using
  finite difference substitutes. Then only function
  values are required.
• Numerous methods Broyden, Davidon-Fletcher-Powell
  (DFP), Broyden-Fletcher-Goldfarb-Shanno (BFGS).

63
Optimization of linear models

• Linear objective function has maximum in infinity
  if the problem is not correctly bounded.
• This is due to the fact that the first
  derivatives of the OF can never be zero. Consider
  f(x) x1x2.
• Normally we require variables to be positive for
  physically meaningful model.
• The constraints are also linear.

64
Optimization of linear models 2
65
Optimization of linear models 3
66
Optimization of linear models 4

• Linear optimization (Linear Programming, LP)
  problems can be huge, like in refinery blending
  optimization several thousand equations and a
  huge number of variables.
• Effective algorithms
• Simplex, looks for a feasible region and a
  solution in one the corners.
• Revised Simplex, more efficient in calculation
• Karmarkar algorithm, not yet proved to be more
  effective in all problem than Simplex.

67
Non-linear optimization with constraints
Often the inequality constraints are transformed
into equality constraints.
68
Non-linear optimization with constraints
2
69
Non-linear optimization with constraints
3
70
Lagrange multiplier method
71
Lagrange multiplier method 2

• Lagrange multiplier method can be extended for
  cases where we have inequality constraints.
• By adding a slack variable sj2 for each
  inequality constraint they are converted to
  equality constraints.

72
Quadratic programming

• The OF is quadratic in n variables.
• There are m linear inequality or equality
  constraints.
• The method is efficient for this class of
  problems.

73
Generalized reduced gradient method

• Uses iterative linearization of the OF and
  constraints.
• Many variants of this method class.
• GRG, GRG2
• GRG2 by prof. Lasdon is one the best performing
  codes. A small version of this is used in Excel
  Solver.

74
Penalty function methods

• Penalty function methods are used to convert
  constrained problems into unconstrained problems.

75
Penalty function methods 2

• User penalty functions are easily integrated into
  objective functions.

76
Penalty function methods 3

• Previous penalty function is so called external
  penalty. Penalty is given once the constraint is
  violated. But in this case it does not work for
  the logarithm term.
• We have to limit x2 before it gets close to value
  -2/3. We use so-called internal penalty function
  for this purpose. Penalty is always given for x2,
  but the more the closer we get to value -2/3.
  This might be called a barrier function that
  prevents the variable to be inside the search
  region.

77
Successive Quadratic Programming, SQP

• Quadratic programming is used sequentially.
• These methods has proved to be extremely usable
  in flowsheeting programs for optimization.
• Different versions available.
• Algorithm includes
• Initialization
• Calculation of active set members
• Termination check. If not satisfied, continue.
• Update the positive definite approximation of the
  Hessian matrix.
• Estimate Lagrangian multipliers.
• Calculate new search direction.
• Move a step in the direction. Step length is
  calculated by minimizing penalty function. Go
  back to second step.

78
Random search methods

• Heuristic random search
• Adaptive random search
• Seppo Palosaari et al. SMIN-code
• Search interval specified
• Objective function values are calculated for a
  large number of even random number distribution.
• The generation of random numbers concentrates
  around the best point found by making the random
  number distribution more and more concentrated as
  the number of objective function value
  evaluations gets larger

79
Random search methods 2

• Genetic algorithms can be used for optimization,
  uses a lot of computational power, but only
  function evaluations required
• Simulated annealing methods, uses a lot of
  computational power, but only function
  evaluations required
• These random search methods are useful in
  searching global optimum (in a specified interval)

80
Optimization of staged and discrete processes

• Dynamic programming
• Integer and mixed integer programming
• Implicit enumeration
• Branch and bound method
• Non-linear mixed-integer programming algorithms
• Treat the discrete variables as continuous ones
  and once solution is found test the nearest
  discrete values
• The discreteness is used as one of the
  restrictions in non-linear optimization

81
Now it is optimal time to stop the theory
presentation...