Introduction to Function Minimization

1
Introduction to Function Minimization
2
Motivation example
Data on height of a group of 10000 people, men
and women Data on gender not recorded, not known
who was man and who woman Can one estimate
number of men in the group?
Asymmetric histogram Non-Gaussian two
subgroups (men woman) ? two superposed
Gaussians ? ______________________________ This
is artificially simulated data, just demo, two
Gaussians with different mean randomly men/women
7/3
3
See error bars!
4
Best Gaussian fit
5
Two Gaussians best fit
This is artificially simulated data, just demo,
two Gaussians were used for simulation, with
different mean, randomly men/women 7/3
6
Two Gaussians best fit
This is artificially simulated data, just demo,
two Gaussians were used for simulation, with
different mean, randomly men/women 7/3
7
Two Gaussians best fit
This is artificially simulated data, just demo,
two Gaussians were used for simulation, with
different mean, randomly men/women 7/3
8
Press-the-button user find the best fit by two
Gaussians but How it is done? Gaussian
Two Gaussians
Find the best values of the parameters
Needs goodness-of-fit criterion
Fit function is nonlinear in its parameters
9
50 histogram bins each bin represents "one data
point"
Goodness-of-fit (least squares)
10
Mathematically, the problem is the following I
have a function of n variables (parameters of
goodness-of-fit) I want to find the point for
which the function achieves its minimum
The function is often non-linear, so analytical
solution of the problem is usually hopeless. I
need numerical methods to attack the problem
11
Numerically, the problem is the following I have
a function (in the sense of program subroutine)
of n parameters I want to find the point for
which the function achieves its minimum. Each
call to evaluate the function value is often
time-consuming, having in mind its definition
like
I need a numerical procedure which can call the
function FCN and by repeating calls with possibly
different values of the parameters finally finds
the minimizing set of parameter values
12
Stepping algorithms

• Start at some point in the parameter space
• Choose direction and step size
• Walk according to some clever strategy doing
  iterative steps and looking for small values
  of the minimized function

13
One dimensional optimization problem
Stepping algorithm with adaptable step
size fcurrent FCN(xcurrent) repeat forever
xtrial xcurrentstep ftrialFCN(xtrial) if
(ftrialltfcurrent) //success
xcurrentxtrial fcurrentftrial step3step
else //failure
step-0.4step
Fast approach to minimum area, slow convergence
at the end
14
(No Transcript)
15
(No Transcript)
16
Success failure method
parabola
17
parabola
line
line ............estimates first derivative
(gradient) parabola ... estimates first as well
as second derivative
18
Stepping method can estimate gradient as well as
second derivative All functions around minimum
look like parabola Newton go straight to minimum
Inverse to the second derivative helps to jump
straight to minimum in the direction of the
negative gradient
19
Any stepping method needs

• starting point
• initial step size
• end criterion (otherwise infinite loop)

• step size lt d
• improvement in the function values lt e

20
Problem of local and/or boundary minima
21
Many-dimensional minimization

• start at point (p0,p1)
• fix value p1
• perform minimization with respect to p0
• fix value p0
• perform minimization with respect to p1

22
axes in wrong directions not in the direction of
gradient
cure rotate axes after each iteration so that
the first axis is in the direction of the
(estimated) gradient
23
(No Transcript)
24
Simplex minimization
trial point
estimated gradient direction
get rid of the worst point
25
(No Transcript)
26
Gradient methods
Stepping methods which use in addition to the
function value at the current point also local
gradient at that point

• The local gradient can be obtained
• by calling a user-supplied procedure which
  returns the vector (n-dimensional array) of
  first order derivatives
• by estimating the gradient numerically
  evaluating the function value at n points in a
  vary small neighborhood of the current point

Performing one-dimensional minimization in the
direction of the negative gradient significantly
improves the current point position
27
Quadratic approximation to the minimized function
at the current point p0 in n dimensions
Knowing g, one can perform one dimensional
optimization with respect to "step size" a in
search for the best point
If G0 were known, the minimum will be at
It would be useful, if the gradient method could
also estimate the matrix G-1
28
If the minimized function is exactly quadratic,
then G is constant in the whole parameter space
If the minimized function is not exactly
quadratic, we expect slow variations of G in the
region not far from the minimum Local numerical
estimate of G is costly, needs many calls to F,
and than matrix inversion is needed Idea can one
iteratively estimate G-1?
29
Example of a variable metric method
for quadratic functions the iteration converges
to minimum and true G-1
30
Variable metric methods

• fast convergence in the almost quadratic region
  around minimum
• added value good knowledge of G-1 at minimum
  what means that the shape of the optimized
  function around minimum is known

31
MINUIT
Author F.James (CERN)

• Complex minimization program (package) comprising
  various minimization procedures as well as other
  useful data-fitting tools
• Among them
• SIMPLEX
• MIGRAD (variable metric method)

32
MINUIT

• Originally FORTRAN program in the CERN library
• Now available in C
• stand alone (SEAL project)
• http//seal.web.cern.ch/seal/MathLibs/Minuit
  2/html/index.html
• contained in the CERN ROOT package
• http//root.cern.ch
• Available in Java (FreeHEP JAIDA project)
• http//java.freehep.org/freehep-jminuit/index.
  html

33
References
F. James and M. Winkler, C MINUIT User's
Guide http//seal.cern.ch/documents/minuit/mnuser
sguide.pdf F. James, Minuit Tutorial on Function
Minimization http//seal.cern.ch/documents/minui
t/mntutorial.pdf F. James, The Interpretation of
Errors in Minuit http//seal.cern.ch/documents/m
inuit/mnerror.pdf
Microsoft Visual c Express is FREE c for
Windows http//www.microsoft.com/express/vc/