Functional Analysis in Data Modelling

1
Functional Analysis in Data Modelling
Tony Dodd t.j.dodd_at_shef.ac.uk
2
Overview

• Metric spaces
• Linear spaces
• Normed, inner product and Hilbert spaces
• Best approximation
• Reproducing kernel Hilbert spaces
• Approximation vs estimation

3
Spaces
F
4
Spaces

• Define some space F
• Points/elements in F denoted fi
• What is F ?
• Euclidean space
• Space of sines and cosines (Fourier)
• Space of bandlimited functions (Paley-Wiener)
• L2
• Can be a nonlinear space

5
Metric spaces

• Put some structure on our space F

F
6
Metric spaces

• Define the distance
  
• if and and only if

7
Why are metric spaces important?

• Allow us to define the distance between functions
• Can then talk about best approximations
• Completeness no holes in the space
• We want to look at spaces that are very similar
  to Euclidean space

8
Linear spaces

• Need two algebraic operations
• Addition f3 f1f2
• f1f2 f2f1 and f1(f2 f3)( f1f2) f3
• Multiplication by scalars f2 af1
• a(f1f2) a f1 a f2
• (aß)f1 af1ßf1
• a(ßf1) (aß)f1
• 1f1 f1 0f1 0

9
Linear spaces and basis

• Set of f1, f2, , fn is linearly independent if
• a1 f1 a2 f2 an fn 0
• Holds only if each ai 0.
• Finite dimensional if only n linearly independent
  elements
• Linear manifold af1ßf2 in F
• Basis can express every f in F in the form
• f a1 f1 a2 f2 an fn

10
Basis
F
0
11
Normed spaces

• Define the notion of the size of an element in F
• Norm f
• Defines a metric

12
Algebraic and geometric

• By defining the metric based on the norm we have
  combined the algebraic (metric) and geometric
  (linear, norm) properties.
• Algebraic the technical bits that we need but
  are difficult!
• Geometric the intuitive bits.

13
Inner product spaces

• Linear space with an inner product
• Define the norm as

with equality iff f 0
14
Hilbert spaces

• These look like Euclidean space.
• An inner product space which is complete with
  respect to the metric defined from the inner
  product is called a Hilbert space.
• In simple terms they have all the nice
  mathematical properties we need so dont worry
  about the complicated bits.

15
Best approximation
f
F
H
16
Best approximation

• For any given f in a Hilbert space F and a closed
  subspace there exists a unique
  best approximation to f out of H.
• In fact
• i.e.

17
Best approximation

• Assume H is finite dimensional with basis
• i.e.
• gives m conditions (i1,,m)

or
18
Reproducing kernel Hilbert spaces (RKHS)

• A particular class of Hilbert space very
  important in machine learning.
• Splines, kernel machines, support vector
  machines, neural networks, Gaussian processes,
  time series analysis, bandlimited signals

19
RKHS

• Hilbert space with even more structure.
• Not worry about technical details here.
• Main properties
• Observations
• k are positive definite functions

20
RKHS approximation
m conditions become
Can then estimate the parameters using
In practise can be ill-conditioned/noise on the
data so minimise
21
Approximation vs estimation
Hypothesis space
Best possible estimate
Estimate
Target space
True function