Advanced Digital Signal Processing

1
DEPARTMENT OF COMPUTER SCIENCE UNIVERSITY OF
JOENSUU JOENSUU, FINLAND

• Advanced Digital Signal Processing
• Lecture 6
• Linear Transforms
• KLT, PCA, SVD, DFT
• Alexander Kolesnikov
• (09.10.2005)

2
Karhunen-Loeve TransformPrincipal Component
Analysis
3
Linear transform of signal
Let X is a discrete set of points
Y is a linear transform of X
4
Autocorrelation (covariance) matrix

• Autocorrelation matrix of the input vector X

(symmetric matrix)

• Autocorrelation matrix of output vector Y

5
Decorrelation of Y
Find a N?N orthonormal projection matrix A such
that covariance matrix Ry is a diagonal matrix
If matrix Rx is symmetric and positive
semidefinite, then A is matix whose rows are the
eigenvectors of Rx.
6
KLT PCA

• Find eigenvectors and eigenvalues ?j for
  a
• matrix Rx

• Construct orthonormal basis with transform
  matrix A

• Transform X into new basis

7
Decorrelation property of KLT

• Correlation matrix for output Y

The coefficients of KLT are decorrelated!
8
Decorrelation property of KLT
9
1st step Find eigenvalues (1)
Solve the characteristic eqaution. Polynom of ?
is characteristic polynom. Set of eigenvalues
?(Rx) is called spectre of Rx.
10
1st step Find eigenvalues (2)
Eigenvalues are sorted in decreasing order.
11
2nd step Find eigenvectors
Solve the system of linear equations for all ?.
12
3rd step Normalization
13
Example 1 KLT, PCA
14
Dimensionality reducing (1)

• Truncate a transformed random vector AX, keeping
  out
• m out of N coefficients and setting the rest to
  zero.
• Then among all linear transforms, the KLT
  provides
• the best approximation in the mean square sense
  to the
• original vector.

Energy compactness property
15
Energy compactness (2)
Matrix Im retains the first components of Y and
sets to zero the last N?m components. We
reconstruct an approximation to X from truncated
set of transform coefficients
Distortion (MSE)
16
Energy compactness (2)
The squared error D is a minimum when the
matrices U and V are the KLT transform and its
inverse, respectively
The k-D space in which the expected energy of the
projection of X is minimized is the space
spanned by the eigenvectors corresponding to the
the k smallest eigenvalues.
17
Example 1 KLT, PCA
18
Example 2 KLT, PCA?
19
All names

• Karhunen-Loeve Transform
• Principal Component Analysis
• Hotelling Transform
• ...

20
KLT is optimal, but...

• Decorrelation of data? Redundancy reduction,
• signal compression
• Energy compactness Dimensionality reduction
• but
• KLT is signal dependent, complexity of
  calculation O(N4)

Lets introduce a model.
21
1st order Markov process

• xn?xn-1 ?n, where ?n is white noise
• Correlation matrix Rx

22
Correlation function
23
KLT for the Markov process N8
T,Leig(Rx) for k1N subplot(2,4,k)
stem(T(,1N-k)) axis(0 N1 -0.8 0.8
) end
24
Discrete Cosine Transform (DCT)
DCT gives asymptotical solution of KLT for Markov
process.
DCT is a core of lossy image compression
algorithm JPEG
25
KLT DCT N8 (k0,1,2,3)
KLT
DCT
Compare KLT and DCT!
26
KLT DCT N8 (k4,5,6,7)
KLT
DCT
Compare KLT and DCT!
27
Singular Value Decomposition
28
Singular Value Decomposition (SVD)

• Any MxN matrix matrix X whose number of rows M is
• greater than or equal to its number of columns N,
  can be
• written as the product of an MxN
  column-orthogonal
• matrix U, an NxN diagonal matrix ? with positive
  or zero
• elements (the singular values), and the transpose
  of an
• NxN orthogonal matrix V.

For example, matrix X consists of is M
N-dimensional vectors.
29
Singular Value Decomposition
Size of matrix ?1 is equal to rank r of matrix X.
Values ?1? ?2?... ? ?N ?0 are singular values.
U and V are left and right singular vectors
30
Singular Value Decomposition
In other words
where ui and vi are the first columns of matrices
U and V, respectively
31
Singular Value Decomposition
If X is approximated by
then matrix is of rank k. It can be shown
that the squared error
is the minimum one with respect to all rank-k
N ?N matrices
32
Singular Value Decomposition
Thus, is the best rank-k approximation of
matrix X in the Frobenius norm
sense. Difference between KLT and SVD KLT
Related to ensemble of samples (covariance) SVD
Related to a single set of samples (signal
itself)
33
Singular Value Decomposition
Solution of linear equations Least-squares
problem Dimensionality reduction Pattern
recognition Noise filtering Signal compression
34
Fourier Transform
35
Four types of Fourier Transform
36
Four types of Fourier Transform
37
Fourier Transform (FT)

• Forward FT

• Inverse FT

38
Fourier Transform Examples
39
Two test signals
a)
?1 10? ?2 20? ?3 40? ?4100?
x(t)cos(?1 t)cos(?2t)cos(?3)cos(?4t)
b)
x1(t)cos(?1t) x2(t)cos(?2t) x3(t)cos(?3t) x4(t)
cos(?4t)
x1(t) x2(t) x3(t) x4(t)
40
Signals are different, spectrums are similar!
a)
b)
Why?
41
What is wrong with the Fourier Transform?
Lets consider two basis functions sin(?t) and
?(t)
Support region In space In frequency
sin(?t) ? 0 ?(t)
0 ? The basis function sin(?t) is not
localized in time! The ?(t) (sample) is not
localized in frequency
Lets introduce basis function which is compact
in time and frequency domains Short-Time
Fourier Transform (STFT)
42
Short-Time Fourier Transform (STFT)
Input signal f(t)
Window g(t)
Result is localized in space and frequency.
43
Problems with STFT
Uncertainity Principle
Improved space resolution? Degraded frequency
resolution Improved frequency resolution?Degraded
space resolution
44
New partition of the time-frequency plane
Frequency, ?
Coordinate, t
45
STFT ? Wavelets
Wavelet transform
Short-time Fourier transform

• Wavelet functions are localized in space and
  frequency
• Hierarchical set of of functions

46
Spectrogram
47
Spectrogram
Spectrogram is graphical representation of
Shport-time Fourier Transform
48
Spectrogram 1 Sine wave at 660 Hz
y(t)Asin(2?f0t)
f0660 Hz
From DSP First by McClellan, Schafer and Yoder
( CD)
49
Spectrogram 2 White noise
50
Spectrogram 3 Chirp signal
51
Spectorgram 4 Music scale
52
Spectrogram 5 Train whistle
53
Spectrogram 6 Human voice Bat
54
Time Frequency Resolution (1)
55
Time Frequency resolution (2)
56
STFT with wide window
Signal and FT
Spectrogram (STFT)
57
STFT with narrow window
Signal and FT
Spectrogram (STFT)
58
STFT with medium window
Signal and FT
Spectrogram (STFT)
59
Time Frequency resolution
h(t) is windowing function
Example