Correntropy%20as%20a%20similarity%20measure

1
Correntropy as a similarity measure

• Weifeng Liu, P. P. Pokharel, Jose Principe

Computational NeuroEngineering Laboratory Universi
ty of Florida http//www.cnel.ufl.edu weifeng_at_cnel
.ufl.edu Acknowledgment This work was partially
supported by NSF grant ECS-0300340 and
ECS-0601271.
2
Outline

• What is correntropy
• Interpretation as a similarity measure
• Correntropy Induced Metric robustness
• Applications

3
Correntropy General Definition

• For random variables X, Y correntropy is
• where K is Gaussian kernel
• Sample estimator

4
Correntropy Correlation Entropy

• Correlation with high order moments
• Taylor expansion of Gaussian kernel
• Kernel size large, second order moment dominates
• Average over dimensions is the argument of
  Renyis quadratic entropy

5
Reproducing Kernel Hilbert Space induced by
Correntropy- (VRKHS)

• V(t,s) is symmetric and positive-definite
• Defines a unique Reproducing Kernel Hilbert
  Space---VRKHS
• Wiener filter is an optimal projection in RKHS
  defined by autocorrelation
• Analytical nonlinear Wiener filter framed as an
  optimal projection in VRKHS

6
Probabilistic Interpretation

• Integration of joint PDF along xy line
• Probability of
• Probability density of XY

7
Probabilistic Interpretation
8
Geometric meaning

• Two vectors
• Define a function CIM

9
Correntropy Induced Metric

• CIM is Non-negative
• CIM is Symmetric
• CIM obeys the triangle inequality
• Therefore it is a metric that is induced in the
  input space when one operates with correntropy

10
Metric contours

• Contours of CIM(X,0) in 2D sample space
• close, like L2 norm
• Intermediate, like L1 norm
• far apart, saturates with large-value elements
• (direction sensitive)

11
CIM versus MSE as a cost function

• Localized similarity measure

12
CIM is robust to outliers

• measure similarity in a small interval Do not
  care how different outside the interval
• Resistant to outliers (in the sense of Hubers
  M-estimation)

13
Application 1 Matched filter

• S transmitted binary signal
• N channel noise
• Y received signal

14
Application 1 Matched filter

• Sampled (1,-1) received signal
• Linear matched filter
• Correntropy matched filter

15
Application 1 Matched filter
BER
SNR (dB)
16
Application 2 Robust Regression

• X input variable
• f unknown function
• N noise
• Y observation

17
Application 2 Robust Regression

• Maximum Correntropy Criterion (MCC)

yg(x)
X
18
MCC is M- Estimation
MCC ?
?
19
Significance

• Correntropy is a building block of
• correntropy nonlinear Wiener filter
• correntropy matched filter
• correntropy nonlinear MACE filter
• correntropy Principal Component Analysis
• Renyis quadratic entropy
• This understanding is crucial to explain the
  behavior of nonlinear algorithms and high-order
  statistics!

20
References

• 1 I. Santamaria, P. P. Pokharel, J. C.
  Principe, Generalized correlation function
  definition, properties and application to blind
  equalization, IEEE Trans. Signal Processing, vol
  54, no 6, pp 2187- 2186
• 2 P. P. Pokharel, J. Xu, D. Erdogmus, J. C.
  Principe, A closed form solution for a nonlinear
  Wiener filter, ICASSP2006
• 3 Weifeng Liu, P. P. Pokharel, J. C. Principe,
  Correntropy Properties and Applications in
  Non-Gaussian Signal Processing, submitted to
  IEEE Trans. Signal Proc.