Nonlinear Dimensionality Reduction with Fuzzy Integral and Applications

1
Nonlinear Dimensionality Reduction with Fuzzy
Integral and Applications
Part B Research Proposal

• Speaker Wang JinFeng
• Supervisors Prof. Leung KwongSak, Prof. Lee
  KinHong
• Date 2006.12.07

2
Outline

• Introduction about dimensionality reduction
• Basic Concepts
• Fuzzy measure
• Fuzzy integral
• Condition matrix
• Main algorithms and methods
• Nonlinear regression with FI
• Condition matrix to learn parameters in FI
• Projection by FI to reduce dimension
• Applications

3
Introduction about dimensionality reduction

• Dimensionality reduction with minimal information
  loss is important for feature extraction, compact
  coding and computational efficiency.
• Classical methods
• Principle component analysis (PCA)
• ---to identfy the dependence structure behind a
  multivariate stochastic observation in order to
  obtain a compact description of it.
• Multidimensional scaling (MDS)
• ---to detect meaningful underlying dimensions
  that allow the researcher to explain observed
  similarities or dissimilarities (distances)
  between the investigated objects.
• Locally linear embedding (LLE)
• ---to compute low-dimensional,
  neighborhood-preserving embeddings of
  highdimensional inputs
• ---be able to learn the global structure of
  nonlinear manifolds, such as imagines of faces or
  documents of text

4
Basic Concepts

• 1. Fuzzy measure
• Generalization of probability measures replacing
  the additive property by the monotonic property
  with respect to set inclusion.
• Let X be a non-empty finite set and P(X) the
  power set of X.
• Definition 1. A set function µ P(X)? 0,1 is a
  fuzzy measure if
• (1)

(2)
(3)
Probability, possibility, belief and plausibility
measures are all special cases of fuzzy measure.
5
Basic Concepts-contd

• 2. Fuzzy integral
• ?Sugeno integral
• Definition 2. Let µ be a fuzzy measure on X. The
  Sugeno integral of a function f X?0,1, with
  respect to µ is defined by

6
Basic Concepts-contd

• 2. Fuzzy integral
• ? Choquet integral
• Definition 1. Let be a fuzzy measure on A.
  The Choquet integral of a function
  with respect to is defined by

indicates that the indices have been permuted so
that
where
The Choquet integral is based on linear operators
to deal with nonlinear space. It is different
with The Sugeno integral which is based on
nonlinear operators.
7
Basic Concepts-contd

• 3. Condition matrix based Genetic Programming
• --- An extension of the instruction matrix for
  generating rule base from datasets.
• --- CM keeps some of characteristics of IM and
  incorporates the information about attributes of
  dataset. In the evolving process,
• --- Adopt an elitist idea to keep the better
  individuals alive to the end.
• ---This algorithm has been applied to Rule
  Learning successfully 8.

8
Main algorithms and methods A new nonlinear
multi-regression model

• Example Let (x1, x2, , xn) be predictive
  attributes and y be the objective attribute.
  Denote Xx1, x2, , xn. The data have a form as
  following.
• x1 x2 xn y
• f11 f12 f1n y1
• f21 f22 f2n y2
• fm1 fm2 fmn ym
• fij the observation of attributes
• yi the value of classes
• m the size of the data

9
Main algorithms and methods-contd A new
nonlinear multi-regression model

• The new nonlinear multi-regression model is
  constructed by
• For the general Choquet integral, this model can
  be transferred to
• The regression residual error can be calculated
  by
• This model will determine the coefficients
  minimizing the square error.

10
Main algorithms and methods-contd Condition
Matrix to learn the parameters in FI
The brief process of algorithm is listed as
follows. Step 1. Initialize condition
matrix Step 2. Select fuzzy measure from CM row
by row according to value of each fuzzy measure
as fitness Step3. Call fuzzy integral to
evaluate. If least square error satisfies one
threshold, stop it else go back to step 2.
11
Main algorithms and methods -contdCondition
Matrix to learn the parameters in FI
u(A(1)) u(A(.)) u(A(.)) u(A(.)) Cu(f(x))
u(A(.)) u(A(.)) u(A(.))
12
Main algorithms and methods-contd Condition
Matrix to learn the parameters in FI

• Initially, all fitness is designed randomly as
• One observation of dataset has three values
• Supposed that the following subsets are selected,
  x2, x1x2, x2x3, x1x3, x2, x1x2x3, x1x2

13
Main algorithms and methods -contd Projection
by fuzzy integral from h-D to l-D

• Projection to 1-D space--the real axis
• Model Projecting the points in the feature
  space onto a real axis through a nonlinear
  fuzzy integral
• M Rn R
• R 1-D space, i.e. a real axis
• Rn the feature space x1,x2,,xn

14
Main algorithms and methods -contd Projection
by fuzzy integral from h-D to l-D

• ????
  u
• But intersection situation may exist in data
  projected by fuzzy integral
• ? ??
  u

15
Main algorithms and methods-contd
Projection by fuzzy integral from h-D to l-D

• Projection onto 2-D space with fuzzy integral

16
Main algorithms and methods -contd Projection
by fuzzy integral from h-D to l-D

• Double fuzzy integral

17
Main algorithms and methods-contd Projection
by fuzzy integral from h-D to l-D

• General case of Projection from h-D to l-D

18
Main algorithms and methods -contd Projection
by fuzzy integral from h-D to l-D

• For a training dataset, we design the square
  error of nonlinear regression with multiple fuzzy
  integral as the evaluation criteria of stopping
  projection.
  .
• pseudo-code
• n the degree of dimension to be projected onto
• m the size of data set
• let n1
• For j1 to m
• Compute fuzzy integral value for each data to
  get
• get the residual
• If , stop and return n as the final degree of
  dimension
• else
• let nn1 and go back to For

19
Application

• Bioinformatics
• Predicting Protein Cellular Localization---Richard
  Mott etc.
• Overlap of Protein structures---David Pelta etc.
• Finance
• Predicting stock market---Hellström, Holmström
  (1998)