Realtime Recognition of Orchestral Instruments

1
Realtime Recognition of Orchestral Instruments

• Ichiro Fujinaga
• McGill University

2
Overview

• Introduction
• Lazy learning (exemplar-based learning)
• k-NN classifier
• Genetic algorithm
• Features
• Results
• Conclusions

3
Introduction

• Realtime recognition of isolated monophonic
  orchestral instruments
• Spectrum analysis by Miller Puckettes fiddle
• Adaptive system based on a exemplar-based
  classifier and a genetic algorithm

4
Overall Architecture
Live mic Input
Sound file Input
Data Acquisition Data Analysis (fiddle)
Recognition K-NN Classifier
Output Instrument Name
Knowledge Base Feature Vectors
Genetic Algorithm K-NN Classifier
Best Weight Vector
Off-line
5
Exemplar-based learning

• The exemplar-based learning model is based on the
  idea that objects are categorized by their
  similarity to one or more stored examples
• There is much evidence from psychological studies
  to support exemplar-based categorization by
  humans
• This model differs both from rule-based or
  prototype-based (neural nets) models of concept
  formation in that it assumes no abstraction or
  generalizations of concepts
• This model can be implemented using k-nearest
  neighbor classifier and is further enhanced by
  application of a genetic algorithm

6
Exemplar-based categorization

• Objects are categorized by their similarity to
  one or more stored examples
• No abstraction or generalizations, unlike
  rule-based or prototype-based models of concept
  formation
• Can be implemented using k-nearest neighbor
  classifier
• Slow and large storage requirements?

7
Exemplar-based learning

• The exemplar-based learning model is based on the
  idea that objects are categorized by their
  similarity to one or more stored examples
• There is much evidence from psychological studies
  to support exemplar-based categorization by
  humans
• This model differs both from rule-based or
  prototype-based (neural nets) models of concept
  formation in that it assumes no abstraction or
  generalizations of concepts
• This model can be implemented using k-nearest
  neighbor classifier and is further enhanced by
  application of a genetic algorithm

8
K-nearest-neighbor classifier

• Determine the class of a given sample by its
  feature vector
• Distances between feature vectors of an
  unclassified sample and previously classified
  samples are calculated
• The class represented by the majority of
  k-nearest neighbors is then assigned to the
  unclassified sample

9
Example of k-NN classifier
10
Example of k-NN classifier
11
Example of k-NN classifier
12
Example of k-NN classifier
13
Distance measures

• The distance in a N-dimensional feature space
  between two vectors X and Y can be defined as
• A weighted distance can be defined as

14
Genetic algorithms

• Optimization based on biological evolution
• Maintenance of population using selection,
  crossover, and mutation
• Chromosomes weight vectors
• Fitness function recognition rate
• Leave-one-out cross validation

15
Features

• Static features (per window)
• pitch
• mass or the integral of the curve (zeroth-order
  moment)
• centroid (first-order moment)
• variance (second-order central moment)
• skewness (third-order central moment)
• amplitudes of the harmonic partials
• number of strong harmonic partials
• spectral irregularity
• tristimulus
• Dynamic features
• means and velocities of static features over time

16
Data

• Original source McGill Master Samples
• Over 1300 notes from 39 different timbres (23
  orchestral instruments)
• Spectrum analysis by fiddle (2048 points)
• First 46232ms of attack (19 windows)
• Each analysis window (46 ms) consists of a list
  of amplitudes and frequencies of the peaks in the
  spectra

17
Results

• Experiment I
• SHARC data
• static features
• Experiment II
• fiddle
• dynamic features
• Experiment III
• more features
• redefinition of attack point

18
Conclusions

• Realtime timbre recognition system
• Analysis by Puckettes fiddle
• Recognition using dynamic features
• Adaptive recognizer by k-NN classifier enhanced
  with genetic algorithm
• A successful implementation of exemplar-based
  classifier in a time-critical environment

19
Future research

• Performer identification
• Speaker identification
• Tone-quality analysis
• Multi-instrument recognition
• Expert recognition of timbre

20
Recognition rate for different lengths of
analysis window
21
Comparison with Human Performance