Realtime Recognition of Orchestral Instruments

1
Realtime Recognition of Orchestral Instruments

• Ichiro Fujinaga and Karl MacMillan
• Peabody Conservatory of Music
• Johns Hopkins University

2
Overview

• Introduction
• Lazy learning (exemplar-based learning)
• k-NN classifier
• Genetic algorithm
• Features
• Results
• Demonstration
• Conclusions

3
Introduction

• Realtime recognition of isolated monophonic
  orchestral instruments
• Spectrum analysis by Miller Puckettes fiddle
• Adaptive system based on a exemplar-based
  classifier and a genetic algorithm

4
Overall Architecture
Live mic Input
Sound file Input
Data Acquisition Data Analysis (fiddle)
Recognition K-NN Classifier
Output Instrument Name
Knowledge Base Feature Vectors
Genetic Algorithm K-NN Classifier
Best Weight Vector
Off-line
5
Exemplar-based categorization

• Objects are categorized by their similarity to
  one or more stored examples
• No abstraction or generalizations, unlike
  rule-based or prototype-based models of concept
  formation
• Can be implemented using k-nearest neighbor
  classifier
• Slow and large storage requirements?

6
K-nearest-neighbor classifier

• Determine the class of a given sample by its
  feature vector
• Distances between feature vectors of an
  unclassified sample and previously classified
  samples are calculated
• The class represented by the majority of
  k-nearest neighbors is then assigned to the
  unclassified sample

7
Example of k-NN classifier
8
Example of k-NN classifier
9
Example of k-NN classifier
10
Example of k-NN classifier
11
Distance measures

• The distance in a N-dimensional feature space
  between two vectors X and Y can be defined as
• A weighted distance can be defined as

12
Genetic algorithms

• Optimization based on biological evolution
• Maintenance of population using selection,
  crossover, and mutation
• Chromosomes weight vector
• Fitness function recognition rate
• Leave-one-out cross validation

13
Features

• Static features (per window)
• pitch
• mass or the integral of the curve (zeroth-order
  moment)
• centroid (first-order moment)
• variance (second-order central moment)
• skewness (third-order central moment)
• amplitudes of the harmonic partials
• number of strong harmonic partials
• spectral irregularity
• tristimulus
• Dynamic features
• means and velocities of static features over time

14
Data

• Original source McGill Master Samples
• Over 1300 notes from 39 different timbres (23
  orchestral instruments)
• Spectrum analysis by fiddle (2048 points)
• First 46232ms of attack (19 windows)
• Each analysis window (46 ms) consists of a list
  of amplitudes and frequencies of the peaks in the
  spectra

15
Results

• Experiment I
• SHARC data
• static features
• Experiment II
• fiddle
• dynamic features
• Experiment III
• more features
• redefinition of attack point

16
Demonstration

• Using stored data
• Using recording
• Using audience

17
Conclusions

• Realtime timbre recognition system
• Analysis by Puckettes fiddle
• Recognition using dynamic features
• Adaptive recognizer by k-NN classifier enhanced
  with genetic algorithm
• A successful implementation of exemplar-based
  classifier in a time-critical environment

18
Future research

• Performer identification
• Speaker identification
• Tone-quality analysis
• Multi-instrument recognition
• Expert recognition of timbre

19
Recognition rate for different lengths of
analysis window
20
Comparison with Human Performance