Concepts for Human Machine Interfaces (HMI)

1
Concepts for Human Machine Interfaces (HMI)
Martin Mienkina Martin.Mienkina_at_udo.edu

• Fakultät für Elektrotechnik und
  Informationstechnik
• Lehrstuhl für Kommunikationstechnik
• Prof. Dr.-Ing. Rüdiger Kays

2
Outline

• The human sensors
• Human Machine Interaction in Cars possible
  approaches
• Input/Output
• Speech recognition
• Fundamentals of the Speech Recognition Process
• Techniques for Noisy Speech Recognition
• Speaker recognition
• Component Communication
• MOST
• HMI2000
• Future Conntextual Car Driver Interface

3
Senses and Stimuli

• The human senses
• exteroceptors
• eye
• ear
• skin
• nose
• tounge
• proprioceptors
• interoceptors

4
HMI in Cars Overview Input

• Human Machine Interaction in Cars possible
  approaches
• Input
• Classic Knobs and switches
• Speech/ Speaker recognition
• Gesture recognition
• Mimics
• Eye tracking

5
HMI in Cars Overview Input

• Human Machine Interaction in Cars possible
  approaches
• Input
• Classic Knobs and switches
• Speech/ Speaker recognition
• Gesture recognition
• Mimics
• Eye tracking

6
HMI in Cars Overview Input

• Human Machine Interaction in Cars possible
  approaches
• Input
• Classic Knobs and switches
• Speech/ Speaker recognition
• Gesture recognition
• Mimics
• Eye tracking

7
HMI in Cars Overview Input

• Human Machine Interaction in Cars possible
  approaches
• Input
• Classic Knobs and switches
• Speech/ Speaker recognition
• Gesture recognition
• Mimics
• Eye tracking

8
HMI in Cars Overview Input

• Human Machine Interaction in Cars possible
  approaches
• Input
• Classic Knobs and switches
• Speech/ Speaker recognition
• Gesture recognition
• Mimics
• Eye tracking

9
HMI in Cars OverviewOutput

• Human Machine Interaction in Cars possible
  approaches
• Output
• Classic instruments
• Tactile control elements
• Speech
• Head up displays

10
HMI in Cars OverviewOutput

• Human Machine Interaction in Cars possible
  approaches
• Output
• Classic instruments
• Tactile control elements
• Speech
• Head up displays

11
HMI in Cars OverviewOutput

• Human Machine Interaction in Cars possible
  approaches
• Output
• Classic instruments
• Tactile control elements
• Speech
• Head up displays

12
HMI in Cars OverviewOutput

• Human Machine Interaction in Cars possible
  approaches
• Output
• Classic instruments
• Tactile control elements
• Speech
• Head up displays

13
Motivation for in-vehicle Speech recognition

• Problem Hand- and eye-free Control for
  in-vehicle HMIs crucial
• Solution Embedded Speech recognition
• Applications
• Navigation system, Telematic Application
• Voice controlled mobile phones
• Dictating e-mails
• Controlling multimedia devices
• Air-condition control, seat positioning, door
  locks, sunroof

14
Recognition modes

• Different applications demand different
  recognition modes
• Modes
• Speaker
• Dependency
• Speaking Style
• Vocabulary Size

15
Error RatesApplication Example HelloIC

• Speaker independent isolated word database for
• 100 words vocabulary 1 error rate
• 600 words 3
• 8000 words 10
• Speaker independent continuous speech database
  for 65000 words vocabulary 15
• Phillips HelloIC key features
• Up to 100 words storable on chip
• Up to 50 words active simultaneously
• Speaker dependent and independent recognition
• Word spotting and key word activation
• Continuous connected word recognition
• gt95 recognition rate in optimised settings

16
The Processing Chain
Dog
Dog
Dog
Dog
Hidden Markov Model
Short Time FT
Filter Bank
Pre- Processing
Classification
Pre- Processing
Classification
Feature Extraction
Feature Extraction
Artificial Neural Network
Noise Reduction Pre- Amplifier
mel-frequency cepstral coefficients
Linear Prediction
17
Feature extractionShort time Fourier
Transformation

• s(t) input signal w(t) window function
• Function of frequency and time
• Absolute value gives spectrogram

18
Sonogramformants and pitch

Wideband (left) and narrowband (right)
spectrograms of "ee ah ee ah" with level pitch
Wideband (left) and (narrowband) spectrgrams of
"aaaaaah" but with wild pitch changes.
19
Feature extractionFilter banks

• Measurement of energy in each frequency band
  provides the parameters
• Different frequency resolutions for each envelope
  
• simulation of cochlea like filtering

Bandpass Filter 1
Non- Linearity
Lowpass Filter
Sampling Rate Reduction
Amplitude Compression
Coef. 1
Speech
Bandpass Filter Q
Non- Linearity
Lowpass Filter
Sampling Rate Reduction
Amplitude Compression
Coef. Q
20
Feature extractionLinear Predictive Coding

• Basis speech is periodic, so we can predict it

predicted speech sample
past speech samples
error

e(n)
s(n)
21
Feature extractionmel-frequency cepstrum
coefficients

• Speech is a convolution of a source (source
  waveform) and a filter(vocal tract)

• Use of DCT provides cepstral co- coefficients
• MFCC Wrapping the frequence axis according to
  human
• pitch perception before the DCT
  
• The first 13 coefficients are characteristic

22
ClassificationBasics of HMM

• Assumption Speech is a statistacal process
• A Hidden Markov Model is
• a graph with N nodes
• determined by 3 Parameters
• start-probability vector
• transition-probability matrix
• emission-probability matrix

23
HMM in ASR

• Using HMM for speech recognition
• state phoneme
• observation feature vector
• emission-probability probability, that a certain
  phoneme is
• represented by a certain feature vector
• transition-probability matrix probability, that
  a phoneme is followed by another phoneme
• finding of most probable state sequence
  recognition
• determination of the optimal matrix training

24
Noisy Speech Resognition

• Biggest Problem for in-vehicle ASR Noise
• Noise sources engine, wind, fan, radio, road
  conditions, weather conditions (wiper), open
  window
• Solutions
• Training data for HMMs is collected from real car
  environments
• Band pass filter for specific, identified noise
  sources (e.g. engine)
• Statistical Noise Modelling
• Computational Auditory Scene Analysis (e.g.
  spatial location)

25
Example for optimised in-vehicle ASR

• Improvement by dirty training data

WER Word Error Rate Condition Level of noise
26
Speaker recognition

• Motivation Driver recognized by voice, personal
  car settings are used
• Recognition engine cepstral coefficients, HM
  Model for each user. Searching model with highest
  probability

Speaker recognition
Task
Speaker identification
Speaker verification
Text dependent
Text independent
Text dependent
Text independent
Method
27
Component Communication 1

• Export of HMI data
• 1st Approach conventional copper cabling
• 2nd Approach MOST Network

Mobile Phone

• Problem too many,
• too heavy cables

CD
HMIs
Navigation

• Solution MOST smaller weight and volume for
  cabling, less components, greater bandwidth,
  better EMC

Mobile Phone
CD
Navigation
HMIs
28
Component Communication 2

• MOST all from physical layer to application
  layer
• Possibility of object-orientated
  implementations
• New Systems get even more complex
• Requirement of a Software Architecture for
  Embedded Multimedia Systems

The HMI2000 Framework
GUI- Toolkit
Control- Logic
Devices
Base Classes
hardware specific software
MOST Base Classes
Application Base
Display- Driver
project independent software
OS-Abstraction Classes
Realtime Operating System
project specific software
Hardware-Abstraction Layer
29
FutureContextual Car Driver Interface

• Make driving a safer and more useful experience
• Context aware environment
• smart car environment and monitoring of driver
  state, with a wide range of input-output
  Modalities
• Input FaceCam, Fingerprint, GPS
• UI drivers Touchscreen, Gesture, Speech
• Output Speakers, Text-to-Speech, HUD
• intelligent agents