Development of protocols WP4

1
Development of protocolsWP4 T4.2

• TRT
• Nancy review
• July 6th 7th 2006

2
Planning

• What was decided in Torino
• What has been already done
• What remains to do

3
Calendar
?
?
2007
2005
2006
m18
m24
m29
m27
T4.34 Evaluation on the fixed and mobile
platforms
Nov.06
Dec.05
June06
Sept.06
T4.2 Development of protocols
M4.1
D4.2
Specification of evaluation protocols
Functional integration on both platforms
completed
M3.2

• TRT (leader , 3 mm)
• Loquendo (2), TUC (2)
• UGR (1), Loria (1), THAV (1)

4
Torinos meeting

• Mobile platform
• Environment
• Grammar
• Extended version
• Panel/ the users
• Colleagues
• 10 to 20
• Location
• An equipped room, noise diffusion
• Factory noise ? hangar noise (ask Airbus)
• Different levels (from clean to ? dB, at the
  microphone capsule level)
• A test scenario
• The maintenance of aircrafts
• Criteria
• Objective measures
• Response time
• SAC
• TCR
• Subjective measures

• Fixed platform
• Environment
• Grammar
• Thav grammar (provided at the end of April)
• Speech input
• Colleagues
• 20 non native speakers (badgtgood accent)
• Location
• The THAV cockpit simulator
• Multi-speaker noise diffusion system
• MM array
• A test scenario
• Depends on the grammar
• Criteria
• Objective measures
• SAC (avg and statistics through speakers)
• Response time
• Subjective measures
• no pilot

5
Evaluation protocol

• Test Environment
• Panel
• Test space
• Equipment
• loudspeakers, sound card, PC, connection, sound
  data, HSPPDA or Cockpit simulator display
• Software
• gathering results, spectral and volume
  calibration
• Grammar
• Evaluation criteria
• Objective measures
• Subjective measures
• Result analysis
• Test scenario
• Preparation
• Test training
• Test
• After test

6
Evaluation protocol

• Test Environment
• Panel
• Test space
• Equipment
• loudspeakers, sound card, PC, connection, sound
  data, HSPPDA or Cockpit simulator display
• Software
• gathering results, spectral and volume
  calibration
• Grammar
• Evaluation criteria
• Objective measures
• Subjective measures
• Result analysis
• Test scenario
• Preparation
• Test training
• Test (depends on the grammar)
• After test

7
Panel

• 20 non-native English speakers
• M/F ratio 50
• Well-balanced origin ratio (France,
  Greece/Creete, Italy, Spain)
• Various English levels
• Names?

8
Test space (fixed pltf.)
L 12m W 7m H 3m DWLS gt1.5m DLS gt1.5m

• Problem
• we do not create a realistic sound field for the
  multi-mic array
• solution we will record noise in a real cockpit
  and test different multi LS diffusion (4, 6, 12,
  ?)
• an acoustic distance is necessary to be computed
  to find the closest LS placing

Cockpit Simulator
Multi-mic device controller HSR
LS-right
PPT
KCCU
Multi-mic array
Command, control and storage area HSR
LS-left
Cockpit Simulator Display Unit PC
9
Test space (mobile pltf.)
L W H DWLS gt1.5m DLS gt1.5m HLS 1.75m
upper view
LS-right

• Where?
• A real hangar would be the best place to have a
  realistic evaluation but it is difficult to
  control the noise level in such a place

Command, control and storage area
LS-left
side view
Command, control and storage area
LSstand
test area
lectern
10
Equipment

• Loudspeakers
• sufficient power for a 300m3 room ? power gt100
  Watts
• wide frequency range
• flat frequency response
• YAMAHA HS80M (x2)

39 cm
? 11 kg
11
Equipment

• Soundcard
• mini 20bits / 48kHz
• SNR gt 100dB
• balanced outputs
• digital PC link
• Behringer F-CONTROL AUDIO FCA202
• 24bits / 96kHz
• SNR 100dB
• Fire Wire PC link
• Connection
• PC to soundcard FireWire cable (x1)
• LS to soundcard Balanced XLR-Jack cable (x2)

• The sound card can be replaced by an ITCs.

12
Equipment

• Sound data
• Sources
• Cockpit noise
• Hangar noise (Aircraft tests, workshop, )
• possible recordings in Le Bourget
• Format
• WAV PCM 20bits / 48kHz
• Computer
• A control PC (laptop)
• to launch sounds
• to gather results

13
Software

• Software
• Objective results gathering
• Plug-in inside the HSR PC for the fixed pltf.
• For the mobile pltf. the plug-in is on the
  control PC, a link between PC and PDA is needed.
• Subjective results gathering
• On the control PC (excel sheet) for the mobile
  pltf.
• LS spectral compensation and volume calibration

14
Fixed platform grammar
15
Mobile platform grammar

• Aircraft maintenance
• large place with high sound level differences
• general observations around the plane before
  first flight (wheels, windows, oil, gaz, motor
  tests)
• noisy environment BUT simple grammar (Yes/no
  answers)
• Repairs in workshop
• little quiet workshops
• Specific unplugged aircraft pieces
• large grammar BUT not noisy environment
• The grammar must present a sufficient variability
  so it will be the second one.

16
Mobile platform grammar

• Exemple

17
Criteria

• Objective measures
• Response time, RT (s)
• Time between the end of speech and the system
  response
• It must be the lower as possible
• Sentence Accuracy, SAC ()
• Ratio between the number of correct sentences and
  the total of pronounced sentences
• It must ideally be 100
• Word accuracy, WAC ()
• Content word accuracy, CWA ()
• Only the words in connection with the topic (non
  link words)
• Task Completion Index, TCI ()
• Average number of repetitions before the sentence
  is understood
• It must be near zero meaning that the sentence is
  immediately understood (no repetition needed)
• Time Before Completed Task, TBCT (s)
• Time between the end of the speech first try and
  the system good response
• Lower is better

18
Filled form
Objective measures of one user test (example for
mobile pltf.)
19
Criteria

• Subjective measures
• Easiness of use, EASE 1-5
• Describes how the tester found the easiness of
  entering information from not easy (1) to very
  easy (5)
• Can be correlated with the user cognitive
  workload, the number of actions before accessing
  to the information, the clarity of the visual
  interface and the synthetic voice.
• Naturalness of interaction, NAT 1-5
• Gives an indication about way it is natural to
  use the system from not natural (1) to very
  natural (5)
• Can be correlated with intuitiveness the closer
  to human-human interaction it is, the more
  natural it is
• Slightly differs from EASE a system can be easy
  to use (because everything is clearly presented)
  but not natural because of a bulky device or
  because the interface (voice or pen) is not
  adapted.
• Completed task performance perception 1-5
• Efficiency 1-5
• Overall assessment 1-5

20
Scenarii

• Mobile platform
• Two session modalities vocal input (VI) and pen
  input (PI).
• Two SNR clean (no noise), noisy
• the background noise level is adjusted at the
  same level as in the hangar
• Some maintenance reports to fill (5 1 for
  training)
• Visual feedback on the PDA
• Fixed platform
• Two kinds of session with close-talking
  microphone (Plantronics or AT?) and with the
  multi-microphone array.
• Four SNR clean (no noise), noisy taxi and
  takeoff
• Comparison with baseline.
• A list of commands to pronounce
• Visual feedback

21
Result analysis
Global result table (mobile plft.)
22
Result analysis
Global result table (fixed plft.)
23
To-do list
24
To-do

• Redaction
• Write a short paragraph for the IP
• Protection of the evaluation protocol
• Put presentation on Twiki
• Justify ICASSP attending
• Recording into a hangar
• Do not need to have a multi-mic
• Stereo recording / diffusion is enough
• Send the aircraft part list to LOQ

25
To-do

• Recording into a cockpit
• 6 1 Shure microphones (left, right, back,
  forward, up, down)
• At least 2 microphones must be at the same
  distance as in the multi-mic array
• Need to access to a cockpit and stick the 7
  microphones
• Recording during taxiing, take-off, en route and
  landing
• Need a self-alimentation for the devices (octamic
  soundcard laptop)

26
To-do

• Measuring evaluation results
• Fixed platform In the HSR PC
• Mobile platform In the monitoring PC
• It is better to keep the scenarii in the
  monitoring PC (easy to modify, )
• Need a link between the PDA and the PC

27
To-do

• Global evaluation
• Propose common measures
• Dominique prefers WER and SER ? ask all the
  partners
• Push for using the Hiwire database
• Need to add cockpit noise artificially (UGR)
• Different noise levels
• artificial noise