Audio/Speech

1
Audio/Speech

• CS376 November 4, 2004
• as presented by Jessica Kuo

2
Audio/Speech Readings

• Speech Acts A Spoken-Language Framework
• The Audio Notebook Paper and Pen Interaction
  with Structured Speech
• A Confederation of Tools for Capturing and
  Accessing Collaborative Activity

3
SpeechActs

• What is it?
• Platform for developing spoken natural language
  applications
• What are its goals?
• Accessibility to developers who are not speech
  experts
• Context awareness
• Technology independence

4
SpeechActs

• Telephone-based applications for business
  travelers
• E-mail reading, calendar lookup, currency
  conversion, weather forecast

• Compare to palmOne Treo Smartphone

• Dictating e-mails
• Hands-free usage

5
SpeechActs

• Overall Structure

6
SpeechActs

• Challenges of simulating human conversation
• Conversational pacing
• Is it pausing because its broken or thinking?
• Explicit error corrections
• User prompting
• Possible user actions are hidden

7
SpeechActs (Questions)

• Stellar examples of voice interaction systems?
• or less than stellar (ATT, airline
  reservations)
• Last comments?

8
Audio Notebook

• User Scenarios
• Student taking notes during class
• Reporter taking notes during interview
• Problem
• Capturing information during an audio
  presentation
• Dividing attention between two sources

9
Audio Notebook

• Synchronization of handwritten notes and audio
• Mechanism
• Ordinary paper notebook
• Digitizing tablet pen, digital audio recorder
• Structured audio recording
• By user through note-taking
• By talker through changes in pitch, pauses, energy

10
Audio Notebook

• Features
• Audio scrollbar of 80 LED lights
  (green-red-orange)
• Audio Snap-to-Grid using phrase detection
• -stormed about it a little bit and we
  brainstormed about it a little bit
• Topic suggestions
• uh really neat thats called phase conjugation
  okay number 4 is going to be metrology and
  microscopy, and uh as an example of this thats
  going to be uh really neat thats called phase
  conjugation

11
Audio Notebook

• Fieldwork Results (2 students, 2 reporters)
• Taking notes normally vs. as an outline
• Selective re-listening for areas of confusion
• Listening at 2x speed
• Locating and transcribing quotes
• Benefits realized when Audio Notebook not
  available for use

12
Audio Notebook (Questions)

• How might this change the way you take notes?
• What happens when you tear off some of your
  notes?
• How useful is Snap-to-grid phrase detection?
• How useful are topic suggestions and the control
  over their granularity?
• Does the audio scrollbar make sense when my
  recording covers a long time? (? LEDs/minute)
• Last comments?

13
Capturing Accessing Collaborative Activity

• Coral - confederation of tools
• Goals
• Support activities during collaborative work
• Capture records of activities
• Provide ways for activities to be indexed
• Provide access to indexed activities to make the
  record an effective resource

14
Capturing Accessing Collaborative Activity

• Types of indices
• Intentional annotations
• Side-effect indices
• Derived indices
• Post hoc indices
• Coral focuses on intentional annotations and
  side-effect indices
• Goal extract indices from structure of activity
  instead of imposing regularity upon process

15
Capturing Accessing Collaborative Activity

• Process of assessing and managing invention
  proposals (IPs) at PARC
• Researchers submit IPs
• Meeting to evaluate IPs (discussion, conclusion,
  summary documentation)
• Setup
• LiveBoard microphones manager with laptop
• Discussion notes are beamed to the LiveBoard
• Conclusion manager notes action items on
  LiveBoard
• Documentation manager re-listens to audio

16
Capturing Accessing Collaborative Activity

• Compare audio access with Audio Notebook

17
Capturing Accessing Collaborative Activity (Q)

• Coral recorded audio only. Would it have been
  beneficial to have video?
• Is using speaker identification as indices
  helpful?
• Contrast with Audio Notebook
• Single user vs. Collaborative environment
• Constraints due to unique domain it was designed
  in?
• Last comments?