Spoken Programs or Programmers say the darndest things

1
SpokenProgramsorProgrammers saythe darndest
things!

• Andrew Begel, Susan L. Graham
• Computer Science Division, EECS
• University of California, Berkeley

VL/HCC 2005
2
Motivation
while (counter lt limit) ?

• Programmers conventionally use keyboard
• Long hours at keyboard leads to higher risk of
  RSI
• Can a programmer code using speech?
• Can a computer understand what the developer says?

1
3
Programming by Voice
while counter is lessthan limit do ...

• Our Goal
• Find out how developers use code verbally. Use
  this to develop a naturally verbalizable input
  form.
• Build development environment that supports
  verbal authoring, navigation, modification.
• Extend conventional compiler analyses to support
  ambiguities generated by speech.
• Learn how developers can use voice-based
  programming, and iterate design.

2
4
Programming by Voice Challenges

• Speech is inherently ambiguous.
• Programming languages and tools were designed to
  be unambiguous.
• Speech tools are poorly suited for programming
  tasks.
• Programmers are not used to verbal software
  development.

3
5
Programming by Voice

• for (int i 0 i lt 10 i )

4
6
Current Tools are Awkward!
VoiceCode Desilets 2004
for loop after left paren declare india of
type integer assign zero after semi recall
one less than ten after semi recall one
increment after left brace

• for ( )

5
7
Current Tools are Awkward!
VoiceCode Desilets 2004
for loop after left paren declare india of
type integer assign zero after semi recall
one less than ten after semi recall one
increment after left brace

• for (int i 0 )

5
8
Current Tools are Awkward!
VoiceCode Desilets 2004
for loop after left paren declare india of
type integer assign zero after semi recall
one less than ten after semi recall one
increment after left brace

• for (int i 0 i lt 10 )

5
9
Current Tools are Awkward!
VoiceCode Desilets 2004
for loop after left paren declare india of
type integer assign zero after semi recall
one less than ten after semi recall one
increment after left brace

• for (int i 0 i lt 10 i )

5
10
How do Programmers Speak Code?

• 10 programmers read Java code out loud (Begel
  05)
• Graduate students in Computer Science
• Five knew Java, five did not
• Five were native English speakers, five were not
• Five were educated in U.S.A., five were not
• Read pre-written code into tape recorder
• As if speaking to a sophomore-level CS undergrad
  who knows Java, but does not know the program
• Most programmers spoke the same way

6
11
Natural Language Verbalization
for int i equals zero i less than ten i plus plus

• for (int i 0 i lt 10 i )

7
12
How Do Programmers Speak Code?
Spoken Words Can Be Hard To Write Down 2
8
13
How Do Programmers Speak Code?
Spoken Words Can Be Hard To Write Down 2 ? 2,
two, to, too
8
14
How Do Programmers Speak Code?
Spoken Words Can Be Hard To Write Down 2 ? 2,
two, to, too print
8
15
How Do Programmers Speak Code?
Spoken Words Can Be Hard To Write Down 2 ? 2,
two, to, too print ? print, Print
8
16
How Do Programmers Speak Code?
Spoken Words Can Be Hard To Write Down 2 ? 2,
two, to, too print ? print, Print drop stack
process
8
17
How Do Programmers Speak Code?
Spoken Words Can Be Hard To Write Down 2 ? 2,
two, to, too print ? print, Print drop stack
process ? drop stack processdrop
stackprocessdropstack processdropstackprocess
8
18
How Do Programmers Speak Code?
Many Ways to Say the Same Thing bari
9
19
How Do Programmers Speak Code?
Many Ways to Say the Same Thing bari ? bar sub
i, bar of i, i from bar
9
20
How Do Programmers Speak Code?
Many Ways to Say the Same Thing bari ? bar sub
i, bar of i, i from bar .
9
21
How Do Programmers Speak Code?
Many Ways to Say the Same Thing bari ? bar sub
i, bar of i, i from bar .? period, dot
9
22
How Do Programmers Speak Code?
Many Ways to Say the Same Thing bari ? bar sub
i, bar of i, i from bar .? period, dot
9
23
How Do Programmers Speak Code?
Many Ways to Say the Same Thing bari ? bar sub
i, bar of i, i from bar .? period, dot ?
right brace, close the if, end method
9
24
How Do Programmers Speak Code?
Many Ways to Say the Same Thing bari ? bar sub
i, bar of i, i from bar .? period, dot ?
right brace, close the if, end method println
9
25
How Do Programmers Speak Code?
Many Ways to Say the Same Thing bari ? bar sub
i, bar of i, i from bar .? period, dot ?
right brace, close the if, end method println ?
print line, print lin, print l n
9
26
How Do Programmers Speak Code?
One Utterance May Mean Many Things object
stack
10
27
How Do Programmers Speak Code?
One Utterance May Mean Many Things object stack
? Object stackobject.stackobject(stack)object(
).stack()
10
28
How Do Programmers Speak Code?
One Utterance May Mean Many Things object stack
? Object stackobject.stackobject(stack)object(
).stack() array sub i plus plus
10
29
How Do Programmers Speak Code?
One Utterance May Mean Many Things object stack
? Object stackobject.stackobject(stack)object(
).stack() array sub i plus plus ?
arrayiarrayi
10
30
How Do Programmers Speak Code?
People Have Trouble Saying Some
Things System.out.println
11
31
How Do Programmers Speak Code?
People Have Trouble Saying Some
Things System.out.println ? system out print
linesystem dot out print linesystem dot out dot
print line
11
32
How Do Programmers Speak Code?
People Have Trouble Saying Some
Things System.out.println ? system out print
linesystem dot out print linesystem dot out dot
print line (int)foo
11
33
How Do Programmers Speak Code?
People Have Trouble Saying Some
Things System.out.println ? system out print
linesystem dot out print linesystem dot out dot
print line (int)foo ? cast foo to integer int
foocast something to integer. that something is
foo.
11
34
How Do Programmers Speak Code?
Sometimes They Describe the Code And then
theres a class.
12
35
How Do Programmers Speak Code?
Sometimes They Describe the Code And then
theres a class. Set all the fields of that
object to null.
12
36
How Do Programmers Speak Code?
Sometimes They Describe the Code And then
theres a class. Set all the fields of that
object to null. All of these are just assignment
operations.
12
37
Design Tradeoffs
Command Language Computer-centric Easy to
analyze, but prescriptive
NaturalLanguage Human-centric Flexible, but
ambiguous
Programming by Voice
13
38
Spoken Java

• Semantically identical to Java
• Syntactically easier to say than Java
• Design methodology generalizable to any computer
  language

14
39
A More Natural Way to Code
public class symbol implements serializable

• public class Symbol implements Serializable

15
40
A More Natural Way to Code
static hash map hash table gets new hash map

• public class Symbol implements Serializable
• static HashMap hashtbl new HashMap()

15
41
A More Natural Way to Code
end the class

• public class Symbol implements Serializable
• static HashMap hashtbl new HashMap()

15
42
A More Natural Way to Code
for int i equals zero i less than ten i plus plus

• for (int i 0 i lt 10 i )

16
43
Too Many Ambiguities

• for (int i 0 i lt 10 i )

16
44
Sometimes Its Non-Obvious
for (times 8 file(2, load) times one)

• fore 8 file.tooLode.times won

4 times ate(file).to(load).equals(1)
17
45
SPEED Speech Editor

• Build an editor that supports naturally
  verbalized programs
• SPEED SPEech EDitor
• Based on Dragon Naturally Speaking, Eclipse IDE,
  Harmonia
• Spoken Java Language for Composition
• Spoken Command language for Navigation, Editing,
  Template instantiation, Refactorings, Search
• Visual feedback for training

18
46
Future Study - SPEED Usability

• Goal Understand how SPEED can be used by
  expert programmers
• Train expert Java programmers on SPEED
• Author new code
• Build a Linked List data structure with
  associated algorithms
• Modify existing code
• Change abstraction representation and update
  algorithms

19
47
Metrics

• User Metrics
• Task completion time
• Vocabulary mistakes, Forgotten vocabulary
• Grammatical mistakes, Grammatical substitutions
• SPEED Metrics
• Word tokenization errors
• Disambiguation errors
• Irresolvable ambiguities

20
48
Spoken Programs

• Learn how developers speak code out loud
• Analyze speech and categorize ambiguities
• Use study to design naturally verbalizable input
  for programming
• Create programming tools designed for spoken
  interaction
• Evaluate design and tools by studying programmers
  using voice for software development

21
49
End of Talk
Any questions? Andrew Begel
abegel_at_cs.berkeley.edu
22