Gilad Bracha

1
Embedding DSLs in NewspeakNewShell, EBNF and
Hopscotch

• Gilad Bracha
• Ministry of Truth

2
Three DSLs

• Shell Scripting
• Parsing
• UI

3
Newspeak Syntax for Java Programmers
4
Newspeak Syntax for Java Programmers
5
Newspeak Syntax for C Programmers
6
Syntax Unary Expressions

• x foo
• means
• x.foo()

7
Syntax Unary Expressions

• x foo
• means
• x.foo()

8
Binary Expressions

• 6 3 factorial
• evaluates to 12.
• In javanese, wed write
• 6.(3.factorial()) or, really
• 6.plus(3.factorial())

9
Keyword Expressions

• Windows version 7 0i .
• is equivalent to all these
• Windows version (7 0i).
• Windows version (7 (0 i)).
• Windows.version(7 0.i())

10
Keyword Expressions

• add 1 to Cobol
• in javanese would be
• add(1, Cobol)
• Big advantage - no arity errors

11
Three DSLs

• Shell Scripting
• Parsing
• UI

12
NewShell

• class NewShell (
• ls value -l.
• ls - lt.
• echo value Hello World.
• svn value ls value svn//myDir.
• (ls value findTokens Character cr)
  collect ls.
• )()

13
Three DSLs

• Shell Scripting
• Parsing
• UI

14
Parser Combinators

• BNF
• id letter (letter digit)

15
Parser Combinators

• BNF
• id letter (letter digit)
• Newspeak
• id letter, (letter digit) star.

16
Parser Combinators

• BNF
• id letter (letter digit)
• Newspeak
• id letter, (letter digit) star.
• Javanese
• id letter().seq(letter().or(digit()).star())

17
How it Works

• id letter, (letter digit) star.

18
How it Works

• id letter, (letter digit) star.

19
How it Works

• id letter, (letter digit) star.

20
How it Works

• id letter, (letter digit) star.

21
How it Works

• id letter, (letter digit) star.

letter
22
How it Works

• id letter, (letter digit) star.

letter
23
How it Works

• id letter, (letter digit) star.

letter
24
How it Works

• id letter, (letter digit) star.

letter
25
How it Works

• id letter, (letter digit) star.

letter
26
How it Works

• id letter, (letter digit) star.

letter
27
How it Works

• id letter, (letter digit) star.

letter
letter
28
How it Works

• id letter, (letter digit) star.

letter
letter
29
How it Works

• id letter, (letter digit) star.

letter
digit
letter
30
How it Works

• id letter, (letter digit) star.

letter

digit
letter
31
How it Works

• id letter, (letter digit) star.

letter
star

digit
letter
32
How it Works

• id letter, (letter digit) star.

,
letter
star

digit
letter
33
Why is this Pretty?

• id letter, (letter digit) star.

34
Why is this Ugly?

• id letter().seq(letter().or(digit()).star())

35
Why is this Ugly?

• id letter().seq(letter().or(digit()).star())
• vs.
• id letter (letter digit)
• vs.
• id letter, (letter digit) star.

36
Why is this Ugly?

• id letter().seq(letter().or(digit()).star())
• vs.
• id letter (letter digit)
• vs.
• id letter, (letter digit) star.

37
Why is it Ugly?

• A programming language is low level when its
  programs require attention to the irrelevant
• - Alan Perlis

38
A Complete Grammar

• class ExampleGrammar1 ExecutableGrammar (
• digit charBetween 0 and 9.
• letter (charBetween a andz)
  (charBetween A andZ).
• id letter, (letter digit) star.
• identifier tokenFor id.
• hat tokenFromChar .
• expression identifier.
• returnStatement hat, expression.
• )()

39
Building an AST

• returnStatement hat, expression
  wrapperr e
  ReturnStatAST on e
  

40
Building an AST

• returnStatement hat, expression
• wrapperr e
• ReturnStatAST
  on e

wrapper
,
expression
41
Grammar

• returnStatement hat, expression
• wrapperr e
• ReturnStatAST
  on e

42
Semantic Action

• returnStatement hat, expression
• wrapperr e
• ReturnStatAST
  on e

43
Factor out Grammar

• Superclass (pure grammar specification)
• returnStatement hat, expression
• Subclass (AST builder)
• returnStatement (
• super returnStatement
• wrapperr e
  ReturnStatAST on e )

44
Modular Parser

• class ExampleParser1 ExampleGrammar1 () (
• id (
• super id
• wrapperfst snd fst asString, (String
  withAll snd)
• )
• identifier (
• super identifier
• wrapperv VariableAST new name v
  token
• 
  start v start end v end.
• )
• r

45
Modular Parser

• returnStatement (
• super returnStatement
• wrapperr e ReturnStatAST new expre
  
• 
  start r start end e end.
• )
• )

46
Extending a Grammar

• class ExampleGrammar2 ExampleGrammar1 (
• if tokenFromSymbolif.
• then tokenFromSymbolthen.
• else tokenFromSymbolelse.
• ifStatement if, expression, then, statement,
  else, statement.
• statement ifStatement returnStatement.
• )()

47
Mutual Recursion
returnStat
IfStmt
if
then
else

stmt
48
Mutual Recursion
digit
nil
letter
nil
nil
id
identifier
nil
hat
nil
expression
nil
returnStatement
nil
if
nil
then
nil
else
nil
ifStatement
nil
statement
nil
49
Mutual Recursion
digit
a parser
letter
nil
nil
id
identifier
nil
hat
nil
expression
nil
returnStatement
nil
if
nil
then
nil
else
nil
ifStatement
nil
statement
nil
50
Mutual Recursion
digit
a parser
letter
a parser
nil
id
identifier
nil
hat
nil
expression
nil
returnStatement
nil
if
nil
then
nil
else
nil
ifStatement
nil
statement
nil
51
Mutual Recursion
digit
a parser
letter
a parser
a parser
id
identifier
nil
hat
nil
expression
nil
returnStatement
nil
if
nil
then
nil
else
nil
ifStatement
nil
statement
nil
52
Mutual Recursion
digit
a parser
letter
a parser
a parser
id
identifier
a parser
hat
nil
expression
nil
returnStatement
nil
if
nil
then
nil
else
nil
ifStatement
nil
statement
nil
53
Mutual Recursion
digit
a parser
letter
a parser
a parser
id
identifier
a parser
hat
a parser
expression
nil
returnStatement
nil
if
nil
then
nil
else
nil
ifStatement
nil
statement
nil
54
Mutual Recursion
digit
a parser
letter
a parser
a parser
id
identifier
a parser
hat
a parser
expression
a parser
returnStatement
a parser
if
nil
then
nil
else
nil
ifStatement
nil
statement
nil
55
Mutual Recursion
digit
a parser
letter
a parser
a parser
id
identifier
a parser
hat
a parser
expression
a parser
returnStatement
a parser
if
a parser
then
nil
else
nil
ifStatement
nil
statement
nil
56
Mutual Recursion
digit
a parser
letter
a parser
a parser
id
identifier
a parser
hat
a parser
expression
a parser
returnStatement
a parser
if
a parser
then
a parser
else
nil
ifStatement
nil
statement
nil
57
Mutual Recursion
digit
a parser
letter
a parser
a parser
id
identifier
a parser
hat
a parser
expression
a parser
returnStatement
a parser
if
a parser
then
a parser
else
a parser
ifStatement
nil
statement
nil
58
Mutual Recursion
digit
a parser
letter
a parser
a parser
id
identifier
a parser
hat
a parser
expression
a parser
returnStatement
a parser
if
a parser
then
a parser
else
a parser
ifStatement
boom!
statement
nil
59
Mutual Recursion

• Addressed with laziness in Haskell
• Can be addressed with closures
• ifStatement if, expression, then,
  statement,
• else, statement.
• statement ifStatement returnStatement.

60
Mutual Recursion

• Addressed with laziness in Haskell
• Can be addressed with closures
• ifStatement if, expression, then,
  statement,
• else, statement.
• statement ifStatement returnStatement.

61
Being Reflective rather than Lazy

• Use Reflection instead of Laziness.
• Framework initializes all slots to stand-in
  parsers which act as forward references

62
Mutual Recursion
digit
fwd ref
letter
fwd ref
fwd ref
id
identifier
fwd ref
hat
fwd ref
expression
fwd ref
returnStatement
fwd ref
if
fwd ref
then
fwd ref
else
fwd ref
ifStatement
fwd ref
statement
fwd ref
63
Mutual Recursion
digit
a parser
letter
fwd ref
fwd ref
id
identifier
fwd ref
hat
fwd ref
expression
fwd ref
returnStatement
fwd ref
if
fwd ref
then
fwd ref
else
fwd ref
ifStatement
fwd ref
statement
fwd ref
64
Mutual Recursion
digit
a parser
letter
a parser
fwd ref
id
identifier
fwd ref
hat
fwd ref
expression
fwd ref
returnStatement
fwd ref
if
fwd ref
then
fwd ref
else
fwd ref
ifStatement
fwd ref
statement
fwd ref
65
Mutual Recursion
digit
a parser
letter
a parser
a parser
id
identifier
fwd ref
hat
fwd ref
expression
fwd ref
returnStatement
fwd ref
if
fwd ref
then
fwd ref
else
fwd ref
ifStatement
fwd ref
statement
fwd ref
66
Mutual Recursion
digit
a parser
letter
a parser
a parser
id
identifier
a parser
hat
fwd ref
expression
fwd ref
returnStatement
fwd ref
if
fwd ref
then
fwd ref
else
fwd ref
ifStatement
fwd ref
statement
fwd ref
67
Mutual Recursion
digit
a parser
letter
a parser
a parser
id
identifier
a parser
hat
a parser
expression
fwd ref
returnStatement
fwd ref
if
fwd ref
then
fwd ref
else
fwd ref
ifStatement
fwd ref
statement
fwd ref
68
Mutual Recursion
digit
a parser
letter
a parser
a parser
id
identifier
a parser
hat
a parser
expression
a parser
returnStatement
a parser
if
fwd ref
then
fwd ref
else
fwd ref
ifStatement
fwd ref
statement
fwd ref
69
Mutual Recursion
digit
a parser
letter
a parser
a parser
id
identifier
a parser
hat
a parser
expression
a parser
returnStatement
a parser
if
a parser
then
fwd ref
else
fwd ref
ifStatement
fwd ref
statement
fwd ref
70
Mutual Recursion
digit
a parser
letter
a parser
a parser
id
identifier
a parser
hat
a parser
expression
a parser
returnStatement
a parser
if
a parser
then
a parser
else
fwd ref
ifStatement
fwd ref
statement
fwd ref
71
Mutual Recursion
digit
a parser
letter
a parser
a parser
id
identifier
a parser
hat
a parser
expression
a parser
returnStatement
a parser
if
a parser
then
a parser
else
a parser
ifStatement
fwd ref
statement
fwd ref
72
Mutual Recursion
digit
a parser
letter
a parser
a parser
id
identifier
a parser
hat
a parser
expression
a parser
returnStatement
a parser
if
a parser
then
a parser
else
a parser
ifStatement
a parser
statement
fwd ref
73
Mutual Recursion
digit
a parser
letter
a parser
a parser
id
identifier
a parser
hat
a parser
expression
a parser
returnStatement
a parser
if
a parser
then
a parser
else
a parser
ifStatement
a parser
statement
a parser
74
Mutual Recursion
returnStat
IfStmt
if
then
else

FwdRef
stmt
75
Being Reflective rather than Lazy

• Use Reflection instead of Laziness.
• Framework initializes all slots to stand-in
  parsers which act as forward references
• When grammar is completely initialized, stand-ins
  are connected to originals and forward calls to
  them

76
Mutual Recursion
returnStat
IfStmt
if
then
else

FwdRef
stmt
77
The Mirror is Mightier Than the Sloth
78
Grammar1
Grammar2
Parser1
Parser2
79
Grammar1
Grammar2
Parser1
Parser2
80
Grammar1
Grammar2
Parser1
Parser2
81
Composing Parsers

• class ExampleParser2 ExampleParser1 mixin gt
  
• 
  ExampleGrammar2 ()
• (
• ifStatement (
• super ifStatement
• wrapperifKw e thenKw s1 elseKw s2
  
• )
• )

82
Grammar is shared executable specification

• Shareable among several IDEs/compilers that
  require distinct ASTs
• Shareable among other language tools syntax
  colorizers, postprocessing tools

83
Three DSLs

• Shell Scripting
• Parsing
• UI

84
Hopscotch

• A GUI application framework
• Higher level of abstraction than widgets
• Provides DSL for composing presentations out of
  pieces known as fragments

85
Fragments

• Fragments of presentation
• Atomic widgets correspond to leaf fragments
• Aggregates represented via compound fragments
• Composed via combinators

86
Fragment Combinators

• blank, filler
• buttonaction, linkaction, image,
  menuWithLabelsAndActions
• row, column, centered, collapsedexpanded
• deferred, draggablesubject

87
Dispatching up the fragment hierarchy

• Dispatching up the visual containment hierarchy
  makes sense
• Old idea (at least since NewtonScript)
• Realized using doesNotUnderstand

88
Demo
89
Connections

• Self Smalltalk

90
Connections

• Parser Combinators.
• Too much to survey some highlights
• Parsec (Haskell parser combinator library)
• Swierstra et al.
• Sparsec (Scala parser combinator library)
• PEGs (Ford, POPL04, ICFP)

91
Connections

• Parser Combinators (continued)
• Tedir (Plesner-Hansens masters thesis)
• Ometa (OOPLA 07 DLS)

92
Conclusions

• Newspeak supports DSLs via
• Unconventional syntax
• keywords
• operators as methods
• implicit receivers
• Dynamic typing and Reflection
• doesNotUnderstand

93
Conclusions

• Newspeak also supports DSLs via
• Mixins, Modules, Hierarchy inheritance
• IDE
• Combinators are a powerful technique
• Compositional
• Useful in libraries and languages

94
Status

• Available at http//newpeaklanguage.org
• Open source under Apache 2.0 license
• Work in Progress
• Expect some tweaks to syntax and semantics
• Implementation still not complete - especially
  libraries

95
Credits

• Peter Ahe
• Vassili Bykov
• Yaron Kashai
• Bill Maddox
• Eliot Miranda

96

• This file is licensed under the Creative Commons
  Attribution ShareAlike 3.0 License. In short you
  are free to share and make derivative works of
  the file under the conditions that you
  appropriately attribute it, and that you
  distribute it only under a license identical to
  this one. Official license.
• The Newspeak eye used in the bullets,
  slide background etc. was designed by Victoria
  Bracha and is used by permission.