JTS: Tools for Implementing DomainSpecific Languages

1
JTS Tools for Implementing Domain-Specific
Languages

• D. Batory, B. Lofaso, Y. Smaradgakis
• Department of Computer Sciences
• University of Texas at Austin

2
Jakarta Tool Suite (JTS)

• DARPA-sponsored project to develop a next
  generation tools for building scalable
  domain-specific component technologies and their
  generators
• GenVoca generators goal is to automate tasks
  that experts would perform manually
• specify and automatically synthesize customized
  applications without coding through high degrees
  of automation

3
Jakarta Tool Suite (JTS)

• JTS to make programming languages extensible
• JTS itself is built using an extensible Java
• automate software development
• OO design pattern transformations
• substantially simplify the evolution of OO
  applications with tools that automatically apply
  design pattens

4
JTS Tools

• Consists of two separate tools
• Jak Language (extensible Java)
• Bali (a GenVoca generator of Jak languages and
  precompilers)

5
The Jak Language

• open, extensible superset of Java
• examine base extension

6
Metaprogramming in Jak

• Extends Java with
• AST constructors
• represent and manipulate programs as data
• adds LISP backquote/comma to Java
• package for AST traversals, manipulation
• arbitrary program transformations
• plus other goodies...

7
AST Constructors

• Added constructors for code fragments of
  different types (expression, statement, )
• similar to LISP backquote

AST_Exp t exp 7 x8 exp AST_Stm s
stm if (ygt4) return r stm t.unparse( )
// outputs 7 x8 s.unparse( ) // outputs
if (ygt4) return r
8
Composing Code Fragments

• Explicit escapes
• LISP comma

AST_Exp t exp 7 gt x8 exp AST_Exp s
stm if (exp(t)) return r stm s.unparse( )
// outputs if (7gtx8)
return r
9
AST Constructors

• Focusing on AST constructors for Java/Jak now
• Add AST constructors for other languages
• CORBA IDL
• Embedded SQL
• (subsets of) C, C

10
AST Traversals

• Package/classes for depth-first searches,
  manipulations
• support arbitrary program transformations

AstCursor c new AstCursor( ) Ast_Node root
// root of AST to search for (c.First(root)
c.More( ) c.PlusPlus( ) ) if (c.node
instanceof AstInterface) c.Delete( )
11
How JTS Works
12
SSTs versus ASTs

• Surface syntax tree (SST) is a syntactically
  correct parse tree
• tree constructors, modifiers guarantee syntactic
  correctness
• SSTs may not be semantically correct
• Abstract syntax tree (AST) is a type-checked SST
• with annotations to symbol table
• invoked by typecheck( ) method to root of SST

13
Extensibility of Jak

• Microsoft IP Intentions
• new grammar rules introduce new AST nodes with
  domain-specific semantics
• P3 cursor, container types, operations
• at reduction time, intention nodes are replaced
  with host language implementations
• P3 replaced abstract cursor, container AST nodes
  with their concrete Java implementations
• Jak/Java inherently open compiler

14
JTS Follows DSL Paradigm
15
How to...

• Produce lexers and parsers?
• Produce transform program?
• Where does GenVoca fit in?answers in remainder
  of talk...

16
Bali

• GenVoca Generator of
• DSL Precompilers

17
Family Languages

• Jak is a family of related languages
• versions with/without AST constructors
• with/without P3-generator extensions
• Classic library scalability problem
• n features with 2n combinations
• cannot build all 2n combinations by hand
• can generate them

18
Bali is a GenVoca Generator

• Assembles variants of Jak from components
• Components encapsulate primitive extensions
• AST constructors
• domain-specific generators like P3
• CORBA IDL
• Compositions of components specifies particular
  variant

19
Two Aspects

• Tool for writing compilers for Domain-Specific
  Languages (DSL)
• looks similar to other DSL-compiler tools
• specify syntax of DSL or language extension using
  annotated, extended BNF grammars
• GenVoca generator
• software component technology
• architectural, extensibility aspects

20
Bali Grammars

• Extended BNF grammar
• extended to handle repetitions
• e.g., POPART

StatementList ( Statement )
ArgumentList Argument ( , Argument )

21
Bali Grammars are Annotated

• by class to instantiate when production is
  recognized
• POPART, DIALECT, common OO design techniques

SelectionStmt IF ( Expr ) Statement
IfStm SWITCH ( Expr ) Block
SwitchStm
22
Bali Specifications

• Jak grammar
• 160 tokens
• 240 productions
• 800 lines
• Bali grammar
• 15 tokens
• 20 productions
• 73 lines

// bali spec lexical patterns grammar rules
23
What can be generated...

• Lexical analyzer
• Jlex (java version of lex)
• Bernie Lofaso (Jakarta Team Member)
• Parser
• Java_Cup (java version of yacc)
• Scott Hudson_at_Georgia Tech
• Inheritance hierarchy and AST classes
• constructors, unparsing, editing methods

24
What cant be generated...

• Type checking, reduction, optimization methods
• AST node specific
• hand code subclasses to Bali-generated class

25
Big Picture

• Traditional approach to building compilers for
  domain-specific languages
• clean model of proven technology
• Problem dont want to hand-code and
  hand-assemble each variant
• too expensive
• Solution GenVoca component technology
• components encapsulate primitive extensions
• compose components to form variants of Jak

26
GenVoca

• Model of parameterized programming
• addresses the architectural aspects of software
• building blocks of domains are refinements of
  fundamental domain abstractions
• components are implementations of refinements
• components composed by parameter instantiation
• visualize component composition as stacking layers

27
Quick Tutorial

• Software system is named composition of
  components called a type equation

System
System
Question what is relationship of components to
classes?
28
Key Idea Layering and Inheritance

• Inheritance hierarchy is an inverted layering
  hierarchy
• calling super classes same as calling lower layers

29
Relationship to GenVoca

• GenVoca components are large scale
• consistent refinements of multiple classes

30
Relationship to Jak
AstNode
T Ast With Java Kernel
31
Conclusions

• JTS integrates different technologies
• Java
• metaprogramming
• domain-specific language compiler tools
• architectures components/generators
• Powerful tool suite for
• building generators/precompilers for embedded,
  non-embedded languages
• used it to build P3 (data structures generator)
• JTS itself (bootstrapping using extensible Java)

32
The End