MonetDB/XQuery Technology Preview 1

1
MonetDB/XQuery Technology Preview 1

• Stefan Manegold
• CWI
• Amsterdam
• http//monetdb.cwi.nl/ - http//pathfinder-xquer
  y.org/

2
European Pathfinder Team

• University of Konstanz (Germany)
• Torsten Grust, Jens Teubner, Jan Rittinger
• University of Twente (Netherlands)
• Maurice van Keulen, Jan Flokstra
• CWI, Amsterdam (Netherlands)
• Peter Boncz, Stefan Manegold, Sjoerd Mullender

3
(No Transcript)
4

Results Performance (1)
5
Results Performance (2)
6
Story

• XQuery Example
• Relational XQuery
• System Architecture
• XML Encoding
• Science Reseach
• Scalability
• Outlook
• Conclusions
• Roadmaps
• Release References

7
XQuery Example

• For each author, return number of books and
  receipts
• for books published in the past 2 years, ordered
  by name
• let cat fndoc(www.bn.com/catalog.xml),
  (Documents)
• sales fndoc(www.publishersweekly.com/
  sales.xml)
• for author in distinct-values(cat//author)
  (Grouping)
• let books cat//book_at_year gt 2003 and
  author a, (Sel.)
• receipts sales/book_at_isbn
  books/_at_isbn/receipts (Join)
• order by author
  (Ordering)
• return
• ltsalesgt
  (XML Construction)
• author
• ltcountgt fncount(books) lt/countgt
  (Aggregation)
• lttotalgt fnsum(receipts) lt/totalgt
• lt/salesgt

8
XQuery Example

• For each author, return number of books and
  receipts
• for books published in the past 2 years, ordered
  by name
• let cat fndoc(www.bn.com/catalog.xml),
  ?Documents?
• sales fndoc(www.publishersweekly.com/
  sales.xml)
• for author in distinct-values(cat//author)
  ?Grouping?
• let books cat//book_at_year gt 2003 and
  author a, ?Sel.?
• receipts sales/book_at_isbn
  books/_at_isbn/receipts ?Join?
• order by author
  ?Ordering?
• return
• ltsalesgt ?XML
  Construction?
• author
• ltcountgt fncount(books) lt/countgt
  ?Aggregation?
• lttotalgt fnsum(receipts) lt/totalgt
• lt/salesgt

9
XQuery Systems 2 Approaches

• Existing native XML/XQuery systems are built
  from scratch
• Galax, Saxon,
• X-Hive, Tamino,
• (Still have to) re-invent optimization technology
• Our approach
• Build XQuery system on top of an RDBMS
• Leverage mature relational technology
• to achieve efficient XQuery processing

10
Architecture
11
XML in an RDBMS XPath Accelerator
12
Science Research

• More research lead to more optimization
• Join Recognition
• Embedded XPath processing
• Order Awareness
• Various scientific publications

13
Results Scalability (3)
14
Conclusions

• Relational approach
• Works
• Is fast
• Is scalable
• Crucial Optimizations
• Join recognition
• Embedded XPath processing
• Order awareness

15
Roadmap

• 30-05-05 MonetDB/XQuery 4.8/0.8 Mercurius
• Developers Release / Technology Preview 1
• 30-09-05 MonetDB/XQuery 4.10/0.10 Venus
• Student Release / Technology Preview 2
• XUpdate, Algebraic Query Optimization
• 30-12-05 MonetDB/XQuery 4.12/1.12 Mars
• Final Release
• Application Programming Interfaces
• End-User Front-Ends

16
Open Source Release

• MonetDB Pathfinder on SourceForge
• Mozilla-like License
• MonetDB homepage
• http//monetdb.cwi.nl/
• Pathfinder homepage
• http//pathfinder-xquery.org/
• Developers website
• http//sf.net/projects/monetdb/

17
(No Transcript)
18
(No Transcript)
19
(No Transcript)
20
(No Transcript)
21
(No Transcript)
22
(No Transcript)
23
Outline

• Basic XML / XQuery
• Introduction of Pathfinder and MonetDB projects
• Relational XQuery
• XPath steps in the pre/post plane
• Translating for-loops, and beyond
• Optimizations
• Order prevention
• Loop-Lifted Staircase join
• Join recognition
• Outlook
• Conclusions
• Roadmaps

24
Outline

• Basic XML / XQuery
• Introduction of Pathfinder and MonetDB projects
• Relational XQuery
• XPath steps in the pre/post plane
• Translating for-loops, and beyond
• Optimizations
• Order prevention
• Loop-Lifted Staircase join
• Join recognition
• Outlook
• Conclusions
• Roadmaps

25
XML

• Standard, flexible syntax for data exchange
• Regular, structured data
• Database content of all kinds Inventory,
  billing, orders,
• Small typed values
• Irregular, unstructured text
• Documents of all kinds Transcripts, books, legal
  briefs,
• Large untyped values
• Lingua franca of B2B Applications
• Increase access to products services
• Integrate disparate data sources
• Automate business processes
• and numerous other application domains
• Bio-informatics, library science,

26
XML A First Look

• XML document describing catalog of books
• lt?xml version"1.0" encoding"ISO-8859-1" ?gt
• ltcataloggt
• ltbook isbn"ISBN 1565114302"gt
• lttitlegtNo Such Thing as a Bad Daylt/titlegt
• ltauthorgtHamilton Jordanlt/authorgt
• ltpublishergtLongstreet Press, Inc.lt/publishergt
• ltprice currency"USD"gt17.60lt/pricegt
• ltreviewgt
• ltreviewergtPublisherlt/reviewergt This book is
  the moving account of one man's successful
  battles against three cancers ... lttitlegtNo Such
  Thing as a Bad Daylt/titlegt is warmly recommended.
  
• lt/reviewgt
• lt/bookgt
• lt!-- more books and specifications --gt
• lt/cataloggt

27
XQuery 1.0

• Functional, strongly-typed query language
• XQuery 1.0
• XPath 2.0 for navigation, selection,
  extraction
• A few more expressions
• For-Let-Where-Order By-Return (FLWOR)
• XML construction
• Operators on types
• User-defined functions modules
• Strong typing

28
XSLT vs. XQuery

• XSLT 1.0 XML ? XML, HTML, Text
• Loosely-typed scripting language
• Format XML in HTML for display in browser
• Must be highly tolerant of variability/errors in
  data
• XQuery 1.0 XML ? XML
• Strongly-typed query language
• Large-scale database access
• Must guarantee safety/correctness of operations
  on data
• Over time, XSLT XQuery may both serve needs of
  many application domains
• XQuery will become a hidden, commodity language

29
XQuery Example

• For each author, return number of books and
  receipts books published in past 2 years, ordered
  by name
• let cat fndoc(www.bn.com/catalog.xml),
  Join
• sales fndoc(www.publishersweekly.com/sal
  es.xml)
• for author in distinct-values(cat//author)
  Grouping
• let books cat//book_at_year gt 2000 and
  author a, S.J.
• receipts sales/book_at_isbn
  books/_at_isbn/receipts
• order by author
  Ordering
• return
• ltsalesgt XML
  Construction
• author
• ltcountgt fncount(books) lt/countgt
  Aggregation
• lttotalgt fnsum(receipts) lt/totalgt
• lt/salesgt

30
Outline

• Basic XML / XQuery
• Introduction of Pathfinder and MonetDB projects
• Relational XQuery
• XPath steps in the pre/post plane
• Translating for-loops, and beyond
• Optimizations
• Order prevention
• Loop-Lifted Staircase join
• Join recognition
• Outlook
• Conclusions
• Roadmaps

31
XQuery Systems 2 Approaches

• Tree-based
• Tree is basic data structure
• Also on disk (if an XQuery DBMS)
• Navigational Approach
• Galax Simeon.., Flux Koch.., X-Hive
• Tree Algebra Approach
• TIMBER Jagadish..
• Relational
• Data shredded in relational tables
• XQuery translated into database query (e.g. SQL)

32
The Pathfinder Project

• Challenge / Goal
• Turn RDBMSs into efficient XQuery engines
• People
• Torsten Grust, Jens Teubner
• University of Konstanz (June 2005 Technical
  University of Munich)
• Maurice van Keulen
• University of Twente
• Jan Rittinger
• University of Konstanz CWI
• Task generate code for MonetDB

33
The Pathfinder Project

• Challenge / Goal
• Turn RDBMSs into efficient XQuery engines
• People
• Torsten Grust, Jens Teubner, ...
• University of Konstanz (June 2005 Technical
  University of Munich)
• Maurice van Keulen, Jan Flokstra, ...
• University of Twente
• Jan Rittinger
• University of Konstanz CWI
• Task generate code for MonetDB
• Peter Boncz, Stefan Manegold, Sjoerd Mullender,
  ...
• CWI, Amsterdam

34

• a decade of query-intensive application
  experience
• image retrieval Peter Bosch ?
  ImageSpotter
• audio/video retrieval Alex van Ballegooij ?RAM
• XML text retrieval de Vries / Hiemstra
  ?TIJAH
• biological sequences Arno Siebes ? BRICKS
• XML databases Albrecht Schmidt ? XMark
• Grust /
  vKeulen ? Pathfinder
• GIS Wilco Quak ?
  MAGNUM
• data warehousing / OLAP / data mining
• SPSS ?
  DataDistilleries
• Univ.
  Massachussetts ? PROXIMITY
• CWI research group successfully spun off
  DataDistilleries (now SPSS)

35
Pathfinder MonetDB
36
Pathfinder MonetDB
37
Open Source

• MonetDB Pathfinder on Sourceforge
• Mozilla License
• MonetDB Homepage
• http//monetdb.cwi.nl/
• Pathfinder Homepage
• http//pathfinder-xquery.org/
• Developers website
• http//sf.net/projects/monetdb/
• RoadMap
• 14-apr-04 initial Beta release MonetDB/SQL
• 30-sep-04 first official release MonetDB/SQL
• 30-may-05 Developer release of MonetDB/XQuery
  (i.e. Pathfinder)
• 30-sep-05 Student release of MonetDB/XQuery
  (incl. XUpdate)
• 30-dec-05 Users release of MonetDB/XQuery (?)

38

• Front-end/back-end
• support multiple data models
• support multiple end-user languages
• support diverse application domains

39

• Front-end/back-end
• support multiple data models
• support multiple end-user languages
• support diverse application domains

40
Architecture
41
The Architecture
42
Outline

• Basic XML / XQuery
• Introduction of Pathfinder and MonetDB projects
• Relational XQuery
• XPath steps in the pre/post plane
• Translating for-loops, and beyond
• Optimizations
• Order prevention
• Loop-Lifted Staircase join
• Join recognition
• Outlook
• Conclusions
• Roadmaps

43
XPath on an RDBMS
44
Pre/Post ? Pre/Level/Size
45
Outline

• Basic XML / XQuery
• Introduction of Pathfinder and MonetDB projects
• Relational XQuery
• XPath steps in the pre/post plane
• Translating for-loops, and beyond
• Optimizations
• Order prevention
• Loop-Lifted Staircase join
• Join recognition
• Outlook
• Conclusions
• Roadmaps

46
Order Prevention
47
Join Recognition
48
Loop-lifted staircase join
49
Scalability
50
Scalability
51
Scalability
52

53
XMark 1GB Pathfinder vs X-Hive
54
Outline

• Basic XML / XQuery
• Introduction of Pathfinder and MonetDB projects
• Relational XQuery
• XPath steps in the pre/post plane
• Translating for-loops, and beyond
• Optimizations
• Order prevention
• Loop-Lifted Staircase join
• Join recognition
• Outlook
• Conclusions
• Roadmaps

55
Conclusions

• Relational approach can be scalable fast
• Crucial Optimizations
• Join recognition
• Loop-lifted XPath steps
• Order awareness

56
Conclusions

• Relational approach can be scalable fast
• Crucial Optimizations
• Join recognition
• Loop-lifted XPath steps
• Order awareness
• Future Roadmap (Scientific/Research)
• Algebraic Query Optimization
• Updates

57
Product Roadmap

• 30-05-05 MonetDB/XQuery 4.8/0.8 Mercurius
• Developers Release / Technology Preview 1
• 30-09-05 MonetDB/XQuery 4.10/0.10 Venus
• Student Release / Technology Preview 2
• XUpdate, Algebraic Query Optimization
• 30-12-05 MonetDB/XQuery 4.12/1.12 Mars
• Final Release
• Application Programming Interfaces
• End-User Front-Ends

58
Loop-lifted staircase join
59
Staircase join