Middleware Diarrhea and Other Ailments

1
Middleware Diarrhea and Other Ailments

• Michael Stonebraker
• Adjunct Professor
• Massachusetts Institute of Technology
• (stonebraker_at_lcs.mit.edu)

2
Outline

• Too much middleware
• XML ailments
• Web services ills
• Our professional sickness

3
Client-Server Got Replaced by N-Tier Computing

• The Web
• Gizmos
• Scalability and management problems with client
  server

4
Humility Lesson

• We all sold client-server hard
• during the 80s
• and even into the 90s
• Less than 10 years later
• it is the worst idea on the planet

We should feel really dumb!
5
N-Tier Computing Produced Lots of Middleware

• App servers
• EAI/messaging
• ETL
• Federators
• Workflow
• CMS
• Portals
• DBMS

6
Middleware Diarrhea

• Average enterprise has
• one (or more) app servers
• one (or more) EAI packages
• one (or more) ETL packages
• one (or more) portal products
• one (or more) application packages
• and maybe someday a federated DBMS

7
All of these systems

• Contain transformation engines
• And often do function activation (app service)
• And often have adapters to legacy systems

Huge overlap in functionality!!
8
Less Moving Parts

• Less systems
• More uniformity
• Less duplication

9
Less Systems

• Less system administrators
• Less training
• Less manuals
• Less bugs
• Less cross system issues

10
More Uniformity

• Every island has
• memory management
• security model
• threading model
• Less is better

11
Less Duplication

• Most of the islands support transformations
• reasonable chance you will do each one 6 or more
  times
• maintenance headache

12
So How To Consolidate

• Converge app server into OR DBMS
• dumbest OR query is execute function

Remember that everything looks like a nail to
the guy with the hammer!
13
Pictorially
client
DBMS
DBMS
14
This Requires.

• DBMS to send queries to other DBMSs
• I.e. be a data federator
• Load balance also requires a federator

15
Best of Breed Federators

• Support schema heterogeneity
• by executing OR functions
• Support materialized views
• to cache static data

16
Less Moving Parts.

• Federators dominate ETL
• ETL only supports push
• federators do both push and pull

17
Workflow

• A collection of rules
• whos allowed to buy what
• and who must approve it
• Best considered as a boxes and arrows diagram
• And compiled into components to run on an app
  server

18
Workflow Framework -- POs
IT?
manager
no
no
PO
Big?
Laven
yes
yes
19
Data Intensive Workflow Should Move Inside an OR
DBMS

• GUI for boxes and arrows
• Compiler for the diagram
• processing steps become components
• business rules become triggers
• all data flow inside the DBMS
• Worked great in Media/360

20
Why?

• Big Big Big performance advantage
• no polling of the DBMS
• no data movement
• easy to change!

Watch for Informix product in this area!
21
Nirvana

• One integrated system that does
• federation
• EAI
• app service
• With a single transformation system
• Based on DBMS technology (or something else.)

22
XML

• Good for content storage and movement
• Good as on the wire format for data movement
• as long as you dont need to send a lot of stuff
  fast
• Bad for data storage!

23
History Lesson

• 1960s
• IMS and IDMS get traction
• customers start complaining about rewriting
  everything when schema changes

24
History Lesson

• 1970
• Codd writes pioneering paper
• starts a decade long argument between IMS/CODASYL
  advocates and Codd supporters

25
Net-Net of Argument

• Putting semantics into data order is bad
• restricts storage options
• Hidden meaning bad
• no self-defining fields

26
Net-Net of Argument

• Data independence is good
• schemas change often
• dont want to rewrite anything when this happens

27
Net-Net of Argument

• Complexity is bad
• high level query languages are good
• KISS arguments
• Call these three premises Codds laws

28
History Lesson

• 1983(?)
• Codd wins Turing award
• acknowledgement for being right

29
XML in This Historical Light

• Most of the bad features of IMS/Codasyl
• allows semantics in data order
• data independence will be a challenge
• try updates on inverted hierarchies
• look at IMS LDBs
• more complex than Codasyl

30
Our Field

• We look a little silly saying
• an idea renounced in the 1970s
• is back
• Leading our colleagues to ask Whats different?
• if somebody disproved Codds laws they didnt
  tell me..

31
How to Win the Turing Award Circa 2020

• 2000s
• XML data storage gets traction
• 2010
• dust off Codds paper
• Wait 10 years to be proven right

32
In Any Case

• In line tags turn 1Tbyte of EMP data into 10
  Tbytes of EMP data
• Wont store anything big in native XML
• will use something else.
• like what?

33
OR DBMS

• XML is merely this years data type
• Next year it will be WML or
• and there will be a next year.

34
XMLSchema

• Contains the kitchen sink
• Complexity run amok
• diarrhea from the SGML types
• Includes lots of known hard stuff
• e.g. union types

35
Xquery

• Mostly syntactic sugar on OR SQL
• // is a user-defined function in Informix OR
  engine
• Try to keep the semantics close to OR SQL

36
Another History Lesson

• Typical enterprise wanted data integration for
  business analysis badly
• needed data in a variety of systems
• in a variety of formats
• often with no unique ids
• often with incompatible semantics
• 2 day delivery means lots of things
• often dirty

37
ETL Warehouse Projects of the 90s

• Well into 8 digits
• Usually a factor of three behind schedule
• Delivering a factor of 3 less stuff
• Everybody dented their pick on semantic
  heterogeneity
• which is hard, hard, hard
• and not solved by the blizzard of 3 letter
  acronyms from Redmond

38
Web Services

• Will be a long time coming outside of simple
  domains (where there is no data integration to
  deal with)
• E.g. catalog management
• Grainger perspiration.

39
The Depressing State of Affairs

• 50-75 of IT projects fail
• if we built bridges, our profession would be
  fired
• and the same mistakes are repeated over and over
  (excessive ambition, rolling specs, bad design,
  failure to load a large data set early)

40
What To Do?

• We typically dont teach this stuff (and do a
  serious disservice to our students)
• probably because we dont (cant) spend any time
  in industry to figure it out

Action item at the very least read a couple of
Robert L. Glasss books
41
The Depressing State of Affairs

• Hardware half-life is 18 months
• Software half-life is 18 years (or more)!
• In 25 years we moved from
• C to Java
• SQL to Xquery

42
What To Do?

• Much higher level design environments
• vis
• workflow
• special purpose languages (report writers,)
• And stop turning down papers on this stuff

43
Grand Challenge

• Improve application productivity (probability of
  success programmer productivity) by 2 this
  decade