Persistent Stream Producers 020902

1
Persistent Stream Producers02/09/02
Andy Cooke / Heriot-Watt University ltceeawc_at_cee.h
w.ac.ukgt
2
Contributors

• Brian Coghlan TCD
• Andy Cooke Heriot-Watt
• Ari Datta QMUL
• Abdeslem Djaoui RAL
• Laurence Field PPARC
• Steve Fisher RAL
• James Magowan IBM-UK
• Werner Nutt Heriot-Watt
• Manfred Oevers IBM-UK
• John Ryan TCD
• Manish Soni PPARC
• Norbert Podhorszki SZTAKI
• Antony Wilson PPARC
• Xiaomei Zhu PPARC

3
Persistent Producers

• Why do we need them?
• WP1 use
• stream producers to publish job status.
• Archivers to collect and store these.
• they cant track jobs if tuples are lost
• Problems
• Servlet might crash
• Servlet buffer is finite
• My approach
• solve (1), and then see if (2) is problem

4
Approaches to Persistence

• Use a database
• Is MySQL fast enough?
• Use Prayler to make RAM objects persistent
• 3000 times faster than MySQL use for
  servlets?
• Write objects to a memory-mapped file.
• E.g. useful interface when making c talk to
  java
• Included in Java 1.4 use for buffer?
• Other approaches
• RVM (recoverable virtual memory, available in
  c)
• didnt investigate this much

5
What is Prevayler?

• Prevaylent System state changed using command
  objects
• Commands are serialized and written to a log
  file.
• Snapshots of system taken periodically
• During crash recovery
• Recover last saved snapshot of the persistent
  system
• Read commands and apply them again to recover
  state
• http//www.prevayler.org
• (LGPL license)

6
Using Prevayler in RGMA

• Some Commands
• AddTuple
• SetBufferSize
• SetReadOffset
• lots of
  refactoring!

7
A New Buffer (Queue)
I
I
Queue (linked list, max size)
I I I I I I I I I
virtual Consumer Queues (read pointer, buferSize)
Only as many tuples as needed are kept
so wont need to write tuples to
file?
8
Summary

• Using Prevayler for making servlets persistent
• Almost completed coding losts of refactoring!
• Not sure yet if we need tuples written to file