OPeNDAP Hyrax BackEnd Server BES

1
OPeNDAP HyraxBack-End Server (BES)

• APAC 2007 OPeNDAP Workshop
• Patrick West (pwest_at_ucar.edu)

2
OPeNDAPs Hyrax Architecture
3
In what ways can you extend the BES?

• Extensions are dynamically loaded from a shared
  object module
• Containers and Definitions, and ways of storing
  them
• Dataset Catalogs
• Interface
• Initialization/Termination
• New commands (like our hello world example)
• New response objects
• New response handlers
• New request handlers (data handlers like netcdf,
  freeform, csv)
• Aggregation engines
• Methods of returning your data (return as netcdf)
• Reporters
• Exception Handlers
• Debugging

4
Modules are dynamically loaded into the BES

• Within the BES configuration you specify the
  modules that you want to load into the BES.
• The module is derived from the C abstract class
  BESAbstractModule.
• The class impelements a c-style function called
  maker that instantiates the module class
• The class implements an initialization and
  termination method to set up and clean up the
  module.
• The module can add named handlers to the
  different pieces of the BES.
• The module can extend already existing modules,
  for example extending the DAP2 module.
• The module can modify already existing modules,
  such as changing the way certain responses are
  built.

5
BES Configuration File/usr/local/etc/bes/bes.conf
6
Running what we have

• Lets see what we have already installed
• besctl start (if its not already running)
• bescmdln -h localhost -p 10002
• show help
• show version
• show keys
• set container in catalog values
  c,data/nc/fnoc1.nc
• define d as c
• get dds for d
• define d as c with c.constraintu
• get dds for d
• get ascii for d

7
Or use the OLFS

• CATALINA_HOME/bin/startup.sh
• Start your browser
• Localhost8080/opendap/
• Select data, then nc, then fnoc1.nc
• Click the check box next to the variable u
• Click the button Get ASCII

8
The CSV Data Handler Module

• /root/src/bes-3.5.1/csv_module-1.0.0

"StationltStringgt","latitudeltFloat32gt","longitudeltF
loat32gt","temperature_KltFloat32gt","NotesltStringgt
"CMWM",-34.7,23.7,264.3, "BWWJ",-34.2,21.5,262.1,"
Foo "CWQK",-32.7,22.3,268.4, "CRLM",-33.8,22.1,27
0.2,"Blah "FOOB",-32.9,23.4,269.69,"FOOBAR"
9
Setting up the example

• cd /root/src/bes-3.5.1/csv_module-1.0.0
• autoreconf
• ./configure
• make install
• This will build and install the module library
  and the sample data
• make bes-config
• This will make the changes to the bes.conf file

10
BES Configuration File/usr/local/etc/bes/bes.conf
11
Restart it all

• CATALINA_HOME/bin/shutdown.sh
• besctl stop
• besctl start
• CATALINA_HOME/bin/startup.sh
• Back in the browser
• localhost8080/opendap/
• Select data, then csv, then temperature.csv
• Click the button Get ASCII

12
CSVModule class

• Example Module class
• /root/src/bes-3.5.1/csv_module-1.0.0
• Implements initialize and terminate methods (and
  dump)
• Adds (typical for data handlers)
• new request handler
• new catalog
• new container storage using the catalog

13
Containers

• A container represents data
• Typically a data file
• Can represent a request to a remote system, such
  as WCS
• Has three main parts
• Symbolic name (cool_data)
• Real name - e.g. filename
• Data Type (nc, h4, h5, ff, csv) - matches the
  name of the data/request handler
• Derive from the abstraction BESContainer
• BESFileContainer built in
• WCSContainer

14
Accessing the container

• Implement the access method
• virtual string access()
• Returns the data container to be accessed,
  typically full path to a file.
• BESFileContainer returns the stored file name
• WCSContainer makes the WCS request to the remote
  server, stores the resulting data file, and
  returns the path to that resulting, cached, data
  file.

15
Storing Containers

• The default storage of containers is volatile for
  the duration of the session
• Can create new persistent stores
• Store in database per user or group or project
• Create class inherited from BESContainerStorage
• Give the new class a unique name (user_store)
• In the Module class add this new storage class to
  the container storage list
• BESContainerStorageListTheList()-gtadd_persisten
  ce( storage )
• Create containers in that storage
• set container in user_store values s,r,t

16
Storing a container

• virtual void add_container( const string
  sym_name,
• const string real_name,
• const string type )
• BESContainerStorageCatalog uses the catalog to
  create the full path to the file and uses the
  file extension to determine the data type.
• WCSContainerStorage creates a WCSContainer whose
  access method makes the WCS request on the remote
  machine, caches the resulting file, and returns
  that as the file to access.
• BESContainerStorageVolatile simply stores the
  container in a volatile stl map for the duration
  of the session.

17
Definitions

• Like a view of data
• Can encompass many containers (available soon)
• Can specify constraints for each container
• Can specify data attributes from each container
• Specify aggregation engine and command
• define d as ltc1gt,,ltcngt with
  ltc1gt.constraint,ltc1gt.attributes,
  ltcngt.constraint,ltcngt.attributes
  aggregate using ltenginegt by ltcommandgt

18
Storing Definitions

• Default storage is volatile for the duration of
  the session
• Can create new persistent stores
• Store in database per user or group or project
• Create class inherited from BESDefinitionStorage
• Give the new class a unique name (user_store)
• In the Module class add this new storage to the
  definition storage list
• DefinitionStorageListTheList()-gtadd_persistence
  ( storage )
• Create definitions in that storage
• define d in user_store as

19
Storing a Definition

• virtual void add_definition( const string
  def_name,
• BESDefine def )
• BESDefinitionStorageVolatile simply stores the
  definition in a volatile stl map for the duration
  of the session.

20
Dataset Catalogs

• OLFS uses BES and THREDDS catalogs for dataset
  catalog traversal
• Can have multiple catalogs in a single BES
  representing any type of catalog (coming soon)
• Filesystem
• Database catalog
• Each Catalog is configurable
• Catalogs inherit from the abstraction BESCatalog
• Add to the list of catalogs in the module class
• BESCatalogListTheCatalogList()-gtadd_catalog(
  catalog )

21
Typical Catalog Configuration

• Data handlers, e.g. nc_module, add a
  BESCatalogDirectory representing a file system.
• In the BES configuration file
• BES.Catalog.catalog.RootDirectory - data root
  directory of the catalog named catalog
• BES.Catalog.catalog.TypeMatch - data types
  matched by file extensions
• BES.Catalog.catalog.Include - what nodes to
  include in the catalog list (usually everything)
• BES.Catalog.catalog.Exclude - what nodes to
  exclude (e.g. dot files)
• Lets look at /usr/local/etc/bes/bes.conf

22
Catalogs and Containers

• Can create containers within the context of
  catalogs using BESContainerStorageCatalog
• Uses BES.Catalog.catalog.TypeMatch to determine
  data type.
• set container in catalog values c,ltcatalog_filegt

23
Putting it together so far
24
The Interface

• There is a single interface into the BES.
• Inherits from the abstraction BESInterface
• Installed version uses BESCmdInterface
• CEDAR uses BESApacheInterface
• Creating BESXMLInterface
• Interface runs a set of steps
• initialize the BES environment
• build the execution plan
• execute the request plan building the response
  object
• transmit the response object
• log the status of the request
• report on the request
• end the request

25
BESDataHandlerInterface

• BESDataHandlerInterface structure is carried
  throughout the execution of the commands
• Holds on to the response handler
• Stores the list of containers
• Holds the action being taken
• Holds map of name/value pairs to be used during
  execution
• Holds an error object if an error/exception occurs

26
Initialize/Termination callbacks

• Can register callbacks during the initialization
  and end steps
• static void BESInterfaceadd_init_callback(
  p_bes_init init )
• static void BESInterfaceadd_end_callback(
  p_bes_end end )
  
• typedef bool (p_bes_init)( BESDataHandlerInterfac
  e dhi )
• typedef void (p_bes_end)( BESDataHandlerInterface
  dhi )
  
• Examples
• Authentication/authorization
• Initialize database connections
• Clean up database connections and files

27
Building the request plan

• Derived classes of BESInterface implement
• virtual void build_data_request_plan()
• BESCmdInterface parses the incoming command
  string
• BESApacheInterface translates information from
  apache and parses the incoming command string
• BESXMLInterface parses the XML document
• Creates a response handler to execute the command

28
Commands

• String commands sent from client to server
• Built in commands
• show help (version, process, status, keys)
• set container
• show containers
• delete container ltcontainer_namegt
• delete containers
• define
• show definitions
• delete definition ltdef_namegt
• delete definitions
• set context ltnamegt to ltvaluegt
• show context

29
DAP Commands

• DAPCommandModule adds
• get das for ltdef_namegt return as type
• get dds
• get ddx
• get dods
• show catalog for node (info)
• dap-server modules (www,usage,ascii) add
• get info_page
• get ascii
• get html_form
• Data handlers (nc, ff, csv) dont add any new
  commands.

30
Response Objects

• Derived from BESResponseObject.
• BESDASResponseObject
• BESInfo
• No methods required to be implemented
• Created by a BESResponseHandler
• Filled in by BESResponseHandler or delegated to
  BESRequestHandler

31
Informational Response Objects

• Built in informational response objects derived
  from BESInfo
• BESTextInfo
• BESHTMLInfo
• BESXMLInfo
• BESSilentInfo
• Each one formats an informational response
  according to its type
• Can add new informational response objects.
• BESInfoListTheList()-gtadd_info_builder( name,
  function )
• Function instantiates the derived BESInfo object.
• Created in BESResponseHandler instances by
  calling
• BESInfo info BESInfoListTheList()-gtbuild_inf
  o()
• Set in BES configuration file
• BES.Info.Typeltnamegt e.g. txt, html, xml

32
Response Handlers

• represent a specific response, such as a DAS
  response, a DDS response, a help response ...
• know how to create the response object (DAS, DDS,
  BESInfo)
• do not necessarily fill in the response object,
  but know how the response should be filled in
• BESDASResponseHandler knows to create a DAS
  object and that it needs to go to each request
  handler for each container.
• A version response handler knows to create an
  informational response object and that it needs
  to go to all registered request handlers.
• A status response handler knows to create a text
  response object and fill it in with the status of
  the server.
• know how to transmit the response object

33
Implementing a Response Handler

• Inherits from the C abstract class
  BESResponseHandler (e.g. BESDASResponseHandler,
  BESHelpResponseHandler)
• Implement the methods
• virtual void execute( BESDataHandlerInterface
  dhi )
  
• Creates the response object
• Does the work to fill in the response object, or
  delegate to a request handler to fill in the
  response object
• virtual void transmit( BESTransmitter
  transmitter,
• BESDataHandlerInterface dhi )
• Transmits the resulting response object using the
  appropriate method on the transmitter (more
  later)
  

34
Request Handlers

• Derived from BESRequestHandler
• Fills in response objects
• Register functions that say I know how to fill
  in some response

35
Registering a Request Handler

• Inside the Module class the request handler is
  registered with the BES
• BESRequestHandlerListTheList()-gtadd_handler(
  name, handler )
• CSVModuleinitialize()
• BESRequestHandlerListTheList()-gt
• add_handler( modname,
• new CSVRequestHandler( modname ) )

36
I know how

• Inside the constructor for the request handler
  you register functions that know how to fill in
  responses
• add_handler( name, function )
• CSVRequestHandlerCSVRequestHandler
• add_handler( DAS_RESPONSE, CSVRequestHandlercsv_
  build_das )
• add_handler( DDS_RESPONSE, CSVRequestHandlercsv_
  build_dds )
• add_handler( DATA_RESPONSE, CSVRequestHandlercsv
  _build_data )
• add_handler( VERS_RESPONSE, CSVRequestHandlercsv
  _build_vers )
• add_handler( HELP_RESPONSE, CSVRequestHandlercsv
  _build_help )

37
How does it work
Containers c1,file1,csv c2,file2,csv
ce,file3,nc c4,file4,ff
command
get das for def
Definitions def as c1, c2
show help
Response Handlers das, BESDASResponseHandler
help, BESHelpResponseHandler
Request Handlers csv, CSVRequestHandler
das, fill_das_func help, fill_help_func
38
The CSV Data Handler Module

• /root/src/bes-3.5.1/csv_module-1.0.0
• Look at CSVModule
• Adds CSVRequestHandler
• Adds BESCatalogDirectory
• Adds BESContainerStorageCatalog
• Look at CSVRequestHandler
• Adds functions to fill in DAS, DDS, DataDDS,
  Version and Help responses
• These functions fill in the respective response
  objects

39
Transmit

• Sends the response object back to the client
• Two built in transmitters
• BESBasicTransmitter
• BESBasicHTTPTransmitter (adds header info)
• These implement methods to transmit informational
  responses (BESInfo)
• BESDapTransmit implements static methods to send
  DAS, DDS, and DataDDS responses
• Adds named static functions to the two built in
  transmitters
• BESResponseHandler instances know what functions
  to call on the transmitter to transmit their
  response objects

40
Adding Transmit Functions

• BESDapTransmit
• BESTransmitter t BESReturnManagerTheManager()
  -gt
• find_transmitter( BASIC_TRANSMITTER )
• if( t )
• t-gtadd_method( DAS_TRANSMITTER,
• BESDapTransmitsend_basic_das )
• t-gtadd_method( DDS_TRANSMITTER,
• BESDapTransmitsend_basic_dds )
• t-gtadd_method( DDX_TRANSMITTER,
• BESDapTransmitsend_basic_ddx )
• t-gtadd_method( DATA_TRANSMITTER,
• BESDapTransmitsend_basic_data )

41
Adding Transmit Functions

• BESTransmitter t BESReturnManagerTheManager()
  -gt
• find_transmitter( HTTP_TRANSMITTER )
• if( t )
• t-gtadd_method( DAS_TRANSMITTER,
• BESDapTransmitsend_http_das )
• t-gtadd_method( DDS_TRANSMITTER,
• BESDapTransmitsend_http_dds )
• t-gtadd_method( DDX_TRANSMITTER,
• BESDapTransmitsend_http_ddx )
• t-gtadd_method( DATA_TRANSMITTER,
• BESDapTransmitsend_http_data )

42
Adding new Transmitter

• Inherit from BESTransmitter
• Implement send_text, send_html
• Add static functions to transmit other response
  objects in the constructor
• E.G. ESG project, transmitter to return data
  object as a netcdf file
• get dods for ltdefgt return as netcdf

43
Aggregation

• You can have multiple aggregation engines
  installed in the BES
• Inherit from BESAggregationServer
• Implement aggregate method
• virtual void aggregate( BESDataHandlerInterface
  dhi )
• In Module class add aggregation function that
  instantiates your instance
• BESAggFactoryTheFactory()-gtadd_handler( name,
  function )
• typedef BESAggregationServer (p_agg_handler)(st
  ring name)

44
Reporters

• After command has been executed and response
  object transmitted
• Inherit from BESReporter
• Implement method
• virtual void report( const BESDataHandlerInterfac
  e dhi )
• Register with reporter list
• BESReporterListadd_reporter( name,
  reporter_instance )

45
Exception Handling

• Register exception handler functions with the
  Exception Manager
• virtual void add_ehm_callback( p_bes_ehm ehm )
• typedef int (p_bes_ehm)( BESException e,
• BESDataHandlerInterface dhi )
• BESExceptionManagerTheEHM-gtadd_ehm_callback(
  func )
• When exception caught, passed to Exception
  Manager
• Iterates through registered functions
• If not handled, default handler
• Set error_info in BESDataHandlerInterface
• This is transmitted in place of response object

46
Logging and Debugging

• Informational and Debug Logging in the BES
• The BES has two forms of logging.
• Event logging in a bes.log file. This is
  configured in the bes.conf file
• BES.LogName/path/to/bes.log
• BES.LogVerboseno
• Debug logging for the BES
• besctl start -d /path/to/bes.debug,nc,bes,ppt

47
BES Debugging

• Format of the command line option -d
• cerrltlog_file_namegt,ltcontext1gt,,ltcontextngt
• E.g. -d /tmp/bes.debug,nc,bes
• Specify a context of all to turn on debugging
  for everything
• Writing debug statements in your code
• Macro BESDEBUG( ltcontextgt, ltoutput to ostreamgt
  )
• E.g. BESDEBUG( nc, reading DAS for ltlt file
  ltlt endl )
• Block of debugging
• If( BESDebugIsSet( nc ) )
• listltstringgtconst_iterator i
  _somelist.begin()
• while( i ! _somelist.end() i )
• BESDEBUG( nc, ltlt (i) )

48
BES Debugging

• Most BES classes derived from BESObj, and some
  libdap class derived from DapObj
• You can dump BESObj instances and DapObj
  instances to ostream
• virtual void dump( ostream strm )
• E.g.
• DAS das
• BESDEBUG( nc, DAS ltlt endl ltlt das )
• Allows for indentation as well
• Register context within your module
• BESDebugRegister( nc )

49
BES Debugging - dump method example
50
Hello World

• cd /root/src/bes-3.5.1/hello_world
• autoreconf
• ./configure
• make
• export BES_CONF./bes.conf
• besctl start
• bescmdln -h localhost -p 10002
• say hello to world

51
Lets write it

• Browse to opendap.hao.ucar.edu/releases/
• Click on newmodule.tar.gz
• cd /root/src/bes-3.5.1
• mv /Desktop/newmodule.tar.gz .
• tar zxvf newmodule.tar.gz
• cd bin
• make install
• cd ../templates
• make install
• cd ..
• mkdir sample
• cd sample

52
besCreateModule

• Creates stub code for a new module
• besCreateModule
• sample
• Sample
• none
• say
• This will create everything you need to get
  started, run autoreconf, configure and make

53
Writing the code names

• SampleResponseNames.h
• define SAY_WHAT say_what
• define SAY_TO say_to

54
Writing the code command

• SamplesayCommand.cc
• Find // Here is where
• Add
• dhi.dataSAY_WHAT my_token
• // Next token should be the token "to
• my_token tokenizer.get_next_token()
• if( my_token ! "to" )
• tokenizer.parse_error( my_token " not
  expected\n" )
• my_token tokenizer.get_next_token()
• if( my_token "" )
• tokenizer.parse_error( my_token " not
  expected\n" )
• dhi.dataSAY_TO my_token

55
Writing the code request handler

• Edit SampleRequestHandler.cc
• Were going to have the response handler fill in
  the response
• Delete in the constructor
• add_handler( say_RESPONSE,
• SampleRequestHandlersample_build_say )
• Delete the function sample_build_say
• Edit SampleRequestHandler.h
• Delete function declaration for sample_build_say

56
Writing the code the response

• Edit SamplesayResponseHandler.cc
• In execute method add after // Here is where
• info-gtbegin_response( say_RESPONSE )
• string str dhi.data SAY_WHAT " "
  dhi.data SAY_TO
• info-gtadd_tag( "text", str ) info-gtend_respo
  nse()

57
New bes.conf file and run!

• export BES_CONF./bes.conf
• besctl start
• bescmdln -h localhost -p 10002
• say hello to world

58
Add a Reporter

• Keep track of what is being said to whom
• cp ../hello_world/SayReporter. .
• Edit SampleModule.cc
• Add includes
• include BESReporterList.h
• include SayReporter.h
• Add Reporter at bottom of initialize
• BESDEBUG( adding Say reporter ltlt endl )
• BESReporterListTheList()-gt
• add_reporter( modname, new SayReporter )

59
Build, restart, say hello

• Add to Makefile.am
• BES_SRCS SayReporter.cc \
• BES_HDRS SayReporter.h \
• Edit bes.conf
• at bottom add Say.LogName./say.log
• make
• besctl stop
• besctl start
• bescmdln -h localhost -p 10002
• say hello to world
• exit
• cat say.log