ECHO

1
ECHO

• A Tutorial for the ESIP Federation

2
Agenda

• What is ECHO?
• ECHO Partner View
• Ingest
• Configuration
• Representing Spatial Information
• ECHO Client View
• Catalog Service
• Query Language

3
What Is ECHO?

• Functionally
• Clearinghouse of spatial, temporal, numeric and
  textual metadata
• Order broker
• User and provider account service
• Services clearinghouse and broker (future)
• Objectives
• ECHO is a portal to Earth Science data and
  services.
• It allows providers of data to share their
  metadata and offload some of their search
  responsibilities.
• It also brokers orders from clients to the
  appropriate providers, providing tracking
  services for both the client and the provider.
• ECHO presents a messaging interface based on XML,
  but does not currently provide a search GUI.

4
System Drivers
Architecture Overview

• Present an API for organizations to connect their
  own user interfaces and programs to
• Make it easy for providers of Earth Science data
  and services to participate in the system
• Provide searches that respond quickly
• Broker orders for both data and services
• Minimize operational costs
• Build upon advances in industry and use
  e-commerce systems as a model
• Build a system that can be scaled up to handle
  large numbers of requests

5
Current Information Architecture
Architecture Overview
EDG
Data Providers
Inventory Metadata Catalog
Collection Metadata Catalog
WWW Browsers
User Interface
Browse Images
Messages Search, Browse, Order
Data Holdings
6
Current Metadata Access
Architecture Overview
Partner down for maintenance, network not
available, etc.
200TB
Partner
Partner
8TB
EDG
Partner
100TB
Partner
1PB
Partner
1GB
Net Result 200 terabytes of data not
searched. Search time dependent on lowest common
denominator.
7
Information Architecture w/ECHO
Architecture Overview
ECHO
Machine And User Interfaces
Data Service Providers
Collection Inventory Catalog
Data Holdings
EDG Others
WWW Browsers
Partner APIs
Orders
Client APIs
Browse Images
8
ECHO Metadata Access
Architecture Overview
Partner down for maintenance, network not
available, etc.
200TB (35MB New)
Partner
Client
Partner
8TB (2MB New)
ECHO
EDG
Partner
100TB (0 New)
Partner
1PB (670MB New)
Client
Partner
1GB (100MB New)
Net Result 35 megabytes of data not
searched. Search time dependent on ECHO only.
9
Simplified ECHO Metadata Model
Collection

Browse
1

Granule

• At the highest level, there are three main
  entities that are part of the ECHO conceptual
  data model
• Collection A grouping of granules typically
  based on a common source of the granules
• Granule The lowest level item retrievable from
  a Partner that is uniquely described in ECHO
• Browse Some kind of binary or ASCII file used
  to provide a user with a quick view of the data.
  This could be a scaled down version of the
  imagery, or a histogram of the data, or some
  other representation

10
High Level ECHO Context
Data
Data Partner
UI Client
ECHO
Order
Query
Order
Order
Data
Data Partner
Results
Machine Client
Service Partner
Web Client
Service Partner
11
Partner Context
Non-service Partner
Service Partner
Search Providers
Metadata Providers
FTP Request/Data
E C H O
Distributed Search Partner Online,
free distribution
Metadata Partner Online, free distribution
Distributed Search Prov. Online,
free distribution
Metadata Partner Online, free distribution
Non-order Distribution
Quote Req.
Quote
Distributed Search Partner Order Distrib. (Quote
optional)
Metadata Partner Order Distrib. (Quote optional)
Order
Distributed Search Prov. Order Distrib. (Quote
optional)
Metadata Partner Order Distrib. (Quote optional)
Order Status
Order Distribution
Stat. Req.
Status
Metadata
Search Result
Data
Search Request
12
Client Context
FTP Req/Data
E C H O
Providers
Search Request
Search Result
DAACs
User Registration
ESIPs
Quote Req.
Quote
RACs
Order
Order Status
Stat. Req.
Etc.
Status
Data
13
Current Partners

• Data Partners
• Operational
• ORNL
• SEDAC
• Partially Operational
• GSFC-ECS
• LPDAAC-ECS
• In Test
• NSIDC-ECS
• LaRC-ECS
• Under Development
• ASF
• Planned
• USGS

• Client Partners
• Operational
• Mercury EOS
• Power UI
• SIMECC
• Partially Operational
• EDG-E
• In Test
• GIZMO
• DVUI
• Under Development
• SNOWI-E
• Nepster
• MODIS Web Sites
• Planned
• Invasive Species
• NEO

14
ECHO Interfaces
Java Toolkit
SOAP to RMI Conversion
Client Interface (RMI) (ECHO API Services)
ECHO
Partner Interface (SOAP) (Quote, Submit, Cancel)
XML to V0/ODL
Business Objects
Client Connector


Catalog
Outside ECHO
Ingest (XML via FTP)
Inside ECHO
Extension
Extension
15
ECHO Interfaces
Client Interface (RMI) (ECHO API Services)
Config Info
ECHO
Submitl Request
Partner Interface (SOAP) (Quote, Submit, Cancel)
Business Objects
Admin API
Cancel Request
User Accts
Place Order
Catalog
Catalog Search
Quote Request
Partner Mgmt
Ingest (XML via FTP)
Full Record Insert, Update, Delete
Partial Record Insert, Update, Delete
16
Current State of ECHO

• Second Operational Version (Release 2 Version
  5.0.1)
• Operational, but not all planned functionality
  has been developed
• Vetted through previous workshops and a year of
  operations
• Clients have provided feedback and will continue
  to
• Changes are coming
• Data Model Review changes will continue to cause
  some basic changes in the conceptual model of the
  system
• XML has been leveraged to try to minimize impact
  of changes
• Some structuring of the messages and tag names
  will be impacted
• New functions will be added
• Some parts of the model that are not being used
  will be removed
• Your feedback is desired, welcome and needed!
• What will make your provider interface work
  better?
• Are there missing requirements?

17
Version 5.5 Snapshot

• Backtrack orbit search
• Capability to allow Partner to specify how
  spatial search is performed for each of their
  collections
• Additional Attribute search performance
  enhancement
• Changed to ISO data format for business API
• Changed phone field format to be free form rather
  than structured

18
Version 6.0 Snapshot

• Data Service architecture with advertised
  services
• Multiple Additional Attribute search capability
  (with and/or functionality)
• Multiple Measured Parameter search capability
  (with and/or functionality)
• And/Or functionality for Platform, Instrument,
  Sensor searches
• Generic path/row functionality for ingest and
  searches
• Data model augmentation DIFID, Center Point, PGE
  Name
• Improved ingest performance
• Web service based order proxies
• Web service based ingest proxies

19
Becoming An ECHO Partner
20
Overview of ECHO Ops Support for Data Partners

• Data Partner Application and Setup
• Data Partner Application
• Initial Setup
• Establish an Operations Agreement (OA)
• Acclimation and Test Support
• API Support
• Metadata Mapping
• Access Control
• Test Support
• Ingest Operations
• Ingest Plan and Schedule
• Support for Metadata Reconciliation
• Partner Relations Management
• High Quality User Services
• High Quality User Support Products
• System Availability and Performance Monitoring
• Outreach Activities

21
Data Partner Resources URLs

• Project web site http//www.echo.eos.nasa.gov
• API documentation http//api.echo.eos.nasa.gov/ec
  ho/message_detail.html
• Operational system access points
• XML Message Test Facility
• http//api.echo.eos.nasa.gov/echo/rmi/EchoTestFac
  ility.jsp
• Client access via SOAP (incl. PUMP)
• api.echo.eos.nasa.gov/soap/servlet/rpcrouter
• Metadata FTP
• ingest.echo.eos.nasa.gov
• Browse imagery
• browse.echo.eos.nasa.gov
• Test system access points
• XML Message Test Facility
• http//beamish.gsfc.nasa.gov5000/echo/rmi/EchoTe
  stFacility.jsp
• Client access via SOAP (incl. PUMP)
• beamish.gsfc.nasa.gov5000/soap/servlet/rpcrouter
  

22
Data Partner Resources Documents and Tools

• ECHO Documentation is maintained online at
• http//www.echo.eos.nasa.gov/echo-docs.shtml
• ECHO 5.0.1 Features/Functionalities
• ECHO API Changes between Version 4.5 and 5.0 and
  5.0.1
• ECHO 5.0.1 Users Guide
• ECHO DTD Tag Directory
• ECHO Acronym List
• The Partner User Management Program (PUMP) is
  available for download from
• http//www.echo.eos.nasa.gov/pump_releases/V_5.0.
  1/dist/webstart/pump.jnlp

23
New Partner Process
Partner Applies
ESDIS Accepts
Configure Test Ingest Process
Configure Test Query
ECHO Accepts
Test Queries
Define Mapping
Partner Sends Sample MD Set
Enable Account
Configure Ingest Process
Configure Query
Partner Sends Historical MD Set
Test Queries
Advertise Partner
Partner Configures to Send Updates
24
New Collection Process
ECHO Accepts
Test Queries
Define Mapping
Partner Sends Sample MD Set
Test Queries
Advertise Collection
Partner Sends Historical MD Set
Partner Configures to Send Updates
Assumption Many of a providers mappings will be
similar and the process of mapping will get
easier and require less work.
25
Metadata Mapping/Ingest Process
(Done once per Partner schema)
Partner Data Source
Transformation Tools (COTS or scripts)
Partner Catalog (Native format)
Interchange Metadata (XML or DTD)
Data Source Metadata (XML)
Data Streamer (XSLT App)
Mapping Tool (Java App)
ECHO Clearing House
Mapper (XSLT Script)
26
ECHO Ingest and Metadata Update Process
27
ECHO Interfaces
Client Interface (RMI) (ECHO API Services)
Config Info
ECHO
Submitl Request
Partner Interface (SOAP) (Quote, Submit, Cancel)
Business Objects
Admin API
Cancel Request
User Accts
Place Order
Catalog
Catalog Search
Quote Request
Partner Mgmt
Full Record Insert, Update, Delete
Partial Record Insert, Update, Delete
Ingest (XML via FTP)
28
Echo Metadata Ingest

• Metadata Preparation Requirements for Data
  Providers
• Metadata Ingest Configuration and Process
• As a clearinghouse of metadata, ECHO must provide
  an efficient mechanism for participating metadata
  providers to share their metadata with ECHO.
  ECHO has implemented this mechanism using an FTP
  interface to allow large files to be efficiently
  transferred to ECHO and to match existing Partner
  distribution capabilities.

29
Metadata Management Through Ingest

• What is currently managed through this interface?
• Collections, granules and browse
• What are the lifecycle metadata management
  capabilities of ingest?
• Insert, update, delete
• What is the granularity of access?
• Record level The entire metadata record for a
  collection or granule is updated as a whole
• How does the Partner express the metadata?
• For collections and granules, an XML file that
  matches the DTD for collections or granule ingest
  files
• For browse, both an XML file that matches the
  browse ingest file DTD and a set of binary browse
  files
• How does the Partner send the metadata to ECHO
• FTP More details to follow

30
Metadata Preparation Requirements for Data
Providers

• Generating input XML files
• ECHO DTD compliant preferred
• BMGT format handled by an ingest proxy
• Data Integrity and Presentation
• Discipline Keywords
• Measured Parameters
• Spatial Data
• Date Format

31
Overview of ECHO Collection Metadata

• Collection
• Basic information (ShortName, VersionID,
  description, processing level)
• Spatial (point, line, polygon, bounding box,
  circle)
• Temporal (range, single shot, periodic)
• Source information (platform, instrument, sensor)
• PSA definitions
• Keywords (discipline (GCMD stack), spatial,
  temporal)
• Campaign
• Science Review
• Collection online resources and online access
  URLs
• Algorithm package
• Contact information
• CSDT information
• Other

32
Overview of ECHO Granule Metadata

• Granule
• Basic information (GranuleUR, )
• Spatial (point, line, polygon, bounding box,
  circle)
• Temporal (range, single shot, periodic)
• Source information (platform, instrument, sensor)
• PSA
• Collection
• Campaign
• Science review
• Online access URLs, granule online resources
• PGE information
• Measured parameters
• Spatial domain Information
• Associated granules (Input, Processing, History,
  QA.)
• Other

33
Understanding the ingest DTDs

• ECHO ingest DTDs
• Collection http//www.echo.eos.nasa.gov/dtd/v5.0/
  collectioningest.dtd
• Granule http//www.echo.eos.nasa.gov/dtd/v5.0/gra
  nuleingest.dtd
• Browse (Currently insert only)
  http//www.echo.eos.nasa.gov/dtd/v5.0/browseingest
  .dtd
• Browse ingest consists of an XML file matched
  with a set of binary files that are the browse
• The XML file matches the GranuleUR (or Collection
  ID) against a set of file names
• ECHO hosts the browse in an http server and
  provides a link within the granules (or
  collections) metadata to the URLs of the browse

34
Understanding the ingest DTDs (Cont)

• Collections and granule XML files have four
  possible sections containing the payload of
  collection or granule metadata insert, update,
  delete and list
• New the payload of metadata is all to be
  inserted into the clearinghouse and that they do
  not already exist
• Update the payload of metadata is to be used to
  find existing records in the clearinghouse and
  change them to match the payload
• Delete the payload is a list of unique
  identifiers that are to be deleted from the
  system
• List the payload of metadata is to be
  synchronized with the clearinghouse (used by
  BMGT)
• Provider delete date/timestamp is used to
  indicate deletion
• Currently, inserts, updates and synchronizations
  all occur in the same way
• If a record already exists, it is updated if it
  doesnt exist, it is created
• The Provider Last Update Date/Timestamp is
  checked on metadata to be updated versus the
  metadata already in the clearinghouse
• Only newer data can update the clearinghouse
  this prevents synchronization issues from
  poisoning the clearinghouse

35
Data Dictionary

• Data dictionary can be found at
  http//www.echo.eos.nasa.gov/documents/ECHO_DTDTAG
  _DICTIONARY1.xls
• The length of the field is represented as VA
  where the is the maximum number of
  characters
• Some salient details
• Date data Format is yyyy-mm-dd hhmmss unless
  otherwise specified
• Keywords GCMD Standard Compliant
• No temporal keywords at this time maintained by
  GCMD
• How to uniquely identify elements in ECHO -
  Assumptions
• Collection Collection short name version
  (obsolete in future)
• Will be simply datasetID in the future
• GranuleUR
• Short name of platform, instrument, sensor
• Browse Internal file name
• Additional attributes name (PSA)
• Provider order ID (future- to be resolved)
• Other

36
Metadata Update Process

• The Metadata Update capability allows certain
  fields to be updated independently of the rest of
  the granule
• In 5.0, this works for granule OnlineURL (for
  Data Pools), granule online resources and QA
  flags only
• The ECHO last update timestamp is used to make
  sure the update is not performed out of sequence
  i.e. if the ProviderLastUpdateDateTime is
  earlier than ECHOs copy, then the update is not
  performed
• If it is performed, then the SaveDateTimeFlag
  indicates to ECHO whether to set its
  ProviderLastUpdateDateTime with the value
  provided in this message

37
Configuring a Partner for Ingest
38
Database Design
system level
error logs
objects
Utility Schema
ingest metadata
staging
model and
procedures
metadata
model
Partner Schema
orderable
Partner
item
profile
information
order
query
user profile
results
ECHO System Schema
39
ECHO Database Architecture

• Partner database account and schema
• Table structure
• Partner specifics
• Date format process
• PSA process
• Order process
• Spatial coordinate system

40
Partner Update Requirements

• Metadata updates occur very soon after the
  Partner submits them (within 5 minutes)
• More frequent updates provide users with more
  up-to-date metadata to search
• Updates should match the rate of change of the
  data, but not to the point of overload
• ECHO staff should work with Partner to determine
  best fit for update schedule
• GOAL ECHO should accurately represent Providers
  metadata so that clients will be able to make
  accurate searches and orders

41
Ingest Process

• Schedule Ingest Process kick off
• Every 5 minutes
• Ingest metadata for every data Partner who is on
  a pre-configured list of data providers
• Check completeness of files in each data
  directory
• Split big files into smaller files (1000
  granules)
• Decompose the XML files to multiple loadable
  ASCII data files
• Load data into Oracle using SQLLoad
• Filter and update operational tables
• Performance
• Ingest Approximately 10 minutes for about 1000
  collection/granules

42
Ingest Performance

• Ingest performance varies based on complexity of
  metadata, the degree the metadata is indexed in
  the clearinghouse, the amount of metadata in the
  clearinghouse (per Partner), the operation
  performed (insert, update, delete) and the query
  load on the system at the time
• Inserts range from 6000-23000 operations per hour
• Ingest updates can be as low as 2000 operations
  per hour

43
FTP Interface Configuration

• Each Partner has an FTP account
• Home directory is parent directory of where
  metadata should go
• FTP directory structure
• data
• collection
• granule
• browse
• How ECHO recognizes completion of FTP
  transmission
• Catch the ending tag in the XML files
• Verify browse XML file parameters (name and size)
  against browse image files

44
Ingest Processing Order

• For each Partner
• Ingest collection files
• Ingest granule files
• Metadata update files
• Ingest browse files

45
The Ingest Summary Report

• Contents
• Lists files received (name and size)
• Number of records processed
• Number of records inserted
• Number of records replaced
• Number of records deleted
• Number of records out of date
• Number of records duplicated
• Number of collections/granules with invalid
  spatial data
• Number of records ignored due to data errors
• Number of records without an associated
  collection
• Number of records that were attempted to be
  deleted, but didnt exist
• Error message(s) if any
• Metadata update detailed error messages
• Anything else?
• Sent to the data Partner specified email address

46
Ingest File Error Summary Report

• Lists of files that are not XML files
• No XML header
• Dont contain XML
• List of files that are not in the correct ingest
  directory
• List of files that did not validate against the
  DTD
• Report is sent to the data Partner via email

47
Representing Spatial Data for Ingest
48
About Spatial Data

• ECHO leverages Oracles spatial capabilities to
  perform spatial search and to represent Partner
  metadata
• Both collections and granules can be spatially
  represented
• If a collection or granule represents the entire
  Earth, there is no need to spatially index it
• ECHO is asking Providers to set a spatial keyword
  to Global in order to indicate this condition
• Version 5.5 will have a special Global flag for
  the Partner to set
• Oracle 9i version 2

49
Understanding Oracle Spatial

• Coordinate system
• Standard Coordinate System Definition (OGC)
• ECHO uses Cartesian and Geodetic models
• User Defined Coordinate System
• Not used by ECHO
• Data type
• Data presentation
• Spatial Indexing
• Quad tree Indexing
• Not used by ECHO
• R-tree Indexing
• ECHO uses R-tree Indexing

50
Cartesian vs. Geodetic Models

• Cartesian Model
• The Earth is represented by a rectangle (-180,
  -90, 180, 90)
• Spatial objects cannot cross the International
  Date Line
• Spatial objects cannot cross the poles
• Polygons can be as large as the whole Earth
• Geodetic Model
• The Earth is represented by a spheroid
• Spatial objects may cross the International Date
  Line
• Spatial objects may cross the poles
• No single polygon may have an area equal to or
  larger than half of the Earths area

51
Oracle Spatial Data Types

• Point and Multiple points
• Line
• Straight line (two points connected by a great
  circle)
• Arc (two points connected by a user specified
  arc)
• Multiple lines (including straight line and/or
  arc)
• Circle
• Three points on the circumference
• Polygon
• Polygon
• Polygon with hole
• Polygon with multiple holes
• Multiple polygons (including all of above
  polygons)
• Bounding Box
• Only valid for the Cartesian model
• Combination of any of the above
• The combination is referred to as a single
  spatial coverage

52
Oracle Spatial Data Presentation

• Polygon
• Outer ring
• Inner ring this represents a hole
• Polygons crossing International Date Line or the
  pole are not supported in the Cartesian model
• No polygon may have an area larger than or equal
  to half of the area of the Earth in the Geodetic
  model
• In general
• In polygon, vertices must be distinct
• No overlap between any two objects in one spatial
  record
• Spatial Data validation Spatial resolution
• The lower the value of spatial resolution, the
  bigger the spatial index will have to be in ECHO
• This should be set to a reasonable value

53
Understanding Spatial Resolution

• Latitude Resolution 0.5, Longitude Resolution
  0.5
• The two blue dots will be the same point
• Latitude Resolution 0.1, Longitude Resolution
  0.1
• The two blue dots will be different points

14
15
16
17
18
19
20
21
18
19.9,14.9
19
20
21
20.1,15.1
22
23
24
25
54
How ECHO applies Oracle Spatial

• Coordinate systems supported
• Flat Cartesian system
• Geodetic system
• Data types supported
• Point and multiple points
• Straight line and multiple lines
• GPolygon(s) connected with straight line (great
  circle if in geodetic model)
• Only allows one hole
• Polygon(s) connected with straight line (great
  circle if in geodetic model)
• Allows holes
• Bounding Box(es)
• Combinations of these
• Indexing R-tree indexing (incremental)

55
What is Expected from Data Providers

• Select appropriate coordinate system for your
  spatial data
• Select appropriate data type for your spatial
  data
• Point
• Line
• Polygon
• Multi polygon
• Circle (input with center point lat/lon and
  radians in meters)
• Bounding box
• This is only appropriate for the Cartesian model
• Future ECHO will be able to convert bounding box
  into polygon for use in the Geodetic model
• Spatial information must provide a reasonable
  approximation of the footprint of the data
• Spatial searches are only as good as the data
  provided
• Densification may be necessary to connect points
  as expected

56
Geodetic View of Sample Polygon
57
Cartesian View of Sample Polygon
58
Cartesian Model Providers

• Apply Flat Cartesian coordinate system
• For GPolygon type, put vertices in a clockwise
  order
• For Polygon type, put vertices in a clockwise
  order for both inner ring and outer ring
• Split the GPolygon if it crosses the
  International Date Line or poles
• Circle data as defined in the DTD (center point,
  radius) will be translated into a polygon using
  Oracle spatial utility

59
Geodetic Data Providers

• Apply Geodetic coordinate system
• Geodetic
• GEOGCS "Longitude / Latitude (WGS 84)", DATUM
  "WGS 84", SPHEROID "WGS 84", 6378137.000000,
  298.257224, PRIMEM "Greenwich", 0.000000 ,
  UNIT "Decimal Degree", 0.01745329251994330
• Prepare data correctly for the Geodetic system
• For Polygon type, put vertices in a clockwise
  order for both inner ring and outer ring
• Provide adequate density of the vertices
• ECHO assumes that points are connected via the
  shortest distance between the points
• Polygon coverage limitation no more than half
  of earth
• Bounding box conversion (future)

60
Partner and ECHO Order Interactions

• ECHO brokers orders from clients to Providers
  with the expectation that the Partner will send
  the data directly to the client. This section
  covers the transactions both from ECHO to the
  Partner and from the Partner to ECHO concerning
  quotes, orders and cancellations.

61
ECHO Interfaces
Client Interface (RMI) (ECHO API Services)
Config Info
ECHO
Submitl Request
Partner Interface (SOAP) (Quote, Submit, Cancel)
Business Objects
Admin API
Cancel Request
User Accts
Place Order
Catalog
Catalog Search
Quote Request
Partner Mgmt
Full Record Insert, Update, Delete
Partial Record Insert, Update, Delete
Ingest (XML via FTP)
62
Components of An Order Sent to Providers

• An order consists of several pieces of
  information
• Order ID
• Provider Tracking ID
• Shipping, Billing and Contact Addresses
• A list of catalog items to be ordered that
  belongs to the Partner
• Quantity of the items
• Options of the items

63
Options in Order

• Options are used here to represent the Partner
  defined settings that are needed to completely
  describe an order but are unique to a Partner
• Order Entry Service allows for options at 3
  levels
• Order Options that apply across the entire
  order
• Currently not used
• Provider Order Options that apply to every item
  being sent to a Partner
• Currently not used, but could be
• Line Item Options that apply only to the
  smallest unit being ordered
• Package options are specified here (media,
  shipping, etc.)
• Also used in other places, but may be called
  policies or preferences.

64
ECHO Option Framework Explained

• The Option Framework is used in a number of
  places throughout ECHO to provide a mechanism for
  introducing new data structures into the system
  during operations.

65
What are Options?

• ECHO uses a common infrastructure for allowing
  abstract, definable items to be specified at
  run-time
• If the structure of something is not known when
  ECHO is developed, then the option infrastructure
  can support representing that structure without
  changes to the ECHO code
• The options infrastructure allows a client to see
  what the template (Option Definitions) for a
  structure is, as well as any previous selections
• It also allows a client to set (select) values in
  the structure according to the rules in the
  Option Definitions Template

66
Options

• Option Definitions - describe the options
  available for a given item. Some options may be
  declared as required.
• Option Selections - the actual value associated
  with an option definition for a particular item.

67
Option Definitions

• Simple Option Attributes
• Option Name- The string that is used to refer to
  this node of the option tree
• MinOccurs The minimum number of times this node
  should appear in the option tree
• MaxOccurs The maximum number of times this node
  should appear in the option tree
• Encrypted When representing this field for
  display or input, use a text box that hides what
  is being typed (password box)
• Primitive Type The type of the option being
  represented
• String, Integer, Boolean, Date, Double
• Valids If a string, an enumeration of valid
  values, if a number, an enumeration of valid
  values
• Min Value If a number, the minimum value, or if
  a string, the minimum length
• Max Value - If a number, the maximum value, or if
  a string, the maximum length
• Complex Option Attributes
• Structure All contained options should be
  present in the Option Selection
• Choice Only 1 of the contained complex options
  should be present in the Option Selection

68
Option Selections

• The Option Selection tree should parallel the
  Option Definitions tree
• Simple Option Attributes
• Option Name Should equal Option Definition name
  at this node in the tree
• Value The value that the user wants assigned
• This will be validated against the information in
  the Option Definition for this node
• Complex Option Attributes
• Option Name Should equal Option Definition name
  at this node in the tree
• ComplexValue This is a container for placing
  the Option Selections for the corresponding child
  Options
• Option Selections are validated against the
  Option Definition when Options are set

69
Options Definition Illustration Example
Name MediaType (ComplexOption) -type
CHOICE
Name FTPPush (ComplexOption) -type
STRUCTURE
Name CD (ComplexOption) Details Removed
Name FTPDir (SimpleOption) -MinOccurs1 -MaxO
ccurs1 -EncryptedNo -PrimitiveTypeString -MaxVa
lue100 -MinValue1
Name UserName (SimpleOption) -MinOccurs1 -
MaxOccurs1 -EncryptedNo -PrimitiveTypeString -M
axValue50 -MinValue1
Name Password (SimpleOption) -MinOccurs1 -M
axOccurs1 -EncryptedYes -PrimitiveTypeString -M
axValue50 -MinValue1
Name Compressed (SimpleOption) -MinOccurs1
-MaxOccurs1 -EncryptedNo -PrimitiveTypeString -
ValidsUnixCompress, GZip
70

• ltOptionSelectiongt
• ltComplexOptionSelectiongt
  ? complex option consists of
• 
  ? an option name and a complex
  value
• ltOptionNamegtECS-TEST
  PKG1lt/OptionNamegt
• ltComplexValuegt
  ? the complex value consists of
• 
  
  ? zero or more simple option
• 
  
  ?and zero or more complex option
  
  
• ltSimpleOptionSelectiongt
  ?simple
  option consists of
• 
  ltOptionNamegtProduction Optionlt/OptionNamegt
  ? an option name
• ltValuegtNative
  Granulelt/Valuegt
  ? and one or more value
• lt/SimpleOptionSelectiongt
• ltComplexOptionSelectiongt
• ltOptionNamegtMedia Typelt/OptionNamegt
• ltComplexValuegt
• ltComplexOptionSelectiongt
• ltOptionNamegtCDROMlt/OptionNamegt
• ltComplexValuegt
• ltSimpleOptionSelectiongt
• ltOptionNamegtCompressionlt/OptionNamegt

71
States of a Provider Order

• Open Order States
• Not Validated
• Validated
• Quoting
• Quoted
• Quote Failed
• Quote Rejected
• Submitting
• Processing
• Cancelling

• Closed Order States
• Deleted
• Submit Failed
• Submit Rejected
• Cancelled
• Closed

72
DeleteProviderOrder
Deleted
DeleteProviderOrder
DeleteProviderOrder
SetOptionSelectionsForProviderOrder
Add, Delete, Update OrderLineItem
UpdateProviderOrderState
QuoteOrder
CreateOrder
No Order
RejectProviderQuote
Add, Delete, Update OrderLineItem
UpdateProviderOrderState
UpdateProviderOrderState
SubmitOrder
SetOptionSelectionsForProviderOrder
SupplyProviderQuote

SetOptionSelectionsForProviderOrder
Add, Delete, Update OrderLineItem
Add, Delete, Update OrderLineItem
SetOptionSelectionsForProviderOrder
SubmitOrder
SubmitOrder
UpdateProviderOrderState
UpdateProviderOrderState
RejectProviderOrderSubmission
UpdateProviderOrderState
AcceptProviderOrderSubmission
CancelOrder
UpdateProviderOrderState
RejectProviderOrderCancellation
AcceptProviderOrderSubmission
CancelProviderOrder
UpdateProviderOrderState
UpdateProviderOrderState
CancelProviderOrder
CloseProviderOrder
Closed
73
How Does ECHO Communicate with Providers?

• User actions on orders that trigger communication
  with providers
• Quote, Submit, Cancel
• Two ways to handle these requests
• Synchronous Action (i.e. ECHO waits for a
  response)
• Asynchronous Action (i.e. contact ECHO with a
  response)
• Retry Mechanism
• If no response received from Partner, ECHO will
  issue another request after waiting for a period
  of time
• Partner can set policy to determine the maximum
  numbers of attempts for retry and waiting time
  between retries
• TrackingID
• ECHO provides a TrackingID on its first
  communication to the Partner
• The Partner has the option of replacing it with
  its own TrackingID
• ECHO will use the current TrackingID for all
  subsequent communications with the Partner and
  make that available to the client

74
What providers give back to users?

• Status Message
• Timestamped message records the status history of
  the Partner order that can be viewed by the user
  using PresentProviderOrder or PresentOrder
  transaction in Order Entry Service
• Additional information for submit response
• Order Information
• total price for the order (including all prices
  below)
• This is the only mandatory item
• price for the data
• price for the media that stores the data
• price for shipping
• price for handling
• any discount made to the total price
• quantity of media used to store the data
• estimated ship date for the order
• Assuming the order is placed immediately
• latest date for cancellation of the order
• additional information
• This is a place for providers to put free text

75
What providers give back to users?

• Additional information for quote response
• Provider Quote
• total price for the order (includes all prices
  below)
• Mandatory
• price for the data
• price for the media that stores the data
• price for shipping
• price for handling
• any discount made to the total price
• recommended media to store the data
• quantity of media used to store the data
• expiration date for the quote
• Mandatory
• additional information

76
Sample XML ECHO?Partner

• ltSubmitProviderOrderRequestgt
• ltOrdergt
• ltProviderIDgtORNLlt/ProviderIDgt
• ltECHOOrderIDgtORNL1234lt/ECHOOrderIDgt
• ltProviderTrackingIDgtORNL1234lt/ProviderTrackingID
  gt
• ltUserInformationgt
• ltECHOUserIDgtlt/ECHOUserIDgt
• ltShippingAddressgtlt/ShippingAddressgt
• ltBillingAddressgtlt/ BillingAddressgt
• ltContactAddressgtlt/ContactAddressgt
• ltNetworkAddressgtlt/NetworkAddressgt
• lt/UserInformationgt
• ltOrderLineItemgt
• ltCatalogItemIDgt9876lt/CatalogItemIDgt
• ltquantitygt2lt/quantitygt
• ltOptionSelectiongtlt/OptionSelectiongt
• lt/OrderLineItemgt
• ..
• lt/Ordergt

77
Synchronous Action

• User sends a request to ECHO to quote, submit, or
  cancel a provider order.
• ECHO sends that message to a specified Partner
  location using the communication protocols
  defined.
• ECHO waits for providers response which will
  come after the Partner performs the requested
  action.
• Partner agrees to TrackingID as discussed on
  previous slide.
• The Partner reacts to the request immediately and
  sends back an answer as part of that response.

78
Synchronous Sample Response

• ltSubmitProviderOrderResponsegt
• ltOrdergt
• ltECHOOrderIDgt1234lt/ECHOOrderIDgt
• ltProviderTrackingIDgtORNL1234lt/ProviderTrackingI
  Dgt
• ltProviderOrderAcceptancegt
• ltlatestCancelDategt01-01-2001
  134200lt/latestCancelDategt
• ltestimatedShipDategt01-02-2001lt/estimatedShipDat
  egt
• ltadditionalInformationgtID5 will ship
  1/4lt/additionalInformationgt
• ltTotalPricegtlt/TotalPricegt
• ltShippingCostgtlt/ShippingCostgt
• ltMediaCostgtlt/MediaCostgt
• ltHandlingCostgtlt/HandlingCostgt
• ltDataCostgtlt/DataCostgt
• ltDiscountgtlt/Discountgt
• lt/ProviderOrderAcceptancegt
• ltStatusMessagegtlt/StatusMessagegt
• lt/Ordergt
• ltSubmitProviderOrderResponsegt

79
Asynchronous Action

• User sends a request to ECHO to quote, submit, or
  cancel a provider order.
• ECHO sends that message to a specified Partner
  location.
• ECHO receives acknowledgement that the message
  was received.
• Partner agrees to TrackingID as discussed on
  previous slide.
• The Partner schedules the action and will later
  initiate a message to ECHO with the result.

80
Asynch. Sample Response

• ltSubmitProviderOrderResponsegt
• ltOrdergt
• ltECHOOrderIDgt1234lt/ECHOOrderIDgt
• ltProviderTrackingIDgtORNL1234lt/ProviderTrackingI
  Dgt
• lt/Ordergt
• ltSubmitProviderOrderResponsegt

81
Introduction to the Client API

• ECHO provides an interface to the outside world
  through XML messages carried over various
  protocols.

82
ECHO Interfaces
Client Interface (RMI) (ECHO API Services)
Config Info
ECHO
Submitl Request
Partner Interface (SOAP) (Quote, Submit, Cancel)
Business Objects
Admin API
Cancel Request
User Accts
Place Order
Catalog
Catalog Search
Quote Request
Partner Mgmt
Full Record Insert, Update, Delete
Partial Record Insert, Update, Delete
Ingest (XML via FTP)
83
What is a Client Developer?

• An ECHO Client Developer uses ECHO APIs to
  provide services to their client community. The
  client interface can make of all or only some
  part of the services that ECHO offers. It is
  intended for the Client Developer to focus on
  providing the best interface possible for their
  target audience and let ECHO focus on keeping the
  Earth Science metadata up to date and dealing
  with the differences among Earth Science Data
  Providers.
• The term Client Partner is used to refer to
  operational systems that use ECHO services

84
Types of Client Providers

• GUIs
• These clients are geared to interface directly
  with a user
• They may use ECHO account services, catalog
  services, order entry services as either guest or
  as a registered user (on behalf of their user)
• They may use only a subset of the above services
• Machine-to-Machine
• These clients do not directly involve a person
• They will typically come from other existing
  system using ECHO as an archive, or from
  specialized GUIs that need some static
  information updated on some schedule
• All ECHO requests can be performed in this way

85
Client Partner Connections

• A Client Partner is running a program X on some
  machine Y that is providing services to a given
  user Z.
• Program X can connect to ECHO in two ways
  currently
• SOAP XML sent over an HTTP connection
• RMI XML sent over an RMI connection
• Program X can represent the user Z in two ways
• Guest user the user will have the capability to
  search public metadata and retrieve, but other
  transactions are limited
• Registered user the user is a known user to
  ECHO, has an account, and is logged in for using
  the system

86
Concept of Logging In

• Most ECHO transactions are available to guests
  (users who have not logged in), but registered
  users (who have logged in) have some additional
  ones they can use
• It is presumed that ECHO Clients will have a user
  that they are representing when they interact
  with ECHO. That user is who the Client should
  log in as, not one representing the Client
  Partner.
• Existing systems may have their own set of users
  which is independent of ECHO
• New systems may decide to rely on ECHO for user
  account management

87
Users and ECHO
Client Partner
ECHO
Registered User
Registered User
Client


88
Seven Basic ECHO Functions

• When a client interacts with ECHO, there are
  really only 7 basic functions
• Login Login to ECHO as a registered user by
  providing userid and password
• If this is not done, then the user is a guest
• Login information if maintained for the length of
  the session (timeout or logout, by IP)
• Logout End an ECHO session
• Perform Send an XML message, get one back
• Identify Provide a client identification string
• SetProviderContext Let ECHO know what Partner
  you are representing
• SetUserContext Let ECHO know what user an admin
  is acting as
• Remove End the session connection with ECHO (or
  set it to NULL)
• Both SOAP and RMI versions are available, others
  are feasible
• Each client will be assigned a string for it to
  use as its identity when communicating with ECHO
• This is useful for debugging issues, and for
  tracking how often a client is used
• The identity of the client will be logged and
  passed along with orders

89
The Session Concept

• As soon as a client grabs a reference to the ECHO
  system, a session is begun
• A session has a user, which may be a guest, and a
  client identification string
• State is maintained until the session ends
• A session ends by calling the remove command or
  by timing out
• The user represented in the session is controlled
  by login and logout
• Until a login command is executed, the user is
  considered a guest user
• After a logout command is executed, the user is
  considered a guest user

90
ECHO Request/Response Paradigm

• ECHOs Perform command allows a client to send a
  string which must be an XML message to the system
• This is referred to as the request message
• The ECHO naming scheme appends Request to the
  message name
• E.g. QueryRequest
• ECHOs Perform command returns a string which
  will be an XML message from the system
• This is referred to as the response message
• The ECHO naming scheme appends Response to the
  message name
• E.g. QueryResponse

91
Understanding XML DTD References

• All Perform transactions take an XML message as
  input
• There is a set of DTDs that describe the
  different messages
• The DTDs are available at http//api.echo.eos.nasa
  .gov/echo/dtd/
• When sending a message, it is important that
  internal DTD references are consistent

92
Client Connectors

• ECHOs preferred method of communication is
  through SOAP
• Since SOAP is a relatively new standard, ECHO has
  provided some code to facilitate sending SOAP
  messages directly from 2 different client
  languages more easily
• Perl, Python
• They offer direct access to a way to call the
  seven different basic ECHO commands login,
  logout and perform, set user context, set
  provider context, identify and remove
• Currently available at http//www.echo.eos.nasa.go
  v/clientpartner-resources.shtml

93
Client API Future Items

• ECHO Services as Web Services
• As ECHO starts to represent Earth Science Data
  Services, we intend to register ECHO services in
  the services directory as well
• ECHO will provide WSDL descriptions of its
  services and provide compliant SOAP access
• We are considering a finer grain object view with
  each ECHO service being its own service, rather
  than the course grained 4 method service
  currently available

94
The Java Client Toolkit (aka Façade)

• An easy mechanism for Java developers to
  communicate with ECHO.

95
Façade Motivation

• In developing management clients to ECHO, the
  development team found out a few things
• A large amount of code was shared among the
  different clients
• The creation and parsing of XML makes for less
  readable code in your development language
• Im coding my client in Java. ECHO is in Java.
  Isnt there an easier way for me to access ECHO
  without going through this XML middle layer?
• ECHO originally chose XML as its message
  interface format because it enables heterogeneous
  clients
• SOAP has provided a new way to leverage XML but
  still have your client interface in your own
  language
• ECHO has a SOAP interface that mimics the way the
  system was originally designed, but will be
  moving to a web service approach
• In the meantime, the Java Client Toolkit (aka
  Façade) can be used by Java developers to
  communicate with ECHO without using XML (except
  in queries and presents)

96
Pre-Façade Code Example

• Document requestDoc new DocumentImpl()
• Element service requestDoc.createElement("UserAc
  countService")
• requestDoc.appendChild(service)
• Element transaction requestDoc.createElement("Ad
  dAddressRequest")
• service.appendChild(transaction)
• Element addressInformationElement
• requestDoc.createElement("AddressInformation")
• transaction.appendChild(addressInformationElement)
  
• Element idElement requestDoc.createElement("Addr
  essId")
• addressInformationElement.appendChild(idElement)
• Element theIdElement requestDoc.createElement(A
  ddressId)
• theIdElement.appendChild(requestDoc.createTextNode
  (addressId))
• IdElement.appendChild(theIdElement)
• Element usFormatElement requestDoc.createElement
  ("USFormat")
• usFormatElement.appendChild(requestDoc.createTextN
  ode(usFormat))
• addressInformationElement.appendChild(usFormatElem
  ent)
• // and so on for Street1-5, City, State, and
  Zip Code

• // submit the message and get the response
• OutputFormat format new OutputFormat(requestDoc)
  
• String dtdAddress
• http//api.echo.eos.nasa.gov/dtd/UserAccountServic
  e.dtd
• format.setDoctype(null, dtdAddress)
• StringWriter stringWriter new StringWriter()
  XMLSerializer serializer new XMLSerializer(strin
  gWriter, format)
• serializer.asDOMSerializer()
• serializer.serialize(requestDoc.getDocumentElement
  ())
• String requestMessage stringWriter.toString()
• String responseMessage null
• try
• responseMessage echoToken.perform(requestMessage
  )
• catch (XMLServiceException e)
• // do something intelligent
• // now parse the response message

97
Façade Code Example

• AddressInformationDO address null
• try
• address new AddressInformationDO(new
  AddressID(home), true, street1, street2,
  street3, street4, street5, city, state, zip,
  country)
• catch (ValidationException e)
• // some validation can be performed client side
  within remote invocation
• try
• UserAccountService.AddAddress(address,
  echoToken)
• catch (XMLServiceException e)
• // some validation can only be performed on the
  server, this is where those errors are reported

98
Service Level Common Functions
99
Façade Primary Classes

• One primary class for each service in the system
• AdministrationService
• CatalogService
• DataManagementService
• GroupManagementService
• OrderEntryService
• ProviderAccountService
• ProviderOrderManagementService
• ProviderProfileService
• RegistrationService
• SubscriptionService
• UserAccountService
• The state of the session is maintained through
  ECHOToken

100
What is the ECHO Token?

• Clients use ECHOs Session Manager to establish
  the session state
• Associate a user
• Identify the client
• Define Partner that the user is representing
• Define user that an admin is representing
• Since SOAP is stateless, the ECHO Token is used
  to embody the state, and is passed in with each
  request
• Instead of calling the Session Manager directly
  in the Façade, you create an ECHO Token

101
Client Toolkit Link

• The Client Toolkit (Façade) can be found linked
  on the main ECHO web page
• http//www.echo.eos.nasa.gov/clientpartner-resourc
  es.shtml
• Javadocs
• Client Toolkit Library
• Source code

102
Registration Service Transactions for Clients

• The Registration Service allows a guest to create
  a new registered user.

103
ECHO Interfaces
Client Interface (RMI) (ECHO API Services)
Config Info
ECHO
Submitl Request
Partner Interface (SOAP) (Quote, Submit, Cancel)
Business Objects
Admin API
Cancel Request
User Accts
Place Order
Catalog
Catalog Search
Quote Request
Partner Mgmt
Full Record Insert, Update, Delete
Partial Record Insert, Update, Delete
Ingest (XML via FTP)
104
User Registration

• CreateUserRequest
• Creates a new registered user within the system
• UserName, password, first name, last name, email
  address are all required
• Address and Phone are optional

105
Sample User Registration
lt?xml version"1.0" encoding"UTF-8"?gt lt!DOCTYPE
RegistrationService SYSTEM 'http//api.echo.eos.na
sa.gov/echo/dtd/RegistrationService.dtd'gt lt!--
Create a new user --gt ltRegistrationServicegt ltCrea
teUserRequestgt ltUserNamegtkwichmalt/UserNamegt ltP
asswordgtpasswordlt/Passwordgt ltUserInformationgt
ltFirstNamegtKeithlt/FirstNamegt ltLastNamegtWichman
nlt/LastNamegt ltEmailAddressgtwichmann_at_gst.comlt/Em
ailAddressgt lt/UserInformationgt lt/CreateUserRequ
estgt lt/RegistrationServicegt
Note The XML version and DOCTYPE tags are
vital