Designing Node 2'0 Data Exchanges

1
Designing Node 2.0 Data Exchanges

• EN National Conference, 4/30/2009
• Bill Rensmith
• Windsor Solutions, Inc.

2
Agenda

• Compare/Contrast Node v1.1 to v2.0
• Migrating Existing Exchanges
• Leveraging Node 2.0 for New and Upgraded
  Exchanges
• Proposed Changes to Design Rules and Guidance
• Documenting Node 2.0 Exchanges

3
Compare/Contrast Node v1.1 to v2.0
4
Node Primitive Methods

• Authenticate
• Submit
• Download
• Query
• Solicit
• Notify
• Execute
• GetStatus

5
Node Specification 1.1 vs. 2.0 - Authenticate

• Optional domain element currently unimplemented
• Not relevant to flow developers (yet)
• securityToken renamed in 2.0, but function is
  unchanged

6
Node Specification 1.1 vs. 2.0 - Submit

• Solicit and Submit have similar changes
• flowOperation tells target node what to do with
  the request or submission. Allows support for
  multiple types of processing.

7
Node Specification 1.1 vs. 2.0 Submit (contd)

• flowOperation is required!
• default used for migrated flows
• Should describe type of data or type of action to
  be performed

8
Node Specification 1.1 vs. 2.0 Submit (contd)

• recipients and notificationURI discussed later
• Returns status immediately
• Instant feedback only available if receiving node
  can process immediately

9
NodeDocuments in Node 2.0

• Implemented in Submit and Download
• DocumentFormat is now enumerated list (XML, Flat,
  Bin, Zip, etc)
• New document ID (optional)
• Unique Identifier,
• GUID form

10
Node Specification 1.1 vs. 2.0 - Download

• Like v1.1, the download method can be used to
  retrieve a specific document
• Rarely used this way in practice
• Requestor includes document name to download
• If FCD specifies document names, then requestor
  could anticipate ahead of time.

11
Node Specification 1.1 vs. 2.0 - Query

• Addition of dataflow a big improvement
• In Node 1.1, request names must be unique across
  all flows
• Parameters are the big difference in Node 2.0
  (next slide explores this)
• Structured ResultSetType response in Node 2.0
• rowId, rowCount, lastSet, results (xml doc)
• Supports more robust result set paging
• What is a row? Flow developer needs to define!

12
Parameters in Node 2.0

• Example request GetFacilityByTypeAndZipCode
• Accepts one facility type
• Accepts one or more zip codes
• Node 1.1
• An array of strings
• NPDES,4051240518
• Node 2.0
• Strong type name (reqd), type (opt), and
  encoding (opt)
• ltparametersgt
• ltparameter nameFacilityTypegtNPDESlt/parametergt
• ltparameter nameZipCodegt40512lt/parametergt
• ltparameter nameZipCodegt40518lt/parametergt
• lt/parametersgt
• Document Allowed Occurrence in FCD!!!

13
Node Specification 1.1 vs. 2.0 - Solicit

• Similar to Query
• Addition of dataflow, recipients,
  notificationURI, and parameters
• returnURL goneno more Solicit w/ Submit!
• Response is different
• Returns instant status (always received?)

14
Node Specification 1.1 vs. 2.0 - Notify

• Usage is same as Node 1.1
• Document Notification
• Event Notification
• Status Notification
• Implemented rarely in practiceexamples anyone?

15
Node Specification 1.1 vs. 2.0 - Notify

• NotificationMessageType is new
• Contains message text and one or more statuses
• Each status has a required ObjectID
• Response is the same as Solicit and Submit

16
Node Specification 1.1 vs. 2.0 - Execute

• Implemented very rarely, but has potential!
• Revamped for Node 2.0
• interfaceName and methodName allow for flexible,
  expandable use
• Return XML must be defined in FCD!

17
Node Specification 1.1 vs. 2.0 - GetStatus

• Similar to Node 1.1, only offers more robust
  status response text

18
Recipients and notificationURI Overview

• Both Recipients and notificationURI
• present on Submit and Solicit requests
• support zero or more node URLs and/or email
  addresses
• relate to communicating results during or after
  processing by a node
• must return error if supplied but not supported
• E_RecipientNotSupported
• E_NotificationURINotSupported
• Up to flow developers to clearly document usage
  in FCD, if supported at all.

19
Recipients Parameter

• Sends the processed submission package on to
  the recipient
• Email Send an email with processed package
  attached
• Email must contain node URLs (originating and
  recipient), transaction id, processed package,
  other info as specified in FCD
• Node use Submit to send processed package
• Specifics determined in FCD
• What is a processed package???
• Up to flow developer to define and document!

20
Recipients Parameter (Contd)

• Challenge Is recipient node a v1.1 or v2.0
  endpoint?
• No straightforward way to communicate this in the
  Submit/Solicit request
• Different nodes use different approaches
• Default Assume it is Node 2.0 since recipients
  is a Node 2.0 construct

21
NotificationURI

• Used to send a status notification when the
  status of the transaction changes
• Optional NotificationType attribute
• Warning
• Error
• Status
• All
• None
• If no NotificationType is specified, assume All
• Assumption is Notify should be used for node
  notifications

22
Example of Recipients and NotificationURI
ltsubmitgt ltrecipientsgtmichael_at_dm.comlt/recipients
gt ltnotificationURIgtdwight_at_dm.comlt/notificationU
RIgt lt/submitgt
Notifications
Recipients
23
Migrating Existing Exchanges to Node 2.0
24
Migrated Exchanges

• Node 2.0 FCD Addenda Available for
• WQX v2.0
• AQS v2.1
• BEACHES v2.1
• FRS v2.3
• UIC v1.0
• All involve Submit operations to EPA
• WQX and FRS implement Query/Solicit
• Addenda defines
• proper dataflow parameter (new in Node 2)
• ParameterName text values
• EPA Ready to accept. See EN Wiki for endpoints.

25
Migrating Existing Exchanges

• See RCRAInfo v4.0 FCD
• Released March 31, 2009
• Supports both Node 1.1 and Node 2.0
• Good example of documenting flow implementation
  details for both endpoint types
• Special considerations
• Support for Header (v0.9 or v2.0?)
• NAAS (Node 2.0 uses NAAS 3.0)
• New dataflow parameter
• Specifying parameter names and occurrence
• Support for recipients and notificationURI

26
Leveraging Node 2.0 for New and Upgraded Exchanges
27
Exchange Network Header v2.0

• Serves as a wrapper for XML files, describing
  metadata (who, when, etc.)
• Usage
• Required for all Node 2.0 Submit operations
• Optional for use on Query responses and other
  operations
• Migrated Exchanges Use Header 0.9 or 2.0?
• See Header Specification v2.0 on EN web site

28
Adding Processing Instructions to a Submission

• Three main ways to instruct a receiver how to
  process a file (using Submit)
• In Submits flowOperation element
• In Header property name/value pairs
• In Header payload operation attribute
• Quiz Which option is new for Node 2.0?

29
Adding Processing Instructions to a Submission
(contd)
SOAP Envelope ltsubmitgt
Document
lt/submitgt
30
Adding Processing Instructions to a Submission
(contd)
SOAP Envelope ltsubmitgt ltdataflowgtMyFlowlt/datafl
owgt ltflowOperationgtPermitlt/flowOperationgt
Document
lt/submitgt
31
Adding Processing Instructions to a Submission
(contd)

• Submits flowOperation element
• indicates the specific processing for the
  document, as defined in the FCD. Node 2.0 Spec
• Headers name/value pairs
• an extension mechanism for the header
• ICIS-NPDES, RCRA use for notification email
  addresses. Other examples?
• Headers payload action attribute
• An operation to be carried out by the
  receiverThe actual value depends on the data
  flow, and should be defined by an IPT.
• See Header v2.0 documentation

32
Naming Data Services in Node 2.0

• Node 1.1 Convention
• FlowName.MethodObjectByParameterName(s)
  _vVersion
• Example WQX.GetActivityByParameters_v2.0
• Early flows did not use prefix, version suffix
• Node 2.0 Convention
• Drop flow name prefix
• No longer needed due to addition of new dataflow
  element in Solicit/Query
• Version still important, references schema
  returned
• Not codified in rules/guidanceyet

33
What is a Row?

• Query allows paging using rowId and maxRows
• but what is a row?
• Flow developer must document in FCD for each root
  schema in the exchange
• More important in Node 2.0 due to new
  ResultSetType, indicates row number, count.
• Quiz How many FCDs currently define a row?

34
So What is New and Cool in Node 2.0?

• More robust query paging
• Status change/error notifications
• Forwarding of processed submissions using
  recipients
• Instant status updates upon submit
• ENDS is an EN Service directory, highlights data
  publishing, more Query traffic? Revamped Execute
  method
• New options available! Use it!
• Can be registered in ENDS

35
One IdeaFacebook meets EN. Execute or RSS to add
facility feed
36
So What is New and Cool in Node 2.0? (contd)

• Not many Node 2.0 flows out there yet
• so be an innovator!

37
Documenting Node 2.0 Exchanges
38
Tips for Documenting Node 2.0 Exchanges

• If FCD supports both Node 1.1 and Node 2.0
  exchanges, note which features are relevant to
  which endpoint (see RCRA v4.0 FCD)
• Data Services
• Indicate min/max occurrence for each parameter
  (0-1, 1, 0-n, 1-n)
• Parameter name must be specific
• Name the data service properly
• Specify Endpoint URLs for regulatory and
  central-node exchanges

39
Tips for Documenting Node 2.0 Exchanges (contd)

• Define a row for each Query/Solicit service
• If recipients is supported define what
  constitutes a processed submission
• If not supported, document that too!
• If notificationURI is supported, define whether
  notificationTypes are supported and behavior for
  each

40
Proposed changes to Exchange Development Guidance

• exchangenetwork.net -gt Build an Exchange
• Currently 19 documents listed
• Even more guidance on different pages
• Header 2.0 Specification
• ENDS 2.0 Specification
• NOB Decision Memoranda
• Very difficult for flow developers to follow
• 500 printed pages

41
Proposed changes to Exchange Development Guidance
(contd)

• Analysis of current rules and guidance complete
• Findings presented to NTG April, 2009
• NTG agreed there is a need to streamline and
  consolidate

42
Proposed changes to Exchange Development Guidance
(contd)

• Next steps
• Adopt DRC-style format for non-schema rules and
  guidelines
• Extract narrative rules and translate to
  specific, succinct, enumerated statement

43
Proposed changes to Exchange Development Guidance
(contd)

• Replace Schema Design Rules and Conventions
  (DRCs) with Exchange Design Rules and
  Conventions
• Existing documents archived
• Single, living repository
• Simplifies flow development
• One stop list
• Searchability (online?)
• Simplifies schema review
• Simple reference to the violated rule
• Much more maintainable

44
Proposed Changes to Schema Conformance Review
Process

• Issues with Schema Conformance Report preparation
  and review process
• Solely focuses on schema, nothing on FCD
• Heavy emphasis on documenting/justifying SSC
  usage
• Burdensome to prepare
• We should be removing barriers to flow
  development, not adding to them
• Difficult to evaluate from reviewer side
• Caused by current disparity in rule/guideline
  documentation

45
Proposed Changes to Schema Conformance Review
Process

• Proposed changes
• Change to Flow Conformance Review
• Expand beyond schema
• Simplify SCR preparation
• Move to checklist format
• Provide explanation for nos
• Deemphasize SSCs
• No schedule yet for implementation
• Changes will require review and approval
• Dependent upon changes to guidance

46
Questions and Discussion