NCAI Exchange Network

1
Tribal Water Quality Data Management

• NCAI Exchange Network
• Tribal Users Meeting
• April 9, 2008

2
Overview

• Tribal 106 Grants Requirements for Monitoring and
  Assessment
• Essential concepts of Water Quality Data
  Management
• Understanding WQX
• Steps to a WQX Submittal
• Accessing Data
• Upcoming Developments
• How to decide where to begin

3
Tribal 106 Guidance Requirement

• All tribes must submit a TAR (Tribal Assessment
  Report) to continue eligibility for 106 funds
  (8-3)
• Monitoring Strategy
• Water Quality Assessment
• STORET-compatible electronic data

4
106 Guidance parameters Fundamental Programs -
Dissolved Oxygen (DO) - Temperature - pH -
Turbidity Intermediate Programs - Nitrogen -
Phosphorus Mature Programs - Pathogens -
Macroinvertebrates - Physical Habitat indicators
5
10 Elements of a Tribal Monitoring Strategy
I. Monitoring Strategy II. Monitoring
Objectives III. Monitoring Design IV. Core
Indicators V. Quality Assurance VI. Data
Management VII. Data Analysis/Assessment
VIII. Reporting IX. Programmatic
Evaluation X. General Support and
Infrastructure
6
Essentials of Data Management

• Monitoring data consists of
• Where sample/measurement was collected
• When it was collected
• What was collected
• How it was collected
• Why it was collected
• Who collected it
• It does not take a lot of information to create a
  good data set.

7
How does data management fit in?
The life of a sample begins here
Field Sheets capture data
Labs generate data
Now what to do with all this data?
8
First, make sense of what you have
Organize data electronically

• Use the tools youre familiar with
• Access
• Excel
• Other higher-powered database management system
  (i.e. Oracle)

Have a plan for growth
Not only will your skill grow, but your data
volume will grow. Be prepared for it.
Find out whats already available
9
Data are valuable, plan for re-use
Electronic data are more valuable than data in
file cabinets
The more data are re-used, the more valuable they
become
Shared data are of even higher value

• Provide for better planning decisions
• Incentivize collaborative efforts
• Make the most use of the monitoring s being
  invested

10
How EPA Describes Monitoring Data
Three categories of data

• Projects (Why data were collected)
• - Brief summary of monitoring plan
• Monitoring Locations (Where data were collected)
• - Describe where monitoring takes place
• Results (When, How and What were collected)
• - Measurements of what were monitored

11
WQX is designed around these concepts
12
The WQX Schema

• The WQX schema is a definition of the structure
  and format that data must have to be submitted
  through WQX.
• Primary components Organization, Project,
  Monitoring Location, Monitoring Activity,
  Samples, and Results
• Data elements currently accommodate physical,
  chemical, and fish tissue data
• Biological monitoring and habitat data flow will
  be incorporated in the very near future

13
Submitting data using WQX

• Water Quality Exchange (WQX) uses the Exchange
  Network to flow physical, chemical and fish
  tissue, and soon biological and habitat data from
  an tribes own database to the existing STORET
  Warehouse
• Tribes that have been awarded Exchange Network
  Grants will be able to submit data through WQX

14
How WQX Works
Approved XML Submission
XML Submission
CDX
WQX Parse and Load Software
Data Context Error Report
Schema Validation Receive Error Responses
Data Load Confirmation and Summary
Data Submitter
WQX ODS
15
WQX in the big picture
16
Steps to Submitting to WQX

• Review the resources that are available
• Getting involved with the Exchange Network
• Getting a NAAS Account
• Getting an Organization ID
• Understanding XML (How does my data relate to an
  XML file?)
• Submitting a file
• Understanding errors
• Understanding the WQX structure
• Interacting with WQX

17
Step 1. Resources Available

• The STORET/WQX team has several resources
  available to help you
• Available at the STORET web site
  www.epa.gov/storet
• Visit the WQX section http//www.epa.gov/storet/w
  qx.html
• Diagrams and presentations about how WQX works
• Presentations from others on how theyve
  implemented WQX (including tribes)
• Lists of allowable values for WQX
• Available at the Exchange Network Site
  http//www.exchangenetwork.net/exchanges/water/wqx
  .htm
• Read the Flow Configuration Document
• See example XML Files
• Examine the Data Exchange Template
• Last but not least, dont be afraid to ask for
  help
• Join the STORET list serve www.epa.gov/storet/li
  stserv.html
• Call the STORET hot line 1-800-424-9067
• end an email to storet_at_epa.gov

18
Step 1. Remember this
All of this information is available at
www.epa.gov/storet
19
Step 2. Getting involved in the Exchange Network

• Ask yourself the following questions
• Does my organization have a node?
• Have we submitted data for another flow?
• Did we use a node? Node client? Or CDX Web?
• Who is my Node Administrator? Do we have one?
• If the answer to these questions is No or I
  dont know visit www.exchangenetwork.net

20
Step 3. Getting a NAAS Account

• The NAAS or Network Authentication Authorization
  Service is a user account that you can use to
  access the Exchange Network. They can be
  specific for an individual node or for a
  particular user
• NAAS accounts are assigned by Node
  Administrators. If you dont have a Node
  Administrator, contact the CDX help desk at
  888-890-1995 Or epacdx_at_csc.com

21
Step 4. Getting an Org ID

• An Organization ID (Org ID) is a unique
  identifier assigned to your organization (i.e.
  YUROK for the Yurok Tribe).
• Org IDs are assigned by EPA upon the request of
  the user.
• You must have an Org ID to be able to submit to
  WQX
• To request an Org ID, send an email to
  storet_at_epa.gov
• Provide the NAAS accounts that you want to have
  access to your Organization

22
Step 5. Understanding XML

• XML can be intimidating at first
• There are many ways to create XML, some things to
  do
• The sample files and Data Exchange Template are
  key to understanding how to create a WQX XML
  file.
• Find a programmer who can spend 20-40 hours on
  designing an XML file
• Have some test data that are representative of
  the data youll be submitting
• Test the procedures that the programmer developed
  (plan for an additional 20-40 hrs testing)
• Start small, it takes time to create an XML file,
  dont waste development time waiting for a
  computer to process a file.
• You should only have to do this once, after
  that, youre just running the procedure.

23
Step 6. Submitting a File

• Once you have a NAAS account and an Org ID, and
  have successfully created an XML file, youre
  ready to submit.
• The original test file should be small, but
  representative of the data youll be submitting
• Submit to CDX test first
• Once you get a successful submission, keep track
  of the transaction ID
• Send an email to the CDX help desk. Requesting
  access to WQX production. Provide them the
  transaction ID of the successful submission.
• Go ahead an begin submitting to production

24
Step 7. Understanding Errors

• There are two types of errors in WQX
• 1. Schema Errors the submitted file does not
  conform to the WQX Schema. (Report is available
  in the file Validation Results which is an xml
  file).
• 2. Processing Errors the submitted file does
  not pass all of the WQX Business Rules (Report is
  available in the file ProcessingReport.zip)

25
Step 8. Understanding the WQX Structure

• WQX files should be organized as follows
• Header (with processing instructions)
• Organization
• Organizational Information
• Projects
• Monitoring Locations
• Activities
• Linkage to Projects
• Linkage to Monitoring Locations
• Results
• Activity Groups

26
Step 9. Interacting with WQX

• WQX is designed to function in a Node-to-Node
  environment (i.e. you manage your data, let your
  computer and Node keep EPA in sync).
• WQX allows Inserts, Updates, and Deletes via a
  batch process
• WQX has Outbound services via the Query and
  Solicit method data visualization and system
  synchronization purposes.

27
Accessing Data
The National STORET Data Warehouse serves as the
national repository for Modern STORET data as
well as WQX data. Users may obtain data from the
warehouse through the STORET home page
www.epa.gov/storet Web services providing direct
access to warehouse data will also provide
enhanced accessibility to the data.
28
STORET Warehouse Updates

• The National STORET Data Warehouse enhancements
  for improved data accessibility
• New Watershed Summary Available Now
• New Web Services Available Now
• Please visit the STORET website for more
  information www.epa.gov/storet

29
(No Transcript)
30
Where do Web Services fit?
Data in/out services via WQX
WH services available via Exch. Net.
Data out services through direct access to the WH
31
All Done with Web Services!!
32
An example using Monitoring Data
33
Whats coming next?

• WQX 2.0 is near completion
• Adds Biological data and Habitat data to the
  schema
• Supports Bio/Habitat metrics and indexes
• EPA will continue to support WQX 1.0 for the
  foreseeable future
• XML Generation Tool
• Tool that will support organizations that dont
  have the ability to create XML
• Batch Geo-location tool
• Tie Monitoring Locations to the National
  Hydrography Dataset

34
Dates to Remember

• WQX Version 1.0 data flow in production now
• Physical, chemical, microbiological, and fish
  tissue data only
• Add Biological/Habitat data to WQX schema
• Bio/Habitat WQX in production (May 08)
• Develop Web tool for XML Generation
• Tool available for general use (September 08)
• Batch Geo-location Tool
• Under development available January 2009

35
What is the XML Generation Tool?

• A web-based application that assists a data
  provider with the creation of a valid submission
  file for the Water Quality Exchange.
• The next generation of SIM and WebSIM, redesigned
  to target WQX.

36
Data-sharing for all

• This tool is meant to allow those who do not have
  Exchange Network Nodes the ability to share data
  over the web
• The tool will be easy to use and accept basic
  input formats (i.e. Excel Spreadsheets).
• Data submitted via this tool will be available on
  the internet within one week of submission.

37
How the XML Generation Tool fits in

• XML Generation Tool
• Import Flat File
• Apply Defaults Translations
• Validate Correct
• Export XML

• CDX Lite
• Submit
• Get Status
• Download

• CDX Web
• Online Registration
• Log In

• WQX System
• Load XML File
• Update WQX Database
• Create Processing Report

CDX Node

• State/Tribal Node
• Submit
• Get Status
• Download

• State/Tribal System
• Retrieve Data from Database
• Export XML

38
Demo
By the Way, what happened to National WebSIM?
39
Batch Geo-Locating

• New development that EPA is working on
• Allow for any new stations submitted via WQX to
  be automatically georeferenced to the NHD
• The Georeferenced reach codes would be returned
  to the data provider, as well an opportunity to
  QA the georeferenced locations
• This will use a concept called Rain Drop to
  ensure that points are tied to the correct NHD
  reach

40

• If the two points represent Accuracy/Precison
  issues then note how raindrop results in same
  indexed location
• Note how upper point with straight snap would go
  to wrong catchment
• Note how the path does trace the topo delineation
  for water

41
Why this is Important?

• Allows for monitoring data to be easily linked to
  other types of data (i.e. Impaired waters,
  assessments, permits, etc.)
• Allows for upstream/downstream analysis
• Greatly expands the usability of monitoring data

42
Batch Indexing Timeline

• This capability should be available January 2009
• Deployed as a service that any data flow could
  use (not just for WQX)
• Also provide some Geospatial QA services (i.e. is
  the lat/long in the county/HUC/state that the
  user specified)

43
Where to Get Started

• Figure out what data you have, and what format it
  is in.
• Find out where you stand with the exchange
  network
• Visit www.epa.gov/storet
• Give us a call, we can walk you through any
  questions.

44
Web Resources

• www.epa.gov/storet
• www.exchangenetwork.net/exchanges/water/wqx.htm
• www.epa.gov/storet/wqx.html

45
Question/Comments

• Contact
• Dwane Young, EPA [phone]
• Young.dwane_at_epa.gov
• Kristen Gunthardt, EPA [phone]
• gunthardt.kristen_at_epa.gov
• STORET Help Desk 800-424-9067
• storet_at_epa.gov
• Join the STORET list serve for more information,
  including monthly conference calls about STORET
  and WQX www.epa.gov/storet/listserv.html