Title: MWCOG GIS Committee Meeting Regional Geospatial Interoperability
1MWCOG GIS Committee MeetingRegional Geospatial
Interoperability
May 25, 2006 Rich Grady, President Applied
Geographics, Inc.
2Introduction
- Applied Geographics, Inc. (AppGeo)
- www.appgeo.com
- GIS Consulting firm in Boston,15 years in
operation, 25 professionals - Service-provider in several Homeland Security and
Emergency Management focused GIS projects - Focused on State Local Government GIS system
planning and implementation -
3General Observations on Homeland Security
- There are many stakeholders involved in Homeland
Security, representing many perspectives. - Stakeholders include
- Different levels of government (federal, state,
and local), and the private sector - Different professional perspectives, including,
public works, public safety, public health,
planning, and GIS - It is essential for a region to communicate and
coordinate on a proactive basis
4Questions to think about
- What data is needed?
- Who has it?
- How do we achieve interoperability?
- Are there some good examples of whats been done
to-date?
5Considerations
- Inventory of Available Data
- Application Objectives
- Standards
- Schema Design
- Data Collection and Integration
- Data Maintenance
- Metadata
- Output
- Security
6Many Different Sources ( schemas, standards,
products, applications, reference systems,
etc.)!!!
Regional
Municipal
Data Repository
Private
County
Military
State
Federal
Data Sources wide variety of quantity and quality
7Project Examples
- Overview of Boston Preparedness Pilot
- Defined First Responder requirements
- Identified data integration issues
- Observations from NCR/HSIP Pilot and involvement
in Project Homeland HSIP refinement - Issues of schema development and normalization
- Issues of practical applications
8The Boston Preparedness Pilot Project, Sept. 2003
- Cover story of GeoIntelligence Magazine Mar/Apr
04 - http//www.appgeo.com/clients/NOAA_HomelandSecurit
y - Phase I Funded by MassGIS
- Poll First Responders for GIS data needs
- Explore data schema design issues
- Phase II Funded by NOAA
- Develop/assemble priority critical infrastructure
(CI) data layers - Offer lessons learned to other 132 Urban Areas
9Pilot Area 102 Cities Towns
10Boston Preparedness Pilot Project Data Quality
Recommendations
- Assembling the most accurate data requires local
input - Local data must rollup to national level, not
vice versa - Assembling the data is only the first step,
mechanisms for maintaining the data are essential - Building high-quality homeland security and
critical infrastructure protection applications
implies a need for laborious data quality
improvement work - Unglamorous data development and data quality
enhancement work is a necessary precursor to more
sexy application development work - On-line, web-based markup tools can greatly
facilitate the ability for widely decentralized
organizations to participate in data collection
and quality assurance efforts
11Boston Preparedness Pilot Project
RecommendationsOn the role of the Federal
Government
- Recommendation that local data should rollup
into national data sets and that the Federal role
in direct data collection might be limited - Purchasing imagery
- Establishing blanket agreements with commercial
data providers - However, the Federal Government plays a vital
role to help orchestrate decentralized data
collection efforts - Setting standards for data format and content
- Local transactions, National Specifications
- Setting ground rules for data collection
practices - Facilitating the development of the recipe that
each Urban Area should follow
12Homeland Security Infrastructure Protection
(HSIP) / National Capital Region (HSIP/NCR) Pilot
Project
- Worked on project for USACE and NGA as sub to
Michael Baker Corp. during 2004 - Explored data schema options and complexities and
piloted development for NCR - Reconcile SDSFIE with HSIP
- Develop a schema for a subset of HSIP and
implement in a personal Geodatabase - HSIP layer list vs. structured schema
- Conceptualize HSIP data warehousing scenarios
and ETL procedures - Develop tools for schema-to-schema mapping and
metadata tracking from multiple sources - Populate target HSIP schema with actual data from
NCR - Document lessons-learned and recommendations
13Project Overview
- Review of prevalent schema literature
- HSIP Schema formalization and SDSFIE, FACC
crosswalks - GIS Data Warehouse Processes and Software Tools
- National Capitol Region Pilot
14Public State and Local GIS Data
- Area of interest 3 states and 8 counties
- Required data request letter from Federal
customer - Initial web-research on local data/metadata
catalogs - Data Use and License Agreements required
- Primarily planimetric data and base mapping
- Lacking in energy/gas/water utilities and
telecommunications
15National Capitol Region
166 Phase Dataset Workflow
- Data Inventorying and Acquisition
- Data Registration, Geometric Transformation, and
Spatial QA/QC - Schema Mapping and Tabular Transformation
- QA/QC, Completeness, Supplemental Data Activity
- Data Load to Central Repository
- Data Distribution and Use
17Phase 1 Data Inventorying and Acquisition
SDSFIE
Federal
Military
EPA Tier II
Dataset Tracker
Inventory Acquisition
FACC
GDT
Commercial
Local
County
18Phase 2 Data Registration, Geometric
Transformation, and Spatial QA/QC
SDSFIE
Geom Transform
FME
Persistent Tabular Transform
EPA Tier II
Maintain Original Primary Key
FACC
Staging Dataset 1
GDT
Spatial QA/QC
Staging Dataset 2
County
ArcMap
19Phase 3 Schema Mapping and Tabular Data
Transformation
Staging Dataset Schema
Schema Map
Mapper
Tabular Schema Mapping
Target HSIP Schema
Staging Dataset 2
Data Complete Check
Tabular Transform
DTS
Staging Dataset 3
Persistent Tabular Transform
20Phase 4 QA/QC, Completeness, and Supplemental
Data Activity
Data Complete Check
Tabular Schema Mapping
Staging Dataset 3
Data Collection
Internet, Misc.
Staging Dataset 4
21Phase 5 Data Load to Central Repository
Merge To Central DB
Staging Dataset 4
Central Repository Dataset 1
ArcGIS DTS
Central Repository Dataset 2
22Phase 6 Data Distribution and Use
Distribute
Data Access and Products
Central Repository Dataset
Desktop or Web Use
Return Improved Data to Provider
23Warehouse Process and Software Tools Cross
Reference
24Warehouse Software Tools
- ESRI ArcGIS ArcMap/ArcEditor (COTS)
- SafeSoftware Feature Manipulation Engine (FME)
(COTS) - Microsoft Data Transformation Services (DTS)
(COTS) - Data Warehouse Dataset Tracker Application
- Schema Mapper Application
- Data Review and Completeness Checker
- HSIP Schema Browser
- HSIP Schema Downloader
25Project Homeland
- Worked under ESRI on contract to National
Geospatial-Intelligence Agency (NGA) during
2004-2005 - Developed a refined Homeland Security
Infrastructure Protection (HSIP) layer list,
including Sectors and Features, with associated
documentation - Built a schema for the refined HSIP
- Implemented in a personal Geodatabase with
associated documentation - Delivered to ESRI for application to Pilot Cities
(e.g. San Francisco, Colorado Springs, Arizona
Border Area)
26Findings across pilot projects
- Schema issues are important but have yet to be
resolved categorically - Project homeland is ongoing
- People continue to wait for the next HSIP
- Simplicity aids in data warehousing and
interoperability - Complexity aids in rich applications
- Highlights importance of developing these schemas
- Data contributors want the roadmap
27Complexity Range Schemas, Data and Applications
Custom Structures/Applications
Statistical Models
Machine Processing
CATS
EPA Tier 2
HAZUS
SDSFIE
Normalized
Network Tracing
Application Sophistication
ESRI HS
HSIP
GeoDatabase
De-normalized
FACC
ArcIms ShapeFile
Human Visual
Basic Data
Features Attributes Domains Relationships
Cost
28Interoperability can benefit from schema
simplification
- Easier to exchange and import into central
repository - The ETL is easier (extraction translation
loading) - Facilitates use of XML
- Easier to QAQC and cleanup
- Easier to use in viewing applications
29Tabular attribute correlations and
transformations the hardest part of the job
By separating geometry from attributes, we have
a wider selection of ETL tools to choose from for
processing tabular data.
30Distributed responsibilities and a common
schema for interoperability are needed
- The authoritative data sources and management
operations are distributed. - The central warehouse authority reviews and helps
coordinate data gathering, but does not actually
gather or QAQC data. - Given a common schema, data can be rolled up and
processed with the same queries and applications.
31Geocoding is not good enough
Actual Building Location
Geocoded Source A (MassGIS)
Geocoded Source B Commercial Vendor
32Data Accuracy Refinement
1. Geocoded Point
2. Visual or Locally Assisted Confirmation of
Asset
3. Footprint-level Feature Capture
33Multi-role facilities need to be handled
Maintain Feature Location 24 Bigelow
St.Contact Info.
Geocode
Facility Records Management
Feature Records Management
Maintain Map Label
Maintain Map Label
Maintain Map Label
Police Station
Mass Care Shelter
Fire Station
34First Responders also require extensive, reliable
attributes
- Contact Info
- One structure may have multiple contacts
- Owner
- Superintendent
- Tenants
- Addresses
- Phone numbers
- Context Info
- Where is the building and what is nearby?
- Is the building a
- School
- Police station
- Hospital
- Nursing home/elder care
- Shelter
- Staging area
- High risk terrorist target
- Etc.
- One building can have multiple contexts
35Summary We need more than GIS Software for
Interoperability
- Coordination between stakeholders
- Cooperative agreements for data sharing and
interoperability - Data warehousing approaches and ETL tools
- Emergency response applicationsIn the past, GIS
products often contained custom built databases - A practical and authorative HSIP schema
- Better attribute integration
- National standards with local implementation
36Questions Discussion
- Thank you
- Rich Grady
- grady_at_appgeo.com