Spatial Database Engine

1
Spatial Database Engine

• Keith T. Weber
• GIS Director
• Idaho State University

2
Todays Topics

• What is SDE?
• Why use SDE?
• SDE Data Structure
• How is data stored within SDE?
• DEMO Meet ArcSDE Professional GDB
• Enterprise workflow Versioning and Replication

3
What is SDE?

• A spatial database engine that works on an RDBMS.
• Helps to serve geospatial data to clients via a
  network

SDE
4
Why use SDE?

• Advantages
• Data loss/integrity degradation through
  versioning
• Centralized data management
• Enterprise GIS
• Geo-spatial data is immediately usable

5
Why use SDE? (contd)

• Disadvantages
• Data management role
• RDBMS administration
• Capital expenditure

6
To Use SDEor Not To Use SDE

• What will help make this decision?
• ROI
• TCO
• Is this the correct technology for the problem?

7
ArcGIS Data Structures

• ArcGIS

• GDB

• Vector Objects
• Shape files
• Coverages

• Raster Objects
• Grids
• Images

8
The GDB

• Can store tables (just information), vector
  feature classes, and raster layers

9
Layers and Layer Files

• All GIS Datasets are considered LAYERs in ArcMap.
• A LAYER FILE is a file that you save in ArcMap to
  retain customized settings.
• This file refers to the LAYER (shape file,
  coverage, grid, or feature class)
• It displays the data with your saved
  visualization settings, textual annotation, etc.

10
Workspaces

• Arc/Info
• Info folder
• Geodata sets (coverages, grids, TINs)
• Collection of ArcView shape files
• Geodatabases

11
Coverages

• Tic
• Bnd
• Arc
• AAT, PAT

12
GeoDatabases

• Personal
• File-based
• ArcSDE (Professional or Enterprise)

13
Personal Geodatabases

• Uses the MS Access Jet Database engine or SQL
  Server Express
• Note Do not open/edit these with MS Access
• Limitations
• 2GB (Access)
• 4GB (SQL Server Express)
• Only vector feature classes are actually stored
  inside the Access database
• 4 users but only one editor

14
File-based Geodatabase

• fGDB
• Stores vector and raster layers in the
  file/folder structure.
• Limitations
• Multi-user (max 10)
• 1 Editor (no versioning)
• Max size is 1 TB

15
ArcSDE Geodatabases

• Uses DB2, Oracle, Informix, SQL Server, etc.
• No software size limits and unlimited number of
  users
• Can accommodate vector and raster data

16
Comparison
17
Given all these differences, there are really
many similarities
18
Geospatial Data Storage(Vector)

• Geo-spatial data are stored as Feature classes
• Non-spatial data are stored as stand-alone tables
• Vector data is handled by DB2s Spatial Extender.
  SDE is a broker.

19
Geo-spatial Data Storage(Raster)

• Two methods
• Stand-alone raster data set
• Mosaic
• ArcSDE is not the best solution to store raster
  GIS data for the Enterprise
• Size considerations
• Performance issues
• Raster data is handled by SDE

20
Internal Data Storage

• Within the DB2 RDBMS
• All data is stored within table spaces referred
  to by Configuration Keyword.
• A Configuration Keyword points to a set of two
  table spaces
• Attribute table space
• Coords table space

21
Loading Vector Data into ArcSDE

• PART 1 Stand-alone feature classes

22
The Spatial Index Grid

• Uniform grid of square tiles
• Like grid reference on a street map
• Each feature (lakes) referenced by one or more
  tiles
• Envelope of feature determines tiles occupied
• Spatial Index Key records occurrences of features
  in tiles
• Empty tiles not stored

23
Loading Vector Data into ArcSDE

• PART 2 Feature classes within a Feature Data Set
• First, you need a Feature Data Set
• What is a Feature Data Set?

24
A Feature Data Set is

• Required to implement Full Topology!
• What?!

25
Full Topology

• The spatial relationship among feature classes
  participating in a topology layer
• Must belong to a feature dataset
• Feature classes share geographic reference
  system,
• and spatial domain.
• More realistic representation of data

26
A Feature Data Set then

• Is an organizational tool used to ensure that all
  feature classes within it use a common
• Geospatial reference system
• Spatial domain

27
Understanding the Spatial Domain

• Low-precision GDB Based upon LONG INTEGER
  (32-bit)
• What is the domain range of a LONG?
• High-precision GDB Based upon 64-bit Integer
• Covers a geographic reference systems Horizon

28
Fitting the World into a LONG

• If we express the X,Y coordinates in the familiar
  Latitude/Longitude system
• By whole degrees, we would use
• Latitude -90 -- 90 (180 units)
• Longitude -180 -- 180 (360 units)
• This is only 0.000009 of the 4B space

29
Problems with this approach

• Resolution to 1 degree is terrible
• Wastes the capacity of LONG INTEGER

30
What if we use Decimal Degrees?

• Hold on! Decimals cannot be stored in an INTEGER
  data type
• Lets just shift the decimal place to the right
  by multiplying the coordinate by a scaling factor
  
• e.g., 10 preserves one decimal place, 100
  preserves decimal places etc.

31
Fitting the World into a LONG (revisited)

• By using a scaling factor of 1M, the world would
  fit nicely into a 3.6B space (theres even a bit
  left over!)
• What is the spatial resolution of 1/1Mth of a
  degree?
• Approximately 1/10th of a millimeter!

32
More about the High-Precision GDB

• Can be pGDB, fGDB, or SDE GDB
• Uses 64-bit integer to encapsulate the spatial
  horizon
• What?
• 64-bit numbers have a range of 18,446,744,073,709,
  551,616
• Thats 18 quintillion!
• http//www.jimloy.com/math/billion.htm

33
The Spatial Horizon?

• Essentially, its a spatial domain large enough
  the contain the entire earth at high-precision

34
Applying this to ArcGIS

• Rule 1, use the high-precision GDB model
  whenever possible.
• Why not always?

35
Hints and Tips

• Optimize the spatial domain by using
  high-precision GDB Feature dataset
• If not, set up your low-precision Feature dataset
  to
• Allow for spatial growth
• Allow for improved instrumentation

36
ArcSDE Professional

• Demo
• Import a vector data set into ArcSDE

37
The Future
38
Coffee talk

• You have seen the geospatial support inherent in
  DB2
• You have seen the capabilities of ArcGIS Server
• Do we need an SDE middle-ware?

• No need to get veclempt
• Discuss amongst yourselves

39
Questions
40
Geodatabases in an Enterprise Workflow

• Keith T. Weber, GISP
• GIS Director, ISU
• GIS Training and Research Center

41
Understanding and managing workflow

• Presentation and Discussion

42
Lets Get Started

• GIS is
• Data-driven
• Powerful
• Dynamic

43
GIS Data Life Cycle

• Create Data

• Change Happens!

44
The Bottleneck

• Distributing the new edition

45
The Solution

• Networks and the Internet

46
A New Problem is Born

• MY version

47
GIS Grows Up!

• RDBMS
• Keep the benefit of network connectivity
• Eliminate the problem of MY version
• Eliminate the bottleneck
• And, change the cycle of events

48
GIS Data Life Cycle

• Create Data

• Change Happens!

49
Backup vs. Versioning

• Backups and archiving are still critical steps
  for the enterprise.
• BUT, not part of the GIS Life Cycle any longer

50
In the Beginning

• Backups were made in case we really messed up
• Edits were made to the original
• Copies of the clean new edition were distributed

51
Today

• The original parent is versioned a child is
  born
• Edits are made to the child, not the parent
• Clean edits are copied synchronized or posted
  to the parent.

52
Benefits Of This Approach

• Brainstorm!!!
• Minimize downtime
• Processes completed within the RDBMS

53
The Role of Backups

• Data retention and deletion
• Legal requirements

54
GIS Data Life CycleToday

• Create Data

• Change Happens!

55
Questions/Discussion?
56
Replication and versioning

• Presentation and Discussion

57
(No Transcript)
58
What is Replication?

• Duplication
• Copying
• Mirroring

• Synonyms

59
True Replication

• Does not need ArcGIS
• Every RDBMS can be replicated natively
• However, using ArcGIS to perform the replication
• Is easy
• Supports GIS workflows better

60
Why Replicate?

• Enable disconnected editing for
• Performance/load balancing
• Network load reduction
• Publishing data to subscribers

61
Network Load Reduction

• The network is a primary bottleneck in a
  High-Capacity Enterprise
• Note capacity refers to how many concurrent
  users a system can support without loss of
  performance

62
How Do I Replicate?

• We will cover this with the hands-on exercise
• As an overview
• Version the database
• Replicate the database
• Edit/update
• Synchronize changes with the parent

63
So Replication is Versioning

• No but replication uses a versioned database

64
What is Versioning?

• One database
• Parent edition (tables) remains live/usable
• Child edition(s) simultaneously edited
• Roll-up is seamless

65
Versioning Principal Concepts

• Edits are stored in Supporting Tables
• Geographic changes (linework) are stored in
  Supporting Vector Tables
• Attribute changes are stored in Supporting Delta
  Tables.

66
Delta Tables

• A Add (insert)
• D Delete
• U Update (delete existing then add)

67
A Tree is Formed

• As versions are created and changes are made, a
  tree grows
• Q What kind of tree?
• A A State Tree

68
Sort of an Upside-down Tree
69
The State Tree

• Default state 0

• Arthurs Court sub-division

• Another sub-division

• Branches

70
Multiple Versions

• Multiple versions are allowed
• Versions can be based upon location (north edits,
  south edits), projects (sub-divisions), or other
  logic decided upon my the GIS Manager.
• Batch reconcile and post are supported

71
The Day of Reconciliation

• Arthurs Court sub-division edits have been
  completed
• Time to reconcile
• This process looks for conflicts
• Once all conflicts have been resolved
• Reconciliation is complete

72
Post

• To roll-up the edits back to the trunk of the
  state tree we Post

73
Considerations

• Performance can degrade with active databases
• Workflow itself can generate unnecessary versions
• Delta tables will become large over time
• DBMS statistics may need to be refreshed or
  reviewed by the DB Admin

74
The Cure

• For many of these ArcGIS-centric performance
  issues is compressing the database
• Moves common rows from delta tables into base
  tables
• Reduces depth of the state tree by removing
  states no longer needed

75
Compression Example
76
Questions/Discussion?
77
Hands-On Exercise

• Practice both replication and versioning

78
Your Assignment

• Complete the exercise handouts
• Connecting to and using SDE on DB2
• Practice both replication and versioning
• Read the PDFs in the SDE exercise folder

79
Key Concepts

• SDE is an engine layer residing between a
  spatially-enabled RDBMS and the GIS desktop.
• SDE enables Enterprise GIS
• SDE reduces data management responsibilities.
• Understand Enterprise workflow