LiDAR Data Products in Indiana

1
LiDAR Data Products in Indiana
Data Overview and Access

• ISPLS Workshop
• January 18th, 2013
• Christopher Morse
• NRCS Indiana GIS Coordinator
• Special thanks to
• Jim Sparks
• Phil Worrall
• R. Wilkinson

2
Digital Elevation Data

• A Short Review
• Raster data (evenly spaced, gridded data)
• Cells hold values for the height of a feature or
  site referenced to a common vertical datum
• Resolution refers to the size of the pixels in
  the data
• A DEM with 30 meter resolution is composed of all
  cells being 30 meters x 30 meters in the x and y
  directions and each cell holds a single elevation
  value (z)
• Elevation value (z) could be stored in a variety
  of units

3
Digital Elevation Data

• Definitions paraphrased from Maune et al, 2nd
  edition DEM Users Manual
• (Terms often used interchangeably)
• DEM Digital Elevation Model Typically Bare
  Earth or terrain
• DTM Digital Terrain Model Similar to DEM with
  the addition of some elevations for significant
  topographic features on the land defined by mass
  points or break lines
• DSM Digital Surface Model Similar to a DEM or
  DTM, but shows the tops of all surfaces including
  buildings, trees, and other features above the
  bare earth

Image from State of Indiana Orthophotography and
LiDAR Program, presentation by R.N. Wilkinson
4
Digital Elevation Data

• Definitions paraphrased from Maune et al, 2nd
  edition DEM Users Manual
• Mass Points are irregularly spaced vector points
  with an x,y, and z value.
• Breaklines are linear features that describe
  changes in the terrain surface (roads, streams,
  building footprints, etc)

5
Digital Elevation Data

• Common Resolutions, locally (not all resolutions
  are available in all areas)
• USGS primarily 30 meter, 10 meter some 3 meter

6
Digital Elevation Data

• Common Resolutions, locally (not all resolutions
  are available in all areas)
• Indiana has 2005 elevation data at 5 foot
  resolution for all parts of the state
• Indiana is generating new DTM data for 2011-2013
  at 5 foot resolution
• Local governments may hold a variety of high
  resolution products

7
Digital Elevation Data

• Why generate new DEMs at the same resolution?
• Funds
• Dramatically Improved Vertical Accuracy
• USGS National Elevation Data (2003) 2.44 meters
  RMSE
• Indiana 2005 Data 6 feet RMSE
• Indiana 2011-2013 Data 18.5 cm RMSE
• Vertical Accuracy is a critical factor when
  considering best supportable contour interval

8
Digital Elevation Data

• What does that mean?
• Best supportable auto-generated contours
• USGS National Elevation Data over 20 foot
  interval
• Indiana 2005 Data 20 foot interval
• Indiana 2011-2013 Data 2 foot interval
• Auto generated results limited without use of
  breaklines

Image source wikipedia.org/wiki/Contour_line
9
Digital Elevation Data

• Indianas new DEMs
• Derived from LiDAR
• Actually a DTM due to the inclusion of breaklines
  for some hydro features
• Rivers greater than 100 feet wide and water
  bodies of 2 acres or greater digitized from
  accompanying imagery
• Rivers digitized in direction of flow and water
  bodies with a set elevation (water bodies will be
  flat and rivers will flow downhill)
• No LiDAR points used within 1.5 meters of a
  digitized breakline
• State Plane Coordinates (NAD 83 feet, NAVD 88)

10
Digital Elevation Data

• Indianas new DEMs
• 5 foot horizontal pixel resolution
• Supports 2 foot contours (thus at least 18.5 cm
  vertical RMSE accuracy met)
• Part of the IndianaMap
• http//www.indianamap.org
• DEMs Available for download from
• Indiana Spatial Data Portal http//gis.iu.edu/

11
Digital Elevation Data

• Resources on Digital Elevation Data
• USGS National Elevation Dataset (NED) Info
• http//ned.usgs.gov/about.asp
• Text
• Digital Elevation Model Technologies and
  Applications The DEM Users Manual, 2nd Edition
• Edited by David F. Maune, PhD, CP
• Published by ASPRS, 2007

12
LiDAR Data

• LiDAR Light Detection And Ranging
• Uses an active sensor to emit energy (light) and
  detect returned energy
• Can be collected day
• or night)

Image from State of Indiana Orthophotography and
LiDAR Program, presentation by R.N. Wilkinson
13
LiDAR Data

• Airborne and Terrestrial capabilities
• Combines GPS and an Inertial Measurement device
  to compute x,y,z positions
• Every point recorded has an x,y,z, and intensity
  value

14
LiDAR Data

• All reflections of emitted energy are returned,
  generating a point cloud of the data
• The point cloud contains data points for scan
  hits at multiple heights on objects, as well as
  some noise due to atmospheric conditions.
• These hits are referred to as returns and are
  referenced in ascending order from highest
  elevation to lowest elevation for a set of
  returns
• Top of a building or tree is the 1st return
• Canopy of a tree or side of a building is 2nd or
  3rd return, and so on as the returned hits
  descend in elevation

15
LiDAR Data

• All returns
• 1st return
• 2nd return
• 3rd return
• 4th return

Image from Lidar Technology Overview,
presentation by USGS, June 2007
16
LiDAR Data

• The vendor uses classification algorithms on the
  data
• Vendor delivers a data product depending on the
  customers specifications

17
LiDAR Data

• Indianas LiDAR Data (2011-2013)
• Classified Point Data
• 1.5 meter Nominal Pulse Spacing (the estimated
  average spacing of irregularly-spaced points in
  both the along-track and cross-track directions
  FEMA)
• LAS files in 5000 x 5000 tiles
• Data Delivered in appropriate State Plane
  Coordinate System (NAD 83 Feet, NAVD88)

18
LiDAR Data

• Indianas LiDAR Data (2011-2013)
• Classification Scheme
• (This is not the same as 1st return, 2nd return)
• 1 Processed but Unclassified
• 2 Bare Earth/Ground
• 7 Noise
• 9 Water
• 10 Ignored Ground (breakline proximity)
• 13 Bridges (over 100 feet in length foot
  bridges not included)

19
LiDAR Data

• Some LiDAR Resources
• USGS LiDAR Guidelines (replaces draft versions 13
  and 14)
• http//pubs.usgs.gov/tm/11b4/
• NDEP Guidelines for Digital Elevation Data,
  Version 1.0 (2004)
• (Currently in work for an updated release)
• http//www.ndep.gov/TechSubComm.html
• Education (most class materials available freely)
• https//www.e-education.psu.edu/lidar/resources/l1
  .html

20
LiDAR Data

• LiDARs Limitations (in designs)
• Site changes snapshot on day of flight
• Grade breaks collection pattern is random and
  not based on changes in grade as a field survey
• Critical elevations may not detect control
  elevations such as building floor elevations,
  edges of concrete, property boundaries or culvert
  inlet/outlet elevations (requires local
  benchmarking at site and adjustment of data to
  benchmark)
• Vegetation May affect readings, dependent on
  quality of the data, density of vegetation.
  Tillage may affect surface smoothness (can affect
  slope calculations)
• Water LiDAR can penetrate water, but type of
  laser and water turbidity can affect this.
  Standing water can invalidate a local elevation
  estimate from LiDAR. If you believe a data
  result is due to influence of water, dont use it
  for an elevation

21
LiDAR Data

• LiDARs Uses (in designs)
• Planning Visualization of data and its
  derivaties (hillshades and contours) can be very
  useful in planning
• Preliminary Design LiDAR relative accuracy is
  typically very high for a site, so preliminary
  design for a number of uses can be done with CAD
  generated surfaces and later tied to a sites
  elevations through adjustment to benchmarks
  obtained in a field survey (if the site has not
  undergone major change since the collect)

22
Indiana Data (2011-2013)
23
Acquiring Indiana LiDAR Data

• IndianaMap Indiana Spatial Data
  Portal(www.IndianaMap.org) View and File-based
  access to point cloud and hydro-flattened DEM
  data
• Open Topography Server (UCSD)
• Key advantages
• User Defined Area of Interest
• Mitigates need for local storage of unneeded data
• Opens the door to Indiana data for all users
• Leverage server side processing for extraction
  and derivatives

24
OT Links

• Open Topography Home Page
• http//www.opentopography.org
• Open Topography Data Page
• http//opentopo.sdsc.edu/gridsphere/gridsphere?cid
  datasets
• Indianas LiDAR Data Home Page
• http//igic.org/projects/lidar/index.html
• Recommend you use this home page Indiana news,
  tips, tricks, documentation, and instructions
  will be poster here.

25
Select a Region
26
Results
27
More
28
Get Data (Top)
29
Get Data (Mid)
30
Get Data (Bottom)
31
Choices

• Thats a lot of boxes!
• How do I get what I need?

32
Choices

• Understand what data you really need
• What task are you trying to accomplish?
• What data do you really need for that task?
• Decisions will depend on uses for LiDAR in which
  you are engaged

33
Choices

• Understand what data you really need
• Do you need LiDAR points (LAS files) for
  particular areas?
• Get it here
• Do you need the bare earth points or the entire
  point cloud?
• Decision driven by your intended use
• Do you need a DEM, TIN or derivatives?
• You may not want to get it here (yet) unless you
  need a custom DEM or cannot process TINs or
  generate derivatives in your own applications

34
Ground Return (LAS, DEM, No TIN) Example

• Section 1 - Basics
• Select Area

35
Ground Return (LAS, DEM, No TIN) Example

• Section 1 - Basics
• Review number of points and modify extent if
  necessary
• Set Ground Classification
• This choice drives the nature of the derived
  DEM/TIN later
• Select Coordinate System
• Note that this choice will affect your units in
  later steps

36
Ground Return (LAS, DEM, No TIN) Example

• Section 2 Point Data Format
• Preference and capabilities
• LAS Larger d/l, no decompression needed
• LAZ smaller d/l, must decompress
• ASCII largest d/l, no decompression needed

37
Ground Return (LAS, DEM, No TIN) Example

• Section 3 DEM Generation
• Gridding Parameters
• Remember Units match projection choice from step
  1c.
• Resolution At least the point spacing of the
  dataset (1.5 m or about 5 feet)
• Radius At least the resolution of the dataset.
  The larger the radius, the more smoothed the
  DEM. Using 2x the resolution guarantees a
  3-cell filter

38
Ground Return (LAS, DEM, No TIN) Example

• Section 3 DEM Generation (contd)
• Methods

39
Ground Return (LAS, DEM, No TIN) Example

• Section 3 DEM Generation (contd)
• Zmean Grid A basic averaging method
• Can be used when making any surface to average
  out data irregularities that would be emphasized
  by Zmin or Zmax
• If you chose all points in step 1b, but you
  still want a DEM of the bare earth, you can
  select Zmin instead, but it will have different
  results than Zmean. The differences may or may
  not be significant to your work, but only you can
  determine that.
• Not clear if this implements any nearest neighbor
  weightings, but from the description it seems not.

40
Ground Return (LAS, DEM, No TIN) Example

• Section 3 DEM Generation (contd)
• Zidw Grid
• IDW Inverse Distance Weighting explicitly
  implements an assumption that things that are
  close to one another are more alike than those
  that are farther apart. Is also an averaging
  method.
• Resulting surface will not pass through the
  sample points.
• No option to control the power factor (possibly
  default of 2)
• IDW in general is not recommended for gridding
  Terrains per Maune et al, Digital Elevation
  Models Technologies and Applications The DEM
  Users Manual, 2nd Edition page 10. It is not
  clear if the particular IDW algorithm and
  settings at Open Topography would contradict
  that.

41
Ground Return (LAS, DEM, No TIN) Example

• Section 3 DEM Generation (contd)
• The Zmean to Zidw difference (2ft contours from 5
  ft derived DEM, smoothed with PAEK, 50 ft)

42
Ground Return (LAS, DEM, No TIN) Example

• Section 3 DEM Generation (contd)
• Which method should I use?
• It depends on your needs
• Zmin can give bare earth values when using ground
  returns or entire point cloud, but will always
  assign the lowest value in the search radius
• Zmax can give a first return surface when using
  all returns
• Zmean is a basic average of points to simulate
  bare earth when using ground returns (can be a
  very reasonable DEM) or of entire surface when
  using all returns
• Zidw is like Zmean but more specialized to weight
  points that are nearer may result in some sharp
  exaggerations or newly introduced inaccuracy.
  Typically not the best for gridding terrain data.

43
Ground Return (LAS, DEM, No TIN) Example

• Section 3 DEM Generation (contd)
• Which method should I use?
• The good news
• Its multiple choice! Try them all at once.
• Alternately, you can generate your own DEM from
  the LiDAR points you download using a variety of
  tools of your own and you do not need to generate
  a DEM here at all (however, this example is for a
  DEM, so this is just a reminder that this step is
  optional and specialized).
• You could also download the final project DEMs
  from ISDP, and in the future, OT, without
  processing here at all and you can resample from
  those.

44
Ground Return (LAS, DEM, No TIN) Example

• Section 3 DEM Generation (contd)
• Formats ArcASCII Grid, GeoTIFF, IMG, or All

45
Ground Return (LAS, DEM, No TIN) Example

• Section 3 DEM Generation (contd)
• Null Filling
• Will fill in small blank areas in the DEM being
  generated at 3, 5, or 7 pixel filter sizes
• Your choice depends on your project needs, but
  setting a value here will minimize tiny holes of
  no data in the resulting surface
• Will not fix large holes from water bodies or
  buildings

46
Ground Return (LAS, DEM, No TIN) Example

• Section 4 Derivative Products
• Hillshade will use altitude of 45 degrees and
  azimuth of 315 degrees
• Slope will be degrees, not percent

47
Ground Return (LAS, DEM, No TIN) Example

• Section 5 Visualization Products
• Optional
• Can generate files for use in Google Earth
• Not used in this example (will uncheck)

48
Ground Return (LAS, DEM, No TIN) Example

• Runtime

49
Ground Return (LAS, DEM, No TIN) Example

• Results

50
Ground Return (LAS, DEM, No TIN) Example

• Results
• Data can be added straight to ArcMap, however
  statistics will not be calculated by default
• You can use tools in ArcGIS to calculate
  statistics
• Alternately, you can force the statistics to
  calculate under the Symbology tab by switching
  back and forth from None to Standard Deviations
  under the Stretch method drop down box

51
Ground Return (LAS, DEM, No TIN) Example

• Results (Zmean, elevation, hillshade)

52
Ground Return (LAS, DEM, No TIN) Example

• Results (Zidw, elevation, hillshade)

53
Limitations

• Area of interest scope is limited to 50 million
  points if you are not logged into OT
• You can increase this to 150 million points when
  logged into OT
• Point density and natural geography will cause
  the approximate area corresponding to that number
  of points to fluctuate
• Ground vs. All will return different point count

54
Limitations

• Working with LAS Data
• Typically requires a tool that supports LAS
  format
• Alternately can convert to a 3d x,y,z format
• Raw LiDAR data can have unclassified points and
  gaps in coverage depending on the return or
  classification selected

55
Limitations

• Derivative products can have similar gaps, such
  as the DEM or hillshade generated at OT
• Break lines are not used to generate data from OT
• This is due to the file being generated from ONLY
  the LiDAR points by the OT site
• DEMs Delivered from the vendor have been post
  processed to fill in such areas and to use some
  breaklines (will be added to OT, currently hosted
  at ISDP)

56
Some Uses for OT Generated DEM

• Analysis in open terrain with few to no
  structures or water bodies
• Can compensate for gaps from buildings and water
  bodies in the OT generated DEM using some
  techniques in ArcGIS Spatial Analyst to further
  fill holes in the surface

57
Fill the blanks in the DEM

• Raster calculator (re-run until no NoData
  returns)
• Arc10
• Con(IsNull(Raster), FocalStatistics(Raster,
  NbrCircle(10, "CELL"), "MEAN"), Raster)
• Where Raster is the name of the DEM layer, Circle
  is the search type, 10 is the radius in cells.
  These criteria can be modified.
• Arc 9
• Con(IsNull(Raster), FocalMean(Raster,
  rectangle,10,10), Raster)
• Where Raster is the name of the DEM layer,
  rectangle is the search type, 10x10 is the
  rectangle size (in cells). These criteria can be
  modified.

58
Fill the blanks in the DEM

• Model Builder (Arc 10)
• Same expression, constructed in model builder

59
Fill the blanks in the DEM

• Python (with Arc 10)
• Add a while loop to continue processing as long
  as IsNull generates any True (1) results

60
Recommended Settings

• To get a bare earth dataset at Open Topography
• Return Classification Ground
• Coordinate System User preference
• Point Format User Preference
• DEM Generation (optional)
• Method Zmean grid
• Resolution at least 1.5 m (UTM) or 5 ft (St.
  Plane)
• Radius at least 3x the resolution
• Null Filling 7 (smooth out small gaps)
• Should produce essentially a bare earth DEM, gaps
  for water and buildings will be present

61
Recommended Settings

• Example of a Zmean surface (no exaggeration)

62
Recommended Settings

• To get a 1st return dataset at Open Topography
• Return Classification All
• Coordinate System User preference
• Point Format User Preference
• DEM Generation (optional)
• Method Zmax grid
• Resolution at least 1.5 m (UTM) or 5 ft (St.
  Plane)
• Radius same as resolution (minimize radius to
  decrease blending of vertical features)
• Null Filling 7 (smooth out small gaps)
• Should produce a surface approximating a DSM, but
  will lack true vertical definition of features,
  gaps for water may be present, may be affected by
  noise in the data

63
Recommended Settings

• Example of a Zmax surface

64
Questions?

• Chris Morse
• USDA-NRCS
• NRCS Indiana GIS Coordinator
• 317-295-5849
• chris.morse_at_in.usda.gov