EUMeTrain LSA SAF Week

1
EUMeTrain LSA SAF Week Ricardo Silva, IM
Session 2 Messing around with the data
2
Who am I?
Ricardo Silva
3
Where do you come from?
4
How often do you use LSA SAF products?
Occasionally
Regularly
Never used, but plan to start
5
Session 2 Summary

• Obtaining LSA SAF products
• General Product Information
• Documentation
• File structure
• Quick Visualization using HDFView
• GIS tools for visualization and analysis
• Quantum GIS
• GRASS GIS
• ILWIS
• Tool for georeferencing LSA SAF products
• Using programming tools Python

6
Obtaining LSA SAF products

• Near Real Time Dissemination via EUMETCast
• Dissemination on-demand
• httplandsaf.meteo.pt
• Products are available for registered users
• Registration is free of charge, but may take up
  to 3 working days for approval
• Downloads use the FTP protocol
• Several download methods according to desired
  temporal extent and download settings

7
How do you obtain LSA SAF products?
LSA SAF (website, FTP, helpdesk)
EUMETCast
Other?
8
(No Transcript)
9
Obtaining LSA SAF products

• Dissemination options
• Ordering methods
• Random slot ordering
• The user can select individual time slots for
  several products
• Automatic dissemination
• Long series ordering
• Ordering of long time series of products.
• Requires human intervention and may take up to
  one week
• Dissemination methods
• LSA SAF FTP server
• The order is placed in the LandSAF server for the
  user to download
• User's personal FTP server
• LSA SAF's system transfers the order directly to
  the user's FTP server

10
Product Documentation

• Product User Manual (PUM)
• Main documentation source
• Physical phenomena
• Algorithm basis
• Output data specifications
• Etc.
• Product Output Format (POF)
• Details concerning description of metadata, file
  naming formats, etc.
• Validation Report (VR)
• Algorithm Theoretical Basis Document (ATBD)

11
Product file structure

• LSA SAF products use the HDF5 format
• http//www.hdfgroup.org/
• Hierarchical Data Format (HDF)
• Groups Containers for datasets and other groups
• Datasets Multidimensional arrays of a
  homogeneous type
• Attributes Metadata on groups and datasets
• HDF files
• Are self-describing. Complex data relationships
  and dependencies
• Can contain binary data
• Support n-dimensional datasets
• Each element in the dataset may itself be a
  complex object

12
Product file structure

• LSA SAF HDF5 files
• A common set of attributes on the root group
• general information
• A dataset for the main product (LST, DSLF, etc)
• Additional datasets for cell metadata (Quality
  flags, etc.)
• Each dataset may have
• Specific attributes (Scaling factor, missing
  value, etc.)
• Different dimensions (Most datasets are 2D)
• Different data types (Most datasets are 16bit
  signed integers)

13
Quick visualization of products

• HDFView software
• http//www.hdfgroup.org/hdf-java-htm
  l/hdfview/index.htmldownload_hdfview
• Java-based HDF Viewer
• Inspection of a HDF5's structure and data
• Visualize data in spreadsheet or image mode
• Basic statistics, line plotting and histogram
  generation
• Export to image formats

14
Have you used HDFView before?
No
Yes
HDFWhat?
15
HDFView

1. Datasets
2. Attributes
3. Table View
4. Table options
5. Image view
6. Image options
7. Pixel values

16
HDFView
Inspecting file and dataset attributes
17
HDFView

• When visualizing products take note
• Cell values are unscaled
• The 'missing value' values have no physical
  meaning
• Product is not georeferenced (line, column
  coordinates only)

18
GIS tools for visualization and analysis

• Quantum GIS
• User friendly interface
• Many plugins created by the community
• GRASS GIS
• Many raster analysis modules
• Python scripting
• ILWIS
• Direct import of LSASAF products through the
  GEONETCast toolbox
• Other tools (ArcGIS, Mapinfo, SAGA, etc.)

19
Quantum GIS

• http//www.qgis.org/
• Open source multi-platform GIS software
• User-friendly interface
• Reads many input formats
• Raster files, Vector files, Web services, etc
• On-the-fly reprojection for vector and raster
  data
• Large and friendly community of users and
  developers

20
Quantum GIS
Integration of different data sources
21
Quantum GIS
Quick inspection of relationships between data
22
GRASS GIS

• Geographic Resources Analysis Support System
• http//grass.fbk.eu
• Open source multi-platform GIS software
• Hundreds of modules for raster and vector
  processing
• Reads many input formats
• Can read HDF5 directly (won't be georeferenced)
• It is preferable to georeference the HDF5 file
  first

23
GRASS GIS
Modular and flexible user interface
24
GRASS GIS

• Raster calculator
• Allows for computing a combination of logical and
  algebraic expressions
• Example Calculate the arithmetic mean LST values
  for a specific day (using hourly sample)
• 24 raster files each with the LSA SAF LST product
  for the hourly timeslots

Approach 1 Using only cells where there is
enough data for the whole day (In GRASS, any math
performed with NULL cells always results in a
NULL value for these cells
Approach 2 Using all the cells where there is
at least one value in the day
25
GRASS GIS
Approach 1
26
GRASS GIS
Approach 2
27
GRASS GIS

• Most commands are available in two interfaces
• Graphical Good for visualization and
  exploratory analysis
• Command-line Nice for automating repetitive
  tasks

28
Are you used to creating small scripts for
automating tasks?
Not really
Yes
I might start using them In the future...
29
GRASS GIS

• Python scripting example
• Automatically import and rename 191 LST files for
  a given day

GRASS 6.4.1 (world4326) gt python gtgtgt example
importing all the LST geotiffs into GRASS gtgtgt
import grass.script as grass gtgtgt from glob import
glob gtgtgt import os gtgtgt import datetime as
dt gtgtgt gtgtgt for path in glob('/home/geo2/test_data/
lsasaf/modificados/georefs/LST.tif') ...
fName os.path.basename(path) ...
nameFieldsfName.split('_') ... product, area
nameFields35 ... timeslot
dt.datetime.strptime(nameFields5,
'YmdHM') ... newName '_'.join((product,
area, timeslot.strftime('Y_m_d_H_M'))) ...
grass.run_command('r.in.gdal', flags'oe',
overwriteTrue, inputpath, outputnewName)
30
ILWIS

• Integrated Land and Water Information System
• http//52north.org/communities/ilwis/
• Open Source GIS software
• Available only for MS Windows (as of now)
• Imports LSA SAF products, using the GEONETCast
  Toolbox
• Several modules for raster and vector processing

31
GEONETCast Toolbox

• GEONETCast Toolbox
• http//52north.org/communities/earth-observation/g
  eonetcast-toolbox-software
• Available as a toolbox for ILWIS
• Developed by ITC University of Twente
• Enables easy access to and management of
  GEONETCast data
• various satellite and environmental data and/or
  resulting products
• Tailored for users running GEONETCast reception
  stations

32
ILWIS/GEONETCast

• Installation
• Follow the GEONETCast Toolbox's user manual
• Read section 1.1 and 1.2 CAREFULLY
• Importing LSA SAF products
• Beware of file names
• Files obtained through GEONETCast (EUMETCast)
• S-LSA_-HDF5_LSASAF_MSG_LST_Euro_201111060000.bz2
• Everything will work OK
• Files obtained through LSA SAF's on-demand
  dissemination
• HDF5_LSASAF_MSG_LST_Euro_201111060000.bz2
• GEONETCast Toolbox expects the 'S-LSA_-' prefix
  and will throw errors
• Solutions
• Uncompress ? rename ? recompress (time
  consuming and boring)
• Modify the GEONETCast's batch import scripts
• Contact LSA SAF's helpdesk and they'll send you
  the modified scripts

33
ILWIS/GEONETCast
Modular user interface, integrating the
GEONETCast Toolbox
34
ILWIS/GEONETCast
Use native NGP projection of the LSA SAF products
35
Have you used these GIS tools before?
No
Yes, some of them
I might start using them In the future...
36
LSA SAF Product Coordinate System

• Normalized Geostationary Projection (NGP)
• Projection of the underlying MSG LRIT source data
• Important parameters for MSG satellite
• Satellite height above ellipsoid 35785831 m
• Central meridian 0º
• Documented on EUMETSAT's LRIT and HRIT Global
  Specification

• http//www.eumetsat.int/groups/cps/documents/docum
  ent/pdf_cgms_03.pdf

37
LSA SAF Product Coordinate System

• LSA SAF products hold coordinate system
  information in the attributes of the root group
• But HDF5 files have no standard way of defining
  coordinate systems
• It's hard to automatically import LSA SAF
  products into a Geographical Information System
  (GIS) software AND have the data correctly
  georeferenced

38
Visualization of LSA SAF products
GIS tools are nice for visualization,
analysis map making, etc.
HDFView is nice for a quick inspection
Most GIS tools cannot recognize LSA
SAF coordinate information
Some GIS tools can open HDF5 directly
BUT ...
The files will not be projected
It won't be possible to overlay different data
UNLESS
39
LSASAF Georeferencer Tool

• A graphical tool for georeferencing LSA SAF
  products
• Still in-development, but already usable
• Open development
• Source code hosted on a public repository
• https//github.com/ricardogsilva/LSASAF-georeferen
  cer
• Uses open source tools
• Python, GDAL, Proj, Qt4
• Multi-platform

40
LSASAF Georeferencer Tool

• Convert LSASAF HDF5 files to GeoTiff
• GeoTiff is a Tiff based interchange format for
  georeferenced raster imagery
• Reproject from the NGP projection to other
  coordinate systems
• Apply scaling factor and missing value
  information to the output file
• Output GeoTiff files are suitable for further
  processing in GIS software
• Process multiple files simultaneously
• Also has a CLI interface, useful for scripting

41
LSASAF Georeferencer Tool

• Graphical Window usage example
• Select the file(s) to process
• Load file information
• Choose the dataset(s) to be processed
• Choose the output projection
• Choose the output directory
• Process the files

42
LSASAF Georeferencer Tool

• Command line usage example

display usage information ricardo_at_geo2
./georef_hdf5.py -h process all the LST files
in /input_dir ricardo_at_geo2 ./georef_hdf5.py
-x -o /my_output_dir /input_dir/LST
43
Guess who is developing the LSA SAF Georeferencer
Tool?
Some other dude...
Ricardo Silva (me) )

• Contributions, bug reports, etc. are very much
  welcome!

44
Programming tools Python

• Python(x, y)
• http//code.google.com/p/pythonxy/
• A (free) scientific and engineering development
  software for
• numerical computations,
• data analysis
• data visualization
• Bundles Python and a big set of extra modules in
  a single package
• Includes modules for manipulating HDF5 files

45
Programming tools Python

• Some useful python modules for working with LSA
  SAF products
• NumPy
• The fundamental package needed for scientific
  computing with Python
• Provides an N-dimensional array object and
  specialized data types and functions
• Scipy
• A library built to work with NumPy arrays
• Provides efficient numerical routines for
  interpolation, statistics, linear algebra, etc.
• Matplotlib
• 2D plotting library which produces publication
  quality figures
• PyTables
• Package for managing HDF5 files
• Designed to efficiently and easily cope with
  extremely large amounts of data
• Gdal
• A translator library for raster geospatial data
  formats.
• Has read/write functionality for many raster
  formats (HDF4/5, GeoTiff, GRIB, netCDF, etc.)
• Allows assigning georeferencing information

46
Programming tools Python

• Example interactive session for manipulating a
  LST product

ricardo_at_geo2 python gtgtgt import tables gtgtgt
import numpy as np gtgtgt import matplotlib.pyplot
as plt gtgtgt product tables.openFile('path_to_HDF
5_file') list attributes of the root group gtgtgt
product.root._v_attrs list attributes of the
LST dataset gtgtgt product.root.LST.attrs import
the LST dataset into a 32bit float numpy
array gtgtgt lst np.array(product.root.LST,
dtypenp.float32) convert missing values to nan
and apply scaling factor gtgtgt lstlst
product.root.LST.attrs'MISSING_VALUE'
np.nan gtgtgt lst lst / product.root.LST.attrs'SCA
LING_FACTOR' plot the data gtgtgt
plt.imshow(lst, interpolation'nearest') gtgtgt
plt.colorbar() gtgtgt plt.show()
47
Programming tools Python

• The script's output

48
END of session 2
Thank you for attending!
Enjoy the rest of the LSA SAF week