Interactive Verification Program Training for AWIPS OB8.2

1
Interactive Verification ProgramTraining for
AWIPS OB8.2

• Hank Herr
• Office of Hydrologic Development

RFC Verification Workshop, 08/15/2007
2
Overview

• Introduction
• IVP Terminology
• Software Components
• Database Usage
• Output Files
• Design
• Demonstration
• Tips
• Future Possibilities
• Exercises
• Discussion

3
IntroductionHistory of IVP

• Original IVP written around 2000-2001 for old HP
  based verify software
• Verification moved to archive database for OB4
• IVP Batch Program software written
• Old IVP GUI still ran, but used pairs files
  created during batch processing it did not
  access the archive database directly
• New IVP GUI written for OB7.2
• Provided ability to build verification products
  via GUI, modify components of a chart and save
  modifications, create batch file to reconstruct
  graphics via IVP Batch Program
• Accessed archive database directly
• Limitations
• Designed for stage data
• Accessed only three data tables of the archive
  database
• Limited number of statistics
• No quality control of pairs
• Inflexibility in building batch files from GUI
• JClass Desktop Views used to render charts
  (limited charting capabilities upgrade cost too
  much money)

4
IntroductionNew for OB8.2

• Designed for verification of stage, flow,
  temperature, and precipitation forecasts
• Can do other data types as long as data is stored
  within accessible tables
• Accesses 10 data tables of the archive database
• Uses all data tables except pempsep, pemrsep, and
  pairedvalues
• Additional statistics
• Moments, CDF Plot, ROC, etc.
• Quality control of pairs via GUI
• View pairs in a time series plot, in addition to
  existing scatter plot
• Mark pairs as bad so that verification ignores
  them
• Change forecast or observed values within a pair
  and save changes to database
• More flexible batch file construction
• Batch file can be built that will construct
  current chart for other combinations of locations
  and analysis periods
• Chart Director used for rendering
• Relatively quick, memory efficient chart
  rendering
• Used previously for HydroGen, so a run-time
  license already exists

5
IVP Terminology

• Verification Location
• A forecast point or location and a data type for
  which verification is to be performed
• Uniquely specified by location id (lid within
  database) and SHEF pedtsep code.
• Includes following information
• Forecast and observed categories
• Observation type (raw or processed)
• Active flag (yes or no)
• National flag (yes or no)
• Sensor preferences (ordered list of observed type
  source, representing preferences for which
  observed values to pair with a forecast)

6
IVP Terminology

• Verification Group
• A set of locations and parameters that define the
  data to load and other parameters of verification
• Locations must have the same number of categories
  and must be of the same type stage (height),
  flow (discharge), temperature, precipitation
• Includes following parameters
• Analysis period and intervals
• Lead time period and intervals
• Issuance time-of-day (0-24 hours) period and
  intervals
• Restrictions on location response time, active
  flag, and SHEF components (forecast type source,
  physical element, duration, and extremum)
• Whether or not to compute statistics
  independently for each lid or forecast type source

7
IVP Terminology

• Forecast-Observed Data Pair
• A forecast value paired with an observed value
• Observed value can be raw or processed
  depending on if the value was posted to the
  database by the raw or processed SHEF decoder
• Each forecast can have one raw pair and one
  processed pair associated with it
• Best candidate observation is chosen for each
  forecast based on following criteria
• SHEF qualifier code (G, M, P, V, S, Z, T, F, and
  Q, in order)
• sensor preference (observation type source) rank
• difference between forecast validtime and
  observation obstime
• A pair can be good or bad
• Bad pairs are not used in verification
  computations

8
Software Components

• Vfyruninfo Editor GUI
• Manage vfyruninfo table
• IVP
• Data quality control and define verification
  products graphically
• IVP Batch Program
• Build pairs and verification products via a batch
  language
• IVP Batch Builder
• Edit IVP Batch Program batch files graphically
• Verify Pairs Ingestor
• Ingest pairs directly from text files of specific
  format into the vfypairs or vfyprocpairs tables
• del_bad_pairs
• Removes bad pairs from the vfypairs and
  vfyprocpairs table

9
Database Usage

• Verification Location Information
• Vfyruninfo table
• Defines locations for which verification can be
  performed.
• Individual entries for each verification location
  and sensor preference component (observed type
  source)
• Edited via Vfyruninfo Editor GUI
• Location table
• Information used in building labels, includes RFC
  name, full location name, etc.
• Rivercrit table
• Flood stages and flows (if location is of
  appropriate data type).

10
Database Usage

• Building Pairs
• Tables of data for pairing
• Forecasts pedfsep, pehfsep, peqfsep, pedcsep
• Raw Observations pecrsep, pedrsep, peoosep
• Processed Observations pedpsep, pehpsep,
  peqpsep, peoosep
• Vaddadjust table
• Adjust observations prior to pairing with a
  forecast
• Only affects forecast-observed pairs
• Is not dated there can be only one entry per
  location
• Sensok table
• Allows for specifying time ranges during which
  sensors should not be used (i.e. observations
  should not be used for pairing)

11
Database Usage

• Storing Pairs
• Vfypairs table
• Stores pairs constructed using raw observations
• Field quality_code defines if the pair is good or
  bad
• Can be very large (millions of records)
• Vfyprocpairs table
• Stores pairs constructed using processed
  observations
• Field quality_code defines if the pair is good or
  bad
• Can be very large, but will likely be a fraction
  of the size of vfypairs
• A forecast can have up to two pairs built for it
  one stored in vfypairs and one stored in
  vfyprocpairs
• Only one of the two pairs can be analyzed at one
  time

12
Output Files

• Output Statistics File (batch only)
• Verification statistics output in an ASCII
  tabular format
• Pairs File
• Pairs used to compute statistics
• ASCII tabular format
• Output Image File (.png, .jpg)
• PNG or JPG files displaying a plot of
  verification statistics
• Image Data File (.dat)
• Statistic values comprising an output image plot
• ASCII tabular format

13
DesignFoundation
User

• Archive database provides the following
• Forecasts and observations
• Verification location information
• Forecast-observed data pairs

14
DesignVfyruninfo Editor
User

• Vfyruninfo Editor is used to specify verification
  locations and saves them to the vfyruninfo table
  of the archive database

Vfyruninfo Editor
Vfyruninfo rows
15
Design IVP Batch Program
User

• IVP Batch Program provides the user the ability
  to
• Pair forecasts with observed values according to
  user specified parameters

Batch file
Vfyruninfo Editor
Batch
parameters
Pairing Processor
forecasts, observations, location info
pairs
Vfyruninfo rows
16
DesignIVP Batch Program
User

• IVP Batch Program provides the user the ability
  to
• Calculate statistics and output results to
  tabular ASCII files

Batch file
Vfyruninfo Editor
Batch
parameters
ASCII
Verification Processor
Pairing Processor
pairs, statistics
forecasts, observations, location info
pairs
pairs, location info
Vfyruninfo rows
17
DesignIVP Batch Program
User

• IVP Batch Program provides the user the ability
  to
• Generate user customizable graphical verification
  products displaying statistics

Graphics
Batch file
Vfyruninfo Editor
Batch
Graphics Processor
parameters
ASCII
params
statistics
Verification Processor
Pairing Processor
pairs, statistics
forecasts, observations, location info
pairs
pairs, location info
Vfyruninfo rows
18
DesignIVP GUI
User

• IVP provides the user the ability to
• Specify, build, and view graphics

Graphics
Batch file
GUI
Vfyruninfo Editor
params
Batch
Graphics Processor
parameters
ASCII
params
statistics
Verification Processor
Pairing Processor
pairs, statistics
forecasts, observations, location info
pairs
pairs, location info
Vfyruninfo rows
19
DesignIVP GUI
User

• IVP provides the user the ability to
• Modify forecast-observed data pairs

Graphics
Batch file
GUI
Vfyruninfo Editor
pairs
params
Batch
Graphics Processor
parameters
ASCII
params
statistics
Verification Processor
Pairing Processor
pairs, statistics
forecasts, observations, location info
pairs
pairs, location info
Vfyruninfo rows
20
DesignIVP GUI
User

• IVP provides the user the ability to
• Create batch files to produce graphics via IVP
  Batch Program

Graphics
batch tokens
Batch file
GUI
Vfyruninfo Editor
pairs
params
Batch
Graphics Processor
parameters
ASCII
params
statistics
Verification Processor
Pairing Processor
pairs, statistics
forecasts, observations, location info
pairs
pairs, location info
Vfyruninfo rows
21
DesignIVP Window Layout
2. View and QC Pairs
1. Load Pairs
4. View Products
3. Define Products
22
Demonstration

• Feature availability
• Both OB7.2 and OB8.2
• OB8.2 only

23
DemonstrationVfyruninfo Editor

• Start Vfyruninfo Editor
• Filter ingestfilter table contents
• Select locations from ingestfilter table
• Create locations from scratch
• Specify location parameters
• Active
• National
• Response time
• Sensor preferences
• Save to archive database

ref Vfyruninfo Editor Users Manual
24
DemonstrationPairing Batch File

• Commands
• Time window START_TIME, END_TIME
• Specify locations LOCATION, DUR, EXTREMUM,
  FCST_TS, PE
• Pairing window PAIRING_WINDOW
• Observation type OBS_TYPE (replaces OBS_TABLE)
• Persistence flag PERSISTENCE
• Action
• Build the pairs BUILD_PAIRS

ref IVP Batch Program Users Manual for Pairing
25
DemonstrationPairing Batch File

• Example (buildpairs_template.bat)

Pair all locations. Use default settings of
ALL for all other location specifying
commands location ALL Build pairs
for a two week window prior to current system
time start_time " - 14 days" end_time
"" Parameters pairing_window
1 obs_type RAW build_pairs true
ref IVP Batch Program Users Manual for Pairing
26
DemonstrationNational Statistics Batch File

• Commands
• Time Window START_TIME, END_TIME
• Specify Locations DUR, EXTREMUM, FCST_TS, PE
• Action
• Compute statistics NATLSTATS
• Recommendation
• Set the national flag for all locations that are
  part of the national program to Y (use
  Vfyruninfo Editor)
• Give the NATLSTATS action a value of NATIONAL

ref IVP Batch Program Users Manual for
Verification
27
DemonstrationNational Statistics Batch File

• Example (natlstats_template.bat)

Compute stats for September, 2003. START_TIME
2003-09-01 000000 END_TIME 2003-09-30
235959 Default Values are ALL FCST_TS
ALL PE ALL DUR ALL
EXTREMUM ALL NATLSTATS NATIONAL
ref IVP Batch Program Users Manual for
Verification
28
DemonstrationIVP Verification Group Manager

• Start IVP
• go vsys_scripts
• ivp
• Specify parameters of a verification group
• Analysis period and intervals
• Lead time period and intervals
• Issuance time period and intervals
• Miscellaneous parameters

ref IVP Users Manual, Section 9
29
DemonstrationIVP Verification Location Manager

• Specify verification locations that comprise a
  group
• Filter available locations
• Choose locations
• Specify categories
• Specify observation type (from which pairs table
  to acquire data)

ref IVP Users Manual, Section 13
30
DemonstrationIVP IVP Data Display

• Create IVP Data Display
• Loading data progress window
• Open the Verification Group Manager and
  Verification Location Manager
• Save the image

ref IVP Users Manual, Section 15
31
DemonstrationIVP IVP Data Display

• Examine data through two views
• Scatter Plot View
• Select a location
• In OB7.2, use Show button in Verification
  Location Manager
• Display info about a location
• Line x y
• Time Series View
• Specify time series to view
• Emphasize a time series

ref IVP Users Manual, Sections 15.1, 15.2
32
DemonstrationIVP IVP Data Display

• Draw a selected region rectangle
• Zoom and the navigator panel
• Zoom in on scatter view
• Zoom in on time series view
• Panning through the time series by zooming
• Shift the zoomed region via the navigator panel
  and keyboard
• Zoom out

ref IVP Users Manual, Sections 15.6, 15.7
33
DemonstrationIVP IVP Data Display

• View Pairs via the Verify Pairs Viewer
• Select pairs
• Sort table
• Create a pairs file
• WARNING Do not view very large (30,000) sets of
  pairs the table rendering may be too slow

ref IVP Users Manual, Section 16
34
DemonstrationIVP IVP Data Display

• Mark pairs as good or bad via IVP Data Display
• Edit pairs via the Verify Pairs Editor
• Select pairs
• Mark pairs as good or bad
• Edit forecast or observed value manually
• Focus on a pair
• Edit forecast or observed value graphically
• Scatter view
• Time series view
• Change the mouse mode
• WARNING Do not select large sets (1000) of
  pairs for editing the table rendering speed may
  be too slow

ref IVP Users Manual, Sections 15.11, 17
35
DemonstrationIVP Verification Plot Definition
Manager

• Open the Plot Definition Manager
• Select statistics
• IVP Statistic Chooser Manager
• Select plot type
• Select x-axis variable
• Select comparison variable
• Edit statistics parameters
• Edit group parameters
• Spawns a window nearly identical to Verification
  Group Manager
• Note how variable selection affects analysis,
  lead time, and issuance time-of-day intervals
• Note how variable selection affects category
  choice boxes

ref IVP Users Manual, Sections 18, 19
36
DemonstrationIVP The Statistics

• Errors
• Categorical Statistics
• Quantiles and Extremes
• Computed relative to whichever variable whose
  category used is set to Do Not Use
• Moments
• Sample Size
• The number of pairs used in the computations
• or
• Number of pairs within the category (if x-axis or
  comparison variable is forecast or observed
  categories)
• Misc.
• Special Plots

ref IVP Batch Builder Users Manual, Appendix B
37
DemonstrationIVP The Statistics

• Categorical Statistics use 3x3 Contingency Table
• Examples
• HFAR D/(D E F)
• TFAR (D F)/(D E F)
• CSI (ABDE)/(ABDEGHCF)
• In a 2x2 contingency table
• HFAR is always 0 for the lowermost category
• TFAR and HFAR are equivalent for uppermost
  category
• POD UFR OFR 1 in all cases

ref IVP Batch Builder Users Manual, Appendix B
38
DemonstrationIVP Verification Plot Definition
Manager
Statistic short name (matches output files)
primary y-axis
comparison variable
secondary y-axis
x-axis
39
DemonstrationIVP Verification Plot Definition
Manager

• Choose categories used
• Disabled if x-axis or comparison variable is
  forecast or observed categories
• Select Do Not Use
• Only one of the two choice boxes can be Do Not
  Use
• Impact of selections on how statistics are
  computed
• Category used restricts pairs used in
  computations to only those within the category
• Some statistics are calculated relative to the
  variable not used to categorize data (e.g.
  quantiles, cdf plot, pdf plot)
• Special case ROC plot
• Category used defines observation threshold

ref IVP Users Manual, Section 18.7
40
DemonstrationIVP IVP Statistic Display

• Create IVP Statistic Display
• Data is always loaded from memory even if the
  overridden group parameters imply additional data
  is needed (such as a wider analysis interval, for
  example)
• Allows for relatively quick plot generation
• Save the image
• View the data via the IVP Statistics Data Viewer
• Create an ASCII format image data file

ref IVP Users Manual, Section 21
41
DemonstrationIVP IVP Statistics Display

• Create a batch file
• Change the analysis period
• Change the included locations
• Create other groups for which to generate
  products
• Specify output and template file names
• Replacement strings are needed if images are to
  be produced for multiple groups in order to avoid
  overwriting output files
• Specify batch file name

ref IVP Users Manual, Section 23
42
DemonstrationIVP Chart Property Manager

• Open up the Chart Property Manager
• Click on menu item or tool bar item
• Click on plot component
• Change labels
• Change axis limits or chart dimensions
• Change legend components
• Change x-axis labels (for IVP Statistic Display)
• Create a template file
• Dont modify manually unless absolutely necessary
• Apply changes from a template file

ref IVP Users Manual, Sections 24, Appendix D
43
DemonstrationIVP Chart Property Manager
Legend Title
Plot Title
Y-Axis 1 Title
Legend
Y-Axis 2 Title
Tick Mark Font
X-Axis Title
44
DemonstrationIVP Batch Builder and the IVP Batch
Program

• Start IVP Batch Builder
• Load batch file
• Insert, append, delete command lines
• Edit command lines
• Comment and uncomment command lines
• Check for errors
• Run batch file

ref IVP Batch Builder Users Manual
45
DemonstrationIVP Batch Builder and the IVP Batch
Program

• Create one batch file to generate products for
  many locations
• _at_FILE Command to refer to other files
• Replacement strings to avoid overwriting output
  files

ref IVP Batch Builder Users Manual
46
DemonstrationIVP Batch Builder and the IVP Batch
Program

• Organize output by using subdirectories of
  (vsys_output)
• If producing graphics for each location, use
  _at_ONELID or _at_ONELIDSHEF to define directories
• OUTPUT_FILE _at_ONELID/output.txt,c
• If producing graphics for time periods, use
  _at_STARTTIME or _at_ENDTIME
• OUTPUT_FILE _at_STARTTIMECCYY/output.txt,c

ref IVP Batch Builder Users Manual
47
TipsVerification Group Manager Usage

• To minimize time spent querying database, load
  all needed pairs at once
• Only specify the overall analysis period and
  choose the locations within the Verification
  Group Manager and Verification Location Manager
• Specify other parameters via the Plot Definition
  version of the Verification Group Manager (click
  on Edit Group Parameters in the Plot Definition
  Manager)
• Important when generating many products
• To minimize the chart rendering time for the IVP
  Data Display, keep the number of pairs small
• Do one location at a time for a couple years
• Do several locations at a time for one month
• Important when quality controlling pairs data
• See the Memory Limitations Tips later for
  recommended numbers of pairs to load

48
TipsDeciding Between PROCESSED or RAW

• Use OBS_TYPE RAW for paired observations if
• Observed data was posted to archive database via
  raw SHEF decoder, AND
• First letter of type source is R.
• Use OBS_TYPE PROCESSED if
• Observed data was posted to archive database via
  processed SHEF decoder, OR
• First letter of type source is not R
• Example
• If processed observations were posted via raw
  SHEF decoder (to peoosep table), then specify
  PROCESSED

49
TipsSystem Settings Files

• Specifies parameters of graphical user interfaces
• Fonts
• Window Sizes
• Colors
• Others
• Location
• Default
• (get_apps_defaults vsys_dir)/app-defaults/_SYSTE
  M_FILE.txt
• Directory can be overridden by exporting
  environment variable SYSTEM_SETTINGS_FILE_DIR
• Can allow for user specific system settings
• Format
• ESPADP/IVP batch language standard lttokengt
  ltvaluegt format

ref IVP Users Manual, IVP Batch Builder Users
Manual, Vfyruninfo Editor Users Manual, Appendix
A
50
Tips Memory Limitations

• Recommended number of pairs
• Limits can be changed by using the ivp and
  ivpbatch script m and s options (run ivp hh
  or ivpbatch hh to read about these options)
• Limits above are for IVP GUI IVP Batch Program
  can deal with more pairs because not all pairs
  need to be stored in memory at one time
• If IVP GUI runs out of memory, an error message
  will pop-up and the pairs will be discarded (in
  most cases)
• The software may crash in other cases
• Verify Pairs Viewer window is a memory hog
• Add about 25 to the recommended value if not
  using the Viewer

ref IVP Users Manual, IVP Batch Builder Users
Manual, Vfyruninfo Editor Users Manual, Appendix
A
51
TipsDebugging and Performance

• Debug problems
• Export vsys_debug token prior to execution to see
  additional output
• Useful levels are 0 (no output), 1, 20, 40, 60,
  80, 100, 120, 128.
• export vsys_debug20 to see additional progress
  output
• export vsys_debug40 to see database query
  where clauses and timing results for chart
  rendering
• export vsys_debug128 to see everything (lots
  and lots of output)
• Performance
• If IVP Data Display chart rendering is slow
• Reduce number of pairs viewed
• Zoom in on regions (points outside current zoom
  area are not rendered)
• Run on faster machines (port code to LX and
  execute software there)
• If database query speed is slow
• Turn off SHEF decoders for a short time (if you
  can)
• Reduce number of locations or width of analysis
  period

ref IVP Users Manual, Section 25
52
Future Possibilities

• Confidence Intervals
• Algorithms need to be developed
• Ability to produce CI values and plot them needs
  to be added
• GUI redesign
• Reduce the number of windows
• Reorganize window so that the Verification Group
  Manager does not need to be in two places
• Add ability to handle multiple comparison
  variables
• Add ability to modify other properties of a chart
• Tick spacing, line style (solid, dashed, etc),
  line width, etc.
• Requires additions to Chart Property Manager
• Create baseline batch files requiring no changes
  in order to produce standard graphics
• Standard graphics need to be determined
• Add ability to put changed forecasts and observed
  values back into original data tables and rebuild
  all affected pairs on-the-fly
• IVP would become a more powerful, general QC tool
  for archive data

53
Documentation

• Available on the web

http//www.nws.noaa.gov/oh/hrl/verification/verifi
cation_doc_ob82.php
54
Exercises

• Described in hand-outs
• The data sets are very large, so while the pairs
  are loading, discuss the plots that are needed in
  order to answer the exercise questions
• What statistics are to be displayed?
• What are the x-axis and comparison variables?
• Are there any observed or forecast category
  restrictions?
• Group assignments

55
Discussion

• Review exercises
• Pre-workshop questions
• Have they been answered?
• General comments on software