Setting up a real-time system

1
Setting up a real-time system

• Kevin.Manross_at_noaa.gov
• National Severe Storms Laboratory
• Norman OK, USA
• http//www.wdssii.org/ Platforms, Real-time
  Monitoring

2
A quick word about hardware

• Running WDSSII is an intensive process
• General hardware/software specs
• (from http//cimms.ou.edu/lakshman/WDSSII/softwar
  e/platforms.html)

Item Minimum Spec for research version Regional-scale (1000 km x 1000 km) algorithm machines CONUS-scale algorithm machines Display -only machine for real-time applications
Processor One Pentium IV or better dual 32-bit Xeon or Athlon dual 64-bit Opteron or Xeon single Xeon or better
Random Access Memory (RAM) 512MB 2GB per processor (ideally 4 GB per processor) 8 GB per processor (ideally 16 GB per processor) 2 GB
Storage (I/O) 1GB per data case RAID 0 (dedicated RAID) -- 4 GB RAID 0 (dedicated RAID for each machine) -- 20 GB 1 GB
Video Card 32MB graphics card supported by Linux None None 128MB NVidia GeForce4 for display machine
3
Generally Speaking

• Get the data
• Real-time
• Archive

• Process the data

4
Obtaining Data Archive Data

• Radar
• Level II III from NCDC
• http//hurricane.ncdc.noaa.gov/pls/plhas/has.dssel
  ect
• OKC TDWR
• Model
• RUC (20km) Hourly Analyses
• http//www.archive.arm.gov/cgi-bin/arm-archive
• Satellite
• 6 Bands
• Mesonet
• http//www.mesonet.ou.edu/data/basic/mesonet/mdf/

Data available in-house at NSSL. Data
only available through NSSL Contact
Kevin.Manross_at_noaa.gov
5
Obtaining Data Real-time

• Typically via Local Data Manager (LDM)
• See your system administrator or LDM
  documentation regarding install/setup of LDM
• PQACT.CONF File
• Format feedtype TAB prodIdPat TAB action TAB
  arg ...
• (from http//www.unidata.ucar.edu/software/ldm/ldm
  -current/basics/pqact.conf.html)
• Typical WDSSII PQACT.CONF File

Data Source Name of file Entry
WSR-88D (NWS radar network) pqact.conf NEXRD2  L2-(/)/(....)/(0-90-90-90-90-10-90-30-90-20-90-50-90-90-9)        PIPE    /home/ldm/bin/nexradII_bz       /home/ldm/data/nexradII/\2/\3.raw
TDWR (FAA weather radar) pqact.conf EXP     (/)/(OKC)(0-90-90-90-90-90-90-90-9(-)0-90-90-90-90-90-9(-)0-9(.)0-90-9(-)0-9(-)0-9(-)0-9(.)(bz2))        FILE    /data/realtime/tdwr/Raw\2/\2\3
RUC (FSL Rapid Update Model) pqact.conf NMC2   //RUC2/252/(0-90-90-90-90-10-90-30-90-20-90-50-9)/F000/(A-Z)/(/)(!) (0-90-90-90-90-90-9)PIPE   /home/wdssii/WDSS2/bin/gribToNetcdf -o /data/realtime/model/ruc2 -P
6
Data Processing
7
Data Processing

• Process the Raw Data
• Results in Base Data
• Reflectivity, Velocity, Spectrum Width, Pressure
  Levels, Cloudtop Temperature, Wind Direction,
  etc
• Process the Base Data
• Run Algorithms on Base Data
• Azimuthal Shear, Reflectivity at Freezing
  Level, TVS Detection Algorithm, etc

8
Processing Data
First rssd

• Setup
• Set environmental variable RMTPORT
• Eg. export RMTPORT50000
• Set rssd.conf
• Syntax
• Client computers allowed access
• Path accessible local paths
• Example
• Client cairo.protect.nsslClient
  172.19..Path /data/radarPath
  /data/raid/satellitePath /tmp

• What it is
• rssd is the server process that provides
  notification to your algorithms and tools.
• WDSS-II algorithms use rssd to provide these
  capabilities
• Read data files on remote machines.
• Let algorithms know when the data they want is
  available

9
Processing Raw Data Command-line Executables

• Note with any executable found in the WDSSII bin
  directory, running the program with no arguments
  will provide a usage commentary
• To Process
• Radar (Level II / TDWR)
• ldm2netcdf
• Satellite
• satIngestor
• Model (RUC GRIB format)
• gribToNetcdf
• Mesonet
• mesonetIngest
• There are others check Tools listing on
  wdssii.org

10
Processing Base Data
Remember processing base data means running
algorithms.

• Processing base data is syntactically the same as
  processing raw data.
• Again, if you are unsure of the arguments needed
  to run a WDSSII executable, run it with no
  arguments and the usage will be displayed.
• Inputs for processing base data are typically the
  outputs from the raw data.
• Look for code_index.lb (binary file), or
  code_index.xml (XML file)
• Depending on which file type you use, specify or
  xmllb/path/to/code_index.lb or
  xml/path/to/code_index.xml
• The path to the code_index file is usually the
  same path specified with the -o argument when
  processing the raw data

11
Processing Data Command-line Steps

• To summarize simple data processing
• Obtain raw data
• Process raw data
• ldm2netcdf radar
• gribToNetcdf model data
• etc.
• Process base data
• w2qcnn
• scit3D
• etc.
• Display output

12
Processing Data Distributed Processing

• You are not limited to processing data (raw or
  base) on only 1 computer.
• If you have multiple computers with WDSSII
  installed on them, you can distribute the
  processing.
• If you are performing WDSSII processing on
  multiple machines, you can link them together (so
  that the output of one process feeds into the
  input of another process)
• First make sure that your rssd.conf is set up to
  allow communication between computers and
  associated directory paths
• Add the computer name to the path of your WDSSII
  program. I.e.
• filetypecomputer/path/to/code_index
• xmllbwdssiicomputer1/home/wdssiiuser/KTLX/code_i
  ndex.lb

13
Processing Data w2algrun

• Often a user wants/needs to process multiple
  sources of data
• Processing of raw data from multiple radars
• Multiple sources (RUC, radar) needed to run a
  complex algorithm like SCIT or HAD
• Can be accomplished at the command line by using
  quotation marks
• scit3D i xmllb/tmp/KTLX/code_index.lb
  xmllb/tmp/KINX/code_index.lb xmllb/tmp/KICT/code
  _index.lb xmllb/tmp/RUC20KM/code_index.lb o
  /tmp/multi
• This way can quickly become cumbersome and
  confusing
• Use w2algrun

14
Processing Data w2algrun

• What is w2algrun?
• A group of scripts (programs) that helps the user
  to set up multiple WDSSII processes (for both raw
  and base data) and control (start/stop) processes
  as a group.
• The user is generally only concerned about 2
  items in the w2algun directory
• The w2alg.conf file
• The w2alg executable
• Be sure to check out http//www.wdssii.org/ and
  click on Real-time Monitoringfor more
  information

15
Processing Data w2alg.conf

• The w2alg.conf file
• This is a config file that allows the user to set
  up multiple processes.
• Common parameters
• PROCESS_EXE the name of the algorithm e.g
  w2qcnn
• PROCESS_PARAMS the command-line parameters to
  the algorithm. Type the name of the command
  (w2qcnn, for example) to find out what parameters
  that program takes.
• NUM_LOGS  default is 10, after which the logs
  will be rotated
• LOG_SIZE  number of lines per log file
• LOG_DIR directory to write logs
• DIR directory to run from
• PROCESS_DWELL number of seconds to pause before
  starting the next algo.
• RESTART_SLEEP number of seconds to wait before
  restarting dead processes

16
Processing Data w2alg.conf

• More Parameters
• Declaring groups
• group GLOBAL sets for all groups
• Any 'set' overrides within hierarchy
• group GLOBAL
• group KMHX,KRAX,KAKQ,KLWX,KFCX
• Setting macros
• Macros are named M1-M9
• Provides a shorthand substitution
• Set a macro to provide the top left and bottom
  right corners of a domain in lat/lon as well as
  vertical grid spacing
• set M1 " -t "40 -82 20" -b "32 -73 0" -s "0.01
  0.01 1" "
• Managers
• Label a process command to be called by w2alg as
  an executable
• manager RSSD
• manager PROCESS-A2

17
Processing Data w2alg.conf

• Example
• set M1 " -t "40 -82 20" -b "32 -73 0" -s "0.01
  0.01 1" "
• group CLOUD
• manager PROCESS-CLOUDCOVERset PROCESS_EXE
  "/home/wdssii/WDSS2/bin/w2cloudcover"set
  PROCESS_PARAMS "-i "xmllbcairo/data/realtime/sat
  ellite/code_index.lb
• xmllbwrwdss/data/ruc2/products/code_index.l
  b" -o /data/realtime
• l /data/realtime/radar/multi/code_index.lb
  -S TMP-HTGL20km2_analysis
• -r -e 5  M1 -I --verbose"
• Notice how we call the macro M1 in the
  PROCESS_PARAMS section 

18
Processing Data w2alg.conf

• Example
• group KMHX,KRAX,KAKQ,KLWX,KFCX    manager
  PROCESS-A2     set PROCESS_EXE "ldm2netcdf"    
  set PROCESS_PARAMS "-i /home/ldm/data/nexradII/GR
  OUP -s GROUP -o "/data/realtime/radar/GROUP
  /data/realtime/radar/U-GROUP" -k -D "   
  manager PROCESS-QCNN     set PROCESS_EXE
  "w2qcnn"     set PROCESS_PARAMS "-i
  xmllb/data/realtime/radar/U-GROUP/code_index.lb
  -o /data/realtime/radar/GROUP -r -u"   
  manager PROCESS-SCIT2D     set PROCESS_EXE
  "swatScit2D"     set PROCESS_PARAMS "-i
  xmllb/data/realtime/radar/U-GROUP/code_index.lb
  -o /data/realtime/radar/GROUP -f -r -s
  GROUP"     set DIR "/home/wdssii/algconfig/GROU
  P/scit"
• In this example, we first declare a group
  consisting of several radars. We then declare
  three different programs to be called with their
  associated parameters.
• Note how instead of writing out the full path for
  each radar and for each command line parameter,
  we simply call GROUP  

19
Processing Data w2algrun

• The w2alg executable
• After the w2alg.conf file is set up, the user can
  control stopping and starting of algorithms in
  various ways
• Syntax
• w2alg command group manager
• Where
• command is one of start, stop, status, or tail
• group one of the members as defined by
  the group parameter in the w2alg.conf
• file, or all to call every
  member of the group declaration
• manager optional specify an executable as
  defined by the manager declaration

20
Processing Data w2algrun

• The w2alg executable
• Examples

Command Action
w2alg start all Starts all the algorithms
w2alg start ktlx Starts all the algorithms in the group KTLX
w2alg start ktlx a2 Starts the a2 algorithms in the group KTLX
w2alg stop all Stops all the algorithms
w2alg stop ktlx Stops all the algorithms in the group KTLX
w2alg status kltx qcnn Provides status of the qc algorithm on KTLX
w2alg status kltx Provides status of all the KTLX algorithms.
w2alg tail kltx qcnn Tails the log of the qc algorithm on KTLX
w2alg tail mainlog Tails the mainlog which tells you which algorithms died and were restarted.
21
Running an Archived Case

• Simple processing (I.e., single input) of an
  archived case can be easily run at a command
  line. However, this is usually done as quickly
  as the computer hardware allows.
• On the other hand, more complex processing,
  requiring multiple inputs and/or hierarchical
  steps, (usually run with w2algrun) requires
  coordinated timing.
• Use w2simulator to simulate a real-time data
  stream.

22
Running an Archived Case

• w2simulator reads in processed data (the
  code_index files), determines the speed of the
  notification updates (based on user settings),
  and writes/updates an output code_index-like file
  at the determined speed.
• Typical use of w2simulator is to read in the base
  data. w2simulator then updates the
  notifications accordingly. The user then
  connects to the w2simulator output as they would
  when running real-time algorithm processing.

23
Running an Archived Case
Real-time
Archive
Output Base Data
24
Running an Archived Case
w2simulator
Output Base Data and Notify based on timing
arguments given to w2simulator
Available Base Data
25
Running an Archived Case

• We can set up w2algrun to run with w2simulator
• Make your first manager section be the
  w2simulator.
• group KMHX,KRAX,KAKQ,KLWX,KFCX
• manager SIMULATION
• set PROCESS_EXE w2simulator set
  PROCESS_PARAMS "-i xmllb/data/realtime/radar/U-G
  ROUP/code_index.lb o /data/simulation/radar/GROU
  P K
•       manager PROCESS-QCNN     set PROCESS_EXE
  "w2qcnn"     set PROCESS_PARAMS "-i
  xmllb/data/simulation/radar/GROUP/index_0.lb -o
  /data/simulation/radar/GROUP -r u l
  /data/simulation/radar/GROUP/index_0.lb "   
  manager PROCESS-SCIT2D     set PROCESS_EXE
  "swatScit2D"     set PROCESS_PARAMS "-i
  xmllb/data/simulation/radar/GROUP/index_0.lb -o
  /data/simulation/radar/GROUP -f -r -s GROUP l
  /data/simulation/radar/GROUP/index_0.lb "    
  set DIR "/home/wdssii/algconfig/GROUP/scit"
• Notice that we do not call ldm2netcdf since the
  data should already be processed
• Notice the input path of the algorithms is the
  output path of w2simulator
• Notice the .lb filename (both input and output)
  is not code_index.lb
• We place the output of the algorithms into the
  simulation directory (-o)

26
Creating Snapshots

• Sometimes it is useful to save images of the data
  you are viewing.
• WDSSII provides the user with two ways to take a
  snapshot of the current view
• Random Snapshot click on the display and take
  a snapshot of the current image
• Auto-snapshot Set WDSSII to create an image of
  each product whenever there is an update of the
  data

27
Creating Snapshots

• Snapshot preferences
• In the wg display
• Under the Options Menu Item
• Click Edit Preferences
• Select the Snapshot tab
• Rule to create the default name for snapshots
• Good idea to include product name (p)
• Include either time (t) or count (c)
• Use custom-size if you want images larger/smaller
  than the screen size.
• Can specify png/gif/jpg etc.

28
Creating Snapshots Random Snapshot

• 1) Right Click on radar image 2) Select
  Take Snapshot

29
Creating Snapshots Auto-Snapshot

• Turn on AutoSnapshot

• Turn on AutoUpdate

30
Question

• How can you create a sequence of images from a
  10-hour data case?
• Answer on next slide.
• Hint how would you capture 10-hours of imagery
  in real-time?

31
Answer

• Dont do things manually!
• Think about using the WDSS-II tools in
  combination.
• The display is just another algorithm
• Use w2simulator and the display
• Simulate the 10-hours of data you have
• Use the display to view the simulation LB
• Switch on auto-update to display data as they
  become available in the simulation LB
• Switch on auto-snapshot to create images from the
  displayed data.
• Let the w2simulator and wg run until the case is
  exhausted
• Run at 10x real-time speed to finish the job in 1
  hour

32
Mirroring Data w2mirror

• Sometimes it is helpful to process the data on
  one machine, but serve the data on another.
• For example, it may be helpful to split data
  processing among many machines, but serve them on
  one machine.
• Less confusing for users to locate data
• Fewer potential security/firewall issues
• Use w2mirror
• Simply copies data from source computer to
  mirror computer and adjusts all relative links.

33
Housekeeping

• Universal WDSS-II command-line options
• --verbose prints out additional information
  that may be useful in locating problems
• --configFilemyConfigFile.xml allows the
  user to run the application with a set of options
  that are specified by the input file. This is
  very useful (and even required in some cases)
  when you have a long command line. For example
• ltfilter path"w2merger"gt
• ltoption letter'i' value"xmllbanubis/data/NSE/c
  ode_index.lb xmllbisis/data/realtime/radar/KABR/
  code_index.lb ..... (long list of LBs here)"gt
• ltoption letter'o' value'/data/realtime/radar/mul
  ti'/gt
• ltoption letter't' value'51 -127 20'/gt
• ltoption letter'b' value'21 -67 0'/gt
• ltoption letter's' value'0.01 0.01 1'/gt
• ltoption letter'C' value'5'/gt
• ltoption letter'a' value""/gt
• ltoption letter'V' value'false'/gt
• ltoption letter'e' value'60'/gt
• ltoption letter'I' value'ReflectivityQC'/gt
• ltoption letter'p' value'0.5'/gt
• ltoption letter'r' value'true'/gt
• lt/filtergt

• Tools for investigating a linear buffer (I.e.,
  code_index.lb)
• lb_cat lb_cat is a way to examine the contents
  of a linear buffer or to write to it from the
  command line or a script.
• lb_create Allows you to create a linear buffer
  from a script. Make sure to use the -n option to
  size the linear buffer to about 32000.
• lb_info lb_info gives you basic information
  about the messages in a Linear Buffer.
• lb_nt lb_nt allows you to troubleshoot event
  notification.
• lb_rm Removes a linear buffer.

• Cleaning up
• w2scour
• scours (deletes) data, similar to unix scour
• Found in src/scripts directory
• Usually run via cron
• Configure with scour.conf

34
Troubleshooting
When running WDSSII fails, there are several
areas to examine

• Are the raw data being processed?
• Check for code_index.lb
• Is code_index.lb in the correct directory path?
• Is there data being written to code_index.lb?
• lb_cat code_index.lb

• Is RSSD running?
• Are the raw data arriving?
• Check LDM setup and watch input flow
• ldmadmin watch
• Is there any available disk space left on the
  hard drive?
• Have we scoured the raw data?
• Have we scoured non-needed processed data?

35
For More Information

• Check out the following websites
• http//www.wdssii.org
• An overview of the entire WDSSII system
• http//forum.nssl.noaa.gov
• A great QA site for WDSSII development and
  administration.
• End of
• WDSSII Setting up a real-time system