Army Research Laboratory

1
Enabling Technologies
James Cogan, Mario Torres, Edward Vidal, Young
Yee, and Jimmy Yarbrough
Army Research Laboratory Computational and
Information Sciences Directorate Battlefield
Environment Division White Sands Missile Range,
NM 88002
2
Definition
Enabling Technologies Hardware, software, or a
combination of both that enable atmospheric
science models, sensors, and related methods and
instruments to interact, perform, and produce
information of value to potential users.
Examples remote access to tower sensors,
automated data collection methods, data
compression (JPEG 2000 based), MetSpaces data
transfer (Java Spaces based),
3
Met Spaces a JavaSpaces concept for
meteorological data distribution and handling.
Primary Developer Mario Torres
4
Requirements

• Tie together a variety of new and legacy
  sensors, models, and data handling software.
• Effectively network an array of loosely
  distributed sensors that may be mobile.
• Handle the distribution of large quantities of
  meteorological data and model output.

Objective
Develop a loosely coupled system for data
distribution between a variety of sensors and
software packages/models that is robust and
flexible over an ad-hoc network.
5
JavaSpaces for Data Collection, Information
Sharing, and Messaging
Any number of data collecting processes can
connect to a (networked shared-memory) space
to Read, write, or take objects of interest from
the space. Processes interact with the space by
simply executing read, write, or take
operations. Data objects reside in the space
until a programmed time-out expires or a process
takes it out of the space. Mechanisms exist to
make contents of the space persistent in case of
system or network failures.
write
Sensor A-2 DataCollector
Sensor A-1 DataCollector
write
read
GUI or Data display
Sensor B DataCollector
Data Archiver
Write
take
6
Met-Spaces
Sample Data-flow for loose-coupling of sensors
and models using a networked Met-Space environment
7
Pros and Cons of Distributed Computing with
JavaSpaces

• Disadvantages
• Setup for optimum performance is not trivial.
• Network latency/load will affect total compute
  time.
• Slowest-worker may affect total compute time.

• Advantages
• Improves over sequential computing times.
• Master-worker mechanism is much easier to
  implement than parallel programming techniques.
• Extremely flexible/scalable
• Master/Tasks are customizable, Workers are
  generic.

8
Dissemination of Data with JavaSpaces

• Advantages
• Provides an easy to implement workflow mechanism.
• Supports large data volumes.
• Distributes Workload
• Very scalable
• Loose-coupling between GUI and business
  logic/servers.

• Disadvantages
• Data objects employed must be predefined.
• Confirmation of services availability needs to be
  devised
• Reliability is network dependent.

9
MetSpaces Summary
Our investigations show that JavaSpaces
technology provides a good model for building
distributed applications. It facilitates
development by providing (1) a simple to use
communications infrastructure and (2) flexible
mechanisms, for the deployment of sophisticated
distributed computing systems of all kinds. Thus
far, MetSpaces, an implementation of JavaSpaces,
has proven to be a very suitable technology for
building distributed data collection systems.
Its scalability allows users to loosely couple
any number of systems from various types of
platforms.
10
Embedded Systems for Meteorological Sensors
Primary Developers Edward Vidal Young Yee Jimmy
Yarbrough
11
Embedded Systems Introduction
A methodology for the collection and management
of a distributed array of 3-dimensional sonic
anemometers has been developed. Issues that need
to be addressed are sensor interface, data
basing, scalability of measurements (number of
sonic instruments), distributed processing of
loosely coupled systems, and performance of the
data acquisition system(s). One of the major
issues in the management and handling of enormous
quantities of micro-met measurements is the time
tagging of the data and location information of
each instrument in a micro-size grid.
12
Requirements

• Deploy large number of sonic anemometers in an
  complex urban environment.
• Effectively network an array of loosely
  distributed sensors together.
• Handle the collection of large quantities of
  meteorological data (144 MB/day per sensor at
  20Hz).

Objective
Develop data collection, storage, signal
processing, and data transfer techniques for
meteorological sensors.
13
Benefits

• Can perform precise time tagging at the sensor
  source.
• Convert slower serial COM port communications to
  faster Ethernet communications (38.4 kbps baud
  rate versus 10/100 Mbps).
• No COM port interrupt conflicts.
• Can quality control the data before sending.
• Can increase the number of sensors without major
  reconfiguration of the system.
• Next generation computers will not have any COM
  ports.

14
(No Transcript)
15
Remote Data Collection Site
REMOTE PC running software for distributed
computing
PHONE LINES
MODEM
HUB
REMOTE SENSORS
16
Move Data via PDA and Cellular Technology
17
Embedded Systems Summary
An architecture of networked sensors has been
demonstrated that is scalable and distributable.
Using microchip technology to collect and process
data at the sensor level, the data from
individual sensors can be networked together.
Assembled data from each local network of sensors
can then be transmitted wirelessly over a bridge
to a central node. Data from individual sensors
are uniquely tagged with GPS location, sensor ID,
and time. Data at the central node can be
checked for incomplete or corrupted data entries
before being saved in a relational data base or
before being applied for model initialization.
Data are distributed between sensors and models
via a flexible and robust Met-Spaces system.
18
COMPRESSIONof MET DATA
Primary ARL Developer Edward Vidal Jr. (Joint
with Univ. of Texas El Paso)
19
Rationale

• Currently gzip GRIB are used, but provide low
  compression ratios of 1.5-31
• High volume meteorological data from remotes
  sensors (e.g., lidar) and high speed in-situ
  sensors (e.g., sonic anemometers) needs
  compression for efficient distribution of Met
  Data products to users.

20
8.68 Hrs 1Gbyte
10.42mins _at_501
21
40Mbytes _at_251
20Mbytes _at_501
22
3D Compression Process

• GRIB or METGM decoder/reader
• Convert binary files to Netcdf format.
• Preprocessing in Z direction using KLT.
• 2D compression on each level using JJ2000.
• Transfer Data
• 2D decompression.
• Post processing in Z direction inverse KLT.
• GRIB/ METGM encoder compressed/decompressed GRIB/
  METGM format.

23
Accuracies
The Potential Temperature data set can achieve an
average compression ratio (CR) of 1131 and
maintain the data to 0.29 K or average
compression ratio of 32.41 and maintain the data
to 0.075 K
CR 1131
CR 32.41
24
Region Of Interest
25
Quality Scalability
0.05 bpp
0.15 bpp
1 bpp
26
Data Compression Summary

• ? Reliable high compression (501) of large set
  of correlated met data is feasible.
• ? Certain parameters are more amenable to data
• compression (e.g. Relative Humidity) than others
• (Water Species Mixing Ratios).
• The efficiency of the Data Compression process
  used here diminishes sharply for smaller data
  sets in which the number of levels of
  decomposition is limited.

27
Enabling Technologies Conclusion
The technologies presented here and others of a
similar nature have the potential to greatly
enhance the availability, distribution, and
usefulness of meteorological information.