DMSS Project

1
NASA Langley Research CenterDistributed Mass
Storage System

• DMSS Project
• HPSS User Forum
• Oakland, California
• June 7-9, 2005

2
DMSS Team

• Frank Thames
• Roger DuBois
• Dave Elliott
• John Kaiser
• Tim Starrin

3
Introduction

• Site Overview Mission
• HPSSs Role
• Hardware and Software Configuration
• Statistics and Projections
• Unique to Our Site
• Activities
• HPSS Development Wish List

4
NASA Langley Research Center

• Established in 1917

5
Langley leads NASA initiatives in

• Aviation safety
• Quiet aircraft technology
• Small aircraft transportation
• Aerospace vehicles system technology
• It supports NASA space programs with
  atmospheric research and technology testing and
  development.

www.larc.nasa.gov
6
Langleys Research Capabilities

• Systems analysis, integration, and assessment
• Aerodynamics and Aerothermodynamics
• Airframe systems and Airborne systems
• Acoustics
• Materials and structures
• Hypersonic air-breathing propulsion
• Atmospheric, radiation, chemistry and dynamics
  remote sensing

7
HPSSs Role

• User Archive
• Virtual Storage
• System Backups
• Repository for Data Management Systems
• Backend Web Services
• Disaster Recovery Solution

8
DVAL provides a centralized capability for
enhancing, visualizing, interpreting, and
exploring experimental and computational datasets
from a variety of disciplines in support of
Langley programs and projects. Highly skilled
specialists working with state-of-the-art
hardware and software can produce effective
data visualizations and custom solutions.
Products can be developed for distributed
platforms, ranging from laptops and desktops to
fully immersive environments.
9
(No Transcript)
10
(No Transcript)
11

Expert assistance in the construction and
analysis of computer-based geometry for numerical
engineering analysis
12
Re-entry Vehicle Design Development
Mach 6 air, ? 40
13
Return to Flight
14
Internal Network
FC
SAN
STK 9940A
STK 9490B
15
Configuration - Software
NASA LaRC Developed Tools Interfaces
16
Statistics Performance - Storage Traffic

• Storage
• 4.6 million files directories 174 TBs of data
• Daily Traffic
• 2,887 transfers, peak 31,291 transfers
• 1.2 TBs data, peak 2.7 TBs
• Weekly Traffic
• 16,724 transfers, peak 54,234 transfers
• 8.2 TBs data, peak 9.7 TBs data
• Monthly - 414 hosts

17
Statistics Performance - Hardware

• Network
• Remote transfers up to gigabit ethernet speeds
• Disk Tape
• FC attached RAID 250 MB/sec
• FC 9940A tape 10 MB/sec
• FC 9940B tape 30 MB/sec
• Migration
• 9940A .7 MB/sec (1 stream)
• 9940B 25 MB/sec (2 streams)

18
Projections

• If current trends continue
• Storage
• 1.6x annual growth rate
• 278 TBs in 1 year, 1.8 PBs in 5 years
• Capacity of silos 3.4 PBs with 9940 media
• Daily Traffic
• 2.2x annual growth rate
• 2.6 TBs in 1 year, 61.8 TBs in 5 years (.7
  GB/sec)

19
Unique to our Site

• User Interface
• EARS command suite
• SRB prototype
• Tools
• Monitoring
• Recovery
• Reporting

20
The BIG Problem

• Amount of data generated stored by applications
  presents serious challenges
• Becomes unmanageable if user has no way of
  knowing what data is or where to find it
• More information about data content is needed
• Ability of scientists to use depends on ability
  to access and manage intelligently and efficiently

Whitepaper on Data Management, R. Sumpter
21
Langleys Problem

• Current management systems of aeronautics data
  at LaRC are ad hoc, file-based, non-searchable
  collections of terabytes of numbers
• Given the growth of technical data sets to tera-
  and petabyte sizes, it will not be possible to
  execute modern, interoperable, collaborative
  efforts on difficult mission tasks if current
  storage/management methods continue to be
  utilized.

J. Z. Pao NASA LaRC
22
SRB Proposal

• Investigate its feasibility as a platform to
  manage the vast amount of aeronautical research
  data at NASA
• Demonstrate ingestion of a select set of current
  aeronautical research data into an information
  system in a brief time period w/ minimal impact
  on Center research staff

23
SRB Features

• Transparent access and management of files and
  metadata in over 95 formats
• Storage resources from HPSS to Unix disk, the
  web, even your own server/system
• Web-accessible and searchable hierachical
  database and filesystem
• Both command-line-driven and GUI

24
SRB Prototype

• Early Configs
• Sun Blade 100, Solaris 8, SRB 3.0.1, Sun Java
  1.4.2, and PostgreSQL 7.3.5
• IBM F50, AIX 5.1, SRB 3.0.1, Sun Java 1.3, and
  PostgreSQL 7.3.5
• Prototype
• MCAT Server - Sun Fire V440, Solaris 9, SRB 3.2,
  Sun Java 1.4, Oracle 9.2.0.5, and OpenSSL-0.9.7c
• Multiple Resource Servers, including HPSS 4.5.0.1

25
SRB Testing Configuration
LAN
Resources
26
SRB Performance Matrix
(Megabytes per second)
27
SRB Data/Metadata Ingestion
Ingested - 15,000 objects - 250,000 metadata - 21
windtunnel tests - 3 wind tunnel facilities
28
SRB Success

• The SRB Prototype project is considered a success
  in the validation of its...
• Ease of use
• Low cost to maintain
• Viability and speed to bring all historical wind
  tunnel data online and searchable by metadata

J. Z. Pao NASA LaRC
29
Tools

• Monitoring
• check-hpss, auto-check-hpss
• timeout
• whpss, xwhpss
• xhpssdf, hpssdf
• xhpssmig, auto-check-mig
• Recovery
• hpsstape, lstape

30
Auto-check-hpss

• If paging is disabled, issue warning message and
  exit
• If current time is during backup window, exit
• If check-hpss returns error, send message to
  pager
• Execute hpss-df, if warning threshold is
  reached, send message to pager
• Check user access, if disabled, send message to
  pager
• Execute HPSS ls (via masls and timeout), if
  it takes more than 5 minutes (configurable) to
  execute, send message to pager
• If non-prime shift, check tape migration via
  auto-check-mig if any storage class has a
  migration tape which hasnt been written to in 3
  hours, send message to pager

31
Activities - Recent

• SRB Prototype
• SAM QFS Evaluation
• StorNext Evaluation
• Centerwide Backup Study

32
Activities - Current

• Timberline to 9940A Transition
  (3.8 million small files on 4,302 tapes)
• End-to-End Disaster Recovery Test
• Cost Analysis of HPSS and Alternate Solutions

33
Activities - Upcoming

• Moving HPSS Metadata to Faster FC Disk
• Core Server Hardware Upgrade
• HPSS 6.2 Upgrade

34
Problems Concerns

• Small File Performance
• Share-able HPSS Filesystem
• Disaster Recovery
• End-to-end DR test
• Import/export of metadata and data
• Converting 2nd copy to primary copy

35
Development Wish List

• (1) Fast Locate support for tapes
• (2) SAN Shared FS support??
• (3) Retry logic in HPSS FTP interface and client
  API, HSI
• (4) Tape VSNs supported as arguments or input to
  reclaim command

36
Questions?