SA @ WV (software assurance research at West Virginia)

1
SA _at_ WV(software assurance research at West
Virginia)

• Kenneth McGill
• NASA IVV Facility Research Lead
• 304.367.8300
• Kenneth.McGill_at_ivv.nasa.gov
• Dr. Tim Menzies Ph.D. (WVU)
• Software Engineering Research Chair
• tim_at_menzies,com

2
Why, what is software assurance?

• Why software assurance?
• bad software can kill good hardware.
• E.g. ARIANE 5 (and many others)
• Software errors in inertial reference system
• Floating point conversion overflow

Ariane 5

• Definition
• Planned and systematic set of activities
• Ensures that software processes and products
  conform to requirements, standards, and
  procedures.
• Goals
• Confidence that SW will do what is needed when
  its needed.

3
OSMA Software Assurance Research Program

• Office of Safety Mission Assurance (Code Q-
  OSMA)
• Five million per year
• Applied software assurance research
• Focus
• Software, not hardware
• SW Assurance
• NASA-wide applicability
• Externally valid results i.e. useful for MANY
  projects
• Organization
• Managed from IVV Facility
• Delegated Program Manager Dr. Linda Rosenberg,
  GSFC

4
Many projects

• Mega highest-level perspective
• e.g. project planning tools like ASK-PETE
  Kurtz
• Macro
• e.g. understanding faults Sigal, Lutz
  Mikulski
• Micro
• e.g. source code browsing Suder
• Applied to basic
• Applied
• (e.g.) MATT/RATT Henry support large scale
  runs of MATLAB
• Basic (not many of these)
• e.g. Fractal analysis of time series data
  Shereshevsky
• Many, many more
• Too numerous to list
• Samples follow
• See rest of SAS!

Horn of plenty
5
Many more projects!
Ratio FY02/FY01
Total proposals 2.2 NASA centers
1.5 Industry 26 University 3.7
6
A survey of 44 FY01 CSIPs
project 1 2 3 4 5 6 7 8 9 10 11 12 13 14 to 44
AATT                         2
ISS                         2
Space Shuttle                         2
ST5                       2
Aura                           1
CHIPS                           1
CLCS                           1
CM2                           1
CMMI                         1
DSMS                           1
EOSDIS                           1
FAMS                           1
GLAST                           1
HSM4                         1
HST                           1
Mars 07                           1
Mars 08                           1
PCS                           1
Space Station                             1
Starlight                           1
Stereo                             1
SWIFT                           1
X-38                           1
5 4 3 2 2 2 2 2 1 1 1 1 1 0
75 with no claim for project connections
Need more transitions! (but dont forget
the theory)
7
Action plan- restructure CSIPS more transitions!

• New (year 1)
• Fund many
• Renewed (year 2)
• Continue funding the promising new projects
• Recommended letter of endorsement from NASA
  project manager
• Transition (year 3)
• Select a few projects
• Aim tools in the hands of project folks
• Required project manager involvement

• Reality check
• Transition needs time
• Data drought

8
Long transition cycles

• Pecheur practical formal methods
• In-Situ Propellant Production project
• Taught developers
• Livingstone model-based diagnosis
• model-checking tool tools
• developed by Reid Simmons, (CMU)
• Technology to be applied to the Intelligent
  Vehicle Health Maintenance (IVMS) for 2nd
  generation shuttles
• Lutz, Mikulski ODC-based analysis of defects
• Deep-space NASA missions
• Found 8 clusters of recurring defects
• Proposed and validated 5 explanations of the
  clusters
• Explanations ? changes to NASA practices
• ODC being evaluated by JPLs defect management
  tool team

Marsatmosphere
on-board
CO2 2H2 gt CH4 O2
fuel
oxidizer
Robyn Lutz JPL, CS-Iowa State
(no photo) Carmen Mikulski JPL
9
The data drought
Gasp need data
10
End the droughtbootstrap off other systems

• Find the enterprise-wide management information
  system
• Insert data collection hooks
• E.g. JPL adding ODC to their defect tracking
  system
• WVU SIAT sanitizer

11
End the droughtContractors as researchers

• Buy N licenses of a defect tracking tool (e.g.
  Clearquest)
• Give away to projects
• In exchange for their data
• Build and maintain a central repository for that
  data
• With a web-based query interface
• Data for all

take me to your data
active data repository
12
End the droughtContractors as researchers (2)

• See also
• Titans new ROI project
• Any contractor proposing an NRA
• Galaxy Globals metric project

experience
1
4
action
reflection
2
Mark Suder Titan, IVV
abstraction
3
Hypertext power browser for source code

SIAT-1
4

Use it.
1
For high-severity errors, recall what SIAT
queries lead to finding those errors
2
SIAT2
Assess each such power queries Reject the less
useful ones
3
Procedures manual for super SIAT or new search
options in interface
4
13
End the droughtraid old/existing projects

• Cancelled projects with public-domain software
• E.g. X-34
• Or other open source NASA projects
• E.g. GSFCs ITOS
• real-time control and monitoring system during
  development, test, and on-orbit operations,
• UNIX, Solaris, FreeBSD, Linux, PC
• Free!!
• NASA project connections
• Triana,
• Swift,
• HESSI,
• ULDB,
• SMEX,
• Formation Flying Testbed,
• Spartan

14
End the droughtsynergy groups

• N researchers
• Same task
• Different technologies
• Share found data
• E.g. IVV business case workers
• E.g. monthly fault teleconferences
• JPL
• Lutz, Nikora
• Uni. Kentucky
• Hayes
• Uni. Maryland
• Smidts
• WV
• Chapman (Galaxy Global) Menzies (WVU)

15
End the droughtTandem experiments

• Technique X finds errors
• So?
• Industrial defect detection capability rates
• TR(min,mean,max)
• TR(0.35, 0.50, 0.65)
• Assumes manual Fagan inspections
• Is X better than a manual 1976technique?
• Need tandemexperimentsto check
• I.e. do it twice
• Once by the researchers
• Once by IVVcontractors (baseline)

fictional data
16
AlternativelyEnd your own drought

• Our duty, our goal
• Work the data problem (e.g. see above)
• Goal of CI project year1 build bridges
• But the more workers, the better
• Myth there is a data truck parked at IVV
• full of goodies, just for you
• Reality Access negotiation takes time
• With contractors, within NASA
• We actively assist
• Each connection is a joy to behold, an occasion
  to celebration
• We dont celebrate much
• Bottom line
• We chase data for dozens of projects
• Researchers have more time, more focus on their
  particular data needs
• Kens law
• chases researchers who chase projects
• CI year2, year3 needs a project connection

17
Alternatively (2), accept the drought and sieve
the dust

• The DUST project
• Assumes a few key options control the rest
• Methodology
• Simulate across range of options
• Data dust clouds
• Too many options what leads to what?
• Summarize via machine learning
• Condense dust cloud
• Improve mean, reduce variance
• Case studies
• JPL requirements engineering
• Feather/JPL Re02
• Project planning
• DART- Raque/ IVV Chaing/UBC
• IVV costing Marinaro/IVV, Smith/WVU
• general Raffo, et.al/PSU Ase02
• An analysis of pair programming Smith/WVU
• Better predictors for
• testability Cukic/WVU, Owen/WVU Issre02, Ase02

The answer my friend, is blowin in the wind
But wait the times they are changing
Each dot 1 random project plan
18
Other WVU SA research
? Testing formal methods ? Bayesian
approach to reliability ? Architectural
metrics ?Risk assessment dynamic
UML ? Reliability
operational profile errors

Bojan Cukic
Hany Ammar
Katerina Goseva Popstojanova
UML (sequence diagrams, state charts)
Software Specs design (early life cycle)
collaborator
UML simulations
Architectural descriptions
Goal accurate, stable, risk assessment early
in the lifecycle
Static (SIAT, Mccabe, entrophy)
Code analysis (ivv,operational usage)
Dynamic (testing, runtime monitoring)
Metrics(complexity,coupling,entropy )
Fault, failure data on components, connectors
Failure data from testing
Severity of failures
19
More WVU research (FY02 UIs)
Architectural metrics Risk assessment dynamic
UML Intelligent flight controllers Testing
formal methods Bayesian approach to
reliability Fractal study of resource
dynamics Reliability operational profile
errors SE research chair interns DUST
Ammar
ISS hub controller, Dryden application
c
cccccc
Cukic
jc
X34
ITOS
jj
F15
SIAT
w
X38
c
FY03 proposals 2.2FY02
Goseva- Popstojanova
Menzies
JPL deep space mission DART KC-2 IVV cost
models
j, ccccccc, w
c conference w workshop j journal
new renewed
20
Function Point Metrics for Safety-Critical
Software

• Thesis
• Traditional function-point cost estimation
• Incorrect for safety-critical software
• gt 1 way to skin a cat
• gt1 way to realize a safety critical function
• NCPN-copy programming
• NVP N-Version Programming ,
• NSCP N Self-Checking Programming,
• With, without redundancy,
• Method
• explore them all!

H2 and C2 effort cost, redundant systemH1
and C1 effort cost, non-redundant system
Afzel Noore
21
Pre-disaster warnings Cukic, Shereshevsky
Can we defer a maintenance cycle and keep doing
science for a while longer?
Mark Shereshevsky
Bojan Cukic
ARTS II
Crash
22
Intelligent flight controllers Napolitano,
Cukic (and menzies)
Marcello Napolitano(Mechanical andAerospace)
Bojan Cukic (CSEE)
Lifecycle opportunities for VV of neural network
based adaptive control systems.
23
The road ahead applied theoretical research
Need both
CSIPs applied research
USIPs applied theoretical research
To boldly go