Department of Electrical and Computer Engineering

1
Software Quality Safety Assessment Using
Bayesian Belief Networks
Joanne Bechta Dugan Susan Donohue, Ganesh
Pai University of Virginia
2
Problems Under Consideration

• GETR How does one decide that a software system
  is good enough to release?
• SWQ-BBN Can I combine process assessment and
  product assessment metrics to predict
  quality/reliability of a software system?

3
Approach Bayesian Belief Networks (BBN)

• We use BBN models as the basis of both projects
• BBN models effectively allow the combination of
  quantitative and qualitative assessment (that is,
  measures and expert judgment) in the same model

4
GETR Approach (with S. Donohue)

• For the GETR (Good Enough to Release) project, we
  are developing a BBN model of the decision
  process
• What evidence is used, and how is it weighed
• Determining conditional probabilities from expert
  opinion (to get probability parameters for the
  model)
• GETR is building a mathematical framework based
  on BBN to understand and facilitate the decision
  making process

5
GETR Decision
6
(No Transcript)
7
Quantifying Judgment for BBN
8
Quantifying Judgment for BBN (QJ BBN)

• Conditional probabilities (NPT entries) are
  generated as a function of the contribution of
  evidence to support a premise. For example,
• Acceptable results from testing supports the
  conclusion that verification is acceptable.
• Unacceptable documentation supports the premise
  that the artifact quality is unacceptable.
• Evidence can overlap, be disjoint or synergistic.
• Proofs of coherence of functions used in QJ
  methodology help assure rational decisions.
• Importance and sensitivity analysis can help
  guide decision makers in seeking new evidence.
• BN model provides a record of evidence analysis.

9
Application to NASA Seal of Approval Process
(SOAP) for PRA tools
10
(No Transcript)
11
SWQ BBN Approach (with G. Pai)

• For the SWQ BBN project, we are developing
  techniques to build a BN to model the software
  development process and the products (artifacts)
• BBN model represents causally related phases and
  activities within the phases.
• Measurements or expert opinion can be used to
  determine probability parameters for the model.
• Model can be used to assess the process/product
  with respect to reliability (defect density) or
  other quality attribute

12
(No Transcript)
13
Candidate BBN for design phase
14
Hypothetical illustrative example

• Hypothetical priors
• Model result
• Medium defectcontent
• Actual values dependenton the mappingbetween
  node statesand range values
• E.g.
• ?Vlow, Low, Medium, High, Vhigh? ? ?0-20, 20-40,
  40-60, 60-80, 80-100?
• Model results ? Defect content would lie in 40
  60 range

15

• Feedback to the designer ? greater value
• Network itself can provide feedback
• Propagation of evidence
• In this case knowledge of high specification
  quality, observation of high defect content
• Change in distribution indicates potential
  problem area

16
Application to IVV (joint work with Titan
(Khalid Lateef))

• Use IVV process for use case analysis, construct
  BBN from process model
• Relevant process parameters and inputs represent
  parent nodes
• Child nodes of BBN represent features desired
  from the requirements specification

17
(No Transcript)
18
Example analysis

• Probabilities reflect either measurement or
  analysts beliefs
• The state true is less than 95 ? Not mature
  enough.

19
Technology Readiness Level
20
Data / Case Study Availability
GETR case study domain lightweight VV for
in-house developed analytical tools being
considered for release to other centers or
research groups. Identified case studies RAP
(JPL), SIAT (IVV), and MATT (IVV) SWQ BBN case
study domain Case study of system development,
including artifacts defect data. OO or ODC
would be great Working with Khalid Lateef to
develop case study for OO requirements analysis
21
Barriers to Research or Application
Case studies