Techniques and tools for VV

1
Techniques and tools for VVA - revisited

Jørgen Moth UNI-C, the Danish IT Centre for
Education and Research DTU, Bldg. 304 DK-2800
Lyngby Denmark Email jorgen.moth_at_uni-c.dk
2
Presentation outline

• REVVA Background
• Introduction
• The seven phases of REVVA
• ToA and ToVV
• Basic VV techniques and tools
• Requirements Engineering
• DMSO techniques
• Multi Criteria Analysis
• Experimentation techniques, DOE
• Techniques and tools for VV in the large
• Argumentation techniques
• Decision techniques
• Future directions
• Conclusions

3
REVVA Background

• Western European Armament Group's THALES
  Memorandum of Understanding
• JP11.20 project (nick named "REVVA")
• Common Validation, Verification and Accreditation
  Framework for Simulation
• March 2003 through October 2004
• France, Italy, Sweden, The Netherlands, Denmark

4
Introduction REVVA basics

• The concepts of Target of Acceptance (ToA) and
  Target of Verification and Validation (ToVV)
• The explicit distinction between the properties
  "correctness" and "validity" of a simulation
  model and the processes of "verification" and
  "validation"
• Hooks for the integration of methods for
  estimation of uncertainty introduced by the
  current inability to prove neither correctness
  nor validity.
• A role model, which clearly distinguishes
  technical and managerial roles and sides from
  which the actors may come.
• A demarcation of "worlds" related to or
  associated with modelling and simulation.

5
The seven Phases of REVVA
6
ToA and ToVV

• VV relevant Acceptability Criteria, which
  exclusively address the models correctness and
  suitability are recorded as the Target of
  Acceptance, ToA.
• The Target of VV (ToVV) is developed based on
  the ToA. For each criterion a strategy is
  developed, which points out how with the
  information at hand and the available technical
  means it can be demonstrated that it is passed or
  failed.

7
Basic VV Techniques and Tools

• Techniques and tools primarily inherited from the
  software engineering and MS disciplines. They
  include
• Requirements Engineering
• Informal, static, dynamic and formal techniques
  for verification and validation
• The basic techniques and tools are mainly
  supporting the first four phases of the REVVA
  process.

8
Requirements Engineering

• Are the requirements -
• complete
• consistent
• comprehensible
• unambiguous
• described in a well structured document
• traceable
• conforming and documented to defined standards?

9
DMSO Techniques

• The RPG document on VV techniques lists 77
  different techniques useful for phase 2 and 4 of
  the REVVA process
• Multi Criteria Analysis (MCA)
• Design of Experiment (DOE)

10
Multi Criteria Analysis

• The main role of the MCA techniques is to deal
  with the difficulties that human decision-makers
  have been shown to have in handling large amounts
  of complex information in a consistent way.
• MCA techniques can be used to identify a single
  most preferred option, to rank options, to
  short-list a limited number of options for
  subsequent detailed appraisal, or simply to
  distinguish acceptable from unacceptable
  possibilities.

11
Experimentation Techniques (DOE)

• Design Of Experiments (DOE) is a powerful
  statistical technique for this purpose introduced
  by R. A. Fisher in England in the 1920's to study
  the effect of multiple variables simultaneously
• DOE was later further developed and improved by
  Genechi Taguchi and is today one of the most
  effective quality building tools used by
  engineers in all types of activities

12
Techniques and Tools for VV in the Large

• REVVA very early concluded, that to support the
  full set of REVVA phases, efficient techniques
  for argumentation construction and decision
  support were needed. This is especially true for
  the last three phases of the REVVA process.

13
Argumentation Techniques
14
Decision Techniques

• Cost/Benefit Analysis (CBA)
• Pareto Analysis
• Paired Comparison Analysis
• Grid Analysis
• Decision Trees
• Plus/Minus/Implications (PMI)
• Force Field Analysis
• Six Thinking Hats
• Multi-criteria analysis (MCA)

15
Future Directions

• Integrate Bayesian Belief Networks (BBNs) with
  the Goal Structuring Notation (GSN) approach.
• Whereas GSN provides a means of presenting
  (essentially non-weighted) inferences between the
  claims of a safety argument,
• BBNs provide a means of deriving, modelling and
  quantifying the belief in the inferences between
  claims.
• Develop a supports tool for REVVA

16
Conclusions

• We have identified a number of techniques that
  complement DMSOs RPGs to cover the full REVVA
  VVA process.
• Still some research remains to locate operational
  techniques for evidence evaluation in the vein of
  jurisprudential science.
• A comprehensive tool to support all the seven
  phases of the REVVA process should soon be
  developed, preferably as part of the coming
  REVVA2 project.