CmpE 550 Advanced Software Engineering

1
CmpE 550 Advanced Software Engineering
Scenario-Based Assessment of Nonfunctional
Requirements
Selim Özyilmaz Elif Sürer
2
INTRODUCTION
Introduction

• Purpose Validation of Requirements Specification
  using Scenarios
• Scenarios are applied to the analysis of
  non-functional requirements using dependency
  tables to assess the relationship between goals
  (functional and non-functional requirements) and
  the agents and tasks that achieve them in the i
  language

3
Intro(Contd)

• Many NFRs are influenced by human properties,
  they inherit the diverse nature of human
  characteristics (System Reliability is influenced
  by human characteristics such as ability)
• This work is a revised version of a previous one
  that prompted designers with questions about
  potential problems in a scenario event sequence.
  (Psychology-based taxonomy of failure)

4
Intro(Contd)

• The problem is that too many scenario variations
  were generated. To prevent, transform the
  taxonomy of human and system failures into a
  model to predict the errors in the system
  design.
• Bayesian Belief Nets (BNs) are used to predict
  the reliability.

5
Related Work

• Model-checking techniques have been used
  extensively to verify and validate requirements.
• Communication problem occurs between
  user-stakeholders and the model developers.

Software Cost Reduction (SCR)
tabular representation
6
Related Work
Related Work

• A combination of visualizations, examples, and
  simulations is necessary to explain complex
  requirements to end users.
• Scenario-based representations and animated
  simulations help users see the implications of
  system behavior and, thereby, improve
  requirements validation.

7
Related Work
Related Work

• Animation simulation tools -gt by Dubois et al. in
  the ALBERT II language
• KAOS language and supporting GRAIL tool which
  enable formal reasoning about dependencies
  between goal model, system behavior and
  constraints.
• Animator-validator tool, TROLL, uses a formal
  object-oriented language for modeling information
  systems

8
Related Work
Related Work

• What is a sufficient set of scenarios to enable
  validation to be completed with confidence?
• While we believe there is no quick answer to this
  vexing problem, one approach is to automate the
  process as far as possible so more scenarios can
  be tested.

9
Related Work
Related Work

• Intent specifications provide a hierarchical
  model to facilitate reasoning about system goals
  and requirements in safety critical systems.
• Goals are decomposed in a means-ends hierarchy,
  widely practiced in requirements engineering.
• Automated support for reasoning about conflicting
  system states and behavior is provided by the
  SpecTRM-RL tool.

10
Related Work
Related Work

• Assessment of nonfunctional system requirements,
  such as system reliability, has to use
  probabilistic reasoning since the range of
  potential system behaviors is either unknown, in
  the early requirements phase, or too large to
  specify.
• Bayesian Nets (BNs) have been developed to assess
  software quality from properties of the code and
  software engineering process.

11
Related Work
Related Work

• BNs have been widely applied as a probabilistic
  reasoning technique in software engineering and
  other domains however, previous work used single
  nets to evaluate a set of discrete states
  pertaining to a software product or development
  process.
• More automated tools for scenario analysis of NFR
  conformance for requirements specifications with
  multiple BN tests have been developed.

12
Modeling Uncertainity
Modeling Uncertainty

• There are four different techniques to model
• Bayesian Probability
• Dempster-Shafer
• Fuzzy Sets
• Possibility Theory
• Bayesian Probability offers an easier combination
  of multiple influences on probability than
  Dempster-Shafer and a sounder reasoning than
  Fuzzy sets.
• Bayesian Probability provides a decision theory
  of how to act on the world in an optimal fashion
  under circumstances of uncertainity.

13
Bayesian Belief Nets

• Directed acyclic graphs of causal influences,
  where the nodes represent the variables and the
  arcs represent the relationships between
  variables.
• Variables can have any number of states in a BN,
  so the choice of measurement is left to the
  analyst.
• Network Probability Table (NPT)

14
BN (Contd)

• Input evidence are propagated through the network
  updating the values of other nodes
  (Computationally complex but effective algorithms
  exist)

15
BN (Contd)

• Three possible ways to compute the probability of
  each node in the network
• Input a posterior probability into a BN as a
  prior observation ?each run should be assumed to
  be independent
• Combine output probabilities from a sequential
  run using a summarizer ?probabilities of
  particular run has to be set initially
• Output probability of each event is compared with
  a threshold value, if surpassed success else
  failure
• Pinpoint some steps in the scenario which are
  weak in reliability
• Sensitiviy analysis can be carried out with
  multiple BN runs by varying environmental
  variables

16
BN Model of System Reliability

• Influencing factors are divided into two
• Slips Attention based lapses and omissions in
  skilled behavior
• Mistakes failures in plans and knowledge of
  processing.
• System environmental variables have an indirect
  effect on an individuals ability whereas
  organizational factors (management culture) have
  a direct effect.
• High Cognitive Complexity ? more prone to
  mistakes
• High Physical Complexity ?more prone to slip
  errors

17
System Reliability

• Remarks
• Input variables are all discrete states
• The BN is a run with a range of scenarios that
  stress-test system design against operational
  variables.
• Scenarios can be either taken from
  domain-specific operational procedures or by
  interviewing with users
• Two different outputs slips and mistakes

18
Operational Performance Time

• Same variables with reliability case, the
  difference is that except two output nodes only
  one output node
• The output is used to increase the best case task
  completion time to reflect the less than ideal
  properties of human and machine agents.
• ET estimated time BT best task completion time
• To reflect the case of reverting to manual when
  an automated technology fails, highly automated
  tasks worst completion time is set greater than
  the those of manual tasks

19
SRA System Architecture

• Analysis starts with the selection of the i
  model to be evaluated and creating the test
  scenarios.
• Scenarios are narratives taken from real life
  experience describing operation of similar
  systems from which event sequences are extracted.

20
SRA Tools Components

• The Session Controller implements the user
  command interface for selecting designs and
  scenarios and executes the algorithm that
  assesses a set of scenarios with the BNs. It
  calls the system reliability or operational
  performance BN assessors to execute the BN runs
  with all possible environmental combinations.

21
SRA Tools Components

• The i model editor allows interactive
  construction of i models with typical CASE
  tool-type functions.
• The Interactive Scenario Constructor produces
  test scenarios from the system model based on
  user directions. Scenarios are stored in a
  database in an array of tuples.

22
SRA Tools Components
SRA Tools Components

• The Model Controller controls the BN models. It
  selects the appropriate BN model for each task
  step, then populates the input nodes, runs the
  model, and receives the belief distributions of
  the output nodes.
• The Model Controller also manages the back
  propagation of the BN model to identify required
  technology and agent characteristics.

23
SRA Tools Components
SRA Tools Components

• The BN assessor modules run the net by calling
  the HUGIN algorithm for each task step and for
  each set of environmental variable combinations.
• The output from each run is compared with the
  desired NFR threshold and the survivor runs are
  passed to the results visualizer.

24
SRA Tools Components
SRA Tools Components

• The Visualizer provides a visual summary of all
  qualified BN runs for a set of scenarios for one
  or more system designs.
• This enables different designs to be compared and
  problem areas in the requirements to be
  identified, i.e., task/technical component
  combinations which show low potential NFR
  assessments. The Visualizer displays results at
  three levels System, Scenario, and Phase views.

25
NFR Analysis Method
NFR Analysis Method

• Remarks
• Scenarios are composed of a number of phases and
  each phase is composed of a number of task-steps
  each modeled as ltAgent,Task,Technologygt
• Phases are used to structure task sequences that
  fulfill a higher order goal.
• The best design is generally the one who has more
  surviving BN runs.
• The best design also needs to be resilient to
  environmental conditions.

26
NFR Analysis Method
NFR Analysis Method

• Impact of Environmental Variables
• (1) Survivor runs with ER x set to best case
• (2) Total survivor runs for all settings
• If an overall design or a particular task fails
  to meet the threshold back propagation alaysis is
  used to discover the necessary settingto achieve
  the NFR value.
• All input nodes are unconstrained (calculation
  for all)
• One/few inputs are unconstrained (calculation for
  unconstrained)

27
Case Study

• The application of the SRA tool in validating the
  operational performance and system reliability of
  a complex socio-technical system.
• The requirements question is to assess the impact
  of new automated technology on the task of
  loading weapons on to aircraft in an aircraft
  carrier.

28
Case Study
Case Study

• Tasks in Design 1 are manual or semiautomated,
  while, in Design 2, they are semi or fully
  automated.
• The second design saves manpower since it can be
  operated by one WA and is potentially more rapid
  to operate, but it is more expensive.

29
Case Study

• When the operational performance times are
  compared, Design 2 is quicker for nearly all
  tasks, which is not surprising since it has more
  automated tasks.

30
Case Study
Case Study

• The critical environmental variables for both
  designs, shows that incentives, motivation, duty
  time concurrency, and time constraints were all
  marked as vulnerable for Design 1.
• Design 2 in contrast, fares better with only
  motivation, concurrency, and maintenance marked
  as vulnerable.

31
Case Study
Case Study

• After identifying the most appropriate design,
  the problematic tasks, and the critical
  environmental variables, the analyst investigates
  the improvements required for the Autoload
  palette component, which was the weakest link in
  Design 2.

32
Validating the BN Models

• Data mining techniques are used to test the
  assumptions in BN models.
• All possible permutations are simulated and
  created a database of reliability and performance
  time predictions
• Data mining Techniques
• Relevance Analysis ranks input parameters of the
  model based on their relevance with output
• Association Rules describe how often two or more
  facts cooccur in a dataset and were employed to
  check the causal associations in our model.
• Classification partitions large quantities of
  data into sets with common characteristics and
  properties

33
Validating BN Models (Contd)
Validating the BN Models

• Results
• Sea state had only a minor influence on system
  error ? relevance analysis
• IF (Duty-timehigh) ? survivedfail
• IF (Workloadhigh) ? survivedfail (Association)
• These rules indicate that causal influences of
  these variables are higher than assumed, alter
  NPT settings to overcome.
• Crew motivation and agent ability problems ?
  classification

34
Discussion Conclusions

• Automated testing of requirements specifications
  and designs for conformance to nonfunctional
  requirements using a set of scenarios and
  variations in the system environment have been
  developed.

35
Discussion Conclusions
Discussion Conclusions

• The SRA could be applied to any class of
  component-based problems where the selection of
  components needs to be optimized for
  nonfunctional requirement types of criteria.
• The SRA tool was a development from previous BN
  requirements analyzer and has partially addressed
  the difficult problem of scenario-based testing

36
Discussion Conclusions
Discussion Conclusions

• The SRA tool is aimed at requirements
  investigation in complex socio-technical systems
  and, hence, it complements model-checking tools
  which are more appropriate to later stages in
  development when specifications of agent behavior
  are available.