A Requirement Verification Framework for Realtime Embedded Systems

1
A Requirement Verification Framework for
Real-time Embedded Systems

• Wei-Tek Tsai
• Department of Computer Science and Engineering
• Arizona State University
• Tempe, AZ 85287

2
Agenda

• Background Information
• The Verification Framework
• Requirement and Verification Patterns
• Framework Evaluation
• Summary
• Questions Answers

3
Main Contributions

• A Requirement Centered Verification Framework for
  the requirement verification of real-time
  embedded systems
• Rationale Requirement, test, verifier
• A three layer architecture
• hardware abstraction, support, test application
• Mechanisms
• Event assembling and event dispatching
• Test execution
• Behavioral verification
• Components
• Requirement, scenario, verifier, event assembler,
  test and test engine etc.

4
Main Contributions

• A Pattern-based Requirement Verification
  Framework
• Novel concept of requirement patterns that
  classify requirement based on the temporal and/or
  cause-effect relations
• Novel concept of verification patterns that
  correspond to requirement patterns.
• Verification patterns implemented as verifiers
  within the verification framework context.
• Eight requirement patterns and their
  corresponding verification patterns
• Verification of a whole real-time system can be
  done by a small number of patternized verifiers

5
Main Contributions

• An Extensible Verification Framework
• Requirement Patterns and Verification Patterns
• Feature Monitors
• A natural extension to event assembler
• Provide system abstraction that is more customer
  friendly.
• Verification Extension with Behavioral Models
• Integration/Interfacing with all kinds of
  behavioral models
• Integration and Interaction with a Test Execution
  Framework
• Pipelining test design, development and execution
• High level of automation

6
Background Information

• Importance of the Verification of Embedded
  Software
• Failures in reactive applications including
  safety-critical devices such as cardiac rhythm
  management devices can be costly and have to be
  avoided.
• Software is playing more and more important role
  of controlling system components in many
  safety-critical systems
• Its testing is critical for ensuring products
  that have higher levels of reliability and
  availability.
• Its testing is the most critical phase of the
  product development where the product is checked
  for its compliance with its requirements through
  its execution
• Search Microsoft Excel 2007 Multiplication Bug
• If you type 85077.1 which gives a result of
  100,000 instead of the correct 65,535.

7
Background Information

• The Concepts of Object-Oriented Framework
• A framework is a partially implemented program
  that provides an overall software architecture,
  implemented object and object components for a
  specific application or domain.
• Object-oriented reuse technique data
  abstraction, encapsulation, polymorphism,
  inheritance
• Inversion of Control
• Pre-implemented components
• Design Patterns
• Reuse of analysis, architecture, design,
  implementation

8
Background Information

• Application Frameworks Review
• Earlier Application Frameworks
• The Smalltalk-80 user interface framework,
  Model-View-Controller (MVC), MacApp, InterViews,
  ET etc.
• General Purpose Frameworks
• Taligent Framework, IBM San Francisco
• MFC, CORBA, JavaBean, OpenStep, Rhapsody, Java
  AWT
• Web Applications
• WebObjects, DIWB
• Testing Related
• Testing frameworks developed in Tsai 93, Onoma
  98, Tsai 99, Poonawala 97, Shao 98, Poonawala 99,
  Vishnuvajjala 01, Tsai 02.
• JUnit, Cactus, HttpUnit

9
The Verification Framework

• Essential Verification Concepts

10
The Verification Framework

• General Requirements of the Framework
• The test framework should be application-independe
  nt.
• The test framework should be easy to adapt
  platform independent.
• The test framework should be easy to expand and
  maintain.
• The complexities of the verification framework
  should be hidden from test developers.
• The test design and the test framework are
  totally separate entities.

11
The Verification Framework

• Rationale for the Verification Framework

12
The Verification Framework

• Architecture

13
The Verification Framework

• Hardware Abstraction
• Insulate higher level components of the framework
  from the hardware details.
• Each device included in the testing system has an
  abstract interface that remains constant from
  project to project.
• The same tests can be applied to different
  versions of test target, although this requires
  some modification of layers directly interfacing
  with the device.

14
The Verification Framework

• Event Assembling
• The essential data composition mechanism of the
  framework
• Makes it possible for testers to write tests at
  domain level instead of working at the raw data
  level.
• A event assembler listens to all the events it
  needed and composes new, higher level events from
  the events it sees.
• With multi-level event assembling, testers can
  observe test output at various levels.

15
The Verification Framework

• Example of Event Assembling PVC event
  assembling

16
The Verification Framework

• Use Case of Architecture Test Execution

17
The Verification Framework

• Use Case of Architecture Behavior Verification

18
The Verification Framework

• Whats inside Framework Components

19
The Verification Framework

• How it works Test Engine

20
The Verification Framework

• The Test
• Scenarios
• Sequential execution
• Multi-level test composition
• Verifiers
• Parallel execution multiple verifiers running
  at the same time to verify multiple features
• Continuously monitor test targets behaviors.
• Separation of Scenarios and Verifiers
• Any verifiers can be executed against any
  scenarios
• Feature interactions can be tested

21
Requirement and Verification Patterns

• Motivation
• Conventionally, requirement verification has
  often been done in a case-by-case approach.
• Each new requirement requires a new verification
  procedure implementation, no matter how similar
  it is to some of the existing requirements.
• If a requirement gets changed, its verification
  procedure has to be changed too.
• Separately implemented verification procedures
  need to be debugged separately, which means
  higher cost for development
• Separately implemented verification procedures
  are often implemented by different people and
  normally in different coding styles, which often
  result in higher cost of maintenance.

22
Requirement and Verification Patterns

• Requirement pattern Definition
• A specific temporal pattern or cause-effect
  relation that can be used in representing a
  number of requirements.
• Describing a requirement pattern
• A timeline depicting the temporal relations of
  the events involved.
• Texts are also helpful for more detailed
  explanations.

23
Requirement and Verification Patterns

• Verification pattern
• A verification pattern is a pre-defined
  verification mechanism that can be used to verify
  a group of behavioral requirements that describe
  similar temporal pattern or cause-effect
  relations, or
• A verification pattern is normally associated
  with a requirement pattern and used to verify all
  the requirements that can be classified as belong
  to that pattern.

24
Requirement and Verification Patterns

• Describing a verification pattern
• Name
• Description and Purpose
• Class Diagram (Also for the requirement
  representation)
• Verification State Machine (representing the
  verification logic)
• Other constraints (Optional)

25
Requirement and Verification Patterns

• Requirement Patterns and their coverage of
  requirements

26
Requirement and Verification Patterns

• Basic Requirement Pattern
• Simplest
• Most-commonly seen
• Example
• When a threshold test is initiated by command,
  the firmware shall request the TEMPORARY brady
  state within 480ms

27
Requirement and Verification Patterns

• Basic Verification Pattern Class Diagram

28
Requirement and Verification Patterns

• Basic Verification Pattern Verification State
  Machine

29
Requirement and Verification Patterns

• Using the Basic Pattern Compounding
• Example REQ1
• (Q1) The active cruising shall be terminated upon
  occurrence of any of the following events
• (P1) Braking
• (P2) Manual Cancellation
• (P3) Cruise Control Off
• We can have
• Solution1
• REQ1 (P1 ? P2 ? P3) ? Q1
• solution2
• REQ1-1 P1 ? Q1
• REQ1-2 P2 ? Q1
• REQ1-3 P3 ? Q1

30
Requirement and Verification Patterns

• Key-Event Driven Requirement Pattern
• Upon the occurrence of a key event, a series of
  events (R1,,Rn) are expected to happen with the
  pre-specified order.
• Optionally, a timeout is associated with the
  pre-condition to require that all the expected
  events should happen before a timer expires.

31
Requirement and Verification Patterns

• Key-Event Driven Verification Pattern Class
  Diagram

32
Requirement and Verification Patterns

• Key-Event Driven Verification Pattern State
  Machine

33
Requirement and Verification Patterns

• Timed Key-Event Driven Requirement Pattern
• Requires a duration after the key event occurred.
  
• The pre-condition is a combination of the key
  event and the event P that occurred during the
  specified duration (t1).
• Normally stated as
• Within the duration t1 after the key event, if P
  then R is expected (before t2).

34
Requirement and Verification Patterns

• Timed Key-Event Driven Verification Pattern
  Class Diagram

35
Requirement and Verification Patterns

• Timed Key-Event Driven Verification Pattern
  State Machine

36
Requirement and Verification Patterns

• Key-Event Driven Time-Sliced Requirement Pattern
• After the key-event occurs, a pre-specified time
  interval is partitioned into many small slices
  (t1, t2, tn).
• Each of the post-conditions (Q1, , Qi, R1, ,
  Tm) of this pattern is expected to see in a
  certain interval.
• But there is no specific order for the results
  within each time slice, as long as they all
  finish before the end of the time slice.
• This pattern is commonly used to represent
  requirements that describe repeated or cyclical
  behaviors, or behaviors that fit into a fixed
  time sequence pattern.

37
Requirement and Verification Patterns

• Key-Event Driven Time-Sliced Verification Pattern
  Class diagram

38
Requirement and Verification Patterns

• Key-Event Driven Time-Sliced Verification Pattern
  State Machine

39
Requirement and Verification Patterns

• Command-Response Requirement Pattern
• Very common in embedded systems
• A command could be aborted.
• A response always follows the command.
• A command could be illegal or non-applicable.
• The response is different depending on both the
  command and the timing of the aborting action.

40
Requirement and Verification Patterns

• Command-Response Verification Pattern Class
  Diagram

41
Requirement and Verification Patterns

• Command-Response Verification Pattern
  Verification State Machine

42
Requirement and Verification Patterns

• Look-back Requirement pattern
• To verify causes when results are found
• To verify certain conditions were met before a
  key event occurs.
• The verifier has to look back and check the
  conditions involved.
• Typical statement of requirement
• Prior to event K, condition C1, C2, Cn must be
  met.

43
Requirement and Verification Patterns

• Look-back Verification Pattern Class Diagram

44
Requirement and Verification Patterns

• Look-back Verification Pattern State Machine

45
Requirement and Verification Patterns

• Verifying requirements using patterns

46
Requirement and Verification Patterns

• Pattern and the framework

47
Framework Evaluation

• Framework Effectiveness
• Effectiveness in Test development
• Effectiveness in Simplifying Debugging
• Easy to maintain
• Flatter learning curve
• Effectiveness in system instantiation

48
Framework Evaluation

• Framework Effectives

49
Framework Evaluation

• Framework Stability
• Architectural Stability of the Framework
• Domain Coverage

50
Framework Evaluation

• Handling Changes
• Adding Requirement
• Classify the requirement, or create a new
  requirement pattern
• Represent the requirement using appropriate
  pattern, or implement a special verifier for
  the requirement.
• Removing Requirement
• Delete it from the pattern verifier it belongs,
  or
• Remove its specialized verifier from the test
  list.
• Change in Requirement
• Timing change - Change the timing parameters in
  the requirement representation
• Sequence change - Change the sequence of the
  requirement representation, or Reclassify the
  changed requirement
• Action change - Update the requirement
  representation using events that represent the
  new actions. Normally the requirement item stays
  with the same requirement pattern.

51
Framework Evaluation

• Framework Applicability
• Was initially targeted on real-time embedded
  systems.
• Can be applied to other application domains such
  as business application or GUI application
  testing, as long as the testing is event-driven.
• Can be applied in a simulated testing environment
  instead of real hardware configuration.
• Not suitable for the testing of the continuous
  control systems or scientific calculation
  systems.
• Not suitable for testing of Internet applications
  or client/server applications as contents, not
  reactions, often make differences for these
  applications.

52
Framework Evaluation Case Study Results
53
Summary

• Framework Summary
• Rationale Requirement, test, verifier
• Architecture Layered hardware abstraction,
  support, application
• Mechanism Event assembling and dispatching
• Components Requirement, scenario, verifier,
  event assembler etc.
• Evaluation
• Effectiveness
• Stability
• Applicability

54
Summary

• Requirement centered, pattern-based requirement
  verification framework for real-time embedded
  systems.
• Different from any other existing test approaches
  and test frameworks, our verification framework
  makes the requirement an essential component.
• This basic idea is further developed and enhanced
  by requirement patterns and verification patterns.