Deriving%20State-Based%20Test%20Oracles%20for%20Conformance%20Testing

1
Deriving State-Based Test Oracles for Conformance
Testing

• Jamie Andrews
• Associate Professor
• Department of Computer Science
• University of Western Ontario
• London, Ontario

2
Plan for Talk

• Testing background
• Log file analysis (LFA)
• Process for developing artifacts
• Refinement of process

3
Testing Background

• 3 main tasks to testing
• Selecting test inputs
• Running tests
• Checking test outputs
• Checking test output not always trivial

Test Input
Software Under Test
Test Output
4
Checking Test Output

• May be too complex to check visually
• May be legitimately different from output of
  previous version

Test Input
Software Under Test
Test Output
5
Test Oracles
Test Input

• Programs that check the output of other programs

Software Under Test
Test Output
Test Oracle
Pass/Fail
6
Test Oracles
Test Input

• Input and output may be difficult to capture
• Oracle may have to parse complex I/O

Software Under Test
Test Output
Test Oracle
Pass/Fail
7
Log File Analysis (LFA)
Test Input

• Log file simple text output
• LFA dynamic analysis for conformance checking

Software Under Test
Log File
Test Output
Log File Analyzer
Pass/Fail
8
LFA Challenges

• Make sure logging doesnt slow down system too
  much
• Can send logging data to another machine
• Decide on logging policy for software
• Write log file analyzer program
• Special-purpose state machine-based language to
  help with this

9
Log File Analysis Language (LFAL)

• Analyzer collection of state machines
• Each machine notices some log file lines, ignores
  others
• Log file lines trigger transitions
• Machine reports error if it
• Notices a line
• Does not have a legal transition on it

10
Processes

• Need a process for getting
• From requirements
• To logging instrumentation, analyzer
• Big-step process
• Used and taught to students
• Small-step process
• Suggested refinement

11
Example Elevator System

• Requirement to check
• The doors are never open when the elevator is in
  motion.

12
Big-Step Process
Requirements
Situations with Permitted and Forbidden Events
SPFEs
Log File Analyzer Program
Logging Policy
13
Example SPFEs

• SPFE1
• Situation Elevator door is open
• Permitted event Door closes
• Forbidden event Elevator starts moving
• SPFE2
• Situation Elevator is moving
• Permitted event Elevator stops moving
• Forbidden event Door opens

14
Relationships

• SPFEs should re-express requirements to be
  checked
• Logging policy should specify that we log all
  events that allow us to determine
• Whether we are in each Situation
• Whether each Permitted/Forbidden Event has
  happened

15
Example Logging Policy

• Log all door open / close events in the form
  door_open, door_close
• Log all elevator move / stop events in the form
  start_move, stop
• Can instrument code based on this

16
From SPFEs to Analyzer

• Situations correspond to states
• Permitted events correspond to transitions
• Forbidden events should not have any
  corresponding transition

17
Example LFAL Analyzer
moving
start_move
closed_stopped
stop
door_open
open
door_close
machine door_safety initial_state
closed_stopped from closed_stopped, on
start_move, to moving from moving, on stop, to
closed_stopped from closed_stopped, on
door_open, to open from open, on door_close,
to closed_stopped final_state Any.
18
Some Past Projects

• Steam boiler simulator and analyzer (X. An)
• WAP client development and testing (V. Liu)
• 5 KLOC 3 protocol layers verified
• Test case generation from oracles (R. Fu)

19
Problems with Big-Step Process

• Not always explicit
• Which requirements are to be checked
• Under what assumptions/conditions
• Sometimes abstract events mentioned in SPFEs
  cannot be directly logged
• e.g. door open event may actually correspond to
  send release command to door lock actuator
• Need more concrete, loggable events

20
Small-Step Process

Checking Assumptions
Requirements to be Checked
Abstract Events
Concrete Events
SPFEs
Log File Analyzer Program
Logging Policy
21
Future Work

• Case studies of small-step process
• Teaching small-step process
• Experiments

22
Potential Benefits and Problems

• Benefits improved reliability, flexibility,
  scalability, traceability
• Problems
• False negatives and positives
• Instrumentation maintenance
• Process weight

23
Summary

• LFA is a method for test result checking
• We have used and taught a process for applying it
• We propose a refined process for future work

24
Frequently Asked Questions

• Do we have to discard regression testing?
• No can use as a complement
• How do we know what to put in log file?
• Recommend identify reqs to check with LFA first
• Develop logging policy, log file analyzer from
  those
• Efficiency? Scalability?
• A few years ago processed 1000 lines/sec
• Biggest analyzer 1200 NLOC, from 10-page spec
• Recommend starting small, seeing if works for you