Metrics for Timed Systems Testing

1
Metrics for Timed Systems Testing

• S. Salva, H. Fouchal, S. Bloch
• LERI/RESYCOM
• University of Reims FRANCE

2
Outline
I Conformance Testing II Timed Automata III Test
quality IV Conclusion
3
Need for Validation

• Systems more and more complex
• Timing constraints are important in critical
  systems
• Need for validation before industrial development
• Beta Test
• Formal validation techniques

4
System validation

• Specification verification
• White box
• Conformance testing
• Spec behavior implementation behavior
  (conformance relation)
• Black box
• Stimuli sending (INPUTS) and observation
  reception (OUTPUT)
• Other testing techniques reliability,
  performance,

5
Software life cycle
6
Modeling
Specs in a high level language LOTOS, ESTELLE,
LDS, Petri nets ...
Automata (I/OFSM, TIOA, )
Wrong pin / refuse
Client Wait
Transaction ask
Insert card / accepted
Ex ATM
Right pin/ welcom
end / card retrieve
Wait for request
verification
retreive/ wait
Total request?/ result
epsilon / accepted
7
testing methodology
Specification
Tester
test sequence
Model
Conforme ?
I/OFSM model
Implementation
Properties for characterization of system parts
Test derivation method (UIO W)
8
Testing Architecture

• IS ISO 9646, a testing architecture composed of
• A tester
• An implementation under test (IUT)
• A test environment
• Points of control and observation (PCO)
• Access points to the à implementation

9
Testability

• " The ability to evaluate, to generate and to
  apply tests to the implementation in order to
  check if this later is conforme to the
  specification" (Bennetts)
• ? all of the defaults should be found
• the test process should be done in the best
  condiftions
• the test has not be blocked
• ? testability degrees for the evalution of the
  test costs
• Testability observable, controlable, timed
  covered

10
Software cycle and testability
Informel functionnal needs
Testability factors
Testable functionnal needs
Functionnal specifications
Testability factors on verified specification
Testable specification
Minimal sequences et maximum fault coverage
Test sequences
11
Testabilité des intéractions
Observability
Specification
1 action
1 or more actions
Many factors for the testability evaluation
depending the observation and the controle
12
The timed automata model
I8 / O1
x0 y0 x ? 1
x0

• dense time(Alur et Dill )
• addiction of clocks (which could be reset)
• clock constraints equations

I3 / e
x0 y0
Idle
Dial. pending
I2 / e
I14 / O8
WUFR
x0
I4 / O3
I1 / O1
I7 / e
y0 x lt 1
x0 y0 x gt 3
I10 / O5
X ? 3
Dial. accepted
Dial. established
I11 / O6
y gt 2
The MAP-DSM protocol part with timed constraints
13
Timed testing methodology
Specification
Implementation
Model
Tester
Clocks

• Sending of clock region bounds
• Checking of the reception time

Test sequences
TIOA
Timed test sequence derivation
Region Graph
Conformance ?
14
Timed System testability 1

• Timed constraints analysis
• Bad properties recognition
• The definition of new testability parameters
• Avoid all of the blocking aspects of the testing
  process
• Avoid all the non conclusive verdicts
• Complete coverage of the system
• Evauate the time required for the testing process

15
Timed System testability 2

• Inifinite time intervals
• The tests should be bound ? Intervals also
• Implementation partialy tested
• Partial fault coverage
• Time reachability
• Implementation partialy tested
• Partial fault coverage

!A
!B
!B cannot be executed
4.1
3
5
3
4
16
Region graph testability 1
Region reachability
Infinity
Infinite regions

• Regions which are not reached by some clocks will
  not be tested
• Partial behavior, low fault coverage
• Reachability degree

• R is infinite There is one or more non bouded
  clocks
• Behavior partialy tested, low fault coverage
• Infinity degree of the region

17
Region graph testability 2
Time Dependancy
Controlability
Y
?A
C
D
wait 0.5s ?A
?A
2
X1 X1.5 X2
A
B
wait 1s ?A
X
Test sequence ? Execution tree
1
3

• Testing of at least one bound of the region
• Depending on the regionn bound
• Time dependency degree

• Maximal test cost (testing time)
• Controlabilitity degree

18
industry testing

• Some softwares
• Geode, Attol (Verilog)
• Kronos (Verimag)
• Spin (Bell labs)
• Industry
• avionics (Dassault)
• Transport (RATP, SNCF)
• Telecommunication (Alcatel, France telecom,
  Motorola)
• Car industry (Daimler Benz)

19
Conclusion

• Study of the timed automata model
• suggestion of some test quality parameters