Aiko Fallas Yamashita

1
Experiences from testing a radiotherapy support
system with QuickCheck
Aiko Fallas Yamashita Simula Research
Laboratory, Norway Andreas Bergqvist, Thomas
Arts IT University of Göteborg, Sweden TAP
2008 April 10, 2008
2
This talk traces the results from using
QuickCheck for testing a radiotherapy support
system
Current practices on medical devices verification
and certification
http//www.hummingbird.com/connectivity/certificat
ion.html
Radiotherapy Support System overview and context
www.micropos.se
Property-based testing and QuickCheck Lessons
learned
3
The industry is using process-centered
verification and certification
Process centered verification is often described
by standards Wallace 1992,Bowen Stavridou
1993
IEEE 1073 and ISO 9001 are some standards
applicable to medical devices


Standards Guidelines of how to verify software
and get certified
4
Well established guidelines for artifact-centered
verification are needed

www.meyerinst.com/html/mediacy/imagepro.htm
General guidelines for processes are insufficient
for delivering a safe product Jetley
2006 Process can be faked Niinimäki
Forsström, 1997 Certification is leaning
towards evidence-based approaches Lee 2006
Artifact Centered
Process Centered
vs
Viewpoints on Software Verification Jones, 2005
5
Possible approach Verification of critical part
through formalization and testing

• We concentrate on a critical property, formalize
  it and generate a test for it
• We still need to test, even if model is proven
  correct!
• In our context, complete formalization may
• Require special skills (usually not available in
  the industry)
• Agerhorm Larsen, 1998
• Only parts of the system will benefit from it
• Agerhorm Larsen, 1998

Module
6
Radio frequency based system for positioning
organs in real-time
Implantable device emits radio signaling
Supports patient positioning before/during
treatment
Radiotherapy for prostate
Signal is captured by receivers located in the
treatment table
Consists of implantable transmitter, receivers
and software
Purpose is to reduce the treatment area by
providing real-time feedback of the organ position
7
Radio frequency based system for positioning
organs in real-time
8
The system has heterogeneous components in
continuous evolution
Interfaces to external systems may change
Model
Coordinates
signals
Position estimator
Hardware is always improving
Underlying mathematical model (reference points)
is under continuous improvement
9
System level testing approach was used
Transmitter can be positioned to specific
coordinates
System level property to verify Radiation hits
the antenna (prostate). Antenna position is
accurately computed in all possible positions of
the patient
www.itpgroup.co.uk
Receiver is placed in relation to the
CMM/Autosetup
10
We focused on one of the critical features
Accuracy
The software component should calculate the 3D
positioning of the transmitter (X, Y, Z)
Functional requirement
Non Functional requirement
The system needs to be accurate
This is expressed in terms of Radial Accuracy
(Euclidian distance)
The system should achieve radial accuracy of 2 1
mm
11
Property-based testing approach relies on a clear
mathematical model
The system should achieve radial accuracy of 2
1 mm
12
QuickCheck enables automation of testing tasks
alongside with formalization
Formula for correctness
It is easy to inspect that the right aspect of
the system has been tested The testing supports
evidence-based certification
modeled as a property
QuickCheck
Failed Test cases simplification
Random test cases generation
13
QuickCheck was connected to the SUT by using a
request-broker
CMM / Autosetup
transmitter
QuickCheck
Broker
Auto Setup
receivers
Position Estimator
Receivers HW controller
Position Estimator
signal
SUT
14
Results
15
Accuracy of the positioning system

• One Sample
• Mean Radial Distance 1.528505mm
• Standard deviation 0.477921,
• 87 passed test cases
• 13 failed test cases
• 11 (2mm 2.4mm)
• 2 (2.4mm 3.4mm)
• Others samples
• 2 failed test cases (2.02mm 2.04mm)

Accuracy much better than the specification
16
Errors found with the test cases generated from
the property

• Errors due to supported types on interfaces
• Autosetup didnt accept decimal points
• Conversion differences between platforms
• LabView handles conversion implicitly, whereas
  .NET C doesnt
• Errors due to the migration between platforms
• Definition of global variables may differ between
  platform

17
QuickCheck also supported

• Quality of underlying models
• Inadequate models were detected
• Communication protocol issues detection
• Request timing issues between Autosetup
  controller and C broker were detected
• Detection of irregular cases
• Some models were found to be sensitive to angling

18
The combined approach for testing and
light-weight methods provides cost-effective
results
Casting operation issues resulted from
cross-platform differences
Errors resulting from the migration process
One property of QuickCheck could detect
Floating point operations, type conversion, and
the use of erroneous types in the drivers'
interfaces
Most of the failures were detected in first test
cases QuickCheck produced
19
QuickChecks results under the presented context
were positive

• Benefits
• Regression testing alongside the development
  process
• Support for certification
• Cost effectiveness / Simplicity
• Support for detection of atypical faults

Work in Progress Usage of normal
distribution Usage of real patient data
Simplification based on test results Testing of
confidence intervals Cover more properties of the
system
20
First attempt of our approach is promising
Addresses key aspects in current medical software
companies Supports cost-effective means for
certification Realistic approach for high-pace
and experimental development
Questions?
21
Discussion Commonalities between Micropos AB and
other medical software practitioners
Start-up company
People
Multidisciplinary team and small software
development unit
Experimental and Commercial versions are being
developed
Technology
Product concept is relatively new
On-going certification process (MDD and FDA)
22
Discussion Which are the values for the industry?
Enables to evaluate the quality of the underlying
model It has potential for integration
testing To be used as a tool for on-site
calibration To be used as evidence for
certification process
23
References
Agerhorm Larsen, 1998 Sten Agerholm and
Peter Gorm Larsen. A Lightweight Approach to
Formal Methods. In Submitted to FM-Trends,
Boppand, Germany, October 1998. Niinimäki
Forsström, 1997 Jaakko Niinimäki and Jari
Forsström, Approaches for certification of
electronic prescription software, International
Journal of Medical Informatics Volume 47, Issue
3, December 1997, pp 175-182. Rest of references
are present in the paper