Image processing algorithm regression testing framework

1
Image processing algorithm regression testing
framework
Soumik Ukil
2
Testing Ground Rules

• Objectives
• Testing is the process of executing a program
  with the intent of finding errors
• A good test case is one that has a high
  probability of finding an error
• A successful test is one that uncovers an error
• However, testing cannot show the absence of
  defects

3

• Unit Testing
• Testing to determine that individual program
  modules perform to specification
• Basic units of software tested in isolation
• Regression Testing
• Selective retesting to detect faults introduced
  during modification of system
• Should be performed after any changes in software
  like bug fixes

4
Motivation

• Projects like NETT and BRP
• Complex algorithms which run on hundreds of
  datasets
• Source is being constantly modified to fix bugs
  etc
• Manual testing infeasible
• Need automated regression testing

5
Testing Strategies

• Dynamic Analysis
• Exercise the software in its execution
  environment
• Black Box
• Treat the system as black -box, ie no knowledge
  of internal structure
• Only need to know legal input and expected output
• White Box
• Depends on Internal structure of program
• Test all paths through code

6
Testing Strategies

• Static Analysis
• Code Reviews
• Walkthroughs
• Inspections

7
Tests on image data

• Gold-standard data used as basis for correctness
  of output
• Black box tests
• Already know 'correct' output
• Generate data from algorithm
• Define tests that compare the two sets of data

8
Tests on image data

• Generic Tests
• Check output is of correct datatype
• Check image dimensions
• Specific tests
• Depends on application being tested
• For airway tree validation we may want to compare
  distance between branchpoints
• For Lung Segmentation we need to compare volumes
  and/or distance between contours

9
Objectives of test framework

• Different applications
• Regression testing after any changes
• Validation of data
• Flexible
• Testers only plug in specific tests
• Test data generation and reporting of test
  results taken care by framework

10
Implementation

• Built on top of PyUnit
• Part of standard Python library for Python 2.1
  and later
• Based on JUnit, a proven testing architecture
• Allows creation of user defined tests,
  aggregation into suites, running tests in textual
  or GUI mode

11
Writing tests with PyUnit

• Basic building blocks called Test Cases
• Created by deriving from base class
  unittest.TestCase
• An instance of a TestCase class is an object that
  can completely run a single test method
• Fixtures
• A set-up and tear down method for each test case
• Many different test cases can use the same
  fixture
• Test Suites
• Test case instances can be grouped together
  according to the features they test
• All tests can be executed together as part of a
  suite
• Test Runner
• A class whose instances run tests and report
  results
• Can be used in text or GUI mode

12
Example

• class LungSegTestCase(unittest.TestCase)
• def initialize(self,fname1,fname2)
  
• A AnaFile.AnaFile()
  Anafile object for reading images
• self.data,hdrA.read(fname1)
  segmentation result
• self.ref_data,ref_hdr A.read(fname2)
  reference mask
• def setUp(self)
• self.labels20,30
  left and right lung labels
• def tearDown(self)
• self.data None
• self.ref_data None
• def test1(self) test method names
  begin 'test'
• """Test to see if mask image datatype
  is uint8."""
• self.assertEquals(self.data.typecode()
  ,'b')

13
Example

• class LungSegTestCase(unittest.TestCase)
• def initialize(self,fname1,fname2)
  
• A AnaFile.AnaFile()
  Anafile object for reading images
• self.data,hdrA.read(fname1)
  segmentation result
• self.ref_data,ref_hdr A.read(fname2)
  reference mask
• def setUp(self)
• self.labels20,30
  left and right lung labels
• def tearDown(self)
• self.data None
• self.ref_data None
• def test1(self) test method names
  begin 'test'
• """Test to see if mask image datatype
  is uint8."""
• self.assertEquals(self.data.typecode()
  ,'b')

14
Example

• class LungSegTestCase(unittest.TestCase)
• def initialize(self,fname1,fname2)
  
• A AnaFile.AnaFile()
  Anafile object for reading images
• self.data,hdrA.read(fname1)
  segmentation result
• self.ref_data,ref_hdr A.read(fname2)
  reference mask
• def setUp(self)
• self.labels20,30
  left and right lung labels
• def tearDown(self)
• self.data None
• self.ref_data None
• def test1(self) test method names
  begin 'test'
• """Test to see if mask image datatype
  is uint8."""
• self.assertEquals(self.data.typecode()
  ,'b')

15
Application dependent tests

• For lung segmentation we use an area overlap
  measure to test correctness
• Check that value is above a given threshold to
  'pass' test
• Similar tests can be defined with distance
  measures between contours
• User has to plug-in appropriate tests

16
Flow diagram
17
Sample configuration file

• DEFAULT
• Directory of gold-standard data
• Refdir ../ValidationData
• Directory of data generated by algorithm
• Maskdir ../NewData
• Output file extension
• MaskExt mask.img
• Name of logfile directory
• Logdir ../logfiles
• ALGORITHM
• Dir. of raw image data to run algorithm on
• Inputdir /home/xyz/lung/Data
• Path to executable for algorithm
• Execpath ../lung

18
Reporting test results

• Textual output directed to log files
• Success
• --------------------------------------------------
  -------------------
• Ran 6 tests in 88.337s
• OK
• Failure
• FAIL Left lung pixel count test (slice by
  slice).
• --------------------------------------------------
  --------------------
• Traceback (most recent call last)
• File "lungsegtests.py", line 101, in test6
• self.failIf(fail1,errormsg)
• File "unittest.py", line 264, in failIf
• if expr raise self.failureException, msg
• AssertionError significant segmentation mismatch
  for left lung on slices
• 458, 462

19
Status

• Testing for Lung segmentation/smoothing has been
  implemented with the following tests
• Type checking of output masks
• Checking that all expected labels are present
• Slice by slice area comparison for both lungs
• Can be used by segmentation algorithms on ANALYZE
  images which produce labeled masks
• Framework can be extended to handle other formats
  like airway tree definitions etc

20
References

• http//pyunit.sourceforge.net
• http//www.xprogramming.com/testfram.htm

Documentation and source code
CVS repository pulmonary/TestFrameWork