ACCEPTANCE TESTS

1
ACCEPTANCE TESTS
Andrea Modigliani
2
OUTLINE

• Why/When
• Priorities
• Acceptance tests
• Experience feedback

3
Why/When

• Why
• Need to define a clear and compact list of
  requirements and tests to verify the compliance
  of DRL deliveries to PAE/COM1.
• Standardise and make transparent the review
  process.
• Possibly distribute the review across a long
  period.
• Ensure that the DRL not only meets scientific
  specs but also is easy to maintain, portable,
  well documented, efficient, robust.
• When
• Every three months to have at least 4-6
  iterations in the period FDR-PAE.
• We encourage the consortium to apply the tests
  during development.
• Tests are performed by ESO after a given DRL
  version has been certified/provided on an
  appropriate data set.

4
Priorities

• The next slides list several requirements
  organized by priority
• Priority 1 tests will be checked on every
  delivery,
• Priority 2 tests are performed from time to time,
  
• Priority 3 tests are verified at the end (PAE
  end of commissioning)together with priority 1
  and priority 2
• Each test is listed and a possible verification
  criteria is indicated in parenthesis,
• references to other presentations are indicated
  as Author

5
Compliance and initial verifications

• The acceptance test should be part of the PAE
  process (executed on DRL 0.5).
• Any action should to be completed before the last
  commissioning run (in DRL 1.0).
• See also R. Palsa, Y. Jung.
• Does the pipeline follow the recipe template Y.
  Jung? (static checks).
• Is the documentation in English? (manual
  inspection)
• Does the software conform to ESO coding standards
  R. Palsa? (manual inspection, compilation with
  no warning with strict compiler options).
• Does the pipeline use only CPL and agreed upon
  libraries C. Izzo, S. Castro ? (static checks).
• Does the code contain only CPL functions/objects?
  If something is missing, ask cpl-help_at_eso.org
  (compute ratio CPL/SLOC).
• Are exported function namespace protected?
  (static checks).

6
Execution tests to verify the delivery is
complete and installable/executable (1)

• Do the recipes exist in the expected version?
  (esorex recipes, esorex man-page)
• Is there a test package covering each combination
  of recipes and major instrument setting? (manual
  inspection, compare with DRL doc)
• Are test data appropriate to verify each recipe
  and DRL relevant function? (manual inspection,
  compare with DRL doc)
• Are recipes executable? (verify on provided test
  package)
• Do the recipes create the expected products S.
  Castro? (compare output versus DRL doc)
• Do the recipes create the QC they should P.
  Ballester, S. Castro? (compare output versus DRL
  doc)

7
Execution tests (2)

• Do the expected DRL functions exist? (nm
  libiiinstrument.so)
• Is there a unit test for each DRL function
  covering each instrument setting and parameter
  space L. Lundin? (manual inspection, compare
  with DRL doc)
• Are there memory leaks J. M. Larsen? (esorex
  mem-check recipe_name recipe_name.sof)
• Are there memory errors J.M. Larsen? (valgrind
  toolmemcheck)
• Is the recipe interface implemented in a standard
  way Y. Jung? (runtest.pl, test with Gasgano)
• Is the doxygen doc complete? (make html, check
  results)
• Do all source files contain addtogroup/defgroup?
  (static checks, make html)
• Are all recipes documented, including In/Out
  tags? (esorex man-page)

8
Detailed validation are results correct?

• Do recipes give correct results (FITS products
  QC parameters)? (run recipe on test data and
  check In/Out, compare with DRL doc)
• Do the interfaces fit together (is it possible to
  run a recipe cascade?)
• Are the final results (after running the cascade)
  correct? (manual inspection, compare with DRL
  doc)
• Does each recipe terminate in a reasonable time ?
  (exposure time/execution time gt1). To profile
  see J.M. Larsen
• Does the pipeline work and give results as
  defined in the DRL design doc? (manual
  inspection, compare results with DRL doc)

9
Detailed validation (2)

• Does the unit test implement the quality
  assessment defined in the DRL doc? (manual
  inspection, compare with DRL doc) L. Lundin
• Does the unit test check the basic validity of
  the function results (including QC parameters)
  and that the accuracy is as required? (manual
  inspection, compare with DRL doc)
• Does the unit test cover relevant parts of input
  parameter space, such as true/false for Boolean
  parameters? (manual inspection)
• Do the unit tests pass? (manual inspection, make
  check)
• Does the recipe verify it receives proper input
  (check for input TAGs)?
• Is the FITS format as described in the DRL doc
  (extensions/keywords)?
• Are the product valid FITS files? (fitsverify)
• Is the documentation and code understandable?
  (manual inspection)

10
Experience and feedback

• Communication is a crucial factor
• Acceptance tests should be part of the
  implementation schedule presented at FDR.
• Acceptance tests may involve additional work in
  the initial implementation phase but will save a
  lot of time for PAE and commissioning and help to
  provide a better product to the users.
• Acceptance tests are in addition to other tests
  which may be applied by the instrument SV team.