Conformance Testing of Interoperability

1
Conformance Testing of Interoperability

• in Health Information Systems in Finland
• Medinfo 2007, Brisbane
• Wednesday, 22 August, Session S103, 11 AM
• Tanja Toroia, Anne Eerolaa, Juha Mykkänenb
• University of Kuopio, Finland
• a Department of Computer Science
• b Health Information System (HIS) RD Unit

2
in this presentation

• Introduction and background
• Survey of conformance testing of interface
  specifications in healthcare applications in
  Finland
• Findings of the survey
• Recommendations based on the results of the survey

3
introductionconformance testing of interfaces

• Conformance testing testing to determine
  whether a system meets a specification or not
• Interface (standard) specifications are central
  for interoperability
• Conformance testing of interfaces includes
• conformity of applications and interfaces to
  standards
• testing of the interfaces between applications
• Does not cover testing of the non
  interoperability-specific features of the
  applications

4
background of the study

• In most healthcare facilities, interoperability
  and safety require attention on cross-application
  level
• Integration, distribution, security and
  interoperability emphasized lately in software
  testing
• Testing related to interoperability
• focus on deployment and integration phases of
  systems lifecycle
• standards compliance
• certification
• Conformance to specifications a key area in
  interoperability
• However, few / no empirical studies on the level
  of conformance testing in software companies in
  healthcare and in health service provider
  organizations
• Survey how much and how often conformance
  testing is performed in relation to the interface
  specifications in healthcare applications

5
main research questions

• How can customers, i.e. hospital districts,
  influence the conformance of the software they
  acquire?
• How much conformance testing do software vendors
  perform and how often are standards used?
• What are the main goals of, problems with, and
  opinions about external interface testing
  services?

6
the survey

• Themes
• General software testing practices
• Conformance testing and standards
• The organization of interface testing services
• Web-based questionnaires with open and structured
  questions
• The questionnaires are shown on
• http//www.cs.uku.fi/ttoroi/questionnaire.html

Topic in this talk
7
sample and responses

• Healthcare software vendors in Finland
• 27 software companies which belong to a Health
  and Welfare IT RD cluster in Finland (response
  rate 52)
• responses focused on companies producing EPR and
  departmental systems consultancy services
• Customers of the software vendors
• 20 Finnish hospital districts, all mainland
  public healthcare districts (response rate 75)
• respondents mostly from specialized care
  facilities

8
results - Conformance testing and standards

• Term 'conformance testing' was misunderstood by
  two respondents
• 1 - 4 new versions each year for most
  applications
• Interoperability standards and open interfaces
  used widely
• Conformance testing performed variedly for entire
  applications and between specific systems
• Demand for the use of UML models and XML in
  standards
• Reported problems
• In some situations, no choice but to accept
  evident errors in applications or new versions
• testing among the first areas to be "cut down" in
  some companies
• often difficult to apply and interpret standards,
  also national specifics and laborious
  implementation hinder the adoption

9
results - Organization of interface testing
services

• Suspiciousness of external interface testing
  services, lack of knowledge about such services
• Interoperability challenges not focused in
  testing
• "testing services cannot help in integration"
• Certification of application interfaces was not
  considered very necessary nor useful
• Respondents were not familiar with the
  international developments of interoperability
  testing or certification
• e.g. IHE Connectathons, CCHIT, European
  certification initiatives

10
discussion and recommendations 1

• Perform interoperability conformance testing more
  rigorously
• surprisingly missing link between quality
  requirements and testing, more time for testing
  and various conformance testing activities
  required for diverse requirements
• Utilize open interfaces and use interface testing
  services
• open interfaces are steadily increasing,
  promoting cost effectiveness in general and in
  testing activities
• Provide proper skills and knowledge
• needed education about the benefits of
  standards-based interfaces, common definitions
  and policies for interoperability testing

11
discussion and recommendations 2

• Enforce the recommendations by authorities
• respondents were in favor of official
  recommendations in relation to interoperability,
  security and usability
• Organize neutral testing services
• little market possibilites for commercial testing
  services
• possibilities automation of conformance testing
• neutral services / national mandates need further
  exploration
• Reuse testing experiences and information
• some encouraging sharing of experiences from
  system introductions, could be extended to expert
  groups
• Improve the testability of specifications and
  standards
• conformance requires accuracy and quality from
  specifications

12
concluding remarks

• Results can not be directly generalized
• Standards perceived necessary but burdensome
• Conformance testing hindered by attitude
  problems and lack of knowledge
• Improvements needed in relation to demand,
  investments and standardization
• But
• Interoperability testing is already part of
  common policies
• National / large-scale initiatives draw attention
  to interoperability and shared requirements,
  solutions and services
• Further discussions and developments encouraging
  (national healthcare IT services and related
  certification, interest and participation to
  interoperability testing on international level)
• first author Tanja.Toroi_at_cs.uku.fi
• speaker Juha.Mykkanen_at_uku.fi

13
other results from the SerAPI project in
Medinfo'07

• Model-Centric Approaches for the Development of
  Health Information Systems (Tuomainen et al.)
• A National Study of eHealth Standardization in
  Finland - Goals and Recommendations (Mykkänen et
  al.)

14
THANK YOU
Tanja Toroi, tanja.toroi_at_uku.fi, Anne
Eerola, Juha Mykkänen, juha.mykkanen_at_uku.fi,
This work is part of the SerAPI and AVOINTA
projects www.serapi.fi/ www.centek.fi/avointa/ Pr
ojects are funded by the Finnish funding agency
for Technology and Innovation TEKES together with
a group of hospital districts / cities / joint
municipal boards for healthcare and companies.