PT and ILC Ankara

1
PT and ILC - Ankara

• Proficiency testing and inter laboratory
  comparisons are one of the powerful tools in
  laboratory quality control.
• You get better knowledge according to the quality
  of laboratory services.
• You get improved possibility to detect errors in
  laboratory testing.
• You shall be able to to some extend to
  determine the actual quality of laboratory
  performance.
• You must participate in PTs and or ILCs to be
  able to achieve accreditation.
• You get based on results ideas for
  improvements.

2
PT and ILC - Ankara

• Definitions and examples according to the use of
  PTs and ILCs
• PT Proficiency testing is the use of inter
  laboratory comparisons with the purpose to
  determine the performance of laboratory tests and
  calibrations (measurements).
• ILC Inter Laboratory Comparison - definition
  Organisation, performance and evaluation of test
  on the same or similar test items.
• ILC examples for calibration-laboratories
  electricity meter (the same meter) sent to a set
  of laboratories the similar for weight,
  flowmeter and other instruments.

3
PT and ILC - Ankara

• ILC - examples for test-laboratories within
  analytical chemistry
• In this case it is normally impossible to use the
  same test-item. The challenge is in this case to
  produce similar test-items as representatively/a
  s accurate as possible.
• In some cases it is practically impossible to
  produse test-material in the form of samples
  generally seen.

4
PT and ILC - Ankara

• Accreditation of providers of PTs
• - documents
• ISO 17025
• ISO Guide 43 PT by interlaboratory comparisons.
• ILAC G13 Guidelines for the requirements for the
  competence of providers of PT schemes.

5
PT and ILC - Ankara

• ISO 17025
• Definitions
• Provider Organisation designing and performing
  PT-programmes.
• Co-operating partner Organisation undertaking
  tasks for the provider.
• Co-ordinator Person responsible for the
  co-ordination of all activities connected to a
  PT-programme.
• PT-programme PT designed and performed with the
  purpose to ensure the quality of the laboratory
  performance within a specified domain (testing,
  measuring or calibration).

6
PT and ILC - Ankara

• Requirements for the management of the
  quality-system.
• Purpose
• Domain
• Statistical design of the PT-programme
• Co-operating partners
• Subcontractors
• Participation fee

7
PT and ILC - Ankara

• Requirements for the management of the quality
  system
• Use of co-operating partners and subcontractors.
• The provider shall ensure, that the requirements
  according to ILAC G13 are met. For instans
  accreditation according to ISO 17025.

8
PT and ILC - Ankara

• Requirements for the management of the quality
  system.
• Registrations
• All measurements, observations and other relevant
  material statistical data-processing.
• All results from participants and reports (as
  well drafts as final reports) for a defined
  period of time ensuring that the specific task
  is completed totally.

9
PT and ILC - Ankara

• Technical requirements personnel
• The management must define the minimum demands
  for qualifications and working experience.
• Meet the requirements according to ISO 17025
  test and/or calibration.
• Ensure the competence of co-operating partners
  and subcontractors (for instans informations
  according to accreditation) - personnel
  equipment methods experience.
• Registrations according to the mentioned items.

10
PT and ILC - Ankara

• Technical requirements Organisation and
  logistics
• Planning
• Identification of processes, which have an impact
  on the quality of the PT.
• Registration of relevant informations concerning
  the PT.
• Contributions from co-operating partners must be
  identified, be documented and evaluated with an
  appropiate regularity

11
PT and ILC - Ankara

• Technical requirements - Organisation and
  logistics
• Planning (continued)
• Formation of a socalled advisory group formed by
  technical specialists and persons with sufficient
  competences within statistics with reference to
  the co-ordinator.
• The advisory group must have specified its
  responsibility for the planning of the actual PT
  and for the choise of material/equipment as for
  the data-processing and reportation.

12
PT and ILC - Ankara

• Technical requirements organisation and
  logistics
• Manufacturing of material/equipment
• Selection and handling of material/equipment.
• Measurement/testing and validation of used
  equipment and methods with special attention in
  case co-operating partners or subcontractors ar
  used.
• Checks of stability and homogenicity of material.
• Packing, storage, labelling, transport and
  distribution.
• Statistical processing of data.
• Reporting.

13
PT and ILC - Ankara

• Technical requirements organisation and
  logistics
• Stability and homogenicity
• Documented procedures for the selection and
  handling of material, which is going to be used
  to prove/describe homogenicity and stability,
  which must be documented.
• The found homogenicity and stability must meet
  the requirements for the use of the PT.

14
PT and ILC - Ankara

• Technical requirements organisation and
  logistics
• Statistical design must take into consideration
• Accuracy and precision.
• Differences which are measurable.
• The number of participants number of samples
  and performed tests/measurements.
• Assignment of true value and identification of
  outliers.
• Characterisation of homogenicity and stability,
  if this influences the final results.

15
PT and ILC - Ankara

• Technical requirements
• Choise of method
• Description of the possibilities for the
  participants to choose method and after that
  registration of the particular choise.

16
PT and ILC - Ankara

• Technical requirements
• Performance
• Instructions
• Detailed and documented instructions containing
  all informations with influence on the planned
  test including storage.
• Instructions according to the reporting of
  results.
• Information saying, that the testing-performance
  must be according to the daily routines, if
  possible.
• No information must be given concerning true
  value.

17
PT and ILC - Ankara

• Technical requirements
• Performance
• Handling of samples material/equipment
• Facilities to separate and store
  test-material/test-equipment in an appropiate
  manner. Monitoring those facilities.
• Facilities for packaging and mailing/transportatio
  n.
• Supervision of storage-facilities

18
PT and ILC - Ankara

• Technical requirements
• Handling of data and interpretation of results
• Appointment af person responsible for the
  handling of data.
• Documented statistical procedure.
• Minimisation of the influence of results
  evaluated as extreme.

19
PT and ILC - Ankara

• Technical requirements
• Handling of data and interpretation of results
• Evaluation of results.
• Specified method for the evaluation, so that the
  credability of the PT is kept.
• Results are evaluated of competent personnel
  according to Variations between samples and
  laboratories, variations between methods,
  variations as a function of time.
• Possible sources of error suggestions
  recommendations.
• Reporting according to agreed time-schedule.

20
PT and ILC - Ankara

• Technical requirements
• Communication
• Detailed information concerning the programme for
  the PT, how to participate, professional domain,
  fee and reporting results.
• Eventual changes are communicated to the
  participants.
• Procedure for handling approaches from
  participants in different matters especially if
  disagreement according to the interpretation of
  results occur.

21
PT and ILC - Ankara

• Technical requirements
• Swindle (fiddle) falcification.
• Programme/procedure to avoid possibilities to
  swindle, to fiddle or even to report false
  results.

22
PT and ILC - Ankara

• Requirements
• The future
• Revision of ILAC G13
• New standard based on ILAC G13 and ISO Guide 43
  in the 17000-serie.

23
PT and ILC - AnkaraI

• ILAC-G132000
• Guidelines for the requirements for the comptence
  of providers of proficiency testing schemes.

24
PT and ILC - Ankara

• ILAC-G132000
• Definitions
• As ISO 17025 plus
• Proficiency testing scheme Interlaboratory
  comparisons designed and operated to assure
  laboratory performance in specified areas of
  testing, calibration and measurement. (A scheme
  might cover a particular type of test or a number
  of tests on paticular products, items or
  materials).
• Proficiency testing round A single operation of
  a proficiency testing scheme.

25
PT and ILC - Ankara

• ILAC-G132000
• Requirements for the management of the system.
• ILAC-G132000 has mainly the same requirements as
  ISO 17025. In the following is stated items,
  where the requirements differ in some way or the
  other.
• The documented quality management system shall
  specify, which activities are undertaken by the
  provider and where relevant, which activities are
  undertaken by collaborators and shall include
  policies and procedures used by the provider to
  ensure, that all activities conducted by
  collaborators comply with the relevant clauses of
  these Guidelines.

26
PT and ILC - Ankara

• ILAC-G132000
• Cordinator requirements (2.2.2)
• Participants choise of method (3.4.1)
• Requested details of method (3.4.2)
• Disclosure of assigned values (3.5.1.6)
• Tests for outliers (3.6.1.6)
• Criteria for non-conforming data (3.6.1.7)
• Evaluation and technical comments on laboratory
  performance (3.6.2.1 ans 3.6.2.2)
• Confidientality of participant identity (3.8.1)
• Confidentiality of test data etc. (3.8.2)

27
PT and ILC - Ankara

• ISO GUIDE 43-1 and ISO GUIDE 43-2
• ISO Guide 43-1. Development and operation of
  proficiency testing schemes. (not covered during
  TR5 except the informative annex A).
• ISO Guide 43-2. Selection and use of proficiency
  testing schemes by laboratory accrditation
  bodies.

28
PT and ILC - Ankara

• ISO Guide 43-2.
• Policies on participation in proficiency testing
  schemes.
• Laboratory accreditation bodies should document
  their policies for participation in proficiency
  testing schemes.
• Whether participation is mandatory or voluntary
  for specific proficiency testing schemes.
• The frequency of which laboratories are expected
  to participate.
• Criteria used by the laboratory accreditation
  body to evaluate the laboratory performance.
• Use of results from PTs in accreditation
  descissions.
• Preservation of PT participants confidentiality.

29
PT and ILC - Ankara

• Action and feedback by laboratories.
• Maintenance of laboratory records showing
  performance.
• Conclusions based on evaluation of performance
  knowing the organisation and the type of PT.

30
PT and ILC - Ankara

• ISO guide 43-2. Annex A.
• Calculation of performance statistics.
  Quantitative results.
• Difference x X. X is the assigned value.
• Percent diference (x X) 100/X
• Percentil or rank.
• Z scores, where z (x X)/s, where s is an
  appropiate estimate/measure of variability.
• En numbers

31
PT and ILC - Ankara

• ISO Guide 43-2.
• Evaluation of performance.
• Are the found results fit for purpose?
• Statistical determination for scores
• For z scores
  z lt 2
  satisfactory, 2 lt z lt 3 questionable,
  z gt 3 unsatisfactory.
• For En numbers
  En lt 1
  satisfactory, En gt 1 unsatisfactory.
• Consensous of participants.

32
PT and ILC - Ankara

• Use of PTs and ILCs
• PTs and ILCs are a very powerful tool to
  improve quality and to detect weeknesses for
  methods in use. Furthermore PTs and ILCs give
  valuble information concerning uncertainty.

33
PT and ILC - Ankara

• Types of PTs/ILCs.
• Bilateral comparisons
• 2 5 participants
• Within testing simultaneous performance.
• Within calibration selection of special
  suitable unit.
• Achieved results are compared taking the
  uncertainty into consideration.
• Usefull when official PTs/ILCs are not
  available.
• Most suitable between participants with
  comparable background and instrumentation.
• It is advisable and an advantage, if one of the
  participants has a higher measurement capability
  or is a reference-laboratory.

34
PT and ILC - Ankara

• Bilateral comparisons.
• Examples.
• Emission (airborne) measurements
• Dust (airborne)
• Sampling (water air)
• Calibration

35
PT and ILC - Ankara

• Star-comparisons
• A large number of participants
• Same sample to all participants.
• Collected data are compared using statistical
  methods. Results are reported as for instans
  En-values or z-scores.
• En-values are normally used within calibration.
• Z-scores are normally used within testing.
• Microbiological samples are mainly seen as
  qualitative or semi-quantitative.

36
PT and ILC - Ankara

• Star comparisons
• Examples for the use of
• Microbiology
• Chemistry (organics in water).
• Food (organic pollutants).
• Fysical/mechanical testing

37
PT and ILC - Ankara

• Round robins
• Many partipants are possible, if the
  sample/test-unit is stable.
• Sample/test-unit is sent to participants
  according to a specified plan.
• Results are evaluated using statistical methods.

38
PT and ILC - Ankara

• Round Robins
• Examples
• Calibration of flowmeters
• Calibration fo electrical measures
• Gasanalysis
• Non-destructive fysical/mechanical testing.

39
PT and ILC - Ankara

• Experiences from DANAK (calibration)

40
PT and ILC - Ankara

• What can go wrong shall go wrong.
• Examples

41
PT and ILC - Ankara

• The distributed Pt-sensors did break during
  mailing to the participants.
• Laboratories with En-values larger than 1 needed
  the measurement-equipment for renewed
  calibration. During the following transport the
  measurement-equipment was damaged, so that
  repeated measurement was impossible.
• Measurement-equipment leaving Denmark did to a
  high extend disappear in the international
  mail-machine

42
PT and ILC - Ankara

• Which kind of equipment shall be chosen for the
  purpose ?

43
PT and ILC - Ankara

• First point choose an equipment which is stable.

44
PT and ILC - Ankara

• Second point choose an equipment which is stable.

45
PT and ILC - Ankara

• What do you believe is the third point ?
• Yes
• Choose an item which is stable.

46
PT and ILC - Ankara

• If the item is not stable
• The unstability/uncertainty of the item is
  contributing so much to the total uncertainty,
  that comparisons are impossible. This situation
  is seen using a magnetic inductiv flow-meter.
• The same situation is seen comparing results
  after calibration of class 2 electricity meter
  when success was achieved using class 0.2
  electricity-meter

47
PT and ILC - Ankara

• Futhermore you must remember
• The item must fit to the measurement capability
  of the laboratory.
• If the measurement capability of the laboratory
  is better than the uncertainty of the reference,
  the outcome of the comparison shall be very
  limited.

48
PT and ILC - Ankara

• I shall now show some overheads.

49
PT and ILC - Ankara
50
PT and ILC - Ankara

• PTs within chemistry.

51
PT and ILC - Ankara

• Traceability.
• SI-units are often not possible to use.
• Method-dependency
• Referencematerials

52
PT and ILC - Ankara

• Purpose
• Establisment of traceability
• Evaluation of conformity between laboratories.
• Evaluation of the quality of the analytical
  performance.
• Selection of approved laboratories.
• Support according to the evaluation of accredited
  laboratories.

53
PT and ILC - Ankara

• International PTs.
• Establishment of traceability.
• Evaluation of conformity between laboratories .
• Support (hopefully !!!!) according to mutual
  recognition.

54
PT and ILC - Ankara

• PTs within chemistry.
• True value does normally not exist.
  Therefore is instead of used
  consensus-value or reference-value.
• Consensus-value.
  The average of
  all results (without outliers).
• Reference-value.
  Based on
  determinations performed by selected laboratories
  or due to certified reference-material.

55
PT and ILC - Ankara

• IMEP program
• IRMM Institute for reference-materials and
  measurements.
• Produces reference-materials.
• MoU with the EU-commission.
• PTs are arranged internationally.
• Training cources are made internationally.

56
PT and ILC - Ankara

• IMEP-20 Heavy metals in tuna fish.
• As, Pb, Hg, Se and MeHg
• Sample from the market is homogenised and
  distributed to 32 countries.
• The limiting value for tuna fish is for Hg 1
  mg/kg.

57
PT and ILC - Ankara

• IMEP-20.
• The distribution of data is relative large
  meaning that the uncertainty seems to be not fit
  for purpose.
• Furthermore it was seen, that some laboratories
  were using equipment not fit for purpose.

58
PT and ILC - Ankara

• ILC Danish Accreditation.
• ILC using certified reference-material.
• Results
• Laboratory 1
  2 3
• Reference-value 0.5 2.0
  6.0 micro-gram/l
• Lab. Nr. 1 0.55
  2.52 7.73
• Lab. Nr. 2 0.46
  1.74 5.19
• Lab. Nr. 3 0.43
  1.70 4.60
• Average (3 labs) 0.48 1.99
  5.84
• RSD 13
  23 29

59
PT and ILC - Ankara

• Remarks to the analysis-results.
• Satisfactory accordance.
  Expectet uncertainty method is
  fit-for-purpose.
• Supplementary information based on ILC.
• Differences caused equipment-differences.
• Differences caused different internal standard.
• Calibration-curves prepared with different
  tiem-schedules.
• Differences in detector-sensitivity.

60
PT and ILC - Ankara

• PT as tool for improvements.
• Participation is not only an exam.
• The important issue for the participation in
  PTs/ILCs should be to improve analytical
  performance and at the same time the education of
  the analyst. Improve the ability to choose the
  most suitable PTs/ILCs.

61
PT and ILC - Ankara

• PT as a tool for improvements.
• If a PT/ILC shows unacceptable result, the
  reasons for that should be evaluated as
• Human faults such as wrong writing/reading/calcu
  lation. Procedure not followed. Instructions not
  understood.
• Problems with the used method (matrix,
  carry-over).
• Problems with the used equipment.
• Problems with the sample (changed after sampling
  and storage.

62
PT and ILC - Ankara

• PT-informations can be found (among others)
• www.eptis.bam.de (commercial)
• www.irmm.irc.be (IMEP-programme)
• www.danak.dk

63
PT and ILC - Ankara

• Naji-plots
• Give a grafical picture of laboratory-performance
  according to precision and uncertainty.
• Suitable, when uncertainties reported seem to
  differ a lot.
• Suitable for all types of z-scores.

64
PT and ILC - Ankara

• Show overheads.

65
PT and ILC - Ankara

• Use og PT and ILC
• Preparation of new/not so experienced
  laboratory.
• The Youden method.
• Praparation of homogenous samples.
• Find/define true value.
• Analysis of twin-samples

66
PT and ILC - Ankara

• The Youden method.
• Same matrix
• Same method equal method.
• CP a measure of total error
• CR a measure of systematically error
• RP a measure of random error.

67
PT and ILC - Ankara

• Systematically errors (effects accuracy).
• Inaccurate calibration-solutions
• Inadequare instrument-calibrations
• Inadequate correction for blank.
• Inadequate selectivity other components
  interfere.
• Wrong calculation.
• Stable dilution-faults.
• Carry-over !!!

68
PT and ILC - Ankara

• Random errors (effects the precision).
• Instrument instability.
• Wrong reading on instrument.
• Wrong calculation (not constantly).
• Inaccuracies during weighing/measuring
  (weight/volume).
• Unstable samples (hygroscopic - -)
• Difference in reaction-time for reagents.
• Carry-over !!!

69
PT and ILC - Ankara

• Youden plot
• Use grahic setup (for instans excel) place
  circels for evaluation purposes.
• Changes can be seen parallel to control-charts

70
PT and ILC - Ankara

• Evaluation of results.
• Use of z-scores. Recommendable when you are using
  the same method, the same type of instrument (and
  the same test-material).
• Use the results 10 20 30 40 as basis for
  z-score-calculation and compare the result from
  those calculations with a informed
  laboratory-uncertainty of 15 .
• Use the results 1,0 2,0 4,0 5,0 as basis
  for z-score-calculation and compare the result
  with a laboratory-uncertainty 20 .

71
PT and ILC - Ankara

• Evaluation as technical assessor.
• Results from PTs and ILCs are normally
  evaluated comparing the calculated z-score, the
  relative error achieved in the PT with the
  unaccuracy for the actual analysis informed by
  the laboratory.