Method comparison

1
Method comparison
Chemometrics
Department of Chemical Pathology, University of
Pretoria,
Dr R Delport 2003
2
Performance characteristics that are taken into
account
Chemometrics
Precision, Accuracy, Interference, Working
Range, andDetection Limit.
http//www.westgard.com/lesson21.htm
3
Chemometrics

• To carry out a good method validation study, you
  need to do the following
• Define a quality requirement for the test in the
  form of the amount of error that is allowable,
  preferably an allowable total error,
• Select appropriate experiments to reveal the
  expected types of analytical errors,
• Collect the necessary experimental data,
• Perform statistical calculations on the data to
  estimate the size of analytical errors,
• Compare the observed errors with the defined
  allowable error, and
• Judge the acceptability of the observed method
  performance.

http//www.westgard.com/lesson21.htm
4
Chemometrics

• An experimental plan can be formulated by
• Recognizing the types of errors that need to be
  assessed for this test and method,
• Identifying the appropriate experiments and the
  amount of data needed to estimate those types of
  errors, then
• Organizing these experiments to perform the quick
  and easy ones first and the ones taking more time
  and effort last.

http//www.westgard.com/lesson21.htm
5
Types of errors
Chemometrics

• Imprecision or random errors,
• Inaccuracy, bias, or systematic errors, which can
  be of two types
• Constant systematic error or
• Proportional systematic error.

http//www.westgard.com/lesson21.htm
6
Chemometrics
The dashed line in the middle of the figure
represents ideal method performance where the
test method and the comparative method give
exactly the same results.
The bottom line in the figure shows the effect of
a proportional systematic error, where the
magnitude of the error increases as the test
result gets higher.
The top line shows the effect of a constant
systematic error, where the whole line is shifted
up and all results are high by the same amount.
Note that these results will also be subject to
the random error of the method, therefore the
actual data points would scatter about the line
as illustrated in the figure. The range of this
scatter above and below the line provides some
idea of the amount of random error that is
present.
http//www.westgard.com/lesson21.htm
7
Chemometrics
Experiments for estimating analytical errors
http//www.westgard.com/lesson21.htm
8
Chemometrics
Experiments for estimating analytical errors
Description of different experiments.

• Replication experiment
• Provides information about random error
• Is performed by making measurements on a series
  of aliquots of the same test samples within a
  specified period of time, usually within an
  analytical run, within a day, or over a period of
  a month.
• Preliminary experiment involves determining
  within-run imprecision.
• Final experiment requires at least 20 working
  days to provide estimate of the total
  imprecision, which includes within and between
  run components.

http//www.westgard.com/lesson21.htm
9
Chemometrics
Experiments for estimating analytical errors
Description of different experiments.

• Interference experiment
• Provides information about the constant
  systematic error caused by the lack of
  specificity of the method.
• One test sample is prepared by adding the
  suspected material to a sample containing the
  analyte.
• A second aliquot of the original sample is
  diluted by the same amount with solvent, then
  both samples are analyzed by the test method and
  the difference determined.

http//www.westgard.com/lesson21.htm
10
Chemometrics
Experiments for estimating analytical errors
Description of different experiments.

• Recovery experiment
• Provides information about the proportional
  systematic error caused by a competitive
  reaction.
• Test sample is prepared by adding a standard
  solution of the analyte being tested to an
  aliquot of a patient specimen.
• A baseline sample is prepared by adding an equal
  amount of the solvent used for the standard
  solution to a second aliquot of the same patient
  specimen.
• The two samples are then analyzed by the test
  method and the amount recovered is compared to
  the amount added.

http//www.westgard.com/lesson21.htm
11
Chemometrics
Experiments for estimating analytical errors
Description of different experiments.

• Comparison of methods experiment
• Is primarily used to estimate the average
  systematic error observed with real patient
  samples.
• Can also reveal the constant or proportional
  nature of that error.
• A series of patient specimens are collected and
  analyzed by both the test method and a
  comparative analytical method.
• The results are compared to determine the
  differences between the methods, which are the
  analytical errors between the methods.

http//www.westgard.com/lesson21.htm
12
Chemometrics
Experiments for estimating analytical errors
Walking tour of the plan.

• Get the method working and establish an operating
  protocol.
• Set up the instrument, prepare the reagents,
  calibrate the methods, and obtain results from
  test samples.
• Check the standards and be sure the method is
  properly calibrated, otherwise calibration errors
  will show up throughout the experimental studies.

http//www.westgard.com/lesson21.htm
13
Chemometrics
Experiments for estimating analytical errors
Walking tour of the plan.

• Determine the working range.
• The working range will vary from test to test and
  must be defined as part of the specifications for
  the method.
• Check by analyzing a series of solutions, in
  duplicate or triplicate, covering the
  concentrations range of interest.
• If detection limit is a critical characteristic,
  it may be assessed at this time or in the next
  phase of preliminary experiments.

http//www.westgard.com/lesson21.htm
14
Chemometrics
Experiments for estimating analytical errors
Walking tour of the plan.

• Determine within-run imprecision, recovery, and
  interference.
• The replication experiment might include 20
  samples of two or three materials whose
  concentrations closely match the medical decision
  levels of interest for the tests.
• Interference experiments should test common
  problems such as hemolysis, lipemia, and high
  bilirubin.
• Recovery experiments assess whether there are any
  competitive reactions due to the matrix or other
  materials in the native specimens.

http//www.westgard.com/lesson21.htm
15
Chemometrics
Experiments for estimating analytical errors
Walking tour of the plan.

• Comparison of methods experiment.
• Fresh patient specimens, and stored specimens.
• Minimum of 40 well-chosen patient samples should
  be tested over a minimum of 5 working days.
• Distributed one-third in the low to low-normal
  range, one-third in the normal range, and
  one-third in the high abnormal range.
• Method acceptability should be judged on the
  basis of the sizes of the random, systematic, and
  total analytical errors.

http//www.westgard.com/lesson21.htm
16
Chemometrics
Experiments for estimating analytical errors
Walking tour of the plan.
Comparison of methods experiment.

• Used as initial graph if the two methods are
  expected to show one-to-one agreement,
• Displays the difference between the test minus
  comparative results on the y-axis versus the
  comparative result on the x-axis, such as shown
  in the accompanying figure. 

Difference plot
http//www.westgard.com/lesson21.htm
17
Chemometrics
Experiments for estimating analytical errors
Walking tour of the plan.
Comparison of methods experiment.

• Used as initial graph if the two methods are
  expected to show one-to-one agreement,
• Displays the difference between the test minus
  comparative results on the y-axis versus the
  comparative result on the x-axis, such as shown
  in the accompanying figure. 
• Differences should scatter around the line of
  zero differences, half being above and half being
  below. 
• Repeat measurements if indicated.

Difference plot
http//www.westgard.com/lesson21.htm
18
Chemometrics
Experiments for estimating analytical errors
Walking tour of the plan.
Comparison of methods experiment.

• The correlation coefficient is a statistic that
  is almost always calculated and reported to
  describe the results from a comparison of methods
  study.
• The new or "test" method values are plotted on
  y-axis and comparisonvalues on x-axis.

Correlation coefficient
http//www.westgard.com/lesson21.htm
19
Chemometrics
Experiments for estimating analytical errors
Walking tour of the plan.
Comparison of methods experiment.

• A value of 1.000 indicates perfect correlation
  between the results of two methods.
• Other statistics (such as slope, intercept, and
  standard deviation of the residuals) can also be
  calculated from the same data to estimate the
  size of errors occurring between the methods.

http//www.westgard.com/lesson15.htm
20
Chemometrics
Analytical errors

• Random error, RE, or imprecision
• Can be either positive or negative.
• Direction and exact magnitude cannot be
  predicted.
• Imprecision is quantitated by calculating the
  standard deviation (SD) from the results of a set
  of replicate measurements.

Random error

http//www.westgard.com/lesson15.htm
21
Chemometrics
Analytical errors

• Random error, RE, or imprecision
• Can be either positive or negative.
• Direction and exact magnitude cannot be
  predicted.
• Imprecision is quantitated by calculating the
  standard deviation (SD) from the results of a set
  of replicate measurements.
• As the SD often increases as the concentration
  increases, the coefficient of variation (CV) is
  calculated to express the SD as a percentage of
  the mean concentration from the replication
  study.
• Maximum size of a random error is commonly
  expressed as a 2 SD or 3 SD estimate to help
  understand the potential size of the error that
  might occur. 

Random error

http//www.westgard.com/lesson15.htm
22
Chemometrics
Analytical errors

• Systematic error, SE, or inaccuracy
• Is always in one direction.
• A systematic shift displaces the mean of the
  distribution from its original value.
• In contrast to random errors that may be either
  negative or positive and whose direction can not
  be predicted, systematic errors are in one
  direction and cause all the test results to be
  either high or low. 

Systematic error
http//www.westgard.com/lesson15.htm
23
Chemometrics
Analytical errors

• Systematic error, SE, or inaccuracy
• How high or how low can be described by the bias,
  which is calculated as the average difference, or
  the difference between averages, between the
  value by the "test" method and a "comparative"
  method in a comparison of methods experiment.
• Alternatively, the expected systematic difference
  may be predicted from the equation of the line
  that best fits the graphical display of test
  method values on the y-axis vs comparative method
  values on the x-axis.

Systematic error
http//www.westgard.com/lesson15.htm
24
Chemometrics
Analytical errors

• Systematic error, SE, or inaccuracy
• How high or how low can be described by the bias,
  which is calculated as the average difference, or
  the difference between averages, between the
  value by the "test" method and a "comparative"
  method in a comparison of methods experiment.
• Alternatively, the expected systematic difference
  may be predicted from the equation of the line
  that best fits the graphical display of test
  method values on the y-axis vs comparative method
  values on the x-axis.
• SE may stay the same over a range of
  concentrations, in which case it can also be
  called constant error, or it may change as
  concentration changes, in which case it can be
  called proportional error.

Systematic error
http//www.westgard.com/lesson15.htm
25
Chemometrics
Analytical errors

• Total Error, TE,
• It is the net or combined effect of random and
  systematic errors, as shown in the accompanying
  figure.

Total error
http//www.westgard.com/lesson15.htm
26
Chemometrics
Analytical errors

• Total Error, TE,
• It is the net or combined effect of random and
  systematic errors, as shown in the accompanying
  figure.
• It represents a "worst-case" situation, or just
  how far wrong a test result might be due to both
  random and systematic errors.
• Because laboratories typically only make a single
  measurement for each test, that measurement can
  be in error by the expected SE, or bias, plus 2
  or 3 SD, depending on how you quantitate the
  effect of RE. 

Total error
http//www.westgard.com/lesson15.htm
27
Chemometrics
28
Chemometrics
29
Chemometrics