Evaluation of Data Quality

1
Evaluation of Data Quality
Food Composition Course Tucuman, Argentina June,
2009
Joanne M. Holden, Research Leader Seema Bhagwat,
Nutritionist Kristine Y. Patterson,
Chemist Nutrient Data Laboratory Beltsville
Human Nutrition Research Center
2
General Process for Evaluating Data

• Was the food representative?
• Was the sample handled properly?
• If necessary, was the sample homogenized?
• Was only the edible portion analyzed?
• Was the sample stored correctly?
• How many samples were analyzed?
• Was the analytical method valid?
• Was analytical quality control adequate?

3
An Approach for Ensuring valid,
well-documented, analytical values

• Documentation is necessary on
• Sampling plan
• Sample handling
• Number of samples
• Analytical methodology
• Analytical Quality Control (QC)
• An evaluation is made of each analytical nutrient
  value based on the documentation

4
Overview of the USDA Evaluation System
For the 5 categories of evaluation

• Each category rated 0 20 on a continuous scale
• Category ratings are combined to give a Quality
  Index (QI)

5
Sampling Plan
Evaluation Criteria Random Selection of
Pickup Locations Number of Regions
Represented Number of Cities / Region
Number of Locations / City Number of Lots
Number of Seasons
6
Sampling Plan
Random Selection of Food Samples All
possible places in the country where a food could
be purchased have a probability of being
selected as pickup locations
According to the population and other
variables. All brands of a food have a
probability of being chosen for analysis with
the probability based on market share (according
to prevalence in the market) This is
designated as a Probability plan
7
Systematic Sampling
Unit 1 Unit 2 Unit 3 Unit 4 Unit 5 Unit 6 Unit
7 Unit 8 Unit 9 Unit 10 Unit 11 Unit 12 Unit
13 Unit 14 Unit 15
Random Start (in sample)
(in sample)
(in sample)
th
8
Systematic Sampling
Cum. Size
Size
8
8
Random Start (in sample)
7
7
15
6
21
(in sample)
6
27
7 15 22
5
32
36
4
(in sample)
40
4
22 15 37
3
43
2
45
Slide 16.5
9
Sampling Plan Evaluation
 
 
 
 
 
10
Sample Handling
Evaluation Criteria Homogenization -
Documentation of equipment used - Validation of
homogeneity Whether only edible portion
analyzed Storage conditions Information on
moisture content
11
Sample Handling Evaluation
Homogenization
Documentation of equipment used (2 points max)
Known 2 points Unknown 0 points
Validation of homogeneity (8 points max)
Homogenization Required?
No
Yes
Homogenization Indicated?
8 points
Yes
No
Homogenization Validated?
0 points
Yes
No
8 points
5 points
12
Sample Handling Evaluation Cont.
Edible Portion Used (3 points max) Yes 3
points No, Unknown 0 points Moisture
Information (3 points max) Yes 3
points No, Unknown 0 points Proper Storage
(4 points max) If storage temp is not
critical 4 points Critical, but not
described 0 points Freezing required Storage
at temp lt0 C 4 points Storage at temp gt0 C 0
points Refrigeration required Storage at temp
lt 4 C 4 points Storage at temp gt 4 C 0 points
13
Number of Samples (Analyzed)
Evaluation Criteria Number of independent
analyses Multiple analyses of a single
composite or sample count as one
analysis That is, analytical replicates at any
level do not get points under this criteria
14
Number of Samples Evaluation
Number of Independent Analyses Points
15
Analytical Method
Evaluation Criteria Validity of the
Method Evaluation of the method against a set
of standard criteria Validity of the Method
as used by the laboratory Demonstrated ability
of laboratory to use the method successfully
16
Analytical Method Validity of the Method
Examples of evaluation Criteria Processing Pre
paration Extraction, digestion,
saponification, etc. Conditions Temperature,
low lights, etc. Analysis and
Quantification Detection of the analytical
signal ID of pks, wavelengths,
etc. Quantification ?3 pt. cal. curve,
calculation algorithms, etc.
17
Analytical MethodValidity of the Method
Evaluation Anal. Prep. B-carotene by HPLC Example
ANALYTICAL PREPARATION
Points given for Yes
Protection from oxidation (BHT,N2,BHA,etc.)? 0.6
Complete extraction reported? 0.5 Temperature
control (evaporation lt 40oC)? 0.3 Samples
protected from UV light? 0.3 Acidic samples
(e.g.fruits and vegetables) neutralized?
0.3 (MgCO3, Na2CO3) Points A sum of the
Points from the Yes responses
18
Analytical MethodValidity of the Method
Evaluation Anal. Prep. Cont. B-carotene by HPLC
Example
19
Analytical MethodValidity of the Method
Evaluation Anal. Quant. B-carotene by HPLC
Example
ANALYSIS AND QUANTIFICATION
How were the carotenoids identified? One
Method 1 Point Two or more Methods 2 Points
Examples of Methods UV/VIS spectral
analysis Co-chromatography 2nd independent HPLC
analysis with diff mobile phase LC/MS or other
MS TLC
20
Analytical MethodValidity of the Method
Evaluation Anal. Quant. Cont.B-carotene by
HPLC Example
For external standardization Yes
points Purity of carotenoid standards
verified? 0.25 Linearity (bracketing
samples concentrations) of standard curve
demonstrated? 0.25 Was the calibration curve
correlation coefficient (r) gt0.99? 0.25 Passage
of standard curve through or very near the
origin? 0.25 Recalibration for each day of
analysis? 0.25 Calibration of UV-visible
spectrophotometer with potassium
dichromate? 0.25 Were there gt3 standard
concentrations used for the calibration
curve? 0.25 Was recovery of the standard
gt90? 0.25 If number of standards lt3 than
Analysis and Quantification score 0
21
Analytical MethodValidity of the Method
Evaluation Anal. Quant. Cont.B-carotene by
HPLC Example
For internal standardization Yes
points Purity of carotenoid standards
verified? 0.25 Linearity (bracketing
samples concentrations) of standard curve
demonstrated? 0.25 Was the calibration curve
correlation coefficient (r) gt0.99? 0.25 Passage
of standard curve through or very near the
origin? 0.25 Internal standard meets
requirements of stability, and chemical and
spectral properties similar to
carotenoids? 0.25 Calibration of UV-visible
spectrophotometer with potassium
dichromate? 0.25 Were there gt3 standard
concentrations used for the calibration
curve? 0.25 Was recovery of the standard
gt90? 0.25 If number of standards lt3 than
Analysis and Quantification score 0
22
Analytical Method Method as Used by the
Laboratory
Evaluation Criteria Validation of the Method
(accuracy)
Acceptable results in comparison with a 2nd
method or 2nd laboratory Acceptable recoveries
Acceptable results with certified reference
materials
Repeatability (precision)
23
Was the method validated?
No
Yes
What was the method of validation? (ranked by
priority?)
Score 0

• Reference Material (SRM, CRM)
• Comparison Method / Lab
• Recovery / Spiking
• Repeatability

24
Analytical Method Method as Used by the
Laboratory Evaluation
Accuracy
Was a certified reference material analyzed?
Yes
No
Were results in certified range?
Were results compared to a 2nd Lab or Method?
No
Yes
No
Yes
Were there recovery results?
Were results within /-15 of cert. value?
Points 9
No
Yes
Yes
Points 0
No
Points 5
Points 0
continued
continued
25
Analytical Method Method as Used by the
Laboratory Evaluation Cont.
Accuracy cont.
Comparison Value
? 10 ?15 ?20 gt20
Recovery
Recovery Points
Points for accuracy are in the table
26
Analytical Method Method as Used by the
Laboratory Evaluation Cont.
Precision (must have an accuracy score gt0)
Reproducibility Points
27
Analytical Quality Control
Evaluation Criteria Results for QC material in
the analytical batch Recovery results for the
batch CV for the QC material QC material
frequency of use With each batch, daily,
weekly, occasionally?
28
Analytical Quality Control Evaluation
Results for QC material in the analytical batch
No
Yes
QC1 Points
Analytical value
QC1 Points 0
29
Analytical Quality Control Evaluation Cont.
QC material frequency of use
CV for the QC material
Points
Points
Recovery results for the batch
Points
Total Points for these three may not exceed 7
30
Example of an Evaluation of an Individual Data
Source
Ratings Category
Optimal Example Sampling
Plan 20 10 Sample Number 20 12
Sample Handling 20 19 Analytical
Method 20 12 Analytical Quality
Control 20 13 QI
100 66 CC A B
31
Combining Evaluation Ratings in Aggregation
Individual Data Sources
Ratings S1 S2 S3 S4 Ratings for Category
Ratings examples Aggregate QI CC Sam.
Plan 10 12 6 8 18 Sam. Num. 12 20 10 8 20
Sam. Handl. 19 15 10 18 16 Anal.
Meth. 12 2 13 17 14 Anal. QC 13
0 12 11 12 80 A
Data omitted in aggregation by food specialist
due to poor Anal. Meth. and QC ratings
32
Reporting the Estimated Reliability of a Nutrient
Value
A Confidence Code (CC) is assigned to a nutrient
value based on the QI to indicate the estimated
reliability of the data
33
Assignment and Meaning of Confidence Codes
34
(No Transcript)
35
(No Transcript)
36
(No Transcript)
37
Nutrient Methodology Needs

• Carotenoids (complete)
• Dietary Fiber (complete)
• Lipids / total fat
• Folate
• Vitamin K (in preparation)
• Riboflavin
• Vitamin B-6
• Vitamin B-12
• Amino acids
• Flavonoids (in preparation)

• Sugars (in preparation)
• Vitamin A
• Vitamin C
• Niacin
• Thiamin
• Nitrogen
• Vitamin E
• Selenium
• Phosphorus
• Minerals (Fe, Zn, etc.)

Committee formed
38
Nutrient Specific Committees

• Vitamin K
• Sarah Booth
• Vieno Piironen
• Martin Shearer
• Lipids/Total fat
• Robert Pawlosky
• Bev Teter
• Andrew Sinclair

• Sugars
• Betty Li
• Gillian Eggleston
• Carotenoids
• Delia Rodriguez-Amaya
• David Southgate
• Paul Hulshof

39
Nutrient Specific Committees cont.

• Niacin
• Wayne Wolf
• Darryl Sullivan
• To be filled
• Folate
• Robert Doherty
• Paul Finglas
• Liisa Vahteristo

• Flavonoids
• Gary Beecher
• Doug Balentine
• Peter Hollman
• Retinol
• Robert Pawlosky
• Neal Craft
• To be filled

40
Method Specific Evaluation CriteriaWell
Characterized Methods
NDL Develop evaluation criteria based on the
literature
Nutrient Specific Evaluate validity of
criteria and Committees give comments
NDL Revise criteria based on comments
External review Verify changes to criteria
NDL Program the database system for evaluation
of data
41
Method Specific Evaluation CriteriaUnique
Methodology
Nutrient Specific Provide critical steps for
methods Committees
NDL Develop evaluation criteria based on
committee report
Nutrient Specific Evaluate validity of
criteria and Committees give comments
NDL Revise criteria based on comments
External review Verify changes to criteria
NDL Program the database system for evaluation
of data
42
What is the Result?

• Standardized / Systematic Approach
• Nutrient / Food Specific Confidence Codes
• Quantitative Evaluation of Data Quality
• Documentation of Data Quality
• Availability of Source
• Information and References
• Easy Update with Changes

43
General Quality General Considerations

• Is all source documentation available?
• Where were samples collected?
• How were they handled (homogenized, edible
  portion)?
• How many samples?
• What types / brands / seasons?
• Who did the analyses?
• When was the work done?
• What methods were used?
• Was analytical quality control used?
• How does the data compare to other / existing
  data?
• Are proximates available?
• Do the proximates add up to 100?

44
Special Publications
45
Emerging Components

• Folates
• Individual Carotenoids
• Isoflavones
• Flavonoids
• Choline/Sphingolipids
• Trans-Fatty Acids
• Omega-3 Fatty Acids
• Phenolic Acids
• Phytic Acid
• Fluoride
• Other.

46
Steps to Improving Data QualityAnalysts

• Validate Methods Identify Critical Points
• Develop a Day-to-Day Quality Control Program
• Select a Representative Sample
• Monitor Accuracy and Precision

47
Steps to Improving Data QualityEditors/Reviewers/
Compilers

• Request QC Information
• Qualitative
• Quantitative
• Review Sampling Plan
• Seek Methods Documentation
• Evaluate Statistical Data

48
Take a break !!
49
Accuracy and Precision
Low Precision High Bias
High Precision High Bias
Precision Quantified by Standard deviation
s Coefficient of variation CV
Accuracy Cannot be quantified Bias can
be estimated by using Various methods based on
different principles Highly sophisticated
methods
High Precision No Bias
Low Precision No Bias
50
Repeatability and Reproducibility
Repeatability minimum variation in conditions
duplicate analyses in one lab in one series,
on the same day by one technician Reproducibilit
y maximum variation in conditions two values
from different labs, by different technicians,
on different days