Validation Study of the USDAs Data Quality Evaluation System

1
Validation Study of the USDAs Data Quality
Evaluation System

• Seema A. Bhagwat, Kristine Y. Patterson
• Joanne M. Holden
• Nutrient Data Laboratory
• ARS/USDA

2
Uses Of Nutrient Data

• Nutrient data is used for nutrition counseling,
  epidemiological studies, government policies,
  food labeling, quality assurance, product
  development etc.
• It is important to have accurate and precise
  analytical data for these purposes

3
Why Do We Evaluate Data?

• Determine reliability of existing data
• Provide documentation of data reliability to
  users / document reliability in the database
• Establish priorities for new research

4
History of Data Quality Evaluation Systems

• Exler et al., 1983 USDA Iron Table
• Holden et al., 1987 Evaluation of Selenium
  
• Schubert et al., 1987 Development of First
  Selenium Table
• Lurie et al., 1989 Evaluation of Copper
• Sadowski Booth, 1993 Vitamin K
• Holden et al, 2002 Multinutrient Eval. System

5
USDAS Data Quality Evaluation System(DQES)

• System evaluates quality of analytical data
• Documentation of 5 categories Sampling plan
• Sample handling
• Analytical method
• Analytical quality control
• Number of samples analyzed

6
DQES, Continued

• Each category is assigned a rating of 0 through
  20.
• A Quality Index (QI) is generated by combining
  the ratings of 5 categories.
• A Confidence Code (CC) is assigned to each
  nutrient and food.

7
Assignment and Meaning of Confidence Codes
8
Introduction

• International community is interested in
  assessing data quality of their food composition
  databases
• Factors affecting quality of data cover 5
  categories
• Menezes et al, 2000 are using evaluation
  categories for Brazilian data
• Harmonization of data quality evaluation scheme
  is necessary

9
Objectives

• To conduct validation study of USDAs Data
  Quality Evaluation System (DQES)
• To measure the variability of ratings assigned by
  evaluators
• To assess the objectivity of the DQES categories
  (critical questions in each category)
• To test the robustness of the rating scale

10
Methods

• Study Group 1 34 students participating in
  graduate courses on Food Composition Data in
  Nutrition and 3 nutritionists from NDL.
• Evaluated an article on phylloquinone (vit. K)
  contents in fruits and vegetables according to
  USDAs DQES

11
Methods-continued

• Study Group 2 23 students evaluated an aritcle
  on catechin contents in Bulgarian fruits.
• Study Group 3 13 students and 3 nutritionists at
  the NDL evaluated an article on riboflavin
  contents in mushrooms.

12
Methods
13
Methods- continued

• Electronic templates of the evaluation system
  were provided
• Unique questions for each analytical method were
  prepared for each nutrient, (e.g.Vit. K, catechin
  and riboflavin)
• Participants chose from multiple choices to
  answer the questions in the 5 categories

14
Sampling Plan Category

• Statistically developed plan based on probability
  sampling (PPS) gets higher ratings.
• The flow of questions
• No. of Regions
• No. of cities/region
• No. of locations /city
• No. of lots (individual samples)/location
• No. of seasons sampled

15
Sample Handling Category
Sample Handling
Homogenization (equipment used, verification)
Storage (Temp., moisture)
Edible portion only
16
Number of Samples Category

• How many samples were analyzed?
• Number of individual samples
• Not repeated analysis (replicates) from the same
  homogenate
• Not sub-samples from the same composite

17
Analytical Quality Control Category
Control/Reference QC material
Yes
No
Commercial or In-house?
Rating 0
Values in Expected or Extended range?
Frequency of use
CV of analysis of QC material
18
Standard Analytical Method Questions

• Optimization of extraction?
• Quantification of the nutrients (External or
  Internal standard method?)
• 3 concentrations for the standard curve?
• linearity of the standard curve?
• Correlation coefficient of the calibration curve
  ( r ) 0.99?
• Frequent checking of the instrument performance?

19
Validation of Analytical Method

• Was CRM/SRM analyzed (if available)?
• Were the values in the expected range?
• What was the coefficient of variation (rsd) for
  repeated analyses of the same material?
• What was the recovery of the nutrient of
  interest?
• Was the method validated by comparing with
  separate independent method or separate
  laboratory? How good was the agreement? (10 to
  20?)

20
Validation of analytical method
CRM/RM analyzed?
No
Yes
Values in expected range?
CV (rsd)
recovery of nutrient
Comparison with another lab or method?
No
Yes
Agreement? 10 to 20
21
Results

• Evaluation responses and Ratings were tabulated
  in Excel files
• Ratings for Analytical QCs corrected by assigning
  0 points if no QC material used
• One outlier excluded while calculating means and
  medians

22
Results Sampling Plan(Max possible score 20)
23
Sampling Plan Issues/Answers

• Probability vs non-probability sampling
• Probability proportional to size (PPS)
• Current sampling plan criteria are suitable for
  large countries like U.S.
• For smaller countries, samples should be
  nationally representative taking into
  consideration consumption relative to
  population centers, total size of population and
  country size (land mass vs population)
• Ratings need consideration/adjustment
• Need to develop module for smaller countries

24
Results Sample Handling(Max possible score 20)
25
Sample Handling Issues/Answers

• Concept of validation of homogenization needs
  clarification
• If analytical samples are not thoroughly
  homogenized results may not be reliable.
• Standard protocols (SOPs) needed

26
Results Number of Samples(Max possible score
20)
27
Number of Samples Issues/Answers

• No. of individual samples analyzed
• If samples procured from various locations are
  mixed to make a single composite, the number of
  samples analyzed is 1
• Does not count replicate sub-samples from the
  same composite are analyzed independently

28
Number of Samples Example
Number of Samples 3
Number of Samples 1
Homogenate
Homogenate
Homogenate
Composite
Sample Aliquot Analyzed
Sample Aliquot Analyzed
Sample Aliquot Analyzed
Sample Aliquot Analyzed
Sample Aliquot Analyzed
Sample Aliquot Analyzed
29
Results Analytical QC(Max possible score 20)
30
Analytical QC Issues/Answers

• Ambiguity about definitions of CRM/SRM and
  In-house quality control materials and reference
  values
• If analyses are performed over a long period of
  time, QC material is required to verify
  consistent performance of analytical method,
  including instrument performance

31
Analytical QC Issues/Answers Continued

• CRM/SRMs may be expensive to use every day or
  with every batch of samples analyzed. In-house
  QC material similar in matrix can be used for
  precision
• Reference values for CRM/SRM are provided with
  uncertainties Analyst has to establish expected
  Values for In-house material

32
Results Analytical Method(Max possible score
20)
33
Analytical Method Issues/Answers

• Some background in chemistry is essential to
  answer questions in this category
• Providing definitions for technical terms like
  CRM/SRM, may help evaluators

34
Quality Indices (Max possible score 100) and
Confidence codes
35
General Issues

• Evaluators should answer all the necessary
  questions
• Evaluation templates need some modifications
• If control material is not used for Analytical
  QC, the category should get 0 (even if the
  evaluators do not skip rest of the questions as
  directed)
• Ranges for CV or rsd should be more specific
• Should all the 5 categories get equal weighting?
• Authors- provide clearly and detailed
  documentation

36
Conclusions

• Consistent results can be obtained by different
  evaluators
• With a few changes system can be adopted
  universally for data quality evaluation
• Clarity in documentation by authors helpful
• Experience valuable for evaluators

37
Suggestions for Future

• Create a web site for multiple users
• Create a reservoir of evaluated articles
• Maintain a list of Reference Materials and values
• Create a glossary of terms
• Registration of multiple evaluators
• Expand sampling plan options