The Quality Measurement and Reporting Project

1
The Quality Measurement and Reporting Project

• Jane Birch, Office for National Statistics

2
Aims of presentation

• To give an overview of the ONS approach to
  statistical quality measurement and reporting.
• To describe how we are using the European
  framework in statistical quality developments at
  the ONS.
• To report progress on the Quality Measurement and
  Reporting Project (QMR).

3
The Code of Practice

• Knowledge of National Statistics will be
  sustained by quality measures, regularly
  published
• and
• Processes and methods used to produce National
  Statistics will be fully documented.

4
The Quality Management Protocol

• Producers of National Statistics will provide
  accessible information on the quality of their
  outputs using, as guidance, the quality
  dimensions defined for the European Statistical
  System
• and
• the presentation of information on quality will
  be tailored to meet the needs of different types
  of users, with more comprehensive quality reports
  prepared for expert users.

5
Definition of quality

• Fitness for purpose.
• Quality is relative - different users have
  different uses and quality requirements of
  official statistics.
• Users need sufficient quality information so that
  they are able to judge strengths, limitations and
  appropriate uses of outputs for themselves.

6
The European Framework

• The ESS dimensions of quality are used as the
  framework for statistical quality at the ONS

Coherence
Accessibility
Comparability
Relevance
Accuracy
Timeliness
7
The QMR Project

• QMR aims to standardise and systematise quality
  measurement and reporting across the ONS and GSS
  in line with European standards, as set out in
  the NS Code of Practice.
• Also, we aim to report quality at different
  levels for different groups of users from
  non-experts to experts.

8
Standardising quality measurement

• The Guidelines for Measuring Statistical Quality
  V2 to standardise statistical quality
  measurement and reporting.
• Using the ESS dimensions, we have identified 132
  quality measures and indicators for Version 2.
• On the National Statistics website
• www.statistics.gov.uk/qualitymeasures

9
Key Quality Measures (KQM)

• 11 compulsory quality measures to be provided as
  a minimum with all outputs, where applicable.
• The smallest set of important, informative,
  achievable quality measures and indicators which
  provide a summary of the overall quality of an
  output.
• For internal and external users.

10
Key Quality Measures

• Describe how the data relate to user needs.
• State the nationally/internationally agreed
  definitions and standards used.
• Unit response rate.
• Key item response rates.
• Editing rate.
• Total contribution to key estimates from imputed
  values.
• Estimated standard error.

11
Key Quality Measures

• Compare estimates with other sources on the same
  theme.
• Estimate mean absolute revision between
  provisional and final statistics.
• Identify known gaps between key user needs, in
  terms of coverage and detail, and current data.
• Time lag from the reference date/period to the
  release of the output.

12
Systematising quality measurement

• ONS Statistical Modernisation Programme (SMP)
• - qualitative measures - in the metadata system
  within the Central ONS Repository for Data
  (CORD) and
• - quantitative measures - in ONS standard
  statistical software.
• Quality metadata and reports on the NS website.
• Implementation by 2006.

13
Quality reports

• Three levels of quality report recommended to
  meet the needs of a wide range of users, from
  expert to non-expert, internal and external
• - Basic Quality Information (for external users)
• - Summary Quality Report (for external users)
  and
• - Full Quality Report (external and internal
  views).

14
Quality reports for external users

• The BQI will consist of release-specific quality
  information, eg KQMs that are calculated for each
  release. It will be integrated into the relevant
  output.
• The SQR will consist of broad descriptions of
  quality and methods that relate to the whole
  series of releases. It will be continuously
  available on the NS website.
• The first SQR is to be piloted for GDP on the
  30th June.
• BQI for GDP to be available by end 2005.

15
Quality Grading

• Aim to report quality to general
  (non-statistical) users.
• This includes
• - how to define levels of quality
• - what to do with low quality data (ie release
  practices) and
• - how to tell general users about statistical
  quality.
• Census 2001 and ASHE (2004) developed quality
  grading systems.

16
Census 2001 imputation grading

• A simple imputation indicator scheme to signify
  the proportion of imputed people in an area

17
Quality grading for ASHE

• ASHE (2004) graded their estimates according to
  the size of the coefficient of variation (CV).
• The CVs were grouped into bands
• - CV lt 5 Precise
• - CVgt 5 and lt 10 Reasonably precise
• - CVgt 10 and lt 20 Acceptable
• - CVgt 20 Unreliable
• Estimates which were unavailable, n/a, nil or
  disclosive were denoted differently.

18
ASHE example
19
Administrative data quality indicators

• Benefits
• - increase available data without increasing
  respondent burden and
• - coverage advantages over sample survey data.
• Quality issue
• - collected by external providers, so we have
  less control over quality.

20
Processing administrative data

• Single gateway for all incoming administrative
  data quality assessment carried out prior to
  processing.
• Quality reports for
• each administrative data source and for
• statistical outputs using administrative data.

21
Administrative data quality indicators
22
Any questions?

• E-mail
• jane.birch_at_ons.gov.uk
• or
• quality.measurement_at_ons.gov.uk
• National Statistics website
• www.statistics.gov.uk

23
References

• Guidelines for Measuring Statistical Quality
• www.statistics.gov.uk/qualitymeasures
• Census 2001 imputation grading
• http//www.statistics.gov.uk/census2001/methodolog
  y.asp
• ASHE downloadable data
• www.statistics.gov.uk/STATBASE/Product.asp?vlnk13
  101
• Guidelines for Assessing the Quality of
  Administrative Data for Statistical Purposes
  (Fourth Draft)
• Internal paper available upon request from
  marilyn.thomas_at_ons.gov.uk