Reliability of the twoquarter LFS flows data

1
Reliability of the two-quarter LFS flows data

• Keith Brook and Catherine Barham, Office for
  National Statistics

2
Background to LFS longitudinal data

• LFS respondents interviewed for 5 consecutive
  quarters, 20 replaced at each quarter
• Longitudinal datasets available since 1993
  looking at flows over 2 and 5 quarters
• Flows variable used to look at changes in main
  economic activity states over the period i.e
  employment to unemployment etc
• Number of articles in LMT presenting findings
  from these datasets

3
Flows from the 2 quarter dataset
4
Biases in the datasets

• 2 types of bias
• Non-response bias
• Compensated for by weighting
• Response error bias or misclassification
• Investigated in this paper

5
Measuring misclassification

• Misclassification measured using a re-interview
  survey designed to measure the true response
• No UK misclassification surveys to date
• 3 studies carried out in the 1908s and 1990s in
  the USA, Canada and Sweden
• Swedish study used as part of extensive
  investigation into misclassification of gross
  flows from UK LFS

6
Re-interview surveys

• Ideally we want a 'true' measurement, but major
  issues about how this can be achieved
• Follow-up survey is probably impractical for UK
  LFS, due to attrition and response burden
• Independent survey preferable

7
Misclassification matrices from re-interview
surveys
Sweden (weighted) 1994 (see Tzavidis
2003) Re-interview survey, 2153 individuals 7
(approx) of sample)
8
Adjusting flows using misclassification matrices

• Involves complex estimation procedures
• Southampton University evaluated a number of
  methods to estimate the bias
• Study concluded that the Maximum Likelihood
  Estimation method (MLE) is the most suitable for
  estimating the effect of response error in the
  flows
• Software supplied to ONS and used to investigate
  the feasibility of estimating response error bias
  in the 2 quarter longitudinal flows

9
Effect of different misclassification matrices on
quarterly flows
10
Effect of different re-interview surveys on 4
quarter average flows
11
Results

• 3 misclassification matrices (Sweden, Canada,
  USA) result in large differences in the MLE
  estimates
• Adjusted flows consistently give lower flows for
  transitions between 2 different states
• Considerable differences in the adjusted flows
  for the 3 matrices with percentage change being
  high for some transitions with small observed
  flows
• Adjusted flows fairly stable over time

12
Sample size of misclassification matrices

• Misclassification matrices obtained from
• re-interview surveys with relatively small
  sample sizes (2-7 of original sample)
• If sample size not large enough, bias will not be
  measured accurately leading to greater
  uncertainty in the unbiased estimated flows
• Issues investigated using linked dataset between
  LFS and 2001 Census

13
Sample size of misclassification matrices

• Number of matrices derived for a range of sample
  sizes by randomly partitioning the available
  sample
• Conclusions suggest accuracy of misclassification
  matrix highly dependent on sample size

14
Effect of seasonality using Canadian matrix
percentage change of MLE estimates
15
Effect of change year on year (4 quarter
averages) using Canadian matrix change
16
Conclusions (1)

• Misclassification matrices from the previous
  studies in the USA, Canada and Sweden applied to
  UK flows
• Analysis indicates there are significant
  variations in the estimated bias for small
  changes in a misclassification matrix. Results
  are based on relatively small re-interview survey
  sample sizes (between 1500 and 2500)
• Studies using the matched LFS / 2001 Census
  dataset as a 're-interview survey' indicate that
  a much higher sample (in excess of 10,000) would
  be needed to achieve an estimation of the bias
  with an acceptable low variance

17
Conclusions (2)

• Misclassification/bias is likely to be seasonal
  and also subject to longer term changes in
  economic activity
• One-off re-interview survey would have limited
  value and a 4 quarter program may be needed to
  investigate seasonal effects and averaging over 4
  quarters
• Resources (and priorities) are unlikely to allow
  a UK re-interview survey to be undertaken in the
  forseeable future
• Recommend that users process longitudinal data
  using 4 quarter averages and to be aware that a
  bias may exist which cannot currently be
  estimated with any degree of reliability.
  However, changes between time periods will in
  principle be un-biased

18
Quality measurement and reporting
Session C

• Albert Suite