Looking a validity and reliability from a different perspective

1
Looking a validity and reliability from a
different perspective

• Bjorklund K, Skold C, Andersson L, Dalen N.
  Reliability of a criterion-based test of athletes
  with knee injuries where the physiotherapist and
  the patient independently and simultaneously
  assess the patient's performance. Knee Surg
  Sports Traumatol Arthrosc. 2006 Feb14(2)165-75.

2
What about validity and reliability from an
athletic training/ physical therapy perspective?

• How do you assess functional ability with
  Anterior Cruciate Ligament (ACL) injuries?
• gtBjörklund et al. 2006 proposed a series of tests
  to assess functional ability ("Tests for Athletes
  with Knee-injuries, or TAK")
• TAK is composed of 8 different tests patient is
  scored on his/her ability to perform these tasks
  (pg 168)
• e.g. I Jogging straightforward
• II Fast running straightforward

3
Study Design
4
How did the evaluations work?
5
Reliability, objectivity, validity

• Fundamental questions they wanted to ask were
• How was reliability of TAK assessed?
• This was done a number of ways
• How was validity of the TAK assessed?
• How was objectivity assessed?
• Also, how did the test rate with 1,2,3?

6
How was reliability assessed?

• Scoring of the 59 subjects by Physiotherapist A
  and Physiotherapist B
• Watched participants go through TAK test at the
  same time
• Physiotherapist A repeated 31 of the subjects 4-6
  days later
• Subjects rated themselves on the two different
  occasions (n 31)
• Physiotherapist A compared to subject
  self-assessments (n 59)

7
Reliability Assessment

• Kappa (K) statistic measure of inter- and
  intra-rater reliability
• Criteria lt0.20 poor
• 0.21 0.40 fair
• 0.41 0.60 moderate
• 0.61 0.80 good
• 0.81 1.00 very good
• Spearman's Rho (rs)
• Correlation coefficient based on rankings
• Don't worry about the details of the calculation
  interpret the same as a Pearson r

8
Kappa Statistic (briefly!)

• Measures the stability of an evaluation

9
Reliability of TAK assessed by Kappa Statistic

• Ability of 2 raters to give the same score is
  moderate
• Reproducibility of a score when separated by 4-6
  days is moderate
• Ability of a subject to reliably rate his/her
  performance is fair

10
Reliability of TAK assessed by correlation

• When assessed by correlation
• 2 raters more reliably assign a score than 1
  rater can score the test twice
• Trained raters are more reliable than the
  participants

11
Validity of TAK?

• Assuming that Biodex is an acceptable measure of
  functional ability for the knee
• The relationship between TAK and Biodex only
  explains 37 of the variance

12
Objectivity of TAK?

• Inter-judge perhaps, but not intra-judge
  assessed!
• In order to assess intra-judge objectivity,
  raters would have to observe the exact same
  performance by the subject
• Did not happen in this case the 2nd assessment
  was the participant repeating the test
• Can you be certain that subjects were able to
  re-produce the tests the same each time?
• Only way to assess objectivity would be to have
  videotaped the performances that way there is no
  possibility that the inability of the subjects to
  reproduce their performance is an issue

13
Overall

• Was reliability truly assessed?
• Physiotherapist A vs. B on same performance may
  be a mix of administrator error and test error
• Physiotherapist A. vs. A error may be partially
  the test administrator error and the inability of
  the athlete to re-produce his/her performance
• Was validity assessed?
• Yes, assuming isokinetic quadriceps strength is a
  valid tool
• Was objectivity assessed?
• Inter-judge, yes
• Intra-judge, no