Multiple Response Questions Allowing for chance in authentic assessments

1
Multiple Response QuestionsAllowing for chance
in authentic assessments

• Mhairi McAlpine
• Robert Clark Centre
• University of Glasgow

Ian Hesketh TOIA Project University of Strathclyde
2
Introduction

• Multiple response questions are a popular method
  of computer assisted assessment however
  questions are being raised about their
  reliability.
• This paper looks at how MRQs are implemented in
  practice, and how this may impact on assessment
  in Higher Education.

3
Methodology

• 637 MRQ questions from 65 tests were reviewed.
• We were interested in
• How big the question guess factors were
  particularly compared with other objective
  formats such as T/F and MCQ
• How this impacted on the test guess factor
• What effect that had on the weightings of the MRQ
  questions within the test

4
Results questions and options

• The examined items ranged from 1-7 keys and 3-18
  response options.
• The majority of items clustered between 2-5 keys
  and 5-9 response options
• The most popular combination of keys and options
  was 3 from 6. This was the default setting on
  the software and accounted for almost 40 of the
  questions examined.

5
Question chance factors

• Question chance factors peaked at 0.5 accounting
  for 57.8 of the questions.
• There was also a smaller peak at 0.4
• Less than 15 of questions had a chance factor
  lower than a standard MCQ (1 key 4 options).
• Nearly 15 of questions had a chance factor
  greater than a True/False question

6
Tests chance factor

• The test chance factors ranged from 0.25 to 0.75
  with the majority of the data clustered between
  0.34 and 0.6.
• Only in 1 test was the chance factor comparable
  to an MCQ test.
• In over a quarter of tests, the chance factor was
  greater than in a T/F test.

7
Impact on Weightings

• In all of the tests examined, each response
  within a MRQ carried at least one mark leading
  to this question type being heavily weighted.
• A high guess factor depresses the discrimination
  of a question
• this in turn depresses the weighting of the
  question, meaning that its intended weight is not
  achieved

8
Effect of intended weighting on test chance
factor

• The chance factors of the questions were weighted
  by the number of marks that each of them carried.
  
• Only in one test did this reduce the overall test
  chance factor
• in some cases it increased the test chance factor
  by 0.05.

9
Discussion

• The use of MRQs in formative assessment has been
  demonstrated, however adjustments may have to be
  made for them to be a valid form of summative
  assessment
• McCabe and Barrett have suggested a formula for
  calculating the chance factor of an MRQ, this
  would make explicit how much random variation an
  author may be introducing.
• This issue becomes more pressing when randomised
  or adaptive tests are given.
• Where the overall test chance factor may vary
  from student to student

10
Recommendations

• It is clear that it is time for a community-based
  approach to identifying and resolving issues of
  analysis.
• Further work should be carried out on how the
  outcomes of computer based questions and tests
  are handled.
• Development of statistical approaches to chance
  calculation guess correction in new question
  formats should be conducted .
• More effort should be focused on the analysis of
  tests and items that exploit the advances made in
  authoring complexity.

11
Conclusions

• Matrix questions may offer a partial solution to
  the chance factor problem in MRQs
• The default software setting has influenced
  practice care must be taken that good practice
  prevails
• Dissemination of item analysis techniques and
  test construction methodology must be prioritised
  within the CAA community.