WIP%20

1
WIP Using Information Technology to Author,
Administer, and Evaluate Performance-Based
Assessments

• Mark Urban-Lurain
• Guy Albertelli
• Gerd Kortemeyer
• Division of Science and Mathematics Education
• Michigan State University
• FIE 2005 Session T2E
• October 20, 2005

2
Acknowledgements

• Based upon work supported in part by
• U.S. Department of Education, under special
  project number P342A030070
• National Science Foundation through the
  Information Technology Research (ITR 0085921) and
  the Assessment of Student Achievement (ASA
  0243126) programs
• Michigan State University
• Any opinion, finding, conclusions or
  recommendations expressed in this presentation
  are those of author(s) and do not necessarily
  reflect the views of any of the supporting
  institutions

3
Performance-Based Assessments (PBA)

• Students construct responses in authentic
  problem-solving contexts
• Can support student learning by
• Focusing on fundamentals of a discipline
• Requiring deeper understanding
• Exposing student thinking
• Providing formative evaluation

4
Performance-Based Assessments

• More expensive and time-consuming to score than
  objective tests
• Scoring entails human judgment
• Scoring requires detailed rubrics
• Training for raters
• Inter-rater reliability

5
Previous Work

• CSE 101
• Non-major CS0 course
• 2000 students semester
• Goal Fluency with Information Technology
  (FITness)
• Modified-mastery PBA Bridge Tasks (BT)
• Database-driven system for creating, delivering
  and evaluating BTs
• 14,000 BTs / semester

6
Creating Assessments

• Each Bridge Task (BT) has dimensions (M) that
  define the skills and concepts being evaluated.
• Within each dimension are some number of
  instances (n) of text describing tasks for that
  dimension.
• A bridge task consists of one randomly selected
  instance from each dimension for that bridge task

Dim 1
Dim 2
Instance i
Instance i
Instance i1
Instance i1
Instance i2
Instance i2
Instance in
Instance in
7
(No Transcript)
8
Evaluation Criteria
Student EvaluationPASS or FAIL
9
Scoring Rubrics

• Software
• Retrieves each students files
• Presents grader scoring rubrics for particular BT
  
• Grader evaluates each criteria pass/fail
• Adds open-ended formative feedback for the
  student
• Backend computes outcome based on pass/fail
  scores as defined by instructor for each BT
• Students view their scored BT with the detailed
  rubrics and comments on the Web

10
Porting to Open Source CMS

• LON-CAPA
• Open source course management system developed at
  MSU
• Content sharing and reuse network
• Randomized homework, quizzes, numeric problems

11
Development Effort Completed

• Mastery model BT assessments
• Multiple pass/fail criteria determine if pass or
  fail BT
• Students repeat given level of BT until pass BT
  grading rubrics
• Scoring rubrics for each part of each BT
• Grader evaluates each pass/fail
• Provide formative feedback to student
• Support for proctored authentication
• Online resources typically completed without
  proctors
• BTs administered in scheduled labs with a proctor
  
• Schedule BTs in particular labs at particular
  dates and times

12
Current Status

• Available in current 2.0.2 release
• Testing with one section in CSE 101
• Authoring and scheduling require
  editing/uploading files

13
(No Transcript)
14
(No Transcript)
15
(No Transcript)
16
Spring 2006

• Improve scheduling interface
• Convert CSE 101 course
• Teacher Education
• Incorporate BT as part of the technology
  requirement for pre-service teachers

17
Future Work

• Other CSE courses adopt LON-CAPA and implement
  PBAs
• Working with faculty to implement and evaluate
  the impact of PBAs on student problem-solving
  across other disciplines

18
Questions

• Mark Urban-Lurainurban_at_msu.edu
• LON-CAPAhttp//www.lon-capa.org