Design Review Paper Towels Assessment

1
Design ReviewPaper Towels Assessment

• Education 334X
• Technology-Based Student Achievement Assessment
• 2001.10.22
• Alim, Josie, Mike G, Bradley, Nina, Yunn Chyi,
  Johnnie, Jim

2
What does this assessment measure?

• Conceptual framework locate assessment
• Conceptual analysis of
• task
• response format
• scoring system
• What does prior research say?
• Observational Data summary of notes from live
  observation and implications for simulation

3
In the conceptual framework
Content knowledge
Rich
Processknowledge
Constrained
Open
PaperTowels
Lean
4
Conceptual Analysis
5

• For this type of performance assessment, prior
  research has found
• Student notebook scores are exchangeable for
  direct observation scores. Notebook scores were
  slightly more reliable for students with prior
  experience with hands-on instruction.
• There is very high interrater reliability.
  Teachers can be trained to consistently judge
  performance. There is somewhat higher reliability
  for observed hands-on performance than for
  notebooks.
• There was moderate correlation between this type
  of assessment and the multiple choice test
  suggesting that different aspects of science
  achievement are being measured.
• There is low to moderate intertask reliability.
  This may be a concern because a number of tasks
  are needed in order to be able to generalize the
  performance to the domain.
• Sources
• Baxter Shavelson 94- Evaluation of
  Procedure-Based Scoring for Hands-On Science
  Assessment
• Shavelson, Gao Baxter 93 - Sampling
  Variability of Performance Assessments

6
n 1
7
Observation

• Consideration Bias due to familiarity with test
  and scoring criteria.
• Single variable procedures only for method.
• Equalization primary concern for Keri.
• First Trial Used dropper to saturate equal
  pieces. Counted drops and compared.
• Second Trial Weighed equal pieces in scale.
• Third Trial Used set amount of water in beaker
  to saturate. Compared amount of water left in
  beaker.

8
Valid for what kinds of instructional goals?
Paper Towels is a Performance Assessment
designed to support a hands-on, inquiry-based
approach to learning and teaching that is
essential in the primary grades. (1)

• To develop scientific reasoning, critical
  thinking, and inquiry/curiosity (3) (2)
• To design and conduct a sound scientific
  experiments, including identifying and
  controlling variables (develop procedural
  understanding) (4)
• To apply previously learned skills to new
  situations(6)
• To develop skills and appreciation for careful
  observation and measurement (5)

9
What is the utility of this hands-on performance
assessment?
(-)
()

• Provides richer feedback regarding the students
  ability to solve problems using scientific
  reasoning
• The experience is instructional and
  non-intimidating
• Allows you to characterize students procedural
  strengths and weaknesses (good feedback to
  teacher)
• Its much more collaborative and fun!

• More costly than multiple-choice, paper-pencil
  test (new and additional hands-on materials)
• Requires more time (cost) to score and be trained
  to score requires trained scorers (harder to
  score)
• Requires more set-up time
• Requires more class time to execute the test
• The lab-book depends on literacy skills
• Its weak in content

10
Reliability

• Inter-raterVery high, .94 (according to authors)
• Inter-taskNot relevant, only one task is offered
  
• Sources of error
• Very few opportunities for discrepancyLimited
  to judgment calls - care taken?- saturated?

11
What is the construct validity of PT assessment?
12

• What is the construct validity of PT assessment?
• Content validity
• The instructional goals of the lesson parallel
  with what PT is measuring (as shown in the table)
• Cognitive process
• PT assess students ability to explain the
  underlying principles, construct a plan (mental
  model) to initiate the strategy, devise
  strategies to solve the problem, monitor their
  thinking and reasoning while PERFORMING the task
  (vs. recalling, reciting). These cognitive
  processes indicate effective learning of the
  subject matter (Baxtel Elder, 1996).
• Performance level
• Research by Baxter et al (1992) indicates that
  PA measures correlated less with traditional
  ability than did a MC achievement test, ie this
  PA doesnt measure dlec
• Exchangeability for other methods (M-C, C-R,
  Teacher observation?)
• Research has shown performance assessment (e.g.
  PT) is exchangeable for direct observation.
• As a content lean performance assessment, PT
  measures different aspects of science achievement
  (problem solving skills) as Multiple Choice (MC)
  and Construct Response (CR), which concentrates
  more on domain knowledge. Hence they are not
  exchangeable.

13
Implications for Virtualizing PT

• Real world mistakes vs. virtual world mistakes -
  How many can/should we accommodate? Drop size,
  scale validity, scattered moisture, size of
  towels
• How many roads (scenarios) can we account for?
• Cross-method results (combining results from
  multiple experiments)
• Can human observation really be replaced by
  computer tracking?
• Now that were not using the notebooks, should we
  bring back tracking care (eg alarms or flags
  set for missing care steps)
• Do we still need to give them some hands-on
  materials?
• Can we really do this??