The Design Phase: Using Evidence-Centered Assessment Design

1
The Design PhaseUsing Evidence-Centered
Assessment Design
Monty Python argument
2
Workshop Flow

• The construct of MKT
• Gain familiarity with the construct of MKT
• Examine available MKT instruments in the field
• Assessment Design
• Gain familiarity with the Evidence-Centered
  Design approach
• Begin to design a framework for your own
  assessment
• Assessment Development
• Begin to create your own assessment items in line
  with your framework
• Assessment Validation
• Learn basic tools for how to refine and validate
  an assessment
• Plan next steps for using assessments

3
The PADI Project padi.sri.com
4
What is Evidence-Centered Design?

• Approach developed by ETS
• Uses evidentiary reasoning to design the
  underlying principles of an assessment
• Answers these questions
• What complex of KSAs should be assessed?
• What types of evidence would we need to show a
  test-taker has these KSAs?
• What kinds of assessment items would allow us to
  gather this evidence?
• Provides a set of tools for structuring a
  systematic approach to doing this

5
Phases of ECD

• Domain Analysis
• Big-picture narrative of the knowledge domain
• Domain Modeling
• Organization of the information in the domain in
  terms of
• The aspect of proficiency of the test-taker in
  the domain
• The kinds of things the test-taker might do to
  provide evidence of their proficiency
• The kinds of situations that might make it
  possible to provide such evidence
• Conceptual Assessment Framework
• Blueprint for the actual assessment that takes
  all of this into account
• The Assessment

6
Phases of ECD for Assessing MKT

• Domain Analysis of MKT
• Big-picture narrative of the knowledge domain

Rate of change

• MKT
• Interpreting unconventional forms or
  representations
• Choosing problems and examples that can
  illustrate key curricular ideas
• Differentiating between colloquial and
  mathematical uses of language
• Linking precise aspects of representations
• Understanding implications of models and
  representations
• Evaluating mathematical statements

yaxb
Proportionality
7
Phases of ECD for Assessing MKT

• Domain Modeling
• Organization of the information in the domain in
  terms of
• The aspect of proficiency of the test-taker in
  the domain
• The kinds of things the test-taker might do to
  provide evidence of their proficiency
• The kinds of situations that might make it
  possible to provide such evidence

Create a Design Pattern
8
Phases of ECD for Assessing MKT

• Conceptual Assessment Framework
• Blueprint for the actual assessment that takes
  all of this into account

9
Types of Relationships to Curriculum/PD
Assessment Development
Curriculum/PD Development
Assessment Development
Curriculum/PD Development
Assessment Development
Curriculum/PD Development
10
Adapting Off-the-Shelf Instruments

• Start with this process for your own assessment
  needs
• Evaluate the fit of the instrument for your needs
• Important Do not simply mix and match!

11
Attributes of a Design Pattern
Framing info Title, Summary, Rationale
What the test-taker should know Focal Knowledge, Skills, and Abilities
Evidence we can collect to show they know it Potential observations Potential work products Potential rubrics
Kinds of situations that can evoke this evidence Characteristic features Variable features
Other info Exemplar items, Online resources, References, Misc
12
Examples of Design Patterns

• MKT for SimCalc
• Co-construct as a group
• Do your own

13
Foundations of proportionality
Find the missing number
Consider the function
Proportionality includes linearity, rate,
function, slope in graphs, interpreting tables
with an underlying rate
14
NCTM Curriculum Focal Points
Students graph proportional relationships and
identify the unit rate as the slope of the
related line. They distinguish proportional
relationships (y/x k, or y kx) from other
relationships, including inverse proportionality
(xy k, or y k/x).

• Grade 7 Developing an understanding of and
  applying proportionality, including similarity.
• Students extend their work with ratios to develop
  an understanding of proportionality that they
  apply to solve single and multistep problems in
  numerous contexts. They use ratio and
  proportionality to solve a wide variety of
  percent problems, including problems involving
  discounts, interest, taxes, tips, and percent
  increase or decrease. They also solve problems
  about similar objects (including figures) by
  using scale factors that relate corresponding
  lengths of the objects or by using the fact that
  relationships of lengths within an object are
  preserved in similar objects. Students graph
  proportional relationships and identify the unit
  rate as the slope of the related line. They
  distinguish proportional relationships (y/x k,
  or y kx) from other relationships, including
  inverse proportionality (xy k, or y k/x).
• Grade 8 Analyzing and representing linear
  functions and solving linear equations and
  systems of linear equations.
• Students use linear functions, linear equations,
  and systems of linear equations to represent,
  analyze, and solve a variety of problems. They
  recognize a proportion (y/x k, or y kx) as a
  special case of a linear equation of the form y
  mx b, understanding that the constant of
  proportionality (k) is the slope and the
  resulting graph is a line through the origin.
  Students understand that the slope (m) of a line
  is a constant rate of change, so if the input, or
  x-coordinate, changes by a specific amount, a,
  the output, or y-coordinate, changes by the
  amount ma. Students translate among verbal,
  tabular, graphical, and algebraic representations
  of functions (recognizing that tabular and
  graphical representations are usually only
  partial representations), and they describe how
  such aspects of a function as slope and
  y-intercept appear in different representations.
  Students solve systems of two linear equations in
  two variables and relate the systems to pairs of
  lines that intersect, are parallel, or are the
  same line, in the plane. Students use linear
  equations, systems of linear equations, linear
  functions, and their understanding of the slope
  of a line to analyze situations and solve
  problems.

They recognize a proportion as a special case of
a linear equation of the form y mx b,
understanding that the constant of
proportionality (k) is the slope and the
resulting graph is a line through the
origin. Students translate among verbal, tabular,
graphical, and algebraic representations of
functions.
15
KSAs

• Common Content Knowledge
• Proportionality concepts
• Specialized Content Knowledge
• Interpreting unconventional forms or
  representations
• Choosing problems and examples that can
  illustrate key curricular ideas
• Differentiating between colloquial and
  mathematical uses of language
• Linking precise aspects of representations
• Understanding implications of models and
  representations
• Evaluating mathematical statements

16

• Design pattern example2, Co-constructed

17
Workshop Flow

• The construct of MKT
• Gain familiarity with the construct of MKT
• Examine available MKT instruments in the field
• Assessment Design
• Gain familiarity with the Evidence-Centered
  Design approach
• Begin to design a framework for your own
  assessment
• Assessment Development
• Begin to create your own assessment items in line
  with your framework
• Assessment Validation
• Learn basic tools for how to refine and validate
  an assessment
• Plan next steps for using assessments

18
Activity 2Create a Design Pattern

• Find Activity 2 in your binder
• Pick a design pattern topic
• Work on your own, with a partner, or with a small
  group
• Complete the Design Pattern form
• Feedback / Review Process
• Discussion to follow
• Show-and-tell of 2 or 3 Design Patterns
• Your insights, questions, challenges

19
Some Useful References(all available on the web)

• Baxter, G. P., Mislevy, R. J. (2005). The Case
  for an Integrated Design Framework for Assessing
  Science Inquiry. PADI Technical Report 5. Menlo
  Park, CA SRI International.
• Embretson, S. E. (Ed) (1985). Test Design
  Developments in psychology and psychometrics.
  New York Academic Press, Inc.
• Mislevy, R. J., Almond, R. G., Lukas, J. F.
  (2003). A brief introduction to
  Evidence-Centered Design. CRESST Technical Paper
  Series. Los Angeles, CA CRESST.
• Mislevy, R. J, Hamel, L. et al. (2003). Design
  Patterns for Assessing Science Inquiry. PADI
  Technical Report 1. Menlo Park, CA SRI
  International.
• PADI Website http//padi.sri.com

20
Feedback

• What worked?
• What didnt work?
• What do you still want to learn?

21
Welcome back! Nicole.shechtman_at_sri.com Teresa
.lara-meloy_at_sri.com
22
Q1. Definition of MKT

• By mathematical knowledge for teaching, we mean
  the mathematical knowledge used to carry out the
  work of teaching mathematics. Examples of this
  work of teaching include explaining terms and
  concepts to students, interpreting students
  statements and solutions, judging and correcting
  textbook treatments of particular topics, using
  representations accurately in the classroom, and
  effects of teachers mathematical knowledge on
  student achievement providing students with
  examples of mathematical concepts, algorithms, or
  proofs (Hill, Rowan, Ball, 2005).
• We look across mathematical knowledge needed for
  or used in teaching, including pure content
  knowledge as taught in secondary, undergraduate,
  or graduate mathematics courses pedagogical
  content knowledge and curricular knowledge (also
  described by Shulman) both possibly taught in
  mathematics methods courses and what is more
  elusive, knowledge that, while also mathematical,
  is not typically taught in undergraduate
  mathematics courses and is not be entirely
  pedagogical. Mathematical knowledge for
  teaching, keeping the emphasis on mathematics and
  acknowledging that teachers may know and use
  mathematics that is different from what is
  required for other professions (Ferrini-Mundy,
  et. al., 2008)

23
Question 2

• Teachers attitudes about being evaluated

24
Q3. Adapting Off-the-Shelf Instruments

• Start with this process for your own assessment
  needs
• Evaluate the fit of the instrument for your needs
• Important Do not simply mix and match!