Assessing Elementary Students

1
Assessing Elementary Students Functional
Thinking Skills The Case of Function Tables

• Katherine L. McEldoon Bethany Rittle-Johnson

2
Project Goals

• Develop an assessment of elementary students
  functional thinking abilities, an early algebra
  math skill
• Develop a model of knowledge progression

3
Functional Thinking

• The table shows how the In numbers are related
  to the Out numbers. When a 38 goes in, what
  number comes out?
• 41 B.51 C. 54 D. 77

• A type of mathematical thinking which focuses on
  the relationship between two (or more) varying
  quantities, specifically the kinds of thinking
  that lead from specific relationships to
  generalizations of that relationship across
  instances. (Smith, 2008)
• Encapsulates important core components of early
  algebraic reasoning, such as generalization and
  covariation. (Carraher, Martinez, Schliemann,
  2008)

77
Out (In x 2) 1
Y 2X 1
4
Functional Thinking Performance Grade 4

• The table shows how the In numbers are related
  to the Out numbers. When a 38 goes in, what
  number comes out?
• 41
• 51
• 54
• 77

National Assessment of Educational Progress
(NAEP), National Performance results in
Mathematics at Grade 4 2007
5
Function Tables

• Focus Functional Tables
• Determining Values and Rules
• Typical Tasks (Carraher Earnest, 2001
  Schliemann Carraher, 2000)
• Fill in the missing values in this table
• What is the rule for this table?
• Asked to select a rule from several choices
• Asked to write the rule verbally or symbolically

Column A Column B
2 6
3 7
4 8
5 9
6 10
14 18
21 25
41 45
6
Function Table Competencies

• Within function table problems, we isolated
  required competencies and used this as a basis
  for our assessment
• Loosely hypothesized order of difficulty
• Apply a Given Rule (prerequisite)
• Determine Next Y value in Sequence
• Determine Near Y value in Sequence
• Determine Far Y value in Sequence
• Recognize a Rule (verbal/symbolic)
• Generate Rule Verbally
• Generate Rule Symbolically

Column A Column B
2 6
3 7
4 8
5 9
6
14
21
41
Column B Column A 4
B A 4
7
Wilsons Construct Modeling Approach
Item Design
Construct Map
Measure ment Model
Item Score

• Wilsons Four Building Blocks
• 1) Construct Map
• 2) Item Design ?
• 3) Item Score
• 4) Measurement Model
• Assess the student performance data to evaluate
  your construct map and items

8
Item Design Assessment

• We designed items that tapped each of these
  competencies
• Modified from e.g. Blanton Schliemann Warren,
  Cooper Lamb
• Items varied in operation used in underlying
  function
• 33 responses to 11 items
• 16 of which had an additive underlying function
• Y X 2
• 10 had a combination underlying function
• Y 2X 2

9
Item Design Assessment

• We developed items that tapped each of these
  competencies
• Apply a Given Rule
• Determine Next Y value in Sequence
• Determine Near Y value in Sequence
• Determine Far Y value in Sequence
• Recognize a Rule
• Generate Rule Verbally
• Generate Rule Symbolically

10
Item Design Assessment

• We developed items that tapped each of these
  competencies
• Apply a Given Rule
• Determine Next Y value in Sequence
• Determine Near Y value in Sequence
• Determine Far Y value in Sequence
• Recognize a Rule
• Generate Rule Verbally
• Generate Rule Symbolically

Column A Column B
2 6
3 7
4 8
5 9
6
14
21
41
11
Item Design Assessment

• We developed items that tapped each of these
  competencies
• Apply a Given Rule
• Determine Next Y value in Sequence
• Determine Near Y value in Sequence
• Determine Far Y value in Sequence
• Recognize a Rule
• Generate Rule Verbally
• Generate Rule Symbolically

Column A Column B
2 6
3 7
4 8
5 9
6
14
21
41
12
Item Design Assessment

• We developed items that tapped each of these
  competencies
• Apply a Given Rule
• Determine Next Y value in Sequence
• Determine Near Y value in Sequence
• Determine Far Y value in Sequence
• Recognize a Rule
• Generate Rule Verbally
• Generate Rule Symbolically

Column A Column B
2 6
3 7
4 8
5 9
6
14
21
41
13
Item Design Assessment

• We developed items that tapped each of these
  competencies
• Apply a Given Rule
• Determine Next Y value in Sequence
• Determine Near Y value in Sequence
• Determine Far Y value in Sequence
• Recognize a Rule Verbal Symbolic
• Generate Rule Verbally
• Generate Rule Symbolically

What is a rule used in the table above to get the
numbers in column B from the numbers in column
A?   A) Multiply the number in column A by 2. B)
Divide the number in column A by 2. C)
Subtract 2 from the number in column A. D) Add
2 to the number in column A.
14
Item Design Assessment

• We developed items that tapped each of these
  competencies
• Apply a Given Rule
• Determine Next Y value in Sequence
• Determine Near Y value in Sequence
• Determine Far Y value in Sequence
• Recognize a Rule
• Generate Rule Verbally
• Generate Rule Symbolically

Column A Column B
2 6
3 7
4 8
5 9
6
14
21
41
The rule is that you add 4 to the A number to
get the B number
What is a rule for figuring out what number
belongs in column B?
15
Item Design Assessment

• We developed items that tapped each of these
  competencies
• Apply a Given Rule
• Determine Next Y value in Sequence
• Determine Near Y value in Sequence
• Determine Far Y value in Sequence
• Recognize a Rule
• Generate Rule Verbally
• Generate Rule Symbolically

Column A Column B
2 6
3 7
4 8
5 9
6
14
21
41
Write this rule as a number sentence, using A
to stand for any number in column A and B to
stand for any number in column B.
B A 4
16
Item Scores Coding

• Coding
• Each response only tapped one competency
• Each was coded as correct or incorrect

Item Design
Construct Map
Measure ment Model
Item Score
17
Data Collection Procedure

• 231 second through sixth grade students
• Middle class suburban community
• Predominantly Caucasian population
• During one 40 minute class period

18
Measurement ModelBased on Item Response Theory

• Item Response Theory encompasses a set of ways to
  mathematically model how both Student Ability
  Estimate and Item Difficulty are related to a
  students Item Responses
• It is a useful methodology to use when evaluating
  an assessment instrument
• both in terms of its ability to accurately
  estimate student ability
• but it also give metrics of the quality of each
  item on the instrument.

19
Measurement ModelWright Map
Student Ability Scores
Item Difficulty Scores
--------------------------------------------------
-------------
5
XX

XXXX

XX
4 XXXXXX

XXXXXXXX

XXXXXX

XXXXXXXXXXXXXXXX
3
XXXXXXXXXXXX

XXXXXXXXXXXXX

XXXXXXXXXXXXXXXXXX

XXXXXXXXXXX
2 XXXXXXXXXXXXXXXXXXXXXXXXXXXX

XXXXXXXXXXXXXXXXXXX

XXXXXXXXXXXXXXXXXXXXXXXXXXXXX

XXXXXXXXXXXXXXXXX
1 XXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXL4
XXXXXXXXXXXXXXXXXXXXXXXXXXXX

XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

XXXXXXXXXXXXXXXXXXXXXXXXXXXXL3
0
XXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXL3
XXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XL3 -1
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXL3

XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXL3

XXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXL2 L2
-2 XXXXXXXXXXXXXXXXXXXX
L2 L2 L2
XXXXXXXXXXXX

XXXXXXXXXXXXXXXXXXXXXXXXXXXX

XXXXXXXXXXXXX
-3
XXXXXXXXXXXXXX

XXXXXXXL1 L1 L1
XXXX

XXXL1 L1
-4
X

X
XX

• An Wright map generated by a Rasch model (a type
  of item response model) and was used in this
  evaluation
• Logit Scale (log-odds ratio)
• Student Ability Estimates
• Item Difficulties

20
Measurement ModelWright Map

• Wright Map
• An Wright map was generated by a Rasch model (a
  type of item response model) and was used in this
  evaluation
• Item difficulties based on the Wright maps were
  used in the development of our Construct map

Student Ability Scores
Item Difficulty Scores
XXXXXXXX
XXXXXXXXXXXItem
E 3
XXXXX-3-Item C Item
D XXXXXXXXXX

XXXXXXXXXXXX
2
XXXXXXXXX-2-

XXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXX
X 1
XXXXXXXXXXXXXXXXXXXX

XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXItem B

XXXXXXXXXXXXXXXXXXXXXXXXXX
0
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXItem A

This item difficulty is 3.1logits, or the average
student has a 0.28 probability of getting it
correct
This item has a difficulty level of .98, meaning
that the average student has a 0.47 probability
of getting it correct
21
Wright Map Addition Functions
XXXXXXXXXXXXXXXXXX

XXXXXXXXXXX
2
XXXXXXXXXXXXXXXXXXXXXXXXXXXX

XXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXX
XXXXXXX
XXXXXXXXXXXXXXXXX
1
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX7-16

XXXXXXXXXXXXXXXXXXXXXXXXXXXX

XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

XXXXXXXXXXXXXXXXXXXXXXXXXXXX6-6
0
XXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXX6-14
XXXXXXXXXXXXXXXXXXXXXXXXX
XXXXX4-15
-1 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX4-13

XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX3-12

XXXXXXXXXXXXXXXXXXXXXXX

XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX3-4 3-5
-2
XXXXXXXXXXXXXXXXXXXX5a-9 2-10 2-11

XXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXX
XXXXXXXXX
XXXXXXXXXXXXX
-3
XXXXXXXXXXXXXX

XXXXXXX1-3 1-7 1-8

XXXX

XXX1-1 1-2
-4 X

X

XX


Each 'X' represents 0.4 cases

0
7) Generate Rule Symbolically 6) Generate Rule
Verbally 5) Recognize a Rule 4) Determine Far Y
value in Sequence 3) Determine Near Y value in
Sequence 2) Determine Next Y value in Sequence 1)
Apply a Given Rule
22
Wright Map Combination Functions
--------------------------------------------------
------------------------------------- 4
XXXXXXXXX

XXXXXXXX

XXXXXXXXXXX4-8
3
XXXXX3-6 4-7 6-9 7-10

XXXXXXXXXX
XXXXXXXXXXXX
2
XXXXXXXXX

XXXXXXXXXXXXXX

XXXXXXXXXXXXXXXXXXXXX
1
XXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXX2-5
XXXXXXXXXXXXXXXXXXXXXXXXXX
0
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX5b-4

XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX5a-3

XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
-1
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
-2
XXXXXXXXXXXXXXXXXXXXXXXXXXX

XXXXXXXXXXXXXXXXXXXXXXXXXXX1-2

XXXXXXXXXXXXXXX
-3
XXXXXXXXX

XXXXXXXXXXX1-1

XXXXXXX
-4 XXXXXXXXX


Each 'X'
represents 0.4 cases
7) Generate Rule Symbolically 6) Generate Rule
Verbally 5) Recognize a Rule 4) Determine Far Y
value in Sequence 3) Determine Near Y value in
Sequence 2) Determine Next Y value in Sequence 1)
Apply a Given Rule
23
Construct Map

• Construct Map
• A representation of the continuum of knowledge
  that people are thought to progress through for
  the target construct (Wilson, 2005)
• Placed competencies into a hierarchy based on
• We used item difficulty scores from IRT measures
• Their clumping on the Wright maps
• From theory

24
Mapping of Competencies into Construct Map Levels
Level Description Competencies
Level 4 Generate Symbolic Rule - Generate an explicit symbolic rule
Level 3 Generate Use Verbal Rule - Generate an explicit verbal rule - Complete a function table with missing values
Level 2 Recognize Rule Determine Next - Select a correct rule out of several choices - Determine the next Y value in a function sequence
Level 1 Apply Rule - Use a given rule to determine new Y values
7. Generate Rule Symbolically 6. Generate Rule
Verbally 5. Recognize a Rule (verbal/symbolic) 4.
Determine Far Y value in Sequence 3. Determine
Near Y value in Sequence 2. Determine Next Y
value in Sequence 1. Apply a Given Rule
25
Benefits of a Construct Modeling Approach

• First, it elucidated the relative difficulty of
  functional thinking abilities, and at times this
  was not in line with our predictions.
• Second, the resulting assessment is a criterion
  referenced measure which is particularly
  appropriate for assessing
• Students ability estimate levels
• Learning gains from an intervention

26
Summary

• Identified key competencies that are important
  for elementary-level functional thinking, with a
  focus on function table problems
• These competencies were then incorporated into an
  assessment
• Student performance data was used to develop a
  construct map, or proposed knowledge progression,
  of elementary-level functional thinking abilities
• The resulting construct map provided insight into
  the acquisition of functional thinking knowledge
  in elementary-school students
• This can be used as a research tool, and to guide
  instructional sequences for students

27
Thank you For more informationhttp//peabody
.vanderbilt.edu/earlyalgebra.xml
The first author is supported by a predoctoral
training grant provided by the Institute of
Education Sciences, U.S. Department of Education,
through Grant R305B040110 to Vanderbilt
University. The opinions expressed are those of
the authors and do not represent views of the
U.S. Department of Education.
28
(No Transcript)
29
Wright Map - Multiplication
--------------------------------------------------
------------------------------------- 5
XXXXXXXXXXXXXXXXX

XXXXXXXXXXXXXXXXXXXXX
4
XXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXX
3 XXXXXXXXXXXXXXXXXXXXXXXXXXX

XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX7-7

XXXXXXXXXXXXXXXXXX3-4
2
XXXXXXXXXXXXXXXXXXXXXXXXXXXXX4-3 3-5

XXXXXXXXXXXXXXXXXX6-6
1 XXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXX5-1
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
X 0
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX2-2

XXXXXXXXXXXXXXXXXXXXXXXXXXXXX
-1 XXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXX
-2 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

XXXXXXXXXXXXXXXXXXXXXXXXXX
-3
XXXXXXXXXXXXXXXXXXXXXXXXXX


Each 'X' represents 0.3
cases

• Apply a Given Rule
• Determine Next Y value in Sequence
• 3. Determine Near Y value in Sequence
• 4. Determine Far Y value in Sequence
• 5. Recognize a Rule (symbolic)
• 6. Generate Rule Verbally
• 7. Generate Rule Symbolically