Three strategies for assessment in autonomous language learning

1
Three strategies for assessment in autonomous
language learning

• Joan Jamieson, Northern Arizona University, USA
• Carol A. Chapelle, Iowa State University, USA

2
Three Strategies

• Adaptivity
• Feedback
• Self-assessment

3
An Adaptive Strategy

• Learner would benefit from more than one form of
  material
• Computer should select appropriate form based on
  responses to questions

4
Overview of LEA
Beginning Reading Beginning Listening Beginning
Writing
Results Recommendations
Interest and Ability Finder
Intermediate Reading Intermediate
Listening Intermediate Writing
Results Recommendations
Advanced Reading Advanced Listening Advanced
Writing
Results Recommendations
5
The Interest Survey

• Select test form
• Select recommendations

6
Items on Interest Survey
7
Example Strategies
8
A Feedback Strategy

• Learner benefits from total scores
• Learner might benefit more from part scores

9
Example Computing Total Score
10
Part Scores Reflect Subskills

• Tests are often made up of subskills
• Each item can be coded according to subskill
• Scores for subskills can be computed by including
  codes

11
Table of Specifications
12
Tags for LEO Tests
TAG What the TAG means L listening LIN listening
for information LID listening for
ideas G grammar G1 grammar point 1 G2 grammar
point 2 G3 grammar point 3 S speaking V vocabulary
R reading P pronunciation P1 pronunciation point
1 P2 pronunciation point 2
13
Tags in Script for Grammar Section
14
Using Tags with System Variables

• score yields percentage correct
• score (tag) yields percentage correct for any
  items with a given tag
• score (G2) yields percentage correct of 2nd point
  of grammarexpressions for suggesting

15
Combining Tags and System Variables
score (L G V S P R) n/m1
rawscore(LIN) / tqw(LIN)   n/m2
rawscore(LID) / tqw(LID)   n/m3 rawscore(G1)
/ tqw(G1)   n/m4 rawscore(G2) / tqw(G2)
n/m5 rawscore(G3) / tqw(G3)
16
Mock-up of Progress Report Screen
Progress Report LEO 3 Test Learners
name Score score (L G V S P
R) Language area Number correct/number of
items Listening for information n/m1 Listening
for ideas n/m2 Grammar (point1) n/m3 Gra
mmar (point2) n/m4 Grammar (point3) n/m5
17
Screen Shot of Progress Report
18
Using Tags to Report Scores
19
A Self-Assessment Strategy

• Learner may benefit by comparing his/her
  perspective of performance with score
• Computer can collect self-confidence data along
  with performance data

20
Example of Self-Confidence Item
Was your answer correct? How sure are you?
Click a circle below. Completely Not
sure at all sure
21
Superimposed Self-Assessment Item
Was your answer correct? How sure are you? Click
in a circle for each answer. 1.
2. 3.
Completely Not
sure sure at all
22
Computing Average Confidence (Tarone and Yule,
1989)
Circle clicked 5 4 3 2 1
total average
confidence correct answers 20 5
3 2 0 29 4.52 incorrect answers
0 0 4 5 2 11 2.00     (205
)(54)(33)(22)(01)/29 4.52  
(43)(52)(20)/11 2.00 Tarone, E., Yule,
G. (1989). Focus on the language learner.
Oxford, UK Oxford University Press.
23
Computing Self-Monitoring Index

• Derived by subtracting self-confidence rating on
  incorrect items from self-confidence rating on
  correct items
• 4.52 2.00 2.52
• Index ranges in value from 4 to - 4
• Messages could be provided instead of numbers

24
Self-Assessment Superimposed onto Progress Report
Self-Assessment You seem to be aware of your
own ability. When you gave the correct answer,
you were very sure you were correct. When you
gave the wrong answer, you were not too sure you
were correct.
25
Implementing Self-Assessment

• Tag self-assessment items ltSAgt
• Save value of rawscore (SA) separately for
  correct and incorrect items
• IF ANSWER 1 THEN SAOK SAOK rawscore (SA)
• IF ANSWER 0 THEN SANO SANO rawscore (SA)

26
Calculating Average Scores

• AVGSAOK SAOK / CORRECT ITEMS
• AVGSANO SANO / INCORRECT ITEMS
• MONITORING INDEX AVGSAOK-AVGSANO

27
Example of Computing Self-Assessment Scores
28
Three Strategies for Individualizing Assessment

• Adapting level, content, and recommendations
  based on learners responses
• Additional feedback in the form of diagnostic
  scores
• Self-assessment to heighten learners
  metacognitive awareness

29
Three strategies for assessment in autonomous
language learning

• Joan Jamieson, Northern Arizona University, USA
• Carol A. Chapelle, Iowa State University, USA