Use SP Xpress to analyze students

1
Use SP Xpress to analyze students performance
evaluate the question items setting

• St. Anthonys School
• 16 Jan 2013

2
Background information
Assessment
Curriculum
3
Enhance the effectiveness of learning and teaching
4
Evaluation after Assessment
Why did students choose A, not B, C or D?
Are the questions too difficult or too easy?
What are the follow-up actions?
Have the students learned? Have they studied
hard?
5
(No Transcript)
6
Data Good to start

• Student MCI
• Item MCI
• Cronbach Alpha
• Graph shape

7
2 Modified Caution Indexes
Students Performance
Student Modified Caution Index
0gt MCIgt 0.3
Question items Performance
Item Modified Caution Index
0gt MCIgt 0.3
8
Student MCIxP Chart
2012-13 P.4 1st Reading Exam
9
Student MCIxP Chart
2012-13 P.4 1st Reading Exam
10
More focused strategies
Type A Type B
The teaching and learning are suitable for them. Encourage them to keep on. Know the difficult questions Miss the simple questions Careless mistakes Dont rely on luck
Type D Type C
Adjust the teaching content Raise their learning motivation Need more consolidation Answering techniques Careless mistakes Focus on the foundation Dont rely on luck
11
Item MCI xP Chart
2012-13 P.4 1st Reading Exam
12
Item MCI xP Chart
Need improvement
Difficult questions
Problem questions
13
MCI item P-value chart
Item Type 1112 P.3 Final Exam 1213 P.4 First Exam
A 20 17
B 1 2
C 1 0
D 3 7
Total no. of questions 25 26
According the Item Modified Caution Index, most
of the items set are normal.
14
Cronbach Alpha (Reliability)
Cronbach Alpha Reliability
0 0.3 low
0.7-0.9 acceptable
gt 0.9 high
15
2011-2012 P.3 First Exam
Blue line Students
Red line Items
Purple line Average
Slightly Easy
16
2012-2013 P.4 First Exam
Blue line Students
Red line Items
Purple line Average
Appropriate
17
Which question items should be improved?
18
Which question items are difficult to our
students?
19
SP Xpress analysis on 2011-2012 Primary 3 Third
Exam Paper

• SP Xpress analysis results
• Type A 20 items
• Type B 1 item
• Type C 1 item
• Type D 3 items
• Lets examine one item from each category (Type
  B, C, D)

20
Exam item example Type B

• On which floor of the centre, can Grandpa find
  presents for Jack?A. 1/F B. 2/F C. 3/F
  D.4/F

21
Possible reasons for item being Type B

• Type B Item is not difficult but requires
  amendments because many students couldnt answer
  it.
• Possible reasons
• Item design flaws
• Not close to what teacher taught in class
• Students lack specific training in reading comic
  strips

22
Teacher examined the Type B item

• Students found it difficult because
• Need to process and connect many ideas all at the
  same time

Question Comic strip
Which floor? 1/F,2/F,3/F,4/F?floor directory
of the Centre Star Shopping Centre
can Grandpa find presents We can buy all the gifts
presents? presents?robots?toys(4/F)
can Grandpa We refers to grandparents
23
Recommended solutions

• Increase students chances in reading English
  comic strips in classroom teaching or leisure
  reading
• Teacher needs to illustrate more clearly in
  connecting ideas
• Teacher needs to teach more explicitly which
  means more visual demonstrations

24
Exam item example Type C

• In which step does Jacks mum add soy sauce and
  wasabi to the sushi?
• A. Step 1 B. Step 3 C. Step 6 D.
  Step 10

25
Possible reasons for item being Type C

• Type C Item is difficult and requires amendments
  because many students couldnt answer it.
• Possible reasons
• Confusing description
• Question misleading students
• Question containing error(s)

26
Teacher examined the Type C item

• Students did not have the relevant personal
  experience in making and/or eating sushi
• The recipe did not mention dipping sushi into
  soy sauce or wasabi after the finish
• Without experience, students could not interpret
  the correct answer which is the final step
• This question contains the error which requires
  outside knowledge uncommon among the students

27
Exam item example Type D

• After cutting the vegetables and crab meat,
  Jacks mum ____.C.takes some rice wine vinegar
  and mixes it with the sushi rice

28
Features of Type D item

• Type D Item is difficult and doesnt require
  amendments. It is suitable for screening the high
  ability students. Low and mid- ability students
  cant answer it.
• Type D items should be kept at a low proportion
  in an assessment.
• The rephrasing is difficult for students to
  interpret the correct answer.

29
Analysis of TSA P.3 Reading Paper

• Predict P.3 students results in the upcoming TSA
  reading assessment
• Compare the overall students ability from
  different years
• Examine the neglected elements in our teaching
• Assumption TSA assessment covers the basic
  competency that all students are supposed to have
  acquired by the end of a learning stage

30
SP Xpress analysis on 2011 TSA P.3 Reading Paper
(3ER1)
2011-2012 P.3 2012-2013 P.3
Mean 19.28 18.95
SD 4.78 4.76
Type A 18 20
Type B 6 3
Type C 0 1
Type D 2 2
31
An Interesting Fact A common Type B item across
2 years

• Part 1B Question 1How many chapters are there
  in this book?
• Chapter / Table of Contents can be an obstacle
• Students never encounter Chapter in storybook
  teaching (replacing textbook teaching)

32
An Interesting Fact A common Type B item across
2 years

• Sam buys two sandwiches with the Rainbow Fast
  Food coupon. How many sandwiches can he get?C.
  three sandwiches ( Buy 2 1 Free)
• Students are not familiar with Buy 2 Get 1 Free
  concept even though they can understand every
  single word

33
An Interesting Fact A common Type D item across
2 years

• too. In the end, John wins the race. All the
  people cheer loudly
• Read lines 11-12. What do the people do when John
  wins the race?C. shout happily
• Students are not familiar with synonyms

cheer loudly
shout happily
34
Personal reflection

• Teachers become aware of the quality of the
  assessments
• Accumulate professional experience in setting
  better assessments
• Teachers prepared for the unexpected and
  unlearned
• Teachers become cautious and adjust their
  teaching

35
Things to do

• Suggest Ts reflect on their exam paper using
• SP Xpress
• Facilitate Ts process the data with technical
  supports
• Set up a school based question bank
• Nurture a culture on giving a more constructive
  feedback to both Ts and Ss after assessment

36
Thank you so much for your time!