Title: Computerised Peer-Assessment
1Computerised Peer-Assessment
Lecturer getting out of doing marking
- Phil Davies
- University of Glamorgan
- South Wales
2Need for assessment?
- As tutors we are trying to separate the sheep
from the goats via the assessment process. - This can often be difficult with the time
constraints imposed on tutors, in what Chanock
describes as more goat-friendly times (Chanock,
2000). - Problem . Feedback against time!
3Defining Peer-Assessment
- In describing the teacher ..
- A tall b, so he was. A tall thin, mean
b, with a baldy head like a lightbulb. Hed
make us mark each others work, then for every
wrong mark we got, wed get a thump. That way
he paused we were implicated in each others
pain
McCarthys Bar (Pete McCarthy, 2000,page 68)
4Why Peer-Assessment
- Working together to learn (and be assessed)
- Overcome Perceptions
- Staff gt Lecturer getting easy life not marking
- Studentgt Lecturer getting easy life not marking
- Student Awareness of Benefits . IMPORTANT
TANGIBILITY - Success of dependant upon scalability
(computerisation) .. Student Numbers have risen
dramatically since 1991 without a concomitant
increase in resources (Pond et al, 1995)
5Computerised Peer-Assessment
- CAP System (2000)
- Permits students to mark comment the work of
other students . (normally 6-8) - Also initial self-assess stage (reflection)
used as a standard of expectation - Internet, not Web-based system (developed in
Visual Basic / Access)
6(No Transcript)
7Having done the marking, what next?
- Students should receive feedback
- What feedback?
- Marks
- Comments
- Which is most important?
- To students or staff
8(No Transcript)
9AUTOMATICALLY EMAIL THE MARKER .. ANONYMOUS
10What should the marker do?Reflect
- Read the essay again
- Take into account the essay owners comments
- Further clarification (if it is needed, then is
this a black mark against the marker?) - Try to appease the essay owner?
- Modify mark based upon reflection?
- Give more constructive feedback
11(No Transcript)
12Exercise One
- In the top box what are the inherent advantages
and disadvantages of peer-assessment - Middle Box Mark (/10) and comment to support
- Bottom Box Mark the marker /10 and comment to
support - Now give the work back!
- Which of the above required higher order skills?
13Must be rewarded for doing the mark for marking
process .. Based on quality
- How to judge?
- Standard of expectation (self-assessment)
- Marking consistency
- Commenting, quality, measure against mark
- Discussion Element
- Need for additional comments black mark?
- Reaction to requests / further clarification
14(No Transcript)
15How easy to get an automated mark for marking?
- Statistically fairly easy to create a mark for
marking based upon marks - Take into account high and low markers
- Standard of expectation
- Consistency judge against final mark awarded
for an essay (compensated median) - What about the comments?
16Feedback Index
- Produce an index that reflects the quality of
commenting - Produce an average feedback index for an essay
(also compensated?) - Compare against marker in a similar manner to
marks analysis - Where does this feedback index come from and is
it valid?
17The way to get the feedback index?
- Develop an application??
- C-Rater?
- Spelling Mistakes
- Similar Meanings?
- That was cool
- Really Choc
- Really Good Essay
- Manually
18(No Transcript)
19Commonality!!
- In the 67 essays that were marked
- Only 96 comments
- 44 positive and 56 negative
- Highly critical if something not explained
properly - Comments grouped into 10 categories
- Need to QUANTIFY these comments .. Feedback index
- Create a database holding positive negative (by
category)
20(No Transcript)
21(No Transcript)
22(No Transcript)
23(No Transcript)
24(No Transcript)
25(No Transcript)
26-7 -6 -5 -4 -3 -2 -1 -0 0 1 2 3 4
34 34 51 56 52 56 61 63 70 64 68 70
39 52 61 46 57 58 62 72 65 70
44 45 52 54 66 59 62 69 68
34 35 57 66 65 65 66 69
56 59 68 64 63 71
61 59 66 66 66 66
55 48 60 69 68
61 57 58 65 73
59 61 60
63
62
68
34 37.8 45.8 59.6 55.6 58.8 62.9 62.8 67.5 68 68 70
27General points of note
- Main criteria used for peer-marking
- Explanations, conclusions, references, examples
- Better students were critical (upper quartile)
- Better understanding permitting criticism?
- Confidence?
- Hostility?
- Weaker student in cloud cuckoo land
consistently - Consistency of markers in marks comments
30-34, 35-39, 55-59 65-69 70-74 - Note 70-74 OVER-Comment Mark
28Time Consuming?
- Can we formulate the marking process
- Take away need for quantification process of
analyzing comments - Is it still peer-assessment if the students are
told what to say?
29(No Transcript)
30Exercise Two
- I think youve missed out a big area of the
research - Youve included a big chunk that you havent
cited - There arent any examples given to help me
understand - Grammatically it is not what it should be like
- Your spelling is atroceious
- You havent explained anything to me
- Youve directly copied my notes as your answer to
the question - Most of what youve said isnt about the question
31(No Transcript)
32STUDENT FRED REFERENCES Positive Negative
.
Personal Valuation 5, 3, 2, 1 3, 1, 2
33Is it my job to teach students how to write
essays, etc?
- Assessment MUST be directed at subject skills
- Why bother writing essays, doing exam questions,
etc. doesnt relate to needs or learning
outcomes of subject - Post HND N-tier Assess the essays of the
final year (last year) - Preparation/Research Judge knowledge against
last years results .. Both marks comments - Mistake!!
34(No Transcript)
35Marks Marking Difference Feedback Difference Mapping for MCQ Essays
5 lt4 lt2 90 - 100
4 lt8 lt4 80 - 89
3 lt12 lt6 60 - 79
2 lt16 lt8 40 - 59
1 lt20 lt10 20 - 39
0 20 or more 10 or more 0 - 19
Who benefited the most by doing this exercise?
Cured plagiarism?
36Can peer-assessment benefit other subject areas?
- Java Programming with Coursemarker
- Stuart Lewis idea
- Students create a solution to a programming
assignment - Submission(s)
- Peer-Evaluate other solutions
- Comments Marks for Marking (weightings)
37CM
Modula-2 Java C
CourseMarker
Computer Assisted Teaching and Assessment
STUDENT
TEACHER
CourseMarker Core
Exercise Developm. System
Student Exercise Environment
- assignments
- exercises
- notes
- questions
File Storage System
- exercise setup
- submission
feedback and mark
- test methods
- solution template
- marking scheme
Marking System
Evaluation System
- final mark
- position in class
- course statistics
- course statistics
- flagging-up of problem cases
immediate support
comments / questions
FEATURES
- UNIX (Linux), Windows, Mac, based all
platforms - Assessment of text I/O assignments only
no marking of graphical output - remote student / teacher access distance
learning, open all hours
Advantages / Disadvantages
TEACHER
STUDENTS
- re-usability
- automated marking
- - fair
- - frees time
- plagiarism check
- steep learning curve
- difficult setup
- (but its getting easier)
- immediate feedback
- fast support
38PeerMarker Screen
39Student while marking
- Exposure to different solutions
- Development of critical evaluative skills
- Useful experience of reading code for future
employment situations - Plagiarism? Good solution / No understanding
40Student while reviewing feedback from peers
- Range of subjective marking
- Confirmation of objective automated marking
- Anonymous discussion between marker and marked
41Current position
- Test system working
- Changes following beta test in progress
- Plans to try sample study again (at a more
convenient time, and with added rewards!) - Employ 2nd Placement Student
- Graphical Interface
42Some Points Outstanding or Outstanding Points
- What should students do if they identify
plagiarism? - Is it ethical to get students to mark the work of
their peers? - Is a computerised solution valid for all?
- At what age / level can we trust the use of peer
assessment? - How do we assess the time required to perform the
marking task? - What split of the marks between creation
marking - BEST STORY
43Contact Information
- pdavies_at_glam.ac.uk
- Phil Davies
- School of Computing
- University of Glamorgan
- Innovations in Education Teaching International
- ALT-J
- CAA Conference site (excellent PDF resource)
44Time for questions?