David Millard

1
Towards a Canonical View of Peer Assessment

• David Millard
• Karen Fill
• Hugh Davis
• Lester Gilbert
• Gary Wills
• Learning Societies Lab, University of
  Southampton, UK

2
PeerPigeon

• PeerPigeon is an out-of-the-box system that runs
  any type of peer review process by supporting a
  generalised case of Peer Review.
• Overview
• Why Peer Assessment / Review?
• How do we provide for the generalised case?
• Introduction to PeerPigeon

3
Peer Assessment

• Peer Assessment has many advantages
• Giving a sense of ownership of the assessment
  process, improving motivation
• Encouraging students to take responsibility for
  their own learning
• Treating assessment as part of learning, so that
  mistakes are opportunities rather than failures
• Practicing the transferable skills needed for
  life-long learning, especially evaluation skills
• Using external evaluation to provide a model for
  internal self-assessment of own learning
• Encouraging deep rather than surface learning.
• Bostock S., Student peer assessment, Higher
  Education Academy Article, 16 Mar 2001

4
Case Studies

• Simple - The simplest form of peer review is
  where authors and reviewers are paired together
• Round Robin - Where participants are grouped, and
  each participant reviews the work of every other
  other participant in their group.
• Group Activity - Where a group of authors work
  together to produce an artefact, and then that
  artefact is reviewed by a third party.
• Group Review - Where a group of authors work
  together to produce an artefact, and then
  individually review the efforts of their group.

5
Case Studies

• Committee Review - Where a group of reviewers act
  together and look at several different artefacts
  in order to produce one review. In the research
  community we are familiar with this as the
  conference committee stage of peer review
• Multiplicity - Where multiple authors create
  multiple artefacts which are then independently
  reviewed by multiple reviewers. For example,
  where students give a paper and presentation and
  are assessed by their classmates on both

6
Common Review Cycle

• All these cases can be thought of as being built
  of common review cycles
• The cycle can be started in any one of its three
  states. For example, to begin an activity the
  student may be asked to Generate an artefact, to
  Submit an existing artefact, or the tutor may
  provide it, in which case the first task is to
  Distribute it.
• The cycles can be interleaved, and occurring in
  parallel as well as in sequence.
• Each stage within the process may involve 1...n
  participants (authors/tutors/reviewers),
  producing 1...m resources (artefacts/reviews/marks
  ).

7

• Example of Multiplicity
• n students, m tutors
• each student delivers a presentation and answers
  questions (i.e. two artefacts)
• students and tutors review/mark the presentations
• only tutors review/mark the answers

8

• Example of Multiplicity
• n students, m tutors
• each student delivers a presentation and answers
  questions (i.e. two artefacts)
• students and tutors review/mark the presentations
• only tutors review/mark the answers

Cycle Creators Authors/ Reviewers Resources Artefacts/ Reviews Receivers Reviewers /Authors
1 1 1 n1
2 n m 2
3 1 m n1
4 n1 n1 1
5 1 1 1
6 1 1 1
9
Use Case for PeerPigeon

• What is PeerPigeon?

• The complexity of the system is encapsulated in a
  Peer Review Plan with three elements
• A Peer Review Pattern (an ordered description
  of the cycles of peer review and the roles of the
  participants in each cycle).
• A number of actual Participants (possibly
  arranged into Groups) that populate the roles in
  the plan.
• A Schedule of upcoming dates and times, that
  ties the pattern to a real timescale.

10
Peer Review Patterns

• We need to formalise the cycles (make them
  machine readable)
• Generation occurs outside of the system
• Submission is simple
• We expect X to submit Y
• Distribution is complex
• Where does Y go next!
• We represent Distribution using transforms
• a participant takes an existing artefact and
  produces a new artefact based on it
• E.g. a reviewer takes a paper and transforms it
  into a review.

11
Peer Review Patterns

• Major challenge with specifying the transforms
• Irreducible complexity
• An infinite number of ways to choose the
  allocation algorithm
• We have used a Domain Specific Language (DSL)
• a programming language for a specific set of
  tasks.
• a clean syntax that can describe the specific
  steps/information
• an agile alternative to XML
• (no need for schema/dtds, a data file, or a
  parser)
• Ruby is an ideal language for DSLs
• Scripting language
• Dynamic programming features
• Lots of support / active community

12
PeerPigeon DSL Example

• Another simple review pattern
• in a group each person produces a paper and
  reviews the paper of someone else
• A simple algorithm
• pass each paper along to the next person
• In our Ruby DSL
• cycle review do c
• c.description 'Write Review'
• c.deadline '2 days from now','4 days from now'
  
• c.transform paper_1, 2, review_1
• c.transform paper_2, 3, review_2
• c.transform paper_3, 1, review_3
• end

• c.distribution reviewers, lambda
  group,person c.transform
• group.wrap('paper', person, -1),
• person,
• group.wrap('review', person, 1)

13
PeerPigeon Prototype
14
Conclusions and Future Work

• We have shown that Peer Review can be generalised
• The building blocks are Peer Review Cycles
• Generate -gt Submit -gt Distribute
• The Distribution can be encoded as a sequence of
  transforms
• Paper 1 is transformed by Person 2 into Review 1
• The Distribution algorithm is irreducibly complex
• But can be handled by a Domain Specific Language
  (DSL)
• We are currently prototyping the PeerPigeon
  system
• Implements the generalised use case
• Written in Ruby on Rails (Ruby DSL)
• An alpha version is available from our website
• Release version (late beta) is planned for end of
  August
• www.peerpigeon.ecs.soton.ac.uk