David Millard - PowerPoint PPT Presentation

1 / 14
About This Presentation
Title:

David Millard

Description:

PeerPigeon is an out-of-the-box system that runs any type of peer review process ... Encouraging students to take responsibility for their own learning ... – PowerPoint PPT presentation

Number of Views:39
Avg rating:3.0/5.0
Slides: 15
Provided by: davidm94
Category:
Tags: david | millard | pigeon

less

Transcript and Presenter's Notes

Title: David Millard


1
Towards a Canonical View of Peer Assessment
  • David Millard
  • Karen Fill
  • Hugh Davis
  • Lester Gilbert
  • Gary Wills
  • Learning Societies Lab, University of
    Southampton, UK

2
PeerPigeon
  • PeerPigeon is an out-of-the-box system that runs
    any type of peer review process by supporting a
    generalised case of Peer Review.
  • Overview
  • Why Peer Assessment / Review?
  • How do we provide for the generalised case?
  • Introduction to PeerPigeon

3
Peer Assessment
  • Peer Assessment has many advantages
  • Giving a sense of ownership of the assessment
    process, improving motivation
  • Encouraging students to take responsibility for
    their own learning
  • Treating assessment as part of learning, so that
    mistakes are opportunities rather than failures
  • Practicing the transferable skills needed for
    life-long learning, especially evaluation skills
  • Using external evaluation to provide a model for
    internal self-assessment of own learning
  • Encouraging deep rather than surface learning.
  • Bostock S., Student peer assessment, Higher
    Education Academy Article, 16 Mar 2001

4
Case Studies
  • Simple - The simplest form of peer review is
    where authors and reviewers are paired together
  • Round Robin - Where participants are grouped, and
    each participant reviews the work of every other
    other participant in their group.
  • Group Activity - Where a group of authors work
    together to produce an artefact, and then that
    artefact is reviewed by a third party.
  • Group Review - Where a group of authors work
    together to produce an artefact, and then
    individually review the efforts of their group.

5
Case Studies
  • Committee Review - Where a group of reviewers act
    together and look at several different artefacts
    in order to produce one review. In the research
    community we are familiar with this as the
    conference committee stage of peer review
  • Multiplicity - Where multiple authors create
    multiple artefacts which are then independently
    reviewed by multiple reviewers. For example,
    where students give a paper and presentation and
    are assessed by their classmates on both

6
Common Review Cycle
  • All these cases can be thought of as being built
    of common review cycles
  • The cycle can be started in any one of its three
    states. For example, to begin an activity the
    student may be asked to Generate an artefact, to
    Submit an existing artefact, or the tutor may
    provide it, in which case the first task is to
    Distribute it.
  • The cycles can be interleaved, and occurring in
    parallel as well as in sequence.
  • Each stage within the process may involve 1...n
    participants (authors/tutors/reviewers),
    producing 1...m resources (artefacts/reviews/marks
    ).

7
  • Example of Multiplicity
  • n students, m tutors
  • each student delivers a presentation and answers
    questions (i.e. two artefacts)
  • students and tutors review/mark the presentations
  • only tutors review/mark the answers

8
  • Example of Multiplicity
  • n students, m tutors
  • each student delivers a presentation and answers
    questions (i.e. two artefacts)
  • students and tutors review/mark the presentations
  • only tutors review/mark the answers

Cycle Creators Authors/ Reviewers Resources Artefacts/ Reviews Receivers Reviewers /Authors
1 1 1 n1
2 n m 2
3 1 m n1
4 n1 n1 1
5 1 1 1
6 1 1 1
9
Use Case for PeerPigeon
  • What is PeerPigeon?
  • The complexity of the system is encapsulated in a
    Peer Review Plan with three elements
  • A Peer Review Pattern (an ordered description
    of the cycles of peer review and the roles of the
    participants in each cycle).
  • A number of actual Participants (possibly
    arranged into Groups) that populate the roles in
    the plan.
  • A Schedule of upcoming dates and times, that
    ties the pattern to a real timescale.

10
Peer Review Patterns
  • We need to formalise the cycles (make them
    machine readable)
  • Generation occurs outside of the system
  • Submission is simple
  • We expect X to submit Y
  • Distribution is complex
  • Where does Y go next!
  • We represent Distribution using transforms
  • a participant takes an existing artefact and
    produces a new artefact based on it
  • E.g. a reviewer takes a paper and transforms it
    into a review.

11
Peer Review Patterns
  • Major challenge with specifying the transforms
  • Irreducible complexity
  • An infinite number of ways to choose the
    allocation algorithm
  • We have used a Domain Specific Language (DSL)
  • a programming language for a specific set of
    tasks.
  • a clean syntax that can describe the specific
    steps/information
  • an agile alternative to XML
  • (no need for schema/dtds, a data file, or a
    parser)
  • Ruby is an ideal language for DSLs
  • Scripting language
  • Dynamic programming features
  • Lots of support / active community

12
PeerPigeon DSL Example
  • Another simple review pattern
  • in a group each person produces a paper and
    reviews the paper of someone else
  • A simple algorithm
  • pass each paper along to the next person
  • In our Ruby DSL
  • cycle review do c
  • c.description 'Write Review'
  • c.deadline '2 days from now','4 days from now'
  • c.transform paper_1, 2, review_1
  • c.transform paper_2, 3, review_2
  • c.transform paper_3, 1, review_3
  • end
  • c.distribution reviewers, lambda
    group,person c.transform
  • group.wrap('paper', person, -1),
  • person,
  • group.wrap('review', person, 1)

13
PeerPigeon Prototype
14
Conclusions and Future Work
  • We have shown that Peer Review can be generalised
  • The building blocks are Peer Review Cycles
  • Generate -gt Submit -gt Distribute
  • The Distribution can be encoded as a sequence of
    transforms
  • Paper 1 is transformed by Person 2 into Review 1
  • The Distribution algorithm is irreducibly complex
  • But can be handled by a Domain Specific Language
    (DSL)
  • We are currently prototyping the PeerPigeon
    system
  • Implements the generalised use case
  • Written in Ruby on Rails (Ruby DSL)
  • An alpha version is available from our website
  • Release version (late beta) is planned for end of
    August
  • www.peerpigeon.ecs.soton.ac.uk
Write a Comment
User Comments (0)
About PowerShow.com