Encouraging reflective learning through formative assessment: a learning outcomes approach

1
Encouraging reflective learning through formative
assessment a learning outcomes approach

• Arlëne G. Hunter
• Centre for Open Learning in Maths, Science,
  Computing and Technology (COLMSCT), The Open
  University

2
Session overview

• brief overview of the project
• development of an online formative assessment
  tool (SOFA)
• review perceptions of the value of the student
  online formative assessment as a reflective
  learning tool
• methods of evaluation
• outcomes and methods of engagement
• preliminary conclusions

3
Background to project the challenge

• new OU level 2 course on Earth System Science
• academically course is conceptually challenging
• students need to draw on broad range of skills
  and prior knowledge (Biology, Chemistry, Earth
  Sciences, Maths, Physics)
• text co-published with external partner altering
  standard method of in-text distance learning
  support
• Challenge
• devise mechanism to support and inform all
  stakeholders of academic progress throughout
  course
• SOFA Student Online Formative Assessment

4
(No Transcript)
5
(No Transcript)
6
Original goals/objectives

• formative
• safe learning environment to test depth of
  learning and application
• enhance student awareness of personal strengths,
  weaknesses, achievements and progression
• adaptive
• match questions to learner competence
  logistically challenging
• integrated
• embed SOFAs into assessment, course and programme
  strategies via learning outcomes approach
• involve all stakeholders in feedback process on
  academic progression
• timely
• offer just in time and self-directed study
  options

7
Initial challenges and problems

• student-course interaction an unknown entity
• how would they engage with an online formative
  assessment
• what academic/ time demands would affect
  engagement
• question setting
• what represents the norm
• adaptivity 10-15 question question bank
• dealing with non-standard characters
• coping with discursive questions and answers
• stepped learning
• designing questions not activities
• time and resource for development

8
Question Types
Question type Method of interaction
assign scientific term free text drag and drop
MCQ (1 or more answers) radial buttons
complete a table free text drag and drop
link geological processes/ interactions free text drag and drop
label diagrams drag and drop
simple/complex calculations (1-2 steps sig. figs sci. not, units) numeric/text entry use or find equation, graph, info
interpret graphs, diagrams, equations varied
multi-format combination
9
e.g. Assign scientific term
10
e.g. MCQ
11
e.g. Complete a table
12
e.g. Link geological processes
13
e.g. Label diagram
14
e.g. Two-step calculation
15
e.g. Interpret graphs
16
e.g. Multi-format question
17
Final formative feedback
18
Learning outcomes summary
19
Evaluation of SOFAs

• Metadata collected by system (all users)
• number of users (62 in total range 15-52 ave
  27), timing and frequency of use per student
• time spent per question/ per assessment (ave 70
  mins)
• all answers (analyse common errors difference
  in scores in chapters vs revision SOFAs)
• Question 11 at end of each SOFA (all users)
• quick feedback questions perceived usefulness
  test understanding, measure progress, revision
  aid
• free-text comments box
• generic questionnaire on Book 1 usage (whole
  community)
• which (if any) SOFAs used
• how did they help learning
• Success Case Method targeted interviews
• specific subsection of community
• exemplar success and non-success cases

20
Success Case Method (SCM) evaluation

• offers quick means of gaining quantitative and
  qualitative insight
• develop models of absolute success and absolute
  failure
• meaning of success will vary between stakeholders
• failure does not equal absence of absolute
  success
• collect extreme robust (provable) stories to
  identify drivers behind experiences
• focus on what, so what, now what?
• how to make improvements

21
SCM process

• students and tutors invited to complete generic
  questionnaire
• gain two perspectives on perceived value of SOFAs
• 12 respondents selected for detailed interviews
• 5 of all SOFA users interviewed
• 9 potential success cases
• 3 potential non-success cases
• (based on total of 270 users who accessed
  at least 1 SOFA between February May 2007
  ideally, 10 is preferred representative quota)

22
SCM Interview buckets

• Success interview buckets
• what was used that worked?
• what results were achieved?
• what good did it do (what was its value)?
• what helped?
• suggestions?

• Non success interview buckets
• barriers?
• suggestions?

23
SCM success case summary

• What was used that worked?
• access via course website Check your
  understanding...
• repeat function and changing questions
• instant feedback on effectiveness of their
  learning approach
• What results were achieved?
• obtained impartial insight into their
  strengths, weaknesses and misunderstandings
• demonstrated ongoing progress
• instant pacing mechanism review learning
  approach/materials
• What was its value?
• focussed learning and helped direct areas in need
  of extra work
• built confidence in personal abilities and
  progress
• provided real measure of personal achievement

? adaptive, timely, formative

• formative, integrated, adaptive

? formative, timely, integrated
24
SCM success case summary cont/.

• What helped?
• flexible, formative structure instantly
  adaptable to own needs targeted feedback on
  specific errors and omission
• positive affirmation of good work
• putting all learning (content and skills) into
  context
• Suggestions
• printable version of questions and final
  explanation for revision BUT recognition that
  targeted feedback needs SOFA to be online
• General benefits
• consolidated and clarified learning
• identified gaps, misunderstandings, unknown
  unknowns before it was too late and offered
  advice on how to improve
• shift from seeing reflective feedback to using
  focussed feedback

? formative, integrated
? formative, adaptable

• formative, adaptive, integrated, timely

25
SCM non success case summary

• Barriers general issues
• working online prefer paper, more flexible,
  more familiar
• academic level too high, so demoralising when
  wrong
• time pressures too much to do in course going
  online is time consuming
• accessing answers prefer immediate access to
  read alongside questions
• Barriers to value
• expected to get all questions right first time
  SAQ experience
• did not like being told wrong or to try again
• too many clicks to get to the final answer
• Suggestions
• focus on prior background skills and knowledge
  gap filling
• downloadable/ print-based question and answer
  immediately accessible
• colourful, fun and tactile

? formative, adaptive
26
Preliminary outcomes

• majority of students who tried them, liked the
  SOFAs and found them helpful
• but only reached on average 27 (total usage
  62) 15 did all chapter SOFAs
• how could remainder be encouraged to engage and
  continue to use
• individual students who did the SOFAs were
  repeating them
• quantitative data indicated general improvement
  in application of learning (content and skills)
  on repeats
• qualitative feedback demonstrates more confident
  learners
• greater personal awareness of efficiency of
  learning approach
• growing confidence in personal skills and sense
  of academic progress
• awareness of personal strengths, weaknesses and
  learning needs
• changing perception from reviewing feedback (i.e.
  what was wrong) to focussing on feeding forward
  (i.e. what needs to be developed)
• majority (success and non-success cases) still
  wanted hard copy version for revision
• outcome of formative versus summative usage?
• familiarity with paper over PC?

? formative, adaptive
? formative, timely
? formative, adaptive, integrated, timely
? formative, adaptive
27
Dr Arlëne G. HunterTeaching FellowCentre for
Open Learning in Maths, Science, Computing and
Technology (COLMSCT)The Open UniversityWalton
HallMilton KeynesEngland. MK7
6AAA.G.Hunter_at_open.ac.uk
28
Links open access to SOFAs

• Chapter specific SOFAs Book 1
• https//students.open.ac.uk/openmark/s279-07.book1
  chapter1world/
• https//students.open.ac.uk/openmark/s279-07.book1
  chapter2world/
• https//students.open.ac.uk/openmark/s279-07.book1
  chapter3world/
• https//students.open.ac.uk/openmark/s279-07.book1
  chapter4world/
• https//students.open.ac.uk/openmark/s279-07.book1
  chapter5world/
• https//students.open.ac.uk/openmark/s279-07.book1
  chapter6world/
• https//students.open.ac.uk/openmark/s279-07.book1
  chapter7world/
• Revision SOFAs random selection of questions
• https//students.open.ac.uk/openmark/s279-07.book1
  chapter8ran1world/
• https//students.open.ac.uk/openmark/s279-07.book1
  chapter8ran2world/
• https//students.open.ac.uk/openmark/s279-07.book1
  chapter8ran3world/
• https//students.open.ac.uk/openmark/s279-07.book1
  chapter8ran4world/

29
Links open access to SOFAs

• Working with diagrams
• https//students.open.ac.uk/openmark/s279-07.book1
  chapter8diagmodworld/
• https//students.open.ac.uk/openmark/s279-07.book1
  chapter8diagtaxingworld/
• Understanding geological processes
• https//students.open.ac.uk/openmark/s279-07.book1
  chapter8geoprocessmodworld/
• https//students.open.ac.uk/openmark/s279-07.book1
  chapter8geoprocesstaxingworld/
• https//students.open.ac.uk/openmark/s279-07.book1
  chapter8geotermsworld/
• Using geological terms
• https//students.open.ac.uk/openmark/s279-07.book1
  chapter8mathmodworld/
• https//students.open.ac.uk/openmark/s279-07.book1
  chapter8mathtaxingworld/
• Summarising information with tables
• https//students.open.ac.uk/openmark/s279-07.book1
  chapter8tablesworld/