Managing the Assessment Lifecycle: principles and practices in the first year

1
Managing the Assessment Lifecycle principles
and practices in the first year

• Prof Keithia Wilson
• Program Lead for the FYE, Griffith Health
• GIHE Senior Fellow for the FYE
• 2010 ALTC National Fellow for the FYE
• 2007 Australian University Teacher of the Year

2
Acknowledgment to Country

• In the Spirit of Reconciliation
• Following on from Sorry Day
• I would like to acknowledge honour the
  Traditional Custodians of this land that our
  Logan campus is built on, the Yugambeh People,
  and pay respect to their Elders past present

3
Overview

• Principles practices for enabling commencing
  students success with FY Assessment
• Designing assessment
• Preparing students to undertake assessment
• Marking Assessment
• Post-Assessment feedback process to enable
  support learning
• 2. Strategies for academic recovery with
  at-risk commencing students

4
Overview

• Current Context for the FYE
• Purposes of FY Assessment Student Motivations
• The FY Assessment Lifecycle from the perspectives
  of
• Students and
• Staff
• Principles Models for enhancing effective
  Practice with FY Assessment

5
Information Sources

• The First Year Experience Assessment literature
  generally
• Findings from a recent ALTC Grant on First Year
  Assessment Practice (2009-2010)
• Findings from a current ALTC National Fellowship
  on the FYE working with FY teaching teams
  (2010-2012)
• Focus on the student voice understanding the
  student experience of assessment

6
Step 1 Context

• How can we improve the assessment process and
  experience for commencing students?
• Understanding the current context for the FYE

7
Federal Govt reform agenda in Higher Education

• Aims to
• Widen student participation in Higher Education
  A FAIR GO!
• Increase the access of students from low SES
  disadvantaged backgrounds to university (higher
  numbers - Target increase from 12 to 20 by
  2020)
• Increase the success of students from low SES
  disadvantaged backgrounds at university (higher
  retention)
• Moving from an elite model of HE (0-15
  population participation) to a mass model of HE
  (16-50) (Trow, 2004)
• 25 participation in Australia (50 USA)
• 1.3 of Indigenous Australians attend university
  (30 lower graduation rate)

8
Evolution of Approaches to FY Orientation
Engagement the FYE

• First Generation Strategies Co-Curricular - A
  focus on designing FYOE supplemental activities
  strategies which are outside of the classroom
• Second Generation Strategies Curricular - A
  focus on enhancing FY curriculum design, pedagogy
  assessment practices
• Third Generation Strategies Whole-of-Institution
  - A focus on an Institution wide approach to 1st
  2nd generation strategies, with practice
  standardisation QA mechanisms for continuous
  improvement
• Whole-of-School/Program - A focus on the
  strategic combination of 1st 2nd generation
  strategies for a particular disciplinary context
  (School or Program)

9
Understanding the Current Context for the FYE

• Meta Model 1 - FY Transition Practice
• Student Diversity
  Student Transition
• Course Design
• Course
  Delivery
• Course
  Assessment

10
Student diversity

• So what is important to understand about student
  diversity?
• Defining student diversity
• Assumed knowledge

11
What is Student Diversity?

• Traditional Students (TS)
• medium-high SES
• second generation
• higher entry levels
• full time
• on-campus
• Elite Model

• Non-Traditional Students (NTS)
• low SES
• first-in-family
• lower entry levels
• full-time working
• not on-campus much
• Indigenous
• NESB International, refugees
• disability
• home care responsibilities
• from rural remote settings
• Mass Model

12
What is assumed knowledge capabilities?

• Academic Skills
• Information Literacy
• Computer Literacy
• Reading Skills
• Written Communication
• Numeracy Skills
• Critical thinking analysis
• Independent Learning (self-regulation) viz. time
  on task, self-study, time management,
  uni-work-social life balance, successful student
  behaviour

13
What is assumed knowledge capabilities?

• Academic/Cultural/Social Capital the Hidden
  Curriculum
• Role Understanding student role expectations
  appropriate effective behaviour (rights
  obligations)
• Success Reading the academic context to
  accurately determine performance requirements re
  studying assessment
• Support Capacity for help-seeking without fear of
  negative labelling (dumb/stupid)
• Personal Identity Sense of belonging personal
  fit with university (overcome the outsider
  within phenomenon A stranger in a foreign
  land)

14
Griffith University context student profile

• Student enrolment of 40,000
• 70 of students are first-in-family at uni (NT) -
  FIF correlates with low SES lower entry
  scores/Ops
• 6th largest low SES student intake in Australia
  (16 overall, 30 Logan campus)
• 3rd largest Indigenous student intake in
  Australia (600)
• 30 International student enrolment (Nthn
  Europe, China, India, Indonesia, Middle East,
  Canada)
• 10 students studying in distance mode
• More of our students work in paid employment
  longer hours in employment than the national
  average (NTS reality)

15
Are NTS capable of being successful at university?

• The national research evidence shows -
• The success rate (or tendency to pass their
  years subjects) of low SES students is 97 of
  the pass rates of their medium high SES peers
  has been stable over the last 5 years (Bradley
  et al, 200830)
• However, they require higher levels of support to
  succeed e.g., financial assistance, academic
  support, mentoring counselling services
  (Commonwealth of Australia, 2009)

16
Therefore....implications for assessment practice

• Taking account of assumed entry level knowledge
  in unit content assessment design making this
  explicit to students
• Building foundational academic skill development
  into unit content assessment design
• Developing commencing students skills as
  independent, self-regulating learners across the
  first year
• Creating a unit program level culture that
  respects values diversity in all of its forms
  (age, gender, race etc.)

17
Student Transition

• So what is important to understand about Student
  Transition?
• An evidence-based model for conceptualising
  student transition
• Senses of success (Lizzio, 2006)
• Predictors of student success

18
Meta-model 2 The Five-Senses of Student Success
(Lizzio, 2006)
Sense of Capability
Sense of Connectedness
Sense of Student Identity
Sense of Purpose
Sense of Resourcefulness
19
What predicts commencing students satisfaction
with their degree program?
Sense of Purpose
Strongly Enhances
Enhances
Sense of Capability
Commencing Student Satisfaction
Sense of Connection
Enhances
Good Teaching
Enhances
Perceived Effectiveness of Orientation
Enhances
Enhances
Time on Task
20
What predicts commencing students academic
outcomes?
Academic Capital Low SES First in Family ESL
Reduces
Competing Demands Time in employment Time as carer
Reduces
Semester 1 GPA
Task Engagement _at_ Uni Attendance at
Orientation Time on task
Strongly Enhances
Prior Academic Achievement Entry Scores (OP)
Enhances
21
What predicts commencing students retention?
Academic Capital -
Competing Demands -
Semester 1 GPA
Student Retention into yr 2
Task Engagement _at_ Uni
Prior Academic Achievement
Sense of Purpose
Student Satisfaction
22
Therefore....Implications for practice

• Curriculum strategies for developing time on task
  self-regulation skills ? assists assessment
  success
• Strategies for enhancing sense of purpose in
  curriculum assessment practices
• Building academic social capital through unit
  assessment design throughout the FY
• Effective Orientation Programs early student
  engagement strategies to encourage attendance at
  Orientation

23
Step 2 Assessment Principles Practices

• So...what do students say about their experience
  of assessment?
• A quick overview of some salient research
  evidence!

24
Assessment characteristics which positively
influence student learning engagement

1. perceived validity of assessment tasks
   approaches
2. perceived extent to which the learning
   environment is empowering
3. perceived fairness of the learning environment,
   especially with assessment tasks practices

25
Students perceptions of the validity of
assessment tasks approaches (Sambell et al.,
1997)

• 3 sets of priorities -
• Educational values authentic/meaningful tasks,
  perceived to have long term benefits, applying
  knowledge
• Educational processes reasonable demands,
  encourages independence by making expectations
  clear
• Educational consequences rewards effort and
  breadth depth in learning

26
Students meta themes in assessment(Savin-Baden
, 2004)

• 2 forms of student disempowerment
• Unrewarded learning the relationship between
  quantity of work its weighting
• Disabling assessment mechanisms including
• Processes lack of information inadequate
  feedback
• Forms assessment methods that do not fit with
  espoused forms of learning

27
Students perceptions of fairness(Lizzio, Wilson
Hadaway, 2008)

• Strongly influenced by
• Extent to which they feel personally respected by
  academic staff (convenors sessional staff) in
  the learning assessment process relational
  culture
• Adequacy of the information support systems
  provided for them to do their job in relation
  to assessment

28
The Reality of Assessment for Commencing Students

• Student feedback from multiple sources indicates
• Strategic nature of Assessment for student
  engagement, success retention
• For many students assessment IS the learning
• Assessment items which are too difficult, not
  meaningful, and not clearly explained are a key
  cause of student drop-out in the first semester
• Well designed implemented assessment is key to
  student engagement, learning retention

29
Assessment Practice in the First Year

• So what is important to our understanding?
• A model of purposes of FY assessment student
  motivation with assessment
• A model for management of the FY Assessment
  Lifecycle informed by
• Student perspective
• Staff perspective
• Principles of Good Practice

30
Purposes of FY Assessment

• Diagnostic ?
• Transition enabling ?
• Motivating ?
• Formative ?
• Learning ?
• Summative/evaluative ?
• Terminal ?
• Meta-reflective ?

• Assessment of student readiness
• Assessment to aid transition to engagement with
  uni
• Assessment as stimulation for learning
• Assessment for learning
• Assessment as learning
• Assessment of learning
• Assessment of capability/mastery
• Assessment of learning process

31
Student view of motivating assessment (Wilson
Lizzio, 2011)

• Engaging Design
• Relevance (personal, academic, professional),
  intellectual challenge, teacher enthusiasm
• Enabling Management
• Task clarity, task scaffolding, formative
  conversations with support from staff
• Teacher Authority
• Clear and firm expectations standards,
  consequences explained, responsibility invoked

32
Meta-model 3 The first-year assessment lifecycle
student experience
6. Academic Outreach and Recovery
33
The first-year assessment lifecycle staff
experience
Design Phase
6. Academic Outreach and Recovery
Post-Assessment Phase
Student Preparation Phase
Assessment Phase
34
Principles of Effective Practice in First Year
Assessment

• Good Practice Principles rely on
• 1. A lifecycle approach to assessment (4 Phases)
• Design phase
• Student Preparation phase (pre-submission)
• Assessment phase (marking)
• Post-Assessment phase (feedback feed-forward)
• 2. A systems approach to assessment (3 Systems
  levels)
• Individual Unit/Course/Subject level
• First Year Program level horizontal vertical
  integration
• Degree Program level

35
1. Design Phase Individual Unit/Subject

• Fit for purpose Optimising student motivation
  engagement by designing assessment tasks which
  are perceived to be relevant meaningful to
  students their learning (sense of purpose)
• Level of difficulty complexity Designing
  tasks with a conscious understanding of the
  assumed entry level knowledge appropriate
  learning level of commencing students
• Progressive knowledge skill development
  Designing assessment tasks across a semester in
  such a way as to ensure effective, cumulative
  knowledge skill development
• Variety Ensuring a variety of different types
  of items both within a unit/subject, and across
  units /subjects in the same semester (FY program
  level planning)

36
1. Design Phase Individual Unit/Subject

• Early success -Optimising an experience of early
  success for students to build academic
  personal efficacy confidence
• Formative Assessment - Emphasising early
  formative assessment designed to develop skill
  confidence viz.
• Early weeks 4-6 is optimal
• Smaller pieces for either no marks, or fewer
  marks (10-20) to encourage recovery from
  possible failure (building hope)
• Speedy, quality feedback (peer or staff,
  individually or collectively) with feedback by
  weeks 7-8
• Self-assessment of all written items by
  self-evaluating on the identified criteria
  (builds meta-competence)

37
1. Design Phase Individual Unit/ Subject Level

• Written presentation of assessment tasks which
  present the task clearly and without ambiguity
• Start with the aim of the task
• Present the sequence of steps involved in the
  task, from the simple to the complex, thus
  breaking down large tasks into manageable chunks
• Refer students to the marking criteria for the
  task
• Ensure that all available information is provided
  as much as possible in the one place

38
1. Design Phase Essays a special case

• There are a range of data sources to indicate
  that essays in the first year are a source of
  great angst for both students and staff, and may
  be more complex than we (staff) think
• Professional Staff feedback
• Academic Staff feedback
• Student feedback
• Quality of FY essays, especially in the first
  semester
• Quality of student essays in the second third
  years beyond!
• Wilson Lizzio Findings from ALTC Grant
  2009-2010

39
Assessment Tasks exams laboratory reports
Engaging Assessment Design
Deep Approach to Learning
Assessment Outcomes Student Grades
Assessment Management Support
Assessment Task Efficacy
40
Assessment Tasks Oral Presentation
Engaging Assessment Design
Deep Approach to Learning
Assessment Outcome - Student Grades
Assessment Management Support
Assessment Task Efficacy
41
Assessment Tasks - Essay
Engaging Assessment Design
Deep Approach to Learning
Assessment Outcome - Student Grades
Assessment Management Support
Assessment Task Efficacy
42
1. Design Phase Essays a special case

• Semester 1 essay tasks that are
• Relatively short (e.g., 750-1000 words)
• Relatively simple (e.g., summary/description
  simple interpretative analysis such as answers to
  how or why questions)
• Include limited research requirements (e.g.,
  simple searches of readily obtainable
  information)
• Semester 2 essay tasks that are
• Longer (1,200-2,500 words)
• Incorporate more complex analysis (e.g., critical
  analysis evaluation tasks compare contrast)
• Include more complex research requirements

43
1. Effective FY Assessment Practice Principles
Design Phase FY Program Level

• Workload distribution Ensuring roughly equal
  workload between all units in a semester to
  assist development of time management skills
• Scheduling coordination of submission dates to
  stagger the student workload
• Threshold/Difficult Units - Identify front-load
  any threshold units in a semester to enable
  student success
• Variety of assessment types Ensure variety
  across units in a semester
• Group Work Coordinate any group-based
  assessment tasks across the FY Program
  preferably reduce to 1 unit only if the
  assessment requires additional meetings outside
  of class

44
Meta-model 4 Progressive Enabling How might we
effectively manage the assessment process?
Systems Awareness Help me by coordinating
the experience

Task Design Provide me with fit for purpose
tasks and roles
Self-Management Help me by managing yourself
45
Reflective Task Assessment Design Phase

• How does this information apply to your FY
  context?
• What are the opportunities for enhancing FY
  assessment design?
• What are the challenges with enhancing FY
  assessment design?

46
2. Student Preparation Phase Individual
Unit/Subject Level

• Motivating students
• 3 Domains of Motivation
• Articulating explicitly the academic relevance of
  each assessment task (the knowledge and skills
  that will be useful to students later in their
  course/degree)
• Articulating explicitly the personal relevance of
  each assessment task (the knowledge and skills
  that will be useful for students in the future)
• Articulating explicitly the professional
  relevance of each assessment task (the knowledge
  and skills that will be useful to students later
  in their career)

47
2. Student Preparation Phase Individual
Unit/Subject Level

• Motivating students
• 2 Types of Relevance
• Articulating future positive relevance
  (personally professionally) of the knowledge
  and skills to be gained from an assessment task
  to the course/degree/future employment
• Articulating future negative relevance of the
  knowledge and skills to be gained from an
  assessment task to the course/degree/future
  employment for students who may not engage
  sufficiently

48
2. Student Preparation Phase Individual
Unit/Subject Level

• Motivating students
• Challenge
• Articulating explicitly the intellectual
  challenge of each assessment task (the challenge
  offered to students to think and learn)
• Articulating to students the investment of work
  required to be successful with the task
  (encourages development of self-regulatory
  behaviour)
• Staff Stance
• Staff conveying to students their own personal
  enthusiasm for the task (staff engagement
  increase student engagement)
• Being clear and well organised (assists anxiety
  management increases performance ability).

49
2. Student Preparation Phase Individual
Unit/Subject Level

• Management Strategies -
• Scaffolding learning -Designing a process for
  scaffolding assessment preparation which prepares
  students for each assessment task
• Providing detailed goals, criteria, standards
  for each task to clarify what good performance is
  viz. clear goals standards
• Providing opportunities for students to actively
  engage with, potentially modify those goals,
  criteria, standards
• Providing high low quality examples of
  performance for each assessment task or a similar
  task
• Providing targeted resources such as practice
  items, quizzes, mini-essay writing, step-by-step
  processes organised sequentially etc.
• Providing multiple regular opportunities for
  discussion of assessment tasks requirements
• Ensuring consistent information resources are
  provided from all teaching staff (convenors
  tutors) on assessment tasks

50
2. Student Preparation Phase First Year Program
Level

• Clear Goals and Standards Ensuring all course
  convenors have detailed criteria for all
  assessment tasks in their units/courses
• Responsive culture Ensuring a responsive FY
  Program culture in all units/courses, including
  training of sessional staff ? consistent messages
  about performance success
• Consistency of referencing style Providing a
  single referencing style only for FY students for
  their FY of study for all units/courses in their
  Program
• Consistency of information storage by unit
  convenors for web-site information for all
  units (same folders)
• Consistent terminology to describe same types of
  assessment tasks across a program e.g., critical
  reflection/critical analysis/essay/critique

51
Progressive Enabling How might we effectively
manage the assessment process?
Facilitating Motivation Help me to engage
Systems Awareness Help me by coordinating
expectations
Task Design Provide me with fit for purpose
tasks and roles
Self- Management Help me by managing yourself
Building Capacity Help me to be task capable
52
Progressive Enabling How might we effectively
manage the assessment process?
Facilitating Motivation Help me to engage
Systems Awareness Help me by coordinating
expectations
Managing Information Help me to understand the
task
Relating Functionally Help me to feel
understood
Task Design Provide me with fit for purpose
tasks and roles
Managing Procedures Help me to navigate the rules
of the game
Facilitating Process Help me to solve problems
Self- Management Help me by managing yourself
Building Capacity Help me to be task capable
53
Reflective Task Student Preparation Phase

• How does this information apply to your FY
  context?
• What are the opportunities for enhancing FY
  student preparation with assessment ?
• What are the challenges with enhancing FY student
  preparation with assessment?

54
3. Assessment Marking Phase Individual
Unit/Subject Level

• Transparency Consistency Ensuring consistency
  of marking standards between markers by
• Marker preparation -
• having agreed, transparent , detailed standards
  for preparing markers to assess
• Marking trial
• initially marking 5 or so then meeting to
  compare standards of marking

55
3. Assessment Marking Phase Individual
Unit/Subject Level

• Marker Moderation -
• facilitating high quality moderation of final
  assessment marks (procedural justice) by staff
  teams viz.
• staff team meeting face-to-face to discuss
  ? education capacity building for staff re
  marking standards
• all staff reading all fails HDs
  (depending on numbers) to clarify understanding
  of high low end standards
• all staff reading re-assessing all
  assessment items on the margins of each grading
  category (high Ps low Cs etc)
• ? final re-adjustment of marks
• keeping copies of high low end examples
  to be used anonymously with future cohorts (with
  student permission)

56
3. Assessment Marking Phase Individual
Unit/Subject Level

• FY Assessment Debriefing Learning for students
• providing speedy feedback (2 weeks optimal)
• Providing quality individual feedback which can
  feed-forward into other assessment tasks -
• identifying achievements (encouragement)
• identifying clearly explicitly what
  students need to do
• to improve their performance in an
  encouraging way
• (developmental)
• sufficient comments to justify the mark
  (fair just)
• summarising cohort strengths weaknesses

57
4. Post-Assessment Phase Individual Unit/Subject
Level

• Timely feedback Ensuring students receive
  feedback within a short time frame (1-2 weeks is
  optimal) during semester
• Empowerment - Ensuring opportunities for
  individual and group discussion of marks and
  grades
• Academic recovery identifying and intervening
  with at-risk students who have failed their first
  assessment item.

58
4. Post-Assessment Phase First Year Program Level

• Empowerment - Creating a first year culture of
  ensuring opportunities for individual and group
  discussion of marks and grades
• Academic recovery Program level strategy by
  identifying front-loading the threshold (most
  difficult) unit/s with the aim of ensuring
  intervening with at-risk students who have failed
  their first assessment item to assist in their
  academic recovery
• .

59
Some Strategies for Academic Recovery

• Preventative Strategies
• Draft submissions for first assessment tasks with
  feedback provided
• Re-submission of a failed assessment task for a
  possible passing grade only (1 only in each
  semester, or semester 1 only, or the first year)

60
Potential hierarchy of academic recovery
interventions

• Students responsible for initiating help-seeking
  without prompting
• Email communication to all students inviting
  contact to discuss first assessment and providing
  information regarding support services
• In-class activity discussing first assessment
  performance and feedback
• Targeted written communication to students who
  failed or nearly failed first assessment inviting
  contact to discuss first assessment and providing
  information regarding support services

61
Potential hierarchy of academic recovery
interventions

• 5. Self-reflective workbook distributed to
  students for independent completion and
  invitation for follow-up contact
• 6. Phone call to students who failed or nearly
  failed first assessment inviting contact to
  discuss first assessment and providing
  information regarding support services
• 7. Targeted invitation to students who failed or
  nearly failed first assessment inviting
  participation in structured face-to-face
  consultation and planning session with tutor.

62
First-Assessment First-Feedback Academic Recovery
Intervention

• Key Idea
• Efficacy building for students who fail or
  marginal pass first assessment in a
  core/threshold course
• Key Aspects
• Students complete a self-directed workbook
• Individual structured session with tutor leading
  to an action plan
• Follow-up phone or email contact
• 40 uptake
• Participation results in a 10 increase in
  submission rates 20 increase in pass rates for
  2nd assessment item, 40 increase in passing
  the course overall

63
First-Assessment First-Feedback Academic Recovery
Intervention

• At-risk students self-reported evaluations of
  the process were uniformly positive
• Academic related learning (5.7/7)
• Personal development (5.02/7)
• Insight into reasons for under-performance
  (5.6/7)
• Increased efficacy optimism (5.6/7)
• Process rated as non-aversive (5.3/7)
• Tutors reported stronger relationships with
  students, higher attendance at tutes by those
  students, greater student engagement

64
Reflective Task Post-Assessment Phase

• How does this information apply to your FY
  context?
• What are the opportunities for enhancing FY
  post-assessment feedback processes?
• What are the challenges with enhancing FY
  post-assessment feedback processes?

65
Working with Diverse Students - implications for
curriculum design assessment

• Identifying our assumptions about the entry
  levels of Assumed Knowledge of our students (x
  course x first semester then first year
  overall) designing course content, structure
  assessment tasks to take account of FY students
  entry levels
• Scaffolding student understanding of assessment
  tasks by providing resources to make expectations
  explicit assist task understanding
• Scaffolding student learning in each unit/course
  by building required attitudes, knowledge
  skills progressively across the first semester
  first year
• TL in the FYE needs to be conceptualised as a
  social experience where students are provided
  with rich varied opportunities for interaction
  dialogue with peers academic staff we are
  building a relational school/program culture