Asynchronous Audio Feedback: Goodbye to the Grading Blues

1
Asynchronous Audio Feedback Goodbye to the
Grading Blues
Lori Kupczynski, Ed.D Jessica Sanchez University
of Texas Pan American
2
Challenges in Online Teaching

• Asynchronous courses do not provide instructors
  with paralinguistic cues that are present in the
  face to face classroom
• Communicating in text can be difficult and
  frustrating as we are unsure of whether or not
  our intent is conveyed

3
Online Learning

• Despite a great amount of research the definition
  of best practice and related elements remains
  somewhat amorphous
• Models that explain learning interactions are not
  as well defined as in the traditional classroom
  setting

4
Community of Inquiry Framework
Social Presence The ability of participants to
identify with the community (e.g., course of
study), communicate purposefully in a trusting
environment, and develop inter-personal
relationships by way of projecting their
individual personalities.
Cognitive Presence The extent to which learners
are able to construct and confirm meaning
through sustained reflection and discourse in a
critical community of inquiry
Teaching Presence The design, facilitation and
direction of cognitive and social processes for
the purpose of realizing personally meaningful
and educationally worthwhile learning outcomes
5
(No Transcript)
6
The Study and Its ContextOriginal study
conducted by Dr. Phil Ice, University of North
Carolina-Charlotte

7
Rationale

• Instructor immediacy behaviors (use of personal
  examples, humor, and openness toward and
  encouragement of student ideas and discussion) in
  online courses were a significant predictor of
  student learning (Arbaugh, 2006)

8
Text Will Work, However

• Various surveys of online learners indicate that
  they prefer multimedia over text only
  presentations of content
• However, they made us question whether applying
  media other than text to online interactions
  would be of benefit in projecting teaching
  presence

9
Audio Feedback

• Audio commenting tool in Adobe Acrobat Pro v.7
  was used to provide feedback on student
  assignments.
• Version 8 is currently available and has the same
  capability.

10
(No Transcript)
11
(No Transcript)
12
What they did (Ice, Curtis, Phillips Wells,
2007)
13
What they wanted to know

• RQ 1 Between audio and text-based student
  feedback in ALN, which do students believe is a
  more effective means of interaction with the
  instructor?
• RQ 2 To what degree do students believe audio
  feedback is an effective replacement of
  instructor/student interaction that typically
  occurs in traditional face to face classes?

14

• RQ 3 How does the use of audio feedback impact
  the sense of community in ALN?
• RQ 4 In what manner is perceived learning
  impacted by the use of audio feedback?
• RQ 5 What relationship exists between the use of
  audio feedback and student satisfaction?

15
What they looked at

• Alternating text-based (6) and audio (5) feedback
  on assignments in online course
• CI 687 Advanced Teaching Strategies
• Single item on course survey
• preference for audio versus text feedback
• Semi-structured post-course interviews with 27 of
  34 enrolled students
• Document analysis of final project
• Relation between feedback modality and level
  (Blooms) of content application
• Unsolicited Feedback (email)

16
Unsolicited Feedback

• Emails started coming in immediately after the
  first use of audio commenting
• Over 40 of students spontaneously emailed us
  about audio commenting 100 expressing
  satisfaction with the technique

17
Unsolicited Feedback - Example

• Weve had written comments twice and verbal
  comments twice now. Let me guess this is
  someones research project right? Let me just
  save you some time. The verbal feedback is much,
  much, much better than the written.

18
End of Course Survey Data

• Response rate 91

• 26 of 31 students preferred audio to text
• 4 students indicated no difference
• 1 student indicated N/A (due to technical
  problems defective sound card)

19
Semi-structured Interviews

• Four themes emerged from analyses of transcripts

• Theme 1 Ability to understand nuance (70 of
  students)
• Students indicated that they were better able to
  understand instructors intent
• Humor, encouragement and emphasis were all much
  more clear

20
Semi-structured Interviews

• Four themes emerged from analyses of transcripts

• Theme 2 Feelings of increased involvement (56
  of students)
• Students felt less isolated and were more
  motivated to participate

21
Semi-structured Interviews

• Four themes emerged from analyses of transcripts

• Theme 3 Content retention (44 of students)
• Students believed they retained audio feedback,
  and the content to which audio feedback was
  related, better than text-based feedback and
  related content.

22
Semi-structured Interviews

• Four themes emerged from analyses of transcripts

• Theme 4 Instructor Caring (30 of students)
• Students felt that audio was more personal than
  text
• Comments frequently related to nuance and tone of
  voice

23
Document Analysis- Quantity

• In final projects (series of thematic integrated
  lesson plans), students used content for which
  audio feedback was received approximately 3 times
  more often than content for which text-based
  feedback received
• Notice triangulation with Theme 3 from interviews

24
Document Analysis- Quality

• Students were 5 to 6 times more likely to apply
  content for which audio feedback was received at
  the higher levels of Blooms Taxonomy than
  content for which text-based feedback was
  received
• Further reinforced interview data

25
Subsequent Data Collection

• Triangulation and novelty effect

• Quantitative 312 students surveyed
• No decrease in preference after multiple
  exposures across semesters
• Qualitative 51 students interviewed
• The same themes emerged from content analyses of
  interview data

26
Efficiency across modalities

• Mean time to provide feedback
• text 13.43 minutes
• audio 3.81 minutes
• Mean quantity of feedback
• text 129.75 words
• audio 331.39 words

27
What weve learned so far

• Students really like audio feedback
• Understood instructor better, felt more cared for
  and more involved in course
• Audio feedback facilitated content acquisition
  and application
• Used content more often and at higher levels of
  Blooms taxonomy
• Audio feedback was efficient
• More feedback in less time

28
Further Research

29
Confirmatory Data

• Quantitative n 1138, Qualitative n 607
• 15 instituions
• Range AA Ph.D.
• 7 Likert-type Items
• Open Qualitative Item probing for more themes

30
Quantitative Items

• When using audio feedback, inflection in the
  instructors voice made his / her intent clear.
• M 4.53, SD .652
• The instructors intent was clearer when using
  audio than text.
• M 4.48, SD .587

31
Quantitative Items

• Audio comments made me feel more involved in the
  course than text based comments.
• M 4.38, SD .683
• Audio comments motivated me more than text based
  comments.
• M 4.46, SD .702

32
Quantitative Items

• I retained audio comments better than text based
  comments.
• M 4.31, SD .568
• Audio comments are more personal than text based
  comments.
• M 4.29, SD .544

33
Quantitative Items

• Receiving audio comments made me feel as if the
  instructor cared more about me and my work than
  when I received text based comments.
• M 4.38, SD .617

34
Additional Finding

• No additional themes revealed by analysis of
  qualitative data
• No difference among learner or institutional
  types
• Some indication that the technique may not be as
  effective if the instructor is not a native
  speaker more data needed

35
(No Transcript)
36
Audio and the CoI

• The following slides compare the findings of the
  Summer, 2007 multi-institutional CoI instrument
  validation (n 287) and responses from the
  aforementioned study (n 1138) that received
  audio feedback In the items addressed there was a
  significant difference (p gt .05) in responses

37
Teaching Presence 1

• The instructor was helpful in identifying areas
  of agreement and disagreement on course topics
  that helped me to learn.
• Summer 2007 / mean 4.12
• Audio group / mean 4.43

38
Teaching Presence 2

• The instructor encouraged course participants to
  explore new concepts in this course.
• Summer 2007 / mean 4.44
• Audio group / mean 4.58

39
Teaching Presence 3

• The instructor provided feedback that helped me
  understand my strengths and weaknesses relative
  to the courses goals and objectives.
• Summer 2007 / mean 4.28
• Audio group / mean 4.57

40
Social Presence

• Online or web-based communication is an excellent
  medium for social interaction.
• Summer 2007 / mean 3.90
• Audio group / mean 4.27

41
Cognitive Presence 1

• I felt motivated to explore content related
  questions.
• Summer 2007 / mean 4.31
• Audio group / mean 4.55

42
Cognitive Presence 2

• Reflection on course content and discussions
  helped me understand fundamental concepts in this
  class.
• Summer 2007 / mean 4.37
• Audio group / mean 4.49

43
Audio Feedback for Developmental Students

• Audio feedback used during the Fall 2007 semester
  at South Texas College
• Used for feedback with Developmental English
  students

44
The Rio Grande Valley
45
South Texas College

• offerings range from Associate Degrees and
  Certificates to a Bachelor of Applied Technology
• 5 campuses covering Hidalgo and Starr Counties
• Area population (Rio Grande Valley) over 85
  Hispanic

46
Student Population

• More than 75 of student population are first in
  family to attend college
• More than 95 of students are Hispanic
• More than 60 of students are female
• More than 75 of students are on some form of
  financial aid

47
Ethnographic Reporting

• Use of Asynchronous Audio Feedback during the
  Fall 2007 Semester
• Four sections of developmental English, ENGL0081,
  were taught using the blended format.
• Two sections of developmental English were taught
  fully online.

48
The Blended Sections-54students

• the format for feedback was explained in class.
• assignments were submitted and returned online.
• the instructor asked students for feedback on the
  use of audio feedback within their assignments
  and noted student responses throughout the
  semester.

49
Student Responses for Blended Sections

• Can you do that for all my papers? I usually
  never understand what teachers are trying to tell
  me. I just know that I got a bad grade.
• I really liked the feedback you gave. Its like
  you were talking to me about my paper and I could
  listen to it over and get what I missed.

50
The Online Sections23 students

• HI Gang,
• When I review and/or grade your essays, I'm
  going to be sending back my comments in pdf
  format. When you download the attachment, make
  sure you have the Adobe Acrobat Reader to be able
  to open up the file. You most likely already have
  it, but if not, you can download it from
  http//www.adobe.com/products/acrobat/readstep2.ht
  ml  This is a free download and will allow you to
  open the file.
• Once you have opened the file, you can click on
  the little speakers throughout. Double click on a
  speaker to hear my audio comments.
• This is a new method that I am trying with you.
  I hope that it assists you all as you work on
  your writing! After you have reviewed your first
  essay, I'd like you all to take a moment to
  comment on the audio feedback in our QA
  discussion forum. I am continually looking for
  new and better ways to help you all, and I think
  this one is really good. I look forward to your
  responses!
• Dr. Kupczynski

51
Student Responses for Online Sections

• I really like the audio feedback. It is a little
  more personal and less cluttered than comments
  littered throughout the document.
• I think you will like my final draft, Ms. K. I
  just listened to what you were saying and did the
  corrections at the same time. Its a lot better.

52
Student Responses for Online Sections contd.

• This really helped me. A lot of times, I dont
  understand what teachers want when they give back
  my papers because English is my second language
  so I dont understand the marks. When you talked,
  its like you were talking to me and I knew what
  I needed to fix.

53
Comparison of Audio-based vs. Text-based feedback

• The instructor has worked in this college for
  over 8 years, teaching developmental courses to a
  large Hispanic population.
• Previous feedback for online and blended sections
  was all text-based using MS Word Track Changes.

54
Differences in Content Revision

• Vague
• Wordy
• Unclear
• Sidetracking
• Redundant
• One idea per paragraph
• Coherency

55
Differences in Grammar Revision

• Run-on
• Fragment
• Comma Splice
• Verb Tense
• Awkward Phrasing
• Pronoun Agreement
• Verb Agreement

56
Instructor Reflection on the Comparison

• Students would often correct grammar errors but
  not content errors based on text-based feedback.
• Students would correct both content and grammar
  errors based on audio-based feedback.

57
The Exploratory Study

• According to an exploratory study at the same
  institution (Kupczynski, Ice, Davis, Callejo,
  2008), it appears that relationship with the
  instructor was far more important in the learning
  process than was the relationship between
  students and the content.

58
Audio-feedback as a signifier of Teaching Presence

• Multiple student comments referred to the idea
  that students felt the feedback was personal and
  that the instructor truly cared about student
  success.

59
Looking Forward

60
Multimedia Feedback

• Using highlighting tool provides students to
  review specific information while listening to
  instructor comments
• Using the pencil tool for brief positive
  affirmation increases student perceptions of
  connectedness with the instructor

61
Offline Applications

• Though embedded audio feedback was originally
  used in the online environment, it is equally
  powerful in the face-to-face classroom
• Term papers, spreadsheets, dissertation advising,
  graphic art projects, etc., etc.

62
Integration with the CS3 Suite

• Student artifacts developed in a host of CS3
  applications can be converted to PDFs and audio
  / multimedia commenting applied
• Photoshop, Illustrator, Dreamweaver, InDesign