Feedback and Presentation in the Microsoft Distributed Classroom Project

1
Feedback and Presentation in the Microsoft
Distributed Classroom Project

• Richard Anderson
• University of Washington and Microsoft Research

2
Overview

• Presentation system
• Develop system for lecture delivery that
  increases instructor flexibility
• Feedback system
• Develop system that supports increased in class
  communication
• Design experiment
• Experiment with system in actual classroom

3
Educational Technology

• in the winter of 1813-1814 I attended a
  mathematical school kept in Bostonon entering
  his room, we were struck at the appearance of an
  ample Blackboard suspended on the wall, with
  lumps of chalk on a ledge below, and cloths
  hanging at either side. I had never heard of
  such a thing before. Mr. May, 1866

4
Classroom Technology

• Black board
• White board
• Overhead projector
• Transparencies
• Persuasion, PowerPoint, LaTeX
• PC with data projector

5
Research goal

• Develop presentation system for PC/Data Projector
• Increase flexibility in lecture delivery
• Support greater communication
• Design for instructional use at the university
  level
• Evaluate based on pedagogical impact

6
Scenarios

• Distributed classroom
• Multi site video conferenced classes
• Data projectors in each room
• Possible desktop access
• Lecture hall
• Public projected display
• Wireless enabled class
• Possible student devices
• Laptop, tablet, handheld

7
Teaching styles

• Wide range of teaching styles and opinions
• Instructors will use technology in different ways
• Instructors will use technology in unexpected
  ways
• We are not comparing between different
  technologies
• PC delivery already established
• Some situations require PC delivery

8
DISC Project

• Internet based video conferenced classes
• Multicast technology to support multiple sites
• Deploy system on a PC
• Simple startup (the big red button)

9
Preliminary Study

• UW Professional Masters Program
• Video conferenced class
• Netmeeting for PowerPoint distribution
• Smartboard Electronic whiteboard
• Instructor interviews
• Student interviews
• Classroom observations
• Review of video archive (5 years of data)

10
Study conclusions

• Need to support handwriting and slides
• Current electronic whiteboards not adequate
• Important to integrate whiteboard and slides
• Every observed use of whiteboard was in the
  context of a slide
• Use pen based device as presenter console

11
Electronic whiteboard comments

• "The whiteboard would be better if both it and
  the slides could be seen at the same time the
  handwriting was captured more accurately if I
  write in anything other than slowly-drawn block
  letters, it looks like a heap of pick-up sticks."
• "The problem with the whiteboard is that, unless
  you draw very carefully, half of it won't come
  out, and similarly when you erase. This is enough
  of a hassle that I wound up not using it much."
•  

12
PowerPoint comments

• "I could do some of what I wanted with PowerPoint
  animations, but these took a huge amount of time
  to prepare. Previously I had used slide overlays
  (transparencies) a lot, frequently presenting a
  slide that was partially filled in (some
  background and a question) and then using class
  discussion to fill it in."
• "PowerPoint is also a pain for the same reason
  it's a pain in non-distance classes - the slides
  impose a rigid structure on the lecture, and make
  it more difficult to adjust to the interactions
  that occur during it. "

13
Presentation system

• Key idea integration of ink and slides
• Lecture from tablet pc
• Mouse input adequate for gesturing, but not for
  writing
• Write on PowerPoint
• Synchronized display of slides
• Real time transmission of ink

14
Implementation

• Export slides as images, broadcast images
• Advantages easy and it worked. Viewer does not
  need to have PowerPoint. Easier support of a
  range of devices.
• Disadvantages
• Loss of fidelity
• Modifying intermediate form
• Loss of animations, transitions

15
Technical Issue Slide distribution

• Current approach
• Broadcast slide deck using multicast
• Interpacket delay introduced to avoid network
  saturation
• Rebroadcast on request
• Issues
• Dealing with clients joining sessions
• Large images (instructors baby pictures)
• Recognized as a challenge for distributed
  presentations

16
Whiteboard features

• Scrollable overlays (virtual mylar)
• Simulate overhead projector
• Minimize or hide slide for whiteboard use
• Ink and transparent ink
• Region highlight
• Remote pointer

17
Classroom use

• Fujitsu tablet
• UW Distance master's class (Professional Master's
  Program, PMP)
• Database
• UW Lecturer, MSR classroom
• 3 hour evening lecture for 10 weeks
• Several sessions in introductory computer
  programming

18
Observation results

• Used for both highlighting and examples
• Lecturers used "dense PowerPoint"
• Presentations not designed for ink annotation
• Novelty wore off quickly
• Instructors soon became comfortable
• Used from podium
• Some difficulty writing on Fujitsu tablet
• User interface issues to be resolved

19
Unexpected use

• Alon Halevy "Could I get volunteers to come up
  and write their sample queries on the tablet"
• Martin Dickey. TV Talk show style. Took the
  tablet to the audience, and had them write on it.
  Wireless tablet.
• Used the device to allow new interaction.
  Technology to enhance classroom collaboration.

20
Current experiment

• Use presentation tool in Introductory Computer
  Programming, Summer 2002
• Evaluate tool in sustained, large lecture use
• Deeper questions
• Does the presentation tool enable a better
  educational experience?
• Does the presentation tool alter the pedagogy?

21
Research questions

• Gestures for lecturing from tablet
• Specific technology very important
• Limited experience with tablet pc
• Specific gestures for presentation
• Constraints imposed by performance environment
• Opportunities for manipulation of slide content

22
Feedback System

• Motivation and challenge
• Increase level of interaction in large lecture
  classes
• Give instructor information on level of
  understanding of lecture
• Real time feedback
• Students annotation of materials on personal
  devices
• Real time summarization for instructor

23
Prior work

• Telep (and others)
• Active class (UCSD)
• Class talk (UMass)
• Classroom 2000
• University of Maryland
• Results mixed

24
Challenges to Interaction in Large Classes

• Student apprehension
• Concerns about interrupting class for
  unimportant questions
• Comment verbalization
• Difficulty in expressing comments
• Feedback lag
• Questions not formed until topic has been changed
• Single speaker paradigm

25
Design Principles

• Non-verbal communication
• Anonymity
• Shared context for feedback
• Rapid, automatic synthesis of feedback
• Simple interfaces
• Value to all users

26
Important questions

• Will instructors be willing to use system?
• Will instructors be able to take advantage of
  feedback during lecture?
• Will students be willing to use system?
• Will students benefits to students outweigh the
  costs of using the system?

27
Feedback System

• Viewers select annotations from context menus
• Annotations associated with slide geometry
• Real time summary of annotations provided to
  viewer
• Potential to deploy on a wide range of devices
• Take advantage of existing student devices

28
(No Transcript)
29
(No Transcript)
30
Feedback categories

• Low impact to make comments
• Allow presenter to solicit particular feedback
• State of mind (Im confused, I understand)
• Suggested action (Explain, Elaborate, Example,
  Define)
• Pacing (Slower, Faster, Just Right)
• Presenter may set categories in a preprocessing
  phase

31
(No Transcript)
32
Summarization of feedback

• Presenter needs summarization if there is
  significant feedback
• Summarization done on "semantic regions"
• Identify slide geometry - rectangles
• Summarize comments in rectangles
• Alternate summarization methods
• Intensity or Histogram
• Comment menus also linked to regions

33
CSE 100 Experiment

• Design experiment
• Done in actual class
• Understand use and effect in real context
• Engineering a learning environment by introducing
  an intervention
• Study environment to guide design
• Conduct pen and paper feedback
• Use laptops in class for feedback

34
Study

• CSE 100 Fluency in information technology
• Large, diverse introductory class
• 150 students, large lecture format
• Observations
• Very limited student participation
• Most students surveyed reported speaking in class
  less than once per week
• Instructor reported few questions, so she could
  not gauge student understanding

35
Pencil and paper study

• Group of students given paper copies of the
  slides
• Students asked to annotate slides from predefined
  categories
• Slow Down, Explain, Question, Cool topic
• Simulate student experience
• Evaluate type of feedback delivered

36
Classroom study

• Limitations
• One class period
• Seven laptops (8 of 100 students participated in
  study)
• Novelty of system and participation in a study
  likely influenced how students used system
• Instructor had a TA mediate feedback (intended
  use of the system is for instructor to get
  feedback directly)

37
Study data

• Logging of system actions
• Classroom observations
• Student questionnaires
• Instructor interviews
• Draw qualitative conclusions from multiple
  sources of data

38
System usage

• Students using laptops gave between 1 and 9
  annotations
• Students using paper gave between 7 and 15
  annotations

Category type Laptop Paper
Slow Down 2 7
Question 2 21
Explain 12 19
Cool topic 16 13
39
Instructor response

• Change in presentation style
• Altering pace of lecture in response to slow down
  comments
• Providing additional explanation in response to
  question or explain
• Examples of terms annotated
• Tracking number forward reference, clarified
  later
• Batch processing old term which students
  werent familiar with

40
Reactions

• Positive instructor interaction
• The students can actually say something right
  away, and if the teacher has the wherewithal or
  time, they can actually make small improvement or
  small changes that will treat those anxieties or
  concerns doing what theyre doing that the
  students like

41
Student reactions

• Students completing survey said that the
  instructor responded to their comments
• Further explanations were given when asked to
• The computer guy the mediator told the
  instructor that there was a question and she did
  respond to it

42
Effect on classroom

• Student participation on day of experiment was
  consistent with previous classes
• 6 student initiated interactions on day of
  experiment, 11 teacher initiated interactions
• Average of 6 student initiated interactions in
  earlier classes, 6 teacher initiated interactions
• New style of interaction
• Teacher response to feedback creating discussion

43
Conclusions

• Positive evidence from design experiment
• Students used system
• Instructor was able to take advantage of
  information
• Linking feedback to content was important
• Selected categories appropriate
• Positive response from instructor and students
• Level of interaction did not drop

44
Future work

• System development
• Larger field tests
• Introductory computing, summer 2002
• Direct instructor use of system
• Sustained use of feedback system
• Feedback infrastructure as platform for
  structured interaction

45
Acknowledgments

• UW
• Tammy Vandergrift, Steve Wolfman, Ken Yasuhara
• MSR DISC Project
• portal.learningwebservices.com/disc