Using Technology to Restructure Spanish Curricula: Texas Techs R2R Project

1
Using Technology to Restructure Spanish
Curricula Texas Techs R2R Project

• PRESENTERS
• Dr. Frederick Suppe, PI
• Tracy A. Rutledge
• Dr. Stephen Corbett
• Dr. Comfort Pratt
• Silvia Peart

• OTHERS ON THE R2R TEAM
• Dr. Greta Gorsuch
• Claudia Diaz
• René Ibarra
• Hilda Salazar

Special thanks to Judy Collazo, Valerie Huggard,
Natasha Ojeda, Terry Martin, Phade Vader, and the
instructors and students who participated in the
experiment.
2
Overview of Texas Tech Beginning Spanish Course
Redesign Project

• Development of a comprehensive instructional
  system for lower-level language instruction.
• System is text-book and language independent.
• System aggressively combines technology with
  highly communicative approaches.
• Individual components can be used rather than the
  whole system.
• Extensive assessment and evaluation component to
  project.

3
Overview of Texas Tech Redesigned Beginning
Spanish Course

• Hybrid 5 credit course with 3 contact hours
  in-class and 2 hours on-line.
• In-class portion devoted to communicative
  exercises and limited to 20 students.
• Workbooks replaced by on-line interactive drill
  with targeted diagnostic feedback and automatic
  grading.
• Semi-automated grading of on-line compositions
• Bi-weekly Language Lab assignments.

4
Overview of Texas Tech Redesigned Beginning
Spanish Course

• Includes adaptive re-use of stripped Destinos
  episodes for in-class communicative exercises.
• Development and validation of ACTFL-based
  assessment instruments for evaluating the 4
  skills.
• Uses Puntos 7 as textbook, but approach can be
  used with any textbook or language

5
WHAT YOU WILL GET FROM THIS SESSION

• Demonstrations of each component of the system.
• Presentation of theory underlying the system
  components and how to information.
• Extensive handouts covering each component.
• Guidance for adapting our approach or its
  components to your own courses.
• Evidence of the effectiveness of the approach

6
Our redesign is part of the Roadmap to
Redesign (R2R) Project

• A project of the National Center for Academic
  Transformation (NCAT)
• Successor to the PEW-funded Program in Course
  Redesign (PCR) project that collaborated with 30
  institutions to demonstrate how colleges and
  universities can redesign their instructional
  approaches using technology to achieve cost
  savings as well as quality enhancements.
• PCR resulted in 5 Redesign Models and 5
  Principles for Successful Course Redesign.

7
R2R Overview

• Three-year, FIPSE-funded project to test results
  of PCR and establish an efficient means for
  spreading the PCR ideas and practices to
  additional institutions.
• Focuses on large-enrollment, introductory courses
  in four disciplines pre-calculus mathematics,
  psychology, Spanish and statistics.

8
R2R Overview

• National competition to select participating
  colleges and universities.
• 40 finalist institutions invited to submit
  proposals.
• 20 selected for the project, 4 in Spanish
• Montclair State University
• Texas Tech University
• Towson University
• University of Alabama

9
The Texas Tech R2R Spanish Redesign Project
10
The Old Spanish 1507 Course for students who have
had at least 2 years of high school Spanish

• 630 students/semester
• 19 instructors
• 12 sections of 32 students
• 5 hours in class per week Language Lab
• 1 section/instructor
• 32 students, 5 contact hours, 5 preparations
• Hand graded workbook exercises compositions
• 110 student Lecture/discussion version
• 40 students in Destinos on-line version

11
The Redesigned Course

• 720 students/semester (vs. 630)
• 18 instructors (vs. 19)
• 36 sections of 20 students/section ADFL
  recommended maximum (vs. 32)
• 3 hours in class per week (vs. 5), 2 hours
  on-line (vs. 0) Language Lab
• 2 sections/instructor (vs. 1)
• 40 students, 6 contact hours, 3 preparations (vs.
  32/5/5)
• automatic grading of on-line drills,
  semi-automated grading of compositions--3-5
  minutes per student per composition (vs. 35
  minutes grading/student/week)
• 45 students on-line version, 60
  Lecture-Discussion

12
REDESIGN GOALS

• Reduce Class Size to ADFL Guidelines
• Reduce workload of instructors
• Increase capacity of course by 19
• Use existing classroom space
• No increase in instructional costs
• No Pedagogical Harm to students
• SPECIFICALLY No significant decline in language
  proficiency under redesign.
• Develop improved testing materials

13
The On-line components were originally developed
for a broadcast TV/web-based distance-education
Beginning Spanish course using Destinos. The
components were tested against conventional
courses in controlled experiments.
14
The On-Line Destinos CourseDeveloped by Tracy
Rutledge and Dr. Frederick Suppe

• Students watch 52 episodes of Destinos telenovela
• Students do on-line interactive drills and submit
  on-line compositions
• Drills cover workbook material with automated
  grading that includes targeted diagnostic
  feedback on wrong answers.
• Compositions are semi-automatically graded with
  targeted diagnostic feedback.
• Class meets physically only for examinations.

15
The Destinos On-line Components

• Uses Web-CT as course management system, but
  approach should work with other systems.
• Requires work-arounds to get Web-CT to do things
  it wasnt designed to do.
• Main on-line components
• On-line compositions with semi-automated
• grading with targeted diagnostic feedback.
• 3-5 minutes grading per student per week with
  detailed diagnostic feedback.

16
The Destinos On-line Components

• Workbooks replaced by parallel
• automatically graded drill exercises.
• Students never given correct answers.
• Targeted diagnostic feedback given for wrong
  answers.
• Students repeat drills as many times as they
  want, with a new drill each time.
• Goal is mastery of vocabulary and grammar in a
  timely fashion.

17
On-line Destinos Effectiveness

• Automated grading of drills is effective and
  efficient.
• Students learn more than in traditional courses
• Reading Comprehension Significantly higher
• (2 SDs)
• Oral comprehension Very significantly
  higher
• (3 SDs)
• Writing No significant difference
• Speaking No significant difference
• Grammar and vocabulary Significantly
  higher (2 SDs)

18
Findings

• Our On-line interactive Drill is more effective
  than traditional class-room use of workbooks.
• On-line submission and semi-automated grading of
  compositions are as effective as traditional
  compositions and far more efficient to grade.
• Automated grading of drills reduces instructor
  grading time.
• These findings form the technological basis for
  our R2R redesign of SPAN 1507.

19
Spring 2005 Pilot Experiment to test the Hybrid
Redesign

• Full Human Subjects Approval, signed informed
  consent
• 8 R2R, 8 Control sections
• 32 students per Control, 20 per R2R
• 160 R2R, 256 Control students
• 4 clusters of 2 R2R and 2 Control section pairs
  on basis of differential teaching ability
  (Exceptional, good, average, weak)
• One pair taught by same instructor
• One pair taught by different instructors

20
Main Pilot Findings

• No Harm Result No statistically significant
  difference in performance between R2R and
  Control.
• R2R students generally did slightly better.
• Some course improvements made and tested in
  Summer 2005.
• Full Implementation of Redesign in Fall 2005
  with 780 students, 21 instructors, 39 sections, 6
  person coordination R2R research team.

21
We now present the various course components,
their underlying design, and offer guidance for
adaptation to your own courses

• On-line interactive Drills
• On-line semi-automatically graded compositions
• Multiple adaptive repurposing of Destinos
  materials
• Standardized in-class communicative exercises
• Mentoring and training of instructors
• ACTFL-derived testing and assessments

22
On-Line Components Interactive DrillsDeveloped
by Claudia Diaz, Silvia Peart, Tracy Rutledge and
Dr. Frederick Suppe coordinated by René
Ibarra
23
On-Line Interactive Drills

• Workbook exercises are converted into different
  but parallel on-line exercises.
• Targeted diagnostic feedback given for wrong
  answers.
• Correct answers revealed only if student gets
  them correct.
• Can repeat drills as many times as one wants
  within window of opportunity.
• Competency-based progression en-echelon Students
  must spend 2 hours a week or get at least 80 on
  one attempt or else their grade is zeroed-out.

24
Students are presented with aseries of questions
25
When they submit their drill attempt they get
targeted diagnostic feedback
26
Multiple Choice Questions
27
Multiple Choice Feedback
28
Matching Question Feedback
29
Short-answer Question
30
Short-Answer Feedback
31
Web-CT limitations do not allow targeted separate
comments for each short-answer option so we have
to provide a generic feedback that covers all
contingencies.
32
Designing On-Line Drills for Targeted Diagnostic
Feedback

• Design of the answer space Conventional workbook
  exercises have to be reconceptualized to meet the
  following criteria
• There must be a fixed answer space with a
  finite set of discrete non-overlapping possible
  answers (cells).
• Each possible incorrect answer must evidence a
  specific error pattern.
• Each cell must have a unique diagnostic feedback
  message associated with it.
• The simplest way to achieve this is to use
  True-False, Multiple-choice, or Matching test
  items.

33
Designing Short-Answer Questions

• Short Answer questions must be
  reconceptualized to meet the fixed discrete
  answer space requirement
• Strategies
• Restricting vocabulary
• Requiring standardized syntactical structures for
  the answers
• where anything not meeting the restrictions
  can receive a default response indicating the
  restrictions were violated.
• In effect this involves making the question a
  disguised multiple-choice question where the
  options are not revealed. This may require as
  many as 40 cells in the answer space.
• Some classroom management systems (e.g., Web
  CT) compromise your ability to give fully
  targeted diagnostic feedback because they do not
  allow you to attach separate feedback to each
  hidden answer option.

34
Typical workbook items can be reconceptualized to
meet the above answer space design requirements.

• Analyze the workbook exercise to determine what
  it is attempting to test or teach.
• If the exercise is testing or teaching more than
  one thing split into separate exercises for each
  learning/testing objective.
• Using the same vocabulary and grammar, rework the
  exercise as a replacement exercise meeting the
  answer-space design requirements.
• Pick your possible answers so as to allow
  response-specific diagnostic feedback.

35
Strategies for converting typical workbook
exercises for on-linewith diagnostic feedback.
36
On-Line Components Written Assignments with
Semi-Automated Grading Developed by Tracy
Rutledge, and Dr. Frederick Suppe, coordinated by
them and René Ibarra
37
Semi-automated grading of on-line compositions

• Basic Empirical Finding If one restricts
  diagnostic feedback to just items related to a
  given lessons learning objectives, then there
  will be 8-12 comments that cover more than 95 of
  relevant comments that need to be made on student
  compositions.

38
Exploitive strategy Archive authoritative
comments targeted to each exercise and supply
them to graders. Example
39
Student down-loads a Word template giving the
assignment and a model response
40
Student adds composition to Word and submits file
via Web-CT assignment (not quiz) modality.
41
Instructor downloads and grades using prefabbed
comments

• Targeted Comments
• 1. No always goes in front of the verb.
• 2. When saying someones age you must use tener
  años.
• 3.Agreement error.
• 4. This verb does not agree with your subject.
• 5. Following a conjugated verb you should use
  the infinitive.
• 6. The verb gustar has a special conjugation.
  It uses an indirect object such as me gusta.
• 7. In Spanish we have letter ñ
• 8. Missing verb.
• 9. Omit
• 10. Word order error
• 11. Vocabulary error
• Generic Comments
• The red underlined words are either proper names
  or misspelled words (a missing accent / tilde( ?
  ) is considered a misspelled word). The green
  are grammar errors. Right click on the word and
  you should get suggestions.
• ?Excelente!?
• ?Muy Bien!?

42
. resulting in an annotated composition to
upload to the student
43
Before uploading, the instructor uses an
ACTFL-derived holistic standard (see your handout
later for details) to assign a composition grade.
44
Repurposing Destinosmaterials
Coordinators and developers Dr. Stephen
Corbett, Dr. Comfort Pratt, Dr. Frederick Suppe,
and Tracy Rutledge
45
Integration into First-Year Spanish

• Students watch 1 episode of the Destinos
  telenovela a week
• Broadcast simultaneously via closed circuit TV
  into high-definition projector- equipped
  classrooms from central control room.
• Didactic material stripped automatically to
  produce a 15 minute story segment.
• Viewing surrounded by in-class communicative
  activities.

46
Structure of a Destinos Session

• 10 Minute preparation period
• Review previous episode

47
Structure of a Destinos Session

• 10 Minute preparation period
• Review previous episode
• Students read composition assignments about
  previous episode.

48
Structure of a Destinos Session

• 10 Minute preparation period
• 15 minute viewing session

49

• 15 minute viewing session
• Other uses of Destinos videos
• In-class communicative activities.
• On-line tele-courses
• Clips from not yet viewed episodes are used in
  listening comprehension tests.

50
Structure of a Destinos Session

• 10 Minute preparation period
• 15 minute viewing session
• 10 minute post-viewing discussion
• Summary of episode done by students
• Instructor asks specific questions to test
  comprehension

51
A variety of activities role-playing relating
events in episode to other events discussing
topics raised in relation to other matters
speculating on what will happen in the next
episode.Essay assignment Summarize episode,
answer specific questions.
52
Each session has a fixed lesson plan (in your
handout packet)
53
Destinos Integration into Second-Year Spanish

• Viewed during fifty minute period of class
  weekly                
•      Pre-viewing activities (teacher comments
  and questions, new vocabulary introduced)
• Viewing with questions (true-false, fill-in the
  blank and short-answer)
• Follow-up in small groups and with whole class to
  answer questions and share opinions
  or expectations
• Evaluation of skills and knowledge via....
  Listening comprehension quizzes
• (multiple-choice format) Written exams
  (short-answer format)
• Student feedback used to revise implementation.

54
Use of Destinos materials in Study- Abroad
intensive immersion courses

• Students do 11 credit hours of lower-level
  Spanish in Seville, living with monolingual
  families.
• Destinos workbook and a street Spanish book used
  as only textbooks.
• (Texts like Puntos have a Latin
  American cultural emphasis that is inappropriate
  for teaching in Spain.)
• One hour a day online Destinos drills using
  servers in Lubbock replace homework.
• Destinos videos not used.
• Culture component provided by excursions and in
  situ class events (in the mercado, etc.).

55
Other in-class Communicative Activities
SPAN 1507 Coordinator Dr. Comfort Pratt
56
Communicative Activities

• 2 principal components
• Instructor training
• Pre-semester Orientation
• Semester-long Training Sessions
• 2 classroom observations per semester
• In-class communicative activities

57
Pre-semester Orientation 4-day training session
for instructors

• Instructors are given an overview of
  communicative language teaching (CLT).
• They are taught how to implement CLT in the
  classroom.
• They learn how to prepare lesson plans.
• They are taught how to use the on-line
  components.
• They have practice and calibrations sessions on
  using grading protocols.

58
Semester-long Training Sessions

• Once a week instructors meet with coordination
  team for 1 hour and 20 minutes.
• Every other week are CLT training sessions that
  focus on Communicative Activities.
• Alternate weeks focus on remaining components of
  the course.

59
Biweekly Communicative Language Teaching Sessions

• Instructors
• receive
• communicative
• lesson plans
• for the
• following two
• weeks and
• accompanying
• materials.
• (Example in handout package)

60
Biweekly Communicative Language Teaching Sessions

• Instructors are taught how to carry out the
  activities in the classroom by means of a
  hands-on training process

61
Classroom Observations

• The coordinator observes each instructor in the
  classroom twice every semester and evaluates
  their performance
• The instructors meet with the coordinator after
  the visit to discuss the evaluation and
  recommendations

62
In-Class Communicative Activities

• Class meets physically for 3 hours a week
• Class time is entirely dedicated to communicative
  activities
• Instructors follow lesson plans designed by
  coordinator (examples in handout)
• Structure of lesson plan
• Communicative goals
• Communicative activities
• Assignments

63
Insert here slides and video of training and then
implementation.
64
Teachers practice the activities they will use in
class.
65
And have their execution of the activities
critiqued
66
Practicing a Total Physical Response activity
67
Practicing a rooms of the house.
68
and daily routines TPR activity.
69
Testing and AssessmentCoordination and design
team Dr. Greta Gorsuch, Silvia Peart, Hilda
Salazar, Dr. Frederick Suppe
70
R2R Assessment and Evaluation

• Pilot study was done as a controlled experiment
  with full Human Subjects approval. More than
  60,000 data points were collected over 411
  students.
• These data included performance data and 5
  surveys.
• The same sorts of data were collected during the
  summer 2005 trial implementation and are being
  collected for the full Fall 2005 implementation
  with 780 students.

71
R2R data collected

• Surveys
• Demographic information
• Academic achievement and motivation survey.
• Technology survey regarding prior experience with
  on-line courses or components.
• Mid-semester and end-of-semester attitudinal
  surveys towards the course.
• Academic performance data.
• Key-stroke and time-on-task records for all
  on-line drill attempts.
• Classroom observation reports using a
  standardized reporting schedule.
• Copies of all exams and graded materials

72
Basic No Harm Pilot Result
We expect better performance in the
implementation phase!
73
The most interesting pilot finding is that
standard ways of testing student performance are
unreliable, of unproven validity, or just invalid!

• We base this conclusion on the basis of standard
  evaluations of tests generated by
  publisher-supplied test banks.
• We did so using sophisticated test validity
  analyses of our pilot data sets

74
Problems with standard testing include

• too few items (causing unreliability).
• test items not having item-discrimination
  validity.
• no construct validity for grading
  protocolsespecially for writing and speaking.
• factor-analysis indicates that virtually all test
  items load on a single undifferentiated
  language factor precluding differentiation of
  performance into differentiated performance on
  each of the four skills.
• in particular such testing methods preclude
  separate evaluation for each of the four skills.

75
Our Texas Tech R2R response to this disturbing
findingDevelop, test, and validate separate
ACTFL-derived evaluations methods for each of the
four skills
76
Example 1 Listening Comprehension(Sample given
in the handouts.)

• Students watch a clip from a Destinos episode
  they have not seen.
• They are given a few minutes to read the
  objective listening comprehension questions
  before they watch the clip.
• After viewing they have time to answer 25
  objective questions.
• In order to differentiate listening from other
  skills, questions are in English and of provable
  reliability.

77
Example 2 Writing Evaluation

• 10 of each sections on-line compositions are
  randomly selected and randomly sent for double
  blind regrading.
• A sophisticated computer program compares grading
  and regrading performance against each other and
  against all 36 sections, producing weekly
  diagnostic reports suggesting grading problems
  accompanied by copies of original and regraded
  essays.
• ACTFL-derived assessments, and distribution of
  prefabbed comments, and flagged diagnosis of
  suspicious disparities, using a strict standard,
  in application of standards are sent in the
  weekly reports.

78
Example 3 Speaking(Mini-SOPI example given in
handout)

• The Center for Applied Linguistics (CAL)
  Simulated Oral Evaluation (SOPI) approach has
  been modified to develop Mini-SOPIs that take
  about 13 minutes to administer, producing 5-7
  minutes of taped responses that are evaluated.
• Unlike SOPIs or ACTFL OPIs, this approach can be
  used twice per semester with about 800 students
  without over-burdening instructors.

79
Mini-SOPIs

• Are based on the SOPI design principles of CAL,
  hence inherit much of their validations.
• Unlike OPIs and SOPIs which attempt to discern
  where a speaker is on the entire ACTFL spectrum,
  our Mini-SPOIs only attempt to place students on
  an empirically-based portion of the ACTFL
  spectrum predicted on the basis of proficiency
  studies.
• This allows us to reduce time from 50-25 minutes
  to 7 minutes of evaluation time by reducing tasks
  to a range appropriate the students target
  levels.
• Recordings use near-professional recording of
  professional or semi-professional actors to
  record the tapes.
• Training sessions in application of ACTFL-derived
  standards use CAL SOPI training tapes.

80
Mini-SOPI Evaluations

• 10 randomized re-evaluation of Mini-SOPIs each
  time.
• Comparisons and reports made similar to those
  for writing evaluations.
• Inter-rater reliability studies to establish
  reliability
• Concurrent validity studies compared with
  official ACTFL OPIs and CAL SOPI evaluations
  will be used to establish construct validity of
  the protocol.

81
Reading, Grammar, Vocabulary

• Primary problem with standardized reading
  comprehension testing is insufficient number of
  test items 10 is minimum for reliability (unlike
  publisher test banks that generate 4).
• Grammar and vocabulary tests generated from test
  banks are ok, but often contain items with poor
  discriminate validity.
• Need to assess all test items and replace by
  ones having suitable reliability and discriminate
  validity.

82
Summary
83
Evaluation of Texas Techs R2R project

• Is a comprehensive teaching system.
• Sufficiently structured as to make teaching
  personnel interchangeable parts that can be
  deployed to produce a fairly uniform product.
• Does not minimize the added value of great
  teaching but insulates the student from damage by
  poor instructors.
• Marries proven on-line approaches pioneered in
  the TTU Destinos on-line project with
  cutting-edge communicative in-class activities.
• Involves extensive on-going mentoring and
  training 30-48 hours per semester.
• Triages using on-line resources where they are
  more effective, communicative classroom
  activities where they are more effective.
• Exploits technology in a sophisticated manner
  in-class and on-line.

84
Evaluation of Texas Techs R2R project

• No Harm result! IT WORKS!!!!
• Reduced class size without cost/student increase.
• Has a very extensive and ambitious evaluation
  component.
• All curricular decisions made on the basis of
  empirical data and target goal performance
  outcomes.
• Massive data sets collected that will be the
  basis for research studies and articles for years
  to come and for dissertations.

85
Continuation of our R2R project

• Pilot and implementation of SPAN 1507
  conversion is so successful that we plan to
  extend the R2R redesign to second-year (3 credit)
  Spanish courses
• Hybrid course, 2 hours inclass, 1 hour on-line.
• Adapt and extend the communicative in-class
  activities including use of Destinos stripped
  episodes in class.
• Extend the ACTFL-derived assessment/grading
  protocols to second year and validate them.
• Collect comparable survey and performance data
  for second-year students

86
Timetable for Second-Year Implementation

• Pilot conversion of SPAN 2301 (first semester of
  second year)
• Spring 2006 with 9 R2R and 9 control
  sections in controlled experiment.
• Depending on outcome of this pilot, implement
  2301 redesign in Fall 2006.
• Determine whether separate SPAN 2302 (second
  semester of second year) pilot is required and
  then either pilot or implement R2R redesign.

87
Longer-term objectives

• As our redesign approach and assessment
  instruments are validated we hope
• To exploit the fact that our design goals have
  been to develop a teaching system and assessment
  instruments that is language and textbook
  independent.
• To port the approach over to other languages in
  a manner where components can be selectively
  deployed.

88
Costs of the project

• The piloting and implementation of the SPAN 1507
  redesign cost about 136,000 spread over 18
  months.
• This does not include salary allotments for the
  three professors involved.
• 30,000 was provided by an external donor the
  remainder came from departmental discretionary
  funds. No special funds were sought from the
  administration.
• Continuation of the project for similar redesign
  of second-year Spanish will raise the special
  costs to about 250,000.

89
Arghh! That is unaffordable!

• For most institutions you are correct.
• However, in the spirit of the R2R project, we
  have tried to provide you through this
  presentation and the extensive handouts with
  enough guidance to
• Decide which elements of our teaching system
  might be adaptable to your local circumstances.
• Experiment with efficient development and
  adaptation of selected components of our system
  to your local needs and purposes
• As with the overarching national R2R project, our
  goal today has been to help you redesign your
  lower-level language courses without having to do
  so from scratch and to share the wisdom we have
  gained in our own experiments and from our
  predecessor PEW PCR partners.

90
Finally!

• We hope we have provided you through this
  presentation and the extensive handouts a basis
  whereby you can consider adapting selected
  components of our Texas Tech foreign language
  teaching system to your own local needs and
  institutional resources.

91
(No Transcript)