Observing patterns of classroom connectivity: Protocol development and implementation

1
Observing patterns of classroom connectivity
Protocol development and implementation

• Stephen J. Pape, University of Florida, Sukru
  Kaya, Karen E. Irving, Vehbi Sanalan, Douglas T.
  Owens, The Ohio State University
• The research reported here was supported by the
  Institute of Education Sciences, U.S. Department
  of Education, through Grant R305K050045 to The
  Ohio State University.  The opinions expressed
  are those of the authors and do not represent
  views of the U.S. Department of Education.

2
Additional Authors

• L. Abrahamson, Better Education Foundation
• S. Granade, Wilkes Community College,
• Sedat Ucar, Cukurova University, Adana, Turkey
• Additional Research Team
• Frank Demana, The Ohio State University
  Christie Boscardin, Joan Herman, Hye Sook Shin
  UCLA, CRESST Mike Kositzke, Project
  Administrative Assistant, OSU
• Ugur Baslanti, University of Florida
• TI Navigator slide adapted from a presentation by
  Eileen Shihadeh, Texas Instruments

3
Changing roles for teachers include

• Thinking beyond skills-based conceptions
• Setting norms for discourse
• Fostering mathematical and scientific reasoning
• Using problem solving and inquiry to support
  knowledge construction
• Developing mathematical and scientific competence
  more broadly defined
• Using formative as well as summative assessment
• Differentiating instruction

4
Context -- Classroom Connectivity in Promoting
Mathematics and Science Achievement

• Interdisciplinary research project
• National Sample of Algebra I and Physical Science
  teachers
• Classroom connectivity technology
• Professional DevelopmentSummer Institute T3
  conference follow-up
• Randomized assignment, cross-over design with
  control group provided treatment during year 2
• Mixed methodology

5
20-Nov-09
The TI-Navigator Connected Classroom
The TI-Navigator System allows the teacher to

• Create a collaborative learning environment
• Engage in formative assessment by way of
  immediate feedback
• Enhance classroom management of TI graphing
  technology
• Quick Poll provides teacher understanding by
  receiving impromptu feedback

5
6
Prior Research on Connected Classrooms
(Roschelle, Penuel, Abrahamson, 2004)

• Students
• Increased student engagement student
  understanding interactivity
• Improved classroom discourse
• Knowledge of classmates learning
• Teachers
• Improved pre- and post- assessment of student
  learning
• Increased awareness of student difficulties
• Improved questioning

7
Owens , Demana , Abrahamson, Meagher, Herman
(2004)
8
The Potential of the Connected Classroom Includes

• Multiple interconnected representations
• Conceptual development supported through
  activity-based learning experiences
• Immediate, anonymous formative assessment
• Public displays of class knowledge
• Teacher identified critical junctures

9
The Potential of the Connected Classroom Includes

• Classroom discourse
• Explanations and justifications
• Focus on process
• Strategic behavior as object of discourse
• Changing classroom atmosphere making possible
• Increased motivation/engagement
• Positive dispositions toward mathematics and
  science

10
Theoretical Perspectives for Observation Protocol
Include

• Goal orientation of motivation social cognitive
  view of learning (e.g., Ames, 1992 Ames
  Archer, 1988 Blumenfeld, 1992 Meece, 1991)
• Classroom context that promote involvement
  (Turner et al., 1998, 2002)
• Classroom discourse practices (Cobb, Boufi,
  McClain, Whitenack, 1997 Meyer Turner, 2002
  Nathan Knuth, 2003 Turner, et al., 1998 Wood,
  1999)
• Contexts that support mathematical understanding
  (Fennema, Sowder, Carpenter, 1999)

11
Theoretical Perspectives for Observation Protocol
Include

• Contexts that support SRL (DeCorte, Verschaffel,
  Eynde, 2000 Pape, 2005 Perry VandeKampe,
  2000)
• Formative Assessment (Bell Cowie, 2001 Black
  Wiliam, 1998)
• Analytic Social Scaffolding (Nathan, 2003)
• Environmental scaffolding of strategic behavior
  (Pape, Bell, Yetkin, 2003)

12
Purpose of Presentation

• To describe the process of developing and
  piloting the CCMS Classroom Observation
  Protocol-Algebra I
• To discuss future implementation strategies

13
Observation Protocols Examined

• Professional Development Observation Protocol
  (Horizon Research, 2004)
• OPAL Observing Patterns of Adaptive learning
  (Patrick, et al., 1997)
• Oregon Mathematics Leadership Institute Classroom
  Observation Protocol (RMC Research Corporation,
  2005)
• Reformed Teaching Observation Protocol (RTOP)
  (Arizona Collaborative for Excellence in the
  Preparation of Teachers, 2000)

14
OPAL Method of Protocol Development

• Goal to investigate associations between
  teacher behaviors and practices and their
  students perceptions of the motivational climate
  of those classrooms (p. 1)
• Format narrative running record
• 9 categories Task, Authority, Recognition,
  Grouping, Evaluation, Time, Social, Help-Seeking,
  Messages

15
OPAL Training

• Project team familiar with literature
• Observation protocol based on literature
• Phase 1
• View videotapes record running records
• Team discussion of observations
• Consensus building on important points of
  observation
• Third person review comment
• Phase 2
• Pair observations of summer school write
  running records
• Compare field notes
• Third person review comment

16
CCMS Protocol Development

• Extensive review of literature including
• Classroom discourse processes
• Classroom contexts that promote self-regulated
  learning
• Classroom contexts that increase student
  motivation
• Formative assessment practices
• Classroom observation protocols
• Focuslist of instructional practices that
  support
• Mathematical understanding
• Self-regulation
• Positive dispositions
• Generate list of instructional strategies with
  descriptions

17
Three-day Training for Research TeamDay One

• Reviewed constructs to establish understanding
• Viewed videotape scripted lesson
• Pairs compared lesson scripts identifying
  similarities and differences
• Initial ratings discussion of ratings
• Generated positive and negative evidence for
  constructs
• Revised protocol based on discussion

18
Three-day Training for Research TeamDay Two
Three

• Day Two Classroom visit
• Day Three Viewed videotaped class
• Groups observed, scripted, rated lesson
• Independently indicated ratings
• Groups discuss ratings, discrepancies, and
  evidence for ratings
• Revise observation protocol

19
Training (Phase 2) Inter-rater Reliability

• Initial focus calibration and construct
  clarification
• 3-4 observers visit class, script, rate
  observation
• Record individual ratings without discussion
• Discussion toward consensus with rationale for
  ratings
• Final revisions deleted/clarified unreliable
  categories

20
Research Team Considerations and Data Collection
Decisions

• Given focus on communication camera focused on
  teacher in relation to students
• Evidence for ratings from observed behavior vs.
  inferences from self-report data
• Three column format time, behavior, analytical
  notes
• Observers consistency of ratings reliability
  number of observers

21
Inter-rater Reliability

• Inconsistent constructs, in terms of large
  variability around the consensus score, included
• Multiple Answers
• S-to-T S-to-S discourse
• Initiate-Respond-Evaluate (IRE)
• Multiple Representations
• Processes
• Autonomy
• Assessment, Knowledge, Community Centeredness
• In addition, one rater consistently higher than
  others one rater consistently lower than others

22
Changes to Protocol during Development

• Feedback during training incorporated to
  elaborate and clarify constructs
• Low reliability constructs revised
  clarification, elaboration, and modification
• Constructs Deleted
• Difficulty of providing evidence e.g.,
  modifying tasks to control challenge, strategy
  application, teacher knowledge
• Relevance to research e.g., conversational
  turns, wait time

23
CCMS Observation Protocol Classroom Context Form

• Background Information Date of observation
  Location (City, State) Observer Observation
  duration
• Classroom artifacts Classroom rules and slogans
  Classroom displays (including graded work)
  Instructional materials
• Classroom layout Provide a labeled map of the
  classroom. Physical arrangement of context in
  terms of specific tasks.
• Scripting the Observation Specific, objective
  descriptions three column format (1) time (2)
  running log stated objectively (3) Reflections,
  theoretical connections, observer notes.

24
CCMS Observation Protocol Observation Rating
Form

• Classroom context summary
• Tasks manner in which teachers structure tasks
  and learning activities
• Structural Content, procedures (explicit
  implicit), materials, participation structure,
  and products of task
• Psychological Student /or teacher affect assoc
  with the task, level of challenge and press,
  instructional support

25
CCMS Classroom Observation Protocol (cont)

• Teacher Designed Questions
• Lower-Order Questions require recalling and
  stating known facts, carrying out a simple
  algorithm or procedure
• Procedural, knowledge, comprehension questions
• Higher-Order Questions require manipulation of
  information, transform information, synthesize,
  generalize, explain, hypothesize
• Justification, classification, comparison,
  exploration/extension, application, analysis,
  synthesis, evaluation questions

26
CCMS Classroom Observation Protocol (cont)

• Sociomathematical norms ways in which teachers
  and students interact within classroom
• Teacher press for student involvement ways
  teacher ensures increases involvement
• Teacher press for elaboration, explanation,
  justification ways teacher pushes students to
  elaborate to make reasoning explicit includes
  uptake of correct incorrect responses
• Soliciting multiple answers or solution methods
• Scaffolding teacher support such as modeling,
  questioning analytic social scaffolding
• Learning strategies explicit discussion of
  techniques for learning
• Performance orientation focus on the
  outcome/products showing competence
• Mastery orientation focus on understanding,
  learning skills with meaning

27
CCMS Classroom Observation Protocol (cont)

• Classroom Discourse -- Relative proportion or
  focus on
• Teacher-to-student teacher-initiated
  interaction
• Student-to-teacher student-initiated
  interaction
• Student-to-student interaction
• Initiation-Response-Evaluation (IRE)
• Classroom management procedures
• Use of multiple representations
• Collective argumentation two or more students
  contribute to establish a claim
• Student discussion of understanding

28
CCMS Classroom Observation Protocol (cont)

• Formative Assessment teacher behaviors targeted
  toward gathering data from students and making
  instructional decision accordingly
• Educational goals associated with instructional
  decisions
• Rules and facts
• Communication
• Application
• Evaluation
• Mathematical processes

29
CCMS Classroom Observation Protocol (cont)

• Knowledge Construction
• Individually vs. group constructed
• Holder of knowledge degree to which teacher or
  students are authority of knowledge construction
• Depth of knowledge superficial vs. central
• Technology Use overall use quick poll screen
  capture learn check student inquiry/data
  aggregation activity center TI presenter
  graphing calculator TI viewscreen Overhead
  projector Computer Other

30
CCMS Classroom Observation Protocol (cont)

• Instructional Practices
• Traditional/Teacher-centered
• NCTM Standards-based, cognitively active
  environment, student-centered
• Inquiry, problem-based learning, discovery
  learning
• How People Learn
• Learner-centeredness
• Assessment-centeredness
• Knowledge-centeredness
• Community-centeredness

31
Update/Implementation/Future Work

• To date, 2 years of observations have been
  conducted (approx. 15 per Algebra I cohort 5
  per Physical Science cohort)
• Observations will be used to triangulate
  self-report data
• Teacher Practices and Beliefs Survey
• Student Perceptions of Instruction Survey
• Implementation rating from Telephone Interviews
• Planned Analyses
• Differences between Algebra I and Physical
  Science teaching
• Changes to protocol to analyze Physical Science
  instruction
• Purposefully chosen cases examined and elaborated
• Examine classroom instruction in relation to
  development of student dispositions and SRL
• Conduct in-depth ethnographies of instruction
  with classroom connectivity technology during
  years 3 4