LINDSAY CLARE MATSUMURA

1
Implementation and Effectiveness of the
Content-Focused Coaching Program

• LINDSAY CLARE MATSUMURA
• HELEN GARNIER
• BRIAN JUNKER
• LAUREN RESNICK
• DONNA DIPRIMA BICKEL
• March 4, 2010
• Society for Research on Educational Effectiveness

2
Context for the Study

• Literacy coaching is widespread
• Little evidence shows that coaching influences
  instruction and student learning
• Research shows that quality of coaching varies
  significantly across schools
• Standards for coach qualifications often not
  followed (IRA, 2004 2006)
• What it means to be a coach is variably defined
  (Duessen et al, 2007)
• Coaching resources used in a diffuse way

3
Content Focused Coaching

• Intensive literacy-coach professional development
  program developed by the Institute for Learning
  (IFL)
• 3 days a month over the academic year led by IFL
  fellows
• Goals of the coach training
• Develop coaching skills
• Build subject matter knowledge and pedagogical
  skills to assist Ts to enact more rigorous
  reading comprehension lessons
• Improving quality of class discussions about
  texts (Questioning the Author, Beck McKeown,
  2006)

4
Content-Focused Coaching
Coaches work with IFL Trainers 3x month
Principals and District staff attend
District
Coaches work with Ts Weekly meetings in
grade-level teams Monthly meeting individually,
modeling in classrooms, observing and
co-teaching
School
Classroom
Ts enact QtA lessons with Ss in their classroom
5
Study Design

• Three year study (2006-2009)
• Urban district in Texas
• 91 of students eligible for free-lunch
• 80 Hispanic, 15 African American
• 40 English language learners
• Lowest-performing schools randomly assigned to
  treatment (n15) and comparison (n14) conditions

6
Data Sources

• Data sources include
• Teacher surveys (baseline and end of each study
  year)
• Frequency of participation in literacy coaching
• Satisfaction with coaching
• Content of the coaching activities
• Coach and principal interviews (once a year)
• Classroom observations (fall and spring of each
  study year)
• Quality of text discussions
• Rigor of text discussions and lesson activities
• Student test scores
• Degrees of Reading Power assessment (fall and
  spring of each study year)
• Texas Assessment of Knowledge and Skills (spring
  of each year)

7
Overview of the Talk

• Part 1 Influence of a schools social resources
  on the implementation of the Content-Focused
  Coaching (CFC) program (Y1)
• Principal leadership
• Norms for the professional community
• Teacher experience
• Part 2 Effectiveness of the CFC program (Y1 and
  Y2)
• Ts participation in coaching
• Observed text discussions
• Ss reading achievement (all Ss ELLs)

8
Part 1 Influence of a Schools Social Resources
on Teachers Coaching Participation

• Analyses
• Regression analyses based on T survey
  responses (N96)
• Social resources assessed at baseline
• Principal leadership
• Norms for the professional community
• Professional experience
• Teachers coaching participation assessed at the
  end of Y1
• Overall frequency of Ts participation in
  coaching
• Ts perception of the usefulness of coaching
• Content of coaching activities
• Qualitative analyses of CFC coach interviews

9
Principal Leadership is Key

• You take the principals to the CFC
  trainings, but honestly, it still comes down to
  if the principal doesnt really want the coach to
  do these things, doesnt value the coach doing
  these things, isnt leading the way so the coach
  can follow, it just isnt going to happen.

10
Principal Leadership

• Principals willingness to share leadership
  predicted
• Greater frequency of T participation in coaching
  (plt.05 )
• Greater emphasis of coaching received by Ts
• Planning and reflecting on instruction (plt.05 )
• Lesson enactment (plt.05)
• Stronger T agreement that the coaching they
  received had improved their practice (plt.01 )

11
Mechanisms by Which Principals Influence
Coaches Work

• Interviews with CFC coaches indicated that
  principals positively influenced their work by
• Actively supporting and participating in the CFC
  program
• The day I came, the P introduced me to the
  faculty. She told them that CFC was vital for us
  to change our ways of thinking and that it was
  going to take some timeand that we would be very
  patient and not despair. They would get it and
  everybody is learning. She was learningI was in
  a learner-student role and they were gonna be in
  the same role.
• CFC isnt the Ps agenda. Its happening at her
  school and she knows that if she tries to block
  it she will get into trouble, but she is not
  going to pave the way for me. If the Ts are
  reluctant or hesitant she is not going to help.
• Publicly identifying the coach as a resource for
  Ts
• Referring Ts to coach for literacy related
  questions
• Holding faculty-wide PD sessions, inviting coach
  to serve on leadership committees, watching the
  coach model lessons in Ts classrooms
• Allowing coaches to manage their own schedules
• Ps negative relationship with Ts impeded coaches
  work

12
Norms of the Professional Community

• Strong existing culture of T collaboration
  predicted
• Less emphasis of coaching received by Ts on
• Planning and reflecting on instruction (plt.05 )
• Lesson enactment (plt.05 )
• Stronger T disagreement that the coaching they
  received had improved their practice (plt.05 )

13
Mechanisms by Which Ts Professional Community
Influence Coaches Work

• Interviews with CFC coaches suggested that
• In a few schools Ts were organized against
  coaching (n2)
• This school has a reputation to oust their
  coach within a year or two. They dont like
  coaches at this campus so the longest a coach has
  been here is two years and then theyre out,
  theyre gone
• In a few schools with strong professional
  communities, reform goals were not aligned with
  CFC goals (n3)
• In some schools with a very weak professional
  culture, Ts were interested in working with
  coaches to alleviate their isolation (n5)
• Contrary pattern of collaborative professional
  community supporting coaching also found in some
  schools (n5)

14
Teachers Years of Experience Teaching

• Less experienced teachers participated more
  frequently in coaching (plt.05 )
• New Ts described by coaches as more receptive to
  coaching than more veteran teachers (n10)
• New teachers are really positive and appreciative
  of getting extra support.
• Im just another person coming into her school
  trying to save her schoolShes seen my kind so
  many times before shes sick of us. So I dont
  expect her to be my best buddy anytime soon.

15
Part 2 Effectiveness of the CFC Program (Y1 and
2)

• What is the influence of the CFC program on
  teachers coaching experiences, reading
  comprehension instruction and students reading
  achievement?

16
Participants

• Students (N1754)
• 4th and 5th grade
• 91 eligible for free or reduced price lunch
• 80 Hispanic 15 African American
• 40 English language learners (ELLs)
• Teachers (N98)
• 7 years average teaching experience
• 38 held masters degree
• 56 taught in both English and Spanish

17
Analyses

• Hierarchical linear growth models
• Amount of coaching and focus of coaching received
  by Ts
• T surveys (baseline and end of each year)
• T belief that coaching helped improve their
  practice
• T surveys (baseline and end of each year)
• Quality of instruction
• Observed text discussions (fall and spring of
  each year)
• Student achievement
• Texas Assessment of Knowledge and Skills (end of
  each year)
• Degrees of Reading Power Assessment (fall and
  spring of each year)

18
Effect of CFC on T Participation in Coaching

• Ts in the CFC schools compared to Ts in
  comparison schools by the end of Y2
• Participated more frequently in coaching (p.000,
  ES.89)
• More strongly agreed that the coaching they
  received was useful to them for improving their
  practice (p.000, ES.95)

19
Figure 1 Coach Observed Teacher for 30 Minutes
20
Effect of CFC on Content of Coaching Activities

• Ts in the CFC schools compared to Ts in
  comparison schools participated in coaching
  activities at the end of Y2 that more strongly
  emphasized
• Building knowledge of the theory underlying
  effective reading instruction (p.016, ES.70)
• Planning and reflecting on instruction (p.002,
  ES.94)
• Lesson enactment (p.001, ES.91)
• Differentiating instruction (p.007, ES.76)

21
Observations of Classroom Discussions

• T and Ss participation
• Percent of Ss participating in the discussion
• T connects Ss contributions
• Ss connect to each others contributions
• T presses Ss to explain their answers using
  evidence from the text
• Ss use evidence from the text to explain their
  answers
• Rigor of the lesson
• Quality (grist) of text discussed
• Ss opportunity to analyze and interpret a text

22
Effect of CFC on Observed Instruction

• T and Ss participation observed in CFC
  schools compared to comparison schools
• Greater proportion of Ss participating in the
  discussion (p.005, ES.35)
• T more often connects Ss contributions (p.003,
  ES.46)
• Ss more often connect to each others
  contributions (p.025, ES.38)
• T more often presses Ss to explain their answers
  using evidence (p.049, ES.33)
• Ss more often use evidence from the text to
  explain their answers (p.006, ES.43)

23
Effect of CFC on Observed Instruction

• Rigor of the lesson observed in CFC schools
  compared to comparison schools
• Higher quality (grist) of the text discussed
  (p.012, ES.52)
• More opportunities for Ss to analyze and
  interpret a text (p.011, ES.39)

24
Figure 2. Observation Ratings of Quality of
Participation in Class Text Discussions,Fall
2006 to Spring 2008 (Cohort 1, N98)Note.
Significant change over time indicated in graph
by plt.01.Within-time comparisons indicated the
following differencesFall 2006, Spring 2007 No
differences detected. Fall 2007, Spring 2008
CFCgtComparison, plt.01.

25



Figure 3. Observation Ratings of Rigor of Class
Text Discussions, Fall 2006 to Spring 2008
(Cohort 1, N98)Note. Significant change over
time indicated in graph by plt.05.Within-time
comparisons indicated the following
differencesFall 2006, Spring 2007, Spring 2008
No differences detected. Fall 2007
CFCgtComparison, plt.01.
26
Effect of CFC on Student Achievement

• Average school-level achievement for students in
  CFC schools compared to students in comparison
  schools
• Higher average gain in reading achievement for
  ELL students (p.013, ES.61)
• No significant effect on the average level of
  reading achievement for all students

27
Dimensions of Instruction Associated with
Improved Achievement for ELL Students

• Higher average reading achievement of ELL
  students in CFC schools associated with
• Greater proportion of Ss participating in the
  discussion (p.000, ES.85)
• Ss more often use evidence from a text to support
  their answers (p.026, ES.33)
• Higher quality of the text discussed (p.008,
  ES.45)

28
Recent NAEP Findings (2007)

• ELLs are the fastest growing subgroup in the U.S.
• The reading achievement of ELL students is very
  low
• 7.5 proficient
• 70 below-basic

29
Summary of Findings

• Contextual factors in schools significantly
  influenced the initial implementation of CFC
• Principal leadership played a key supportive role
• Less experienced Ts were more receptive to
  coaching
• Stronger existing professional norms negatively
  impacted some of the coaches work with Ts
• CFC produced positive results in desired teacher
  and student outcomes
• Strong effect on Ts coaching experiences and
  attitude toward coaching
• Moderate effect on reading comprehension
  instruction
• Moderate effect on reading achievement of ELL
  students (40 of the sample)

30

• Thank You!
• For further information about the study please
  contact me at
• Lclare_at_pitt.edu