LINDSAY CLARE MATSUMURA

1
Evidence on the Implementation and Effectiveness
of the Content-Focused Coaching Program

• LINDSAY CLARE MATSUMURA
• HELEN GARNIER
• BRIAN JUNKER
• LAUREN RESNICK
• DONNA DIPRIMA BICKEL
• June 30, 2010
• Institute of Educational Sciences Conference

2
Context for the Study

• Literacy coaching is widespread, little
  evidence that coaching influences instruction and
  student learning
• Research shows that quality of coaching varies
  significantly across schools
• Standards for coach qualifications often not
  followed (IRA, 2004 2006)
• What it means to be a coach is variably defined
  (Duessen et al, 2007)
• Coaching resources used in a diffuse way

3
Content Focused Coaching

• Intensive literacy-coach professional development
  program developed by the Institute for Learning
  (IFL)
• 3 days a month over the academic year led by IFL
  fellows
• Goals of the coach training
• Develop coaching skills
• Build subject matter knowledge and pedagogical
  skills to assist Ts to enact more rigorous
  reading comprehension lessons
• Improving quality of class discussions about
  texts (Questioning the Author, Beck McKeown,
  2006)

4
Content-Focused Coaching
Coaches work with IFL Trainers 3x month
Principals and District staff attend
District
Coaches work with Ts Weekly in grade-level
teams and individually Monthly in classrooms
to model, observe and co-teach
School
Classroom
Ts enact QtA lessons with Ss in their classroom
5
Study Design

• Three year study (2006-2009)
• Urban district in Texas
• 91 of students eligible for free-lunch
• 80 Hispanic, 15 African American
• 40 English language learners
• Lowest-performing schools randomly assignment
  to treatment (n15) and comparison (n14)
  conditions

6
Data Sources

• Data sources include
• Teacher surveys (baseline and end of each study
  year)
• Frequency of participation in literacy coaching
• Satisfaction with coaching
• Content of the coaching activities
• Coach and principal interviews (once a year)
• Classroom observations (twice a year)
• Quality of text discussions
• Rigor of text discussions and lesson activities
• Student test scores
• Degrees of Reading Power assessment (twice a
  year)
• Texas Assessment of Knowledge and Skills

7
Overview of the Talk

• Part 1 Influence of a schools social resources
  on the implementation of the Content-Focused
  Coaching (CFC) program (Y1)
• Principal leadership
• Norms for the professional community
• Teacher experience
• Part 2 Effectiveness of the CFC program (Y1 and
  Y2)
• Ts participation in coaching
• Observed text discussions
• Ss reading achievement (all Ss ELLs)

8
Part 1 Influence of a Schools Social Resources
of Teachers Coaching Participation

• Regression analyses based on T survey
  responses (N96)
• Social resources assessed at baseline
• Principal leadership
• Norms for the professional community
• Professional experience
• Teachers coaching participation assessed at the
  end of Y1
• Overall frequency of Ts participation in
  coaching
• Ts perception of the usefulness of coaching
• Type of coaching activities
• Qualitative analyses of CFC coach interviews

9
Principal Leadership is Key

• You take the principals to the CFC
  trainings, but honestly, it still comes down to
  if the principal doesnt really want the coach to
  do these things, doesnt value the coach doing
  these things, isnt leading the way so the coach
  can follow, it just isnt going to happen.

10
Principal Leadership

• Principals willingness to share leadership
  significantly predicted
• Frequency of T participation in coaching (plt.05 )
• Type of coaching received by Ts
• Planning and reflecting on instruction (plt.05 )
• Lesson enactment (plt.05)
• T belief that the coaching they received had
  improved their practice (plt.01 )
• Principals past active participation in literacy
  reforms did NOT predict T participation in
  coaching

11
Mechanisms by Which Principals Influence
Coaches Work

• Interviews with CFC coaches indicate that
  principals positively influenced their work by
• Actively supporting and participating in the CFC
  program
• The day I came the P introduced me to the
  faculty. She told them that CFC was vital for us
  to change our ways of thinking and that it was
  going to take some timeand that we would be very
  patient and not despair. They would get it and
  everybody is learning. She was learningI was in
  a learner-student role and they were gonna be in
  the same role.
• CFC isnt the Ps agenda. Its happening at her
  school and she knows that if she tries to block
  it she will get into trouble, but she is not
  going to pave the way for me. If the Ts are
  reluctant or hesitant she is not going to help.
• Publicly identifying the coach as a resource for
  Ts
• Referring Ts to coach for literacy related
  questions
• Holding faculty-wide PD sessions, inviting coach
  to serve on leadership committees, watching the
  coach model lessons in Ts classrooms
• Allowing coaches to manage their own schedules
• Ps negative relationship with Ts impeded coaches
  work

12
Norms of the Professional Community

• Strong existing culture of T collaboration
  negatively predicted
• Type of coaching received by Ts
• Planning and reflecting on instruction (plt.05 )
• Lesson enactment (plt.05 )
• T belief that the coaching they received had
  improved their practice (plt.05 )

13
Mechanisms by Which Ts Professional Community
Influence Coaches Work

• Interviews with CFC coaches suggest that
• In a few schools Ts were organized against
  coaching (n2)
• This school has a reputation to oust their
  coach within a year or two. They dont like
  coaches at this campus so the longest a coach has
  been here is two years and then theyre out,
  theyre gone
• In a few schools with strong professional
  communities reform goals were not aligned with
  CFC (n3)
• In some schools with a very weak professional
  culture, Ts were interested in working with
  coaches to alleviate their isolation (n5)
• Contrary pattern detected in some schools (n5)

14
Teachers Years of Experience

• Less experienced teachers participated more
  frequently in coaching (plt.05 )
• New Ts were reported to be more receptive to
  coaching than more veteran teachers (n10)
• New teachers are really positive and appreciative
  of getting extra support.
• Im just another person coming into her school
  trying to save her schoolShes seen my kind so
  many times before shes sick of us. So I dont
  expect her to be my best buddy anytime soon.

15
Part 2 Effectiveness of the CFC Program (Y1 and
2)

• What is the influence of the CFC program on
  teachers coaching experiences, reading
  comprehension instruction and students reading
  achievement?

16
Participants

• Students (N1754)
• 4th and 5th grade
• 91 eligible for free or reduced price lunch
• 80 Hispanic 15 African American
• 40 English language learners (ELLs)
• Teachers (N98)
• 7 years average teaching experience
• 38 masters degree
• 56 teach in both English and Spanish

17
Analyses

• Hierarchical linear growth models
• Amount and type of coaching received by Ts
• T belief that coaching helped improve their
  practice
• T surveys (baseline and end of each year)
• Quality of instruction
• Observed text discussions (fall and spring of
  each year)
• Student achievement
• Texas Assessment of Knowledge and Skills (end of
  each year)
• Degrees of Reading Power Assessment (fall and
  spring of each year)

18
Effect of CFC on T Participation in Coaching

• Ts in the CFC schools by the end of Y2
• Participated more frequently in coaching
• ES.89 plt.000
• Were more likely to believe that the coaching
  they received was useful to them for improving
  their practice
• ES.95 plt.000

19
Figure 1Coach Observed Teacher for 30 Minutes
20
Effect of CFC on Types of Coaching Activities

• Ts at the end of Y2 in the CFC schools
  participated in coaching activities that
  emphasized
• Building knowledge of the theory underlying
  effective reading instruction
• ES.70, plt.016
• Planning and reflecting on instruction
• ES.94, plt.002
• Lesson enactment
• ES.91, plt.001
• Differentiating instruction
• ES.76, plt.007

21
Observations of Classroom Discussions

• T and Ss participation
• Percent of Ss participating in the discussion
• T connects Ss contributions
• Ss connect to each others contributions
• T presses Ss to explain their answers using
  evidence from the text
• Ss use evidence from the text to explain their
  answers
• Rigor of the lesson
• Quality (grist) of text discussed
• Ss opportunity to analyze and interpret a text

22
Effect of CFC on Instruction

• T and Ss participation
• Percent of Ss participating in the discussion
• ES.35, plt.005
• T connects Ss contributions
• ES.46, plt.003
• Ss connect to each others contributions
• ES.38, plt.025
• T presses Ss to explain their answers using
  evidence
• ES.33, plt.049
• Ss use evidence from the text to explain their
  answers
• ES.43, plt.006

23
Effect of CFC on Instruction

• Rigor of the lesson
• Quality (grist) of the text discussed
• ES.52, plt.012
• Ss opportunity to analyze and interpret a text
• ES.39, plt.011

24
Figure 2. Observation Ratings of Quality of
Participation in Class Text DiscussionsFall 2006
to Spring 2008 (Cohort 1, n98)Note.
Significant change over time indicated in graph
by plt.01. Within-time comparisons
indicated the following differencesFall 2006,
Spring 2007 No differences detected. Fall
2007, Spring 2008 CFCgtComparison, plt.01.

25



Figure 3. Observation Ratings of Rigor of Class
Text Discussions Fall 2006 to Spring 2008
(Cohort 1, n98)Note. Significant change over
time indicated in graph by plt.05.
Within-time comparisons indicated the following
differencesFall 2006, Spring 2007, Spring 2008
No differences detected. Fall 2007
CFCgtComparison, plt.01.
26
Effect of CFC on Student Achievement

• No effect of CFC was detected on student
  achievement for the total sample of students
  (N1754)
• CFC positively predicts ELL student achievement
  on the TAKS
• ES.61, plt.01

27
Dimensions of Instruction Associated with
Improved Achievement for ELL Students

• Percent of Ss participating in the discussion
• ES.85, plt.000
• Ss use evidence from a text to support their
  answers
• ES.33, plt.026
• Quality of the text discussed
• ES.45, plt.08

28
Recent NAEP Findings (2007)

• ELLs are the fastest growing subgroup in the U.S.
• The reading achievement of ELL students is very
  low
• 7.5 proficient
• 70 below-basic

29
Summary of Findings

• Contextual factors in schools significantly
  influenced the initial implementation of CFC
• Principal leadership played a key role
• Less experienced Ts were more receptive to
  coaching
• Some evidence of a negative relationship between
  the strength of the existing professional norms
  and coaches work with Ts
• CFC mostly showed positive effects on desired
  outcomes
• Strong effect on Ts coaching experiences and
  attitude toward coaching
• Moderate effect on reading comprehension
  instruction
• Moderate effect on reading achievement of ELL
  students only (40 of the sample)

30

• Thank You!
• For further information about the study please
  contact me at
• Lclare_at_pitt.edu

31
Publications

• Matsumura, L.C., Garnier, H., Resnick, L.B. (in
  press). Implementing literacy coaching The role
  of school social resources. Educational
  Evaluation and Policy Analysis
• Matsumura, L.C., Garnier, H.E., Correnti, R.,
  Junker, B., Bickel, D.D. (in press).
  Investigating the effectiveness of a
  comprehensive literacy-coaching program in
  schools with high teacher mobility. Elementary
  School Journal.
• Matsumura, L.C., Sartoris, M., Bickel, D.,
  Garnier, H. (2009). Leadership for literacy
  coaching The principals role in launching a new
  coaching program. Educational Administration
  Quarterly, 45(5), 655-93.
• For more information Lclare_at_pitt.edu