Studying Lesson Study

1
Studying Lesson Study

• Results from Michigans Round 1
  Mathematics/Science Partnership Grant Projects

2
Contents

• Introduction to Lesson Study and MSP
• Round 1 projects
• What we have learned so far
• Qualitative data
• Quantitative data

3
What is Lesson Study?

• A view from Macomb

4
What is a Mathematics/Science Partnership Grant
(MSP- Title IIB)?

• Increase the academic achievement of students in
  mathematics and science by enhancing the content
  knowledge and teaching skills of classroom
  teachers.
• Partnerships between high-need school districts
  and the science, technology, engineering, and
  mathematics (STEM) faculty in institutions of
  higher education are at the core of these
  improvement efforts.

5
What is a Mathematics/Science Partnership Grant
(cont.)

• Scientifically-based professional development
• Quasi-experimental design
• Treatment and control groups
• Quantitative and qualitative data

6
Michigan MSPs

• In Michigan our first round applicants were asked
  to focus on
• Mathematics K-8
• Teachers that needed to become highly qualified

7
Michigan MSPs

• 1st round received award in February, 2004
• 4 recipients
• 18 months
• Projects ended August 30, 2005
• All received a 2-year continuation which ends
  August, 2007.
• All had a Lesson Study Component

8
Michigan MSPs

• Similarities
• Shared a common control group
• Worked together to develop a common content
  knowledge measuring tool
• Planned a common introduction workshop to Lesson
  Study
• Summer Institutes
• Middle school
• Lesson study topic determination based on
  weaknesses in student achievement

9
Michigan MSPs

• Differences
• of teachers receiving treatment
• Team configurations
• Interventions
• Lesson creation/research
• Implications
• Success with lesson study may be affected by
  these variables

10
Michigan MSPs

• Purpose of Lesson Study in our Projects
• Reinforce the content teachers learned from the
  Math Institutes/content courses
• Help teachers use this content knowledge in their
  classrooms
• Increase teacher collaboration around student
  learning

11
What have we learned so far?

• What our experiences have taught us so far about
  implementing Lesson Study.
• What qualitative data suggests about the
  effectiveness of Lesson Study in improving
  mathematics instruction.

12
Qualitative Measuring tools

• Teachers
• Surveys
• Journals
• Observations
• Final reports
• Students
• Observations, including videos
• Work samples
• Student presentations

13
Lessons Studied, Lessons Learned

• Lessons Learned

14
Project Jugyoukenkyuu

• Difficulties
• Teachers were not all volunteers
• Lack of STEM leadership/involvement as
  knowledgeable others
• Scheduling and time out of the classroom for
  teachers
• Different priorities for PD within a district

15
Project Jugyoukenkyuu

• What we have learned
• Teachers are examining their questioning skills
• Rethinking their day to day lessons
• Become more aware of the involvement level of
  their students in a lesson
• Start to think more about the student reaction
  and student response to activities

16
Project TEAM2

• Transforming Education and Achievement
• in Middle School Mathematics
• Benton Harbor Area Schools
• Grades 4-5
• Summer Content Institutes
• School Year Everyday Math PD
• Grades 6-8
• Summer Technology Institutes
• School Year Lesson Study (2 cycles)

17
Project TEAM2

• Results of the Professional Development Feedback
  Likert Scale (5 pt)
• mean score differences showed that while teachers
  feedback was positive from both groups, the
  grades 6-8 teachers scores were significantly
  higher in 4 of 5 categories.

18
Project TEAM2

• Teachers viewed their Lesson Study Professional
  Development Experience as having a greater impact
  upon
• Knowledge about the Michigan GLCEs (4.2gt3.7)
• Use of a variety of instructional strategies
  (4.4gt3.9)
• Their own mathematics content knowledge
  (4.4gt3.7)
• Their understanding about how students learn
  mathematics (4.0gt3.3)

19
Challenges in Implementing Lesson Study in
Mathematics in an Urban School District

• Administrative Issues
• Teacher Issues
• Substitute Issues
• Student Issues
• Restructuring Issues
• Community Issues

20
Project TEAM2

• What was gained from ?
• Planning the Research Lesson collaboratively
• Observing and debriefing the Research lesson
• Revising and re-teaching the Research Lesson

21
Sustained Professional Development Student
Achievement

• Lesson Study is valuable because
• a valuable opportunity for teachers
  collaboratively discuss issues, research content,
  and plan a lesson.
• promotes openness to other perspectives of
  teaching mathematics.
• directly relevant to day to day classroom
  teaching.
• changes the focus from teaching a lessons
  content to how students respond to and learn that
  lessons content.
• increases teacher knowledge of how students
  respond to a lesson/learn.
• provides an opportunity for students to see
  teachers model professional inquiry and
  collaboration.

22
Sustained Professional Development Student
Achievement

• Our Lesson Study could be improved by
• A stronger assessment of student learning to
  evaluate the effectiveness of the lesson and
  improve formative assessment.
• A more detailed lesson script of possible teacher
  student interactions.
• Scheduling is difficult. In the future make
  lesson study integral to the school professional
  learning environment and allocated PD time.
• Administration needs to be included in the lesson
  observation and debriefing so they understand
  Lesson Study and learn how to support process.
• Lesson study is currently supported by outside
  funding (see third bullet).

23
Sustained Professional Development Student
Achievement

• Changes in Practice Due to Lesson Study
• I listen more closely to students.
• It is okay for students to make mistakes if they
  can explain their thinking. It is a part of the
  learning process.
• I ask students to explain or justify their
  answers.
• I think more about lesson objectives and student
  response/learning in planning.
• I use more group work when I teach.

24
What have we learned so far?

• What quantitative data suggests about the
  effectiveness of Lesson Study in improving
  mathematics instruction.

25
Quantitative Measuring tools

• Teachers
• Content
• Michigan Test for Teacher Certification (MTTC)
• Learning for Mathematics Teaching (LMT)
• Instructional Techniques
• Science and Mathematics Program Improvement
  (SAMPI)
• Survey of Enacted Curriculum (SEC)
• Students
• Michigan Educational Assessment Program (MEAP)
• Standardized tests
• STAR Math

26
Student Results

• Three sites reviewed MEAP data
• All tracked trend data (increased or decreased
  percents proficient)
• One site found no change at grade 4(no Lesson
  Study) and an increase at grade 8 (Lesson Study)
• One site found the treatment group increased and
  noted the control group decreased
• One site found an upward slope for the treatment
  group

27
Student Results

• One site used district administered standardized
  NRT data for the five of seven districts that
  used standardized tests
• Data for three districts were compared to a
  control group
• Data for two districts were compared pre/post
• No significant differences were found
• A second site used the Star Math assessment
  pre/post no significant differences and small
  effect sizes were found

28
Teacher Results

• Four sites used the Michigan Test for Teacher
  Certification (MTTC) and conducted pre/post
  treatment/common control comparisons
• The four sites reported no significant
  differences
• One site found that being highly qualified had a
  significant influence on pre/post MTTC as did
  being highly qualified and participating in
  Lesson Study and being male
• A second site found no significant difference in
  participating in Lesson Study

29
Teacher Results

• Two sites used the Learning for Mathematics
  Teaching (LMT)
• Both used a pre/post design
• One site found no significance
• One site found significant differences on two of
  three subtests (geometry and numbers/operations)
  but not for algebra

30
Teacher Results

• Two sites used Science and Mathematics Program
  Improvement (SAMPI)
• both used a pre/post design
• One site found an overall downward trend
• One site found no significant differences
• small effect sizes for content and lesson overall
  
• medium effect sizes for implementation of lesson

31
Teacher Results

• Four sites used the Survey of Enacted Curriculum
  (SEC)
• all used a pre/post treatment and control
  design
• Three sites grouped items to form a variety of
  subsets
• One site reported statistics for individual items
  related to that sites goals

32
Teacher Results

• SEC (contd)
• Most results were mixed with some increases and
  some decreases.
• There appears to have been a positive change for
  the scale measuring active teacher engagement in
  professional development

33
Lessons Studied Lessons Learned
Specific Quantitative Data From the Survey of
Enacted Curriculum, scale scores were derived
from Treatment participants (n47) as compared to
the Control group (n45) at the post-assessment
and used in the final Evaluation Report. These
scale scores represent a cluster of questions
from the SEC survey which are accompanied with
reliability coefficients determined from the
Wisconsin Center for Teacher Research based on a
massive data base.
34
Lessons Studied Lessons Learned

• Specific Quantitative Data
• Further post-assessment within the Treatment
  group, a subgroup of Lesson Study participants
  (n26) was compared to non-lesson study
  participants (n21) as to the effectiveness of
  the Lesson Study intervention.
• Again scale scores were derived from the SEC
  survey.

35
Lessons Studied Lessons Learned

• Active Teacher Engagement Criteria from SEC
• Observed demonstrations of teaching techniques
• Led group discussions
• Developed curricula or lesson plans which other
  participants or the activity leader reviewed
• Reviewed student work or scored assessments
• Developed assessments or tasks as part of a
  professional development activity
• Practiced what you learned received feedback as
  part of a PD activity
• Received coaching or mentoring in the classroom
• Given a lecture or presentation to colleagues

36
Lessons Studied Lessons Learned

• Active Teacher Engagement Criteria from SEC
  (contd)
• Our results Treatment M1.37,SD.55
• Non-treatment M .89, SD .61 p .012
• Reliability coefficient 0.767

37
Lessons Studied Lessons Learned

• Conclusion
• Lesson Study protocol seems to be an effective
  method of active teacher engagement in
  professional development as determined from our
  preliminary data.
• IF the goal of the intervention includes the
  criteria as listed in the scale cluster of the
  SEC survey, then a noted influence has been
  observed.

38
Limitations of Quantitative Data

• Small n sizes
• Local evaluators were not required to use a
  common format
• MEAP test individual student pre/post results
  not possible
• Matching common control group students with
  program students was a challenge

39
Key Recommendations

• Continue quasi-experimental treatment vs.
  comparison group design
• Address issues related to internal validity based
  on comparison group choice
• Continue to use established performance measures
  for students and teachers
• Identify additional performance measures for
  Lesson Study

40
Key Recommendations (cont)

• Increase number of participants
• Address SEC administration procedures
• Consider issues of congruency between school
  mathematics texts and PD offerings
• Continue the provision of manipulatives for
  classroom use
• Encourage university instructors to incorporate
  more of the PD techniques in their instruction

41
Conclusions

• Qualitative data suggests PD well received
• Quantitative data suggests limited improvement
  for teachers
• No discernable change in student achievement

42
Conclusions

• Dysfunctional systems that are characteristic of
  high needs schools
• Statistical power of quantitative data limited
  by
• n size
• Sensitivity of measuring tools
• Time
• Longitudinal studies are necessary, especially
  with respect to student data

43
Measurable Aspects of Lesson Study

• Specific content that is the topic of the lesson
  studied
• Hard to define because of the individual needs of
  the teams, i.e. manipulatives, student
  questioning
• But this is the strength of Lesson Study
• The work that teachers do in each lesson varies
• But over time we would see changes in classroom
  culture that supports student learning ultimate
  measure

44
For more information -

• Michigan MSP website
• www.michigan.gov/mspartnership
• MTTC Study Guide
• http//www.mttc.nesinc.com/MI_viewSG_opener.asp
• Survey of Enacted Curriculum (SEC)
• www.seconline.org
• Learning Mathematics for Teaching (LMT)
• http//sitemaker.umich.edu/lmt
• SAMPI
• http//www.wmich.edu/sampi/
• Ruth Anne Hodges
• Michigan Department of Education
• hodgesra_at_michigan.gov

45
www.misd.net/lessonstudy

• To learn more about Lessons Studied Lessons
  Learned and to view the lesson study videos
  visit
• www.misd.net/lessonstudy

46
Proposal

• Does Lesson Study really work? Is there
  quantifiable evidence to suggest that it is an
  effective teacher development tool? Funded by an
  MSP grant four different groups in Michigan
  studied Lesson Study using a scientifically-based
  research model. The meta-analysis of this data
  will be presented supported by participant
  vignettes.

47
Objectives of the Presentation

• In Michigans first round of Mathematics/Science
  Partnership grants all grantees chose to evaluate
  Lesson Study as a teacher development tool using
  a scientifically based research model. Although a
  lot has been written about Lesson Study there is
  little actual quantitative data to support its
  effectiveness. We feel that it is important that
  we share our results from these studies with
  those responsible for designing and implementing
  teacher development.

48
Evaluation Designs
Michigan MSPs

• Quasi-experimental employing a common control
  group and pre/post measurement
• Multiple measurement instruments
• All used MTTC compilation and Survey of Enacted
  Curriculum (SEC)
• 2 used Learning Mathematics for Teaching (LMT)
  and SAMPI