Status of NCATEs Program Review Process

1
Status of NCATEs Program Review Process

• Wendy Wiggins
• Margie Crutchfield
• May, 2005

2
Why change program review?

• Reduce differences across SPAs
• Need for consistent requirements for evidence
• Explicitly tie together unit and program review

3
NCATE Unit Standard 1

• Candidates preparing to work in schools as
  teachers or other school personnel know the
  content of their fields, demonstrate professional
  and pedagogical knowledge, skills, and
  dispositions and apply them so that students
  learn. Assessments indicate that candidates meet
  professional, state, and institutional standards.

4
New Program Review Process

• Increases uniformity among SPAs, identical
  templates
• Explicitly links unit and program review
• Streamlines process and report
• Limits number of assessments (6-8)
• System is administered by NCATE staff
• Is web-based

5
6 - 8 AssessmentsThe rules

• Institution must submit a minimum of six
  assessments, unless the SPA specifies more than
  six
• Institution may submit additional assessments
  when SPA does not specify all eight assessments
• Five specific types of assessments are required
  by all SPAs
• Several SPAs have additional required assessments
  (outlined on templates available on NCATE web
  site)

6
Required Assessments

• State licensure exam for program area (if
  availableotherwise another content based
  assessment)
• Content Assessment
• Assessment of Planning (e.g., unit plan)
• Student teaching/internship assessment
• Assessment of candidate impact on student
  learning or providing a supporting learning
  environment

7
Documentation for each assessment

• Complete description of assessment
• Scoring guide
• Data chart
• Section IV 2-page roadmap for reviewers
• how the assessment meets its assigned standards
  and what faculty have learned from the data for
  that specific assessment.

8
What Have We Learned So Far?

• Fall 2004 Pilot Program Reviews

9
Fall 2004 Pilot Group

• 103 reports were reviewed
• Pilots included 35 institutions from 18 states
• Pilot reports were submitted for review to all
  but three SPAs
• About 40 of the decisions were positive (either
  recognition or close)
• All SPAs recognized at least one or two programs
  in the pilot group

10
Survey of the Pilot Review Process

• Some findings from the reviewer survey
• Reviewer training was accurate and useful (85)
  76 felt prepared to conduct reviews
• Program report categories clear and
  understandable (78)
• Easy to navigate report to find necessary
  information for the review (52)

11
Survey of the Pilot Review Process

• Some findings from the survey of program
• report compilers
• Evidence fits the standards (84)
• Evidence provided in report will help with the
  writing of the institutional report for the unit
  (82)
• NCATE and SPA advice were consistent (59)

12
Time Spent on Reports

• Compilers of program reports
• 20-200 hours 81 hours average
• Reviewers of program reports
• 5-150 hours 22.4 hours average

13
Interesting Comparison

• Programs
• We felt prepared to provide the information
  requested 69
• SPA Reviewers
• Program report information provides an adequate
  demonstration that candidates are proficient in
  SPA standards and preparation programs are of
  high quality 50

14
Problems Reviewers Found in Program Reports
Organizational

• Reports missing required sections or incorrectly
  prepared
• Reports not logically organized
• Programs not following directions for required
  assessments
• Programs often did not take advantage of Section
  IV to explain the assessment

15
Problems Reviewers Found in Program Reports
Assessments

• Generic assessments sometimes same assessment
  used more than once
• Grades used as assessments without adequate
  description of what they measure
• Inappropriate assessments (Praxis I scores, entry
  GPAs)
• Alignment to SPA standards not given
• No correlation of assessment ? scoring guides ?
  data

16
Problems Reviewers Found in Program Reports
Scoring Guides

• In many cases, the weakest link in the report
• Aexcellent is not enough the reviewer must
  know how the program distinguishes an A from a B.
  
• Journal10 pts is not enough the reviewer must
  know what qualities in a journal would earn 10
  pts, 5 pts, 0 pts

17
Problems Reviewers Found in Program Reports Data

• Data not broken down into subscores or assessment
  categories (not telling the story)
• Data not disaggregated when appropriate
• Poor data not reflected upon in Section IV or V
• Data difficult to interpret

18
One More Important Lesson
19

• It is possible for programs to demonstrate that
  their candidates have mastered the program
  standards using a limited number of comprehensive
  assessments.

20

• An Example

21
An English EducationProgram Report

• Received National Recognition from NCTE
• Doesnt mean it is perfect!

22
Used 8 assessments

• State Licensure Test Data
• English Unit Planning Assignment (Content)
• Classroom Planning Assignment
• Teaching Competence
• Student Teaching Assessment
• P-12 Outcomes Assessment
• GPA in English Courses
• Writing Inquiry Project

23
Lets look at one standard
24
Assessment 1 State Licensure Test

• Competency 004. The teacher understands reading
  processes and teaches students to apply these
  processes.
• Competency 005. The teacher understands reading
  skills and strategies for various types of
  nonliterary texts and teaches students to apply
  these skills and strategies to enhance their
  lifelong learning.
• Competency 007. The teacher understands
  strategies for reading literary texts and
  provides students with opportunities to
  formulate, express, and support responses to
  literature.

25
Assessment 2 Unit Plan

• Partial Instructions
• Elements carried over (extended, elaborated) from
  stage I
• Guiding question
• Use of reading process in approaching each main
  reading
• Use of "thinking activities" described by Burke
• Thematic approach unit as ongoing inquiry
• Use of the ELA cycle so that speaking, listening,
  reading, writing, and thinking reinforce each
  other.
• Application of principles of responsible
  assessment (must specify method of assessment for
  each activity, overall weighting of various
  activities into a unit grade, and at least one
  rubric for a major assignment)
• Application of Burke's principles Construction,
  Occupation, Negotiation, and Conversation

26

• Additional required elements
• Variety of reading at least two different genres
  of literature, plus nonfiction, and
  multicultural readings. Consider also using film,
  and/or internet.
• Use of young adult literature
• Study of media, and activities which would fit
  the "viewing and representing" TEKS
• Speaking/listening activities including both
  instruction and practice
• Description of how major assignments/activities
  will be modified for English Language Learners

27
Scoring Guide and Data for Assessment 2
28
Assessment 7 Grades in English Language Arts
Courses

• Standard 3.3 Met by
• ENGL 3326 Advanced Expository Writing
• ENGL 4314 Critical Theory
• ENGL 3321 Issues in Teaching Language and
  Literature
• Reading 3326 Reading in the Content Areas
• Reading 4310 Diagnostic and Prescriptive Reading

29

• Descriptions provided for each course

30
GPA is calculated on a 4 point scale, with 4.0 as
the maximum possible. A GPA of 2.5 in the major
field is required by the state.
31
What Well Be Doing This Summer

• Adding examples of reports, assessments, and data
  charts to the NCATE web site
• Working with SPAs to revise reviewer guidelines
• Revising directions to institutions on report
  preparation and what constitutes appropriate
  evidence

32
Other ways to inform programs about the changes

• Training events sponsored by States
• South Carolina
• Ohio
• Oklahoma
• Hawaii

33

• Regional training events sponsored by
  universities
• University of Mississippi
• Central Missouri State University
• University of Delaware
• University of Hawaii

34

• Training by all SPAs at their annual conferences

35
More Information?

• Margie Crutchfield
• margie_at_ncate.org
• Wendy Wiggins
• wendy_at_ncate.org
• 202-466-7496
• www.ncate.org