New NCATE Program Review Process

1
New NCATE Program Review Process

• Donna M. Gollnick
• May 12, 2004

2
Principles for the Reform of the Program Review
Process

• Maintain focus on subject content preparation
• Continue to rely on expertise of SPAs
• Reduce costs and burdens for SPAs
• Reduce costs and burdens for institutions
• Use candidate performance evidence consistently

3
Task Force on Program Review

• Camilla Benbow, Vanderbilt
• Rosalind Hale, Xavier
• Kathe Rasch, Maryville
• Jerry Robbins, Eastern Michigan
• Carol Vukelich, University of Delaware

• Alan Farstrup, IRA, chair
• Josué Cruz, NAEYC
• Steve Gilbert, NSTA
• Jim, Rubillo, NCTM
• Gerald Terozzi, NASSP

• Martha Gage, Kansas
• Charlie Coble, ECS
• Patsy Garriott, Public
• Jennie Pilato, Maryland
• Fran Watkins, Georgia

• Katherine Wright Knight, Teacher in Little Rock
• Blake West, Kansas NEA

4
QUESTIONS THAT PROGRAM REVIEWS SHOULD ANSWER

• Have candidates mastered the necessary knowledge
  for the subjects they will teach or the jobs they
  will perform?
• Do candidates understand teaching and learning
  and can they plan their teaching or fulfill other
  professional education responsibilities?
• Can candidates implement their conceptual plan
  with students and colleagues?
• Can candidates apply their knowledge in
  classrooms and schools?
• Do candidates meet state licensure requirements?

5
Section IContextual Information

• Submit descriptions of the following
• Relevant state or institutional policies and
  practices affecting the program
• Field and clinical experiences
• Admission criteria, including GPA requirements
• The relationship of the program to the units
  conceptual framework
• The relationship of assessments used in the
  program to the units assessment system

6
Section I (continued)

• Attach the following
• The program of study
• Table with number of candidates and completers in
  the program
• Table on faculty expertise and experience in the
  specialty field and in schools

7
Section II Assessments Related Data
When Admin- istered
Scoring Guide/ Criteria
Type of Assess- ment
Name of Assessment
Data Table
Assess- ment
1. 2. 3. 4. 5. 6. 7. 8.
? ? ? ? ? ? ? ?
? ? ? ? ? ? ? ?
? ? ? ? ? ? ? ?
8
6-8 Assessments Must Be Submitted

• 6-8 Required Assessments (depending on the SPA)
• 2 Additional Assessments of Your Choice May Be
  Submitted

9
Section II Assessments Related Data
When Admin- istered
Scoring Guide/ Criteria
Type of Assess- ment
Name of Assessment
Data Table
Assess- ment
? ? ? ? ? ? ? ? ?
1. Praxis II-Elem Ed 2. Comprehensive Ex 3.
Lesson Plan Assign 4. Student Tchng Eval 5.
Student Work Sample 6. Case Study 7.
Diversity Project 8. IEP Project
? ? ? ? ? ? ? ? ?
? ? ? ? ? ? ? ? ?
? Test ? Exam ? Portfolio ? Eval ? Work
Sample ? Case ? Portfolio ? Portfolio
?Before ST ?End of Prog ?ED 320 ?During
ST ?During ST ?ED 330 ?ED 330 ?SPED 310
10
Section II Assessments Related Data
When Admin- istered
Scoring Guide/ Criteria
Type of Assess- ment
Name of Assessment
Data Table
Assess- ment
1. Praxis II-Elem Ed 2. Comprehensive Ex 3.
Lesson Plan Assign 4. Student Tchng Eval 5.
Student Work Sample 6. Case Study 7.
Diversity Project 8. IEP Project
? ? ? ? ? ? ? ? ?
? Test ? Exam ? Portfolio ? Eval ? Work
Sample ? Case ? Portfolio ? Portfolio
?Before ST ?End of Prog ?ED 320 ?During
ST ?During ST ?ED 330 ?ED 330 ?SPED 310
? ? ? ? ? ? ? ? ?
? ? ? ? ? ? ? ? ?
11
Examples of Tables for Aggregating Assessment
Data
12
Teacher Candidates in Science (45 Candidates)
13
Teaching Performance During Internship Final
Evaluation  
 
14
Effectiveness of the Program for the Initial
Preparation of Teachers in Developing Candidate
Knowledge, Skills, and Dispositions Scale 1
strongly disagree 2 moderately disagree 3
disagree 4 agree 5 moderately agree
6strongly agree
15
Section III Alignment of SPA Standards
Assessments
ALIGNMENT with UNIT STANDARD 1
SPECIALTY STANDARD
RELATED ASSESSMENTS
? Content ? Professional Pedagogical KSD ?
Student Learning
Standard 1 Standard 2
?1 ?2 ?3 ?4 ?5 ?6 ?7 ?8
? Content ? Professional Pedagogical KSD ?
Student Learning
?1 ?2 ?3 ?4 ?5 ?6 ?7 ?8
16
SECTION IV Organization of Proficiency Areas

• 1. Content knowledge
• 2. Pedagogical and professional knowledge,
  skills and dispositions
• 3. Effects on student learning

17
Section IV Write the following in 2 pages or less

• Indicate the assessment(s) from Section II that
  provides information about the area and describe
  how it addresses the applicable program
  standards.
• 2. Summarize the data presented in the table(s)
  related to the assessment submitted in Section II
  and interpret the results in terms of the
  standards.

18
1 (Required)-CONTENT KNOWLEDGE

• Data from licensure tests or professional
  exam-inations of content knowledge.

19
2 (Required)-CONTENT KNOWLEDGE

• Assessment of content knowledge in the
  discipline to be taught, teaching field, or other
  education professional field.

20
3 (Required)-PEDAGOGICAL PROFESSIONAL
KNOWLEDGE, SKILLS, AND DISPOSITIONS

• Assessment that demonstrates candidates can
  effectively plan classroom-based instruction, or
  fulfill identified professional responsibilities
  in other professional education roles.

21
4 (Required)- PEDAGOGICAL PROFESSIONAL
KNOWLEDGE, SKILLS, AND DISPOSITIONS

• Assessment that demonstrates candidates'
  knowledge, skills, and dispositions are applied
  effectively in clinical practice.

22
5 (Required)-EFFECTS ON STUDENT LEARNING

• Assessment that demonstrates candidate
  effects on student learning, or on providing
  supportive learning environments for student
  learning.

23
6 (Required)

• Assessment that demonstrates candidates are
  proficient in content knowledge professional and
  pedagogical knowledge, skill, and dispositions
  and/or student learning.
• Examples of assessments include evaluations of
  field experiences, case studies, portfolio tasks,
  licensure tests not reported in 1, and follow-up
  studies.

24
7 (May be Optional)

• Assessment that demonstrates candidates are
  proficient in content knowledge professional and
  pedagogical knowledge, skill, and dispositions
  and/or student learning.
• Examples of assessments include evaluations of
  field experiences, case studies, portfolio tasks,
  licensure tests not reported in 1, and follow-up
  studies.

25
8 (May be Optional)

• Assessment that demonstrates candidates are
  proficient in content knowledge professional and
  pedagogical knowledge, skill, and dispositions
  and/or student learning.
• Examples of assessments include evaluations of
  field experiences, case studies, portfolio tasks,
  licensure tests not reported in 1, and follow-up
  studies.

26
Examples of SPA-specific Assessments
27
ISTE Assessment

• A hardware/software needs assessment survey in
  which the candidate gives the survey, analyzes
  the data collected, develops a budget for
  creating the environment using the suggested
  hardware/software, and develops a multimedia
  presentation that conveys the results to the
  educational community

28
Science Assessment

• An assessment that demonstrates candidate
  knowledge of safety in the classroom, laboratory,
  preparation and storage areas and field (9b) of
  legal liability (9b) and of the use and care of
  living things (9c). This could include a score on
  a composite test covering these areas portfolio
  data or performance in a seminar specifically
  covering these topics.

29
Science Assessment

• An assessment that demonstrates that
  candidates can design, conduct, and present
  research in their disciplines, and use
  mathematics to process and explain data. This
  could include science projects, theses, research
  course or seminar grades, a rigorous test on
  research design, occupational experience in
  research, or similar confirmed experiences in the
  design of research (Standard 1c, 1d).

30
Section VUse of Assessment Results

• Evidence must be presented in this section
  that assessment results have been analyzed and
  have been or will be used to improve candidate
  performance and strengthen the program.

31
(No Transcript)
32
Alignment of Program Review with Standard 1

• Content

• Rubric elements 1-2

Professional Pedagogical Knowledge, Skills,
Dispositions
Rubric elements 3-5
P-12 Student Learning
Rubric elements 6-7
33
Features of Program Report

• Limited to 25-35 pages of text plus 3 attachments
  for each of the 6-8 assessments
• Consistent categories of evidence across programs
• Common report formats from institutions SPAs
• Will not include samples of candidate work

34

• Web-based program report and SPA report

35
The NCATE Report on National Recognition

• Part A Recognition Decision
• A.1 SPA Decision
• A.2 Test Results
• A.3 Summary of Strengths
• Part B Status of Meeting the SPA Standards

36
The NCATE Report on National Recognition (cont.)

• Part C Evaluation of Program Report Evidence
• C.1 Content Knowledge
• C.2 Pedagogical Professional
• Knowledge, Skills,
• Dispositions
• C.3 Effects on Student Learning

37
The NCATE Report on National Recognition (cont.)

• Part D Evaluation of the Use of Assessment
  Data
• Part E Additional Comments
• E.1 Comments on Context, etc.
• E.2 Concerns for Possible Follow-
• up by the BOE Team
• Part F Directions for Preparing a Rejoinder

38
Management by NCATE
Institutions submit electronic program reports
approximately six months before the visit.
NCATE notifies three SPA reviewers that program
reports are available.
Each reviewer prepares a report with a
recognition recommendation.
39
Lead reviewer compiles the three reviews into
one report with a recommendation on national
recognition.
Single SPA report is electronically submitted by
lead reviewers to NCATE.
NCATE notifies institutions that their SPA
reports are available.
40
When the three reviewers can not agree on the
recognition decision, a SPA Audit Committee makes
the recognition decision.
Institutions may electronically submit a
rejoinder to their SPA reports if a program has
not been nationally recognized.
BOE team accesses SPA report for the on-site
visit.
41
Timeline for Implementation

• September 2004

• Pilot testing of new program review system

• Submission of
• program reports
• delayed by 5-7
• months to next cycle

• Program reports
• due September
• 2004 to February
• 2006

42
Next Steps

• Complete the customization of template to SPA
  Standards
• Template available on the web for access in
  August
• Piloting the new process in fall 2004