Criteria 2000 Evaluator Training IEEE

1
Program Preparation Workshopfor ABET Engineering
Program Review Part 2 Presentation

2
Learning Objectives for Workshop

• By the end of the workshop you will
• Understand the ABET Engineering Criteria
• Be able to describe program constituencies
• Be able to formulate measurable program
  attributes such as objectives and outcomes
• Be able to define continuous improvement
  processes relative those measurable attributes
• Understand the ABET review process
• Be able to critically evaluate the program
  against the Criteria from the perspective of an
  external evaluator.

3
Plan for the Workshop Day 2

• Examine and learn from the ABET review process
• Consider the expectations of program evaluator(s)

4
Substantial Equivalency Process Institution
Actions - I

• Request substantial equivalency review
• Prepare self-study that
• represents the program through its purposes
• describes its curriculum and supplemental
  co-curricular opportunities in relation to its
  purposes
• describes its continuous improvement processes
  with respect to fulfillment of its purposes
• describes the (evolving) level of achievement of
  its purposes and the responsive changes to
  (further) improve the level of achievement
• responds to the expectations of the program
  evaluator(s) and the review team chair

5
Substantial Equivalency Process Institution
Actions -II

• Provide self-study, supporting and requested
  documentation, and a statement of materials to be
  made available at the time of the review visit to
  the program evaluator(s) and the review team
  chair
• Respond with alacrity to requests of the program
  evaluator(s) and the review team chair
• Prior to the review visit
• During the review visit
• Following the review visit

6
The EAC Review Timeline
May - June Necessary changes to statement, if
any, are made
March - June Team chairs assigned, dates set,
team members chosen

• January
• Institution requests
• review
• for programs

August Institutions notified of this action
January - February Draft statements edited and
preliminary statements sent to institutions
Year 1
Year 2
July EAC meets to take final action
February - May Institution prepares self-evaluatio
n (Program Self-Study Report)
March - April Institutions respond to draft
statement and return to ABET
September - December Visits take place, draft
statements written and finalized following 14-day
response period
7
Evaluator and Team Provide

• Disciplinary expectations of the programs and
  institution
• Professional preparation for review of the
  programs and the institution
• Experience with continuous quality improvement
  practices

8
ABET Engineering Accreditation Criteria

• 1. Students
• 2. Program Educational Objectives
• 3. Program Outcomes and Assessment
• 4. Professional Component
• 5. Faculty
• 6. Facilities
• 7. Institutional Support Financial Resources
• 8. Program Criteria

9
ABET Engineering Accreditation Criteria

• 1. Students
• 2. Program Educational Objectives
• 3. Program Outcomes and Assessment
• 4. Professional Component
• 5. Faculty
• 6. Facilities
• 7. Institutional Support Financial Resources
• 8. Program Criteria

10
CRITERION 2
11
Issues for Criterion 2

• Educational objectives not published or readily
  accessible to the public
• Educational objectives not related to
  institutional mission
• No evidence of constituency input in objective
  setting or periodic evaluation
• No process for evaluating the extent to which
  educational objectives are attained
• No data available on the attainment of
  educational objectives

12
CRITERION 2
13
Issues for Criterion 2

• Results of evaluation not used to develop and
  improve the program outcomes

14
Questions Evaluator May Ask

• Objectives
• How were the objectives determined?
• Are they consistent with the mission statement?
• Are they measurable?
• How do you accomplish your objectives?
• How do you know when you have achieved your
  objectives?
• What is your process to review and update?
• Constituencies
• Who are your constituencies?
• How do you involve your constituencies in the
  continuous improvement process?
• Are faculty members involved? If so, why? If not,
  why not?

15
Hints for Evaluation of Program Objectives

• The following tools/instruments/methods are
  useful in assessment and evaluation of program
  objectives
• alumni surveys
• employer surveys and placement data
• graduate program surveys and placement data (at
  other universities)
• industrial/professional (including academics and
  alumni) advisory boards
• Programs may choose to use these or other
  appropriate assessment and evaluation
  tools/instruments/methods
• Provide the rationale for each means of
  assessment and evaluation

16
ABET Engineering Accreditation Criteria

• 1. Students
• 2. Program Educational Objectives
• 3. Program Outcomes and Assessment
• 4. Professional Component
• 5. Faculty
• 6. Facilities
• 7. Institutional Support Financial Resources
• 8. Program Criteria

17
CRITERION 3
18
Issues for Criterion 3

• No evidence demonstrating one or more outcomes
• Outcomes (or some) are not assessed directlythat
  is, are not based on objective evaluation of
  student performancebut are validated indirectly
  using
• anecdotal versus measured results
• reliance on course grades
• reliance on self-assessment (e.g., surveys,
  entire or partial cohort)

19
Issues for Criterion 3

• No systematic assessment process
• No process or process not documented
• Plans developed but not implemented
• Little or no faculty support for the process
• No evidence that assessment results are being
  applied to develop programthat is, updating
  objectives and refining outcomes and curriculum
• Assessment results not used
• Assessment and development cycle not complete

20
Questions Evaluator May Ask

• Outcomes
• What should students know and be able to do upon
  graduation?
• Do students know the required outcomes?
• How do you know that the student know?
• Processes
• What is your process to achieve your outcomes?
  How is it documented?
• What does it do? How do you know it does what you
  say it does?
• Who maintains and improves the process? What is
  your involvement?

21
Questions Evaluator May Ask

• Assessment
• Are data being collected for each outcome?
• What is measured? How often?
• How do you use the data you collect?
• Does your evaluation provide the information that
  you need to improve the program?
• What is your feedback process to continuously
  improve the program?
• What actions have been taken to improve the
  program as a result of the assessment process?
• Have past improvement actions been effective?
  What is the basis of that judgment?

22
Questions Evaluator May Ask

• Results
• How do you demonstrate that outcomes are
  attained?
• What is your evaluation of the quality of the
  program?
• What evidence do you have to demonstrate that
  your efforts to improve the program are producing
  results?
• Based on your evaluation of current assessment
  data, what are your plans for additional
  improvement?

23
Outcomes versus Objectives

• Based upon criteria definitions
• Objectives require longer term evaluation
• Will be based on needs of constituencies
• May be based on alumni and employer surveys
• Evaluation is required
• Program development is required
• Outcomes require shorter term evaluation
• Should involve student work as the primary means
  of assessment
• Surveys and other opinion polls can be used as
  secondary evidence
• Outcomes should be measured against an
  expectation
• Closing the continuous improvement loop is
  required

24
Evidence to Be Provided

• Self-study should describe the materials that
  will be available for review during the visit to
  demonstrate (or validate) achievement of the
  program outcomes described in the self-study
• The evaluators will be looking for results of
  assessment and evaluation. Evaluators will not
  assess student work to determine what outcomes
  were covered
• Admonition Taking a course does not guarantee
  achievement of outcomes

25
AssessmentTools and Measures

• The primary outcomes assessment should be based
  on direct measures of student learning
• Senior exit surveys, alumni surveys, and employer
  surveys as means of outcomes assessment are
  indirect measures. The evaluated data can be used
  as supportive evidence. It is not acceptable as
  the primary means of assessment, but may auger
  the results of evaluation of direct measures not
  yet available.

26
Level of Expectation

• Exactly which outcomes must each graduate attain?
• A system must be in place to ensure that all
  graduates have attained the prescribed outcomes
  at some minimum level
• The minimal level of achievement may vary,
  consistent with program educational objectives

27
What to Expect

• Processes in place that provide for
• Definition of desired, measurable outcomes
• Collection of data linked to the outcomes
• Analysis of the data and evaluation of the
  results
• Implementation of change
• Repetition of the cycle, through the review which
  in second and later cycles considers improvement
  or lack thereof

28
What to Look For

• Documentation of results and evidence that
  results are being used to develop the program,
  for example
• Student portfolios
• Subject content examinations
• Performance observations
• Performance evaluations of internships and/or
  co-operative education appointments
• Note Some of the above direct means of
  assessment may be construed as indirect measures
  for a cohort as a whole with respect to some
  particular set of outcomes. How might this be
  rectified?

29
Evaluator Admonitions

• You do not have to be an expert on assessment.
  The institution must provide evidence that it has
  a working and effective continuous improvement
  system in place
• The institution must describe a clear
  relationship between program objectives,
  outcomes, and measurable indicators of success
  with required levels of achievement
• You are assessing the program based on the
  criteria and the strength of the evidence
  provided by the institution, not your own
  personal preferences

30
Exercise 2-A

• A faculty should determine the essential
  knowledge and skill components of each outcome
  that are to be covered in the curriculum. Why is
  this important?
• Having earlier considered a courses to outcomes
  implication matrix, it is suggested that the
  faculty should focus on a subset of required
  courses to associate with a particular outcome.
  Why focus on the required courses?

31
Exercise 2-B

• The faculty should distribute components of this
  outcome across these (required) courses, with the
  components being assigned to several courses. Why
  is this important?
• Criterion 3 calls for each and every student to
  demonstrate achievement of each and every
  outcome. This is difficult. Why?

32
Exercise 2-C

• The EAC white paper on Criterion 3, in essence,
  indicates that
• if the program has pathways that each and every
  student must take,
• if the pathways include learning opportunities
  with respect all the components of each and every
  outcome,
• if the presence of the opportunities (or modified
  opportunities) is regularly validated,
• if a regular assessment process reasonably leads
  to the conclusion that the likelihood is suitably
  low of a student failing to achieve each outcome,
  and
• if a periodic assessment process leads to the
  further conclusion that the likelihood is
  suitably low of a student failing to achieve each
  and every outcome,
• then Criterion 3 is deemed to be satisfied.

33
Exercise 2-C

• Propose a process that fulfills the first four
  bulleted if statements.
• Propose a processmost likely a variation on the
  preceding processthat fulfills all five bulleted
  if statements.

34
ABET Engineering Accreditation Criteria

• 1. Students
• 2. Program Educational Objectives
• 3. Program Outcomes and Assessment
• 4. Professional Component
• 5. Faculty
• 6. Facilities
• 7. Institutional Support Financial Resources
• 8. Program Criteria

35
CRITERION 4
36
Issues for Criterion 4

• Quality of the major design experience
• No culminating experience
• analysis or research instead of design
• several courses with elements of design
• Multiple capstone courses with widely varying
  quality
• Design experience does not address engineering
  standards
• Design experience does not address multiple
  constraints
• Engineering topics can be and typically appears
  to be satisfied by electives, but advising
  doesnt assure adequate coverage

37
Exercise 2-D

• Devise a strategy to ensure that knowledge of and
  ability in design expands for students as they
  progress through the curriculum, culminating in a
  major design experience.
• Consider and describe how constraintsmultiple
  realistic, in the case of the major design
  experiencegenerally, and engineering standards
  in particular, could be incorporated in design
  across your curriculum?

38
Level of Expectation

• The course-of-study of each graduate must satisfy
  all elements of the professional component
  however, the course-of-study may vary, consistent
  with options in the curriculum.

39
ABET Engineering Accreditation Criteria

• 1. Students
• 2. Program Educational Objectives
• 3. Program Outcomes and Assessment
• 4. Professional Component
• 5. Faculty
• 6. Facilities
• 7. Institutional Support Financial Resources
• 8. Program Criteria

40
CRITERION 1, 5, 6, AND 7
41
Issues for Criterion 1

• Problems with student advising (often cited with
  Criterion 5 Faculty)
• ineffective and inconsistent advising
• lack of understanding of curricular requirements
  especially if many options are available
• Problems with student monitoring
• Lack of documentation for
• course substitutions
• after the fact or missing prerequisites

42
Issues for Criterion 1

• Problems with transfer students
• No documentation on acceptability of transfer
  credits (primarily for engineering topics courses)

43
Student Transcripts

• Provide direct evidence that the institutions
  program requirements are met
• Provide evidence that the ABETs program
  criteriamost notably, its professional
  componentare met

44
Transcript Evaluation

• Determine if curriculum is being followed
• Note that curriculum may differ from the one
  being evaluated
• If no Worksheet is provided, ask if it exists
• Does institutional process assure that course
  substitutions meet ABET requirements?
• The transcript should indicate the title of the
  program being evaluated
• Consult team chair for programs in transition

45
Exercise 2-E

• With respect to evaluating, monitoring, and
  advising students, what might be done to
• insure that transfer students satisfy the same
  program requirements and
• address special needs at the time of
  matriculation and graduation.

46
Exercise 2-F

• Describe several ways evaluating, monitoring, and
  advising of students often fall short and propose
  procedures that avoid these faults.

47
ABET Engineering Accreditation Criteria

• 1. Students
• 2. Program Educational Objectives
• 3. Program Outcomes and Assessment
• 4. Professional Component
• 5. Faculty
• 6. Facilities
• 7. Institutional Support Financial Resources
• 8. Program Criteria

48
CRITERION 1, 5, 6, AND 7
49
Issues for Criterion 5

• Insufficient number to
• support concentrations, electives, etc.
• provide student advising
• Poor faculty morale, affecting the program, as
  evidenced by
• the lack or paucity of professional development
• excessive workloads
• the retention/turnover rate
• a low salary structure that negatively impacts
  retention and recruitment (often cited with
  Criterion 7 Institutional Support and Financial
  Resources)

50
Issues for Criterion 5

• Faculty quality
• For teaching design (often specified in Criterion
  8 Program Criteria)
• Excessive reliance on and/or insufficient
  supervision of adjunct faculty members

51
Exercise 2-G

• With due consideration
• of a faculty members obligations other than
  teaching and
• of other factors you determine to be important,
• describe a method to determine the sufficiency,
  or not, of a faculty.

52
ABET Engineering Accreditation Criteria

• 1. Students
• 2. Program Educational Objectives
• 3. Program Outcomes and Assessment
• 4. Professional Component
• 5. Faculty
• 6. Facilities
• 7. Institutional Support Financial Resources
• 8. Program Criteria

53
CRITERION 1, 5, 6, AND 7
54
Issues for Criterion 6

• Insufficient Space
• overcrowded classrooms and/or laboratories
• Classrooms
• lack of appropriate instructional technology
• Laboratories
• unsafe conditions
• inoperable equipment and/or instrumentation
• insufficient modern equipment and/or
  instrumentation (including computer software as
  well as hardware)
• lack of funds for maintaining and/or upgrading
  equipment and/or instrumentation (often cited
  with Criterion 7 Institutional Support and
  Financial Resources)
• Computing/Information Infrastructure
• lack of funds for maintaining and/or upgrading
  these learning resources (cited with Criterion 7)

55
Exercise 2-H

• Identify ten unsafe conditions in a laboratory.
• Identify five indicators of safety conscious-ness
  in a laboratory.

56
ABET Engineering Accreditation Criteria

• 1. Students
• 2. Program Educational Objectives
• 3. Program Outcomes and Assessment
• 4. Professional Component
• 5. Faculty
• 6. Facilities
• 7. Institutional Support Financial Resources
• 8. Program Criteria

57
CRITERION 1, 5, 6, AND 7
58
Issues for Criterion 7

• Unstable leadership affecting programs
• Dean and/or Department Head positions open or
  filled by interim appointments for an extended
  period of time
• frequent turnover of university administration
  and engineering school leadership
• Inadequate operating budget affecting
• acquisition and maintenance of classroom,
  laboratory, and computing resources (including
  space and embedded resources)
• faculty salaries, promotions, and professional
  development, all affecting recruitment and/or
  retention

59
Issues for Criterion 7

• Insufficient support staff
• teaching assistants
• technicians for instructional laboratories,
  machine shops, and laboratory services
• administrative and/or clerical

60
Exercise 2-I

• Outline a needs assessment process to determine
  resources that are needed
• by information services and computing services
  and
• for classrooms and instructional laboratories
• to facilitate the achievement of program
  educational objectives and outcomes.

61
ABET Engineering Accreditation Criteria

• 1. Students
• 2. Program Educational Objectives
• 3. Program Outcomes and Assessment
• 4. Professional Component
• 5. Faculty
• 6. Facilities
• 7. Institutional Support Financial Resources
• 8. Program Criteria

62
CRITERION 1, 5, 6, 7, AND 8
63
Issues for Criterion 8

• Failure to conform to the curriculum requirement
  of the applicable program criteria, particularly
  with respect to
• mathematics and basic sciences conditions
• distribution requirements with respect to
  disciplinary technical areas

64
Exercise 2-J

• For the ABET program criteria applicable to your
  degree program, create a checklist of
  requirements.
• Use the checklist to determine what requirements
  are not met.

65
CRITERIA 2000
66
Matrix for Implementation Assessment

• Self assessment of readiness of nominally 3 on
  the scale of 1 to 5 on the six categories
• education objectives,
• constituents,
• processes,
• outcomes assessment,
• results, and
• system
• is a strong indicator of readiness for an ABET
  review.

67
(No Transcript)
68
Alternative to Matrix

• Another route to setting the scale value for each
  implementation factor is to respond to the
  request Select a numerical value on the scale of
  1 to 5 that most accurately describes the extent
  to which

69

• Program Educational Objectives have been
  established and maintained
• Constituents are involved in helping set program
  objectives and in evaluating the level to which
  they are being achieved
• The required Processes are operational
• Outcomes Assessment is being practiced
• Results of outcomes and the various processes are
  being used to improve programs and assure
  objectives are being achieved
• An overall System is in place to meet the
  accreditation requirement

70
Exercise 2-K

• Estimate your readiness in each of the six
  categories
• education objectives _____
• constituents _____
• processes _____
• outcomes assessment _____
• results _____
• system _____
• What should you work on
• first?
• second?

71
Self- Study

• The institution may employ any means it chooses
  to represent itself to ABET and the visiting
  team. Consequently, the references to specific
  tables in the instructions are for guidance only.
  The information may be presented in any manner
  the institution chooses.

72
Program Self-Study Report

• Background Information
• Degree titles
• Program modes
• Actions to correct previous deficiencies
• Contact information
• Accreditation Summary
• How each element of criteria is met (in order)
• Appendix I Additional Program Information
• Appendix II Institutional Profile
• Background information relative to the
  institution
• Background information relative to the
  engineering unit
• Tabular data for the engineering unit

73
Accreditation Summary

• For each of the eight criteria
• A complete description of how the program
  satisfies each of the EC2000 requirements
• Evidence that demonstrates that each of the
  EC2000 requirements has been fulfilled
• A description of additional evidence that will be
  provided during the campus visit
• General advanced-level program criterion

74
Appendix I Additional Program Information

• Tabular Data
• Basic-Level Curriculum Table
• Course and Section Size Summary Table
• Faculty Workload Summary Table
• Faculty Analysis Table
• Support Expenditures Table
• Course Syllabi
• Faculty Curriculum Vitae (2 pages)

75
Appendix II - Institutional Profile
Information about the institution and the
engineering unit, identified as Appendix II, may
be attached to each Program Self-Study Report or
supplied as a separate document.
76
Review ProcessPre-Visit
77
Evaluator Responsibilities

• Analysis
• Review Program Self-Study Report and other
  materials sent by institution to assess areas of
  apparent strength and shortcoming
• Evaluate transcripts
• Make preliminary assessment of which ABET
  Criteria appear to be satisfied and which may not
  be met
• Develop plan for additional analysis during
  campus visit
• Compile questions and identify matters for which
  additional information is desired (notify Team
  Chair)
• Compile list of desired interviews and meetings
  with faculty, students, and others (notify
  Program Head)
• Compile list of visits by team members to support
  units (notify Team Chair)
• Documentation
• Prepare required Pre-Visit forms
• Begin Draft (Final) Report preparation

78
Communications Admonitions

• While it is appropriate for the Program Head to
  assist in making arrangements, the Program
  Evaluator should specify who is to be interviewed
  and the order desired.
• The program must demonstrate compliance with the
  criteria. Give them the flexibility and latitude
  required to accomplish this.

79
Communications Admonitions

• Based on review of self-study materials, the
  Evaluator should convey to the Program Head a
  clear sense of additional information needed for
  a complete analysis. Avoid last minute requests.
• Do not conduct interviews or share preliminary
  conclusions prior to Campus Visit.
• Keep the Team Chair informed of all
  communications.

80
Review ProcessVisit
81
Objectives Of On-Campus Visit

• Conduct a detailed examination of the materials
  compiled by the Institution
• Make a qualitative assessment of factors that
  cannot be found in a written document
• Provide the institution with a preliminary
  assessment of its strong points and shortcomings

82
Campus Visit Activities

• Day 0
• Initial team meeting
• Review visit plan
• Provide pre-visit documentation to Team Chair
• Report pre-visit assessment
• Review criteria and policy changes
• Visit program to begin evaluation of
• materials
• classroom facilities
• laboratory facilities
• computing and information infrastructure
• other visit activities (including interviews)

83
Evening Team Meeting

• Day 0
• Program evaluator identifies all potential
  shortcomingsdeficiencies, weaknesses, and
  concernsto the Team
• Program Evaluators are agents of the Team
• Continue Draft (Final) Report preparation

84
Campus Visit Activities

• Day 1 (morning)
• Team meeting with President, Vice President of
  Academic Affairs, and/or Dean and designated
  guests
• Evaluator meetings with
• Program Head
• program teams (task forces, committees, etc.),
  faculty members, students, and support staff
  members (based on pre-visit requested
  appointments)
• Luncheon with institutional officials and guests
  (at the discretion of the institution)
• Typical and selected guests at an evaluators
  table include
• leaders of supporting academic and administrative
  service units
• leaders and faculty members of the program
• members of any program-relevant advisory board
• alumni of the program
• student leaders of the program

85
Campus Visit Activities

• Day 1 (afternoon)
• Continue meetings and interviews (including of
  students)
• Continue visits of facilities and reviews of
  materials

86
Evening Team Meeting

• Day 1
• Report findings relative to previously identified
  potential shortcomingsdeficiencies, weaknesses,
  and concerns to the Team
• Discuss possible accreditation action
• Continue Draft (Final) Report preparation

87
Campus Visit Activities

• Day 2 (morning)
• Provide to the Team Chair the Draft (Final)
  Report of program, which will also be the basis
  of the Exit Interview Statement
• Complete meetings and interviews
• Complete visits of facilities and reviews of
  materials
• Brief Program Head on findings

88
Campus Visit Activities

• Day 2 (lunchtime)
• Complete and provide Team Chair with program
  evaluator report, including
• updated pre-visit report forms
• list of persons interviewed
• recommended accreditation action form
• Draft (Final) Report program statement, which
  will be basis of the Exit Interview Statement
  (sometimes with strengths redacted)

89
Campus Visit Activities

• Day 2 (afternoon)
• Exit Interview with President and others at
  his/her discretion
• Read Exit Interview Statement for the program
• Leave a copy of the Program Audit form for the
  program

90
Admonitions for Evaluator

• Do not mention proposed recommended accreditation
  action
• Read written statement and do not ramble
• As time permits, read positive as well as
  negative observations
• If there are any deficiencies or weaknesses, they
  must be specifically and clearly stated and
  correlate exactly with those summarized in the
  Program Audit form
• Be courteous and professional

91
Draft (Final) Statement
A. Introduction B. For the Institution 1.
Strengths 2. Deficiencies 3. Weaknesses 4.
Concerns 5. Observations C. For each Program 1.
Strengths 2. Deficiencies 3. Weaknesses 4.
Concerns 5. Observations
92
Review ProcessPost-Visit
93
Post-Visit Process

• 14-Day Response from institution
• Draft Statement prepared and sent to institution
• 30-Day Due Process Response from institution
• Revised Draft Statement prepared
• ABET International Activities Committee takes
  final action on substantial equivalency status
• ABET sends Final Statement and substantial
  equivalency status letter to institution

94
Review LanguageandActions
95
Levels of Criteria Compliance

• Key Terms
• Compliance satisfies criterion
• Concern criterion is currently satisfied, but
  potential exists for situation to change such
  that the criterion may not be satisfied.
• Weakness program lacks strength of compliance
  to ensure the quality of the program will not be
  compromised. Remedial action is required to
  strengthen compliance with the criterion
• Deficiency criterion is NOT satisfied,
  therefore, the program is not in compliance.

96
Use of Key Terms

• Use key terms only in reference to overall
  evaluation of each criterion

97
Compliance with Criteria

• Compliance is based on Criteria, not opinion
  however,
• in the overall determination of compliance with
  each criterion the evaluator is called on to use
  good judgment in
• weighing the contributions of all elements of a
  criterion and
• deciding whether the resulting recommended action
  is consistent with the nature of the shortcoming

98
Prior Assessment Statistics
99
Accreditation Actions

• NGR Next General Review
• IR Interim Report
• IV Interim Visit
• SC Show Cause
• RE Report Extended
• VE Visit Extended
• SE Show Cause Extended
• NA Not to Accredit

100
Terminology vs ActionGeneral Reviews
 
101
Interim Action Recommendations IR vs IV

• Interim Visit recommended when degree of
  resolution cannot be determined by review of a
  report or when previous written information has
  not been effective in providing the necessary
  evidence. Requires a review by an evaluator on
  campus (e.g., student work, lab safety)
• Interim Report recommended when resolution of
  shortcomings can be described by a report (e.g.,
  faculty hiring). The current program evaluator
  and team chair may review the interim report,
  assess progress, prepare a statement, and
  recommend accreditation action

102
Actions and DurationsGeneral Review
103
Terminology vs ActionInterim Review
 
104
Interim Visits

• Generally focus only on identified deficiencies
  (only if show cause) and weaknesses
• If a concern has been addressed, then the team
  should evaluate its resolution
• significant deterioration surrounding a concern
  may become a weakness or deficiency
• If a concern has not been addressed, then the
  team should determine if the situation has
  deteriorated significantly
• again, significant deterioration surrounding a
  concern may become a weakness or deficiency

105
Actions and DurationsInterim Review
106
Consistency Is Important

• PEV Report
• Recommended Action
• Exit Interview
• Draft Statement

107
Consistency Checks

• Overall considerations
• Accreditation actions must be consistent across
  all institutions
• Accreditation actions must be consistent with
  those given to other programs with similar
  shortcomings (weaknesses, deficiencies)
• Consistency is checked at five levels to various
  degrees of detail

108
Consistency Checks
EAC Consistency Committee Final check
EAC Meeting
ABET HQ Accreditation Director
Editor 2 checks among all reports received
Editor 2
Director checks higher-level consistency
Editor 1 checks among all reports received
Editor 1
Team Chairs check among evaluators
Professional Societies
Team Chair
Team Chair
Team Chair
Team
Team
Team
Team
Team
109
Consistency Issues for Teams

• The depth and completeness of the evaluation from
  program to program
• The assignment of appropriate key terms
  (deficiency, weakness, concern) to describe
  shortcomings
• Consistency across all programs in an institution
  
• Consistency on interim recommendations for
  weaknesses of an IR versus an IV

110
Consistency Issues for Criteria 2 and 3

• Most consistency issues have centered on Criteria
  2 and 3
• Team chairs and evaluators should have a clear
  understanding of
• The requirements of the criteria
• the institutions terminology relevant to these
  criteria

111
Criterion 2 Program Educational Objectives

• Broad statements that describe the career and
  professional accomplishments that the program is
  preparing graduates to achieve
• Consistent with the mission of the institution
• Allows differentiation between programs in the
  same discipline

112
Criterion 2 Program Educational Objectives

• Consider a deficiency if the general intent of
  Criterion 2 is not met
• Contributing factors may include
• No involvement of constituencies
• Educational program does NOT prepare students to
  attain program outcomes (links to curriculum)
• No process for evaluating the objectives
• No data that demonstrate the extent to which
  objectives are attained
• No evidence of development and improvement of
  program outcomes

113
Criterion 2 Program Educational Objectives

• Consider a weakness if the general intent of
  Criterion 2 is met but lacks the strength of
  compliance to ensure program quality
• Contributing factors may include
• Objectives are published but are not accessible
  to constituencies and potential students
• Limited involvement of constituencies
• Incomplete links to curriculum or links are not
  clear
• Incomplete process for evaluating objectives
• Incomplete evidence of development and
  improvement of program outcomes

114
Criterion 2 Program Educational Objectives

• Consider a concern if the general intent of the
  criterion is fully met, but potential exists for
  change such that criterion may not be satisfied
  in the future
• Contributing factors may include
• Objectives are published, but are changed
  frequently
• Objectives are evaluated, but there is limited
  involvement of constituencies in this process or
  it varies from cycle to cycle (2b)
• Program development processes may rely too
  heavily on one person

115
Criterion 3 Program Outcomes and Assessment

• Narrower statements that describe what students
  are expected to know or be able to do by the time
  of graduation (Relate to skills, knowledge, and
  behaviors that students acquire from the
  program.)
• The attainment of all outcomes indicates that the
  student is equipped to achieve the program
  educational objectives
• ABET designated outcomes a-k included in some way

116
Criterion 3 Program Outcomes and Assessment

• Consider a deficiency if the general intent of
  Criterion 3 is not met
• Contributing factors may include
• No documented working process(es) to produce
  outcomes
• Loop not closed on any outcomes
• Absence of defined goals and documented
  assessment results
• No assessment evidence that outcomes are attained
  by students
• No evidence of efforts at program development
  based on assessment

117
Criterion 3 Program Outcomes and Assessment

• Consider a weakness if the general intent of
  Criterion 3 is met but lacks the strength of
  compliance to ensure program quality
• Contributing factors may include
• Absence of a working process(es) to produce some
  outcomes
• Loop closed on some outcomes
• Defined goals and documented assessment results
  for some outcomes
• Absence of assessment evidence for a small number
  of outcomes
• Incomplete evidence of efforts at program
  development based on assessment

118
Criterion 3 Program Outcomes and Assessment

• Consider a concern if the general intent of the
  criterion is fully met, but potential exists for
  change such that criterion may not be satisfied
  in the future
• Contributing factors may include
• Process to produce some outcomes is possibly
  inconsistent and may lead to circumstances in
  which their quality is insufficient to meet
  program metrics
• Loop closed on most outcomes, but some important
  evaluation results have not been acted upon
• Inconsistent coverage or assessment of a small
  number of outcomes, may be overly dependent on
  one person

119
Plan for the Workshop Part 2

• Examine and learn from the ABET review process
• Consider the expectations of program evaluator(s)