Title: Outcomes Assessment 2 Program Assessment
1Outcomes Assessment 2Program Assessment
- Joseph A. Shaeiwitz
- West Virginia University
- joseph.shaeiwitz_at_mail.wvu.edu
Daina M. Briedis Michigan State
University briedis_at_egr.msu.edu
2Outline
- ABET and engineering criteria
- Program objectives
- Program outcomes
- Assessment
- performance criteria
- assessment measures direct and indirect
- rubrics
- Review
3Outline
- ABET and engineering criteria
- Program objectives
- Program outcomes
- Assessment
- performance criteria
- assessment measures direct and indirect
- rubrics
- Review
4Initial Quiz
- What do you know about ABET?
- Describe the engineering criteria?
- According to ABETs definitions, what is the
difference between outcomes and objectives? - Identify four assessment methods. Classify each
as direct or indirect.
5Initial Quiz
- What do you know about ABET?
- Describe the engineering criteria?
- According to ABETs definitions, what is the
difference between outcomes and objectives? - Identify four assessment methods. Classify each
as direct or indirect.
6Initial Quiz
- What do you know about ABET?
- Describe the engineering criteria?
- According to ABETs definitions, what is the
difference between outcomes and objectives? - Identify four assessment methods. Classify each
as direct or indirect.
7Initial Quiz
- What do you know about ABET?
- Describe the engineering criteria?
- According to ABETs definitions, what is the
difference between outcomes and objectives? - Identify four assessment methods. Classify each
as direct or indirect.
8Initial Quiz
- What do you know about ABET?
- Describe the engineering criteria?
- According to ABETs definitions, what is the
difference between outcomes and objectives? - Identify four assessment methods. Classify each
as direct or indirect.
9ABET and Engineering Criteria
- ABET Accreditation Board for Engineering and
Technology - Engineering criteria changed to assessment basis
(TQI) around 2000 - Must prove that students are achieving objectives
and outcomes - measure output, feedback model
- previously, feed forward model
10Feed Forward Model
curriculum
input
output assumed
11Feedback Model
education process
one class
one course
graduate
alumnus
one class
entering college
one course
12The Two Loops of the Engineering Criteria
13ABET and Engineering Criteria
- 8 criteria
- students
- program educational objectives
- program outcomes and assessment
- professional component
- faculty
- facilities
- institutional support and financial resources
- program criteria
14ABET and Engineering Criteria
- Focus of this workshop
- program educational objectives
- definition
- how to establish
- how to assess
- program outcomes and assessment
- definition
- how to establish
- how to assess
15Minute PaperClearest vs. Muddiest Point
- What have you just learned about ABET and
assessment? - What points are the clearest?
- What points are the muddiest?
16Outline
- ABET and engineering criteria
- Program objectives
- Program outcomes
- Assessment
- performance criteria
- assessment measures direct and indirect
- rubrics
- Review
17Program Objectives
- broad statements that describe the career and
professional accomplishments that the program is
preparing the graduates to achieve. - Our graduates will be successful
- must define successful
- Criteria for Accrediting Engineering Programs,
2007-2008 Cycle, ABET, Inc., Baltimore, MD,
http//www.abet.org
18Program Objectives
- must be
- detailed and published
- include constituencies/periodically evaluated
- educational program to achieve outcomes (defined
later) and to prepare graduates for
accomplishments that achieve objectives - ongoing evaluation to determine extent objectives
attained, use results for program improvement - Criteria for Accrediting Engineering Programs,
2007-2008 Cycle, ABET, Inc., Baltimore, MD,
http//www.abet.org
19Exercise
- New program in nanobiomolecular engineering
- Define two program objectives.
20Outline
- ABET and engineering criteria
- Program objectives
- Program outcomes
- Assessment
- performance criteria
- assessment measures direct and indirect
- rubrics
- Review
21Program Outcomes
- statements of what students are expected to
know and be able to do by the time of their
graduation. - Outcomes must foster attainment of the program
objectives - Process to produce outcomes
- Assessment process, with documented results
- demonstrating measurement
- demonstrating degree of achievement
- Evidence results used for program improvement
- Criteria for Accrediting Engineering Programs,
2007-2008 Cycle, ABET, Inc., Baltimore, MD,
http//www.abet.org
22Program Outcomes
- Minimum outcomes a-k plus program criteria
- Opportunity to be unique, i.e., define unique
outcomes, not just repeat a-k - Helpful to map program-defined outcomes into a-k
- Helpful to map outcomes into classes
- Necessary to map outcomes into objectives
23 WVU ABET 1 chemical process 2 communicate 3 computers 4 learn independently and group work 5 lab and data analysis 6 continuing education 7 safety, societal, environmental 8 ethics 9 broad education
a. apply math, sci, engr ?
b. expts - design, conduct, analyze, interpret data ?
c. design system ?
d. multidisciplinary teams ?
e. identify, formulate, solve engineering problems ?
f. professional and ethics ?
g. communication ?
h. broad education - global impact ?
i. life-long learning ? ?
j. contemporary issues ? ?
k. use techniques, skills, modern engineering tools ? ?
24 WVU outcome WVU class 1 chemical process 2 communicate 3 computers 4 learn independently and group work 5 lab and data analysis 6 continuing education 7 safety, societal, environmental 8 ethics 9 broad education
CHE 201 ? ? ? ? ? ?
CHE 202 ? ? ? ? ? ?
CHE 230 ? ? ? ? ?
CHE 310 ? ? ? ? ? ?
CHE 311 ? ? ? ? ? ?
CHE 312 ? ? ? ? ? ?
CHE 315 ? ? ? ? ? ?
CHE 320 ? ? ? ? ? ?
CHE 325 ? ? ? ? ? ?
CHE 450/451 ? ? ? ?
CHE 455/456 ? ? ? ? ? ? ? ?
25Exercise
- Define three unique program outcomes for the
nanobiomolecular engineering program.
26Outline
- ABET and engineering criteria
- Program objectives
- Program outcomes
- Assessment
- performance criteria
- assessment measures direct and indirect
- rubrics
- Review
27Performance Criteria
- What will students do to demonstrate achievement
of outcome - Example An ability to communicate
effectively. (3g in ABET list) - What are attributes of effective communication?
28Performance Criteria for Effective Communication
- When making an oral presentation, students will
- maintain eye contact
-
- A written report will
- follow a prescribed format
- demonstrate proper grammar and punctuation
- adhere to commonly accepted word usage
29Performance Criteria
Learning Outcome
1.Has knowledge of current technological issues
related to XXX engineering and society 2. Is
able to discuss major political and societal
issues and their pertinence to XXX engineering
Educational Objective
Contempo-rary Issues
Professionally responsible
30Exercise
- Define two or three performance criteria for one
of the outcomes previously defined.
31Outline
- ABET and engineering criteria
- Program objectives
- Program outcomes
- Assessment
- performance criteria
- assessment measures direct and indirect
- rubrics
- Review
32Indirect vs. Direct Assessment
- Indirect based mostly on student
self-evaluation - surveys
- interviews
- focus groups
- Direct by faculty or some other means of
evaluation of student performance (advisory
boards evaluate design projects)
33Indirect vs. Direct Assessment
- Indirect
- necessary, but not sufficient provides
uncalibrated snapshot - self-assessment not necessarily reliable
- Terminology may be unfamiliar
- Direct
- necessary for quality assessment plan
- not now for all, but new Program Evaluators are
being trained to look for this feature - relies on faculty experience, expertise, and
judgment
34How do we know if the students have the requisite
outcomes?
- When the students think they do, based on student
survey results - With direct evidence from student work
- Evidence is the key to accreditation
- Faculty evaluation of student work is the key to
providing evidence
35Assessment Measures
- Primary assessment of student outcomes should be
based on student work (direct) - e.g., student portfolios, student projects,
assignments and exams, some employer surveys
where skill is observed - Secondary evidence
- Senior exit surveys, alumni surveys, employer
surveys (qualitative evidence based on opinion),
other - Combination of both methods triangulation
36Indirect Assessment Measures
- Surveys
- Interviews
- Course satisfaction surveys
37Direct Assessment Measures
- End-of-Course Assessments
- Targeted Assignments
- Capstone Experiences
- Capstone Exams
- Portfolios
38End-of-Course Assessments
- Course should have objectives
- perhaps set by department committee
- Related to program outcomes
- Assigned problems (assignments, exams, projects)
each related to course objectives - Evaluation/reflection by instructor
39End-of-Course Assessments
- Advantages
- quick and easy
- assessment can be done in parallel with grading
- Disadvantage
- not comprehensive
- no big picture
- Opinion
- a component of assessment plan
40Targeted Assignments/Problems
- Key assignments that relate to specific program
outcomes - Multiple assignments per outcome recommended
- Integrate through curriculum
- can demonstrate progress toward achievement of
program outcome
41Targeted Assignments/Problems
- Advantages
- quick and easy
- assessment can be done in parallel with grading
- Disadvantages
- none really
- need consistent evaluation method with reliable
inter-rater reliability
42Capstone Experiences
- Can be design, laboratory, research
- All programs have them
- Where students are supposed to demonstrate and
synthesize what learned - Usually includes teamwork, communication
43Capstone Experiences
- Advantages
- already required in program
- assessment can be done in parallel with grading
- Disadvantages
- none really
- need consistent evaluation method with reliable
inter-rater reliability
44Capstone Exams
- FE Exam
- detailed, subject-related results available
- Department-generated
45Capstone Exams
- Advantages
- FE is standardized
- the most direct measure possible
- Disadvantages
- FE may not set bar as high as some want
- students may not take department-generated exam
seriously if results do not impact grades or
graduation
46Portfolios
- Collection of student work demonstrating outcomes
- Must also evaluate the portfolio
- Can also have students reflect on work in
portfolio
47Portfolios
- Advantage
- comprehensive
- Disadvantages
- portfolio evaluation is additional work
- need consistent evaluation method with reliable
inter-rater reliability
48Other Direct Assessment Methods
- Journals
- Concept maps
- Oral presentations with follow-up questions (like
M.S./Ph.D. defense)
49Exercise
- Select an assessment method to be used for direct
assessment of program outcomes previously
defined.
50Outline
- ABET and engineering criteria
- Program objectives
- Program outcomes
- Assessment
- performance criteria
- assessment measures direct and indirect
- rubrics
- Review
51Rubrics
- A set of categories developed from the
performance criteria that define extent to which
the performance criteria are met (progression
towards attaining).
52Developing Rubrics
- Define levels of performance for each performance
criterion - best first to define top and bottom levels of
performance then fill in middle ground - 3-5 levels of performance recommended
53Developing Rubrics
- Standardized method to ensure inter-rater
reliability - Specific definitions of terms like excellent,
understand, not acceptable, exceeds expectations - Initial effort and periodic review required
54Developing Rubrics
- If have five levels of performance, can make each
level a grade - if have three levels, can make each level A/C/F,
and interpolate for B/D - Use rubric for assessment and grading
- Advantage Students and faculty have clearly
defined criteria for grading of what appears to
be subjective (lab and design reports, oral
reports, etc.)
55Application of math science
56Problem Assessment Form
- Connects physical model with math model
- Able to write unsteady state mass balance
- Makes appropriate substitution for flow terms
- Makes appropriate simplification for flow terms
- Converts differential equation into Laplace form
correctly - Understands idea for output/input form of
transfer function - Correctly combines Laplace transforms in series
- Correct answer
57Rubrics Assessment ScaleApplication of math
science (5high)
- Level 5
- Formulates models correctly
- Applies calculus or linear algebra to solve
problems - Correct calculations
- Correct statistical analysis
- . . .
- Level 3
- Formulates models with some trouble
- Some understanding of calc/linear algebra
applications
- Minor calc errors
- Minor statistical errors
- . . .
- Level 1
- Not able to model
- Cannot apply calc/linear algebra
- Incorrect calculations
- Does not apply statistics
- . . .
58Performance Criteria
Learning Outcome
1.Has knowledge of current technological issues
related to XXX engineering and society 2. Is
able to discuss major political and societal
issues and their pertinence to XXX engineering
Educational Objective
Contempo-rary Issues
Professionally responsible
59Outcome A knowledge of contemporary issues
Performance Criteria Scoring Rubrics
Rating Scale Element Needs Improvement 1 2 Met Expectations 3 4 Exceeded Expectations 5 N/A
Has knowledge of current technological issues related to chemical engineering and society (global warming, resource depletion, waste proliferation, etc.) Has minimal knowledge of technological issues and their relevance to chemical engineering has weak connection between the issue and scientific principles for analysis and has trouble developing solutions. Has reasonable knowledge of technological issues some may not be directly relevant to chemical engineering can apply scientific principles to analysis and suggest solutions when guided. Has thorough knowledge of current technology issues related to chemical engineering and is able to analyze them and propose solutions using scientific principles
Has knowledge of and is able to discuss major societal and political issues and their pertinence to chemical engineering Has minimal knowledge of societal and political issues if given an issue, does not see its connection to engineering without instruction is minimally effective in discussion and presentation of such issues Has reasonable knowledge of societal and political issues recognizes some connection to chemical engineering, but misses the details is somewhat effective in discussing and presenting such issues when prompted Has thorough knowledge of societal and political issues related to chemical engineering recognizes the big picture and the details presents strong discussion of such issues
60(No Transcript)
61(No Transcript)
62(No Transcript)
63Rubrics
- Additional rubrics at
- http//www.che.cemr.wvu.edu/ugrad/outcomes/rubric
s/index.php - Instructions on rubrics
- http//webquest.sdsu.edu/rubrics/weblessons.htm
64Exercise
- For the two or three performance criteria for one
of the outcomes previously defined, and assuming
the assessment method previously defined, begin
to develop an evaluation rubric.
65Outline
- ABET and engineering criteria
- Program objectives
- Program outcomes
- Assessment
- performance criteria
- assessment measures direct and indirect
- rubrics
- Review
66Recommendation
- Strongly recommend adopting direct assessment
measures into assessment activities - Will be expected in the near future!
67Exercise
- What are the two most important things that you
learned in this workshop? - What is still unclear to you about program
assessment?
68Questions