Helping students learn through the assessment of student outcomes: A faculty responsibility

1
Helping students learn through the assessment of
student outcomes  Afaculty responsibility

• Edwin C Jones, Jr
• University Professor Emeritus, Iowa State
  University
• Adjunct Professor, University of St Thomas

2
Quote from Derek Bok

• I would have liked more discussion among the
  faculty as a whole in really discussing methods
  of teaching. But they don't engage in what
  well-run organizations in every other part of
  society do, that is, a continual process of
  self-improvement--identifying problems and
  experimenting with ways to solve those problems.
  The professors,, spend almost all their time
  figuring out what courses should be required and
  in what sequence. They spend almost no time
  discussing how the courses should be taught, even
  though most people say the lasting influence of
  colleges is focused much more on how classes
  are taught.
• U S News World Report, 3/6/06.

3
Early work on instructional development in
engineering

• 1970swe considered Mager
• We wrote instructional objectives at all 6 levels
• We developed personalized self instruction
• We worked on computer-based education
• We considered the role of the laboratory
• Some publishers worked with authors to
  incorporate the ideas
• But, was it enough?

4
Is Engineering Different?

• Engineering Education has been well studied
• Wickenden Report
• Grinter Report
• Goals Report
• 1990sThree Studies
• National Academy of Engineering
• Deans Council of ASEE
• National Science Foundation

5
Observations and Activities

• The early studies were primarily content
• 1990s studies considered process
• In addition, many, especially larger,
  institutions were questioning ABET
• ABET responded to the institutional concerns and
  to the studies
• Industry/Government concerns also considered

6
What was concluded?

• Major reform of engineering education needed
• The field is changing so rapidly that emphasis on
  content was inappropriate
• Engineering educators needed to develop BS
  graduates who can be productive for a career
• Graduate study provides this for some, not all
• ABET had a major role to play

7
ABETs Response

• EC2000 Criteria
• Major Components
• Students
• Program Educational Objectives
• Program Outcomes and Assessment
• Professional Component
• Faculty
• Facilities
• Institutional Support and Resources

8
Comments

• ABETs use of terms not used in all
  literaturegoals, objectives, mission, outcomes,
  etc.
• Major Criteria Change is addition of 3, Program
  Outcomes and Assessment
• Educational Objectives had long been there, but
  had generally been downplayed
• Early on, ABET did not define objectives and
  outcomes. Has led to much confusion. Definitions
  added about 2003.

9
Definitions

• Program Educational Objectives
• Although institutions may use different
  terminology, for purposes of Criterion 2, program
  educational objectives are broad statements that
  describe the career and professional
  accomplishments that the program is preparing
  graduates to achieve.

10
Definitions (contd)

• Program Outcomes (and Assessment)
• Although institutions may use different
  terminology, for purposes of Criterion 3, program
  outcomes are statements that describe what
  students are expected to know and be able to do
  by the time of graduation. These relate to the
  skills, knowledge, and behaviors that student
  acquire in their matriculation through the
  program.

11
Comments on Objectives

• They are marketing tools
• Potential students
• Employers
• They define the program
• What it is
• What it is not--implicitly
• Assessment is straight forward
• Find out what the graduates are doing
• Find out what suggestions they have for program
  improvement

12
More Comments on Objectives

• Criterion 2-a. Must be published
• Criterion 2-b. Developed with constituents
• Criterion 2-c. Map curriculum to objectives
• Criterion 2-d. Show achievement of
  objectivesfaculty responsibility. Show also
  improvements.

13
Program Outcomes (ABET/EAC)

• Each program must formulate program outcomes that
  foster attainment of the program objectives
  articulated in satisfaction of Criterion 2 of
  these criteria. There must be processes to
  produce these outcomes and an assessment process,
  with documented results, that demonstrates that
  these program outcomes are being measured and
  indicates the degree to which the outcomes are
  achieved. There must be evidence that the results
  of this assessment process are applied to the
  further development of the program.

14
Comment

• Attaining outcomes does not guarantee achievement
  of objectives
• Not all graduates need to have achieved all
  objectives
• Significant changes are possible and have been
  observed

15
What are the desired outcomes?

• a) an ability to apply knowledge of mathematics,
  science, and engineering
• (b) an ability to design and conduct experiments,
  as well as to analyze and interpret data
• (c) an ability to design a system, component, or
  process to meet desired needs within realistic
  constraints such as economic, environmental,
  social, political, ethical, health and safety,
  manufacturability, and sustainability

16
Outcomes--continued

• (d) an ability to function on multi-disciplinary
  teams
• (e) an ability to identify, formulate, and solve
  engineering problems
• (f) an understanding of professional and ethical
  responsibility
• (g) an ability to communicate effectively

17
Outcomes--continued

• (h) the broad education necessary to understand
  the impact of engineering solutions in a global,
  economic, environmental, and societal context
• (i) a recognition of the need for, and an ability
  to engage in life-long learning
• (j) a knowledge of contemporary issues
• (k) an ability to use the techniques, skills,
  and modern engineering tools necessary for
  engineering practice.

18
Additional Proviso

• In addition, an engineering program must
  demonstrate that its students attain any
  additional outcomes articulated by the program to
  foster achievement of its education objectives.

19
Where did these a-k come from?

• Development
• Other professions developing similar outcomes
  (possibly with other names)
• Not limited to professional education

20
Principal types of assessment

• Direct
• Student work evaluated by professionals
  considering the degree to which the work shows
  the designated ability
• Homework, tests, laboratory work, design
  projects, papers, portfolios
• Assessment is of designated ability. Goes beyond
  gradesdeeper, focused.

21
Principal types of assessment

• Indirect
• Surveys
• Other self assessment tools
• A common opinion of assessment consultants3
  measurements needed
• One may be indirect
• Direct measurements should be independent
• There are new companies doing assessment
• Remains a faculty responsibility

22
How can we assume our responsibility?

• Design courses with abilities in mind
• e.g., EE 333 addresses 3-b, experiment design and
  data interpretation
• It uses content to do thiscontent relevant to
  the program
• Made clear to students and faculty
• Not too much responsibility in each
  coursesuggest a H/M/L classification

23
How can we assume our responsibility? Course Level

• Instructional activity reviewed by, say, 3
  qualified personsincluding instructor
• When appropriate, usually soon after the end of
  the course, the 3 persons review student work
• Evaluate work on a Likert scale
• Compile and interpret data
• Suggest improvements
• Repeatdid improvements work?

24
How can we assume our responsibility? Course Level

• Not all changes we make are effective. This is OK
• Some might even make it worse. Again, this is OK
• Some changes do work, happily
• Continue assessment on a regular basis

25
How can we assume our responsibility? Program
Level

• Design Projectcommon to all engineering
  programs. Other programs usually have some sort
  of integrating project.
• Be sure project requirements exercise all of the
  abilities
• Instead of, or in addition to grades, ask project
  evaluators to evaluate attainment of the outcomes
  in the projectLikert scale

26
How can we assume our responsibility? Program
Level

• Portfolios
• Common in some disciplines
• New venture in engineering
• Start as first year students
• Organize by content and abilities
• Ask evaluators to consider portfolios with
  respect to a-kagain, Likert scale
• Can be done on the web

27
How can we assume our responsibility? What do we
do with the data?

• Faculty responsible for setting targets and for
  evaluating the data
• Some results will be outstanding
• Some results will be OK
• There will be room for improvementthere always
  is
• Cautiondo not overdo it!

28
Interesting Comparison with the Deming Cycle for
Continuous Improvement
PLAN
ACT
DO
CHECK
29
Implementation Matrix

• Program Educational Objectives have been
  established and maintained
• Constituents are involved in helping set program
  objectives and in evaluating the level to which
  they are being achieved
• The required Processes are operational
• Outcomes Assessment is being practiced
• Results of outcomes and the various processes are
  being used to improve programs and assure
  objectives are being achieved
• An overall System is in place to meet the
  accreditation requirement

30
Implementation Matrix

• HorizontalObjectives and Outcomes
• VerticalLevel of attainment, 1-5
• A good accreditation suggest 3 or higher

31
Part of the Matrix
32
Part of the Matrix
33
(No Transcript)
34
Summary

• Assessment is a developing faculty responsibility
• Good techniques existwill get better
• Considerable startup required
• Eventually, readily accomplished while the
  process improves
• Results are worth the effort