Understanding Student Learning Outcomes and Assessment

1
Understanding Student Learning Outcomes and
Assessment

• Bob Barr
• Andrew LaManque
• Rob Johnstone

Breakout Session, District Opening Day, September
22, 2005
Adapted from material prepared by Fred Trapp,
Long Beach CC, the RP Group, and James O.Nichols,
University of Mississippi
2
Session Outcomes

• Describe a multi-step process for conducting
  learning outcomes assessment and name the purpose
  of each step.
• Distinguish among levels of assessment work.

3
Session Outcomes

• Critique a student learning outcome statement.
• Describe two means of assessment and match them
  to an outcome statement.

4
What Is Student Learning Outcomes Assessment?

• An ongoing process of systematically, gathering,
  analyzing, and interpreting evidence aimed at
  understanding and improving student learning.
• Involves making learning expectations explicit
  and public.
• Involves setting appropriate criteria and high
  standards for learning quality.

5
Why Do Assessment?

• The Reason Improve student learning and
  institutional support services.
• The Effort Continuous reflection on the
  teaching-learning process
• An Occasion to Our Display Work Regional or
  program accreditation.

6
Accrediting Commission for Community Junior
Colleges (ACCJC)

• Standard II Student Learning Programs and
  Services
• The institution engages in ongoing, systematic
  evaluation and integrated planning to assure
  currency and measure achievement of its stated
  student learning outcomes for courses, degrees,
  certificates, and general education and
  vocational programs.

7
The Assessment Process
8
Assessment Process and Key Questions

• Link of GE and program to mission institutional
  outcomes course to program outcomes.
• Where does the ______ program fit into the
  mission and learning outcomes of the college?
• Specify intended learning outcomes
• What learning comes of completing the course,
  program, or degree?

9
Assessment Process and Key Questions

• Identify means of assessment and criteria for
  success
• How shall we discover what students have learned?
  How good is good enough?
• Assemble the results and make meaning from them
• How well did they do relative to expectations?

10
Assessment Process and Key Questions

• Using the results
• What shall we do with these results?
• Document, document, document
• Suggestion
• Department/program plan
• Program review

11
Levels of SLO Assessment Work

• College-wide
• General education or core competencies
• Instructional Program or pre-major or Student
  Service Program
• Course or sections of a commonly taught course

12
Step 1. Linking to Mission -Expanded Statement
of Purpose

• Mission of the college
• Broad statements of defining purposes and
  outcomes
• GE, program, course purpose statements
• Links the GE package and program to the mission
  of the college
• Links the course to GE package or program goals
• Cornerstone of self-assessment ( self-study)
• How well are you accomplishing your purposes?

13
Step 1. Linking to Mission -Linking the Three
Levels

• College or GE Outcome
• Students will be able to speak and write
  effectively.
• Academic Program Outcome (Psychology)
• Students can speak and write effectively using
  the principles and concepts of the discipline.
• Course Outcome (Psych Measurement)
• Students can prepare a written summary and
  interpretation of standardized test results.

14
Two Types of Educational Outcomes(Institutional/P
rogram Performance)

• Traditional Achievement Outcomes
• Success rates
• Retention or persistence rates
• Number and rates of graduates or transfers
• Student Learning Outcomes
• What faculty intend students to know (cognitive),
  think or believe (affective) or be able to do
  (behavioral) when they have completed
• Degree (general education)
• Field of concentration (program)
• Course

15
Step 2. Specify Intended Learning Outcomes

• Ideally, the curriculum and courses are designed
  by working backwards from the ultimate outcomes.
• What roles will program graduates hold and what
  must they be able to do out there in the real
  world or in upper division work?
• How will the outcomes be demonstrated in here?
• What skills and concepts need to be learned?

16
Step 2. Specify Intended Learning Outcomes

• Helpful hints
• Use concrete, action verbs
• See Blooms Taxonomy of Educational Objectives
• Anticipate the deliverable
• What is it that students must do to convince you
  that they are where you want them to be with
  respect to learning?
• Enabling vs. terminal
• Target key concepts or change points in the
  program
• Major aspects of learning central to the
  discipline

17
Example Specify Intended GE Learning Outcomes

• Be exposed to good writing examples. (in need of
  repair)
• Upon completion of a degree, the student will be
  able to
• Write effectively
• Compose a range of written products designed for
  different audiences and purposes.

18
Example Specify Intended Psychology Program
Learning Outcomes

• Hear and read about the scientific method of
  inquiry as used in the social sciences. (in need
  of repair)
• Upon completion of the program, the student will
  be able to
• Use critical thinking, skeptical inquiry, and,
  when possible, the scientific approach to solve
  problems related to behavior and mental processes.

19
Example Auto Tech Program Intended Student
Learning Outcomes

Expanded Statement of Institutional
Purpose Identity Statement Your Community
College is an open-admission, community-based,
comprehensive college. Mission
Statement Serve persons of all ages in
preparing for entry and careers in a variety of
fields, such as automotive technology.
Program Intended Educational Outcomes 1.
Graduates of the Automotive Technology Program
will be successfully employed in the field.
(achievement outcome, not a SLO) 2. Graduates
of the Automotive Technology Program will be
technically proficient. 3. Employers of
Automotive Technology Program graduates will rate
them competent based on the education received in
the program and would employ others.
20
Step 2. Specify Intended Student Services Program
Learning Outcomes

• Know how to be a more prepared student. (in need
  of repair)
• Upon completion of the EOPS summer transition
  program, the student will be able to
• Develop and use a personal schedule that
  incorporates regular study time, regular study
  location and study groups to match their class
  schedule.

21
Characteristics of Intended Student Learning
Outcomes

• Student-focused, not teacher focused
• Focus on learning resulting from an activity, not
  the activity itself
• Focus on important, non-trivial aspects of
  learning
• Focus on skills abilities central to the
  discipline, based on professional standards of
  excellence

22
Characteristics of Intended Student Learning
Outcomes

• General enough to capture important learning but
  clear and specific enough to be measurable
• Focus on aspects of learning that will develop
  and endure but that can be assessed in some form
  now

23
Aligning Outcomes,Teaching-Learning and
Assessment
Outcome statements Methods of teaching/ learning Assumptions about teaching/ learning Methods of assessing Assumptions about methods
What do you expect students to know, understand, be able to do as a result of your teaching? What methods develop/ foster your primary outcomes? What assumptions underlie your methods? What methods do you believe provide evidence of desired student outcomes? What assumptions underlie your belief that each method will elicit students demonstration of your desired outcomes?
24
Step 3. Some Means of Assessment

• Embedded Narrative Assignments
• Case study
• Portfolio (paper or electronic)
• Capstone project
• Senior exit essay
• Performances
• Critical situation scenario
• Debate
• Demonstration
• Oral presentation
• Online monitoring
• Tests
• Standardized test
• Homegrown test
• In class objective test
• In class essay

• Student Centered
• Exit interview
• Focus group
• Self-rating of progress
• Archival
• Library use, on-line activity
• Transcript analysis
• Transfer records
• Stakeholders Perspective
• Alumni, employee, employer surveys
• Experience of faculty who inherit our students

Weve always done this, but now systematically to
improve student learning rather than just to
assess the student and assign a grade.
25
Step 3. Means of Assessment

• Levels of assessment
• College (General Education, Core Abilities)
• Program (Deg, cert, or course sequence)
• Course
• Types of evidence
• Direct
• Indirect and supportive

26
Step 3. Means of Assessment

• Direct evidence of student learning outcomes
  (think in the aggregate)
• Passage of license, certification or subject area
  exams
• Results of general education exams
• Capstone projects or performances
• Employer or internship supervisor ratings of
  student performance
• Class assignments, projects, etc.
• Departmental exam results
• But, NOT grades.

27
Step 3. Means of Assessment

• Indirect evidence of student learning outcomes
  (think in the aggregate)
• Student self-reflective reports
• Surveys of perception
• Alumni, graduating or current students
• Focus group interviews

28
Step 3. Means of Assessment

• Targets for Assessment
• Cognitive outcomes (knowledge, thinking
  disciplines)
• Attitudinal outcomes (values, beliefs)
• Behavioral/performance outcomes (use of knowledge
  or values)
• or Knowledge, Skills, and Abilities (KSAs)

29
Grading is not Assessment

• Evaluation of individual students assessment
• Focus is individual not groups of students
• A summative, not formative act
• Objectivity of single evaluator vs. group
• Generally not accepted as direct evidence
• Uses of the grading process
• Agreed upon course exam or part of exam
• Row and column model for assignments

30
Grading and SLO Assessment(Embedded Assignment)
Total down the column for individual grading.
Analyze across the row for assessment of intended
outcomes from the group.
James Nichols
31
Step 3. Means of Assessment - Course Exams

• Agreed upon course exam questions
• Monitor student responses to identify topics of
  weakness
• Item analysis to discern what groups are having
  difficulty with selected items
• Revisit teaching approach to the topic

32
Step 3. Means of Assessment - Embedded
Assignments

• Embedded narrative class assignments
• Evaluated for the grade mark
• Later a sample is re-inspected to
• Assess program accomplishment against expected
  outcomes
• Explore the learning process
• Agreed upon performance criteria
• The fruits of campus dialogue
• Expert judgment of the faculty
• Inter rater reliability

33
Step 3. Means of Assessment -Rubrics

• Key Questions
• What essential elements must be present in
  student work to ensure high quality?
• What does student work look like at each level of
  mastery?

34
Step 3. Means of Assessment -Elements of A Useful
Rubric

• Dimensions/criteria of rating (the rows)
• Levels of mastery (the columns) e.g.,
• Beginner
• Developed
• Accomplished
• Commentary describing features of work at each
  level of mastery (the cells)
• Build from strengths and weaknesses teachers see
  in student work over the years

35
Step 3. Means of Assessment - Portfolios

• A collection of student work
• Selective or all inclusive
• Displays what students can do
• Best if student is required to reflect upon and
  evaluate their own work and learning
• Caution in program use
• Require the same work product from all students
  at pre-determined points in time.

36
Step 3. Means of Assessment

• Attitudinal Learning
• Surveys of students, graduates, alumni
• Affirmation of accomplishments
• Indirect measure
• Exam or task to explore attitudes and apply values

37
Step 3. Means of Assessment

• Behavioral and Performance Student Learning
  Outcomes
• Specific motor skills
• Computer applications use
• Writing talent
• Computational skills
• Speaking ability

38
Step 3. Means of Assessment

• Behavioral and Performance
• Case studies
• Internships
• Simulations
• Projects
• Classroom work in a capstone course

39
Step 3. Means of Assessment - Final Performances

• Most frequently used means
• Portfolio review
• Public performance
• Juried competition
• Drawn from embedded assignments that require
  synthesis and application of learning

40
Step 3. Criteria for Success

• Identify the criteria before assessment is done
• How well ought the completers (on average)
  perform?
• Role of dialogue, professional judgment and
  collective responsibility

41
Step 3. Criteria for Success

• Primary student performance areas
• Minimum overall score, rating, response expected
  if program is functioning at an acceptable level
• Sub areas of student performance
• More detailed minimum sub-scale or performance
  which faculty need to review to ascertain likely
  cause and make improvement

42
Example Criteria for Success - Automotive
Technology Program

Means of Program Assessment and Criteria for
Success 2a. At the close of their final term,
90 of the graduates will be able to identify
and correct within a given period of time all of
the mechanical problems in five test cars that
have been "prepared" for the students by
Automotive Technology Program faculty. No single
automotive malfunction will fail to be identified
and corrected by more than 20 of students. 2b.
80 of Automotive Technology Program graduates
will pass the National Automotive Test. On no
subscale will students average missing 30 or
more of the items. 3. 80 of the respondents to
an Employer Survey conducted every 3 years by the
college will respond that they find graduates
competent and would employ others.
Program Intended Educational Outcomes 2.
Graduates of the Automotive Technology Program
will be technically proficient. 3.
Employers of the Automotive Technology Program
graduates will be rate them competent based on
the education received in the program.
43
Step 4. Assessing, Assembling Results and Making
Meaning of Them

• Assessing and Assembling Results
• Logistics of doing assessment
• Recording results
• Making meaning
• The creative tension between ought and is

44
Example Recording ResultsGraduates of the
Automotive Technology Program will be
technically proficient.

Summary of Data Collected 2a. 79 overall
success rate. Electrical system malfunction
undetected by 34 of students. 2b. Pass
rate on National Automotive Test was 83
however, on hydraulic theory subscale students
missed an average of 34 of questions.
Means of Program Assessment and Criteria for
Success 2a. At the close of their final term,
90 of the graduates will be able to
identify and correct within a given period of
time all of the mechanical problems in five test
cars that have been "prepared" for the students
by Automotive Technology Program faculty. No
single automotive malfunction will fail to be
identified and corrected by more than 20 of
students. 2b. 80 of Automotive Technology
Program graduates will pass the National
Automotive Test. On no subscale will students
average missing 30 or more of the items.
45
Step 5. Documenting and Using the Results -
Closing the Loop

• Fitting the response to the results
• Faculty adjust what we teach
• Faculty adjust how we teach
• Faculty adjust the means of assessment
• Faculty adjust where we put the bar of success
• Document

46
Example Auto Tech Program(Using Results
Documenting Those Actions)

Use of Results 2a. Expanded electrical
trouble-shooting component of AT 202 to include
automotive electrical systems. 2b. Modified
means of teaching hydraulic theory during AT 102
(Basic Auto Systems). 3. Added body shop
representative to Advisory Committee and are
reviewing curriculum to determine if separate
program is needed.
Summary of Data Collected 2a. 79 overall
success rate. Electrical system malfunction
undetected by 34 of students. 2b. Pass rate
on National Automotive Test was 83 however, on
hydraulic theory subscale students missed an
average of 34 of questions. 3. 90 reported
willingness to employ graduates, but only 50 of
body shops.
47
The Bottom Line, A Call To Learning

• A college is an organization composed of
  learners
• (End)

48
Automotive Technology Program
Program Intended Educational Outcomes 1.
Graduates of the Automotive Technology
Program will be successfully employed in the
field. (achievement, not Student Learning
Outcome) 2. Graduates of the Automotive
Technology Program will be technically
proficient. 3. Employers of the Automotive
Technology Program graduates will rate them
competent based on the education received in the
program.
Use of Results 1a. Revised criteria for
success to 70. 1b. No action necessary at
this time, however, will continue to
monitor. 2a. Expanded electrical
trouble-shooting component of AT 202 to include
automotive electrical systems. 2b. Modified
means of teaching hydraulic theory during AT 102
(Basic Auto Systems). 3. Added body shop
representative to Advisory Committee and are
reviewing curriculum to determine if separate
program is needed.
Means of Program Assessment and Criteria for
Success 1a. 50 of the responding graduates of
the Automotive Technology Program will report
employment in the field on the Graduating Student
Survey administered at the time of program
completion. 1b. 80 of the graduates of the
Automotive Technology Program will report
employment in the field on the recent Alumni
Survey administered one year after
graduation. 2a. At the close of their final
term, 90 of the graduates will be able to
identify and correct within a given period of
time all of the mechanical problems in five test
cars that have been "prepared" for the students
by Automotive Technology Program faculty. No
single automotive malfunction will fail to be
identified and corrected by more than 20 of
students. 2b. 80 of Automotive Technology
Program graduates will pass the National
Automotive Test. On no subscale will students
average missing 30 or more of the items. 3.
80 of the respondents to an Employer Survey
conducted every 3 years by the college will
respond that they find the graduates competent
and would employ others .
Summary of Data Collected 1a. 73 reported
employment. 1b. 81 reported employment one
year after graduation. 2a. 79 overall
success rate. Electrical system malfunction
undetected by 34 of students. 2b. Pass rate on
National Automotive Test was 83 however, on
hydraulic theory subscale students missed an
average of 34 of questions. 3. 90 reported
willingness to employ graduates, but only 50 of
body shops.

Expanded Statement of Institutional
Purpose Mission Statement Your Community
College is an open-admission, community-based,
comprehensive college. Goal Statement Serve
persons of all ages in preparing for job entry
and careers in automotive technology.
49
Transfer Program, Your CC(CC with no majors and
few transfers)

Use of Results 1. Expansion of
microcomputer use in introductory accounting
problems at Your CC is underway. 2. Math 107
(college algebra) is being strengthened to better
relate with calculus at four-year
institutions.. 3. No change necessary.
Program Intended Educational Outcomes 1.
Students transferring will find courses taken
fully accepted as prerequisites for junior and
senior level courses at four-year colleges. (not
a SLO) 2. After one year of adjustment to the
four-year college, the grades of students
transferring will be similar to those of students
who initially enrolled at four-year colleges. 3.
Graduates transferring to a four-year college as
a full-time student will complete degrees at
almost the same rate as those students originally
enrolling at the four-year college. (not a SLO)
Means of Program Assessment and Criteria for
Success 1. Each year one of the colleges six
academic departments will contact their
counterparts at the three four-year colleges
where most students transfer and all courses
designed to support the transfer will be found
fully accepted as prerequisites by the four-year
institutions. 2. Analysis of data concerning
the grades of students transferring to four-year
colleges will indicate that the differences
between the average of transfer students GPAs
and that of native students is statistically
insignificant one year after enrollment at the
four-year college. 3. Analysis of data
received from each of Your CC three primary
transfer student destinations will indicate that
the difference in the average number of semesters
to the BA degree completion of full-time transfer
students from Your CC and students originally
enrolling in the four-year college is
statistically insignificant.
Summary of Data Collected 1. Problem noted in
introductory accounting where microcomputer
technology is now used at two primary transfer
institutions. 2. Overall GPA of transfers found
comparable to native students, but significantly
less in math classes. 3. Degree completing
time of Your CC transfer students found to be
virtually identical to native students.
Expanded Statement of Institutional
Purpose Mission Statement an
open-admission, community-based, comprehensive
college designed to provide inexpensive, quality
educational opportunities (transfer,
career/technical and continuing
education) Goal Statement Serve traditional
students seeking the first two years of
instruction leading to a bachelors degree