Assessment

1
Assessment

• Is like a dancers mirror.
• It improves ones ability to see and improve
  ones performance.
• Alexander Astin
• 1993

2

• ASSESSMENT . . .
• a rich conversation
• about student learning
• informed by data.
• -- Ted Marchese --
• AAHE

3
Assessment of Individual Student Development

• Assessment of basic skills for use in advising
• Placement
• Counseling
• Periodic review of performance with detailed
  feedback
• End-of-program certification of competence
• Licensing exams
• External examiners
• CLAST

4
Key Results of Individual Assessment

• Faculty can assign grades
• Students learn their own strengths and weaknesses
• Students become self-assessors

5
A Second Look

• Across students
• Across sections
• Across courses

6

• Where is learning satisfactory?
• What needs to be retaught?
• Which approaches produce the most learning for
  which students?

7
Group Assessment Activities

• Classroom assignments, tests, projects
• Questionnaires for students, graduates, employers
• Interviews, focus groups
• Program completion and placement
• Awards/recognition for graduates
• Monitoring of success in graduate school
• Monitoring of success on the job

8
Use of Results of Group Assessment

• Program improvement
• Institutional and / or state peer review
• Regional and / or national accreditation

9
Outcomes Assessment

• The process of providing credible
• evidence of the processes
• and outcomes of higher education
• undertaken for the purpose of
• improving programs and services
• within the institution.
• Banta, T. W.

10
Some Purposes of Assessment

• 1. Students learn content
• 2. Students assess own strengths
• 3. Faculty improve instruction
• 4. Institutions improve programs/services
• 5. Institutions demonstrate accountability

11
Assessment of Learning

• 1. Faculty agreement on outcomes and
  performance standards
• 2. Evidence in syllabi and assignments that
  outcomes are taught
• 3. Collective faculty review of group
  performance
• 4. Use of findings to improve instruction and
  curriculum

12
Most Faculty Are Not Trained as Teachers

• Faculty Development
• Can Help Instructors
• Write clear objectives for student learning in
  courses and curricula
• Individualize instruction using a variety of
  methods and materials
• Ask questions that make students active learners
• Develop assessment tools that test higher order
  intellectual skills

13
Taxonomy of Educational Objectives(Bloom and
Others, 1956)

• Cognitive domain categories
• Knowledge
• Comprehension
• Application
• Analysis
• Synthesis
• Evaluation

• Sample verbs for outcomes
• Identifies, defines, describes
• Explains, summarizes, classifies
• Demonstrates, computes, solves
• Differentiates, diagrams, estimates
• Creates, formulates, revises
• Criticizes, compares, concludes

14
Organizing for Assessment
15
Planning for Learning and Assessment
16
Faculty and Staff Development

• Focus faculty and student affairs professionals
  on improving learning in and outside class
• Attend conferences together
• Study literature on student learning
• Provide workshops on teaching and learning
• Provide resources (e.g., grants, summer salary,
  release time)

17

• Direct Measures of Learning
• Assignments, exams, projects, papers
• Indirect Measures
• Questionnaires, inventories, interviews
• - Did the course cover these objectives?
• - How much did your knowledge increase?
• - Did the teaching method(s) help you learn?
• - Did the assignments help you learn?

18
Select or DesignAssessment Methods

• 1. Match with goals
• 2. Use multiple methods
• 3. Combine direct and indirect measures
• 4. Combine qualitative and quantitative
  measures
• 5. Consider pre - post design to assess gains
• 6. Use built-in points of contact with students

19
Outcomes Assessment Requires Collaboration

• In setting expected program outcomes
• In developing sequence of learning experiences
  (curriculum)
• In choosing measures
• In interpreting assessment findings
• In making responsive improvements

20
Barriers to Collaboration in the Academy

• Graduate schools prepare specialists
• Departments hire specialists
• Much of our scholarship is conducted alone
• Promotion and tenure favor individual
  achievements -- interdisciplinary work is harder
  to evaluate

21
Change the Focus

• from
• TEACHING
• to
• LEARNING
• Barr Tagg
• 1995

22
Functions of the Work of the Professoriate

• Scholarship of Discovery
• Scholarship of Integration
• Scholarship of Application
• Scholarship of Teaching
• Ernest Boyer - 1990

23
Good assessment is good research . . .

• An important question
• An approach to answer the question
• Data collection
• Analysis
• Report
• -Gary R. Pike (2000)

24
Campus Interest in Assessment

• WHAT WORKS in.
• increasing student retention?
• general education?
• use of technology in instruction?
• curriculum in the major?

25
Measures of Critical Thinking

• 1. Academic Profile (ETS)
• 2. Collegiate Assessment of Academic Proficiency
  (ACT)
• 3. California CT Dispositions Inventory
  (Faciones)
• 4. California CT Skills Test (P. Facione)
• 5. Cornell CT Test (Ennis Millman)
• 6. Tasks in CT (ETS)
• 7. Reflective Judgment Inventory (King
  Kitchener)
• 8. Watson Glaser CT Appraisal (Psych Corp)

26
Collegiate Assessment of Academic
Proficiency(CAAP from ACT)

• Individual Modules
• Reading
• Writing (MC and Essay)
• Mathematics
• Science Reasoning
• Critical Thinking

27
COLLEGE BASE
Competences
Interpretive Reasoning
Strategic Reasoning
Adaptive Reasoning
English
S U B J E C T S
English Subject Clusters
Interpretive Reasoning
Math
Writing Skills
Science
Reading/Literature Skills Reading
Critically Reading Analytically Understanding
Literature
Social Studies
28
Are Standardized Tests the Answer?

• Not available in many fields
• Do not measure all that is taught
• Usually assess knowledge, not performance
• May be standardized on unrepresentative norm
  group
• Provide few, if any, subscores
• Do not indicate why scores are low

29
Start with Measures You Have

• Assignments in courses
• Course exams
• Work performance
• Records of progress through the curriculum

30
Primary Trait Scoring

• Assigns scores to attributes (traits) of a task
• STEPS
• Identify traits necessary for success in
  assignment
• Compose scale or rubric giving clear definition
  to each point
• Grade using the rubric

31
Can Develop a Research Paper
   

• Narrows and defines topic
• Produces bibliography
• Develops outline
• Produces first draft
• Produces final draft
• Presents oral defense

 
32
Bibliography

• Outstanding References current, appropriately
  cited, representative, relevant
• Acceptable References mostly current, few
  citation errors, coverage adequate, mostly
  relevant
• Unacceptable No references or containing many
  errors in citation format, inadequate coverage or
  irrelevant

33
Mapping Course Outcomes to Program Outcomes
34
Sophomore Competence in Mathematics(Multiple
choice responses supporting work)

• Score
• 3
• 2
• 1
• 0

• Criterion
• Clear conceptual understanding, consistent
  notation, logical formulation, complete solution
• Adequate understanding, careless errors, some
  logic missing, incomplete solution
• Inadequate understanding, procedural errors,
  logical steps missing, poor or no response
• Problem not attempted or conceptual understanding
  totally lacking
• Ball State University

35
Journal Evaluation

• Entries accurately and vividly record objective
  observations of site experiences (events, people,
  actions, setting)
• Entries convincingly record subjective responses
  to site experience (thoughts, emotions, values,
  judgments)
• Entries effectively analyze/ evaluate your
  experiences (find insights, patterns, meaning,
  causes, effects)

36
Assessment in Sociology and Anthropology

• Focus groups of graduating students
• Given a scenario appropriate to the discipline, a
  faculty facilitator asks questions related to
  outcomes faculty have identified in 3 areas
  concepts, theory, methods.
• 2 faculty observers use 0-3 scale to rate each
  student on each question
• GROUP scores are discussed by all faculty
• Murphy Goreham
• North Dakota State University

37
Methods of Assessment

• Paper and pencil tests
• Individual or group projects
• Portfolios
• Observation of practice
• Observation of simulated practice
• Analysis of case studies
• Attitude or belief inventories
• Interviews and focus groups
• Surveys

38
Assessing Student GrowthThe Portfolio - Some
Examples of Content

• Course assignments
• Research papers
• Materials from group projects
• Artistic productions
• Self-reflective essays (self-assessment)
• Correspondence
• Taped presentations

39
Student Electronic Portfolio

• Students take responsibility for demonstrating
  core skills
• Unique individual skills and achievements can be
  emphasized
• Multi-media opportunities extend possibilities
• Metacognitive thinking is enhanced through
  reflection on contents
• - Sharon J. Hamilton
• IUPUI

40
PRINCIPLES OF UNDERGRADUATE LEARNING (PULs)

• Core communication and quantitative skills
• Critical thinking
• Integration and application of knowledge
• Intellectual depth, breadth, and adaptiveness
• Understanding society and culture
• Values and ethics
• Approved by IUPUI Faculty Council
  May 1998

41
ePort Goals

• Promote clearer understanding among faculty and
  students of how the curriculum supports
  increasing mastery of the PULs.
• Contribute to assessment of student learning of
  the PULs at multiple levels of aggregation.
• Support student engagement with the PULs over
  their entire undergraduate experience.

42
PULs Levels of Competence

• Introductory What all undergraduate students
  should know and be able to do within the first 26
  credit hours.
• Intermediate What all undergraduate students
  should know and be able to do within the first
  56 credit hours.
• Advanced What all baccalaureate recipients
  should know and be able to do in their major or
  profession or academic program.
• Experiential Connecting curricular and
  co-curricular learning

43
Background Knowledge Probe(Pre-Test Indirect
Measure)

• ARCHAEOLOGY
• A. Have never heard of this
• B. Have heard of it, but dont really know
  what it means
• C. Have some idea what it means, but not
  too clear
• D. Have a clear idea what this means and
  can explain it
• - Classroom Assessment
• Angelo and Cross

44
Fast Feedback(at end of every class)

• Most important thing learned
• Muddiest point
• Helpfulness of advance reading assignments for
  days work in class
• Suggestions for improving class / assignments
• Bateman Roberts
• Graduate School of Business
• University of Chicago

45
Student Suggestions for Improvement

• Install a portable microphone
• Increase type size on transparencies
• Leave lights on when using projector
• Dont cover assigned reading in detail
• Provide more examples in class

46
Outcomes of Classroom Assessment

• 1. Classes more student centered
• 2. Focus on learning, not teaching
• 3. Students take ownership for learning
• 4. Faculty look for data to make improvements
  continuously
• - Jann Freed
• Assessment Update - 1999

47
In a Comprehensive Assessment Program...

• INVOLVE
• Students
• Faculty
• Student Affairs Staff
• Administrators
• Graduates
• Employers

48
Involve Students

• 1. Set learning expectations in recruiting
• 2. Communicate learning outcomes in orientation
• 3. Involve student leaders in promoting learning
• 4. Involve students in evaluating
  courses/curricula
• 5. Let students know their recommendations are
  used.

49
Student Advisory Council at MontevalloA way to
provide continuous student assessment

• Student Recommendations
• Develop a statement of expected ethical behaviors
  for students
• Add a second research course with lab
• Increase comparative psychology
• Add terminals for statistics lab
• Increase opportunities for research, writing, and
  speaking

50
Guidance from Alumni

• Alumni surveys emphasized that graduates valued
  skills in writing, speaking, working
  collaboratively, and information literacy
• Now the Faculty Senates General Education
  Committee has developed 5 learning elements, at
  least 3 of which must be integrated in any course
  approved for general education
• -Michael Dooris
• Penn State University

51
Junior Course in Professional Communication

• Teams devise promotional event for non-profit
  agency
• Campaign theme and rationale
• Logo and other visual materials
• Event outline
• Faculty, agency representative, peers evaluate
  the plan
• - A. J. Johnson
• Alverno College

52
Assessment in Fine Arts

• Portfolio review and/or audition
• for every student every semester
• by
• Panel of faculty, students, community
• representatives, staff or faculty outside fine
  arts
• Results - Students creativity,
  conceptualization, technique have improved
• Alec Testa
• Eastern New Mexico University

53
Involving Employers

• Combination of survey and focus groups for
  employers of business graduates
• Identified skills, knowledge, personality
  attributes sought by employers
• Encouraged faculty to make curriculum changes
• Motivated student to develop needed skills
• Strengthened ties among faculty, students,
  employers
• - Kretovics McCambridge
• Colorado State University

54
Colorado State UniversityCollege of Business

• Curriculum changes based on employer suggestions
• 1 credit added to Business Communications for
  team training and more
  presentations
• Ethics social responsibility now discussed in
  intro courses
• New Intro to Business course emphasizing career
  decision-making
• More teamwork, oral written communication,
  problem-solving in Management survey courses
• - Kretovics McCambridge

55
Authentic AssessmentAt

• Southern Illinois University - Edwardsville
• Business - Case Study Analysis with Memo
• Education - Professional Portfolio
• Psychology - Poster on Research Project
• Engineering - Senior Design Project
• Nursing - Plan of Care for Patient

56
Responses to AssessmentAt

• Southern Illinois University - Edwardsville
• Business - More case studies and research
• Education - More practice in classroom management
• Psychology - Curriculum change in statistics
• Engineering - More practice in writing and
  speaking
• Nursing - Simulation lab with computerized
  patients

57
Virginia Commonwealth University

• 1. First-year English students wrote 1
• response / week for 15 weeks
• 2. Faculty-student affairs teams read
• essays
• 3. Sample of writers interviewed 18
• months later
• Responses to findings
• Central advising center, new advising
  handbook, multicultural workshops

58
2004-2005

• The Year of Accountability
• . . . Peter Ewell
• Assessment Update
• September 2005

59
National Center for Public Policy inHigher
Education

• Measuring Up 2004-5-State Pilot
• National Adult Literacy Survey
• Licensure and grad admissions scores
• College Learning Assessment
• (or ACT Work Keys)

60
3 Themes Common to 4 Reports

• Accountability is now about the outcomes of
  student learning
• Accountability is more focused on serving the
  public interest
• Accountability now emphasizes public disclosure
• . . . Peter Ewell
• Assessment Update
• September 2005

61
External Accountability Measures are Summative
rather than Formative and Lead to Compliance
or Creative LOCAL Initiatives
62
The Future

• Need for evidence of accountability will increase
• More faculty will recognize benefits of
  assessment
• More electronic assessment methods will be
  developed
• More sharing of assessment methods will take
  place
• Faculty will learn more about learning and
  student learning will improve