Fractal Thoughts on Student Evaluations of Professors

1
Fractal Thoughts on Student Evaluations of
Professors
2
Fractal Thoughts on Student Evaluations Ratings
of Professors
3
The challenge

• Understand nature of the complex subject we are
  trying to measure
• Decide what we want our tool to do
• Decide instructional dimensions we want to
  examine
• Create good questions
• Test and tweak the tool

4
Plan for today

• Understand the fractal nature of what we are
  trying to measure
• Understand summative formative
• Learn about instructional dimensions
• Learn about creating good questions
• Consider what we want to have happen

5
Fools rush in.

• But, could fractal thinking tell us anything
  about student evaluations?

6
Whats a fractal?
7
(No Transcript)
8
All learning produces complex interconnected affec
tive and cognitive synaptic wiring.
9
Cognitive Domain?

• The products of the brain we think of commonly as
  the products of learning in formal education such
  as knowledge, concepts, skills, evaluative
  thinking based on evidence.

10
Affective Domain

• The product of the brain that produces the sense
  of feelings and emotions that are "complex but
  internally consistent qualities of character and
  conscience."

(Krathwohl et al., 1964, p.7)
11
The affective domain influences

• Attitudes
• Biases
• Enthusiasm
• Ethics
• Likes dislikes
• Motivation
• Self awareness
• Self-esteem
• Sense of responsibility

12
Some qualities of learning that involve cognitive
plus affective

• Confidence
• Knowledge
• Preconceptions
• Self assessment
• Skills
• Values including priorities

13
There is probably no cognitive learning that does
not involve an affective component.
14
Let length of coastline be L. Let divider width
used to measure L be r.
15
(No Transcript)
16
(No Transcript)
17
A curious quality of fractals they have no
absolute dimensions
18
In a fractal form, size depends on the tool of
measure.
They don't have absolute dimensions, but they
have order.
19
"The harder I try to measure it, the bigger it
seems to get."
20
"The more I study it, the less I seem to know."
21
Teaching is so complex that we cant really
identify good teaching.
22
Reflection Exercise 1

• How might fractal thinking explain the following
  observation by Bob Leamnson?

23
Turn to your neighbor. Evaluate the described
practice from a fractal thinker's view.
Over 25 years I was evaluated by more than 100
groups of students. For about 18 of those years,
I participated in annual reviews of my
departmental colleagues and in those of all Arts
and Science departments at the college level. The
methods were virtually invariable. Whenever a
rating form had a global question, it alone was
considered and all else simply ignored! If, in
the real world, you put a global question on the
rating form...the global question is the end-all
and be-allthe alpha and the omega. Leamnson,
November, 2005
24
The brain learns by building and stabilizing
neural connections (see Leamnson, 1999).
25
Quiz time!

• How many letters are in the English alphabet?
• How many words can those letters produce?
• How many ideas can be expressed by those words?
• How many stories are possible?
• And we can consider the above for just one
  discipline in one language!
• And even consider the extent to which all human
  knowledge/experience can be described by words.

26
A difference between content and learned content

• Content is roughly finite and quantifiable

• Once in the brain, learned content becomes part
  of a network of thought and feeling that is
  infinite

27
All learning produces complex interconnected affec
tive and cognitive synaptic wiring.
28
How do we diagnose MBTI or Multiple Intelligences?
What makes us think we can evaluate a teacher
from one global question--or even one tool?
29
From Petersen, S.E., Fox, P.T., Posner, M.I.,
Mintun, M. Raichle, M.E. (1989) Positron
emission tomographic studies of the processing of
single words, Journal of Cognitive Neuroscience,
1, 153-170.
30
Generators arent usually recognized for their
true power. Through recursive operations their
influence grows, persists, pervades and shapes
the final outcomes.
31
(No Transcript)
32
Illustration of the power of generators

• Ambadys and Rosenthals (1993) "thin slices"
  studies determined that students arrive at
  ratings for teachers after watching 30 seconds of
  silent content-free video that are highly
  consistent (r 0.76) with end-of-semester
  ratings.
• What insight does this provide to understanding
  student ratings?

33
Practical realizations that arise from this
concept

• Single measures can't work
• Single instructional methods are limited
• Single tools likely can't tell us nearly enough

34
Given this background, let's now concentrate on
student ratings forms
35
Historical Benchmarks

• First ratings studies Herman Remmers, 1928
• Formative/summative Mike Scriven, 1967
• Dimensions of Teaching Ken Feldman, 1989

36
Consider the nature of summative surveysOn a
scale from 4 (outstanding) to 1 (poor) rate

1. The course instructor
2. The course
3. The overall learning experience

37
Pause to Look at Instructional Dimensions and
Their Definitions
38
Pedagogy Formative Surveys

• These look for useful teaching traits in terms of
  the dimensions and the degree to which each is
  visible to students.
• Provide a pedagogical fingerprint
• Multiple measures -- specifics without global
  generalizations
• Firmly grounded in the research

39
Pause to look at a formative survey
Then, consider how the summative and formative
might differ in the nature of the sample that
each obtains from a neural network.
40
Concepts of Teaching Evaluation based on Single
Global Versus Multiple Formative Measures
41
Graphic output from formative evaluation tool
42
Teaching for Learning Satisfaction (after K.
Feldman, 1997 1998)
43
(No Transcript)
44
(No Transcript)
45
(No Transcript)
46
Next, let's map a form onto the dimensions

• We want to know what dimensions we want to
  measure
• Mapping gives us a check
• Based on the fractal character that shows dangers
  of single measures, we would like to have two or
  three for each dimension we choose.
• Work time follows--take out the sheet that looks
  like the slide that follows

47
(No Transcript)
48
Do a summative item "My overall satisfaction
with this workshop now isRate from 4
(outstanding) to 1 (poor)--Rate that now.
Next,do a formative item rate from 4 (strongly
agree) to 1 (strongly disagree) "The instructor
uses examples and illustrations." Rate that now.
49
Reflection Exercise 4

• Contrast the thinking and feeling that occurred
  in each rating. How were they similar? How did
  they differ?

50
Considerations

• Do we want to use this for personnel decisions?
• Do we want a bank of items for faculty
  improvement?
• Do we want option for supplemental faculty items?
• Do we want option for supplemental unit level
  items?
• Do we want to align our evaluation/rewards with
  our mission?
• Let's finally look at a sample of a university
  doing the last-University of Minnesota--a work in
  progress.

51
But"Student satisfaction and pedagogical
practices are not what my course is about

• So, enter the knowledge survey--which is a
  special kind of student evaluation

52
Concept of a Knowledge Survey
1. I have insufficient knowledge to answer this
question. 2. I have partial knowledge or know
where to quickly (20 minutes or less) obtain a
complete answer to this question. 3. I can
fully answer this question with my present
knowledge.
53
Consider a concept. (GCI item). First, rate your
confidence to explain the concept within the item
to follow on a scale from 3 (I know it and can
teach it right now) to 1 (I'm not at all sure)

• Are rocks and minerals alive?
• (A) Yes, rocks and minerals grow
• (B) Yes, rocks are made up of minerals, and
  minerals are analogous to plant cells
• (C) Yes, rocks and minerals are always changing
• (D) No, rocks and minerals don't reproduce
• (E) No, rocks and minerals are not made up of
  atoms

Next, answer the question.
54
.what knowledge surveys sample
55
All learning produces complex interconnected affec
tive and cognitive synaptic wiring,
and knowledge surveys sample a mix of both.
56
Enough!

• This is not a knowledge survey workshop.
• However, consider the nature of the three student
  ratings we have examined

57
Suggestion triangulate a global summative item
with a profile of pedagogical practices and a
profile of learned material.
58
END! (phew!!)

• Think about and discuss these ideas over lunch.

59
Reliability of 40 pt Formative Survey