Critical Thinking Assessment

1
Critical Thinking Assessment

• Judy Ruland PhD
• FCTL Summer Workshop
• 2003

2
Assessment

• Accurate assessment in any area requires the
  examination and understanding of the phenomenon
  to be studied

3
Definitions of critical thinking

• See FCTL Web Page for detailed definitions from a
  variety of sources.

4
Thinking about ones thinking while thinking to
make ones thinking better. (Paul, 2001).
5
Critical thinking is thinking which

• Operates on itself
• ?
• To improve itself
• ?
• Is self correcting
• ?
• Continually assesses itself
• Paul 2001

6
Critical thinking is

• reasonable, reflective thinking that is focused
  on deciding what to believe or to do.
• Ennis, 1995

7
Critical Thinking is

• Sensitive to context
• Reliant on criteria
• Self correcting
• Conducive to judgment
• Lipman 1999

8
Three components of CT

• A number of mental operations
• Certain kinds of knowledge
• Certain attitudes
• Beyers 1997

9
Mental operations

• Cognitive operations
• Metacognitive operations

10
Cognitive operations

• Those operations used to generate or find
  meaning.
• Includes decision making, problem solving,
  synthesizing, reasoning

11
Meta-cognitive operations

• The operations by which we direct or control
  those meaning-making strategies or skills.
• Includes skills such as planning, monitoring,
  assessing ones thinking self-awareness

12
Knowledge

• General heuristicsrules of thumb for how to
  execute various thinking operations
• Nature of knowledge itself knowing that what we
  believe is highly selective, fragmentary, and
  interpretive
• Domain-specific knowledgediscover favors the
  well-prepared mind. (Bruner, 1960)

13
Attitudes or dispositions

• Habitual inclinations of individuals to behave in
  a certain way
• Attitudes and dispositions support skillful
  thinking, guide it, in effect drive it.

14
Attitudes or dispositions

• Exhibit curiosity to explore and willingness for
  uncertainty and challenge
• Demonstrate a fundamental dissatisfaction with
  pat, plugged-in responses

15
Attitudes or dispositions

• Seek a clear statement of problem
• Question the assumptions on which hypotheses are
  based.
• Look for alternatives
• Deliberately examine a variety of viewpoints.
• Use credible sources and mention them

16
Attitudes or dispositions

• Secure as much information as possible before
  making a judgment.
• Suspend judgment when evidence is lacking
• Keep in mind the original and/or basic concern
• Be open minded

17
Attitudes or dispositions

• Consider seriously other points of view than
  ones own (dialogical thinking).
• Reason from premises with which one
  disagreeswithout letting the disagreement
  interfere with ones reasoning (dialectical
  thinking).
• Judge in terms of situation, issues, purposes and
  consequences rather than in terms of fixed,
  dogmatic or emotional thinking.

18
Attitudes or dispositions

• Take a position (and change a position) when
  evidence and reasons so warrant
• Persist in carrying out a thinking task
• Seek as much precision as the subject permits
• Deal in an orderly manner with the parts of the
  complex whole

19
Attitudes or dispositions

• Be slow to believe exhibit a health skepticism
• Seek a number of alternatives after an apparently
  acceptable alternative has been proposed.
• Be sensitive to the feelings, level of knowledge,
  and degree of sophistication of others.

20
As a process CT involves

• recognizing and researching the assumptions that
  undergrid thoughts and actions
• Brookfield (1997)

21
Assumptions are

• The taken-for-granted beliefs about the world and
  our place within it that seem so obvious to use
  that they do not seem to need to be stated
  explicitly.
• Give meaning and purpose to who we are and what
  we do.

22
Assumptions

• Are frequently not recognized for the provisional
  understandings that they really are.
• Ideas and actions regarded as commonsense
  conventional wisdoms are often based on
  uncritically accepted assumptions.
• Brookfield, 1997

23
Examining Assumptions

• When we think critically, we start to research
  these assumptions for the evidence and
  experiences that inform them.
• The purpose of critical thinking is the
  recognition and researching of the assumptions
  that guide our lives.

24
Assessing Critical thinking

• Standardized tests?????
• Watson-Glaser (1980), California Critical
  Thinking Skills Test (Facione, 1994, 2001) and
  the California Critical Thinking Dispositions
  Inventory (Facione, 1994, 2001) Cornell Critical
  Thinking Tests, Ennis-Wier (1995)

25
Assessing Critical thinking

• Context bound????
• The same person can be highly critical in one
  situation, or with regard to one set of ideas,
  but completely closed to reappraising another
  situation or idea critically.

26
Social process

• Learning to think critically is an irreducibly
  social process.
• Happens best when we enlist the help of others to
  see our ideas and actions in new ways.
• Very few of us get very far probing assumptions
  on our own.

27
Dog chasing its tail

• No matter how much we may think we have an
  accurate sense of ourselves, we are stymied by
  the fact that we are using our own interpretive
  filters to become aware of our own interpretive
  filters!!
• Brookfield 1997

28
Self confirming cycles

• It is very difficult to stand outside ourselves
  to see how some of our most deeply held values
  and beliefs lead us into distorted and
  constrained ways of being.
• A self confirming cycle develops where out
  uncritically accepted assumptions shape actions
  that then serve only to confirm the truth of
  those assumptions.

29
Critical mirrors needed

• If CT is an irreducibly social process. Then our
  peers (and teachers) become important critical
  mirrors.
• When our stories are listened to and then
  reflected back to us, we are often presented with
  a version of ourselves and our actions that comes
  as a surprise.

30
Writing or talking about our assumptions

• Helps us become more aware of how much we take
  for granted about our own ideas
• Alerts us to our judgmental ways of seeing
• Confirms correctness of instincts that we felt
  privately but doubted because they contradicted
  conventional wisdom

31
Assessment of CT

• Is also a social process.
• Is best done, when locally crafted in the
  specific context in which the thinking is to take
  place.
• Should involve multiple methods and should take
  into account multiple experiences, contributions
  and perceptions.

32
Assumptions about CT Assessment

• CT can only be assessed in specific contexts.

33
Assumptions about CT Assessment

• CT can often be best assessed by peers who
  function as critical mirrors.

34
Assumptions about CT Assessment

• Assessment of CT should allow learners to
  document, demonstrate, and justify their own
  engagement in critical thinking.

35
Assessment Methods

• Reflective writing
• Scenario analysis
• Critical incident
• Critical debate
• Storytellers and detectives

36
Reflection

• Reflective writing with question prompts.
• Journals
• One pagers before class (e-mailed)
• Free writes in class
• One minute papers at end of class
• Essay questions
• Others..

37
Reflection

• Most students will not be reflective unless given
  specific prompts and frequent feedback.
• Holistically grade these over time to see growth
  in critical thinking.
• Provide students with grading rubrics in syllabus
  so they are clear about what you expect.
• Provide examples of good work and not good work.

38
Reflection and sharing

• Think-pair-share with comparison of original
  reflection
• Discussion boards on Web CT
• Question of the week on Web CT
• Others..

39
Scenario Analysis

• Hypothetical scenario in which a central
  character is making some kind of decision or
  initiating some kind of action.
• Learners are asked to put themselves in the place
  of the protagonist in the scenario and to write
  down assumptions under which they think the
  person is acting.

40
Scenario Analysis

• Learners are then asked to take each of the the
  assumptions they have identified and say how the
  protagonist might check them for accuracy and
  validity.
• Finally they are asked to make an alternative
  interpretation of the scenario that the
  protagonist would disagree with if she or he were
  confronted with it.

41
Scenario Analysis-method

• What assumptionsexplicit and implicit do you
  think ____ is operating under in this situation?
  List as many as you can.
• Of the assumptions you have listed, which ones
  could ____ check by simple research and inquiry?
  How could she do this?
• Give an alternate interpretation of this
  scenarioa version that is consistent with the
  events described, but that you think ___ would
  disagree with.

42
Scenario Analysis

• This could be adapted to a pretest postest
  approach.
• Could start out with a general scenario not
  discipline specific at beginning of semester or
  program, and then later in term could have them
  do it again with a context specific scenario.

43
Scenario Analysis

• Over time allows teacher to ascertain if the
  student is able to
• identify more assumptions
• propose more diverse methods to research them
• generate increasingly greater numbers of
  alternative readings.

44
Experiential Approach-Critical incident Audits

• Anytime student is asked to
• analyze situations,
• reflect on past experience,
• critique previously made judgments and decisions
• take actions without the benefit of standard
  protocol or uniform response

45
Critical Incident

• A critical incident is defined as an event that
  can be called to mind easily and quickly because
  it is remembered vividly
• Such an incident is usually unexpected and takes
  us by surprise.
• Sometimes hey are wonderful highs, sometimes they
  are demoralizing lows.
• Often they are a mix of both.

46
Critical Incident Audits

• Think back over the past seven days. As you
  review your clinical practice, think about the
  critical incidents that have occurred .
• 1. Write a brief description of one such
  incident, including details of
• what happened,
• who was involved,
• where and when it took place, and
• what made the incident critical for you.

47
Critical Incident Assumptions

• 2. List the assumptions you have as a teacher
  that were confirmed by this incident.
• 3. List the assumptions that you have as a
  teacher that were challenged by this incident.
• 4. How did you try to check the accuracy of your
  assumptions that were challenged?

48
Critical Incident

• 5. What different perspectives could be taken on
  the incident?
• 6. In retrospect, are there different responses
  you might have made to the incident? If so what
  would these responses be, and why would you make
  them?

49
Critical Incident Uses/benefits

• As learners do these on a weekly basis, are
  documenting their growing capacity for, and
  struggles with critical thinking.
• Similar to scenario analysis in that learner is
  asked to focus on how their actions in specific
  circumstances reveal the assumptions they hold.
• Difference is that the actions are real not
  fictional.

50
Critical Incident Adaptations

• Weekly journals with question prompts
• Double entry journals
• Process recordings
• Others

51
Critical Debate-benefits

• A theatrical device with an element of playful
  swagger.
• Doesnt involve their real selves in any
  serious consideration of new ideas.
• As process evolves, students find themselves
  deeply engaged in alternative perspectives on
  familiar ideas.

52
Critical Debate-method

• Find a contenious issue on which opinion is
  divided among students.
• Be SURE that it has RELEVANCE for the students.
• Frame the issue as a debate motion
• By a show of hands, get volunteers to serve on
  each side of the debate issue

53
Critical Debate-method

• Announce that all those who volunteered for the
  supporting team will actually be on the opposing
  team and vice versa.
• Give each side time for research
• Conduct the debatehave formal guidelines for the
  time frames etc.

54
Critical Debate-method

• Debrief the debate.
• Discuss with the participants their experience of
  the exercise.
• Focus on how it felt to argue against positions
  to which they are committed.
• Identify what new ways of thinking were opened
  up.
• Ask if they changed their positions at all.

55
Follow-up reflective writing focusing on

• What assumptions about the issue were clarified
  or confirmed for you by the debate?
• Which of your assumptions were you surprised by
  during the debate?
• How could you check these new assumptions?
  Evidence?
• What new perspectives about the issue did you
  realize?
• In what ways were your existing assumptions
  challenged or changed by the debate?

56
Storytellers and Detectives

• One of hardest processes of CT for students to
  learn and faculty to assess is the ability to
  give challenging but respectful critical
  commentary on another persons ideas or actions.
• CT assessment must include assessment of ability
  to participate in critical conversions.

57
A critical conversation is

• a focused conversation in which someone is helped
  to
• come to an awareness of the assumptions under
  which she is operating,
• to investigate the extent to which these
  assumptions are well grounded in critically
  examined reality
• to look at her ideas and actions from different
  viewpoints and
• to think about implications of the conversation
  for her future actions.

58
Storytellers and Detectives

• People play one of three roles
• The storyteller
• Detectives
• Umpire
• All members play all there roles at different
  times.

59
Storyteller

• The person who is willing to make herself the
  focus of critical conversation by first
  describing some part of her experience.

60
Detectives

• Others in the group whose purpose is to help the
  storyteller examine her experience so that she
  comes to amore fully informed understanding of
  the assumptions that inform her ideas and actions.

61
Umpire

• The group member who monitors the conversation
  with a view to pointing out when people are
  talking to each other in a judgmental way.

62
Method

• Storyteller tells the tale
• As concretely and specifically as possible
• Describes a critical incident (good or bad)
• No questions or interruptions during story

63
Detectives listen purposefully

• identify any and all assumptions that are
  operating for the storyteller.
• Asked to imagine themselves inside the heads of
  the other characters in the story and try to see
  the events through their eyes.
• Detectives make mental or written notes about
  plausible alternate interpretations of the story
  that might come as a surprise to the storyteller

64
Detectives ask questions

• At end of story, detectives ask any questions
  they may have about the events describe.
• Questions are only to clarify what they have
  heard.
• Ground Rule May only request information make no
  judgments or offer any opinions or suggestions.

65
Umpire role

• Umpire points out any questions that may be
  judgmental. For example
• Did you really believe that?
• Didnt you think to?
• Do you mean to tell us that
• Also points out any body language or voice tone
  that may indicate judgment.

66
Storyteller answers

• Tries to answer as honestly as possible
• She may ask detectives WHY they have asked
  certain questions.

67
Detectives report assumptions

• Once all questions are answered, the detectives
  tell the storyteller what assumptions they think
  she holds.
• Non-judgmentally in a reporting back mode.
• I wonder if you might be holding this
  assumption or Is it possible you assumed.

68
Detectives give alternate interpretations

• Based on detectives attempts to see the story
  through the eyes of the other participants, they
  offer their alternative interpretations.
• Umpire points out when psyhoanalytic second
  guessing is occurring.

69
Storyteller rebutes

• Following the alternate perspective presentation
  the storyteller is allowed to give any additional
  information that would cast doubt on these
  interpretations.

70
Experiential audit/reflection

• At the end of the exercise all of the
  participants recount what they have learned, what
  insights they realized, and how their future
  actions will be affected.
• The umpire also gives a summary of what he/she
  saw happening.

71
Experiential audit assessment

• During the exercise and in the audit phase the
  teacher is able to gather important CT assessment
  data.
• Notes taken, videotaped etc.
• Participants willingness to engage in the
  critical conversation.

72
Modeling as an assessment

• Students learn CT by watching experienced
  teachers.
• Teachers need to engage in a public self
  assessment of their facility for critical
  analysis.

73
Modeling as an assessment

• Out loud self talk about
• assumptions held
• How their instincts and preferences shape a
  groups agenda
• Discuss how they might omit ideas or evidence
  that contradicts their positions
• Note questions that went unanswered.
• Report on a critical incident analysis

74
Information available

• Slides are available on FCTL webpage
• Several other handouts related to
• Critical thinking definitions
• Assessment practices
• Classroom environment factors that foster
  critical thinking.

75
Thanks for your attention.

• Judy Ruland
• COPHA, School of Nursing
• 407 823-3372
• Jruland_at_mail.ucf.edu