Tools for Supporting Increased Rigor

1
Tools for Supporting Increased Rigor
High-Quality Assessment and Instruction

• Verona Public Schools
• Karin K. Hess, Ed.D.
• www.karin-hess.com

2
Track your reflections as we work

• Ways I am refining my thinking about DOK/rigor
• ?
• ?

• Scaffolding strategies for getting students to
  deeper thinking
• ?
• ?

3
Before we begin

• Take a minute to jot down words/phrases that come
  to mind when you think of cognitive rigor as it
  relates to instruction, learning, and/or
  assessment.

4
Lets apply your rigor definitions

• Your class has just read some version of Little
  Red Riding Hood.
• What is a basic comprehension question you might
  ask?
• What is a more rigorous question you might ask?

5
The Hess Cognitive Rigor Matrix integrates Bloom
Webb

• Different states/schools/teachers use different
  models to describe cognitive rigor. Each
  addresses something different.
• Bloom What type of thinking (verbs) is needed
  to complete a task?
• Webb How deeply do you have to understand the
  content to successfully interact with it? How
  complex is the content?

6
Merging Bloom Webb The thinking behind the
development of the Hess Cognitive Rigor Matrix
7
Blooms Taxonomy 1956 Blooms Cognitive
Process Dimensions 2001
Knowledge -- Define, duplicate, label, list, name, order, recognize, relate, recall Remember Retrieve knowledge from long-term memory, recognize, recall, locate, identify
Comprehension -- Classify, describe, discuss, explain, express, identify, indicate, locate, recognize, report, review, select, translate Understand -- Construct meaning, clarify, paraphrase, represent, translate, illustrate, give examples, classify, categorize, summarize, generalize, predict
Application -- Apply, choose, demonstrate, dramatize, employ, illustrate, interpret, practice, write Apply -- Carry out or use a procedure in a given situation carry out or use /apply to an unfamiliar task
Analysis -- Analyze, appraise, explain calculate, categorize, compare, criticize, discriminate, examine Analyze -- Break into constituent parts, determine how parts relate
Synthesis -- Rearrange, assemble, collect, compose, create, design, develop, formulate, manage, write Evaluate -- Make judgments based on criteria, check, detect inconsistencies/fallacies, critique
Evaluation -- Appraise, argue, assess, choose, compare, defend, estimate, explain, judge, predict, rate, core, select, support, value Create -- Put elements together to form a coherent whole, reorganize elements into new patterns/ structures
8
Webbs Depth-of-Knowledge Levels

• DOK-1 Recall Reproduction - Recall of a fact,
  term, principle, concept, or perform a routine
  procedure
• DOK-2 - Basic Application of Skills/Concepts -
  Use of information, conceptual knowledge, select
  appropriate procedures for a task, two or more
  steps with decision points along the way, routine
  problems applying 2 concepts, organize/display
  data, interpret/use simple graphs
• DOK-3 - Strategic Thinking - Requires reasoning,
  developing a plan or sequence of steps to
  approach problem requires some decision making
  and justification abstract, complex, or
  non-routine often more than one possible answer
  or approach
• DOK-4 - Extended Thinking - An original
  investigation or application to real world
  requires time to research, problem solve, and
  process multiple conditions of the problem or
  task non-routine manipulations, across
  disciplines/content areas/multiple sources

9
DOK Misconception 1 All kids cant do this or
Kids dont need scaffolding to get up there.

• Engaging in a complex task with supports/
  scaffolding is an essential step along the way to
  proficiency (Vygotskys ZPD)
• Do it with others first DOK 3 and 4 are not
  meant to only be done alone/independently,
  especially at first
• Oral language meaningful discourse support
  deeper thinking and increase initial exposures to
  the content and student engagement. This is NOT
  cheating!
• One strategy Plan questioning formative probes
  from DOK 1-2-3-4 over the course of a lesson or
  unit of study. Consider all DOK levels in your
  planning.

10
Vygotsky Zone of Proximal Development(What a
child can do with assistance today)
What a child can do independently tomorrow/future
What a child can do independently now ENTRY
Actual Development Area
Potential Development Area
The ZONE
LEARNING PROGRESSIONS ZONE Dynamic area Causes
development to move forward Social interaction
essential (scaffolding)
11
DOK Misconception 2Webbs DOK model is a
taxonomy

• Blooms is a taxonomy, intended to be a hierarchy
• Primary Weaknesses of Bloom generic verbs (void
  of content) same verbs at different levels
• Webbs DOK model is nominative
• It names how you interact with content
• It differentiates varying levels of engagement
  with content and suggests what tasks might look
  like
• DOK 4 is not better than DOK 3, or DOK 2, or DOK 1

12
DOK Misconception 3Bloom verbs levels Webb
DOK
The DOK Wheel of Misfortune implies that a DOK
level is indicated by a particular verb or set of
verbs.
Norman Webb, Its what comes after the verb,
that indicates the complexity of a task.
13
DOK Misconception 4 DOK is about difficulty.

• The intended student learning outcome determines
  the DOK level. What mental processing must occur?
  DOK Complexity, not difficulty!
• While verbs may appear to point to a DOK level,
  it is what comes after the verb that is the best
  indicator of the rigor/DOK level and complexity
  of the task.
• Describe the information contained in graphics or
  data tables in the text or the rule for rounding
  a number
• Describe how the two story characters are alike
  and different.
• Describe the data or text evidence that supports
  your solution, reasoning, or conclusions
• Describe varying perspectives on global climate
  change using supporting scientific evidence, and
  identify the most significant effects it might
  have on the planet in 100 yrs.

14
DOK LEVEL 1 Recall and Reproduction
DOK LEVEL 3 Reasoning
DOK 2 Skills and Concepts
DOK 4 Extended Thinking
Understand
Create
Remember
Evaluate
Analyze
Apply
Recall, locate basic facts, definitions, details
and events
Select appropriate word when intended meaning is
clear
Explain relationships Summarize Central ideas
Explain, generalize or connect ideas using
supporting evidence (quote, text, evidence, data,
etc.)
Explain how concepts relate to other content
domains
Use context to find meaning Obtain and use
information in text features
Use language structure or word relationships
(synonyms/anto-nyms)
Use concepts to solve non-routine problems and
justify solutions with evidence
Devise an approach among alternatives to research
a novel problem
Analyze or interpret authors craft (e.g.,
literary devices, viewpoint, or potential bias)
to critique a text
Compare literary elements, facts, terms and
events. Analyze format, organization text
structures
Analyze multiple sources or texts Analyze complex
abstract themes.
Identify information in a graphic, table, visual,
etc.
UGs
Cite evidence and develop a logical argument for
conjectures based on one text or problem.
Evaluate relevancy, accuracy and completeness of
information
Synthesize information across multiple sources
Articulate a new voice, theme, perspective.
Develop a complex model or approach for a given
situation Develop an alternative solution
Generate conjectures or hypotheses based on
observations or prior knowledge
Brainstorm ideas, concepts, problems, or
perspectives related to a topic
Blooms Taxonomy Webbs DOK the Hess CRM
15
The Hess Cognitive Rigor Matrix Applies Webbs
DOK to Blooms Cognitive Process Dimensions
Depth Thinking Level 1 Recall Reproduction Level 2 Skills Concepts Level 3 Strategic Thinking Level 4 Extended Thinking
Remember - Recall, locate basic facts, details, events
Understand - Select appropriate words to use when intended meaning is clearly evident - Specify, explain relationships - summarize identify main ideas - Explain, generalize, or connect ideas using supporting evidence (quote, example, data ) - Explain how concepts or ideas specifically relate to other content domains or concepts
Apply - Use language structure (pre/suffix) or word relationships (synonym/antonym) to determine meaning Use context to identify meaning of word - Obtain and interpret information using text features - Use concepts to solve non-routine problems - Devise an approach among many alternatives to research a novel problem
Analyze - Identify whether information is contained in a graph, table, text feature, etc. Compare literary elements, terms, facts, events analyze format, organization, text structures - Analyze or interpret authors craft (literary devices, viewpoint, or potential bias) to critique a text Analyze multiple sources - Analyze complex/abstract themes
Evaluate UG unsubstantiated generalization UG unsubstantiated generalization Cite evidence and develop a logical argument for conjectures - Evaluate relevancy, accuracy, completeness of information
Create - Brainstorm ideas about a topic - Generate conjectures based on observations or prior knowledge - Synthesize information within one source, data set, or text - Synthesize information across multiple sources or texts
16
DOK Misconception 5All DOK levels can be
assessed with a multiple choice question

• Thats just dumb!
• Weak DOK 3 multiple choice items are possible
  but does selecting the best option (e.g., locate
  supporting evidence for a theme) provide as much
  insight as seeing HOW a student formulates and
  reveals thinking?
• By their nature, DOK 3 and 4 questions/tasks are
  more open-ended, generally take longer to respond
  to/solve, and may have more than one appropriate
  right answer

17
2. The DOK Matrix Instructional Paths
Instruction Assessment Decisions
Selected Response
Each standard has an assigned Depth of Knowledge.
Constructed Response
Performance Tasks
The DOK determines the cognitive level of
instruction.
18
Lets practice using the CRM-back to Little Red
Riding Hood

• Your sample questions a basic and more
  rigorous question
• Handout 2 Linking Research, Tool 1 CRM
  template for Close Reading (page 4)

19
Depth Thinking Level 1 Recall Reproduction Level 2 Skills Concepts Level 3 Strategic Thinking/ Reasoning Level 4 Extended Thinking
Remember What color was Reds cape? Who is this story about?
Understand Who are the main characters? What was the storys setting? Retell or summarize the story in your own words. What is the authors message or theme? Justify your interpretation using text evidence.
Apply Identify words/phrases that helped you to know the sequence of events in the story.
Analyze Is this a realistic or fantasy story? Compare the wolf character to the character of Red. How are they alike-different? Is this a realistic or fantasy story? Justify your interpretation using text evidence. Are all wolves (in literature) like the wolf in this story? Support your response using evidence from this and other texts.
Evaluate What is your opinion about the cleverness of the wolf? Justify your opinion using text evidence. Which version has the most satisfying ending? (establish criteria first, then locate evidence)
Create Write text messages between Red her mother explaining the wolf incident.
20
Some other content examples

• Your class will be learning about
• Fractions/decimals Data use
• Data use scientific investigation
• Elements of art principles of design
• Come up with a basic understanding and more
  rigorous question you could pose.
• Use a CRM Template to Plan Units

21
Depth Thinking Level 1 Recall Reproduction Level 2 Skills Concepts Level 3 Strategic Thinking/ Reasoning Level 4 Extended Thinking
Remember What is slope? What is white space?
Understand Read, write, and represent these fractions Describe why white space is used. Explain how you solved this problem. Why control variables in the investigation? Construct an argument to show equivalence using area, set, and linear models
Apply Convert this fraction to a decimal Add these fractions Use these data to graph your solution Conduct the investigation, interpret results, and support conclusions with data
Analyze What kind of graph or model is this? Which data point shows ____? Find examples Compare these methods. Which graph shows how the data would be displayed? Interpret what was happening in the event? Justify your interpretation using what you know about slope. Analyze more than one product (same time period, medium, theme drawing from multiple contexts source materials for the analyses
Evaluate UG - Which team is the best? How would you rank these ___? Justify your rankings using data that supports your criteria. Some say the NFL settlement for player brain injury is not adequate. Evaluate both sides using data to determine the validity of this claim.
Create How would you demonstrate each technique? Create a card game using fractions. Create scenario explained by a data display. Integrate multiple source materials with intent to develop a product
22
DOK Misconception 6Higher order thinking
deeper learning

• What we have thought of as higher order
  (analysis, evaluation, creative thinking) might
  only be engaging or funand not always deeper
• Many critical thinking examples do not go deep or
  get to DOK 3 or 4 (e.g., interpret/solve and
  justify)
• Shift our thinking from higher order to deeper
  learning, and that can mean
• deeper understanding
• deeper application
• deeper analysis, etc.

The Hess CRM illustrates this shift
23
Some general rules of thumb

• If there is one correct answer, it is probably
  level DOK 1 or DOK 2
• DOK 1 you either know it (can recall it, locate
  it, do it) or you dont know it
• DOK 2 (conceptual) apply one concept, then make
  a decision before going on applying a second
  concept express relationship (if-then
  cause-effect)
• If more than one answer/approach, requiring
  evidence, it is DOK 3 or 4
• DOK 3 Must interpret, provide supporting
  evidence and reasoning (not just HOW solved, but
  WHY it works explain reasoning for each
  step/decision made)
• DOK 4 all of 3 use of multiple sources/data/
  texts initiate complete an investigation

24
DOK Misconception 7Multi-step or longer tasks,
multiple texts, or complex texts always means
deeper thinking

• DOK 2 is not simply more than one step, its
  applying more than one concept DOK 2 is still
  routine/typical (main idea, word problems, etc.)
• Simply reading more complex texts, but NOT
  delving deeply into the texts meaning/style/etc.,
  is likely to still be DOK 1 or 2
• DOK 3 requires some aspect of open-endedness and
  interpretation with justification or support DOK
  4 drawing from multiple sources

25
Reading Standards
Depth Thinking Level 1 Recall Reproduction Level 2 Skills Concepts Level 3 Strategic Thinking Level 4 Extended Thinking
Remember KEY DETAILS Decode, read orally
Understand KEY DETAILS WORD MEANINGS- fill in CENTRAL IDEAS SUMMARIZE predict, infer REASONING SUPPORT interpret theme, purpose, pt of view/perspective REASONING SUPPORT multiple texts
Apply WORD STRUCTURE -roots, affixes, structure, RELATIONSHIPS synonyms-antonyms WORD MEANINGS-use in context USE TEXT STRUCTURES FEATURES TEXT STRUCTURES FEATURES how is the message structured/ presented? REASONING SUPPORT multiple texts -compare treatments
Analyze USE of TEXT STRUCTURE or FEATURES LANGUAGE USE-identify non- literal usage ANALYSIS REASONING WITHIN TEXTS LANGUAGE USE-interpret non- literal usage ANALYSIS REASONING ACROSS TEXTS
Evaluate AUTHORs CRAFT WITHIN A TEXT (e.g., language use impact/intent bias, reasoning) EVALUATE AUTHORs PURPOSE or CRAFT ACROSS TEXTS
Create
26
Reading Writing

Depth Thinking Level 1 Recall Reproduction Level 2 Skills Concepts Level 3 Strategic Thinking Level 4 Extended Thinking
Remember KEY DETAILS
Understand KEY DETAILS WORD MEANINGS- fill in CENTRAL IDEAS SUMMARIZE predict, infer REASONING SUPPORT DEVELOP theme, or pt of view/ perspective topic REASONING SUPPORT use multiple texts - compare or elaborate
Apply WORD STRUCTURE RELATIONSHIPS EDIT/CLARIFY USE TECHNOLOGY CITE SOURCES WORD MEANINGS-LANGUAGE USE USE OF TEXT STRUCTURES FEATURES INTEGRATE TEXT STRUCTURES FEATURES into Compositions REASONING SUPPORT
Analyze USE of TEXT STRUCTURES or FEATURES LANGUAGE USE ANALYSIS REASONING WITHIN TEXTS RESEARCH for Writing Develop reasoning ANALYSIS REASONING ACROSS TEXTS RESEARCH for Writing Comparing themes
Evaluate AUTHORs CRAFT WITHIN A TEXT Evaluate credibility of sources EVALUATE AUTHORs PURPOSE or CRAFT ACROSS TEXTS
Create WRITE/EDIT BRIEF TEXTS COMPOSE /REVISE FULL TEXTS COMPOSE FULL TEXTS-sources
27
Math Content Standards Math Practices
Depth Thinking Level 1 Recall Reproduction Level 2 Skills Concepts Level 3 Strategic Thinking (support with data, equations, models, etc.) Level 4 Extended Thinking (cross domains)
Remember Know math facts, terms
Understand Attend to precision Evaluate expressions, plot point Model with mathematics Estimate, predict, observe, explain relationships Construct viable arguments Geometry proof Integrate concepts across domains
Apply Calculate, measure, make conversions Make sense of routine problems Make sense of non-routine problems Design conduct a project
Analyze Identify a pattern Locate information in table Use tools strategically Classify, organize data, extend a pattern Reason abstractly Generalize a pattern Analyze multiple sources of evidence
Evaluate Critique the reasoning of others
Create Design a complex model
28
For each assessment task (or rubric)ask

• What is its purpose? (What content/skill is being
  assessed? is there a right answer?)
• What is the implied/intended rigor? (What mental
  processing would you expect students to engage
  in? Use the CRM to find descriptors.)
• Which standards does it REALLY assess? (content
  intended rigor)
• Does the scoring guide/rubric match content
  intended rigor?
• What would student responses tell a teacher if
  students could/could not do all or part of this
  task? (open-ended tasks, reasoning used) next
  instructional decisions are clear

29
Take-Away Messages Cognitive Rigor Some
Implications for Assessment

• Begin with daily DOK3 classroom discourse!
• Assessing only at the highest DOK level (the
  ceiling) will miss opportunities to know what
  students do dont know go for a range end
  high in selected/prioritized content
• Performance assessments can offer varying levels
  of DOK embedded in a larger, more complex task
• Planned formative assessment strategies and tools
  can/should focus on differing DOK levels

30
Some resources worth exploring

• www.nms.org national math science initiative
  gr 3 HS also has ELA and SS performance tasks
  with DOK designations
• www.readworks.org gr k-8 short literary
  informational texts with CC Qs
• http//www.louisianabelieves.com/resources/library
  /common-core-state-standards - LA gr 3-HS math
  ELA tasks (some math samples are weak and not all
  math DOK levels are correct, but generally good
  examples)
• http//ccgpsmathematicsk-5.wikispaces.com/K-5Form
  ativeAssessmentLessons28FALs29 - GA gr K-5
  math
• www.exemplars.com sample math PAs for k-12
  science PAs for k-8
• Dan Meyer blog math PAs for MS-HS kids have to
  build the problems by deciding whats needed to
  solve them good strategic thinking required
• Hess Gong (2014). Ready for college and career?
  Achieving the Common Core Standards and beyond
  through deeper, student-centered learning
  http//www.nmefoundation.org/resources/scl-2/ready
  -for-college-and-career
• Links on www.karin-hess.com
• Karins YouTube video Text Complexity Tools
  (qualitative text analysis)
• Karins vimeo DOK video http//vimeo.com/1828141
  5
• Karins YouTube DOK video excerpt from a CT
  workshop