The DATA WISE Process and Data-Driven Dialogue

1
The DATA WISE Process and Data-Driven Dialogue
2
Using data effectively does not mean getting
good at crunching numbers. It means getting good
at working together to gain insights from
student-assessment results and to use the
insights to improve instruction.- Kathryn
Boudett, Elizabeth City, Richard Murnane, When
19 Heads Are Better Than One, Education Week,
December 7, 2005.
3
The DATA WISE Process and Data-Driven Dialogue
4
Organize for Collaborative Work
Prepare

• Build Data Teams
• Establish team structure to allow for data
  discussions
• Establish norms
• Utilize protocols
• Complete a Data Inventory

5
Data Teams
Prepare

• Raise important questions about student learning
  and achievement
• Assist in organizational aspects of data
• Dialogue about multiple data sources
• Examine and interpret data
• Investigate ways to improve teaching and learning

6
Curriculum CouncilsInitial Data Teams
Prepare

• October and November
• - Setting Norms Protocol
• - Compass Points Protocol

7
Setting Norms Protocol
Prepare

• What norms do we need?
• Brainstorm
• Discuss
• Synthesize
• Build consensus

8
We need to build emotional safety to reach
cognitive complexity. B. Wellman and L.
Lipton
9
Curriculum CouncilsInitial Data Teams

• October and November
• - Setting Norms Protocol
• - Compass Points Protocol
• December and January
• - Data Inventory
• - Data Analysis Tools

10
Purpose of Data Inventory
Prepare

• Summarizes all the types of data that are
  available and helps to determine what other data
  is needed
• Builds assessment literacy
• Assists in the planning of using data effectively
• Begins conversation about educational questions

11
Data Inventory
Prepare
Data Source Dates of Collection Students Assessed Purpose Current Data Use More Effective Use
ELA State Asmt January (results in summer) Grades 3, 4, 5, 6, 7, 8 State Accountability purposes and to evaluate program and students Program evaluation and intervention placement Data Team analyzes data to inform instruction
Other Student Level-Information Other Student Level-Information Other Student Level-Information Other Student Level-Information Other Student Level-Information Other Student Level-Information
Race/Ethnicity Race/Ethnicity Race/Ethnicity Race/Ethnicity Race/Ethnicity Race/Ethnicity
Data Wish List Data Wish List Data Wish List Data Wish List Data Wish List Data Wish List

12
Data Inventory
Prepare
Data Source Dates of Collection Students Assessed Purpose Current Data Use More Effective Use
Running Record Ongoing Grades K - 2 Evaluate a students reading skill and level To inform planning of guided rdg. lessons and inform new teacher of rdg level
District Writing Folder 3 times per year per grade Grades K-12 Document a students writing progress over time To archive student work Teacher review regularly to inform instruction
13
The DATA WISE Process and Data-Driven Dialogue
14
Assessment Literacy
Prepare
scaled score
norm-referenced
cohort
reliability
validity
cut score
measurement error
performance levels
raw score
grade equivalents
sampling
standards-referenced
percentile rank
criterion-referenced
15
Data Analysis Tools

• COGNOS Report Net
• COGNOS Power Play Cubes
• Data Mentor
• nySTART
• NYS State Report Card Databases and ELA and Math
  Media Databases
• Student Management System

Demonstrate tools at Curriculum Councils, Grade
Level and Department Meetings Offer training
opportunities
16
The DATA WISE Process and Data-Driven Dialogue
17
Data Overview

• Determine audience
• Decide on educational questions
• Create graphic displays of standardized test
  results
• Engage in conversations around initial data set

18
2006 English Language Arts Performance
Data Sources CNYRIC COGNOS PowerPlay Cubes,
NYSED School Report Card and ELA Assessment
Databases
19
2006 Mathematics Performance
Data Sources CNYRIC COGNOS PowerPlay Cubes,
NYSED School Report Card and Math Assessment
Databases
20
The DATA WISE Process and Data-Driven Dialogue
21
Without an investigation of the data, schools
risk misdiagnosing the problem.Data Wise, 2005
22
Data Analysis Protocol

• Activate and Engage
• Set norms
• Articulate predictions and assumptions
• Explore and Discover
• Begin with a single data source
• First describe what you see
• Ask questions
• Identify additional data needs

23
The DATA WISE Process and Data-Driven Dialogue
24
Data Analysis

• Writers Club

25
Informing Planning for Writers Club Use COGNOS
PowerPlay to identify needs of struggling
students.
Item Difficulty( of points earned out of total possible points) Item Difficulty( of points earned out of total possible points) Item Difficulty( of points earned out of total possible points) 2006 Grade 3 English Language Arts Assessment  2006 Grade 3 English Language Arts Assessment  2006 Grade 3 English Language Arts Assessment  2006 Grade 3 English Language Arts Assessment 
Item Difficulty( of points earned out of total possible points) Item Difficulty( of points earned out of total possible points) Item Difficulty( of points earned out of total possible points) SCHOOL DISTRICT  BOCES  Region 
MC 01 Level_3 L3_Low 89.7 90.0 92.4 92.6
MC 01 Level_3 Level_3 93.8 95.5 95.2 95.1
MC 01 Level_2 L2_High 85.7 74.1 86.4 86.7
MC 01 Level_2 L2_Med 100.0 85.7 81.9 80.9
MC 01 Level_2 L2_Low 100.0 100.0 72.5 72.5
MC 01 Level_2 Level_2 91.7 79.5 82.5 82.2
MC 01 Level_1 L1_High 80.0 55.6 54.0 56.6
MC 01 Level_1 L1_Med /0 /0 23.5 20.5
MC 01 Level_1 L1_Low /0 /0 20.0 14.3
MC 01 Level_1 Level_1 80.0 55.6 51.6 53.6
MC 01 All Performance Levels All Performance Levels 94.1 92.7 90.1 89.6
MC 02 Level_3 L3_Low 100.0 100.0 97.6 97.5
MC 02 Level_3 Level_3 100.0 99.5 98.3 98.4
MC 02 Level_2 L2_High 100.0 85.2 95.3 95.6
MC 02 Level_2 L2_Med 100.0 71.4 93.2 93.9
MC 02 Level_2 L2_Low 100.0 100.0 91.0 91.7
MC 02 Level_2 Level_2 100.0 84.6 93.9 94.4
MC 02 Level_1 L1_High 80.0 88.9 72.1 74.6
MC 02 Level_1 L1_Med /0 /0 23.5 17.9
MC 02 Level_1 L1_Low /0 /0 20.0 14.3
MC 02 Level_1 Level_1 80.0 88.9 68.3 70.1
MC 02 All Performance Levels All Performance Levels 99.2 97.4 95.6 95.6
26
A distracter analysis may help you understand
childrens incorrect thought processes.
Distracter Analysis Item Countas values Distracter Analysis Item Countas values Distracter Analysis Item Countas values 2006 Grade 3 English Language Arts Assessment  2006 Grade 3 English Language Arts Assessment  2006 Grade 3 English Language Arts Assessment  2006 Grade 3 English Language Arts Assessment  2006 Grade 3 English Language Arts Assessment 
Distracter Analysis Item Countas values Distracter Analysis Item Countas values Distracter Analysis Item Countas values ELEMENTARY SCHOOL  ELEMENTARY SCHOOL  ELEMENTARY SCHOOL  ELEMENTARY SCHOOL  ELEMENTARY SCHOOL 
Distracter Analysis Item Countas values Distracter Analysis Item Countas values Distracter Analysis Item Countas values Choice 1  Choice 2  Choice 3  Choice 4  Blank 
MC 01 Level_3 L3_Low 1 26 2 0 0
MC 01 Level_3 Level_3 2 76 3 0 0
MC 01 Level_2 L2_High 0 6 1 0 0
MC 01 Level_2 L2_Med 0 4 0 0 0
MC 01 Level_2 L2_Low 0 1 0 0 0
MC 01 Level_2 Level_2 0 11 1 0 0
MC 01 Level_1 L1_High 1 4 0 0 0
MC 01 Level_1 L1_Med 0 0 0 0 0
MC 01 Level_1 L1_Low 0 0 0 0 0
MC 01 Level_1 Level_1 1 4 0 0 0
MC 01 All Performance Levels All Performance Levels 3 112 4 0 0
MC 02 Level_3 L3_Low 0 0 29 0 0
MC 02 Level_3 Level_3 0 0 81 0 0
MC 02 Level_2 L2_High 0 0 7 0 0
MC 02 Level_2 L2_Med 0 0 4 0 0
MC 02 Level_2 L2_Low 0 0 1 0 0
MC 02 Level_2 Level_2 0 0 12 0 0
MC 02 Level_1 L1_High 1 0 4 0 0
MC 02 Level_1 L1_Med 0 0 0 0 0
MC 02 Level_1 L1_Low 0 0 0 0 0
MC 02 Level_1 Level_1 1 0 4 0 0
MC 02 All Performance Levels All Performance Levels 1 0 118 0 0
27
Data Analysis

• Writers Club
• Physical Education program
• Intervention Analysis

28
Intervention Analysis
2006 Cohort ELA 8 Performance of students who
performed at Level 1 or 2 on ELA 4 and remained
in district (n43)
Tracking cohort performance may give you some
information about program or student growth.
29
Data Analysis

• Writers Club
• Physical Education program
• Intervention Analysis
• English Language Arts Analyses

30
English Language Arts

• Educational question Is student performance
  declining in reading comprehension but increasing
  in listening comprehension?
• Utilize COGNOS PowerPlay item analysis and
  Scoring Key to classify questions by subtest
• Compile data using formulas and functions
• Study trends over time
• Use comparative data to inform analysis

31
(No Transcript)
32

• The graph above illustrates the importance of
  using comparative data to inform analysis.

33
English Language Arts Analyses
COGNOS Report Net now houses analysis reports
which can provide you information about student
performance.
34
English Language Arts Analyses
Last Name First Name School Teacher Reading Comp (MC-28) Listening Writing (5) Reading Writing (5) Writing Mechanics (3) Level Total Score
Sample Abby EH Eladata 18 3 3 1 Level 3 652
Sample Bob EH Eladata 26 3 3 2 Level 3 695
Sample Catherine EH Eladata 28 3 3 2 Level 3 711
Sample David EH Eladata 27 3 4 2 Level 3 711
Sample Erin EH Eladata 28 3 4 2 Level 4 721
Sample Frank EH Eladata 11 2 1 1 Level 1 606
Sample Gabby EH Eladata 22 3 3 2 Level 3 673
Sample Harold EH Eladata 27 3 3 2 Level 3 703
Sample Iris EH Eladata 15 3 2 1 Level 2 632
Sample Jacob EH Eladata 24 4 4 3 Level 3 711
Sample Karen EH Eladata 20 2 2 2 Level 3 652
Sample Luke EH Eladata 28 2 3 2 Level 3 703
35
Sampling Principle

• Because a test is not a direct measure of a
  students degree of mastery of an entire domain,
  any conclusion you reach about proficiency in
  that domain is based on an inference from
  proficiency on the smaller sample. Even a test
  that provides good support for one inference may
  provide weak support for another.
• Data Wise, 2005

36
Data Analysis

• Writers Club
• Physical Education program
• Intervention Analysis
• English Language Arts Analyses
• Math Analyses

37
Data Overview of NYS Assessment
Performance

• Considerations when reviewing summary data
• Different cohort groups
• Different samples of items each year
• Different test blueprints
• Importance of comparison data sets

38
Go to www.emsc.nysed.gov/osa and the Report Card
link to access data for similar schools.
39
Enrollment in College Level Math Courses
Note Dual enrollments taken into account for
total annual percentage
Utilize your own student management system to
analyze additional data as well.