Steve Donatti

1

• Steve Donatti
• Phil Green
• School Performance Branch

2
Session Outline

• Background
• Key Concepts
• Software Update
• Software Demonstration
• Activities
• Quick Quiz

3
Meeting expectations

• Did you attend last years Data Club PD?
• Have you had time to load your disk and
  investigate your data?
• What are you hoping to get out of the session?

4
What is the Data Club?

• A support program designed to assist school
  leaders in making performance judgements based on
  WALNA data
• Additional analyses of the Year 3, 5 and 7
  Reading, Writing, Spelling and Numeracy test
  results
• Professional development program and support line

5
History

• Pilot 1999, managed by ECU 2000-2002 under a
  variety of funding arrangements
• DET fully responsible from 2003 with CD produced
  for the first time
• CD updated each year since based on school leader
  feedback made available to schools in February
• In 2007 the Software and installer was available
  through the secure environment of Schools Online

6
What should you have?

• School DataClub Software in your computer
• School information sheet
• Socio-economic index and band
• How many sat each test, in each strand, in each
  year
• DataClub2007 download instructions
• PowerPoint slides
• Software screen shots
• Activity handout

7
Data Club 2007 whats new?

• Five years of WALNA results
• 5 years of data - 2002, 2003, 2004, 2005, 2006
• 7 sets of student progress data
• Year 3-5 2002-2004, 2003-2005, 2004-2006
• Year 5-7 2002-2004, 2003-2005, 2004-2006
• Year 3-7 2002-2006
• The progress made by each student is calculated
  in the View Data panel of graph 8
• Many other minor software enhancements

8
The analyses

• Longitudinal
• Did our latest Year 3, 5 and 7 cohorts do better
  than those from previous years?
• Are there any trends?
• Comparative
• How did our Year 3, 5 and 7 cohorts perform
  compared with others? Are there any trends?
• How did individual students perform compared with
  others?

9
The analyses

• Progress
• Did our students progress enough?
• Standardised residuals
• Did our year 3 cohort perform as expected, taking
  into account the schools socio-economic index?
• Did our year 5 and 7 cohorts perform as expected,
  taking into account both earlier cohort test
  scores and the schools socio-economic index?

10
Key concepts

• Presenting distributions
• Box and whisker plots
• percentiles, median, mean, types of distributions
• WAMSE scale
• School, State (government schools), District,
  Band
• Comparing with expected performance
• Standardised residuals
• Correlation, line of best fit, residuals,
  value-added
• Year 3 v socio-economic, Years 5 and 7 v previous
  performance and socio-economic

11
Box and Whisker Plots
450
Median or middle score
400
350
300
250
200
Mean score
150
School
12
(No Transcript)
13
Sample Data

• 1. John A 200 11. Barry M 300
• 2. Sally B 240 12. Mark N 220
• 3. Barry C 290 13. Leyton O 170
• 4. Carrie D 295 14. Kylie P 360
• 5. Alan F 270 15. Andrew R 270
• 6. Macy G 340 16. Nicola S 190
• 7. John H 100 17. Adam T 245
• 8. Simon J 230 18. Melanie U 310
• 9. Peter K 260 19. Tina V 270
• 10. Carmen L 200 20. Stuart V 280
• Total 5040 Mean 252

14
Sample Data

• 1. Kylie P 360 11. Peter K 260
• 2. Macy G 340 12. Adam T 245
• 3. Melanie U 310 13. Sally B 240
• 4. Barry M 300 14. Simon J 230
• 5. Carrie D 295 15. Mark N 220
• 6. Barry C 290 16. Carmen L 200
• 7. Stuart V 280 17. John A 200
• 8. Andrew R 270 18. Nicola S 190
• 9. Tina V 270 19. Leyton O 170
• 10. Alan F 270 20. John H 100
• Total 5040 Mean 252

15
Sample Data
Percentile
Percentile

• 1. Kylie P 360 11. Peter K 260
• 2. Macy G 340 12. Adam T 245
• 3. Melanie U 310 13. Sally B 240
• 4. Barry M 300 14. Simon J 230
• 5. Carrie D 295 15. Mark N 220
• 6. Barry C 290 16. Carmen L 200
• 7. Stuart V 280 17. John A 200
• 8. Andrew R 270 18. Nicola S 190
• 9. Tina V 270 19. Leyton O 170
• 10. Alan F 270 20. John H 100
• Total 5040 Mean 252

- 90th (337)
- 75th (293.75)
- 25th (205)
- 10th (172)
- 50th (265)
16
Box and Whisker Plots
WAMSE Scale
17
Socio-economic index

• Dimensions derived from ABS Census 2001
• Education x 2
• Occupation x 2
• Aboriginality x 2 ( based on actual in
  school)
• Family income ( adjusted with RPI)
• Single parent family
• Best socio-economic index on balance for
  predicting WALNA performance

18
Socio-economic Index - Bands
Band Index Range No. Schools (602) 0 lt
77 29 1 77 - 88 52 2
89 93 58 3 94 - 97 85 4
98 101 136 5 102 - 105 104
6 106 - 109 74 7 110 115
40 8 gt 116 24
19
2005 Performance in Bands - Means
Band Year 3 Reading Year 7 Numeracy 0
164 351 1 240 421 2
269 440 3 283 455 4
288 467 5 306 478 6
316 488 7 327 509 8
348 538
20
(No Transcript)
21
(No Transcript)
22
Comparing with expected performance -
Prediction models

• Year 3 prediction model
• We attempt to take into account socio-economic
  status no prior performance available
• 2005 correlations between 0.56 and 0.69
• Year 5 and 7 prediction models
• We attempt to take into account both
  socio-economic status and cohort performance two
  years earlier
• Year 5 correlations between 0.81 and 0.86
• Year 7 correlations between 0.80 and 0.86

23
Correlations
SES v Year 3 School Reading Mean
r 0.60
SES Index
24
School Performance
SES v Year 3 School Reading Mean
School B
School A
SES Index
25
School A Performance
SES v Year 3 School Reading Mean
School A
SES Index
26
School A Performance
SES v Year 3 School Reading Mean
School A
SES Index
27
School A Performance
SES v Year 3 School Reading Mean
School A
SES Index
28
School A Performance
SES v Year 3 School Reading Mean
School A
ve value-added
SES Index
29
School B Performance
SES v Year 3 School Reading Mean
School B
SES Index
30
School B Performance
SES v Year 3 School Reading Mean
School B
SES Index
31
School B Performance
SES v Year 3 School Reading Mean
-ve value-added
School B
SES Index
32
School Performance
SES v Year 3 School Reading Mean
negative residual
School B
School A
positive residual
SES Index
33
Normal Curve
Mean
-1 SD
1 SD
-2 SD
2 SD
68
34
Normal Curve
2.5
2.5
Mean
-1 SD
1 SD
-2 SD
2 SD
68
95
35
Standardised Residuals
Horizontal line at 0 is expected performance
Scale is Standard Deviations
36
Loading the Software

• The Data Club software and all of the data will
  been copied to your C drive and are held under
  Program FilesgtDataClub.
• The software is not designed to run on a network.
• The Data Club software can be uninstalled by
  selecting Start gt Programs gt DataClub gt Uninstall
  DataClub.

37
Software demonstration

• Refer to the Software Screen Shots handout
• Record your notes against each slide graphic

38
Anomalous data

• Disappearing box and whiskers
• Small schools
• Restricted distributions
• Spurious significant residuals

39
Small schools
40
Restricted distributions
41
Spurious significant residuals

• 26 Year 3s
• 27 Year 5s
• 19 match
• Too many have declining performance
• Year 3 result?
• What else do you know?

Standardised Residual -3.04
42
Activities

• 1. Using Customise
• 2. Transiency Impact
• 3. School Report

43
1. Using Customise

• Setting Colours
• Show ESID students

44
2. Transiency impact

• How can I get a handle on how the students we
  had for the past two years went the ones we are
  responsible for?
• How did they perform in comparison with the
  cohort, the Band and the State?
• How well did they progress?

45
3. The School Report

• What makes sense to be included in your annual
  School Report?
• From your interpretation present data as
  statements about trends refer suggestion sheet
• What about percentages meeting the benchmark?
• What about including some graphs? How do I do it?
• What does Schools Online have to offer?

46
New National Reporting Requirements

• The School Assistance Act (2004) and Regulations
  (2005) require schools to report on 13
  performance indicators under the headings
• Professional engagement
• Key student outcomes
• Satisfaction

47
New National Reporting Requirements

• School Performance Branch has been working with
  schools to ensure that the impact on schools will
  be minimised.
• The school report is the vehicle for addressing
  these reporting requirements.

48
Schools Online
49
What now?

• After looking at the Data Club analyses Ive got
  a feel for where we need to make some
  improvements. What now?
• Using the Student Results spreadsheet
• Linking with teacher professional judgements
• Developing a whole school approach that is
  evidenced based
• The Assessment Literacy Project

50
Quick Quiz

• What is the purpose of the Data Club?
• Which graph do think will be the most useful to
  you?
• What does a box plot show?
• Which scale is used in the display of test
  results?
• Whats a like-school Band?
• Which graph shows the Benchmarks and WAGSATs?
• Why are there two standardised residual graphs?
• Approximately what percentage of schools will
  have a standardised residual gt1.00?
• What do you click on to get help for a graph?
• What is one thing you will do with the DC
  resource when you get back to school ?

51
Data Club Contact Points

• Help Line - 9264 4966
• Website
• Go to the Data Club link under School
  Performance at
• http//www.eddept.wa.edu.au/accountability/
• New National Reporting Requirements
• Go to School Accountability link under School
  Performance at the same address

52
Meeting expectations

• How did we go?

53
Acknowledgements

• Data Analysis, PD Design and Presentation
• School Performance Team
• Software
• Mike Smith
• School Performance Team
• WA Government School Principals and Leaders