Dissimilarities of Students Gradings Distributions

1
Dis/similarities of Students Gradings
Distributions
Matev Bren, University of Maribor, Slovenia
Institute of Mathematic, Physics and
Mechanics, Slovenia Darko Zupanc, National
Examinations Centre, Slovenia
ICSEI 2009 Vancouver, BC, Canada, Januar 4-7
2
Motivation

• To compare achievements through gradings in upper
  secondary education in Slovenia
• School vs. national grades
• Teachers vs. national grades
• Teacher vs. teacher in the same subject in a
  school
• Subject vs. subject in the same class or school
• And to define schools, teachers, students or
  class profile.

3
List of profiles
4
Outline

• Data on students' achievement at the end of
  schooling in Slovenia
• Assessment of/for Learning Analytic Tool
• Mann-Whitney U test, r - statistic and
  dissimilarity
• List of other possible dissimilarities
• Test data sets and perceptual dissimilarity
• Dissimilarity matrices
• Discussion and Suggestions

5
Data on students' achievement at the end of
schooling in Slovenija

• Upper secondary education (USE) in Slovenia.
• Data have been gathered for the last seven years
  
• Matura exams 7 x 2 14 exam sessions, 80,000
  candidates, or 64,000 of those who sat for all
  five subjects of Matura for the first time.
• Data are gathered for 64,000 x 5 320,000
  exams.
• Vocational Matura 7 x 3 - 1 20 exam sessions,
  113,000 candidates, 85,000 of those who sat for
  all four subjects of Vocational Matura for the
  first time. Data are gathered for 85,000 x 4
  340,000 Vocational Matura exams
• Data on students' achievement at the end of
  schooling include the achievement of the entire
  yearly cohort, i.e. 150,000 secondary school
  students

6
Assessment of/for Learning Analytic Tool

• National Examinations Centre with the support of
  the Slovene Ministry of Education and Sport and
  the European Social Fund developed the Assessment
  of/for Learning Analytic Tool - ALA Tool.
• Teachers and heads in Slovenia and other
  professionals are able to access examination
  database, analyse assessment results and teachers
  grades, interpret achievement of their students,
  and analyse efficiency of teaching and learning
  in classrooms and schools in USE.
• The ALA Tool allows online access to data for
  seven types of analyses and for different
  combinations between them.
• Selected data can be displayed as histograms,
  line graphs or pie charts that enable the
  representation of the distribution and allow for
  a comparison of several distributions. Analyses
  are user friendly, numerical displays can be
  exported as charts for further processing in the
  Microsoft Excel program.

7
Assessment of/for Learning Analytic Tool
8
(No Transcript)
9
Grades fall into ordinal scale

• Measurement involves scaling
• Which type of measurement scale is appropriate
  for educational grades?
• Is it appropriate to compute the mean and STD for
  grade distributions?
• The basic in grading is rank ordering
• Educational measurement in grades fall into
  ordinal scale
• (Bucik, V., 1997 Nunnally, J.C. Bernstein,
  I.H., 1994 Zupanc, D., 2005).

10
(No Transcript)
11
(No Transcript)
12
Mann-Whitney U test

• One of the best known non-parametric significance
  test is the Mann-Whitney U test also called the
  Mann-Whitney-Wilcoxon (MWM)
• U test remains the logical choice when the data
  are ordinal but not interval. It is often
  recommended for situations where the
  distributions of the two samples are very
  different
• In the case of small samples, the distribution is
  tabulated, but for samples above 20 there is a
  good approximation using the normal distribution
• In educational grade distributions proportion of
  ties is quite large thus the correction for ties
  must be applied
• The two samples difference is
• marginally significant (Plt0.05),
• significant (Plt0.01),
• highly significant (Plt0.002) or NOT significant
  (Pgt0.05)

13
r statistic and directed dissimilarity

• r is linearly related to U r is calculated by
  dividing U by maximum value for the given sample
  sizes, nA and nB

• r is a non-parametric measure of the overlap
  between two distributions if r 0.5 it
  represents complete overlap
• We can calculate the directed dissimilarity DAB
  and DBA between two distributions

• Both extreme values DAB 1 and DBA - 1
  represent complete separations, while D 0
  represent complete overlap

14
School vs. national grades in MATH

• The difference between the two distributions is
  significant (P 0.0022)
• rA 0.571 rB 0.429 DAB 0.142 DBA -
  0.142

15

• The difference between the two distributions is
  highly significant (P 0.00075)
• rA 0.428 rB 0.572 DAB -0.143 DBA
  0.143

16
Teacher vs. teacher in the same subject in a
school
17
Teacher vs. teacher in the same subject in a
school

• (L - M) Not significantly different (P0.052)
• rA 0.6252 rB 0.3748 DAB 0.2505
  DBA - 0.2505
• (M - R) Difference marginally significant
  (P0.021)
• rA 0.6471 rB 0.3529 DAB 0.2943
  DBA - 0.2943
• (L - R) Difference highly significant
  (P0.00005)
• rA 0.7565 rB 0.2435 DAB 0.5129
  DBA - 0.5129

18
Teacher vs. teacher in MATURA - national
Teacher vs. teacher in FINAL YEAR
19
Teacher vs. national grades

• The difference between the two distributions is
  highly significant (P0.000000002)
• rA 0.0740 rB 0.9260 DAB - 0.8521
  DBA 0.8521