Should we report subscores

1
Should we report subscores?
2
Outline

• Subscore as Skills Diagnostic information
• Cognitive Diagnostic Models (CDMs)
• Dimensionality Analysis
• Classical Test Theory based subscore analysis
• Analysis of Classification Accuracy based on
  synthesized empirical data

3
Subscores as Skills Diagnostic information

• Why use subscore?
• Can provide more useful information about
  teaching and learning
• Example
• Rather than reporting Candy being in the 80th
  percentile in statistics
• Report Candy is proficient in multivariate
  analysis, experimental design, but needs to work
  on time series, sampling.
• Follow up such profile with quality
  individualized instruction for Candy.

4
Cognitive Diagnostic Models

• Fusion model
• Simplified from Unified model (DiBello et al.,
  1995)
• Good side
• Can link the observable performance to latent
  skills specified by expert/theory/previous
  experience (Q-matrix)
• Limitation
• Estimation
• Parameters can be estimated using Hierarchical
  Bayesian approach (Arpeggio).

5
Dimensionality Analysis

• To find the dimension structure of the test
• DETECT and NOHARM

6
Classical Test Theory based subscore analysis

• Goal To predict the true subscore from the
  observed score
• Predictors of true subscore
• Observed subscore
• Observed total score
• To report subscore in a test?

7
Method Classical Test Theory

• Habermans Method (Haberman, 2008)
• Proportional Reduction in MSE (PRMSE)

8
Research Questions

• Q1
• Can raw subscore obtain comparative student skill
  profiles as fusion model does?
• Q2
• Can four subscores (observed subscore, mastery
  probability, and expected subscores based on CTT,
  MLE of ?) produce comparative classification
  accuracy?

9
Analysis procedure for Q1

• Step 1 Dimensionality analysis
• Step 2 Cluster analysis
• Derive profiles based on raw subscores,
  transformed raw subscores, and mastery
  probability
• Step 3 Comparison of profiles by cluster analysis

10
Data

• Sample size 14874
• Number of subtests 3
• Content area Mathematics, Reading, Science

11
Results of Dimensionality Analysis

• DETECT
• DETECT index 0.22
• NOHARM
• Correlation between dimensions

12
Results of Cluster analysis (NOT in same scale)

• Comparison of cluster solutions

13
Results of Cluster analysis ( in same scale)

• Comparison of cluster solutions

14
Analysis procedure for Q2

• Step 1 Set the cutting scores based on ?cuts
• Step 2 Compute the subscores from each method
• Observed subscore (obs)
• Expected subscore based on either observed
  subscore or total score (hab)
• Maximum-likelihood estimation of ?(mle)
• Posterior probability of mastery (fus)
• Step 3 Classify students to be proficiency or
  not based on the cutting scores
• Step 4 Calculate the exact agreement among the
  classifications of each method

15
Haberman (2007) analysis based upon Classical
Test Theory

• Summary statistics for Subscores

16
Analysis of mastery/non-mastery classification

• Classification for Math

17
Analysis of mastery/non-mastery classification

• Classification for Reading

18
Analysis of mastery/non-mastery classification

• Classification for Science

19
Summary of Classification analysis
20
Future Study

• Simulation study 1 How does dimension structure
  affect the similarity of profiles based on raw
  subscore and mastery probability?
• Study 2 Can the standard setting information
  enhance the classification accuracy of fusion
  model?