Investigating Learning over Time

1
Investigating Learning over Time
Longitudinal Analysis on Assistment Data

• Mingyu Feng
• Neil Heffernan

2
Purpose

• To answer two type of questions
• Do students learn over time?
• To characterize each persons pattern of change
  over time
• Within-individual change over time
• Do students differ on learning rate? If yes, what
  impact learning?
• To examine if people differ on within-individual
  change and ask for the association between
  predictors and patterns of change
• Inter-individual differences in change

3
Important Features of Study of Change

• Three or more waves of data
• Why three?
• Describe process the change
• Tell the shape of individual growth trajectory
• Is change steady?
• The more the better
• A sensible metric for time
• Make sure you still got enough waves of data
• An outcome whose values change systematically
  over time

4
Method

• Our work follows the approach presented in
• Applied Longitudinal Data Analysis (Modeling
  change and event occurrence), Singer Willett
  (2003)
• Statistical software package - SPSS was used to
  run all the analysis.

5
Data from the Assistment System

• Log data of year 2004
• 841 students
• 2 Worcester Public Schools
• 8 Teachers with 4 from each school
• Students went to labs about every other week from
  Sept., 04 to June,05
• On average, 5.70 measurement occasions
• ranges from 1 to 9

6
The Three Features in the Data

• Longitudinal data - Person-period structured
• Each student has multiple records-one for each
  measurement occasion
• Metric for time
• CenteredMonth Month centered around Sept.
  (value of months since Sept. 0 10)
• multiple sessions in one month are aggregated
  into one
• The outcome
• correct on the main question.
• Transformed to MCASScore
• MCASScore correct 54 (full score of MCAS
  test)

7
Sample Data
Student ID
Gender
Free lunch?
MCASScore
2 73 n n 104 y 74 1 950 m n n 2 16.61538
3 73 n n 104 y 74 1 950 m n n 3 20.25
4 73 n n 104 y 74 1 950 m n n 5 41.72727
5 73 n n 104 y 74 1 950 m n n 6 23.14286
6 73 n n 104 y 74 1 950 m n n 7 32.4
7 73 n n 104 y 74 1 950 m n n 8 19.63636
9 73 n n 104 y 74 1 951 f y y 2 18
10 73 n n 104 y 74 1 951 f y y 3 33.42857
11 73 n n 104 y 74 1 951 f y y 5 31.90909
Centered Month
Teacher ID
Class ID
School ID
Class Level
Special Ed.
Note 1. class level was determined class average
initial score in Oct. If avg (class
score in Oct.) gt global mean, then class_level
1 else class_level 0.2. Given the way class
level is calculated, we filtered out data waves
in Sept. and Oct.
8
Explore the data set

• Mean (MCASScore) increased across time

• T-test showed that students from School 73 has
  got significant higher scores (p lt .001)

School 73 School 75
Mean 24.3295 20.6751
Std. Dev. 13.69403 13.60057
9
Individual Change over Time

• Empirical growth plots for 24 students

10
Individual Change over Time

• Smooth nonparametric summaries of how individuals
  change over time

11
Individual Change over Time
Parameters of regression line for some students
Mean (Intercept) 16.555 Mean (Slope) 1.4586
gt 0 Correlation (intercept, slope) -.81

• Fitted trajectories (linear regression line)

12
Use Multilevel Model for Change

• We need a model that embodies two types of
  research questions
• Level-1 question about within-person change
• Level-2 question about between-person differences
• Multilevel statistical model
• Level-1 submodel that describes how individuals
  change over time
• Level-2 submodel that describes how these changes
  varies across individuals

13

• A multilevel model
• Level-1 submodel (individual growth model)
• Level-2 submodel
• Composite model

Mixed Effect Model
Interpretation Yij score for person i at time
j, a linear function of TIMEij (CenteredMonth
here) ?0i/ ?1i Intercept/Slope of true change
trajectory of student i (?0i score in
Sept.) r00/r10 Population average of level-1
intercept/slope r01/r11 Population average
difference in ?0i/ ?1i for a unit difference in
level-2 predictor (LEVELi) eij
random measurement error for person i at occasion
j ?0i, ?1i parameter residual which permits
the level-1 parameters of one person to differ
stochastically from those of others
Fixed effect
Random effect (Variance Components)
14
Fit Multilevel Model to Assistment Data

• Baseline
• Unconditional means model
• Just means and variations (no predictor)

Estimates of Fixed Effects (a)
estimated overall average MCASScore
Parameter Estimate Std. Error df t Sig. 95 Confidence Interval 95 Confidence Interval
Parameter Estimate Std. Error df t Sig. Lower Bound Upper Bound
Intercept 24.1687396 .3258244 768.537 74.177 .000 23.5291282 24.8083511
a Dependent Variable MCASScore.
within-person variance
Estimates of Covariance Parameters(a)
between-person variance
Parameter Parameter Estimate Std. Error Wald Z Sig. 95 Confidence Interval 95 Confidence Interval
Lower Bound Upper Bound
Residual Residual 126.8945794 3.1537978 40.235 .000 120.8613877 133.2289377
Intercept subject studentID Variance 55.8742819 4.2160555 13.253 .000 48.1929604 64.7799047

This tells us MCASScore varies over time and
students differ from each other on MCASScore (p lt
.001) sufficient variation at both levels to
warrant further analysis
15
More models

• Introduce first predictor TIME
• Unconditional growth model
• This factor is important BIC diff 84
• Try more factors as predictors
• school, teacher, class, class_level

estimate of students average MCASScore in Sep.
Estimates of Fixed Effects(a)
Parameter Estimate Std. Error df t Sig. 95 Confidence Interval 95 Confidence Interval
Lower Bound Upper Bound
Intercept 20.8622499 .5366397 677.170 38.876 .000 19.8085722 21.9159277
CenteredMonth .6415105 .0948756 630.321 6.762 .000 .4552001 .8278210
Average score increase .64 points every month
Reject hypothesis of no relationship between
score and TIME
16
What did We Learn?

• School matters?
• Teacher
• Class
• Class level

17
Fit Multilevel Model to Assistment Data
Model Predictors BIC params
Model A 31711.79 3
Model B CenteredMonth 31627.67 6
Model D CenteredMonthSchoolID 31616.67 8
Model E CenteredMonthTeacherID 31671.87 20
Model F CenteredMonthClassID 31668.08 70
Model K CenteredMonthClassLevel 31457.92 8
Model L CenteredMonthClassLevelSchoolID 31454.602 10
Model M CenteredMonthClassLevelSchoolID (intercept) 31449.059 9
Model N CenteredMonthClassLevelTeacherID 31516.433 22
Model O CenteredMonthClassLevelTeacherID(intercept) 31485.309 15
BIC of Model M (highlighted in bold) is the
lowest among all models A through O.
Predictor TIME, SCHOOL and Class_Level School
was only used as predictor of intercept (changing
rate is not distinguishable between schools (p gt
.05, ns)
18
Result of Model M Tells
Estimates of Fixed Effects (a)
Parameter Estimate Std. Error df t Sig. 95 Confidence Interval 95 Confidence Interval
Lower Bound Upper Bound
schoolID73.0 17.1389235 .7693093 868.727 22.278 .000 15.6290012 18.6488458
schoolID75.0 14.7419300 .7969344 968.325 18.498 .000 13.1780125 16.3058475
ClassLevel 9.4549425 1.0083900 678.784 9.376 .000 7.4750039 11.4348810
CenteredMonth .8172511 .1385169 755.640 5.900 .000 .5453273 1.0891749
CenteredMonth ClassLevel -.3473341 .1899717 650.737 -1.828 .068 -.7203657 .0256974
a Dependent Variable MCASScore.

• Interpretation
• Estimate of average initial score of students
  from lower level classes of school 73 is 17.1389
  the score is 14.7419 for students from lower
  level classes of school 75
• From higher level class adds 9 points to average
  initial score
• Estimate of change rate of lower level classes is
  .8173
• It seems students from higher level classes
  learns slower (.3473 points lower every month)
• How to use this to calculate students score?
• Students from 73 Score at Month
  17.13899.45Level (0.817-0.347Level)Month
• Students from 75 Score at Month
  14.74199.45Level (0.817-0.347Level)Month

19
A New Data Set

• Include transfer model information
• Does learning rate differ on knowledge
  components?
• Use the basic model MCAS5
• Outcome MCASScore for a single standard
• Time metric season ( every 3 months)
• For more stable estimate of student performance
  on different knowledge status
• Include pretest score from 09/2004
• Paper and pencil test
• given in original format of MCAS 04

20
Sample Data (II)
schoolID Teacher ID Class ID studentID Season KC Name MCASScore Pretest
73 104 74 950 0 G-Geometry 30.375 8
73 104 74 950 0 M-Measurement 27 8
73 104 74 950 0 N-Number-Sense-Operations 34.2692 8
73 104 74 950 0 P-Patterns-Relations-Algebra 24.3 8
73 104 74 950 0 D-Data-Analysis-Statistics-Probability 40.5 8
73 104 74 950 1 G-Geometry 43.2 8
73 104 74 950 1 N-Number-Sense-Operations 45 8
73 104 74 950 1 P-Patterns-Relations-Algebra 28.4211 8
73 104 74 950 1 D-Data-Analysis-Statistics-Probability 36 8
73 104 74 950 2 G-Geometry 27 8
73 104 74 950 2 M-Measurement 0 8
73 104 74 950 2 P-Patterns-Relations-Algebra 33.9429 8
For one student 950
21
Fit Multilevel Model to Assistment Data (II)
MODEL BIC params Predictors
Model A2 66207.548 3
Model B2 66016.383 6 Season
Model C2 65406.461 10 season KC name (intercept)
Model C2' 65722.122 10 season KC name (slope)
Model D2 65287.17 14 season KC name
Model E2 44588.375 8 season pretest
Model F2 44580.103 7 season pretest (intercept)
Model G2 44042.376 15 season pretest (intercept) KC name

• TIME is still significant (BIC diff 191)
• Knowledge Components as a predictor lead to a big
  improvement (more than 700 BIC decrease)
• Pretest is a even better differentiator (see the
  big gap between Model E and Model D)
• Why?

22
Result of Mode G2
Estimates of Fixed Effects(a)
Parameter Estimate Std. Error df t Sig. 95 Confidence Interval 95 Confidence Interval
Lower Bound Upper Bound
KCNameD-Data-Analysis-Statistics-Probability 9.2048856 .9340674 1058.357 9.855 .000 7.3720511 11.0377201
KCNameG-Geometry 12.6313817 .9237818 1020.453 13.674 .000 10.8186526 14.4441108
KCNameM-Measurement 8.0682218 .9397700 1082.318 8.585 .000 6.2242445 9.9121992
KCNameN-Number-Sense-Operations 18.8854008 .9026647 937.580 20.922 .000 17.1139238 20.6568779
KCNameP-Patterns-Relations-Algebra 19.0073128 .9173294 988.721 20.720 .000 17.2071766 20.8074491
PretestScore .6272369 .0438227 498.598 14.313 .000 .5411371 .7133367
Season (KCNameD-Data-Analysis-Statistics-Probability) 5.2260148 .5272740 3239.974 9.911 .000 4.1921905 6.2598390
Season (KCNameG-Geometry ) -.0723080 .5603555 3488.254 -.129 .897 -1.1709659 1.0263499
Season (KCNameM-Measurement) 1.4231592 .6093210 3864.869 2.336 .020 .2285379 2.6177805
Season (KCNameN-Number-Sense-Operations) 2.5131848 .4729417 2715.492 5.314 .000 1.5858227 3.4405468
Season (KCNameP-Patterns-Relations-Algebra ) -1.3867194 .4529750 2490.712 -3.061 .002 -2.2749657 -.4984732
The estimate of average initial score on
Patterns is 19.007, the highest
Pretest score increases by 1, the estimated
initial score increases by .627
estimate of average rate of change on
Data-Analysis is 5.226
Negative learning rate indicates un-learning
a Dependent Variable MCASScore.
23
Future work

• Try other predictors
• Gender, class_level, finer grained transfer
  models
• Use fitted model to predict post-test score or
  even further real MCAS score
• Introduce Assistment metrics
• Performance on scaffolds, hints
• Weigh outcome by time spent

24

• Thank you!

Details about this analysis are available at
http//www.cs.wpi.edu/mfeng/analysis