SGP Precision and Guidance

Northeastern Educational Research Organization · April 2019

SGP Precision and Guidance — Northeastern Educational Research Organization, April 2019.
Published

April 23, 2019

Event
Conference
Slides
31
Format
Converted from PowerPoint

This deck is an image-based copy of the original slides. Builds and animations are flattened; the slide text is reproduced below for search and screen readers.

Slide text
  1. SGP Precision and Guidance NEERO Conference April 25, 2019
  2. Overview of Presentation • Criticism of SGP Accuracy and Reliability • SGP Properties and Uses • SGP Precision for Student-Level Results o Calculation of standard errors o Simulation of standard errors in categorizing student growth o R Square for prediction using SGPs o Use Case: Decision matrix for using SGP results • SGP Precision for Aggregate-Level Results o Calculation of standard errors o Use Case: Local evaluation of district or school programming o Use Case: Quick comparison across schools, districts Massachusetts Department of Elementary and Secondary Education
  3. SGP Criticism • SGPs are unreliable o Using the same or similar dataset of student results, consistent SGPs are calculated o SGP error is modeled from random and non-random error in testing • There is no validity evidence to support the use of SGPs o Use of SGPs in mode-adjusted studies (MA DESE, in publication) • SGPs violate Standards for Educational and Psychological Testing o Advice encourages descriptive use of SGPs, not intended for generating teacher evaluation results From: Sireci, S. G., Wells, C. S., & Keller, L. A. (2016). Why we should abandon student growth percentiles. Massachusetts Department of Elementary and Secondary Education 3
  4. STUDENT-LEVEL SGPS
  5. What are SGPs (Student Growth Percentiles?) Slides and interpretation from Betebenner, D. (October 2013). An Overview of the Student Growth Percentile Methodology. MCAS Technical Assessment Committee Meeting.
  6. SGP Computation • Computed using an R routine: o Quantile Regression generates 100 regression lines where students are assigned a SGP from 1 to 99 o All students are used in the analysis – there is no “small N” problem regarding cohorts (cohorts are a way to explain the estimation using prior performance, but that is not exactly the way SGPs are calculated) o B-Spline smoothing to provide better fit at the end of the distributions o Distribution of the SGPs is largely uniform with slightly more enrollments in the 1st and 99th quantiles Massachusetts Department of Elementary and Secondary Education
  7. SGP Distributions by Educational Level Massachusetts Department of Elementary and Secondary Education
  8. SGPs Model Changes in Student Test Scores Massachusetts Department of Elementary and Secondary Education 8
  9. Grades combine
  10. Nearly-Equal Opportunity to Grow
  11. Scatterplot Matrix: ELA Change scores are negatively correlated with prior year’s MCAS SSs (r=-.28 to -.49 ) SGPs are NOT correlated with prior MCAS SSs (r=0), but are positively correlated with current (r=.48 to .57) and future (r=.23 to .25) MCAS SSs Meaning: Student growth is unrelated to prior achievement levels
  12. SGPs 5X Less Correlated with Student Demographic Variables than Achievement Scores Comparison of Correlations: Achievement and Growth, Students Average r Race_O Race_M Race_H ELAG.13 ELAG.12 Race_W ELAG.11 esp.13 Race_A esp.12 esgp.11 Race_AA SWD_Aver LEP_Aver Free_Red_Aver Attendance_Aver Massachusetts Department of Elementary and Secondary Education 12
  13. SGP Standard Errors for Students Average Standard Errors Average SEs by MSGP SGP Standard Errors, Two Years ELA Math ELA Math Grade 4 5.26 5.33 7.21 6.15 Grade 5 5.89 5.78 7.18 6.95 Grade 6 5.93 5.74 6.86 6.82 Grade 7 5.94 5.79 7.14 7.72 Grade 8 6.17 5.81 7.51 7.25 Grade 10 5.64 6.22 7.65 7.43 Massachusetts Department of Elementary and Secondary Education 13
  14. Modeling SGP Error • SGP SEMs are calculation using scaled score CSEMs provided as part of the state’s assessment data used for SGP calculation. • Scale scored CSEMs are used to perturb the observed scales so that 100 simulated SGPs are calculated for each student. • The 100 simulated SGPs are used to calculate a standard error for each SGP reported. • In general, state assessment systems show standard errors for SGPs between 5 and 15. Massachusetts Department of Elementary and Secondary Education 14
  15. Modeling SGP Error • SGPs, being percentiles, are uniformly distributed. • However, the distribution of observed SGPs based upon a normal CSEM is normal. • That is, a for a student with a true SGP of 50, observed SGPs occur most often near 50. • For example, a student with an SGP a true SGP of 50, would expect to see an observed SGP of 20 or less 2.5% of the time (assuming an SGP SEM of 15). Massachusetts Department of Elementary and Secondary Education 15
  16. Modeling SGP Error SGP with Measurement Error: SGP of 50, SE of 15 Probability of SGP Occurring SGP Massachusetts Department of Elementary and Secondary Education 16
  17. Modeling SGP Error • In situations where students demonstrate low growth (e.g., SGP <= 20), the chances of them having true growth >= 50 (the dividing line for demonstrating a year’s growth) is < 2%. • Because SGPs are uniformly distributed, 20% of students have SGPs <= 20. • Using low SGPs as a flag for investigation of students in possible need of remediation would be justified as those students likely didn’t demonstrate a year’s worth of growth. Massachusetts Department of Elementary and Secondary Education 17
  18. Modeling SGP Error SGP with Measurement Error: SGP of 20, SE of 15 0.030 SGP = 20: Observed Growth SGP ≥ 50: At least a year's growth Probability of SGP Occurring AGP = 70: Growth to Catch−Up SGP Massachusetts Department of Elementary and Secondary Education 18
  19. SGPs are Significant Predictors of Student Performance R Square Estimates 2018, Grade 5 ELA by Grade and Subject ELA b se B t Sign. Prior Score Curr. SGP Demos R^2 Diff. Grade 4 .56 .96 .97 .00 (Constant) -19.93 0.04 0.00 -556.9 0.000 Grade 5 .61 .91 .92 .01 ELA Theta 2017 0.04 0.00 0.73 553.9 0.000 Grade 6 .62 .91 .92 .01 SGP 2018 0.02 0.00 0.54 488.4 0.000 Grade 7 .66 .94 .94 .00 LEP2017.2 -0.14 0.00 -0.03 -29.0 0.000 Grade 8 .67 .93 .93 .00 IEP2017 -0.18 0.00 -0.06 -50.4 0.000 Math ECODIS2017 -0.07 0.00 -0.03 -25.4 0.000 Prior Score Curr. SGP Demos R^2 Diff. Grade 4 .66 .97 .97 .00 Race_W 0.02 0.01 0.01 3.9 0.000 Grade 5 .69 .93 .94 .00 Race_B -0.05 0.01 -0.01 -7.4 0.000 Grade 6 .73 .94 .95 .00 Race_H -0.03 0.01 -0.01 -5.7 0.000 Grade 7 .77 .95 .95 .00 Race_A 0.05 0.01 0.01 7.2 0.000 Grade 8 .75 .95 .95 .00 Outcome Variable: 2018 MCAS Theta Demographics: IEP, EL, and socioeconomic status, Race: W, B, A, H
  20. Use Case: SGP Guidelines by Educational Level and Prior Results Student Guidance: SGPs by Prior Performance SGP Quintile/Median Quintile Prior Achievement Level 0-19 20-39 40-59 60-79 80+ Key Not Meeting More Concern Partially Meeting Some Concern Meeting Little Concern Exceeding No Concern SGPs are used alongside prior or current student results to identify students who may need additional supports. Massachusetts Department of Elementary and Secondary Education
  21. AGGREGATED SGPS
  22. SGPs in Aggregations • Properties of SGPs different for student-level and for aggregated levels • SGPs provide key information about student progress • SGPs are much less correlated with student demographic data than achievement scores • At the student level, allow for near-equal opportunity to grow o However, it is important to understand that students are not randomly assigned; consequently, low-growing students are somewhat more likely in certain situations than others Massachusetts Department of Elementary and Secondary Education
  23. Standard Deviations and Year-to-Year Correlations by Educational Level (From 2013 Paper) Level Median St. Dev. # Units Year-to-Year Average Student 50 29 ~66,000/ Correlations grade Teacher 49 17 ~3000 in Level ELA Math elem. And Student 0.00 0.00 ~1000 in MS Teacher 0.43 0.53 School 51 11 ~900 in elem. And School 0.54 0.58 ~500 in MS District 0.60 0.61 District 51 8 ~270/ grade Massachusetts Department of Elementary and Secondary Education
  24. Redo w/MLM -- SGPs are Significant Predictors of Student Performance at All Educational Levels Teacher-Level Anchored SGP Models, Grade 5 English Language Arts R R^2 B Beta t Sign. Baseline* 2011 (N=2383) B1: Prior Achieve Levels.11 0.829 0.687 A.ELA.2011_TMean 3.14 0.12 Baseline P.ELA.2011_TMean 2.16 0.09 0.47 23.28 NI.ELA.2011_TMean 0.89 0.10 15% 8.98 0.18 SGPs S2: ESGP.11 ESGP.12 0.898 0.807 1% BMdn4_esgp.B.2012_MAX 0.00 0.00 Demographics TMdn4_esgp.B.2013_MAX 0.01 0.00 0.33 33.87 B3: Demographics 0.902 0.813 TchMean_Attendance4.2 0.20 0.00 0.28 0.78 TchMeanFree_Red4.2 -0.22 -0.10 -6.46 0.00 TchMeanLEP4.2 -0.20 -0.04 -3.08 0.00 TchMean_SWD4.2 -0.40 -0.09 32% SGPs No Baseline (N=2383) S1: Prior SGPs 0.470 0.221 68% TMdn4_esgp.B.2012_MAX Demographics TMdn4_esgp.B.2013_MAX S3: Demographics 0.834 0.695 TchMean_Attendance4.2 5.27 0.08 5.88 0.00 TchMeanFree_Red4.2 -0.97 -0.45 -29.20 0.00 TchMeanLEP4.2 -1.01 -0.21 -15.16 0.00 TchMean_SWD4.2 -1.38 -0.31 -26.92 0.00 Massachusetts Department of Elementary and Secondary Education
  25. mean sd School 1 School 2 Statistics SGP1 SD1 N1 SGP2 SD2 N2 St. Error t Comparing Aggregate SGPs: Comparisons to Schools mean sd School 1 School 2 Statistics SGP1 SD1 N1 SGP2 SD2 N2 St. Error t
  26. where df (degrees of freedom)=n-1 s=29 n=sample n SGP n sqrt N t critical State 50 School1 60 100 9.949874 3.430991 School2 45 55 7.348469 -3.80093 Comparing Aggregate SGPs: Comparisons to State where df (degrees of freedom)=n-1 s=29 n=sample n SGP n sqrt N t critical State 50 School1 60 100 9.949874 3.430991 School2 45 55 7.348469 -3.80093
  27. Considerations in analyzing SGPs for Aggregated Units Cross-sectional studies Multi-year /Time Series Studies • Consider who you are comparing to • Change in growth indicates change o Like schools, districts (DART) in average scores – if an increase is o Compared to the state observed see if the school/district (examining gaps, for example) can maintain that growth over time • For a simple aggregate (e.g., 60 in year 1, an average of comparisons to the state, consider using a standard error across the 50 in subsequent years will indicate state based on the distribution of a positive change in performance aggregate units at each level of that is sustained) analysis • Consider graphing results • Variance differs by educational level – consider how to sample • Consider using average SGPs over classrooms, N ~ time
  28. INFORMATION FOR GUIDANCE MATERIALS
  29. Reporting Recommendations 1. Use SGP reports to enhance instructional/curricular planning 2. Report SGPs alongside contextual information (courses, grades, etc.) 3. SGPs should be reported alongside information on prior (baseline) achievement to enhance appropriate interpretations 4. Reporting multiple years of SGPs will reduce spurious decisions 5. At the aggregated levels of reporting, filters may improve the interpretation of SGPs 6. At the aggregated levels, the range of low, moderate, and high SGPs should be adjusted according to the distributions Massachusetts Department of Elementary and Secondary Education
  30. Teacher-Level Alternate Views
  31. Math SGP by Course and Attendance Level Massachusetts Department of Elementary and Secondary Education