SGP Precision and Guidance NEERO Conference April 25, 2019

Overview of Presentation • Criticism of SGP Accuracy and Reliability • SGP Properties and Uses • SGP Precision for Student-Level Results o Calculation of standard errors o Simulation of standard errors in categorizing student growth o R Square for prediction using SGPs o Use Case: Decision matrix for using SGP results • SGP Precision for Aggregate-Level Results o Calculation of standard errors o Use Case: Local evaluation of district or school programming o Use Case: Quick comparison across schools, districts Massachusetts Department of Elementary and Secondary Education

SGP Criticism • SGPs are unreliable o Using the same or similar dataset of student results, consistent SGPs are calculated o SGP error is modeled from random and non-random error in testing • There is no validity evidence to support the use of SGPs o Use of SGPs in mode-adjusted studies (MA DESE, in publication) • SGPs violate Standards for Educational and Psychological Testing o Advice encourages descriptive use of SGPs, not intended for generating teacher evaluation results From: Sireci, S. G., Wells, C. S., & Keller, L. A. (2016). Why we should abandon student growth percentiles. Massachusetts Department of Elementary and Secondary Education 3

STUDENT-LEVEL SGPS

What are SGPs (Student Growth Percentiles?) Slides and interpretation from Betebenner, D. (October 2013). An Overview of the Student Growth Percentile Methodology. MCAS Technical Assessment Committee Meeting.

SGPs provide a norm-referenced interpretation of student growth.

There is a conditional distribution associated with each scale score history:
This is the conditional distribution of students with a 2011 scale score of 256.
Students with a 2012 scale score of 252 have a percentile of 40 on that distribution (SGP=40)
Students with a 2012 scale score of 262 have an SGP of 75.

SGP Computation • Computed using an R routine: o Quantile Regression generates 100 regression lines where students are assigned a SGP from 1 to 99 o All students are used in the analysis – there is no “small N” problem regarding cohorts (cohorts are a way to explain the estimation using prior performance, but that is not exactly the way SGPs are calculated) o B-Spline smoothing to provide better fit at the end of the distributions o Distribution of the SGPs is largely uniform with slightly more enrollments in the 1st and 99th quantiles Massachusetts Department of Elementary and Secondary Education

SGP Distributions by Educational Level Massachusetts Department of Elementary and Secondary Education

SGPs Model Changes in Student Test Scores Massachusetts Department of Elementary and Secondary Education 8

Grades combine

Nearly-Equal Opportunity to Grow

Scatterplot Matrix: ELA Change scores are negatively correlated with prior year’s MCAS SSs (r=-.28 to -.49 ) SGPs are NOT correlated with prior MCAS SSs (r=0), but are positively correlated with current (r=.48 to .57) and future (r=.23 to .25) MCAS SSs Meaning: Student growth is unrelated to prior achievement levels

SGPs 5X Less Correlated with Student Demographic Variables than Achievement Scores Comparison of Correlations: Achievement and Growth, Students Average r Race_O Race_M Race_H ELAG.13 ELAG.12 Race_W ELAG.11 esp.13 Race_A esp.12 esgp.11 Race_AA SWD_Aver LEP_Aver Free_Red_Aver Attendance_Aver Massachusetts Department of Elementary and Secondary Education 12

Student-level standard errors for SGPs are generated in the SGP package using an iterative method of applying the conditional standard error of measurement to the students’ current test scores, and generating 100 estimates of the students’ scores. These perturbed scores are used to estimate 100 SGPs. The standard deviation of each set of 100 SGPs is used to identify a unique standard error band for each student.

SGP Standard Errors for Students Average Standard Errors Average SEs by MSGP SGP Standard Errors, Two Years ELA Math ELA Math Grade 4 5.26 5.33 7.21 6.15 Grade 5 5.89 5.78 7.18 6.95 Grade 6 5.93 5.74 6.86 6.82 Grade 7 5.94 5.79 7.14 7.72 Grade 8 6.17 5.81 7.51 7.25 Grade 10 5.64 6.22 7.65 7.43 Massachusetts Department of Elementary and Secondary Education 13

Student-level standard errors for SGPs are generated in the SGP package using an iterative method of applying the conditional standard error of measurement to the students’ current test scores, and generating 100 estimates of the students’ scores. These perturbed scores are used to estimate 100 SGPs. The standard deviation of each set of 100 SGPs is used to identify a unique standard error band for each student.

Modeling SGP Error • SGP SEMs are calculation using scaled score CSEMs provided as part of the state’s assessment data used for SGP calculation. • Scale scored CSEMs are used to perturb the observed scales so that 100 simulated SGPs are calculated for each student. • The 100 simulated SGPs are used to calculate a standard error for each SGP reported. • In general, state assessment systems show standard errors for SGPs between 5 and 15. Massachusetts Department of Elementary and Secondary Education 14

Given average SGP standard errors in 2018 of about 7 points, we can set student-level categories at low, moderate, and high growth using the criteria below. A similar approach can be used to set growth categories for group levels.
Low: 39 or less
Medium: 40 to 60
High: 61 or higher

Using these ranges and the Center for Assessment’s 2016 template for understanding SGP standard errors, we can conclude with 84% certainty that a typical student with an SGP of 32 has an SGP that is below moderate-level growth (Betebenner, DePascale, Marion, Domaleski, & Martineau, 2016). As the Center for Assessment pointed out in their response, guidance materials suggest educators use SGPs in a descriptive manner, such as using them alongside other educational indicators such as baseline results, and using mean SGPs across two or more years. This guidance acknowledges the nonlinear nature of growth for all educational levels (e.g., it can oscillate or display other nonlinear dynamic patterns), and therefore all guidance cautions against overreacting to any one data point.

Modeling SGP Error • SGPs, being percentiles, are uniformly distributed. • However, the distribution of observed SGPs based upon a normal CSEM is normal. • That is, a for a student with a true SGP of 50, observed SGPs occur most often near 50. • For example, a student with an SGP a true SGP of 50, would expect to see an observed SGP of 20 or less 2.5% of the time (assuming an SGP SEM of 15). Massachusetts Department of Elementary and Secondary Education 15

Given average SGP standard errors in 2018 of about 7 points, we can set student-level categories at low, moderate, and high growth using the criteria below. A similar approach can be used to set growth categories for group levels.
Low: 39 or less
Medium: 40 to 60
High: 61 or higher

Using these ranges and the Center for Assessment’s 2016 template for understanding SGP standard errors, we can conclude with 84% certainty that a typical student with an SGP of 32 has an SGP that is below moderate-level growth (Betebenner, DePascale, Marion, Domaleski, & Martineau, 2016). As the Center for Assessment pointed out in their response, guidance materials suggest educators use SGPs in a descriptive manner, such as using them alongside other educational indicators such as baseline results, and using mean SGPs across two or more years. This guidance acknowledges the nonlinear nature of growth for all educational levels (e.g., it can oscillate or display other nonlinear dynamic patterns), and therefore all guidance cautions against overreacting to any one data point.

Modeling SGP Error SGP with Measurement Error: SGP of 50, SE of 15 Probability of SGP Occurring SGP Massachusetts Department of Elementary and Secondary Education 16

Given average SGP standard errors in 2018 of about 7 points, we can set student-level categories at low, moderate, and high growth using the criteria below. A similar approach can be used to set growth categories for group levels.
Low: 39 or less
Medium: 40 to 60
High: 61 or higher

Using these ranges and the Center for Assessment’s 2016 template for understanding SGP standard errors, we can conclude with 84% certainty that a typical student with an SGP of 32 has an SGP that is below moderate-level growth (Betebenner, DePascale, Marion, Domaleski, & Martineau, 2016). As the Center for Assessment pointed out in their response, guidance materials suggest educators use SGPs in a descriptive manner, such as using them alongside other educational indicators such as baseline results, and using mean SGPs across two or more years. This guidance acknowledges the nonlinear nature of growth for all educational levels (e.g., it can oscillate or display other nonlinear dynamic patterns), and therefore all guidance cautions against overreacting to any one data point.

Modeling SGP Error • In situations where students demonstrate low growth (e.g., SGP <= 20), the chances of them having true growth >= 50 (the dividing line for demonstrating a year’s growth) is < 2%. • Because SGPs are uniformly distributed, 20% of students have SGPs <= 20. • Using low SGPs as a flag for investigation of students in possible need of remediation would be justified as those students likely didn’t demonstrate a year’s worth of growth. Massachusetts Department of Elementary and Secondary Education 17

Given average SGP standard errors in 2018 of about 7 points, we can set student-level categories at low, moderate, and high growth using the criteria below. A similar approach can be used to set growth categories for group levels.
Low: 39 or less
Medium: 40 to 60
High: 61 or higher

Using these ranges and the Center for Assessment’s 2016 template for understanding SGP standard errors, we can conclude with 84% certainty that a typical student with an SGP of 32 has an SGP that is below moderate-level growth (Betebenner, DePascale, Marion, Domaleski, & Martineau, 2016). As the Center for Assessment pointed out in their response, guidance materials suggest educators use SGPs in a descriptive manner, such as using them alongside other educational indicators such as baseline results, and using mean SGPs across two or more years. This guidance acknowledges the nonlinear nature of growth for all educational levels (e.g., it can oscillate or display other nonlinear dynamic patterns), and therefore all guidance cautions against overreacting to any one data point.

Modeling SGP Error SGP with Measurement Error: SGP of 20, SE of 15 0.030 SGP = 20: Observed Growth SGP ≥ 50: At least a year's growth Probability of SGP Occurring AGP = 70: Growth to Catch−Up SGP Massachusetts Department of Elementary and Secondary Education 18

Given average SGP standard errors in 2018 of about 7 points, we can set student-level categories at low, moderate, and high growth using the criteria below. A similar approach can be used to set growth categories for group levels.
Low: 39 or less
Medium: 40 to 60
High: 61 or higher

Using these ranges and the Center for Assessment’s 2016 template for understanding SGP standard errors, we can conclude with 84% certainty that a typical student with an SGP of 32 has an SGP that is below moderate-level growth (Betebenner, DePascale, Marion, Domaleski, & Martineau, 2016). As the Center for Assessment pointed out in their response, guidance materials suggest educators use SGPs in a descriptive manner, such as using them alongside other educational indicators such as baseline results, and using mean SGPs across two or more years. This guidance acknowledges the nonlinear nature of growth for all educational levels (e.g., it can oscillate or display other nonlinear dynamic patterns), and therefore all guidance cautions against overreacting to any one data point.

SGPs are Significant Predictors of Student Performance R Square Estimates 2018, Grade 5 ELA by Grade and Subject ELA b se B t Sign. Prior Score Curr. SGP Demos R^2 Diff. Grade 4 .56 .96 .97 .00 (Constant) -19.93 0.04 0.00 -556.9 0.000 Grade 5 .61 .91 .92 .01 ELA Theta 2017 0.04 0.00 0.73 553.9 0.000 Grade 6 .62 .91 .92 .01 SGP 2018 0.02 0.00 0.54 488.4 0.000 Grade 7 .66 .94 .94 .00 LEP2017.2 -0.14 0.00 -0.03 -29.0 0.000 Grade 8 .67 .93 .93 .00 IEP2017 -0.18 0.00 -0.06 -50.4 0.000 Math ECODIS2017 -0.07 0.00 -0.03 -25.4 0.000 Prior Score Curr. SGP Demos R^2 Diff. Grade 4 .66 .97 .97 .00 Race_W 0.02 0.01 0.01 3.9 0.000 Grade 5 .69 .93 .94 .00 Race_B -0.05 0.01 -0.01 -7.4 0.000 Grade 6 .73 .94 .95 .00 Race_H -0.03 0.01 -0.01 -5.7 0.000 Grade 7 .77 .95 .95 .00 Race_A 0.05 0.01 0.01 7.2 0.000 Grade 8 .75 .95 .95 .00 Outcome Variable: 2018 MCAS Theta Demographics: IEP, EL, and socioeconomic status, Race: W, B, A, H

Use Case: SGP Guidelines by Educational Level and Prior Results Student Guidance: SGPs by Prior Performance SGP Quintile/Median Quintile Prior Achievement Level 0-19 20-39 40-59 60-79 80+ Key Not Meeting More Concern Partially Meeting Some Concern Meeting Little Concern Exceeding No Concern SGPs are used alongside prior or current student results to identify students who may need additional supports. Massachusetts Department of Elementary and Secondary Education

AGGREGATED SGPS

SGPs in Aggregations • Properties of SGPs different for student-level and for aggregated levels • SGPs provide key information about student progress • SGPs are much less correlated with student demographic data than achievement scores • At the student level, allow for near-equal opportunity to grow o However, it is important to understand that students are not randomly assigned; consequently, low-growing students are somewhat more likely in certain situations than others Massachusetts Department of Elementary and Secondary Education

Standard Deviations and Year-to-Year Correlations by Educational Level (From 2013 Paper) Level Median St. Dev. # Units Year-to-Year Average Student 50 29 ~66,000/ Correlations grade Teacher 49 17 ~3000 in Level ELA Math elem. And Student 0.00 0.00 ~1000 in MS Teacher 0.43 0.53 School 51 11 ~900 in elem. And School 0.54 0.58 ~500 in MS District 0.60 0.61 District 51 8 ~270/ grade Massachusetts Department of Elementary and Secondary Education

Redo w/MLM -- SGPs are Significant Predictors of Student Performance at All Educational Levels Teacher-Level Anchored SGP Models, Grade 5 English Language Arts R R^2 B Beta t Sign. Baseline* 2011 (N=2383) B1: Prior Achieve Levels.11 0.829 0.687 A.ELA.2011_TMean 3.14 0.12 Baseline P.ELA.2011_TMean 2.16 0.09 0.47 23.28 NI.ELA.2011_TMean 0.89 0.10 15% 8.98 0.18 SGPs S2: ESGP.11 ESGP.12 0.898 0.807 1% BMdn4_esgp.B.2012_MAX 0.00 0.00 Demographics TMdn4_esgp.B.2013_MAX 0.01 0.00 0.33 33.87 B3: Demographics 0.902 0.813 TchMean_Attendance4.2 0.20 0.00 0.28 0.78 TchMeanFree_Red4.2 -0.22 -0.10 -6.46 0.00 TchMeanLEP4.2 -0.20 -0.04 -3.08 0.00 TchMean_SWD4.2 -0.40 -0.09 32% SGPs No Baseline (N=2383) S1: Prior SGPs 0.470 0.221 68% TMdn4_esgp.B.2012_MAX Demographics TMdn4_esgp.B.2013_MAX S3: Demographics 0.834 0.695 TchMean_Attendance4.2 5.27 0.08 5.88 0.00 TchMeanFree_Red4.2 -0.97 -0.45 -29.20 0.00 TchMeanLEP4.2 -1.01 -0.21 -15.16 0.00 TchMean_SWD4.2 -1.38 -0.31 -26.92 0.00 Massachusetts Department of Elementary and Secondary Education

mean sd School 1 School 2 Statistics SGP1 SD1 N1 SGP2 SD2 N2 St. Error t Comparing Aggregate SGPs: Comparisons to Schools mean sd School 1 School 2 Statistics SGP1 SD1 N1 SGP2 SD2 N2 St. Error t

where df (degrees of freedom)=n-1 s=29 n=sample n SGP n sqrt N t critical State 50 School1 60 100 9.949874 3.430991 School2 45 55 7.348469 -3.80093 Comparing Aggregate SGPs: Comparisons to State where df (degrees of freedom)=n-1 s=29 n=sample n SGP n sqrt N t critical State 50 School1 60 100 9.949874 3.430991 School2 45 55 7.348469 -3.80093

Considerations in analyzing SGPs for Aggregated Units Cross-sectional studies Multi-year /Time Series Studies • Consider who you are comparing to • Change in growth indicates change o Like schools, districts (DART) in average scores – if an increase is o Compared to the state observed see if the school/district (examining gaps, for example) can maintain that growth over time • For a simple aggregate (e.g., 60 in year 1, an average of comparisons to the state, consider using a standard error across the 50 in subsequent years will indicate state based on the distribution of a positive change in performance aggregate units at each level of that is sustained) analysis • Consider graphing results • Variance differs by educational level – consider how to sample • Consider using average SGPs over classrooms, N ~ time

INFORMATION FOR GUIDANCE MATERIALS

Reporting Recommendations 1. Use SGP reports to enhance instructional/curricular planning 2. Report SGPs alongside contextual information (courses, grades, etc.) 3. SGPs should be reported alongside information on prior (baseline) achievement to enhance appropriate interpretations 4. Reporting multiple years of SGPs will reduce spurious decisions 5. At the aggregated levels of reporting, filters may improve the interpretation of SGPs 6. At the aggregated levels, the range of low, moderate, and high SGPs should be adjusted according to the distributions Massachusetts Department of Elementary and Secondary Education

Teacher-Level Alternate Views

Math SGP by Course and Attendance Level Massachusetts Department of Elementary and Secondary Education

About this copy

SGP Precision and Guidance — Northeastern Educational Research Organization, April 2019.

This is an image-based copy of the original PowerPoint slides; builds and animations are flattened.

• Talk page
• Download PDF
• Original file
• All talks