Talks archive, 2015–2019 – SBAC Student Growth Norms

SBAC Student Growth Norms: Theoretical Considerations and Results Damian Betebenner Center for Assessment Minneapolis, Minnesota May 15th, 2019

Background • Beginning in Spring 2018, the Center for Assessment began working with SBAC to create growth norms and SGPs using SBAC consortium data. • Project was led by Drs. Damian Betebenner and Joseph Martineau from the Center for Assessment. • Project was conducted between June, 2018 and February, 2019.

Background • Project goals – Provide a brief overview of student growth and the models used nationally. – Analyze SBAC data using 3 student growth models: • Gain scores • Gain score norms. • Norms conditioning on prior score(s) (Student Growth Percentiles) – Present summary results for the different models using SBAC data – Provide recommendations on how growth norms could best be used.

SBAC Data • Three years of longitudinal SBAC data was provided: 2014-2015, 2015-2016, and 2016-2017. • Data comprised student results from 10 SBAC member states. • No personally identifiable information was provided as part of the data. • Limited student demographic information was provided as part of the data. • No school or district identifiers were provided in the data.

SBAC Data: Valid Scaled Score Frequencies

SBAC Data: Valid Scaled Scores in consecutive years/grades

Growth Analyses • Three classes of growth analyses were conducted with the SBAC data: – Gain score analyses using the SBAC vertical scale – Gain score norm analyses – Norms conditioning on prior score(s) (Student Growth Percentiles) • Using current year score for dependent variable and single prior (with and without measurement error correction) • Using gain score for dependent variable and single prior (with and without measurement error correction)

Growth Analyses • All analyses were successfully conducted using the R software environment. • No substantial issues with ceiling/floor (HOSS/ LOSS) were observed. – However, there appear to be different LOSS/HOSS rules being applied in different states. – At the extremes of the scales where LOSS/HOSS rules differ, growth results would not be comparable across states with different ceilings/floors.

Gain Score Analyses • The most intuitive and straightforward growth analysis method is gain/difference scores. • The SBAC vertical scale permits scores from adjacent grades in the same content area to be subtracted to provide a gain/loss for each student with two such scores. • Because each score is an unbiased estimate of the student’s true score, the gain score is an unbiased estimate of the student’s true gain score.

Gain Score Analyses Mean student gain by grade and content area

Gain Score Analyses Mean student gain by grade & prior score: ELA SBAC 2016 to 2017 Mean Scale Score Gains: ELA Correlations: Gains with Pr ior Achievement Grade 4: −0.19 Grade 5: −0.19 75 Grade 6: −0.27 Grade 7: −0.17 Grade 8: −0.23 Scale Score Gain (2016 to 2017) 2016 Scale Score (Pr ior Scale Score) Prior Score Deciles indicated b y circles

Gain Score Analyses Mean student gain by grade & prior score: Math SBAC 2016 to 2017 Mean Scale Score Gains: Math Correlations: Gains with Pr ior Achievement Grade 4: −0.17 Grade 5: −0.03 75 Grade 6: 0.02 Grade 7: −0.1 Grade 8: −0.03 Scale Score Gain (2016 to 2017) 2016 Scale Score (Pr ior Scale Score) Prior Score Deciles indicated b y circles

Gain Score Analyses Percentage of students with negative gains Percentage of students with negative gains ELA Math Grade 4 Grade 5 Grade 6 Grade 7 Grade 8

Gain Score Analyses • Despite their intuitive nature, gain scores are difficult to interpret because their magnitude depends upon: – Content area – Grade – Year – Prior score • In addition, because of these dependencies aggregation (and comparison) of gains is problematic.

Gain Score Norm Analyses • One way of eliminating the gain score dependency on content area, grade, and year is to create gain norms (percentile ranks for gains). • Gain score norms maintain a dependency upon prior scores.

Student Growth Percentiles • To eliminate dependency upon prior scores one can condition upon prior score(s). • Student growth percentiles (SGPs) are growth norms conditioned upon prior score(s). • There are several ways of calculating SGPs: – With or without measurement error correction. – Using the final score or the gain score for the dependent variable in the regression analyses.

Student Growth Percentiles • Properties of SGPs – Uniformly distributed – Uncorrelated with prior achievement – Uncorrelated from year-to-year. That is, no relationship between a student’s SGP from one year to the next. – Central tendency of SGPs is 50. That is, mean/median SGPs for groups tend toward 50. – Growth norms don’t address questions of ”good enough” or ”adequate” growth. See criterion-referenced growth.

Student Growth Percentiles Distribution of SGPs by prior score decile. 2017 Grade 5 ELA

Student Growth Percentiles 2017 Mean SGP-Gain by State, Grade and Content Area

Student Growth Percentiles 2017 Mean SGP-Gain by Demographic subgroup, Grade and Content Area

Student Growth Percentiles Mean SGP-Gain by state and prior score group: 2017 ELA Grade 4

Student Growth Percentiles • There are several types of norm-groups that can be used to create the growth norms: – Consortium level norms using all students in the consortium. – State level norms that create a different growth norms for each state. – Weighted consortium level norms where states are proportionally weighted to adjust for state size. • Analyses for this report used consortium norms.

Recommendations • The SBAC vertical scale provides the foundation for monitoring student year-to-year growth. • Technical and non-technical stakeholders will calculate gain scores. • Supplementing gain-scores with growth norm information will help stakeholders interpret the gain. • There are many ways in which growth norm information can be communicated besides a percentile rank.

About this copy

SBAC Student Growth Norms — SBAC, May 15, 2019.

This is an image-based copy of the original PowerPoint slides; builds and animations are flattened.