Hawaii SGP Calculations in 2021

Hawaii Department of Education · September 2, 2021

Hawaii SGP Calculations in 2021 — Hawaii Department of Education, September 2, 2021.
Published

September 2, 2021

Event
State agency
Slides
41
Format
Converted from PowerPoint

This deck is an image-based copy of the original slides. Builds and animations are flattened; the slide text is reproduced below for search and screen readers.

Slide text
  1. Hawaii SGP Calculations in 2021 Background and Results Damian Betebenner & Adam VanIwaarden Center for Assessment Hawaii Department of Education September 2nd, 2021
  2. Hawaii Testing in 2021 • Hawaii administered both SBA and WIDA-ACCESS in spring 2021. • Both assessments were full form, in person administrations. • SBA administered in Grades 3 to 8 and 11 in ELA and Mathematics. NOT administered in spring 2020. • WIDA-ACCESS administered in K to 12. It was administered in spring Hawaii SGP Calculations 2021
  3. SBA SGP Operational Considerations in 2021 • Due to no SBA testing in Spring 2020, SBA growth analyses will be skip- year/gap year growth from 2019 to 2021. • Two types of SGP analyses were conducted for SBA data: • Cohort referenced analyses: These analyses create growth norms from the 2019 to 2021 students. These growth norms yield cohort means/median SGPs of 50 and obscure COVID-19 academic impact. • Baseline referenced analyses: Historical growth norms were created using 2019 and prior data. These growth growth analyses don’t have mean/median SGPs of 50 and can be used to investigate overall COVID-19 academic impact • Projections (lagged and straight) were calculated for 2021 data using baseline growth norms in case they were needed. Hawaii SGP Calculations 2021
  4. WIDA-ACCESS SGP Operational Considerations in 2021 • Two types of SGP analyses were conducted for WIDA-ACCESS data: • Cohort referenced analyses: These analyses create growth norms from the 2020 to 2021 students. These growth norms yield cohort mean/median SGPs of 50 and obscure COVID-19 academic impact. • Baseline referenced analyses: Historical growth norms were created using 2020 and prior data. These growth growth analyses don’t have mean/median SGPs of 50 and can be used to investigate overall COVID-19 academic impact. • Use composite score from WIDA-ACCESS data. No analyses performed using sub-scores. • Projections (lagged and straight) were calculated for 2021 data using both cohort and baseline growth norms in case they were needed. Hawaii SGP Calculations 2021
  5. SGP Summary of Cohort versus Baseline Referenced • Cohort referenced SGPs • Growth norms created from the current cohort of students. • Mean/median will by 50 for all grade x content area analyses • Cannot investigate overall impact of the pandemic on students. • Can investigate differential impact of the pandemic on students. • Does not require scale stability across years. • Baseline referenced SGPs. • Growth norms created from historical (i.e., pre-COVID) cohorts of students. • Mean/median 50 for historical cohort, not necessarily 50 for current cohort. • Extent of deviation from (i.e. below) 50 is indicative of less learning/pandemic related academic impact. • Mean SGP deviation of 5 corresponds to an effect size of 0.18. • Can investigate BOTH overall impact and differential impact • Requires scale stability across years. Hawaii SGP Calculations 2021
  6. SGP Assumptions • SGP analyses presume valid scale scores • Students forming the growth norm group are presumed to have taken the same (or equivalent) test at the same time. Scores have to mean the same thing. • SGPs, however, are often used as a post-hoc check of scale score validity/ comparability via Reductio ad absurdum • In 2021, check form comparability and mode-of-administration comparability. • RIDE has extensive experience (e.g., PARCC paper/pencil versus online) investigating and accommodating mode-effects. • If scores are not comparable then we either attempt to make them comparable or run separate analyses. • We assume something close to population invariance with respect to IRT scaling. • The population of students tested in 2021 and differences to 2019 is a concern so we begin with data on the population of students tested. Hawaii SGP Calculations 2021
  7. SBA Participation ELA Mathematics Grade Hawaii SGP Calculations 2021
  8. WIDA-ACCESS Participation (Not sure I have all data for this) Overall Overall Grade Grade 2021 Hawaii SGP Calculations 2021
  9. SGP Calculations • SGP calculations were performed using the SGP package and R software environment based upon data supplied by the Hawaii Department of Education. • To facilitate the calculation of baseline SGPs across multiple states, we created an additional R package, SGPmatrices, that contains Hawaii matrices, amongst many others. • https://centerforassessment.github.io/SGPmatrices/ • Source code associated with calculations are available on GitHub: • SBA: https://github.com/CenterForAssessment/Hawaii • WIDA-ACCESS: https://github.com/CenterForAssessment/WIDA_HI Hawaii SGP Calculations 2021
  10. SGP Results • The following slides show SGP results for 2021. • We report only baseline referenced SGPs in these slides • Differential impact shown in baseline SGPs will mirror that found in cohort referenced SGPs. • Due to the highly skewed nature of the SGP results, we recommend median SGP over mean SGP in this instance. • We also provide 2019 results to understand pandemic related academic impact. Hawaii SGP Calculations 2021
  11. SGP Analyses and Academic Impact • Understanding pandemic related impact necessarily requires looking back to pre-pandemic results. • The impulse of stakeholders to want to compare 2021 results with 2019 results is the correct impulse at both the individual, school, district, and state level. • Wanting to look at 2019 to 2021 change is an attempt to investigate impact. • The challenge is making 2019 to 2021 comparisons apples-to-apples comparisons. • Supporting these comparisons requires having a complete understanding is who constitutes the missing data. More on that after some initial comparisons. Hawaii SGP Calculations 2021
  12. SGP Analyses for 2021 • With comparable results we can quantify pandemic related academic impact: • At the state level • Demographic subgroups including ethnicity, poverty • Academic subgroups (special education, low/high achievers) • By region/district/school • By mode of education (remote/hybrid/in-person). • Attendance • English Language Learner impact can be evaluated using WIDA-ACCESS results. Hawaii SGP Calculations 2021
  13. SBA Percent Proficient by Grade ELA Mathematics Grade Hawaii SGP Calculations 2021
  14. SBA Median Skip-Year SGPs by Grade ELA Mathematics Grade 3 NA NA NA NA 4 NA NA NA NA Hawaii SGP Calculations 2021
  15. Discussion • Median baseline SGPs in ELA were in the mid 30s/lower 40s and median baseline SGPs in mathematics were in the 20s (grade 11 median SGP of 38). • Deviation from normal growth (i.e., 50) represents the extent to which academic growth slowed down from 2019 to 2021, that is, academic impact. • Larger academic impact in ELA than mathematics. • Academic impact is fairly uniform across grades. • Academic impact is very large likely requiring multiple years and additional educational supports/interventions to recover. Hawaii SGP Calculations 2021
  16. WIDA-ACCESS Median SGPs by Grade Overall Overall Grade Grade K NA NA 6 52 (1,268) 38 (1,008) Hawaii SGP Calculations 2021
  17. Discussion of Grade Level Results • Median baseline SGPs are low in elementary school grades and are close to 50 in the high school grades. • Deviation from normal growth (i.e., 50) represents the extent to which academic growth slowed down from 2020 to 2021, that is, academic impact. • Academic impact is very large in elementary and middle school and close to non-existent in high school. • Lack of impact is consistent with: • Great academic support for the group during the pandemic. • Impoverished learning conditions prior to the pandemic such that the pandemic didn’t alter the status quo. Hawaii SGP Calculations 2021
  18. SBA Median/Mean Skip-Year SGPs by Ethnicity ELA Mathematics Ethnicity Am. In./AL Native Asian African American Hispanic or Lat. Multiple Ethnicities Native HI/PI No Race reported White Hawaii SGP Calculations 2021
  19. Discussion of Ethnicity Results Hawaii SGP Calculations 2021
  20. SBA Median/Mean Skip-Year SGPs by Free/Reduced Lunch Status ELA Mathematics Free/Reduced Lunch Status FRL Yes FRL No Hawaii SGP Calculations 2021
  21. Discussion of Free/Reduced Lunch Results Hawaii SGP Calculations 2021
  22. SBA Median/Mean Skip-Year SGPs by IEP Status ELA Mathematics IEP Status Non-IEP IEP Students with 504 Hawaii SGP Calculations 2021
  23. Discussion of IEP Results Hawaii SGP Calculations 2021
  24. SBA Median/Mean Skip-Year SGPs by ELL Status ELA Mathematics ELL Status ELL No ELL Yes Hawaii SGP Calculations 2021
  25. Discussion of ELL Results Hawaii SGP Calculations 2021
  26. SBA Median/Mean Skip-Year SGPs by Gender ELA Mathematics Gender Female Male Hawaii SGP Calculations 2021
  27. Discussion of Gender Results Hawaii SGP Calculations 2021
  28. SBA Median Skip-Year SGPs by Prior Achievement: Grade 5 & 6 Mathematics Hawaii SGP Calculations 2021
  29. SBA Median Skip-Year SGPs by Prior Achievement: Grade 7 & 8 Mathematics Hawaii SGP Calculations 2021
  30. SBA Median Skip-Year SGPs by Prior Achievement: Grade 11 Mathematics Hawaii SGP Calculations 2021
  31. SBA Median Skip-Year SGPs by Prior Achievement: Grade 5 & 6 ELA Hawaii SGP Calculations 2021
  32. SBA Median Skip-Year SGPs by Prior Achievement: Grade 7 & 8 ELA Hawaii SGP Calculations 2021
  33. SBA Median Skip-Year SGPs by Prior Achievement: Grade 11 ELA Hawaii SGP Calculations 2021
  34. Discussion of Growth by Prior Achievement Results • Except in Grades 5 and 6 ELA, a fairly consistent pattern is that higher achieving students demonstrate lower growth (have greater academic impact) than lower achieving students. • We seen this results in several states, though there are some where the impact is fairly uniform across prior achievement level. Hawaii SGP Calculations 2021
  35. Participation Rates and Interpretation • An issue impacting the interpretation of growth (and status) results in 2021 is participation in the state assessment. • Low participation doesn’t impact calculation of cohort referenced growth norms nor use of baseline referenced growth norms. • Low participation does impact comparisons of group level results with historical (e.g., 2019) results based upon higher participation rates. • For example, how should one compare growth and status results for school in 2019 and 2021 with very different rates of participation. Hawaii SGP Calculations 2021
  36. Multiple Imputation • Given the rich historical data available for students missing in 2021, and the observed performance of comparable students from 2019 to 2021, it is possible to simulate what we think their performance might have been. • Combining multiple simulations can inform us about potential bias in what we observed at state/district/school levels in 2021, as well as inform us about levels of uncertainty around them. • Multiple imputation (MI) of missing data is widely adopted and well-developed statistical analysis technique used in many fields to deal with missing data problems. Hawaii SGP Calculations 2021
  37. Multiple Imputation • MI uses information from the observed data to generate parameter estimates in three steps: • Imputation: A prediction model generates a set of plausible values for the missing observations, resulting in M imputed data sets. • Analysis: The analysis (e.g., baseline student growth percentiles) is conducted on each of the M data sets. • Pooling: Parameter and (more importantly) uncertainty estimates are constructed by pooling across the M analyses. Hawaii SGP Calculations 2021
  38. Multiple Imputation • In the context of COVID impact analyses, we use the multiple (plausible) parameter estimates to inform us about the uncertainty around what was observed. That is, we do not use MI to produce “corrected” parameter estimates, but rather to help determine whether we trust the statistics calculated from the observed (with missing) data. • Statistics of particular interest: • Mean/median scale scores • Mean/median baseline SGPs • Effect size estimates Hawaii SGP Calculations 2021
  39. Multiple Imputation • Before performing a MI analysis with SBA data, missing data patterns were investigated. • Patterns suggest some relationship to factors such as prior achievement and economic status. • These relationships suggest that missingness in SBA is non- ignorable, but does not preclude the use of MI. • Data can be considered “Missing-at-random” – relationships exist between missingness and the observed data (and so can not be considered “missing completely-at-random”) but the relationships are not systematic (e.g. all students below proficiency in 2019 were missing in 2021). Hawaii SGP Calculations 2021
  40. SGP Analysis Plans post 2021 (2022 and beyond) • Spring 2021 represents an ideal time to get a snapshot of student attainment as we are presumably at or near the point of maximal impact on student attainment. • That is, recovery has not really begun for most students in earnest. • It’s impossible to talk about recovery sensibly without understanding how big the impact is. • Monitoring recovery will be at least as critical as monitoring the impact of the pandemic on student achievement. • Beware the K-shaped recovery! Hawaii SGP Calculations 2021
  41. Communication of Results in 2021 • Technical challenges to the calculation of growth in 2021 are likely minor compared to communication challenges. • State reporting is, in general, highly aligned with accountability. • Even though analyses outlined herein deal with status/growth, they don’t necessarily align with accountability narratives. • It is critical for leadership (at the highest level) to map out a communication plan and then build analytics and reporting to support that plan. • Without a well considered, disciplined and coordinated communication plan, accountability narratives are likely to take over. Hawaii SGP Calculations 2021