Illinois 2020 Skip-Year SGP Analyses

Illinois Technical Advisory Committee · June 3, 2020

Illinois 2020 Skip-Year SGP Analyses — Illinois Technical Advisory Committee, June 3, 2020.
Published

June 3, 2020

Event
Technical advisory committee
Slides
11
Format
Converted from PowerPoint

This deck is an image-based copy of the original slides. Builds and animations are flattened; the slide text is reproduced below for search and screen readers.

Slide text
  1. Illinois 2020 Skip-Year SGP Analyses Damian Betebenner National Center for the Improvement of Educational Assessment Illinois State Board of Education (ISBE) Assessment and Accountability Technical Advisory Committee (TAC) June 4, 2020
  2. COVID-19 Pandemic • Oh what a difference a couple of months make. • Most Spring 2020 tests have been cancelled: – State assessments in ELA/Math have been cancelled – Many ELP assessments (WIDA/ACCESS) were completed, at least by most students in the state • States, including Illinois, have been given assessment and accountability waivers by ED for 2020. • Because student academic growth as measured by state summative assessments utilizes prior assessment data in its calculation, the impacts of the 2020 pandemic will have ripple effects in 2021. • My colleague Adam VanIwaarden and I wrote a blog post on this topic in April which can be found here: https://www.nciea.org/blog/sgp/issues-and- considerations-covid-19-pandemic-presents-measuring-student-growth [Presentation Title - Footer] 2
  3. 2021 SGP Calculation • I have discussed whether growth can be calculated in 2021 and whether it can be used for accountability (e.g., school or teacher) with more than two-dozen states. • Simple answer: Yes, SGPs can be calculated using 2019 (and earlier data) as priors. – Several states have been calculating two-year growth for years when there is not annual sequence of tests. For example, 8th to 10th grade growth in ELA or Mathematics or End of Course testing. – Growth projections which yield Adequate Growth Percentiles (AGPs) will also be calculable in • The harder question is whether these SGPs can/should be used as part of state accountability. – Are two-year SGPs valid as indicators of annual growth? – Beyond technical viability, is it tenable (e.g., politically) to use two-year SGPs as a substitute for one-year SGPs? [Presentation Title - Footer] 3
  4. 2021 SGP Calculation • Are two-year SGPs valid as indicators of annual growth? – The paper on which this presentation is based shares some findings and will be discussed hereafter. • Is it tenable (e.g., politically) to use two-year SGPs as a substitute for one-year SGPs? – A much harder question that likely differs from state to state. – Answering it well requires technical due diligence which, hopefully, this report/presentation begins to address. [Presentation Title - Footer] 4
  5. Illinois SGP Calculation • Illinois, as a member of the PARCC consortium has had SGPs calculated since Spring of 2016. • Illinois assesses students in ELA and Mathematics in grade 3, 4, 5, 6, 7 and 8. • As mentioned previously, there is no summative data in 2020 due to the COVID-19. • To investigate the use of two-year SGPs in lieu of one- year SGPs, we used historical data as follows [Presentation Title - Footer] 5
  6. Illinois SGP Calculation • Using historical data from 2016 to 2019, two-year SGPs from 2017 to 2019 were calculated using 2019 as the dependent variable and 2016 and 2017 as the independent variables (i.e., order 1 and 2 SGPs) – Two-year SGPs are calculated for grades 5, 6, 7, 8 in ELA and Mathematics. – SGPs are calculated separately for each norm-group. • One-year SGPs from 2018-2019 were available from previous SGP analyses. – One-year SGPs are calculated for grades 4, 5, 6, 7, and 8. – SGPs are calculated separately for each norm-group. [Presentation Title - Footer] 6
  7. Illinois SGP Results • Frequencies associated with SGPs. • In general, there are between 135,000 and 140,000 students with SGPs in each grade and content area. • Note when more priors are used, the count decreases slightly due to some students not having the additional prior. [Presentation Title - Footer] 7
  8. Illinois SGP Results: Individual • Correlations between two-year SGPs and one-year SGPs (of order 2) were all high. – Correlations ranged from 0.85 to 0.9 across grades and content areas. – When one-year/order 1 SGPs are considered, correlations drop to 0.65 to 0.7. This is likely due to the fact that the 2017 score is not a part of both one- and two-year analyses. • Differences between individual SGPs can be large. [Presentation Title - Footer] 8
  9. Illinois SGP Results: School Level • School level data was not provided as part of Pearson data sets. We substituted another states data for this part until IL school data becomes available. • Correlations between two-year mean SGPs and one-year mean SGPs (of order 2) were all high, higher than at the individual level. – Correlations ranged from 0.9 to 0.95 across grades and content areas. • Differences between two- and one-year mean SGPs can be large. [Presentation Title - Footer] 9
  10. Illinois SGP Results: School Level • For context, the year-to-year correlations between mean school SGPs is approximately 0.5 to 0.6. • Model-to-model (i.e., one-year to two-year) SGP correlations are much higher than year-to-year correlations implying that changing student populations • Correlations of 0.9 are common in comparisons between SGP and VAM. • SGP differences of 5 correspond to an effect size of 0.18. – The majority of mean SGP difference correspond to small effect sizes. [Presentation Title - Footer] 10
  11. For more information: Center for Assessment www.nciea.org