An Investigation of 2024 PSAT/SAT Results

Rhode Island Technical Advisory Committee · July 30, 2024

An Investigation of 2024 PSAT/SAT Results — Rhode Island Technical Advisory Committee, July 30, 2024.
Published

July 30, 2024

Event
Technical advisory committee
Slides
22
Format
Converted from Keynote

This deck is an image-based copy of the original slides. Builds and animations are flattened; the slide text is reproduced below for search and screen readers.

Slide text
  1. An Investigation of 2024 PSAT/SAT Results Damian Betebenner The National Center for the Improvement of Educational Assessment July 30th, 2024 RIDE TAC Meeting This work is licensed under a Creative Commons Attribution 4.0 International License.
  2. Background
  3. PSAT/SAT Test Administration • Since 2017 the Rhode Island Department of Education has administered the PSAT as part of the spring test administration to 10th graders • Since 2018 the Rhode Island Department of Education has administered the SAT to 11th graders. • Both the PSAT and SAT include sections in ELA and Mathematics.
  4. Transition to Digital SAT • In 2024, the College Board transitioned to the Digital SAT administration platform. • Several aspects of the assessment changed during this transition in terms of both user-facing features (computer versus P&P administration) as well as backend features (adaptive versus fixed- form). • Changes in the results in 2024 from 2023 led to concern amongst some states about test equivalence. • The analyses in this presentation are intended to support discussions about the equivalence of tests and comparability of results.
  5. Summary PSAT/SAT Data
  6. Introduction • We examine student performance on the PSAT/SAT in several ways • Participation frequencies • Mean/Standard Deviations of Scores by year • Mean/Median Baseline SGP by year (SAT only) • Baseline SGPs are normed using pre-pandemic, PSAT-SAT growth norms.
  7. Rhode Island PSAT/SAT Participation PSAT-ELA PSAT-Math SAT-ELA SAT-Math
  8. Frequencies • In general, there was a substantial decrease in participation due to the pandemic. • For the PSAT, the number of students tested has roughly returned to pre-pandemic level. • For the SAT, in 2024 there was a substantial (approximate 8%) increase in the number of students tested.
  9. Rhode Island Scaled Score Mean/SD PSAT-ELA PSAT-Math SAT-ELA SAT-Math
  10. Density Plot: ELA PSAT ELA PSAT 10 Density 0.003 Scale Score
  11. Density Plot: Mathematics PSAT MATHEMATICS PSAT 10 Density 0.003 Scale Score
  12. Density Plot: ELA SAT
  13. Density Plot: Mathematics SAT
  14. Summary Measures • Both ELA and Mathematics PSAT saw decreases in mean scaled scores over the last six years. The drop in mathematics in 2023-2024 from 436.4 to 418.4 was particularly large. • Both ELA and Mathematics PSAT saw increases in the standard deviation in 2023-2024 from prior years. • Both ELA and Mathematics SAT saw decreases in mean scaled scores, particularly in 2023-2024. Mathematics dropped from 461.5 to 445.9. • Both ELA and Mathematics SAT saw increases in standard deviation in • The number of LOSS (200) scores in the SAT saw a large increase in
  15. Rhode Island: Mean SGP/Count SAT-ELA SAT-Math
  18. Baseline SGPs • Because growth is conditioned on prior scores, baseline SGPs are much more robust to changes in the population of students tested than status measures. • Baseline SGPs were fairly stable across the last several years in ELA, less so in mathematics. The drop in the mean SGP to 44.4 in 2023-2024 in mathematics is synonymous with the drop in achievement observed. • The low median SGP in mathematics was caused (largely) by low SGPs of students in the 6th, 7th, and 8th prior score deciles. • The Center works with another state using the PSAT/SAT tests in high school and observed a very similar pattern (as evidenced by the following two goodness of fit plots)
  21. Conclusions • Results comparing 2023-2024 SAT results with prior years indicate discrepancies in mathematics which suggest assessment changes may have • In general, assessment changes are always accompanied by score discrepancies such as those observed here. • Causal conclusions are not substantiated by these secondary analyses. However, further investigations into the digital SAT format — particularly for higher achieving students where lower growth was observed — may suggest causal factors for lower scores.
  22. Questions