An Investigation of 2024 PSAT/SAT Results

Rhode Island Technical Advisory Committee · July 30, 2024

An Investigation of 2024 PSAT/SAT Results — Rhode Island Technical Advisory Committee, July 30, 2024.
Published

July 30, 2024

Event
Technical advisory committee
Slides
22
Format
Converted from Keynote

This deck is an image-based copy of the original slides. Builds and animations are flattened; the slide text is reproduced below for search and screen readers.

Slide text
  1. An Investigation of 2024 PSAT/SAT Results Damian Betebenner The National Center for the Improvement of Educational Assessment July 30th, 2024 RIDE TAC Meeting This work is licensed under a Creative Commons Attribution 4.0 International License.
  2. Background
  3. PSAT/SAT Test Administration • Since 2017 the Rhode Island Department of Education has administered the PSAT as part of the spring test administration to 10th graders • Since 2018 the Rhode Island Department of Education has administered the SAT to 11th graders. • Both the PSAT and SAT include sections in ELA and Mathematics.
  4. Transition to Digital SAT • In 2024, the College Board transitioned to the Digital SAT administration platform. • Several aspects of the assessment changed during this transition in terms of both user facing features (computer versus P&P administration) as well as backend features (adaptive versus fixed- form). • Changes in the results in 2024 from 2023 led to concern amongst some states about test equivalence. • The analyses in this presentation are intended to support discussions about the equivalence of tests and comparability of results.
  5. Summary PSAT/SAT Data
  6. Introduction • We examine student performance on the PSAT/SAT in several ways • Participation frequencies • Mean/Standard Deviations of Scores by year • Mean/Median Baseline SGP by year (SAT only) • Baseline SGPs are normed using pre-pandemic, PSAT-SAT growth norms. Pandemic Related Academic Recovery in Rhode Island 6
  7. Rhode Island PSAT/SAT Participation PSAT-ELA PSAT-Math SAT-ELA SAT-Math
  8. Frequencies • In general, there was a substantial decrease in participation due to the pandemic. • For the PSAT, the number of students tested has roughly returned to pre- pandemic level. • For the SAT, in 2024 there was a substantial (approximate 8%) increase in the number of students tested. Pandemic Related Academic Recovery in Rhode Island 8
  9. Rhode Island Scaled Score Mean/SD PSAT-ELA PSAT-Math SAT-ELA SAT-Math
  10. Density Plot: ELA PSAT ELA PSAT 10 Density 0.003 Scale Score
  11. Density Plot: Mathematics PSAT MATHEMATICS PSAT 10 Density 0.003 Scale Score
  12. Density Plot: ELA SAT ELA SAT Density 0.003 Scale Score
  13. Density Plot: Mathematics SAT MATHEMATICS SAT Density 0.003 Scale Score
  14. Summary Measures • Both ELA and Mathematics PSAT saw decreases in mean scaled score over the last six years. The drop in mathematics in 2023-2024 from 436.4 to 418.4 was particularly large. • Both ELA and Mathematics PSAT saw increases in the standard deviation in 2023-2024 from prior years. • Both ELA and Mathematics SAT saw decreases in mean scaled score, particularly in 2023-2024. Mathematics dropped from 461.5 to 445.9. • Both ELA and Mathematics SAT saw increases in standard deviation in • The number of LOSS (200) scores in the SAT saw a large increase in Pandemic Related Academic Recovery in Rhode Island 14
  15. Rhode Island: Mean SGP/Count SAT-ELA SAT-Math
  16. Student Growth Percentile Goodness−of−Fit Descriptives 2023−2024 ELA SAT SGP, Grade EOCT (N = 9,256) SGP Deciles by Prior Achievement Level (ELA PSAT 10) Exceeding (12.1%) Prior Achievement Level Meeting (45.9%) Partially Meeting (22.6%) Not Meeting (19.4%) Median Student Growth Percentile Ceiling / Floor Effects Test Student Growth Percentile Range Current Observed Score 5 9 to to to to to to to to to to to to 10 20 30 40 50 60 70 80 90 95 Lowest OSS: [ 200 ] 100.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 N = 220 Extremes (Range) Highest OSS: [760,770) 0.0 0.0 0.0 0.0 0.0 0.0 0.0 16.7 8.3 16.7 33.3 25.0 N = 24 Student Growth Percentile Range QQ−Plot: Student Growth Percentiles 19 29 39 49 59 69 79 89 99 Theoretical SGP Distribution 9 to to to to to to to to to to 10 20 30 40 50 60 70 80 90 0 10 20 30 40 50 60 70 80 90 100 1st/[160,340) 26.27 11.21 7.59 6.91 7.47 5.78 9.29 7.47 8.38 9.63 (9.5%) 100 2nd/[340,370) 18.54 7.20 10.58 10.14 6.65 9.92 9.49 10.47 7.20 9.81 (9.9%) Prior Scale Score Decile/Range * 3rd/[370,390) 13.65 9.91 5.89 14.19 11.38 7.23 7.23 10.31 9.24 10.98 (8.1%) Empirical SGP Distribution 4th/[390,420) 9.14 7.75 13.31 8.14 7.65 14.90 8.84 9.14 10.43 10.72 (10.9%) 5th/[420,450) 9.05 8.04 12.46 11.36 11.86 8.64 8.64 7.64 13.27 9.05 (10.7%) 6th/[450,480) 11.23 10.36 11.01 14.07 7.09 7.20 7.63 14.07 9.27 8.07 (9.9%) 7th/[480,510) 11.63 6.66 10.99 13.11 8.77 10.36 7.51 13.21 8.67 9.09 (10.2%) 8th/[510,540) 8.53 12.14 6.98 8.91 9.56 11.11 12.66 6.98 10.08 13.05 (8.4%) 9th/[540,600) 12.33 10.46 9.92 9.65 6.34 9.47 8.40 6.97 8.40 18.05 (12.1%) 10th/[600,760] 10.83 8.20 7.89 9.04 8.31 8.73 10.09 10.73 12.20 13.99 (10.3%) *Prior score deciles can be uneven depending upon the prior score distribution
  17. Student Growth Percentile Goodness−of−Fit Descriptives 2023−2024 Math SAT SGP, Grade EOCT (N = 9,284) SGP Deciles by Prior Achievement Level (Mathematics PSAT 10) Exceeding (3.4%) Meeting (25.7%) Prior Achievement Level Partially Meeting (49.5%) Not Meeting (21.4%) Median Student Growth Percentile Ceiling / Floor Effects Test Student Growth Percentile Range Current Observed Score 5 9 to to to to to to to to to to to to 10 20 30 40 50 60 70 80 90 95 Lowest OSS: [ 200 ] 99.5 0.5 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 N = 203 Extremes (Range) Highest OSS: [750,760) 0.0 0.0 0.0 13.3 0.0 0.0 13.3 6.7 0.0 26.7 13.3 26.7 N = 15 Student Growth Percentile Range QQ−Plot: Student Growth Percentiles 19 29 39 49 59 69 79 89 99 Theoretical SGP Distribution 9 to to to to to to to to to to 10 20 30 40 50 60 70 80 90 0 10 20 30 40 50 60 70 80 90 100 1st/[160,350) 12.26 5.69 9.48 6.07 6.57 7.59 13.65 16.69 9.23 12.77 (8.5%) 100 2nd/[350,370) 12.85 5.34 6.32 12.45 5.53 14.62 17.39 7.51 9.29 8.70 (5.5%) Prior Scale Score Decile/Range * 3rd/[370,400) 13.29 8.10 9.99 5.67 14.91 13.29 10.73 10.19 7.62 6.21 (16%) Empirical SGP Distribution 4th/[400,420) 13.91 14.25 15.62 15.51 13.68 6.04 4.79 7.87 3.19 5.13 (9.4%) 5th/[420,430) 15.46 11.13 13.58 16.47 14.74 6.65 4.91 6.79 5.78 4.48 (7.5%) 6th/[430,450) 19.68 13.12 16.20 12.05 11.65 5.76 7.90 3.88 5.35 4.42 (8%) 7th/[450,470) 19.94 18.73 13.70 13.23 5.31 7.83 6.06 4.57 4.01 6.62 (11.6%) 8th/[470,500) 20.57 11.11 16.86 7.70 10.82 8.09 3.80 6.24 6.82 7.99 (11.1%) 9th/[500,540) 12.99 11.71 9.25 7.97 7.87 8.07 10.73 7.09 11.81 12.50 (10.9%) 10th/[540,760] 6.15 7.54 5.87 10.06 9.59 10.06 11.36 14.99 11.92 12.48 (11.6%) *Prior score deciles can be uneven depending upon the prior score distribution
  18. Baseline SGPs • Because growth conditions on prior scores, baseline SGPs are much more robust to changes in the population of students tested than status measures. • Baseline SGPs were fairly stable across the last several years in ELA, less so in mathematics. The drop in the mean SGP to 44.4 in 2023-2024 in mathematics is synonymous with the drop in achievement observed. • The low median SGP in mathematics was caused (largely) by low SGPs of students in the 6th, 7th, and 8th prior score deciles. • The Center works with another state using the PSAT/SAT tests in high school and observed a very similar pattern (as evidenced by the following two goodness of fit plots) Pandemic Related Academic Recovery in Rhode Island 18
  19. Student Growth Percentile Goodness−of−Fit Descriptives 2024 SAT ELA SGP, Grade 11 (N = 49,466) SGP Deciles by Prior Achievement Level (ELA PSAT 10) Level 5 (1.5%) Level 4 (13.7%) Prior Achievement Level Level 3 (52.4%) Level 2 (13%) Level 1 (19.5%) Median Student Growth Percentile Ceiling / Floor Effects Test Student Growth Percentile Range Current Observed Score 5 9 to to to to to to to to to to to to 10 20 30 40 50 60 70 80 90 95 Lowest OSS: [ 200 ] 100.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 N = 416 Extremes (Range) Highest OSS: [ 780 ] 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 6.4 19.1 10.6 63.8 N = 47 Student Growth Percentile Range QQ−Plot: Student Growth Percentiles 19 29 39 49 59 69 79 89 99 Theoretical SGP Distribution 9 to to to to to to to to to to 10 20 30 40 50 60 70 80 90 0 10 20 30 40 50 60 70 80 90 100 1st/[160,350) 19.45 7.93 8.09 6.68 7.61 9.58 10.11 9.47 9.58 11.50 (7.6%) 100 2nd/[350,390) 13.06 6.36 5.93 13.18 8.66 10.02 11.16 10.72 9.54 11.37 (11.9%) Prior Scale Score Decile/Range * 3rd/[390,420) 7.73 8.59 10.57 7.50 13.26 10.42 7.29 11.43 10.40 12.82 (9.6%) Empirical SGP Distribution 4th/[420,450) 9.09 10.15 11.04 7.06 8.99 9.50 10.96 10.47 10.41 12.34 (9.5%) 5th/[450,480) 8.24 11.14 11.44 11.07 10.04 8.63 9.96 10.14 9.27 10.08 (10.2%) 6th/[480,510) 8.61 8.93 10.99 12.89 12.50 9.12 7.91 12.22 6.79 10.05 (10.3%) 7th/[510,540) 8.16 12.86 9.60 8.63 9.24 16.73 6.99 7.22 8.14 12.44 (9.6%) 8th/[540,570) 12.45 12.52 6.50 12.00 10.61 7.80 7.03 7.32 10.61 13.16 (8.9%) 9th/[570,620) 13.33 10.67 12.24 8.29 6.99 6.94 10.79 7.73 10.23 12.79 (11.2%) 10th/[620,750] 11.18 9.50 7.48 8.47 9.07 10.51 11.68 10.24 11.23 10.64 (11.2%) *Prior score deciles can be uneven depending upon the prior score distribution
  20. Student Growth Percentile Goodness−of−Fit Descriptives 2024 SAT Math SGP, Grade 11 (N = 49,466) SGP Deciles by Prior Achievement Level (Mathematics PSAT 10) Level 5 (0.5%) Level 4 (6%) Prior Achievement Level Level 3 (34.1%) Level 2 (23.3%) Level 1 (36.1%) Median Student Growth Percentile Ceiling / Floor Effects Test Student Growth Percentile Range Current Observed Score 5 9 to to to to to to to to to to to to 10 20 30 40 50 60 70 80 90 95 Lowest OSS: [ 200 ] 100.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 N = 362 Extremes (Range) Highest OSS: [ 790 ] 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.7 13.0 29.6 53.7 N = 54 Student Growth Percentile Range QQ−Plot: Student Growth Percentiles 19 29 39 49 59 69 79 89 99 Theoretical SGP Distribution 9 to to to to to to to to to to 10 20 30 40 50 60 70 80 90 0 10 20 30 40 50 60 70 80 90 100 1st/[160,370) 7.24 6.79 3.31 7.06 7.53 11.54 14.26 15.26 14.46 12.54 (9.9%) 100 2nd/[370,390) 6.70 7.64 4.73 12.45 7.98 9.27 19.37 10.86 11.96 9.04 (5.3%) Prior Scale Score Decile/Range * 3rd/[390,420) 9.12 10.01 13.21 13.79 14.90 10.03 7.33 6.92 6.98 7.71 (13.8%) Empirical SGP Distribution 4th/[420,430) 12.63 9.71 15.76 15.71 12.20 5.94 9.77 7.85 4.09 6.35 (7%) 5th/[430,460) 15.58 17.25 13.21 10.00 12.20 8.33 4.88 6.04 4.21 8.30 (13.1%) 6th/[460,480) 17.92 18.09 12.25 7.25 9.19 5.73 7.86 4.73 7.30 9.69 (10.3%) 7th/[480,500) 17.65 16.20 9.25 10.99 7.58 6.31 6.39 8.19 7.32 10.12 (7.7%) 8th/[500,520) 10.89 13.50 8.19 12.76 7.35 7.76 6.88 6.81 10.60 15.26 (8.5%) 9th/[520,560) 8.33 7.74 11.05 7.09 7.53 9.03 10.72 9.92 12.52 16.07 (11.6%) 10th/[560,760] 4.15 5.21 5.41 6.26 11.23 11.12 11.12 13.76 13.49 18.26 (12.9%) *Prior score deciles can be uneven depending upon the prior score distribution
  21. Conclusions • Results comparing 2023-2024 SAT results with prior years indicates discrepancies in mathematics which suggest assessment changes may have • In general, assessment changes are always accompanied by score discrepancies such as those observed here. • Drawing firm causal conclusions isn’t substantiated with these secondary analyses. However, further investigations into digital SAT test format — particularly for higher achieving students where lower growth was observed — may lead to causal factors leading to lower scores. Pandemic Related Academic Recovery in Rhode Island 21
  22. Questions