Rhode Island Growth Analyses Using PSAT

Rhode Island Department of Education · September 2018

Rhode Island Growth Analyses Using PSAT — Rhode Island Department of Education, September 2018.
Published

September 11, 2018

Event
State agency
Slides
18
Format
Converted from Keynote

This deck is an image-based copy of the original slides. Builds and animations are flattened; the slide text is reproduced below for search and screen readers.

Slide text
  1. RHODE ISLAND GROWTH ANALYSES USING PSAT Damian W Betebenner National Center for the Improvement of Educational Assessment Providence, Rhode Island May 15th, 2018
  2. BACKGROUND • Rhode Island’s switch from PARCC to RI-CAS/PSAT10/SAT leads to changes in the way that growth is calculated. • In general, SGPs are using a students’ current test scores combined with at least one prior test score subject to the condition that scores come from the same test taken at the same time. • That is, test transitions like those happening in Rhode Island do not prevent the calculation of growth. However, communicating how growth is calculated across a test transition can be more difficult. • This presentation provides an overview of due diligence analyses conducted for PARCC to PSAT10.
  3. BACKGROUND • Despite the versatility of SGP analyses not requiring a common scale across years, problems can arise with analyses. • The most common problems are related to ceiling/floor issues on tests as well as too few score points on which to calculate the 99 conditional quantile curves. • All source code associated with these analyses (no data) can be found on the Center’s GitHub repository for Rhode Island analyses at: https:// github.com/CenterForAssessment/Rhode_Island
  4. 2016 ELA Growth Analyses PARCC G3 PARCC G4 PARCC G4 PARCC G5 PARCC G5 PARCC G6 Growth Calculations PARCC G6 PARCC G7 PARCC G7 PARCC G8 PARCC G8 PARCC G9 PARCC G9 PARCC G10
  5. 2016 Mathematics Growth Analyses PARCC G3 PARCC G4 PARCC G4 PARCC G5 PARCC G5 PARCC G6 Growth PARCC G6 PARCC G7 Calculations PARCC G7 PARCC G8 PARCC G7 PARCC ALGEBRA I PARCC G8 PARCC ALGEBRA I PARCC ALGEBRA I PARCC GEOMETRY
  6. 2017 ELA Growth Analyses PARCC G3 PARCC G4 PARCC G3 PARCC G4 PARCC G5 PARCC G4 PARCC G5 PARCC G6 Growth PARCC G5 PARCC G6 PARCC G7 Calculations PARCC G6 PARCC G7 PARCC G8 PARCC G7 PARCC G8 PARCC G9 PARCC G8 PARCC G9 PARCC G10 PARCC G9 PSAT10
  7. 2017 Mathematics Growth Analyses PARCC G3 PARCC G4 PARCC G3 PARCC G4 PARCC G5 PARCC G4 PARCC G5 PARCC G6 PARCC G5 PARCC G6 PARCC G7 2017 PARCC G6 PARCC G7 PARCC G8 Growth Calculations PARCC G6 PARCC G7 PARCC ALGEBRA I PARCC G7 PARCC G8 PARCC ALGEBRA I PARCC G7 PARCC ALGEBRA I PARCC GEOMETRY PARCC G8 PARCC ALGEBRA I PARCC GEOMETRY PARCC G8 PARCC ALGEBRA I PSAT 10 PARCC ALGEBRA I PARCC GEOMETRY PSAT 10 Note: Several analyses using PSAT 10 in 2018 were conducted including: PARCC ALGEBRA I 2015 to PSAT 10 2017 without the GEOMETRY in 2016
  8. 2018 ELA Growth Analyses PARCC G3 RICAS G4 PARCC G3 PARCC G4 RICAS G5 2018 PARCC G4 PARCC G5 RICAS G6 Growth Calculations PARCC G5 PARCC G6 RICAS G7 PARCC G6 PARCC G7 RICAS G8 PARCC G9 PSAT10 PARCC G8 PSAT10 Note: Several analyses using PSAT 10 in 2018 were conducted including: PARCC G8 (2016) to PSAT 10, PARCC G10 (2017) to PSAT 10
  9. 2018 Mathematics Growth Analyses PARCC G3 RICAS G4 PARCC G3 PARCC G4 RICAS G5 PARCC G4 PARCC G5 RICAS G6 Growth PARCC G5 PARCC G6 RICAS G7 Calculations PARCC G6 PARCC G7 RICAS G8 PARCC ALGEBRA I PSAT10 PARCC GEOMETRY PSAT 10 PARCC G8 PSAT 10
  10. 2019 ELA Growth Analyses (Proposed) RICAS G3 RICAS G4 PARCC G3 RICAS G4 RICAS G5 2019 PARCC G4 RICAS G5 RICAS G6 Growth Calculations PARCC G5 RICAS G6 RICAS G7 PARCC G6 RICAS G7 RICAS G8 PSAT 10 SAT PARCC G8 PSAT10
  11. 2019 Mathematics Growth Analyses (Proposed) RICAS G3 RICAS G4 PARCC G3 RICAS G4 RICAS G5 Growth PARCC G4 RICAS G5 RICAS G6 Calculations PARCC G5 RICAS G6 RICAS G7 PARCC G6 RICAS G7 RICAS G8 PSAT 10 SAT
  12. ANALYSES -2017 • All analyses using 2017 PSAT data ran without any analytic issues • No ceiling or floor effects were observed in any of the analyses. • No issues related to “granularity” of scale were observed.
  13. ANALYSES -2018 • All RICAS and PSAT/SAT analyses ran without any analytic issues. Analyses conducted included cohort referenced and measurement error corrected (SIMEX adjusted) SGPs as well and growth to standard projections. • No ceiling or floor effects were observed in any of the analyses. • No issues related to “granularity” of scale were observed. • Following discussions with another state regarding year-to-year changes after transition to PSAT/SAT, an examination of cross-year.
  14. ANALYSES -2018: POSSIBLE PSAT/SAT BIAS • Colorado noticed large changes in district level median SGPs in ELA and Math after transitioning from PARCC Grade 9 & 10 to PSAT/9, PSAT/10, SAT • Upon further examination in Colorado, other large changes emerged most prominently large changes in growth depending upon the gender of the student. • For example, in 2017, growth from ELA Grade 8 to ELA Grade 9 had median SGPs of 50/51 for males/females. In 2018 from ELA Grade 8 to PSAT 9 median SGPs were 56/44 for males/females. • Within several large districts that saw changes in median SGP, males went from underperforming females (median SGP) to outperforming. • The result is reminiscent of the paper versus online issue that emerged in PARCC.
  15. ANALYSES -2018: POSSIBLE PSAT/SAT BIAS • Rhode Island doesn’t have the exact same assessment transition as Colorado but is transitioning away from PARCC and including PSAT/SAT assessments in high school. • Preliminary analysis of Rhode Island data shows similar differences as Colorado: Cohort Median SGP Males Median SGP Females Median SGP Overall 2014_2015/ELA_9; 2015_2016/ELA_10 46 (3,863) 54 (3,480) 50 2015_2016/ELA_9; 2016_2017/ PSAT_EBRW_EOCT 2016_2017/ELA_PSAT_10_EOCT; 2017_2018/ELA_SAT_EOCT 2016_2017/ELA_9; 2017_2018/ ELA_PSAT_10_EOCT
  16. ANALYSES -2018: POSSIBLE PSAT/SAT BIAS • Historically, for ELA, across all grades for which growth is calculated, females consistently demonstrate equal or slightly higher growth than males. • As the previous results show, this pattern reverses dramatically when the state switched from PARCC to PSAT/SAT. • Note that the results diminish when the PSAT/SAT become both the independent (prior) and dependent (current) variable. • The optimal window to detect such bias occurs in the year that the test is switched (the first row in the previous table) • The results support recent research by Reardon et al. (2018) on achievement gender gaps and test item format. Education Researcher, Vol 47, No. 5, pp. 284-294
  18. 2018 RICAS ELA: Grade 4, 5, 6, 7, 8 Student Growth Percentile Goodness−of−Fit Descriptives Student Growth Percentile Goodness−of−Fit Descriptives Student Growth Percentile Goodness−of−Fit Descriptives 2017−2018 ELA SGP, Grade 4 (N = 10,044) 2017−2018 ELA SGP, Grade 5 (N = 10,250) 2017−2018 ELA SGP, Grade 6 (N = 9,871) SGP Deciles by Prior Achievement Level (ELA) SGP Deciles by Prior Achievement Level (ELA) SGP Deciles by Prior Achievement Level (ELA) Level 5 (3.5%) Level 5 (7.4%) Level 5 (3%) Prior Achievement Level Prior Achievement Level Prior Achievement Level Level 4 (37%) Level 4 (33.7%) Level 4 (38.2%) Level 3 (22.4%) Level 3 (27.3%) Level 3 (27.5%) Level 2 (19.5%) Level 2 (18.2%) Level 2 (18.7%) Level 1 (17.5%) Level 1 (13.4%) Level 1 (12.5%) Median Student Growth Percentile Median Student Growth Percentile Median Student Growth Percentile Ceiling / Floor Effects Test Ceiling / Floor Effects Test Ceiling / Floor Effects Test Student Growth Percentile Range Student Growth Percentile Range Student Growth Percentile Range Current Observed Score Current Observed Score Current Observed Score 5 9 to to to to to to to to to to 5 9 to to to to to to to to to to 5 9 to to to to to to to to to to to to to to 10 20 30 40 50 60 70 80 90 95 to to Lowest OSS: [−3.15,−3.13] 78.4 21.6 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 N = 37 Lowest OSS: [−3.34,−3.27] 81.8 15.2 3.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 N = 33 Lowest OSS: [−3.17,−3.1] 82.6 13.0 4.3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 N = 23 Extremes (Range) Extremes (Range) Extremes (Range) Highest OSS: [ 2.7382 ] 2.3 0.0 0.0 0.0 0.0 0.0 2.3 2.3 7.0 7.0 16.3 62.8 N = 43 Highest OSS: [ 2.555 ] 2.0 0.0 0.0 0.0 0.0 0.0 0.0 3.9 3.9 11.8 13.7 64.7 N = 51 Highest OSS: [2.512,2.648) 0.0 0.0 0.0 0.0 0.0 0.0 2.4 7.3 7.3 17.1 7.3 58.5 N = 41 Student Growth Percentile Range QQ−Plot: Student Growth Percentiles Student Growth Percentile Range QQ−Plot: Student Growth Percentiles Student Growth Percentile Range QQ−Plot: Student Growth Percentiles 19 29 39 49 59 69 79 89 99 Theoretical SGP Distribution 19 29 39 49 59 69 79 89 99 Theoretical SGP Distribution 19 29 39 49 59 69 79 89 99 Theoretical SGP Distribution 9 to to to to to to to to to 9 to to to to to to to to to 9 to to to to to to to to to to 10 20 30 40 50 60 70 80 90 0 10 20 30 40 50 60 70 80 90 100 1 to 10 20 30 40 50 60 70 80 90 0 10 20 30 40 50 60 70 80 90 100 1 to 10 20 30 40 50 60 70 80 90 0 10 20 30 40 50 60 70 80 90 100 100 1st/[−5.5,−1.5) 9.71 10.98 10.88 9.71 10.29 10.39 9.61 8.53 9.41 10.49 (10%) 100 100 1st/[−15,−1.367) 10.77 11.08 10.26 10.97 9.74 10.05 9.33 9.54 8.92 9.33 (9.7%) 1st/[−5.624,−1.488) 9.33 10.71 10.82 9.23 9.54 10.18 10.07 9.33 9.86 10.92 (9.6%) 2nd/[−1.367,−0.8713) 8.53 10.10 9.61 9.02 9.90 9.71 10.20 10.20 10.39 12.35 (10.2%) 2nd/[−1.5,−0.96) 9.17 9.56 9.27 10.44 9.46 9.27 8.68 12.39 11.22 10.54 (10%) 2nd/[−1.488,−0.911) 9.74 9.35 8.96 10.81 10.22 9.64 10.13 10.13 9.93 11.10 (10.4%) Prior Scale Score Decile/Range * Prior Scale Score Decile/Range * Prior Scale Score Decile/Range * 3rd/[−0.8713,−0.4928) 8.13 7.42 9.55 9.25 10.16 12.20 9.55 10.98 11.99 10.77 (9.8%) 3rd/[−0.96,−0.56) 9.74 10.34 9.74 10.24 9.84 9.74 12.23 8.35 9.54 10.24 (9.8%) 3rd/[−0.911,−0.511) 8.71 10.33 10.84 10.13 10.13 9.52 9.63 11.04 10.64 9.02 (10%) Empirical SGP Distribution Empirical SGP Distribution Empirical SGP Distribution 4th/[−0.4928,−0.178) 10.34 9.84 11.05 10.34 11.26 10.95 9.94 8.62 6.90 10.75 (9.8%) 4th/[−0.56,−0.24) 10.88 9.71 10.29 10.20 10.00 10.10 8.82 8.92 10.10 10.98 (10%) 4th/[−0.511,−0.1947) 9.31 10.26 9.31 11.01 9.31 10.79 10.58 8.89 10.16 10.37 (9.6%) 5th/[−0.178,0.1279) 10.46 12.65 9.26 9.96 8.67 8.37 8.47 11.95 11.35 8.86 (10%) 5th/[−0.24,0.05) 8.59 9.58 10.27 9.08 10.17 11.06 9.28 11.25 9.58 11.15 (9.9%) 5th/[−0.1947,0.1056) 10.23 9.83 10.33 9.42 9.63 10.13 10.13 9.93 9.42 10.94 (10%) 6th/[0.1279,0.399) 8.79 10.79 8.79 11.59 9.69 8.89 9.99 9.09 10.89 11.49 (10%) 6th/[0.05,0.31) 8.28 10.06 10.45 10.55 10.45 8.68 10.06 11.05 10.06 10.36 (9.9%) 6th/[0.1056,0.402) 9.39 9.97 9.10 10.26 10.65 9.97 9.87 11.04 9.58 10.16 (10.5%) 7th/[0.399,0.7032) 9.88 8.10 11.36 8.50 11.86 10.77 10.87 9.68 7.91 11.07 (10.1%) 7th/[0.31,0.6) 10.32 10.32 9.09 10.70 9.75 10.89 10.13 9.38 10.04 9.38 (10.3%) 7th/[0.402,0.6637) 10.49 9.97 10.39 9.55 9.86 10.07 10.28 8.71 10.39 10.28 (9.7%) 8th/[0.7032,1.04) 8.86 9.90 10.67 9.43 9.62 11.14 8.67 12.38 9.90 9.43 (10.5%) 8th/[0.6,0.94) 9.27 10.06 10.06 9.47 9.37 9.86 10.95 8.48 10.45 12.03 (9.9%) 8th/[0.6637,1.014) 8.57 10.56 10.46 11.55 9.46 9.76 9.26 9.86 9.46 11.06 (10.2%) 9th/[1.04,1.46) 10.03 9.63 8.61 10.44 8.31 10.64 10.03 9.32 11.35 11.65 (9.8%) 9th/[0.94,1.4) 10.07 10.36 10.65 9.68 8.71 10.07 9.87 10.26 9.20 11.13 (10.1%) 9th/[1.014,1.491) 10.28 8.97 9.07 8.47 10.99 9.48 10.79 10.28 10.89 10.79 (10%) 10th/[1.46,6.139] 9.46 10.34 10.63 10.34 10.34 9.56 10.34 9.66 9.17 10.15 (10.2%) 10th/[1.4,6.9] 9.34 9.15 9.91 10.39 10.49 10.20 9.63 10.49 10.58 9.82 (10.2%) 10th/[1.491,4.36] 9.30 10.50 9.90 10.40 9.80 10.30 10.00 10.00 9.70 10.10 (10.1%) *Prior score deciles can be uneven depending upon the prior score distribution *Prior score deciles can be uneven depending upon the prior score distribution *Prior score deciles can be uneven depending upon the prior score distribution Student Growth Percentile Goodness−of−Fit Descriptives Student Growth Percentile Goodness−of−Fit Descriptives 2017−2018 ELA SGP, Grade 7 (N = 9,876) 2017−2018 ELA SGP, Grade 8 (N = 10,051) SGP Deciles by Prior Achievement Level (ELA) SGP Deciles by Prior Achievement Level (ELA) Level 5 (5.4%) Level 5 (11%) Prior Achievement Level Prior Achievement Level Level 4 (32.3%) Level 4 (28.4%) Level 3 (28.5%) Level 3 (24.7%) Level 2 (18.3%) Level 2 (22.1%) Level 1 (11.7%) Level 1 (17.5%) Median Student Growth Percentile Median Student Growth Percentile Ceiling / Floor Effects Test Ceiling / Floor Effects Test Student Growth Percentile Range Student Growth Percentile Range Current Observed Score Current Observed Score 5 9 to to to to to to to to to to 5 9 to to to to to to to to to to to to 10 20 30 40 50 60 70 80 90 95 to to 10 20 30 40 50 60 70 80 90 95 Lowest OSS: [−3.12,−3.08] 57.7 11.5 23.1 7.7 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 N = 26 Lowest OSS: [−2.961,−2.957] 18.4 18.4 26.5 24.5 10.2 2.0 0.0 0.0 0.0 0.0 0.0 0.0 N = 49 Extremes (Range) Extremes (Range) Highest OSS: [ 2.251 ] 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.6 10.5 10.5 23.7 52.6 N = 38 Highest OSS: [2.342,2.622) 0.0 0.0 0.0 0.0 0.0 0.0 0.0 7.3 12.2 29.3 12.2 39.0 N = 41 Student Growth Percentile Range QQ−Plot: Student Growth Percentiles Student Growth Percentile Range QQ−Plot: Student Growth Percentiles 19 29 39 49 59 69 79 89 99 Theoretical SGP Distribution 19 29 39 49 59 69 79 89 99 Theoretical SGP Distribution 9 to to to to to to to to to 9 to to to to to to to to to to 10 20 30 40 50 60 70 80 90 0 10 20 30 40 50 60 70 80 90 100 1 to 10 20 30 40 50 60 70 80 90 0 10 20 30 40 50 60 70 80 90 100 1st/[−4.891,−1.474) 7.42 14.63 10.60 9.97 9.65 9.97 9.33 9.65 9.01 9.76 (9.5%) 100 1st/[−5.661,−1.688) 4.27 15.42 13.85 10.42 8.23 8.44 9.69 9.69 9.69 10.31 (9.6%) 100 2nd/[−1.474,−1.014) 12.55 7.02 9.29 10.28 9.68 9.88 9.88 9.68 10.18 11.56 (10.2%) 2nd/[−1.688,−1.106) 12.51 7.83 8.69 9.17 10.60 9.36 10.89 10.32 10.70 9.93 (10.4%) Prior Scale Score Decile/Range * Prior Scale Score Decile/Range * 3rd/[−1.014,−0.6135) 8.09 8.69 10.59 10.39 10.59 10.99 9.49 10.99 10.79 9.39 (10.1%) 3rd/[−1.106,−0.7056) 8.14 8.54 10.85 10.45 10.25 11.56 10.05 9.85 9.85 10.45 (9.9%) Empirical SGP Distribution Empirical SGP Distribution 4th/[−0.6135,−0.285) 8.17 11.44 10.52 9.81 10.21 9.40 9.91 9.91 10.01 10.62 (9.9%) 4th/[−0.7056,−0.3461) 9.56 10.44 8.97 10.94 9.85 9.46 9.46 10.64 10.44 10.25 (10.1%) 5th/[−0.285,0.0344) 11.94 9.62 9.92 9.72 9.11 10.02 10.43 8.81 9.72 10.73 (10%) 5th/[−0.3461,−0.0668) 10.54 11.05 9.61 9.50 11.05 9.61 10.43 8.99 8.78 10.43 (9.6%) 6th/[0.0344,0.3365) 9.82 10.24 9.82 10.13 9.31 10.44 9.82 11.17 9.51 9.72 (9.8%) 6th/[−0.0668,0.2107) 8.49 10.46 10.46 8.98 8.49 10.07 9.28 10.66 11.94 11.15 (10.1%) 7th/[0.3365,0.6398) 8.78 9.37 10.65 9.86 10.75 9.47 9.27 9.07 11.14 11.64 (10.3%) 7th/[0.2107,0.4981) 9.95 8.99 9.95 11.40 10.82 10.43 9.86 9.57 9.57 9.47 (10.3%) 8th/[0.6398,0.9782) 8.78 9.90 9.59 9.49 10.92 10.82 11.33 9.90 8.57 10.71 (9.9%) 8th/[0.4981,0.8308) 8.56 10.50 9.38 9.38 10.60 9.99 9.79 10.40 9.58 11.82 (9.8%) 9th/[0.9782,1.463) 10.46 9.85 9.14 11.27 8.83 10.86 9.14 10.66 9.44 10.36 (10%) 9th/[0.8308,1.308) 10.47 9.68 10.67 10.28 8.99 10.08 9.98 9.98 10.28 9.58 (10.1%) 10th/[1.463,6.122] 9.04 10.03 10.53 9.53 10.82 9.33 10.03 9.73 10.53 10.43 (10.2%) 10th/[1.308,5.236] 9.17 9.95 9.46 10.05 10.34 10.05 9.76 9.95 10.15 11.12 (10.2%) *Prior score deciles can be uneven depending upon the prior score distribution *Prior score deciles can be uneven depending upon the prior score distribution