RHODE ISLAND GROWTH ANALYSES USING PSAT
Damian W Betebenner
National Center for the Improvement of Educational Assessment
Providence, Rhode Island May 15th, 2018
BACKGROUND
• Rhode Island’s switch from PARCC to RI-CAS/PSAT10/SAT leads to
changes in the way that growth is calculated.
• In general, SGPs are using a students’ current test scores combined with at
least one prior test score subject to the condition that scores come from
the same test taken at the same time.
• That is, test transitions like those happening in Rhode Island do not
prevent the calculation of growth. However, communicating how growth is
calculated across a test transition can be more difficult.
• This presentation provides an overview of due diligence analyses
conducted for PARCC to PSAT10.
BACKGROUND
• Despite the versatility of SGP analyses not requiring a common scale
across years, problems can arise with analyses.
• The most common problems are related to ceiling/floor issues on tests as
well as too few score points on which to calculate the 99 conditional
quantile curves.
• All source code associated with these analyses (no data) can be found on
the Center’s GitHub repository for Rhode Island analyses at: https://
github.com/CenterForAssessment/Rhode_Island
2016 ELA Growth Analyses
PARCC G3 PARCC G4
PARCC G4 PARCC G5
PARCC G5 PARCC G6
Growth
Calculations
PARCC G6 PARCC G7
PARCC G7 PARCC G8
PARCC G8 PARCC G9
PARCC G9 PARCC G10
2016 Mathematics Growth Analyses
PARCC G3 PARCC G4
PARCC G4 PARCC G5
PARCC G5 PARCC G6
Growth PARCC G6 PARCC G7
Calculations
PARCC G7 PARCC G8
PARCC G7 PARCC ALGEBRA I
PARCC G8 PARCC ALGEBRA I
PARCC ALGEBRA I PARCC GEOMETRY
2017 ELA Growth Analyses
PARCC G3 PARCC G4
PARCC G3 PARCC G4 PARCC G5
PARCC G4 PARCC G5 PARCC G6
Growth PARCC G5 PARCC G6 PARCC G7
Calculations
PARCC G6 PARCC G7 PARCC G8
PARCC G7 PARCC G8 PARCC G9
PARCC G8 PARCC G9 PARCC G10
PARCC G9 PSAT10
2017 Mathematics Growth Analyses
PARCC G3 PARCC G4
PARCC G3 PARCC G4 PARCC G5
PARCC G4 PARCC G5 PARCC G6
PARCC G5 PARCC G6 PARCC G7
2017 PARCC G6 PARCC G7 PARCC G8
Growth
Calculations PARCC G6 PARCC G7 PARCC ALGEBRA I
PARCC G7 PARCC G8 PARCC ALGEBRA I
PARCC G7 PARCC ALGEBRA I PARCC GEOMETRY
PARCC G8 PARCC ALGEBRA I PARCC GEOMETRY
PARCC G8 PARCC ALGEBRA I PSAT 10
PARCC ALGEBRA I PARCC GEOMETRY PSAT 10
Note: Several analyses using PSAT 10 in 2018 were conducted including: PARCC ALGEBRA I 2015 to PSAT 10 2017 without the GEOMETRY in 2016
2018 ELA Growth Analyses
PARCC G3 RICAS G4
PARCC G3 PARCC G4 RICAS G5
2018 PARCC G4 PARCC G5 RICAS G6
Growth
Calculations PARCC G5 PARCC G6 RICAS G7
PARCC G6 PARCC G7 RICAS G8
PARCC G9 PSAT10
PARCC G8 PSAT10
Note: Several analyses using PSAT 10 in 2018 were conducted including: PARCC G8 (2016) to PSAT 10, PARCC G10 (2017) to PSAT 10
2018 Mathematics Growth Analyses
PARCC G3 RICAS G4
PARCC G3 PARCC G4 RICAS G5
PARCC G4 PARCC G5 RICAS G6
Growth PARCC G5 PARCC G6 RICAS G7
Calculations
PARCC G6 PARCC G7 RICAS G8
PARCC ALGEBRA I PSAT10
PARCC GEOMETRY PSAT 10
PARCC G8 PSAT 10
2019 ELA Growth Analyses (Proposed)
RICAS G3 RICAS G4
PARCC G3 RICAS G4 RICAS G5
2019 PARCC G4 RICAS G5 RICAS G6
Growth
Calculations PARCC G5 RICAS G6 RICAS G7
PARCC G6 RICAS G7 RICAS G8
PSAT 10 SAT
PARCC G8 PSAT10
2019 Mathematics Growth Analyses (Proposed)
RICAS G3 RICAS G4
PARCC G3 RICAS G4 RICAS G5
Growth PARCC G4 RICAS G5 RICAS G6
Calculations
PARCC G5 RICAS G6 RICAS G7
PARCC G6 RICAS G7 RICAS G8
PSAT 10 SAT
ANALYSES -2017
• All analyses using 2017 PSAT data ran without any analytic issues
• No ceiling or floor effects were observed in any of the analyses.
• No issues related to “granularity” of scale were observed.
ANALYSES -2018
• All RICAS and PSAT/SAT analyses ran without any analytic issues. Analyses
conducted included cohort referenced and measurement error corrected
(SIMEX adjusted) SGPs as well and growth to standard projections.
• No ceiling or floor effects were observed in any of the analyses.
• No issues related to “granularity” of scale were observed.
• Following discussions with another state regarding year-to-year changes
after transition to PSAT/SAT, an examination of cross-year.
ANALYSES -2018: POSSIBLE PSAT/SAT BIAS
• Colorado noticed large changes in district level median SGPs in ELA and Math
after transitioning from PARCC Grade 9 & 10 to PSAT/9, PSAT/10, SAT
• Upon further examination in Colorado, other large changes emerged most
prominently large changes in growth depending upon the gender of the student.
• For example, in 2017, growth from ELA Grade 8 to ELA Grade 9 had median
SGPs of 50/51 for males/females. In 2018 from ELA Grade 8 to PSAT 9 median
SGPs were 56/44 for males/females.
• Within several large districts that saw changes in median SGP, males went from
underperforming females (median SGP) to outperforming.
• The result is reminiscent of the paper versus online issue that emerged in
PARCC.
ANALYSES -2018: POSSIBLE PSAT/SAT BIAS
• Rhode Island doesn’t have the exact same assessment transition as
Colorado but is transitioning away from PARCC and including PSAT/SAT
assessments in high school.
• Preliminary analysis of Rhode Island data shows similar differences as
Colorado:
Cohort Median SGP Males Median SGP Females Median SGP Overall
2014_2015/ELA_9; 2015_2016/ELA_10 46 (3,863) 54 (3,480) 50
2015_2016/ELA_9; 2016_2017/
PSAT_EBRW_EOCT
2016_2017/ELA_PSAT_10_EOCT;
2017_2018/ELA_SAT_EOCT
2016_2017/ELA_9; 2017_2018/
ELA_PSAT_10_EOCT
ANALYSES -2018: POSSIBLE PSAT/SAT BIAS
• Historically, for ELA, across all grades for which growth is calculated, females
consistently demonstrate equal or slightly higher growth than males.
• As the previous results show, this pattern reverses dramatically when the state
switched from PARCC to PSAT/SAT.
• Note that the results diminish when the PSAT/SAT become both the
independent (prior) and dependent (current) variable.
• The optimal window to detect such bias occurs in the year that the test is
switched (the first row in the previous table)
• The results support recent research by Reardon et al. (2018) on achievement
gender gaps and test item format. Education Researcher, Vol 47, No. 5, pp. 284-294
2018 RICAS ELA: Grade 4, 5, 6, 7, 8
Student Growth Percentile Goodness−of−Fit Descriptives Student Growth Percentile Goodness−of−Fit Descriptives Student Growth Percentile Goodness−of−Fit Descriptives
2017−2018 ELA SGP, Grade 4 (N = 10,044) 2017−2018 ELA SGP, Grade 5 (N = 10,250) 2017−2018 ELA SGP, Grade 6 (N = 9,871)
SGP Deciles by Prior Achievement Level (ELA) SGP Deciles by Prior Achievement Level (ELA) SGP Deciles by Prior Achievement Level (ELA)
Level 5 (3.5%) Level 5 (7.4%) Level 5 (3%)
Prior Achievement Level Prior Achievement Level Prior Achievement Level
Level 4 (37%) Level 4 (33.7%) Level 4 (38.2%)
Level 3 (22.4%)
Level 3 (27.3%) Level 3 (27.5%)
Level 2 (19.5%)
Level 2 (18.2%) Level 2 (18.7%)
Level 1 (17.5%) Level 1 (13.4%) Level 1 (12.5%)
Median Student Growth Percentile Median Student Growth Percentile Median Student Growth Percentile
Ceiling / Floor Effects Test Ceiling / Floor Effects Test Ceiling / Floor Effects Test
Student Growth Percentile Range Student Growth Percentile Range Student Growth Percentile Range
Current Observed Score Current Observed Score Current Observed Score
5 9 to to to to to to to to to to 5 9 to to to to to to to to to to 5 9 to to to to to to to to to to
to to to to 10 20 30 40 50 60 70 80 90 95 to to
Lowest OSS: [−3.15,−3.13] 78.4 21.6 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 N = 37 Lowest OSS: [−3.34,−3.27] 81.8 15.2 3.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 N = 33 Lowest OSS: [−3.17,−3.1] 82.6 13.0 4.3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 N = 23
Extremes (Range) Extremes (Range) Extremes (Range)
Highest OSS: [ 2.7382 ] 2.3 0.0 0.0 0.0 0.0 0.0 2.3 2.3 7.0 7.0 16.3 62.8 N = 43 Highest OSS: [ 2.555 ] 2.0 0.0 0.0 0.0 0.0 0.0 0.0 3.9 3.9 11.8 13.7 64.7 N = 51 Highest OSS: [2.512,2.648) 0.0 0.0 0.0 0.0 0.0 0.0 2.4 7.3 7.3 17.1 7.3 58.5 N = 41
Student Growth Percentile Range QQ−Plot: Student Growth Percentiles Student Growth Percentile Range QQ−Plot: Student Growth Percentiles Student Growth Percentile Range QQ−Plot: Student Growth Percentiles
19 29 39 49 59 69 79 89 99 Theoretical SGP Distribution 19 29 39 49 59 69 79 89 99 Theoretical SGP Distribution 19 29 39 49 59 69 79 89 99 Theoretical SGP Distribution
9 to to to to to to to to to 9 to to to to to to to to to 9 to to to to to to to to to
to 10 20 30 40 50 60 70 80 90 0 10 20 30 40 50 60 70 80 90 100 1
to 10 20 30 40 50 60 70 80 90 0 10 20 30 40 50 60 70 80 90 100 1
to 10 20 30 40 50 60 70 80 90 0 10 20 30 40 50 60 70 80 90 100
100 1st/[−5.5,−1.5) 9.71 10.98 10.88 9.71 10.29 10.39 9.61 8.53 9.41 10.49 (10%) 100 100
1st/[−15,−1.367) 10.77 11.08 10.26 10.97 9.74 10.05 9.33 9.54 8.92 9.33 (9.7%) 1st/[−5.624,−1.488) 9.33 10.71 10.82 9.23 9.54 10.18 10.07 9.33 9.86 10.92 (9.6%)
2nd/[−1.367,−0.8713) 8.53 10.10 9.61 9.02 9.90 9.71 10.20 10.20 10.39 12.35 (10.2%) 2nd/[−1.5,−0.96) 9.17 9.56 9.27 10.44 9.46 9.27 8.68 12.39 11.22 10.54 (10%) 2nd/[−1.488,−0.911) 9.74 9.35 8.96 10.81 10.22 9.64 10.13 10.13 9.93 11.10 (10.4%)
Prior Scale Score Decile/Range * Prior Scale Score Decile/Range * Prior Scale Score Decile/Range *
3rd/[−0.8713,−0.4928) 8.13 7.42 9.55 9.25 10.16 12.20 9.55 10.98 11.99 10.77 (9.8%) 3rd/[−0.96,−0.56) 9.74 10.34 9.74 10.24 9.84 9.74 12.23 8.35 9.54 10.24 (9.8%) 3rd/[−0.911,−0.511) 8.71 10.33 10.84 10.13 10.13 9.52 9.63 11.04 10.64 9.02 (10%)
Empirical SGP Distribution Empirical SGP Distribution Empirical SGP Distribution
4th/[−0.4928,−0.178) 10.34 9.84 11.05 10.34 11.26 10.95 9.94 8.62 6.90 10.75 (9.8%) 4th/[−0.56,−0.24) 10.88 9.71 10.29 10.20 10.00 10.10 8.82 8.92 10.10 10.98 (10%) 4th/[−0.511,−0.1947) 9.31 10.26 9.31 11.01 9.31 10.79 10.58 8.89 10.16 10.37 (9.6%)
5th/[−0.178,0.1279) 10.46 12.65 9.26 9.96 8.67 8.37 8.47 11.95 11.35 8.86 (10%) 5th/[−0.24,0.05) 8.59 9.58 10.27 9.08 10.17 11.06 9.28 11.25 9.58 11.15 (9.9%) 5th/[−0.1947,0.1056) 10.23 9.83 10.33 9.42 9.63 10.13 10.13 9.93 9.42 10.94 (10%)
6th/[0.1279,0.399) 8.79 10.79 8.79 11.59 9.69 8.89 9.99 9.09 10.89 11.49 (10%) 6th/[0.05,0.31) 8.28 10.06 10.45 10.55 10.45 8.68 10.06 11.05 10.06 10.36 (9.9%) 6th/[0.1056,0.402) 9.39 9.97 9.10 10.26 10.65 9.97 9.87 11.04 9.58 10.16 (10.5%)
7th/[0.399,0.7032) 9.88 8.10 11.36 8.50 11.86 10.77 10.87 9.68 7.91 11.07 (10.1%) 7th/[0.31,0.6) 10.32 10.32 9.09 10.70 9.75 10.89 10.13 9.38 10.04 9.38 (10.3%) 7th/[0.402,0.6637) 10.49 9.97 10.39 9.55 9.86 10.07 10.28 8.71 10.39 10.28 (9.7%)
8th/[0.7032,1.04) 8.86 9.90 10.67 9.43 9.62 11.14 8.67 12.38 9.90 9.43 (10.5%) 8th/[0.6,0.94) 9.27 10.06 10.06 9.47 9.37 9.86 10.95 8.48 10.45 12.03 (9.9%) 8th/[0.6637,1.014) 8.57 10.56 10.46 11.55 9.46 9.76 9.26 9.86 9.46 11.06 (10.2%)
9th/[1.04,1.46) 10.03 9.63 8.61 10.44 8.31 10.64 10.03 9.32 11.35 11.65 (9.8%) 9th/[0.94,1.4) 10.07 10.36 10.65 9.68 8.71 10.07 9.87 10.26 9.20 11.13 (10.1%) 9th/[1.014,1.491) 10.28 8.97 9.07 8.47 10.99 9.48 10.79 10.28 10.89 10.79 (10%)
10th/[1.46,6.139] 9.46 10.34 10.63 10.34 10.34 9.56 10.34 9.66 9.17 10.15 (10.2%) 10th/[1.4,6.9] 9.34 9.15 9.91 10.39 10.49 10.20 9.63 10.49 10.58 9.82 (10.2%) 10th/[1.491,4.36] 9.30 10.50 9.90 10.40 9.80 10.30 10.00 10.00 9.70 10.10 (10.1%)
*Prior score deciles can be uneven depending upon the prior score distribution *Prior score deciles can be uneven depending upon the prior score distribution *Prior score deciles can be uneven depending upon the prior score distribution
Student Growth Percentile Goodness−of−Fit Descriptives Student Growth Percentile Goodness−of−Fit Descriptives
2017−2018 ELA SGP, Grade 7 (N = 9,876) 2017−2018 ELA SGP, Grade 8 (N = 10,051)
SGP Deciles by Prior Achievement Level (ELA) SGP Deciles by Prior Achievement Level (ELA)
Level 5 (5.4%)
Level 5 (11%)
Prior Achievement Level Prior Achievement Level
Level 4 (32.3%)
Level 4 (28.4%)
Level 3 (28.5%) Level 3 (24.7%)
Level 2 (18.3%)
Level 2 (22.1%)
Level 1 (11.7%) Level 1 (17.5%)
Median Student Growth Percentile Median Student Growth Percentile
Ceiling / Floor Effects Test Ceiling / Floor Effects Test
Student Growth Percentile Range Student Growth Percentile Range
Current Observed Score Current Observed Score
5 9 to to to to to to to to to to 5 9 to to to to to to to to to to
to to 10 20 30 40 50 60 70 80 90 95 to to 10 20 30 40 50 60 70 80 90 95
Lowest OSS: [−3.12,−3.08] 57.7 11.5 23.1 7.7 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 N = 26 Lowest OSS: [−2.961,−2.957] 18.4 18.4 26.5 24.5 10.2 2.0 0.0 0.0 0.0 0.0 0.0 0.0 N = 49
Extremes (Range) Extremes (Range)
Highest OSS: [ 2.251 ] 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.6 10.5 10.5 23.7 52.6 N = 38 Highest OSS: [2.342,2.622) 0.0 0.0 0.0 0.0 0.0 0.0 0.0 7.3 12.2 29.3 12.2 39.0 N = 41
Student Growth Percentile Range QQ−Plot: Student Growth Percentiles Student Growth Percentile Range QQ−Plot: Student Growth Percentiles
19 29 39 49 59 69 79 89 99 Theoretical SGP Distribution 19 29 39 49 59 69 79 89 99 Theoretical SGP Distribution
9 to to to to to to to to to 9 to to to to to to to to to
to 10 20 30 40 50 60 70 80 90 0 10 20 30 40 50 60 70 80 90 100 1
to 10 20 30 40 50 60 70 80 90 0 10 20 30 40 50 60 70 80 90 100
1st/[−4.891,−1.474) 7.42 14.63 10.60 9.97 9.65 9.97 9.33 9.65 9.01 9.76 (9.5%) 100 1st/[−5.661,−1.688) 4.27 15.42 13.85 10.42 8.23 8.44 9.69 9.69 9.69 10.31 (9.6%) 100
2nd/[−1.474,−1.014) 12.55 7.02 9.29 10.28 9.68 9.88 9.88 9.68 10.18 11.56 (10.2%) 2nd/[−1.688,−1.106) 12.51 7.83 8.69 9.17 10.60 9.36 10.89 10.32 10.70 9.93 (10.4%)
Prior Scale Score Decile/Range * Prior Scale Score Decile/Range *
3rd/[−1.014,−0.6135) 8.09 8.69 10.59 10.39 10.59 10.99 9.49 10.99 10.79 9.39 (10.1%) 3rd/[−1.106,−0.7056) 8.14 8.54 10.85 10.45 10.25 11.56 10.05 9.85 9.85 10.45 (9.9%)
Empirical SGP Distribution Empirical SGP Distribution
4th/[−0.6135,−0.285) 8.17 11.44 10.52 9.81 10.21 9.40 9.91 9.91 10.01 10.62 (9.9%) 4th/[−0.7056,−0.3461) 9.56 10.44 8.97 10.94 9.85 9.46 9.46 10.64 10.44 10.25 (10.1%)
5th/[−0.285,0.0344) 11.94 9.62 9.92 9.72 9.11 10.02 10.43 8.81 9.72 10.73 (10%) 5th/[−0.3461,−0.0668) 10.54 11.05 9.61 9.50 11.05 9.61 10.43 8.99 8.78 10.43 (9.6%)
6th/[0.0344,0.3365) 9.82 10.24 9.82 10.13 9.31 10.44 9.82 11.17 9.51 9.72 (9.8%) 6th/[−0.0668,0.2107) 8.49 10.46 10.46 8.98 8.49 10.07 9.28 10.66 11.94 11.15 (10.1%)
7th/[0.3365,0.6398) 8.78 9.37 10.65 9.86 10.75 9.47 9.27 9.07 11.14 11.64 (10.3%) 7th/[0.2107,0.4981) 9.95 8.99 9.95 11.40 10.82 10.43 9.86 9.57 9.57 9.47 (10.3%)
8th/[0.6398,0.9782) 8.78 9.90 9.59 9.49 10.92 10.82 11.33 9.90 8.57 10.71 (9.9%) 8th/[0.4981,0.8308) 8.56 10.50 9.38 9.38 10.60 9.99 9.79 10.40 9.58 11.82 (9.8%)
9th/[0.9782,1.463) 10.46 9.85 9.14 11.27 8.83 10.86 9.14 10.66 9.44 10.36 (10%) 9th/[0.8308,1.308) 10.47 9.68 10.67 10.28 8.99 10.08 9.98 9.98 10.28 9.58 (10.1%)
10th/[1.463,6.122] 9.04 10.03 10.53 9.53 10.82 9.33 10.03 9.73 10.53 10.43 (10.2%) 10th/[1.308,5.236] 9.17 9.95 9.46 10.05 10.34 10.05 9.76 9.95 10.15 11.12 (10.2%)
*Prior score deciles can be uneven depending upon the prior score distribution *Prior score deciles can be uneven depending upon the prior score distribution
About this copy
Rhode Island Growth Analyses Using PSAT — Rhode Island Department of Education, September 2018.
This is an image-based copy of the original Keynote slides; builds and animations are flattened.