Assessment Changes/Transitions using SGPs

New Meridian TAC · October 3, 2019

Assessment Changes/Transitions using SGPs — New Meridian TAC, October 3, 2019.
Published

October 3, 2019

Event
Invited talk
Slides
7
Format
Converted from Beamer

This deck is an image-based copy of the original slides. Builds and animations are flattened; the slide text is reproduced below for search and screen readers.

Slide text
  1. Background Approach References Assessment Changes/Transitions using SGPs Damian W. Betebenner & Adam Van Iwaarden National Center for the Improvement of Educational Assessment New Meridian TAC Meeting — Austin, Texas October 3rd, 2019 Betebenner & Van Iwaarden Assessment Changes & SGPs
  2. Background Approach References SGP Calculations with Pearson/PARCC/New Meridian History of SGP Calculations SGPs [Betebenner, 2009] calculated for 2015-2016, 2016-2017, 2017-2018, and 2018-2019 SGPs, Measurement error corrected SGPs, SGP standard errors, and student growth projections are calculated at both the state and consortium level. For 2018-2019, SGPs were calculated for NM, IL, BIE, NJ, DoDEA, DC, MD Several states/organizations utilize consortium level SGPs and were interested in investigating whether changing consortium membership and/or changes in test format impacts the validity of SGP calculated. Betebenner & Van Iwaarden Assessment Changes & SGPs
  3. Background Approach References SGP Calculations with Pearson/PARCC/New Meridian History of SGP Calculations In general, because SGPs are norm referenced, they are scale agnostic only requiring students to take the same test at the same time. Given changing consortium state membership together with different test/form types, there are questions regarding the comparability of SGPs both within and across years. Issues related to mode of administration effect (within year, different groups) have been investigated by PARCC previously. Issues related to across year differences in growth due to changes in consortium membership or test form change (e.g., 2019 short-form use) have not been investigated. For 2019 spring analyses, NM, IL, NJ, BIE utilized an abbreviated form of the PARCC test. DC, MD, DoDEA utilized the flagship (long) form. NJ also post-equated its ELA test. We investigated whether changing state membership as well as test form differences led to appreciable differences in SGP results. Betebenner & Van Iwaarden Assessment Changes & SGPs
  4. Background Approach References SGP Properties SGP Properties SGPs are uniformly distributed and uncorrelated with prior achievement at the individual level. State and consortium level SGPs (i.e., growth norms) are calculated. For large groups, entire states (state norms), or the consortium, mean(SGP ) ≈ 50. “Large” deviations from 50 or between groups usually indicates something is going on and worth investigating further. A state’s recent transition to different assessments at the high school level led to 10 to 15 point mean SGP disparities between males and females (prior tests were near parity between males/females) Within year mean SGP differences between groups taking paper-pencil versus computer administered have been Betebenner & Van Iwaarden Assessment Changes & SGPs
  5. Background Approach References Investigation of mean SGP differences Two approaches Probabilistic Difference Given sufficient historical data, mean SGPs can be investigated relative to the distribution of previous observations. A t-distribution or similar sample distribution can be used to formally calculate a p-values. In general, too few observations (3 years of mean SGPs for these analyses) were available for significance testing. Also, statistical significance doesn’t imply practical significance. Practical Difference Cohen’s D can be easily calculated for groups using SGPs. Because SGPs are uniform, the pooled within group standard deviation that forms the denominator is approximately 28.8 for most groups investigated. A mean(SGP ) difference of 5 (which is often used as a rule of thumb for practical differences) corresponds to a Cohen’s D of 0.17. Betebenner & Van Iwaarden Assessment Changes & SGPs
  6. Background Approach References Investigation of mean SGP differences Results Historical consortium SGP results were recalculated using 2019 state membership to examine year-to-year variation. Year-to-year differences uniformly negligible using 5 as the practical significance threshold. For state continuing with dual mode administration, mode effects continue to be detected. Betebenner & Van Iwaarden Assessment Changes & SGPs
  7. Background Approach References References Betebenner, D. W. (2009). Norm- and criterion-referenced student growth. Educational Measurement: Issues and Practice, 28(4):42–51. Betebenner & Van Iwaarden Assessment Changes & SGPs