Rhode Island Mode Effect Analysis

Rhode Island Department of Education · July 18, 2016

Rhode Island Mode Effect Analysis — Rhode Island Department of Education, July 18, 2016.
Published

July 18, 2016

Event
State agency
Slides
10
Format
Converted from Keynote

This deck is an image-based copy of the original slides. Builds and animations are flattened; the slide text is reproduced below for search and screen readers.

Slide text
  1. RHODE ISLAND MODE EFFECT ANALYSIS OF 2016 PARCC RESULTS Damian W Betebenner National Center for the Improvement of Educational Assessment Providence, RI July 18th, 2016
  2. RECAP/OVERVIEW • 2015 NECAP to PARCC SGP analyses were conducted in late 2015/early 2016. • Based upon reports from the eld that online and paper & pencil forms were not equivalent, SGPs were analyzed between the two modes of administration. • Signi cant mode effects were discovered as part of the SGP analyses where paper and pencil test takers generally had an advantage of those taking the online administered test. • The mode effects were larger in English Language Arts than in Mathematics. Median SGP for P&P were often at/above 60 whereas Median SGPs for Online were in mid 40s. • Pearson analyses con rmed the existence of mode effects. fi fi fi
  3. 2016 OVERVIEW • Dual mode administration was again allowed in Rhode Island • Students were not required to take the test using the same mode as in 2015. Thus 4 combinations across two years are possible: • 2015 Online/2016 Online • 2015 Online/2016 Paper • 2015 Paper/2016 Online • 2015 Paper/2016 Paper • We know mode effects exist with prior scores, also possible this year with current scores. • Currently (to the best of my knowledge) PARCC decided not to pursue a mode adjustment but allowed states to work with Pearson to implement a mode adjustment with their data (Colorado only state pursuing a mode adjustment — possibly for both years).
  4. 2016 OVERVIEW • Analyses of 2016 PARCC data conducted using preliminary le provided by RIDE. • SGPs were calculated using scaled scores (not thetas) under two conditions: • single prior (only PARCC data) • up to 5 priors (including NECAP data) • SGPs calculated with thetas will demonstrate the same mode effect properties as those calculated with scale scores. • Analyses conducted for Mathematics, Algebra I, Geometry, and English Language Arts fi
  5. QUESTIONS TO BE ANSWERED TODAY • Are we making any mode adjustments that will impact SGPs? • If so, for both 2015 and 2016? And how? • Individual SGPs – potentially impacts GMV and accountability. • Aggregate – could result in discrepancies between aggregates in GMV and accountability. • What SGPs will be used in GMV? Current assumption is that we will use the ones that are presented in ISRs. • Need cutscores for GMV. Should we try to be consistent between GMV and accountability. If so, need to nalize cuts very soon. • Current GMV cuts are: • High: 71-99 • High typical: 56-70 • Typical: 46-55 • Low typical 31-45 • Low: 1-30 fi
  6. PARCC 2016 MODE EFFECT ANALYSES ELA: 1 PRIOR SCORE P&P 2016 Online 2016 GRADE MEDIAN SGP COUNT GRADE MEDIAN SGP COUNT ALL 43 6729 ALL 32.0 5731 GRADE MEDIAN SGP COUNT GRADE MEDIAN SGP COUNT Online 2015 6 ALL 65 3442 ALL 52 39818
  7. PARCC 2016 MODE EFFECT ANALYSES MATHEMATICS: 1 PRIOR SCORE P&P 2016 Online 2016 GRADE MEDIAN SGP COUNT GRADE MEDIAN SGP COUNT ALL 47 5760 ALL 41 4560 GRADE MEDIAN SGP COUNT GRADE MEDIAN SGP COUNT Online 2015 6 ALL 51 2989 ALL 51 31940
  8. SUMMARY • Signi cant mode effects persist in 2016 data. • Given the permutations of different modes across 2 years, dif cult to tease out. • Because of small N size in some groups, impossible to just run separate SGP analyses for each group. • If mode adjustments are desired two possibilities • Adjust scale scores for both 2015 and 2016 (what Colorado is doing) • Run SGP analyses with an additional dummy variable indicating mode (crossing 2015 and fi fi
  9. QUESTIONS TO BE ANSWERED TODAY • Are we making any mode adjustments that will impact SGPs? • If so, for both 2015 and 2016? And how? • Individual SGPs – potentially impacts GMV and accountability. • Aggregate – could result in discrepancies between aggregates in GMV and accountability. • What SGPs will be used in GMV? Current assumption is that we will use the ones that are presented in ISRs. • Need cutscores for GMV. Should we try to be consistent between GMV and accountability. If so, need to nalize cuts very soon. • Current GMV cuts are: • High: 71-99 • High typical: 56-70 • Typical: 46-55 • Low typical 31-45 • Low: 1-30 fi