FILE UNDER COMPARABILITY: USING SGP TO EVALUATE ADMINISTRATION MODE EFFECTS Damian W Betebenner National Center for the Improvement of Educational Assessment Dover, New Hampshire January 5th, 2016

MODE EFFECTS • Many states are transitioning to computer administered testing. • Not all states have transitioned completely to 100% online administration. • For example, many PARCC states, allowed both online and paper administration (Rhode Island, Massachusetts, New Mexico, …) • Other states (e.g., Indiana) also allowed both online and paper administration.

MODE EFFECTS • Rhode Island was the rst state I worked with that had nalized PARCC data. • During a November 23rd meeting, the RI DOE communicated reports from the eld that the online administration was much harder than the paper and pencil. • At the heart of that claim is an impossible to test counterfactual. • However, SGPs allow states a simple and immediate check on the claim of a mode effect. fi fi fi

SGP: RELEVANT PROPERTIES • Student growth percentiles (SGPs) are a norm-referenced quantity. • SGPs use (quantile) regression to approximate the conditional density associated with each students achievement history. • Similar to residualized gain scores. • SGPs are uncorrelated from year-to-year. • For a random sample of students, the median SGP associated with the group is expected to be 50. • “Treatment effects” would be associated with deviations from 50.

THE TRANSITION TO PARCC • SGP analyses are norm-referenced and don’t require a vertical scale to be conducted. Depending upon whether results are disseminated, a likely cause of confusion: “How can you calculate growth across two different tests?” • Other PARCC SGP states we’re conducting analyses for: Massachusetts, New Jersey, Colorado • The Center conducted meetings with both SBAC and PARCC consortium members discussing issues related to calculating and using growth during assessment transitions.

RI PARCC: By Content Area Online Paper Median SGP ELA Mathematics (not Algebra I)

ELA by Grade Online Paper Median SGP Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9

RI PARCC: Mathematics by Grade Online Paper Median SGP Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Algebra I

CONCLUSIONS • SGPs provide an easy way for states to comb through the data and examine the impact that various “treatments” have on the outcomes of students. • Some states (e.g., MA) immediately looked at this metric while others weren’t sure how to examine. • Need to give more thought to helping states use the data that they have available to quickly test relevant hypotheses.

About this copy

File Under Comparability — Center for Assessment staff meeting, January 5, 2016.

This is an image-based copy of the original Keynote slides; builds and animations are flattened.