NEW JERSEY PARCC 2016 SGP ANALYSES Damian W Betebenner National Center for the Improvement of Educational Assessment Trenton, NJ September 23rd, 2016

“It is a common misconception that to measure student growth in education, the subject matter and grades over which growth is examined must be on the same scale—referred to as a vertical scale. Not only is a vertical scale not necessary, but its existence obscures fundamental concepts necessary to understand growth.” Me Norm- and Criterion-Referenced Student Growth

RED HERRINGS • Student growth is not that dif cult to understand and use, but numerous red herrings exist that obfuscate and mislead non- technical (and technical) users: • Gain scores: Subtraction isn’t a growth model. • Dif cult to understand because so complicated to calculate. • Can’t be used for evaluation (not causal). fi fi

“If you can’t solve a problem, then there is an easier problem you can solve: nd it.” George Pólya Mathematical Discovery on Understanding, Learning, and Teaching Problem Solving, Volume I fi

A Math Story Problem • Whose high jump changed more? Explain your answer. • Whose high jump change is more remarkable? Explain your answer. • Which high jumper improved more? Explain your answer.

A Math Story Problem Properties that don’t hold with simple examples will not hold in complicated situations. This simple example illustrates how minimally helpful a scale is to understanding change. • A vertical scale allows one to answer a very narrow question: Who changed more? • Remarkable & improvement need not be synonymous with change. • This example (ratio scale, no measurement error) illustrates that a vertical scale is of little help in answering substantive questions.

RED HERRING 1: SUBTRACTION ISN’T A GROWTH MODEL • PARCC doesn’t have a vertical scale so subtract isn’t even justified. • People will subtract even though they aren’t supposed to. The PARCC scale runs from 650 to 850 across all content areas and grades. • Even if you could subtract, the previous example shows that subtracting (and then comparing gains) is highly problematic. • States with vertical scales (e.g., SBAC states) all demonstrate the “high jumper phenomenon”: Low achievers gain more than high achievers.

RED HERRING 1: SUBTRACTION ISN’T A GROWTH MODEL • Peer-based analyses are also called norm-referenced analyses. • Like with infant height/weight charts, peers are children/student with similar characteristics. • For growth analyses using large scale assessments, peers are students with (at least) the same scale score history on the same tests. • Peer based norms are invaluable in providing context for stakeholders to understand student growth. • Peer based norms are possible even across test transitions like New Jersey’s transition from NJASK to PARCC.

RED HERRING 2: DIFFICULT TO CALCULATE SO CAN’T UNDERSTAND • The concept of student growth isn’t as difficult to understand and popular perception suggests. • Calculations may be complicated but knowing how to calculate student growth is not synonymous with understanding student growth: – The number π (pi=3.14159…) is difficult to calculate but is simple to understand conceptually. – Height and weight percentiles associated with children are difficult to calculate but are simple to understand conceptually. • Computational details are important but often obscure a simple conceptual basis.

RED HERRING 3: NOT CAUSAL SO INAPPROPRIATE FOR EVALUATION • There is active debate about whether it is possible to isolate the the unique contribution of a teacher from the data derived from state summative assessments like PARCC (i.e., are causal estimates) • The pro-argument: If they are causal, then it is hard to deny using them for evaluation purposes. • The con-argument: If they aren’t causal, then they shouldn’t be used for evaluation purposes. • The con-argument is a red herring (even if estimates are not causal). • Society employs numerous non-causal quantities for evaluation purposes (e.g., win/loss percentage, student satisfaction, …). • In fact, I’m at a loss to come up with a single example where an actual causal estimator is used for actual institutional or personnel evaluation.

RESULTS: SGP PROPERTIES • SGPs are uncorrelated with prior achievement at the student level (i.e., a conditional probability) • SGPs are correlated with current achievement. I.e., higher growth is associated with higher achievement and vice-versa. • SGPs for students are uncorrelated from year to year.

BACKGROUND • 2016 SGP analyses of New Jersey PARCC data were conducted in September 2016. • This follows 2015 SGP analyses of New Jersey PARCC data conducted in late 2015/early 2016. • SGP analyses of New Jersey data have been performed since 2010. • Analyses were conducted using PARCC theta values (instead of scaled scores) and scale scores for NJASK as priors to avoid any potential issues with ceiling/ oor effects. • Of the 4 PARCC states we work with on SGP analyses, 3 are using theta values. • PARCC SGP analyses are conducted using theta values. • Results from SGP analyses (only ELA and Mathematics) are used, in part, for educator evaluation. fl

RESULTS: DATA PREPARATION • 2016 NJ PARCC SGPs were calculated for: • ELA and Mathematics EOGTs • Algebra I, Algebra II and Geometry EOCTs. • Valid cases by Grade and Content area (2015 top/2016 bottom) Content 3 4 5 6 7 8 9 10 Content Area Area Algebra I 91,716 ELA 95,288 93,830 94,651 92,668 90,301 88,479 82,683 73,165 Algebra II 58,022 Math 95,997 94,555 95,360 93,239 87,375 58,097 Geometry 71,130 Content 3 4 5 6 7 8 9 10 11 Content Area Area Algebra I 106,030 ELA 98,998 96,802 95,710 96,888 95,974 94,268 95,406 86,974 69,777 Algebra II 74,582 Math 99,800 97,604 96,406 97,633 93,117 60,743 Geometry 84,532

RESULTS: DATA ANALYSIS • EOGTs are analyzed based upon same content area and consecutive grades. • EOCTs are analyzed based upon same/similar content area (e.g., Mathematics to Algebra I) but not necessarily consecutive grades: • 2015 Math, Grade 8 to 2016 Algebra I; 2015 Math, Grade 7 to 2016 Algebra 1 • 2015 Geometry to 2016 Algebra II; 2015 Algebra I to 2016 Algebra II • 2015 Algebra I to 2016 Geometry: 2015 Algebra II to 2016 Geometry

RESULTS: DATA ANALYSIS • 2016 NJ PARCC median (mean) SGPs by content area and grade: Content 4 5 6 7 8 9 10 11 Area ELA 50 (50) 50 (49.9) 50 (49.8) 50 (49.8) 50 (49.8) 50 (50.1) 50 (50.2) 50 (50.1) Math 50 (50) 50 (50.1) 50 (50.1) 50 (50.2) 50 (50.2) Content Area Algebra I 50 (50.2) Algebra II 50 (50.0) Geometry 50 (50.1)

RESULTS: DATA ANALYSIS NJ_ORIGINAL School Performance NJ_ORIGINAL Growth & Prior Achievement2016 NJ_ORIGINAL ELA Higher Achievement Higher Achievement School Size Lower Growth Higher Growth 2015 Prior Percent at/above Proficient 50 Students 100 Students 250 Students 500 Students Achievement Lower Growth Higher Growth Lower Achievement Lower Achievement Growth 2016 Median Student Growth Percentile

RESULTS: DATA ANALYSIS New Jersey School Performance PARCC Growth & Prior Achievement 2015 PARCC ELA Higher Achievement Higher Achievement School Size Lower Growth Higher Growth 2014 Prior Percent at/above Proficient 50 Students 100 Students 250 Students 500 Students Achievement Lower Growth Higher Growth Lower Achievement Lower Achievement Growth 2015 Median Student Growth Percentile

RESULTS: DATA ANALYSIS NJ_ORIGINAL School Performance NJ_ORIGINAL Growth & Achievement 2016 NJ_ORIGINAL Mathematics Higher Achievement Higher Achievement School Size Lower Growth Higher Growth 50 Students 100 Students 2016 Percent at/above Proficient 250 Students 500 Students Achievement Lower Growth Higher Growth Lower Achievement Lower Achievement Growth 2016 Median Student Growth Percentile

RESULTS: DATA ANALYSIS New Jersey School Performance PARCC Growth & Prior Achievement 2015 PARCC Mathematics Higher Achievement Higher Achievement School Size Lower Growth Higher Growth 2014 Prior Percent at/above Proficient 50 Students 100 Students 250 Students 500 Students Achievement Lower Growth Higher Growth Lower Achievement Lower Achievement Growth 2015 Median Student Growth Percentile

About this copy

New Jersey PARCC 2016 SGP Analyses — New Jersey Department of Education, September 23, 2016.

This is an image-based copy of the original Keynote slides; builds and animations are flattened.