Growth During Transition

New Jersey Department of Education · May 2015

Growth During Transition — New Jersey Department of Education, May 2015.
Published

May 26, 2015

Event
State agency
Slides
21
Format
Converted from Keynote

This deck is an image-based copy of the original slides. Builds and animations are flattened; the slide text is reproduced below for search and screen readers.

Slide text
  1. GROWTH DURING TRANSITION Damian W Betebenner National Center for the Improvement of Educational Assessment New Jersey Technical Advisory Committee June 1, 2015
  2. GROWTH DURING TRANSITION: TWO DISTINCT CHALLENGES • Challenges to the calculation of student growth. • Challenges to the use of student growth for: • Teacher evaluation • School/District/State evaluation • Other diagnostic purposes
  3. THE TRANSITION “SPEED BUMP” (At least) 4 stages to traverse when considering growth
  4. THE TRANSITION “SPEED BUMP” (At least) 4 stages to traverse when considering growth Before: 2013-2014
  5. THE TRANSITION “SPEED BUMP” (At least) 4 stages to traverse when considering growth During 1: 2014-2015
  6. THE TRANSITION “SPEED BUMP” (At least) 4 stages to traverse when considering growth During 1I: 2015-2016
  7. THE TRANSITION “SPEED BUMP” (At least) 4 stages to traverse when considering growth After: 2016-2017
  8. September October November December January February March April May June July August PARCC PBA PARCC EOY Field Testing Field Testing PBA EOY Standard Window Window Setting 2016 Scores & 2015 Scores PBA EOY Data Files Returned Window Window Returned PBA EOY 2017 Scores & Window Window Data Files Returned September October November December January February March April May June July August
  9. REVIEW: FLAVORS OF GROWTH • SGP accommodates both Norm- and Criterion-Referenced growth • Norm-Referenced: Cohort referenced SGPs • Criterion-Referenced: Baseline referenced/anchored SGPs, student growth projections/targets (i.e., growth to standard), catch-up/keep-up/move-up/stay-up. • Based upon state preference, policy, and data, different states have created different combinations of norm- and criterion-referenced growth metrics. • These growth metrics often have different data requirements. • SGP calculation is robust to scale transformations. It unlikely that the transition from a state assessment scale to a PARCC scale will prevent the calculation of an SGP. However, this has to be validated on a state by state, content area by content area, and grade by grade basis.
  10. Level 5 Advanced Level 4 Move Up Target ●● Keep Up ● ● ● Catch Up Target Level 3 Target Proficient ● Level 2 Part Proficient Level 1 Unsatisfactory −− Grade 4 Grade 5 Grade 6 Grade 7 Scale Score −− 491 506 1499 Achievement Level Achievement Target −− Proficient Proficient Level 2 Level 4 (1586)/Level 5 (1586) Level 4 (1593) Achievement Growth Percentile −− −− 21 Growth Level Growth Target −− −− Low Keep Up (99)/Move Up (99) Catch Up (99) Growth
  11. REVIEW: THREATS TO CALCULATION • SGPs don’t require a vertical scale for calculation. Regardless of whether PARCC uses a vertical scale, SGPs can be calculated using both vertical and non-vertical scales. • A signi cant ceiling/ oor associated with a test (either PARCC or a state test used as a prior) can lead to bad model t. These growth metrics often have different data requirements. • Barring signi cant ceiling/ oor, norm-referenced analyses will very likely yield a uniform distribution of percentiles. • SGP calculation is robust to scale transformations. It unlikely that the transition from a state assessment scale to a PARCC scale will prevent the calculation of an SGP. However, this has to be validated on a state by state, content area by content area, and grade by grade basis. fi fl fl fi fi
  12. REVIEW: THREATS TO CALCULATION Student Growth Percentile Goodness−of−Fit Descriptives Student Growth Percentile Goodness−of−Fit Descriptives 2014 ELA SGP, Grade 6 (N = 95,825) 2014 Mathematics SGP, Grade 6 (N = 96,072) SGP Deciles by Prior Achievement Level SGP Deciles by Prior Achievement Level Advanced (8.8%) Prior Achievement Level (Mathematics) Advanced (43.6%) Prior Achievement Level (ELA) Proficient (53.2%) Proficient (37.1%) Part Proficient (38.1%) Part Proficient (19.3%) Median Student Growth Percentile Median Student Growth Percentile Student Growth Percentile Range QQ−Plot: Student Growth Percentiles Student Growth Percentile Range QQ−Plot: Student Growth Percentiles 19 29 39 49 59 69 79 89 99 Theoretical SGP Distribution 19 29 39 49 59 69 79 89 99 Theoretical SGP Distribution 9 to to to to to to to to to 9 to to to to to to to to to to 10 20 30 40 50 60 70 80 90 0 10 20 30 40 50 60 70 80 90 100 1 to 10 20 30 40 50 60 70 80 90 0 10 20 30 40 50 60 70 80 90 100 1st/[100,169) 9.89 9.94 9.85 10.26 9.62 9.83 10.15 9.89 10.10 10.48 (9.6%) 100 1st/[100,179) 9.61 10.41 10.13 9.87 9.81 10.21 9.51 9.69 10.07 10.70 (9.6%) 100 2nd/[169,181) 9.93 10.46 10.07 9.94 10.16 9.95 9.76 10.00 9.40 10.33 (8.7%) 2nd/[179,200) 9.40 9.66 9.97 10.12 9.85 9.85 10.32 10.31 9.68 10.83 (9.7%) Prior Scale Score Decile/Range * Prior Scale Score Decile/Range * 3rd/[181,190) 9.65 10.02 10.27 10.21 10.01 9.48 10.46 10.26 10.16 9.48 (8.9%) 3rd/[200,214) 9.89 10.18 9.82 9.82 10.17 10.00 9.71 9.54 10.46 10.42 (10.7%) Empirical SGP Distribution Empirical SGP Distribution 4th/[190,200) 9.77 10.19 10.01 9.44 10.02 9.90 9.75 9.79 10.13 11.00 (10.8%) 4th/[214,226) 9.41 9.86 10.38 10.35 9.43 9.83 9.41 10.05 10.71 10.57 (7.6%) 5th/[200,205) 9.96 10.10 10.01 10.38 10.19 9.88 9.60 9.76 9.66 10.45 (8.4%) 5th/[226,241) 9.22 10.33 9.77 9.47 10.37 10.07 10.51 10.21 8.74 11.31 (12.1%) 6th/[205,214) 9.77 10.04 10.15 9.90 9.73 10.28 10.10 10.20 9.68 10.15 (12.9%) 6th/[241,253) 9.96 9.72 10.01 10.41 9.75 9.74 9.68 9.75 10.82 10.14 (10.2%) 7th/[214,221) 9.26 10.09 9.77 9.79 9.74 9.60 9.86 10.50 10.99 10.39 (8.6%) 7th/[253,263) 9.86 9.83 10.12 9.94 9.93 9.80 9.14 9.84 11.62 9.93 (7.4%) 8th/[221,232) 9.79 10.24 9.87 9.89 10.11 10.38 10.06 9.66 9.60 10.39 (11.5%) 8th/[263,280) 9.31 10.22 9.95 9.51 10.07 9.77 10.54 10.38 9.15 11.11 (11.4%) 9th/[232,243) 9.88 9.92 10.14 10.06 10.18 9.66 10.01 9.65 9.66 10.83 (9.4%) 9th/[280,295) 8.80 9.63 9.78 10.03 10.40 10.63 10.38 9.28 4.30 16.76 (7.8%) 10th/[243,300] 9.23 10.16 9.75 10.00 9.94 10.15 9.94 10.23 10.16 10.45 (11.2%) 10th/[295,300] 9.80 10.23 10.40 9.41 5.96 5.18 3.59 2.58 0.95 41.90 (13.5%) *Prior score deciles can be uneven depending upon the prior score distribution *Prior score deciles can be uneven depending upon the prior score distribution
  13. REVIEW: REQUIREMENTS FOR CALCULATING GROWTH: SGP • Annual growth? Bi-annual growth? Tri-annual growth? • Annual growth (norm-referenced SGPs based upon a single cohort) requires (at a minimum) two consecutive annual measures (i.e., scale scores) of student achievement (usually taken from the same content area). • Annual growth (anchored/baseline-referenced SGPs) requires (at a minimum) two consecutive annual measures (i.e., scale scores) of student achievement (usually taken from the same content area) from the same assessment (i.e., an assessment with a time-invariant scale). • Annual targets (SGP Projections/Targets/Trajectories) require (at a minimum) two consecutive annual measures of student achievement from the same assessment (i.e., an assessment with a time-invariant scale).
  14. REVIEW: REQUIREMENTS FOR CALCULATING GROWTH: SGP • States primarily focus on annual growth, particularly for accountability (exceptions include, for example, student growth from 8th to 10th grade). • PARCC eld testing will impact a state’s ability to calculate growth for students due to the simple fact that student will not receive back scale scores on the PARCC eld test. • Absent double testing, students participating in the eld test cannot receive annual SGPs in BOTH 2013-2014 (no current year score) AND 2014-2015 (no prior year score). • If a student has a scale score in 2012-2013, one could, for example, calculate a two year SGP for a students participating in eld testing between 2012-2013 and 2014-2015. • In the years after the transition, absent double testing, students will have different numbers of priors available for SGP analyses. fi fi fi fi
  15. REVIEW: REQUIREMENTS FOR CALCULATING GROWTH: SUMMARY • Easier to conduct norm-referenced SGP analyses during transition than criterion-referenced growth to standard analyses. • Criterion-referenced projections require linking SGP analyses done exclusively on the PARCC assessment in order to extrapolate forward how much growth is required to, for example catch-up/keep-up/move-up/stay-up. • Analogously growth-to-standard (e.g., career and college-readiness) will require at least two years data on the new scale as well as xed achievement levels/ standards de ned on the new scale. • Establishing baseline referenced SGPs will likely require even more time (e.g., 4 years) to establish a stable baseline from which to compare going forward. fi fi
  16. September October November December January February March April May June July August PARCC PBA PARCC EOY Field Testing Field Testing PBA EOY Standard Window Window Setting 2016 Scores & 2015 Scores PBA EOY Data Files Returned Window Window Returned PBA EOY 2017 Scores & Window Window Data Files Returned September October November December January February March April May June July August
  17. REQUIREMENTS FOR THE USE OF GROWTH • Two distinct challenges: Calculation and Use • Just because we can calculate an SGP doesn’t mean it is defensible to use it for previously existing accountability purposes. • Validating SGP use will require de ning the purpose. • Current state uses include: • Teacher evaluation (required by RttT, ESEA waiver, state policy/rule making, ...) • School and District accountability (ESEA waiver, state policy, ...) • Charter school accreditation. fi
  18. REQUIREMENTS FOR THE USE OF GROWTH • During the assessment transition, a number of challenges confront states using growth for accountability purposes. • Data time span (e.g., two-year growth) not aligned with accountability time span (annual) • Changes in types of growth analyses done for accountability during transition. Criterion-referenced growth not available until year 2 of PARCC. • Assessment results from different assessments (state and PARCC). Depending upon the number of priors used by the state, this will be an ongoing issue for many years. • Missing data due to PARCC eld testing. fi
  19. REQUIREMENTS FOR THE USE OF GROWTH • Data time span (e.g., two-year growth) not aligned with accountability time span (annual) • Most accountability systems use an observational pre-post design to hold teachers/schools/districts accountable. • Often, the pre-test occurs the spring prior to enrollment in the unit of accountability and the post-test the spring during enrollment. • An extended growth time span (e.g., two-year) undermines the attribution of responsibility implicit within the accountability determination. • Mis t between accountability and growth time span impact school and district accountability as well, but likely will have less impact because eld testing does not occur school wide. fi fi
  21. Questions?