Talks archive, 2015–2019 – Understanding Student Growth

UNDERSTANDING STUDENT GROWTH Damian W Betebenner National Center for the Improvement of Educational Assessment Hartford, CT April 28th, 2016

Why Student Growth? If you care about student achievement, then you care about student growth. ๏ Where you are is a function of where you started and how quickly you’ve travelled since. ๏ Student growth and student attainment/ achievement are related in this way. ๏ Distance=Rate x Time ๏ Those who start behind tend to stay behind and those that start ahead tend to stay ahead.

“If you can’t solve a problem, then there is an easier problem you can solve: find it.” George Pólya Mathematical Discovery on Understanding, Learning, and Teaching Problem Solving, Volume I

A 5th Grade Story Problem ๏ Whose high jump changed more? Explain your answer. ๏ Whose high jump change is more remarkable? Explain your answer. ๏ Which high jumper improved more? Explain your answer.

A 5th Grade Story Problem ๏ Judy improved 2 inches. ๏ Anna Improved 6 inches. ๏ 6 inches > 2 inches. ๏ An inch is an inch!!! ๏ But is an inch really an inch???

A 5th Grade Story Problem ๏ Are we comparing apples to apples? ๏ Anna’s “beginner’s gains” happen in many sports. ๏ 2 inches for an elite jumper is far more remarkable than 6 inches for a novice.

A 5th Grade Story Problem Properties that don’t hold in simple situations will not hold in complicated situations. This simple example illustrates how minimally helpful a scale is to understanding change. ๏ A vertical scale allows one to answer a very narrow question: Who changed more? ๏ Remarkable improvement need not be synonymous with a large change. ๏ This example illustrates that a vertical scale is of little help in answering substantive questions.

Another Story ๏ Suppose your child showed no change in height between the ages of 3 and 5. ๏ Upon discovering this your pediatrician responds: “Don’t worry, my height hasn’t changed for the last 10 years and I’m perfectly healthy.” ๏ What do you think of this Doctor’s diagnosis? ๏ Again, 0 inches = 0 inches. But is an inch an inch??? ๏ It’s remarkable (and extremely concerning!) for a toddler not to grow. It’s not remarkable for an adult to show no change in height. ๏ Context matters!

Student Academic Growth ๏ The same holds true for the vertically scaled tests (e.g., SBAC, STAR). Average Student Scale Score Change 2014 to 2015 Top 10% Reading Grade 4 2014 -7.5 Bottom 10% Reading Grade 4 2014 +24.5 Top 10% Mathematic Grade 4 2014 -12.6 Bottom 10% Mathematics Grade 4 2014 +18.4 ๏ Just as an inch isn’t necessarily an inch; a scale score point isn’t necessarily a scale score point. ๏ Low achieving students tend to show much larger scale score gains than high achieving students! ๏ This isn’t a defect of test scales! It’s just the way it is (like high jumping).

Moral of the Story Subtraction isn’t a growth model. You have to go beyond subtraction to understand student academic growth.

So how can one understand student growth?

Context is Important ๏ Analogous to height/weight measures for infants. ๏ Growth amounts are tricky to interpret and require context. ๏ Peer based comparisons used by growth analyses are a way to acquire that context. An infant male toddler is measured at 2 and 3 years of age and is shown to have grown 4 inches. The magnitude of increase—4 inches—is a well understood quantity that any parent can grasp and calculate at home using a simple yardstick. However, parents leaving their pediatrician’s office knowing only how much their child has grown would likely be wanting for more information: Parents are not interested in an absolute magnitude of growth, but instead in a norm-referenced quantity locating that 4 inch increase alongside the height increases of similar children.

Context is Important ๏ Peer-based analyses are sometimes called norm-referenced analyses. ๏ Like with infant height/weight charts, peers are infants/toddlers with similar characteristics (e.g., sex and age). ๏ For growth analyses using large scale assessments, peers are students with (at least) the same scale score history on the same tests. ๏ Multiple priors give more accurate starting point. ๏ Demographics not used but could be. ๏ For interim assessments like STAR, time elapsed between tests is a peer characteristic. ๏ Peer based norms are invaluable in providing context for stakeholders to understand student growth. ๏ Student growth percentiles (SGP) were created to provide a peer- based context to understand student academic growth.

Student Growth Percentiles ๏ Research on the SGP model began in 2007 (Betebenner, 2008). ๏ Based upon work in Colorado, the norm-referenced model was extended to include a criterion-referenced component. (Betebenner, 2009). ๏ Work in other states incorporating the model into state reporting and accountability systems occurred in subsequent years and student growth became more common across state assessment systems. (Betebenner, 2012). ๏ Used in approximately 30 states for various diagnostic and accountability purposes. ๏ Extended for use with interim assessments in collaboration with Renaissance Learning (2015).

6 0 co T S re im 600 0 le e 1 4 0 ca ca 2 le 800 0 e S 2 0 im core

8 co re 0 0 le 6 ca th S ro nt i le er ce 4 im th re co le ca im

8 co re th 0 0 le ro w 6 ca nt i le S er ce 2 th 75 4 im re co le ca im

e m Sc o a a le A Sc 7 5 75th er ce nt i le G rowth 8 0 6 0 co T S re im 600 0 le e1 4 0 S ca ca 2 le 800 0 e S 2 0 T im core

Student Growth Percentiles (SGPs) ๏ Similar to height and weight percentiles for infants/toddlers, SGPs range from 1 to ๏ 1st percentile growth is impoverished and 99th percentile growth is exemplary. ๏ Peer-based growth calculations (SGPs) are only as good as the assessments on which they are based. ๏ SGPs are uncorrelated with students beginning level of achievement and allow for all students to demonstrate growth regardless of whether they are low or high achieving. ๏ SGPs can be aggregated for any group of interest to see whether the group is demonstrating, on average, high or low growth. ๏ School/District ๏ Demographic subgroup ๏ Classroom/Teacher ๏ Curriculum/Program

Student Growth Percentiles ๏ Peer based comparisons (norms) are helpful but limited by their very nature. ๏ Norms tell you where you are in a distribution but not whether the ranking is good or bad. ๏ One needs criteria as well as norms to have a complete understanding. ๏ A criterion-referenced interpretation examines the quantity against a set of standards. ๏ A criterion-referenced interpretation tries to answer: ๏ What is “good” growth? ๏ What is “bad” growth? ๏ What is “good enough” growth?

Student Growth Percentiles ๏ Criterion-referenced growth is also referred to as growth-to-standard. ๏ The standards used to examine growth are often the state’s performance standards. ๏ Peer based analyses (e.g., SGP) are used in many states as part of growth-to- standard analyses. ๏ Growth-to-standard analyses produce growth targets determining whether a student’s growth is sufficient to reach/maintain state standards. ๏ STAR interim assessments also provide growth to standard information where the standards are state standards linked to the STAR scale. ๏ Absent norms, criterion-referenced growth produces results similar to achievement indicating: ๏ Students that start behind tend to stay behind. ๏ Students that start ahead tend to stay ahead.

Growth & Accountability ๏ Most of the controversy surrounding student growth analyses centers on their (high stakes) uses: ๏ Teacher evaluation ๏ School evaluation/accreditation ๏ Principal evaluation ๏ There is broad disagreement on whether results derived from large scale standardized assessments (state summative, interim) are appropriate for personnel or institutional evaluation. ๏ Indicators/measures and uses often get conflated leading to push back against the measures (assessments) themselves. ๏ There are many great uses for student growth results (SGPs) besides personnel/institutional evaluation.

Growth & Accountability Non-Accountability Uses of Growth ๏ Progress monitoring of individual students: Is a student on track to college readiness? ๏ Curricula evaluation: Do students receiving different curricula demonstrate different growth? ๏ Ed-System evaluation: Do students at different grade levels in the district grow at the same rate? ๏ Test monitoring: Is their a mode effect in the test administration? ๏ Monitoring progress during standards transition: Were some schools more successful at transition to new standards than other schools in terms of the growth shown?

Growth & Teacher Evaluation ๏ Likely the most controversial use of tests scores. ๏ Began with Tennessee Value-Added Assessment System (TVAAS now EVAAS) in early 90s. ๏ With the availability of longitudinal student assessment data from NCLB, data was available in every state for the purpose. ๏ Increased dramatically in late 2000s/early 2010s under Obama/ Duncan Accountability Policy. ๏ Numerous states also adopted laws requiring teacher evaluation based in some part on test scores. ๏ Might have been the straw (along with Common Core) that broke the “Ed Reform” back.

Growth & Teacher Evaluation Argument #1 for Teacher Evaluation ๏ Value-added methods (VAMs) measure the teacher’s contribution to student learning. ๏ Because student learning is important and we can measure the teacher’s contribution to it, teachers should be held accountable for student learning. ๏ This argument is often proffered by economists. ๏ Arguments against it often focus on the technical characteristics of the VAMs: ๏ Unreliability: Values can change a lot from year-to-year ๏ Invalidity: Examples abound of great teachers receiving low VAM scores. ๏ Campbell’s Law: “The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.”

Growth & Teacher Evaluation Argument #2 for Teacher Evaluation ๏ Student learning is important. ๏ Because student learning is important, providers of education should be held accountable, in part, for student learning. ๏ Indicators of student learning can be derived from assessment scores (e.g., SGP). ๏ Similar to holding a sports coach responsible, in part, for the win/loss percentage of his/her team. ๏ Arguments against it often focus on the technical characteristics of the VAMs: ๏ Unreliability: Values can change a lot from year-to-year. ๏ Invalidity: Examples abound of great teachers receiving low growth scores. ๏ Campbell’s Law (what often happens when winning is valued too much?)

Growth & Teacher Evaluation ๏ Student growth is usually only a minority part of the overall teacher evaluation. ๏ Weights applied to student growth in teacher evaluation vary widely (e.g., not at all to 50%). ๏ ESSA has decreased pressure for new teacher evaluation systems in states. ๏ Evaluation in non-tested grades/subjects has yielded some questionable practices. ๏ Unintended consequences (e.g., Atlanta cheating scandal) provide cautionary tales. ๏ What might Accountability 2.0 look like?

Accountability 2.0 ๏ Based upon general dissatisfaction with accountability systems emerging from the NCLB/Waiver era and new flexibility under ESSA, states are asking what an accountability system might look like. ๏ Design principles: ๏ Accountability done With stakeholders instead of To stakeholders. ๏ Accountability as a flashlight instead of a hammer. ๏ Data used for improvement instead of punishment. ๏ The change from Accountability 1.0 to 2.0 is more cultural than technical.

References ๏ Betebenner, D. W. (2008). Toward a normative understanding of student growth. In Ryan, K. E. and Shepard, L. A., editors, The Future of Test-Based Educational Accountability, pages 155-170. Taylor & Francis, New York. ๏ Betebenner, D. W. (2009). Norm- and criterion-referenced student growth. Educational Measurement: Issues and Practice, 28(4): 42-51. ๏ Betebenner, D. W. (2012). Growth, standards, and accountability. In Cizek, G., editor, Setting Performance Standards: Foundations, Methods, and Innovations, pages 439-450. Routledge, New York. ๏ Betebenner, D. W., VanIwaarden, A., Domingue, B., and Shang, Y. (2016). SGP: Student Growth Percentiles & Percentile Growth Trajectories. (R package version 1.5-0.0)

About this copy

Understanding Student Growth — Center for Assessment RLI, May 2016.

This is an image-based copy of the original Keynote slides; builds and animations are flattened.