Talks archive, 2020–2026 – A Focus on Student Learning

A Focus on Student Learning An Introduction to Student Growth Percentiles and their use in New Mexico Dr. Damian Betebenner National Center for the Improvement of Educational Assessment September 10th, 2024

Success is a journey, not a destination. The doing is often more important than the outcome. Arthur Ashe

Know what you want to accomplish and focus on the process rather than the outcome. Nick Saban

Education is not the filling of a pail, but the lighting of a fire. William Butler Yeats

Outcomes, outcomes, outcomes! • I want to be successful. • I want to be thin. • I want to be admired. We live in a society • I want to be powerful. obsessed with ends • I want to be famous. • I want to have a million followers. • I want to be the best. • I want to be liked. • I want to have it all. • I want to be the fastest. • I want to have the perfect family. • I want to win.

Just win, baby! Al Davis

A Focus on Outcomes (at the expense of a focus on process) Process vs. Product Means vs. Ends Journey vs. Destination Development vs. Result Learning vs. Attainment Growth Vs. Achievement

So what does all of this have to do with student academic growth?

A Focus on Outcomes (at the expense of a focus on process) Process vs. Product Means vs. Ends Journey vs. Destination Development vs. Result Learning vs. Attainment Growth Vs. Achievement

Accountability Today

Accountability and Outcomes • The central focus of educational accountability systems since NCLB has been academic outcomes, largely derived from large scale, state administered summative assessments. • Student academic proficiency has been the main determinant of education quality. • NCLB mandated universal proficiency (100% proficient) by 2014 • ESSA mandates more modest proficiency targets. • I am highly critical of the way that educational accountability has been conceived over the last two decades, but not for the reasons you might think: • I am a strong proponent of annual, large scale, summative assessments. • I am a strong proponent of reporting achievement outcomes like student academic proficiency (I.e., percent proficient) • Outcomes from large scale, standardized summative assessments like academic proficiency are important, but there’s a rub.

The Problem • Not all outcomes derived from state summative assessments are created equal. • Distal outcomes: Achievement/attainment such as academic proficiency • Proximal outcomes: Growth/learning such as student growth percentiles (SGP) • Growth & Achievement (Learning & Attainment) are often treated as conceptually separate but they are intricately related: Growth/Learning leads to Achievement/Attainment • A desire for high achieving students is necessarily a desire for students who grow/learn a lot and vice- versa. • I often say Growth/Learning and Achievement/Attainment are opposite sides of the same coin. • Current accountability systems ignore this relationship between growth and achievement and often present growth and achievement as a value-laden choice (“Which do you prefer, growth or achievement?”) • Treating growth and achievement separately leads to system incoherence: Desired outcomes (proficiency) are divorced from the efforts of teachers, schools, … to support and maximize the learning of students.

Remember: Know what you want to accomplish and focus on the process rather than the outcome. Nick Saban

Remember: Know what you want to accomplish and focus on the process rather than the outcome. Nick Saban • Yes, we want all students to be proficient. • That’s analogous to declaring that we want to win all of our games. • Accomplishing such goals requires a focus on process (highly effective processes). • The process by which students become proficient is learning. • Coherent accountability focuses on student learning without losing sight of the end goal of student proficiency.

Student Growth • Student growth connects process with outcomes. • Student growth is to learning as student attainment is to achievement. • It is important to note that no single assessment measures student learning/growth. • Assessments (plural), including MSSA, measure achievement/attainment: What a student knows and can do at a given point in time. • Learning is a process that happens over time. • We use multiple assessments of achievement to infer whether or not learning has occurred.

The Solution

Student Growth/Learning: If you care about student achievement, then you care about student growth. • Where you are is a function of where you started and how quickly you’ve travelled since. • Student growth and student attainment/achievement are related in this way. • Distance=Rate x Time • Those who start behind tend to stay behind and those that start ahead tend to stay ahead.

Student Growth - Calculations • There is a common misconception that because student growth can be difficult to calculate it is difficult to understand. • Calculations may be complicated but knowing how to calculate student growth is not synonymous with understanding student growth: – The number π (pi=3.14159…) is difficult to calculate but is simple to understand conceptually. – Height and weight percentiles associated with children are difficult to calculate but are simple to understand conceptually. – Credit scores are difficult to calculate but we all know what they mean practically. • Computational details are important but often obscure a simple conceptual basis. • Not knowing how something is calculated does not imply that one cannot understand the quantities.

A Math Story Problem • Whose high jump changed more? Personal Bests in 2023 and 2024 for Two High Jumpers Explain your answer. World Record: 6 feet, 10 1/2 inches • Whose high jump change is more remarkable? Explain your answer. 7 6 feet, 6 inches Judy • Which hit jumper improved more? Height (feet) 6 6 feet, 4 inches Explain your answer. 4 feet, 0 inches 4 Anna 3 feet, 6 inches Spring 2023 Spring 2024 Track Season

Student Growth • Growth as an amount of change: – What could be more simple? – Like height or weight just calculate the amount of change. – Requires a vertical scale (which New Mexico DOESN’T have). – Growth = Score Time 2 - Score Time 1 • Not as simple as it seems: – Comparisons between students often leads to wrong conclusions. – Interpreting change requires context. – Even if New Mexico MSSA had a vertical scale, you’ll see that it doesn’t give you want you think.

Student Growth Back to the story problem! Personal Bests in 2023 and 2024 for Two High Jumpers World Record: 6 feet, 10 1/2 inches 6 feet, 6 inches Judy Height (feet) 6 6 feet, 4 inches 4 feet, 0 inches 4 Anna 3 feet, 6 inches Spring 2023 Spring 2024 Track Season

Student Growth • Which high jumper improved more: – Judy improved 2 inches. Personal Bests in 2023 and 2024 for Two High Jumpers World Record: 6 feet, 10 1/2 inches – Anna improved 6 inches. 8 – An inch is an inch!!! 6 feet, 6 inches – But is an inch really an inch??? Judy Height (feet) – 2 inches for an elite jumper is far more 6 6 feet, 4 inches remarkable than 6 5 inches for a novice. 4 feet, 0 inches 4 Anna 3 feet, 6 inches Spring 2023 Spring 2024 Track Season

Student Growth • The same holds true for the vertically scaled tests (e.g., SBAC). Average Scale Score Change 2022 to 2023 Top 10% ELA Grade 4 2023: -7.5 Bottom 10% ELA Grade 4 2023: +24.5 Top 10% Mathematic Grade 4 2023: -12.6 Bottom 10% Mathematics Grade 4 2023: +18.4 • Large increases for the lowest 10% of achievers, small (negative) increases for high achievers. • Just as an inch isn’t necessarily an inch; a scale score point isn’t necessarily a scale score point. • Moral of the story: There’s more to student growth than subtraction (and in New Mexico you aren’t supposed to subtract student scores with the MSSA)

Student Growth • This is analogous to height/weight measures for infants. • Growth amounts are tricky to interpret and require context. • Peer based comparisons used by growth analyses are a way to acquire that context. An infant male toddler is measured at 2 and 3 years of age and is shown to have grown 4 inches. The magnitude of increase—4 inches—is a well understood quantity that any parent can grasp and calculate at home using a simple yardstick. However, parents leaving their pediatrician’s office knowing only how much their child has grown would likely be wanting for more information: Parents are not interested in an absolute magnitude of growth, but instead in a norm-referenced quantity locating that 4 inch increase alongside the height increases of similar children.

Student Growth • Peer-based analyses are also called norm-referenced analyses. • Like with infant height/weight charts, peers are children/student with similar characteristics. • For growth analyses using large scale assessments, peers are students with (at least) the same scale score history on the same tests. • Peer based norms are invaluable in providing context for stakeholders to understand student growth. • Peer based norms are possible even across test content areas or test transitions. • This is the mathematical basis of Student Growth Percentiles, as the following slides illustrate.

Student Growth Percentiles - Academic Peers 460 co 23 360 re S le S ca le ca S re 24 co

Student Growth Percentiles - Academic Peers th er nt i ce le G wth ro 460 co 23 360 re S le S ca le S ca S re 20 24 co

Student Growth Percentiles - Academic Peers Pe 3 i re = de m co Ac cale 202 75% 75th er nt i ce le G wth ro 460 co 23 360 re S le S ca le S ca S re 20 24 co

Student Growth Percentiles - Academic Peers Th is d 7 5 t h e r s: 6 0 dat erive perc acaa and from ntile Pe 3 dem co a cu i re = ic p vers ll the t de m co eer a ll g ro Ac cale ups th er nt i ce le G wth ro 460 co 23 360 re S le S ca le S ca S re 20 24 co

Student Growth Percentiles - Academic Peers Pe 3 Th i re = e is d 7 5 t h de m co dat erive perc acaa and from ntile Ac cale dem co a cu ic p vers ll the t 3 S eer a ll 20 2 g ro ups 75% 75th er nt ce le G wth ro 460 co 23 360 re le ca le ca S re 24 co

Student Growth • Peer-based growth calculations are only as good as the assessments on which they are based. • With quality assessments, peer based analyses can allow for all students to demonstrate growth. • However, ceiling or floor effects on a test can impede the analysis of growth for students at the extremes. • The goal is to be able to allow various stakeholders groups to investigates student growth/learning and answer questions relevant to them. • What are some of these questions (Yen, 2007).

Relevant Questions for Parents • Did my child make a year’s worth of learning in a year? • Is my child growing appropriately toward meeting state standards? • Is my child growing as much in Math as Reading? • Did my child grow as much this year as last year?

Relevant Questions for Teachers • Did my students make a year’s worth of progress in a year? • Did my students grow appropriately toward mee ng state standards? • How close are my students to becoming Pro cient? • Are there students with unusually low growth who need special a en on? ti tt ti fi

Relevant Questions for Administrators • Did the students in our district/school make a year’s worth of progress in all content areas? • Are our students growing appropriately toward mee ng state standards? • Does this school/program show as much growth as that one? • Can I measure student growth even for students who do not change pro ciency categories? • Can I pool together results from di erent grades to draw summary conclusions? ti ff fi

Growth is achievement over time

Growth is achievement over time • The figure depicts a student’s ELA achievement in Grade 3/2022 and Grade 4/2023. • The student was Near Proficient in both years but their growth (SGP = 72) was high (> 1 year of learning). • Evidence suggests the student in on fire in ELA (remember the Yeats quote). • Compare this with the student’s mathematics performance.

Growth is achievement over time

Growth is achievement over time • Like ELA, the student’s is Near Proficient in both Grade 3/2022 and Grade 4/2023. • Yet their growth from 2022 to 2023 is < 1 year. • Why robust learning in ELA but not in math? • What strategies can 5th grade teachers use to continue the ELA trend and reverse the math trend?

Carina Roper Sample School How to interpret this student Advanced English growth & achievement report Language Arts Achievement Proficient MSSA MSSA English Language Arts > 1 Year Scale Score Scale Score Growth Level Percentiles About 1 Year MSSA Near Proficient > 1 Year 66th − 99th Achievement Levels < 1 Year About 1 Year35th − 65th < 1 Year 1st − 34th Novice Student Growth Grade 3 Grade 4 Grade 5 Percentile Scale Score 341 453 Achievement Level Near Proficient Near Proficient Achievement Growth Percentile 72 Growth Level > 1 Year Growth Suggested Uses Advanced Review past growth to assess student academic progress toward Math MSSA achievement goals. Achievement Proficient Develop remediation or enrich− MSSA Math ment plans based on rate of Scale Score > 1 Year growth needed to reach higher MSSA achievement levels. Growth Level Percentiles Identify the rate of progress Near Proficient needed in order to reach or About 1 Year > 1 Year 66th − 99th maintain proficient status on the MSSA next year. About 1 Year35th − 65th < 1 Year Novice < 1 Year 1st − 34th Grade 3 Grade 4 Grade 5 Scale Score 352 448 Achievement Level Near Proficient Near Proficient Achievement Growth Percentile 34 Growth Level < 1 Year Growth For more information please visit the New Mexico Public Education Department (NMPED) at http://ped.state.nm.us or contact 505−827−5800 Cooperatively developed by NMPED & the Center for Assessment, Inc. Distributed by NMPED.

Changing conversations about education • Accountability together with disconnected point in me assessments have led to a focus on outcomes (achievement) at the expense of learning/ growth. • This focus has led to incoherence in the way we understand the performance of our educa on systems and the way in which we hold them accountable. • How much students grow/learn is directly connected to later levels of achievement in the same way velocity is related to distance traveled. • The purpose of New Mexico’s Growth Model/SGPs is to make these connec ons apparent and change stakeholder conversa ons about educa on and student performance. ti ti ti ti ti

2024 US Roadtrip

Thank you for Inviting me

www.nciea.org

About this copy

A Focus on Student Learning — New Mexico Department of Education, September 10, 2024.

This is an image-based copy of the original Keynote slides; builds and animations are flattened.