Death to Proficiency! Long Live Proficiency!

Center for Assessment staff meeting · July 9, 2024

Death to Proficiency! Long Live Proficiency! — Center for Assessment staff meeting, July 9, 2024.
Published

July 9, 2024

Event
Staff session
Slides
32
Format
Converted from Keynote

This deck is an image-based copy of the original slides. Builds and animations are flattened; the slide text is reproduced below for search and screen readers.

Slide text
  1. Death to Proficiency! Long Live Proficiency! Where did we go wrong with accountability Damian Betebenner Center for Assessment July 9th, 2024
  2. Introduction
  3. An Unfulfilled Promise • I think that most experts (and non-experts) would agree that educational accountability has not lived up to its promise since it was introduced more than two decades ago as part of NCLB. • There are many takes on why accountability didn’t work. I’ll be discussing mine today. • What was the promise? • What would have happened had accountability systems “worked”. • My argument is that current systems are incoherent: The desired outcomes implicit in the accountability indicator system do not connect with the actions (broadly) of the actors in the system. • My argument is not that accountability systems would have fulfilled their promise had they been designed in a coherent fashion. I can’t re-live the past and find out. However, building incoherent systems dooms the project from the outset.
  4. An Unfulfilled Promise • I see evidence of this incoherence in almost every interaction that I have with clients when discussing accountability and growth. • These interactions take me back to my time when I was a teacher and I’d hear students mis-use terms in discussions. • This misuse reveals a lack of understanding. • What are some of these terms: “Improvement”, “Growth”, “Performance”, “Continuous Improvement” • This misuse rarely gets corrected. • I would argue that accountability systems have matured to a point that if we don’t nail down vocabulary, systems will continue to be incoherent.
  5. An Unfulfilled Promise • Can we use systematically derived evidence (data) to improve education for students? • This is the technocratic mission at the heart of educational accountability. • Some argue that if we used different data than what is currently collected, then we’d have a better chance of fulfilling that mission. • My argument is that we haven’t even used the data we collect coherently, so I am skeptical that using other data (often of lower quality) will lead to a different outcome.
  6. Background Concepts
  7. Carlson’s 2x2 Table • Dale Carlson, former assessment director for the California Department of Education and former Center board member wrote a paper more than 20 years ago for the Center that stands (I believe) as an under-appreciated seminal work in our field. • Titled Focusing on State Educational Accountability Systems: Four Methods of Judging School Quality and Progress, at the heart of the paper is a 2x2 table that describes indicators at the heart of educational accountability.
  8. Carlson’s 2x2 Table (modified slightly by me) Point in Time: Over time: Classroom, school, district, state (or Classroom, school, district, state (or subgroup thereof) subgroup thereof) Status Improvement What is the level of achievement of students in Is the level of achievement of students in this this classroom, school, district, state (or classroom, school, district, state (or subgroup subgroup thereof) at this point in time? thereof) increasing/decreasing over time? Achievement Growth Acceleration What is the level of growth/learning of Is the level of growth/learning of students in students in this classroom, school, district, this classroom, school, district, state (or state (or subgroup thereof) at this point in subgroup thereof) increasing/decreasing over time? time? Growth/Learning
  9. What is the desired outcome • Improvement is a word often used to describe the goal. • The term has a common meaning • The term also has a specific technical meaning that is indicated in Carlson’s 2x2 table • Accountability system discussions rarely align with the technical definitions of these terms (e.g., student growth and improvement are often conflated — all change isn’t growth) • I would argue that accountability systems have matured to a point that if we don’t adhere to technical definitions we will continue to create incoherent accountability systems.
  10. Mind the Gap • It’s all about closing gaps: • Two types of gaps: • Aspirational achievement gaps (e.g., percent proficient lags behind what we aspire to). • Subgroup achievement gaps (e.g., demographic subgroups differ in terms of outcomes). • Improvement is synonymous with the closing of these gaps. NCLB, for example, mapped out long term goals to reach universal proficiency. ESSA mapped out more modest long term goals. • But what does improvement entail besides a number (e.g., percent proficient) just going up over time?
  11. Improvement • What needs to happen in order for percent proficient to increase? • For a state education system (or school or district) to improve (demonstrate higher rates of achievement over time) at least one of following two things needs to happen: • Students entering the system need to be higher achieving and/or • Students in the system need to demonstrate higher rates of learning. • The latter is the implicit goal of improvement. • That is, improvement is synonymous with acceleration in student learning.
  12. Carlson’s 2x2 Table (modified slightly by me) Point in Time: Over time: Classroom, school, district, state (or Classroom, school, district, state (or subgroup thereof) subgroup thereof) Status Improvement What is the level of achievement of students in Is the level of achievement of students in this this classroom, school, district, state (or classroom, school, district, state (or subgroup subgroup thereof) at this point in time? thereof) increasing/decreasing over time? Achievement Growth Acceleration What is the level of growth/learning of Is the level of growth/learning of students in students in this classroom, school, district, this classroom, school, district, state (or state (or subgroup thereof) at this point in subgroup thereof) increasing/decreasing over time? time? Growth/Learning
  13. Two achievement gap scenarios for schools potentially leading to incorrect inferences regarding school quality Expanding aspirational achievement gaps at School X Persistent, large subgroup achievement gaps at School Y White State Percent Pro cient Percent Pro cient African American School X Time Time fi fi
  14. Improvement • By definition, School X is not improving. • School X’s “performance is stagnant”. • Is that bad? Why?
  15. Two Axioms 1. Achievement and Growth (i.e., learning) are different but are connected: Learning precedes achievement. Learning is a proximal indicator whereas achievement is distal. 2. Students enter and exit the education system (and its schools and districts) • Despite the obviousness of these two statements, I don’t believe that current accountability systems take account of them. Two things are changing at one time making it hard to conceptualize.
  16. An illustration
  17. An illustration • Achievement is to water temperature as growth is to the heat supplied. • Status based accountability systems judge the systems based upon “water temperature.” • What the school provides is the heat!
  18. An illustration • These systems will reach a state of thermal equilibrium. • This is similar to schools: If the achievement level of students entering the school and the rates of learning remain constant over time, achievement will reach equilibrium.
  19. Improvement • School X’s low achievement MAY mask high rates of student learning. • Demanding that School X improve is equivalent to demanding that they increase rates of student learning (i.e., acceleration) • If the students at School X are already learning a lot (i.e., high velocity), there is no room for improvement (i.e., acceleration). • Status based systems are incoherent with what the education system and its components do: Support learning.
  20. Criticisms of Current Accountability Systems
  21. Two types of criticisms • There is no shortage of criticisms of current accountability systems. • Two prominent classes of criticisms include: 1. Indicators: There is significant criticism regarding the narrow range of indicators, primarily state summative assessment data, that are heavily relied upon to assess educational quality. 2. Design: The design of accountability systems has also been widely critiqued for lacking inclusivity in the development process, consequently detaching the system from the realities faced by educational practitioners.
  22. Indicators • Solutions to the over-reliance on state summative testing have centered on reducing their weight and/or supplementing them with additional indicators: • With regard to supplementation, I focus on two classes of indicators) • Student level indicators (e.g., student perseverance/grit) • System level indicators (parental satisfaction)
  23. Student Level Indicators • The difficulty with adding additional student level indicators is that these indicators need to obey the logic of Carlson’s 2x2 table just as summative assessment results. • As I have argued, we don’t have a coherent system with the high quality indicators from status summative tests. Adding
  24. System Level Indicators • Though system level indicators don’t obey the same “Carlson logic”, they add another layer of complexity: 1. Are they included because they are believed to positively influence student-level performance indicators, such as academic achievement or growth? If so, is there credible evidence to validate these claims. 2. Are they valued independently of their impact on academic outcomes—appreciated for their intrinsic benefits to the educational environment? 3. How susceptible are these indicators to being manipulated within the system?
  25. Design • Though I am wholeheartedly committed to inclusive and purposeful design, I am skeptical that a better and more inclusive process alone will yield a coherent accountability system.
  26. A New Hope
  27. Three Principles Principle 1: Learning/growth is the foundation: Principle 2. Extend Accountability to include Equity & Justice: Principle 3: Design Purposively and with Inclusivity:
  28. Equity & Justice
  29. Growth/Learning is Key “My point here is that we won’t fall o an information cli without state test scores for every student every year; in fact, we can deliver much better information about student learning.” Scott Marion Do We Really Need to Test All Students Every Year? • Student learning ≠ Student Achievement (i.e. attainment) • Assessments measure achievement, not learning • Learning is an unobserved process that takes place over time. • We infer learning via multiple observations/assessments taken over time. ff ff
  30. Toward Statistical Process Control • The achievement of students (i.e., status) is the ultimate criterion by which we should judge our education system. • There are two levers to increase achievement of students: • Increase initial levels of achievement of students (early childhood education) • Accelerate rates of learning of students. • Goal is to provide a system level view for leadership and policy makers to understand how growth (i.e., learning) and achievement are related. • Desired achievement outcomes can be aligned with the rates of learning necessary to reach them. • Learning/growth is key.
  31. “This is the difference between a retrospective question of identifying fault as opposed to a prospective strategy to engineer some corrective measure, almost independent of considering whether there was blame-worthiness. And to move away from the blame-worthiness paradigm toward something that is more regulatory in nature where one might seize upon disparities or circumstances that are for some reason deemed unacceptable and engineer the interventions needed to bring about the necessary change. . . . It’s the no-fault gap closing strategy in which the effort is to build a consensus about a vision of an improved society rather than gure out where’s the person we want to pillory.” Christopher Edley (2006) fi
  32. Questions