Death to Proficiency! Long Live Proficiency!
Where did we go wrong with accountability
Damian Betebenner
Center for Assessment
July 9th, 2024
An Unfulfilled Promise
• I think that most experts (and non-experts) would agree that educational
accountability has not lived up to its promise since it was introduced more than two
decades ago as part of NCLB.
• There are many takes on why accountability didn’t work. I’ll be discussing mine today.
• What was the promise?
• What would have happened had accountability systems “worked”.
• My argument is that current systems are incoherent: The desired outcomes implicit
in the accountability indicator system do not connect with the actions (broadly) of
the actors in the system.
• My argument is not that accountability systems would have fulfilled their promise had
they been designed in a coherent fashion. I can’t re-live the past and find out.
However, building incoherent systems dooms the project from the outset.
An Unfulfilled Promise
• I see evidence of this incoherence in almost every interaction that I have
with clients when discussing accountability and growth.
• These interactions take me back to my time when I was a teacher and
I’d hear students mis-use terms in discussions.
• This misuse reveals a lack of understanding.
• What are some of these terms: “Improvement”, “Growth”,
“Performance”, “Continuous Improvement”
• This misuse rarely gets corrected.
• I would argue that accountability systems have matured to a point that if
we don’t nail down vocabulary, systems will continue to be incoherent.
An Unfulfilled Promise
• Can we use systematically derived evidence (data) to improve
education for students?
• This is the technocratic mission at the heart of educational
accountability.
• Some argue that if we used different data than what is
currently collected, then we’d have a better chance of fulfilling
that mission.
• My argument is that we haven’t even used the data we collect
coherently, so I am skeptical that using other data (often of
lower quality) will lead to a different outcome.
Carlson’s 2x2 Table
• Dale Carlson, former assessment director for the California
Department of Education and former Center board member wrote a
paper more than 20 years ago for the Center that stands (I believe) as
an under-appreciated seminal work in our field.
• Titled Focusing on State Educational Accountability Systems: Four
Methods of Judging School Quality and Progress, at the heart of the
paper is a 2x2 table that describes indicators at the heart of
educational accountability.
Carlson’s 2x2 Table (modified slightly by me)
Point in Time: Over time:
Classroom, school, district, state (or Classroom, school, district, state (or
subgroup thereof) subgroup thereof)
Status Improvement
What is the level of achievement of students in Is the level of achievement of students in this
this classroom, school, district, state (or classroom, school, district, state (or subgroup
subgroup thereof) at this point in time? thereof) increasing/decreasing over time?
Achievement
Growth Acceleration
What is the level of growth/learning of Is the level of growth/learning of students in
students in this classroom, school, district, this classroom, school, district, state (or
state (or subgroup thereof) at this point in subgroup thereof) increasing/decreasing over
time? time?
Growth/Learning
What is the desired outcome
• Improvement is a word often used to describe the goal.
• The term has a common meaning
• The term also has a specific technical meaning that is indicated in
Carlson’s 2x2 table
• Accountability system discussions rarely align with the technical
definitions of these terms (e.g., student growth and improvement are
often conflated — all change isn’t growth)
• I would argue that accountability systems have matured to a point that
if we don’t adhere to technical definitions we will continue to create
incoherent accountability systems.
Mind the Gap
• It’s all about closing gaps:
• Two types of gaps:
• Aspirational achievement gaps (e.g., percent proficient lags behind what
we aspire to).
• Subgroup achievement gaps (e.g., demographic subgroups differ in
terms of outcomes).
• Improvement is synonymous with the closing of these gaps. NCLB, for
example, mapped out long term goals to reach universal proficiency. ESSA
mapped out more modest long term goals.
• But what does improvement entail besides a number (e.g., percent
proficient) just going up over time?
Improvement
• What needs to happen in order for percent proficient to increase?
• For a state education system (or school or district) to improve
(demonstrate higher rates of achievement over time) at least one of
following two things needs to happen:
• Students entering the system need to be higher achieving and/or
• Students in the system need to demonstrate higher rates of
learning.
• The latter is the implicit goal of improvement.
• That is, improvement is synonymous with acceleration in student
learning.
Carlson’s 2x2 Table (modified slightly by me)
Point in Time: Over time:
Classroom, school, district, state (or Classroom, school, district, state (or
subgroup thereof) subgroup thereof)
Status Improvement
What is the level of achievement of students in Is the level of achievement of students in this
this classroom, school, district, state (or classroom, school, district, state (or subgroup
subgroup thereof) at this point in time? thereof) increasing/decreasing over time?
Achievement
Growth Acceleration
What is the level of growth/learning of Is the level of growth/learning of students in
students in this classroom, school, district, this classroom, school, district, state (or
state (or subgroup thereof) at this point in subgroup thereof) increasing/decreasing over
time? time?
Growth/Learning
Two achievement gap scenarios for schools potentially
leading to incorrect inferences regarding school quality
Expanding aspirational achievement gaps at School X Persistent, large subgroup achievement gaps at School Y
White
State
Percent Pro cient Percent Pro cient
African American
School X
Time Time
fi
fi
Improvement
• By definition, School X is not
improving.
• School X’s “performance is
stagnant”.
• Is that bad? Why?
Two Axioms
1. Achievement and Growth (i.e., learning) are different but are
connected: Learning precedes achievement. Learning is a proximal
indicator whereas achievement is distal.
2. Students enter and exit the education system (and its schools and
districts)
• Despite the obviousness of these two statements, I don’t believe that
current accountability systems take account of them. Two things are
changing at one time making it hard to conceptualize.
An illustration
• Achievement is to water
temperature as growth is to the
heat supplied.
• Status based accountability
systems judge the systems
based upon “water
temperature.”
• What the school provides is the
heat!
An illustration
• These systems will reach a state
of thermal equilibrium.
• This is similar to schools: If the
achievement level of students
entering the school and the
rates of learning remain
constant over time,
achievement will reach
equilibrium.
Improvement
• School X’s low achievement MAY mask
high rates of student learning.
• Demanding that School X improve is
equivalent to demanding that they
increase rates of student learning (i.e.,
acceleration)
• If the students at School X are already
learning a lot (i.e., high velocity), there
is no room for improvement (i.e.,
acceleration).
• Status based systems are incoherent
with what the education system and
its components do: Support learning.
Criticisms of Current Accountability
Systems
Two types of criticisms
• There is no shortage of criticisms of current accountability systems.
• Two prominent classes of criticisms include:
1. Indicators: There is significant criticism regarding the narrow
range of indicators, primarily state summative assessment data,
that are heavily relied upon to assess educational quality.
2. Design: The design of accountability systems has also been
widely critiqued for lacking inclusivity in the development
process, consequently detaching the system from the realities
faced by educational practitioners.
Indicators
• Solutions to the over-reliance on state summative testing have
centered on reducing their weight and/or supplementing them with
additional indicators:
• With regard to supplementation, I focus on two classes of indicators)
• Student level indicators (e.g., student perseverance/grit)
• System level indicators (parental satisfaction)
Student Level Indicators
• The difficulty with adding additional student level indicators is that
these indicators need to obey the logic of Carlson’s 2x2 table just as
summative assessment results.
• As I have argued, we don’t have a coherent system with the high
quality indicators from status summative tests. Adding
System Level Indicators
• Though system level indicators don’t obey the same “Carlson logic”,
they add another layer of complexity:
1. Are they included because they are believed to positively influence
student-level performance indicators, such as academic achievement
or growth? If so, is there credible evidence to validate these claims.
2. Are they valued independently of their impact on academic
outcomes—appreciated for their intrinsic benefits to the educational
environment?
3. How susceptible are these indicators to being manipulated within the
system?
Design
• Though I am wholeheartedly committed to inclusive and purposeful
design, I am skeptical that a better and more inclusive process alone
will yield a coherent accountability system.
Three Principles
Principle 1: Learning/growth is the foundation:
Principle 2. Extend Accountability to include Equity & Justice:
Principle 3: Design Purposively and with Inclusivity:
Growth/Learning is Key
“My point here is that we won’t fall o an information cli without state test
scores for every student every year; in fact, we can deliver much better
information about student learning.”
Scott Marion
Do We Really Need to Test All Students Every Year?
• Student learning ≠ Student Achievement (i.e. attainment)
• Assessments measure achievement, not learning
• Learning is an unobserved process that takes place over time.
• We infer learning via multiple observations/assessments taken over time.
ff
ff
Toward Statistical Process Control
• The achievement of students (i.e., status) is the ultimate criterion by which
we should judge our education system.
• There are two levers to increase achievement of students:
• Increase initial levels of achievement of students (early childhood
education)
• Accelerate rates of learning of students.
• Goal is to provide a system level view for leadership and policy makers to
understand how growth (i.e., learning) and achievement are related.
• Desired achievement outcomes can be aligned with the rates of learning
necessary to reach them.
• Learning/growth is key.
“This is the difference between a retrospective question of
identifying fault as opposed to a prospective strategy to engineer
some corrective measure, almost independent of considering
whether there was blame-worthiness. And to move away from the
blame-worthiness paradigm toward something that is more
regulatory in nature where one might seize upon disparities or
circumstances that are for some reason deemed unacceptable and
engineer the interventions needed to bring about the necessary
change. . . . It’s the no-fault gap closing strategy in which the effort
is to build a consensus about a vision of an improved society rather
than gure out where’s the person we want to pillory.”
Christopher Edley (2006)
fi
About this copy
Death to Proficiency! Long Live Proficiency! — Center for Assessment staff meeting, July 9, 2024.
This is an image-based copy of the original Keynote slides; builds and animations are flattened.