The Idaho Growth Model
Changing Conversations about Education
www.schoolview.org
Richard J. Wenning & Damian W. Betebenner
rwenning99@gmail.com Center for Assessment
www.twitter.com/rwenning Dover, New Hampshire
Accountability Complexity
• Accountability for educator effectiveness
now layered onto systems for student,
school, district, state and federal
accountability
• Better when these multiple layers are
aligned to support the business we are in
What Business Are We In?
• Maximize student progress toward college and
career readiness
– Implies new bright line: all kids ready by exit
– Requires a definition of readiness and the
standards leading there
– Requires measurement system that determines
how well students are progressing toward and
reaching the destination
Policy Perspective on Growth
• Why is measuring student growth so
important?
– NCLB (Accountability 1.0) had right intent but…
• AYP metric not useful for school performance
management
• Incentives focused on short-term increases in percent
proficient, on “bubble” kids, invited moral hazard
• Instead of long-term effectiveness and progress for all
kids toward college & career readiness
– ESEA waivers & reauthorization provides
opportunity to get the measures & incentives right
Desired Level of Performance?
• Dramatic, not incremental improvements
required for students that need to catch
up to become college and career ready
• Implies that our accountability systems
should should provide information that
fuels a consensus for change and
capacity for improvement
Desired System: Accountability 2.0
• Coherent system focused on learning and
building student, educator, school, district,
state and federal performance management
capacity.
– Maximize local ownership of high quality
information to drive insight and action
• We should ensure educator effectiveness
conversation not stuck in Accountability 1.0
Designing a Coherent System for
Performance Management
What Models?
• What statistical models of longitudinal student
growth will promote the most coherence and
alignment in our accountability system?
Some Framing Ideas
• We understand best those things we see
emerge from their very beginnings.
--Aristotle
• All Models are wrong but some are useful.
--George E. P. Box
• It is better to have an approximate answer to
the right question than a precise answer to the
wrong question.
--John Tukey
Consequential Validity
• Henry Braun (2008)
– Assessment practices and systems of
accountability are consequentially valid if they
generate useful information and constructive
responses that support one or more policy goals
(Access, Quality, Equity, Efficiency) within an
education system, without causing undue
deterioration with respect to other goals.
Marshaling a Consensus for Change
“This is the difference between a retrospective question of
identifying fault as opposed to a prospective strategy to
engineer some corrective measure, almost independent of
considering whether there was blame-worthiness. And to
move away from the blame-worthiness paradigm toward
something that is more regulatory in nature where one
might seize upon disparities or circumstances that are for
some reason deemed unacceptable and engineer the
interventions needed to bring about the necessary change.
. . . It’s the no-fault gap closing strategy in which the effort is
to build a consensus about a vision of an improved society
rather than figure out where’s the person we want to pillory.”
Christopher Edley (2006)
Questions Set the Table
• Growth models address specific questions
– Different techniques are good at answering
different questions
– Different questions lead to different conversations
which lead to different uses and outcomes
– Starting with the right questions simplifies
development and motivates the proper use of the
growth model results
Policy Q’s: Educator Effectiveness
• What questions do we want to answer about growth
rates of students associated with educators?
– What matters and when? Student growth rates as
evidence of effectiveness or ineffectiveness?
• How many categories of effectiveness and
ineffectiveness are important?
– Which categories are consequential and for what?
• What body of evidence will be used to evaluate and
infer teacher value-added?
• Normative and criterion-referenced growth?
How much growth did a student
make and is it enough?
• Describing growth versus ascribing responsibility
– The Colorado Growth Model began by separating the
description of growth from discussions of responsibility/
accountability
– Incorporating growth into accountability followed from the
accepted description of growth
– The description of growth facilitated stakeholder engagement
and investigations of responsibility for good/bad growth
– That in turn led to greater stakeholder support for particular
forms of accountability
Describing Student Growth
• Discussing student growth, even with a vertical scale,
is not a simple task
• Growth and change require context. Consider, for
example, height:
– A child might grow 4 inches between ages 3 and 4
• 4 inches is a well understood quantity
– The 4 inch increase becomes meaningful only when
understood alongside the growth of other 3 to 4 year olds
• Student growth percentiles were developed to
provide a norm-referenced basis for describing
student growth
Who/What is Responsible for Student
Growth?
• Some analyses of student growth attempt to
determine the amount of student progress that can
be attributed to the school or teacher
– Called value-added analyses, these techniques attempt to
estimate the teacher/school contribution to student
academic growth
• Value added is an inference – a causal conclusion
drawn from the data
• All growth models can be used for value-added
purposes
Colorado Growth Model Asks…
What is? How much growth did a child make in
one year?
What should be? How much growth is enough to reach
college & career readiness?
What could be? How much growth have other students
made with the same starting point?
Student Growth Percentiles
• Should we be surprised with a child’s current achievement given
their prior achievement?
– Student growth percentiles answer this question
• Consider a low achieving student with 90th percentile growth and
a high achieving student with 10th percentile growth
– The low achieving student grew at a rate exceeding 90 percent of similar
students
– The high achieving student grew at a rate exceeding just 10 percent of
similar students
– The low achiever’s growth is more exemplary (probabilistically) than the
high achiever’s
• Judgments about the adequacy of student growth require
external criteria together with standard setting
600 co
09 600 re
le
ca le ca
S re 10
co
600 co
09 600 re
le
ca le ca
S re 10
co
600 co
09 600 re
le
ca le ca
S re 10
co
co
th
re
ro
i le
le
ce
nt ca
er
th
re
600 co
le
ca
Establishing Growth Standards Based
Upon Growth Norms
• The most common adequacy criterion is judging growth
toward an achievement goal (i.e., growth-to-standard)
• Results from student growth percentile analyses can be
used to calculate percentile growth trajectories for each
student
• These trajectories indicate what future rates of growth
will lead to and are used to make adequacy judgments
• This growth-to-standard approach was approved as part
of Colorado’s successful application to the Growth Model
Pilot Program
Understanding Student Growth Percentiles
Academic
Peers
My prior CSAP
Achievement
Prior Year
= Student
Growth
Percentile ( Low
Typical
My Growth Compared
High
CSAP Achievement to My Academic Peers
What is Student Adequate Growth Percentile (AGP)?
My Prior CSAP
Distance to or from Proficiency
3 Years or
By Grade 10*
Adequate
Growth
Percentile
Achievement
*Whichever comes first.
One Student’s Growth Percentiles
Students in a Grade in a School
Views of Schools within a District
15 States with MOU
• 15 states have signed MOU to use the Colorado
Growth Model growth percentile methodology
and SchoolView display tools:
– Arizona, Colorado, Georgia, Indiana, Kansas,
Massachusetts, Missouri, Nevada, New Hampshire, New
York, Rhode Island, Virginia, Washington, West Virginia,
Wisconsin
• Pursue cloud-based analytics and visualizations
• Creative Commons Share Alike Noncommercial
Use: http://creativecommons.org/licenses/
Colorado Growth Model Shows…
• Growth not correlated with how close a student is
to proficient
• Normative growth gaps narrowing among student
groups
• Most low achieving students not growing fast
enough to catch up
• Many proficient students not keeping up
• Many high achieving schools show low growth and
many low achieving schools show high growth
Growth Gaps in Reading Closing over Time
Gap in MGP
4 Black-White Gap
3 Hispanic-White Gap
2 FRL-Non FRL Gap
Growth Gaps in Math Closing over Time
Gap in MGP
4 Black-White Gap
3 Hispanic-White Gap
2 FRL-Non FRL Gap
Percentage of Students Growing Enough to
Catch Up and Keep Up in Reading
80 Keeping Up
% of All Students
50 Non-FRL
30 FRL
20 Catching Up
Percentage of Students Growing Enough to
Catch Up and Keep Up in Math
% of All Students
Keeping Up
30 Non-FRL
20 Catching Up FRL
About this copy
The Idaho Growth Model: Changing Conversations about Education — Idaho Department of Education, August 2, 2011.
This is an image-based copy of the original PowerPoint slides; builds and animations are flattened.