The Big Picture
Guiding Principals and Questions
The Colorado Growth Model
New Directions
The Colorado Growth Model
Changing Conversations about Education
Damian W. Betebenner
National Center for the Improvement of Educational Assessment
Dover, New Hampshire
Colorado Assessment Review Stakeholders Meeting
Denver, Colorado December 11th, 2009
Damian W. Betebenner The Colorado Growth Model
The Big Picture
The Data Revolution
Guiding Principals and Questions
Why Growth?
The Colorado Growth Model
Accountability and Growth
New Directions
Data Data Everywhere
Enhanced data acquisition and management has enabled:
Historical records of student achievement
Historical records of student demographics, teachers,
schools, educational programs, . . .
Stakeholder interest in an examination of this longitudinal
data
Interest in examining student achievement over time (student
growth) derives from data availability.
Damian W. Betebenner The Colorado Growth Model
The Big Picture
The Data Revolution
Guiding Principals and Questions
Why Growth?
The Colorado Growth Model
Accountability and Growth
New Directions
The attractiveness of growth
What is growth and why measure it?
Student learning is a central goal of education.
Assessments of student achievement provide evidence of
the current status of student knowledge and
understanding.
Learning is demonstrated by growth in student
achievement from one point in time to another point in
time–not by status at either point time alone.
Damian W. Betebenner The Colorado Growth Model
The Big Picture
The Data Revolution
Guiding Principals and Questions
Why Growth?
The Colorado Growth Model
Accountability and Growth
New Directions
Accountability & Growth
Growth versus Status
Enthusiasm for growth in accountability stems from the
belief that growth and teacher/school quality are more
closely related than status.
Enthusiasm for growth also stems from its potential
diagnostic uses.
How do we judge the use of growth related measures
within an accountability system?
Damian W. Betebenner The Colorado Growth Model
The Big Picture
Guiding Principals and Questions Guiding Principals
The Colorado Growth Model Guiding Questions
New Directions
Guiding principals
John Tukey
It is better to have an approximate answer to the right question
than a precise answer to the wrong question.
George E. P. Box
All models are wrong but some are useful.
Start at the end and work backward
The best growth model answers the questions of relevance to
stakeholders.
Damian W. Betebenner The Colorado Growth Model
The Big Picture
Guiding Principals and Questions Guiding Principals
The Colorado Growth Model Guiding Questions
New Directions
Guiding questions
Growth models address specific questions
Different growth analysis techniques are good at
answering different questions.
It is critical to understand these different questions.
Different questions lead to different conversations which
lead to different uses and outcomes.
Changing conversations about education
Starting with the right questions simplifies development
and motivating the proper use of the growth model results.
It’s all about the conversations you want to have. The
questions set the table for those conversations.
Damian W. Betebenner The Colorado Growth Model
The Big Picture
Guiding Principals and Questions Guiding Principals
The Colorado Growth Model Guiding Questions
New Directions
What are the relevant questions for parents?
Yen (2007), from a state survey of parents, teachers and
administrators, compiled a list of frequently voiced
questions/concerns by stakeholder group.
Parent Questions
Did my child make a year’s worth of progress in a year?
Is my child growing appropriately toward meeting state
standards?
Is my child growing as much in Math as Reading?
Did my child grow as much this year as last year?
Damian W. Betebenner The Colorado Growth Model
The Big Picture
Guiding Principals and Questions Guiding Principals
The Colorado Growth Model Guiding Questions
New Directions
What are the relevant questions for teachers?
Yen (2007), from a state survey of parents, teachers and
administrators, compiled a list of frequently voiced
questions/concerns by stakeholder group.
Teacher Questions
Did my students make a year’s worth of progress in a
year?
Did my students grow appropriately toward meeting state
standards?
How close are my students to becoming Proficient?
Are there students with unusually low growth who need
special attention?
Damian W. Betebenner The Colorado Growth Model
The Big Picture
Guiding Principals and Questions Guiding Principals
The Colorado Growth Model Guiding Questions
New Directions
What are the relevant questions for administrators?
Yen (2007), from a state survey of parents, teachers and
administrators, compiled a list of frequently voiced
questions/concerns by stakeholder group.
Administrator Questions
Did the students in our district/school make a year’s worth
of progress in all content areas?
Are our students growing appropriately toward meeting
state standards?
Does this school/program show as much growth as that
one?
Can I measure student growth even for students who do
not change proficiency categories?
Can I pool together results from different grades to draw
summary conclusions?
Damian W. Betebenner The Colorado Growth Model
The Big Picture
Describing Student Growth
Guiding Principals and Questions
Combining Actual and Aspirational Growth
The Colorado Growth Model
Growth & Effectiveness
New Directions
The Colorado Growth Model
How much growth did a student make? Is it enough?
An important first step in the development of the Colorado
Growth Model was to separate the description of growth
from discussions of responsibility (i.e., accountability).
Incorporating growth into accountability follows from the
viable description of growth.
A viable description of growth facilitates stakeholder
engagement and investigations of responsibility for
good/bad growth.
This leads to greater stakeholder support for
accountability.
Damian W. Betebenner The Colorado Growth Model
The Big Picture
Describing Student Growth
Guiding Principals and Questions
Combining Actual and Aspirational Growth
The Colorado Growth Model
Growth & Effectiveness
New Directions
Describing student growth
Measuring student growth, even with a vertical scale, is not a
simple task.
Some believe a vertical scale simplifies the task of measuring
student growth.
Even with an interval (or ratio) scale, growth is not easy to
interpret. Consider, for example, height.
A child might grow 4 inches between ages 3 and 4.
4 inches is a well understood quantity.
The 4 inch increase becomes really meaningful only when
understood alongside the growth of other 3 to 4 year olds.
Student growth percentiles were developed to provide a
normative context for describing student growth.
Damian W. Betebenner The Colorado Growth Model
The Big Picture
Describing Student Growth
Guiding Principals and Questions
Combining Actual and Aspirational Growth
The Colorado Growth Model
Growth & Effectiveness
New Directions
Student Growth Percentiles
Should we be surprised with a child’s current achievement given
their prior achievement?
Student growth percentiles answer this question.
Consider a low achieving student with 90th percentile growth
and a high achieving student with 10th percentile growth.
The low achieving student grew at a rate exceeding 90 percent
of similar students.
The high achieving student grew at a rate exceeding just 10
percent of similar students.
The low achiever’s growth is more exemplary (probabilistically)
than the high achiever’s.
Judgments about the adequacy of student growth require
external criteria.
Damian W. Betebenner The Colorado Growth Model
The Big Picture
Describing Student Growth
Guiding Principals and Questions
Combining Actual and Aspirational Growth
The Colorado Growth Model
Growth & Effectiveness
New Directions
Combining norms and standards
Growth adequacy is determined by whether a student’s
growth is sufficient to reach/maintain desired achievement
levels (e.g., proficiency).
Percentile growth projections/trajectories are calculated for
each student.
These “growth-to-standard” trajectories indicate what it will
take for the student to reach/maintain proficiency and other
achievement levels.
The Colorado Growth Model was recently approved as part of
the Growth Model Pilot Program using this approach.
Damian W. Betebenner The Colorado Growth Model
Advanced Reading
High
Achievement
CSAP Reading
● Typical
Scale Score
Proficient Growth
Low
● Level Percentiles
High 66th − 99th
Typical 35th − 65th
Part Proficient
● Low 1st − 34th
Unsatisfactory
Grade 3 Grade 4 Grade 5 Grade 6 Next Year
Scale Score 462 539 563 609
Achievement Level Unsatisfactory Part Proficient Part Proficient Proficient Achievement
Growth Percentile 66 66 90
Growth Level High High High Growth
Example Student 12 (1234567812) Sample Middle School
Advanced
Math
Achievement
High
CSAP Math
● Scale Score
Typical
● Growth
Proficient
● Level Percentiles
Low
High 66th − 99th
● Typical 35th − 65th
Part Proficient
Low 1st − 34th
Unsatisfactory
Grade 3 Grade 4 Grade 5 Grade 6 Next Year
Scale Score 379 388 444 462
Achievement Level Part Proficient Part Proficient Part Proficient Part Proficient Achievement
Growth Percentile 16 69 51
Growth Level Low High Typical Growth
Advanced Reading
High
Achievement
CSAP Reading
● Typical
Scale Score
Proficient Growth
Low
● Level Percentiles
High 66th − 99th
Typical 35th − 65th
Part Proficient
● Low 1st − 34th
Unsatisfactory
Grade 3 Grade 4 Grade 5 Grade 6 Next Year
Scale Score 462 539 563 609
Achievement Level Unsatisfactory Part Proficient Part Proficient Proficient Achievement
Growth Percentile 66 66 90
Growth Level High High High Growth
Advanced
Writing
Achievement
High
CSAP Writing
● Scale Score
Typical
● Growth
Proficient
Level Percentiles
● Low
High 66th − 99th
Part Proficient ● Typical 35th − 65th
Low 1st − 34th
Unsatisfactory
Grade 3 Grade 4 Grade 5 Grade 6 Next Year
Scale Score 425 449 469 504
Achievement Level Part Proficient Part Proficient Part Proficient Part Proficient Achievement
Growth Percentile 55 53 79
Growth Level Typical Typical High Growth
Copyright 2009, Colorado Department of Education/Center for Assessment, Inc.
The Big Picture
Describing Student Growth
Guiding Principals and Questions
Combining Actual and Aspirational Growth
The Colorado Growth Model
Growth & Effectiveness
New Directions
Going from students to groups of students
It’s of interest to examine schools where students
demonstrate, on average, extraordinarily high and low student
growth.
To summarize the student growth percentiles associated with
a school (or other grouping) calculate the median of the
student growth percentiles.
If students were randomly assigned to schools, expect to see
a median of 50.
Values greatly above or below 50 are of interest in identifying
best practices or providing extra support.
Examining growth with achievement sheds new light on
school performance.
Damian W. Betebenner The Colorado Growth Model
District C: 2008 CSAP Math School Results
Student Growth versus Student Achievement by Percent Free/Reduced Lunch
School Percent
Higher Achievement Higher Achievement Free/Reduced Lunch
Lower Growth Higher Growth
Percent at/above Proficient in School
Less than 20 percent
● 20 to 40 percent
● 40 to 60 percent
● 60 to 80 percent
● More than 80 percent
School Size
50 Students
100 Students
200 Students
30 500 Students
1,000 Students
10 Lower Growth Higher Growth
Lower Achievement Lower Achievement
Median of Student Growth Percentiles in School
The Big Picture
Describing Student Growth
Guiding Principals and Questions
Combining Actual and Aspirational Growth
The Colorado Growth Model
Growth & Effectiveness
New Directions
Growth, Effectiveness, and Value-Added
Fundamental Premise
“Good” schools bring about student growth in excess of that
found at “bad” schools.
“Good schools” are often called highly effective schools.
What’s the relationship between growth and effectiveness?
Effectiveness indicates who/what is responsible for the growth
(value-added models).
Yen’s questions make no mention of effectiveness.
Yen’s questions indicate stakeholder interest in a description
of student growth.
Damian W. Betebenner The Colorado Growth Model
The Big Picture
Growth & Accountability
Guiding Principals and Questions
Web 2.0 Technologies
The Colorado Growth Model
References
New Directions
Next Steps for the Colorado Growth Model
Integration of the results into an accountability system via the
School Performance Framework currently under development
at CDE.
Using Web 2.0 data visualization and social networking tools
to distribute the data and change conversations about
education.
Extending the work of Colorado nationally. Currently
Massachusetts, Indiana, and Arizona have adopted the model
with numerous others investigating.
Damian W. Betebenner The Colorado Growth Model
The Big Picture
Growth & Accountability
Guiding Principals and Questions
Web 2.0 Technologies
The Colorado Growth Model
References
New Directions
Descriptive Accountability
“Accountability system results can have value without making
causal inferences about school quality, solely from the results of
student achievement measures and demographic characteristics.
Treating the results as descriptive information and for identification
of schools that require more intensive investigation of
organizational and instructional process characteristics are
potentially of considerable value. Rather than using the results of
the accountability system as the sole determiner of sanctions for
schools, they could be used to flag schools that need more
intensive investigation to reach sound conclusions about needed
improvements or judgments about quality.”
R. L. Linn (2008)
Damian W. Betebenner The Colorado Growth Model
The Big Picture
Growth & Accountability
Guiding Principals and Questions
Web 2.0 Technologies
The Colorado Growth Model
References
New Directions
Descriptive Accountability
“This is the difference between a retrospective question of
identifying fault as opposed to a prospective strategy to engineer
some corrective measure, almost independent of considering
whether there was blame-worthiness. And to move away from the
blame-worthiness paradigm toward something that is more
regulatory in nature where one might seize upon disparities or
circumstances that are for some reason deemed unacceptable and
engineer the interventions needed to bring about the necessary
change. . . . It’s the no-fault gap closing strategy in which the effort
is to build a consensus about a vision of an improved society rather
than figure out where’s the person we want to pillory.”
C. Edley (2006)
Damian W. Betebenner The Colorado Growth Model
The Big Picture
Growth & Accountability
Guiding Principals and Questions
Web 2.0 Technologies
The Colorado Growth Model
References
New Directions
Web 2.0: Data Visualization and Social Networking
The Colorado Department of Education and the Center for
Assessment have been working for the last two year on
developing next generation data visualization to accompany
growth model data.
The goal: Transform conversations about education through
active engagement with data (i.e., evidence).
Our efforts have received tremendous interest and
recognition:
Recognized by Adobe for innovative uses of their technology
as an Adobe Max Award finalist in October, 2009.
Multiple states signing MOUs to co-develop a cloud-based
reporting platform in a non-proprietary fashion.
Colorado recently devoted $2.5 million of stimulus funds to the
development efforts
Damian W. Betebenner The Colorado Growth Model
The Big Picture
Growth & Accountability
Guiding Principals and Questions
Web 2.0 Technologies
The Colorado Growth Model
References
New Directions
Web 2.0: Data Visualization and Social Networking
With a collaborative spirit, with a collaborative platform where
people can upload data, explore data, compare solutions, discuss
the results, build consensus, we can engage passionate people,
local communities, media and this will raise–incredibly–the amount
of people who can understand what is going on.
And this would have fantastic outcomes: the engagement of
people, especially new generations; it would increase knowledge,
unlock statistics, improve transparency and accountability of public
policies, change culture, increase numerary, and in the end,
improve democracy and welfare.
E. Giovannini, Chief Statistician, OECD. June 2007
Damian W. Betebenner The Colorado Growth Model
The Big Picture
Growth & Accountability
Guiding Principals and Questions
Web 2.0 Technologies
The Colorado Growth Model
References
New Directions
References
Edley, C. (2006).
Educational “Opportunity" is the highest civil rights priority. So
what should researchers and lawyers do about it?
Retrieved June 22, 2006 from the World Wide Web: http:
//www.softconference.com/MEDIA/WMP/260407/#43.010.
Linn, R. L. (2008).
Educational accountability systems.
In The Future of Test-Based Educational Accountability, pages
3–24. Taylor & Francis, New York.
Yen, W. M. (2007).
Vertical scaling and No Child Left Behind.
In Dorans, N. J., Pommerich, M., and Holland, P. W., editors,
Linking and Aligning Scores and Scales, pages 273–283.
Springer, New York.
Damian W. Betebenner The Colorado Growth Model
About this copy
The Colorado Growth Model: Changing Conversations about Education — Colorado Department of Education, December 11, 2009.
This is an image-based copy of the original PDF slides; builds and animations are flattened.