The Big Picture
Guiding Principles and Questions
The Colorado Growth Model
New Directions
Exploring Student Growth in Missouri
Changing Conversations about Education
Damian W. Betebenner
National Center for the Improvement of Educational Assessment
Dover, New Hampshire
Missouri Growth Model Technical Advisory Committee
Missouri DESE, Kansas City, Missouri September 16th, 2010
Damian W. Betebenner Student Growth in Missouri
The Big Picture
Guiding Principles and Questions
The Colorado Growth Model
New Directions
About Me
Dr. Damian Betebenner, PhD
- Senior Associate at the Center for Assessment (NCIEA).
- Developed student growth percentiles and percentile
growth trajectories to help states and educational
associations employ student growth in decision
making [Betebenner, 2008, Betebenner, 2009].
- Currently refining and sharing these techniques with other
states including Colorado, Massachusetts, Arizona,
Indiana, and 15 other states in various stages of
investigation/adoption.
Damian W. Betebenner Student Growth in Missouri
The Colorado Growth Model
Adopted Model
Near Model Adoption
Interested in Model
The Big Picture
The Data Revolution
Guiding Principles and Questions
Why Growth?
The Colorado Growth Model
Accountability and Growth
New Directions
From a Paucity to a Plethora of Data
The Economist: The Data Deluge
Historical records of student achievement
Historical records of student demographics, teachers,
schools, educational programs, . . .
Stakeholder interest in an examination of this longitudinal
data
Interest in examining student achievement over time (student
growth) derives from data availability.
Damian W. Betebenner Student Growth in Missouri
The Big Picture
The Data Revolution
Guiding Principles and Questions
Why Growth?
The Colorado Growth Model
Accountability and Growth
New Directions
The Attractiveness of Growth
What is growth and why measure it?
Student learning is a central goal of education.
Assessments of student achievement provide evidence of
the current status of student knowledge and
understanding.
Learning is demonstrated by growth in student
achievement from one point in time to another point in
time–not by status at either point time alone.
Damian W. Betebenner Student Growth in Missouri
The Big Picture
The Data Revolution
Guiding Principles and Questions
Why Growth?
The Colorado Growth Model
Accountability and Growth
New Directions
Accountability & Growth
Growth versus Status
Enthusiasm for growth in accountability stems from the
belief that growth and teacher/school quality are more
closely related than status.
Enthusiasm for growth also stems from its potential
diagnostic uses.
How do we judge the use of growth related measures
within an accountability system?
Damian W. Betebenner Student Growth in Missouri
The Big Picture
Guiding Principles and Questions Guiding Principles
The Colorado Growth Model Guiding Questions
New Directions
Guiding Principles
John Tukey
It is better to have an approximate answer to the right question
than a precise answer to the wrong question.
George E. P. Box
All models are wrong but some are useful.
Start at the end and work backward
The best model answers the questions of greatest relevance to
stakeholders.
Damian W. Betebenner Student Growth in Missouri
The Big Picture
Guiding Principles and Questions Guiding Principles
The Colorado Growth Model Guiding Questions
New Directions
Guiding Questions
Growth models address specific questions
Different growth analysis techniques are good at
answering different questions.
It is critical to understand these different questions.
Different questions lead to different conversations which
lead to different uses and outcomes.
Changing conversations about education
Starting with the right questions simplifies development
and motivating the proper use of the growth model results.
The questions set the table for those conversations. See
Yen (2007) for an excellent list of questions derived from a
survey of parent, teachers and administrators
Damian W. Betebenner Student Growth in Missouri
The Big Picture Describing Student Growth
Guiding Principles and Questions Model & Software for Student Growth Percentiles
The Colorado Growth Model Combining Actual and Aspirational Growth
New Directions Growth & Effectiveness
The Colorado Growth Model
How much growth did a student make? Is it enough?
An important first step in the development of the Colorado
Growth Model was to begin discussions by separating the
description of growth from discussions of
responsibility/accountability.
Incorporating growth into accountability followed more
easily from the accepted description of growth.
The description of growth facilitated stakeholder
engagement and investigations of responsibility for
good/bad growth.
That in turn led to greater stakeholder support for
particular forms of accountability.
Damian W. Betebenner Student Growth in Missouri
The Big Picture Describing Student Growth
Guiding Principles and Questions Model & Software for Student Growth Percentiles
The Colorado Growth Model Combining Actual and Aspirational Growth
New Directions Growth & Effectiveness
Describing Student Growth
Discussing student growth, even with a vertical scale, is not a
simple task.
Some believe a vertical scale simplifies the task of measuring
student growth.
Even with an interval (or ratio) scale, growth and change
require context. Consider, for example, height.
A child might grow 4 inches between ages 3 and 4.
4 inches is a well understood quantity.
The 4 inch increase becomes really meaningful only when
understood alongside the growth of other 3 to 4 year olds.
Student growth percentiles were developed to provide a
norm-referenced basis for describing student growth.
Damian W. Betebenner Student Growth in Missouri
20 core
ca ca le
le 400
S re 2006
co
20 core
ca ca le
le 400
S re 2006
co
20 core
ca ca le
le 400
S re 2006
co
20 core
ca ca le
le 400
S re 2006
co
The Big Picture Describing Student Growth
Guiding Principles and Questions Model & Software for Student Growth Percentiles
The Colorado Growth Model Combining Actual and Aspirational Growth
New Directions Growth & Effectiveness
Student Growth Percentiles
Should we be surprised with a child’s current achievement given
their prior achievement?
Student growth percentiles answer this question.
Consider a low achieving student with 90th percentile growth
and a high achieving student with 10th percentile growth.
The low achieving student grew at a rate exceeding 90 percent
of similar students.
The high achieving student grew at a rate exceeding just 10
percent of similar students.
The low achiever’s growth is more exemplary (probabilistically)
than the high achiever’s.
Judgments about the adequacy of student growth require
external criteria together with standard setting.
Damian W. Betebenner Student Growth in Missouri
The Big Picture Describing Student Growth
Guiding Principles and Questions Model & Software for Student Growth Percentiles
The Colorado Growth Model Combining Actual and Aspirational Growth
New Directions Growth & Effectiveness
Model for Student Growth Percentiles
Student growth percentiles are calculated using quantile
regression with B-spline smoothing, within a generalized
additive model [Hastie and Tibshirani, 1990].
Given assessment scores for t occasions, (t ≥ 2), the τ-th
conditional quantile for Yt based upon Yt −1 , Yt −2 , . . . , Y1 is
given by
QYt (τ|Yt −1 , . . . , Y1 ) = φij (Yj )βij (τ)
where φi ,j , i = 1, 2, 3 and j = 1, . . . , t − 1 denote the B-spline
basis functions.
Quantile regression is used to model the complete distribution
of the response variable (current achievement), as opposed to
just the conditional mean.
B-spline smoothing is uses to accommodate non-linearity and
heteroscedasticity of the data.
The model/method is descriptive and all about data fit—it’s a
data mining procedure.
The model quantifies distance = rate · time probabilistically.
Damian W. Betebenner Student Growth in Missouri
The Big Picture Describing Student Growth
Guiding Principles and Questions Model & Software for Student Growth Percentiles
The Colorado Growth Model Combining Actual and Aspirational Growth
New Directions Growth & Effectiveness
Student Growth Percentile Model Properties
Does not require a vertical scale
No distributional assumptions associated with achievement.
Robust to ceiling/floor effects on the exam.
Invariant to mono-tone transformations of scale—does not
require interval scale.
Robust to outliers. For example, states with LOSS scores at 0
but a scale range that “really” begins at 300 to 400.
Uses full historical record of student achievement data based
upon a 1 grade/1 year progression.
Quantile crossing is adjusted for via the procedure of Dette
and Volgushev (2009). [Dette and Volgushev, 2009]
Damian W. Betebenner Student Growth in Missouri
The Big Picture Describing Student Growth
Guiding Principles and Questions Model & Software for Student Growth Percentiles
The Colorado Growth Model Combining Actual and Aspirational Growth
New Directions Growth & Effectiveness
Accuracy and Precision of SGP results
Discussing accuracy and precision of estimates/calculations
requires stipulation of a chance process.
There are several possible chance processes at work. For
example, measurement, students, equating,
Measurement error seems most relevant. Results in a “double
whammy” [Carroll et al., 1995, p. 22], impacting both accuracy
and precision.
Bias is negligible at individual level, particularly in situations
with 2 or more prior year scores.
Currently using simulation to express precision based upon
published CSEM statistics for state assessments at both
individual and group level.
Based upon multiple conversations with states, fear that
confidence intervals lead people to make “bad inferences with
confidence”.
Damian W. Betebenner Student Growth in Missouri
The Big Picture Describing Student Growth
Guiding Principles and Questions Model & Software for Student Growth Percentiles
The Colorado Growth Model Combining Actual and Aspirational Growth
New Directions Growth & Effectiveness
Software for Student Growth Percentiles
Student Growth Percentiles and Percentile Growth
Trajectories are calculated using the open source R Software
Environment [R Development Core Team, 2010] and the SGP
package [Betebenner, 2010]
In working with state departments of education, the goal is to
build state capacity so states can perform and validate their
own analyses using software available and easily used by the
state.
Colorado, Massachusetts, Indiana, and Wisconsin are four
states (that I know of) who have state staff capable of
generating the student growth percentiles together with
aggregations that follow from their calculation.
State ownership/oversight of the SGP calculations has been a
big plus in terms of buy-in from the state, enhancing their
ability to explain and work with local stakeholders.
Damian W. Betebenner Student Growth in Missouri
The Big Picture Describing Student Growth
Guiding Principles and Questions Model & Software for Student Growth Percentiles
The Colorado Growth Model Combining Actual and Aspirational Growth
New Directions Growth & Effectiveness
Establishing growth-standards based upon growth-norms
The most common adequacy criterion are judging growth
toward an achievement goal (i.e., growth-to-standard).
Results from student growth percentile analyses can be used
to calculate percentile growth trajectories for each student.
These trajectories indicate what future rates of growth will
lead to and are used to make adequacy judgments.
This growth-to-standard approach was approved as part of
Colorado’s successful application to the Growth Model Pilot
Program.
Damian W. Betebenner Student Growth in Missouri
Advanced Reading
High
Achievement
CSAP Reading
● Typical
Scale Score
Proficient Growth
Low
● Level Percentiles
High 66th − 99th
Typical 35th − 65th
Part Proficient
● Low 1st − 34th
Unsatisfactory
Grade 3 Grade 4 Grade 5 Grade 6 Next Year
Scale Score 462 539 563 609
Achievement Level Unsatisfactory Part Proficient Part Proficient Proficient Achievement
Growth Percentile 66 66 90
Growth Level High High High Growth
NCIEA
CDE
Colorado Department of Education
High
Math
Advanced ●
Typical
Achievement
● Low
CSAP Math
● Scale Score
● ● Growth
Level Percentiles
Proficient
High 66th − 99th
Typical 36th − 65th
Part Proficient
Low 1st − 35th
Unsatisfactory
Grade 4 Grade 5 Grade 6 Grade 7 Next Year
Scale Score 515 517 589 603
Achievement Level Proficient Proficient Advanced Proficient Achievement
Growth Percentile 15 92 76
Growth Level Low High High Growth
2008 CSAP Growth & Achievement Report
Advanced High
Reading
Typical Achievement
CSAP Reading
Low ● Scale Score
Proficient Growth
Level Percentiles
● High 66th − 99th
Typical 36th − 65th
Part Proficient
Low 1st − 35th
Unsatisfactory
Grade 4 Grade 5 Grade 6 Grade 7 Next Year
Scale Score 567 620 652 657
Achievement Level Part Proficient Proficient Proficient Proficient Achievement
Growth Percentile 72 89 72
Growth Level High High High Growth
Advanced
High Writing
Achievement
● Typical
CSAP Writing
● Scale Score
Low
● ● Growth
Proficient
Level Percentiles
Example Student 3020
High 66th − 99th
Part Proficient Typical 36th − 65th
Low 1st − 35th
Unsatisfactory
Grade 4 Grade 5 Grade 6 Grade 7 Next Year
Scale Score 478 517 519 574
Achievement Level Part Proficient Proficient Proficient Proficient Achievement
Growth Percentile 77 44 86
Growth Level High Typical High Growth
2008 CSAP Mathematics
Growth and Achievement Chart
2008 Achievement Percentile Percentile Growth Trajectory
Scale Score
Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10
The Big Picture Describing Student Growth
Guiding Principles and Questions Model & Software for Student Growth Percentiles
The Colorado Growth Model Combining Actual and Aspirational Growth
New Directions Growth & Effectiveness
From Students to Groups of Students
Aristotle
We understand best those things we see emerge from their very
beginnings.
It’s of interest to examine schools where students
demonstrate, on average, extraordinarily high and low student
growth.
To summarize the student growth percentiles associated with
a school (or other grouping) one calculates the median of the
student growth percentiles.
If students were randomly assigned to schools, one expects to
see a median of 50.
Values greatly above or below 50 are of interest in identifying
best practices or providing extra support.
Examining growth with achievement sheds new light on
school performance.
Damian W. Betebenner Student Growth in Missouri
District C: 2008 CSAP Math School Results
Student Growth versus Student Achievement by Percent Free/Reduced Lunch
School Percent
Higher Achievement Higher Achievement Free/Reduced Lunch
Lower Growth Higher Growth
Percent at/above Proficient in School
Less than 20 percent
● 20 to 40 percent
● 40 to 60 percent
● 60 to 80 percent
● More than 80 percent
School Size
50 Students
100 Students
200 Students
30 500 Students
1,000 Students
10 Lower Growth Higher Growth
Lower Achievement Lower Achievement
Median of Student Growth Percentiles in School
Boston−Charlestown School Performance
Growth and Achievement MCAS ELA by School Poverty
Higher Achievement X X Higher Achievement School Size
Lower Growth Higher Growth 50 Students
100 Students
2009 Percent at/above Proficient
250 Students
500 Students
X X X X 1000 Students
XX X
60 X XX X X
X XX
X X X XX X X
School Percent
X X X Free/Reduced Lunch
X XX
X ● Less than 20 percent
XX
● 20 to 40 percent
XX X X X
X X X X X ● 40 to 60 percent
X X XX
XX ● 60 to 80 percent
Achievement
X XX
XXX X ● More than 80 percent
Lower Growth X
XX X
Higher Growth
Lower Achievement Lower Achievement
Growth 2009 Median Student Growth Percentile
The Big Picture Describing Student Growth
Guiding Principles and Questions Model & Software for Student Growth Percentiles
The Colorado Growth Model Combining Actual and Aspirational Growth
New Directions Growth & Effectiveness
Growth, Effectiveness, and Value-Added
Fundamental Premise
“Good” schools (or teachers) bring about student growth in ex-
cess of that found at “bad” schools.
“Good schools” are often called highly effective schools.
What’s the relationship between growth and effectiveness?
Effectiveness indicates who/what is responsible for the growth
(value-added models).
Assuming the above premise is, at least, partly true. How do
we judge the correctness of a value-added model?
Damian W. Betebenner Student Growth in Missouri
2009 CSAP Elementary School Mathematics:
School Poverty by School Achievement (r = −0.71)
80 Higher Achievement
Percent at/above Proficient
Lower Achievement
Percent Free/Reduced Lunch Students at School
2009 CSAP Elementary School Mathematics:
School Poverty by School Median SGP (r = −0.22)
80 Higher Average Growth
Median Student Growth Percentile
20 Lower Average Growth
Percent Free/Reduced Lunch Students at School
2009 CSAP Elementary School Reading:
School Poverty by School Median SGP (r = −0.25)
80 Higher Average Growth
Median Student Growth Percentile
20 Lower Average Growth
Percent Free/Reduced Lunch Students at School
Economic Disadvantage vs School Effectiveness
TN Elementary Schools 2008
Average Yearly Achievement Gain 1
Math & Reading/Language Arts in Z units from
TN Current State Average
Percent Free & Reduced Lunch
Composite VA Gain by %
Poverty
School % Poverty
The Big Picture
Growth & Accountability
Guiding Principles and Questions
Web 2.0 Technologies
The Colorado Growth Model
References
New Directions
Next Steps for Growth Data
Confucius
Tell me and I will forget, show me and I may remember, involve
me and I will understand.
Calculating growth quantities is the first and easiest step in
the development and deployment of a growth model.
The end goal is to change conversations about education
through the examination and use of data.
To this end, Colorado has been inspired by the Web 2.0
user-centered design philosophies and technologies.
The goal is to produce data visualization and social
networking tools to turn data into information for all education
stakeholders and ultimately change conversations about
education.
Damian W. Betebenner Student Growth in Missouri
The Big Picture
Growth & Accountability
Guiding Principles and Questions
Web 2.0 Technologies
The Colorado Growth Model
References
New Directions
Descriptive Accountability
“Accountability system results can have value without making
causal inferences about school quality, solely from the results of
student achievement measures and demographic characteristics.
Treating the results as descriptive information and for identification
of schools that require more intensive investigation of
organizational and instructional process characteristics are
potentially of considerable value. Rather than using the results of
the accountability system as the sole determiner of sanctions for
schools, they could be used to flag schools that need more
intensive investigation to reach sound conclusions about needed
improvements or judgments about quality.”
R. L. Linn (2008)
Damian W. Betebenner Student Growth in Missouri
The Big Picture
Growth & Accountability
Guiding Principles and Questions
Web 2.0 Technologies
The Colorado Growth Model
References
New Directions
Web 2.0: Data Visualization and Social Networking
The Colorado Department of Education and the Center for
Assessment have been working for the last two year on
developing next generation data visualization to accompany
growth model data.
The goal: Transform conversations about education through
active engagement with data (i.e., evidence).
Our efforts have received tremendous interest and
recognition:
Received (just announced) the 2010 annual award for
Outstanding Dissemination of Educational Measurement
Concepts to the Public by the National Council on
Measurement in Education.
Recognized by Adobe for innovative uses of their technology
as an Adobe Max Award finalist in October, 2009.
Multiple states signing MOUs to co-develop a cloud-based
reporting platform in a non-proprietary fashion.
Colorado recently devoted $2.5 million of stimulus funds to the
development efforts
Damian W. Betebenner Student Growth in Missouri
The Big Picture
Growth & Accountability
Guiding Principles and Questions
Web 2.0 Technologies
The Colorado Growth Model
References
New Directions
Web 2.0: Data Visualization and Social Networking
With a collaborative spirit, with a collaborative platform where
people can upload data, explore data, compare solutions, discuss
the results, build consensus, we can engage passionate people,
local communities, media and this will raise–incredibly–the amount
of people who can understand what is going on.
And this would have fantastic outcomes: the engagement of
people, especially new generations; it would increase knowledge,
unlock statistics, improve transparency and accountability of public
policies, change culture, increase numeracy, and in the end,
improve democracy and welfare.
E. Giovannini, Chief Statistician, OECD. June 2007
Damian W. Betebenner Student Growth in Missouri
The Big Picture
Growth & Accountability
Guiding Principles and Questions
Web 2.0 Technologies
The Colorado Growth Model
References
New Directions
References
Betebenner, D. W. (2008).
Toward a normative understanding of student growth.
In Ryan, K. E. and Shepard, L. A., editors, The Future of
Test-Based Educational Accountability, pages 155–170. Taylor
& Francis, New York.
Betebenner, D. W. (2009).
Norm- and criterion-referenced student growth.
Educational Measurement: Issues and Practice, 28(4):42–51.
Betebenner, D. W. (2010).
SGP: Student Growth Percentile and Percentile Growth
Projection/Trajectory Functions.
R package version 0.0-5.
Carroll, R. J., Ruppert, D., and Stefanski, L. A. (1995).
Measurement Error in Nonlinear Models.
Chapman & Hall, London.
Dette, H. and Volgushev, S. (2009).
Non-crossing non-paramatric estimates of quantile curves.
Journal of the Royal Statistical Society B, 70(3):609–627.
Hastie, T. J. and Tibshirani, R. J. (1990).
Generalized Additive Models.
Chapman & Hall/CRC, New York, NY.
Linn, R. L. (2008).
Educational accountability systems.
In The Future of Test-Based Educational Accountability, pages
3–24. Taylor & Francis, New York.
R Development Core Team (2010).
R: A language and environment for statistical computing.
R Foundation for Statistical Computing, Vienna, Austria.
Yen, W. M. (2007).
Vertical scaling and No Child Left Behind.
In Dorans, N. J., Pommerich, M., and Holland, P. W., editors,
Linking and Aligning Scores and Scales, pages 273–283.
Springer, New York.
Damian W. Betebenner Student Growth in Missouri
About this copy
Exploring Student Growth in Missouri: Changing Conversations about Education — Missouri Department of Education, September 16, 2010.
This is an image-based copy of the original PDF slides; builds and animations are flattened.