Intro
Student Growth & Value-Added
Criterion Referenced Growth
Visualization and Analysis
References
Student Growth & Accountability
Student Growth Percentiles: One Model, Many Uses
Damian W. Betebenner
National Center for the Improvement of Educational Assessment
Dover, NH
December 7th, 2010
Damian W. Betebenner Student Growth & Accountability
Intro
Student Growth & Value-Added
Criterion Referenced Growth
Visualization and Analysis
References
Growth Model Purposes
Institutional Evaluation
Adequate Yearly Progress
Personnel Evaluation
Diagnostic Information
Advocacy (Point out disparities and what should be)
Damian W. Betebenner Student Growth & Accountability
Intro
Student Growth & Value-Added
Criterion Referenced Growth
Visualization and Analysis
References
Metrics
Better than average (and hopefully good enough)
No Child Left Behind on Average
No Child Left Behind
Damian W. Betebenner Student Growth & Accountability
Intro
Student Growth & Value-Added The Colorado Growth Model
Criterion Referenced Growth Value-Added Inferences
Visualization and Analysis Description, Inference, and Causality
References
The Colorado Growth Model
How much growth did a student make? Is it enough?
An important first step in the development of the Colorado
Growth Model was to begin discussions by separating the
description of growth from discussions of
responsibility/accountability.
Incorporating growth into accountability followed more easily
from the accepted description of growth.
The description of growth facilitated stakeholder engagement
and investigations of responsibility for good/bad growth.
That in turn led to greater stakeholder support for particular
forms of accountability.
Damian W. Betebenner Student Growth & Accountability
Intro
Student Growth & Value-Added The Colorado Growth Model
Criterion Referenced Growth Value-Added Inferences
Visualization and Analysis Description, Inference, and Causality
References
Value-added Analyses/Inferences
Who/What is responsible for student growth?
Often, analyses of student growth attempt to determine the amount
of student progress/growth that can be attributed to the school or
teacher.
Called Value-added analyses, these techniques attempt to estimate
the teacher/school contribution to student achievement.
Value-added models quantify effectiveness—purporting to measure
the impact on achievement that this school or teacher would have,
on average, upon similar students assigned to them.
These models and their effectiveness claims are controversial for a
number of reasons (EPI (2010), NRC/NAE (2010), Braun (2005),
Rubin and Stuart (2004)) .
Damian W. Betebenner Student Growth & Accountability
Intro
Student Growth & Value-Added The Colorado Growth Model
Criterion Referenced Growth Value-Added Inferences
Visualization and Analysis Description, Inference, and Causality
References
Descriptive Questions
Note that the questions from Yen’s (2007) survey are primarily
descriptive.
The questions are only peripherally associated with causality.
High stakes accountability has transformed questions about student
growth into questions about responsibility/cause: Teacher and
School Effectiveness.
Beginning with description doesn’t prevent subsequent discussions
about responsibility/cause.
Beginning with responsibility/cause often prevents discussions about
description.
Damian W. Betebenner Student Growth & Accountability
Intro
Student Growth & Value-Added The Colorado Growth Model
Criterion Referenced Growth Value-Added Inferences
Visualization and Analysis Description, Inference, and Causality
References
Descriptive Accountability
Accountability system results can have value without making
causal inferences about school quality, solely from the results of
student achievement measures and demographic
characteristics. Treating the results as descriptive information
and for identification of schools that require more intensive
investigation of organizational and instructional process
characteristics are potentially of considerable value. Rather
than using the results of the accountability system as the sole
determiner of sanctions for schools, they could be used to flag
schools that need more intensive investigation to reach sound
conclusions about needed improvements or judgments about
quality [Linn, 2008, p. 21].
Damian W. Betebenner Student Growth & Accountability
District C: 2008 CSAP Math School Results
Student Growth versus Student Achievement by Percent Free/Reduced Lunch
School Percent
Higher Achievement Higher Achievement Free/Reduced Lunch
Lower Growth Higher Growth
Percent at/above Proficient in School
Less than 20 percent
● 20 to 40 percent
● 40 to 60 percent
● 60 to 80 percent
● More than 80 percent
School Size
50 Students
100 Students
200 Students
30 500 Students
1,000 Students
10 Lower Growth Higher Growth
Lower Achievement Lower Achievement
Median of Student Growth Percentiles in School
Intro
Student Growth & Value-Added
Criterion Referenced Growth
Visualization and Analysis
References
Establishing growth-standards based upon growth-norms
The most common adequacy criterion are judging growth toward an
achievement goal (i.e., growth-to-standard).
Results from student growth percentile analyses can be used to
calculate percentile growth trajectories for each student.
These trajectories indicate what future rates of growth will lead to
and are used to make adequacy judgments.
This growth-to-standard approach was approved as part of
Colorado’s successful application to the Growth Model Pilot
Program.
Damian W. Betebenner Student Growth & Accountability
Advanced Reading
High
Achievement
CSAP Reading
● Typical
Scale Score
Proficient Growth
Low
● Level Percentiles
High 66th − 99th
Typical 35th − 65th
Part Proficient
● Low 1st − 34th
Unsatisfactory
Grade 3 Grade 4 Grade 5 Grade 6 Next Year
Scale Score 462 539 563 609
Achievement Level Unsatisfactory Part Proficient Part Proficient Proficient Achievement
Growth Percentile 66 66 90
Growth Level High High High Growth
2008 CSAP Mathematics
Growth and Achievement Chart
2008 Achievement Percentile Percentile Growth Trajectory
Scale Score
Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10
Intro
Student Growth & Value-Added
Criterion Referenced Growth Concluding Remarks
Visualization and Analysis
References
Software for Student Growth Percentiles
Student Growth Percentiles and Percentile Growth Trajectories are
calculated using the open source R Software
Environment [R Development Core Team, 2010] and the SGP
package [Betebenner, 2010]
In working with state departments of education, the goal is to build
state capacity so states can perform and validate their own analyses
using software available and easily used by the state.
Colorado, Massachusetts, Indiana, and Wisconsin are four states
(that I know of) who have state staff capable of generating the
student growth percentiles together with aggregations that follow
from their calculation.
State ownership/oversight of the SGP calculations has been a big
plus in terms of buy-in from the state, enhancing their ability to
explain and work with local stakeholders.
Damian W. Betebenner Student Growth & Accountability
Intro
Student Growth & Value-Added
Criterion Referenced Growth Concluding Remarks
Visualization and Analysis
References
What’s Next?
Colorado, Massachusetts, and Indiana have publicly released
growth results.
The response has been overwhelmingly positive.
Calculating of the quantities is the easy part. Getting people to
understand them and use them correctly is the hard part.
Colorado, Massachusetts, Indiana, Virginia, and other states are
investigating a multi-state platform for data visualization and
reporting built from a common growth analysis.
Damian W. Betebenner Student Growth & Accountability
The Colorado Growth Model
Adopted Model
Near Model Adoption
Interested in Model
Intro
Student Growth & Value-Added
Criterion Referenced Growth
Visualization and Analysis
References
References
Betebenner, D. W. (2010).
SGP: Student Growth Percentile and Percentile Growth
Projection/Trajectory Functions.
R package version 0.0-6.
Braun, H. I. (2005).
Using student progress to evaluate teachers: A primer on
value-added models.
Technical report, Educational Testing Service, Princeton, New
Jersey.
Economic Policy Institute (2010).
Problems with the use of student test scores to evaluate teachers.
Technical report, Washington, D.C.
Downloaded August 30th, 2010 from
http://epi.3cdn.net/724cd9a1eb91c40ff0_hwm6iij90.pdf.
Linn, R. L. (2008).
Educational accountability systems.
In The Future of Test-Based Educational Accountability, pages 3–24.
Taylor & Francis, New York.
National Research Council and National Academy of Education
Getting value of out value-added: Report of a workshop.
Technical report, Washington, D.C.
Commitee on Value-Added Methodology for Instructional
Improvement.
R Development Core Team (2010).
R: A language and environment for statistical computing.
R Foundation for Statistical Computing, Vienna, Austria.
Rubin, D. B., Stuart, E. A., and Zanutto, E. L. (2004).
A potential outcomes view of value-added assessment in education.
Journal of Educational and Behavioral Statistics, 29(1):103–116.
Damian W. Betebenner Student Growth & Accountability
About this copy
Student Growth & Accountability — Student Growth Percentiles: One Model, Many Uses — New York Department of Education, December 7, 2010.
This is an image-based copy of the original beamer slides; builds and animations are flattened.