Intro
Describing Student Growth
Student Growth Percentiles and Accountability
References
The Idaho Growth Model
Changing Conversations about Education
Damian W. Betebenner
National Center for the Improvement of Educational Assessment
Dover, NH
2011 Annual Superintendents’ Meeting
Meridian, Idaho
August 2nd, 2011
Damian W. Betebenner The Idaho Growth Model
Intro
Describing Student Growth
Student Growth Percentiles and Accountability
References
About Me
Dr. Damian Betebenner, PhD
- Senior Associate at the Center for Assessment (NCIEA).
- Developed student growth percentiles and percentile growth
trajectories to help states and educational associations employ
student growth in decision
making [Betebenner, 2008, Betebenner, 2009].
- Currently refining and sharing these techniques with other
states including Colorado, Massachusetts, Indiana, Virginia and
15 other states in various stages of investigation/adoption.
Damian W. Betebenner The Idaho Growth Model
Intro
Describing Student Growth
Student Growth Percentiles and Accountability
References
About Today’s Talk
On Questions John Tukey
It is better to have an approximate answer to the right question than a precise answer to the
wrong question.
On Statistical Models George E. P. Box
All models are wrong but some are useful.
On Understanding Aristotle
We understand best those things we see emerge from their very beginnings.
On Accountability Sherlock Holmes
It is a capital mistake to theorize before you have all the evidence. It biases the judgment.
On Theories of Action Confucius
Tell me and I will forget, show me and I will remember, involve me and I will understand.
Damian W. Betebenner The Idaho Growth Model
Intro
Student Growth Percentiles
Describing Student Growth
Model & Software for Student Growth Percentiles
Student Growth Percentiles and Accountability
Criterion Referenced Growth
References
Describing Student Growth
Measuring student growth, even with a vertical scale, is not a simple
task.
Some believe a vertical scale simplifies the task of measuring
student growth.
Even with an interval (or ratio) scale, growth is not easy to interpret.
Consider, for example, height.
A child might grow 4 inches between ages 3 and 4.
4 inches is a well understood quantity.
The 4 inch increase becomes really meaningful only when understood
alongside the growth of other 3 to 4 year olds.
Student growth percentiles were developed to provide a normative
context for describing student growth.
Damian W. Betebenner The Idaho Growth Model
600 co
09 600 re
le
ca le ca
S re 10
co
600 co
09 600 re
le
ca le ca
S re 10
co
600 co
09 600 re
le
ca le ca
S re 10
co
co
th
re
ro
i le
le
ce
nt ca
er
th
re
600 co
le
ca
Intro
Student Growth Percentiles
Describing Student Growth
Model & Software for Student Growth Percentiles
Student Growth Percentiles and Accountability
Criterion Referenced Growth
References
Student Growth Percentiles
Should we be surprised with a child’s current achievement given their prior
achievement?
Given a student’s prior scale scores and the associated conditional
density, their current scale score corresponds to a percentile of that
conditional distribution.
This percentile is the student’s growth percentile.
Growth percentiles are closely related to estimating the probability of
observing a student’s current achievement taking account of their
past achievement:
Pr(Current Achievement |Past Achievement ).
As such, growth percentiles describe the rarity of a student’s current
achievement conditional upon their prior achievement.
Damian W. Betebenner The Idaho Growth Model
Intro
Student Growth Percentiles
Describing Student Growth
Model & Software for Student Growth Percentiles
Student Growth Percentiles and Accountability
Criterion Referenced Growth
References
Student Growth Percentiles
Should we be surprised with a child’s current achievement given their
prior achievement?
Student growth percentiles answer this question.
Consider a low achieving student with 90th percentile growth and a
high achieving student with 10th percentile growth.
The low achieving student grew at a rate exceeding 90 percent of
similar students.
The high achieving student grew at a rate exceeding just 10 percent of
similar students.
The low achiever’s growth is more exemplary (probabilistically) than
the high achiever’s.
Judgments about the adequacy of student growth require external
criteria.
Damian W. Betebenner The Idaho Growth Model
Intro
Student Growth Percentiles
Describing Student Growth
Model & Software for Student Growth Percentiles
Student Growth Percentiles and Accountability
Criterion Referenced Growth
References
Software for Student Growth Percentiles
Student Growth Percentiles and Percentile Growth Trajectories are
calculated using the open source R Software
Environment [R Development Core Team, 2011] and the SGP
package [Betebenner and VanIwaarden, 2011]
In working with state departments of education, the goal is to build
state capacity so states can perform and validate their own analyses
using software available and easily used by the state.
Colorado, Massachusetts, Indiana, and Wisconsin are four states
(that I know of) who have state staff capable of generating the
student growth percentiles together with aggregations that follow
from their calculation.
State ownership/oversight of the SGP calculations has been a big
plus in terms of buy-in from the state, enhancing their ability to
explain and work with local stakeholders.
Damian W. Betebenner The Idaho Growth Model
Intro
Student Growth Percentiles
Describing Student Growth
Model & Software for Student Growth Percentiles
Student Growth Percentiles and Accountability
Criterion Referenced Growth
References
Establishing growth-standards based upon growth-norms
The most common adequacy criterion are judging growth toward an
achievement goal (i.e., growth-to-standard).
Results from student growth percentile analyses can be used to
calculate percentile growth trajectories for each student.
These trajectories indicate what future rates of growth will lead to
and are used to make adequacy judgments.
This growth-to-standard approach was approved as part of
Colorado’s successful application to the Growth Model Pilot
Program.
Damian W. Betebenner The Idaho Growth Model
Advanced Reading
High
Achievement
CSAP Reading
● Typical
Scale Score
Proficient Growth
Low
● Level Percentiles
High 66th − 99th
Typical 35th − 65th
Part Proficient
● Low 1st − 34th
Unsatisfactory
Grade 3 Grade 4 Grade 5 Grade 6 Next Year
Scale Score 462 539 563 609
Achievement Level Unsatisfactory Part Proficient Part Proficient Proficient Achievement
Growth Percentile 66 66 90
Growth Level High High High Growth
Example Student 12 (1234567812) Sample Middle School
Advanced
Math
Achievement
High
CSAP Math
● Scale Score
Typical
● Growth
Proficient
● Level Percentiles
Low
High 66th − 99th
● Typical 35th − 65th
Part Proficient
Low 1st − 34th
Unsatisfactory
Grade 3 Grade 4 Grade 5 Grade 6 Next Year
Scale Score 379 388 444 462
Achievement Level Part Proficient Part Proficient Part Proficient Part Proficient Achievement
Growth Percentile 16 69 51
Growth Level Low High Typical Growth
Advanced Reading
High
Achievement
CSAP Reading
● Typical
Scale Score
Proficient Growth
Low
● Level Percentiles
High 66th − 99th
Typical 35th − 65th
Part Proficient
● Low 1st − 34th
Unsatisfactory
Grade 3 Grade 4 Grade 5 Grade 6 Next Year
Scale Score 462 539 563 609
Achievement Level Unsatisfactory Part Proficient Part Proficient Proficient Achievement
Growth Percentile 66 66 90
Growth Level High High High Growth
Advanced
Writing
Achievement
High
CSAP Writing
● Scale Score
Typical
● Growth
Proficient
Level Percentiles
● Low
High 66th − 99th
Part Proficient ● Typical 35th − 65th
Low 1st − 34th
Unsatisfactory
Grade 3 Grade 4 Grade 5 Grade 6 Next Year
Scale Score 425 449 469 504
Achievement Level Part Proficient Part Proficient Part Proficient Part Proficient Achievement
Growth Percentile 55 53 79
Growth Level Typical Typical High Growth
Copyright 2009, Colorado Department of Education/Center for Assessment, Inc.
2008 CSAP Mathematics
Growth and Achievement Chart
2008 Achievement Percentile Percentile Growth Trajectory
Scale Score
Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10
Intro
School Accountability
Describing Student Growth
Growth & Effectiveness
Student Growth Percentiles and Accountability
Concluding Remarks
References
Going from Students to Schools/Classrooms
It’s of interest to examine schools or classrooms/teachers where
students demonstrate, on average, extraordinarily high and low
student growth.
To summarize the student growth percentiles associated with a
school (or other grouping) calculate the median of the student
growth percentiles.
If students were randomly assigned to schools, expect to see a
median of 50.
Values greatly above or below 50 are of interest in identifying best
practices or providing extra support.
Examining growth with achievement sheds new light on school
performance.
Damian W. Betebenner The Idaho Growth Model
District C: 2008 CSAP Math School Results
Student Growth versus Student Achievement by Percent Free/Reduced Lunch
School Percent
Higher Achievement Higher Achievement Free/Reduced Lunch
Lower Growth Higher Growth
Percent at/above Proficient in School
Less than 20 percent
● 20 to 40 percent
● 40 to 60 percent
● 60 to 80 percent
● More than 80 percent
School Size
50 Students
100 Students
200 Students
30 500 Students
1,000 Students
10 Lower Growth Higher Growth
Lower Achievement Lower Achievement
Median of Student Growth Percentiles in School
Intro
School Accountability
Describing Student Growth
Growth & Effectiveness
Student Growth Percentiles and Accountability
Concluding Remarks
References
Growth, Effectiveness, and Value-Added
“Good schools” are often called highly effective schools.
What’s the relationship between growth and effectiveness?
Effectiveness indicates who/what is responsible for the growth
(value-added models).
RttT has placed teacher responsibility for student progress as a
primary issue.
In teacher performance evaluations, even when large scale
assessment results are available, depending upon the state they
constitute, on average, about a third of the overall teacher rating.
Damian W. Betebenner The Idaho Growth Model
Intro
School Accountability
Describing Student Growth
Growth & Effectiveness
Student Growth Percentiles and Accountability
Concluding Remarks
References
What’s Next?
Colorado, Massachusetts, Indiana and Nevada have publicly
released growth results.
The response has been overwhelmingly positive.
Calculating of the quantities is the easy part. Getting people to
understand them and use them correctly is the hard part.
Colorado, New Hampshire, Nevada, and several other states are
investigating a multi-state platform for data visualization and
reporting built from a common growth analysis.
Damian W. Betebenner The Idaho Growth Model
The Colorado Growth Model
Adopted Model
Near Model Adoption
Interested in Model
Intro
Describing Student Growth
Student Growth Percentiles and Accountability
References
References
Betebenner, D. W. (2008).
Toward a normative understanding of student growth.
In Ryan, K. E. and Shepard, L. A., editors, The Future of Test-Based
Educational Accountability, pages 155–170. Taylor & Francis, New
York.
Betebenner, D. W. (2009).
Norm- and criterion-referenced student growth.
Educational Measurement: Issues and Practice, 28(4):42–51.
Betebenner, D. W. and VanIwaarden, A. (2011).
SGP: An R Package for the Calculation and Visualization of Student
Growth Percentiles.
R package version 0.3-0.0.
R Development Core Team (2011).
R: A language and environment for statistical computing.
R Foundation for Statistical Computing, Vienna, Austria.
Damian W. Betebenner The Idaho Growth Model
About this copy
The Idaho Growth Model: Changing Conversations about Education — Idaho Department of Education, August 2, 2011.
This is an image-based copy of the original PDF slides; builds and animations are flattened.