References
On Student Growth Percentiles & Accountability
in Hawaii
Damian W. Betebenner
National Center for the Improvement of Educational Assessment
Dover, NH
Educator Evaluation Advisory Group, Honolulu, Hawaii
February 12, 2014
Damian W. Betebenner On Student Growth Percentiles & Accountability in Hawaii
References
About Me
Dr. Damian Betebenner, PhD
- Senior Associate at the Center for Assessment (NCIEA).
- Developed student growth percentiles and percentile growth
trajectories to help states and educational associations employ
student growth in decision making [Betebenner, 2008,
Betebenner, 2009, Betebenner, 2012, Betebenner et al., 2014].
- Currently refining and sharing these techniques with other
states including Colorado, Massachusetts, Indiana, Nevada,
Hawaii and 25 other states in various stages of
investigation/adoption.
Damian W. Betebenner On Student Growth Percentiles & Accountability in Hawaii
References
Student Growth Percentiles
2007 Began develop of the Colorado Growth Model (student
growth percentiles).
2008 Began work with Indiana on development of growth model.
2008 Successfully application of Colorado Growth Model
(criterion-referenced component) as part of growth model
pilot program.
2009-2010 Began/continued work with approximately two-dozen other
states on developing state growth models.
2010-2014 Worked with states on incorporating growth into RttT
applications and with successful states on their use
(Hawaii, Massachusetts, Georgia, Rhode Island).
2011-2014 Worked with a number of states on incorporating student
growth percentiles into the ESEA waiver applications and
teacher evaluation systems.
2014- Working with states transitioning to PARCC, SBAC
assessment systems.
Damian W. Betebenner On Student Growth Percentiles & Accountability in Hawaii
Student Growth Percentile Adoption
Adopted Model
Considering Adoption
Investigating Model
References
Lessons Learned
On Questions John Tukey
It is better to have an approximate answer to the right question than a precise answer
to the wrong question.
On Statistical Models George E. P. Box
All models are wrong but some are useful.
On Understanding Aristotle
We understand best those things we see emerge from their very beginnings.
On Accountability Sherlock Holmes
It is a capital mistake to theorize before you have all the evidence. It biases the judg-
ment.
On Theories of Action Confucius
Tell me and I will forget, show me and I will remember, involve me and I will understand.
Damian W. Betebenner On Student Growth Percentiles & Accountability in Hawaii
References
About the growth model
The Student Growth Percentiles Methodology is a norm- and
criterion-referenced growth analysis method.
The norm-referenced component of the model is the student growth
percentile which indicates the relative growth a student makes.
The criterion-referenced component of the model is the student
growth percentile target which indicates the amount of growth
required by the student to reach/maintain desirable achievement
goals in a specified time frame. For example, catching-up to
proficiency within 3 years.
The norm- and criterion-referenced components together allow
stakeholders to understand what is, what should be, and what is
reasonable simultaneously.
The goal is to create a durable metric to assist in moving beyond
debates over what thermometer is best to discussions of how best to
use the thermometer.
Damian W. Betebenner On Student Growth Percentiles & Accountability in Hawaii
References
Built to maximize sharing
A number of analytic tools (for doing calculations) and visualization tools
have been developed with the goal to promote maximum sharing. Data
isn’t valuable! It’s what one does with data that is potentially valuable!
The analytic and visualization tools to calculate all the quantities
associated with student growth percentiles are open source and
freely available to all states.
Conduct training sessions at national conferences to help other
calculate these quantities on their own.
Build enhanced visualization and reporting systems to make data
come alive to stakeholders and lead to better stakeholder
engagement with data.
Goal is to re-invent the user experience with education data and
enable a “growth mindset”.
Damian W. Betebenner On Student Growth Percentiles & Accountability in Hawaii
References
Describing Student Growth
Measuring student growth, even with a vertical scale, is not a simple
task.
Some believe a vertical scale simplifies the task of measuring
student growth.
Even with an interval (or ratio) scale, growth is not easy to interpret.
Consider, for example, height.
A child might grow 4 inches between ages 3 and 4.
4 inches is a well understood quantity.
The 4 inch increase becomes really meaningful only when understood
alongside the growth of other 3 to 4 year olds.
Student growth percentiles were developed to provide a normative
context for describing student growth.
Damian W. Betebenner On Student Growth Percentiles & Accountability in Hawaii
Advanced
Reading
Achievement
Proficient ●
DEMO Reading
Scale Score
● Catch Up
Target Growth
Part Proficient Level Percentiles
Very High 81st − 99th
High 61st − 80th
Typical 40th − 60th
Unsatisfactory
Low 20th − 39th
Very Low 1st − 19th
Grade 3 Grade 4
Scale Score 449
Achievement Level
Achievement Target
Unsatisfactory
Proficient (529)
Achievement
Growth Percentile
Growth Level
Growth Target Catch Up (64)
Growth
Advanced Reading
Achievement
DEMO Reading
● Scale Score
Proficient
● Catch Up
Growth
Target
Level Percentiles
Very High 81st − 99th
Part Proficient High 61st − 80th
● Typical 40th − 60th
Low 20th − 39th
Unsatisfactory
Very Low 1st − 19th
Grade 3 Grade 4 Grade 5
Scale Score 449 533
Achievement Level
Achievement Target
Unsatisfactory Part Proficient
Proficient (562)
Achievement
Growth Percentile 69
Growth Level
Growth Target
High
Catch Up (76)
Growth
Advanced
Reading
Achievement
Proficient ● ●
● Catch Up
Target
DEMO Reading
Scale Score
Growth
Part Proficient Level Percentiles
Very High 81st − 99th
High 61st − 80th
Typical 40th − 60th
Unsatisfactory
Low 20th − 39th
Very Low 1st − 19th
Grade 3 Grade 4 Grade 5 Grade 6
Scale Score 449 533 579
Achievement Level
Achievement Target
Unsatisfactory Part Proficient Part Proficient
Proficient (586)
Achievement
Growth Percentile 69 85
Growth Level
Growth Target
High Very High
Catch Up (71)
Growth
Advanced Reading
● Move Up
Achievement
Target
DEMO Reading
● Scale Score
Keep Up
● Target
Proficient Growth
Level Percentiles
● Very High 81st − 99th
High 61st − 80th
Part Proficient Typical 40th − 60th
Low 20th − 39th
Very Low 1st − 19th
Unsatisfactory
Grade 3 Grade 4 Grade 5 Grade 6 Grade 7
Scale Score 449 533 579 602
Achievement Level
Achievement Target
Unsatisfactory Part Proficient Part Proficient Proficient
Proficient (620)/Advanced (661)
Achievement
Growth Percentile 69 85 74
Growth Level
Growth Target
High Very High High
Keep Up (76)/Move Up (99)
Growth
References
Going from Students to Classrooms/Schools
It’s of interest to examine schools/classrooms where students
demonstrate, on average, extraordinarily high and low student
growth.
To summarize the student growth percentiles associated with a
school/classroom (or other grouping) Colorado calculates the
median of the student growth percentiles.
If students were randomly assigned to classrooms/schools, expect
to see a median of 50.
Values greatly above or below 50 are of interest in identifying best
practices or providing extra support.
Examining growth with achievement sheds new light on
school/classroom performance.
Damian W. Betebenner On Student Growth Percentiles & Accountability in Hawaii
Hawaii School Performance
HSA Growth & Achievement 2013 HSA Mathematics
Higher Achievement Higher Achievement School Size
Lower Growth Higher Growth 50 Students
100 Students
2013 Percent at/above Proficient
250 Students
500 Students
Achievement
Lower Growth Higher Growth
Lower Achievement Lower Achievement
Growth 2013 Median Student Growth Percentile
Hawaii School Performance
HSA Growth & Prior Achievement 2013 HSA Mathematics
Higher Achievement Higher Achievement School Size
2012 Prior Percent at/above Proficient
Lower Growth Higher Growth 50 Students
100 Students
250 Students
500 Students
Achievement
Lower Growth Higher Growth
Lower Achievement Lower Achievement
Growth 2013 Median Student Growth Percentile
Hawaii School Performance
HSA Growth & Achievement 2013 HSA Reading
Higher Achievement Higher Achievement School Size
Lower Growth Higher Growth 50 Students
100 Students
2013 Percent at/above Proficient
250 Students
500 Students
Achievement
Lower Growth Higher Growth
Lower Achievement Lower Achievement
Growth 2013 Median Student Growth Percentile
Hawaii School Performance
HSA Growth & Prior Achievement 2013 HSA Reading
Higher Achievement Higher Achievement School Size
2012 Prior Percent at/above Proficient
Lower Growth Higher Growth 50 Students
100 Students
250 Students
500 Students
Achievement
Lower Growth Higher Growth
Lower Achievement Lower Achievement
Growth 2013 Median Student Growth Percentile
Hawaii Teacher Median SGPs
Frequency 400
Median SGP
Hawaii: 2013 Mathematics
Norm & Criterion Referenced Growth & Achievement
No Grade 9 Assessment
2013 Achievement Percentile
Percentile Growth Trajectory
Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 10
Hawaii: 2013 Reading
Norm & Criterion Referenced Growth & Achievement
No Grade 9 Assessment
2013 Achievement Percentile
Percentile Growth Trajectory
Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 10
References
Web 2.0: Data Visualization and Social Networking
The Hawaii Department of Education and the Center for
Assessment have been working for the last two years on developing
next generation data visualization to accompany growth model data.
The goal: Transform conversations about education through active
engagement with data (i.e., evidence).
Our efforts have received tremendous interest and recognition:
Recognized by Adobe for innovative uses of their technology as an
Adobe Max Award finalist in October, 2009.
2010 NCME Award for Outstanding Dissemination of Educational
Measurement Concepts to the Public.
Damian W. Betebenner On Student Growth Percentiles & Accountability in Hawaii
References
References
Betebenner, D. W. (2008).
Toward a normative understanding of student growth.
In Ryan, K. E. and Shepard, L. A., editors, The Future of Test-Based
Educational Accountability, pages 155–170. Taylor & Francis, New
York.
Betebenner, D. W. (2009).
Norm- and criterion-referenced student growth.
Educational Measurement: Issues and Practice, 28(4):42–51.
Betebenner, D. W. (2012).
Growth, standards, and accountability.
In Cizek, G., editor, Setting Performance Standards: Foundations,
Methods, and Innovations, pages 439–450. Routledge, New York.
Betebenner, D. W., Van Iwaarden, A., Domingue, B., and Shang, Y.
SGP: An R Package for the Calculation and Visualization of Student
Growth Percentiles & Percentile Growth Trajectories.
R package version 1.2-2.0.
Damian W. Betebenner On Student Growth Percentiles & Accountability in Hawaii
About this copy
On Student Growth Percentiles & Accountability in Hawaii — Hawaii Department of Education, February 12, 2014.
This is an image-based copy of the original beamer slides; builds and animations are flattened.