Intro
Growth & Accountability: Part I
Growth & Accountability: Part II
References
An Overview of Growth & Value-Added Models
Damian W. Betebenner
National Center for the Improvement of Educational Assessment
Dover, NH
Georgia RTTT Accountability Symposium
Athens, Georgia
January 24th, 2011
Damian W. Betebenner Growth & Value-Added Models
Intro
Growth & Accountability: Part I
Growth & Accountability: Part II
References
About Me
Dr. Damian Betebenner, PhD
- Senior Associate at the Center for Assessment (NCIEA).
- Developed student growth percentiles and percentile growth
trajectories to help states and educational associations employ
student growth in decision
making [Betebenner, 2008, Betebenner, 2009].
- Currently refining and sharing these techniques with other
states including Colorado, Massachusetts, Indiana, Virginia and
15 other states in various stages of investigation/adoption.
Damian W. Betebenner Growth & Value-Added Models
Intro
Why Growth?
Growth & Accountability: Part I
Different Models/Different Questions
Growth & Accountability: Part II
Relevant Questions/Concepts
References
The Attractiveness of Growth
What is growth and why measure it?
Student learning is a central goal of education.
Assessments of student achievement provide evidence of the
current status of student knowledge and understanding.
Learning is demonstrated by growth in student achievement
from one point in time to another point in time–not by status at
either point time alone.
Growth versus Status
Enthusiasm for growth in accountability stems from the belief
that growth and teacher/school quality are more closely related
than status.
Enthusiasm for growth also stems from its potential diagnostic
uses.
Damian W. Betebenner Growth & Value-Added Models
Intro
Why Growth?
Growth & Accountability: Part I
Different Models/Different Questions
Growth & Accountability: Part II
Relevant Questions/Concepts
References
Guiding Questions
Growth/Value-Added models address specific questions
Different growth analysis techniques often address different but
related questions.
It is critical to understand these questions.
Different questions lead to different conversations which lead to
different uses and outcomes.
Changing conversations about education
Starting with the right questions simplifies development.
The questions set the table for conversations: What questions
are going to be answered?
Conversations built around these questions provide implicit use
cases for the growth model results.
Yen (2007) provides and excellent list of questions
Damian W. Betebenner Growth & Value-Added Models
Intro
Why Growth?
Growth & Accountability: Part I
Different Models/Different Questions
Growth & Accountability: Part II
Relevant Questions/Concepts
References
What are the relevant questions for parents?
Yen (2007), from a state survey of parents, teachers and administrators,
compiled a list of frequently voiced questions/concerns by stakeholder
group.
Parent Questions
Did my child make a year’s worth of progress in a year?
Is my child growing appropriately toward meeting state
standards?
Is my child growing as much in Math as Reading?
Did my child grow as much this year as last year?
Damian W. Betebenner Growth & Value-Added Models
Intro
Why Growth?
Growth & Accountability: Part I
Different Models/Different Questions
Growth & Accountability: Part II
Relevant Questions/Concepts
References
What are the relevant questions for teachers?
Yen (2007), from a state survey of parents, teachers and administrators,
compiled a list of frequently voiced questions/concerns by stakeholder
group.
Teacher Questions
Did my students make a year’s worth of progress in a year?
Did my students grow appropriately toward meeting state
standards?
How close are my students to becoming proficient?
Are there students with unusually low growth who need special
attention?
Damian W. Betebenner Growth & Value-Added Models
Intro
Why Growth?
Growth & Accountability: Part I
Different Models/Different Questions
Growth & Accountability: Part II
Relevant Questions/Concepts
References
What are the relevant questions for administrators?
Yen (2007), from a state survey of parents, teachers and administrators,
compiled a list of frequently voiced questions/concerns by stakeholder
group.
Administrator Questions
Did the students in our district/school make a year’s worth of
progress in all content areas?
Are our students growing appropriately toward meeting state
standards?
Does this school/program show as much growth as that one?
Can I measure student growth even for students who do not
change proficiency categories?
Can I pool together results from different grades to draw
summary conclusions?
Damian W. Betebenner Growth & Value-Added Models
Intro Value-Added
Growth & Accountability: Part I Description, Inference, and Causality
Growth & Accountability: Part II Criterion Referenced Growth
References Growth & Effectiveness
Growth & Accountability
Fundamental Premise
“Good” schools/teachers bring about growth in student achievement
in excess of that found at “bad” schools/teachers.
The inherent believability of this premise is at the heart of current
enthusiasm to incorporate growth models into accountability
systems.
Note that current NCLB accountability invokes the same premise
(with achievement substituted for growth).
“Good schools/teachers” are commonly referred to as highly
effective schools/teachers.
It is critical to understand the distinction between growth and
effectiveness.
Damian W. Betebenner Growth & Value-Added Models
Intro Value-Added
Growth & Accountability: Part I Description, Inference, and Causality
Growth & Accountability: Part II Criterion Referenced Growth
References Growth & Effectiveness
Value-added Analyses/Inferences
Who/What is responsible for student growth?
Often, analyses of student growth attempt to determine the amount
of student progress/growth that can be attributed to the school or
teacher.
Called Value-added analyses, these techniques attempt to estimate
the teacher/school contribution to student achievement.
A value-added analysis quantifies effectiveness—purporting to
measure the impact on achievement that this school or teacher or
program would have, on average, upon similar students assigned to
them.
Damian W. Betebenner Growth & Value-Added Models
Intro Value-Added
Growth & Accountability: Part I Description, Inference, and Causality
Growth & Accountability: Part II Criterion Referenced Growth
References Growth & Effectiveness
Value-added Analyses/Inferences
Who/What is responsible for student growth?
What is a “value-added assessment” or a “value-added model”.
Value-added is an inference – a causal conclusion drawn from the
data.
Value-added isn’t a type of statistical model or type of assessment (if
I showed you the statistical model formulation or an assessment,
from the model formulation or assessment you wouldn’t be able to
tell me whether it’s value-added).
All growth models can be used for value-added purposes but not
vice-versa.
The question is whether the inference/conclusion is sound. This is
where the detective work begins.
Damian W. Betebenner Growth & Value-Added Models
Intro Value-Added
Growth & Accountability: Part I Description, Inference, and Causality
Growth & Accountability: Part II Criterion Referenced Growth
References Growth & Effectiveness
Descriptive Questions
Note that the questions from Yen’s (2007) survey are primarily
descriptive and only peripherally causal.
Accountability initiatives like RttT have transformed questions about
student growth into questions about responsibility/cause:
Teacher/Classroom and Principal/School Effectiveness.
Beginning with description doesn’t prevent subsequent discussions
about responsibility/cause.
Damian W. Betebenner Growth & Value-Added Models
Intro Value-Added
Growth & Accountability: Part I Description, Inference, and Causality
Growth & Accountability: Part II Criterion Referenced Growth
References Growth & Effectiveness
How much growth/effectiveness is “enough”
It is important to recognize the difference between norm and
criterion referenced models.
Value-added models make judgments of relative effectiveness.
Many growth models (especially those approved as part of the
growth model pilot program) make criterion referenced judgments of
effectiveness.
Growth toward a standard (e.g., proficiency or career and college
readiness) is a criterion referenced concept.
Growth above statistical expectation is a norm-referenced concept
Damian W. Betebenner Growth & Value-Added Models
District C: 2008 CSAP Math School Results
Student Growth versus Student Achievement by Percent Free/Reduced Lunch
School Percent
Higher Achievement Higher Achievement Free/Reduced Lunch
Lower Growth Higher Growth
Percent at/above Proficient in School
Less than 20 percent
● 20 to 40 percent
● 40 to 60 percent
● 60 to 80 percent
● More than 80 percent
School Size
50 Students
100 Students
200 Students
30 500 Students
1,000 Students
10 Lower Growth Higher Growth
Lower Achievement Lower Achievement
Median of Student Growth Percentiles in School
Intro Value-Added
Growth & Accountability: Part I Description, Inference, and Causality
Growth & Accountability: Part II Criterion Referenced Growth
References Growth & Effectiveness
Growth, Effectiveness, and Value-Added
“Good schools” are often called highly effective schools.
What’s the relationship between growth and effectiveness?
Effectiveness indicates who/what is responsible for the growth
(value-added models).
RttT has placed teacher responsibility for student progress as a
primary issue.
In teacher performance evaluations, even when large scale
assessment results are available, depending upon the state they
constitute, on average, about a third of the overall teacher rating.
Damian W. Betebenner Growth & Value-Added Models
2009 CSAP Elementary School Mathematics:
School Poverty by School Achievement (r = −0.71)
80 Higher Achievement
Percent at/above Proficient
Lower Achievement
Percent Free/Reduced Lunch Students at School
2009 CSAP Elementary School Mathematics:
School Poverty by School Median SGP (r = −0.22)
80 Higher Average Growth
Median Student Growth Percentile
20 Lower Average Growth
Percent Free/Reduced Lunch Students at School
2009 CSAP Elementary School Reading:
School Poverty by School Median SGP (r = −0.25)
80 Higher Average Growth
Median Student Growth Percentile
20 Lower Average Growth
Percent Free/Reduced Lunch Students at School
Economic Disadvantage vs School Effectiveness
TN Elementary Schools 2008
Average Yearly Achievement Gain 1
Math & Reading/Language Arts in Z units from
TN Current State Average
Percent Free & Reduced Lunch
Intro
Growth & Accountability: Part I
Growth & Accountability: Part II
References
References
Betebenner, D. W. (2008).
Toward a normative understanding of student growth.
In Ryan, K. E. and Shepard, L. A., editors, The Future of Test-Based
Educational Accountability, pages 155–170. Taylor & Francis, New
York.
Betebenner, D. W. (2009).
Norm- and criterion-referenced student growth.
Educational Measurement: Issues and Practice, 28(4):42–51.
Yen, W. M. (2007).
Vertical scaling and No Child Left Behind.
In Dorans, N. J., Pommerich, M., and Holland, P. W., editors, Linking
and Aligning Scores and Scales, pages 273–283. Springer, New
York.
Damian W. Betebenner Growth & Value-Added Models
About this copy
An Overview of Growth & Value-Added Models — Georgia Department of Education, January 24, 2011.
This is an image-based copy of the original beamer slides; builds and animations are flattened.