The Big Picture
Validity & Accountability Systems
References
Measuring Teacher Effectiveness in a Comprehensive Teacher
Evaluation System: Fair, Accurate, & Legally Defensible?
Damian W. Betebenner
National Center for the Improvement of Educational Assessment
Dover, NH
NATD/NCME Symposium
April 9th, 2011
Damian W. Betebenner Measuring Teacher Effectiveness
The Big Picture
Validity & Accountability Systems
References
Student Growth: An Intersection of Three Topics
Longitudinal Data Analysis/Applied Statistics
Accountability/Education Policy/Data Use
Measurement/Psychometrics
Damian W. Betebenner Measuring Teacher Effectiveness
The Big Picture
Validity & Accountability Systems
References
Validity & Accountability
A Fair, Accurate, & Legally Defensible Teacher Evaluation
System?
Instead, what about a valid teacher evaluation system?
What constitutes such a system?
How would we distinguish a valid teacher evaluation system
from an invalid one?
Damian W. Betebenner Measuring Teacher Effectiveness
The Big Picture
Validity & Accountability Systems
References
Validity & Accountability (Braun 2008)
Validity as a scientific enterprise
Validity is an ongoing argument that seeks to clarify what
a measurement means and to understand the limitations
of each score interpretation. (Cronbach,1988)
Validity as consideration of consequences
Validity is an overall evaluative judgment, founded on
empirical evidence and theoretical rationales, of the
adequacy and appropriateness of inferences and actions
based on test scores. (Messick, 1989)
Damian W. Betebenner Measuring Teacher Effectiveness
The Big Picture
Validity & Accountability Systems
References
Systemic Validity (Braun, 2008)
Assessment practices and systems of accountability are
systemically valid if they generate useful information and
constructive responses that support one or more policy
goals (Access, Quality, Equity, Efficiency) within an
education system, without causing undue deterioration
with respect to other goals.
In validating an accountability system, the
theory-of-action plays the same role as does the
construct in test validation.
Consequential validity is the ultimate criterion by which
we should judge an accountability system.
Damian W. Betebenner Measuring Teacher Effectiveness
The Big Picture
Validity & Accountability Systems
References
Theory of Action
What is the theory of action that turns data into information
that increases educational efficacy?
Articulating that theory of action is essential to validating the
teacher evaluation system.
Example 1: Publishing VAM scores in the LA Times.
Example 2: Incentivizing behavior through monetary
compensation (see
http://www.youtube.com/watch?v=u6XAPnuFjJc).
Perhaps this is what Box was referring to when he said, “all
models are wrong, some are useful.”
Damian W. Betebenner Measuring Teacher Effectiveness
The Big Picture
Validity & Accountability Systems
References
Re-framing the questions presented to the panel
What are necessary elements for assessing the construct of
“teacher effectiveness,” and where should student performance
data fit into these determinations?
Is “teacher effectiveness” actually a construct?
To what extent are we performing measurement when we
assess teacher quality?
In qualifying teacher effectiveness, should this be done in a
norm-referenced fashion or should it be, like student
academic achievement, criterion referenced? Is average
effectiveness “good enough”?
Damian W. Betebenner Measuring Teacher Effectiveness
The Big Picture
Validity & Accountability Systems
References
Re-framing the questions presented to the panel
How do these elements impact instructional practice?
How can we create an innovative and responsible system that
inspires confidence from educators, parents, and the general
public?
What are some of the land mines and unintended
consequences of using State assessment results to evaluate
teacher effectiveness?
All of these questions speak to consequences. That is, answering
these questions, by definition, greatly increases the likelihood of
implementing a valid teacher evaluation model.
Damian W. Betebenner Measuring Teacher Effectiveness
The Big Picture
Validity & Accountability Systems
References
References
Braun, H. I. (2008).
Viccissitudes of the validators.
Presentation made at the 2008 Reidy Interactive Lecture
Series, Portsmouth, NH, September, 2008.
Damian W. Betebenner Measuring Teacher Effectiveness
About this copy
Measuring Teacher Effectiveness in a Comprehensive Teacher Evaluation System: Fair, Accurate, & Legally Defensible? — NCME Annual Meeting, April 9, 2011.
This is an image-based copy of the original beamer slides; builds and animations are flattened.