Overview
Validity vis-à-vis Accountability Systems
Data Quality vis-à-vis Data Use
Data is not equal to Information
The Medium is the Message
References
Data Visualization/Reporting and Valid Accountability System Design
Damian W. Betebenner
National Center for the Improvement of Educational Assessment
Dover, NH
CCSSO National Conference on Student Assessment
Orlando, Florida June 21st, 2011
Damian W. Betebenner Data Visualization/Reporting and Valid Accountability System Design
Overview
Validity vis-à-vis Accountability Systems
Data Quality vis-à-vis Data Use
Data is not equal to Information
The Medium is the Message
References
Topics of Today’s Presentation
Validity vis-à-vis Accountability Systems.
Data Quality vis-à-vis Data Use.
Data , Information.
The Medium is the Message.
Damian W. Betebenner Data Visualization/Reporting and Valid Accountability System Design
Overview
Validity vis-à-vis Accountability Systems
Data Quality vis-à-vis Data Use
Data is not equal to Information
The Medium is the Message
References
Validity & Accountability (Braun 2008)
Validity as a scientific enterprise
Validity is an ongoing argument that seeks to clarify what a
measurement means and to understand the limitations of each
score interpretation. (Cronbach,1988)
Validity as consideration of consequences
Validity is an overall evaluative judgment, founded on empirical
evidence and theoretical rationales, of the adequacy and
appropriateness of inferences and actions based on test
scores. (Messick, 1989)
Damian W. Betebenner Data Visualization/Reporting and Valid Accountability System Design
Overview
Validity vis-à-vis Accountability Systems
Data Quality vis-à-vis Data Use
Data is not equal to Information
The Medium is the Message
References
Systemic Validity (Braun, 2008)
“Assessment practices and systems of accountability are
systemically valid if they generate useful information and
constructive responses that support one or more policy goals
(Access, Quality, Equity, Efficiency) within an education system,
without causing undue deterioration with respect to other goals.”
“In validating an accountability system, the theory-of-action
plays the same role as does the construct in test validation.”
“Consequential validity is the ultimate criterion by which we
should judge an accountability system.”
Damian W. Betebenner Data Visualization/Reporting and Valid Accountability System Design
Overview
Validity vis-à-vis Accountability Systems
Data Quality vis-à-vis Data Use
Data is not equal to Information
The Medium is the Message
References
Theory of Action
What is the theory of action that turns data into information that
increases educational efficacy?
Articulating that theory of action is essential to validating the teacher
evaluation system.
Example 1: Publishing VAM scores in the LA Times.
Example 2: Incentivizing behavior through monetary compensation
(see http://www.youtube.com/watch?v=u6XAPnuFjJc).
Perhaps this is what Box was referring to when he said, “All models
are wrong, some are useful.”
Damian W. Betebenner Data Visualization/Reporting and Valid Accountability System Design
Overview
Validity vis-à-vis Accountability Systems
Data Quality vis-à-vis Data Use
Data is not equal to Information
The Medium is the Message
References
Creating a Virtuous Cycle
Assessment Data Quality Assessment Data Use
Damian W. Betebenner Data Visualization/Reporting and Valid Accountability System Design
Overview
Validity vis-à-vis Accountability Systems
Data Quality vis-à-vis Data Use
Data is not equal to Information
The Medium is the Message
References
The Data to Knowledge Continuum
Damian W. Betebenner Data Visualization/Reporting and Valid Accountability System Design
Overview
Validity vis-à-vis Accountability Systems
Data Quality vis-à-vis Data Use
Data is not equal to Information
The Medium is the Message
References
From Paper to Interactive Media
We are witnessing a revolution in the way in which humans interact
with media. From static (paper) to interactive (Web 2.0)
Confirmatory versus Exploratory data analysis.
Causal Attribution (man as researcher, intra-psychic) versus Causal
Explanation (conversational, inter-psychic).
Rendering verdicts about education/educator quality or promoting
conversations about education/educator quality.
UX (User Experience) Design: What use cases do we want to
promote? (Telling versus Involving).
Tell me and I will forget,
show me and I will remember,
involve me and I will understand.
(Confucius)
Damian W. Betebenner Data Visualization/Reporting and Valid Accountability System Design
Overview
Validity vis-à-vis Accountability Systems
Data Quality vis-à-vis Data Use
Data is not equal to Information
The Medium is the Message
References
References
Braun, H. I. (2008).
Viccissitudes of the validators.
Presentation made at the 2008 Reidy Interactive Lecture Series,
Portsmouth, NH, September, 2008.
Damian W. Betebenner Data Visualization/Reporting and Valid Accountability System Design
About this copy
Data Visualization/Reporting and Valid Accountability System Design — CCSSO, June 21, 2011.
This is an image-based copy of the original beamer slides; builds and animations are flattened.