VERSCHLIMMBESSERN
Damian W Betebenner
National Center for the Improvement of Educational Assessment
Boulder, CO May 19th, 2015
GERMAN: A LANGUAGE OF UNIQUE WORDS
• Schadenfreude: The pleasure derived from the misfortune of
others.
• Kummerspeck: Weight gained from emotional overeating.
• Backpfeifengesicht: A face that cries out for a st in it.
• Verschlimmbessern: To make something worse in an attempt
to improve it
fi
Are we actually pulling a Verschlimmbessern
with our work in education
“You’re just lucky you didn’t hire the guy down the
street to do your plastic surgery.”
VERSCHLIMMBESSERN
• I’m not convinced that we aren’t engaged in what historically will be recognized
as verschlimmbessern.
• Over the last several years, there are several examples in my own area that
caused me heartburn that I’d like to discuss today:
• (Ab)use of the term measure/measurement.
• (Ab)use of the term “on track”.
• In writing this I see that a major problem is in how ideas are
(mis)communicated.
WHAT’S A MEASURE (AND WHAT ISN’T)
“Not everything that counts can be counted, and not
everything that can be counted counts”
Attributed to Einstein
• Our eld has been lax on policing the boundaries of what is (and isn’t a
measure).
• Stevens: “measurement is the assignment of numerals to objects or events
according to rule”.
• See Joel Michell’s Measurement in Psychology (2005) for a critical examination of
Steven’s concept and psychological measurement in general.
• When nearly everything’s a measure, it’s great for our business.
fi
WHAT’S A MEASURE (AND WHAT ISN’T)
• For example, are we producing “measures” of effective teaching?
• Measures versus indicators. (How large a dinner table is, square feet, versus
how many people it seats)
• You say tomato I say tomato. What difference does it make.
• How does the validation of an indicator differ from the validation of a measure?
WHAT DOES “ON TRACK” MEAN
• I’ve been carrying on a discussion with Kathryn Castellano and (recently)
Andrew Ho about “growth to standard” and how it is validated.
• After a lot of back and forth, I’d say that the issue comes down to de ning
what “on track” means:
• “In this ‘growth to standards’ framework (Betebenner & Linn, 2009), only students who are not
currently pro cient are help up to the growth criterion and classi ed as either ‘on track’ or ‘not
on track’ to pro ciency in a future grade level. In other words, the growth models are intended
to identify students who will be pro cient in the future and give schools credits for moving
students toward pro ciency [emphasis added].
• Students are classi ed as either ‘on track’ or ‘not on track’ to pro ciency in a future grade level
depending upon whether students are predicted to be pro cient in the future.
• Students are classi ed as either ‘on track’ or ‘not on track’ to pro ciency if they are “proceeding
as planned, as expected, or in a manner consistent with an established pattern.”
fi
fi
fi
fi
fi fi
fi
fi
fi fi
fi
WHAT DOES “ON TRACK” MEAN
• An obese individual in consultation with their physician could establish a one year weight
reduction plan with monthly weight targets leading to their ultimate weight reduction goal.
Moreover, on track would only be construed to mean “predicted” to meet their weight
reduction goal contingent upon maintenance of the plan.
• Individuals with a 401K plan establish age and nancial goals for retirement. These goals result in
contribution plans that, subject to market conditions, will yield the desired result. Being “on
track” to reach a retirement goal does not mean “will reach your retirement goal” as market
conditions, as well as employment, can change that can signi cantly alter what subsequently
occurs. Moreover, on track would only be construed to mean “predicted” to meet their
retirement goal based upon a number of factors including market conditions, employer and
employee contributions to the plan, …
fi
fi
WHAT DOES “ON TRACK” MEAN
• Some example from real life that motivated the SGP idea of “on track”.
• In the Ph.D program at CU, a four or ve year time line is established mapping out a sequence leading from
admission to completion of the Ph.D.
• Year 1: First year of course work.
• Year 2: Second year (completion) of course work.
• Year 3: Comprehensive examinations.
• Year 4: Prospectus writing/defense.
• Year 5: Dissertation writing/defense/completion
• On track indicates status with regard to the timeline established.
• A student completing their course work at the end of the 2nd year would be characterized correctly as being
“on track” to nish their Ph.D in 5 years. However, that shouldn’t/wouldn’t be construed to mean they will or are
predicted to nish in 5 years.
fi
fi
fi
About this copy
Staff Sharing — Center for Assessment staff meeting, May 19, 2015.
This is an image-based copy of the original Keynote slides; builds and animations are flattened.