Norm- and Criterion-Referenced Student Growth

NCME Annual Meeting · March 2008

Norm- and Criterion-Referenced Student Growth — NCME Annual Meeting, March 2008.
Published

March 26, 2008

Event
Conference
Slides
18
Format
Converted from PDF

This deck is an image-based copy of the original slides. Builds and animations are flattened; the slide text is reproduced below for search and screen readers.

Slide text
  1. bg=White Growth, Status, and Accountability Normative Student Growth Descriptive/Prescriptive Growth References Norm- and Criterion-Referenced Student Growth Damian W. Betebenner National Center for the Improvement of Educational Assessment Dover, NH NCME 2008 Annual Meeting March 26th, 2008 D. W. Betebenner Norm- and Criterion-Referenced Student Growth
  2. bg=White Growth, Status, and Accountability Normative Student Growth Growth and Accountability Models Descriptive/Prescriptive Growth Growth & Status Models References Growth & Accountability Bob’s Axiom Scratch a criterion and you (should) find a norm. Accountability mandates—like NCLB’s “bright line” (i.e., universal proficiency) requirements—dominate discussions of assessment outcomes. In these discussions, it is important distinguish between the judgments made by the accountability system and the quantities on which the judgment is based. With the analysis of growth, it is important to distinguish between a description of growth provided by the model and the qualification of that value as (in)adequate by the accountability system. Growth-to-standard methodologies qualify, for accountability purposes, whether a student is on track to proficient and use this qualification as evidence of school quality. These criterion referenced growth models define adequate growth without establishing a norm and thus are in violation of Bob’s Axiom. D. W. Betebenner Norm- and Criterion-Referenced Student Growth
  3. bg=White Growth, Status, and Accountability Normative Student Growth Growth and Accountability Models Descriptive/Prescriptive Growth Growth & Status Models References Growth & Status Unconditional Projected Conditional Achievement Achievement Achievement Status Growth Current growth-to-standard methodologies judge individual student progress based upon projections of future student achievement. In this sense, growth-to-standard models are projected achievement models. Given the representational continuum connecting status with growth, growth-to-standard models are hybrids, neither growth nor status. Not surprisingly, results derived from their use correlate highly with results derived from existing status models [Dunn, 2007]. D. W. Betebenner Norm- and Criterion-Referenced Student Growth
  4. bg=White Growth, Status, and Accountability Student Growth Percentiles Normative Student Growth Describing Actual Student Growth Descriptive/Prescriptive Growth Exploring the Range of Aspirational Outcomes References Actual versus Aspirational Student Growth Conditional achievement (i.e., growth) models can serve the same ends as projected achievement/growth-to-standard models. Doing so requires distinguishing actual growth from aspirational growth. Actual What is a current year’s growth? Aspirational What should a current year’s growth be? Answering the second question establishes a threshold distinguishing adequate from inadequate growth. To make such a distinction requires an answer to the first question which defines a norm: What is the range of growth currently observed? Aspirational growth should be possible—Ought implies Can (Kant) or Linn’s existence proof [Linn, 2003]. D. W. Betebenner Norm- and Criterion-Referenced Student Growth
  5. bg=White Growth, Status, and Accountability Student Growth Percentiles Normative Student Growth Describing Actual Student Growth Descriptive/Prescriptive Growth Exploring the Range of Aspirational Outcomes References Describing Actual Student Growth Measuring student growth, even with a vertical scale, is not a simple task. Numerous states believe that placing their tests on a vertical scale will simplify the task of measuring student growth. Even with an interval (or ratio) scale, growth is not easy to interpret. Consider, for example, height. A child might grow 4 inches between ages 3 and 4. 4 inches is a well understood quantity. However, the 4 inch increase becomes fully understandable only when placed in context of growth of other children between ages 3 and 4. The following introduces student growth percentiles as a normative quantity useful to interpreting student growth. D. W. Betebenner Norm- and Criterion-Referenced Student Growth
  6. bg=White Growth, Status, and Accountability Student Growth Percentiles Normative Student Growth Describing Actual Student Growth Descriptive/Prescriptive Growth Exploring the Range of Aspirational Outcomes References Student Growth Percentiles A quantity of fundamental interest to norming student progress (i.e., conditional achievement) is: Pr(Current Achievement |Past Achievement ). (1) A student’s growth percentile is the conversion of 1 to a percentile. Estimation of the conditional density used to calculate student growth percentiles is performed using quantile regression (Betebenner & Shang, 2007; Koenker, 2005). Consider a low achieving student demonstrating 90th percentile growth and a high achieving student demonstrating 10th percentile growth. The low achieving student grew at a rate exceeding 90 percent of similar students. The high achieving student grew at a rate exceeding just 10 percent of similar students. In this sense, the low achiever’s growth is more exemplary than the high achiever’s. Growth percentiles allow for easy understanding and comparison of student growth. However, judgment of whether the student’s growth was adequate requires external criteria. D. W. Betebenner Norm- and Criterion-Referenced Student Growth
  7. bg=White Growth, Status, and Accountability Student Growth Percentiles Normative Student Growth Describing Actual Student Growth Descriptive/Prescriptive Growth Exploring the Range of Aspirational Outcomes References Performance Standards and Growth Percentiles Performance standards are the most obvious external criteria needed to define what adequate growth is. If proficiency within 3 years is the achievement goal, then one can examine prior growth to estimate what growth percentile is sufficient to reach the desired outcome. For high achieving students, modest growth percentiles are required to maintain their position above proficiency whereas for some low achieving student sustained high growth percentiles are necessary. The process of defining (in)adequate growth is a standard setting procedure. Clearly, defining adequate growth for some students as 80th percentile or better sets a very high standard. And defining a year’s growth for some students as 25th percentile or better sets a low standard. Growth standards should be equitable, reasonable, and desirable. D. W. Betebenner Norm- and Criterion-Referenced Student Growth
  8. bg=White Growth, Status, and Accountability Normative Student Growth Achievement Projections/Forecasts Descriptive/Prescriptive Growth References Achievement Projections/Forecasts Growth-to-standard analyses project future student achievement and use these determinations for accountability purposes. Such projections implicitly assume (but rarely explicate) interim rates of growth for the student. The result of growth-to-standard analyses are almost fatalistic statements about the future status of students. Such conversations often leave teachers/educators wondering what their role is if the student’s future status is known. Conversations about future status must turn from “where will (s)he be” to “what will it take”. Using student growth percentile estimation techniques it is possible to forecast different growth scenarios that allow stakeholders to entertain conversations of “what it will take”. These forecasts establish a basis for growth standard setting. Only by considering what is, what should be, and what is possible simultaneously, can growth models and their associated standards by equitable, just and truly informed. D. W. Betebenner Norm- and Criterion-Referenced Student Growth
  9. Math Growth Percentile Trajectories Growth Trajectories for 25th, 40th, 50th, 60th, 75th, & 90th Percentiles 2007 Achievement Percentile Growth Percentile (Consecutive) Scale Score Grade
  10. Math Growth Percentile Trajectories Growth Trajectories for 25th, 40th, 50th, 60th, 75th, & 90th Percentiles 2007 Achievement Percentile Growth Percentile (Consecutive) Scale Score Grade
  11. Math Growth Percentile Trajectories 90 Growth Trajectories for 25th, 40th, 50th, 60th, 75th, & 90th Percentiles 2007 Achievement Percentile Growth Percentile (Consecutive) Scale Score Grade
  12. Reading Growth Percentile Trajectories Growth Trajectories for 10th, 25th, 40th, 50th, 60th, 75th, & 90th Percentiles Growth Percentile (Consecutive) 2007 Achievement Percentile Scale Score Grade
  13. Reading Growth Percentile Trajectories Growth Trajectories for 10th, 25th, 40th, 50th, 60th, 75th, & 90th Percentiles Growth Percentile (Consecutive) 2007 Achievement Percentile Scale Score Grade
  14. Reading Growth Percentile Trajectories Growth Trajectories for 10th, 25th, 40th, 50th, 60th, 75th, & 90th Percentiles90 Growth Percentile (Consecutive) 2007 Achievement Percentile Scale Score Grade
  15. Writing Growth Percentile Trajectories Growth Trajectories for 10th, 25th, 40th, 50th, 60th, 75th, & 90th Percentiles 2007 Achievement Percentile Growth Percentile (Consecutive) Scale Score Grade
  16. Writing Growth Percentile Trajectories Growth Trajectories for 10th, 25th, 40th, 50th, 60th, 75th, & 90th Percentiles 2007 Achievement Percentile Growth Percentile (Consecutive) Scale Score Grade
  17. Writing Growth Percentile Trajectories Growth Trajectories for 10th, 25th, 40th, 50th, 60th, 75th, & 90th Percentiles 2007 Achievement Percentile Growth Percentile (Consecutive) Scale Score Grade
  18. bg=White Growth, Status, and Accountability Normative Student Growth Descriptive/Prescriptive Growth References References Betebenner, D. W. and Shang, Y. (2007). Reference growth charts for educational outcomes. Paper presented at the 2007 AERA Annual Conference, Chicago, IL. Retrieved May 12th, 2007 from http://monticello.bc.edu/Betebenner. Dunn, J. (2007). When does a “growth model" act the same as a "status model": Lessons learned from some empirical growth model comparisons. Paper presented at the Systems and Reporting SCASS, September 25, 2007, Nashua, NH. Koenker, R. (2005). Quantile Regression. Cambridge University Press, Cambridge. Linn, R. L. (2003). Accountability: Responsibility and reasonable expectations. Technical report, Center for the Study of Evaluation, CRESST, Los Angeles, CA. D. W. Betebenner Norm- and Criterion-Referenced Student Growth