Growth in Student Achievement: Issues of Measurement, Longitudinal Analyses & Accountability

CCSSO · June 23, 2010

Growth in Student Achievement: Issues of Measurement, Longitudinal Analyses & Accountability — CCSSO, June 23, 2010.
Published

June 23, 2010

Event
Conference
Slides
38
Format
Converted from PowerPoint

This deck is an image-based copy of the original slides. Builds and animations are flattened; the slide text is reproduced below for search and screen readers.

Slide text
  1. Growth in Student Achievement: Issues of Measurement, Longitudinal Analyses & Accountability Damian W. Betebenner NCIEA CCSSO NCSA, June 23, 2010
  2. Discussions of student growth lie at the intersection of three topics Longitudinal Data Analysis/Applied Statistics Overview Accountability/Education Policy/Data Use Measurement/Psychometrics
  3. Measurement/Psychometrics Examining student growth requires multiple measurements of the same individual  Growth in what? Overview  How much growth? (How is scaling involved in answering this question?)  Is it enough growth?
  4. Longitudinal Data Analysis/Applied Statistics Many methods for analysis of longitudinal data  What are the relevant questions?  Are the analytic techniques capable of Overview answering those questions?  Does the data possess properties sufficient for the analytic techniques employed? (e.g., vertical scale)  Does the analysis sustain the inferences made from the data?
  5. Accountability/Education Policy/Data Use Education Policy & Accountability have many goals and purposes  Why growth in accountability? Overview  What are the goals and purposes of accountability?  What is the theory of action behind accountability?  How can we judge the validity of the accountability system?  What about the current policy context?
  6. Measurement/Psychometric Issues Technical Considerations
  7. Measurement/Psychometric Issues Technical Considerations  Growth in what?  How much growth?  Scales for measuring growth  Ordinal (within-year, across year)  Interval (within-year, across year)  Vertical  Growth magnitude versus growth norm  Is it enough growth? Norm- versus criterion- referencing (intersection of Accountability and Measurement)
  8. Growth in what? Technical Considerations  Beneath any notion of change (i.e., growth) is a construct that is changing over time  Height and weight are common points of reference  Constructs in education are “slippery”  Need, at a minimum, an underlying semantical referent (e.g. reading or math)
  9. How much growth? Technical Considerations  Are growth magnitudes possible in education?  If calculable, are they interpretable absent some norm?  Approaches to growth magnitudes:  Performance standards  Vertical scale with interval properties  Learning progressions (qualitative growth)
  10. How much growth? Technical Considerations Performance Standards Strengths Limitations  Anchors reference  Few levels, mask points for discussions substantial range within about performance levels thus masking student growth within  Growth is embedded level in accountability metric  Vary greatly in stringency from state to state so that “proficient” performance lacks meaning
  11. How much growth? Technical Considerations Scale Scores Strengths Limitations  Semi-continuous scores  Difficult to interpret or (many score points) explain to users  Can be used to create  Vertical scales are hard vertical scales across to defend grade levels  Claims of interval  Give the appearance of measurement interval scales needed properties don’t hold to by some analytical close scrutiny models
  12. How much growth? Technical Considerations Vertical Scale Vertical & Interval scales required for some analytic techniques:  Gain score calculation (magnitude of growth)  Growth curve analysis (rate of growth) (e.g., Willett & Singer, 2003) Vertical & Interval scales required for some questions:  Matthew effects: Do higher achievers grow faster than lower achievers?  Growth rates relative to student age: Do students grow more in later grades than earlier grades?
  13. How much growth? Technical Considerations Vertical Scale Vertical and/or Interval scales NOT required for some analytic techniques:  Value-Added analyses: Most require interval, but not vertical, scale. See Ballou (2008), Briggs & Betebenner  Auto-regressive analyses, growth norms Vertical and/or Interval scales NOT required for some questions:  Is a student’s progress (ab)normal?  Is a student’s growth sufficient to put them on track to reach/maintain proficiency?  See Yen (2007) for an excellent list of questions
  14. How much growth? Technical Considerations Magnitudes versus Norms Physical growth Two Growth Quantities  9 year old boy grew 5 inches in past year  Magnitude of growth  Average increase in height for boys between years 8  Relative amount of growth and 9 is 4 inches How much growth? Achievement growth  4th grader grew 25 scale  People expect an answer score points since 3rd grade of magnitude  Average 4th grade scale  People need magnitude score is 21 points higher than average 3rd grade score embedded within a norm
  15. How much growth? Technical Considerations Growth norms Although normative comparisons are spurned by criterion-referenced and standards-based measurement advocates, norms can provide a useful interpretive framework, especially in the interpretation of student growth “Scratch a criterion and you find a norm” W. H. Angoff (1974)
  16. Longitudinal Data Analysis Issues Technical Considerations
  17. Many Questions Technical Considerations  How much annual growth did this (these) student(s) make in reading?  Is (Are) this (these) student(s) making sufficient growth to reach/maintain desired achievement targets? (Growth-to-standard & Growth Model Pilot Program)  Are students in particular subgroups (e.g., minority students) making as much progress as other students?  How much did this teacher/school contribute to students’ growth over the last year? (Value-Added)  Again, see Yen (2007) for an excellent list of questions
  18. Many Techniques Technical Considerations Numerous data analysis techniques for use with longitudinal data:  Gain scores (suitable scale required)  Cross-tabulation based upon prior and current categorical achievement level attainment (e.g., value-tables, transition matrices)  Regression based approaches: growth-curve analysis (HLM), fixed/mixed-effects models, growth norms
  19. Questions 1st, Analyses 2nd Technical Considerations  Different growth analysis techniques often address different questions  Different questions lead to different conversations which lead to different uses and outcomes “It is better to have an approximate answer to the right question than a precise answer to the wrong question.” J. W. Tukey
  20. Model Purpose Technical Considerations Three general uses associated with statistical models (Berk, 2004): Description: An account of the data. Model is true to the extent that it is useful. Model quality judged by craftsmanship (de Leeuw, 2004) Inference: Sample to Population. Model is true to the extent that the assumed chance process reflects reality (super-population fallacy) Causality: A causes B to happen. Model is true to the extent that plausible causal theory exists and design criteria are met  Models are rarely descriptive despite minimal requirements  Inference and causality require information external to the data. Can’t be validated solely from data  Models are often causal in nature but rarely meet rigorous criteria necessary for such inferences
  21. Value-Added Models Technical Considerations Causality  Value-Added Models (e.g., EVAAS) are a frequently discussed type of growth model  Value-Added Models attempt to quantify the portion of student progress attributable, usually to a teacher or school  Value-Added is about the inferences made and not the actual model  Causal attributions make value-added models well suited for accountability discussions  In the absence of random assignment causal attributions are always suspect and subject to challenges (see, for example, Raudenbush, 2004; Rubin, Stuart & Zanutto, 2004)
  22. Value-Added Models Technical Considerations Causality  Value-added models return norm-referenced effectiveness quantities  With regard to schools, quantities indicate whether a school is significantly more or less effective than the mean school effectiveness in the district or state  In a standards based assessment environment, how much effectiveness is enough?  Especially important in light of universal proficiency policy mandates  Growth-to-standard models created to provide criterion-referenced growth models
  23. Growth Model Pilot Program Technical Considerations Growth-to-standard  In response to requests for growth model use as part of AYP, USED allowed states to apply to use growth models  Fifteen states had models accepted  Models required to adhere to the “bright line principle” of universal proficiency (growth-to- standard)  Yen (2009) provides an excellent overview of the models  Growth-to-standard models returned, in general, results that closely aligned with AYP status results.
  24. Growth versus Value-Added Models Technical Considerations Description & Causality  Growth measures are descriptive  Accountability has skewed discussions of growth from description toward responsibility (i.e., causality)  All measures (even VAM) are potentially descriptive. However, some measures are specially crafted for causal inference/attribution  Good descriptive measures are interpretable, informative and capable of multiple uses
  25. Growth versus Value-Added Models Technical Considerations Description: Colorado Growth Model  The Colorado Growth Model uses student growth percentiles to quantify student growth  Percentiles are familiar to stakeholders  Separating description from responsibility has led to broad public acceptance including teacher’s unions  Asking what schools or teachers are associated with students demonstrating the highest growth percentiles moves from description toward value-added
  26. Growth versus Value-Added Models Technical Considerations Description: Colorado Growth Model  Analysis employs quantile regression to calculate conditional quantile relationships between current and prior achievement  Student growth percentiles can also be criterion referenced to accommodate growth-to-standard  This approach formed the basis of Colorado’s successful application as part of the Growth Model Pilot Program  Student growth percentiles provide a bridge connecting value-added and criterion-referenced interests
  27. Validating Models Technical Considerations  There is no “gold standard” against which to judge value-added or growth model results  Statistical model specification goes only part way toward validation  Results should have credibility (i.e. face validity)  Because of their importance in accountability, utility is a primary component of model validity “All models are wrong but some are useful.” G. E. P. Box
  28. Accountability/Policy/Data Use Technical Considerations
  29. Accountability Systems Technical Considerations Purpose and requirements  Intended to improve education  Increase student achievement  Reduce achievement gaps  Increase efficiencies  Externally mandated and designed to hold educators responsible for student learning  Impose sanctions and rewards based upon results from large scale assessment outcomes
  30. Accountability and Growth Technical Considerations What type of growth?  Value-added provides a norm-referenced lens judging growth/effectiveness against district/state averages.  Growth-to-standard provides a criterion-referenced lens judging growth toward community endorsed achievement goals  Inferences about education quality based upon value-added make judgments relative to students reaching statistical expectation  Inferences about education quality based upon growth-to-standard make judgments relative to students reaching criterion-referenced destinations
  31. Accountability and Growth Technical Considerations What type of growth?  States currently employ a variety of growth models in service of accountability  Current policy mandates like NCLB are criterion- referenced---establishing achievement targets/destinations for all students  Need BOTH norm- and criterion-referenced growth to reconcile individual focused policies like NCLB with imperatives to judge education quality at the group level (e.g., teacher or school)
  32. Accountability Systems Technical Considerations Theory of action  Theory of action connects interpretations, uses, and consequences (Gong, 2008)  Details connecting punishments/rewards and outcomes are usually vague/incomplete  It is exactly these details that are critical to validating the theory of action associated with the accountability system’s use of growth any growth/value-added metric
  33. Accountability System Validity Technical Considerations Systemic Validity “Assessment practices and systems of accountability are systemically valid if they generate useful information and constructive responses that support one or more policy goals (Access, Quality, Equity, Efficiency) within an education system, without causing undue deterioration with respect to other goals.” H. Braun (2008)
  34. Descriptive Accountability Technical Considerations Building in systemic validity “Accountability system results can have value without making causal inferences about school quality, solely from the results of student achievement measures and demographic characteristics. Treating the results as descriptive information and for identification of schools that require more intensive investigation of organizational and instructional process characteristics are potentially of considerable value. Rather than using the results of the accountability system as the sole determiner of sanctions for schools, they could be used to flag schools that need more intensive investigation to reach sound conclusions about needed improvements or judgments about quality.” R. L. Linn (2008)
  35. Descriptive Accountability Technical Considerations  Part of a broader research program  Helpful in spotting provocative associations  A part of advocacy/informative discussions (e.g, growth-gaps by ethnicity)  Informs policy goals and initiatives The descriptive growth norms of the Colorado Growth Model are an example of this type of accountability metric
  36. Bibliography  Angoff, W. H. (1974). Criterion-referencing, norm-referencing and the SAT. College Board Review, 92:2–5.  Ballou, D. (2008). Test Scaling and Value-Added Measurement. Presented at National Conference on Value-Added Modeling, Madison, Wisconsin. Bibliography  Berk, R. A. (2004). Regression analysis: A constructive critique. Sage Publications, Thousand Oaks, CA.  Braun, H. (2008). Viccissitudes of the Validators. Presented at the Reidy Interactive Lecture Series, Portsmouth, New Hampshire.  Breiter, A & Light, D (2006). Data for School Improvement: Factors for designing effective information systems to support decision-making in schools. Educational Technology & Society, 9 (3), 206-217.  Briggs, D. C. M. & Betebenner, D. W. (2009) Is growth in student achievement scale dependent?. Paper presented at the NCME annual conference, April 2009. San Diego, CA.  Campbell, D. T. (1976), Assessing the Impact of Planned Social Change. The Public Affairs Center, Dartmouth College, Hanover New Hampshire, USA. December, 1976.  De Leeuw, J. (2004) Preface to Berk’s “Regression analysis: A constructive critique”. Sage Publications, Thousand Oaks, CA.
  37. Bibliography  Downloaded January 20th, 2009 from http://www.education- consumers.com/VAAA/Poverty%20vs%20School%20Effectivness%202 006%20chart.pdf  Downloaded January 20th, 2009 from http://www.education- consumers.org/tnproject/poverty_vs_effectiveness_2008.pdf Bibliography  Edley, C. (2006). Educational “Opportunity" is the highest civil rights priority. So what should researchers and lawyers do about it? Retrieved June 22, 2006 from the World Wide Web: http://www.softconference.com/MEDIA/WMP/260407/#43.010  Gong, B. (2008). Validating accountability systems. Presented at the Reidy Interactive Lecture Series, Portsmouth New Hampshire  Linn, R. L. (2000). Assessments and accountability. Educational Researcher 29(2): 4-16.  Linn, R. L. (2008). Educational accountability systems. In K. E. Ryan & L. A. Shepard (Eds.), The Future of Test-Based Educational Accountability, pages 3–24. Taylor & Francis, New York.  Mintrop, H. & Sunderman, G. L. (2009). Why high stakes accountability sounds good but doesn’t work—and why we keep on doing it anyway. Los Angeles, CA: The Civil Rights Project/Proyecto Derechos Civiles at UCLA.
  38. Bibliography  Raudenbush, S. W. (2004). What are value-added models estimating and what does this imply for statistical practice? Journal of Educational and Behavioral Statistics, 29 (1), 121–129. Bibliography  Rubin, D. B., Stuart, E. A., & Zanutto, E. L. (2004). A potential outcomes view of value-added assessment in education. Journal of Educational and Behavioral Statistics, 29 (1), 103–116.  Singer, J. D., & Willett, J. B. (2003). Applied longitudinal data analysis: Modeling change and event occurrence. Oxford University Press, USA.  Yen, W. M. (2007). Vertical scaling and No Child Left Behind. In N. J. Dorans, M. Pommerich, & P. W. Holland (Eds.), Linking and Aligning Scores and Scales, pages 273–283. Springer, New York.  Yen, W. M. (2009, March) Growth Models Approved for the NCLB Growth Model Pilot. ETS.