The Importance of Performance Standards in Measures of Student Growth

CCSSO · June 2005

The Importance of Performance Standards in Measures of Student Growth — CCSSO, June 2005.
Published

June 24, 2005

Event
Conference
Slides
83
Format
Converted from PDF

This deck is an image-based copy of the original slides. Builds and animations are flattened; the slide text is reproduced below for search and screen readers.

Slide text
  1. The Importance of Performance Standards in Measures of Student Growth Damian W. Betebenner Lynch School of Education Boston College CCSSO 2005 National Conference on Large-Scale Assessment D. W. Betebenner Performance Standards and Growth
  2. Introduction No Child Left Behind Longitudinal Analysis of Student Data Important Considerations Standards Based Growth Institutional or Individual Growth Big Picture No Child Left Behind NCLB requirements for testing have left states buried in data with mandates to use the data to improve education. D. W. Betebenner Performance Standards and Growth
  3. Introduction No Child Left Behind Longitudinal Analysis of Student Data Important Considerations Standards Based Growth Institutional or Individual Growth Big Picture No Child Left Behind NCLB requirements for testing have left states buried in data with mandates to use the data to improve education. The most frequently cited mandate is that by 2014 all students will perform at a proficient or higher level in reading and math. D. W. Betebenner Performance Standards and Growth
  4. Introduction No Child Left Behind Longitudinal Analysis of Student Data Important Considerations Standards Based Growth Institutional or Individual Growth Big Picture No Child Left Behind NCLB requirements for testing have left states buried in data with mandates to use the data to improve education. The most frequently cited mandate is that by 2014 all students will perform at a proficient or higher level in reading and math. Concern that this mandate is unrealizable has led some experts to propose modifying NCLB’s AYP criteria to incorporate student growth. [Kingsbury et al., 2004, Hill, 2003, Thum, 2003] D. W. Betebenner Performance Standards and Growth
  5. Introduction No Child Left Behind Longitudinal Analysis of Student Data Important Considerations Standards Based Growth Institutional or Individual Growth Big Picture No Child Left Behind NCLB requirements for testing have left states buried in data with mandates to use the data to improve education. The most frequently cited mandate is that by 2014 all students will perform at a proficient or higher level in reading and math. Concern that this mandate is unrealizable has led some experts to propose modifying NCLB’s AYP criteria to incorporate student growth. [Kingsbury et al., 2004, Hill, 2003, Thum, 2003] Linn (2004), for example, suggests a combination of longitudinal student analysis with an annual NCLB-like status measure. D. W. Betebenner Performance Standards and Growth
  6. Introduction No Child Left Behind Longitudinal Analysis of Student Data Important Considerations Standards Based Growth Institutional or Individual Growth Big Picture Important Considerations The primary goal of longitudinal data analysis should be to return worthwhile information that informs, refines, alters, and ultimately improves practice. D. W. Betebenner Performance Standards and Growth
  7. Introduction No Child Left Behind Longitudinal Analysis of Student Data Important Considerations Standards Based Growth Institutional or Individual Growth Big Picture Important Considerations The primary goal of longitudinal data analysis should be to return worthwhile information that informs, refines, alters, and ultimately improves practice. Currently, standards are the lingua franca of instruction, assessment, and accountability. D. W. Betebenner Performance Standards and Growth
  8. Introduction No Child Left Behind Longitudinal Analysis of Student Data Important Considerations Standards Based Growth Institutional or Individual Growth Big Picture Important Considerations The primary goal of longitudinal data analysis should be to return worthwhile information that informs, refines, alters, and ultimately improves practice. Currently, standards are the lingua franca of instruction, assessment, and accountability. For growth analyses to be both understandable and useful, they should synchronize with these standards. D. W. Betebenner Performance Standards and Growth
  9. Introduction No Child Left Behind Longitudinal Analysis of Student Data Important Considerations Standards Based Growth Institutional or Individual Growth Big Picture Important Considerations The primary goal of longitudinal data analysis should be to return worthwhile information that informs, refines, alters, and ultimately improves practice. Currently, standards are the lingua franca of instruction, assessment, and accountability. For growth analyses to be both understandable and useful, they should synchronize with these standards. The purpose of this presentation is to better situate current analyses of longitudinal data within the context of performance standards. D. W. Betebenner Performance Standards and Growth
  10. Introduction No Child Left Behind Longitudinal Analysis of Student Data Important Considerations Standards Based Growth Institutional or Individual Growth Big Picture No Child Left Behind Institutional or Individual Growth NCLB’s AYP mandates stipulate growth. The growth, however, is institutional instead of individual. For example, next year’s 5th grade should outperform this year’s 5th grade. D. W. Betebenner Performance Standards and Growth
  11. Introduction No Child Left Behind Longitudinal Analysis of Student Data Important Considerations Standards Based Growth Institutional or Individual Growth Big Picture No Child Left Behind Institutional or Individual Growth NCLB’s AYP mandates stipulate growth. The growth, however, is institutional instead of individual. For example, next year’s 5th grade should outperform this year’s 5th grade. If cohorts are identical from year-to-year, then NCLB based growth will likely imply institutional improvement. D. W. Betebenner Performance Standards and Growth
  12. Introduction No Child Left Behind Longitudinal Analysis of Student Data Important Considerations Standards Based Growth Institutional or Individual Growth Big Picture No Child Left Behind Institutional or Individual Growth NCLB’s AYP mandates stipulate growth. The growth, however, is institutional instead of individual. For example, next year’s 5th grade should outperform this year’s 5th grade. If cohorts are identical from year-to-year, then NCLB based growth will likely imply institutional improvement. A second way to assess institutional improvement is to ensure that the “amount” students learn in a given grade increases over time. That is, for a given grade, there is an increasing rate of change in student growth. D. W. Betebenner Performance Standards and Growth
  13. Introduction No Child Left Behind Longitudinal Analysis of Student Data Important Considerations Standards Based Growth Institutional or Individual Growth Big Picture No Child Left Behind Institutional or Individual Growth NCLB’s AYP mandates stipulate growth. The growth, however, is institutional instead of individual. For example, next year’s 5th grade should outperform this year’s 5th grade. If cohorts are identical from year-to-year, then NCLB based growth will likely imply institutional improvement. A second way to assess institutional improvement is to ensure that the “amount” students learn in a given grade increases over time. That is, for a given grade, there is an increasing rate of change in student growth. A fundamental question is how to relate individual growth to NCLB conceived institutional growth. D. W. Betebenner Performance Standards and Growth
  14. Introduction Overview Longitudinal Analysis of Student Data Value-Added Analysis Standards Based Growth Teacher/School Comparisons Big Picture Longitudinal Data Analysis Annual state testing of students provides mountains of longitudinal data. D. W. Betebenner Performance Standards and Growth
  15. Introduction Overview Longitudinal Analysis of Student Data Value-Added Analysis Standards Based Growth Teacher/School Comparisons Big Picture Longitudinal Data Analysis Annual state testing of students provides mountains of longitudinal data. The most prominent type of longitudinal analysis currently employed on student data are the value-added analyses developed by Dr. William Sanders. D. W. Betebenner Performance Standards and Growth
  16. Introduction Overview Longitudinal Analysis of Student Data Value-Added Analysis Standards Based Growth Teacher/School Comparisons Big Picture Longitudinal Data Analysis Annual state testing of students provides mountains of longitudinal data. The most prominent type of longitudinal analysis currently employed on student data are the value-added analyses developed by Dr. William Sanders. In education, value-added analysis is synonymous with longitudinal analysis of data. D. W. Betebenner Performance Standards and Growth
  17. Introduction Overview Longitudinal Analysis of Student Data Value-Added Analysis Standards Based Growth Teacher/School Comparisons Big Picture Longitudinal Data Analysis Annual state testing of students provides mountains of longitudinal data. The most prominent type of longitudinal analysis currently employed on student data are the value-added analyses developed by Dr. William Sanders. In education, value-added analysis is synonymous with longitudinal analysis of data. Using this familiarity as a starting point, consider two basic questions: Value-added by whom/what? Value-added relative to what? D. W. Betebenner Performance Standards and Growth
  18. Introduction Overview Longitudinal Analysis of Student Data Value-Added Analysis Standards Based Growth Teacher/School Comparisons Big Picture Value-Added by Whom/What? Usually the value-added inference is to the teacher or school. D. W. Betebenner Performance Standards and Growth
  19. Introduction Overview Longitudinal Analysis of Student Data Value-Added Analysis Standards Based Growth Teacher/School Comparisons Big Picture Value-Added by Whom/What? Usually the value-added inference is to the teacher or school. The term teacher effect implies that teacher actions are responsible for a given effect. That is, that the teacher caused the effect. D. W. Betebenner Performance Standards and Growth
  20. Introduction Overview Longitudinal Analysis of Student Data Value-Added Analysis Standards Based Growth Teacher/School Comparisons Big Picture Value-Added by Whom/What? Usually the value-added inference is to the teacher or school. The term teacher effect implies that teacher actions are responsible for a given effect. That is, that the teacher caused the effect. Determination of cause usually requires random assignment. For example, in determining whether a particular curriculum is effective, the gold standard employs random assignment of subjects to treatment in the form of an experimental design. D. W. Betebenner Performance Standards and Growth
  21. Introduction Overview Longitudinal Analysis of Student Data Value-Added Analysis Standards Based Growth Teacher/School Comparisons Big Picture Value-Added by Whom/What? Usually the value-added inference is to the teacher or school. The term teacher effect implies that teacher actions are responsible for a given effect. That is, that the teacher caused the effect. Determination of cause usually requires random assignment. For example, in determining whether a particular curriculum is effective, the gold standard employs random assignment of subjects to treatment in the form of an experimental design. There is a great deal of debate as to whether value-added analyses using observational data can accurately estimate the contribution by whom/what [Rubin et al., 2004]. D. W. Betebenner Performance Standards and Growth
  22. Introduction Overview Longitudinal Analysis of Student Data Value-Added Analysis Standards Based Growth Teacher/School Comparisons Big Picture Value-Added Relative to What? Even if one can validly attribute responsibility for a measured effect to a teacher/school, just what does the effect mean? D. W. Betebenner Performance Standards and Growth
  23. Introduction Overview Longitudinal Analysis of Student Data Value-Added Analysis Standards Based Growth Teacher/School Comparisons Big Picture Value-Added Relative to What? Even if one can validly attribute responsibility for a measured effect to a teacher/school, just what does the effect mean? Value-added usually denotes residual performance above (or below) a statistically determined expectation. D. W. Betebenner Performance Standards and Growth
  24. Introduction Overview Longitudinal Analysis of Student Data Value-Added Analysis Standards Based Growth Teacher/School Comparisons Big Picture Value-Added Relative to What? Even if one can validly attribute responsibility for a measured effect to a teacher/school, just what does the effect mean? Value-added usually denotes residual performance above (or below) a statistically determined expectation. What does growth above expectation represent in terms of standards based performance? D. W. Betebenner Performance Standards and Growth
  25. Introduction Overview Longitudinal Analysis of Student Data Value-Added Analysis Standards Based Growth Teacher/School Comparisons Big Picture Value-Added Relative to What? Even if one can validly attribute responsibility for a measured effect to a teacher/school, just what does the effect mean? Value-added usually denotes residual performance above (or below) a statistically determined expectation. What does growth above expectation represent in terms of standards based performance? An answer to this question greatly enhances the interpretability and utility with respect to current standards based accountability initiatives. D. W. Betebenner Performance Standards and Growth
  26. Introduction Overview Longitudinal Analysis of Student Data Value-Added Analysis Standards Based Growth Teacher/School Comparisons Big Picture Teacher Comparisons Caterpillar Plot Mathematics Exam 40 Grade 7 Teachers (N = 60 ) ●● Residual BLUP District Mean ●●●● Rank of Residual D. W. Betebenner Performance Standards and Growth
  27. Introduction Overview Longitudinal Analysis of Student Data Value-Added Analysis Standards Based Growth Teacher/School Comparisons Big Picture Teacher Comparisons Caterpillar Plot Questions District Mean: Mathematics Exam Relation to AYP? Grade 7 Teachers (N = 60 ) Relation to Standards? Residual BLUP District Mean Zero Sum Game? Rank of Residual D. W. Betebenner Performance Standards and Growth
  28. Introduction Overview Longitudinal Analysis of Student Data Value-Added Analysis Standards Based Growth Teacher/School Comparisons Big Picture Value Added Relative to What? For a teacher adding value significantly above (or below) the district average, what can be expected of this teacher’s students in terms of performance standards? D. W. Betebenner Performance Standards and Growth
  29. Introduction Overview Longitudinal Analysis of Student Data Value-Added Analysis Standards Based Growth Teacher/School Comparisons Big Picture Value Added Relative to What? For a teacher adding value significantly above (or below) the district average, what can be expected of this teacher’s students in terms of performance standards? There is an incongruence between many presently used growth models and NCLB. NCLB achievement mandates are standards based. Growth models that rely upon scale scores return normative comparisons. D. W. Betebenner Performance Standards and Growth
  30. Introduction Overview Longitudinal Analysis of Student Data Value-Added Analysis Standards Based Growth Teacher/School Comparisons Big Picture Value Added Relative to What? For a teacher adding value significantly above (or below) the district average, what can be expected of this teacher’s students in terms of performance standards? There is an incongruence between many presently used growth models and NCLB. NCLB achievement mandates are standards based. Growth models that rely upon scale scores return normative comparisons. If normative results for students are inadequate, why should normative comparisons of teachers be any different? D. W. Betebenner Performance Standards and Growth
  31. Introduction Overview Longitudinal Analysis of Student Data Value-Added Analysis Standards Based Growth Teacher/School Comparisons Big Picture Which Growth Metric Scale Scores or Performance Standards Scale Score Based Growth Conceptually simple interval scale based results. Scale score based results require translation to standards based criteria. Difficult to define “how much growth” is enough without normative references. D. W. Betebenner Performance Standards and Growth
  32. Introduction Overview Longitudinal Analysis of Student Data Value-Added Analysis Standards Based Growth Teacher/School Comparisons Big Picture Which Growth Metric Scale Scores or Performance Standards Scale Score Based Growth Conceptually simple interval scale based results. Scale score based results require translation to standards based criteria. Difficult to define “how much growth” is enough without normative references. Standards Based Growth Conceptually simple probability based results. Amenable to standards based accountability initiatives. Simple to define “how much growth” with regard to NCLB-like policy mandates. D. W. Betebenner Performance Standards and Growth
  33. Introduction Transitions and their Probabilities Longitudinal Analysis of Student Data A Simple Illustration Standards Based Growth Benefits Big Picture Value-Added Standards Based Growth Change (growth) is modeled via student movement between performance categories. Student movement can refer to annual change. Student movement can refer to long term change. For example, between the beginning and end of middle school D. W. Betebenner Performance Standards and Growth
  34. Introduction Transitions and their Probabilities Longitudinal Analysis of Student Data A Simple Illustration Standards Based Growth Benefits Big Picture Value-Added Standards Based Growth Change (growth) is modeled via student movement between performance categories. Student movement can refer to annual change. Student movement can refer to long term change. For example, between the beginning and end of middle school Growth estimates are quantified as transition probabilities. That is, the probability of observing a transition between performance categories. D. W. Betebenner Performance Standards and Growth
  35. Introduction Transitions and their Probabilities Longitudinal Analysis of Student Data A Simple Illustration Standards Based Growth Benefits Big Picture Value-Added Standards Based Growth Performance Level Transitions Diagram Performance Level Transitions PL4 PL3 PL2 PL1 D. W. Betebenner Performance Standards and Growth
  36. Introduction Transitions and their Probabilities Longitudinal Analysis of Student Data A Simple Illustration Standards Based Growth Benefits Big Picture Value-Added Standards Based Growth Performance Level Transitions Diagram Performance Level Transitions Most students maintain their PL4 performance level from year to year. PL3 PL2 PL1 D. W. Betebenner Performance Standards and Growth
  37. Introduction Transitions and their Probabilities Longitudinal Analysis of Student Data A Simple Illustration Standards Based Growth Benefits Big Picture Value-Added Standards Based Growth Performance Level Transitions Diagram Performance Level Transitions Most students maintain their PL4 performance level from year to year. Some students move up in performance level from year to year. PL3 PL2 PL1 D. W. Betebenner Performance Standards and Growth
  38. Introduction Transitions and their Probabilities Longitudinal Analysis of Student Data A Simple Illustration Standards Based Growth Benefits Big Picture Value-Added Standards Based Growth Performance Level Transitions Diagram Performance Level Transitions Most students maintain their PL4 performance level from year to year. Some students move up in performance level from year to year. PL3 Some students move down in performance level from year to year. PL2 PL1 D. W. Betebenner Performance Standards and Growth
  39. Introduction Transitions and their Probabilities Longitudinal Analysis of Student Data A Simple Illustration Standards Based Growth Benefits Big Picture Value-Added Standards Based Growth Performance Level Transitions Diagram Performance Level Transitions Most students maintain their PL4 performance level from year to year. Some students move up in performance level from year to year. PL3 Some students move down in performance level from year to year. Rarely do students jump one or PL2 more performance levels. PL1 D. W. Betebenner Performance Standards and Growth
  40. Introduction Transitions and their Probabilities Longitudinal Analysis of Student Data A Simple Illustration Standards Based Growth Benefits Big Picture Value-Added Standards Based Growth Transition Probabilities Diagram Transition Probabilities PL4 PL3 PL2 PL1 D. W. Betebenner Performance Standards and Growth
  41. Introduction Transitions and their Probabilities Longitudinal Analysis of Student Data A Simple Illustration Standards Based Growth Benefits Big Picture Value-Added Standards Based Growth Transition Probabilities Diagram Transition Probabilities For example, there might be a 70% .7 PL4 chance for a student to remain at the highest performance level. PL3 PL2 PL1 D. W. Betebenner Performance Standards and Growth
  42. Introduction Transitions and their Probabilities Longitudinal Analysis of Student Data A Simple Illustration Standards Based Growth Benefits Big Picture Value-Added Standards Based Growth Transition Probabilities Diagram Transition Probabilities For example, there might be a 70% .7 PL4 chance for a student to remain at the highest performance level. Similarly, there might be a 50% PL3 chance of moving from the lowest to the next to lowest category. PL2 PL1 D. W. Betebenner Performance Standards and Growth
  43. Introduction Transitions and their Probabilities Longitudinal Analysis of Student Data A Simple Illustration Standards Based Growth Benefits Big Picture Value-Added Standards Based Growth Transition Probabilities Diagram Transition Probabilities For example, there might be a 70% .7 PL4 chance for a student to remain at the .3 highest performance level. Similarly, there might be a 50% PL3 chance of moving from the lowest to the next to lowest category. These conditions impose constraints PL2 upon two other performance level transitions. .5 PL1 D. W. Betebenner Performance Standards and Growth
  44. Introduction Transitions and their Probabilities Longitudinal Analysis of Student Data A Simple Illustration Standards Based Growth Benefits Big Picture Value-Added Transition Probabilities An Illustration Diagram Transition Matrix PL4 Time 2 PL1 PL2 PL3 PL4 PL1 PL2   PL3 Time 1   PL3   PL4 PL2 PL1 D. W. Betebenner Performance Standards and Growth
  45. Introduction Transitions and their Probabilities Longitudinal Analysis of Student Data A Simple Illustration Standards Based Growth Benefits Big Picture Value-Added Transition Probabilities An Illustration Diagram Transition Matrix .64 PL4 Time 2 PL1 PL2 PL3 PL4 PL1 .66 PL2  .59  .82 PL3 Time 1   PL3  .82  PL4 .64 .59 PL2 .66 PL1 D. W. Betebenner Performance Standards and Growth
  46. Introduction Transitions and their Probabilities Longitudinal Analysis of Student Data A Simple Illustration Standards Based Growth Benefits Big Picture Value-Added Transition Probabilities An Illustration Diagram Transition Matrix .64 PL4 Time 2 PL1 PL2 PL3 PL4 PL1 .66 .34 PL2  .59 .31  .82 PL3 Time 1   PL3  .82 .09  .31 PL4 .64 .59 PL2 .66 PL1 D. W. Betebenner Performance Standards and Growth
  47. Introduction Transitions and their Probabilities Longitudinal Analysis of Student Data A Simple Illustration Standards Based Growth Benefits Big Picture Value-Added Transition Probabilities An Illustration Diagram Transition Matrix .64 PL4 Time 2 PL1 PL2 PL3 PL4 PL1 .66 .34 PL2  .82 PL3 Time 1  PL3  .09 .82 .09  .31 .09 PL4 .36 .64 .59 PL2 .66 PL1 D. W. Betebenner Performance Standards and Growth
  48. Introduction Transitions and their Probabilities Longitudinal Analysis of Student Data A Simple Illustration Standards Based Growth Benefits Big Picture Value-Added Transition Probabilities An Illustration Diagram Transition Matrix .64 PL4 Time 2 PL1 PL2 PL3 PL4 PL1 .66 .34 .00 .00 PL2  .82 PL3 Time 1  PL3  .00 .09 .82 .09  .31 .09 PL4 .00 .00 .36 .64 .59 PL2 .66 PL1 D. W. Betebenner Performance Standards and Growth
  49. Introduction Transitions and their Probabilities Longitudinal Analysis of Student Data A Simple Illustration Standards Based Growth Benefits Big Picture Value-Added Transition Probabilities An Illustration Transition Matrix Multiplication Current Transition Future Status Matrix Status D. W. Betebenner Performance Standards and Growth
  50. Introduction Transitions and their Probabilities Longitudinal Analysis of Student Data A Simple Illustration Standards Based Growth Benefits Big Picture Value-Added Transition Probabilities An Illustration Transition Matrix Multiplication Current Transition Future Status Matrix Status PL1 .10 PL2  PL3  .50  PL4 .20 D. W. Betebenner Performance Standards and Growth
  51. Introduction Transitions and their Probabilities Longitudinal Analysis of Student Data A Simple Illustration Standards Based Growth Benefits Big Picture Value-Added Transition Probabilities An Illustration Transition Matrix Multiplication Current Transition Future Status Matrix Status PL1 .10 .66 .34 .00 .00 PL2  PL3  .50   .00 .09 .82 .09  PL4 .20 .00 .00 .36 .64 D. W. Betebenner Performance Standards and Growth
  52. Introduction Transitions and their Probabilities Longitudinal Analysis of Student Data A Simple Illustration Standards Based Growth Benefits Big Picture Value-Added Transition Probabilities An Illustration Transition Matrix Multiplication Current Transition Future Status Matrix Status PL1 .10 .66 .34 .00 .00 PL2  .20   .10 .59 .31 .00   .179  PL3  .50   .00 .09 .82 .09   .452  PL4 .20 .00 .00 .36 .64 .283 D. W. Betebenner Performance Standards and Growth
  53. Introduction Transitions and their Probabilities Longitudinal Analysis of Student Data A Simple Illustration Standards Based Growth Benefits Big Picture Value-Added Benefits Growth, denoted by transition probabilities in the transition matrix, readily connects changes in percentages of students at different performance standards over time. D. W. Betebenner Performance Standards and Growth
  54. Introduction Transitions and their Probabilities Longitudinal Analysis of Student Data A Simple Illustration Standards Based Growth Benefits Big Picture Value-Added Benefits Growth, denoted by transition probabilities in the transition matrix, readily connects changes in percentages of students at different performance standards over time. Transition probabilities are consonant with standards based instruction, assessment and accountability initiatives. Vertically articulated performance standards make transitions and their associated probabilities that much more instructionally meaningful. D. W. Betebenner Performance Standards and Growth
  55. Introduction Transitions and their Probabilities Longitudinal Analysis of Student Data A Simple Illustration Standards Based Growth Benefits Big Picture Value-Added Benefits Growth, denoted by transition probabilities in the transition matrix, readily connects changes in percentages of students at different performance standards over time. Transition probabilities are consonant with standards based instruction, assessment and accountability initiatives. Vertically articulated performance standards make transitions and their associated probabilities that much more instructionally meaningful. Transition probabilities, being statements of probability, are easy to understand by stakeholders: Teachers, parents, administrators. D. W. Betebenner Performance Standards and Growth
  56. Introduction Transitions and their Probabilities Longitudinal Analysis of Student Data A Simple Illustration Standards Based Growth Benefits Big Picture Value-Added Standards Based Growth Value-Added? Again, we are confronted with two essential questions: Value-added by whom/what? Value-added relative to what? D. W. Betebenner Performance Standards and Growth
  57. Introduction Transitions and their Probabilities Longitudinal Analysis of Student Data A Simple Illustration Standards Based Growth Benefits Big Picture Value-Added Standards Based Growth Value-Added? Again, we are confronted with two essential questions: Value-added by whom/what? Value-added relative to what? Value-added by whom/what D. W. Betebenner Performance Standards and Growth
  58. Introduction Transitions and their Probabilities Longitudinal Analysis of Student Data A Simple Illustration Standards Based Growth Benefits Big Picture Value-Added Standards Based Growth Value-Added? Again, we are confronted with two essential questions: Value-added by whom/what? Value-added relative to what? Value-added by whom/what Even with transition probabilities, observational data makes causal inferences tenuous. D. W. Betebenner Performance Standards and Growth
  59. Introduction Transitions and their Probabilities Longitudinal Analysis of Student Data A Simple Illustration Standards Based Growth Benefits Big Picture Value-Added Standards Based Growth Value-Added? Again, we are confronted with two essential questions: Value-added by whom/what? Value-added relative to what? Value-added by whom/what Even with transition probabilities, observational data makes causal inferences tenuous. That is not to suggest that such measures are not relevant even if determination of cause/responsibility is not absolute. D. W. Betebenner Performance Standards and Growth
  60. Introduction Transitions and their Probabilities Longitudinal Analysis of Student Data A Simple Illustration Standards Based Growth Benefits Big Picture Value-Added Standards Based Growth Value-Added? Value-added relative to what? D. W. Betebenner Performance Standards and Growth
  61. Introduction Transitions and their Probabilities Longitudinal Analysis of Student Data A Simple Illustration Standards Based Growth Benefits Big Picture Value-Added Standards Based Growth Value-Added? Value-added relative to what? If policy dictates the future status of the system, then growth should be defined relative to those mandates. D. W. Betebenner Performance Standards and Growth
  62. Introduction Transitions and their Probabilities Longitudinal Analysis of Student Data A Simple Illustration Standards Based Growth Benefits Big Picture Value-Added Standards Based Growth Value-Added? Value-added relative to what? If policy dictates the future status of the system, then growth should be defined relative to those mandates. This level of growth relates to stakeholders how current status turns into future status. D. W. Betebenner Performance Standards and Growth
  63. Introduction Transitions and their Probabilities Longitudinal Analysis of Student Data A Simple Illustration Standards Based Growth Benefits Big Picture Value-Added Standards Based Growth Value-Added? Value-added relative to what? If policy dictates the future status of the system, then growth should be defined relative to those mandates. This level of growth relates to stakeholders how current status turns into future status. If the rate of growth is realistic, then this can be used as the growth standard for schools, teachers, etc. D. W. Betebenner Performance Standards and Growth
  64. Introduction Transitions and their Probabilities Longitudinal Analysis of Student Data A Simple Illustration Standards Based Growth Benefits Big Picture Value-Added Standards Based Growth Value-Added? Defining Adequate Progress Current Future Status × System Status (Reality) Growth (Policy) D. W. Betebenner Performance Standards and Growth
  65. Introduction Transitions and their Probabilities Longitudinal Analysis of Student Data A Simple Illustration Standards Based Growth Benefits Big Picture Value-Added Standards Based Growth Value-Added? Defining Adequate Progress Current Future Status × System Status (Reality) Growth (Policy) D. W. Betebenner Performance Standards and Growth
  66. Introduction Transitions and their Probabilities Longitudinal Analysis of Student Data A Simple Illustration Standards Based Growth Benefits Big Picture Value-Added Standards Based Growth Value-Added? Defining Adequate Progress Current Future Status × System Status (Reality) Growth (Policy) PL1 .10 PL2   .05  PL3  .50  PL4 .20 .30 D. W. Betebenner Performance Standards and Growth
  67. Introduction Transitions and their Probabilities Longitudinal Analysis of Student Data A Simple Illustration Standards Based Growth Benefits Big Picture Value-Added Standards Based Growth Value-Added? Defining Adequate Progress Current Future Status × System Status (Reality) Growth (Policy) PL1 .10 PL2   ? ? ? ?   .05  PL3  .50  ? ? ?   .60  PL4 .20 ? ? ? ? .30 D. W. Betebenner Performance Standards and Growth
  68. Introduction Transitions and their Probabilities Longitudinal Analysis of Student Data A Simple Illustration Standards Based Growth Benefits Big Picture Value-Added Standards Based Growth Value-Added? Defining Adequate Progress: Good School 1 Year Current Future Status × System Status (Reality) Growth (Policy) PL1 .10 .59 .41 .00 .00 PL2  .20   .06 .44 .50 .00   .15  PL3  .50   .00 .04 .89 .07   .60  PL4 .20 .00 .00 .27 .73 .18 D. W. Betebenner Performance Standards and Growth
  69. Introduction Transitions and their Probabilities Longitudinal Analysis of Student Data A Simple Illustration Standards Based Growth Benefits Big Picture Value-Added Standards Based Growth Value-Added? Defining Adequate Progress: Good School 2 Years Current Future Status × System Status (Reality) Growth (Policy) PL1 PL2  .20   .06 .44 .50 .00   .12  PL3  .50   .00 .04 .89 .07   .66  PL4 .20 .00 .00 .27 .73 .17 D. W. Betebenner Performance Standards and Growth
  70. Introduction Transitions and their Probabilities Longitudinal Analysis of Student Data A Simple Illustration Standards Based Growth Benefits Big Picture Value-Added Standards Based Growth Value-Added? Defining Adequate Progress: Good School 3 Years Current Future Status × System Status (Reality) Growth (Policy) PL1 PL2  .20   .06 .44 .50 .00   .10  PL3  .50   .00 .04 .89 .07   .69  PL4 .20 .00 .00 .27 .73 .17 D. W. Betebenner Performance Standards and Growth
  71. Introduction Transitions and their Probabilities Longitudinal Analysis of Student Data A Simple Illustration Standards Based Growth Benefits Big Picture Value-Added Standards Based Growth Value-Added? Defining Adequate Progress: Good School 4 Years Current Future Status × System Status (Reality) Growth (Policy) PL1 PL2  .20   .06 .44 .50 .00   .09  PL3  .50   .00 .04 .89 .07   .71  PL4 .20 .00 .00 .27 .73 .17 D. W. Betebenner Performance Standards and Growth
  72. Introduction Transitions and their Probabilities Longitudinal Analysis of Student Data A Simple Illustration Standards Based Growth Benefits Big Picture Value-Added Standards Based Growth Value-Added? Defining Adequate Progress: Good School 5 Years Current Future Status × System Status (Reality) Growth (Policy) PL1 PL2  .20   .06 .44 .50 .00   .08  PL3  .50   .00 .04 .89 .07   .72  PL4 .20 .00 .00 .27 .73 .18 D. W. Betebenner Performance Standards and Growth
  73. Introduction Transitions and their Probabilities Longitudinal Analysis of Student Data A Simple Illustration Standards Based Growth Benefits Big Picture Value-Added Standards Based Growth Growth Targets? Providing Growth Targets Observed Growth D. W. Betebenner Performance Standards and Growth
  74. Introduction Transitions and their Probabilities Longitudinal Analysis of Student Data A Simple Illustration Standards Based Growth Benefits Big Picture Value-Added Standards Based Growth Growth Targets? Providing Growth Targets Observed Target Growth Growth D. W. Betebenner Performance Standards and Growth
  75. Introduction Transitions and their Probabilities Longitudinal Analysis of Student Data A Simple Illustration Standards Based Growth Benefits Big Picture Value-Added Standards Based Growth Growth Targets? Providing Growth Targets Observed Institutional Target Growth Improvement Growth D. W. Betebenner Performance Standards and Growth
  76. Introduction Summary Longitudinal Analysis of Student Data Goals Standards Based Growth References Big Picture Summary When measuring growth, the choice of metric impacts the results of the analyses and their interpretation. D. W. Betebenner Performance Standards and Growth
  77. Introduction Summary Longitudinal Analysis of Student Data Goals Standards Based Growth References Big Picture Summary When measuring growth, the choice of metric impacts the results of the analyses and their interpretation. Scale Scores Growth analyses using scale scores often yield results in that metric. The abstract nature of the scale requires interpretation of results to be based in normative comparisons. Results are not congruent with many standards based accountability initiatives. D. W. Betebenner Performance Standards and Growth
  78. Introduction Summary Longitudinal Analysis of Student Data Goals Standards Based Growth References Big Picture Summary When measuring growth, the choice of metric impacts the results of the analyses and their interpretation. Performance Standards Growth analyses using performance standards yield results in that metric. Transitions between performance categories are interpretable without recourse to normative comparisons. Results are easily mapped onto current standards based accountability initiatives. D. W. Betebenner Performance Standards and Growth
  79. Introduction Summary Longitudinal Analysis of Student Data Goals Standards Based Growth References Big Picture Goals John Tukey It is better to have an approximate answer to the right question than a precise answer to the wrong question. D. W. Betebenner Performance Standards and Growth
  80. Introduction Summary Longitudinal Analysis of Student Data Goals Standards Based Growth References Big Picture Goals John Tukey It is better to have an approximate answer to the right question than a precise answer to the wrong question. NCLB has defined the question: How do we get all students to the proficient level or better? D. W. Betebenner Performance Standards and Growth
  81. Introduction Summary Longitudinal Analysis of Student Data Goals Standards Based Growth References Big Picture Goals John Tukey It is better to have an approximate answer to the right question than a precise answer to the wrong question. NCLB has defined the question: How do we get all students to the proficient level or better? The question assumes the metric of performance standards. The natural way to analyze the question is to look at movement between performance levels. D. W. Betebenner Performance Standards and Growth
  82. Introduction Summary Longitudinal Analysis of Student Data Goals Standards Based Growth References Big Picture Goals John Tukey It is better to have an approximate answer to the right question than a precise answer to the wrong question. NCLB has defined the question: How do we get all students to the proficient level or better? The question assumes the metric of performance standards. The natural way to analyze the question is to look at movement between performance levels. Transition probabilities locate growth within NCLB mandates and map directly back to performance standards around which curriculum and instruction are based. D. W. Betebenner Performance Standards and Growth
  83. Introduction Summary Longitudinal Analysis of Student Data Goals Standards Based Growth References Big Picture References Hill, R. (2003). Using longitudinal designs with NCLB. Technical report, Center for Assessment, Dover, NH. Kingsbury, G. G., Olson, A., McCahon, D., and McCall, M. Adequate yearly progress using the Hybrid Success Model: A suggested improvement to No Child Left Behind. Technical report, Northwest Evaluation Association, Porland, OR. Linn, R. L. (2004). Rethinking the No Child Left Behind accountability system. Paper prepared for a forum on No Child Left Behind sponsored by the Center on Education Policy, Washington DC, July 28, Rubin, D. B., Stuart, E. A., and Zanutto, E. L. (2004). A potential outcomes view of value-added assessment in education. Journal of Educational and Behavioral Statistics, Thum, Y. M. (2003). No Child Left Behind: Methodological challenges and recommendations for measuring adequate yearly progress. Technical report, National Center for Research on Evaluation, Standards, and Student Testing, Los Angeles. CSE Technical Report 590. D. W. Betebenner Performance Standards and Growth