Hawaii Student Growth Analyses

Hawaii Department of Education · August 15, 2019

Hawaii Student Growth Analyses — Hawaii Department of Education, August 15, 2019.
Published

August 15, 2019

Event
State agency
Slides
20
Format
Converted from Keynote

This deck is an image-based copy of the original slides. Builds and animations are flattened; the slide text is reproduced below for search and screen readers.

Slide text
  1. HAWAII STUDENT GROWTH ANALYSES 2018-2019 & BEYOND Damian W Betebenner & Adam VanIwaarden National Center for the Improvement of Educational Assessment Honolulu, Hawaii August 15th, 2019
  2. TOPICS FOR DISCUSSION • About Us • What is student growth? A quick overview. • A brief history of student growth analyses in Hawaii. • SGP analyses associated with the 2018-2019 academic year.
  3. ABOUT US • Damian Betebenner • Joined Center for Assessment in 2007 • Co-Developed SGPs and the SGP package with Adam VanIwaarden • Adam VanIwaarden • Intern at Center for Assessment in 2010 • Working full time at Center as part of SGP development and implementation
  4. Student Growth: Introductory Concepts • The concept of student growth isn’t as difficult to understand and popular perception suggests. • Calculations may be complicated but knowing how to calculate student growth is not synonymous with understanding student growth: – The number π (pi=3.14159…) is difficult to calculate but is simple to understand conceptually. – Height and weight percentiles associated with children are difficult to calculate but are simple to understand conceptually. • Computational details are important but often obscure a simple conceptual basis.
  5. Student Growth Introductory Concepts A Math Story Problem • Whose high jump changed more? Explain your answer. • Whose high jump change is more remarkable? Explain your answer. • Which high jumper improved more? Explain your answer.
  6. Student Growth: Introductory Concepts • There are 3 general approaches of calculating student growth: – Growth as an amount of change (i.e., gain). – Growth as a norm/peer referenced quantity. – Growth as a criterion referenced quantity.
  7. Student Growth: Introductory Concepts • Growth as an amount of change: – What could be more simple? – Like height or weight just calculate the amount of change. – Requires a vertical scale (which Arizona now has) – Growth = Score Time 2 - Score Time 1 • Not as simple as it seems: – Comparisons between students often leads to wrong conclusions – Interpreting change requires context.
  8. Student Growth: Introductory Concepts Back to the story problem!
  9. Student Growth: Introductory Concepts • Which high jumper improved more: – Judy improved 2 inches. – Anna improved 6 inches. – An inch is an inch!!! – But is an inch really an inch??? – 2 inches for an elite jumper is far more remarkable than 6 inches for a novice.
  10. Student Growth: Introductory Concepts • The same holds true for the vertically scaled tests (e.g., SBAC). Average Scale Score Change 2017 to 2018 Top 10% ELA Grade 4 2017 +8.76 Bottom 10% ELA Grade 4 2017 +31.5 Top 10% Mathematic Grade 4 2017 +16.7 Bottom 10% Mathematics Grade 4 2017 +44.5 • Just as an inch isn’t necessarily an inch; a scale score point isn’t necessarily a scale score point.
  11. Student Growth: Introductory Concepts •This is analogous to height/weight measures for infants. •Growth amounts are tricky to interpret and require context. •Peer based comparisons used by growth analyses are a way to acquire that context. An infant male toddler is measured at 2 and 3 years of age and is shown to have grown 4 inches. The magnitude of increase—4 inches—is a well understood quantity that any parent can grasp and calculate at home using a simple yardstick. However, parents leaving their pediatrician’s office knowing only how much their child has grown would likely be wanting for more information: Parents are not interested in an absolute magnitude of growth, but instead in a norm- referenced quantity locating that 4 inch increase alongside the height increases of similar children.
  12. Student Growth: Introductory Concepts • Peer-based analyses are also called norm-referenced analyses. • Like with infant height/weight charts, peers are children/ student with similar characteristics. • For growth analyses using large scale assessments, peers are students with (at least) the same scale score history on the same tests. • Peer based norms are invaluable in providing context for stakeholders to understand student growth. • The Hawaii SGP Model creates growth norms for all students using regression (i.e., peer-based) analyses. • These norms allow stakeholders to understand what an ambitious yet reasonable expectation is for each student.
  13. Student Growth Percentiles • Research on the SGP model began in 2007. • Based upon work in Colorado, the norm-referenced model was extended to include a criterion-referenced component. • Work in other states incorporating the model into state reporting and accountability systems occurred in subsequent years and student growth became more common across state assessment systems. • Goal or the model is to create an individual level growth metric that is informative to different stakeholders ranging from parents to policy makers.
  14. HISTORY OF STUDENT GROWTH IN HAWAII • Begun in 2012 as part of Hawaii’s successful RttT grant. • Originally focused on teacher evaluation. • Included use of interactive visualizations to help communicate SGP results. • Included the calculation of student growth percentiles and student growth projections (not sure whether SGTs were used). • Used (and not used) in different ways over the last decade.
  15. CALCULATION OF STUDENT GROWTH PERCENTILES • Student Growth Percentiles and Projections are calculated using the SGP package (2019) which utilized the R statistical software environment. • Both the SGP package and R are open source (free!) software. • All source code for the SGP package as well as the Hawaii SGP analyses are open source and posted on the Center for Assessment Github site: • https://github.com/CenterForAssessment • https://github.com/CenterForAssessment/SGP • https://github.com/CenterForAssessment/Hawaii • Details about the R software environment and SGP package are available in an accompanying presentation.
  16. CALCULATION OF STUDENT GROWTH PERCENTILES • Calculation of SGPs is conducted in two broad steps: • Data Preparation: • Raw student level assessment data and raw student-teacher roster data are provided by Hawaii DOE. These data are modestly cleaned and prepared for SGP analysis. • Data Analysis: • One data is prepared, the SGP package and its functions are used to conduct all SGP analyses
  17. CALCULATION OF STUDENT GROWTH PERCENTILES • Data preparation: • Student level data preparation: • The R script used to create the nal student level data for SGP analysis is available on the Hawaii SGP Github website: https://github.com/ CenterForAssessment/Hawaii/blob/master/ Hawaii_Data_LONG_2019.R • Student-teacher roster data preparation: • The R script used to create the nal student-teacher roster lookup le is available on the Hawaii SGP Github website: https://github.com/ CenterForAssessment/Hawaii/blob/master/ Hawaii_Data_LONG_2019_INSTRUCTOR_NUMBER.R fi fi fi
  18. CALCULATION OF STUDENT GROWTH PERCENTILES • Data analysis: • The R script used to run the entire SGP is available on the Hawaii SGP Github website: https://github.com/CenterForAssessment/Hawaii/blob/ master/Hawaii_SGP_2019.R • The SGP R script includes a number of internal steps creating SGPs, AGPs (adequate growth percentiles), school level summaries, instructor level summaries, and outputting the results. • Previously, individual student reports were generated for the Hawaii DOE as well using the SGP package (see accompanying sample reports).
  19. CALCULATION OF STUDENT GROWTH PERCENTILES • Data validation and transfer • SGP analyses produce numerous goodness-of- t plots used to investigate whether SGP analyses produce valid results. • Output data are shared with Hawaii DOE via the Center for Assessment’s secure ShareFile website. fi