HAWAII STUDENT GROWTH ANALYSES 2018-2019 & BEYOND Damian W Betebenner & Adam VanIwaarden National Center for the Improvement of Educational Assessment Honolulu, Hawaii August 15th, 2019

TOPICS FOR DISCUSSION • About Us • What is student growth? A quick overview. • A brief history of student growth analyses in Hawaii. • SGP analyses associated with the 2018-2019 academic year.

ABOUT US • Damian Betebenner • Joined Center for Assessment in 2007 • Co-Developed SGPs and the SGP package with Adam VanIwaarden • Adam VanIwaarden • Intern at Center for Assessment in 2010 • Working full time at Center as part of SGP development and implementation

Student Growth: Introductory Concepts • The concept of student growth isn’t as difficult to understand and popular perception suggests. • Calculations may be complicated but knowing how to calculate student growth is not synonymous with understanding student growth: – The number π (pi=3.14159…) is difficult to calculate but is simple to understand conceptually. – Height and weight percentiles associated with children are difficult to calculate but are simple to understand conceptually. • Computational details are important but often obscure a simple conceptual basis.

Student Growth Introductory Concepts A Math Story Problem • Whose high jump changed more? Explain your answer. • Whose high jump change is more remarkable? Explain your answer. • Which high jumper improved more? Explain your answer.

Student Growth: Introductory Concepts • There are 3 general approaches of calculating student growth: – Growth as an amount of change (i.e., gain). – Growth as a norm/peer referenced quantity. – Growth as a criterion referenced quantity.

Student Growth: Introductory Concepts • Growth as an amount of change: – What could be more simple? – Like height or weight just calculate the amount of change. – Requires a vertical scale (which Arizona now has) – Growth = Score Time 2 - Score Time 1 • Not as simple as it seems: – Comparisons between students often leads to wrong conclusions – Interpreting change requires context.

Student Growth: Introductory Concepts Back to the story problem!

Student Growth: Introductory Concepts • Which high jumper improved more: – Judy improved 2 inches. – Anna improved 6 inches. – An inch is an inch!!! – But is an inch really an inch??? – 2 inches for an elite jumper is far more remarkable than 6 inches for a novice.

Student Growth: Introductory Concepts • The same holds true for the vertically scaled tests (e.g., SBAC). Average Scale Score Change 2017 to 2018 Top 10% ELA Grade 4 2017 +8.76 Bottom 10% ELA Grade 4 2017 +31.5 Top 10% Mathematic Grade 4 2017 +16.7 Bottom 10% Mathematics Grade 4 2017 +44.5 • Just as an inch isn’t necessarily an inch; a scale score point isn’t necessarily a scale score point.

Student Growth: Introductory Concepts •This is analogous to height/weight measures for infants. •Growth amounts are tricky to interpret and require context. •Peer based comparisons used by growth analyses are a way to acquire that context. An infant male toddler is measured at 2 and 3 years of age and is shown to have grown 4 inches. The magnitude of increase—4 inches—is a well understood quantity that any parent can grasp and calculate at home using a simple yardstick. However, parents leaving their pediatrician’s office knowing only how much their child has grown would likely be wanting for more information: Parents are not interested in an absolute magnitude of growth, but instead in a norm- referenced quantity locating that 4 inch increase alongside the height increases of similar children.

Student Growth: Introductory Concepts • Peer-based analyses are also called norm-referenced analyses. • Like with infant height/weight charts, peers are children/ student with similar characteristics. • For growth analyses using large scale assessments, peers are students with (at least) the same scale score history on the same tests. • Peer based norms are invaluable in providing context for stakeholders to understand student growth. • The Hawaii SGP Model creates growth norms for all students using regression (i.e., peer-based) analyses. • These norms allow stakeholders to understand what an ambitious yet reasonable expectation is for each student.

Student Growth Percentiles • Research on the SGP model began in 2007. • Based upon work in Colorado, the norm-referenced model was extended to include a criterion-referenced component. • Work in other states incorporating the model into state reporting and accountability systems occurred in subsequent years and student growth became more common across state assessment systems. • Goal or the model is to create an individual level growth metric that is informative to different stakeholders ranging from parents to policy makers.

HISTORY OF STUDENT GROWTH IN HAWAII • Begun in 2012 as part of Hawaii’s successful RttT grant. • Originally focused on teacher evaluation. • Included use of interactive visualizations to help communicate SGP results. • Included the calculation of student growth percentiles and student growth projections (not sure whether SGTs were used). • Used (and not used) in different ways over the last decade.

CALCULATION OF STUDENT GROWTH PERCENTILES • Student Growth Percentiles and Projections are calculated using the SGP package (2019) which utilized the R statistical software environment. • Both the SGP package and R are open source (free!) software. • All source code for the SGP package as well as the Hawaii SGP analyses are open source and posted on the Center for Assessment Github site: • https://github.com/CenterForAssessment • https://github.com/CenterForAssessment/SGP • https://github.com/CenterForAssessment/Hawaii • Details about the R software environment and SGP package are available in an accompanying presentation.

CALCULATION OF STUDENT GROWTH PERCENTILES • Calculation of SGPs is conducted in two broad steps: • Data Preparation: • Raw student level assessment data and raw student-teacher roster data are provided by Hawaii DOE. These data are modestly cleaned and prepared for SGP analysis. • Data Analysis: • One data is prepared, the SGP package and its functions are used to conduct all SGP analyses

CALCULATION OF STUDENT GROWTH PERCENTILES • Data preparation: • Student level data preparation: • The R script used to create the nal student level data for SGP analysis is available on the Hawaii SGP Github website: https://github.com/ CenterForAssessment/Hawaii/blob/master/ Hawaii_Data_LONG_2019.R • Student-teacher roster data preparation: • The R script used to create the nal student-teacher roster lookup le is available on the Hawaii SGP Github website: https://github.com/ CenterForAssessment/Hawaii/blob/master/ Hawaii_Data_LONG_2019_INSTRUCTOR_NUMBER.R fi fi fi

CALCULATION OF STUDENT GROWTH PERCENTILES • Data analysis: • The R script used to run the entire SGP is available on the Hawaii SGP Github website: https://github.com/CenterForAssessment/Hawaii/blob/ master/Hawaii_SGP_2019.R • The SGP R script includes a number of internal steps creating SGPs, AGPs (adequate growth percentiles), school level summaries, instructor level summaries, and outputting the results. • Previously, individual student reports were generated for the Hawaii DOE as well using the SGP package (see accompanying sample reports).

CALCULATION OF STUDENT GROWTH PERCENTILES • Data validation and transfer • SGP analyses produce numerous goodness-of- t plots used to investigate whether SGP analyses produce valid results. • Output data are shared with Hawaii DOE via the Center for Assessment’s secure ShareFile website. fi

About this copy

Hawaii Student Growth Analyses — Hawaii Department of Education, August 15, 2019.

This is an image-based copy of the original Keynote slides; builds and animations are flattened.