Talks archive, 2010–2014 – An Overview of Student Growth Percentile Methodology: Massachusetts TAC

Intro References An Overview of Student Growth Percentile Methodology Damian W. Betebenner National Center for the Improvement of Educational Assessment Dover, NH Massachusetts TAC Cambridge, Massachusetts October 25th, 2012 Damian W. Betebenner Student Growth Percentile Methodology

Describing Student Growth Intro Student Growth Percentiles References Model & Software for Student Growth Percentiles Criterion Referenced Growth About Me Damian Betebenner - Senior Associate at the Center for Assessment (NCIEA). - Developed student growth percentiles and percentile growth trajectories to help states and educational associations employ student growth in decision making [Betebenner, 2008, Betebenner, 2009, Betebenner, 2012]. - Currently refining and sharing these techniques with other states including Colorado, Hawaii, Indiana, New York, Nevada, Rhode Island and approximately 25 other states in various stages of investigation/adoption. Damian W. Betebenner Student Growth Percentile Methodology

Describing Student Growth Intro Student Growth Percentiles References Model & Software for Student Growth Percentiles Criterion Referenced Growth Describing Student Growth Measuring student growth, even with a vertical scale, is not a simple task. Some believe a vertical scale simplifies the task of measuring student growth. Even with an interval (or ratio) scale, growth is not easy to interpret. Consider, for example, height. A child might grow 4 inches between ages 3 and 4. 4 inches is a well understood quantity. The 4 inch increase becomes really meaningful only when understood alongside the growth of other 3 to 4 year olds. Student growth percentiles were developed to provide a norm-referenced description of student growth (i.e., a growth norm). Damian W. Betebenner Student Growth Percentile Methodology

20 core ca ca le le 280 S re 2012 co

co re le ca ro th nt i le 12 ce er th 260 re co 240 le ca

co re th S ro le le ca nt er ce S th 260 re co 240 le ca

Peer 2 5 4 m i re ade Sco Ac cale 201 75% 75 th Per 200 nt i ce le G wth ro 20 core ca ca le le 280 S re 20 12 co

Th is d 75th s: dat erive perc eer 2 5 4 acaa and from ntile c P = dem co a cu ic p vers ll the t m i re eer all ade Sco g ro Ac cale ups 201 75% 75 th Per 200 nt i ce le G wth ro 20 core ca ca le le 280 S re 20 12 co

eer 2 2 6 cPe= Th m i r is d 75th ade Sco dat erive perc acaa and from ntile Ac cale dem co a cu ic p vers ll the t 1S eer all g ro ups th Per 200 nt i ce le G wth ro 20 core ca ca le le 280 S re 2012 co

Th is e dade 75 a c t a r iv t h a d a ed p emnd f er ic coro cmen peve atil errs ll e c gralltheut ou ps 75 th Pe 20 r ce 0 nt il e Gr ow th Sc 0 or al Sc Sc 0 0 al or 22 Sc

Th is e dade 75 a c t a r iv t h a d a ed p emnd f er ic coro cmen peve atil errs ll e c gralltheut ou ps % 7 5 th P 20 75 er ce 0 nt 0 il e Gr ow th Sc 0 or al Sc Sc 0 0 al or 22 Sc

Describing Student Growth Intro Student Growth Percentiles References Model & Software for Student Growth Percentiles Criterion Referenced Growth Student Growth Percentiles Should we be surprised with a child’s current achievement given their prior achievement? Given a student’s prior scale scores and the associated conditional density, their current scale score corresponds to a percentile of that conditional distribution. This percentile is the student’s growth percentile. Growth percentiles are closely related to estimating the probability of observing a student’s current achievement taking account of their past achievement: Pr(Current Achievement |Past Achievement ). As such, growth percentiles describe the rarity of a student’s current achievement conditional upon their prior achievement. Damian W. Betebenner Student Growth Percentile Methodology

Describing Student Growth Intro Student Growth Percentiles References Model & Software for Student Growth Percentiles Criterion Referenced Growth Student Growth Percentiles Should we be surprised with a child’s current achievement given their prior achievement? Student growth percentiles answer this question. Consider a low achieving student with 50th percentile growth and a high achieving student with 50th percentile growth. The low achieving student grew at a rate exceeding 50 percent of similar students. The high achieving student grew at a rate exceeding just 50 percent of similar students. The low achiever’s growth is probabilistically equivalent to the high achiever’s. Equally rare does not imply equally good! Judgments about the adequacy of student growth require external criteria. That is, criterion referencing. Damian W. Betebenner Student Growth Percentile Methodology

Describing Student Growth Intro Student Growth Percentiles References Model & Software for Student Growth Percentiles Criterion Referenced Growth Model for Student Growth Percentiles Student growth percentiles are calculated using quantile regression with B-spline smoothing, a generalized additive model. Quantile regression is used to model the complete distribution of the response variable (current achievement) B-spline smoothing is uses to accommodate non-linearity and heteroscedasticity of the data. The model/method is descriptive and all about data fit—it’s a data mining procedure. The model quantifies distance = rate · time probabilistically. Damian W. Betebenner Student Growth Percentile Methodology

Describing Student Growth Intro Student Growth Percentiles References Model & Software for Student Growth Percentiles Criterion Referenced Growth Model for Student Growth Percentiles Conditional Decile Regression Curves Conditional Decile Regression Lines 2006−07 Math: Grade 5 versus 6 2006−07 Math: Grade 5 versus 6 1000 Unsatisfactory Partially Prof. 1000 Unsatisfactory Partially Prof. Proficient Proficient Partially Prof. Proficient Advanced Partially Prof. Proficient Advanced Grade 6 Scale Score Grade 6 Scale Score Grade 5 Scale Score Grade 5 Scale Score Damian W. Betebenner Student Growth Percentile Methodology

Grade 3 & 4 B−Spline/Linear Conditional Decile Curves/Lines with OLS Regression Line ● ● ● 2009 4th Grade CSAP Scale Score 200 10th 20th 30th 40th ● 50th 60th 70th 80th 90th 2008 3rd Grade CSAP Scale Score

Describing Student Growth Intro Student Growth Percentiles References Model & Software for Student Growth Percentiles Criterion Referenced Growth Software for Student Growth Percentiles Student Growth Percentiles and Percentile Growth Trajectories are calculated using the open source R Software Environment [R Development Core Team, 2012] and the SGP package [Betebenner et al., 2012] Licensed under Creative Commons Share-and-Share Alike Licenses: In working with state departments of education, the goal is to build state capacity so states can perform and validate their own analyses using software available and easily used by the state. Colorado, Massachusetts, Nevada, Indiana, West Virginia, Wisconsin are states who have state staff capable of generating the student growth percentiles together with aggregations that follow from their calculation. State ownership/oversight of the SGP calculations has been a big plus in terms of buy-in from the state, enhancing their ability to explain and work with local stakeholders. Damian W. Betebenner Student Growth Percentile Methodology

Describing Student Growth Intro Student Growth Percentiles References Model & Software for Student Growth Percentiles Criterion Referenced Growth Establishing growth-standards based upon growth-norms The most common adequacy criterion are judging growth toward an achievement goal (i.e., growth-to-standard). Results from student growth percentile analyses can be used to calculate percentile growth trajectories for each student. These trajectories indicate what future rates of growth will lead to and are used to make adequacy judgments. This growth-to-standard approach was approved as part of Colorado’s successful application to the Growth Model Pilot Program. Damian W. Betebenner Student Growth Percentile Methodology

Massachusetts: 2009 ELA Norm & Criterion Referenced Growth & Achievement 2009 Achievement Percentile No Grade 9 Assessment Baseline Referenced Percentile Growth Trajectory Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 10

Massachusetts: 2009 Mathematics Norm & Criterion Referenced Growth & Achievement Baseline Referenced Percentile Growth Trajectory No Grade 9 Assessment 2009 Achievement Percentile Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 10

Massachusetts: 2009 ELA Norm & Criterion Referenced Growth & Achievement Baseline Referenced Percentile Growth Trajectory No Grade 9 Assessment 2009 Achievement Percentile Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 10

Massachusetts: 2009 Mathematics Norm & Criterion Referenced Growth & Achievement No Grade 9 Assessment Baseline Referenced Percentile Growth Trajectory 2009 Achievement Percentile Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 10

Describing Student Growth Intro Student Growth Percentiles References Model & Software for Student Growth Percentiles Criterion Referenced Growth Precision of SGPs In the past year there questions have arisen about the precision associated with SGPs Interest has been on quantifying the precision as well as reporting the precision as part of the student reports. In Spring 2012, representatives from DESE went to UMASS Amherst and participated in a colloquium on the growth model. Together with baseline referenced SGPs, DESE has been researching SGP precision as part of its ongoing effort to improve the growth model. Damian W. Betebenner Student Growth Percentile Methodology

Describing Student Growth Intro Student Growth Percentiles References Model & Software for Student Growth Percentiles Criterion Referenced Growth Precision of SGPs Based upon an idea of Kathy Flannagan and recently seen in a paper under review for JEBS. Through simulation create standard error of prediction. Recall from standard OLS regression: 1 (xh + x̄ )2 Standard Error of the Prediction ≡ MSE 1 + +P n (xi − x̄ )2 Using the CSEMs associated with MCAS scale scores we estimate the standard error associated with SGPs by running multile (100) analyses using CSEM perturbed scores. Damian W. Betebenner Student Growth Percentile Methodology

Describing Student Growth Intro Student Growth Percentiles References Model & Software for Student Growth Percentiles Criterion Referenced Growth Precision of SGPs Before running on MCAS, ran analyses on smaller data set (real data) embedded in SGP package. Ran 100 full analyses of the data using CSEM perturbed scale scores. Using the coefficient matrices derived in each of the 100 analyses, calculate SGP for student. Using those 100 different SGPs for each student, calculate their standard deviation to generate the standard error of estimate. Damian W. Betebenner Student Growth Percentile Methodology

Describing Student Growth Intro Student Growth Percentiles References Model & Software for Student Growth Percentiles Criterion Referenced Growth Baseline referenced Student Growth Percentiles Using data from 2008, 2009, 2010, 2011, and 2012, Massachusetts constructed baseline referenced SGPs using super cohorts of students with up to three priors in each grade and content area. Baseline referenced SGPs allow for, in theory, all students to have high growth since they are being compared to a fixed baseline reference set of students. Baseline SGPs require a stable scale from year to year. Preliminary investigations of baseline SGPs have confirmed achievement results some grades and content areas which suggest that students in the state of Massachusetts are growing at higher rates, on average, than students in previous years. Damian W. Betebenner Student Growth Percentile Methodology

Intro References References Betebenner, D. W. (2008). Toward a normative understanding of student growth. In Ryan, K. E. and Shepard, L. A., editors, The Future of Test-Based Educational Accountability, pages 155–170. Taylor & Francis, New York. Betebenner, D. W. (2009). Norm- and criterion-referenced student growth. Educational Measurement: Issues and Practice, 28(4):42–51. Betebenner, D. W. (2012). Growth, standards, and accountability. In Cizek, G., editor, Setting Performance Standards: Foundations, Methods, and Innovations, pages 439–450. Routledge, New York. Betebenner, D. W., Iwaarden, A. V., and Domingue, B. (2012). SGP: An R Package for the Calculation and Visualization of Student Growth Percentiles & Percentile Growth Trajectories. R package version 0.9-0.0. R Development Core Team (2012). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. Damian W. Betebenner Student Growth Percentile Methodology

About this copy

An Overview of Student Growth Percentile Methodology: Massachusetts TAC — Massachusetts Technical Advisory Committee, October 25, 2012.

This is an image-based copy of the original beamer slides; builds and animations are flattened.