Agenda
■ (30 min) DAMIAN Overview of SGPs – where we are, where we are
going
Baseline SGPs – showcase grade 10 results in ELA
■ Efficacy of the B-Splines for adjusting growth expectations at the tails of the
distributions
Growth and achievement charts –
■ discussion of purpose of the SGPs – discussion of the descriptive capacity of
SGPs
■ (30 min) Bob – History and Current Application of the SGPs
■ (30 min) Kathy – Utility and Validity of SGPs
■ (30 min) DAMIAN Reliability: Accuracy (How much are we off from
the true value) & Precision (Signal to noise)
■ Arise at the individual levels for summary data:
For individuals or groups
■ A discussion about use cases and its relationship to precision & reliability
■ Future Applications
Baseline SGPs
Proposed for use in educator evaluation (2012, informally – DESE will
provide guidance on this in 2013)
Future validity studies
An Overview of the Student
Growth Percentile Methodology
About me
Damian Betebenner
■ Senior Associate at the Center for Assessment
■ Developed student growth percentiles and percentile
growth trajectories to help states and educational
associations utilize student growth in decision making.
■ Currently refining and sharing these techniques with over
30 states including Colorado, New York, Rhode Island,
New Hampshire, Maine, …
Describing Student Growth
■ Measuring student growth, even with a vertical scale, is not a simple
task.
■ Some believe a vertical scale simplifies the task of measuring
student growth.
■ Even with an interval (or ratio) scale, growth is not easy to interpret.
■ Consider, for example, height.
A child might grow 4 inches between ages 3 and 4.
4 inches is a well understood quantity.
The 4 inch increase becomes really meaningful only when understood alongside
the growth of other 3 to 4 year olds.
■ Student growth percentiles were developed to provide a norm-
referenced description of student growth (i.e., a growth norm).
History and Current
Applications of the SGPs
Why did we need a growth
model
■ NCLB required ELA and Mathematics tests in all grades
3-8 beginning in 2006
■ Longitudinal data systems allowed schools and districts
to view assessment trends over time
■ Inferences made by comparing scaled scores from one
grade to another are invalid, but were being made with
increasing frequency
2012 MCAS Percent Proficient by Grade
ELA Mathematics
Growth Model Roll-out Recap
Spring 2009 SGP Pilot
■ A Department staff-member was assigned to each of 10
volunteer districts to help principals and curriculum
coordinators interpret data and use the Education Data
Warehouse
■ Reports were developed and modified based on feedback
from pilot districts
■ Districts struggled to interpret Growth to Standard Data,
which was removed from the system
■ Pilot districts helped Department staff develop FAQ and
interpretive materials for the statewide release
Growth Model Roll-out Recap
October 2009: Statewide release
■ Growth Reports from 2008 and 2009 added to School and
District Profiles
■ The Department pledged not to use Growth Data in the
accountability system while school and district officials
familiarized themselves with the data
■ Student level reporting in the Education Data Warehouse
hampered by very slow technology
■ 475 Data Warehouse users statewide were able to run 3,000
reports in a peak week
Growth Model Roll-out Recap
■ 2010 First Commendations given to schools with two consecutive
years of median Growth above 60 in ELA & Math
■ Massachusetts’ Race to the Top application proposes using
Growth data as significant part of the educator evaluation system –
Two thirds of the state’s districts sign up for RttT
17% of teachers in Massachusetts have 20 or more growth scores in their
primary subject
■ June 2010-2011 Student Assessment oversees $460,000 project to
upgrade and renovate the EDW
■ The Department pilots the collection of student, course and
educator data
Growth Model Roll-out Recap
■ Use of the renovated EDW burgeons to 150,000 reports in
September 2011 then 268,000 in September 2012
About half of all reports feature growth
■ 99,000 users per hour review public MCAS Growth
and Performance Reports on September 19, 2012
■ Every Principal in Massachusetts is required to
attend one of seven MCAS workshops where a
growth model is explained in detail – 96.5% attend
Growth Model Roll-out Recap
■ The state’s NCLB waiver uses ELA and
Mathematics growth as two of 11 indicators in the
Progress and Performance Index used to make
accountability ratings
■ The state proposes using SGPs along with at least
one other indicator to identify trends and patterns
in Educator Effectiveness
Growth in the NCLB Waiver
Median SGP Range Alternate pathways Number of points
60-99 15+ point change in SGP 100
50.5-59.5 10-14 point change 75
41-50 1-9 point change 50
Schools must average 75 points across all indicators and
for their High Needs Subgroup to be classified in Level 1
Growth in the Educator Evaluation
Proposal
2010 to 2011 English and Elementary Teachers’ two year
growth trends (Moderate = 40 to 59.5)
ELA 2011 Teacher Growth Category
Higher Moderate Lower Total
Count 210 126 31 367
Higher
% of Total 18.7% 11.2% 2.8% 32.7%
ELA 2010 Pilot Growth Count 144 307 90 541
Moderate
Category % of Total 12.8% 27.3% 8.0% 48.1%
Count 19 107 90 216
Lower
% of Total 1.7% 9.5% 8.0% 19.2%
Count 373 540 211 1124
Total
% of Total 33.2% 48.0% 18.8% 100.0%
R = .52; The attrition of teachers with low growth in the 2010 pilot
was higher (19.8%) than for teachers with high growth (14.7%)
2010 to 2011 Mathematics and Elementary Teachers’ two
year growth trends (Moderate = 40 to 59.5)
Math Teachers 2011 Growth Category
Higher Moderate Lower Total
Count 195 132 14 341
Higher
% of Total 18.7% 12.6% 1.3% 32.6%
Math Teachers 2010 Count 127 244 103 474
Moderate
Growth Category % of Total 12.2% 23.3% 9.9% 45.4%
Count 27 82 121 230
Lower
% of Total 2.6% 7.8% 11.6% 22.0%
Count 349 458 238 1045
Total
% of Total 33.4% 43.8% 22.8% 100.0%
R = .56; The attrition of teachers with low growth in the 2010 pilot
was higher (18.4%) than for teachers with high growth (12.8%)
SGP Utility and Validity
Validity and Utility of SGPs – An examination of
consequential validity evidence
Overall: Promote a defensible way of describing and understanding
changes in achievement.
Intended Outcomes Degree Goal is Achieved
1. Growth indicator that is correlated 1. SGPs are highly correlated to gains
with improvement on MCAS exams on MCAS and to changes in
performance levels
2. Growth indicator that is
uncorrelated to student 2. SGPs have very low correlations
demographic measures with student demographic
measures at both the individual
and the school level
3. All students have an equal
opportunity to show growth 3. Due to the use of B-Spline fitting,
unfettered by floor and ceiling SGPs are largely unaffected by
effects floor and ceiling effects; almost all
students have an equal opportunity
to grow
SGPs have standardized measurement properties: Scatterplot matrix: ELA
Change scores are
negatively correlated with
prior year’s MCAS SSs
(r=-.29 to -.50 )
Meaning: Regression to
the Mode/Mean
SGPs areSGPs
NOT arecorrelated with prior
NOT correlated
MCAS SSs (r=0),
with eachbut are (r=0),
other positively
but
correlatedhighly
with current (r=.49
correlated withto .56)
and futurechange
(r=.22 scores
to .25) MCAS SSs
(.75 to .81)
Meaning: Student growth is
unrelated to prior achievement levels
Sample: ~61,000+
students starting in
grade 5 in 2009 and
ending in grade 8 in
SGPs explain student achievement patterns
Math Proficiency Levels 2009 and 2012
by Student Average Growth 2010 to 2012
Math Growth Average #10#3 SGP Categories:
Very Grand Very Low: 1-16
Math PL 2012 Very Low Low Moderate High
High Total Low : 17-33
Warning 1,076 3,450 5,840 247 30 10,643 Moderate: 34-66
Needs Improvement 529 3,549 12,943 1,535 175 18,731 High: 67-82
Proficient 202 2,373 14,514 3,030 421 20,540 Very High:83-99
Advanced 19 452 9,234 4,466 1,023 15,194
Grand Total 1,826 9,824 42,531 9,278 1,649 65,108
Math Proficiency Levels 2009 and 2012
by Student Average Growth 2010 to 2012
The majority of
Math Growth Average #10#3
students achieving
Math PL 2012 Very Low Low Moderate High Very High
high growth from Warning 59% 35% 14% 3% 2%
2010-12 end up Needs Improvement 29% 36% 30% 17% 11%
scoring Advanced Proficient 11% 24% 34% 33% 26%
in 2012. Advanced 1% 5% 22% 48% 62%
Grand Total 100% 100% 100% 100% 100%
SGPs explain student achievement patterns
Math Proficiency Levels 2009 and 2012
by Student Average Growth 2010 to 2012
While students in all
Math PL 2012 performance
Math Growth
Average Needs
categories in 2009
Math PL 2009 #10#2 Warning Improvement Proficient Advanced Grand Total have an equal chance
Warning Low 18% 1% 19%
Moderate 46% 18% 0% 0% 64%
at growth…
High 3% 12% 3% 0% 17%
Needs
Improvement
Low 11% 6% 0% 0% 17% Students achieving
Moderate 7% 45% 14% 0% 66%
high average growth
High 0% 3% 12% 2% 18%
Proficient Low 2% 11% 3% 0% 17% are 5X more likely to
Moderate 0% 12% 44% 9% 65% move out of the Failing
High 0% 0% 5% 13% 18% level.
Advanced Low 0% 3% 11% 3% 17%
Moderate 0% 0% 17% 48% 65%
High 0% 18% 18% Students achieving low
Grand Total 16% 29% 31% 23% 100% average growth are 4X
more likely to fall from
the Proficient level.
SGPs help identify students who gain or lose ground on MCAS
performance levels
Students 2009: 216
starting out in 2012: 240 SGP10: 99
needs SGP11: 20
SGP10: 99 SGP12: 16
improvement in SGP11: 62
Mod. Growth 2012: 228
2009 with high SGP12: 23
Students
average growth Mod. Growth SGP10: 1
move up a out in
starting
SGP11: 28
failing in 2009
performance SGP12: 78
level with high
in 2012 Mod. Growth
average growth
move up a
performance
level in 2012
SGPs predominantly describe growth, not student demographic information
Proportion of Explained Variance, Stepwise Linear
Regression
R R^2 Variables Entered
0.777 0.604 MB1: MATG2009
0.945 0.893 MB2: MATG2009
*Math SGPs 2010-12
0.948 0.899 MB3: MATG2009
*MSGP 2010-12
*Student Demographics
Student
demographics
explains .6%
residual variance.
SGPs provide an important indicator of school effectiveness
Hierarchical Regression Coefficients &
(Standard Errors)* Variance Components of Hierarchical Regression Models*
Null Model MB1 MB2 MB3
Null Model MB1 MB2 MB3
Intercept -.081 -.031 -013 .120
(.023) (.011) (.004) (.005) Variance Components
MATG2009 (GMC) .858 .869 .812 L1 Student (W) 1.422 0.602 0.178 0.167
(.003) (.002) (.002) L2 School (B) 0.443 0.107 0.008 0.008
MSGP 2010 (GMC) .011 .011 Additional Variance Explained
L1 Student (W) 57.64% 87.51% 88.26%
MSGP 2011 (GMC) .010 .010
(.000) (.000) L2 School (B) 75.88% 98.13% 98.13%
MSGP2012 (GMC) .019 .019 Deviance (-2LL) 197,274 144,388 68,422 65,261
(.000) (.000) Change in -2LL 52,886 75,966 3,161
freelunch_sum -0.02 # of parameters 3 4 7 20
Significance 0.000 0.000 0.000 0.000
title1_sum -.033
(.002) Intra-class Correlation
lep_sum .007 ICC 23.8%
(.006) *Dependent Variable: Math Theta 2012
lep_flep_sum -.022
sped_sum -.053
Race_AA -.086
Race_A .060 24% of Variance is
Race_H
-.025 87.5% of
between schools Addition
varianceof for
Race_M
demographic
students and 98% of
Race_O
variables
variance for explains
schools is an
accom_l_sum
additional
described by .75% of
the 2009
accom_c_sum
variance
MCAS scores forthe
and students
accom_m_sum
to 2011 and
SGPs .001% of
(.006) variance for schools
SGPs maximize benefits and minimize weaknesses of student growth data
■ Prior to the SGPs,
individuals had no easy way
to determine if they
performed better or worse on
MCAS from one grade to the
next
■ Change scores demonstrate
regression to the mode
■ SGPs are strongly correlated
with change (r = .85), but are
more stable
■ SGPs are strongly
associated with future
performance on MCAS
■ SGPs are very weakly
correlated with student
demographics
■ Given MCAS’ lack of floor/
ceiling effects, SGPs allow
all students ~equal chance
to grow
Size denotes change scores –
range from -42 to 48