IDAHO’S STUDENT GROWTH
TRAJECTORY MODEL
Damian W Betebenner
National Center for the Improvement of Educational Assessment
Boise, ID March 6th, 2018
BACKGROUND
• Idaho’s new vision for accountability is sharing clear, understandable
data about school performance using a variety of indicators
• Outlined in IDAPA Rule 08.02.03.112. We have provided a copy of this in your
TAC materials.
• The state plans to make this information available via a new online
Report Card
BACKGROUND
• For reporting, and identification, schools will be separated into three
categories:
• K-8 schools
• High Schools
• Alternative High Schools
• Each category of schools will have their own set of academic and
school quality measures
BACKGROUND
• Idaho Code outlines the required measures, but with limited detail
• For example, the K-8 reporting must include “ISAT growth toward
proficiency based on a trajectory model approved by the State Board
of Education”
• There are several ways in which we could design a “trajectory” model.
THE TRAJECTORY MODEL
• Trajectory model is a broad term that could encompass several
different types of “growth to standard” approaches.
• In some instances, student trajectories are based upon:
• Predicted future outcome(s): Is the student predicted to reach the desired
standard.
• Desired future outcome(s): Is the student making annual progress sufficient to
reach the desired standard.
• Based on previous conversations with the Board’s Accountability
Oversight Committee, our understanding is that the Board is
interested in the following type of model…
THE TRAJECTORY MODEL
Image from “A Practitioner’s Guide to Growth Models” by Castellano and Ho.
http://www.ccsso.org/Documents/2013GrowthModels.pdf
THE TRAJECTORY MODEL
• This model is conceptually straightforward
• Additionally, the model is consistent with the Idaho Standards
Achievement Tests (ISATs) in ELA and Math scoring structure
• Scale is vertically aligned and ranges from roughly 2,000 to 3,000
THE TRAJECTORY MODEL
• However, we still need to determine:
• What standard is the student expected to reach?
• Over what time period?
• Using what growth assumption?
WHAT STANDARD IS THE STUDENT
EXPECTED TO REACH?
• The ISAT has four performance levels: Level 1, Level 2, Level 3, and
Level 4
• Informally, these levels are interpreted as representing: Below Basic,
Basic, Proficient, and Advanced performance
• Consequently, we have interpreted Idaho Code’s reference to “future
proficiency” as using the Level 3 cut score
• These scores are illustrated on the next slide
WHAT STANDARD IS THE STUDENT
EXPECTED TO REACH?
WHAT STANDARD IS THE STUDENT
EXPECTED TO REACH?
• Potential considerations
• Internally consistent with other measures of school achievement (e.g.
proficiency rate), but also contains the same limitations as the proficiency
calculation, such as creating a binary distinction between scores that are
statistically equivalent (e.g. one point below and one point above the cut
score). This results in a loss of information.
• Potentially too ambitious for low achieving students.
OVER WHAT TIME PERIOD?
• Regardless of the target the state adopts, we also need to determine
the time period over which a student is expected to reach that
threshold
• For example, should a student be expected to reach this target over a
set number of years (e.g. three years) or by a consistent end point
(e.g. 8th grade)
• And for either of these approaches, we would still need to make specific
decisions about the number of years or end point to use
OVER WHAT TIME PERIOD?
• Idaho Code offers less guidance on this question than the it does for
the growth standard
• Internally, conversations have generally focused on reaching the target
by 8th grade
• However, this was for simplicity and we have not addressed this issue in detail
OVER WHAT TIME PERIOD?
• Potential considerations
• A set target, such as 8th grade, is easier to communicate because of its
consistency but is generally more forgiving to students at earlier grades
• In most growth to standard implementations, time frame has been 3 years or
by the final grade, whichever came first. The desire for a fixed time frame was
due to a desire to not provide too long a time frame so that schools “pass the
buck”.
• As a point of comparison. Many ELP testing programs use growth-to-standard.
5 year time frame is typical but empirically derived.
USING WHAT GROWTH ASSUMPTION?
• Even with a growth target and timeframe in place, Idaho would need to determine
how to establish future achievement targets (annual achievement targets can be
converted to growth targets via simple subtraction).
• Generally, trajectory models base future achievement targets in two ways:
• Predicted future outcome(s): Based upon linear trajectory assumptions or regression.
• Desired future outcome(s): Based upon a trajectory leading toward proficiency.
• However, even within that framework, there are additional details we would need to
address:
• Does a student’s “historical growth” mean last year’s growth, the student’s average growth, a
weighted composite of all past growth?
• Should that growth be projected forward using linear growth assumptions or with
adjustments for the “expected” scale score growth at different grade levels?
USING WHAT GROWTH ASSUMPTION?
• This topic is again something Idaho Code does not address in any
detail
• Internal discussions have tended to focus on a student’s growth over
the past year, with a linear projection forward
• For example, if a student’s ELA scale score went up 25 points from Grade 3 to
Grade 4, the accountability targets would establish a scale score increase by 25
points each year in the future.
• This is the projection seen in the Castellano and Ho chart
• This trajectory might not lead to proficiency.
USING WHAT GROWTH TRAJECTORY
ASSUMPTIONS?
• Potential considerations
• Castellano and Ho state the linear projections in their trajectory model
example are “not empirical as much as descriptive and aspirational.”
• In fact, they categorize more refined, regression-based projection models separately from
the trajectory approach
• Increasing accuracy may result in more complicated calculations that are less
understandable
• Regardless of the approach, scale score targets are established that are used
for determining whether progress is (un)satisfactory.
INVESTIGATION OF SUMMARY RESULTS
• Assuming achievement targets are established leading to proficiency
in (i) 3 years or (ii) 8th grade, calculations with 2017 data were
performed.
• Individual level results
• School level results
SUMMARY OF INDIVIDUAL LEVEL RESULTS
• Growth-to-Standard results show:
• ELA: 55.3% of students reaching 3 year targets, 56.7% of students reaching 8th
grade targets, and 57.1% reaching 10th grade targets.
• Mathematics: 47.3% of students reaching 3 year targets, 49.3% of students
reaching 8th grade targets, and 46.1% reaching 10th grade targets.
• More time increases chances of reaching the proficiency target. 7th
grade students have only until either grade 8 or 10 to reach the target
(1 or 2 time periods)
• Low achieving students have much more difficult targets to reach than
higher achieving students.
• Economically disadvantaged students (who tend to be lower achieving
than their non-disadvantaged counterparts) have more difficult targets.
• Special education students have more difficult targets as well.
SCHOOL LEVEL RESULTS
• Used for school level accountability, several summary statistics are
available to examine probable impact including.
• Correlation between percent meeting target and mean prior achievement
• Correlation between percent economically disadvantaged and percent
meeting target.
• Correlation between percent special education and percent meeting target.
SUMMARY OF SCHOOL LEVEL RESULTS
• Growth-to-Standard results show:
• Correlation between percent of students reaching growth target and prior
achievement is strong. This is true regardless of whether 3 year, 8th grade, or
10th grade is used and regardless of whether mean standardized prior
achievement of percent proficient prior is used.
• When broken out by content area, school level correlations remain strong.
• Strong correlations between prior achievement and growth indicate that high
achieving schools will mostly be flagged for high growth. This is generally a
problematic finding.
• Moderate to low correlations exist between percent of student reaching
growth target and percent economically disadvantaged.
• Moderate to low correlations exist between percent of student reaching
growth target and percent special education.
• Results (particularly with prior achievement) are consistent with what
is found in other states.
Percent Students Reaching 2017 Target: 4th
Grade
2016 Math 3 Year Math 8th Math 10th ELA 3 Year ELA 8th Grade ELA 10th Grade
(beginning) (N=21,665) Grade Grade (N=21,675) (N=21,675) (N=21,675)
Achievement (N=21,665) (N=21,665)
Level
Level 1 31.5% 50.5% 52.1% 32.4% 47.5% 51.3%
Level 2 42.6% 54.8% 53.8% 46.0% 53.3% 55.6%
Level 3 59.0% 63.6% 60.5% 60.2% 62.0% 62.9%
Level 4 75.2% 71.1% 66.8% 72.5% 66.8% 65.2%
Percent Students Reaching 2017 Target: 5th
Grade
2016 Math 3 Year Math 8th Math 10th ELA 3 Year ELA 8th Grade ELA 10th Grade
(beginning) (N=21,290) Grade Grade (N=21,303) (N=21,303) (N=21,303)
Achievement (N=21,290) (N=21.290)
Level
Level 1 21.9% 30.0% 31.1% 38.8% 47.7% 49.1%
Level 2 38.8% 42.8% 44.5% 55.0% 58.8% 59.1%
Level 3 63.2% 62.8% 65.2% 65.5% 65.9% 64.3%
Level 4 81.0% 76.0% 77.0% 79.0% 75.1% 74.1%
Percent Students Reaching 2017 Target: 6th
Grade
2016 Math 3 Year Math 8th Math 10th ELA 3 Year ELA 8th Grade ELA 10th Grade
(beginning) (N=21,355) Grade Grade (N=21,368) (N=21,368) (N=21,368)
Achievement (N=21,355) (N=21,355)
Level
Level 1 26.8% 26.8% 28.2% 37.0% 37.0% 38.1%
Level 2 45.6% 45.6% 47.3% 48.0% 48.0% 49.2%
Level 3 58.5% 58.5% 57.4% 58.4% 58.4% 59.3%
Level 4 76.4% 76.4% 77.5% 69.9% 69.9% 71.5%
Percent Students Reaching 2017 Target: 7th
Grade
2016 Math 3 Year Math 8th Math 10th ELA 3 Year ELA 8th Grade ELA 10th Grade
(beginning) (N=20,893) Grade Grade (N=20,925) (N=20,925) (N=20,925)
Achievement (N=20,893) (N=20,893)
Level
Level 1 15.1% 14.3% 15.1% 24.5% 23.8% 24.5%
Level 2 35.6% 36.8% 35.6% 49.0% 48.1% 49.0%
Level 3 66.7% 64.5% 66.7% 73.7% 72.0% 73.7%
Level 4 89.2% 87.0% 89.2% 90.0% 88.7% 90.0%
Percent Students Reaching 2017 Target: 8th
Grade
2016 Math 3 Year Math 8th Math 10th ELA 3 Year ELA 8th Grade Math 10th
(beginning) (N=20,488) Grade Grade (N=20,518) (N=20,518) Grade
Achievement (N=20,488) (N=20,488) (N=20,518)
Level
Level 1 5.2% 0.8% 5.2% 8.2% 3.1% 8.2%
Level 2 19.4% 16.4% 19.4% 31.4% 29.2% 31.4%
Level 3 69.4% 67.2% 69.4% 82.2% 79.1% 82.2%
Level 4 98.8% 97.9% 98.8% 99.1% 98.8% 99.1%
Percent Students Reaching 2017 Target: 4th
Grade by Economically Disadvantaged Status
2016 Math 3 Year Math 8th Math 10th ELA 3 Year ELA 8th Grade ELA 10th Grade
(beginning) (N=21,665) Grade Grade (N=21,675) (N=21,675) (N=21,675)
Achievement (N=21,665) (N=21,665)
Level
Economically 44.4% 54.4% 56.8% 45.5% 52.1% 54.6%
Disadvantaged
Not 61.4% 66.8% 69.1% 61.8% 64.0% 66.5%
Economically
Disadvantaged
Percent Students Reaching 2017 Target: 5th
Grade by Economically Disadvantaged Status
2016 Math 3 Year Math 8th Math 10th ELA 3 Year ELA 8th Grade ELA 10th Grade
(beginning) (N=21,290) Grade Grade (N=21,303) (N=21,303) (N=21,303)
Achievement (N=21,290) (N=21,290)
Level
Economically 42.0% 44.8% 45.6% 52.2% 56.2% 58.1%
Disadvantaged
Not 60.3% 60.9% 62.9% 672% 67.9% 68.7%
Economically
Disadvantaged
Percent Students Reaching 2017 Target: 6th
Grade by Economically Disadvantaged Status
2016 Math 3 Year Math 8th Math 10th ELA 3 Year ELA 8th Grade ELA 10th Grade
(beginning) (N=21,355) Grade Grade (N=21,368) (N=21,368) (N=21,368)
Achievement (N=21,355) (N=21,355)
Level
Economically 38.3% 38.3% 39.2% 46.4% 46.4% 48.2%
Disadvantaged
Not 59.9% 59.9% 60.4% 60.6% 60.6% 62.3%
Economically
Disadvantaged
Percent Students Reaching 2017 Target: 7th
Grade by Economically Disadvantaged Status
2016 Math 3 Year Math 8th Math 10th ELA 3 Year ELA 8th Grade ELA 10th Grade
(beginning) (N=20,893) Grade Grade (N=20,925) (N=20,925) (N=20,925)
Achievement (N=20,893) (N=20,893)
Level
Economically 36.1%% 34.5% 36.1% 49.9% 48.6% 49.9%
Disadvantaged
Not 58.2%% 56.5% 58.2% 68.3% 66.9% 68.3%
Economically
Disadvantaged
Percent Students Reaching 2017 Target: 8th
Grade by Economically Disadvantaged Status
2016 Math 3 Year Math 8th Math 10th ELA 3 Year ELA 8th Grade ELA 10th Grade
(beginning) (N=20,488) Grade Grade (N=20,518) (N=20,518) (N=20,518)
Achievement (N=20,488) (N=20,488)
Level
Economically 28.2% 26.7% 28.2% 44.2% 41.2% 44.2%
Disadvantaged
Not 53.4% 50.3% 53.4% 64.5% 63.7% 64.5%
Economically
Disadvantaged
Correlation between percent reaching target
and other school level indicators (N >= 10)
School Level Indicator 3 Year Growth Target 8th Grade Growth Target 10 Grade Growth Target
Mean Standardized prior 0.76 0.69 0.65
achievement
Percent prior year 0.77 0.71 0.67
proficient
Percent current year 0.91 0.85 0.82
proficient
Percent Economically -0.43 -0.37 -0.32
Disadvantaged
Percent Special Education -0.40 -0.36 -0.28
Correlation between percent reaching target
and other school level indicators (N>=10)
School Level Math 3 Year Math 8th Grade Math 10th ELA 3 Year ELA 8th Grade ELA 10th Grade
Indicator (516) (516) Grade (516) (516) (516) (516)
Mean prior 0.69 0.61 0.59 0.69 0.62 0.59
standardized
achievement
Percent prior 0.73 0.68 0.66 0.70 0.62 0.60
year proficient
Percent -0.44 -0.37 -0.33 -0.41 -0.35 -0.31
economically
disadvantaged
Percent special -0.28 -0.24 -0.26 -0.31 -0.25 -0.24
education
About this copy
Idaho’s Student Growth Trajectory Model — Idaho Department of Education, March 6, 2018.
This is an image-based copy of the original Keynote slides; builds and animations are flattened.