Idaho’s Student Growth Trajectory Model

Idaho Department of Education · March 6, 2018

Idaho’s Student Growth Trajectory Model — Idaho Department of Education, March 6, 2018.
Published

March 6, 2018

Event
State agency
Slides
34
Format
Converted from Keynote

This deck is an image-based copy of the original slides. Builds and animations are flattened; the slide text is reproduced below for search and screen readers.

Slide text
  1. IDAHO’S STUDENT GROWTH TRAJECTORY MODEL Damian W Betebenner National Center for the Improvement of Educational Assessment Boise, ID March 6th, 2018
  2. BACKGROUND • Idaho’s new vision for accountability is sharing clear, understandable data about school performance using a variety of indicators • Outlined in IDAPA Rule 08.02.03.112. We have provided a copy of this in your TAC materials. • The state plans to make this information available via a new online Report Card
  3. BACKGROUND • For reporting, and identification, schools will be separated into three categories: • K-8 schools • High Schools • Alternative High Schools • Each category of schools will have their own set of academic and school quality measures
  4. BACKGROUND • Idaho Code outlines the required measures, but with limited detail • For example, the K-8 reporting must include “ISAT growth toward proficiency based on a trajectory model approved by the State Board of Education” • There are several ways in which we could design a “trajectory” model.
  5. THE TRAJECTORY MODEL • Trajectory model is a broad term that could encompass several different types of “growth to standard” approaches. • In some instances, student trajectories are based upon: • Predicted future outcome(s): Is the student predicted to reach the desired standard. • Desired future outcome(s): Is the student making annual progress sufficient to reach the desired standard. • Based on previous conversations with the Board’s Accountability Oversight Committee, our understanding is that the Board is interested in the following type of model…
  6. THE TRAJECTORY MODEL Image from “A Practitioner’s Guide to Growth Models” by Castellano and Ho. http://www.ccsso.org/Documents/2013GrowthModels.pdf
  7. THE TRAJECTORY MODEL • This model is conceptually straightforward • Additionally, the model is consistent with the Idaho Standards Achievement Tests (ISATs) in ELA and Math scoring structure • Scale is vertically aligned and ranges from roughly 2,000 to 3,000
  8. THE TRAJECTORY MODEL • However, we still need to determine: • What standard is the student expected to reach? • Over what time period? • Using what growth assumption?
  9. WHAT STANDARD IS THE STUDENT EXPECTED TO REACH? • The ISAT has four performance levels: Level 1, Level 2, Level 3, and Level 4 • Informally, these levels are interpreted as representing: Below Basic, Basic, Proficient, and Advanced performance • Consequently, we have interpreted Idaho Code’s reference to “future proficiency” as using the Level 3 cut score • These scores are illustrated on the next slide
  10. WHAT STANDARD IS THE STUDENT EXPECTED TO REACH?
  11. WHAT STANDARD IS THE STUDENT EXPECTED TO REACH? • Potential considerations • Internally consistent with other measures of school achievement (e.g. proficiency rate), but also contains the same limitations as the proficiency calculation, such as creating a binary distinction between scores that are statistically equivalent (e.g. one point below and one point above the cut score). This results in a loss of information. • Potentially too ambitious for low achieving students.
  12. OVER WHAT TIME PERIOD? • Regardless of the target the state adopts, we also need to determine the time period over which a student is expected to reach that threshold • For example, should a student be expected to reach this target over a set number of years (e.g. three years) or by a consistent end point (e.g. 8th grade) • And for either of these approaches, we would still need to make specific decisions about the number of years or end point to use
  13. OVER WHAT TIME PERIOD? • Idaho Code offers less guidance on this question than the it does for the growth standard • Internally, conversations have generally focused on reaching the target by 8th grade • However, this was for simplicity and we have not addressed this issue in detail
  14. OVER WHAT TIME PERIOD? • Potential considerations • A set target, such as 8th grade, is easier to communicate because of its consistency but is generally more forgiving to students at earlier grades • In most growth to standard implementations, time frame has been 3 years or by the final grade, whichever came first. The desire for a fixed time frame was due to a desire to not provide too long a time frame so that schools “pass the buck”. • As a point of comparison. Many ELP testing programs use growth-to-standard. 5 year time frame is typical but empirically derived.
  15. USING WHAT GROWTH ASSUMPTION? • Even with a growth target and timeframe in place, Idaho would need to determine how to establish future achievement targets (annual achievement targets can be converted to growth targets via simple subtraction). • Generally, trajectory models base future achievement targets in two ways: • Predicted future outcome(s): Based upon linear trajectory assumptions or regression. • Desired future outcome(s): Based upon a trajectory leading toward proficiency. • However, even within that framework, there are additional details we would need to address: • Does a student’s “historical growth” mean last year’s growth, the student’s average growth, a weighted composite of all past growth? • Should that growth be projected forward using linear growth assumptions or with adjustments for the “expected” scale score growth at different grade levels?
  16. USING WHAT GROWTH ASSUMPTION? • This topic is again something Idaho Code does not address in any detail • Internal discussions have tended to focus on a student’s growth over the past year, with a linear projection forward • For example, if a student’s ELA scale score went up 25 points from Grade 3 to Grade 4, the accountability targets would establish a scale score increase by 25 points each year in the future. • This is the projection seen in the Castellano and Ho chart • This trajectory might not lead to proficiency.
  17. USING WHAT GROWTH TRAJECTORY ASSUMPTIONS? • Potential considerations • Castellano and Ho state the linear projections in their trajectory model example are “not empirical as much as descriptive and aspirational.” • In fact, they categorize more refined, regression-based projection models separately from the trajectory approach • Increasing accuracy may result in more complicated calculations that are less understandable • Regardless of the approach, scale score targets are established that are used for determining whether progress is (un)satisfactory.
  18. INVESTIGATION OF SUMMARY RESULTS • Assuming achievement targets are established leading to proficiency in (i) 3 years or (ii) 8th grade, calculations with 2017 data were performed. • Individual level results • School level results
  19. SUMMARY OF INDIVIDUAL LEVEL RESULTS • Growth-to-Standard results show: • ELA: 55.3% of students reaching 3 year targets, 56.7% of students reaching 8th grade targets, and 57.1% reaching 10th grade targets. • Mathematics: 47.3% of students reaching 3 year targets, 49.3% of students reaching 8th grade targets, and 46.1% reaching 10th grade targets. • More time increases chances of reaching the proficiency target. 7th grade students have only until either grade 8 or 10 to reach the target (1 or 2 time periods) • Low achieving students have much more difficult targets to reach than higher achieving students. • Economically disadvantaged students (who tend to be lower achieving than their non-disadvantaged counterparts) have more difficult targets. • Special education students have more difficult targets as well.
  20. SCHOOL LEVEL RESULTS • Used for school level accountability, several summary statistics are available to examine probable impact including. • Correlation between percent meeting target and mean prior achievement • Correlation between percent economically disadvantaged and percent meeting target. • Correlation between percent special education and percent meeting target.
  21. SUMMARY OF SCHOOL LEVEL RESULTS • Growth-to-Standard results show: • Correlation between percent of students reaching growth target and prior achievement is strong. This is true regardless of whether 3 year, 8th grade, or 10th grade is used and regardless of whether mean standardized prior achievement of percent proficient prior is used. • When broken out by content area, school level correlations remain strong. • Strong correlations between prior achievement and growth indicate that high achieving schools will mostly be flagged for high growth. This is generally a problematic finding. • Moderate to low correlations exist between percent of student reaching growth target and percent economically disadvantaged. • Moderate to low correlations exist between percent of student reaching growth target and percent special education. • Results (particularly with prior achievement) are consistent with what is found in other states.
  23. Percent Students Reaching 2017 Target: 4th Grade 2016 Math 3 Year Math 8th Math 10th ELA 3 Year ELA 8th Grade ELA 10th Grade (beginning) (N=21,665) Grade Grade (N=21,675) (N=21,675) (N=21,675) Achievement (N=21,665) (N=21,665) Level Level 1 31.5% 50.5% 52.1% 32.4% 47.5% 51.3% Level 2 42.6% 54.8% 53.8% 46.0% 53.3% 55.6% Level 3 59.0% 63.6% 60.5% 60.2% 62.0% 62.9% Level 4 75.2% 71.1% 66.8% 72.5% 66.8% 65.2%
  24. Percent Students Reaching 2017 Target: 5th Grade 2016 Math 3 Year Math 8th Math 10th ELA 3 Year ELA 8th Grade ELA 10th Grade (beginning) (N=21,290) Grade Grade (N=21,303) (N=21,303) (N=21,303) Achievement (N=21,290) (N=21.290) Level Level 1 21.9% 30.0% 31.1% 38.8% 47.7% 49.1% Level 2 38.8% 42.8% 44.5% 55.0% 58.8% 59.1% Level 3 63.2% 62.8% 65.2% 65.5% 65.9% 64.3% Level 4 81.0% 76.0% 77.0% 79.0% 75.1% 74.1%
  25. Percent Students Reaching 2017 Target: 6th Grade 2016 Math 3 Year Math 8th Math 10th ELA 3 Year ELA 8th Grade ELA 10th Grade (beginning) (N=21,355) Grade Grade (N=21,368) (N=21,368) (N=21,368) Achievement (N=21,355) (N=21,355) Level Level 1 26.8% 26.8% 28.2% 37.0% 37.0% 38.1% Level 2 45.6% 45.6% 47.3% 48.0% 48.0% 49.2% Level 3 58.5% 58.5% 57.4% 58.4% 58.4% 59.3% Level 4 76.4% 76.4% 77.5% 69.9% 69.9% 71.5%
  26. Percent Students Reaching 2017 Target: 7th Grade 2016 Math 3 Year Math 8th Math 10th ELA 3 Year ELA 8th Grade ELA 10th Grade (beginning) (N=20,893) Grade Grade (N=20,925) (N=20,925) (N=20,925) Achievement (N=20,893) (N=20,893) Level Level 1 15.1% 14.3% 15.1% 24.5% 23.8% 24.5% Level 2 35.6% 36.8% 35.6% 49.0% 48.1% 49.0% Level 3 66.7% 64.5% 66.7% 73.7% 72.0% 73.7% Level 4 89.2% 87.0% 89.2% 90.0% 88.7% 90.0%
  27. Percent Students Reaching 2017 Target: 8th Grade 2016 Math 3 Year Math 8th Math 10th ELA 3 Year ELA 8th Grade Math 10th (beginning) (N=20,488) Grade Grade (N=20,518) (N=20,518) Grade Achievement (N=20,488) (N=20,488) (N=20,518) Level Level 1 5.2% 0.8% 5.2% 8.2% 3.1% 8.2% Level 2 19.4% 16.4% 19.4% 31.4% 29.2% 31.4% Level 3 69.4% 67.2% 69.4% 82.2% 79.1% 82.2% Level 4 98.8% 97.9% 98.8% 99.1% 98.8% 99.1%
  28. Percent Students Reaching 2017 Target: 4th Grade by Economically Disadvantaged Status 2016 Math 3 Year Math 8th Math 10th ELA 3 Year ELA 8th Grade ELA 10th Grade (beginning) (N=21,665) Grade Grade (N=21,675) (N=21,675) (N=21,675) Achievement (N=21,665) (N=21,665) Level Economically 44.4% 54.4% 56.8% 45.5% 52.1% 54.6% Disadvantaged Not 61.4% 66.8% 69.1% 61.8% 64.0% 66.5% Economically Disadvantaged
  29. Percent Students Reaching 2017 Target: 5th Grade by Economically Disadvantaged Status 2016 Math 3 Year Math 8th Math 10th ELA 3 Year ELA 8th Grade ELA 10th Grade (beginning) (N=21,290) Grade Grade (N=21,303) (N=21,303) (N=21,303) Achievement (N=21,290) (N=21,290) Level Economically 42.0% 44.8% 45.6% 52.2% 56.2% 58.1% Disadvantaged Not 60.3% 60.9% 62.9% 672% 67.9% 68.7% Economically Disadvantaged
  30. Percent Students Reaching 2017 Target: 6th Grade by Economically Disadvantaged Status 2016 Math 3 Year Math 8th Math 10th ELA 3 Year ELA 8th Grade ELA 10th Grade (beginning) (N=21,355) Grade Grade (N=21,368) (N=21,368) (N=21,368) Achievement (N=21,355) (N=21,355) Level Economically 38.3% 38.3% 39.2% 46.4% 46.4% 48.2% Disadvantaged Not 59.9% 59.9% 60.4% 60.6% 60.6% 62.3% Economically Disadvantaged
  31. Percent Students Reaching 2017 Target: 7th Grade by Economically Disadvantaged Status 2016 Math 3 Year Math 8th Math 10th ELA 3 Year ELA 8th Grade ELA 10th Grade (beginning) (N=20,893) Grade Grade (N=20,925) (N=20,925) (N=20,925) Achievement (N=20,893) (N=20,893) Level Economically 36.1%% 34.5% 36.1% 49.9% 48.6% 49.9% Disadvantaged Not 58.2%% 56.5% 58.2% 68.3% 66.9% 68.3% Economically Disadvantaged
  32. Percent Students Reaching 2017 Target: 8th Grade by Economically Disadvantaged Status 2016 Math 3 Year Math 8th Math 10th ELA 3 Year ELA 8th Grade ELA 10th Grade (beginning) (N=20,488) Grade Grade (N=20,518) (N=20,518) (N=20,518) Achievement (N=20,488) (N=20,488) Level Economically 28.2% 26.7% 28.2% 44.2% 41.2% 44.2% Disadvantaged Not 53.4% 50.3% 53.4% 64.5% 63.7% 64.5% Economically Disadvantaged
  33. Correlation between percent reaching target and other school level indicators (N >= 10) School Level Indicator 3 Year Growth Target 8th Grade Growth Target 10 Grade Growth Target Mean Standardized prior 0.76 0.69 0.65 achievement Percent prior year 0.77 0.71 0.67 proficient Percent current year 0.91 0.85 0.82 proficient Percent Economically -0.43 -0.37 -0.32 Disadvantaged Percent Special Education -0.40 -0.36 -0.28
  34. Correlation between percent reaching target and other school level indicators (N>=10) School Level Math 3 Year Math 8th Grade Math 10th ELA 3 Year ELA 8th Grade ELA 10th Grade Indicator (516) (516) Grade (516) (516) (516) (516) Mean prior 0.69 0.61 0.59 0.69 0.62 0.59 standardized achievement Percent prior 0.73 0.68 0.66 0.70 0.62 0.60 year proficient Percent -0.44 -0.37 -0.33 -0.41 -0.35 -0.31 economically disadvantaged Percent special -0.28 -0.24 -0.26 -0.31 -0.25 -0.24 education