Idaho SMART Project

Idaho Department of Education · March 5, 2025

Idaho SMART Project — Idaho Department of Education, March 5, 2025.
Published

March 5, 2025

Event
State agency
Slides
60
Format
Converted from Keynote

This deck is an image-based copy of the original slides. Builds and animations are flattened; the slide text is reproduced below for search and screen readers.

Slide text
  1. Idaho SMART Project An investigation of program efficacy using IRI data Damian Betebenner Center for Assessment March 5th, 2025
  2. Introduction
  3. SMART Background • Launched in May 2021, Idaho Striving to Meet Achievement in Reading Together (SMART) is a multi-year cohort project training K-3 educators in science-based reading instruction and intervention. • Participants engage in facilitated learning with Speech to Print by Dr. Louisa Moats, guided implementation of evidence-based practices, and coaching for sustained instructional improvement. • The goal is to equip K-3 educators with science-aligned reading strategies to improve student outcomes on the Idaho Reading Inventory (IRI).
  4. SMART: IRI Impact • To investigate whether SMART benefits IRI outcomes, we examine whether students learn more in SMART classrooms than non-SMART classrooms. • Using 2022-2023 and 2023-2024 IRI assessment results we calculate the growth (i.e., learning) of students and compare rates of learning between groups to examine SMART impact. • NOTE: Simply comparing IRI outcomes between SMART and non-SMART students can potentially provide misleading results as students are not randomly-assigned to the SMART program.
  5. SMART: IRI Impact Positive SMART Impact: No SMART Impact: SMART students learn more SMART students learn as much than non-SMART students as non-SMART students over over the course of the year the course of the year SMART SMART IRI non-SMART IRI non-SMART Fall Spring Fall Spring
  6. IRI: Processes to Impacts to Outcomes Assessment Indicators Proximal-Leading Distal-Lagging Growth/Learning Achievement/Status lead to lead to Processes Impacts Outcomes The diverse set of factors and activities including instructional practices, professional climate, and student experiences, that are implemented or cultivated to support student learning.
  7. Research Questions: • Do students instructed by SMART participating teachers demonstrate more learning (i.e., greater growth) than other students. • Does student growth/learning differ by grade of student/ teacher? • Does student growth/learning differ by number of years of teacher participation in SMART? • Does student growth/learning differ by number of years of student receiving SMART instruction?
  8. Background on Student Growth • Calculation of student growth requires longitudinal, individual results from at least two assessments (usually in the same content area administered at the same time). • When thinking about student growth/learning, most users immediately think of gain (i.e., change) scores. • Calculation of change across two tests requires the test be scaled vertically placing the results of both tests on the same scale. • The IRI is vertically scaled. • However, gain scores are not a viable growth model because student change is dependent upon where a student starts. • All defensible growth models utilize regression to accommodate dependency upon where a student starts.
  9. Student Growth Percentiles (SGPs) • SGPs were derived to obviate issues related to the interpretation of change on score scales. • The SGP properties I discuss are associated with those derived via the methodology implemented in the SGP package for the R software environment. • SGPs are integers ranging from 1 to 99 where 1st percentile growth is the lowest and 99th percentile growth is the highest. • SGPs are independent of prior achievement meaning that low and high achieving students have the same chances of demonstrating low or high growth. • SGPs utilize non-parametric (B-spline) based quintile regression to accommodate both non- linearity of the multi-variate score distribution as well as heteroskedasticity. • SGPs are invariant to mono-tone transformations of scale. • In general, assessment changes resulting in the change of score scale do not impact the calculation of SGPs (cohort referenced).
  10. IRI Data • For SGP calculations, IRI data was utilized from the 2022-2023 and 2023-2024 academic years. • Fall and Spring IRI data was utilized to calculate Fall-to- Spring SGPs in both 2022-2023 and 2023-2024 • Students with appropriate Fall and Spring scaled scores in grade K, 1, 2, and 3 received SGPs.
  11. Potential Issues related to SGPs and inferences regarding SMART • Because SGP calculations utilize scaled scores, properties of the scaled score distribution can impact the results. • Ceiling/Floor effects: SGPs model the multi-variate distribution and attempt to model 99 conditional quantile curves based upon the scaled score supplied. When the score distribution is capped and disproportionate numbers of students are assigned HOSS/LOSS, SGPs will be impacted. • Inferences regarding SMART impact (using SGPs or any metric) are only as good as the data on which they are based. • If the IRI does not assess what SMART instruction teaches children, then results from the IRI will not be sensitive to changes in those outcomes.
  12. SMART-IRI Data & SGP Calculation
  13. SMART Participation • IRI data for the 2022-2023 and 2023-2024 academic years was examined. • At the start of 2022-2023, between 3.2% (Grade 3) and 5.3% (Grade 1) students were associated with teachers participating in the SMART program. • At the start of 2023-2024, between 5.5% (Grade 3) and 11.7% (Grade 1) students were associated with teachers participating in the SMART program. • The achievement level of students participating in the program in 2022-2023 was slightly lower in 3 out of 4 grades and in 2023-2024 was slightly lower in 1 out of 4 grades.
  14. Starting Achievement Level*: SMART-IRI Participant All Students SMART Students Non-Participating Grade Mean Scale Score Mean Scale Score Students Mean Scale Percentile (Count) Percentile (Count) Score Percentile (Count) Fall 2023 2 50.2 (22,966) 50.4 (819) 50.2 (22,147) Fall 2024 2 50.3 (23,339) 50.9 (1,818) 50.2 (21,521) * Idaho only normed achievement
  15. SGP Calculation • Though initial achievement level of students associated with SMART instruction was, in general, comparable to those not associated with the program, we examine growth/learning by conditioning upon prior (Fall) achievement. • SGPs were calculated using IRI reading data supplied by IDOE for 2022-2023 and 2023-2024 in grades K, 1, 2, and 3. • SGPs were calculated for Fall-to-Spring in 2022-2023 and 2023-2024. • SGPs for Spring 2023 utilized a Fall 2022 prior. • SGPs for Spring 2024 utilized a Fall 2023 prior and Spring 2023 prior (if available) • Due to the multi-week testing window associated with IRI administration, time between the spring and fall administration was incorporated into the analyses to account for variability in time-lag between test administration. • Analyses were conducted using the R software environment (4.4.3) and the SGP package (2.2-1.93). • All analyses ran without error and no technical issues emerged in the calculation of the SGPs.
  16. SGP Calculation: IRI Data SMART SGP SGP SMART Grade Student Student Student Student Instructor Count Count Count Count (SMART) Count Fall 2022 2 22,966 819 41 Spring 2023 2 23,240 830 22,434 801 41 Fall 2023 2 23,339 1,818 90 Spring 2024 2 23,411 1,859 22,681 1,813 93
  17. SGP Calculation: Goodness of Fit • SGP results show excellent fit in both years and in all grades. • No ceiling or floor effects were observed on any of the assessments. • SGPs were uniformly distributed across all prior achievement levels. • By definition, the mean/median of the SGPs (for the entire grade) is • However, subgroups (e.g., students associated with SMART instruction) can deviate from 50. • Results above 50 indicate increased rates of learning/growth. • Results below 50 indicate decreased rates of learning/growth
  18. SMART/IRI SGP Results
  19. SGP Results • Did students associated with SMART instruction learn more than their counterparts not associated with SMART instruction? • The following two tables display: • Mean SGPs by Year, Grade, and SMART subgroup. • Mean SGPs by Number of Years teacher in SMART program by grade.
  20. SMART-IRI SGP Results All Students SMART Students Non-Participating Grade Mean SGP Mean SGP Students Mean SGP (Count) (Count) (Count) Spring 2023 2 50.0 (22,434) 52.3 (801) * 49.9 (21,633) Spring 2024 2 50.4 (22,681) 48.7 (1,813) * 50.4 (20,868) * Indicates statistically significant at the .05 level
  21. SMART-IRI SGP Results by Number of Years Teaching in SMART 0 Years in SMART 1 Year in SMART 2 Years in SMART Grade Mean SGP Mean SGP Mean SGP (Count) (Count) (Count) Spring 2024 2 50.4 (20,868) 47.7 (1,247) 50.8 (566)
  22. SGP Results • In 2022-2023, Grades K and 2 showed statistically significant increases in growth/learning. • In 2023-2024, Grades K, 1, and 3 showed statistically significant increases in growth/learning while Grade 2 showed a statistically significant decrease. • Though the results are statistically significant due to large sample size, the results are not practically significant. • Results based upon number of years participating in SMART mirror the increases but do not show overall increases in learning with two years participation over one year.
  23. Instructor Level Performance • De-identified IDs were provided for instructors participating in the program. • We examined performance (student growth/learning by entering level of achievement). • The results are shown for 2023-2024 for K, 1, 2, and 3 in the following figures.
  24. SMART Instructor Performance: Grade K 2023⌧2024 Growth & Achievement Growth & Prior Achievement Higher Achievement Higher Achievement Class Size Lower Growth Higher Growth 10 Students 25 Students Mean Scale Score Percentile 50 Students 100 Students Prior Achievement Lower Growth Higher Growth Summary Statistics Correlation: −0.08 Lower Achievement Lower Achievement Correlation (N ≥ 10): −0.11 Growth Mean Student Growth Percentile
  25. SMART Instructor Performance: Grade K 2023⌧2024 Growth & Achievement Growth & Prior Achievement Higher Achievement Higher Achievement Class Size Lower Growth Higher Growth 10 Students 25 Students Mean Scale Score Percentile 50 Students 100 Students Prior Achievement Lower Growth Higher Growth Summary Statistics Correlation: −0.08 Lower Achievement Lower Achievement Correlation (N ≥ 10): −0.11 2 Year Instructors Highlighted Growth Mean Student Growth Percentile
  26. SMART Instructor Performance: Grade 1 2023⌧2024 Growth & Achievement Growth & Prior Achievement Higher Achievement Higher Achievement Class Size Lower Growth Higher Growth 10 Students 25 Students Mean Scale Score Percentile 50 Students 100 Students Prior Achievement Lower Growth Higher Growth Summary Statistics Correlation: 0.14 Lower Achievement Lower Achievement Correlation (N ≥ 10): 0 Growth Mean Student Growth Percentile
  27. SMART Instructor Performance: Grade 1 2023⌧2024 Growth & Achievement Growth & Prior Achievement Higher Achievement Higher Achievement Class Size Lower Growth Higher Growth 10 Students 25 Students Mean Scale Score Percentile 50 Students 100 Students Prior Achievement Lower Growth Higher Growth Summary Statistics Correlation: 0.14 Lower Achievement Lower Achievement Correlation (N ≥ 10): 0 2 Year Instructors Highlighted Growth Mean Student Growth Percentile
  28. SMART Instructor Performance: Grade 2 2023⌧2024 Growth & Achievement Growth & Prior Achievement Higher Achievement Higher Achievement Class Size Lower Growth Higher Growth 10 Students 25 Students Mean Scale Score Percentile 50 Students 100 Students Prior Achievement Lower Growth Higher Growth Summary Statistics Correlation: 0.03 Lower Achievement Lower Achievement Correlation (N ≥ 10): 0.03 Growth Mean Student Growth Percentile
  29. SMART Instructor Performance: Grade 2 2023⌧2024 Growth & Achievement Growth & Prior Achievement Higher Achievement Higher Achievement Class Size Lower Growth Higher Growth 10 Students 25 Students Mean Scale Score Percentile 50 Students 100 Students Prior Achievement Lower Growth Higher Growth Summary Statistics Correlation: 0.03 Lower Achievement Lower Achievement Correlation (N ≥ 10): 0.03 2 Year Instructors Highlighted Growth Mean Student Growth Percentile
  30. SMART Instructor Performance: Grade 3 2023⌧2024 Growth & Achievement Growth & Prior Achievement Higher Achievement Higher Achievement Class Size Lower Growth Higher Growth 10 Students 25 Students Mean Scale Score Percentile 50 Students 100 Students Prior Achievement Lower Growth Higher Growth Summary Statistics Correlation: −0.01 Lower Achievement Lower Achievement Correlation (N ≥ 10): −0.08 Growth Mean Student Growth Percentile
  31. SMART Instructor Performance: Grade 3 2023⌧2024 Growth & Achievement Growth & Prior Achievement Higher Achievement Higher Achievement Class Size Lower Growth Higher Growth 10 Students 25 Students Mean Scale Score Percentile 50 Students 100 Students Prior Achievement Lower Growth Higher Growth Summary Statistics Correlation: −0.01 Lower Achievement Lower Achievement Correlation (N ≥ 10): −0.08 2 Year Instructors Highlighted Growth Mean Student Growth Percentile
  32. Instructor Level Performance • In general, the results suggest broad variability in rates of student learning associated with factors that extend beyond the curriculum/pedagogy (SMART) being utilized. • The slightly negative correlations suggest SMART instruction is potentially of greater benefit to lower achieving students.
  33. SMART Impact by Prior Achievement • The summary results suggest that student associated with SMART participants demonstrated greater rates of learning if they were lower achieving. • Discussions with IDE (Meghan) indicated that SMART emphasizes learning to read over reading to learn which suggest greater efficacy for lower achieving readers. • To investigate this hypothesis, we examine student growth/learning of students associated with SMART by prior achievement quintile. • In some grades, the results, in general, indicate slightly better rates of learning for lower achieving students. • The following figures illustrate results for Grade 1, 2023-2024
  34. SMART Instructor Performance: Grade 1 2023⌧2024 Growth & Achievement Growth & Prior Achievement Higher Achievement Higher Achievement Class Size Lower Growth Higher Growth 10 Students 25 Students Mean Scale Score Percentile 50 Students 100 Students Prior Achievement Lower Growth Higher Growth Summary Statistics Correlation: −0.02 Lower Achievement Lower Achievement Correlation (N ≥ 10): −0.16 Growth Mean Student Growth Percentile
  35. SMART Instructor Performance: Grade 1 2023⌧2024 Growth & Achievement Growth & Prior Achievement Higher Achievement Higher Achievement Class Size Lower Growth Higher Growth 10 Students 25 Students Mean Scale Score Percentile 50 Students 100 Students Prior Achievement Lower Growth Higher Growth Summary Statistics Correlation: −0.03 Lower Achievement Lower Achievement Correlation (N ≥ 10): 0.11 Growth Mean Student Growth Percentile
  36. SMART Instructor Performance: Grade 1 2023⌧2024 Growth & Achievement Growth & Prior Achievement Higher Achievement Higher Achievement Class Size Lower Growth Higher Growth 10 Students 25 Students Mean Scale Score Percentile 50 Students 100 Students Prior Achievement Lower Growth Higher Growth Summary Statistics Correlation: 0 Lower Achievement Lower Achievement Correlation (N ≥ 10): 0.1 Growth Mean Student Growth Percentile
  37. SMART Instructor Performance: Grade 1 2023⌧2024 Growth & Achievement Growth & Prior Achievement Higher Achievement Higher Achievement Class Size Lower Growth Higher Growth 10 Students 25 Students Mean Scale Score Percentile 50 Students 100 Students Prior Achievement Lower Growth Higher Growth Summary Statistics Correlation: −0.07 Lower Achievement Lower Achievement Correlation (N ≥ 10): −0.05 Growth Mean Student Growth Percentile
  38. SMART Instructor Performance: Grade 1 2023⌧2024 Growth & Achievement Growth & Prior Achievement Higher Achievement Higher Achievement Class Size Lower Growth Higher Growth 10 Students 25 Students Mean Scale Score Percentile 50 Students 100 Students Prior Achievement Lower Growth Higher Growth Summary Statistics Correlation: 0.04 Lower Achievement Lower Achievement Correlation (N ≥ 10): −0.18 Growth Mean Student Growth Percentile
  39. IRI School Level Results
  40. IRI School Performance 2022⌧2023 Growth & Achievement Growth & Prior Achievement Higher Achievement Higher Achievement School Size Lower Growth Higher Growth 25 Students 100 Students Mean Scale Score Percentile 250 Students 500 Students Prior Achievement Lower Growth Higher Growth Summary Statistics Correlation: 0.18 Lower Achievement Lower Achievement Correlation (N ≥ 10): 0.17 Growth Mean Student Growth Percentile
  41. IRI School Performance 2023⌧2024 Growth & Achievement Growth & Prior Achievement Higher Achievement Higher Achievement School Size Lower Growth Higher Growth 25 Students 100 Students Mean Scale Score Percentile 250 Students 500 Students Prior Achievement Lower Growth Higher Growth Summary Statistics Correlation: 0.17 Lower Achievement Lower Achievement Correlation (N ≥ 10): 0.18 Growth Mean Student Growth Percentile
  42. IRI School Performance 2023⌧2024 Growth & Achievement Growth & Prior Achievement Higher Achievement Higher Achievement School Size Lower Growth Higher Growth 25 Students 100 Students Mean Scale Score Percentile 250 Students 500 Students Prior Achievement Lower Growth Higher Growth Summary Statistics Correlation: 0.17 Lower Achievement Lower Achievement Correlation (N ≥ 10): 0.18 SMART > 20% highlighted Growth Mean Student Growth Percentile
  43. IRI School Performance: Grade KG.3 2023⌧2024 Growth & Achievement Growth & Prior Achievement Higher Achievement Higher Achievement School Size Lower Growth Higher Growth 25 Students 100 Students Mean Scale Score Percentile 250 Students 500 Students Prior Achievement Lower Growth Higher Growth Summary Statistics Correlation: −0.05 Lower Achievement Lower Achievement Correlation (N ≥ 10): −0.09 Growth Mean Student Growth Percentile
  44. IRI School Performance: Grade 1.3 2023⌧2024 Growth & Achievement Growth & Prior Achievement Higher Achievement Higher Achievement School Size Lower Growth Higher Growth 25 Students 100 Students Mean Scale Score Percentile 250 Students 500 Students Prior Achievement Lower Growth Higher Growth Summary Statistics Correlation: 0.01 Lower Achievement Lower Achievement Correlation (N ≥ 10): 0.04 Growth Mean Student Growth Percentile
  45. IRI School Performance: Grade 2.3 2023⌧2024 Growth & Achievement Growth & Prior Achievement Higher Achievement Higher Achievement School Size Lower Growth Higher Growth 25 Students 100 Students Mean Scale Score Percentile 250 Students 500 Students Prior Achievement Lower Growth Higher Growth Summary Statistics Correlation: 0.08 Lower Achievement Lower Achievement Correlation (N ≥ 10): 0.07 Growth Mean Student Growth Percentile
  46. IRI School Performance: Grade 3.3 2023⌧2024 Growth & Achievement Growth & Prior Achievement Higher Achievement Higher Achievement School Size Lower Growth Higher Growth 25 Students 100 Students Mean Scale Score Percentile 250 Students 500 Students Prior Achievement Lower Growth Higher Growth Summary Statistics Correlation: 0.05 Lower Achievement Lower Achievement Correlation (N ≥ 10): 0.01 Growth Mean Student Growth Percentile
  47. Summary
  48. Conclusions • I commend IDE for investigating the efficacy of the SMART program using IRI data in “real time”. • Overall, the results suggest, at best, modest improvement in student learning for students/teachers in the SMART program. • These results confirm, I think, some things that we all know: • “Everything works somewhere. Nothing works everywhere.” Dylan Wiliam • There are no silver bullets. • There are likely numerous pedagogical factors beyond SMART participation at play that also need to be addressed to substantially improve student rates of learning.
  50. Appendix
  51. Goodness of Fit (Kindergarten) Student Growth Percentile Goodness−of−Fit Descriptives Student Growth Percentile Goodness−of−Fit Descriptives 2022−2023.3 Reading SGP, Grade KG.3 (N = 20,764) 2023−2024.3 Reading SGP, Grade KG.3 (N = 19,404) Ceiling / Floor Effects Test Ceiling / Floor Effects Test Student Growth Percentile Range Student Growth Percentile Range Current Observed Score Current Observed Score 5 9 to to to to to to to to to to 5 9 to to to to to to to to to to to to 10 20 30 40 50 60 70 80 90 95 to to 10 20 30 40 50 60 70 80 90 95 Lowest OSS: [100,134) 80.0 16.0 4.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 N = 25 Lowest OSS: [100,137) 76.0 8.0 16.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 N = 25 Extremes (Range) Extremes (Range) Highest OSS: [504,522) 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.0 8.0 16.0 20.0 52.0 N = 25 Highest OSS: [504,521) 0.0 0.0 0.0 0.0 0.0 4.0 4.0 0.0 0.0 20.0 12.0 60.0 N = 25 Student Growth Percentile Range QQ−Plot: Student Growth Percentiles Student Growth Percentile Range QQ−Plot: Student Growth Percentiles 19 29 39 49 59 69 79 89 99 Theoretical SGP Distribution 19 29 39 49 59 69 79 89 99 Theoretical SGP Distribution 9 to to to to to to to to to 9 to to to to to to to to to to 10 20 30 40 50 60 70 80 90 0 10 20 30 40 50 60 70 80 90 100 1 to 10 20 30 40 50 60 70 80 90 0 10 20 30 40 50 60 70 80 90 100 1st/[100,198) 9.88 10.17 10.36 10.32 10.07 9.83 9.98 10.22 9.39 9.78 (9.9%) 100 1st/[100,201) 10.24 10.18 10.40 10.29 9.76 9.71 9.97 9.82 9.50 10.13 (9.8%) 100 2nd/[198,218) 9.26 10.14 9.41 9.12 9.70 10.19 10.09 10.09 10.24 11.75 (9.9%) 2nd/[201,220) 8.62 9.83 9.83 9.67 9.83 10.15 10.41 10.04 10.94 10.68 (9.8%) Prior Scale Score Decile/Range Prior Scale Score Decile/Range 3rd/[218,231) 8.76 9.34 10.01 10.79 10.30 10.01 9.82 9.34 11.32 10.30 (10%) 3rd/[220,233) 9.62 9.57 9.98 9.98 10.96 10.49 9.26 9.93 9.52 10.70 (10%) Empirical SGP Distribution Empirical SGP Distribution 4th/[231,242) 9.81 10.45 10.06 10.06 9.72 9.86 10.55 9.81 9.81 9.86 (9.9%) 4th/[233,244) 9.50 10.38 9.60 9.08 9.24 9.86 10.22 10.64 10.74 10.74 (9.9%) 5th/[242,252) 10.12 10.37 10.37 9.68 10.72 10.22 9.18 10.67 8.78 9.88 (9.7%) 5th/[244,255) 9.47 9.91 10.40 11.24 9.76 10.06 10.01 9.71 9.57 9.86 (10.5%) 6th/[252,263) 9.28 9.48 9.62 9.57 9.23 10.06 10.54 10.40 10.16 11.66 (9.9%) 6th/[255,265) 10.49 10.27 9.44 9.55 10.82 9.06 9.99 9.77 9.33 11.26 (9.3%) 7th/[263,275) 9.18 10.03 9.93 10.22 10.55 9.89 10.08 9.18 10.03 10.93 (10.2%) 7th/[265,277) 8.96 10.03 10.18 10.08 9.98 10.18 10.08 9.69 10.86 9.98 (10.6%) 8th/[275,289) 10.04 10.27 10.42 10.23 9.42 9.56 9.75 10.27 10.56 9.47 (10.2%) 8th/[277,291) 9.63 9.89 9.78 9.53 9.63 11.22 9.73 10.71 9.48 10.40 (10.1%) 9th/[289,309) 9.13 9.55 9.87 9.97 10.52 10.34 10.11 9.92 9.87 10.71 (10.4%) 9th/[291,310) 9.11 9.91 10.76 10.49 10.17 8.74 10.28 9.54 9.80 11.18 (9.7%) 10th/[309,440] 9.78 10.21 9.97 10.01 9.78 10.01 9.97 10.06 9.73 10.49 (10.1%) 10th/[310,464] 9.73 10.03 9.58 10.03 9.93 10.38 9.98 10.18 10.18 9.98 (10.4%) *Prior score deciles can be uneven depending upon the prior score distribution *Prior score deciles can be uneven depending upon the prior score distribution
  52. Goodness of Fit (1st grade) Student Growth Percentile Goodness−of−Fit Descriptives Student Growth Percentile Goodness−of−Fit Descriptives 2022−2023.3 Reading SGP, Grade 1.3 (N = 22,040) 2023−2024.3 Reading SGP, Grade 1.3 (N = 21,732) Ceiling / Floor Effects Test Ceiling / Floor Effects Test Student Growth Percentile Range Student Growth Percentile Range Current Observed Score Current Observed Score 5 9 to to to to to to to to to to 5 9 to to to to to to to to to to to to 10 20 30 40 50 60 70 80 90 95 to to 10 20 30 40 50 60 70 80 90 95 Lowest OSS: [100,184) 83.3 4.2 12.5 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 N = 24 Lowest OSS: [100,170) 76.0 8.0 12.0 4.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 N = 25 Extremes (Range) Extremes (Range) Highest OSS: [580,595) 0.0 0.0 0.0 0.0 0.0 0.0 4.0 4.0 4.0 16.0 12.0 60.0 N = 25 Highest OSS: [578,593) 0.0 0.0 0.0 0.0 3.8 0.0 0.0 3.8 3.8 15.4 26.9 46.2 N = 26 Student Growth Percentile Range QQ−Plot: Student Growth Percentiles Student Growth Percentile Range QQ−Plot: Student Growth Percentiles 19 29 39 49 59 69 79 89 99 Theoretical SGP Distribution 19 29 39 49 59 69 79 89 99 Theoretical SGP Distribution 9 to to to to to to to to to 9 to to to to to to to to to to 10 20 30 40 50 60 70 80 90 0 10 20 30 40 50 60 70 80 90 100 1 to 10 20 30 40 50 60 70 80 90 0 10 20 30 40 50 60 70 80 90 100 1st/[100,269) 9.36 9.92 9.68 10.34 10.10 10.47 10.34 9.78 9.64 10.38 (9.7%) 100 1st/[100,271) 10.00 9.86 10.74 9.76 9.81 9.25 10.97 9.53 10.04 10.04 (9.9%) 100 2nd/[269,290) 9.88 10.20 10.16 9.79 9.74 9.65 9.32 10.48 9.74 11.04 (9.8%) 2nd/[271,292) 8.87 10.71 9.25 10.15 10.38 11.00 8.64 10.29 9.86 10.85 (9.8%) Prior Scale Score Decile/Range Prior Scale Score Decile/Range 3rd/[290,304) 9.10 9.96 10.32 9.28 10.32 9.64 10.46 9.73 11.19 10.00 (10%) 3rd/[292,306) 9.27 9.54 9.63 10.13 9.81 10.41 9.45 10.50 10.31 10.95 (10.1%) Empirical SGP Distribution Empirical SGP Distribution 4th/[304,316) 9.96 9.24 10.00 10.90 10.09 10.00 10.31 9.69 9.42 10.40 (10.1%) 4th/[306,318) 9.46 9.84 9.94 9.84 10.32 9.89 10.89 9.42 10.03 10.37 (9.7%) 5th/[316,328) 9.04 10.81 10.10 9.66 10.46 10.19 9.35 10.50 9.17 10.72 (10.2%) 5th/[318,330) 9.43 10.38 10.83 9.66 9.56 8.98 10.83 9.88 10.20 10.24 (10.2%) 6th/[328,340) 9.52 10.26 9.43 9.52 9.47 10.07 10.17 10.12 11.18 10.26 (9.8%) 6th/[330,342) 10.46 10.18 9.68 10.87 9.82 10.50 9.09 10.00 9.36 10.05 (10.1%) 7th/[340,353) 10.03 9.43 10.49 10.44 9.29 10.17 10.44 9.84 9.47 10.40 (9.8%) 7th/[342,356) 9.09 9.32 10.07 10.30 9.74 10.16 10.39 9.88 10.11 10.95 (9.9%) 8th/[353,369) 8.93 9.82 10.13 9.91 10.62 9.60 9.60 10.04 10.31 11.03 (10.2%) 8th/[356,372) 9.31 10.63 9.90 9.22 10.36 9.63 9.72 10.22 10.49 10.54 (10.1%) 9th/[369,391) 9.95 10.76 9.50 10.04 9.90 10.31 9.95 9.72 9.95 9.95 (10.1%) 9th/[372,393) 10.08 9.48 10.04 9.48 10.77 10.31 9.76 10.13 10.04 9.90 (10%) 10th/[391,549] 9.42 9.69 10.22 10.09 9.96 9.91 10.09 10.13 9.91 10.58 (10.3%) 10th/[393,605] 9.19 10.21 9.95 10.52 9.41 9.99 10.39 9.86 9.72 10.75 (10.4%) *Prior score deciles can be uneven depending upon the prior score distribution *Prior score deciles can be uneven depending upon the prior score distribution
  53. Goodness of Fit (2nd Grade) Student Growth Percentile Goodness−of−Fit Descriptives Student Growth Percentile Goodness−of−Fit Descriptives 2022−2023.3 Reading SGP, Grade 2.3 (N = 22,434) 2023−2024.3 Reading SGP, Grade 2.3 (N = 22,681) Ceiling / Floor Effects Test Ceiling / Floor Effects Test Student Growth Percentile Range Student Growth Percentile Range Current Observed Score Current Observed Score 5 9 to to to to to to to to to to 5 9 to to to to to to to to to to to to 10 20 30 40 50 60 70 80 90 95 to to 10 20 30 40 50 60 70 80 90 95 Lowest OSS: [105,196) 88.0 0.0 12.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 N = 25 Lowest OSS: [100,186) 60.0 20.0 8.0 8.0 4.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 N = 25 Extremes (Range) Extremes (Range) Highest OSS: [716,762) 0.0 0.0 0.0 4.0 0.0 0.0 0.0 0.0 0.0 4.0 12.0 80.0 N = 25 Highest OSS: [743,794) 0.0 0.0 0.0 4.0 0.0 0.0 0.0 0.0 4.0 16.0 8.0 68.0 N = 25 Student Growth Percentile Range QQ−Plot: Student Growth Percentiles Student Growth Percentile Range QQ−Plot: Student Growth Percentiles 19 29 39 49 59 69 79 89 99 Theoretical SGP Distribution 19 29 39 49 59 69 79 89 99 Theoretical SGP Distribution 9 to to to to to to to to to 9 to to to to to to to to to to 10 20 30 40 50 60 70 80 90 0 10 20 30 40 50 60 70 80 90 100 1 to 10 20 30 40 50 60 70 80 90 0 10 20 30 40 50 60 70 80 90 100 1st/[130,322) 9.42 10.00 9.96 10.18 10.18 10.18 10.18 9.28 10.18 10.45 (9.9%) 100 1st/[100,320) 9.81 10.26 9.77 10.30 10.08 9.99 9.42 9.06 10.17 11.14 (10%) 100 2nd/[322,353) 9.38 10.02 10.65 9.07 9.79 9.43 9.29 11.06 10.20 11.11 (9.8%) 2nd/[320,351) 9.78 8.69 9.42 9.37 9.65 9.51 9.96 11.51 10.69 11.42 (9.7%) Prior Scale Score Decile/Range Prior Scale Score Decile/Range 3rd/[353,373) 10.41 9.45 9.18 11.38 10.02 10.54 10.33 10.11 9.18 9.40 (10.1%) 3rd/[351,372) 9.53 9.30 9.26 9.85 9.85 10.52 10.16 9.94 10.57 11.02 (9.8%) Empirical SGP Distribution Empirical SGP Distribution 4th/[373,389) 8.70 11.35 10.18 9.29 10.54 9.42 10.81 8.79 10.00 10.90 (9.9%) 4th/[372,388) 9.82 10.26 10.21 10.12 9.90 10.48 9.86 9.86 9.15 10.34 (10%) 5th/[389,403) 9.63 8.79 9.72 10.12 9.77 10.53 9.23 10.48 10.88 10.84 (10%) 5th/[388,403) 9.14 10.63 10.12 10.20 9.61 9.48 9.74 9.78 10.29 11.01 (10.4%) 6th/[403,416) 9.43 10.61 11.50 9.34 9.10 9.67 9.19 10.84 9.90 10.42 (9.5%) 6th/[403,416) 9.45 9.99 9.27 10.39 10.12 9.36 10.48 10.03 10.30 10.62 (9.8%) 7th/[416,431) 9.68 9.84 8.46 11.14 10.43 10.31 10.85 9.93 9.80 9.55 (10.6%) 7th/[416,430) 9.43 8.99 10.39 9.21 10.61 10.13 9.96 10.70 9.74 10.83 (10.1%) 8th/[431,447) 9.13 9.86 10.64 8.81 10.32 9.73 10.14 9.86 9.68 11.83 (9.8%) 8th/[430,446) 10.81 9.03 9.81 10.89 9.07 10.20 9.98 9.42 9.94 10.85 (10.2%) 9th/[447,472) 10.01 10.50 9.62 10.10 9.75 10.28 10.15 9.31 10.23 10.06 (10.1%) 9th/[446,472) 8.98 11.22 10.12 9.15 9.60 10.12 9.90 10.43 9.86 10.61 (10%) 10th/[472,709] 9.36 9.67 10.32 10.28 10.10 9.89 9.67 10.37 9.97 10.37 (10.2%) 10th/[472,676] 9.55 9.50 9.98 9.98 10.37 9.85 9.81 9.94 10.20 10.81 (10.1%) *Prior score deciles can be uneven depending upon the prior score distribution *Prior score deciles can be uneven depending upon the prior score distribution
  54. Goodness of Fit (3rd Grade) Student Growth Percentile Goodness−of−Fit Descriptives Student Growth Percentile Goodness−of−Fit Descriptives 2022−2023.3 Reading SGP, Grade 3.3 (N = 22,505) 2023−2024.3 Reading SGP, Grade 3.3 (N = 22,850) Ceiling / Floor Effects Test Ceiling / Floor Effects Test Student Growth Percentile Range Student Growth Percentile Range Current Observed Score Current Observed Score 5 9 to to to to to to to to to to 5 9 to to to to to to to to to to to to 10 20 30 40 50 60 70 80 90 95 to to 10 20 30 40 50 60 70 80 90 95 Lowest OSS: [100,224) 88.0 12.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 N = 25 Lowest OSS: [110,212) 96.2 3.8 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 N = 26 Extremes (Range) Extremes (Range) Highest OSS: [817,847) 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 12.5 20.8 66.7 N = 24 Highest OSS: [845,882) 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.8 3.8 11.5 80.8 N = 26 Student Growth Percentile Range QQ−Plot: Student Growth Percentiles Student Growth Percentile Range QQ−Plot: Student Growth Percentiles 19 29 39 49 59 69 79 89 99 Theoretical SGP Distribution 19 29 39 49 59 69 79 89 99 Theoretical SGP Distribution 9 to to to to to to to to to 9 to to to to to to to to to to 10 20 30 40 50 60 70 80 90 0 10 20 30 40 50 60 70 80 90 100 1 to 10 20 30 40 50 60 70 80 90 0 10 20 30 40 50 60 70 80 90 100 1st/[181,374) 9.39 10.37 9.52 9.43 9.79 10.06 10.10 10.19 10.15 11.00 (9.9%) 100 1st/[141,375) 9.67 9.75 10.37 9.75 9.97 9.62 9.80 10.19 10.37 10.50 (10%) 100 2nd/[374,407) 10.02 9.76 10.20 10.91 10.47 9.67 10.02 9.62 9.76 9.58 (10%) 2nd/[375,409) 9.47 10.42 9.51 10.64 10.01 10.55 10.73 9.87 9.10 9.69 (9.7%) Prior Scale Score Decile/Range Prior Scale Score Decile/Range 3rd/[407,426) 8.86 9.74 10.76 9.51 9.88 10.39 9.46 10.76 9.60 11.04 (9.6%) 3rd/[409,430) 9.24 10.30 10.13 10.05 9.96 9.79 8.91 10.05 10.47 11.10 (10.4%) Empirical SGP Distribution Empirical SGP Distribution 4th/[426,441) 9.34 10.28 9.70 9.74 9.92 10.28 10.41 10.54 10.01 9.79 (10%) 4th/[430,444) 9.29 9.38 9.38 9.24 9.29 10.28 11.09 10.46 10.14 11.45 (9.7%) 5th/[441,456) 10.12 9.90 10.12 10.20 9.77 9.29 9.90 8.98 10.51 11.21 (10.2%) 5th/[444,460) 10.51 10.02 10.95 10.37 10.64 9.44 9.62 9.49 9.71 9.26 (9.9%) 6th/[456,471) 9.06 10.05 9.96 10.05 10.41 9.96 9.91 9.46 10.50 10.64 (9.9%) 6th/[460,475) 8.97 10.49 10.36 9.98 10.58 10.58 9.48 9.69 9.81 10.07 (10.3%) 7th/[471,488) 9.85 9.98 9.98 10.62 9.94 10.45 11.00 10.58 8.32 9.30 (10.4%) 7th/[475,492) 10.30 10.08 9.33 11.04 9.73 9.55 9.86 9.77 9.29 11.04 (10%) 8th/[488,508) 9.58 10.31 9.95 9.04 10.04 9.63 8.68 9.86 11.30 11.62 (9.8%) 8th/[492,511) 8.52 9.59 9.64 8.65 9.23 10.62 10.62 10.89 11.21 11.03 (9.8%) 9th/[508,538) 9.10 9.40 9.66 10.19 9.71 10.58 10.32 10.27 10.23 10.54 (10.2%) 9th/[511,541) 8.96 9.52 9.95 10.04 10.39 9.65 10.26 10.51 10.47 10.26 (10.1%) 10th/[538,765] 9.81 10.30 10.16 10.25 10.12 9.63 10.03 9.90 9.68 10.12 (10%) 10th/[541,787] 10.32 10.45 10.36 10.28 9.98 10.06 9.59 9.16 9.34 10.45 (10.2%) *Prior score deciles can be uneven depending upon the prior score distribution *Prior score deciles can be uneven depending upon the prior score distribution
  55. SMART Instructor Performance: Grade K 2022⌧2023 Growth & Achievement Growth & Prior Achievement Higher Achievement Higher Achievement Class Size Lower Growth Higher Growth 10 Students 25 Students Mean Scale Score Percentile 50 Students 100 Students Prior Achievement Lower Growth Higher Growth Summary Statistics Correlation: −0.11 Lower Achievement Lower Achievement Correlation (N ≥ 10): −0.13 Growth Mean Student Growth Percentile
  56. SMART Instructor Performance: Grade 1 2022⌧2023 Growth & Achievement Growth & Prior Achievement Higher Achievement Higher Achievement Class Size Lower Growth Higher Growth 10 Students 25 Students Mean Scale Score Percentile 50 Students 100 Students Prior Achievement Lower Growth Higher Growth Summary Statistics Correlation: −0.2 Lower Achievement Lower Achievement Correlation (N ≥ 10): −0.22 Growth Mean Student Growth Percentile
  57. SMART Instructor Performance: Grade 2 2022⌧2023 Growth & Achievement Growth & Prior Achievement Higher Achievement Higher Achievement Class Size Lower Growth Higher Growth 10 Students 25 Students Mean Scale Score Percentile 50 Students 100 Students Prior Achievement Lower Growth Higher Growth Summary Statistics Correlation: 0.14 Lower Achievement Lower Achievement Correlation (N ≥ 10): −0.1 Growth Mean Student Growth Percentile
  58. SMART Instructor Performance: Grade 3 2022⌧2023 Growth & Achievement Growth & Prior Achievement Higher Achievement Higher Achievement Class Size Lower Growth Higher Growth 10 Students 25 Students Mean Scale Score Percentile 50 Students 100 Students Prior Achievement Lower Growth Higher Growth Summary Statistics Correlation: 0.38 Lower Achievement Lower Achievement Correlation (N ≥ 10): 0.23 Growth Mean Student Growth Percentile
  59. Know what you want to accomplish and focus on the process rather than the outcome. Nick Saban
  60. IRI: Processes to Impacts to Outcomes Assessment Indicators Proximal-Leading Distal-Lagging Growth/Learning Achievement/Status lead to lead to Processes Impacts Outcomes The diverse set of factors and activities including instructional practices, professional climate, and student experiences, that are implemented or cultivated to support student learning.