Georgia Summary SGP Results

Georgia Department of Education · October 2016

Georgia Summary SGP Results — Georgia Department of Education, October 2016.
Published

October 17, 2016

Event
State agency
Slides
55
Format
Converted from Keynote

This deck is an image-based copy of the original slides. Builds and animations are flattened; the slide text is reproduced below for search and screen readers.

Slide text
  1. GEORGIA SUMMARY SGP RESULTS CORRELATIONS BETWEEN GROWTH AND PRIOR ACHIEVEMENT Damian W Betebenner & Adam VanIwaarden National Center for the Improvement of Educational Assessment Dover, New Hampshire September 15th, 2016
  2. ISSUE: BACKGROUND • Initial SGP analyses with preliminary 2016 ELA and Mathematics EOG data were completed in late July. Preliminary EOC data was also available • As part of validation of results, correlations between mean prior achievement (mean of prior scale score standardized within grade) and mean SGP (unadjusted and SIMEX adjusted) were examined at the school level. • In all but one content area, the correlations increased from 2015 to 2016. • We want to explore whether these increases reflect a change in the system or are a manifestation of the model (e.g., measurement error).
  3. ASSESSMENT: BACKGROUND • 2015 was the first year for the Georgia Milestones (CRCT/EOCT independent variables/Milestones dependent variable) • 2016 was the second year for the Georgia Milestones (CRCT & Milestones independent variables/Milestones dependent variable)
  4. ASSESSMENT: BACKGROUND • In an attempt to isolate the cause of the increases in correlation, we examined Mathematics, Social Studies and Science. • In 2015, ELA analyses used Reading & ELA from 2014 and earlier CRCT as priors (two-priors/year). In 2016, ELA analyses used one 2015 ELA prior and multiple 2014 priors (CRCT reading and ELA). • The new Milestones ELA test incorporates reading, language arts, and writing. • Result for 2016 EOG tests (ELA, Mathematics, Science, and Social Studies) have been finalized. EOC test results are not finalized yet (it is unexpected that the results will change significantly). • The best “apples-to-apples” comparisons emerged from the EOG tests with consistent priors (Mathematics, Social Studies, Science)
  5. ANALYSIS: RESULTS • From 2015 to 2016 there is a definite trend upward in correlations between prior achievement and growth. • Examination of data (see supplementary scatterplots) rule out outliers/ changes in marginal distributions as the reason. • For example, non-random assignment of students to school by level of achievement is one of the factors for measurement error bias. Yet no marked changes in stratification of schools based upon achievement was observed. • The increasing correlations observed are synonymous with larger increases in mean SGP for schools serving higher achieving students (higher mean achievement) than their counterparts serving lower achieving students.
  6. ANALYSIS: RESULTS • Examination of data (see supplementary scatterplots) rule out outliers/changes in marginal distributions as the reason. • For example, non-random assignment of students to school by level of achievement is one of the factors for measurement error bias. Yet no marked changes in stratification of schools based upon achievement was observed. • The increasing correlations observed are consistent with changes in percent proficient for schools between 2015 and 2016. Schools serving higher achieving students had, on average, higher rates of improvement than schools serving lower achieving students (see supplementary scatterplots)
  7. ANALYSIS: RESULTS • To investigate more broadly, we examined results associated with assessment transitions in 7 states: • 2 non-consortium affiliated states (including Georgia) • 2 PARCC affiliated states • 3 SBAC affiliated states • All summary statistics calculated using a 15 student minimum.
  8. CORRELATIONS: MEAN PRIOR ACHIEVEMENT BY MEAN SGP
  9. GA: SCHOOL LEVEL CORRELATIONS SIMEX ADJUSTED/MEAN PRIOR ACHIEVEMENT 2014/2015/2016 2014 (CRCT/EOCT) 2015 (Milestones) 2016 (Milestones) American Lit 0.28 (427) 0.18 (427) 0.5 (436) Analytic Geometry 0.34 (426) 0.37 (436) 0.45 (346) Biology 0.11(438) 0.21(449) 0.23 (447) Coordinate Algebra 0.23 (649) 0.23 (725) 0.21(246) Economics 0.09 (426) 0.05 (427) 0.29 (428) ELA 0.38 (1,745) 0.3 (1,760) 0.44 (1.742) Grade 9 Lit 0.22 (465) 0.25 (540) 0.36 (584) Mathematics 0.35 (1,744) 0.38 (1,760) 0.42 (1,752) Physical Science 0.22 (465) 0.25 (540) 0.27 (584) Science 0.27 (1,747) 0.3 (1,760) 0.39 (1,752) Social Studies 0.24 (1,747) 0.21(1,760) 0.34 (1,752) US History 0.03 (423) 0.06 (426) 0.18 (431)
  10. GA: SCHOOL LEVEL CORRELATIONS MEAN SIMEX ADJUSTED SGP TO PRIOR MEAN SCALE SCORE BY GRADE Mathematics ELA Social Studies Science Grade 4 0.18/0.23/0.26 0.23/0.09/0.29 0.09/0.10/0.18 0.18/0.20/0.32 Grade 5 0.11/0.13/0.20 0.11/0.11/0.24 -0.05/-0.10/0.10 0.06/0.06/0.13 Grade 6 0.27/0.18/0.24 0.28/0.22/0.25 0.08/0.06/0.15 0.16/0.18/0.20 Grade 7 0.35/0.22/0.29 0.23/0.36/0.32 0.10/0.07/0.08 0.04/-0.01/0.06 Grade 8 0.23/0.23/0.30 0.38/0.22/0.34 0.00/-0.07/0.07 0.08/0.06/0.21 Entire School 0.35/0.38/0.42 0.38/0.30/0.44 0.24/0.21/0.34 0.27/0.30/0.39
  11. NON-ALIGNED 1 SCHOOL LEVEL CORRELATIONS MEAN SGP TO PRIOR MEAN SCALE SCORE BY GRADE Mathematics ELA Grade 4 0.14/0.12/0.02/0.03 0.31/0.23/0.09/0.10 Grade 5 0.13/0.07/0.12/0.13 0.31/0.22/0.19/0.19 Grade 6 0.13/0.03/0.03/0.09 0.13/0.15/0.18/0.06 Grade 7 0.26/0.18/0.14/0.13 0.24/0.22/0.17/0.09 Grade 8 —/0.26/0.15/0.18/ 0.17/0.19/0.17/0.12 Entire School 0.26/0.26/0.23/0.26 0.34/0.40/0.33/0.21
  12. NON-ALIGNED 2 SCHOOL LEVEL CORRELATIONS MEAN SGP TO PRIOR MEAN SCALE SCORE BY GRADE Mathematics ELA Grade 4 0.06/0.10/0.10 0.08/0.19/0.20 Grade 5 0.09/0.12/0.18 0.12/0.19/0.18 Grade 6 0.13/0.18/0.15 0.15/0.11/0.23 Grade 7 0.24/0.17/0.27 0.19/0.28/0.31 Grade 8 0.13/0.35/0.32 0.17/0.26/0.33 Entire School 0.27/0.33/0.34 0.24/0.35/0.39
  13. PARCC 1 SCHOOL LEVEL CORRELATIONS MEAN SGP TO PRIOR MEAN SCALE SCORE BY GRADE Mathematics ELA Grade 4 0.32/0.25/0.18 0.50/0.40/0.28 Grade 5 0.34/0.27/0.30 0.42/0.25/0.28 Grade 6 0.34/0.20/0.25 0.35/0.24/0.21 Grade 7 0.20/0.24/0.17 0.33/0.26/0.22 Grade 8 0.31/0.31/0.21 0.30/0.15/0.12 Grade 9 —/0.14/0.13 Entire School 0.43/0.33/0.31 0.58/0.29/0.29
  14. PARCC 2 SCHOOL LEVEL CORRELATIONS MEAN SGP TO PRIOR MEAN SCALE SCORE BY GRADE Mathematics ELA Grade 4 0.11/0.19/0.12 0.23/0.11/0.16 Grade 5 0.08/0.11/0.21 0.10/0.03/0.18 Grade 6 0.20/0.16/0.26 0.15/0.06/0.22 Grade 7 0.28/0.18/0.19 0.17/0.17/0.19 Grade 8 0.15/0.33/0.37 0.26/0.17/0.17 Grade 9 0.21/0.20/0.25 Grade 10 0.25/0.26/— Entire School 0.34/0.25/0.39 0.33/0.24/0.29
  15. PARCC 3 SCHOOL LEVEL CORRELATIONS MEAN SGP TO PRIOR MEAN SCALE SCORE BY GRADE Mathematics ELA Grade 4 0.14/0.15/0.31 0.32/0.26/0.27 Grade 5 0.07/0.14/0.20 0.24/0.32/0.32 Grade 6 0.17/0.29/0.31 0.32/0.14/0.31 Grade 7 0.45/0.15/0.36 0.17/0.33/0.49 Grade 8 0.43/0.30/0.58 0.36/0.18/0.51 Grade 9 —/0.07/0.33 Entire School 0.43/0.37/0.53 0.32/0.25/0.45
  16. SBAC 1 SCHOOL LEVEL CORRELATIONS MEAN SGP TO PRIOR MEAN SCALE SCORE BY GRADE Mathematics ELA Grade 4 0.12/0.20/0.05 0.44/0.15/0.15 Grade 5 0.05/0.07/0.06 -0.05/-0.04/0.17 Grade 6 -0.05/0.15/0.14 0.04/0.23/0.11 Grade 7 -0.19/0.04/0.07 0.08/0.29/0.27 Grade 8 0.07/0.25/0.28 -0.10/0.22/0.12 Entire School 0.07/0.28/0.18 0.13/0.29/0.29
  17. SBAC 2 SCHOOL LEVEL CORRELATIONS MEAN SGP TO PRIOR MEAN SCALE SCORE BY GRADE Mathematics ELA Grade 4 0.14/0.02/0.15 0.17/0.15/0.19 Grade 5 0.24/0.08/0.04 0.34/0.14/0.17 Grade 6 0.25/0.05/0.08 0.41/0.14/0.11 Grade 7 0.15/0.03/-0.04 0.26/0.16/0.09 Grade 8 0.16/0.13/0.15 0.21/0.06/0.21 Entire School 0.40/0.16/0.18 0.32/0.22/0.26
  18. SBAC 3 SCHOOL LEVEL CORRELATIONS MEAN SGP TO PRIOR MEAN SCALE SCORE BY GRADE Mathematics ELA Grade 4 -0.05/0.14/0.00 0.10/-0.04/0.13 Grade 5 0.14/0.00/0.06 0.24/0.01/0.09 Grade 6 -0.10/0.21/0.21 0.32/0.11/0.11 Grade 7 0.22/0.23/0.25 0.30/0.25/0.35 Grade 8 0.10/0.28/0.11 0.09/0.26/0.30 Grade 9 0.36/0.21/0.20 0.30/0.07/0.20 Grade 10 0.25/0.08/0.26 0.20/0.18/-0.03 Grade 11 0.31/0.16/0.23 0.30/0.01/-0.02 Entire School 0.24/0.27/0.27 0.30/0.16/0.30
  19. CORRELATIONS: CURRENT ACHIEVEMENT BY PERCENT FREE/REDUCED LUNCH
  20. GA SCHOOL LEVEL CORRELATIONS CURRENT SCALE SCORE TO CURRENT PERCENT ED 2014/2015/2016 Mathematics ELA Social Studies Science Grade 4 -0.50/-0.77/-0.75 -0.56/-0.80/-0.78 -0.50/-0.72/-0.71 -0.53/-0.74/-0.73 Grade 5 -0.50/-0.72/-0.70 -0.58/-0.81/-0.79 -0.56/-0.73/-0.70 -0.56/-0.74/-0.72 Grade 6 -0.52/-0.70/-0.68 -0.53/-0.72/-0.71 -0.50/-0.61/-0.64 -0.49/-0.65/-0.64 Grade 7 -0.48/-0.72/-0.71 -0.51/-0.72/-0.74 -0.45/-0.64/-0.67 -0.47/-0.67/-0.69 Grade 8 -0.49/-0.67/-0.68 -0.55/-0.71/-0.69 -0.50/-0.62/-0.64 -0.50/-0.64/-0.51 Entire School -0.48/-0.69/-0.68 -0.52/-0.70/-0.68 -0.51/-0.65/-0.65 -0.52/-0.67/-0.67
  21. PARCC I SCHOOL LEVEL CORRELATIONS CURRENT SCALE SCORE TO CURRENT PERCENT ED 2014/2015/2016 Mathematics ELA Grade 4 -0.68/-0.67/-0.70 -0.78/-0.71/-0.71 Grade 5 -0.57/-0.58/-0.66 -0.74/-0.63/-0.65 Grade 6 -0.48/-0.45/-0.49 -0.62/-0.44/-0.43 Grade 7 -0.52/-0.46/-0.48 -0.61/-0.43/-0.43 Grade 8 -0.57/-0.25/-0.34 -0.55/-0.46/-0.45 Entire School -0.73/-0.49/-0.51 -0.75/-0.57/-0.57
  22. PARCC 2 SCHOOL LEVEL CORRELATIONS CURRENT SCALE SCORE TO CURRENT PERCENT ED 2014/2015/2016 Mathematics ELA Grade 4 -0.69/-0.58/-0.65 -0.72/-0.63/-0.63 Grade 5 -0.70/-0.64/-0.56 -0.70/-0.68/-0.57 Grade 6 -0.64/-0.64/-0.65 -0.64/-0.62/-0.56 Grade 7 -0.65/-0.55/-0.53 -0.63/-0.56/-0.51 Grade 8 -0.60/-0.37/-0.45 -0.68/-0.50/-0.53 Entire School -0.71/-0.64/-0.64 -0.73/-0.62/-0.62
  23. PARCC 3 SCHOOL LEVEL CORRELATIONS CURRENT SCALE SCORE TO CURRENT PERCENT ED 2014/2015/2016 Mathematics ELA Grade 4 -0.73/-0.68/-0.40 -0.74/-0.69/-0.43 Grade 5 -0.76/-0.57/-0.52 -0.66/-0.67/-0.45 Grade 6 -0.73/-0.43/-0.46 -0.66/-0.48/-0.43 Grade 7 -0.53/-0.72/-0.23 -0.51/-0.52/-0.35 Grade 8 -0.43/-0.40/-0.43 -0.54/-0.54/-0.56 Entire School -0.66/-0.68/-0.39 -0.74/-0.62/-0.49
  24. SBAC 1 SCHOOL LEVEL CORRELATIONS CURRENT SCALE SCORE TO CURRENT PERCENT ED 2014/2015/2016 Mathematics ELA Grade 4 -0.55/-0.57/-0.61 -0.66/-0.69/-0.69 Grade 5 -0.47/-0.60/-0.61 -0.63/-0.63/-0.59 Grade 6 -0.46/-0.60/-0.65 -0.48/-0.64/-0.63 Grade 7 -0.49/-0.40/-0.60 -0.52/-0.42/-0.58 Grade 8 -0.30/-0.57/-0.43 -0.39/-0.63/-0.44 Entire School -0.59/-0.65/-0.66 -0.71/-0.68/-0.68
  25. SBAC 2 SCHOOL LEVEL CORRELATIONS CURRENT SCALE SCORE TO CURRENT PERCENT ED 2014/2015/2016 Mathematics ELA Grade 4 -0.42/-0.47/-0.54 -0.34/-0.41/-0.53 Grade 5 -0.45/-0.46/-0.55 -0.52/-0.54/-0.60 Grade 6 -0.36/-0.39/-0.43 -0.38/-0.43/-0.50 Grade 7 -0.41/-0.53/-0.48 -0.36/-0.44/-0.57 Grade 8 -0.48/-0.49/-0.53 -0.40/-0.49/-0.49 Entire School -0.50/-0.52/-0.56 -0.50/-0.54/-0.60
  26. CORRELATIONS: MEAN SGP AND PERCENT FREE/REDUCED LUNCH
  27. GA SCHOOL LEVEL CORRELATIONS MEAN SIMEX ADJUSTED SGP TO CURRENT PERCENT ED 2014/2015/2016 Mathematics ELA Social Studies Science Grade 4 -0.17/-0.41/-0.36 -0.18/-0.19/-0.33 -0.14/-0.21/-0.34 -0.16/-0.20/-0.35 Grade 5 -0.07/-0.20/-0.17 -0.12/-0.15/-0.22 -0.11/-0.04/-0.15 -0.12/-0.10/-0.14 Grade 6 -0.28/-0.28/-0.25 -0.19/-0.22/-0.22 -0.14/-0.10/-0.21 -0.11/-0.18/-0.24 Grade 7 -0.13/-0.21/-0.30 -0.12/-0.30/-0.25 -0.05/-0.08/-0.17 -0.01/-0.06/-0.15 Grade 8 -0.09/-0.09/-0.16 -0.30/-0.19/-0.23 -0.11/-0.01/-0.05 -0.07/-0.05/-0.12 Entire School -0.14/-0.31/-0.29 -0.18/-0.18/-0.27 -0.17/-0.14/-0.26 -0.13/-0.16/-0.26
  28. PARCC I SCHOOL LEVEL CORRELATIONS MEAN SGP TO PERCENT ED 2014/2015/2016 Mathematics ELA Grade 4 -0.31/-0.29/-0.24 -0.53/-0.33/-0.26 Grade 5 -0.22/-0.09/-0.21 -0.40/-0.19/-0.10 Grade 6 -0.12/-0.07/-0.12 -0.39/-0.19/-0.10 Grade 7 -0.25/-0.08/0.00 -0.28/-0.04/0.02 Grade 8 -0.24/-0.01/-0.04 -0.11/0.00/0.00 Entire School -0.34/-0.13/-0.18 -0.49/-0.14/-0.07
  29. SBAC 2 SCHOOL LEVEL CORRELATIONS MEAN SGP TO PERCENT ED 2014/2015/2016 Mathematics ELA Grade 4 -0.19/-0.18/-0.14 -0.09/-0.31/-0.18 Grade 5 -0.16/-0.20/-0.17 -0.31/-0.31/-0.16 Grade 6 -0.20/-0.07/-0.08 -0.16/-0.14/-0.05 Grade 7 -0.11/-0.22/-0.06 -0.07/-0.22/-0.09 Grade 8 -0.05/-0.16/-0.12 -0.03/-0.19/-0.20 Entire School -0.21/-0.24/-0.17 -0.20/-0.31/-0.20
  30. CORRELATIONS: MISCELLANEOUS CORRELATIONS
  31. GA STUDENT LEVEL CORRELATIONS PRIOR SCALE SCORE TO CURRENT SCALE SCORE 2014/2015/2016 Mathematics ELA Social Studies Science Grade 4 0.79/0.81/0.84 0.78/0.78/0.82 0.76/0.77/0.77 0.79/0.79/0.79 Grade 5 0.78/0.82/0.84 0.76/0.80/0.83 0.80/0.80/0.80 0.79/0.79/0.80 Grade 6 0.79/0.80/0.84 0.77/0.78/0.84 0.78/0.79/0.80 0.80/0.80/0.80 Grade 7 0.85/0.86/0.87 0.76/0.79/0.84 0.82/0.83/0.84 0.82/0.82/0.83 Grade 8 0.81/0.84/0.80 0.77/0.78/0.83 0.80/0.79/0.82 0.81/0.79/0.78
  32. GA SCHOOL LEVEL CORRELATIONS PRIOR SCALE SCORE TO CURRENT SCALE SCORE 2014/2015/2016 Mathematics ELA Social Studies Science Grade 4 0.89/0.90/0.93 0.94/0.93/0.95 0.85/0.87/0.88 0.92/0.92/0.93 Grade 5 0.87/0.90/0.92 0.95/0.95/0.96 0.88/0.87/0.88 0.91/0.90/0.91 Grade 6 0.89/0.90/0.91 0.95/0.95/0.96 0.84/0.83/0.86 0.89/0.89/0.89 Grade 7 0.96/0.96/0.96 0.96/0.96/0.97 0.89/0.89/0.91 0.91/0.90/0.92 Grade 8 0.93/0.93/0.90 0.96/0.95/0.97 0.89/0.87/0.90 0.93/0.88/0.89
  33. ANALYSIS: CONCLUSIONS • Results suggest that increasing correlations are not associated with model anomalies but reflect actual changes occurring with Georgia schools. • The results suggest that in the second year of Georgia Milestones anchored to new achievement standards, schools serving higher achieving students are transitioning better than schools serving lower achieving students. • We welcome any advice on how we might continue to investigate this phenomenon to reach definitive conclusions.
  34. SCATTER PLOTS: MEAN PRIOR ACHIEVEMENT BY MEAN SIMEX SGP
  45. SCATTER PLOTS: MEAN PRIOR ACHIEVEMENT BY YEAR-TO-YEAR INCREASE IN GROWTH
  52. SCATTER PLOTS: MEAN PRIOR ACHIEVEMENT BY YEAR-TO YEAR IMPROVEMENT