An Investigation of Growth Norms using the Smarter Balanced Assessment

SBAC · September 26, 2018

An Investigation of Growth Norms using the Smarter Balanced Assessment — SBAC, September 26, 2018.
Published

September 26, 2018

Event
Invited talk
Slides
17
Format
Converted from Beamer

This deck is an image-based copy of the original slides. Builds and animations are flattened; the slide text is reproduced below for search and screen readers.

Slide text
  1. Background Current State Analyses Growth Norm Analyses References An Investigation of Growth Norms using the Smarter Balanced Assessment Damian W. Betebenner National Center for the Improvement of Educational Assessment Dover, NH Dover, New Hampshire September 26th, 2018 Damian W. Betebenner SBAC: Growth Norms
  2. Background Current State Analyses Growth Norm Analyses References Purpose Investigate and produce annual, summative assessment growth norms for SBAC ELA and Mathematics assessments. Investigate reporting metrics in use of results from growth norm results. Investigate fit associated with growth norm results, particularly issues related to misfit at LOSS/HOSS of assessment. Investigate properties of growth norms related to content areas and performance levels where students demonstrate low growth. Damian W. Betebenner SBAC: Growth Norms
  3. Background Current State Analyses Growth Norm Analyses References Timeline May, 2018 SBAC contracts with Center for Assessment to investigate growth norms using Smarter Balance Mathematics and ELA assessment data. July & August, 2018 Receive data from SBAC. Conduct growth norm analyses. September & October, 2018 Finalize report summarizing methodology and findings. Damian W. Betebenner SBAC: Growth Norms
  4. Background Current State Analyses Growth Norm Analyses References Why Growth Norms? The Smarter Balanced Assessment has a vertical/developmental scale allowing for cross grade comparisons of scores. The Smarter Balanced Assessment vertical scale allows for scale score gains to be calculated indicating magnitude of change for students from year to year. Gains are negatively correlated (-0.2 to -0.3) with initial status (larger gains for low achieving students) making interpretation difficult. Approximately 30 percent of gain scores are negative from 2017 to Growth norms obviate this negative correlation phenomenon so that growth comparisons between students are meaningful. Damian W. Betebenner SBAC: Growth Norms
  5. Background Current State Analyses Growth Norm Analyses References Growth Norms “Growth norms" are most familiar to lay persons from pediatrics where height and weight norms are provided for children conditioning upon their sex and age. In education, norm-referenced tests have a long history with norms provided for students conditioning upon their grade and content area tested. Growth norms are calculated conditioning upon grade, content area and one or more prior achievement scores. Norms are often reported in terms of percentiles but can equivalently be reported as z-scores or normal curve equivalent (NCE) scores. For this study, percentiles, z-scores and NCE scores were calculated for student growth scores. The following slides illustrate a growth norm using a single prior on the ELA assessment. Damian W. Betebenner SBAC: Growth Norms
  6. 20 co 17 0 S re S le ca le 0 2 ca S re 20 18 co
  7. 20 co 17 0 S re S le ca le 0 2 ca S re 20 18 co
  8. 20 co 17 0 S re S le ca le 0 2 ca S re 20 18 co
  9. th 0 Per ce nt i le G th 0 row 20 co 17 0 S re S le ca le 0 2 S ca S re 20 18 co
  10. Background Current State Analyses Growth Norm Analyses References SBAC states currently utilize different growth models, some growth norm based, others not HI, MI, NV, SD, VT, WA Student growth Percentiles (norm-referenced) CT, OK, USVI Transition table or variation thereof (hybrid norm/criterion-referenced) CA Residual Gain (norm-referenced) DE Value-added model (norm-referenced) ID Growth-trajectory using SBA vertical scale. (criterion-referenced) MT None Damian W. Betebenner SBAC: Growth Norms
  11. Background Current State Analyses Growth Norm Analyses References Analyses conducted Consortium wide growth norms were created for the following groups with associated N counts: ELA Grade 4 with Grade 3 as prior (639,519), Grade 5 with Grades 3 & 4 as priors (652,742), Grade 6 with Grades 4 & 5 as priors (635,850), Grade 7 with Grades 5 & 6 as priors (629,475), Grade 8 with Grades 6 & 7 as priors (625,474), Grade 10 with Grade 8 as a prior (66,552), and Grade 11 with Grade 10 as a prior (12,566). Mathematics Grade 4 with Grade 3 as prior (641,149), Grade 5 with Grades 3 & 4 as priors (654,041), Grade 6 with Grades 4 & 5 as priors (636,687), Grade 7 with Grades 5 & 6 as priors (629,821), Grade 8 with Grades 6 & 7 as priors (625,221), Grade 10 with Grade 8 as a prior (27,881, and Grade 11 with Grade 10 as a prior (5,105). Damian W. Betebenner SBAC: Growth Norms
  12. Background Current State Analyses Growth Norm Analyses References Analytic Details Method 1: Quantile regression is used to model the complete distribution of the response variable (current achievement) B-spline smoothing is uses to accommodate non-linearity and heteroscedasticity of the data of the multivariate data. See Betebenner (2009) for more detail on the model used to create the growth norms. Measurement error corrected norms were also calculated as part of the comprehensive analysis of the data [Shang et al., 2015]. Damian W. Betebenner SBAC: Growth Norms
  13. Background Current State Analyses Growth Norm Analyses References Analytic Details Method 2: Ordinal least squares (OLS) regression is used to model the distribution of the response variable using 1 or 2 priors. Residuals are calculated from the OLS conditional mean. Residuals can be reported in native scale score units, percentiles, standardized residuals, normal curve equivalents. Damian W. Betebenner SBAC: Growth Norms
  14. Background Current State Analyses Growth Norm Analyses References Analytic Fit & Results Method 1: All analyses converged yielding no analytic issues related to quantile regression estimation. With the exception of the Grade 11 growth norm calculations (where the number of students was small), model fit was extremely good for all norm-groups analyzed. Results showed variation among states that will be detailed in the final report. Damian W. Betebenner SBAC: Growth Norms
  15. Background Current State Analyses Growth Norm Analyses References Analytic Fit & Results Method 2: All analyses converged yielding no analytic issues related to OLS estimation. Model fit was generally good with consistent misfit at the low and high end of the distribution due to OLS assumption of multi-variate normality. Results showed variation amongst states that will be detailed in the final report. Damian W. Betebenner SBAC: Growth Norms
  16. Background Current State Analyses Growth Norm Analyses References Conclusions Growth norm are used by several SBAC states. Smarter Balanced Assessments had suitable N-sizes, Deviations from multi-variate normality at the bottom/top of the distribution led to some misfit with procedures assuming a linear relationship. Several reporting metrics are possible for norms. Percentiles are the most frequently. Percentiles can be easily converted to scale score gains, standardized scores (z-scores), normal curve equivalents. From an interpretation perspective, percentiles are most broadly understood. Depending upon desired use cases, growth norms can also support criterion-referenced outcomes via growth-norm based projections. Depending upon desired use cases, growth norms can also be produced for within year interim assessments taking account of prior achievement and time lag between assessments. Damian W. Betebenner SBAC: Growth Norms
  17. Background Current State Analyses Growth Norm Analyses References References Betebenner, D. W. (2009). Norm- and criterion-referenced student growth. Educational Measurement: Issues and Practice, 28(4):42–51. Shang, Y., VanIwaarden, A., and Betebenner, D. W. (2015). Covariate measurement error correction for student growth percentiles using the simex method. Educational Measurement: Issues and Practice, 34(1):4–14. Damian W. Betebenner SBAC: Growth Norms