Talks archive, 2020–2026 – Evolving Assessments: Implications for Educational Accountability

Evolving Assessments: Implications for Educational Accountability Challenges and Opportunities in Policy and Practice Dr. Damian Betebenner National Center for the Improvement of Educational Assessment This work is licensed under a Creative Commons Attribution 4.0 International License.

About Me • Recovering Academic • At the Center for Assessment since 2007 • Developed student growth percentile (SGP) methodology. • Currently live just outside of Boston, Massachusetts (Needham, MA) with wife, daughter, and dog.

Overview of Presentation • Background • Evolving Assessments • Implications for Accountability Systems • Equity and Decision-Making • Policy and Practice Implications • Conclusion and Discussion

Background

It’s not just semantics • Nitpicking over the meaning of words may seem trivial, but for complicated topics, it’s essential to understand exactly what we’re talking about. • In education, terminology carries weight because it shapes how we think, communicate, and make decisions. • Misunderstandings or differing interpretations of key terms can lead to misaligned policies, ineffective practices, or even public mistrust in the system. • For example, what do we mean by “assessment”? Is it a test, a portfolio, an observation? How we define it determines how we design it and what we expect from it.

It’s not just semantics • Over the last decade (and even further), the term "assessment" has expanded far beyond its traditional scope—a shift many of you have likely observed. • Historically, "assessment" primarily referred to standardized testing, often used for accountability purposes or to evaluate large-scale trends in student achievement. • The term, however, encompasses a wide range of tools and practices: formative assessments, performance tasks, classroom checks for understanding, adaptive assessments, through-course assessments, and even measures of non-academic skills like social-emotional learning. • I embrace this broader application of the term "assessment" as it reflects the reality of the diverse ways we assess in education.

So what’s the big deal? • Assessments, I would argue, regardless of what type, are point in time exercises in data collection. • The term assessment data often connotes something quantitative, derived through standardized procedures. • However, assessment data can encompass qualitative data as well. • As many of you are aware, current assessments often promise much more than just reporting data on what student know and can do. • Current assessments often claim to inform instruction. • Is this assessment or assessment use

Assessment & Learning • The term assessment has been and is currently used in some ways unfamiliar (and wrong) to me. A case in point: • The terms Assessment and Learning are appearing more frequently together: Assessment of Learning, Assessments to measure student learning. • Learning is an unobserved process that happens over time whereas an assessment indicates what a student knows and can do at a given point in time. • We infer that learning has occurred by changes in what a student knows and can do (which is what assessments, at their best, tell us). I do not think that assessments measure learning. • We make inferences about learning based upon observations (i.e., assessments, plural) over time.

Time Learning to walk Observation Time 1 Observation Time 2 Child can crawl Child can walk The combination of the two observations allows us to conclude that learning occurred where the child learned to walk. If the time span is short we say the “child learned quickly” emphasizing the fact that learning is something that occurs over time.

One last peeve • If I had a dollar for every time I’ve heard: “That assessment wasn’t designed for that purpose and therefore can’t be used that way.” • It’s the quick and easy way for “experts” to shut down the use of an assessment that they find questionable. • Yes, design is important, however, life is filled with examples of tools originally designed for one purpose being “validated” for others (my wife uses a screw driver to open tight jar lids) • And note, tools designed for a purpose don’t always work for that purpose (who remembers Clippy?). • That said, assessment use absolutely does need to be validated (regardless if it was designed or not designed for that purpose).

Evolving Assessments Having set the stage with the complexi es of assessment terminology and applica on, let’s now explore how assessments are evolving to meet the demands of today’s educa on landscape. ti ti ti

Innovation in Assessment • Assessments have evolved significantly in the last two decades: • Modality: Transition from traditional paper-and-pencil tests to online platforms, enabling greater efficiency, flexibility, and accessibility. • Adaptivity: Emergence of adaptive testing, where items are dynamically adjusted based on student responses, providing more precise measures of ability. • Automated Scoring: Advances in technology have enabled automated scoring for written responses, offering faster feedback and reducing human scoring biases. • Complex Item Types: Incorporation of interactive and multimedia-rich items that assess higher-order skills and mirror real-world problem-solving. • Intererim & Through-Course Assessment: Introduction of modular assessment designs that evaluate student progress throughout the course, rather than relying solely on end-of-course exams.

Accountability & Assessment • Due to the way in which NCLB inextricably linked assessment results and accountability, the two are often conflated. • Dislike of accountability systems often overlaps with dislike of the assessments that provide the data for the accountability systems. • Innovations in large scale assessment, particularly over the last decade, have been in large part driven by frustration associated with their (ab)use in accountability systems.

What’s the problem? • Criticisms of current large scale assessments include: • Delayed feedback: Results aren’t timely to impact instruction. • Lack of instructional utility: Results aren’t instructionally useful. • Unintended consequences: Teaching to the test, narrowing/altering the scope and sequence of the curriculum. High stakes and stress.

What’s the problem? • Is the problem the assessment or the way in which the assessment is used (or both in some measure). • The likely answer is both. • I’m of the opinion that the much bigger issue is the way in which the data derived from the assessments is used. • Accountability being the driving force for how data gets used is largely to blame for this. • I’m skeptical that the “right” assessment will correct the woes associated with inadequate or faulty data use.

Example: Through Course Assessment (TCA) • One of the latest ideas to improve large scale education assessment is with through course assessment designs. • Through-course assessment refers to a system where students complete multiple assessments spaced throughout the academic year, as opposed to a single, end-of-year summative test. • These assessments are designed to collectively measure the same standards and competencies typically addressed by a traditional summative assessment.

Example: Through Course Assessment (TCA) Modular Assessments • The curriculum is divided into units or instruc onal segments, each aligned to speci c standards or learning objec ves. • Students take an assessment a er comple ng each segment. Integrated Timing • Assessments are strategically scheduled at various points during the academic year, o en ed to instruc onal pacing. Cumula ve Measurement • Each through-course assessment contributes to an overall evalua on of student pro ciency, aggrega ng results into a nal score or report. • The system provides both forma ve (to inform instruc on) and summa ve (to measure outcomes) informa on. Diverse Item Types • Includes a mix of tradi onal (e.g., mul ple choice) and innova ve (e.g., performance tasks, wri en responses, or project-based tasks) items. • Emphasizes real-world applica ons and cri cal thinking. Alignment with Instruc on • Each assessment is designed to mirror the instruc onal content recently covered, ensuring close alignment with what students have been taught. ti ti ti tt ti ft fi fi ti ti ti ti ti ti ti ti ti ti ti ft ti ti ti fi ti

Example: Through Course Assessment (TCA) Con nuous Feedback: Provides mely data to inform instruc on, allowing teachers to address gaps in understanding throughout the year. Reduced Test-Day Pressure: Spreading assessments across the year reduces the high stakes and anxiety associated with a single end-of-year test. Enhanced Curriculum Alignment: Encourages teaching the full breadth of the curriculum by assessing standards throughout the year. Opportuni es for Deeper Assessment: Enables the inclusion of complex tasks that might be imprac cal in a single summa ve test. Support for Forma ve Use: Interim results can help students and teachers understand progress and adjust learning strategies. ti ti ti ti ti ti ti

Implications for Accountability Systems

What’s the problem? • Criticisms of current large scale assessments include: • Delayed feedback: Results aren’t timely to impact interventions and instruction. • Lack of instructional utility: Results aren’t instructionally useful due to curricular (mis)alignment, grain size of results, etc. • Unintended consequences: Teaching to the test, narrowing/ altering the scope and sequence of the curriculum. High stakes and stress. Shaming of schools. • Time allotted to testing and test-prep takes away from instruction.

What’s the solution? • To some extent, assessment design can solve some of these problems. • Delayed feedback: Use of technology to expedite scoring can greatly decrease turn-around time. • However, it isn’t clear that large scale assessment design will solve: • Lack of instructional utility: Results aren’t instructionally useful due to curricular (mis)alignment, grain size of results, • Unintended consequences: Teaching to the test, narrowing/ altering the scope and sequence of the curriculum. High stakes and stress.

Instructional Utility • Instructional utility has become a focus of many debates regarding assessment and accountability. • Two of my colleagues Carla Evans and Scott Marion wrote a book on the topic, Understanding Instructionally Useful Assessment • Whereas my colleagues take a more micro-view of instructional utility (useful for day-to-day instructional decision making), I take a more macro- view (useful for instructional decision making across some time frame). • Clearly, large scale, annual summative assessments are not useful for day- to-day instructional decision making. • However, across larger time frames, those tests can be used for more coarse grained instructional decisions.

Instructional Utility • Do current assessment designs solve the instructional utility problem? • Instructional utility is NOT a property of an assessment. It’s a function of how the assessment (if properly designed) is used. • I would argue that current assessments are not utilized even close to their full potential. • I’m skeptical that new assessments will change that. • TCAs still only occur 3 times a year — rather infrequent for instructional decision making.

Opportunity to Learn • Assessment designs that aren’t strictly summative (e.g., TCAs) involve components that are more formative/benchmark. • For example, an assessment might be administered early in the year to see where students are in terms of knowledge and skills (before they’ve been formally instructed) • How does accountability utilize student results where opportunity to learn is questionable?

Assessment as Intervention • Earlier I mentioned that, at their core, assessments are data collection activities. • Assessments, especially those claiming to be instructionally useful, are both data collection and intervention. • Without a THOROUGH explication of how intervention is coupled with the assessment results, I remain skeptical that new assessment designs will improve assessment results any more than previous annual summative designs. • Due to accountability systems that emphasize student attainment, our use of data to inform student learning is stuck.

Assessment as Intervention • As an intervention, such assessment systems should be evaluated with regard to whether student performance improves in comparison to other (e.g., annual, summative assessments) systems. • Given that new assessment systems often involve resetting performance standards, a dubious outcome would be to see higher student performance (e.g., % proficient) based solely upon new (possible lower) standards being established.

Equity & Decision Making

Equity in Education: A Shared Understanding • Definition: Equity in education means ensuring that every student has access to the resources and opportunities they need to succeed, recognizing and addressing disparities caused by systemic inequities. • Key Aspects of Equity: • Fairness: Providing what students need to achieve comparable outcomes, rather than treating everyone identically. • Inclusion: Ensuring all students, regardless of background or circumstances, have meaningful access to learning. • Responsiveness: Identifying and addressing the unique barriers faced by different student groups.

Equity in Education: A Shared Understanding

Equity as a Guiding Principle for Decisions • Impact of Equity on Decision-Making: • Shapes resource allocation (e.g., funding, staffing, and materials). • Informs the design and use of assessments to highlight disparities. • Ensures policies and practices are inclusive and culturally relevant. • The Stakes: • Inequitable systems perpetuate disparities in achievement, opportunity, and outcomes. • Equitable decision-making can close gaps and improve success for marginalized groups.

Challenges in Operationalizing Equity • Common Challenges: • Access: Digital divide or lack of resources for technology- based assessments. • Bias: Cultural and linguistic bias in test design or administration. • Opportunity to Learn: Holding students accountable for content they haven’t had the chance to learn. • Standardization vs. Individualization: Balancing the need for comparability with the need for personalized approaches.

Equity-Focused Decision-Making in Action • Practical Strategies to Promote Equity • Data-Driven Decisions: • Use disaggregated data to identify and address disparities. • Focus on root causes, not just symptoms, of inequities. • Inclusive Policies: • Include diverse stakeholders in decision-making processes. • Develop policies that prioritize underrepresented or underserved groups. • Iterative Improvement: • Regularly review assessment tools and policies for unintended consequences. • Adjust based on feedback and outcomes.

Policy & Practice Implications

Policy & Practice Implications: Framing the Discussion Why Policy and Practice Matter • Key Point: Policies shape the frameworks within which assessments are designed, implemented, and interpreted. Practices determine how those policies affect students and educators. • Objective: Explore how evolving assessments require shifts in policy and practice to maximize their potential.

Policies Driving Assessment Innovation Key Policy Drivers • Federal policies like ESSA (Every Student Succeeds Act) emphasize flexibility in assessments, including innovative designs. • State Initiatives: Many states are piloting competency-based or through-course assessments to move beyond traditional summative tests. • Equity Mandates: Increasing attention to ensuring that assessments are accessible and meaningful for all students.

Challenges for Policymakers Barriers to Policy Implementation • Scalability: Pilots of innovative assessments are difficult to scale while maintaining quality and equity. • Cost: Developing and implementing innovative assessments require significant investment in technology, training, and scoring. • Standardization vs. Innovation: Balancing the need for comparability across districts with the flexibility to innovate. • Example: Challenges faced by states implementing performance-based or through-course assessments under ESSA waivers.

Practical Implications for Educators Impacts on Teaching and Learning • Instructional Alignment: Ensuring that innovative assessments align with curriculum and teaching practices. • Professional Development: Teachers need training to interpret results and integrate them into instruction effectively. • Shifts in Practice: Moving from test-prep-centered teaching to a more holistic approach that emphasizes deeper learning and skill development.

The Role of Technology Leveraging Technology for Better Outcomes • Technology enables adaptive assessments, automated scoring, and real-time feedback. • Data systems must be robust to handle disaggregated reporting and privacy concerns. • Access equity is critical to ensure no student is left behind.

Accountability Reimagined Rethinking Accountability Systems • Shift focus from punitive to improvement-based accountability. • Use multiple measures of success, including formative and through-course assessments. • Incorporate growth metrics to evaluate student learning (as opposed to student attainment). • Use growth data to help identify what works as well as impediments to student learning.

Policy Recommendations for the Future Designing Policies for Evolving Assessments • Provide funding for innovative pilot programs and scaling successful designs. • Require equity audits to ensure all assessments meet accessibility and cultural relevance standards. • Foster collaboration between policymakers, educators, and assessment designers to create cohesive and coherent systems.

Moving from Policy to Practice Bridging Policy and Practice • Establish feedback loops where policy decisions are informed by classroom realities and vice versa. • Invest in professional learning communities to share best practices for assessment use. • Pilot innovative practices in diverse contexts to identify scalable solutions.

Conclusion & Discussion

Recap of Key Themes Reflecting on What We’ve Explored • The evolving landscape of educational assessments. • What an assessment does versus what is done with the results from the assessment. • The critical interplay between assessment design and accountability. • Opportunities and challenges in policy and practice.

Lessons Learned Key Takeaways • Assessment Design: Innovative approaches must balance validity, reliability, and accessibility. • Accountability Systems: Should shift from punitive to improvement-oriented frameworks. • Equity: Disaggregated data and inclusive practices are essential for addressing disparities. • Utility: Assessments are most impactful when their results TRULY inform (as opposed to potentially inform in some dream scenario) beneficial, educational decisions.

Looking Ahead The Future of Assessments • Innovations to Watch: Personalization in assessments, adaptive assessments, real-time reporting systems, the role of AI in assessment generation and reporting. • Policy Directions: Investments in scalable pilots, alignment with instructional goals, and increased focus on equity audits. • Role of Stakeholders: Collaboration among educators, policymakers, and assessment designers is critical.

Open Questions for Discussion • What would true innovation in assessment design look like • Would such an assessment be practical? • What does meaningful equity look like in large-scale assessments? • How do we ensure that assessments drive improvements in teaching and learning?

About this copy

Evolving Assessments: Implications for Educational Accountability — Massachusetts Educational Research Association, November 2024.

This is an image-based copy of the original Keynote slides; builds and animations are flattened.