# Audit sheet: two-dimension recode (6 Oct 2026)

## 1. The 11 items where the two coders disagree (n = 98 sample)

| id | title | coder 1 | coder 2 | coder 1 rationale | coder 2 rationale |
|---|---|---|---|---|---|
| P008 | What Are We Assessing in Human-AI Interaction? (Keynote: Dragan Gaševi | product only | neither | Assessing learning when performance is produced through human-AI interaction in tasks. | Assessing human learning amid AI use; AI not product nor professional tool |
| P036 | Rethinking Pilot Data: Evaluating LLM Synthetic Data for Scale Develop | both | profession only | LLM synthetic respondents for early psychometric analysis in scale development. | LLM synthetic data for psychometric scale development analysis |
| P073 | Using Generative AI to Simulate Item Responses by Skills Insight Abili | both | profession only | GPT simulating item responses by ability band for early item evaluation. | GenAI simulated responses for early item evaluation before field testing |
| P138 | Introduction to AI for Measurement | both | profession only | Title only: introduction to AI for measurement. | Title only: introduction to AI for measurement |
| P143 | Building Responsible AI Practice in Language Assessment: Anticipating  | both | profession only | Title only: responsible AI practice in language assessment. | Title only: responsible AI practice in language assessment |
| P155 | Human–AI Collaboration in Educational Measurement: Transforming Assess | both | profession only | Title only: human-AI collaboration in measurement turning assessment data into action. | Title only: human-AI collaboration in measurement work |
| P233 | Measuring Collaborative Reasoning with LLMs | product only | profession only | LLMs classifying collaborative reasoning in student discussions. | LLMs labeling student discussion reasoning; coding of discourse data |
| P257 | S2A3: Thompson Sampling and Stochastic Exposure Control for High-Stake | product only | neither | Bayesian Thompson sampling adaptive testing with continuous calibration. | Bayesian adaptive testing and exposure control; no clear AI |
| P282 | The Analysis of Artificial Intelligence’s Decade-Long Impact on Educat | both | profession only | Content analysis of AI's decade-long evolution in educational measurement. | Synthesis of AI's impact on the measurement field |
| P283 | Transformer-Aided Detection of Gaming in Constructed-Response English  | product only | both | Transformer detection of gaming in automatically scored responses. | Transformer detection of gaming in constructed responses; scoring and QC |
| P304 | SarphieSense: A Pipeline-Based AI Chatbot for Navigating Living System | neither | profession only | Title only: AI chatbot for systematic review evidence; not assessment or measurement work. | Title only: chatbot navigating systematic review evidence for researchers |

## 2. Old 'product' (A) items now coded profession only (coder 1)

| id | title | rationale |
|---|---|---|
| P010 | AI-Enabled Quality Assurance for Multiple-Choice Assessment Items | AI-enabled quality assurance of multiple-choice items: AI as item reviewer. |
| P011 | Predicting Item-to-RPLD Matches with Structured AI Reasoning | LLMs classifying items to performance level descriptors to support assessment design. |
| P012 | Can AI Detect Item-Metadata (Mis)alignment? An Empirical Study Using State Asses | LLMs classifying items to standards for assessment quality assurance. |
| P013 | Achievement-Level Descriptor Conditioned Option-Selection Probability Modeling w | LLM-supported item evaluation predicting option selection and item difficulty. |
| P038 | Prior-Informed 3PL Calibration: Reducing Sample Size via Predictive Modeling | Predictive item-parameter priors to reduce calibration samples; AI implied by session. |
| P051 | Putting it Together: Using Embeddings Clusters to Identify Inconsistent Raters | Title only: embedding clusters to flag inconsistent human raters; operational quality control. |
| P056 | Exploring AI-driven Methods for Pre-calibrating Difficulty of Mathematics Items  | AI pre-calibration of item difficulty replacing field-testing labor. |
| P058 | What LLMs Capture and Miss About Cognitive Sources of L2 Item Difficulty | LLM zero-shot item difficulty estimates versus cognitive attributes. |
| P059 | Math Item Difficulty Prediction with Multimodal Input | Multimodal LLMs predicting math item difficulty. |
| P060 | Beyond Item Text: Modeling Latent Cognitive Demand for Math Item Parameter Model | Modeling cognitive demand to improve item parameter prediction. |
| P087 | LLM-Based Pairwise Judgment for Math Item Parameter Modeling using Workflows and | LLM pairwise judgments to recover item parameter scales. |
| P088 | LLM as Investigator and Judge in Pairwise Comparisons for Item Parameter Modelin | LLM investigator and judge for item parameter modeling. |
| P089 | Validating LLM Rating of Item Features for Explanatory Item Response Model . | LLM rating of item features for explanatory IRT difficulty prediction. |
| P107 | Partial Identification with Multiple Nonlinear Measurements of a Latent Regresso | Estimator combining disagreeing AI scores for downstream research analyses. |
| P108 | Demonstrating FairGAIte: An Agentic LLM Tool for Detecting Construct-Irrelevant  | Agentic LLM tool for fairness review of pilot items. |
| P110 | Using LLM Judges’ Paired Comparisons to Estimate Mathematics Item Difficulty | LLM paired comparisons to estimate item difficulty without field testing. |
| P111 | Evaluating Multiple Models for Predicting Item Difficulty in a Principled Assess | ML models predicting item difficulty in principled assessment design. |
| P112 | Consensus without Accuracy: Investigating LLMs Recovery of Item Difficulty Using | LLM pairwise comparisons recovering item difficulty for AI-assisted calibration. |
| P113 | When One Benchmark Hides Many Truths: Mixture IRT and LLM Difficulty Prediction | Validating LLM difficulty prediction against mixture IRT classes. |
| P117 | A Multimodal AI Analysis of Teacher Praise in Online Tutoring | Multimodal AI coding teacher praise quality in tutoring research data. |
| P118 | Breaking the Learning System Wall Using Multimodal Tutoring Transcriptions | AI transcription of tutoring sessions to build research logs. |
| P125 | AI-Assisted Rater Training Modules: Effects on Qualification and Scoring Quality | Title only: AI-assisted training modules for human raters. |
| P128 | Using Natural Language Processing to Explore Alignment Between Skill Taxonomies | NLP analysis of alignment across skill taxonomies. |
| P129 | Pre-Data Embedding Diagnostics for Construct Validity in Multi-Domain Instrument | Embedding diagnostics for construct boundaries in instrument development. |
| P130 | Auxiliary Information for Semantic Clustering of Assessment Items | Transformer clustering of operational items for blueprint and SME review. |
| P131 | Supporting Distractor Quality Review Through Interpretable Semantic and Lexical  | NLP diagnostic tool supporting distractor quality review. |
| P135 | Automatic Pre-Testing of Mathematics Items: predicting difficulty parameter with | ML automated pretesting to predict IRT difficulty for calibration. |
| P137 | Using XGBoost to Construct a Vertically Skill Difficulty Scale | XGBoost model building a vertical skill difficulty scale. |
| P145 | Interpretable Humans, Alien LLMs: Expert Analysis of Latent Structures in Assess | Factor analysis comparing human and LLM respondents interpreted by experts. |
| P149 | Predicting IRT Parameters for Passage-Based Reading Items Using NLP | NLP features predicting IRT parameters of reading items. |
| P150 | Predicting Item Statistics Using LLM-Derived Difficulty Ratings and Response-Opt | LLM-derived features predicting item statistics for certification items. |
| P151 | Anchored Bradley-Terry Calibration Using LLM Comparative Judgments | LLM comparative judgments for early item difficulty calibration. |
| P156 | Understanding Spatial Cognitive Process Using Time-Embedded N-Grams Model with M | Title only: machine learning analysis of spatial cognitive process data. |
| P165 | Hierarchical Attention Network Architecture for Automated Response Time Predicti | Deep learning predicting item response times from metadata. |
| P168 | Fine-tuning DeBERTa-v3 to Automate Spatial Language Classification | Fine-tuned DeBERTa replacing human coders for spatial language classification. |
| P187 | A Specialized Multimodal Transformer for Classroom Discourse Classification | Multimodal transformer classifying classroom discourse data. |
| P192 | Measurement-Informed Difficulty Priors for Competition Math Items | LLM features for item difficulty priors. |
| P197 | Lost Without Translation? Multilingual Sentence Embeddings for Linguistic-integr | Multilingual embeddings replacing translation in scoring reliability auditing. |
| P227 | AI-Assisted Evidence-Centered Design to Build a Blueprint for a Semiconductor Cr | Title only: AI-assisted ECD to build a credential blueprint. |
| P234 | An exploration of using an AI-based approach to shorten an assessment | AI shortening an assessment compared with expert-shortened form. |
| P244 | Does including images improve multimodal language models’ accuracy on item discr | Multimodal models predicting item discrimination. |
| P245 | Image-Based Representation of Item Response Patterns for Test integrity | Deep learning anomaly detection for test integrity. |
| P246 | Listening for Meaning: Evaluating an ASR-LLM Pipeline for Thematic Audio Analysi | ASR-LLM pipeline coding student audio themes to inform assessment development. |
| P265 | Beyond Latent Factors: Machine Learning Approaches to Response Clustering | ML clustering of survey responses as alternative to factor analysis. |
| P277 | Why AI Demands More Principled Assessment Design, Not Less | Title only: principled design practice for AI-assisted assessment design. |
| P279 | AI-Generated Distractor Plausibility Ratings as Predictors of Mathematics Item D | Title only: AI distractor plausibility ratings predicting item difficulty. |
| P280 | Predicting Item Difficulty: How PAD Features Shape Model Performance | Title only: PAD features in item difficulty prediction models. |
| P295 | Design Choices in Encoder-Based IRT-3PL Parameter Prediction for Passage-Based M | Encoder-based IRT parameter prediction for item calibration workflows. |
| P300 | Automated Item Evaluation: Predicting Item Acceptance and Rejection using LLM-Ge | Transformer models predicting operational item acceptance from critiques. |
| P303 | EquiFrame: An AI-based Program that Assesses Disability Language in Educational  | Title only: AI tool reviewing disability language in measurement instruments. |
| P309 | From Validation to Resilience: AI-assisted Enemy Item Identification in Operatio | Title only: AI-assisted enemy item identification in operations. |
| P311 | Predicting Race and Ethnicity DIF Using Generative AI in Large-Scale Assessment | Generative AI DIF screening supporting item review in test development. |
| P315 | Fairness Sentinel: An AI Agent for Early DIF Risk Screening | AI agent screening draft items for DIF risk before field testing. |
| P324 | Offloading the White Elephant: Assembling with AI-estimated IRT Parameters Inste | AI-estimated IRT parameters replacing pretesting in form assembly. |

## 3. The 6 norms disagreements (n = 98 sample)

| id | title | coder 1 | coder 2 | coder 1 rationale | coder 2 rationale |
|---|---|---|---|---|---|
| P154 | To Err is not only Human: Human/LLM Collaboration in Assessment | 1 | 0 | Panel on human/LLM collaboration and error in assessment work; implies division of verification responsibilities. | Human/LLM collaboration panel; no abstract |
| P173 | AI-Augmented Form Assembly from a Secure Item Bank: A Faculty Pilot | 1 | 0 | Faculty test builders using AI; examines their judgment boundaries in AI-assisted form assembly. | Faculty AI-assisted form assembly pilot; judgment boundaries tangential |
| P177 | Building Teacher Capacity to Generate and Evaluate Math Items with Sma | 1 | 0 | Professional development model building teacher capacity to generate and evaluate items with LLMs. | Teacher PD for AI item generation; teachers not measurement professionals |
| P277 | Why AI Demands More Principled Assessment Design, Not Less | 0 | 1 | Argues AI requires more principled assessment design; design methodology rather than professional norms. | Argues AI demands more principled assessment design practice |
| P282 | The Analysis of Artificial Intelligence’s Decade-Long Impact on Educat | 0 | 1 | Bibliometric synthesis of AI research trends in measurement; field content, not norms or roles. | Charts AI's decade-long impact on the measurement field; responsible adoption |
| P335 | PsyMAS: A Human-in-the-Loop Multi-Agent System for Auditable Test Secu | 0 | 1 | Human-in-the-loop test security system keeping misconduct decisions human; system design foremost. | Analyst adjudication, auditability; AI must not make misconduct decisions |

## 4. The 22 items coded norms = 1 (coder 1)

| id | title | rationale |
|---|---|---|
| P004 | Integrating Generative AI into R Workflows: From APIs to Shiny Apps | Frames a training gap for measurement professionals expected to adopt AI while upholding standards. |
| P007 | Writing an AI-Native Dissertation | Reimagines doctoral training: building an AI-native dissertation, a change in how researchers are formed. |
| P033 | Industry Leader Perspectives: How AI is Changing K12 Measurement Roles | Panel explicitly about how AI is changing K12 measurement roles. |
| P046 | Responsible use of generative AI when creating reading comprehension questions:  | Argues responsible-use practice: AI item evaluations should document prompts and generation steps. |
| P085 | Our Future with AI: Graduate Student Perspectives on a Changing Profession | Panel on how AI changes the profession, graduate training and career preparation. |
| P086 | A Jurisdictional Scan of Automated Scoring Practices in K-12 Summative Assessmen | Scan of automated-scoring practices across state programs; describes operational practice landscape of the field. |
| P109 | Human-Centered AI Applications: Bridging Learning Science, Assessment, and Ethic | Panel on governance choices sharing responsibility and authority between humans and AI in assessment. |
| P125 | AI-Assisted Rater Training Modules: Effects on Qualification and Scoring Quality | AI-assisted rater training and qualification; subject is training of the scoring workforce. |
| P139 | Human agency before and after Generative Artificial Intelligence in education, a | Human agency in measurement before and after GenAI; plausibly addresses professionals' authority and role. |
| P143 | Building Responsible AI Practice in Language Assessment: Anticipating What’s Nex | Panel on building responsible AI practice in language assessment; professional norms. |
| P154 | To Err is not only Human: Human/LLM Collaboration in Assessment | Panel on human/LLM collaboration and error in assessment work; implies division of verification responsibilities. |
| P173 | AI-Augmented Form Assembly from a Secure Item Bank: A Faculty Pilot | Faculty test builders using AI; examines their judgment boundaries in AI-assisted form assembly. |
| P177 | Building Teacher Capacity to Generate and Evaluate Math Items with Small LLMs | Professional development model building teacher capacity to generate and evaluate items with LLMs. |
| P189 | Interweaving Assessment: A Field-Generated R&D Agenda for AI and Measurement | Field-generated R&D agenda for AI and measurement; the field setting its own direction. |
| P221 | Implementing Generative AI Assessment Guidelines in a Mexican University | University policy process for GenAI assessment guidelines, transparency, and faculty development; governance of AI in assessment. |
| P225 | Educational Measurement as an AI-Native Profession | Panel on professional norms, training, accountability and standards of an AI-native measurement profession. |
| P238 | Trust Calibration in AI-Assisted Content Verification | How SMEs calibrate trust in AI verification concerns; professional review responsibilities. |
| P252 | Is AI a Foundational Competency or Does AI Elevate Aspects of Competencies? | Whether AI is a foundational competency for measurement professionals (NCME competencies). |
| P253 | Don’t Throw the BAIby Out with the Bathwater: Connecting FCEM to AI | Connecting NCME foundational competencies to AI; professional competencies. |
| P254 | AI Competencies in Education and Credentialing: Aligning Expectations with Evolv | AI competencies for measurement and credentialing professionals aligned with evolving practice. |
| P255 | Technical AI Competencies as Measurement Competencies | Technical AI competencies as measurement competencies; professional competency framework. |
| P310 | Responsible AI in Digital Assessment: Case Studies with the Duolingo English Tes | Responsible AI case studies at a testing program; governance of AI use in assessment practice. |