Review item parameters and total-score correlations before Batch 3

The advice. “What you are using is real item parameters from the test. The parameters are different for the subtests. Before you do batch 3, review the item parameters and correlations between total scores.”

Why. Raised in response to a committee question about why Batch 1 conditions produced such similar results while Batch 2 produced noticeable differences — was that a property of the methods, or of how the items relate to their subcategories?

The simulation draws true theta from a mixture model but uses operational item parameters, which differ by subtest. So subtest-level differences in the results may be an artefact of the item pools rather than of the classification methods being compared. Until the item parameters and each subtest’s correlation with total score are characterised, the comparison confounds method with instrument.

A related committee question sharpens it: “Why is life science different than the others; are other subtests more related to the total score?”

Status. Open. Explicitly framed as a precondition for a third batch.