How AI reshapes the archetypal roles of the psychometrician

Position brief — Frank Rijmen, Cambium Assessment

AIME-Con 2026 panel, Educational Measurement as an AI-Native Profession, 7 October 2026

Marking: citable in public.

1. The argument in brief

Discussion of AI in educational measurement has focused on specific processes and products: item generation, automated scoring, and AI-generated feedback. Less attention has gone to how AI changes the roles of the people who do the work.

This contribution looks at three professional archetypes familiar from large-scale assessment: the Guardian, the Catcher, and the Architect. It argues that AI changes not only the knowledge and technical skills these roles require, but also their behavioral skill profiles: the dispositions that make someone good at the work.

AI shifts these profiles differently for each archetype. It also changes how the three depend on each other.

Many of these changes did not start with AI. Data science was entering psychometrics, data volumes and digital score reporting were growing, and QC was increasingly automated well before AI became a working tool. AI’s role is mainly as an accelerant. It speeds up shifts that were already underway, to the point where organizations can no longer adapt to them gradually.

2. The three archetypes

• The Guardian’s main objective is to safeguard the reliability, validity, and fairness of standardized tests. The Guardian treats the Standards as first principles and is skeptical of any innovation that cannot be justified within the established measurement framework.

• The Catcher is defined by operational vigilance. Detail-oriented, highly conscientious, and risk-averse, the Catcher focuses on what must not go wrong in calibration, equating, scoring, and anomaly detection. The Catcher works best in a “closed” environment, where tasks are well defined and predictable and the criteria for success are explicit.

• The Architect bridges psychometrics with data science and emerging technology, and is restless to redesign systems rather than run them. High on openness, the Architect loves thinking outside the box, bringing in ideas from other fields, and collaborating across disciplines.

These are archetypes, not job titles. Nor is the list exhaustive: there are others, such as the Communicator, who translates technical results for clients, policymakers, and the public. This contribution focuses on these three. Most psychometricians combine elements of several archetypes, and a healthy organization needs them in balance.

3. How AI shifts each archetype

In each row, the shift was already underway before AI; AI accelerates it.

	Before AI	With AI	Behavioral skills that gain weight
Guardian	Applies a stable framework; the main task is judging whether a new method fits it.	Must reckon with evolving standards. New methods arrive faster than the framework is revised, and the Guardian has to explain why a principle matters, not just enforce it. At a deeper level, AI’s ability to observe behavior across time and contexts challenges one of the framework’s founding assumptions (see below).	Receptiveness to new methods; the ability to explain principles and teach them; tolerance for ambiguity
Catcher	Checks outputs, often by hand, against known rules. Diligence and thoroughness carry the role.	Moves from box-ticking to forensics. The volume of outputs makes manual QC infeasible, so checks are automated or given to AI agents. The Catcher’s job becomes imagining what could go wrong and specifying what the agent must look for. This takes the Catcher out of the closed environment where they work best: the edge cases that matter are the ones no one has defined yet.	Anticipating failure modes; comfort with open-ended problems; skepticism; calibrated trust in automation (avoiding complacency)
Architect	Moves slowly from idea to working system; building is the bottleneck.	Goes from design to working prototype much faster. The bottleneck moves to making sure what is built is grounded in (evolving) standards and established measurement principles.	Self-skepticism; verification discipline; willingness to take responsibility for results one did not produce by hand

Across all three rows, AI pushes each archetype to take on some of another’s traits: the Guardian some of the Architect’s receptiveness to new ideas, the Catcher some of the Architect’s comfort with the undefined, and the Architect some of the Guardian’s and Catcher’s discipline.

A deeper challenge for the Guardian. Much of test theory rests on a practical constraint: we cannot observe a person all the time, across all the situations we care about. So we sample behavior under standardized conditions and infer an underlying trait that is assumed to predict behavior and success across situations. AI weakens that constraint. When behavior can, at least in principle, be observed across time and contexts, the need to infer a trait from a small, standardized sample is no longer self-evident. This goes beyond evolving standards: it questions an assumption on which the framework itself was built. It also raises questions of its own: observations made continuously over time, but with limited measurement precision on each occasion, need dynamic measurement models.

4. The unresolved problem

Developing behavioral skills may require a different approach than training cognitive skills. What should professional development look like when the goal is new ways of working, not new knowledge?

The deep subject-matter expertise psychometricians already have is the foundation for this. The question is how to build on it. This is hard for three reasons:

• Different kinds of learning: knowledge and technical skills are learned through instruction and practice. Behavioral skills, such as a Catcher’s comfort with open-ended problems or an Architect’s verification discipline, tend to develop through experience, modeling, feedback, and the working environment. Standard training formats are not designed for that.

• Learning by doing: qualities such as forensic skepticism are usually learned by doing the work by hand. If AI does more of that work, it is unclear how early-career psychometricians will build them.

• Learning from each other: each archetype already has what another needs. The Architect’s openness, the Catcher’s vigilance, and the Guardian’s grounding in principles can be developed through collaboration across archetypes, such as pairing, mentoring, and shared projects, keeping all three in balance.

5. Connections to the panel’s guiding questions

• Q1 (what has changed): carried by the provocation itself. Many of the shifts predate AI, and AI acts as the accelerant (section 1). What changes is not only the speed of the work but the roles and dispositions it requires.

• Q2 (what this demands of professionals): the most direct fit. Q2 asks which knowledge, skills, and dispositions define psychometric expertise, and how employers should develop and assess them. The unresolved problem in section 4 is a sharper version of that question: developing those dispositions may require a different approach than training cognitive skills.

• Q3 (what must be verified and disclosed): the Catcher’s shift to specifying checks for AI agents.

• Q4 (what measurement science owes the evaluation of AI): two links. The archetypes describe what makes a good human partner in a human–AI team, which Q4 names explicitly. And the deeper challenge in section 3 means measurement science has to supply models for what AI makes observable, such as dynamic models for many observations of limited precision.

• Q5 (where accountability remains human): the Architect’s and Catcher’s willingness to own results one did not produce by hand.

• Links to other panelists: Abulela (curricula: how graduate programs could develop these profiles); Betebenner (verification demands in AI-integrated workflows); Briggs (teaching graduate students to use AI without outsourcing judgment).

• Link to the discussant: Oswald. How people develop behavioral skills at work is core I/O psychology, and his field has long debated whether behavior is consistent across situations. That bears on both the unresolved problem and the deeper challenge.