What Am I Actually Doing?

AI-Supported Workflows in Educational Measurement

Damian W. Betebenner

National Center for the Improvement of Educational Assessment

NCIEA Staff Retreat · Brown-Bag · July 2026

The next 15 minutes

What a workflow actually is
What workflows look like in the wild — and where they get structured
How AI supports a workflow
Why workflows loop — and why the direction of the loop is everything

One question runs through the whole talk: what am I actually doing here? You can’t get help — from a colleague or from AI — with work you can’t yet see.

What is a workflow?

A workflow is just: how the work actually gets done

Not the org chart. Not the project plan. The real sequence of steps — clicks, copies, saves, sends — that turns a request into a finished thing.
For most of us it’s undocumented and lives only in our heads and our habits.
And it is almost never one tool. It’s a relay across many tools, with you carrying the baton between them.

If you’ve ever exported from one app just to paste into another, you were the connector between two steps of a workflow.

Workflows in the wild

A person labeled 'You' at the center, connected by tangled lines to app chips: Outlook, Word, Excel, Chrome, Teams, Zoom, PDFs, Slides, and R. Caption: you are the integration layer.

A typical day: one person hand-carrying work between a dozen apps, across laptop, phone, and browser.

The tell: coordination is invisible work

The apps are visible. The coordination between them is not.
It lives in memory, sticky notes, and “I’ll remember to update that later.”
It’s where things fall through: the stale number, the version that didn’t get sent, the step you redo every single time.

The reframe

The bottleneck usually isn’t any one app. It’s the hand-offs — and those are exactly what a human keeps in their head.

Where structure exists

Some fields wrote their workflow down

Software developers hit the same problem years ago: too many manual hand-offs, too many things falling through. Their answer was to make the workflow explicit and automatic.

A pipeline: saving a change triggers Test, then Build, then Deploy, each with an automatic green check underneath.

Every change runs the same named steps automatically, and each step reports pass or fail before anything moves forward.

The same idea, in three lines

when:  a change is saved
do:    run the checks
then:  publish only if ✓

Not real code you need to read — just the shape: a trigger, the steps, and a gate that won’t let broken work through.

What structure buys you: the workflow becomes named (you can point at each step), automatic (it runs without you remembering), and observable (you can see where it passed or failed).

Most of our workflows have none of these three. That’s the gap AI can help close.

How AI supports a workflow

First move: introspection, not automation

Before automating anything, answer the title question — what am I actually doing? Turn the sea of apps into a handful of named components and the connections between them.

On the left a scattered cloud of apps; an arrow labeled 'what am I actually doing?' points to five named, connected steps on the right: gather, analyze, draft, review, share.

Naming the components turns an undifferentiated pile of apps into a workflow you can actually reason about.

AI plugs in at the seams

Once the steps are named, AI helps inside a step and between steps:

Inside: draft the memo, clean the columns, summarize the PDF, write the R snippet.
Between: carry context from one step to the next so you aren’t the only thing holding it.

The role AI plays

Not a replacement for the human in the loop — a second worker you delegate pieces to, while you keep judgment, direction, and the final call.

A real example: going deeper, not just faster

In my own research I’ve been chasing a genuinely hard methods question — one I’d normally have to simplify just to make tractable.

With AI inside the workflow I could:

stress-test many more cases and assumptions than I’d check by hand,
formalize arguments more carefully, and
keep going down paths I’d usually abandon for time.

The surprising part

The value wasn’t mostly speed. It was reach — AI let me go deeper into the problem than I could have alone.

Depth is the real dividend

For a sea-of-apps workflow, AI’s first gift is clarity — seeing what you do. For a hard analytic workflow, its gift is depth — going further into what you do than time and working memory normally allow.

Same tool, two different wins, depending on where you start.
Both only land if you’ve done the introspection first — you have to know what you’re doing to hand any of it off.

Workflows loop

Good workflows feed back into themselves

Most real work isn’t one pass — it repeats. This quarter’s report is next quarter’s starting point.
What you learn in one pass can be fed back to make the next pass better.
That feedback is what turns a workflow into a loop — and loops compound.

Compounding is the whole game. A small improvement fed back every cycle grows; a small error fed back every cycle also grows.

The same loop runs both ways

Two loops. Left, a virtuous cycle in green spiraling up: do, learn, verify, improve. Right, a vicious cycle in red spiraling down: do, skip the check, error compounds, drift. Caption: same loop, what you feed back sets the direction.

The machinery is identical; only the thing you feed back differs. Checked learning compounds upward; unchecked output compounds downward.

AI makes the loop faster — in both directions

The risk

AI makes each pass faster and easier, so a vicious loop can spin just as fast: unchecked output feeds the next prompt, small errors harden into assumptions, and quality drifts while it still feels productive.

The safeguard is boring and it works: keep a human check on what gets fed back. Verify before you loop. The check is what points a fast loop up instead of down.

The efficiency paradox

The promise of automation is a dividend: time saved on rote work, freed up for more thoughtful, dedicated work.

But efficiency on a task can quietly invite more of that task.
Email is the clean case: AI helps you read, compose, and reply faster — so you send more. Your recipients, also using AI, reply more.
The loop clears faster and fills faster. The dividend gets spent on volume, not depth.

A dividend you don’t spend deliberately gets spent for you. Decide what the freed-up time is for — or the loop just manufactures more work.

This talk is a workflow, too

Everything you’ve seen came out of one attempt: build a repeatable workflow for turning ideas into talks — by asking the same question, what am I actually doing?

A pipeline: Idea, then Outline, then Draft and design, then a Review gate marked 'human check' with a green check, then Present.

The same shape as the software pipeline from earlier — named steps, AI helping inside each, and a human review gate before it ships.

Takeaways

Name your workflow. Ask what am I actually doing? until the sea of apps becomes a short list of steps.
Find the seams. The hand-offs between steps are the hidden work — and where AI helps most.
Put AI in the loop, keep yourself in it too. Delegate the pieces; hold the judgment and the final call.
Watch the direction of the cycle. Verify what you feed back — that’s what keeps a fast loop virtuous.
Expect it to feel different. Working with AI is a back-and-forth, not solo craft. If you love the craftsmanship of doing it yourself, the chatbot to-and-fro can feel uncomfortable — even off-putting. It’s a genuinely different way of working. It’s also, most likely, where the work is heading.

AI didn’t replace my workflow. It made me finally look at it — and then it let me go deeper than I could alone.

If you want to go further

Take this home: hand the talk to your own AI and let it interview you.

Step 1Give the AI the talk — paste this URL: github.com/dbetebenner/NCIEA_2026_Retreat/blob/main/ai_workflows.qmd

Step 2Give it this prompt, then paste it right after the URL:

I just watched a 15-minute talk, "What Am I Actually Doing? — AI-Supported Workflows." The slides are here: https://github.com/dbetebenner/NCIEA_2026_Retreat/blob/main/ai_workflows.qmd

Please read that presentation first so we share its language — workflows, the hand-offs (or "seams") between steps, the introspection question "what am I actually doing?", virtuous vs. vicious loops, and keeping a human check on whatever gets fed back into the loop.

Then be my thinking partner — don't hand me a plan yet. Start by interviewing me. Ask several questions, a few at a time, to help me surface what I actually do in a normal week: the tasks I repeat, the apps I move between, where work gets stuck or falls through the cracks, and what I would do with more time if I had it.

Once you understand my real workflow, help me:
1. Name it — turn my sea of apps into a short list of steps and the hand-offs between them.
2. Find the seams where AI could help most, both inside a step and between steps.
3. For each, suggest concretely how to use AI to get better outcomes — faster, higher quality, and more broadly shared — while I stay in the loop with a human check on what I feed back.

Keep it specific to me, ask before assuming, and tell me honestly if I'm describing a vicious loop rather than a virtuous one.