Transcript (partial): Educational Measurement as an AI-Native Profession
AIME-Con 2026 · Panel · Wednesday 7 October 2026, 11:00–12:30 EDT · Commonwealth 2, Wyndham Grand Pittsburgh Downtown
Machine transcription (speech to text) of the room audio, unedited. It covers 10:53:29–11:43:55 EDT. Nothing after 11:43:55 was captured: the chair’s laptop, which carried the capture, was disconnected then so the discussant could connect his own. The discussant’s remarks (Fred Oswald) and the audience questions are therefore not in this transcript. Times are approximate.
Speakers are identified from the run of show (the chair’s framing and his provocation, then Derek Briggs, Frank Rijmen and Mohammed A. A. Abulela) and the timing of the machine’s speaker labels. Short interjections the timing cannot place are marked “Unidentified”. Before the session began (11:00) the room was settling in.
[11:00:04] Damian Betebenner: Good morning. Thank you for coming to this session. For the next… What do we have, ninety minutes? Yeah. Hour and a half, we’re going to be talking about, education measurement as an AI native profession.
[11:00:25] Damian Betebenner: Oh,
[11:00:27] Damian Betebenner: what is this conference about? I dragged the PDF of the conference into an AI agent to try to come up with some distinctions between what is actually being discussed at this conference. And a lot of sessions, as you know, are address tools that we’re building with AI that address measurement problems, things like automated item generation and things like that. And as you can see from this bar chart, the majority of presentations are about that. Some of the sessions are about uses of AI in your own work, and a small percentage talks about what AI means for the profession. Sort of the norms, roles, training, and accountability that are associated with that. We’re going to be talking about the latter. So I’m gonna start with just a I guess, being the chair of my own provocation. I believe that AI is going to have a much greater impact on what we actually do based upon us as measurement professionals than the little the tools that we actually make to conduct measurement with. And I would go on to say that either now or in the very near future all of the big breakthroughs that are going to be coming in educational measurement are going to be based working with AI to actually drive the field forward. And if that’s the case, that means that the people in this room that are doing this need to become expert of AI to drive their work forward.
[11:02:15] Damian Betebenner: So
[11:02:17] Damian Betebenner: the idea behind this session came from a chat that I’m a part of with Derek and Andrew Ho, and it was based upon something I saw that I think came out in June called the Leiden Declaration on AI and Mathematics, and you can look it up, but if you’re interested in mathematics you know that AI is really pretty much disrupted mathematics, and that if you’re a professional mathematician, a theoretical one, that generally sort of proves things, I don’t see how you can work in that field any longer and not basically have an AI agent on your desktop. The idea that you basically are gonna get coffee and some blank paper and go and write out proofs is just not a viable way to work in that profession any longer. And my suspicion is that’s going to be happening fairly quickly in our profession as well. So there’s five guiding questions that sort of going to discuss today. So what has changed in the substance and organization of work in educational measurement? What that demands of us as professionals, division of labor, expertise and training, what the profession must redefine, what is verified documented and disclosed, what measurement science owes to the evaluation of AI, including this panel, as we think about what we’re actually doing here today, how do we sort of conduct a evaluation of it? And where authority and accountability remain human and on what grounds. So to try to illustrate this notion of AI nativeness, what I have been dreaming of for a while and I just decided that since I have AI and I’m pretty good at using it, I’m going to sort of vibe engineer sort of an application. And I like the idea that as we get together as professionals, we try to capture what actually happens in this moment and discussions with all of the documentation, and we build an environment in which AI actually helps us do this at conduct our business here in a sense. That being AI native can be a single person thing but it can also be a group thing and so I’m very interested in sort of understanding how we as professionals collaborate AI and build it into the things that we actually do. So how is Lane the AI agent that’s participating in this taking part? So the idea isn’t to just have this little chat bot that’s running amok and and talking about things.
[11:05:06] Damian Betebenner: In fact, one second here.
[11:05:25] Damian Betebenner: So part of the design process behind building this app was to think how do how do we want an agent to actually participate in these discussions amongst professionals? My goal isn’t to create something that says, oh, look at how smart the AI agents are and how sort of maybe not so smart we are. The idea is to use it to in a sense interrogate our ideas that we’re putting forward. To ask us questions about what’s going on to find commonalities that we actually might miss. And so working with the AI, building out design specs, you know, one of the things that you can do with AI with advanced skills is just start building web applications. There’s all sorts of tools that you can plug in and essentially you put a credit card on there and start buying API all of a sudden you have sort of working applications. And this is available to anybody in this room. So this is sort of the future of what’s possible right now and only gonna get easier and easier to do this. And as we conduct ourselves as measurement professionals this is what people can do. How do we thrive in an environment in which this is possible?
[11:06:41] Damian Betebenner: So,
[11:06:42] Damian Betebenner: this was set up so hopefully it’s actually transcribing things in the background. Right now through this. I’ve spent a lot of time having ins and outs because eventually the transcriptions then get sent to an AI and come back as text to speech so that it can actually talk back. And conduct itself in this. A way you can think of this maybe is like something from like the Marvel movies like Jarvis, where you have this entity so to speak that you can interact with and it’s basically giving you ideas and throwing things back.
[11:07:19] Damian Betebenner: So,
[11:07:20] Damian Betebenner: before we start, this session is being recorded and and I hope people aren’t sort of like, opposed to that. I think it’s a public meeting more or less so that’s not a bad thing. And I have not tested this out but if you go to the top QR code, I think you can actually go in and ask questions. And ultimately I think what’s going to occur hopefully a company comes along like Zoom and just makes this so it’s an easy plug and play thing so it becomes part of our natural way of working. Where we spin up things and when we get together to actually present these materials, we have an AI there to help us, like I said, interrogate them. So we’re gonna have a lot of time for questions, After I’m done framing, we’re gonna have four or five minute provocations. Then an AI synthesis, then Fred’s gonna come up and be the discussant, Then we’re gonna have a panel and audience dialogue, and towards the end, try to come up with some norm building activities in terms of how we force feed our profession in this AI era.
[11:08:43] Damian Betebenner: Okay. So where are we? Eight minutes after. Okay.
[11:08:52] Damian Betebenner: Okay.
[11:09:06] Damian Betebenner: Any questions? Okay. So I’ll start my… With my provocations. I asked this group to be here because I wanted a broad variety of people that come at the measurement profession from different angles, from industry, from academia, academia, from consulting, because we work in different ways even though we work on the same topic, but ultimately I wanted to sort of get a broad variety of people to say how they think it’s going to change their professions. I’m a consultant and some of you know me as the person that invented SGPs and that’s been what I’ve been doing for almost twenty years and AI is completely changing what my job is. And in exciting ways and in frightening ways. So AI isn’t just helping me do my old job faster, I think a lot of people approach this as sort of a productivity tool that oh, this will allow me to write emails and do things. It’s really changing what my job is and going much deeper than I ever could before. In general problems that used to require teams of specialists and just remained unexplored because it required too much effort or too much time really are within the grasp of people now with AI. Many of you know the term rabbit holes, where you get an idea and you think oh I better not go down that rabbit hole because I’m gonna end up burning a whole afternoon. I’m much more open to going down rabbit holes now with AI because I can actually get somewhere very, very quickly. And then ultimately make judgments about whether it’s good to go further down the rabbit hole or just say, that’s probably not much low hanging fruit there to pick. So how has my work changed? Before AI and with AI? In terms of theoretical development, previously a lot of derivation is slow and locally constrained. You know, I’m pretty good at math, but working through long mathematic equations takes a lot of time. And it’s error prone and you have to check things. AI is extraordinarily good at math. And in fact I just saw something on, Twitter just now that OpenAI decided to release their vault of all of the unsolved math problems that they’ve been sort of holding on to. They’ve gone through the field of mathematics and essentially solved some of the biggest problems that are out there. So it’s better than humans at math. May not be better at all things but at math it’s really, really good and it’s helped me tremendously. In seeing relationships that I had… Would never have been able to
[11:12:16] Damian Betebenner: before. Code.
[11:12:20] Damian Betebenner: AI has completely revolutionized coding. Most developers now never write code by hand any longer. When you talk to people and that’s a paradigm shift for that My belief is that ultimately it’s we’re also going to see it in terms of handwritten text. That this idea that we sit down at a typewriter and and chisel out on a page is going to be like the Flintstones to the Jetsons. You’re going to go in and explain conceptually what you want and the AI will do the writing for you. Simulation allows you to run all sorts of complex simulations, sets up pipelines, sets up databases to do things. Critique excellent at critiquing things, setting up adversarial reviews, It’s amazing at documentation. It’s one of the veins of a coder’s existence of documenting their stuff. AI is really, really good at that. And my role before was producing these technical artifacts and now it’s really to sort of specify what you want done, be able to interrogate, verify, decide, ultimately take responsibility.
[11:13:34] Damian Betebenner: And
[11:13:35] Damian Betebenner: some of you I’ve told about the recent work that I’ve done with copula based growth. So that was really just I’d like to take responsibility for it but it would probably be I may have started the initial prompt and kept prompting, but most of the insights actually came from having the AI map out the mathematics, me learning it was saying, and then interrogate it, having it explain more to me. And previously that would have taken, I wouldn’t have been able to do it, but years even if I had ability to do it, and now it’s something that can be done in months. So what’s become scarce Mathematical derivations are cheap, code production and factoring, simulation, documentation, critiques. The more valuable skills now are problem formulation, model adversarial testing, recognizing nonsense, these are skills that many in this room have because we have very high levels of metacognition. And that allows us to work effectively with AIs. The unresolved problem the question isn’t whether you can do the analysis, it’s whether I can warrant that the analysis that AI has made possible. And mathematicians are reaching this level of where they get a proof Math is interesting because they have an algorithm that can check proofs for correctness called lean. So in a way you can know that the proof is right even if maybe you don’t understand it, and that’s part of their peer review process. So you get a proof that AI produced that says it’s right and you’re an expert in the field, you spend weeks just trying to understand what the AI has actually done. And it it does things that are going to be defensible beyond our ability to comprehend. It’s already happening in mass and like I said, I think the empirical sciences are going to be swept up in this quite soon. Okay. So I’ll turn… That’s my provocation.
[11:15:52] Damian Betebenner: Who’s next on the list?
[11:16:03] Damian Betebenner: Derek.
[11:16:09] Damian Betebenner: So Derek is now talking. Eventually, if things get set up, you have mics and the AI knows who the mics go to so that the transcription is actually associated with with authors, which allows it to be even more personalized.
[11:16:26] Derek Briggs: What’s up Lane? Derek Briggs speaking here. Treat me with respect. So I have my provocation as something very, very concrete. Which is I want to argue that peer review should not only use AI but it should use AI twice. Once for the reviewer, once for the editor, both by the journal. I wanna just give you quick my street cred on this. I have been a journal editor. I have reviewed hundreds of journal articles. I have written many, many, many journal articles. I’ve been in all stages a part of the process and peer review process was in a state of peril ten years ago. It’s only getting worse and we need to sort of think very carefully about what we’re gonna do it. So review a strain before generative AI There in 2025 there were 8,000,000 scientific publications, double the count from five years earlier. In general, there are 4.5 invitations per completed review in 2025. Again, twice the 2018 figure. And 53% of reviewers used AI per review task in 2025, up from 29% a year before. So it’s already happening. Right? People are… Reviewers are using AI. However, they’re not supposed to. No journal, if you actually look, sets its own reviewer rules. So I actually had Claude do an inventory and search through 14. And if you look at my slides, have a QR code later. I have another document where I looked at 14 journals that published work in psychometrics, education measurement. 13 of the 14 journals require AI, authors that is, to disclose generative AI use. The only one that doesn’t just doesn’t say. Doesn’t have a specific policy. 12 of the 14 bar reviewers from putting manuscript text into an AI tool. And the two that don’t, the that are not part of that, again, don’t say you can, they just don’t have a policy for it. And zero of the journals of the 14, set their own reviewer or editor rules. They really just go back to what the publisher argues and they don’t really seem to have really thought this out at the journal level. And so the stated reason not using AI in the review process is confidentiality. It’s not AI quality. Yet publishers typically in the… As a first stage of their, system, their pipeline, typically use AI as a quick check on the journal submission So there’s an inconsistency there. There’s actually AI being used by the publishers at one end. We know that authors themselves are actually using AI on their manuscript and they just need to disclose it. And yet on the review side of things purportedly because of confidentiality confidentiality, reviewers are not supposed to be using AI as part of the review process. So here’s my proposal. Two uses both run by the journal. Use one, for the reviewer. Review first then audit the AI reviews. The idea I would have is that a reviewer still, I I’m not at all trying to advocate that humans shouldn’t be involved. In the peer review process. But I think it should go like this. Have a human write an independent review then read the journal’s AI review of code. So before you submit, before you’re allowed to submit, you click a button, it generates the AI review. Before you submit your review, you have to read the AI review. Then you have the opportunity to adjust your review and explain to what extent your review has been adjusted after seeing the AI review. Right? And you tell the editor how that happened, you document all that. Use to would be for the editor. Ask the AI how the human reviews fit together. So the editor would receive the human reviews and then ask the AI whether the reviewers agree, conflict, and what no one decided. And decide then with the AI report as one input how to write a coherent letter to the the author that submitted the manuscript. So in both cases, humans write a review, they make all the decisions, but AI is playing a really critical supporting role. We actually have some evidence from a randomized trial that when reviewers receive AI feedback actually it makes things better. 20,000 human reviews got optional AI feedback on their draft. 27% of the reviewers who got feedback revised using 12,000 plus AI suggestions And two out of three of the revised reviews were judged better by independent panel. Than before they had the AI feedback. So there’s a caveat on that. The AI commented on the reviewer’s drafts, not the actual manuscripts. Right? We still don’t know everything about this. But to me, is at least least compelling suggestive evidence that using AI in the review process can make things better. Whose judgment is it? For accountability and this is somewhat resolved. When a reviewer revises after reading the AI’s critique, who answers for the review? Right? So there’s no journal policy yet. Our field that covers a review the journal itself supplies. Then there’s questions of independence. If every reviewer reads the same AI the reviews are no longer independent ratings. The editor may see more agreement but have less independent evidence. I’m not so sure that’s a bad thing in all cases. And so just the last a last point I’ll just can I, do I, does this look good? No. Oh, I guess I thought I had another slide with my QR code. But check my LinkedIn post that talks about this thing and has a link to both the slides and then my investigation of the 14 journals. And what their policies are. I just want to say this, which is that there is nothing at this stage in my career, nothing that I write that has any consequence or stakes attached that I don’t submit to AI for a critical review. Because every time I do, I learn something really important about it, a thing that I miss It catches a course, any simple logical errors, mathematical errors, things like that immediately catches. But it’s often giving me actually deeper insights that I can address and I can address them pretty well. So I think that the argument here about when we come compare an AI review to a human review, the question is not whether the AI review is better than any possible human review. But will it be better than the median review? I’m quite sure that it would.
[11:22:35] Damian Betebenner: Thanks, Derek.
[11:22:37] Damian Betebenner: Okay. Next, I think, is Frank. So I should just get all these up.
[11:23:00] Damian Betebenner: Thanks Damian. So, I’m Frank Raymond, Committee of Assessments.
[11:23:04] Frank Rijmen: Representing the perspective from a large testing company. And I wanna focus on today is, I mean, as Damian said, at the conference, we’ve been hearing a lot of presentations about item development, passive automated scoring, some of the processes, workflows, pipelines. And I know like gonna have to come up, we gonna have to make sure that everything is grounded in research, that there are guardrails and all of that. But, what I wanna talk about today is what about the people who are working at testing company. I have about 100 psychometricians, data scientists, stat analysts, measurement scientists in my team and I think that with advance of AI, it will not just change what they are doing, but also how they are doing and what behavioral skills are needed to be successful in the job. So I can Next page.
[11:24:13] Frank Rijmen: Another Nancy. Oh, the… Right. Alright. Yep.
[11:24:17] Frank Rijmen: So, I’m gonna focus on three archetypical roles, just, these are not this is not exhaustive. These are just three roles I thought of and most psychometricians combine a kind of a mixture of these roles every testing organization also need needs them in balance. So, the first role, and I’m sure if you can think, I mean we all know, we can think of people who are, like, prototypical for those roles. So, the first one is the guardian, that’s typically like a senior person. They they pay their dues, working on a lot of assessments, sometimes became a little bit jaded, and they see their role as most
[11:24:55] Frank Rijmen: mostly
[11:24:56] Frank Rijmen: safeguarding the reliability, validity, and the fairness of the So, that the standards are treated as a first print and anything new and innovation is judged against that framework. Now with AI, and I will talk a little bit more about that in a minute, there is they must reckon with evolving standards. Right? Standards are kind of the crystallization of measurement theory and of best practices in the industry. As the industry is changing a lot and and the and use of AI questions some of the principles of our measurement theory I think the Guardian has to kind of shift focus from defending to explaining. Then the catcher, that’s the your typical operational psychometrician, very very detail oriented, as my friend Richard Roberts personality psychologist would say, you guys are whole… All high end, high c, meaning very conscientious, but also a little bit anxious when things are open ended. So, these are the people who who love, who work best in a a predictable environment, like think of the checklist, or a subset of processes to go through, another person will do the same thing, come up with the same conclusion and then the scores go out. So, it’s kind of trying to to prevent scores falling off the cliff. It’s constantly affording disaster. And then, the architect, that’s the kind of the innovator, the person who kinda looks across the field, works with data scientists, other fields, and and basically, he wants to redesign the systems rather than run them. So, he gets bored with operational work, always looking for something new, and I think with AI, I mean, it’s kind of kid in a candy store, I mean, there’s like all the tools available the challenge is gonna be, to some extent, like how how do we keep all those kids from, like, making too much noise, and how do we make them play together, and have their stuff grounded in measurement theory. So again, the Guardian, and so now I want to focus a little bit more on the behavioral skills and my claim is that they shift differently for each archetype. So, Guardian, I think, before AI, it was more like it was a person who would focus on adherence, which judge fit to a stable framework and would mostly interact with junior psychometricians. I would basically go around and say, oh, what are you doing? Well, know, have you thought about this? You can’t really do that. And I think right now, I mean, with all the changes coming our way, with a lot of assessment happening outside traditional testing companies and a lot of new players there. It’s it’s gonna communication, is gonna be more important, so not just judging whether or not something adheres to a principle, but being able why a principle is important. And so some tolerance for ambiguity. I think there’s also a bigger challenge for the garden, which I will talk about a little bit later. So the catcher before very diligent thrown us in checking out foods, I think the switch there is gonna be more becoming rather than a traffic cop becoming like an FBI agent, right, because a lot of the QC is automated, can be further automated, but the new role for the catcher is gonna be, like, trying to think of possible scenarios where things can go wrong and then work with an agent to check for those things. And so, what is needed in terms of behavioral skills is being more comfortable with open ended problems like find a way to to manage that anxiety inducing part of the of the job. And, kind of a trusting automation. A lot of catchers, they still work with… They have that code on their laptop, and they worked on it. For a long time, and if a new person comes in, the step one is to kinda replicate that code. I think that that’s that’s all changing now. The architect, where before the bottleneck really was getting things built, so kind of, in a way, one would say that the wind track clipped a little bit because mean, people would say, well, yeah, you already… Sounds great. Show me a prototype. And then, it would take a lot of effort and involvement of other people to actually come up with that prototype. I mean, now it’s all easy. You can do it
[11:29:39] Frank Rijmen: a
[11:29:40] Frank Rijmen: a couple of days, but I think, what is important is kind of verification, discipline, also owning results one need not produce by hand. I mean, can’t say, well, know, for I didn’t really build it. It was AI. And I think also, making sure, like, seeing the bigger picture, so it’s not just about building little things but there’s like an overarching system that things should fit into and and and we do we do operate in a high stakes context so grabbing a dataset from the Internet and some code from Hugging Face, Showing that it works is is is not really a prototype, I think that or a or a proof of concept, there is more work involved. And so that discipline, I think, is gonna be important like, not getting bored too quickly, but hanging in and doing all the work that sometimes can be nitty gritty and and and boring. So, with respect to Guardian, I think a deeper challenge is like some of what is possible with AIs to some extent, question some of the basic principles that led to our motion… Modern measurement theory, right, because the original idea was, well, we cannot observe a person all the time across all situations, so we’re gonna bring the into a sterilized environment, sample behavior, that’s gonna be predictable for all that other behavior, but now, in many cases, we actually can in principle, observe a lot of behavior across time, across situation and so why do I need like a formative assessment if I can just kinda see what the student has been doing, and all the work products, I mean, I can’t, I mean, the AI kinda knows where the student is. Right? So so that’s, I think, something to think about. I mean, I’m sure, there’s an answer, but it’s something to think about. And then on the other hand… And then the other thing is what does even mean to have a standardized test in in an environment where everything is hyper customized mean, I do remember one of the conversation I had at the previous place I worked where of the guardians said, like, you… Like, an adaptive standardized test is an oximerone.
[11:31:59] Frank Rijmen: So,
[11:32:02] Frank Rijmen: because once you adapt it, it’s not sterilized anymore. So, but now everything is hyper customized, right? So, what does it mean? Are we talking about standardized experience? What does that even mean? So it becomes a little bit of an epistemological question. On the other hand, and I think where where there’s a… What AI makes possible because we… It’s so easy to observe behavior now, collect it and score it, we basically can move from this one measurement or pre post test, pre post test, delayed post test. We can basically move the model where we continue observe behavior and sample a little bit of behavior every time. So, we have a lot of data, over time, but at any given moment, have very little data and think the challenge is there that, it is time Some researchers have worked on these dynamic measurement models, but I think time for those models to become integrated in the column part of the standards. So essentially the trajectory becomes becomes the score, right? Now, what is the… The unresolved problem is
[11:33:12] Frank Rijmen: So,
[11:33:14] Frank Rijmen: I think we know how to train people in terms of skills and stuff or we’ll figure it out. I mean, all the tools I’m sure we’ll figure it out. But, how do we develop behavioral skills? I mean, that those require a different approach than the than the cognitive skills. What should professional development look like when the actual goal is like changing the way we work rather than how we work, but how we work together, how can we balance the different roles at the testing organization.
[11:33:46] Frank Rijmen: And that’s it for me.
[11:33:52] Damian Betebenner: Thank you, Frank. Okay. So last up, we have Mohammed.
[11:34:02] Damian Betebenner: Let’s go here.
[11:34:12] Damian Betebenner: I’m just going to put up your PDF. Is that fine?
[11:34:23] : Okay. Let’s see. Can I there we go? First, I would like to thank Damian and Marina for his invitation for
[11:34:37] Mohammed A. A. Abulela: this panel. And it’s really hard to be in a panel with this great panel, so I’ll try my best. This is a running research project in fact that we started early this year when we started from… Like, we see from industry, to publication, but what about academic preparation for major main professions? Before I start, I would like to thank my great co authors Doctor. Brian French Washington State, Doctor. Brian Livingsel, James Madison University, and doctor Gohair Gurgan. University of Georgia and Doctor. Masso Gaster Metametrics and Corporation. And also I would like to thank thoughtful short conversation I had with Doctor. Andrew Ho early on when he we had this discussion about academic preparation. So my idea was the following. Coming from academia to industry, like, I feel both. I was in academia and our industry, so I think of the academic preparation of graduate students in the AI era. So one of the meetings that I had with Damon early on when we started this panel, he says like a student who started PhD five years ago, and graduated now, they’re graduating now but they finished, likely finished their coursework during the first two years, which may be like twenty one, twenty two, At that time, AI and large language models were not like that. So I’m graduating as a graduate, as a major one professional, going to the market. I don’t know where I’m going. Like, all jobs request something. And what should I do? It’s kind of make me think about the training of graduate students now. How should we incorporate AI responsibly? That’s the most important. In order to promote and help our graduate professionals in measurement to be able to cope with what we are in now. And now we don’t expect the future for AI. Really important. So the claim we have is is this, AI preparation should no longer be treated as an optional excitation. It’s not an optional excitation. That’s kind of my heart, to have this claim, but we have evidence. Exitation of what… Of our graduate training in measurement and psychometrics should not be optional. But it should become an an intentional part of profession. Professional preparation while this is the most important. While remaining grounded in validity, fairness, measurement theory, and psychometrics. So we’re not saying that we’re promoting AI to be taught for our graduates and replace the foundational measurement that we have been trained to be on and so This is the claim. I wanna disclose AI use. I use the Charge GBT because we are an AI native profession. So I use the ChargeGBT to create these. That’s why when you have Mac versus Windows, it’s for them. Right? The evidence that we have, the one concrete example where… Why we came up with this claim We started, like, thank you, NSC and me, and that they have, like, they updated now, by the way. They have my… Their nine programs of educational measurement, and these programs are available on any CME website. So we’d like to know how many programs teach what courses, and if they are required or optional. So we started to review all these courseware, I think only two programs were not able to reach like, the kind of the actual syllabus and program. It was really I’m not saying time consuming. It’s worthy, but it took a lot of time going to the website because the the link is that they are to the program, not to the course. So for each of these, we started to look for what courses are offered and if they are like an option or, like, kind of elective or just required to pay the program, and we found the following. Only sick esteem programs. That have AI, like AI related kind of coursework offered, and most of these, if not all, are optional. And focused on machine learning and mostly in data science focused programs. Has like this kind of data science and measurement. So the question now, even only 16 programs have these machine learning mostly on data science programs What about the training of educational measurement? I can’t go to computer science and take a machine learning, but that is… Is this really a major main focus course? Do I will receive as a grad student the applications or the implications of learning machine learning in a computer science program when I do measurement. This is kind of the thing. And that’s why I like like, this is really tough somehow for now. And make me think, oh, k. Let’s go to the job market. The second piece of evidence. We reviewed what is currently what we have at hand and we find… We found, like, 45 jobs in industry, 35 of them requires AI related skills. That means if I was trained in a program that has AI, it will make me more competitive compared to another graduate and it’s not my fault that I was trained at that program. But I graduated, I wanna go, I wanna compete, I want a job, But, unfortunately, I was I was not trained the way I should. Is it my fault? Now, this means the programs now can really attract and recruit students when they promote what they are doing what course work is offering and so on. This is important. Right? And also, again, the focus not on AI only, but validity, fairness, foundational psychometrics and measurement, that’s really so important. So this is the gap, right? So the gap between this which leads us to the next. And reserved for professional problem. This is really important for all people in academia to think, oh, what is the meaning? I will say it emotionally because it really touches my heart. When I say it. It’s really touches my heart when I think of graduate students whether they are in programs, they will go to it, like training, like, which start, like, a a question of measurement. Is the minimum AI related preparations at every measurement every measurement professional should receive. Is there a minimum? Should we think? What should be required and what should be elective? We need to think of that. Right? How do we integrate AI with our displacing validity?
[11:41:20] Mohammed A. A. Abulela: Fairness,
[11:41:22] Mohammed A. A. Abulela: psychometric foundations will learn in our educational measurement programs. My thinking when I had a chance to speak with doctor Brian Finch, he’s doing a good job now. He has an independent study course with the students. He invite people from industry to teach black students. And it’s really one of the solutions in our kind of an independent course. But basically, it’s the following. I’ve I’ve taken my first speech like, here in my BCD program, I took a course like education… Principles of Education and Psychological Measurement. The course like, covered, like, touches on the topics, like, factor analysis, IRT, like, all these kind of scoring, performance assessment, and all of these. So this kind of the principles that led me to think which course I need to take to have advanced training on. So can we have an AI and educational measurement course as a starting point and be required so that the students know the ethics and responsible use and validity and bias and fairness And then if I’d like to learn more more about machine learn, and educational measurement, then this found course give me the starting point I feel that I need advanced training, then I can look for another course that’d be like, for example, an advanced training in medication measurement. That’s kind of my thinking and that’s kind of our conversation. It’s a running project. We’ll continue this to be a publication, so hopefully you are doing a good job now. I would also thank again my co authors. You’re welcome a lot. For this, and thank you for listening.
[11:42:57] Damian Betebenner: Okay. I’m afraid that this thing is not working. So I might just ask Fred to come up.
[11:43:10] Damian Betebenner: Yeah, Fred, you come up.
[11:43:19] Damian Betebenner: Okay, our experiment has wrinkles, I guess. And so do I. Well, thanks for having me here. Let me
[11:43:29] Damian Betebenner: yeah. Yeah. That’s gonna break
[11:43:35] Mohammed A. A. Abulela: it. Just a second. Just break it. We’ll break it. All those things are hooked in to, like, the
[11:43:41] Damian Betebenner: speakers and then out with the audience.
[11:43:48] Frank Rijmen: On top. Maybe that’s okay.
[11:43:53] Frank Rijmen: Nope. It’s not okay. Oh,