Oral assessment should be designed as measurement

The first question is not whether the assessment will be live, recorded, or AI-mediated. It is what the institution intends to infer. Kane's argument-based account of validation (2013) makes the governing principle explicit: interpretations and uses of scores are validated, not formats in the abstract, and more ambitious claims require more support.

“Reasoning” is too broad. A defensible claim names observable moves. For example, can the student:

• identify the consequential issue in a case;
• distinguish evidence from assumption;
• connect a cue to a governing principle;
• compare plausible alternatives;
• state what would change the conclusion;
• respond to a counterexample without abandoning coherence; or
• make and justify a commitment under uncertainty?

Oral communication should enter the score only if it is part of the intended competence. If the outcome is disciplinary reasoning, accent, social ease, verbal speed, and polished delivery are potential contamination. If rapid oral response is itself a professional requirement, they may be relevant—but that relevance must be stated, not smuggled into a global impression.

The working paper reaches a deliberately narrow conclusion: oral assessment can elicit evidence relevant to reasoning. The score's meaning still depends on construct definition, content sampling, response processes, scoring consistency, relationships with other evidence, and consequences.

One deep conversation feels informative, but measurement asks how much of the performance belongs to the student and how much to this case, examiner, moment and prompt chain. A student may reason well in one familiar context and poorly in another; two examiners can agree while sampling too little of the domain.

A stronger design uses a blueprint across content, difficulty, cognitive demand and reasoning moves, with more than one case or occasion when stakes justify it. A medical structured oral using two problems and trained examiners reported G = .793 for relative decisions but Φ = .696 for absolute decisions; examiner variance was omitted because each student saw one examiner (Sabqat et al., 2026). Structure did not make sampling error disappear.

For most courses, the practical answer is not a two-hour viva. It is several short, deliberately different observations: a rationale after group work, a changed-condition probe after a case, a sampled defense of an artifact, and a later unaided transfer task. Together they carry a stronger claim than one grand conversation.

Follow-up questions are the distinctive strength of oral assessment. They are also its largest uncontrolled variable.

A useful protocol can separate four stages:

1. Initial response: the student answers the common prompt without help.
2. Clarification: the examiner resolves ambiguity without adding substantive content.
3. Challenge: the examiner introduces a prespecified counterexample, changed condition, or request for evidence.
4. Support: if the assessment is formative, the examiner may teach, cue, or reframe after the scorable evidence is preserved.

This makes help visible. The record can distinguish what the student produced independently, what survived challenge, and what emerged only after support. It also prevents a helpful examiner from giving one student a path that another never received.

The distinction is not merely procedural. Fox, Ericsson and Best (2011) found that neutral concurrent think-aloud had no detected average accuracy effect, r = −.03, 95% CI [−.10, .03], whereas directed description or explanation changed current-task performance, r = .23 [.14, .31]. A substantive “why?” prompt can become part of the performance it elicits.

The rule is not that every interaction must be robotic. It is that adaptation should have a documented purpose and a scoring consequence. A bounded probe can reveal reasoning. An open-ended rescue mission changes the construct.

An analytic rubric should map each claim to observable evidence. “Strong reasoning” is not enough. A criterion might require a relevant premise, a justified connection between evidence and conclusion, recognition of uncertainty, and an explicit revision condition.

High rater correlation does not solve the scoring problem by itself. In a 443-student psychology project interview, Turner and Davila-Ross (2015) reported rater correlations of r = .94–.95, while second raters still awarded about half a point more on average, d = .06. Rank consistency and absolute agreement answer different questions.

For every criterion, the scorer should be able to identify where the evidence occurred—in the recording, transcript, or structured observation. If no evidence is locatable, the system should not manufacture a level from fluency or general impression. Evidence absence must also be represented honestly: “not observed” is not always “low competence,” especially when the task never created the opportunity.

The evidence-before-score review finds substantial prior art: human and automated procedures have marked criterion-specific evidence before assigning levels, and LLM systems have required evidence-bearing rationales. What remains untested is whether a criterion-level hard gate improves reliability or validity when isolated from the other changes bundled with it.

That means an evidence citation should be audited on at least three dimensions:

• recoverability: does the cited passage or time region exist?
• relevance: does it address the criterion?
• sufficiency: does it warrant the assigned level rather than merely mention the topic?

A quotation can make a wrong score look transparent. Auditability must include the inference from evidence to level, not only a clickable span.

AI can make oral assessment more measurable in several useful ways. It can administer common prompts, apply bounded follow-up policies, create transcripts, separate pre-help from post-help evidence, link criterion scores to exact passages, and repeat the same scoring procedure across many performances.

The current direct study illustrates the distinction. In two NYU cohorts totaling 73 students, Ipeirotis and Rizakos (2026) reported post-deliberation inter-model α = .86 and .90 at the rubric-dimension level, without confidence intervals. The models had seen one another's ratings before that agreement was calculated. The human comparison was explicitly informal, not a blinded exercise in which faculty and the system applied the same prespecified rubric. Agreement among models is not agreement with the intended construct, and a technically accurate transcript can still support a substantively wrong interpretation.

Consequential use requires a frozen model, prompt, rubric, follow-up policy and aggregation rule; blinded independent human ratings; preserved pre-adjudication disagreement; criterion and total-score reliability; subgroup and missingness analysis; threshold review; and drift checks after changes.

AI should standardize only the parts of the process that require standardization. Human experts should define the construct, write and review cases, judge novel or ambiguous performances, examine disagreement, and decide when the rubric itself has become wrong.

An oral can remove a writing barrier for one student and create a speaking barrier for another. Anxiety, language background, disability, sensory access, processing time, and familiarity with the response genre can affect the performance. Ringeisen and colleagues (2019) observed higher anxiety and cortisol on an authentic oral-exam day than on a matched control day, but had no written-mode comparator and found no relation between anxiety level and grade in the final model. Nieminen, Moriña, and Biagiotti's meta-ethnography (2024) synthesized 42 qualitative studies and 868 disabled students, finding that assessment could enable and exclude; it did not isolate oral assessment. The sources identify validity threats more clearly than they establish successful mitigation.

A defensible design offers practice with the format, predictable timing, accessible interfaces, appropriate response time, disclosed data policies, and accommodations that preserve the intended construct. Alternative modes should be equivalent with respect to the claim, not superficially identical. The aim is to avoid scoring barriers absent from the learning outcome.

For each use, faculty should be able to answer:

1. What precise reasoning claim will the oral support?
2. Which cases and prompts sample that claim?
3. What is the common first question?
4. Which follow-ups are clarification, challenge, or support?
5. What evidence is recorded before help?
6. Which rubric criteria are separate from delivery?
7. How many observations are needed for the decision's stakes?
8. How are raters or scoring systems calibrated and monitored?
9. What happens when evidence is insufficient, the score is borderline, or the student appeals?
10. What accessibility alternatives preserve the construct?

This specification also prevents oral assessment from being misused as authorship detection. A student who cannot defend a paper may warrant another sample of learning. That discrepancy does not prove misconduct. The assessment-integrity essay keeps those inferences separate.

Oral assessment can reveal something a finished artifact cannot: how a student responds when a premise is challenged, a condition changes, or a justification is requested. That is valuable evidence of present performance.

But a conversation does not measure reasoning merely because reasoning may occur inside it. It becomes an assessment when the claim is defined, the domain is sampled, help is bounded, evidence is preserved, scoring is calibrated, uncertainty is handled, and consequences are reviewed. Those conditions support an interpretation of performance under specified prompts; they do not turn speech into direct access to cognition or evidence of durable learning.

The next tests are concrete. Multi-case and multi-rater designs can estimate task, examiner and occasion variance. Delayed, unaided tasks can test transfer beyond the exchange. Blinded faculty and automated systems can apply the same prespecified rubric before adjudication. Response-process and subgroup analyses can examine whether prompting, transcription and scoring carry the same meaning across learners.

The goal is not to standardize every human conversation. It is to make every consequential score explainable: what was asked, what the student showed, what support entered, what rule connected evidence to level, and how much uncertainty remains. That is how faculty can use the responsiveness of conversation without pretending that speech speaks for itself.