Oral exams vs multiple choice: what speaking measures

Everything in our previous two pieces on assessment shared an assumption: that an answer is either right or wrong. That is how multiple choice works, and it is a substantial simplification of what happens when a student answers a question.

In a viva, an examiner does not record a bit. They form a judgement across a range — the differential was broad enough but the priorities were not justified, the mechanism was sound but the delivery rambled. We wanted to know what that extra resolution is actually worth, and where it stops being worth anything.

The setup

The tool for partial credit is Samejima’s graded response model, the standard extension of item response theory to items scored on a scale rather than as right/wrong.

The model estimates boundary curves — the probability of scoring at least level 2, at least level 3, and so on — and derives the item’s information from how sharply those boundaries separate ability levels.

We held discrimination constant at 1.0 and spaced the rubric thresholds evenly, then varied only the number of score levels: 2 (right/wrong), 3, 4, 5, 6. A two-level graded item is mathematically identical to a dichotomous one, which gives us a clean baseline.

Partial credit helps, then stops

Score levels	Information	Relative	Items for the same precision
2 (right/wrong)	0.250	1.00×	45
3	0.295	1.18×	38
4	0.306	1.22×	37
5	0.307	1.23×	37
6	0.307	1.23×	37

Going from right/wrong to a three-level judgement — wrong, partial, right — buys 18% more information per item. Adding a fourth level adds a little more. Adding a fifth adds essentially nothing, and a sixth adds nothing at all.

The curve is flat past four, and that is a genuinely useful negative result. A ten-point rubric does not measure anything a four-point rubric misses. It just asks the marker to make distinctions the data cannot support, which costs time and adds noise while looking more rigorous.

If you are building a marking scheme, four bands is where the returns stop.

What actually doubles the signal

The larger gain does not come from finer gradations of one judgement. It comes from making more than one.

A spoken answer supports two judgements that are close to independent: was the medicine correct, and was it delivered clearly. Knowing something and being able to say it under pressure are different skills — which is precisely why viva candidates who know the material still fail.

Information from independent dimensions adds. Two five-level rubrics on the same answer give:

Assessment	Information	vs one MCQ	Items for the same precision
One MCQ (5 options)	0.167	1.00×	67
One right/wrong item	0.250	1.50×	45
Content + delivery, graded	0.614	3.68×	19

Against a five-option multiple-choice question, a spoken answer marked on both dimensions carries 3.7× the information. Against a clean right/wrong item, 2.46×.

In practical terms: 19 spoken questions locate your ability as precisely as 67 multiple-choice ones.

The examiner is not being thorough for its own sake. Two judgements about one answer is the cheapest way to learn twice as much.

The catch, stated plainly

A spoken question is not a click. Nineteen of them, answered aloud with follow-ups, takes far longer than 67 taps through a question bank — this is a comparison per item, not per minute.

There is also an independence assumption doing real work in that 2.46×. Content and delivery correlate: students who understand something explain it better. If they correlate at 0.5 rather than 0, the combined information is meaningfully lower than the clean sum we report. We would put the honest range somewhere between 1.8× and 2.5×, and we are reporting the top of it.

The right reading is not that spoken practice replaces question banks. It is that spoken practice is information-dense, so a small amount of it is worth doing even when time is short — and that it measures something the other formats cannot reach at all.

What this is for

Live Talk grades content and delivery separately for exactly this reason, and asks adaptive follow-ups rather than moving on after one answer. A follow-up on a thin response is the same trick again: another judgement about the same underlying knowledge, which is where the information density comes from.

It is also the only format on this list that rehearses the thing being examined. A viva tests whether you can construct an argument aloud, in real time, while someone waits. No amount of silent multiple choice practises that, which is why students who have done thousands of questions still freeze in the room.

The transcript matters for the same reason the page citations in Notes matter — a score you cannot inspect is a score you cannot learn from. Being told your delivery scored 68 is useless. Being shown the thirty seconds where you lost the thread is not.

Limits of the model

We assume evenly spaced rubric thresholds. Real rubrics are lumpy — the gap between “wrong” and “partially right” is usually wider than between “good” and “excellent” — and uneven thresholds shift where the saturation point falls, though not that it exists.

The independence assumption is the significant one, and we have flagged it above rather than buried it.

The graded response model also says nothing about reliability of marking. It assumes the rubric is applied consistently. Human examiners disagree with each other and with themselves; automated marking is consistent but can be consistently wrong. Neither problem is visible in these numbers, and both are real.

Model: Samejima graded response model, discrimination 1.0, evenly spaced thresholds (width 1.6), evaluated at θ = 0. Code in research/question-quality/graded.py.