MCQ vs free response: what each format measures

Almost all self-testing that medical students do is multiple choice. Question banks are multiple choice, most flashcard apps are effectively multiple choice, and Step 1 itself is one-best-answer. So it is reasonable to treat MCQ as the format and stop thinking about it.

We wanted to know what that choice costs, so we measured four formats against the same model that real licensing exams use to convert answers into scores.

The setup

The standard tool here is item response theory, specifically the three-parameter logistic model. Every operational licensing exam is scored with some version of it.

The model says your chance of getting an item right is:

P = c + (1 − c) / (1 + exp(−a(θ − b)))

θ is your ability, b is the item’s difficulty, a is how sharply it separates strong students from weak ones, and c is the pseudo-guessing floor — the probability of getting it right while knowing nothing at all.

That last parameter is where the formats differ. Free response has c = 0; you cannot accidentally write the right answer. A five-option MCQ has c = 0.20. True/false has c = 0.50.

We held difficulty and discrimination identical across all four formats, so the only thing varying is the guessing floor. Then we computed Fisher information — the standard measure of how much an item tells you about the person answering it.

What each format is worth

Format	Guessing floor	Information	Items for the same precision
Free response	0.00	0.250	45
MCQ, 5 options	0.20	0.167	67
MCQ, 4 options	0.25	0.150	75
True / false	0.50	0.083	134

A five-option MCQ carries 67% of the information of a free-response item on the same content. To locate your ability equally precisely you need 67 MCQs where 45 short answers would do — half again as many questions for the same knowledge about yourself.

True/false is worth a third of a written answer. A hundred-item true/false quiz tells you roughly what thirty-three written answers would.

Where the information goes

Into the guess. Every point of c is probability mass assigned to outcomes that carry no signal about whether you knew anything, and the model discounts the item accordingly.

The practical version is visible in scores:

You actually know	Free response	MCQ (5)	MCQ (4)	True/false
20%	20%	36%	40%	60%
40%	40%	52%	55%	70%
50%	50%	60%	62%	75%
80%	80%	84%	85%	90%

Knowing half the material returns a 60% on a five-option paper. The ten-point gap is guessing, and it is not distributed evenly — at 20% true knowledge the inflation is 16 points, at 80% it is 4.

That asymmetry is the part worth sitting with. The format flatters weakest where the stakes of self-deception are highest. A student with real gaps is the one whose practice scores most overstate them.

A multiple-choice score is your knowledge plus a loan from chance, and the loan is largest when you can least afford it.

It also blurs comparisons

We ran one more test: two students whose true abilities differ by 0.30 logits — a real but modest gap — sitting a fixed-length test. How often does the test rank them backwards?

Format	40 items	100 items
Free response	21.7%	12.7%
MCQ, 5 options	25.1%	17.6%
True / false	30.4%	24.4%

Even at a hundred items, a five-option paper puts the weaker student ahead 17.6% of the time. This is not a defect of MCQ specifically — it is measurement noise, and it is why operational exams are long. It is, however, a strong argument against reading much into a twenty-question quiz you took on a Tuesday.

Why MCQ still deserves its place

None of this says stop doing multiple choice, and we would be misrepresenting our own results if we implied it.

Three things weigh the other way. MCQ marks itself, so you can do 67 of them in the time 45 written answers would take to mark honestly. Step 1 is one-best-answer, so practising the format is practising the exam. And a well-built clinical vignette tests reasoning that a short-answer prompt often cannot reach — the discrimination parameter we held constant is not really constant in the wild, and good vignettes earn a high one.

The finding is narrower and more useful than “MCQ is bad”: per item, multiple choice tells you less, and it tells you least about the students who most need telling.

We ran the mixed case too, holding total items at 60:

Sixty questions, spent as	Standard error
60 MCQ	0.316
40 MCQ + 20 free response	0.293
30 MCQ + 30 free response	0.283
60 free response	0.258

Swapping half your MCQs for written answers buys a 10% tighter estimate from the same number of questions.

What we do with it

This is why KoiSwarm ships SAQ alongside MCQ rather than treating short answer as a lesser sibling of the format everyone actually uses. They answer different questions about you: MCQ tells you whether you can recognise the right answer among plausible company; SAQ tells you whether you can produce it.

USMLE exists because format practice genuinely matters when the exam is one-best-answer — and because the discrimination of a vignette depends entirely on whether its distractors work, which is a problem we took apart separately.

The practical suggestion, if you take one: when a practice score comes back higher than your gut says it should, the gap is probably the guessing floor. Rewrite a handful of those questions as free recall and see whether the number survives.

Limits of the model

We held discrimination equal across formats to isolate the guessing effect. In practice a superb vignette discriminates better than a mediocre short-answer prompt, and format is not destiny — item quality does much of the work. Our comparison is between formats of equal craft, which is the fair comparison but not always the real one.

The 3PL model also treats guessing as blind. Real students eliminate options, so effective c sits somewhere between the blind floor and zero depending on partial knowledge, and our MCQ figures are therefore slightly pessimistic for a well-prepared student and slightly optimistic for an unprepared one.

Finally, information is computed at average ability. Items are most informative near their own difficulty, so a real adaptive test does better than these fixed numbers imply — for every format.

Model: 3PL item response theory, discrimination 1.0, difficulty 0.0, evaluated at θ = 0. Misranking figures from 40,000 simulated pairs per condition. Code in research/question-quality.