Beyond Multiple Choice: Building Assessments That Measure Real Understanding

A multiple-choice item asks a student to recognize the right answer among distractors. Recognition is a weaker cognitive act than recall, and recall is weaker than explanation, and explanation is weaker than application to a new situation. A learner who cannot write a correct sentence about osmosis may still pick "the movement of water across a semipermeable membrane" from four options, because the phrase is familiar. Familiarity is not comprehension, and an assessment that cannot tell them apart will report understanding a student does not have.

This is not an argument against multiple choice as such. Selected-response items are efficient, reliable to score, and — as retrieval practice — genuinely strengthen memory. The problem is the inference: treating a score on recognition items as evidence of understanding. The format constrains what can be inferred from it, and most testing draws inferences the format does not license.

The revised taxonomy Anderson, Krathwohl and colleagues published in 2001 is often reduced to a poster of verbs. Its actual contribution is a two-dimensional map: a cognitive-process dimension (Remember, Understand, Apply, Analyze, Evaluate, Create) crossed with a knowledge dimension (factual, conceptual, procedural, and — newly — metacognitive). An objective is located at an intersection, and so is every item meant to assess it. The design failure the taxonomy exposes is the common one where an objective sits at "Analyze" but the test items all live at "Remember." The assessment is misaligned with its own stated goal, and no amount of item-writing polish fixes that.

Used seriously, the taxonomy is a coverage audit. Tag each item to a cognitive level, and the distribution becomes visible. Most course assessments, audited this way, turn out to cluster in the bottom two rows — Remember and Understand — regardless of what the learning objectives claim. The higher-order levels are underrepresented not because they do not matter but because they are harder and more expensive to write and to score. That cost, not pedagogy, is what keeps assessment shallow.

What "understanding" ultimately means, operationally, is transfer — the ability to apply knowledge in a context different from the one it was learned in. Barnett and Ceci's 2002 taxonomy of far transfer breaks this down along dimensions of context: transfer across knowledge domains, physical settings, social contexts, and time. Near transfer, to a nearly identical situation, is common and easy to teach for. Far transfer is rare, hard, and the thing education actually claims to produce. An assessment that measures understanding has to present a problem the student has not already seen solved.

Physics education research offers the cleanest demonstration of why this matters. The Force Concept Inventory, built by Hestenes and colleagues in 1992, uses ordinary-language questions to probe whether students hold correct conceptual models of force — not whether they can compute. Richard Hake's 1998 study of more than 6,000 students found that traditional lecture courses produced weak conceptual gains on the FCI even where students passed conventional exams, while interactive-engagement courses produced roughly double the normalized gain. The conventional exams had been measuring something other than conceptual understanding, and only an instrument designed for transfer revealed the gap.

What higher-order items require

• Novelty: the problem context differs from the worked examples, so retrieval alone cannot solve it.
• Justification: the student must show reasoning, not just produce an answer that could have been guessed.
• Discrimination: distractors, where used, target specific misconceptions rather than filling space.
• Authenticity: the task resembles how the knowledge is used outside the test.

Grant Wiggins named the alternative to fill-in-the-blank testing "authentic assessment" in 1990: tasks that are realistic, require judgment, and ask students to use a repertoire of knowledge to negotiate a complex problem of the kind adults actually face. A portfolio, a design brief, a data analysis, a defended argument — these assess understanding because they cannot be completed by recognition. Wiggins and McTighe's Understanding by Design turns this into a method: start from the evidence of understanding you would accept, design the assessment that would produce that evidence, and only then build the instruction. Backward design makes the assessment the specification for the course rather than an afterthought bolted onto it.

The objection to authentic assessment has always been cost. Performance tasks are expensive to build, expensive to score reliably, and hard to scale — which is precisely why so much assessment retreats to multiple choice. The retreat is economic, not pedagogical, and it is the economics that have changed.

Automatic question generation is a mature enough field to have its own systematic review: Kurdi and colleagues surveyed the literature in 2020 and documented steady progress from template-based generation toward semantically richer items. Large language models have since made it practical to generate not just recall questions but scenario-based, transfer-oriented items — novel contexts around a fixed learning objective, misconception-targeted distractors, and prompts that ask for justification rather than selection. The expensive part of authentic assessment, the item writing, is the part AI can most credibly assist.

The caution is real and follows directly from the earlier articles in this series. Generation without validation reintroduces every hazard of automated assessment: items that look higher-order but test recall in disguise, distractors that give the answer away, cultural or linguistic bias baked into the framing, and factual errors stated with fluency. AI-generated items have to be treated as drafts, subject to the same review and psychometric scrutiny as human-written ones — piloted, checked for difficulty and discrimination, and audited for bias before they count.

• Generate against the objective, not the topic: anchor each item to a specific cognitive level so the output is aligned by construction.
• Vary the context, fix the construct: produce many novel scenarios probing the same transfer, defeating memorization and item leakage.
• Human review as gate, not garnish: an educator validates before deployment; the machine drafts and scales, the human certifies.

Used this way, LearnLab's item-generation engines lower the cost of the assessments that measure understanding — scenario items, justification prompts, misconception-targeted questions — until they are cheap enough to use routinely rather than reserve for the one high-stakes exam. The pedagogy was never in doubt. What AI removes is the excuse that measuring real understanding is too expensive to do at scale.