Automated Grading: Accuracy, Bias, and Where AI Grading Works

Automated essay scoring is usually evaluated against a single number: quadratic weighted kappa, or QWK, the agreement between the engine's scores and a human rater's, corrected for chance and weighted so that near-misses count less than large disagreements. Operational scoring programs typically require QWK of at least 0.70 before an engine is allowed near live scoring — a threshold chosen because it is roughly the agreement two trained human raters reach with each other. The bar, in other words, is not perfection. It is "as consistent as a second human."

That framing is both the strength and the weakness of the field. QWK measures whether the machine reproduces human scores. It says nothing about whether the machine reproduces them for the right reasons. An engine can agree with humans on a well-behaved set of essays while relying on features — length, vocabulary sophistication, sentence-boundary counts — that have no causal relationship to writing quality. High agreement can coexist with an invalid construct. This is the central tension in every honest account of automated grading.

The optimistic evidence is genuine and worth stating plainly. The 2012 Automated Student Assessment Prize, sponsored by the Hewlett Foundation and run on Kaggle, benchmarked commercial and open engines against human scores on thousands of essays. Shermis and Hamner reported that, on the extended-response prompts studied, the better engines produced scores statistically comparable to trained human raters. ETS's e-rater work, documented by Attali and Burstein, reaches similar agreement on large-volume, well-defined prompts.

The pattern in that evidence is consistent: automated scoring is most trustworthy where the target is structured and the rubric is concrete.

• Short constructed responses with a defined correct-content model — a paragraph naming the causes of an event, a math justification — where scoring is close to content matching.
• High-volume, low-stakes writing where a fast, consistent score with human moderation beats no feedback at all.
• Convention and mechanics — spelling, grammar, organization signals — which engines detect reliably because they are surface-observable by design.
• Second-reader roles, where the engine flags essays on which it disagrees sharply with a human so a person can adjudicate, rather than assigning the grade itself.

In these settings automated scoring is not a compromise. Consistency is a virtue in assessment, and a tireless engine applying the same rubric to the ten-thousandth essay as to the first can be more consistent than a fatigued human at hour six.

The failure cases are equally well documented, and they cluster where the construct is hardest to observe from the surface. Les Perelman's work is the sharpest demonstration. Observing that length is the single strongest predictor of machine scores on timed prompts, he and students built the BABEL generator, which assembles grammatically intact but semantically empty prose. BABEL essays — gibberish by any human reading — earned high marks from scoring engines, because the engines cannot read meaning or check facts. They measure proxies, and proxies can be gamed.

That is not a bug in one vendor's product; it is a property of scoring writing by surface features. It means automated grading is riskiest exactly where writing matters most:

• Creative, argumentative, or original work, where quality lives in ideas, evidence and voice — none of which are surface-observable.
• High-stakes, consequential decisions — admissions, certification, graduation — where a gameable proxy becomes an incentive to game it, and a single misscore has real cost.
• Factual accuracy and reasoning, which conventional engines do not evaluate at all and which even large language models assess unreliably.

The National Council of Teachers of English put the objection formally in its 2013 position statement: machines cannot read, so scoring writing by machine measures the measurable and calls it writing. That critique has aged well.

Automated scoring learns from human-scored training data, and it inherits whatever is in that data — including systematic disadvantage for writing that departs from the dominant norm. The most rigorous recent evidence comes from adjacent territory: Liang and colleagues, publishing in Patterns in 2023, tested GPT detectors and found they misclassified 61.3% of essays by non-native English writers as machine-generated, against 5.1% for native writers. The mechanism generalizes to scoring: models penalize the vocabulary range and syntactic patterns typical of second-language and dialect writers, confusing linguistic difference with lower quality.

The fairness literature on essay scoring specifically — including the 2024 BEA workshop work on fairness in AES — finds measurable score gaps across demographic groups even when human-judged content is equivalent. This is why validity evaluation of automated scoring, as Williamson, Xi and Breyer's ETS framework insists, cannot stop at overall agreement. It has to test agreement within subgroups, because an engine can hit 0.75 QWK overall while systematically under-scoring a protected group whose essays are a minority of the sample.

"Human-in-the-loop" has become a reassurance rather than a design. A human who rubber-stamps machine scores at scale adds no protection; the loop has to be built so the human's attention lands where it matters. Several design commitments distinguish a real loop from a nominal one.

• Confidence-gated routing. The engine scores what it is confident about and routes low-confidence or high-disagreement responses to a person. Effort follows uncertainty instead of spreading evenly.
• Adjudication on disagreement. Where the engine and a human diverge beyond a threshold, a second human resolves it — the same discrepancy procedure operational programs already use between two human raters.
• Subgroup validity monitoring. Agreement is tracked within demographic groups, not just overall, so a fairness gap surfaces as a metric rather than a complaint.
• Stakes-tiered authority. The engine's role narrows as consequences rise: primary scorer for low-stakes practice, advisory only for decisions that gate a credential.
• Transparency to the learner. Feedback shows what was evaluated, so students are not left inferring how to satisfy a proxy they cannot see.

LearnLab's grading engines are built to this shape: automated scoring carries the reliable, high-volume work — short answers, conventions, first-pass structure — while confidence thresholds surface the essays a human should actually read, and subgroup agreement is monitored as a first-class metric. The engine is fastest where speed is safe and quietest where judgment is required. That is the honest position: automated grading is a strong tool for a bounded set of tasks and a liability outside it, and the design job is to keep it inside its bounds.