Education rarely dominates the conversation about high-stakes AI. The headlines belong to credit scoring, medical diagnosis and criminal justice. But AI is now woven into the daily experience of millions of students: automated essay scoring, adaptive learning platforms that decide what to teach next, proctoring systems that watch for cheating, and detection tools that judge whether a piece of writing was produced by a human or a machine. Each of these makes a consequential decision about a person who is often young, rarely consulted, and poorly placed to contest the outcome. That combination — real stakes, limited recourse — is exactly why education deserves the same assurance discipline as any regulated sector.
Where the failures actually happen
The most visible failure mode in education right now is AI writing detection. These tools do not prove authorship; they estimate a probability by analysing statistical patterns such as sentence rhythm, vocabulary predictability and structural regularity. The trouble is that a probability is not a verdict, and the error rates are far from negligible. Independent studies have reported false-positive rates ranging from roughly 3% to 17% depending on the tool — meaning genuine, human-written work is regularly flagged as machine-generated.
Worse, those errors are not evenly distributed. A Stanford-led study found that widely used detectors misclassified writing by non-native English speakers as AI-generated at dramatically higher rates than writing by native speakers, because the formal, structured, formulaic style many second-language writers are taught reads as "machine-like" to the algorithm. Neurodivergent students and anyone who uses grammar checkers, translation aids or accessibility tools can be flagged for the same reason: those tools smooth writing into the statistically regular patterns detectors associate with AI. The result is a system that most penalises the students who are already most vulnerable.
The consequences are not abstract. A false accusation of academic dishonesty can mean a failing grade, a lost scholarship, a disciplinary record, or expulsion — alongside genuine psychological harm. When a probabilistic score becomes the sole basis for a disciplinary decision, the institution has effectively outsourced a life-altering judgement to a tool its own vendor warns should never be used that way.
Beyond detection: grading and adaptive systems
Automated grading carries a quieter version of the same risk. Proponents argue that algorithms grade more consistently than tired, distracted or unconsciously biased humans, and in high-volume standardised testing that consistency has real value. But a model trained on past scores can also inherit and amplify whatever bias was present in that history, and it can be gamed by essays that hit the statistical markers of quality without the substance. A grade that looks objective because a machine produced it is still only as fair as the data and the design behind it.
Adaptive learning systems introduce a subtler issue: they decide what each student sees next. If the model consistently routes certain groups toward easier material or away from advanced tracks, it can quietly narrow opportunity in ways no single teacher decision ever would — and because the logic is embedded and personalised, the pattern is hard to see without deliberately looking for it.
What assurance looks like in education
The good news is that education does not need exotic controls. It needs the same evidence-based assurance applied elsewhere, adapted to the setting.
Never let a probabilistic score be the sole basis for a high-stakes decision. This is the single most important control. A detection flag should be a trigger for a human conversation and a review of drafts, version history and process — not a verdict. The leading institutions that have examined these tools, including several universities that have restricted or abandoned AI detectors, have reached the same conclusion.
Test for disparate impact before deployment, not after complaints. Any grading, detection or routing system used on students should be evaluated for how its error rates differ across language background, disability status and other protected characteristics. If a detector is wrong far more often for non-native speakers, that is a design defect to be measured and managed, not an unfortunate footnote.
Preserve process evidence. Assessment designs that capture drafts, outlines and revision history give both students and instructors something more reliable than a single opaque score — and they make AI-assisted misconduct harder to hide without punishing the innocent.
Keep a human accountable. Whoever assigns the grade or brings the accusation owns the decision, regardless of which tool informed it. Assurance frameworks must reinforce that accountability rather than let it dissolve into "the system flagged it."
Log what the model saw and produced. When a decision is challenged — as it should be able to be — the institution needs to reconstruct what the system was given, what it output and who reviewed it. Without that record, there is no way to learn from errors or to demonstrate that due care was taken.
The broader lesson
Education is a revealing case precisely because so many assume the stakes are low. They are not. A wrongful cheating accusation at eighteen can reshape a life as surely as a denied loan. The underlying dynamic is the same one that appears wherever AI produces a confident output that feeds a consequential decision: fluency and apparent objectivity invite over-trust, and over-trust is most damaging when the output is wrong and the subject has the least power to push back. Assuring AI in education is not about resisting the technology — used well, it can genuinely widen access and free teachers to teach. It is about insisting that any system which judges a student be held to the standard of evidence we would demand of any other high-stakes decision, and never treated as an oracle simply because it produces a number.