Few professional fields illustrate the risks of generative AI as starkly as law. The work is language-intensive, expensive and repetitive in exactly the ways generative models handle well: drafting contracts, summarising case law, producing first drafts of briefs. And yet law has also produced the most public and embarrassing catalogue of AI failures, because the profession has an unforgiving test for accuracy — a court — and a paper trail that makes every mistake permanent.
The now-familiar pattern began when lawyers submitted court filings containing citations to cases that did not exist. The AI tool had generated plausible-looking case names, reporter numbers and even quoted holdings — all fabricated. Judges discovered the invented authorities, and the lawyers involved faced sanctions, public reprimand and reputational damage. What made these incidents so instructive was not that the technology failed, but *how* it failed: confidently, fluently, and in a form indistinguishable from correct work to anyone who did not check the source.
Why legal drafting is uniquely exposed
Several features of legal work combine to make hallucination especially dangerous.
Authority is the product. A legal argument derives its force from citing real, binding precedent. A fabricated citation is not a cosmetic error — it undermines the entire submission and can constitute a misrepresentation to the court.
Fluency masks fabrication. Legal language is formulaic. A model that has read millions of judgments can produce a citation in perfect form — correct court, plausible year, realistic case name — with nothing real behind it. The very fluency that makes the output look professional is what makes the fabrication hard to spot.
The verifier must be an expert. You cannot check a legal citation by reading the draft. You have to look up the actual case in an authoritative database and confirm it exists, says what the draft claims, and remains good law. That is skilled, time-consuming work — precisely the work AI was supposed to eliminate.
The consequences are formal and personal. Unlike a marketing error that damages a brand, a fabricated citation can lead to professional sanction of an individual practitioner, dismissal of a client's case, and lasting harm to real people relying on the outcome.
The assurance principles that actually work
The lesson from the sanctioned cases is not "never use AI for legal work." It is that legal AI requires a verification discipline proportionate to the stakes. Several principles have emerged.
Ground generation in a verified corpus. A model drafting from an authoritative, current legal database — and citing only sources it retrieved from that database — cannot invent a case, because every citation traces to a real document. This retrieval-grounded approach is the single most important architectural control. Free-form generation from a model's memory should never be trusted for citations.
Verify every authority independently. Every citation in an AI-assisted draft must be checked against the primary source before filing: does the case exist, does it stand for the stated proposition, and has it been overturned or distinguished? This check cannot be delegated to the same system that produced the draft.
Preserve the lawyer's accountability. Courts have been consistent on one point: the human who signs the filing is responsible for its contents, regardless of what tool produced them. Assurance frameworks must reinforce rather than dilute this. The AI is a drafting aid; the lawyer is the author of record.
Log what the model was given and what it produced. When something goes wrong, reconstructing what source material the model had access to, what it generated, and what the human reviewed is essential — both for improving the system and for demonstrating diligence.
Beyond citations: the subtler risks
Fabricated citations are the visible failure, but subtler ones matter as much. A model may summarise a case accurately but omit the crucial exception that makes it inapplicable. It may draft a contract clause that is grammatically perfect but legally unenforceable in the relevant jurisdiction. It may miss that a cited statute was amended last year. These errors do not announce themselves the way an invented case name eventually does — they surface only when the deal fails or the argument collapses.
This is why assurance for legal AI cannot stop at "check the citations are real." It has to extend to whether the substance is correct, current and jurisdiction-appropriate — judgements that still require legal expertise. The technology can dramatically accelerate the drafting; it cannot yet assume the responsibility.
The broader lesson for every profession
Law is a vivid warning precisely because its verification test is so public. But the underlying dynamic applies wherever generative AI produces authoritative-sounding output that feeds a high-stakes decision. The more fluent and confident the output, the more tempting it is to skip verification — and the more damaging it is when the output is wrong. Legal drafting simply makes visible what is true everywhere: fluency is not reliability, and the assurance layer is not optional. The professions that adopt AI safely will be the ones that treat every confident output as a claim to be checked, not a fact to be trusted.