Cite11 — How we check quotations

A citation can pass every check that only asks “does this case exist, and does it resolve to the name given” and still carry language the opinion never wrote. That is an invented or altered quotation: the case is real, correctly cited, and the words attributed to it are not in the opinion — wholesale invented, or a real holding reworded until it says something the court didn’t. In the 30 sanctions orders we tested, 16 of 64 named fake citations were exactly this: a real case, not an invented one, with an invented quote. Data: AI Hallucination Cases database, Damien Charlotin, CC BY 4.0. Analysis: Cite11.

Why “does the case exist” doesn’t catch it

A tool that only looks up the citation number stops the moment it finds a real opinion at that address. It never reads the opinion against the sentence a lawyer wrote. The case existing, and resolving to the name given, says nothing about whether the words in quotation marks are the court’s own.

How Cite11 checks it

Cite11 looks for a quotation in three places: directly before a citation, inside a parenthetical after a citation, and directly before an “Id.” or a short-form citation. Each one found is compared, word for word, against the actual text of every opinion in the case’s cluster (a majority, concurrence or dissent can each carry it). The comparison tolerates punctuation, spacing and hyphenation differences, and a citation the brief itself says it left out — nothing else. Each quotation is reported as one of:

Found — the words are in the opinion.
Found — minor punctuation difference — the words match; only punctuation, hyphenation or spacing differs.
Wording differs from the opinion — every available opinion in the cluster was read in full and the words are not in any of them. This is the finding whether a quotation was invented outright or only a few words were changed — Cite11 does not currently distinguish the two; both read exactly the same way.
Not checked, with the specific reason — no opinion text could be fetched, the per-document checking limit was reached, or the citation has no opinion text to compare against at all (for example, a case cited only by docket number).

What it can’t check

A quotation outside the three detected patterns above — a second quotation before the same citation, or one that only follows the citation later in the text — is not detected at all, not flagged, not mentioned.
When wording differs, Cite11 does not show the nearest matching passage from the opinion, or where in the opinion a found quotation appears. It reports that the words are, or are not, there — nothing closer than that today.
Quote checking for one document is capped at a share of the account’s shared CourtListener request budget (45 requests by default). Once reached, remaining quotations in that document read “not checked,” with that reason stated plainly — never silently skipped.

No catch-rate for invented or altered quotations is published yet. The 16-of-64 figure above describes how often this shows up in real sanctions orders, not how often Cite11 catches it; see a sample report for what the finding looks like on the page.