Benchmark — run history
Results — all runs
Run 1: 2026-09-07, before any fix. Run 2: 2026-09-08, after root causes F (short-form case names) and G (Westlaw/LEXIS coverage). Run 5: 2026-09-08, the first full re-run after root causes A/B/C (Ex parte/year-first/ docket-infix extraction) — Run 5's row is highlighted.
| Label | Run | Testable | Flagged | Falsely verified | Not checked | Outside coverage |
|---|---|---|---|---|---|---|
| FAKE_NONEXISTENT | 1 | 34 | 31 (91.2%) | 0 (0.0%) | 2 (5.9%) | 1 (2.9%) |
| FAKE_NONEXISTENT | 2 | 34 | 21 (61.8%) | 0 (0.0%) | 2 (5.9%) | 11 (32.4%) |
| FAKE_NONEXISTENT | 5 | 34 | 22 (64.7%) | 0 (0.0%) | 0 (0.0%) | 12 (35.3%) |
| FAKE_WRONG_CASE | 1 | 15 | 5 (33.3%) | 0 (0.0%) | 8 (53.3%) | 2 (13.3%) |
| FAKE_WRONG_CASE | 2 | 15 | 5 (33.3%) | 0 (0.0%) | 8 (53.3%) | 2 (13.3%) |
| FAKE_WRONG_CASE | 5 | 15 | 9 (60.0%) | 0 (0.0%) | 1 (6.7%) | 5 (33.3%) |
| REAL | 1 | 8 | 1 (12.5%) false-flagged | 6 (75.0%) verified | 1 (12.5%) | 0 |
| REAL | 2 | 8 | 0 (0.0%) false-flagged | 7 (87.5%) verified | 1 (12.5%) | 0 |
| REAL | 5 | 8 | 0 (0.0%) false-flagged | 8 (100.0%) verified | 0 (0.0%) | 0 |
| REAL_FROM_ORDER | 1 | 57 | 17 (29.8%) false-flagged | 28 (49.1%) verified | 10 (17.5%) | 2 (3.5%) |
| REAL_FROM_ORDER | 2 | 57 | 8 (14.0%) false-flagged | 32 (56.1%) verified | 10 (17.5%) | 7 (12.3%) |
| REAL_FROM_ORDER | 5 | 57 | 8 (14.0%) false-flagged | 38 (66.7%) verified | 4 (7.0%) | 7 (12.3%) |
Detection, in-coverage subset only
The 63.3%/73.5% headline numbers above are "all testable fabricated citations," coverage gaps included. Restricted to fabricated citations CourtListener's own index actually had evidence to check against — every outside-coverage exclusion, not just the WL/LEXIS ones — this is the detection performance on the citations it had something to compare against.
| Label | Run | In-coverage testable | Flagged | Detection rate |
|---|---|---|---|---|
| FAKE_NONEXISTENT | 1 | 33 | 31 | 93.9% |
| FAKE_NONEXISTENT | 2 | 23 | 21 | 91.3% |
| FAKE_NONEXISTENT | 5 | 22 | 22 | 100.0% |
| FAKE_WRONG_CASE | 1 | 13 | 5 | 38.5% |
| FAKE_WRONG_CASE | 2 | 13 | 5 | 38.5% |
| FAKE_WRONG_CASE | 5 | 10 | 9 | 90.0% |
| All fabricated | 1 | 46 | 36 | 78.3% |
| All fabricated | 2 | 36 | 26 | 72.2% |
| All fabricated | 5 | 32 | 31 | 96.9% |
Every fabricated citation there was coverage to check, it caught, in Run 5 — 31 of 32; the one miss (Menocal, a bare name the extractor deliberately refuses to guess at) is disclosed on the benchmark page, not a coverage question. Root causes A/B/C (Ex parte, year-first, and docket-infix extraction) drive the jump from Run 2's 72.2% to 96.9%.
Detection by category
Flag rate on the two fabricated-citation labels, Run 2 vs. Run 5 (current). FAKE_WRONG_CASE nearly doubles — D-117's extraction fixes, confirmed at full scale; see "Run 5" below.
Every remaining false flag and miss, with root cause
Per instruction, everything that isn't a disclosed coverage boundary is listed plainly here, not tuned around. The 8 false-flag instances below are 5 distinct citations — 4 of the 8 are the same citation, Mata v. Avianca's 678 F. Supp. 3d 443, at four different pincites. Counted once per distinct citation instead of once per instance: 5 of 59 distinct real citations false-flag (8.5%), against the raw per-instance rate of 8 of 65 (12.3%) in the table above. 12.3% answers "how often does a false flag happen across real filings"; 8.5% answers "how many distinct real citations currently get flagged wrong."
- Cooter & Gell v. Hartmax Corp. (496 U.S. 384) → mismatch against COOTER & GELL v. HARTMARX CORP. Et Al. — disclosed, not new: a genuine one-letter misspelling in the source filing itself ("Hartmax" vs. the real "Hartmarx"), not an abbreviation difference the short-form-name fix addresses. Confirmed as a known, accepted limitation by the resolver's own test suite.
- Mars Steel Corp. v. Cont'l Bank N.A. (880 F.2d 928) → mismatch against a caption that appends four individual objector-appellants by name — not disclosed anywhere found; a real residual, listed as instructed, not tuned. The short-form-name fix's abbreviation expansion (Cont'l → Continental) doesn't by itself resolve an entire additional clause of named parties.
- Mata v. Avianca, Inc. (678 F. Supp. 3d 443, four separate appearances), Williams v. The Ests. LLC (663 F. Supp. 3d 466), and Malkeet Lnu v. Blanche (177 F.4th 1014) — six false-flag instances total, all not_found. Disclosed, a known coverage boundary: real, correctly-cited, recent reporter citations CourtListener's own citation-lookup index doesn't yet carry. The Westlaw/LEXIS fix only pattern-matches WL/LEXIS-numbered citations; none of these five are WL/LEXIS-numbered, so they fall outside that fix's scope — exactly the risk this project's own strategy doc already named as "the single most likely embarrassment," observed directly, not new.
Why the fabrication-detection rate went down, not up
The Westlaw/LEXIS fix makes any WL/LEXIS-numbered citation always report "outside coverage," without running the fabrication-detection check at all. In Run 1, several genuinely fabricated citations happened to be WL/LEXIS-numbered and got "flagged" — correctly, but only because the old code ran its full check and got lucky, the same check that also produced Run 1's false flags on real WL/LEXIS citations. The fix trades that luck away deliberately: 11 of Run 1's flagged FAKE_NONEXISTENT entries are now an honest "not enough evidence to call this fabricated" instead of a lucky flag — a direct, intended consequence of eliminating false positives on real citations, not a new gap. Zero of either run were ever falsely verified; the trade is entirely between two non-"verified" outcomes.
Run 3 — subset verification of a fix, not a full re-run
Run 2's false-flag analysis above proposed a resolver change for three of the root causes it found. That change shipped, and Run 3 re-tests a named 23-entry subset against it: all 15 FAKE_WRONG_CASE entries, the 7 REAL-set entries the fix's own acceptance test required to confirm clean, and the one FAKE_NONEXISTENT entry the same fix also touches. Not a re-run of the full 136 — a first attempt at a full re-run (Run 4) was abandoned unfinished on account exhaustion; Run 5, above, is the full re-run that moves the 72.2% in-coverage headline.
| Citation (asserted) | Run 2 | Run 3 |
|---|---|---|
| Ex parte Vasquez | not_checked | mismatch |
| Ex parte Clayton | not_checked | mismatch |
| Ex parte Martinez | not_checked | mismatch |
| Dino v. Pelliccioni | not_checked | mismatch |
| Menocal | not_checked | not_checked |
| Tinch / Burton / Ricks (WL-numbered) | not_checked | outside_coverage |
| 5 ordinary two-party entries | mismatch | mismatch (unchanged) |
4 new catches (Vasquez, Clayton, Martinez, Dino). Menocal is unchanged, on purpose — see below. Tinch/Burton/Ricks shifted from an uninformative blank miss to an honestly labeled coverage-gap miss, not a catch: they're still Westlaw-numbered, so the deliberate WL/LEXIS gate above still applies once a name is extracted. In-coverage detection on this 15-entry subset: 9 of 10 (90.0%), up from 5 of 13 (38.5%) on the same subset — a different denominator (3 entries moved into the coverage-gap exclusion), not a directly comparable percentage.
All 7 REAL-set guardrails now confirm clean (In re Fisher, Matter of Rottkamp, Cummings v. Dessel, In re World Trade Ctr. Disaster Site Litig., In re Keegan Mgmt. Co., In re Daou Systems, In re Girardi) — each moved from not_checked to verified. None was ever at risk of a false flag before or after; this is a pure reduction in unhelpful "not checked" reports, confirmed on live data, not just the unit tests written alongside the fix. Zero new false flags, inside or outside this subset.
What this run does not fix: Menocal (a bare name with no recognized prefix) is left unchanged on purpose — accepting a bare capitalized word as a candidate case name risks pulling ordinary prose in as if it were one, a precision regression the acceptance test didn't require. Tinch/Burton/Ricks (Westlaw-numbered) aren't caught by design either: doing so would mean reopening the search ladder for WL/LEXIS citations the WL/LEXIS gate deliberately closed, the same lever that produced 5 of Run 1's false flags on real WL citations.