spancheck

Overview

spancheck scores a grounded-answer system — any RAG or retrieval-then-answer pipeline — on the four things that decide whether its answers can be trusted: citation accuracy, abstention correctness, hallucination rate, and cost and latency. It runs offline, hard-codes no model provider, and emits a versioned audit log a compliance reviewer can read. pip install spancheck gets a three-command CLI, and a composite GitHub Action drops it into CI to fail a pull request on a regression.

It is the spinoff of an earlier project, the LLM Evaluation & Guardrails Harness — the version that proved the method against one real target, with judge agreement of 1.00 on a human gold set. spancheck is that method extracted from an in-repo lab, generalised to any corpus, and packaged so that someone else can point it at their own system. What carried over, what was rewritten, and what changed is recorded in the repository’s LINEAGE.md, written as the port happened rather than reconstructed afterwards.

It has since been pointed at systems I did not build. On Kotaemon, a published third-party citation-RAG application, 18 of 18 verbatim citations were source-scoped to the exact document they named, and the run exposed an assumption in this tool’s own support metric that I then fixed — Evaluating in the wild is that story.

The problem

Teams shipping RAG can tell you their retriever’s hit rate, but not whether the answer’s citations actually hold.

Two failure modes slip through an ordinary eval. An answer cites a passage that does not support it — including the insidious case where the answer is right and the citation is wrong. Or an answer should have said “I don’t know” and didn’t. Both destroy the one thing a grounded system exists to provide: an auditable trail from claim to source. Measuring them takes more than string overlap, and measuring them credibly means the eval itself has to be trustworthy first.

Constraints

Deterministic first. Anything checkable by a rule is checked by a rule — no model in the loop, no drift, no cost. The judge is reserved for the one thing a rule cannot do, entailment, and it is calibrated before it is trusted.

No vendor lock-in. The library never hard-codes a model provider. A caller supplies a plain input -> answer callable, and the judge reaches a model only through a caller-supplied provider, stubbed by default.

Every metric computable offline from a captured run. Capture a run once — answers, contexts, citations, usage, latency — and re-score it any number of times without touching the system again.

No invented numbers. Every figure in the docs is measured. Where a number would have to be assumed — a per-token dollar cost, for instance — the tool reports a blank rather than a guess.

Architecture

A small, layered, dependency-free package:

adapter.py   normalise any system's return into a typed Output {answer, contexts, citations, usage, latency_ms}
core.py      Case / evaluate / Run / Scorecard / diff / gate  (ported and generalised from the eval lab)
graders.py   deterministic graders: abstention, groundedness, PII-leak, latency, contains/regex, recall
span.py      citation-span verification — the capability the tool is named for
judge.py     the calibrated LLM-judge (entailment), prompts loaded from version-controlled files
audit.py     cost/latency aggregation and the versioned, compliance-readable audit log
cli.py       run / score / gate — thin wrappers over the library

Citation verification splits each citation into two deterministic checks, both of which must hold. Provenance asks whether the cited span is present, verbatim, in the retrieved context — which catches a fabricated quote. Support asks whether the span covers the claim — which catches a real citation that does not support what it is attached to. The default support check is a lexical proxy; the opt-in judge upgrades it to true entailment. Provenance is always deterministic and is never delegated to a model.

When the caller supplies a {source_id: text} map, provenance is scoped to the document a citation actually names, so a real span attributed to the wrong document fails as misattributed rather than passing because it exists somewhere in the retrieved set. That distinction only matters on a corpus where documents disagree — which is why the second dogfood corpus was built to disagree with itself.

Support scoring has a matching escape hatch. By default a citation is scored against the whole answer, which is right for a system that cites per claim and wrong for one that answers in detail with a few short evidence quotes. claim_split scores such a citation against the answer sentence its span is most relevant to instead. It is opt-in because the two citation styles genuinely need different questions asked of them, and guessing which is which would be the sort of hidden assumption this tool exists to catch.

Key decisions

Chose to extract from the lab, not from the product. Three eval codebases already existed across the portfolio. The most reusable was the dependency-free lab core, not the eval coupled to a product’s pipeline — that one imported the application directly and scored by substring assertion, so extracting it would have meant rewriting every metric anyway. spancheck generalises the lab and adds the one genuinely new capability.

Chose to pin the output shape early. Normalising every system’s return into one typed Output is what makes offline re-scoring, cost and latency accounting, and the audit log fall out for free rather than being bolted on later. The cost is an adapter every caller has to write.

Chose version-controlled prompt files over inline strings. A rubric change is a reviewable diff, and every judge verdict records the prompt version and the model that produced it. Without this, a judge’s score is unreproducible the moment anyone edits a string.

Chose MIT for the tool and source-available for the products. A measuring instrument is worth more the more widely it is used, so spancheck is licensed for anyone to install, vendor, or modify without asking. Project Suver and the applied-AI engines stay source-available, because a product is not an instrument. The cost is that this is the one piece of the portfolio whose reuse cannot be tracked or withdrawn.

Chose to name the limits in tests. The lexical support proxy cannot see a contradiction that shares a claim’s words. That false positive is pinned by a test and stated in the README, with the judge as the documented upgrade — a known blind spot with a test holding it in place, rather than a surprise.

How it’s verified

Measured, not asserted. spancheck was run end-to-end against a real pipeline as a black box — Project Suver’s grounded-answer tools on claude-sonnet-5 — and the judge was calibrated against a human-labelled gold set. Two dogfood runs: one tidy contract, and a deliberately messy four-document corpus — a Master Services Agreement, a superseding Amendment that changes the notice period and the governing law, a Statement of Work that cross-references both, and a Data Processing Addendum.

What	Result
Single-doc run, overall pass-rate across 12 cases	0.967 — it caught a false abstention, where the system declined an answerable, paraphrased question
Multi-doc run, overall pass-rate across 21 cases over 4 conflicting documents	1.00
Multi-doc source-scoped citation accuracy	47 of 47 citations verified against the specific document they named — both sides of every conflict cited to the right source
Hallucination rate, both runs	0 hallucinations
Judge calibration against the gold set, stub	0.75 — a zero-cost floor with no false positives
Judge calibration against the gold set, claude-sonnet-5	1.00, 12 of 12 — closes exactly the paraphrase cases the stub missed
Test suite	72 tests, green on a clean install and in CI on Python 3.11 and 3.12

The two runs answer different questions, which is why both are kept. The easy one found a real recall gap in a shipping system — a measured behaviour, not a spancheck bug. The hard one confirmed correct attribution where the documents contradict each other, and a perfect score there is only evidence because the run is provably against the real model rather than the extractive stub, which would pass provenance trivially.

What I’d do differently

A second real target. Both dogfoods score the same pipeline, so the adapter generalises in principle rather than in public. A differently-built target is the thing that would actually demonstrate it.

Push the judge past a twelve-item gold set. The real judge scored 12 of 12, which is a clean number and a small one. A larger held-out slice where the judge is sometimes wrong would sharpen the claim that this reports its own error rate — though that edges toward benchmark territory, which is deliberately out of scope.

Limits

Deterministic support is a lexical proxy. It cannot see a span that shares a claim’s words but contradicts it. The opt-in, calibrated judge is the upgrade for true entailment.

Provenance requires a verbatim quote. A paraphrased citation fails by design — a deliberate incentive to quote sources exactly.

Source-scoping needs per-source text. Attribution is checked against the named document only when the caller supplies a {source_id: text} map. Without one, provenance falls back to matching against all retrieved contexts, and a misattributed citation passes.

The metrics are only as good as the test set brought to them. spancheck scores a system against cases someone else authors; it does not generate them.

Links

The package is on PyPI under MIT, so pip install spancheck is the whole install; the repository is linked at the top of this page. python dogfood/run_dogfood.py reproduces the pipeline run in the table above, and python -m spancheck.calibrate --real reproduces the judge calibration; both print the numbers rather than asserting them. calibration/RESULTS.md carries the confusion matrix behind each agreement figure, docs/AUDIT-LOG.md documents the audit-log schema and its versioning rule, and LINEAGE.md is the ledger of what this inherited from the harness and what it corrects.