Evaluating in the wild

Overview

Building an evaluation library is one thing. The claim that it is system- and corpus-agnostic only holds if it works on something it was not designed for.

So I pointed spancheck — the citation-grade RAG eval in the previous case study — at two real, published, third-party RAG applications, as a black box, over a public healthcare corpus. One cracked into a full scored run and, in the process, improved spancheck itself. The other stood up cleanly and then hit a wall I chose to document rather than break through. Both chapters are the point: this is what integrating with and measuring systems you did not write actually looks like.

The problem

A grounded-answer system’s most important property — do its citations hold, and does it decline when it should — is also the least portable to measure, because every system represents “sources” differently.

A credible eval has to meet each target where it is, run against real environments with their proxies and version pins and container quirks, and be honest when a target will not cooperate. The test of “corpus-agnostic” is not a claim in a README. It is a run against someone else’s code.

Constraints

Public data only, source cited. A five-document corpus of US healthcare policy from Wikipedia (CC BY-SA): Medicare, Medicaid, health insurance, HIPAA, prior authorization. Multi-document and overlapping, so a citation has to attribute to the right article.

Neutral, reproducible, small-set. The subject is spancheck’s generality, not either target’s quality. Every run is a measurement on roughly twenty authored cases with the audit log attached — a field report, explicitly not a benchmark.

A real, hostile environment. The machine runs a TLS-inspecting corporate proxy that breaks almost every tool’s outbound HTTPS; only clients that use the OS trust store, like curl, pass unaided. It is the recurring adversary across both chapters.

Time-box the unknowns. Each target gets a feasibility pre-flight. If it is not answering a question within a couple of hours, it gets documented and left, not chased.

Chapter one — Kotaemon, a full scored run

Kotaemon (Apache-2.0, Cinnamon Labs) is a real citation-RAG application whose citation model — a direct quote from the context, as a substring of the original content — is an exact match for what spancheck verifies.

Standing it up meant clearing six environment hurdles in order. It needs Python 3.11 or lower, and the machine runs 3.13, so a 3.11 interpreter via uv. The proxy broke uv’s installer, which --native-tls fixed, and then tiktoken’s and Anthropic’s TLS, which truststore fixed. Its settings loader wanted the application’s UI package, resolved with an environment override. A loose langchain pin resolved to a version its code does not use, so the install had to come from its lockfile with uv sync. And the model rejected a deprecated parameter. Then it ran: its real citation pipeline, on claude-sonnet-5, over the corpus, 21 cases.

Provenance and attribution ported with no tuning. Of the verbatim citations Kotaemon produced, 18 of 18 were source-scoped to the exact document they named. spancheck verified a system it had never seen and got attribution completely right.

It caught a real defect. On an adversarial false-premise question conflating HIPAA with prior-authorization rates, Kotaemon answered instead of abstaining, and produced a fabricated citation — an evidence quote that is not verbatim in the retrieved context. The provenance check flagged it.

It exposed an assumption in spancheck’s own metric. The support score came in low — 0.29 on the lexical proxy, 0.43 judged — but the failures were all unsupported rather than fabricated: the quotes were real and on topic. The cause was a design assumption. The support check judges each citation against its claim, which defaults to the whole answer. Kotaemon answers in detail with a few short, answer-level evidence quotes rather than per-claim citations, so “does this fifteen-word quote support the entire paragraph?” scores low by construction. The dogfood scored 1.00 precisely because its citations were claim-scoped.

Using the tool on a system it was not designed for made the tool better. That was the run’s most valuable output.

Chapter two — AnythingLLM, an honest blocker

AnythingLLM (MIT) is a different shape: a Docker container exposing a REST API.

It stood up cleanly, its outbound HTTPS worked through the proxy, and its chat API and developer key both functioned — confirming it as a partial target. It returns retrieved source chunks rather than span-level citations, so spancheck would score abstention, groundedness and source attribution, which was the intended contrast with Kotaemon.

But every document-ingestion call failed with an opaque internal error: a failed integrity signature check, and a 500 the collector never explains. Setting the backend-to-collector signing keys and restarting did not resolve it. The most likely cause is the same proxy meeting the collector’s processing step, but the service does not surface enough to confirm that. I documented the fix path — mount the proxy CA into the container — and stopped, because knowing when to stop is the discipline the pre-flight exists to enforce.

How it’s verified

What	Result
Kotaemon — citations source-scoped to the correct document	18 of 18 verbatim citations, on a system spancheck had never seen
Kotaemon — real defect caught	a fabricated citation and a failed adversarial abstention, both provenance-flagged
Kotaemon — the metric fix, measured	citation accuracy 0.286 → 0.762 after shipping claim_split
AnythingLLM — pre-flight	container, API, key and model access all working; ingestion blocked, documented with a fix path

The fix was to score an unscoped citation against the answer sentence the span is most relevant to, rather than the whole answer. Re-scoring the same captured run moved citation accuracy from 0.286 to 0.762, and the failures that remain are the genuinely fabricated citations — the eval’s real catches. The run is stored, so the before and after are the same data scored two ways, not two runs.

What broke

The instructive break was not an environment fight. It was the assumption in my own metric.

spancheck’s citation support check quietly assumed citations are claim-scoped, which is true of the system I built it against and false of the first real system I pointed it at. A 0.29 looked like a Kotaemon defect; it was a mismatch between how spancheck asked the question and how Kotaemon answers. Naming that rather than publishing the low number as a verdict is what turned it into a fix.

Limits

Not a benchmark, and not a verdict on either project. Two systems, roughly twenty authored cases, one corpus, one retrieval choice.

One target was never scored. The corpus-agnostic claim held on the system that let me in, and went untested on the one that did not. That is a weaker result than two scored targets, and a truer one than quietly dropping the target that refused.

The environment is not reproducible. The proxy that shaped most of this work is specific to the machine it ran on, so another person’s six hurdles would be six different hurdles.

What I’d do differently

Unblock AnythingLLM properly by mounting the proxy CA into the container, for a second scored target and a real partial data point.

Add a provenance-only mode for systems that emit chunk-level sources rather than claims, so the metric fits the citation style instead of needing a workaround.

Links

The tool is spancheck, MIT-licensed and installable with pip install spancheck; this work lives in its repository rather than one of its own. showcase/kotaemon/FINDINGS.md and showcase/anythingllm/FINDINGS.md carry the full write-ups and the audit logs, and docs/EXTERNAL-SHOWCASE.md is the repeatable protocol both chapters followed.