LLM Evaluation & Guardrails Harness

Overview

The harness tests an LLM system the way you would test software. A test set of cases runs through the target via a thin adapter, each output is scored by a mix of deterministic rule checks and a calibrated LLM judge, and runs are stored so a later version can be compared against an earlier one to flag regressions. The first wired target is a regulatory RAG copilot, and it is the only one. Grading a different system means writing an adapter inside a running deployment rather than installing anything — the harness is an app, not a library, which is a real constraint on who can pick it up.

The part that makes it usable is that it answers its own hardest question first. An evaluator whose judge is miscalibrated, or whose checks are buggy, is worse than no evaluator, because it manufactures confidence. So before the harness grades anything else, make selfcheck grades the harness: it measures the judge against human labels, exercises every deterministic check against known-good and known-bad fixtures, and confirms that a deliberately injected regression is actually caught.

This method was later generalised into an installable library — spancheck — which adds citation-span verification and ships as a CLI and a CI gate.

The problem

Anyone can get an LLM feature to look good in a demo. Knowing whether it is actually good — and whether a prompt tweak, a model swap, or a new release quietly broke it — requires measurement, and most projects do not have it. The gap is between “it worked when I tried it” and “here are the numbers, and here is what changed.”

Standing up an evaluator does not close that gap on its own. A judge that disagrees with human reviewers, or a check that silently passes bad output, produces numbers that are worse than no numbers, because they are believed. So it is two problems: measure the target, and prove the instrument doing the measuring can be trusted.

Constraints

Synthetic and public test data only. No PHI, no internal systems. The gold set and the fixtures are authored for this project.

The judge is non-deterministic, so a single judged run is a sample rather than a measurement, and the design had to treat it as one.

Deterministic checks carry the load; the judge is scoped to the qualitative. A property a rule can check must not be handed to a model.

It has to run offline. Without a provider key the self-check falls back to an offline mock judge and must say so, because a mock number cannot be allowed to read as calibration.

A blind spot must not read as a failure. When the harness cannot measure something, it has to report that rather than blame the target for the instrument’s own gap.

Architecture

The pipeline is a test set of cases — input, expected behaviour, and the checks that should pass, including adversarial and guardrail cases — then a runner that executes the target through a thin adapter and captures each output, then two-layer scoring, then per-case and per-suite gates, then a comparison layer that stores runs and flags deltas.

The two layers divide the work by what each is good at. Rule checks are deterministic: must-include and must-not-include, format and schema, citation-present, abstention and refusal behaviour, and PII-leak detection. An abstention check with no marker match and no judge to adjudicate returns unmeasured — an instrument gap — rather than a confident FAIL.

The judge handles only the qualitative — groundedness, correctness against a reference, helpfulness — against a versioned rubric. Because it is non-deterministic on borderline cases, JUDGE_RUNS=N scores each judged check N times and reports the distribution plus a stability flag: a split verdict shows as UNSTABLE rather than a clean pass, and a score inside a band around the threshold is flagged near-threshold.

Regression detection is the comparison layer: run the same suite against a baseline and a candidate, and surface the per-check deltas. A separate coherence detector checks the judge’s verdict against its own written rationale, so a verdict that contradicts its stated reasoning is caught rather than trusted.

Key decisions

Chose deterministic checks for the load and the judge for the qualitative only. Most of what a test needs to assert — a citation is present, a schema holds, an identifier leaked — is a rule, cheap and reproducible. The cost is that the judge’s error is irreducible on what remains, which is exactly why it is calibrated rather than trusted.

Chose to validate the evaluator before trusting it. The self-check measures judge-human agreement, exercises the rule-check fixtures, and confirms an injected regression is caught. The cost is that the human gold set is small at 12 cases — directional rather than a benchmark — so disagreements are spot-checked by hand.

Chose to report unmeasured rather than a false FAIL. When a check has no marker and no judge to adjudicate, the harness reports that it could not measure the property instead of failing the target for the harness’s own blind spot. The cost is that an unmeasured result needs a human to resolve rather than a clean red or green.

Chose to treat a single judged run as a sample. JUDGE_RUNS plus a stability flag turns a lucky pass into a visible UNSTABLE, and a borderline score into near-threshold. The cost is N times the judge spend on the checks where stability matters.

Chose to ship an offline mock judge that says so. The whole harness and its self-check run without a key, so CI is free and a reviewer can clone and run it. The mock’s agreement is stamped model=mock… and carries a loud caveat, because a mock number is not evidence of calibration.

How it’s verified

make selfcheck runs the harness against itself. This is real output against the configured judge, Claude Opus 4.8:

Check	Result
Judge agreement with the 12-case human gold set	1.00 (12/12, no disagreements)
Coherence — the judge’s verdict against its own rationale	detector OK; no self-contradictions in the gold set
Rule-check fixtures — citation, PII-leak, abstention, refusal, format, include/exclude	23 / 23
Regression — an injected degraded target is flagged	yes: citation_present 1.00 → 0.00, case pass 89% → 33% over 11 checks
Judge meta	model=anthropic:claude-opus-4-8 rubric=g1
Automated suite	66 tests

The gates are hard where they can be: fixtures must be 100%, the regression must flag, and the coherence detector must fire. The judge gate is soft for the offline mock — a loud caveat — and hard for a real provider, where agreement below 0.90 fails the run. Run without a key, the self-check uses the offline mock judge and stamps it as such.

What I’d do differently

Build the stability check first. The failure mode that would have quietly undermined everything was the judge’s own non-determinism: a single judged run read like a measurement when it was a sample, so a borderline check could pass or fail by luck and the report looked clean either way. Scoring each judged check JUDGE_RUNS times and surfacing a stability flag is the thing to build first next time, because an evaluator’s own variance is a first-class property rather than a footnote.

Grow the gold set, and add a second labeller. Twelve cases is directional. A second labeller would let the harness report inter-rater agreement instead of leaning on one person’s labels.

Limits

Not a benchmark or a certification. The judge is calibrated against a 12-case human gold set. Agreement of 1.00 means the judge and the labels agree on that set.

The judge has irreducible error, which is why the deterministic checks carry most of the load and the judge is scoped to the qualitative and spot-checked.

Self-validation proves the harness behaves as designed, not that a suite is complete. Coverage for any given target is a separate, ongoing effort.

Synthetic and public test data only. No PHI, no internal systems.

Links

The repository is linked at the top of this page. make selfcheck prints the readout above — judge agreement, the fixture count, and the caught regression — and runs offline on a mock judge with no key, or against a real provider for a calibrated number. EVAL.md carries the full method: the gold set, the rubric, the fixtures, and the threshold each gate holds to.