Payment-Integrity Claims Reviewer

Overview

The reviewer takes a batch of synthetic claim lines and flags three payment-integrity issues: duplicate billing, unbundling — a panel’s components billed separately to bill more than the panel — and out-of-network claims adjudicated as in-network. Each flag records the rule that fired and the exact fields that triggered it, with a confidence derived from the rule rather than from a model. An LLM turns that trigger into a plain-English rationale, and a human approves, dismisses or escalates each flag from a prioritised queue. A dashboard tallies the dollars identified, labelled as an estimate.

The design choice the whole project is built around is where the model sits: the LLM has no authority over what gets flagged or how it ranks. Detection is deterministic Python. The model only explains, grounded strictly in the rule and the triggering fields, with the model and prompt version recorded on every explanation. In a regulated review workflow, why a claim was flagged has to be auditable, and an opaque model verdict is not.

The problem

Payment integrity hinges on finding the small share of claims worth a closer look, without drowning reviewers in false positives or making opaque automated calls on people’s claims. Access to the claims is not the hard part. The hard parts are two: explaining why a claim was flagged, so a reviewer can trust it and act, and measuring whether the flagging is any good — because a detector that looks impressive in a demo can be catching nothing the demo didn’t stage.

Constraints

Synthetic and public data only. No PHI, no real claims, no real fee schedules. Every identifier, date and amount is invented, and the reference tables are hand-curated and small enough to audit.

Detection had to be deterministic and auditable. A reviewer, and a compliance function behind them, has to see a concrete, inspectable reason a claim was flagged. No model call could sit in the detection path.

The model gets no authority. The LLM explains and nothing else. It cannot flag, clear or rank a claim, and its prompt forbids any new reason or approve/deny recommendation.

The reviewer never sees the answer key. The synthetic claims carry ground-truth labels for the eval, but the review UI hides them. A reviewer who could see the label would not be a real human in the loop.

The hosted demo makes no model call. Explanations are cached, so a public URL cannot run up an API bill through traffic, and the host needs no key.

Architecture

Detection is a pure function over the claim lines. Three rules group and compare lines — DUP-01 on member, provider, CPT and date; UNB-01 on a panel’s components sharing a date; OON-01 on network status against how the line was paid — and each emits a flag carrying the rule id, the exact triggering fields, and a rule-derived confidence. It reads nothing from the model layer and makes no external call, so the same claims always produce the same flags.

The explanation layer is separate and downstream. For each flag it sends the model only the rule description and the recorded triggering fields. A fixed system prompt constrains it to explain those, forbids new reasons, and forbids any recommendation. Each explanation stores the model id and a prompt version, so it is reproducible.

The reviewer queue is a prioritised worklist: a per-flag dollar figure that is a triage signal rather than a recovered amount, filterable by decision and sortable by priority, amount or confidence. Every approve, dismiss or escalate is an append-only decision row. The dashboard shows Identified — everything flagged — against Confirmed, which is approved only, with the dollar total single-sourced from one ROI module and badged as an estimate.

Detection and the served app run offline. The model is touched only when explanations are generated, and on the hosted demo they are already cached.

Key decisions

Chose rules that detect and a model that only explains. Every flag is produced by inspectable Python and traces to concrete fields, so detection is auditable and the model cannot invent a flag. The cost is that the rules are only as good as they are written — which is what the holdout below measures.

Chose precision over recall. A missed overpayment leaks a dollar; a wrong flag can cost the relationship. A false positive that reaches a provider is a demand to return money on a legitimately paid claim, which generates appeals and erodes trust in the integrity programme itself — the two errors are not symmetric. So the rules fire only on the patterns they are confident about: exact-date duplicates, a whole panel billed as components, an unambiguous network mismatch. The cost is the 0.47 holdout recall, and it is the posture rather than a shortfall — the rules would rather miss a real-world variant than flag a clean claim, and every miss is listed with its cause.

Chose to score against a holdout, not only the seed. The demo seed was authored alongside the rules, so it scores a perfect 1.00, which measures that the rules behave as designed rather than what they cover. A separately written holdout, reusing only the row schema, probes claim shapes the rules were not tuned on. The cost is two datasets to keep; what it buys is a coverage number that means something.

Chose deterministic grounding first, an LLM judge second. A deterministic check guarantees an explanation names the rule and cites a real triggering field; a judge with a versioned rubric then catches a fluent invented reason the keyword check would miss. The cost is that the judge has its own error rate — on the run below, its one unfaithful verdict was itself the error.

Chose to hide the ground-truth label from the reviewer. It keeps the human accountable for the call and preserves a labelled decision history for a future feedback loop. This one cost nothing.

Chose to host from a committed, pre-explained database. The explanations are static over a fixed seed, so the image ships them cached: the running app makes no model call, needs no key, and costs nothing to host. The cost is that refreshing the demo is a regenerate-and-recommit step rather than a live call.

How it’s verified

Because the synthetic claims carry ground-truth labels, both halves are measurable, and every check is a deterministic assertion over the pipeline’s own output.

Check	Result
Detector precision / recall / F1 on the demo seed (make eval)	1.00 / 1.00 / 1.00
Near-misses — issue-like but legitimate, correctly not flagged	17 / 17
Detector precision / recall / F1 on a separately written holdout (make holdout)	0.85 / 0.47 / 0.61
Holdout by rule — duplicate · unbundling · out-of-network	0.57/0.36 · 1.00/0.43 · 1.00/1.00
The three rules re-expressed in SQL, proven equal to the Python engine (make sql)	flag-for-flag parity — a bidirectional EXCEPT, empty both ways on the seed (34 flags) and the holdout (20 flags)
Holdout after two SQL gap-closers — DUP-02 date-drift, UNB-02 partial unbundling (make sql)	0.91 / 0.86 / 0.89 — recall 0.47 → 0.86, precision held, no new false positives
Explanation faithfulness — deterministic grounding, then an LLM judge	0.97 (33/34; the one miss was a judge error)
Automated suite, offline, including SQL parity, boundary and malformed-input tests	61 tests

The seed’s 1.00 measures that the rules behave as designed, not what they cover: its positives are exactly the patterns the rules look for, and its near-misses use exactly the modifiers the rules know.

The holdout is where the detector’s shape shows. It takes the textbook cases — exact-date duplicates, full-panel unbundling, and out-of-network, all at 100% — and misses the real-world variants: modifier-59 abuse, a true duplicate stamped with a distinct-service modifier it did not earn, caught 0 of 3; date-drift duplicates, the same service billed a day apart, caught 0 of 4; and partial unbundling, where 2 of 3 panel components are split out, caught 0 of 6. All three trace to one root: DUP-01 keys on an exact date and trusts modifier 59, and UNB-01 requires the whole panel inside one date group. Every missed line is listed in EVAL.md with its fix direction.

Two of those three are now closed. Writing the same three rules in SQL made it cheap to add two more — DUP-02 for date-drift duplicates, written twice over as a self-join and as a LAG window and proven to return the same flags, and UNB-02 for partial unbundling. Holdout recall goes 0.47 to 0.86 and precision does not fall; it rises to 0.91, because the new rules added only true positives. Modifier-59 abuse, the next-day tail of a cross-date split, and the bilateral false positive below are still open, and are still the stated fix direction.

The gap-closers live in the SQL layer, so make holdout — the Python engine — still reports 0.47. The 0.86 comes from make sql. Two implementations of each shared rule is the cost of that, which is why the parity check is a gate rather than a note: the SQL and the Python have to agree flag-for-flag before either is trusted.

Precision leaks the same way. Modifier 50, bilateral, is a legitimately distinct repeat that is not in the rule’s distinct-service set, so the rule fires on it. A test pins that behaviour rather than papering over it.

What I’d do differently

Two of the holdout’s early failures were in the eval, not the system. The metric counted a fabrication for any adversarial case that failed — but a case that fails by abstaining emits no lure, so the count now fires only when a forbidden value actually appears. A second grader demanded a spelled-out number the model had answered as a digit. Both fixes were to the checks, not to the rules.

The detector’s own gap is scoped rather than closed. Recall on near-date and partial variants needs a date window on DUP-01, partial-panel detection on UNB-01, and a modifier policy that treats 59 as scrutiny-worthy rather than exculpatory. Closing it would need a fresh holdout, because the current one would no longer be testing what the rules are not tuned on.

Limits

Not an adjudication engine. It assembles evidence and a cited rationale for a human reviewer. It does not decide, price or pay a claim. No auto-clear, no auto-deny.

Synthetic, hand-labelled data only. No PHI, no real fee schedules, no payer policy library. The results show whether the rules behave as designed, not real-world performance on live claims.

Three narrow rules, at 0.47 holdout recall. Modifier-59 abuse, date-drift duplicates and partial or cross-date unbundling are missed today, and are the stated fix direction.

The dollars figure is an estimate, not recovered dollars — labelled Identified, split from Confirmed, with the assumptions surfaced on the dashboard.

The judge has its own error rate. The LLM only explains, and its explanations are scored for faithfulness against a judge that is spot-checked rather than trusted, which is why the deterministic grounding check comes first.

Links

The repository and the live demo are linked at the top of this page. The demo serves the reviewer queue and the flag detail with cached rationales, on a free tier that sleeps when idle, so the first request after a quiet period can take a moment to wake. Inside the repo, make eval prints the seed metrics, make holdout prints the coverage number, and EVAL.md carries the full breakdown: every miss, its root cause, what would close it, and the argument for why precision comes before recall on this kind of work.