Project Suver

Overview

Suver is a hub of sixteen AI tools across four platforms — Documents, Communications, Data & Analysis, and Learning — sharing one no-prompt shell. Each tool does one job: you bring the input, the tool returns the output. There is no chat box, no system prompt, nothing to configure.

What every tool shares is a trust contract. Every claim is grounded to a span of your source and cited, or it is withheld; and any PII or PHI is detected and tokenised locally, before a single character reaches a model.

I did not build sixteen applications. I built one reusable shell over five engine cores — a sanitisation boundary, a summariser, a template drafter, a typed extractor, a two-source comparator — and held the same contract as the shell went from one tool to sixteen and from one platform to four. A new vertical, whether legal, finance or HR, is a configuration: a field set, a draft kind, a policy. Not a new engine, and not a fork.

The problem

Most AI tools hand a user a blank box and hope they know how to prompt it, and hope the model does not invent an answer. For anyone working with documents that matter, both halves of that are the problem. A summary that quietly includes a clause the contract never contained is worse than no summary. An answer that cites the wrong section is worse than an honest “I don’t know.” And a tool that ships a customer’s medical record or SSN to a third-party model to do its job is a non-starter before the quality question is asked.

The work is not access to a model. It is making the model’s output checkable: grounded to the source, silent when the source is silent, and safe on data that can never leave the building.

Constraints

Synthetic and public data only. Every document in the repository is invented or public. The design has to be demonstrable without touching a real client’s data.

The model may only ever see sanitised text. PII and PHI are tokenised before egress and re-hydrated only in the user’s view. The boundary is not a step the model can choose to skip.

Grounding is enforced in code, not asked of the model. “Cite your sources” in a prompt is a request. The guarantee had to be a deterministic gate the model cannot talk its way past.

New verticals are configuration, not forks. The shell had to generalise, or every domain would become a rewrite.

It has to run without a key. The whole suite and its test set run offline on a deterministic stub, so CI is free and a reviewer can clone and run it in one command. The real model is a config flip.

Architecture

Every tool runs the same pipeline: ingest → sanitise → retrieve/split → draft (LLM) → ground (cite-or-abstain) → re-hydrate → show. The three steps carrying the trust guarantee — sanitisation, splitting, and grounding — are deterministic and never call a model. The model only ever sees text the boundary has already tokenised.

Sanitisation runs eight detectors — SSN, credit-card-like numbers, MRN, email, US phone, dates and DOB, person names, US street addresses — and is recall-first: it over-redacts rather than under-redacts, because for a trust boundary a false positive is cheap and a miss is not.

Retrieval is lexical, stemmed token overlap, widened by model-assisted query expansion: the model paraphrases the sanitised question so differently-worded facts still match. An optional additive re-ranker can reorder a lexical hit but never drop one. The grounding gate then keeps only claims whose content tokens are actually present in their best retrieved span; anything below the support threshold is dropped, or the tool abstains.

Across a set of documents the rule is map, don’t blend: the question is answered strictly within each document and attributed per document, so a fact from one can never contaminate another’s answer. The discipline across the whole system is one line — improve retrieval, never loosen the grounding gate.

Key decisions

Chose abstaining over answering. Cite-or-abstain on answers, cite-or-block on drafts: a section that cannot be grounded is withheld rather than faked. The product never fabricates confidently. The cost is over-caution on messy inputs, where recall drops rather than risk being wrong.

Chose sanitising before egress, in code. The boundary tokenises PII locally before any model call, because a policy the model enforces is a policy the model can violate. The cost is a recall-first heuristic that sometimes over-redacts — a long order number can be tokenised as a card number.

Chose composing engine cores over forking. The tools vendor five lean engine cores behind one shell, so a fifth platform needed no new contract field at all. The cost is that the vendored cores have to be kept in sync with their sources.

Chose a deterministic grounding gate over a model self-report. Support is measured by token presence in the cited span, so the guarantee is code rather than a prompt. The cost is that exact-token grounding can reject a correct paraphrase and abstain — the safe direction, and a real recall cost.

Chose to ship the re-ranker off by default. On clean documents, query expansion already reaches the fact, so the extra model call earned nothing and risked a regression. The cost is a retrieval-depth miss on harder documents where the re-ranker would help — and turning it on to fix that miss made a neighbouring case answer a question it should have refused. The trade is paid on one set of documents rather than everywhere.

How it’s verified

A labelled evaluation scores the product on the real model across four categories: answerable (recall), unanswerable (abstention), adversarial (no fabrication, no cross-document contamination), and sensitive (PII handled end to end). Every check is a deterministic assertion over the pipeline’s own output. No model call scores the model.

Check	Result
Labelled trust set, real model (python -m eval.run)	20 / 20
Recall — answerable cases surfaced the fact	6 / 6
Abstention — unanswerable cases refused	5 / 5
No fabrication or contamination — adversarial cases	5 / 5
PII handled — sensitive items tokenised before the model	4 / 4
Hallucination incidents · fabrication incidents	0 · 0
Hard set, deliberately messy documents (python -m eval.run_hard)	7 / 9 — one over-generalisation, no fabrications
Automated suite, offline stub	225 tests

The hard set is built from the documents that break grounded retrieval in the field: superseded facts, OCR noise, tables, distractor-dense unanswerables, near-duplicate documents, paraphrase gaps. On those documents the real model is non-deterministic at the margins — individual cases flip between runs, which is why the hard set hovers around 78% while the clean set is 20/20. The clean set is unambiguous; the hard set is deliberately not.

The demo shows the refusal rather than describing it: in Ask this document, the “Try one it can’t answer” button puts a question to a document that does not contain the answer, and the tool declines with a reason instead of guessing. It is the first thing worth checking, because a confident wrong answer is the failure that hides itself.

Across both sets: zero fabrications and zero confident wrong citations. Every miss is the system being too cautious, or in one case over-generalising a figure the document states generally to a question that asked specifically. Each is root-caused in eval/FINDINGS-HARD.md.

What I’d do differently

Two of the hard set’s failures were my checks, not the model. One counted an abstention as a fabrication, because the metric flagged any failed adversarial case rather than one that actually emitted a lure. Another demanded the word “twelve” when the model had correctly answered “12”. Both were the same trap — a red number that was a measurement bug — and the fix in each case was to the check rather than to the document. I would build the graders to assert the behaviour rather than its surface form from the start.

Retrieval over near-duplicate documents over-abstains. Two near-identical contracts confuse lexical retrieval enough that the system withholds rather than risk contamination. No lure is ever emitted, which is the safe direction, but it is a real recall miss. Lexical retrieval plus query expansion is not enough there; a dense-embedding seam is the fix, deferred on cost and on taking a new vendor, and recorded as a limit.

Limits

Not an adviser. Suver assembles cited, grounded output. It makes no legal, clinical or financial determination. A human reads and decides.

The 20/20 is a measurement, not a certification. It is a labelled 20-case set on the real model, against synthetic and public documents. Reproduce it on your own documents before trusting it on them.

The sanitisation boundary is recall-first and heuristic. Names outside its gazetteer, non-US addresses, and identifier types it does not model — passport, driver’s licence, bank account and routing, IP address — can pass through. It over-redacts by design; it is not NER-complete.

Links

The repository and the live demo are linked at the top of this page. The repo runs in one command, offline on a deterministic stub, with no key required. Inside it, eval/SCORECARD.md carries the 20/20 and eval/FINDINGS-HARD.md carries the hard-set error analysis: every miss, its root cause, and what changed.

The repo also carries docs/ENGAGEMENT.md, a worked engagement from discovery and requirements through integration, evidence, a runbook and the first thirty days — the delivery side of the same project.

The demo runs the real model under a cost cap, on a free tier that sleeps when idle — the first request after a quiet period can take around half a minute to wake.