biomed-evidence-mcp

Overview

Six MCP servers expose ClinicalTrials.gov, four FDA datasets and PubMed to Claude as callable tools over one Postgres table — nine distinct tools across 212,919 documents.

The shared schema is the part that matters. Because every source normalises into the same documents row, a question like “what FDA context exists for this trial’s drugs” becomes a single SQL join rather than three API calls stitched together by a model.

The problem

Evaluating a cancer treatment means working across the trial registry, FDA approval and safety data, and the published literature. All three are free and public. Using them together is slow: each has its own query syntax and record shape, and nothing links a trial to its drug’s label or to the papers about it.

The work isn’t access. It’s retrieval, normalisation, and the joins between sources.

Constraints

Public and synthetic sources only. No patient data enters the system by construction.

Regulated content carries obligations. FDA and PubMed responses need a disclaimer a model cannot choose to omit, so it is appended in code after generation.

Upstream is not under my control. ClinicalTrials.gov sits behind Akamai Bot Manager, which 403s ordinary HTTP clients on TLS fingerprint. openFDA and NCBI both throttle anonymous callers, with higher ceilings behind a free key. Backoff, caching and incremental fetch are load-bearing, not decoration — the three clients share one retry policy that honours Retry-After, retries 429s, 5xx and transport errors, and never retries any other 4xx.

The client is Claude Desktop, which speaks MCP over stdio — one process per source.

Retrieval has to be inspectable. A tool that returns documents is only useful if someone can check whether they’re the right ones.

Architecture

Each connector implements a three-method DataSource protocol — fetch, normalize, get_raw — and declares itself in a server.toml. The ingest runner pages the upstream API, normalises records into canonical Documents, embeds them in batches through Voyage, and upserts into documents. core.refresh tracks a per-source watermark and only advances it on success, so a failed window retries rather than being skipped.

At query time, core/server.py reads the toml and registers whichever generic tools the vertical declared plus its custom tools. Every call routes through one choke point that applies the authorization gate and records a metering row.

Search is hybrid: pgvector cosine and Postgres full-text (websearch_to_tsquery), fused with Reciprocal Rank Fusion. Vector alone misses exact identifiers; full-text alone misses paraphrase.

Three of the nine tools summarise rather than return records: they retrieve first and pass only the fetched text to Claude Haiku 4.5. The rest, including every cross-source tool, run no model at all.

That is the retrieval half. documents answers “what should I cite?”, and it cannot answer “how many distinct adverse-event cases, as opposed to reports?” — one embedded row per citable record has no grain, and nothing joins to anything.

So the staging package loads the same upstreams’ relational files into real tables beside it: four quarterly FAERS extracts — all seven tables plus the FDA’s withdrawn-case list, with case-version dedup — and five tables from a pinned monthly AACT archive of ClinicalTrials.gov. 29,497,843 rows across 14 tables. Every table carries its partition column first, so re-loading a quarter replaces that quarter and nothing else, and every table has a declared grain that a gate has to prove.

The architecture diagram is committed in the repo’s README as Mermaid source.

Key decisions

Chose one canonical documents table over a database per source. Because the interesting questions cross sources — a trial’s drug against its FDA label, adverse events and recalls — and those are a single SQL join only if the rows live together. The cost is a canonical schema every connector must map into, and per-source detail has to survive as a JSON structured payload rather than typed columns. That mapping is most of the per-vertical work.

Chose a relational staging layer beside documents, not instead of it. Because the two shapes answer different questions and neither subsumes the other: retrieval wants one embedded row per citable thing, analysis wants keys, partitions and joins. The cost is a second copy of overlapping data and a second load path to maintain. I would collapse them if the staged tables ever started serving the MCP tools directly, at which point the duplication stops paying for itself.

Chose to keep FAERS partial dates rather than pad them to a DATE. 200906 is a month, and padding it to 2009-06-01 invents a day that nothing downstream can tell from an observed one. Each date lands as a raw value, a precision flag, and a real DATE only at day precision. The cost is three columns instead of one and a filter in every date query. It is not a marginal case: 26.8% of therapy start dates would have been fabricated.

Chose to read AACT’s archive over HTTP range requests. The archive is 2.5 GB and this layer needs five of its 49 members. A zip’s directory sits at the end of the file and every member is an addressable byte range, so the loader fetches 332 MB of 2,534 MB. The cost is about 150 lines of zip parsing instead of one zipfile call — and the same class reads local files, so there is still one code path.

Chose deterministic SQL over LLM synthesis for the cross-source join. Because the same question should return the same answer, and a reader needs to see which records matched rather than trust a paraphrase. The cost is that the output is a structured report, not prose — anyone wanting narrative pipes it into the model themselves.

Chose hybrid retrieval over pure vector search. Because registry text is full of identifiers and acronyms that embeddings blur (NCT ids, KEYNOTE-937, MK-3475) while the questions people ask are conceptual. RRF needs no tuning and degrades to pure vector when the lexical arm matches nothing. The cost is two index scans per query instead of one.

Chose curl_cffi browser impersonation over a proxy or a scraping service for ClinicalTrials.gov. Because Akamai fingerprints the TLS handshake, so header spoofing with httpx fails regardless of User-Agent. The cost is a dependency that carries its own libcurl and CA store, which is why TLS verification for that client is separately configurable.

Chose a bundled 300-document corpus and a keyless search fallback over “bring your own keys”. Because a reader who has to provision two API keys and ingest the full corpus before seeing anything never sees anything. The cost is a 1.9 MB artefact in git and a demo slice that is explicitly not a mirror of the real corpus.

How it’s verified

Check	Result
Contract and unit tests	433 passing, 182 of them contract or failure-path — every tool’s registered schema via list_tools(), plus HTTP 429 with Retry-After, 5xx, timeouts, malformed bodies, unreachable database, empty corpus
Retrieval spot-check, 20 labelled questions	hit@1 70%, hit@3 80%, hit@10 95%, MRR 0.77
Clean-clone CI	Every push: docker compose up, six servers selftest, a keyless search must return real trials
End-to-end tour	All nine tools over real MCP stdio against live data, 14/14 steps
Staging gates — grain, conservation, dedup, integrity, rejects	5 of 5 pass over 31M rows checked (31,031,531), every number published in docs/STAGING.md
Staging integration tests	Loader, gates and report against a real Postgres rather than a fake — 8 tests, run in the clean-clone CI job, skipped without a database
Static analysis	ruff lint + format, mypy over core, servers, staging, mcp_platform

The retrieval set is scored against the full 212,919-document corpus, not the demo slice, and the report names every miss with what outranked it. The one outright miss is root-caused: DESTINY-Breast03 is registered under the development code “DS-8201a”, never “trastuzumab deruxtecan”, so it ranks 28th for a question using the generic name while its sibling trial ranks first.

The questions were written by the person who built the corpus, and twenty items is a spot-check rather than a benchmark. Both facts are in the published report.

Each staging gate is written so that a clean database returns no rows: the pass condition is an empty result, and the row count beside it proves the gate had data to look at.

What broke

I blamed the wrong component for an hour. The first real staging load ran at 450 rows a second — 438,512 rows in sixteen minutes — and Postgres was the obvious suspect: 36 columns, two indexes, a container on a Windows host. Measuring instead of guessing took ten minutes and cleared all of it. COPY into the same shape did 390,000 rows a second. The parser did 118,000. The download took twelve seconds.

The problem was one line of mine. The line reader accumulated each decompressed chunk in a buffer and consumed it with pending = pending[index + 1:], copying everything left in the buffer once per line. An 8 MiB chunk inflates to about 40 MB and holds roughly 300,000 lines, so that slice moved terabytes to read a file. Splitting once per chunk instead of slicing once per line made the same load take 21 seconds — 47× faster, and the four-quarter load went from an overnight job to about eight minutes. The regression test asserts 200,000 lines parse in under ten seconds, which the quadratic version missed by minutes.

A gate disproved a key I had declared. I gave the FAERS drug table a grain of (primaryid, drug_seq) and the grain gate rejected it: 7,294,235 rows carry only 6,569,196 distinct pairs, and the worst key has 103 rows on it — the same drug, differing only in the reported manufacturing lot. It is a join key, not an identity. Anything that counts rows after that join counts lot reports rather than drugs, which is exactly the kind of error the document store could never have surfaced.

Two independent checks landed on the same 62 records. The integrity gate found 857 orphaned AACT child rows. The reject lane, separately, had set aside 62 study rows where an unescaped | inside an official title widened them past their column count. Those 62 studies are precisely the ones whose sponsors, conditions, interventions and sites are orphaned — all 857 of them. Neither check knows about the other, so agreeing is the evidence that both work.

What I’d do differently

Embed the interventions field from the start. The DS-8201a miss was avoidable — the synonyms were sitting in the structured payload the whole time. Fixing it now means re-embedding the 84,600 trials already in the corpus.

Write the end-to-end smoke test before the unit tests. Four defects hid behind a fully-mocked suite: the JSON codec that made every cross-source tool crash on its first real call, an openFDA date filter that 500s because httpx percent-encodes +, a safety-profile prompt that carried no label text so the model filled the gap from its own knowledge and asserted a boxed warning the record doesn’t have, and a server that exited at startup when Postgres was down — which Claude Desktop reports as “Connection closed”, pointing at the wrong component. Each had passing unit tests over it.

Limits

No clinical evaluation. This is not decision support and has not been evaluated clinically.

The corpora are scoped subsets, not mirrors: cancer-focused trials, sampled FDA data, one PubMed topic.

The staging layer is pinned, not live. Four named FAERS quarters and one named AACT snapshot, loaded on demand. The retrieval corpus has a refresh pipeline; this does not, and the staged counts are four quarters of FAERS and five of AACT’s 49 tables rather than all of either.

It is not a pharmacovigilance pipeline. It loads FAERS and computes nothing about causality — no disproportionality statistic, no incidence rate. FAERS reports are voluntary, unverified and duplicated across reporters, so the counts are counts of reports, not of events in the world.

No hosted instance. The try-it path is local — docker compose up -d brings up a queryable instance with no API keys.

Links

The repo is linked at the top of this page. Inside it, docs/EVAL_RETRIEVAL.md carries the retrieval evidence — every question, every miss, and the error analysis — and the README’s “Try it in five minutes” section is the docker compose up -d path.