filing-delta

Overview

A company’s annual report repeats most of last year’s risk factors and management discussion, word for word or nearly. The few paragraphs that are new, gone or rewritten are what a reader actually needs, and they are buried in a hundred pages of text that did not change.

filing-delta compares a company’s filings period over period and shows what changed, with every change tied to the exact character span it came from. It covers 10-K filings with a period of report from 2018-01-01 to 2025-12-31 and 10-Q filings filed 2025-01-01 to 2026-09-30, for 11 companies — 144 filings in all, 88 10-K and 56 10-Q. No 8-Ks: they were planned for the question set and dropped.

It was built with Claude Code, and every human label in it is mine.

The problem

A summary that says “the risk factors now emphasise supply-chain concerns” is no help unless the reader can see the sentence it came from, because the next question is always “where does it say that?”

Without a way to check, a reader either re-reads everything or trusts a paraphrase they cannot verify. Comparing the documents by hand takes hours and is easy to get wrong.

Constraints

The source has rules. SEC’s EDGAR asks for a declared identity on every request and at most 10 requests a second. The client refuses to start without the identity, holds itself to 5 a second, and caches every response. A re-run over the full corpus makes zero requests.

The text is unaudited. Risk factors and MD&A are management’s narrative, and MD&A figures are often non-GAAP. The tool can report what the text says and how it changed — never whether it is true, and never what it means for a security.

A stranger has to be able to check every number without an API key. Every model response is recorded by request hash and shipped as a 63 MB data release, so the published numbers replay exactly.

One labeller. Every human label is mine, made at gates alongside the build. That limits how many labels exist, and it shaped how they were collected.

Architecture

The design splits on one line: everything that produces evidence is deterministic, and the one stage that makes a judgment is a model, scored separately.

parse       narrative sections out of filer HTML; every section ends in exactly one
            outcome — ok, partial, failed or absent — with a reason
diff        paragraph level, period over period: added, removed, reworded, moved.
            No model. Byte-identical on every run
classify    each change material or cosmetic. The only model judgment in the diff
            half; reads the diff's output and never feeds back into it
answer      questions with citations that must be exact substrings of the stored
            section, or abstain

The answer type admits nothing else: one unverifiable citation turns the whole answer into an abstention.

Key decisions

Chose a deterministic diff over handing a model both years’ sections. Asking a model what changed is much less work to build, but its output cannot be tested, will not reproduce, and does not point at a span. The diff is the evidence, and evidence has to come out the same every time.

Chose an outcome model before writing a parser. A survey of 18 real filings found four layouts where one was expected. JPMorgan and Exxon make Item 7 a single sentence pointing elsewhere. GE has no Item headings in the body at all, only a cross-reference index at the back. A naive “find Item 7” would have returned a one-sentence stub as JPMorgan’s entire MD&A and reported success. So pointers, cross-reference indexes and items a filer marks “not applicable” each became a named outcome, and a section that cannot be located has to say why.

Chose to make removing text need structural evidence. Two of the early parser fixes deleted less, not more. A rule treating any bare number as a page number had been eating table cells; a rule treating text repeated near page breaks as a running header had eaten JPMorgan’s “COVID-19 Pandemic” subheading. Removing text is the dangerous operation.

Chose one honest row in a log over a special case nobody can see. Pfizer’s FY2018 and FY2019 MD&A are incorporated from an exhibit outside the primary document. They stay failed, with that reason, rather than being fetched from the exhibit by a special case. The same root cause returned three milestones later — Pfizer FY2019’s primary document carries only 77 tagged facts, the cover page — and the financial-data join uses the SEC’s company-facts API for that one filing and records why.

Chose the instance criterion for paragraph alignment. “Same type of disclosure” and “same instance” give different answers when a template is reused for a new event. The instance criterion is the one the product claims. It is the stricter reading: on the same labels, the best simple rule fell from an F1 of about 0.90 to about 0.82.

How it’s verified

Every result regenerates on a fresh Linux runner with no API key, on every push.

Check	Result
Section boundaries	28 human-confirmed golden sections across 14 filings, every layout archetype and HTML era
Parse rate	270 of 272 present sections ok (99.3%), 16 verified absent, 2 failed; in 184 of 288 sections the heading and the table-of-contents anchor agree
Diff	17 mutation-checked unit tests; re-diffs byte-identical
Paragraph alignment	Against 70 blind, stratified human labels: precision 0.83 (95% CI 0.74–0.93), recall 0.80 (0.66–0.93)
Material vs. cosmetic	122 blind human labels, scored once with a frozen prompt: agreement 77%, “material” precision 0.89 (0.81–0.96), recall 0.80 (0.68–0.89)
Financial data	Inline XBRL read from each filing against the SEC company-facts API: 65,243 of 65,243 values equal; 132 of 132 document period ends match; 44 mutation-checked tests
MD&A headline claims	No model. Of 1,085 claims, 194 match a tagged value, 12 use non-GAAP wording, 1 has no match, and 878 cannot be checked because they are about segments or other periods
Cite-or-abstain	16 forged-citation tests — fabricated, altered, wrong offsets, wrong filing, one bad citation among good — all caught
Questions	On 60 questions: 37 of 37 answerable answered, each citing the key’s passage; 23 of 23 correct abstentions; 37 of 37 citations verified as exact substrings
spancheck on the recorded run	Provenance 91 of 91 citations; abstention 60 of 60; answer entailed by its own citations 59 of 60; same decision on a re-run 60 of 60
Clean-clone reproduction	On a fresh Linux runner with no API key, every committed report and golden file regenerates byte for byte

Three of those are agreement figures, not independent measurements, and that matters more than any single number above.

The question-answering key is model-built: two models agreed on 58 of 60 questions and I decided the other 2 — both among the 12 questions I wrote rather than the 48 Claude drafted. The classifier and alignment figures are agreement with one human labeller, me. And the hand check of the MD&A claim matcher is not evidence at all: the labels were made with the matcher’s answer already shown, and 0 of 99 were overridden at about five seconds each.

The parse rate has a limit of its own. 99.3% is the parser grading itself through its own sanity checks. Two things made it believable, and both came from outside the parser: 28 section boundaries confirmed by hand against the filings, and the diff catching an error the parser had passed — JPMorgan’s MD&A had lost its last page in 5 of 8 years, because JPMorgan prints its running header only on alternate pages. Inside one filing a missing page is invisible. Across years it shows up as text appearing and disappearing at the edge of the section.

What broke

The wrong assumption that took the most iterations: that a prompt rule could decide when it is safe to say “the filing doesn’t say this.”

Questions like “does the 10-K discuss X?” often have the true answer “not in those words, but here is what it says nearby.” Version 2 answered those in that form, quoting the nearby text. On one question about Netflix paid sharing, that turned an honest abstention into a confident false one — “the passages never mention paid sharing” — because retrieval had missed the filing’s own wording, “multi-household usage”. Every grader passed it, because the quote was real and the answer followed from what had been retrieved.

Version 4 makes the model list other wordings and search again before any “doesn’t say” answer. The first lexical check for that was a no-op — “sharing” and “shares” stem to the same word — and was caught only because the test demanded the case actually trigger. Then the second search over-answered two questions that asked for a specific figure, by finding nearby figures. The fix there was a rule in code: a “doesn’t say” answer to a figure question is refused.

Each fix moved the failure somewhere else, until the boundary moved out of the prompt and into code. The report now also lists every answer whose quotes miss the key’s passage, so a false absence cannot hide inside a 100% abstention score.

The first cross-check of the financial data said 99.83%. The extractor had kept a rounded duplicate — “$4.9 billion” — over the exact value. It reads 65,243 of 65,243 now.

The materiality error analysis tested its own first explanation and mostly lost. The classifier’s misses cluster on restructured passages, and the first explanation was that the labeller had not seen the context the model saw. A re-check with the context shown tested that: I held my original label on 7 of the 9 items it applied to. The misses are a real difference in reading the guide, and they are reported as one.

Limits

It is not investment advice and makes no forward-looking statement. It reports what filings say and how the text changed.

It does not verify management’s narrative. Only a narrow slice of MD&A figures is checked against tagged values — whole-company headline figures for the filing’s own period, which is about 18% of such claims.

It cannot prove something is absent. “Doesn’t say” means “wasn’t found”, after a second search for other wording.

The accuracy figures are agreement, not independent measurement. The question-answering figures are agreement with a model-built key; the classifier and alignment figures are agreement with one human labeller.

“Material” is a labelled text-change category, not a legal or accounting materiality determination.

Coverage is exactly the corpus. 10-K with a period of report from 2018-01-01 to 2025-12-31, and 10-Q filed 2025-01-01 to 2026-09-30, for 11 companies. No 8-Ks, no amendments, no exhibits, and no financial statements read as narrative. Questions outside that range get an abstention.

What I’d do differently

Settle what a label means before anyone labels. The type-versus-instance question cost a second labelling pass, and it was the real design decision behind the alignment threshold.

Treat “the filing doesn’t say X” as a retrieval problem from the start. A claim of absence needs a stronger search guarantee than a positive answer does, and three pipeline versions taught that the slow way.

Links

The repository is linked at the top of this page. One command takes a fresh clone to the read view with no API key, in about four minutes including the 63 MB data release:

git clone https://github.com/tjromack/filing-delta.git && cd filing-delta && uv run filing-delta demo

docs/parse-failures.md is the log the build grew around, docs/CASE-STUDY.md is the long-form write-up, and tools/reproduce.py rebuilds every published number and fails if any committed report changes by a byte.