All documents · source file docs/reports/2026-11-09-extraction-use-report.md · changed 2026-11-10 · sha256 18818e00c868

What I did with 202,638 paper extractions after the model test failed

2026-11-09. Plan: docs/plans/2026-11-09-extraction-use-plan.md. Everything here ran on this machine, at no API cost, and nothing was pushed.

1. The short version

The setup. A model read the title and abstract of 202,638 lab papers and extracted structured facts, for $47.75. The one planned test asked whether those facts improve the lab→company ranker. They did not (mean AUC gain −0.0007).
The decision. Stop asking the extractions to predict. Use them to describe. I built the **Science Lens**: a point-in-time search and science profile over labs and their linked companies.
It works, with measured accuracy.
- Extraction accuracy on 200 fresh papers: claims supported 98%, paper type right 97%, model system right 95%.
- Search: precision at 10 is 0.86 with the extraction filter against 0.58 for plain keyword search on the same abstracts (+0.28, 95% interval +0.19 to +0.36, 19 queries, 0 losses).
No predictive win is claimed. No unspent past labels exist to test one. I froze one small forward test that reads on 2028-09-30.
The main weakness. Every accuracy number comes from AI readers. No person has checked the extractions.

2. What I decided and why

Option	Verdict	Reason
Back up and load the data	Done first	Unblocks everything, costs nothing.
Search and profile tool (Science Lens)	Primary	Its value is retrieval and structure. Accuracy can be measured now. It ships whatever any backtest shows.
Company-stage model on past labels	Cut	Every company label from 2016 to 2026-03 already chose v2, v3 and v4. A "win" there would be a search result, not evidence.
Freeze v4 + extraction for labs	Cut	The same features just failed for labs.
Forward test on the 2026-03-31 cohorts	Kept as upside	The only labels nothing has used. Small, and it reads in 2028.
Post about the null alone	Folded in here	Nobody uses a null. Section 6 reports it.

Success and the stop rule were written before any build step. The stop rule (claims supported below 80%) did not fire.

3. What I built

A backup. data/backups/diaspore_extractions-2026-11-09.db.zst, 197 MB, verified. It is on the same disk as the original.
The load. The extractions are now in each panel database's extractions table, with as_of_date = the paper's publication date: 43,551 rows (main, 2014 panel), 51,394 (2019), 96,240 (2021), 92,041 (live 2025). That is all 202,638 distinct papers. New columns as_of_date, output_type and source_id are added by db.migrate; db/schema.sql is untouched.
The Science Lens (src/diaspore/lens.py). It answers three questions as of any date:
- Search: which labs have work on a topic, with what evidence.
- Lab profile: what one lab's extracted papers claim, in what model system, with what effect and what stated limits.
- Company profile: the science of a company's founding labs. This is the "feed in a target" use.
- It has a command line, three MCP tools (lens_search, lens_lab, lens_company) and a Markdown brief. Every dated row passes through diaspore.backtest.gate. Every output prints the extraction model, the prompt version, "no human has verified" and the measured accuracy.
A forward test. docs/prospective/lens-ladder.frozen.json, with its pre-registration.

Try it:

PYTHONPATH=src ~/.venvs/diaspore-sk122/bin/python -m diaspore.lens brief "KRAS inhibitor" --as-of 2025-12-31 --types product

Three sample briefs are in data/processed/lens/. They are not committed, because they pair named PIs with name-matched companies and some of those matches are wrong.

4. Results, with baselines

Extraction accuracy (200 fresh papers, two AI readers, both must agree)

Check	Share correct	95% interval	Threshold set in advance
Claim supported by the abstract	97.9%	93.6% to 99.3%	≥ 95%, met
Paper type right	96.9%	92.2% to 98.8%	≥ 85%, met
Model system right	94.9%	89.6% to 97.6%	≥ 85%, met
Scale right	97.4%	92.9% to 99.1%	none
Effect numbers supported	98.6%	92.1% to 99.8%	none
Limitations supported	93.1%	85.4% to 96.8%	none

Across all 202,638 extractions, 99.3% of the 637,539 quoted numbers appear verbatim in the source text. Detail: docs/methodology/lens-accuracy.md.

Arm	Precision at 10	Relevant labs in top 10
Titles only (what the database could do before)	0.60	3.1
Title and abstract keyword search	0.58	5.5
Lens: same search, product-directed papers only	0.86	8.0

The Lens beat keyword search on 15 queries, tied on 4 and lost on none. One query ("wearable glucose biosensor") found nothing in any arm. Detail: docs/methodology/lens-search-eval.md.

Forward test (no result yet)

A fixed, unfitted "evidence ladder" score for companies with a known founding lab, frozen on the 2026-03-31 cohorts: 109 of 871 (S1), 231 of 783 (S2), 144 of 391 (S3) companies scored. It reads once, on or after 2028-09-30, against v4, random, the rubrics and the plain-LLM ranking on the same rows. I expect about 40 events, so only a large effect can show.

5. How a skeptical investor should attack this

"AI graded AI." The extractor, the readers and the judge are all Claude models. They can share blind spots. A human spot check of 20 to 30 items is the cheapest fix and is on the owner queue.
"The search test is partly circular." The judge counted a lab as relevant only if its work was product-directed, and the filter selects papers the extractor called product-directed. The result shows two models agree on reading abstracts. It does not show the labs are good investments.
"Three abstracts is not a lab." Correct. The corpus is each lab's three most recent last-author papers at four year-ends, and 32% of papers had no abstract. A lab's key paper can be missing.
"Your company links are name matches." Correct. Lab-to-company links come from officer names on SEC Form D filings. The first sample brief links a well-known immunology lab to an unrelated LLC. The Lens labels every link "possible" with its match confidence. This is the weakest part of the company profile.
"Human evidence is mostly clinicians." Many "therapeutic candidate, human" papers are clinical studies of existing drugs, not spin-out science. The evidence level describes the paper, not its novelty.
"It needs every query word." There are no synonyms. Odd phrasing finds nothing.
"None of this predicts anything." Correct, and the report does not claim it does.

6. What failed

The original test. Five extraction features added nothing to the lab→company ranker: AUC gain +0.0024 and −0.0037 in the two folds. Record: docs/prospective/extraction-preregistration.md.
A retrospective company-stage test was not possible. No past company label is unspent.
My first pre-registration draft was wrong and loose. The auditor found a misstated horizon (36 months where the frozen files say 24), an unspecified tie rule and bootstrap, and unpinned inputs. All were fixed before the freeze. A second audit asked for the secondaries to be marked descriptive; done.
My first Lens version leaked existence. It showed a company's name for a date before its first filing, and accepted a malformed date as a late cutoff. The auditor caught both. Fixed, with tests.
Two guard tests caught my code. One forbids attaching databases; one requires every reader of company tables to be listed. I rewrote the loader and listed the three readers.
One search query returns nothing, and "GLP-1 obesity" scores 3 of 10.
Two tests fail in this environment and did before this work: tests/test_release_s1.py, a last-digit float difference on this Python. I confirmed it on the commit before my first change. Two more (test_site, test_views_gate) fail unless the markdown package is added to the test run; I added it. All four are already on the owner queue. Final run: 1,963 passed, 2 failed, 5 skipped.

7. Checks by independent agents

Milestone reviewers checked M0–M1 (pass), M2 (pass with notes), M3–M4 (pass with notes). The M3–M4 reviewer recomputed every number with its own script and found no mismatch, redrew the sample from its seed, and agreed with 15 of 16 spot-checked verdicts.
**backtest-auditor** audited the freeze before it was written, twice, and the Lens's point-in-time behaviour. It computed nothing on real features.
A final acceptance checker ran the six acceptance steps from the plan on the real data and returned "ACCEPTANCE: PASS", with the two known test failures as the disclosed exception. It also caught that this report had understated the known test failures; corrected above.

8. What is left, and what needs you

Off-machine backup. Both copies of the extractions are on this Mac. Say where a copy may go.
Push and timestamp. The freeze's only date proof is a local commit. A push and an OpenTimestamps stamp need your approval.
Human spot check. 20 to 30 items from data/interim/lens/m3/items.json.
Decision-log lines (I did not commit docs/decisions.md, which holds the other session's edits):
- 2026-11-09: extractions loaded into every panel's extractions table; as_of_date = published_date; columns added through db.ADDS, schema.sql untouched.
- 2026-11-09: extraction data is used for description (Science Lens), not prediction; no retrospective company-stage test, because no company label is unspent.
- 2026-11-09: lens-ladder frozen on the 2026-03-31 cohorts, sha256 7d5384d6…, unfitted, read once on or after 2028-09-30.
- 2026-11-09: a Lens entity with no dated row by as_of is not shown; a malformed as_of is refused.
For the IDEA-015 session. lens.company_science(con, company_id, as_of) and lens.lab_science(...) return gated, sourced science rows it can use. I edited none of its files.
Next steps I would take, in order: fix link quality (it limits the company profile); add synonyms or embeddings to search; extract full text for labs that reach a shortlist (needs API spend); put a Lens page on the site.

9. Addendum, 2026-11-10: fixes after the owner read a brief

Search ranking and filters did not change, so the measured precision still describes the tool.

Placeholder text is now null. A field that starts with "Null", "Not stated", "N/A" or similar is shown as empty in every output, not only in briefs and not only for limitations.
Company links are tiered. Each link carries a basis. Only pi_officer (the PI named as an officer or director, full name and middle initial matching) is shown as a company link. A lab member's name match, a PI match without the middle-initial check, and an S-1 licence mention are shown as "weaker, unconfirmed".
Meeting abstracts are marked. A supplement DOI is labelled "(meeting abstract)".
The evidence level says when it may be too low. If a product-directed paper has no model system classified, the level reads "at least".
**Every lab carries freshness,** measured at the as-of date: its latest last-author paper, the gap in months, and a "possibly inactive" flag at 24 months or more.
Not fixed, and why:
- A wrong model_system in an extraction can only be fixed by extracting again, which costs API credit.
- The institution comes from OpenAlex when the panel was built. It is not dated and can be a parent organisation (a Duke-NUS lab shows as Duke University). Fixing it needs dated affiliations per paper.
- Ranking still breaks ties by text score. Changing it would void the measured precision.

For the science pipeline session: call lens.search_labs, lens.lab_science and lens.company_science. New keys since the first version: freshness, evidence_caveat, linked_companies[].basis, papers[].meeting_abstract. Placeholder fields arrive as null.