Model and rigor

1. Amendments leaked the future. A Form D amendment filed after the cutoff changed what a pre-cutoff round looked like. Fix: every filing is stored with its own public date, and rounds are rebuilt "as of" any date.
2. The S1 universe was selected on its future. Companies with only SBIR awards entered the candidate set because they later filed a Form D. Fix: S1 requires a Form D by the cutoff.
3. Industry tags from later filings chose who was a candidate. Fix: scope is decided per filing, through the date gate.
4. Names leak outcomes. Given company names, the LLM baseline's S2 AUC rose from 0.74 to 0.84; some scorers cited "known" companies. Lab names showed no measurable leak. Rule: company-stage LLM scoring is anonymised.
5. Product names leak too. Technology scorers named products from FDA and trial text. Fix: a redaction pass using a vocabulary built from 2010–2014 paper titles.
6. Two of seven top-ranked spinout links were false. One was a Michigan namesake of a Stanford PI; the other was a "C." initial matched to "Clare". Fix by rule, not by hand: a PI match needs a state match, and an initial-only match needs an agreeing middle initial. It removed 146 of 569 S0 labels and cut the rubric's P@25 from 0.28 to 0.16.
7. Firms were merged retroactively. Investor-tier clusters used fund filings from after the cutoff. Fix: clusters are built per date.
8. Ties were broken by database id. Id order is ingestion order, which correlates with filing age. Fix: P@k is the expected value over tie orderings, for every ranker.
9. A filter was written after seeing outcomes. The non-startup filter came after top-25 tables with outcomes had been viewed. It is disclosed, and we checked the direction: it lowered rubric lift.
10. The filter also deleted real startups. Bare "imaging" and "dialysis" removed eXo Imaging and Home Dialysis Plus. Fix: facility words need facility context.
11. "Underfunded technology" didn't survive. It looked real on FDA milestones, which were an artefact of devices accruing clearances. On VC-relevant milestones it is chance.
12. Long biomedical summaries were blocked. A safety classifier stopped about a third of the blind economics replies. 40 of 200 labs have no economics profile, recorded as missing, not filled in.
13. The fix for #6 threw away real founders. We hand-checked 20 of the 146 removed labels: 9 were real (PIs who moved, companies registered in a hub state or abroad). False-negative rate 0.45 (95% CI 0.26–0.66).
14. The fix for the fix failed its holdout. A "rare name" exemption restored 61 links; on 15 fresh restorations, 6 were namesakes (an investor, a Toronto businessman, an MIT Sloan professor). Two further ideas from that holdout made things worse at scale and were dropped the same day. What we kept: an out-of-state match counts only if a co-author of the PI is on the same company's filings. It adds 21 links and removes none; S0 still undercounts out-of-state spinouts.
15. Amendments of pre-2009 paper filings were dropped. An amendment whose original was a pre-2009 paper Form D had nothing to attach to, so its money vanished (Gliknik: $47.8M counted, $50.7M filed). An amendment of an amendment also attached to the wrong offering.
16. Prediction wording was edited after registration. The resolution criteria for 17 of 18 calls were reworded a day after registration and after a check of the gap period. It changes one outcome: Gliknik's September 2026 amendment voids call #10 under the registered text but not under the edited one. The registered text governs; the edit is disclosed in the ledger.
17. The provenance hashes could not be reproduced. Three aggregate source hashes had no written recipe and one could not be recomputed at all. Fix: one documented recipe, recomputed rows, and a hash on every cache-backed source.
18. The first IPO-filing founder rule was 74% right. Reading "our scientific co-founders" out of S-1s found Icosavax's founders, but a fresh 35-pair hand check showed the word "founder" in a director's bio usually belonged to another company. The stricter rule (the phrase must tie the person to this company) scored 27/30 on new pairs and is the one in use.
19. Our "fresh" lab test was not fresh at first. The learned model's lab design validated on outcomes that overlapped the holdout's years. The auditor caught it before any lab result existed; the design was re-issued (v2.1) with a shorter validation window and committed before the test.
20. The first learned model was tested on dates we had already looked at. Its result was real but pseudo-prospective. The second model was scored once on dates nobody had looked at (labs end-2021, companies end-2022).
21. SEC filings miss many academic founders. Of the 25 top "no company yet" labs on the current list, 10 heads had already founded companies (TScan, Epic Bio, Synthekine, Century, …) that appear in no filing our linker reads. Every test's labels undercount hits for this reason.
22. The hand-set score never beat simple rules. In all eight original tests it was within noise of the best one-line rule. What finally beat it was learning from outcomes, tested properly: on the fresh tests the learned model passed for labs and for new companies → seed, failed for seed → A and A → B, and tied a blind AI model given the same evidence.
23. New data on who bought each round did not clear the bar. Model v3 added Form D purchase terms (investor count, check size, brokers, convertible notes) and was pre-registered against end-2023. Its top 25 for new companies was the best yet (8 of 25), but no stage's interval cleared zero against our hand-set score, so it was reported and not used.
24. The end-2023 test was not fully fresh. Many of its outcomes (2024-25) had already been seen as outcomes of the end-2022 test before v3 was designed: 20 of 55 new-company hits, 59 of 76 seed → A. The audit caught it; the site now calls it an overlapping re-test, not a replication.
25. Grant types told us nothing new about labs. SBIR/STTR grants held by the lab head, NIH Pioneer and New Innovator awards, career-stage awards, milestone-driven grants and the trend in NIH dollars added nothing over the existing lab signals (mean average precision +0.0004 across three seen tests), so they stayed out of model v4.
26. There is no fresh past date left to test on. Every cutoff with known outcomes has now been looked at. The next model (v4) is therefore tested on the future: its rankings of today's labs and companies were frozen, hashed and committed on 22 October 2026, and are scored once the outcome windows close (companies 2028, labs 2030). Its first spec claimed no outcome existed yet; the pre-freeze audit found early-2026 filings already in the data, so the spec was re-issued before any v4 output, with a read that ignores outcomes public before the freeze. The citation baseline at the end-2021 lab test is still not run (OpenAlex budget: about 850 of 27,000 labs fetched).
27. Our labels missed most spinouts at the top. A blind web audit of the end-2021 lab test's top 25s (protocol committed first) found lab heads who founded companies in 2022-25 that no SEC filing links to them (Doudna, June, Porteus, Gray, ...). Corrected top-25 counts on a strict reading: learned model 3 → 9, our score 1 → 9, the most-papers rule 2 → 3, random labs 1 → 1. The labels missed about 70% of spinouts at the top; the simple rule looked closer to our methods than it is, and the learned model's edge over our score vanishes. My first tally re-checked a hand-picked subset and showed the model at 12; the auditor caught it, and one rule now applies to every lab. Famous lab heads are easier to find on the web, so part of the gap may be findability.
28. The model's lab edge did not survive better labels. A second blind audit sampled 400 labs across the whole 2021 ranking and weighted them back to all 27,415. SEC links find an estimated 41% of 2022-25 spinouts (26-68%). On corrected labels the learned model and our hand-set score tie (AUC 0.747 vs 0.746); the model's published +4.5-point lead on labs vanishes. Both still beat the most-papers rule by about 12 points. Twice the auditor caught my tally first: undated companies counted as non-hits, and a claim that rested on one lab.

What did not fail: every backtest number has baselines, the date split lives in one function, and an independent auditor agent signs off before a number is reported.