Methodology

Question

Given only what was public at a cutoff date, can a scoring system rank candidates at each step of the biotech pipeline — lab → pre-seed → seed → Series A → Series B — so that those that later advanced appear near the top, better than simple baselines?

Stages (config/project.yaml → stages)

Company stages are re-run at a 2016-12-31 cutoff (robustness: the outcome window ends before the 2020–21 funding bubble).

The gate (no lookahead)

diaspore.backtest.gate.split(events, cutoff, horizon) is the only place an event date is compared to a cutoff: dated ≤ cutoff → may be a feature; dated in (cutoff, cutoff + horizon] → may be a label; undated → neither. Signals are written with as_of_date and read only through score.features.features_as_of. Tests in tests/test_gate.py, tests/test_signals.py and a leakage trap in tests/test_run.py (a post-cutoff round that must not change features).

The "knowable" date of each fact:

• paper: publication date · grant: NIH award notice date · financing: Form D filing date (not first-sale date) · IPO: first public-company-only filing on EDGAR · company↔lab link: date of the earliest Form D naming the person.

Data (all public)

• Papers / labs: OpenAlex, in-scope subfields (config), 40-university US panel chosen by 2010–2014 output.
• Companies / rounds: SEC Form D quarterly datasets, industry Biotechnology|Pharmaceuticals, pooled funds excluded. Every filing (original and amendment) is stored with its filing date in form_d_filings. Rounds are derived as of a date (form_d.rounds_as_of): only filings public by that date count; amendments raise the amount of the round they amend; $0 and post-IPO filings are not rounds; an original within 90 days of the previous round is a tranche unless its own amount implies a higher stage. Features use the view at the cutoff; labels use the view at cutoff + horizon. So an amendment filed after a cutoff cannot change a round at the cutoff (backtest audit B1; regression test in tests/test_form_d.py).
• Stages: inferred from Form D amount and sequence (bands in config, typed inferred). Never go backwards.
• Public status: EDGAR submissions (424B4/10-Q/10-K…) across the full filing history (older-history files included). Company-stage runs refuse to start if any candidate lacks a complete EDGAR check.
• Grants: NIH RePORTER, matched to lab PIs by name; SBIR/STTR matched to companies by normalized name.
• Company ↔ lab: Form D related persons (officers/directors) name-matched to lab PIs/members; full-name matches 0.8 (0.9 with agreeing middle initial), initial-only 0.5; names shared by >3 authors rejected; officers of public companies ignored.

Scoring

Rubrics are pre-declared JSON (config/rubrics/v0.2-S*.json), registered in rubric_versions, and cannot be edited in place (the runner refuses). A component = mean percentile rank of its signals among that stage's candidates; missing components are excluded, not zeroed.

Metrics

precision@k (with 95% Wilson interval), recall@k, lift vs random; base rate and n always reported. k = min(25, n/10).

Baselines

A run's verdict is INCOMPLETE unless every declared baseline ran. plain_llm is compared on its sample by average precision and ROC AUC (precision@k on a small sample is meaningless).

plain_llm (Claude Opus 5.5) gets the same pre-cutoff evidence with names withheld, on a seeded sample; the rubric is re-scored on the same sample (AP, ROC AUC). Two routes: the Batches API, or in-session blind subagents (/llm-baseline) that see only anonymized batch text under opaque ids; any batch whose scorer used a tool is discarded. The report states which route produced the number. A --named variant measures memorization. Baselines that could not run are stored as not_run with the reason, never silently dropped.

Known threats

1. Label incompleteness. Stealth, non-US and never-filed companies are invisible → some misses are hits.
2. Link precision. Name matching can attach a common-named person to the wrong lab; needs a hand-audited sample (evals/) before headline numbers are published.
3. Board seats ≠ founding. A PI on a startup board may be an advisor, not a founder.
4. Inferred stages. A $16M "seed" is labelled Series A by amount.
5. Bubble years. 2020–21 inflates company-stage base rates (robustness cutoff addresses).
6. Author disambiguation is today's OpenAlex, not the cutoff's.
7. LLM memorization for plain_llm (names withheld; named variant quantifies it).
8. Small k. With k = 25, precision differences under ~10 points are noise. Average precision is reported too.
9. Dead features at company cutoffs. Papers are ingested for 2010–2014 only, so origin-lab paper signals are ~0 at the 2016/2018 cutoffs (neutral in percentile scoring, but they add nothing).
10. Label rule chosen after a dry run. The PI/trainee-only link rule was introduced after a dry run on partial data (see decisions.md 2026-10-14). Disclosed; the rule is principled but was not pre-registered.

Technology value vs capital (added 2026-10-15)

Money raised measures investor belief, not technology value. Diaspore scores technology separately and compares the two.

• Scope. Companies: every issuer that ever filed a Form D tagged Biotechnology, Pharmaceuticals, Other Health Care, Hospitals and Physicians or Health Insurance (devices, diagnostics, digital health and care delivery included). Services usually have no technology to score and are reported as such.
• Evidence pack (value/evidence.py), all dated ≤ as-of, company name masked, **no funding, no location**: SBIR/STTR titles and abstracts (award notice date), FDA 510(k)/PMA/NDA/BLA decisions (decision date), industry clinical trials (first-posted date), papers of linked founder labs, SEC industry tag.
• Assessment (prompts/tech_value_v0.1.md): four anchored 0–4 scales — evidence strength, problem value, defensibility, translation feasibility — plus category and an explicit insufficient-evidence flag. Scored by blind subagents (opaque ids, no tools; batches with tool use discarded). No dollar figures.
• Tech score = weighted mean (config/rubrics/tech-v0.1.json, equal weights, pre-declared), 0–1.
• Capital–value gap = percentile(tech) − percentile(capital raised as of the date), within category. Positive = underfunded relative to technology; negative = overfunded.
• Validation (value/validate.py): over the next 60 months, milestones (FDA clearance/approval, trial phase advance, IPO, acquisition, licensing) and survival. Technology adds information only if AUC(tech + capital) > AUC(capital) and tech separates companies within capital terciles. Over- vs underfunded milestone rates reported with CIs.
• Known limits. De Novo decisions are not in openFDA. Acquisition prices and licensing upfronts are not yet ingested (needs press/8-K sourcing). Name matching to FDA/trial sponsors is exact-normalized and ambiguous names are skipped, so coverage is incomplete. LLM judgment is the measuring instrument; inter-rater agreement (re-scoring a sample) should be checked before publishing.

Corrections and known limits after the 2026-10-15 audit (docs/audits/2026-10-15-runs1-7.md)

Fixed (all runs rebuilt after these changes; earlier runs 1–7 are void):

• Scope is point-in-time (H1). Each Form D filing stores its own industry tag. A company is a company-stage candidate only if an in-scope filing is public by the cutoff; an S0 hit needs an in-scope filing by the window end. A later re-tag can no longer pull a company into an earlier universe.
• Sampled baselines are compared on their sample (H2, M1). plain_llm keeps no full-population P@k; the report and site compare it with the rubric on the same items by ROC AUC, with a paired bootstrap CI of the difference. Per-item scores are stored in baseline_scores.
• "New company" for S0 (M2, L4). Any public trace by the cutoff disqualifies a hit: Form D, SBIR, FDA, clinical trial, IPO, or the same trace of a same-name entity (LLC→Inc. gets a new CIK). Age uses every filing's incorporation statement public by the window end, including "over five years ago".
• Imports are guarded (L1). An LLM import refuses if any exported item's evidence differs from what the DB generates now, or if sampled ids are not candidates of the run; missing scores are dropped from both sides instead of counting as 0.
• Trial rows (L2) no longer store the registry's current status or start date; SBIR matching skips ambiguous names like FDA and trial matching do.
• S3 label (L3) reads its non-round components from config (label_components: series_b_plus, ipo).

Known limits (not fixed; stated in every report's caveats):

• IPO = first 10-Q/10-K/8-A in EDGAR, so reverse mergers and 12(g) registrations count as IPOs.
• Acquisitions are not ingested: an acquired company is a miss, and status is never used.
• S0 labels rest on name matches. Hits are reported by distinct company, and top-k links below 0.8 confidence are counted in each report; they have not been hand-verified.
• Lab signals at company cutoffs (2016, 2018) are constant because papers stop at 2014-12-31; the report lists constant rubric inputs. Extending paper ingest is future work.
• Rubric pre-declaration (v0.2) cannot be proven from git: rubrics and the first hindcasts were committed together. The order in which results were seen is disclosed in docs/decisions.md.
• Memorization: measured with --named and --masked (titles hidden) variants on a fixed subset.

IDEA-001 M9(i) estimator notes (added 2026-10-27, re-read r-m4)

• Non-finite bootstrap draws are dropped from every interval. In scripts/strat_audit.py, ci removes non-finite draws before taking the 2.5 and 97.5 percentiles, and returns [None, None] if none are left. A paired cell's draw is invalid when its paired set holds no hit; those draws are counted, per cell, in valid_resamples, and the triage makes any cell with fewer than 2,000 valid draws of 2,000 untriaged (iii). The unpaired lines (each ranker's own interval, the hidden-event count and the label-recall interval) carry no valid count, so a dropped draw there is not visible in the output, and a [None, None] interval reads as "no value" (triage (iv)). The RNG stream and every point estimate are unchanged: only the percentile input differs, and only when a draw is non-finite.