Demand pull v2: no detectable gain (IDEA-006, draft, not published)
Result. Adding five dated buyer-demand features to the v4 lab ranker did not improve it. Mean AUC gain over two folds: -0.0015 (95% CI -0.0051 to +0.0021). The test failed all four required conditions and the reported fifth. No v5 is built on demand grounds; the ranker is unchanged. Run once on 2026-11-03 under config/rubrics/model-v4d-S0.1.json. File: data/processed/idea006_v2_result.json.
| fold 1 (2019) | fold 2 (2021) | |
|---|---|---|
| v4 AUC | 0.7417 | 0.7424 |
| v4 + demand AUC | 0.7403 | 0.7408 |
| v4 average precision | 0.0387 | 0.0368 |
| v4 + demand average precision | 0.0386 | 0.0353 |
| hits in top 100, v4 / v4 + demand | 7 / 8 | 9 / 9 |
| precision at 25, v4 / v4 + demand | 0.08 / 0.08 | 0.12 / 0.12 |
| required condition | needed | got | met |
|---|---|---|---|
| lower 95% bound above 0 | > 0 | -0.0051 | no |
| gain above area-shuffle p95 | > 0.0045 | -0.0015 | no |
| gain above 0 in each fold | both | -0.0014, -0.0016 | no |
| gain above 0 removing each one area | all 17 | 2 of 17 above 0 | no |
Reported, not required: cutoff-permutation p95 is 0.0011, and the gain is below it. The gain is also inside the range of white-noise tables (median -0.0026, p95 0.0037).
Baselines on the same rows (AUC).
| fold 1 | fold 2 | |
|---|---|---|
| random | 0.534 | 0.513 |
| citation rank | 0.679 | not run |
| rubric without economics (v0.4) | 0.692 | 0.700 |
| plain LLM | not run | 0.721 on a case-control sample, where v4 is 0.744; not comparable with the full-fold row |
Hard rule 6 is only partly met on stored data: citation rank has no fold 2 read, and plain LLM has no fold 1 read and no full-fold read. No fresh LLM run was made (no spend).
The most important caveat: this design could hardly have passed. The post-run audit planted a real signal to check the harness, and it registered cleanly (+0.030 AUC at a moderate strength, +0.094 at a strong one). So there is no bug flattening an effect. But the audit also tried an oracle-style area-level feature, each area's own actual hit rate, one static number per area on all five features. It added only about +0.004 AUC (+0.0025 and +0.0054 by fold), about the size of the area-shuffle threshold of 0.0045. This is a rough benchmark, not a ceiling: it can't represent timing, and the raw-exposure secondary below came out slightly higher (+0.0054). Still, it points the same way: area-level features seem to have only about +0.005 of room on these folds, whatever they measure. "Adequate power" was met only formally, by a proxy MDE of about 0.025 that this benchmark sits far below. So this closes only this retrospective, area-level formulation. It is weak evidence against demand pull as an idea, and useful evidence that a coarse area signal is the wrong instrument: a lab-level or asset-level match is what would have room to matter.
What this does and does not say. It is "no detectable gain", not "no effect". The interval here is tighter than the MDE, but it is anti-conservative: features vary across about 51 area-by-cutoff cells, not 50,000 labs. Company-clustered intervals are wider: fold 1 -0.0033 to 0.0005, fold 2 -0.0102 to 0.0070. It says nothing about the thesis in its full form. This tested dated trial counts by area, not the matched buyer problems of the problems catalog or the economics layer, which are still untested. It also doesn't mean v4 is good in any new sense: v4's own skill claim waits on the prospective reads.
Secondaries, descriptive only; they cannot rescue the primary.
- All-sponsor table: -0.0013 (-0.0041 to 0.0016). Drug types only: -0.0010 (-0.0045 to 0.0026).
- Features alone: level -0.0010, growth +0.0001, breadth -0.0009, depth +0.0006, per-lab -0.0003. All intervals include 0.
- v4 + the 17 raw area exposures, no demand numbers: +0.0054 (-0.0013 to 0.0120). The interval includes 0. It suggests that which therapeutic area a lab works in may carry more than the demand numbers attached to it, but this was one of many looks and needs its own pre-registered test before it means anything.
Why to be careful with this result.
- This is a second vocabulary. The first missed its coverage floor at 69.16%; this one was written knowing that aggregate miss and the first check's false rows, and reached 78.5% and 78.3%, with blind precision of 94% and 82% (trials are 86% if a single-organ cancer counts only as oncology, still above the 85% floor). A second vocabulary raises the chance of a false pass; it did not pass.
- One required condition was moved to "reported" before the read, after the audit showed it was nearly unsatisfiable. That lowered the bar. The test failed anyway, including that line.
- The cutoffs were already used to choose v4, so this was never a fresh test.
- Registry sponsor, conditions and phase are 2026 state; only
postedis dated at registration. - Untested: v4 already holds
area_heatand 14kw_modality shares, so overlap with what v4 knows might explain part of this.
Rule that fired, quoted from the plan. It is applied to this owner-authorized second test; the first attempt was cancelled at its coverage floor and no test ran then. "an executed test at adequate power (MDE <= 0.03, white-noise placebo OK, vocab floors passed) that fails its pass rule closes the retrospective prediction line: no v5 on demand grounds, R3 closed, R1 stays shipped, the null ships as a dated what-failed."