Diaspore

All documents · source file docs/methodology/demand-pull-v2-result.md · changed 2026-11-03 · sha256 4896dd4b23a0

Demand pull v2: no detectable gain (IDEA-006, draft, not published)

Result. Adding five dated buyer-demand features to the v4 lab ranker did not improve it. Mean AUC gain over two folds: -0.0015 (95% CI -0.0051 to +0.0021). The test failed all four required conditions and the reported fifth. No v5 is built on demand grounds; the ranker is unchanged. Run once on 2026-11-03 under config/rubrics/model-v4d-S0.1.json. File: data/processed/idea006_v2_result.json.

fold 1 (2019)fold 2 (2021)
v4 AUC0.74170.7424
v4 + demand AUC0.74030.7408
v4 average precision0.03870.0368
v4 + demand average precision0.03860.0353
hits in top 100, v4 / v4 + demand7 / 89 / 9
precision at 25, v4 / v4 + demand0.08 / 0.080.12 / 0.12
required conditionneededgotmet
lower 95% bound above 0> 0-0.0051no
gain above area-shuffle p95> 0.0045-0.0015no
gain above 0 in each foldboth-0.0014, -0.0016no
gain above 0 removing each one areaall 172 of 17 above 0no

Reported, not required: cutoff-permutation p95 is 0.0011, and the gain is below it. The gain is also inside the range of white-noise tables (median -0.0026, p95 0.0037).

Baselines on the same rows (AUC).

fold 1fold 2
random0.5340.513
citation rank0.679not run
rubric without economics (v0.4)0.6920.700
plain LLMnot run0.721 on a case-control sample, where v4 is 0.744; not comparable with the full-fold row

Hard rule 6 is only partly met on stored data: citation rank has no fold 2 read, and plain LLM has no fold 1 read and no full-fold read. No fresh LLM run was made (no spend).

The most important caveat: this design could hardly have passed. The post-run audit planted a real signal to check the harness, and it registered cleanly (+0.030 AUC at a moderate strength, +0.094 at a strong one). So there is no bug flattening an effect. But the audit also tried an oracle-style area-level feature, each area's own actual hit rate, one static number per area on all five features. It added only about +0.004 AUC (+0.0025 and +0.0054 by fold), about the size of the area-shuffle threshold of 0.0045. This is a rough benchmark, not a ceiling: it can't represent timing, and the raw-exposure secondary below came out slightly higher (+0.0054). Still, it points the same way: area-level features seem to have only about +0.005 of room on these folds, whatever they measure. "Adequate power" was met only formally, by a proxy MDE of about 0.025 that this benchmark sits far below. So this closes only this retrospective, area-level formulation. It is weak evidence against demand pull as an idea, and useful evidence that a coarse area signal is the wrong instrument: a lab-level or asset-level match is what would have room to matter.

What this does and does not say. It is "no detectable gain", not "no effect". The interval here is tighter than the MDE, but it is anti-conservative: features vary across about 51 area-by-cutoff cells, not 50,000 labs. Company-clustered intervals are wider: fold 1 -0.0033 to 0.0005, fold 2 -0.0102 to 0.0070. It says nothing about the thesis in its full form. This tested dated trial counts by area, not the matched buyer problems of the problems catalog or the economics layer, which are still untested. It also doesn't mean v4 is good in any new sense: v4's own skill claim waits on the prospective reads.

Secondaries, descriptive only; they cannot rescue the primary.

Why to be careful with this result.

Rule that fired, quoted from the plan. It is applied to this owner-authorized second test; the first attempt was cancelled at its coverage floor and no test ran then. "an executed test at adequate power (MDE <= 0.03, white-noise placebo OK, vocab floors passed) that fails its pass rule closes the retrospective prediction line: no v5 on demand grounds, R3 closed, R1 stays shipped, the null ships as a dated what-failed."