Grading a science brief's supported records, in session, with negative controls

2026-11-10. Script: scripts/dd_sci_grade.py. Result: docs/methodology/scibrief-grading.json. Items and verdicts: data/dd/scigrade/ (gitignored).

Why this exists

The SX1 test (docs/dd/sx1-results.md) measured one thing: whether the reader finds a planted misstatement and names the paper that shows it. It measured nothing about the other direction, the records that say a claim is supported, which are most of a real brief. The owner does not grade by hand, so the grading is done in session by agents and is labelled model-graded throughout.

Method

1. build samples 20 supported records at random (seed 20261110) from the briefs named on the command line, taking only records the brief's own checker kept. Each becomes an item: the claim as asserted, the reader's quote, and the full text of the paper it quoted.
2. For each item, code builds one negative control from the same material: the claim altered to disagree with its own paper (a direction word flipped, or a stated number changed), or, when neither applies, the same claim paired with a different paper from the same pack. The control is the point of the exercise: a grader that accepts everything scores 100% on the real records and fails here.
3. Two graders rule on every item, 80 runs in all. They see one item each, with no hint of which group it is in: the wrapper text is byte-identical and the instructions never mention controls. A record counts as holding only if both graders say so.
4. score reports exact (Clopper-Pearson) intervals. A bootstrap over items cannot: with no variation among items it returns a point, which is how an earlier draft of this check produced a meaningless "100% to 100%".

Result

Controls rejected by kind: altered direction 3 of 3, altered number 7 of 7, wrong paper 10 of 10.

What this does and does not show

• Does: on 20 real records drawn from five briefs, an independent re-read of the whole paper agreed with every supported verdict, and the same graders rejected every control built from the same papers. The check is not a yes-machine.
• Does not: the controls are plain contradictions, a flipped direction word, a number changed by about a factor of two, or an unrelated paper. A subtly wrong supported record, the same result measured in a different population or at a different timepoint, is not represented. Nothing here says the graders would catch one.
• Does not: 20 items give a lower bound of 83%, not a guarantee. One wrong record in 20 would have moved the lower bound to about 75%.
• Does not: reader, graders and extractor are all Claude models. Two instances of one model agreeing is agreement, not independence. No person has read any of it.