fix(research): require the v2 reproduction, and correct the P1-cap denominator

Two review findings, plus a lost-edit repair.

v2_reconstruction is now a required variant. It carries every published figure
the reproduction rests on (avg, p80, max, P3-pegged, W1-live), so a run without
it could emit a confident, non-provisional recommendation having checked nothing
against v2 at all -- while the methodology doc claims v2 and v3 are reproduced
first. The default invocation is now derived from REQUIRED_VARIANTS so the two
cannot drift, and a test asserts the default satisfies its own requirement.

The doc and the P1_TREND_BREAK_ANCHORS comment still justified skipping the
P1_SCORE_CAP with 17/408 = 4.2%, which is the all-session share and does not
evaluate the rule. The rule names sessions with State >= 40: 47 of them, P1 sole
argmax on 17 = 36.2%, against P2's 16 and P3's 14. Conclusion unchanged -- well
under the 80% trigger -- but the published rationale now states the metric that
actually decided it.

Root cause of that survival: the earlier correction WAS made, but in a script
that applied several substitutions and wrote the file once at the end. A later
substitution raised, so the successful edits were discarded with it. The
"Unlike P3 and V1 ... P3's do not" fix was lost the same way and is restored.

Also adds tests for the refusal paths themselves -- missing required variant,
unknown variant, custom window with no calendar anchor. They were verified by
hand last round but left unpinned, which is the same shape of problem as the
optional gates they exist to enforce. All return before any network call.

Deliberately not done, as not load-bearing: recording the oas400 variant's
missing-credit session count (the truncation conclusion rests on the
distribution mismatch, which is already recorded), and generalising
_pipeline_gates for arbitrary --end/--sessions windows.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-08-08 23:16:03 +02:00
co-authored by Claude Opus 5
parent 87224a1451
commit 43ee619412
4 changed files with 60 additions and 9 deletions
@@ -139,3 +139,45 @@ class TestScenarios:
# ...and it must clear the chosen threshold under either P2 assumption.
assert min(rows["S3a"]["state"], rows["S3b"]["state"]) > 65.0
assert rows["S1"]["state"] < 20.0
class TestRefusals:
"""The harness's safety contract: it must decline rather than under-report.
Both paths return before any network call, so these are fast and offline.
"""
def _run(self, argv, monkeypatch):
import asyncio
import sys
monkeypatch.setattr(sys, "argv", ["run_regime_monitor_calibration.py", *argv])
return asyncio.run(calib._main())
def test_refuses_without_every_required_variant(self, monkeypatch):
assert self._run(["--methodology", "v3,v4"], monkeypatch) == 2
assert self._run(["--methodology", "v2_reconstruction,v3"], monkeypatch) == 2
def test_refuses_an_unknown_variant(self, monkeypatch):
assert self._run(["--methodology", "v3,v4,nonsense"], monkeypatch) == 2
def test_refuses_a_custom_window_without_a_calendar_anchor(self, monkeypatch):
"""--sessions alone can only ever be tautological, so the start date must
be supplied explicitly once the published window is left behind."""
assert self._run(
["--methodology", ",".join(calib.REQUIRED_VARIANTS), "--sessions", "100"],
monkeypatch,
) == 2
assert self._run(
["--methodology", ",".join(calib.REQUIRED_VARIANTS), "--end", "2026-01-05"],
monkeypatch,
) == 2
def test_the_default_invocation_satisfies_its_own_requirement(self, monkeypatch):
"""A default that the requirement rejects would make every bare run fail."""
import sys
monkeypatch.setattr(sys, "argv", ["run_regime_monitor_calibration.py"])
default = calib._parse_args().methodology.split(",")
assert set(calib.REQUIRED_VARIANTS) <= set(default)
assert set(calib.REQUIRED_VARIANTS) <= set(calib.VARIANTS)