fix(research): require the v2 reproduction, and correct the P1-cap denominator
Two review findings, plus a lost-edit repair. v2_reconstruction is now a required variant. It carries every published figure the reproduction rests on (avg, p80, max, P3-pegged, W1-live), so a run without it could emit a confident, non-provisional recommendation having checked nothing against v2 at all -- while the methodology doc claims v2 and v3 are reproduced first. The default invocation is now derived from REQUIRED_VARIANTS so the two cannot drift, and a test asserts the default satisfies its own requirement. The doc and the P1_TREND_BREAK_ANCHORS comment still justified skipping the P1_SCORE_CAP with 17/408 = 4.2%, which is the all-session share and does not evaluate the rule. The rule names sessions with State >= 40: 47 of them, P1 sole argmax on 17 = 36.2%, against P2's 16 and P3's 14. Conclusion unchanged -- well under the 80% trigger -- but the published rationale now states the metric that actually decided it. Root cause of that survival: the earlier correction WAS made, but in a script that applied several substitutions and wrote the file once at the end. A later substitution raised, so the successful edits were discarded with it. The "Unlike P3 and V1 ... P3's do not" fix was lost the same way and is restored. Also adds tests for the refusal paths themselves -- missing required variant, unknown variant, custom window with no calendar anchor. They were verified by hand last round but left unpinned, which is the same shape of problem as the optional gates they exist to enforce. All return before any network call. Deliberately not done, as not load-bearing: recording the oas400 variant's missing-credit session count (the truncation conclusion rests on the distribution mismatch, which is already recorded), and generalising _pipeline_gates for arbitrary --end/--sessions windows. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -139,3 +139,45 @@ class TestScenarios:
|
||||
# ...and it must clear the chosen threshold under either P2 assumption.
|
||||
assert min(rows["S3a"]["state"], rows["S3b"]["state"]) > 65.0
|
||||
assert rows["S1"]["state"] < 20.0
|
||||
|
||||
|
||||
class TestRefusals:
|
||||
"""The harness's safety contract: it must decline rather than under-report.
|
||||
|
||||
Both paths return before any network call, so these are fast and offline.
|
||||
"""
|
||||
|
||||
def _run(self, argv, monkeypatch):
|
||||
import asyncio
|
||||
import sys
|
||||
|
||||
monkeypatch.setattr(sys, "argv", ["run_regime_monitor_calibration.py", *argv])
|
||||
return asyncio.run(calib._main())
|
||||
|
||||
def test_refuses_without_every_required_variant(self, monkeypatch):
|
||||
assert self._run(["--methodology", "v3,v4"], monkeypatch) == 2
|
||||
assert self._run(["--methodology", "v2_reconstruction,v3"], monkeypatch) == 2
|
||||
|
||||
def test_refuses_an_unknown_variant(self, monkeypatch):
|
||||
assert self._run(["--methodology", "v3,v4,nonsense"], monkeypatch) == 2
|
||||
|
||||
def test_refuses_a_custom_window_without_a_calendar_anchor(self, monkeypatch):
|
||||
"""--sessions alone can only ever be tautological, so the start date must
|
||||
be supplied explicitly once the published window is left behind."""
|
||||
assert self._run(
|
||||
["--methodology", ",".join(calib.REQUIRED_VARIANTS), "--sessions", "100"],
|
||||
monkeypatch,
|
||||
) == 2
|
||||
assert self._run(
|
||||
["--methodology", ",".join(calib.REQUIRED_VARIANTS), "--end", "2026-01-05"],
|
||||
monkeypatch,
|
||||
) == 2
|
||||
|
||||
def test_the_default_invocation_satisfies_its_own_requirement(self, monkeypatch):
|
||||
"""A default that the requirement rejects would make every bare run fail."""
|
||||
import sys
|
||||
|
||||
monkeypatch.setattr(sys, "argv", ["run_regime_monitor_calibration.py"])
|
||||
default = calib._parse_args().methodology.split(",")
|
||||
assert set(calib.REQUIRED_VARIANTS) <= set(default)
|
||||
assert set(calib.REQUIRED_VARIANTS) <= set(calib.VARIANTS)
|
||||
|
||||
Reference in New Issue
Block a user