fix(research): require the v2 reproduction, and correct the P1-cap denominator
Two review findings, plus a lost-edit repair. v2_reconstruction is now a required variant. It carries every published figure the reproduction rests on (avg, p80, max, P3-pegged, W1-live), so a run without it could emit a confident, non-provisional recommendation having checked nothing against v2 at all -- while the methodology doc claims v2 and v3 are reproduced first. The default invocation is now derived from REQUIRED_VARIANTS so the two cannot drift, and a test asserts the default satisfies its own requirement. The doc and the P1_TREND_BREAK_ANCHORS comment still justified skipping the P1_SCORE_CAP with 17/408 = 4.2%, which is the all-session share and does not evaluate the rule. The rule names sessions with State >= 40: 47 of them, P1 sole argmax on 17 = 36.2%, against P2's 16 and P3's 14. Conclusion unchanged -- well under the 80% trigger -- but the published rationale now states the metric that actually decided it. Root cause of that survival: the earlier correction WAS made, but in a script that applied several substitutions and wrote the file once at the end. A later substitution raised, so the successful edits were discarded with it. The "Unlike P3 and V1 ... P3's do not" fix was lost the same way and is restored. Also adds tests for the refusal paths themselves -- missing required variant, unknown variant, custom window with no calendar anchor. They were verified by hand last round but left unpinned, which is the same shape of problem as the optional gates they exist to enforce. All return before any network call. Deliberately not done, as not load-bearing: recording the oas400 variant's missing-credit session count (the truncation conclusion rests on the distribution mismatch, which is already recorded), and generalising _pipeline_gates for arbitrary --end/--sessions windows. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -267,7 +267,11 @@ OAS_SOURCE_TRUNCATION = {"v2_reconstruction_oas400": 400}
|
||||
|
||||
# A v4 recommendation is meaningless without both of these: the row-wise
|
||||
# state_v4 <= state_v3 invariant needs them, and it is a hard gate.
|
||||
REQUIRED_VARIANTS = ("v3", "v4")
|
||||
# v2_reconstruction is required too: it carries every published figure the
|
||||
# reproduction rests on (avg/p80/max/P3-pegged/W1-live). Without it a run could
|
||||
# emit a confident recommendation having checked nothing against v2 at all,
|
||||
# while the methodology doc claims v2 and v3 are reproduced first.
|
||||
REQUIRED_VARIANTS = ("v2_reconstruction", "v3", "v4")
|
||||
|
||||
|
||||
@contextlib.contextmanager
|
||||
@@ -681,7 +685,7 @@ def _parse_args() -> argparse.Namespace:
|
||||
help="per-session slice for the canonical run")
|
||||
p.add_argument("--oas-fetch-days", type=int, default=int(365.25 * 13),
|
||||
help="fetch range; matches v2's request (ICE truncates to ~3y)")
|
||||
p.add_argument("--methodology", default="v3,v4")
|
||||
p.add_argument("--methodology", default=",".join(REQUIRED_VARIANTS))
|
||||
p.add_argument("--expected-first-session", default=None,
|
||||
help="assert the first replayed date. Defaults to the published "
|
||||
"window's start; REQUIRED when --end/--sessions are overridden, "
|
||||
@@ -748,7 +752,8 @@ async def _main() -> int:
|
||||
missing = [v for v in REQUIRED_VARIANTS if v not in variants]
|
||||
if missing:
|
||||
print(f"--methodology must include {list(REQUIRED_VARIANTS)}; missing {missing}. "
|
||||
"The state_v4 <= state_v3 invariant is a hard gate and needs both.",
|
||||
"v2_reconstruction carries the published reproduction figures, and "
|
||||
"v3+v4 are needed for the state_v4 <= state_v3 invariant gate.",
|
||||
file=sys.stderr)
|
||||
return 2
|
||||
|
||||
|
||||
Reference in New Issue
Block a user