Files
signal-platform/tests/unit/test_regime_calibration_script.py
T
dennisthiessenandClaude Opus 5 3143477a62 feat(regime): cut the risk monitor to v4 — desaturate VIX and the trend break
Two sensors saturated in exactly the range where resolution matters, and the top
State band had no headroom. Calibrated with scripts/run_regime_monitor_calibration.py
over the 408 sessions ending 2026-07-24; the shipped code reproduces that run's
band shares exactly (78.9 / 13.0 / 4.7 / 3.4).

V1 read VIX 30, 50 and 82 as an identical 100 — the same defect v3 had just
removed from P3, left in place one sensor over. In the window it flattened five
distinct April-2025 prints (52.33, 46.98, 45.31, 40.72, 38.57) into one value.
Now an anchor table reaching full scale at 55, not at 2020's ~82: anchoring the
top at a once-in-a-generation print would make VIX 50 read only ~70. Pegged on
14 of 408 sessions before; none now.

_under_200 returned a bare 0/100, so P1 printed 100 the moment SMH and QQQ were
both under their average — and since the price pillar takes max(P1, P2, P3),
that pinned the pillar and stopped P3's ladder resolving for the whole of a
selloff. Now graded by depth below the 200-DMA, with a deliberate floor of 20 at
the crossing: the break is a genuine binary event, only its depth is graded.
Pegged on 46 of 408 sessions before; none now. A 2% break reads ~30, not 100.

max() was KEPT — the defect was the step function feeding it, not the vote, and
v3's "one capped vote for correlated reads" rationale still holds. P1 is the sole
price argmax on 17 of 408 sessions (4.2%), so the P1_SCORE_CAP fallback drafted
during design was measured as unnecessary and not shipped.

STATE_BANDS breaking 80 -> 65, and only that threshold. Credit returns 0.0 (not
None) when calm, so it holds its 20 points pinned at zero and price + breadth +
volatility at literal maximum summed to exactly 80.0 — v3's threshold to the
decimal, with nothing above it. The sensor is deliberately unchanged: a
calm-credit selloff genuinely is less stressed. What was stale is the band, fit
on v2 while credit's since-removed percentile leg still contributed. A
2022-style AI/tech drawdown with calm credit computes to 70.3 (no death cross) or
74.0 (with one); 70 would have left 0.33 points of headroom, reproducing the
defect. Chosen by scenario arithmetic, and the realized breaking share then lands
on 3.4% — the same as v3's, arrived at independently.

"v4" added to CATEGORICAL_FUNDAMENTAL_METHODOLOGIES in this same commit, which is
load-bearing: that set is checked against the STORED blob, so bumping without it
discards the collected observation on first write, leaving fetched_at null and
locked false — and update_regime_monitor then fires a paid LLM refresh on every
run, forever. Now guarded by a test parametrised over v2 and v3 stored blobs.

SENSOR_REVISION deliberately stays 2: a METHODOLOGY change already forces a full
reseed via _parse_snapshot, and bumping both would imply the reseed was
revision-driven.

QUADRANT_STATE_DIVIDER stays 50 because only breaking moved, so alert_service,
RegimeChart and the quadrant tests need no change. A new test enforces
divider == band boundary on both axes, which nothing did before.

Doc renamed to regime-monitor-v4.md with a tombstone at the old path (commit
messages cite it), the three open questions converted to resolved with the
reasoning that closed them, and indexed in docs/research/README.md for the first
time. The P2 limit is stated honestly: _death_cross pegs at a -5% MA gap, so a
deep selloff still reaches 100 via P2 — v4 repairs the shallow-to-moderate break,
not "the price pillar no longer pegs".

DEPLOY: the first run reseeds ~464 sessions. Expect one phantom quadrant alert
(the dedup key carries basket_hash, not methodology) and re-run the Event Study
manually — its cached report self-invalidates but does not self-regenerate.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-08 20:34:25 +02:00

142 lines
6.2 KiB
Python

"""The calibration harness's pure helpers. No network, no data.
The harness's whole value is that its numbers can be trusted, so the parts that
decide whether a run is trustworthy — the variant patching and the gate
evaluation — are worth pinning even though the script is research-only.
"""
import importlib.util
from pathlib import Path
import pytest
from app.services import regime_monitor_service as rms
_SPEC = importlib.util.spec_from_file_location(
"regime_calibration",
Path(__file__).resolve().parents[2] / "scripts" / "run_regime_monitor_calibration.py",
)
calib = importlib.util.module_from_spec(_SPEC)
_SPEC.loader.exec_module(calib)
class TestVariantPatching:
def test_patching_restores_module_state_exactly(self):
"""A leaked patch would silently contaminate every later variant."""
before = {
name: getattr(rms, name)
for name in ("p1_trend_break", "p5_volatility", "p3_drawdown",
"f2_credit_spreads", "HY_OAS_WINDOW_DAYS")
}
with calib.patched(calib.VARIANTS["v2_reconstruction"]):
assert rms.p3_drawdown is not before["p3_drawdown"]
assert rms.HY_OAS_WINDOW_DAYS == 3653
for name, original in before.items():
assert getattr(rms, name) is original or getattr(rms, name) == original
def test_patching_restores_even_when_the_body_raises(self):
original = rms.p5_volatility
with pytest.raises(RuntimeError):
with calib.patched(calib.VARIANTS["v3"]):
raise RuntimeError("boom")
assert rms.p5_volatility is original
def test_v4_variant_patches_nothing(self):
"""The shipped methodology must be exercised as live code, not a copy,
or the harness and the service can drift apart silently."""
assert calib.VARIANTS["v4"] == {}
class TestCandidateFormulas:
def test_graduated_trend_break_never_exceeds_the_binary_one(self):
"""Half of the provable state_v4 <= state_v3 invariant."""
closes = [100.0] * 200
for last in (105.0, 100.0, 99.0, 92.0, 80.0, 50.0):
series = closes[:-1] + [last]
v4 = rms._under_200(series) # shipped
v3 = calib._v3_under_200(series) # retired
assert v4 <= v3, f"close={last}: v4 {v4} > v3 {v3}"
def test_anchored_vix_never_exceeds_the_live_formula(self):
for vix in (10, 15, 17, 20, 25, 30, 40, 55, 82):
assert rms.p5_volatility(vix) <= calib._v3_p5_volatility(vix), f"vix={vix}"
def test_the_vix_table_keeps_resolving_past_thirty(self):
assert rms.p5_volatility(30) < rms.p5_volatility(40) < rms.p5_volatility(50)
assert rms.p5_volatility(55) == 100.0
# the retired formula's defect, kept as the contrast
assert calib._v3_p5_volatility(30) == calib._v3_p5_volatility(82) == 100.0
def test_a_shallow_break_no_longer_pegs(self):
closes = [100.0] * 199 + [98.0] # ~2% below a flat 200-DMA
assert calib._v3_under_200(closes) == 100.0 # retired: pegged
assert rms._under_200(closes) < 40.0 # shipped: graded
def test_candidate_tables_are_well_formed(self):
for table in (calib.P5_VIX_ANCHORS_A, calib.P5_VIX_ANCHORS_B,
calib.P1_TREND_BREAK_ANCHORS):
xs = [x for x, _ in table]
ys = [y for _, y in table]
assert xs == sorted(xs) and len(set(xs)) == len(xs)
assert ys == sorted(ys)
assert 0.0 <= min(ys) and max(ys) <= 100.0
class TestGates:
def _row(self, date_str, state, coverage=100.0, w1=5.0):
return {
"date": date_str,
"state": {"score": state, "coverage": coverage, "pillars": []},
"warning": {"score": 10.0, "coverage": 100.0, "pillars": [
{"id": "breadth_divergence", "sensors": [{"id": "W1", "score": w1}]}
]},
"data_quality": {"stale_inputs": []},
}
def test_the_invariant_gate_catches_a_v4_row_above_its_v3_row(self):
v3 = [self._row("2026-01-02", 40.0), self._row("2026-01-05", 50.0)]
v4 = [self._row("2026-01-02", 38.0), self._row("2026-01-05", 55.0)]
gate = calib._invariant_gate(v3, v4)
assert gate["passed"] is False
assert gate["detail"][0]["date"] == "2026-01-05"
def test_the_invariant_gate_passes_when_v4_is_never_higher(self):
v3 = [self._row("2026-01-02", 40.0)]
v4 = [self._row("2026-01-02", 40.0)]
assert calib._invariant_gate(v3, v4)["passed"] is True
def test_sensor_lookup_searches_both_axes(self):
"""W1 lives under warning; a state-only search silently returns None and
made the W1 census read 0."""
row = self._row("2026-01-02", 40.0, w1=7.5)
assert calib._sensor_score(row, "breadth_divergence", "W1") == 7.5
class TestBandShares:
def test_shares_use_the_candidate_bands_not_the_imported_default(self):
"""band_for binds bands=STATE_BANDS as an import-time default, so the
grid must pass candidates explicitly or every row scores identically."""
scores = [10.0, 30.0, 55.0, 75.0]
loose = calib._band_shares(scores, (20.0, 50.0, 80.0))
tight = calib._band_shares(scores, (20.0, 50.0, 60.0))
assert loose["breaking"] == 0.0
assert tight["breaking"] == 25.0
def test_shares_sum_to_one_hundred(self):
shares = calib._band_shares([5.0, 25.0, 55.0, 85.0], (20.0, 50.0, 80.0))
assert sum(shares.values()) == pytest.approx(100.0)
class TestScenarios:
def test_the_decisive_scenario_is_computed_not_asserted(self):
rows = {s["scenario"].split()[0]: s for s in
calib._scenarios(calib.P5_VIX_ANCHORS_A, calib.P1_TREND_BREAK_ANCHORS)}
# A 2022-style AI/tech drawdown with genuinely calm credit. This is the
# meaning anchor for the breaking threshold, so it is machine-checked.
assert rows["S3a"]["C1"] == 0.0
assert rows["S3a"]["state"] == pytest.approx(70.33, abs=0.01)
assert rows["S3b"]["state"] == pytest.approx(74.00, abs=0.01)
# ...and it must clear the chosen threshold under either P2 assumption.
assert min(rows["S3a"]["state"], rows["S3b"]["state"]) > 65.0
assert rows["S1"]["state"] < 20.0