dennisthiessenandClaude Opus 5
333989eeab
The Warning study measured a fitted percentile crossing that nothing consumes. What reaches Telegram is a quadrant change: fixed 50/40 dividers, hysteresis, two-session confirmation, 3-day cooldown. Those thresholds are constants, not fits, so there is no training set to protect and all 11 detected corrections are evaluable instead of the 4 that fell in a holdout. Replaying it: 1/10 corrections, 0.9 false alarms/year. Random alarms at the same firing rate match or beat that in 65% of draws. The panel now carries ablations (does the quadrant machinery earn its place?), external baselines (does the score earn its complexity?), and that null, because a bare "2 of 4" was unreadable in either direction. Nothing in the alert path was retuned on the strength of it. Fundamentals become a third channel rather than a term in either score. v3 cut them arguing 12+8 of 100 points "could not change any published conclusion" -- true only when every technical sensor reads zero; weighted they moved the bar for the 40 divider from 40 to 25. But no fusion weight is measurable either: with ~10 events and no fundamental history, any weight is a policy preference presented as a measurement. So the read is a categorical state (supportive/neutral/adverse/ unknown) with an evidence grade, derived by fixed rules from stored facts, read by confluence. The LLM extracts and explains; it does not score. Absence stays absence throughout. `unknown` is unreachable by averaging, a stale or empty observation may display but never confirm, extraction failures map to `unknown` rather than `mixed`, and the study rows are coverage-matched and marked not-measurable until enough corrections are covered -- otherwise a fortnight of observations renders as 0/10 and reads as a failed test. Observations become a real time series (migration 033); they lived in a single overwritten settings slot, so no history existed to replay. Pre-rename snapshots are adapted rather than discarded. METHODOLOGY stays v4 -- no score changed -- so no reseed; STUDY_SCHEMA moves to 3 and discards the cached report. Post-deploy: re-run Event Study from Admin -> Jobs. The panel reads "not run yet" until then. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
887 lines
39 KiB
Python
887 lines
39 KiB
Python
"""Offline replay of the AI/Tech Risk Monitor, for calibrating a methodology cut.
|
|
|
|
Reproduces the State/Warning series session by session from the same inputs the
|
|
live job uses -- Alpaca for prices, FRED for VIX and HY OAS -- with no database,
|
|
so a sensor change can be measured against real history before it ships.
|
|
|
|
v3 was calibrated this way ad-hoc and the harness was never committed, which is
|
|
why its published numbers cannot be re-derived today. This is that harness.
|
|
|
|
**It never reimplements an unchanged live sensor.** ``_compute_index``,
|
|
``_score_pillars``, breadth, divergence, P2, P4 and the Warning sensors are
|
|
imported and called. Only *candidate* formulas (proposed for v4) and *retired*
|
|
ones (v2, no longer in the codebase) are defined here and patched onto the
|
|
service for the duration of a variant. Once a candidate ships, delete it here and
|
|
import the shipped function instead, or the two will drift.
|
|
|
|
The script refuses to emit a band recommendation unless every hard gate passes.
|
|
That is deliberate: it must be structurally impossible to read a calibration
|
|
result out of a run whose pipeline did not validate.
|
|
|
|
**Fundamental channel note.** The sourced capex / earnings read is a separate
|
|
categorical channel and is never a term in State or Warning, so every variant and
|
|
gate below is unaffected by it. This harness passes no observation, which means
|
|
the ``fundamental_context`` on each replayed row reads ``unknown`` -- correct, and
|
|
the same thing production reports for a session nobody observed. Calibrating
|
|
anything *about* that channel needs an observation series passed through
|
|
``_compute_index(..., observations=...)``, and enough history to be worth
|
|
calibrating against.
|
|
|
|
Research branch only. Example:
|
|
|
|
.\\.venv\\Scripts\\python.exe scripts\\run_regime_monitor_calibration.py ^
|
|
--end 2026-07-24 --sessions 408 --methodology v3,v4 ^
|
|
--cache-dir .calib-cache
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import argparse
|
|
import asyncio
|
|
import contextlib
|
|
import json
|
|
import math
|
|
import statistics
|
|
import subprocess
|
|
import sys
|
|
from collections.abc import Iterator
|
|
from copy import deepcopy
|
|
from datetime import date, datetime, timedelta
|
|
from pathlib import Path
|
|
from typing import Any, Callable
|
|
|
|
ROOT = Path(__file__).resolve().parents[1]
|
|
if str(ROOT) not in sys.path:
|
|
sys.path.insert(0, str(ROOT))
|
|
|
|
from app.config import settings # noqa: E402
|
|
from app.providers.alpaca import AlpacaOHLCVProvider # noqa: E402
|
|
from app.services import breadth_service # noqa: E402
|
|
from app.services import regime_monitor_service as rms # noqa: E402
|
|
|
|
# Published v3/v2 figures from docs/research/regime-monitor-v3.md. The v3 pair is
|
|
# DERIVED there (22.6 - 0.4; 91.2 - 3.8), not measured, so its tolerance is loose
|
|
# on purpose -- anything tighter would be false precision.
|
|
PUBLISHED = {
|
|
"window_end": "2026-07-24",
|
|
# The session count alone is tautological -- the harness slices the tail of
|
|
# leader_series, so it can only ever equal what was asked for. The start date
|
|
# is what actually validates the calendar.
|
|
"window_first": "2024-12-05",
|
|
"sessions": 408,
|
|
"w1_live_sessions": 108,
|
|
"v2_state_avg": 22.6,
|
|
"v2_state_p80": 35.1,
|
|
"v2_state_max": 91.2,
|
|
"v2_p3_pegged": 39,
|
|
# No v3 average is published: the doc's "-0.4" is measured against
|
|
# v3-with-the-percentile-leg, not against v2, so only the max is checkable.
|
|
"v3_state_max": 87.4,
|
|
}
|
|
|
|
# Every symbol comes from one source on one split-adjustment basis. Mixing a
|
|
# sqlite snapshot for the basket with Alpaca for the leaders would splice two
|
|
# adjustment bases mid-200-DMA for any symbol that split in between.
|
|
LEADER, CONFIRM, MARKET = "SMH", "QQQ", "SPY"
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Candidate formulas (proposed for v4) -- patched in, never shipped from here
|
|
# ---------------------------------------------------------------------------
|
|
|
|
P5_VIX_ANCHORS_A = ((15.0, 0.0), (20.0, 20.0), (25.0, 38.0), (30.0, 55.0), (40.0, 80.0), (55.0, 100.0))
|
|
P5_VIX_ANCHORS_B = ((15.0, 0.0), (20.0, 25.0), (25.0, 45.0), (30.0, 65.0), (40.0, 85.0), (55.0, 100.0))
|
|
P1_TREND_BREAK_ANCHORS = ((0.0, 20.0), (3.0, 35.0), (8.0, 55.0), (15.0, 75.0), (25.0, 100.0))
|
|
|
|
|
|
def _candidate_under_200(closes: list[float], anchors=P1_TREND_BREAK_ANCHORS) -> float | None:
|
|
"""Graduated trend break: 0 above the 200-DMA, else scaled by depth below it.
|
|
|
|
The live version returns a bare 0/100, which pins the price pillar's max()
|
|
at 100 through any real selloff and stops P3's ladder resolving. The step at
|
|
the crossing (0 -> 20) is kept deliberately: the break itself is a genuine
|
|
binary event and deserves a floor; only the depth past it is graduated.
|
|
"""
|
|
sma200 = rms._sma(closes, 200)
|
|
if sma200 is None or sma200 <= 0:
|
|
return None
|
|
pct_below = (sma200 - closes[-1]) / sma200 * 100.0
|
|
if pct_below <= 0:
|
|
return 0.0
|
|
return rms._clamp(rms._interpolate(pct_below, anchors))
|
|
|
|
|
|
def _candidate_p1(anchors=P1_TREND_BREAK_ANCHORS) -> Callable:
|
|
def p1_trend_break(smh, qqq, leader_weight: float = 2.0):
|
|
return rms._blend(
|
|
_candidate_under_200(smh, anchors), _candidate_under_200(qqq, anchors), leader_weight
|
|
)
|
|
|
|
return p1_trend_break
|
|
|
|
|
|
def _candidate_p5(anchors) -> Callable:
|
|
def p5_volatility(vix: float | None) -> float | None:
|
|
if vix is None:
|
|
return None
|
|
return rms._clamp(rms._interpolate(vix, anchors))
|
|
|
|
return p5_volatility
|
|
|
|
|
|
def _capped(fn: Callable, cap: float) -> Callable:
|
|
"""P1_SCORE_CAP fallback: cap the sensor score after the blend, before max()."""
|
|
|
|
def wrapped(smh, qqq, leader_weight: float = 2.0):
|
|
value = fn(smh, qqq, leader_weight)
|
|
return None if value is None else min(value, cap)
|
|
|
|
return wrapped
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Retired formulas -- reconstructed, no longer in the codebase
|
|
# ---------------------------------------------------------------------------
|
|
|
|
def _v3_under_200(closes: list[float]) -> float | None:
|
|
"""v3's binary trend break, retired when v4 graduated it."""
|
|
sma200 = rms._sma(closes, 200)
|
|
if sma200 is None:
|
|
return None
|
|
return 100.0 if closes[-1] < sma200 else 0.0
|
|
|
|
|
|
def _v3_p1_trend_break(smh, qqq, leader_weight: float = 2.0) -> float | None:
|
|
return rms._blend(_v3_under_200(smh), _v3_under_200(qqq), leader_weight)
|
|
|
|
|
|
def _v3_p5_volatility(vix: float | None) -> float | None:
|
|
"""v3's linear VIX ramp, retired when v4 anchored it. Saturated at 30."""
|
|
if vix is None:
|
|
return None
|
|
return rms._clamp((vix - 15.0) / 15.0 * 100.0)
|
|
|
|
|
|
|
|
def _v2_drawdown(closes: list[float]) -> float | None:
|
|
if len(closes) < 30:
|
|
return None
|
|
peak = max(closes[-252:])
|
|
if peak <= 0:
|
|
return None
|
|
return rms._clamp((peak - closes[-1]) / peak * 100.0 * 5.0)
|
|
|
|
|
|
def _v2_p3_drawdown(smh, qqq, leader_weight: float = 2.0) -> float | None:
|
|
"""v2 took max() across the legs, so the more volatile leader always won."""
|
|
vals = [v for v in (_v2_drawdown(smh), _v2_drawdown(qqq)) if v is not None]
|
|
return max(vals) if vals else None
|
|
|
|
|
|
def _v2_divergence_series(breadth, benchmark_closes, lookback: int = 20):
|
|
"""v2's hard price gate: the sensor ZEROED during any decline.
|
|
|
|
v3 replaced this with a taper, which is why v2 shows W1 nonzero on only 108
|
|
of 408 sessions while v3 shows it nonzero far more often. Reconstructing it
|
|
is the only way to check that published figure.
|
|
"""
|
|
bench = {d: c for d, c in benchmark_closes}
|
|
common = sorted(d for d in bench if d in breadth)
|
|
out: dict[date, float] = {}
|
|
for i in range(lookback, len(common)):
|
|
d, d0 = common[i], common[i - lookback]
|
|
if bench[d0] <= 0:
|
|
continue
|
|
price_ret = (bench[d] / bench[d0] - 1.0) * 100.0
|
|
deterioration = max(0.0, -(breadth[d] - breadth[d0]))
|
|
score = deterioration * 5.0 if price_ret >= 0 else 0.0
|
|
out[d] = max(0.0, min(100.0, round(score, 2)))
|
|
return out
|
|
|
|
|
|
def _v2_f2_credit_spreads(oas_values: list[float]) -> float | None:
|
|
"""70% named anchors + 30% upper-tail percentile over whatever window it got."""
|
|
if not oas_values:
|
|
return None
|
|
latest = oas_values[-1]
|
|
absolute = rms._oas_absolute_score(latest)
|
|
if len(oas_values) < 30:
|
|
return round(absolute, 2)
|
|
less = sum(1 for v in oas_values if v < latest)
|
|
equal = sum(1 for v in oas_values if v == latest)
|
|
percentile = (less + 0.5 * equal) / len(oas_values) * 100.0
|
|
relative = rms._clamp((percentile - 50.0) / 45.0 * 100.0)
|
|
return round(absolute * 0.7 + relative * 0.3, 2)
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Variants
|
|
# ---------------------------------------------------------------------------
|
|
|
|
VARIANTS: dict[str, dict[str, Callable]] = {
|
|
# Retired since the v4 cutover -- "nothing patched" is now v4, so v3 has to
|
|
# be reconstructed like v2 to stay comparable.
|
|
"v3": {
|
|
"p1_trend_break": _v3_p1_trend_break,
|
|
"p5_volatility": _v3_p5_volatility,
|
|
},
|
|
# SHIPPED as of v4 -- nothing patched, so this variant exercises live code.
|
|
# Keeping a private copy here would let the harness and the service drift.
|
|
"v4": {},
|
|
"v4-vix-b": {
|
|
"p1_trend_break": _candidate_p1(),
|
|
"p5_volatility": _candidate_p5(P5_VIX_ANCHORS_B),
|
|
},
|
|
"v4-p1-capped": {
|
|
"p1_trend_break": _capped(_candidate_p1(), 50.0),
|
|
"p5_volatility": _candidate_p5(P5_VIX_ANCHORS_A),
|
|
},
|
|
"v4-vix-only": {"p1_trend_break": _v3_p1_trend_break},
|
|
"v4-p1-only": {"p5_volatility": _v3_p5_volatility},
|
|
# (4) v2 as production actually fetched it: a 400-calendar-day OAS source,
|
|
# which left the oldest rows with no credit at all. Truncating the SERIES is
|
|
# the only faithful simulation -- patching the per-session window is not,
|
|
# because the data was simply absent.
|
|
"v2_reconstruction_oas400": {
|
|
"p1_trend_break": _v3_p1_trend_break,
|
|
"p5_volatility": _v3_p5_volatility,
|
|
"p3_drawdown": _v2_p3_drawdown,
|
|
"f2_credit_spreads": _v2_f2_credit_spreads,
|
|
"HY_OAS_WINDOW_DAYS": 3653,
|
|
},
|
|
# v2 State sensors + the v2 divergence gate that feeds W1. The v2 *Warning
|
|
# composition* (F1/F3 fundamentals, 20 of 100 points) is NOT reconstructed,
|
|
# so only State statistics and the W1 census are comparable to the published
|
|
# v2 figures -- not the Warning score.
|
|
"v2_reconstruction": {
|
|
# v2 shared v3's binary trend break and linear VIX ramp verbatim, so both
|
|
# are retired now and must be restored here too -- otherwise a "v2" replay
|
|
# silently picks up v4's graded sensors.
|
|
"p1_trend_break": _v3_p1_trend_break,
|
|
"p5_volatility": _v3_p5_volatility,
|
|
"p3_drawdown": _v2_p3_drawdown,
|
|
"f2_credit_spreads": _v2_f2_credit_spreads,
|
|
# v2 sliced HY_OAS_REFERENCE_YEARS = 10.0 per session. The percentile leg
|
|
# ranks the current spread against that window, so replaying it against
|
|
# a 700-day slice gives systematically different mid-distribution scores.
|
|
"HY_OAS_WINDOW_DAYS": 3653,
|
|
},
|
|
}
|
|
|
|
# Variants needing the retired divergence formula rather than the live one.
|
|
V2_DIVERGENCE_VARIANTS = {"v2_reconstruction", "v2_reconstruction_oas400"}
|
|
|
|
# Variants whose OAS *source series* is truncated before replay, in calendar days.
|
|
OAS_SOURCE_TRUNCATION = {"v2_reconstruction_oas400": 400}
|
|
|
|
# A v4 recommendation is meaningless without both of these: the row-wise
|
|
# state_v4 <= state_v3 invariant needs them, and it is a hard gate.
|
|
# v2_reconstruction is required too: it carries every published figure the
|
|
# reproduction rests on (avg/p80/max/P3-pegged/W1-live). Without it a run could
|
|
# emit a confident recommendation having checked nothing against v2 at all,
|
|
# while the methodology doc claims v2 and v3 are reproduced first.
|
|
REQUIRED_VARIANTS = ("v2_reconstruction", "v3", "v4")
|
|
|
|
|
|
@contextlib.contextmanager
|
|
def patched(overrides: dict[str, Callable]) -> Iterator[None]:
|
|
"""Swap functions on the service module, then restore exactly."""
|
|
original = {name: getattr(rms, name) for name in overrides}
|
|
try:
|
|
for name, fn in overrides.items():
|
|
setattr(rms, name, fn)
|
|
yield
|
|
finally:
|
|
for name, fn in original.items():
|
|
setattr(rms, name, fn)
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Inputs
|
|
# ---------------------------------------------------------------------------
|
|
|
|
def _basket(config: dict) -> list[str]:
|
|
return list(config["breadth_basket"])
|
|
|
|
|
|
def _all_symbols(config: dict) -> list[str]:
|
|
return list(dict.fromkeys(_basket(config) + [LEADER, CONFIRM, MARKET]))
|
|
|
|
|
|
async def _load_prices(
|
|
symbols: list[str], start: date, end: date, cache_dir: Path | None, quiet: bool
|
|
) -> dict[str, list[tuple[date, float]]]:
|
|
cache = None
|
|
if cache_dir:
|
|
cache_dir.mkdir(parents=True, exist_ok=True)
|
|
cache = cache_dir / f"prices-{start}-{end}.json"
|
|
if cache.exists():
|
|
raw = json.loads(cache.read_text(encoding="utf-8"))
|
|
if set(raw) >= set(symbols):
|
|
if not quiet:
|
|
print(f"prices: cache hit ({len(raw)} symbols)", flush=True)
|
|
return {
|
|
s: [(date.fromisoformat(d), float(c)) for d, c in raw[s]] for s in symbols
|
|
}
|
|
|
|
provider = AlpacaOHLCVProvider(settings.alpaca_api_key, settings.alpaca_api_secret)
|
|
out: dict[str, list[tuple[date, float]]] = {}
|
|
for index, symbol in enumerate(symbols, 1):
|
|
bars = await provider.fetch_ohlcv(symbol, start, end)
|
|
out[symbol] = sorted((b.date, float(b.close)) for b in bars)
|
|
if not quiet:
|
|
print(f" [{index}/{len(symbols)}] {symbol}: {len(out[symbol])} bars", flush=True)
|
|
if cache:
|
|
cache.write_text(
|
|
json.dumps({s: [[d.isoformat(), c] for d, c in v] for s, v in out.items()}),
|
|
encoding="utf-8",
|
|
)
|
|
return out
|
|
|
|
|
|
async def _load_fred(series_id: str, start: date, end: date, cache_dir: Path | None):
|
|
cache = cache_dir / f"{series_id}-{start}-{end}.json" if cache_dir else None
|
|
if cache and cache.exists():
|
|
raw = json.loads(cache.read_text(encoding="utf-8"))
|
|
return [(date.fromisoformat(d), float(v)) for d, v in raw]
|
|
series = await rms._fetch_fred_series(series_id, start, end)
|
|
if cache and series:
|
|
cache.write_text(
|
|
json.dumps([[d.isoformat(), v] for d, v in series]), encoding="utf-8"
|
|
)
|
|
return series
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Replay
|
|
# ---------------------------------------------------------------------------
|
|
|
|
def _replay(
|
|
variant: str,
|
|
prices: dict[str, list[tuple[date, float]]],
|
|
vix, oas, config: dict, sessions: list[date],
|
|
breadth_series, divergence_by_variant: dict[str, Any], breadth_counts,
|
|
) -> list[dict]:
|
|
# Divergence is computed outside _compute_index, so the retired v2 gate has
|
|
# to be selected here rather than patched onto the module.
|
|
divergence_series = divergence_by_variant[
|
|
"v2" if variant in V2_DIVERGENCE_VARIANTS else "live"
|
|
]
|
|
truncate_days = OAS_SOURCE_TRUNCATION.get(variant)
|
|
if truncate_days is not None and oas:
|
|
cutoff = max(d for d, _ in oas) - timedelta(days=truncate_days)
|
|
oas = [(d, v) for d, v in oas if d >= cutoff]
|
|
rows: list[dict] = []
|
|
with patched(VARIANTS[variant]):
|
|
for as_of in sessions:
|
|
rows.append(
|
|
rms._compute_index(
|
|
prices, vix, oas, {}, deepcopy(config), as_of,
|
|
breadth_series, divergence_series, breadth_counts,
|
|
)
|
|
)
|
|
return rows
|
|
|
|
|
|
def _percentile(values: list[float], pct: float) -> float | None:
|
|
if not values:
|
|
return None
|
|
ordered = sorted(values)
|
|
k = (len(ordered) - 1) * pct / 100.0
|
|
lo, hi = math.floor(k), math.ceil(k)
|
|
if lo == hi:
|
|
return ordered[int(k)]
|
|
return ordered[lo] + (ordered[hi] - ordered[lo]) * (k - lo)
|
|
|
|
|
|
def _sensor_score(row: dict, pillar_id: str, sensor_id: str) -> float | None:
|
|
"""Search both axes: W1/W2/W3 live under ``warning``, P*/B1/C1/V1 under ``state``."""
|
|
for axis in ("state", "warning"):
|
|
for pillar in row[axis]["pillars"]:
|
|
if pillar["id"] == pillar_id:
|
|
for sensor in pillar["sensors"]:
|
|
if sensor["id"] == sensor_id:
|
|
return sensor["score"]
|
|
return None
|
|
|
|
|
|
def _band_shares(scores: list[float], bands: tuple[float, float, float]) -> dict[str, float]:
|
|
if not scores:
|
|
return {}
|
|
counts = {"stable": 0, "watch": 0, "elevated": 0, "breaking": 0}
|
|
for score in scores:
|
|
counts[rms.band_for(score, bands)] += 1 # bands passed explicitly -- see module docstring
|
|
return {k: round(v / len(scores) * 100.0, 1) for k, v in counts.items()}
|
|
|
|
|
|
def _stats(rows: list[dict], label: str) -> dict:
|
|
states = [r["state"]["score"] for r in rows if r["state"]["score"] is not None]
|
|
warnings = [r["warning"]["score"] for r in rows if r["warning"]["score"] is not None]
|
|
|
|
def pegged(pillar: str, sensor: str) -> int:
|
|
return sum(1 for r in rows if (_sensor_score(r, pillar, sensor) or 0) >= 100.0)
|
|
|
|
argmax_sole, argmax_tied, ties = {"P1": 0, "P2": 0, "P3": 0}, {"P1": 0, "P2": 0, "P3": 0}, 0
|
|
# The P1_SCORE_CAP rule is "sole argmax on >80% of sessions with State >= 40",
|
|
# so the all-session count does not evaluate it. Track the conditional
|
|
# population separately rather than deciding off the wrong denominator.
|
|
stressed_sole, stressed_total = {"P1": 0, "P2": 0, "P3": 0}, 0
|
|
for row in rows:
|
|
legs = {s: _sensor_score(row, "price", s) for s in ("P1", "P2", "P3")}
|
|
live = {k: v for k, v in legs.items() if v is not None}
|
|
if not live:
|
|
continue
|
|
top = max(live.values())
|
|
winners = [k for k, v in live.items() if v == top]
|
|
if len(winners) > 1:
|
|
ties += 1
|
|
for w in winners:
|
|
argmax_tied[w] += 1
|
|
if len(winners) == 1:
|
|
argmax_sole[winners[0]] += 1
|
|
if (row["state"]["score"] or 0) >= 40.0:
|
|
stressed_total += 1
|
|
if len(winners) == 1:
|
|
stressed_sole[winners[0]] += 1
|
|
|
|
return {
|
|
"label": label,
|
|
"sessions": len(rows),
|
|
"state": {
|
|
"avg": round(statistics.fmean(states), 2) if states else None,
|
|
"median": round(statistics.median(states), 2) if states else None,
|
|
"p80": round(_percentile(states, 80), 2) if states else None,
|
|
"p90": round(_percentile(states, 90), 2) if states else None,
|
|
"max": round(max(states), 2) if states else None,
|
|
"scored": len(states),
|
|
},
|
|
"warning_avg": round(statistics.fmean(warnings), 2) if warnings else None,
|
|
"saturation_census": {
|
|
"p3_pegged": pegged("price", "P3"),
|
|
"p2_pegged": pegged("price", "P2"),
|
|
"v1_pegged": pegged("volatility", "V1"),
|
|
"p1_pegged": pegged("price", "P1"),
|
|
},
|
|
"price_argmax_sole": argmax_sole,
|
|
"price_argmax_tie_inclusive": argmax_tied,
|
|
"price_argmax_ties": ties,
|
|
"price_argmax_when_state_ge_40": {
|
|
"sessions": stressed_total,
|
|
"sole": stressed_sole,
|
|
"p1_sole_share_pct": round(stressed_sole["P1"] / stressed_total * 100.0, 1)
|
|
if stressed_total else None,
|
|
"cap_rule": "P1_SCORE_CAP warranted if p1_sole_share_pct > 80",
|
|
},
|
|
"w1_nonzero_sessions": sum(
|
|
1 for r in rows if (_sensor_score(r, "breadth_divergence", "W1") or 0) > 0
|
|
),
|
|
"band_shares_current": _band_shares(states, rms.STATE_BANDS),
|
|
}
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Gates
|
|
# ---------------------------------------------------------------------------
|
|
|
|
def _completeness_gates(
|
|
prices: dict, symbols: list[str], sessions: list[date], breadth_counts: dict, basket_size: int
|
|
) -> list[dict]:
|
|
"""Without these, every gate below can pass on partial data.
|
|
|
|
_breadth_with_counts publishes on min_tickers=20, so 20 of 30 basket names
|
|
still yields 100% State coverage and a plausible W1 count.
|
|
"""
|
|
first, last = sessions[0], sessions[-1]
|
|
missing = [s for s in symbols if not prices.get(s)]
|
|
thin = [
|
|
s for s in symbols
|
|
if len([d for d, _ in prices.get(s, []) if d < first]) < 252
|
|
]
|
|
truncated = [s for s in symbols if not prices.get(s) or prices[s][-1][0] < last]
|
|
short_basket = sorted(
|
|
d.isoformat() for d in sessions if breadth_counts.get(d, 0) != basket_size
|
|
)
|
|
return [
|
|
{"gate": "symbols_fetched", "expected": len(symbols),
|
|
"measured": len(symbols) - len(missing), "passed": not missing, "detail": missing},
|
|
{"gate": "per_symbol_warmup_252_bars", "expected": "all",
|
|
"measured": len(symbols) - len(thin), "passed": not thin, "detail": thin},
|
|
{"gate": "per_symbol_reaches_last_session", "expected": last.isoformat(),
|
|
"measured": len(symbols) - len(truncated), "passed": not truncated, "detail": truncated},
|
|
{"gate": "breadth_counts_full_basket", "expected": basket_size,
|
|
"measured": f"{len(sessions) - len(short_basket)}/{len(sessions)} sessions",
|
|
"passed": not short_basket, "detail": short_basket[:20]},
|
|
]
|
|
|
|
|
|
def _pipeline_gates(rows: list[dict], sessions: list[date], expected_first: str) -> list[dict]:
|
|
coverage_bad = [
|
|
r["date"] for r in rows if (r["state"]["coverage"] or 0) < 100.0
|
|
]
|
|
w1_available = sum(
|
|
1 for r in rows if _sensor_score(r, "breadth_divergence", "W1") is not None
|
|
)
|
|
stale = [r["date"] for r in rows if r["data_quality"]["stale_inputs"]]
|
|
gates = [
|
|
{"gate": "sessions_scored", "expected": PUBLISHED["sessions"],
|
|
"measured": len(rows), "passed": len(rows) == PUBLISHED["sessions"]},
|
|
{"gate": "last_scored_date", "expected": PUBLISHED["window_end"],
|
|
"measured": rows[-1]["date"], "passed": rows[-1]["date"] == PUBLISHED["window_end"]},
|
|
# Availability, not the published "W1 live 108" -- that figure counts
|
|
# NONZERO sessions under v2's hard price gate and is checked there.
|
|
{"gate": "w1_available_every_session", "expected": len(rows),
|
|
"measured": w1_available, "passed": w1_available == len(rows)},
|
|
{"gate": "state_coverage_100_every_row", "expected": 0,
|
|
"measured": len(coverage_bad), "passed": not coverage_bad, "detail": coverage_bad[:20]},
|
|
{"gate": "no_stale_inputs", "expected": 0,
|
|
"measured": len(stale), "passed": not stale, "detail": stale[:20]},
|
|
]
|
|
# Unconditional: sessions_scored is tautological when the harness slices the
|
|
# tail of leader_series, so the start date is the only real calendar check.
|
|
# An optional gate is not a gate.
|
|
gates.append({
|
|
"gate": "first_scored_date", "expected": expected_first,
|
|
"measured": rows[0]["date"], "passed": rows[0]["date"] == expected_first,
|
|
})
|
|
return gates
|
|
|
|
|
|
def _invariant_gate(v3_rows: list[dict], v4_rows: list[dict]) -> dict:
|
|
"""state_v4 <= state_v3 on every aligned row.
|
|
|
|
Provable, not heuristic: graduated P1 never exceeds binary P1, anchored VIX
|
|
never exceeds (vix-15)/15*100, max() is monotone in its arguments, and no
|
|
other State sensor or weight changes. A violation means the harness is
|
|
mis-wired, not that the calibration is interesting.
|
|
"""
|
|
violations = []
|
|
for a, b in zip(v3_rows, v4_rows):
|
|
assert a["date"] == b["date"], "row misalignment"
|
|
s3, s4 = a["state"]["score"], b["state"]["score"]
|
|
if s3 is not None and s4 is not None and s4 > s3 + 1e-9:
|
|
violations.append({"date": a["date"], "v3": s3, "v4": s4})
|
|
return {
|
|
"gate": "state_v4_le_v3_every_row", "expected": 0,
|
|
"measured": len(violations), "passed": not violations, "detail": violations[:20],
|
|
}
|
|
|
|
|
|
def _soft_gates(stats: dict, variant: str) -> list[dict]:
|
|
if variant == "v3":
|
|
# The doc states no v3 average: its "-0.4" is measured against
|
|
# v3-with-the-percentile-leg, not against v2. Only the max is checkable.
|
|
pairs = [("v3_state_max", stats["state"]["max"], 0.5)]
|
|
else:
|
|
pairs = [("v2_state_avg", stats["state"]["avg"], 0.3),
|
|
("v2_state_p80", stats["state"]["p80"], 0.5),
|
|
("v2_state_max", stats["state"]["max"], 0.5),
|
|
("v2_p3_pegged", stats["saturation_census"]["p3_pegged"], 0),
|
|
("w1_live_sessions", stats["w1_nonzero_sessions"], 0)]
|
|
out = []
|
|
for key, measured, tol in pairs:
|
|
expected = PUBLISHED[key]
|
|
ok = measured is not None and abs(measured - expected) <= tol
|
|
out.append({"gate": key, "expected": expected, "measured": measured,
|
|
"tolerance": tol, "passed": ok})
|
|
return out
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Scenarios -- pillar arithmetic, stated as explicit sensor scores
|
|
# ---------------------------------------------------------------------------
|
|
|
|
def _scenarios(vix_anchors, p1_anchors) -> list[dict]:
|
|
"""The meaning anchors for the band choice, machine-checked rather than prose.
|
|
|
|
Stated as explicit sensor scores because drawdown + VIX + OAS does not
|
|
determine State: the price pillar is max(P1, P2, P3) and P2 is set by the
|
|
50/200-DMA gap, which no drawdown figure implies.
|
|
"""
|
|
def state(p1, p2, p3, breadth, c1, v1):
|
|
price = max(p1, p2, p3)
|
|
return round((price * 40 + breadth * 25 + c1 * 20 + v1 * 15) / 100, 2)
|
|
|
|
def p1_at(pct_below):
|
|
return round(rms._interpolate(pct_below, p1_anchors), 2) if pct_below > 0 else 0.0
|
|
|
|
def p3_at(dd):
|
|
return round(rms._interpolate(dd, rms.P3_DRAWDOWN_ANCHORS), 2)
|
|
|
|
def v1_at(vix):
|
|
return round(rms._interpolate(vix, vix_anchors), 2)
|
|
|
|
rows = [
|
|
("S1 ordinary tape", 0.0, 0.0, p3_at(3), rms.breadth_level_score(65), 0.0, v1_at(16)),
|
|
("S2 10% correction, calm credit", p1_at(2), 0.0, p3_at(10), rms.breadth_level_score(35), 0.0, v1_at(24)),
|
|
("S3a 2022-style, calm credit, no death cross", p1_at(20), 0.0, p3_at(35), rms.breadth_level_score(8), 0.0, v1_at(32)),
|
|
("S3b 2022-style, calm credit, death cross", p1_at(20), 100.0, p3_at(35), rms.breadth_level_score(8), 0.0, v1_at(32)),
|
|
("S4 credit event on top", p1_at(25), 100.0, p3_at(40), rms.breadth_level_score(5), rms.f2_credit_spreads([6.0]), v1_at(45)),
|
|
("S5 March 2020, everything pegged", 100.0, 100.0, 100.0, 100.0, 100.0, 100.0),
|
|
]
|
|
return [
|
|
{"scenario": name, "P1": p1, "P2": p2, "P3": p3, "price": max(p1, p2, p3),
|
|
"breadth": br, "C1": c1, "V1": v1, "state": state(p1, p2, p3, br, c1, v1)}
|
|
for name, p1, p2, p3, br, c1, v1 in rows
|
|
]
|
|
|
|
|
|
def _markdown(report: dict) -> str:
|
|
"""Scannable sibling to the JSON. Never written over a curated docs/ file."""
|
|
lines = ["# Regime Monitor v4 calibration", ""]
|
|
src = report["source"]
|
|
dirty = " **(dirty working tree)**" if src.get("git_dirty") else ""
|
|
lines.append(f"Generated {report['generated_at']} at `{src['git_rev']}`{dirty}, "
|
|
f"{report['provenance']['scored_range'][0]} → {report['provenance']['scored_range'][1]}.")
|
|
lines += ["", "Source hashes (sha256, first 16):", ""]
|
|
for rel, digest in src["source_sha256"].items():
|
|
lines.append(f"- `{rel}` — `{digest}`")
|
|
lines += ["", "## Hard gates", "", "| gate | expected | measured | |", "|---|---|---|---|"]
|
|
for g in report["hard_gates"]:
|
|
lines.append(f"| {g['gate']} | {g['expected']} | {g['measured']} | {'ok' if g['passed'] else '**FAIL**'} |")
|
|
lines += ["", "## Distributions", "", "| variant | avg | median | p80 | p90 | max |", "|---|---|---|---|---|---|"]
|
|
for name, v in report["variants"].items():
|
|
st = v["state"]
|
|
lines.append(f"| {name} | {st['avg']} | {st['median']} | {st['p80']} | {st['p90']} | {st['max']} |")
|
|
lines += ["", "## Saturation census (sessions pegged at 100)", "",
|
|
"| variant | P1 | P2 | P3 | V1 |", "|---|---|---|---|---|"]
|
|
for name, v in report["variants"].items():
|
|
c = v["saturation_census"]
|
|
lines.append(f"| {name} | {c['p1_pegged']} | {c['p2_pegged']} | {c['p3_pegged']} | {c['v1_pegged']} |")
|
|
for name, v in report["variants"].items():
|
|
if v.get("soft_gates"):
|
|
lines += ["", f"## Reproduction gates — {name}", "",
|
|
"| figure | published | measured | |", "|---|---|---|---|"]
|
|
for g in v["soft_gates"]:
|
|
lines.append(f"| {g['gate']} | {g['expected']} | {g['measured']} | {'ok' if g['passed'] else '**miss**'} |")
|
|
if "v4" in report["variants"]:
|
|
lines += ["", "## v4 band-share grid (watch 20 / elevated 50)", "",
|
|
"| breaking | stable | watch | elevated | breaking |", "|---|---|---|---|---|"]
|
|
for row in report["variants"]["v4"]["band_grid"]:
|
|
w, e, b = row["bands"]
|
|
if w == 20.0 and e == 50.0:
|
|
sh = row["shares"]
|
|
lines.append(f"| {b:.0f} | {sh['stable']} | {sh['watch']} | {sh['elevated']} | {sh['breaking']} |")
|
|
lines += ["", "## Scenarios (pillar arithmetic, explicit sensor scores)", "",
|
|
"| scenario | price | breadth | C1 | V1 | State |", "|---|---|---|---|---|---|"]
|
|
for sc in report["scenarios"]["vix_a"]:
|
|
lines.append(f"| {sc['scenario']} | {sc['price']} | {sc['breadth']} | {sc['C1']} | {sc['V1']} | **{sc['state']}** |")
|
|
lines += ["", f"Recommendation: `{report['v4_recommendation']}`", ""]
|
|
return "\n".join(lines)
|
|
|
|
|
|
def _band_grid(states: list[float]) -> list[dict]:
|
|
grid = []
|
|
for watch in (15.0, 20.0, 25.0):
|
|
for elevated in (40.0, 50.0):
|
|
for breaking in (60.0, 65.0, 70.0):
|
|
grid.append({
|
|
"bands": [watch, elevated, breaking],
|
|
"shares": _band_shares(states, (watch, elevated, breaking)),
|
|
})
|
|
return grid
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Main
|
|
# ---------------------------------------------------------------------------
|
|
|
|
def _parse_args() -> argparse.Namespace:
|
|
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
|
p.add_argument("--end", default=PUBLISHED["window_end"])
|
|
p.add_argument("--sessions", type=int, default=PUBLISHED["sessions"])
|
|
p.add_argument("--history-days", type=int, default=1200, help="matches production _fetch_prices")
|
|
p.add_argument("--oas-window-days", type=int, default=rms.HY_OAS_WINDOW_DAYS,
|
|
help="per-session slice for the canonical run")
|
|
p.add_argument("--oas-fetch-days", type=int, default=int(365.25 * 13),
|
|
help="fetch range; matches v2's request (ICE truncates to ~3y)")
|
|
p.add_argument("--methodology", default=",".join(REQUIRED_VARIANTS))
|
|
p.add_argument("--expected-first-session", default=None,
|
|
help="assert the first replayed date. Defaults to the published "
|
|
"window's start; REQUIRED when --end/--sessions are overridden, "
|
|
"since the count alone is tautological.")
|
|
p.add_argument("--cache-dir", default=None)
|
|
p.add_argument("--out", default=None)
|
|
p.add_argument("--quiet", action="store_true")
|
|
return p.parse_args()
|
|
|
|
|
|
def _git_rev() -> str:
|
|
try:
|
|
return subprocess.run(["git", "rev-parse", "--short", "HEAD"], cwd=ROOT,
|
|
capture_output=True, text=True, check=True).stdout.strip()
|
|
except Exception:
|
|
return "unknown"
|
|
|
|
|
|
def _source_state() -> dict:
|
|
"""Identify the code that produced this run, not just the commit HEAD names.
|
|
|
|
A run from a dirty tree is not reproducible by checking out git_rev -- which
|
|
is exactly how the first v4 artifact was generated, with HEAD still on the
|
|
harness commit while the v4 sensors lived only in the working tree. The
|
|
hashes make that visible instead of implied.
|
|
"""
|
|
import hashlib
|
|
|
|
tracked = [
|
|
"app/services/regime_monitor_service.py",
|
|
"app/services/breadth_service.py",
|
|
"scripts/run_regime_monitor_calibration.py",
|
|
]
|
|
digests = {}
|
|
for rel in tracked:
|
|
path = ROOT / rel
|
|
digests[rel] = hashlib.sha256(path.read_bytes()).hexdigest()[:16] if path.exists() else None
|
|
try:
|
|
dirty = bool(subprocess.run(["git", "status", "--porcelain"], cwd=ROOT,
|
|
capture_output=True, text=True, check=True).stdout.strip())
|
|
except Exception:
|
|
dirty = None
|
|
return {"git_rev": _git_rev(), "git_dirty": dirty, "source_sha256": digests}
|
|
|
|
|
|
async def _main() -> int:
|
|
args = _parse_args()
|
|
end = date.fromisoformat(args.end)
|
|
cache_dir = Path(args.cache_dir) if args.cache_dir else None
|
|
config = deepcopy(rms.DEFAULT_CONFIG)
|
|
symbols = _all_symbols(config)
|
|
variants = [v.strip() for v in args.methodology.split(",") if v.strip()]
|
|
expected_first = args.expected_first_session
|
|
if not expected_first:
|
|
if args.end != PUBLISHED["window_end"] or args.sessions != PUBLISHED["sessions"]:
|
|
print("--expected-first-session is required when --end or --sessions "
|
|
"differ from the published window.", file=sys.stderr)
|
|
return 2
|
|
expected_first = PUBLISHED["window_first"]
|
|
unknown = [v for v in variants if v not in VARIANTS]
|
|
if unknown:
|
|
print(f"unknown variant(s): {unknown}; known: {sorted(VARIANTS)}", file=sys.stderr)
|
|
return 2
|
|
missing = [v for v in REQUIRED_VARIANTS if v not in variants]
|
|
if missing:
|
|
print(f"--methodology must include {list(REQUIRED_VARIANTS)}; missing {missing}. "
|
|
"v2_reconstruction carries the published reproduction figures, and "
|
|
"v3+v4 are needed for the state_v4 <= state_v3 invariant gate.",
|
|
file=sys.stderr)
|
|
return 2
|
|
|
|
if not args.quiet:
|
|
print(f"fetching {len(symbols)} symbols from Alpaca...", flush=True)
|
|
prices = await _load_prices(symbols, end - timedelta(days=args.history_days), end, cache_dir, args.quiet)
|
|
vix = await _load_fred("VIXCLS", end - timedelta(days=args.history_days), end, cache_dir)
|
|
oas = await _load_fred("BAMLH0A0HYM2", end - timedelta(days=args.oas_fetch_days), end, cache_dir)
|
|
|
|
leader = prices.get(LEADER, [])
|
|
if not leader:
|
|
print("no leader (SMH) price data — cannot replay", file=sys.stderr)
|
|
return 2
|
|
sessions = [d for d, _ in leader if d <= end][-args.sessions:]
|
|
|
|
breadth, breadth_counts = breadth_service._breadth_with_counts(
|
|
{s: prices[s] for s in _basket(config) if prices.get(s)}, window=200, min_tickers=20
|
|
)
|
|
# Required glue: _item_asof breaks on the first date > as_of, so unsorted
|
|
# input silently returns a wrong value rather than erroring.
|
|
breadth_series = rms._mapping_series(breadth)
|
|
divergence_by_variant = {
|
|
"live": rms._mapping_series(breadth_service.compute_divergence_series(breadth, leader)),
|
|
"v2": rms._mapping_series(_v2_divergence_series(breadth, leader)),
|
|
}
|
|
|
|
if args.oas_window_days != rms.HY_OAS_WINDOW_DAYS:
|
|
rms.HY_OAS_WINDOW_DAYS = args.oas_window_days
|
|
|
|
results: dict[str, Any] = {}
|
|
rows_by_variant: dict[str, list[dict]] = {}
|
|
for variant in variants:
|
|
if not args.quiet:
|
|
print(f"replaying {variant} over {len(sessions)} sessions...", flush=True)
|
|
rows = _replay(variant, prices, vix, oas, config, sessions,
|
|
breadth_series, divergence_by_variant, breadth_counts)
|
|
rows_by_variant[variant] = rows
|
|
results[variant] = _stats(rows, variant)
|
|
results[variant]["soft_gates"] = _soft_gates(results[variant], variant) \
|
|
if variant in ("v3", "v2_reconstruction") else []
|
|
states = [r["state"]["score"] for r in rows if r["state"]["score"] is not None]
|
|
if variant.startswith("v4"):
|
|
results[variant]["band_grid"] = _band_grid(states)
|
|
|
|
canonical = rows_by_variant.get("v3") or next(iter(rows_by_variant.values()))
|
|
hard_gates = _completeness_gates(prices, symbols, sessions, breadth_counts, len(_basket(config)))
|
|
hard_gates += _pipeline_gates(canonical, sessions, expected_first)
|
|
if "v3" in rows_by_variant and "v4" in rows_by_variant:
|
|
hard_gates.append(_invariant_gate(rows_by_variant["v3"], rows_by_variant["v4"]))
|
|
|
|
blocked_by = [g["gate"] for g in hard_gates if not g["passed"]]
|
|
provisional = any(
|
|
not g["passed"] for v in results.values() for g in v.get("soft_gates", [])
|
|
)
|
|
|
|
report = {
|
|
"generated_at": datetime.now().isoformat(timespec="seconds"),
|
|
"source": _source_state(),
|
|
"params": vars(args),
|
|
"provenance": {
|
|
"symbols": {s: {"bars": len(prices.get(s, [])),
|
|
"first": prices[s][0][0].isoformat() if prices.get(s) else None,
|
|
"last": prices[s][-1][0].isoformat() if prices.get(s) else None}
|
|
for s in symbols},
|
|
"vix": {"points": len(vix or []),
|
|
"first": vix[0][0].isoformat() if vix else None,
|
|
"last": vix[-1][0].isoformat() if vix else None},
|
|
"oas": {"points": len(oas or []),
|
|
"first": oas[0][0].isoformat() if oas else None,
|
|
"last": oas[-1][0].isoformat() if oas else None},
|
|
"scored_range": [sessions[0].isoformat(), sessions[-1].isoformat()],
|
|
},
|
|
"hard_gates": hard_gates,
|
|
"blocked_by": blocked_by,
|
|
"provisional": provisional,
|
|
"variants": results,
|
|
"scenarios": {
|
|
"note": "pillar arithmetic on explicit sensor scores; State weights unchanged",
|
|
"vix_a": _scenarios(P5_VIX_ANCHORS_A, P1_TREND_BREAK_ANCHORS),
|
|
},
|
|
"diagnostics": {
|
|
"vix_top10": sorted(((d.isoformat(), v) for d, v in (vix or [])),
|
|
key=lambda x: -x[1])[:10],
|
|
"candidate_anchors": {
|
|
"P5_VIX_ANCHORS_A": P5_VIX_ANCHORS_A,
|
|
"P5_VIX_ANCHORS_B": P5_VIX_ANCHORS_B,
|
|
"P1_TREND_BREAK_ANCHORS": P1_TREND_BREAK_ANCHORS,
|
|
},
|
|
},
|
|
# Structurally impossible to read a recommendation out of a run whose
|
|
# pipeline did not validate.
|
|
"v4_recommendation": None if blocked_by else {
|
|
"state_bands_candidate": [20.0, 50.0, 65.0],
|
|
"provisional": provisional,
|
|
"note": "confirm against band_grid + scenarios before shipping",
|
|
},
|
|
}
|
|
|
|
stamp = datetime.now().strftime("%Y%m%d-%H%M%S")
|
|
out = Path(args.out) if args.out else ROOT / "reports" / f"regime-monitor-v4-calibration-{stamp}.json"
|
|
out.parent.mkdir(parents=True, exist_ok=True)
|
|
tmp = out.with_suffix(out.suffix + ".tmp")
|
|
tmp.write_text(json.dumps(report, indent=2, default=str), encoding="utf-8")
|
|
tmp.replace(out)
|
|
md = out.with_suffix(".md")
|
|
md_tmp = md.with_suffix(".md.tmp")
|
|
md_tmp.write_text(_markdown(report), encoding="utf-8")
|
|
md_tmp.replace(md)
|
|
if not args.quiet:
|
|
print(f"wrote {out}", flush=True)
|
|
for gate in hard_gates:
|
|
mark = "ok " if gate["passed"] else "FAIL"
|
|
print(f" [{mark}] {gate['gate']}: expected {gate['expected']}, got {gate['measured']}")
|
|
if blocked_by:
|
|
print(f"HARD GATES FAILED: {blocked_by} — no v4 recommendation emitted", file=sys.stderr)
|
|
return 1
|
|
return 0
|
|
|
|
|
|
if __name__ == "__main__":
|
|
raise SystemExit(asyncio.run(_main()))
|