dennisthiessen/signal-platform
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(regime): reseed stored history on a sensor change, and stop faking an observation

Browse Source 
Two review findings on 46ace50.

[P1] Raising HY_OAS_WINDOW_DAYS to 700 only reached newly computed rows. A
routine run recomputes the latest trading date alone, and `rebuilding` was keyed
on "no v3 snapshot exists at all", which is false once the cutover has run --
so every row already written kept the credit gap the wider window exists to
close, indefinitely.

Adds SENSOR_REVISION: stamped into each snapshot, absent on pre-marker rows
(read as 1), and a stored revision below the current one triggers exactly one
reseed. Deliberately not METHODOLOGY, which would partition the history API and
discard the cached event study -- neither warranted, since the study recomputes
its Warning series from source rather than reading snapshots and so cannot be
staled by a reseed.

The reseed is bounded by REBUILD_LOOKBACK_DAYS in calendar days rather than a
session count, because the binding constraint is the OAS fetch: each replayed
row needs W3's 20-business-day lookback inside HY_OAS_WINDOW_DAYS. Replaying by
session count would have left the oldest stored rows unrepaired -- the exact
rows the fix targets. At 672 days the replay covers ~464 sessions, W3's oldest
requirement lands on the first fetched OAS day, and the ~400-session series the
cutover wrote is fully covered. A test asserts that relationship so the two
constants cannot drift back into recreating the gap.

[P2] With nothing ever collected, current_observation returned available=true
and the default placeholders -- "unknown" for every hyperscaler, "mixed" for the
reaction -- so the card announced a reading that never happened. Those are the
absence of an observation, not an observation of absence. Gated on `observed`
(non-null fetched_at, the one field every path writing real content stamps),
which blanks the content and drives a proper empty state naming where an admin
collects one. This was a regression from 46ace50; fundamental_overlay never had
it, since no observation means no effective date means pending.

Also renames the leftover v2 identifiers in the touched paths
(rewrite_existing_v2, latest_v2).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
dennisthiessen
2026-08-07 18:22:25 +02:00
co-authored by Claude Opus 5
parent 7dc804be2b
commit 3483797e75
5 changed files with 233 additions and 25 deletions
Download Patch File Download Diff File Expand all files Collapse all files
app/services/regime_monitor_service.py
+62 -17
View File
@@ -52,7 +52,14 @@ METHODOLOGY = "v3"
# Snapshots are reseeded on a methodology bump, but fundamental observations are
# collected by hand/LLM and carried across it when the format is compatible.
CATEGORICAL_FUNDAMENTAL_METHODOLOGIES = frozenset({"v2", "v3"})
REBUILD_SESSIONS = 400
# Bumped when a fix changes what historical rows *should* contain without
# changing the live formula, so stored history needs one reseed. Deliberately
# not METHODOLOGY: that partitions the history API and discards the cached event
# study, neither of which is warranted here -- the study recomputes its Warning
# series from source rather than reading snapshots, so a reseed cannot stale it.
# Snapshots written before this marker existed carry no key and read as 1.
SENSOR_REVISION = 2
MIN_COVERAGE = 75.0
SOURCE_MAX_LAG_DAYS = 7
@@ -91,6 +98,14 @@ HY_OAS_STRESSED = 7.0
# live scores are unchanged and this needs no methodology bump. Stays under
# ICE's ~3-year cap so FRED still honours the request.
HY_OAS_WINDOW_DAYS = 700
# A rebuild replays every session inside this window. Bounded by calendar days
# rather than a session count because the binding constraint is the OAS fetch:
# each replayed row needs W3's 20-business-day lookback (~28 calendar days)
# inside HY_OAS_WINDOW_DAYS, so replaying further back would recreate the exact
# credit gap a reseed exists to close. 672 days is ~464 trading sessions, which
# comfortably covers the 400-session series the v3 cutover wrote.
REBUILD_LOOKBACK_DAYS = HY_OAS_WINDOW_DAYS - 28
W3_OAS_LOOKBACK = 20
W3_OAS_FULL_SCALE_PCT = 35.0
@@ -536,19 +551,27 @@ def current_observation(overrides: dict, config: dict, as_of: date) -> dict:
published number; the stored snapshot keeps the gate.
"""
effective, pending, age, stale = _overlay_timing(overrides, config, as_of)
# The default override carries "unknown"/"mixed" placeholders for every
# hyperscaler. Those are the absence of an observation, not an observation
# of absence, and must never be presented as collected. ``fetched_at`` is
# the collection timestamp and is the only field written on every path that
# produces real content (LLM refresh and manual save both stamp it).
observed = bool(overrides.get("fetched_at"))
return {
"observed": observed,
# Live availability is about usefulness, not effectiveness: a pending
# observation is the freshest thing we have.
"available": not stale,
# observation is the freshest thing we have -- but nothing collected is
# never available.
"available": observed and not stale,
"pending": pending,
"stale": stale,
"effective_date": effective.isoformat() if effective else None,
"age_days": age,
"capex": overrides.get("capex"),
"good_news_stock_down": overrides.get("good_news_stock_down"),
"capex_stress": overrides.get("f1_score"),
"earnings_stress": overrides.get("f3_score"),
"reasoning": overrides.get("reasoning"),
"capex": overrides.get("capex") if observed else None,
"good_news_stock_down": overrides.get("good_news_stock_down") if observed else None,
"capex_stress": overrides.get("f1_score") if observed else None,
"earnings_stress": overrides.get("f3_score") if observed else None,
"reasoning": overrides.get("reasoning") if observed else None,
"source": overrides.get("source"),
"fetched_at": overrides.get("fetched_at"),
}
@@ -680,6 +703,8 @@ def _compute_index(
return {
"methodology": METHODOLOGY,
# Not part of the history filter -- only the reseed trigger.
"sensor_revision": SENSOR_REVISION,
"date": as_of.isoformat(),
"state": state,
"warning": warning,
@@ -939,7 +964,7 @@ async def _upsert_snapshot(
db: AsyncSession,
result: dict,
*,
rewrite_existing_v2: bool,
rewrite_existing: bool,
) -> tuple[bool, dict]:
snapshot_date = date.fromisoformat(result["date"])
existing = await db.execute(select(RegimeSnapshot).where(RegimeSnapshot.date == snapshot_date))
@@ -956,15 +981,23 @@ async def _upsert_snapshot(
created_at=datetime.now(timezone.utc),
))
else:
existing_v2 = _parse_snapshot(row.breakdown_json)
if existing_v2 is not None and not rewrite_existing_v2:
return False, existing_v2
existing_parsed = _parse_snapshot(row.breakdown_json)
if existing_parsed is not None and not rewrite_existing:
return False, existing_parsed
row.total_score = float(state_score or 0.0)
row.band = state_band or "unavailable"
row.breakdown_json = payload
return True, result
def _snapshot_revision(snapshot: dict) -> int:
"""Sensor revision of a stored snapshot; pre-marker rows read as 1."""
try:
return int(snapshot.get("sensor_revision") or 1)
except (TypeError, ValueError):
return 1
def _parse_snapshot(raw: str) -> dict | None:
try:
parsed = json.loads(raw)
@@ -984,7 +1017,9 @@ async def _latest_snapshot_row(db: AsyncSession) -> tuple[RegimeSnapshot, dict]
return None
async def update_regime_monitor(db: AsyncSession, rebuild_sessions: int = REBUILD_SESSIONS) -> dict:
async def update_regime_monitor(
db: AsyncSession, rebuild_lookback_days: int = REBUILD_LOOKBACK_DAYS
) -> dict:
config = await get_regime_config(db)
overrides = await get_fundamental_overrides(db)
if _fundamentals_stale(overrides, config) and not overrides.get("locked"):
@@ -1018,10 +1053,18 @@ async def update_regime_monitor(db: AsyncSession, rebuild_sessions: int = REBUIL
logger.warning("Regime monitor: fixed-basket breadth skipped: %s", exc)
breadth, breadth_counts, divergence = {}, {}, {}
latest_v2 = await _latest_snapshot_row(db)
rebuilding = latest_v2 is None and bool(leader_series)
latest_snapshot = await _latest_snapshot_row(db)
# A stored series written under an older sensor revision is reseeded once.
# Without this, raising HY_OAS_WINDOW_DAYS would only ever reach newly
# computed rows: routine runs touch the latest date alone, so every older row
# would keep the credit gap indefinitely.
rebuilding = bool(leader_series) and (
latest_snapshot is None
or _snapshot_revision(latest_snapshot[1]) < SENSOR_REVISION
)
if rebuilding:
dates = [d for d, _ in leader_series[-max(1, rebuild_sessions):]]
floor = end - timedelta(days=rebuild_lookback_days)
dates = [d for d, _ in leader_series if d >= floor] or [latest_date]
else:
# Routine PIT rule: only the latest trading date may be inserted/updated.
dates = [latest_date]
@@ -1045,7 +1088,9 @@ async def update_regime_monitor(db: AsyncSession, rebuild_sessions: int = REBUIL
written, latest_result = await _upsert_snapshot(
db,
computed,
rewrite_existing_v2=rebuilding or snapshot_date == latest_date,
# True for *every* replayed date on a reseed, or it would write one
# row and leave the rest at the old revision.
rewrite_existing=rebuilding or snapshot_date == latest_date,
)
snapshots_written += int(written)
await db.commit()