dennisthiessen/signal-platform
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(risk-monitor): measure the rule that fires, and give fundamentals their own channel
Deploy / lint (push) Failing after 11s
Details
Deploy / test (push) Skipped
Details
Deploy / deploy (push) Skipped
Details

Browse Source 
The Warning study measured a fitted percentile crossing that nothing consumes.
What reaches Telegram is a quadrant change: fixed 50/40 dividers, hysteresis,
two-session confirmation, 3-day cooldown. Those thresholds are constants, not
fits, so there is no training set to protect and all 11 detected corrections are
evaluable instead of the 4 that fell in a holdout.

Replaying it: 1/10 corrections, 0.9 false alarms/year. Random alarms at the same
firing rate match or beat that in 65% of draws. The panel now carries ablations
(does the quadrant machinery earn its place?), external baselines (does the score
earn its complexity?), and that null, because a bare "2 of 4" was unreadable in
either direction. Nothing in the alert path was retuned on the strength of it.

Fundamentals become a third channel rather than a term in either score. v3 cut
them arguing 12+8 of 100 points "could not change any published conclusion" --
true only when every technical sensor reads zero; weighted they moved the bar for
the 40 divider from 40 to 25. But no fusion weight is measurable either: with ~10
events and no fundamental history, any weight is a policy preference presented as
a measurement. So the read is a categorical state (supportive/neutral/adverse/
unknown) with an evidence grade, derived by fixed rules from stored facts, read
by confluence. The LLM extracts and explains; it does not score.

Absence stays absence throughout. `unknown` is unreachable by averaging, a stale
or empty observation may display but never confirm, extraction failures map to
`unknown` rather than `mixed`, and the study rows are coverage-matched and marked
not-measurable until enough corrections are covered -- otherwise a fortnight of
observations renders as 0/10 and reads as a failed test.

Observations become a real time series (migration 033); they lived in a single
overwritten settings slot, so no history existed to replay. Pre-rename snapshots
are adapted rather than discarded. METHODOLOGY stays v4 -- no score changed --
so no reseed; STUDY_SCHEMA moves to 3 and discards the cached report.

Post-deploy: re-run Event Study from Admin -> Jobs. The panel reads "not run yet"
until then.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
dennisthiessen
2026-08-13 11:15:09 +02:00
co-authored by Claude Opus 5
parent 3033ad83fd
commit 333989eeab
18 changed files with 3494 additions and 403 deletions
Download Patch File Download Diff File Expand all files Collapse all files
app/services/regime_monitor_service.py
+423 -34
View File
@@ -7,11 +7,17 @@ two deliberately separate outputs:
* Warning: deterioration/divergence that may precede State (breadth divergence,
relative strength, credit impulse).
Both scores are quantitative and daily. The sourced hyperscaler capex and
earnings-reaction observations are a qualitative *overlay* since v3 rather than
weighted sensors: at a combined 20 points they could not reach the event
study's alarm threshold even when both pegged, so refreshing them appeared to
do nothing. They are reported next to the scores instead of inside them.
* Fundamental context: a categorical channel (supportive / neutral / adverse /
unknown) with an evidence-quality grade, derived by fixed rules from the
sourced hyperscaler capex and earnings-reaction observations.
Both scores are quantitative and daily. The fundamental channel is deliberately
**not** a term in either: the three are read together by confluence, because
adding a slow categorical judgement to a fast continuous score manufactures
precision by summing unlike things, and any fusion weight would be a policy
preference presented as a measurement until there is enough point-in-time
history to fit one. A missing observation therefore stays ``unknown`` instead of
silently redistributing its weight onto the technical sensors.
Daily snapshots are the point-in-time record. The first run under a new
``METHODOLOGY`` rewrites every session inside ``REBUILD_LOOKBACK_DAYS`` once;
@@ -35,6 +41,7 @@ from sqlalchemy.ext.asyncio import AsyncSession
from app.config import settings
from app.exceptions import ProviderError, ValidationError
from app.models.regime_fundamental_observation import RegimeFundamentalObservation
from app.models.regime_snapshot import RegimeSnapshot
from app.providers.alpaca import AlpacaOHLCVProvider
from app.services import breadth_service, settings_store
@@ -55,7 +62,11 @@ METHODOLOGY = "v4"
# against the *stored* blob, so omitting the current one discards the observation
# on its first write, which leaves fetched_at null and locked false -- and then
# update_regime_monitor refreshes it via the LLM on every single run, forever.
CATEGORICAL_FUNDAMENTAL_METHODOLOGIES = frozenset({"v2", "v3", "v4"})
# "v5" is listed although no v5 scoring exists: a v5 was briefly built (a weighted
# fundamental modifier on Warning) and reverted, so a development box can have
# that string sitting in its settings blob. Keeping it costs nothing; omitting it
# costs the failure above.
CATEGORICAL_FUNDAMENTAL_METHODOLOGIES = frozenset({"v2", "v3", "v4", "v5"})
# Bumped when a fix changes what historical rows *should* contain without
# changing the live formula, so stored history needs one reseed. Deliberately
@@ -173,6 +184,34 @@ WARNING_WEIGHTS = {
"credit_impulse": 25.0,
}
# The sourced fundamental read is a **separate channel**, never a term in either
# score. It is reported as a categorical state beside State and Warning, and the
# three are read together by confluence rather than added up.
#
# Two things had to be true at once and only this shape gets both.
#
# **v3's reason for removing it was wrong.** v3 argued that F1+F3, at 12+8 of 100
# Warning points, "could not change any published conclusion" because pegged they
# produced a Warning of exactly 20.0. That holds only when every technical sensor
# reads exactly zero. Weighted, those points added +10 to +20 across the
# realistic range and moved the technical score needed to reach the 40 quadrant
# divider from 40 to 25. So the observation was not inert, and demoting it to
# decoration was not justified by that argument.
#
# **But no weight is measurable either.** A weighted modifier was built (v5,
# reverted) and its size could not be derived from anything: with ~10 correction
# events and essentially no fundamental history, any fusion weight is a policy
# preference presented as a measurement. Adding a slow categorical judgement to a
# fast continuous score also manufactures precision by summing unlike things, and
# it forces a missing observation to silently redistribute its weight onto the
# technical sensors -- the opposite of leaving it unknown.
#
# So the read gets a channel, not a coefficient. Revisit only with enough
# point-in-time history to test whether the state improves prediction
# *conditional on* Warning; a fitted model then has something to fit.
FUNDAMENTAL_STATES = ("supportive", "neutral", "adverse", "unknown")
EVIDENCE_QUALITY = ("complete", "partial", "stale", "manual", "unavailable")
# Fixed at the v2 launch. These are liquid S&P 500/Nasdaq AI, semiconductor,
# infrastructure, cloud, and enterprise-software names that the platform's
# normal universe sync already stores.
@@ -196,7 +235,12 @@ DEFAULT_CONFIG: dict = {
}
CAPEX_STATES = ("raising", "holding", "cutting", "unknown")
GNSD_STATES = ("yes", "no", "mixed")
# "mixed" is a genuinely observed mixed reaction; "unknown" is nobody looked or
# the extraction failed. They were the same value until 2026-08-13, so a failed
# LLM parse silently became neutral *evidence* -- an observation of normality
# manufactured out of a parse error. Same distinction the capex map already made
# with its own "unknown", and the same one the whole channel is built on.
GNSD_STATES = ("yes", "no", "mixed", "unknown")
# v2 scored raising and holding identically at 0, so in a capex boom the reading
# was pinned at 0 and could not express the raising -> holding deceleration that
# is the actual early warning. Display-only in v3, but it should still describe.
@@ -435,6 +479,104 @@ def score_warning_sensors(sensors: dict[str, float | None]) -> float | None:
return sum(s * w for s, w in live) / sum(w for _, w in live)
def _capex_signal(capex: dict[str, str] | None, names: list[str]) -> str:
"""Categorical read of hyperscaler capex direction. Never an average.
Averaging is what this must not do: it would let two ``cutting`` reads and
two ``unknown`` ones land on "neutral", presenting missing evidence as
evidence of normality. Any cut is adverse on partial evidence; only a fully
known, uniformly rising basket is supportive.
"""
states = [str((capex or {}).get(name, "unknown")).strip().lower() for name in names]
known = [state for state in states if state in ("raising", "holding", "cutting")]
if not known:
return "unknown"
if "cutting" in known:
return "adverse"
if "holding" in known:
return "neutral"
return "supportive"
def _reaction_signal(good_news_stock_down: str | None) -> str:
"""Good earnings being sold is a late-cycle tell; not being sold is healthy.
Anything that is not one of the three observed categories -- including the
explicit ``"unknown"`` an extraction failure now writes -- falls through to
``unknown`` rather than to ``mixed``. A parse error is not a reading.
"""
return {
"yes": "adverse",
"no": "supportive",
"mixed": "neutral",
}.get(str(good_news_stock_down or "").strip().lower(), "unknown")
def combine_fundamental_signals(capex_signal: str, reaction_signal: str) -> str:
"""Confluence, not arithmetic: precedence over the two categorical reads.
``unknown`` is deliberately unreachable by combination -- it survives only
when *nothing* was observed. A single adverse read carries, because partial
evidence of deterioration is still evidence of deterioration; supportive
requires every observed signal to agree.
"""
signals = (capex_signal, reaction_signal)
if "adverse" in signals:
return "adverse"
observed = [signal for signal in signals if signal != "unknown"]
if not observed:
return "unknown"
return "supportive" if all(signal == "supportive" for signal in observed) else "neutral"
def _usable_context(observed: bool, pending: bool, stale: bool, state: str) -> bool:
"""Whether a fundamental reading may count as evidence.
One definition, called by both the point-in-time record and the live
reading, because they publish the same field name to the same consumers and
a second copy would drift. Distinct from `available`, which is about timing
alone: an observation whose extraction failed on everything is effective and
fresh, and still knows nothing.
"""
return observed and not pending and not stale and state != "unknown"
def _evidence_quality(
capex: dict[str, str] | None,
good_news_stock_down: str | None,
names: list[str],
*,
observed: bool,
stale: bool,
source: str | None,
) -> str:
"""How much to trust the state above, as one field the reader can act on.
Ordered by what an operator most needs to know: nothing collected beats
everything else, then a reading too old to be current, then a hand override,
then completeness.
"""
if not observed:
return "unavailable"
if stale:
return "stale"
if str(source or "").strip().lower() == "manual":
return "manual"
known = sum(
1
for name in names
if str((capex or {}).get(name, "unknown")).strip().lower() != "unknown"
)
# `bool(names)` matters: with an empty basket `known == len(names)` is
# vacuously true, so nothing observed would grade as complete.
complete = (
bool(names)
and known == len(names)
and _reaction_signal(good_news_stock_down) != "unknown"
)
return "complete" if complete else "partial"
def _sensor(sensor_id: str, label: str, score: float | None, **details: object) -> dict:
return {
"id": sensor_id,
@@ -572,26 +714,66 @@ def _overlay_timing(
return effective, pending, age, stale
def fundamental_overlay(overrides: dict, config: dict, as_of: date) -> dict:
"""Point-in-time qualitative overlay. Never feeds State or Warning since v3.
def fundamental_context(overrides: dict, config: dict, as_of: date) -> dict:
"""Point-in-time fundamental channel. Never a term in State or Warning.
Called an "overlay" until 2026-08-12, which undersold it: it is the third
channel of the model, read alongside the two scores by confluence rather than
decorating them. The categorical ``state`` is what a reader and the chart
consume; ``evidence_quality`` is how far to trust it.
Both are derived from the stored categorical facts by fixed rules, not from
an LLM's numeric judgement. The LLM's job is extraction and explanation --
find the capex guidance, classify it, cite it -- and the rules turn those
facts into a state, so the same observation always yields the same category.
The effective-date gate stays even though nothing is scored from this: the
400-session rebuild replays historical dates, and stamping today's LLM read
onto 2024 snapshots would be plain lookahead in the stored record.
rebuild replays historical dates, and stamping today's read onto 2024
snapshots would be plain lookahead in the stored record.
This is the *record*. For "what do we know right now", use
This is the *record*. For "what do we know right now", use
``current_observation`` -- do not add a bypass flag here, because this runs
for every replayed date during a rebuild.
"""
effective, pending, age, stale = _overlay_timing(overrides, config, as_of)
names = list(config["tickers"]["hyperscalers"])
capex = None if pending else overrides.get("capex")
reaction = None if pending else overrides.get("good_news_stock_down")
observed = not pending and bool(overrides.get("fetched_at"))
capex_signal = _capex_signal(capex, names) if observed else "unknown"
reaction_signal = _reaction_signal(reaction) if observed else "unknown"
state = combine_fundamental_signals(capex_signal, reaction_signal)
return {
"state": state,
"evidence_quality": _evidence_quality(
capex, reaction, names,
observed=observed, stale=stale, source=overrides.get("source"),
),
"capex_signal": capex_signal,
"reaction_signal": reaction_signal,
# Two different questions, and conflating them is a trap:
#
# `available` is about *timing* -- there is an effective, non-stale record
# to display. `usable` is about *content* -- it also actually says
# something. A collected observation whose extraction failed on every
# hyperscaler is available (show it, with its date) but not usable: it
# knows nothing, so it must never count as evidence.
#
# The distinction is load-bearing for the event study. Coverage is
# measured in sessions with usable context, and if repeated extraction
# failures counted, they would slowly accumulate "exposure" until the
# fundamental rows flipped to measurable 0/8 -- a failed result reported
# for a channel that never knew anything, which is the exact confusion
# coverage-matching exists to prevent.
"available": not pending and not stale,
"usable": _usable_context(observed, pending, stale, state),
"pending": pending,
"stale": stale,
"effective_date": effective.isoformat() if effective else None,
"age_days": age,
"capex": None if pending else overrides.get("capex"),
"good_news_stock_down": None if pending else overrides.get("good_news_stock_down"),
"capex": capex,
"good_news_stock_down": reaction,
"capex_stress": None if pending else overrides.get("f1_score"),
"earnings_stress": None if pending else overrides.get("f3_score"),
"reasoning": None if pending else overrides.get("reasoning"),
@@ -603,7 +785,7 @@ def fundamental_overlay(overrides: dict, config: dict, as_of: date) -> dict:
def current_observation(overrides: dict, config: dict, as_of: date) -> dict:
"""The observation as it stands now, for the live reading only.
Same shape as ``fundamental_overlay``, but the effective date is *reported*
Same shape as ``fundamental_context``, but the effective date is *reported*
rather than used to blank the content. A refresh stamps
``_next_weekday(today)``, so gating the live card hid a just-collected read
for one day -- three over a weekend -- and refreshing appeared to do
@@ -611,14 +793,36 @@ def current_observation(overrides: dict, config: dict, as_of: date) -> dict:
published number; the stored snapshot keeps the gate.
"""
effective, pending, age, stale = _overlay_timing(overrides, config, as_of)
# The default override carries "unknown"/"mixed" placeholders for every
# The default override carries "unknown" placeholders for every
# hyperscaler. Those are the absence of an observation, not an observation
# of absence, and must never be presented as collected. ``fetched_at`` is
# the collection timestamp and is the only field written on every path that
# produces real content (LLM refresh and manual save both stamp it).
observed = bool(overrides.get("fetched_at"))
names = list(config["tickers"]["hyperscalers"])
capex_signal = _capex_signal(overrides.get("capex"), names) if observed else "unknown"
reaction_signal = (
_reaction_signal(overrides.get("good_news_stock_down")) if observed else "unknown"
)
state = combine_fundamental_signals(capex_signal, reaction_signal)
return {
"observed": observed,
"state": state,
"evidence_quality": _evidence_quality(
overrides.get("capex"), overrides.get("good_news_stock_down"), names,
observed=observed, stale=stale, source=overrides.get("source"),
),
"capex_signal": capex_signal,
"reaction_signal": reaction_signal,
# Same shape as the record means the same *fields*, not just the same
# ones this function happens to need: the frontend types both payloads
# identically, so an omission here is an undefined at runtime that
# TypeScript cannot catch across a trusted server boundary.
#
# Note this is stricter than the `available` directly below: a pending
# observation is the freshest thing we have and worth showing, but it is
# not yet in force, so it is not yet evidence.
"usable": _usable_context(observed, pending, stale, state),
# Live availability is about usefulness, not effectiveness: a pending
# observation is the freshest thing we have -- but nothing collected is
# never available.
@@ -656,8 +860,16 @@ def _compute_index(
breadth_series: Series | None = None,
divergence_series: Series | None = None,
breadth_counts: dict[date, int] | None = None,
observations: list[dict] | None = None,
) -> dict:
"""Compute the complete State/Warning snapshot as of one trading date."""
"""Compute the complete State/Warning snapshot as of one trading date.
``observations`` is the point-in-time fundamental series and is authoritative
when supplied; ``overrides`` is the single-slot fallback for callers that
predate the table (the calibration harness). Either way the reading is scored
into the same ``fundamental_context`` -- only where it is read from differs,
so the live monitor and the event study cannot report different states.
"""
tickers = config["tickers"]
smh = _closes_asof(prices.get(tickers["leaders"][0], []), as_of)
qqq = _closes_asof(prices.get(tickers["confirm"][0], []), as_of)
@@ -682,7 +894,10 @@ def _compute_index(
sensors = warning_sensor_scores(divergence, smh, spy, oas_window)
relative_strength = sensors["relative_strength"]
credit_impulse = sensors["credit_impulse"]
overlay = fundamental_overlay(overrides, config, as_of)
observation = (
observation_asof(observations, as_of) if observations is not None else overrides
) or {}
context = fundamental_context(observation, config, as_of)
state_pillars = [
{
@@ -768,7 +983,7 @@ def _compute_index(
"date": as_of.isoformat(),
"state": state,
"warning": warning,
"fundamental_overlay": overlay,
"fundamental_context": context,
"quadrant_config": {
"state_divider": QUADRANT_STATE_DIVIDER,
"warning_divider": QUADRANT_WARNING_DIVIDER,
@@ -790,8 +1005,8 @@ def _compute_index(
"breadth_pct_above_200": round(breadth_pct, 1) if breadth_pct is not None else None,
"breadth_date": breadth_item[0].isoformat() if breadth_item else None,
"fundamentals_fetched_at": overrides.get("fetched_at"),
"fundamentals_effective_date": overlay.get("effective_date"),
"fundamentals_age_days": overlay.get("age_days"),
"fundamentals_effective_date": context.get("effective_date"),
"fundamentals_age_days": context.get("age_days"),
},
"data_quality": {
"minimum_coverage": MIN_COVERAGE,
@@ -859,7 +1074,7 @@ async def get_fundamental_overrides(db: AsyncSession) -> dict:
"f1_score": None,
"f3_score": None,
"capex": {name: "unknown" for name in names},
"good_news_stock_down": "mixed",
"good_news_stock_down": "unknown",
"locked": False,
"reasoning": None,
"fetched_at": None,
@@ -880,9 +1095,9 @@ async def get_fundamental_overrides(db: AsyncSession) -> dict:
if stored.get("methodology") not in CATEGORICAL_FUNDAMENTAL_METHODOLOGIES:
return default
capex = _normalise_capex_states(stored.get("capex"), names)
reaction = str(stored.get("good_news_stock_down", "mixed")).strip().lower()
reaction = str(stored.get("good_news_stock_down", "unknown")).strip().lower()
if reaction not in GNSD_STATES:
reaction = "mixed"
reaction = "unknown"
return {
**default,
**stored,
@@ -922,6 +1137,100 @@ def _score_capex_states(capex: dict[str, str], names: list[str]) -> float | None
return round(score, 1) if score is not None else None
async def record_fundamental_observation(db: AsyncSession, observation: dict) -> None:
"""Append the observation to the point-in-time series, keyed on effective date.
Upsert rather than insert: re-saving on the same effective date is a
correction to that day's reading, not a second observation of it.
Silently does nothing without an effective date or a ``fetched_at``. Those
are the default placeholder blob -- the absence of an observation, which must
never enter the series as though someone had looked.
Deliberately does **not** commit. ``update_regime_monitor`` calls this inside
a run that owns its transaction and commits once after the snapshot loop;
committing here would take that boundary away from it. The two override
writers commit for themselves.
"""
effective = _parse_date(observation.get("effective_date"))
fetched_raw = observation.get("fetched_at")
if effective is None or not fetched_raw:
return
try:
fetched = datetime.fromisoformat(str(fetched_raw))
except ValueError:
fetched = datetime.now(timezone.utc)
if fetched.tzinfo is None:
fetched = fetched.replace(tzinfo=timezone.utc)
existing = await db.execute(
select(RegimeFundamentalObservation).where(
RegimeFundamentalObservation.effective_date == effective
)
)
row = existing.scalar_one_or_none()
payload = {
"f1_score": observation.get("f1_score"),
"f3_score": observation.get("f3_score"),
"capex_json": json.dumps(observation.get("capex") or {}),
"good_news_stock_down": str(observation.get("good_news_stock_down") or "unknown")[:10],
"reasoning": observation.get("reasoning"),
"source": str(observation.get("source") or "unknown")[:30],
"fetched_at": fetched,
}
if row is None:
db.add(RegimeFundamentalObservation(
effective_date=effective,
created_at=datetime.now(timezone.utc),
**payload,
))
else:
for key, value in payload.items():
setattr(row, key, value)
async def get_fundamental_observations(db: AsyncSession) -> list[dict]:
"""The whole observation series, oldest first, for point-in-time scoring."""
result = await db.execute(
select(RegimeFundamentalObservation).order_by(
RegimeFundamentalObservation.effective_date.asc()
)
)
out: list[dict] = []
for row in result.scalars().all():
try:
capex = json.loads(row.capex_json)
except (TypeError, ValueError):
capex = {}
out.append({
"effective_date": row.effective_date,
"f1_score": row.f1_score,
"f3_score": row.f3_score,
"capex": capex,
"good_news_stock_down": row.good_news_stock_down,
"reasoning": row.reasoning,
"source": row.source,
"fetched_at": row.fetched_at.isoformat() if row.fetched_at else None,
})
return out
def observation_asof(observations: list[dict] | None, as_of: date) -> dict | None:
"""Latest observation effective on or before ``as_of``.
This *is* the effective-date gate now. The settings-blob version had to
recompute it per call because there was only ever one observation to gate;
with a series, "which reading was live that day" is just a lookup.
"""
chosen: dict | None = None
for observation in observations or []:
if observation["effective_date"] <= as_of:
chosen = observation
else:
break
return chosen
async def set_fundamental_overrides(
db: AsyncSession,
capex: dict[str, str] | None = None,
@@ -954,7 +1263,15 @@ async def set_fundamental_overrides(
"fetched_at": now.isoformat(),
"effective_date": _next_weekday(now.date()).isoformat(),
})
await update_setting(db, KEY_FUNDAMENTALS, json.dumps(current))
# The blob (what the live card reads) and the series row (what the
# point-in-time replay reads) are the same observation. Committed together:
# `update_setting` commits internally, so using it here would leave a window
# where a failure publishes the reading to the card but not to the record,
# and the two would disagree permanently with nothing to detect it.
await settings_store.upsert_setting(db, KEY_FUNDAMENTALS, json.dumps(current))
if observation_changed:
await record_fundamental_observation(db, current)
await db.commit()
return current
@@ -1058,12 +1375,59 @@ def _snapshot_revision(snapshot: dict) -> int:
return 1
def _context_from_legacy_overlay(overlay: dict) -> dict:
"""Rebuild the categorical channel from a pre-rename snapshot's overlay.
The channel was called ``fundamental_overlay`` until 2026-08-12 and stored
the same underlying facts -- the capex map, the earnings reaction, the
effective date. The rename shipped without a methodology bump (no score
changed), so those rows are still served and were never reseeded: reading
only the new key would turn every one of them into ``unknown`` and silently
discard real recorded evidence -- historical Path colours, and any exposure
the event study could legitimately count.
Derived, not guessed. The hyperscaler list comes from the overlay's own
capex keys, which is exactly the basket that was observed at the time rather
than today's configured one.
"""
capex = overlay.get("capex") or {}
reaction = overlay.get("good_news_stock_down")
names = list(capex)
pending = bool(overlay.get("pending"))
stale = bool(overlay.get("stale"))
observed = not pending and bool(overlay.get("fetched_at"))
capex_signal = _capex_signal(capex, names) if observed else "unknown"
reaction_signal = _reaction_signal(reaction) if observed else "unknown"
state = combine_fundamental_signals(capex_signal, reaction_signal)
return {
**overlay,
"state": state,
"evidence_quality": _evidence_quality(
capex, reaction, names,
observed=observed, stale=stale, source=overlay.get("source"),
),
"capex_signal": capex_signal,
"reaction_signal": reaction_signal,
"usable": _usable_context(observed, pending, stale, state),
}
def _parse_snapshot(raw: str) -> dict | None:
try:
parsed = json.loads(raw)
except (TypeError, ValueError):
return None
return parsed if parsed.get("methodology") == METHODOLOGY else None
if parsed.get("methodology") != METHODOLOGY:
return None
# Normalise here rather than at each call site: every reader of a stored
# snapshot goes through this function, so a legacy row cannot reach one of
# them un-adapted.
if "fundamental_context" not in parsed and "fundamental_overlay" in parsed:
parsed["fundamental_context"] = _context_from_legacy_overlay(
parsed["fundamental_overlay"] or {}
)
return parsed
async def _latest_snapshot_row(db: AsyncSession) -> tuple[RegimeSnapshot, dict] | None:
@@ -1082,6 +1446,10 @@ async def update_regime_monitor(
) -> dict:
config = await get_regime_config(db)
overrides = await get_fundamental_overrides(db)
# Carries the pre-v5 single-slot observation into the series on first run, so
# a deployment does not lose the live reading. A no-op once recorded, and a
# no-op for the placeholder blob (no fetched_at).
await record_fundamental_observation(db, overrides)
if _fundamentals_stale(overrides, config) and not overrides.get("locked"):
try:
overrides = await refresh_fundamental_overrides(db, config=config)
@@ -1131,6 +1499,9 @@ async def update_regime_monitor(
breadth_series = _mapping_series(breadth)
divergence_series = _mapping_series(divergence)
# Loaded once, after any refresh, so a reseed scores each replayed date with
# the observation that was effective on it rather than with today's.
observations = await get_fundamental_observations(db)
latest_result: dict | None = None
snapshots_written = 0
for snapshot_date in dates:
@@ -1144,6 +1515,7 @@ async def update_regime_monitor(
breadth_series,
divergence_series,
breadth_counts,
observations=observations,
)
written, latest_result = await _upsert_snapshot(
db,
@@ -1221,16 +1593,22 @@ async def get_regime_monitor(db: AsyncSession) -> dict:
quality["is_fresh"] = bool(quality.get("inputs_fresh")) and snapshot_age <= 4
result["data_quality"] = quality
# The snapshot's overlay is the point-in-time record; the reader also wants
# the current observation even when it is not effective until the next
# session, because otherwise refreshing it looks like it did nothing.
# The snapshot's `fundamental_context` is the point-in-time record; the
# reader also wants the current observation even when it is not effective
# until the next session, or refreshing it looks like it did nothing.
config = await get_regime_config(db)
overrides = await get_fundamental_overrides(db)
live = current_observation(overrides, config, date.today())
# Deliberately reads the *snapshot's* overlay, not the live one: this is how
# Deliberately reads the *snapshot's* record, not the live one: this is how
# the reader tells "shown here" from "in the stored record".
live["observed_in_snapshot"] = bool((result.get("fundamental_overlay") or {}).get("available"))
result["fundamental_context"] = live
live["observed_in_snapshot"] = bool(
(result.get("fundamental_context") or {}).get("available")
)
# `fundamental_context` is the stored channel and stays the snapshot's;
# `fundamental_live` is what we know right now. Collapsing the two under one
# key is what made a just-collected observation look like it had been
# backdated into history.
result["fundamental_live"] = live
result["available"] = True
return result
@@ -1248,10 +1626,17 @@ async def get_regime_history(db: AsyncSession, days: int = 800) -> list[dict]:
if data is None:
continue
state, warning = data.get("state") or {}, data.get("warning") or {}
context = data.get("fundamental_context") or {}
out.append({
"date": row.date.isoformat(),
"state": state.get("score") if state.get("band") is not None else None,
"warning": warning.get("score") if warning.get("band") is not None else None,
# The third channel, carried per point so the Path view can colour a
# dot by the fundamental context that was on the record that day.
# Rows written before the channel existed carry nothing, which reads
# as "unknown" -- correct, since nothing was observed then either.
"fundamental_state": context.get("state") or "unknown",
"evidence_quality": context.get("evidence_quality") or "unavailable",
"state_coverage": state.get("coverage"),
"warning_coverage": warning.get("coverage"),
"basket_hash": (data.get("basket") or {}).get("hash"),
@@ -1383,7 +1768,7 @@ async def refresh_fundamental_overrides(
f1 = _score_capex_states(capex, names)
reaction = str(parsed.get("good_news_stock_down", "")).strip().lower()
if reaction not in GNSD_STATES:
reaction = "mixed"
reaction = "unknown"
f3 = _GNSD_SCORES.get(reaction)
now = datetime.now(timezone.utc)
result = {
@@ -1398,7 +1783,11 @@ async def refresh_fundamental_overrides(
"locked": False,
"source": llm.get("provider"),
}
await update_setting(db, KEY_FUNDAMENTALS, json.dumps(result))
# One transaction: see set_fundamental_overrides on why these two writes must
# not be able to land separately.
await settings_store.upsert_setting(db, KEY_FUNDAMENTALS, json.dumps(result))
await record_fundamental_observation(db, result)
await db.commit()
logger.info(json.dumps({
"event": "regime_fundamentals_refreshed",
"f1": result["f1_score"],