feat(risk-monitor): measure the rule that fires, and give fundamentals their own channel
Deploy / lint (push) Failing after 11s
Deploy / test (push) Skipped
Deploy / deploy (push) Skipped

The Warning study measured a fitted percentile crossing that nothing consumes.
What reaches Telegram is a quadrant change: fixed 50/40 dividers, hysteresis,
two-session confirmation, 3-day cooldown. Those thresholds are constants, not
fits, so there is no training set to protect and all 11 detected corrections are
evaluable instead of the 4 that fell in a holdout.

Replaying it: 1/10 corrections, 0.9 false alarms/year. Random alarms at the same
firing rate match or beat that in 65% of draws. The panel now carries ablations
(does the quadrant machinery earn its place?), external baselines (does the score
earn its complexity?), and that null, because a bare "2 of 4" was unreadable in
either direction. Nothing in the alert path was retuned on the strength of it.

Fundamentals become a third channel rather than a term in either score. v3 cut
them arguing 12+8 of 100 points "could not change any published conclusion" --
true only when every technical sensor reads zero; weighted they moved the bar for
the 40 divider from 40 to 25. But no fusion weight is measurable either: with ~10
events and no fundamental history, any weight is a policy preference presented as
a measurement. So the read is a categorical state (supportive/neutral/adverse/
unknown) with an evidence grade, derived by fixed rules from stored facts, read
by confluence. The LLM extracts and explains; it does not score.

Absence stays absence throughout. `unknown` is unreachable by averaging, a stale
or empty observation may display but never confirm, extraction failures map to
`unknown` rather than `mixed`, and the study rows are coverage-matched and marked
not-measurable until enough corrections are covered -- otherwise a fortnight of
observations renders as 0/10 and reads as a failed test.

Observations become a real time series (migration 033); they lived in a single
overwritten settings slot, so no history existed to replay. Pre-rename snapshots
are adapted rather than discarded. METHODOLOGY stays v4 -- no score changed --
so no reseed; STUDY_SCHEMA moves to 3 and discards the cached report.

Post-deploy: re-run Event Study from Admin -> Jobs. The panel reads "not run yet"
until then.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-08-13 11:15:09 +02:00
co-authored by Claude Opus 5
parent 3033ad83fd
commit 333989eeab
18 changed files with 3494 additions and 403 deletions
+398 -1
View File
@@ -1,17 +1,27 @@
"""Tests for v3 correction events, warning alarm episodes, and report caveats."""
"""Tests for correction events, alarm episodes, the shipped-rule replay, and caveats."""
from __future__ import annotations
from copy import deepcopy
from datetime import date, timedelta
import pytest
from app.services.breadth_service import _breadth_from_closes, compute_divergence_series
from app.services.event_study_service import (
MIN_EVENTS_FOR_CONFIDENCE,
STRESS_QUADRANT,
WARNING_QUADRANTS,
_era_split,
_null_model,
_percentile,
_reliability,
alarm_episodes,
below_average_series,
detect_events,
entry_alarms,
evaluate_alarms,
replay_quadrant_changes,
)
@@ -19,6 +29,33 @@ def _days(count: int, start: date = date(2021, 1, 1)) -> list[date]:
return [start + timedelta(days=index) for index in range(count)]
def _row(
warning: float,
state: float = 0.0,
*,
warning_coverage: float = 100.0,
state_coverage: float = 100.0,
fresh: bool = True,
) -> dict:
return {
"state": state,
"warning": warning,
"state_coverage": state_coverage,
"warning_coverage": warning_coverage,
"inputs_fresh": fresh,
}
def _rows(
dates: list[date], warnings: list[float], patch: dict[int, dict] | None = None
) -> dict[date, dict]:
"""One publishable row per date, with per-position replacements."""
built = {day: _row(value) for day, value in zip(dates, warnings)}
for index, replacement in (patch or {}).items():
built[dates[index]] = replacement
return built
def test_detect_events_uses_rising_edge_and_cooldown():
closes = [100.0] * 300 + [85.0] * 5 + [100.0] * 50 + [85.0] * 5
events = detect_events(closes, _days(len(closes)), threshold_pct=15.0, cooldown=40)
@@ -88,6 +125,366 @@ def test_evaluate_alarms_counts_episodes_not_alarm_days():
assert result["median_lead_days"] == 17.5
# ---------------------------------------------------------------------------
# The shipped quadrant rule, replayed
# ---------------------------------------------------------------------------
def test_replay_seeds_silently_and_needs_two_sessions():
"""A one-session spike is not an alert; the second session confirms it.
The alarm is therefore dated at the confirmation rather than at the first
crossing, which costs one session of lead. That is what ships.
"""
dates = _days(10)
spike = _rows(dates, [30] * 5 + [70] + [30] * 4)
assert replay_quadrant_changes(spike, dates) == []
held = _rows(dates, [30] * 5 + [70, 70] + [30] * 3)
fires = replay_quadrant_changes(held, dates)
# The rule alerts on quadrant changes in both directions, so the return to
# calm fires too. Only the entry is a warning about anything.
assert [(f["index"], f["from"], f["to"]) for f in fires] == [
(6, "3", "1"),
(9, "1", "3"),
]
assert entry_alarms(fires, WARNING_QUADRANTS) == [6]
def test_confirmation_classifies_the_prior_session_against_the_baseline():
"""Not against its own predecessor -- the distinction changes the answer.
Warning 42 sits inside the hysteresis deadband. Measured from the standing
"3" baseline it is still "3", so it cannot confirm a move to "1". A chain
that classified each session against the one before it would read 42 as "1"
(having just seen 70) and fire a day later, which production does not do.
"""
dates = _days(10)
rows = _rows(dates, [30, 30, 30, 30, 70, 42, 70, 30, 30, 30])
assert replay_quadrant_changes(rows, dates) == []
def test_cooldown_suppresses_and_the_baseline_only_advances_on_a_fire():
dates = _days(10)
rows = _rows(dates, [30, 30, 30, 30, 70, 70, 30, 30, 30, 30])
fires = replay_quadrant_changes(rows, dates)
# Entry confirmed on day 5. The exit confirms on day 7 but lands inside the
# 3-day cooldown, so it is re-evaluated and fires on day 8 instead.
assert [(f["index"], f["from"], f["to"]) for f in fires] == [
(5, "3", "1"),
(8, "1", "3"),
]
assert entry_alarms(fires, WARNING_QUADRANTS) == [5]
def test_low_coverage_sessions_cannot_confirm():
"""The confirmation source has to be a session that published a band."""
dates = _days(10)
warnings = [30, 30, 30, 30, 30, 70, 70, 30, 30, 30]
visible = replay_quadrant_changes(_rows(dates, warnings), dates)
assert entry_alarms(visible, WARNING_QUADRANTS) == [6]
# Day 5 is the only session that could confirm the entry on day 6; below
# MIN_COVERAGE it never published a band, so day 4 is the prior instead.
hidden = _rows(dates, warnings, {5: _row(70, warning_coverage=70.0)})
assert replay_quadrant_changes(hidden, dates) == []
def test_stale_inputs_block_todays_alert_but_not_tomorrows_confirmation():
"""is_fresh gates the live reading only; the prior session comes from history."""
dates = _days(10)
rows = _rows(dates, [30] * 4 + [70, 70, 70] + [30] * 3, {5: _row(70, fresh=False)})
fires = replay_quadrant_changes(rows, dates)
assert entry_alarms(fires, WARNING_QUADRANTS) == [6]
def test_entry_alarms_ignore_movement_inside_the_set():
fires = [
{"index": 3, "from": "3", "to": "1"},
{"index": 9, "from": "1", "to": "2"},
{"index": 20, "from": "2", "to": "4"},
]
assert entry_alarms(fires, WARNING_QUADRANTS) == [3]
assert entry_alarms(fires, STRESS_QUADRANT) == [9]
def test_below_average_series_needs_a_full_window():
series = list(zip(_days(6), [10.0, 10.0, 10.0, 10.0, 4.0, 20.0]))
indicator = below_average_series(series, window=3)
assert _days(6)[1] not in indicator # warm-up
assert indicator[_days(6)[4]] == 100.0 # 4 is under the 3-day mean of 8
assert indicator[_days(6)[5]] == 0.0
def test_null_model_is_seeded_and_drawn_from_evaluable_sessions_only():
dates = _days(300)
events = [100, 180, 260]
first = _null_model(6, events, dates, horizon=20, start_index=50, observed_warned=2, draws=200)
second = _null_model(6, events, dates, horizon=20, start_index=50, observed_warned=2, draws=200)
assert first == second # a re-run must not move the report
assert 0.0 <= first["p_at_least_observed"] <= 1.0
assert first["alarms_per_draw"] == 6
assert first["mean_warned"] <= len(events)
# More alarms than there are sessions to place them on is not a null.
assert _null_model(500, events, dates, 20, 50, 2, draws=10) is None
assert _null_model(6, [], dates, 20, 50, 0, draws=10) is None
def test_era_split_reports_the_two_sensor_eras_separately():
"""The fuller sample is mostly pre-credit, where Warning is W1+W2 only."""
dates = _days(400)
eras = _era_split(
alarms=[80, 300],
event_indices=[90, 310],
dates=dates,
horizon=20,
start_index=10,
credit_from=dates[200],
)
assert eras["pre_credit"]["events"] == 1
assert eras["pre_credit"]["events_warned"] == 1
assert eras["full_coverage"]["events"] == 1
assert eras["full_coverage"]["events_warned"] == 1
assert eras["credit_from"] == dates[200].isoformat()
# No credit series at all means there is no boundary to split on.
assert _era_split([80], [90], dates, 20, 10, None) is None
def _business_days(count: int, end: date = date(2026, 8, 7)) -> list[date]:
out: list[date] = []
cursor = end
while len(out) < count:
if cursor.weekday() < 5:
out.append(cursor)
cursor -= timedelta(days=1)
return list(reversed(out))
def _synthetic_path(sessions: int) -> list[float]:
"""A rising leader with two deep drawdowns, so corrections exist to detect."""
closes: list[float] = []
for index in range(sessions):
if index < 350:
closes.append(100.0 + index * 0.25)
elif index < 400:
closes.append(187.5 - (index - 350) * 0.9)
elif index < 650:
closes.append(142.5 + (index - 400) * 0.4)
elif index < 700:
closes.append(242.5 - (index - 650) * 1.1)
else:
closes.append(187.5 + (index - 700) * 0.3)
return closes
async def test_report_assembles_every_rule_from_synthetic_inputs(monkeypatch):
"""End-to-end: the shipped replay, ablations, baselines and null all score.
Synthetic rather than recorded because the point is the wiring -- that every
rule is measured on the same events over the same sessions and the report
carries what the panel reads. The numbers are meaningless by construction.
"""
import app.services.event_study_service as ess
sessions = 900
dates = _business_days(sessions)
closes = _synthetic_path(sessions)
leader = list(zip(dates, closes))
# SPY grinds up throughout, so the leader's relative strength rolls over
# exactly when it falls.
market = list(zip(dates, [100.0 + index * 0.12 for index in range(sessions)]))
# Breadth deteriorates ~15 sessions ahead of each decline, which is the
# divergence W1 exists to catch.
breadth = {}
for index, day in enumerate(dates):
weak = 335 <= index < 400 or 635 <= index < 700
breadth[day] = 30.0 if weak else 70.0
vix = [(day, 32.0 if (350 <= i < 400 or 650 <= i < 700) else 15.0) for i, day in enumerate(dates)]
# Credit starts late, exactly as ICE's 3-year cap makes it in production.
oas = [(day, 4.2 if (650 <= i < 700) else 3.0) for i, day in enumerate(dates) if i >= 500]
async def fake_config(_db):
return deepcopy(ess.rms.DEFAULT_CONFIG)
async def fake_prices(_config, _start, _end):
return {"SMH": leader, "QQQ": leader, "SPY": market}
async def fake_fred(series_id, _start, _end):
return {"VIXCLS": vix, "BAMLH0A0HYM2": oas}.get(series_id)
async def fake_breadth(_db, _symbols, window=200, min_tickers=20):
return breadth, {day: 30 for day in dates}
async def fake_observations(_db):
return []
monkeypatch.setattr(ess.rms, "get_regime_config", fake_config)
monkeypatch.setattr(ess.rms, "_fetch_prices", fake_prices)
monkeypatch.setattr(ess.rms, "_fetch_fred_series", fake_fred)
monkeypatch.setattr(ess.rms, "get_fundamental_observations", fake_observations)
monkeypatch.setattr(ess.breadth_service, "compute_breadth_details", fake_breadth)
monkeypatch.setattr(ess, "NULL_DRAWS", 100)
report = await ess.run_event_study(None)
assert report["available"] is True
assert report["schema"] == ess.STUDY_SCHEMA
# The shipped rule is measured on the whole sample, not a 30% holdout.
shipped = report["shipped"]
assert shipped["metrics"]["events"] == report["sample"]["events_evaluable"]
assert report["sample"]["events_evaluable"] >= 2
assert shipped["metrics"]["events"] >= report["fitted"]["metrics"]["events"]
assert len(shipped["events"]) == shipped["metrics"]["events"]
assert {row["kind"] for row in report["comparison"]} == {
"ablation", "baseline", "fundamental",
}
# Market rows share the headline's events, or the table lies. Fundamental
# rows deliberately do not: they are coverage-matched to the sessions the
# channel actually existed on, which is a different (here empty) window.
for row in report["comparison"]:
if row["kind"] != "fundamental":
assert row["events"] == shipped["metrics"]["events"]
assert row["false_alarms_per_year"] >= 0
else:
# No eligible sessions means the rate is undefined, not zero. A
# tiny-divisor fallback here printed 5e9 alarms/year.
assert row["false_alarms_per_year"] is None
# The credit sensor starts mid-sample, so the era split must be populated.
eras = shipped["by_era"]
assert eras["credit_from"] == dates[500].isoformat()
assert eras["pre_credit"]["events"] + eras["full_coverage"]["events"] == shipped["metrics"]["events"]
if report["null_model"] is not None:
assert 0.0 <= report["null_model"]["p_at_least_observed"] <= 1.0
assert report["null_model"]["observed_warned"] == shipped["metrics"]["events_warned"]
# With an empty observation series the fundamental rows are *untested*, not
# failed, and the report has to carry that distinction or a 0/10 in the table
# reads as a measured result.
coverage = report["fundamental_coverage"]
assert coverage["observations"] == 0
assert coverage["sessions_eligible"] == 0
assert coverage["events_covered"] == 0
assert coverage["measurable"] is False
fundamental_rows = [r for r in report["comparison"] if r["kind"] == "fundamental"]
assert {r["id"] for r in fundamental_rows} == {
"fundamental_adverse", "confluence", "market_over_covered",
}
assert all(row["measurable"] is False for row in fundamental_rows)
# Coverage-matched denominators: with no exposure these rows must not claim
# to have been scored against the market rows' 10 corrections.
assert all(row["events"] == 0 for row in fundamental_rows)
# Market rows are unaffected: their inputs exist for the whole window.
assert all(
row["measurable"] is True
for row in report["comparison"]
if row["kind"] != "fundamental"
)
def test_fundamental_rows_are_scored_only_on_their_own_exposure():
"""One day of coverage must not render as 0/10.
A fundamental rule scores zero whether it is wrong or merely absent, so
scoring it against corrections it could never have seen manufactures a
failed result out of a thin one — the same mistake the `measurable` flag
prevents for an empty table, arriving one observation later.
"""
import app.services.event_study_service as ess
dates = _days(300)
events = [50, 120, 200, 280]
# Context exists for a single stretch, covering only the 120 event's horizon.
rows = {
day: {
"fundamental_state": "adverse",
"fundamental_usable": 105 <= index <= 115,
}
for index, day in enumerate(dates)
}
covered = ess.covered_events(events, rows, dates, horizon=20)
assert covered == [120]
assert ess.eligible_sessions(rows, dates, start_index=0) == 11
# A stale stretch counts for nothing, however adverse it reads.
stale = {
day: {"fundamental_state": "adverse", "fundamental_usable": False}
for day in dates
}
assert ess.covered_events(events, stale, dates, horizon=20) == []
assert ess.eligible_sessions(stale, dates, start_index=0) == 0
assert ess.adverse_episodes(stale, dates, 0) == []
assert ess.confluence_episodes([120], stale, dates) == []
# And neither does a *fresh* observation that determined nothing. Repeated
# extraction failures would otherwise accumulate exposure until the rows
# flipped to a measurable 0/8 for a channel that never knew anything —
# the same tested-versus-unavailable confusion, arriving by a slower route.
empty = {
day: {"fundamental_state": "unknown", "fundamental_usable": False}
for day in dates
}
assert ess.covered_events(events, empty, dates, horizon=20) == []
assert ess.eligible_sessions(empty, dates, start_index=0) == 0
async def test_the_fundamental_channel_never_moves_the_warning_score():
"""The channel is compared, never fused. Warning must be identical either way.
A weighted modifier was built and reverted: with ~10 correction events and
almost no fundamental history any fusion weight is a policy preference
presented as a measurement.
"""
import app.services.event_study_service as ess
end = date(2026, 6, 26)
dates = _business_days(400, end)
rising = [(day, 100.0 + index * 0.2) for index, day in enumerate(dates)]
prices = {"SMH": rising, "QQQ": rising, "SPY": rising}
args = (prices, [(end, 20.0)], [(day, 4.0) for day in dates])
config = deepcopy(ess.rms.DEFAULT_CONFIG)
names = config["tickers"]["hyperscalers"]
tail = (rising, [(day, 20.0) for day in dates], dates, config)
def adverse(effective: date) -> list[dict]:
return [{
"effective_date": effective,
"f1_score": 100.0,
"f3_score": 100.0,
"capex": dict.fromkeys(names, "cutting"),
"good_news_stock_down": "yes",
"fetched_at": "2026-01-01T00:00:00+00:00",
}]
bare = ess._axis_rows(*args, *tail, None)
observed = ess._axis_rows(*args, *tail, adverse(dates[-20]))
latest, early = dates[-1], dates[-90]
assert observed[latest]["warning"] == bare[latest]["warning"]
assert observed[latest]["fundamental_state"] == "adverse"
assert bare[latest]["fundamental_state"] == "unknown"
# Sessions before the effective date stay unknown, so a rebuild cannot stamp
# today's reading onto history.
assert observed[early]["fundamental_state"] == "unknown"
# The confluence rule keeps only crossings the channel agrees with, and the
# fundamental rule fires on the transition into adverse -- both rising-edge,
# so both stay comparable with the market rows.
adverse_alarms = ess.adverse_episodes(observed, dates, 0)
assert [dates[i] for i in adverse_alarms] == [dates[-20]]
assert ess.adverse_episodes(bare, dates, 0) == []
assert ess.confluence_episodes([dates.index(early), dates.index(latest)], observed, dates) == [
dates.index(latest)
]
def test_breadth_from_fixed_closes_and_tapered_divergence():
dates = _days(10)
closes_by_symbol = {
+289 -33
View File
@@ -28,7 +28,7 @@ from app.services.regime_monitor_service import (
drawdown_pct,
f2_credit_spreads,
current_observation,
fundamental_overlay,
fundamental_context,
p1_trend_break,
p2_death_cross,
p3_drawdown,
@@ -40,6 +40,24 @@ from app.services.regime_monitor_service import (
)
async def _no_observations(_db):
return []
async def _skip_recording(_db, _observation):
return None
class _CommitOnlyDB:
"""Enough session for writers that own their own transaction boundary."""
def __init__(self) -> None:
self.commits = 0
async def commit(self) -> None:
self.commits += 1
def _dated(values: list[float], end: date = date(2026, 6, 26)) -> list[tuple[date, float]]:
return [
(end - timedelta(days=len(values) - 1 - index), value)
@@ -215,7 +233,7 @@ def test_score_pillars_gates_band_below_75_percent_coverage():
assert result["band"] is None
def test_fundamental_overlay_never_replays_before_effective_date_and_expires():
def test_fundamental_context_never_replays_before_effective_date_and_expires():
overrides = {
"f1_score": 0.0,
"f3_score": 100.0,
@@ -226,19 +244,19 @@ def test_fundamental_overlay_never_replays_before_effective_date_and_expires():
}
config = {**DEFAULT_CONFIG, "fundamental_staleness_days": 80}
pending = fundamental_overlay(overrides, config, date(2026, 6, 1))
pending = fundamental_context(overrides, config, date(2026, 6, 1))
assert pending["pending"] is True
assert pending["available"] is False
assert pending["capex"] is None
# The effective date is still reported so a pending refresh is visible.
assert pending["effective_date"] == "2026-06-02"
live = fundamental_overlay(overrides, config, date(2026, 6, 2))
live = fundamental_context(overrides, config, date(2026, 6, 2))
assert live["available"] is True
assert live["good_news_stock_down"] == "yes"
assert live["earnings_stress"] == 100.0
expired = fundamental_overlay(overrides, config, date(2026, 8, 22))
expired = fundamental_context(overrides, config, date(2026, 8, 22))
assert expired["stale"] is True
assert expired["available"] is False
@@ -263,7 +281,7 @@ def test_live_observation_is_visible_before_its_effective_date():
config = {**DEFAULT_CONFIG, "fundamental_staleness_days": 80}
before = date(2026, 6, 1)
record = fundamental_overlay(overrides, config, before)
record = fundamental_context(overrides, config, before)
now = current_observation(overrides, config, before)
# Same day, same observation: the record hides it, the live reading shows it.
@@ -314,35 +332,256 @@ def test_an_uncollected_observation_is_not_reported_as_collected():
assert current_observation(collected, DEFAULT_CONFIG, date(2026, 8, 7))["observed"] is True
def test_fundamentals_do_not_move_the_warning_score():
"""The v3 complaint: a maxed-out LLM read must not silently do nothing.
def test_fundamental_state_never_averages_unknown_into_neutral():
"""Missing evidence must not present as evidence of normality.
It no longer feeds Warning at all, so Warning is identical either way and
the observation is reported beside the score instead of buried in it.
This is the trap that mattered when the channel replaced the weighted
modifier: treating ``unknown`` as a middle value would let two ``cutting``
reads and two ``unknown`` ones land on "neutral". A single adverse read
carries on partial evidence; ``unknown`` survives only when *nothing* was
observed.
"""
names = DEFAULT_CONFIG["tickers"]["hyperscalers"]
assert rms._capex_signal(dict.fromkeys(names, "unknown"), names) == "unknown"
assert rms._capex_signal(dict.fromkeys(names, "raising"), names) == "supportive"
assert rms._capex_signal(dict.fromkeys(names, "holding"), names) == "neutral"
half_cut = {names[0]: "cutting", names[1]: "cutting", **dict.fromkeys(names[2:], "unknown")}
assert rms._capex_signal(half_cut, names) == "adverse"
assert rms._reaction_signal("yes") == "adverse"
assert rms._reaction_signal("no") == "supportive"
assert rms._reaction_signal("mixed") == "neutral"
assert rms._reaction_signal(None) == "unknown"
combine = rms.combine_fundamental_signals
assert combine("unknown", "unknown") == "unknown"
assert combine("adverse", "supportive") == "adverse" # one adverse read carries
assert combine("supportive", "unknown") == "supportive"
assert combine("neutral", "unknown") == "neutral"
assert combine("supportive", "neutral") == "neutral"
# Nothing combines *into* unknown -- that would be inventing missing evidence.
assert "unknown" not in {
combine(a, b)
for a in rms.FUNDAMENTAL_STATES
for b in rms.FUNDAMENTAL_STATES
if not (a == "unknown" and b == "unknown")
}
def test_fundamental_context_is_a_channel_not_a_term_in_warning():
"""The read is reported beside the scores and never added into them.
A weighted modifier was built and reverted: with ~10 correction events and
almost no fundamental history, any fusion weight is a policy preference
presented as a measurement, and adding a slow categorical judgement to a fast
continuous score manufactures precision by summing unlike things.
"""
end = date(2026, 6, 26)
rising = [100.0 + index * 0.2 for index in range(700)]
prices = {"SMH": _dated(rising, end), "QQQ": _dated(rising, end), "SPY": _dated(rising, end)}
args = (prices, [(end, 20.0)], [(end - timedelta(days=i), 4.0) for i in reversed(range(100))])
tail = (copy.deepcopy(DEFAULT_CONFIG), end, [(end, 55.0)], [(end, 20.0)], {end: 25})
names = DEFAULT_CONFIG["tickers"]["hyperscalers"]
quiet = _compute_index(*args, {"f1_score": None, "f3_score": None}, *tail)
screaming = _compute_index(
*args,
{
"f1_score": 100.0,
"f3_score": 100.0,
"capex": dict.fromkeys(DEFAULT_CONFIG["tickers"]["hyperscalers"], "cutting"),
"good_news_stock_down": "yes",
def observed(capex_state: str, reaction: str) -> dict:
return {
"capex": dict.fromkeys(names, capex_state),
"good_news_stock_down": reaction,
"effective_date": "2026-06-01",
},
*tail,
)
"fetched_at": "2026-06-01T00:00:00+00:00",
"source": "openai",
}
assert quiet["warning"]["score"] == screaming["warning"]["score"]
assert {p["id"] for p in quiet["warning"]["pillars"]} == set(WARNING_WEIGHTS)
assert screaming["fundamental_overlay"]["available"] is True
assert screaming["fundamental_overlay"]["capex_stress"] == 100.0
unobserved = _compute_index(*args, {"f1_score": None, "f3_score": None}, *tail)
supportive = _compute_index(*args, observed("raising", "no"), *tail)
adverse = _compute_index(*args, observed("cutting", "yes"), *tail)
# Every Warning is identical: the channel is not a term in the score.
scores = {
snapshot["warning"]["score"]
for snapshot in (unobserved, supportive, adverse)
}
assert len(scores) == 1
assert {p["id"] for p in unobserved["warning"]["pillars"]} == set(WARNING_WEIGHTS)
# And it never touches coverage, so a missing observation cannot suppress a
# band or silently redistribute weight onto the technical sensors.
assert len({s["warning"]["coverage"] for s in (unobserved, supportive, adverse)}) == 1
assert unobserved["fundamental_context"]["state"] == "unknown"
assert unobserved["fundamental_context"]["evidence_quality"] == "unavailable"
assert supportive["fundamental_context"]["state"] == "supportive"
assert adverse["fundamental_context"]["state"] == "adverse"
assert adverse["fundamental_context"]["evidence_quality"] == "complete"
def test_a_fresh_but_empty_observation_is_available_to_show_and_not_usable():
"""Collected-but-determined-nothing must not count as evidence.
`available` is about timing (there is an effective, non-stale record to
display); `usable` is about content. An LLM run that failed to extract
anything produces a perfectly fresh observation that knows nothing — and if
that counted, repeated extraction failures would slowly accumulate study
exposure until the fundamental rows reported a measurable 0/8 for a channel
that had never seen a thing.
"""
config = copy.deepcopy(DEFAULT_CONFIG)
names = config["tickers"]["hyperscalers"]
as_of = date(2026, 6, 26)
base = {
"effective_date": "2026-06-01",
"fetched_at": "2026-06-01T00:00:00+00:00",
"source": "openai",
}
empty = fundamental_context(
{**base, "capex": dict.fromkeys(names, "unknown"), "good_news_stock_down": "unknown"},
config, as_of,
)
assert empty["state"] == "unknown"
assert empty["available"] is True # there is a record, and it has a date
assert empty["usable"] is False # but it says nothing
# One real signal is enough to be usable, on partial evidence.
partial = fundamental_context(
{
**base,
"capex": {names[0]: "cutting", **dict.fromkeys(names[1:], "unknown")},
"good_news_stock_down": "unknown",
},
config, as_of,
)
assert partial["state"] == "adverse"
assert partial["usable"] is True
assert partial["evidence_quality"] == "partial"
# Stale is neither available nor usable — `available` means effective *and*
# non-stale. What survives is `state`, which the card renders on its own
# (with the stale badge) so the last thing observed stays visible.
stale = fundamental_context(
{
**base,
"effective_date": "2026-01-01",
"capex": dict.fromkeys(names, "cutting"),
"good_news_stock_down": "yes",
},
config, as_of,
)
assert stale["state"] == "adverse"
assert stale["stale"] is True
assert stale["available"] is False
assert stale["usable"] is False
# Nothing collected at all: neither.
absent = fundamental_context({}, config, as_of)
assert (absent["available"], absent["usable"]) == (False, False)
def test_the_live_reading_publishes_the_same_fields_as_the_record():
""""Same shape" has to mean the same fields, not the same ones it needs.
The frontend types both payloads as one interface, so a field present on the
record and missing from the live reading is an undefined at runtime that
TypeScript cannot catch across a trusted server boundary.
"""
config = copy.deepcopy(DEFAULT_CONFIG)
names = config["tickers"]["hyperscalers"]
as_of = date(2026, 6, 26)
observation = {
"effective_date": "2026-06-01",
"fetched_at": "2026-06-01T00:00:00+00:00",
"source": "openai",
"capex": dict.fromkeys(names, "cutting"),
"good_news_stock_down": "yes",
}
record = fundamental_context(observation, config, as_of)
live = current_observation(observation, config, as_of)
assert set(record) <= set(live)
assert (live["state"], live["usable"]) == ("adverse", True)
# A just-collected observation is shown but is not yet in force, so it is
# available to read and not yet usable as evidence.
pending = current_observation(
{**observation, "effective_date": "2026-07-01"}, config, as_of
)
assert (pending["pending"], pending["available"], pending["usable"]) == (True, True, False)
# And an extraction that determined nothing is never usable, however fresh.
empty = current_observation(
{**observation, "capex": dict.fromkeys(names, "unknown"), "good_news_stock_down": "unknown"},
config, as_of,
)
assert (empty["state"], empty["usable"]) == ("unknown", False)
def test_pre_rename_snapshots_keep_their_recorded_fundamental_evidence():
"""The rename shipped without a methodology bump, so those rows were never reseeded.
Reading only the new key would turn real observations into `unknown` and
silently drop historical Path colours and legitimate study exposure.
"""
names = DEFAULT_CONFIG["tickers"]["hyperscalers"]
legacy = {
"methodology": rms.METHODOLOGY,
"date": "2026-07-01",
"state": {"score": 10.0, "band": "stable"},
"warning": {"score": 20.0, "band": "stable"},
"fundamental_overlay": {
"available": True,
"pending": False,
"stale": False,
"effective_date": "2026-06-20",
"capex": {names[0]: "cutting", **dict.fromkeys(names[1:], "raising")},
"good_news_stock_down": "yes",
"source": "openai",
"fetched_at": "2026-06-19T00:00:00+00:00",
},
}
parsed = rms._parse_snapshot(json.dumps(legacy))
context = parsed["fundamental_context"]
assert context["state"] == "adverse"
assert context["evidence_quality"] == "complete"
assert context["usable"] is True
assert context["effective_date"] == "2026-06-20"
# A pending legacy overlay carried no facts, so it stays unknown rather than
# inventing an observation for a session nobody had looked at.
blank = json.loads(json.dumps(legacy))
blank["fundamental_overlay"] = {"pending": True, "stale": False, "capex": None}
blank_context = rms._parse_snapshot(json.dumps(blank))["fundamental_context"]
assert blank_context["state"] == "unknown"
assert blank_context["evidence_quality"] == "unavailable"
assert blank_context["usable"] is False
# A row already carrying the new key is left exactly as written.
modern = json.loads(json.dumps(legacy))
modern["fundamental_context"] = {"state": "supportive", "usable": True}
assert rms._parse_snapshot(json.dumps(modern))["fundamental_context"]["state"] == "supportive"
def test_evidence_quality_ranks_what_an_operator_needs_first():
names = DEFAULT_CONFIG["tickers"]["hyperscalers"]
config = copy.deepcopy(DEFAULT_CONFIG)
full = dict.fromkeys(names, "raising")
partial = {names[0]: "raising", **dict.fromkeys(names[1:], "unknown")}
def quality(capex, reaction, *, observed=True, stale=False, source="openai"):
return rms._evidence_quality(
capex, reaction, names, observed=observed, stale=stale, source=source
)
assert quality(full, "no") == "complete"
assert quality(partial, "no") == "partial"
assert quality(full, None) == "partial" # reaction unknown
assert quality(full, "no", source="manual") == "manual"
assert quality(full, "no", stale=True) == "stale"
# Nothing collected outranks every other grade.
assert quality(full, "no", observed=False, stale=True, source="manual") == "unavailable"
assert set(rms.EVIDENCE_QUALITY) >= {quality(full, "no"), quality(partial, "no")}
assert config["tickers"]["hyperscalers"] == names
def test_capex_score_separates_holding_from_raising():
@@ -375,10 +614,12 @@ async def test_legacy_numeric_fundamentals_do_not_leak_into_v4(monkeypatch):
result = await rms.get_fundamental_overrides(object())
assert result["methodology"] == "v4"
assert result["methodology"] == rms.METHODOLOGY
assert result["f1_score"] is None
assert result["f3_score"] is None
assert result["good_news_stock_down"] == "mixed"
# Not "mixed": an unreadable blob is an absence of an observation, and
# "mixed" is a genuinely observed mixed reaction.
assert result["good_news_stock_down"] == "unknown"
@pytest.mark.asyncio
@@ -442,11 +683,12 @@ async def test_unlock_does_not_redate_a_fundamental_observation(monkeypatch):
async def fake_update(_db, _key, value):
saved.update(json.loads(value))
return None
monkeypatch.setattr(rms, "get_fundamental_overrides", fake_get)
monkeypatch.setattr(rms, "update_setting", fake_update)
monkeypatch.setattr(rms.settings_store, "upsert_setting", fake_update)
result = await rms.set_fundamental_overrides(object(), locked=False)
result = await rms.set_fundamental_overrides(_CommitOnlyDB(), locked=False)
assert result["locked"] is False
assert result["fetched_at"] == stored["fetched_at"]
@@ -476,14 +718,22 @@ async def test_manual_fundamentals_are_categorical_and_derived(monkeypatch):
async def fake_update(_db, _key, value):
saved.update(json.loads(value))
return None
monkeypatch.setattr(rms, "get_fundamental_overrides", fake_get)
monkeypatch.setattr(rms, "update_setting", fake_update)
monkeypatch.setattr(rms.settings_store, "upsert_setting", fake_update)
# A manual save now also appends to the point-in-time series.
monkeypatch.setattr(rms, "record_fundamental_observation", _skip_recording)
capex = {names[0]: "cutting", **dict.fromkeys(names[1:], "holding")}
db = _CommitOnlyDB()
result = await rms.set_fundamental_overrides(
object(), capex=capex, good_news_stock_down="mixed"
db, capex=capex, good_news_stock_down="mixed"
)
# The series row is a second write after update_setting's own commit, so the
# writer has to take one -- record_fundamental_observation deliberately does
# not, or it would steal update_regime_monitor's transaction boundary.
assert db.commits == 1
assert result["f1_score"] == 62.5 # one cutting (100) + three holding (50)
assert result["f3_score"] is None
@@ -498,7 +748,9 @@ async def test_manual_fundamentals_are_categorical_and_derived(monkeypatch):
async def test_prior_snapshot_is_immutable_without_explicit_rebuild(db_session):
snapshot_date = date(2026, 6, 26)
first = {
"methodology": "v4",
# Must be the *current* methodology: a foreign row does not parse, so it
# reads as absent and the rewrite guard never comes into play.
"methodology": rms.METHODOLOGY,
"date": snapshot_date.isoformat(),
"state": {"score": 10.0, "band": "stable"},
"warning": {"score": 20.0, "band": "stable"},
@@ -571,6 +823,8 @@ async def test_routine_can_refresh_latest_trading_session_after_civil_day_rolls(
monkeypatch.setattr(rms.breadth_service, "compute_breadth_details", fake_breadth)
monkeypatch.setattr(rms, "_latest_snapshot_row", fake_latest)
monkeypatch.setattr(rms, "_upsert_snapshot", fake_upsert)
monkeypatch.setattr(rms, "get_fundamental_observations", _no_observations)
monkeypatch.setattr(rms, "record_fundamental_observation", _skip_recording)
result = await rms.update_regime_monitor(FakeDB())
@@ -636,6 +890,8 @@ async def test_a_stale_sensor_revision_reseeds_stored_history(
("_fetch_fred_series", fake_fred),
("_latest_snapshot_row", fake_latest),
("_upsert_snapshot", fake_upsert),
("get_fundamental_observations", _no_observations),
("record_fundamental_observation", _skip_recording),
):
monkeypatch.setattr(rms, name, value)
monkeypatch.setattr(rms.breadth_service, "compute_breadth_details", fake_breadth)
@@ -722,7 +978,7 @@ def test_compute_index_uses_one_max_price_vote_and_has_no_combined_score():
price = next(p for p in result["state"]["pillars"] if p["id"] == "price")
sensor_scores = [sensor["score"] for sensor in price["sensors"] if sensor["score"] is not None]
assert price["score"] == max(sensor_scores)
assert result["methodology"] == "v4"
assert result["methodology"] == rms.METHODOLOGY
assert "combined" not in result
assert result["basket"]["members_available"] == 25
+143
View File
@@ -5,10 +5,16 @@ different realized ranges -- Warning never exceeded 64.9 in the 408 calibration
sessions, so a shared 60 left the whole upper half of that axis unreachable.
"""
import pytest
from app.services import alert_service
from app.services.alert_service import (
CONFLUENCE_TYPE,
FUND_TYPE,
QUAD_X_DIV,
QUAD_Y_DIV,
_classify_quadrant,
_collect_regime_fundamental,
_parse_quadrant_log_key,
_quadrant_log_key,
)
@@ -48,3 +54,140 @@ def test_quadrant_key_carries_basket_hash_and_parses_legacy_keys():
assert _parse_quadrant_log_key(key) == ("abc123", "3", 32.4, 54.6)
assert _parse_quadrant_log_key("3:32.4:54.6") == (None, "3", 32.4, 54.6)
assert _parse_quadrant_log_key("3") == (None, "3", None, None)
# ---------------------------------------------------------------------------
# Fundamental-context and confluence alerts
# ---------------------------------------------------------------------------
def _monitor(
warning_score: float, state: str, *, coverage: float = 100.0, usable: bool = True
) -> dict:
return {
"available": True,
"warning": {"score": warning_score, "coverage": coverage},
"fundamental_context": {
"state": state,
"evidence_quality": "complete" if usable else "stale",
# The state survives going stale so the card can still show it, and
# a failed extraction is fresh but knows nothing; `usable` is what
# says whether it may still confirm anything.
"available": usable,
"usable": usable,
},
"data_quality": {"is_fresh": True},
"quadrant_config": {"warning_divider": QUAD_Y_DIV},
}
class _LogSpyDB:
"""Records what would be logged; returns a canned "last logged key"."""
def __init__(self, last: dict[str, str | None]) -> None:
self.last = last
self.logged: list[tuple[str, str]] = []
@pytest.fixture
def patched(monkeypatch):
def apply(data: dict, last: dict[str, str | None]):
db = _LogSpyDB(last)
async def fake_monitor(_db):
return data
async def fake_last(_db, alert_type):
return db.last.get(alert_type)
def fake_log(_db, alert_type, key, value=None):
db.logged.append((alert_type, key))
import app.services.regime_monitor_service as rms
monkeypatch.setattr(rms, "get_regime_monitor", fake_monitor)
monkeypatch.setattr(alert_service, "_last_logged_key", fake_last)
monkeypatch.setattr(alert_service, "_log_alert", fake_log)
return db
return apply
@pytest.mark.asyncio
async def test_first_run_seeds_both_channels_without_alerting(patched):
db = patched(_monitor(60.0, "adverse"), {FUND_TYPE: None, CONFLUENCE_TYPE: None})
assert await _collect_regime_fundamental(db) == []
assert dict(db.logged) == {FUND_TYPE: "adverse", CONFLUENCE_TYPE: "yes"}
@pytest.mark.asyncio
async def test_fundamental_change_and_confluence_are_separate_messages(patched):
db = patched(_monitor(60.0, "adverse"), {FUND_TYPE: "neutral", CONFLUENCE_TYPE: "no"})
out = await _collect_regime_fundamental(db)
assert [alert_type for alert_type, _, _ in out] == [FUND_TYPE, CONFLUENCE_TYPE]
assert "neutral → adverse" in out[0][2]
assert "Confluence" in out[1][2]
# Neither message reports a fused score; they name which channel moved.
assert "not a score" in out[0][2]
@pytest.mark.asyncio
async def test_unknown_never_alerts(patched):
"""Absence of evidence is not a change in the evidence."""
db = patched(_monitor(60.0, "unknown"), {FUND_TYPE: "neutral", CONFLUENCE_TYPE: "no"})
assert await _collect_regime_fundamental(db) == []
@pytest.mark.asyncio
async def test_adverse_alone_is_not_confluence(patched):
"""A calm tape with adverse fundamentals is a context change, not confluence."""
db = patched(_monitor(10.0, "adverse"), {FUND_TYPE: "neutral", CONFLUENCE_TYPE: "no"})
out = await _collect_regime_fundamental(db)
assert [alert_type for alert_type, _, _ in out] == [FUND_TYPE]
@pytest.mark.asyncio
async def test_leaving_confluence_rebaselines_quietly(patched):
db = patched(_monitor(10.0, "neutral"), {FUND_TYPE: "neutral", CONFLUENCE_TYPE: "yes"})
assert await _collect_regime_fundamental(db) == []
assert (CONFLUENCE_TYPE, "no") in db.logged
@pytest.mark.asyncio
async def test_low_coverage_or_stale_inputs_stay_quiet(patched):
thin = _monitor(60.0, "adverse", coverage=50.0)
assert await _collect_regime_fundamental(
patched(thin, {FUND_TYPE: "neutral", CONFLUENCE_TYPE: "no"})
) == []
stale = _monitor(60.0, "adverse")
stale["data_quality"]["is_fresh"] = False
assert await _collect_regime_fundamental(
patched(stale, {FUND_TYPE: "neutral", CONFLUENCE_TYPE: "no"})
) == []
@pytest.mark.asyncio
async def test_a_stale_observation_cannot_confirm_a_new_crossing(patched):
"""The state is kept for display, but it stops being evidence.
Without this, one adverse read corroborates every Warning crossing for the
rest of time — the strongest claim the channel makes, from the data with the
least right to make it.
"""
stale = _monitor(60.0, "adverse", usable=False)
db = patched(stale, {FUND_TYPE: "adverse", CONFLUENCE_TYPE: "no"})
assert await _collect_regime_fundamental(db) == []
# It also rebaselines to "no", so recollecting the observation re-arms it.
assert (CONFLUENCE_TYPE, "no") not in db.logged # already "no"; nothing to log
fresh = _monitor(60.0, "adverse", usable=True)
db2 = patched(fresh, {FUND_TYPE: "adverse", CONFLUENCE_TYPE: "no"})
out = await _collect_regime_fundamental(db2)
assert [alert_type for alert_type, _, _ in out] == [CONFLUENCE_TYPE]
@pytest.mark.asyncio
async def test_a_stale_state_change_does_not_alert(patched):
db = patched(_monitor(10.0, "adverse", usable=False), {FUND_TYPE: "neutral", CONFLUENCE_TYPE: "no"})
assert await _collect_regime_fundamental(db) == []