feat(risk-monitor): measure the rule that fires, and give fundamentals their own channel
The Warning study measured a fitted percentile crossing that nothing consumes. What reaches Telegram is a quadrant change: fixed 50/40 dividers, hysteresis, two-session confirmation, 3-day cooldown. Those thresholds are constants, not fits, so there is no training set to protect and all 11 detected corrections are evaluable instead of the 4 that fell in a holdout. Replaying it: 1/10 corrections, 0.9 false alarms/year. Random alarms at the same firing rate match or beat that in 65% of draws. The panel now carries ablations (does the quadrant machinery earn its place?), external baselines (does the score earn its complexity?), and that null, because a bare "2 of 4" was unreadable in either direction. Nothing in the alert path was retuned on the strength of it. Fundamentals become a third channel rather than a term in either score. v3 cut them arguing 12+8 of 100 points "could not change any published conclusion" -- true only when every technical sensor reads zero; weighted they moved the bar for the 40 divider from 40 to 25. But no fusion weight is measurable either: with ~10 events and no fundamental history, any weight is a policy preference presented as a measurement. So the read is a categorical state (supportive/neutral/adverse/ unknown) with an evidence grade, derived by fixed rules from stored facts, read by confluence. The LLM extracts and explains; it does not score. Absence stays absence throughout. `unknown` is unreachable by averaging, a stale or empty observation may display but never confirm, extraction failures map to `unknown` rather than `mixed`, and the study rows are coverage-matched and marked not-measurable until enough corrections are covered -- otherwise a fortnight of observations renders as 0/10 and reads as a failed test. Observations become a real time series (migration 033); they lived in a single overwritten settings slot, so no history existed to replay. Pre-rename snapshots are adapted rather than discarded. METHODOLOGY stays v4 -- no score changed -- so no reseed; STUDY_SCHEMA moves to 3 and discards the cached report. Post-deploy: re-run Event Study from Admin -> Jobs. The panel reads "not run yet" until then. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -39,6 +39,162 @@ session, the calendar anchors, 100% coverage on every row, and a row-wise
|
||||
`state_v4 <= state_v3` invariant. Reading a calibration result out of a run whose
|
||||
pipeline did not validate is meant to be structurally impossible.
|
||||
|
||||
## The fundamental channel (2026-08-12)
|
||||
|
||||
The monitor has **three channels**, not two scores with a decoration:
|
||||
|
||||
- **State** — current observable technical stress (price, breadth, credit, volatility).
|
||||
- **Warning** — observable deterioration that may precede stress (breadth
|
||||
divergence, relative strength, credit impulse).
|
||||
- **Fundamental context** — a categorical state (`supportive` / `neutral` /
|
||||
`adverse` / `unknown`) with an `evidence_quality` grade.
|
||||
|
||||
The third is **never a term in the other two**. They are read together by
|
||||
confluence:
|
||||
|
||||
| Warning | Fundamentals | Reading |
|
||||
|---|---|---|
|
||||
| Calm | Supportive/neutral | Normal |
|
||||
| Elevated | Supportive/neutral | Technical warning, not fundamentally confirmed |
|
||||
| Calm | Adverse | Fundamental concern; tape has not confirmed |
|
||||
| Elevated | Adverse | Confluence — highest attention |
|
||||
|
||||
`METHODOLOGY` stays **v4**: no score changed, so partitioning the history API and
|
||||
discarding the event study cache would be churn. `STUDY_SCHEMA` moved to 3
|
||||
instead, and is now the only thing that discards a stale report.
|
||||
|
||||
### Why the read is a channel and not a weight
|
||||
|
||||
Two things are true at once, and only this shape honours both.
|
||||
|
||||
**v3's reason for removing fundamentals from the score was wrong.** Not stale —
|
||||
wrong. v3 argued that F1 (capex) and F3 (good-news-stock-down), carrying 12 + 8
|
||||
of 100 Warning points, "could not change any published conclusion" because pegged
|
||||
they produced a Warning of exactly 20.0, below the alarm threshold. That
|
||||
arithmetic holds only when *every* technical sensor reads exactly zero, which is
|
||||
the one case that never matters. Warning is a weighted average, so the sensors
|
||||
add:
|
||||
|
||||
| technical Warning | without fundamentals | with them pegged | delta |
|
||||
|---|---|---|---|
|
||||
| 0 | 0.0 | 20.0 | +20.0 |
|
||||
| 20 | 20.0 | 36.0 | +16.0 |
|
||||
| 25 | 25.0 | **40.0** | +15.0 |
|
||||
| 35 | 35.0 | **48.0** | +13.0 |
|
||||
| 50 | 50.0 | 60.0 | +10.0 |
|
||||
| 80 | 80.0 | 84.0 | +4.0 |
|
||||
|
||||
Pegged fundamentals lowered the technical Warning needed to reach the 40 quadrant
|
||||
divider from 40 to 25. That is a 15-point shift in where the alert fires, which
|
||||
is emphatically a changed conclusion. The v3 section below is kept as written,
|
||||
with this correction attached, because its reasoning is cited elsewhere in this
|
||||
file and a silent overwrite would hide that the error was ever made.
|
||||
|
||||
**But no weight is measurable either.** A weighted modifier was built and
|
||||
reverted: 0–25 points added onto the technical Warning, sized so a maxed-out read
|
||||
carried a calm tape over the 40 divider on its own. Nothing could justify the 25.
|
||||
With ~10 correction events and essentially no fundamental history, any fusion
|
||||
weight is a policy preference presented as a measurement — and the debate it
|
||||
invites ("does the read deserve 10%, 20%, 30%?") has no evidence that can settle
|
||||
it. Adding a slow categorical judgement to a fast continuous score also
|
||||
manufactures precision by summing unlike things, and it forces a missing
|
||||
observation to silently redistribute its weight onto the technical sensors, which
|
||||
is the opposite of leaving it unknown.
|
||||
|
||||
So: the read gets a channel, not a coefficient. Both facts survive — the v3
|
||||
removal was badly argued *and* no weight is defensible — because "report it
|
||||
separately" is the only design that neither buries the observation nor invents a
|
||||
number for it.
|
||||
|
||||
### Derivation
|
||||
|
||||
Deterministic, from the stored categorical facts. The LLM is an **extraction and
|
||||
explanation layer**: it finds the capex guidance, classifies it, and cites it.
|
||||
Fixed rules turn those facts into a state, so the same observation always yields
|
||||
the same category.
|
||||
|
||||
`capex_signal`: any `cutting` → adverse; else any `holding` → neutral; else all
|
||||
known `raising` → supportive; nothing known → unknown.
|
||||
`reaction_signal`: `yes` → adverse, `mixed` → neutral, `no` → supportive,
|
||||
`unknown` → unknown.
|
||||
|
||||
`mixed` and `unknown` are different reaction states and were merged until
|
||||
2026-08-13. A failed LLM parse fell back to `mixed`, so an extraction error
|
||||
became *neutral evidence* — an observation of normality manufactured out of a
|
||||
bug. `mixed` now means an observed mixed reaction; anything unreadable, missing
|
||||
or unattempted is `unknown` and contributes nothing.
|
||||
|
||||
Combined by precedence, never by averaging: **any adverse read carries**; both
|
||||
unknown → unknown; every observed signal supportive → supportive; otherwise
|
||||
neutral.
|
||||
|
||||
`unknown` is deliberately unreachable by combination. Averaging would let two
|
||||
`cutting` reads and two `unknown` ones land on "neutral", presenting missing
|
||||
evidence as evidence of normality — the same conflation `current_observation`
|
||||
already refuses between "no observation" and "an observation of zero". Two cuts
|
||||
and two unknowns read **adverse with `evidence_quality: partial`**.
|
||||
|
||||
`evidence_quality` is ordered by what an operator needs first: `unavailable`
|
||||
(nothing collected) → `stale` (past `fundamental_staleness_days`) → `manual`
|
||||
(hand override) → `complete` / `partial`.
|
||||
|
||||
### Presentation and alerts
|
||||
|
||||
The Path view colours each dot by the fundamental state recorded that day; the
|
||||
axes are untouched, because context is confluence information rather than a
|
||||
position on either axis. The card leads with the state and evidence grade.
|
||||
|
||||
Alerts stay **separate**, off one toggle:
|
||||
|
||||
- quadrant change — the market axes moved (existing);
|
||||
- `regime_fundamental` — the context changed, e.g. neutral → adverse;
|
||||
- `regime_confluence` — Warning elevated *and* fundamentals adverse.
|
||||
|
||||
`unknown` never alerts: an absence of evidence is not a change in the evidence,
|
||||
and alerting on it would train the reader to ignore the channel. Both new
|
||||
triggers seed silently on first run, as the quadrant alert does.
|
||||
|
||||
### The observation is now a real time series
|
||||
|
||||
`regime_fundamental_observations` (migration 033), one row per `effective_date`,
|
||||
upserted. Before this it lived in a single `SystemSetting` slot that every
|
||||
refresh overwrote, so no history existed at all — which made the read impossible
|
||||
to replay, impossible to backtest, and meant a rebuild recorded every historical
|
||||
session as if nothing had been observed. `update_regime_monitor` carries the
|
||||
pre-existing single-slot observation into the series on its next run.
|
||||
|
||||
### What this does not establish
|
||||
|
||||
The table starts empty and fills one observation at a time, so the fundamental
|
||||
rows are **untested, not failed**. Two things enforce that rather than one:
|
||||
|
||||
- they are **coverage-matched** — scored only on sessions where the channel had
|
||||
usable context and on corrections whose warning horizon fell inside it, with a
|
||||
market-only comparator over the identical window so any difference between them
|
||||
is the channel and not the window;
|
||||
- `measurable` stays false until `MIN_EVENTS_FOR_CONFIDENCE` corrections are
|
||||
covered, and the panel prints "insufficient exposure" rather than a ratio.
|
||||
|
||||
Without the first, one day of coverage would render as 0/10 — recreating, one
|
||||
observation later, exactly the tested-versus-unavailable confusion the flag was
|
||||
added to prevent. The market rows are unchanged, and the 1/10 shipped-rule figure
|
||||
remains a verdict on the technical sensors and the alert machinery alone.
|
||||
|
||||
The rationale for expecting the read to matter is the operator's: hyperscaler
|
||||
capex is the demand side of the entire AI trade, and good earnings being sold is
|
||||
a classic late-cycle tell. Both are plausible. Neither is measured here, and this
|
||||
file's convention is that published numbers are reproducible.
|
||||
|
||||
**The path forward is accumulation, then a test — in that order.** Once enough
|
||||
point-in-time observations exist, test whether the state improves prediction
|
||||
*conditional on* Warning. If it does, a fitted and calibrated model has something
|
||||
to fit; until then there is nothing to calibrate against. Backfilling would get
|
||||
there faster: capex direction is derivable from the 10-Q/10-K capex line, which
|
||||
the SEC fundamentals import already carries, and "good news, stock down" from
|
||||
earnings dates plus next-day returns, which the Dolt earnings import already
|
||||
carries. That last one is worth computing deterministically rather than asking
|
||||
the LLM to judge, for the same reason the state derivation is rule-based.
|
||||
|
||||
## What changed in v4
|
||||
|
||||
**V1 stopped saturating at VIX 30.** `(vix - 15) / 15` reached 100 at VIX 30 —
|
||||
@@ -81,6 +237,16 @@ a qualitative overlay reported beside the scores. Capex also stopped scoring
|
||||
`raising` and `holding` identically at 0: `holding` is the deceleration case and
|
||||
now scores 50, so a boom no longer reads the same as a stall.
|
||||
|
||||
> **Corrected 2026-08-12.** The claim in this paragraph is false. "Pegged
|
||||
> they produced a Warning of exactly 20.0" describes only the case where every
|
||||
> technical sensor reads zero; Warning is a weighted average, so in the general
|
||||
> case those 20 points added +10 to +20 and moved the technical score needed to
|
||||
> reach the 40 quadrant divider from 40 to 25. The observation was removed for
|
||||
> being *underweighted*, on reasoning that mistook a corner case for the whole
|
||||
> range. See "The fundamental channel" above for what replaced it — a separate
|
||||
> categorical channel, not a restored weight. The capex `holding` rescale in the second half
|
||||
> of this paragraph stands and is still live.
|
||||
|
||||
**The drawdown sensor stopped saturating.** v2 used `dd_pct * 5`, reaching 100 at
|
||||
a 20% drawdown — the 90th percentile of the observed distribution. 39 of 408
|
||||
sessions sat at exactly 100 with no resolution left, and the price pillar showed
|
||||
@@ -126,7 +292,11 @@ upper half of the Warning axis was unreachable.
|
||||
- 60-session SMH/SPY relative-strength deterioration, 30%.
|
||||
- HY OAS 20-session widening, 25%.
|
||||
|
||||
Combined, RSP/SPY (former F4), and the NVDA canary (former P6) do not enter v3 or v4.
|
||||
**Fundamental context** — a categorical third channel, not a term in either
|
||||
score. See "The fundamental channel" above.
|
||||
|
||||
Combined, RSP/SPY (former F4), and the NVDA canary (former P6) do not enter v3
|
||||
or v4.
|
||||
|
||||
## Calibration
|
||||
|
||||
@@ -279,31 +449,73 @@ reseed exists to close. The history API and main chart show only snapshots match
|
||||
the current methodology, so a bump reseeds the series rather than splicing two
|
||||
formulas into one line.
|
||||
|
||||
The fundamental overlay keeps its effective date (normally the next session after
|
||||
The fundamental channel keeps its effective date (normally the next session after
|
||||
collection) and is never replayed backward, so a rebuild cannot stamp today's
|
||||
observation onto historical snapshots. Because the observation is stored in a
|
||||
single slot, a refresh replaces the previously effective record: the snapshot
|
||||
therefore reports the overlay as `pending` until the new effective date.
|
||||
observation onto historical snapshots. Since the observations became a real
|
||||
series (`regime_fundamental_observations`, migration 033), the effective-date
|
||||
lookup *is* the gate: a replayed session gets whichever observation was live on
|
||||
it, and sessions before the first one read `unknown`.
|
||||
|
||||
Two functions, deliberately: `fundamental_overlay` is the **record** and keeps
|
||||
Two functions, deliberately: `fundamental_context` is the **record** and keeps
|
||||
the gate — it runs for every replayed date during a rebuild, so it must never
|
||||
grow a bypass flag. `current_observation` is the **live reading** behind
|
||||
`fundamental_context`, and *reports* the effective date instead of blanking the
|
||||
`fundamental_live`, and *reports* the effective date instead of blanking the
|
||||
content.
|
||||
|
||||
Until 2026-08-07 the live reading called the gated function, so a just-collected
|
||||
observation stayed hidden until the next weekday — three days over a weekend —
|
||||
and refreshing appeared to do nothing. That was the opposite of what this section
|
||||
already claimed. Showing it early cannot leak into a published number, because
|
||||
nothing in the overlay is scored (see "Fundamentals left the score").
|
||||
already claimed. Showing it early cannot leak into a published score, because
|
||||
nothing in the channel is scored.
|
||||
|
||||
`current_observation` gates on `observed` (a non-null `fetched_at`, the one field
|
||||
every path writing real content stamps). Without it, the default override —
|
||||
`unknown` for every hyperscaler and `mixed` for the reaction — was reported as a
|
||||
live observation with `available: true`, so the card presented placeholders as a
|
||||
collected reading. Those are the absence of an observation, not an observation of
|
||||
absence. `fundamental_overlay` never had this problem: no observation means no
|
||||
effective date, which means `pending`, which already blanks the content.
|
||||
`unknown` for every hyperscaler and, since 2026-08-13, `unknown` for the reaction
|
||||
— was reported as a live observation with `available: true`, so the card
|
||||
presented placeholders as a collected reading. Those are the absence of an
|
||||
observation, not an observation of absence. `fundamental_context` never had this
|
||||
problem: no observation means no effective date, which means `pending`, which
|
||||
already blanks the content.
|
||||
|
||||
**`usable` is what may confirm; `available` is only what to display.** Three
|
||||
distinct things, and collapsing any two of them is a bug:
|
||||
|
||||
- `state` — the last thing observed. Survives going stale, so the card can show it.
|
||||
- `available` — *timing*: there is an effective, non-stale record to display.
|
||||
- `usable` — *content*: available **and** the observation actually determined
|
||||
something (`state != "unknown"`).
|
||||
|
||||
The confluence alert and all three coverage-matched study rules gate on `usable`.
|
||||
Gating on `available` instead has two failure modes, and both were live at some
|
||||
point in this design:
|
||||
|
||||
1. a reading past `fundamental_staleness_days` would corroborate every Warning
|
||||
crossing indefinitely — the strongest claim this channel makes, from the data
|
||||
with the least right to make it;
|
||||
2. an LLM run that failed to extract anything produces a perfectly fresh
|
||||
observation that knows nothing. Counting it as exposure means repeated
|
||||
extraction failures slowly accumulate coverage until the fundamental rows flip
|
||||
to a *measurable* 0/8 — a failed result published for a channel that never saw
|
||||
a thing, which is precisely what coverage-matching exists to prevent.
|
||||
|
||||
**Pre-rename snapshots are adapted, not discarded.** The channel was stored as
|
||||
`fundamental_overlay` until 2026-08-12. The rename shipped without a methodology
|
||||
bump — no score changed — so those rows are still served and were never reseeded.
|
||||
Reading only the new key would have turned every one of them into `unknown`,
|
||||
silently dropping real recorded evidence: historical Path colours, and exposure
|
||||
the event study can legitimately count. `_parse_snapshot` derives the channel
|
||||
from a legacy overlay's own stored facts (its capex map supplies the basket, so
|
||||
the derivation uses the names observed at the time rather than today's config).
|
||||
Normalising there rather than at each call site means no reader can receive an
|
||||
un-adapted row. Delete only after a reseed has rewritten the whole window.
|
||||
|
||||
**The blob and the series row are one transaction.** They are the same
|
||||
observation seen by the live card and by the point-in-time replay; committing
|
||||
them separately leaves a window where a failure publishes one and not the other,
|
||||
and the two then disagree permanently with nothing to detect it. Both writers use
|
||||
`settings_store.upsert_setting` (which does not commit) plus a single commit;
|
||||
`record_fundamental_observation` deliberately takes no commit of its own so
|
||||
`update_regime_monitor` keeps its own transaction boundary.
|
||||
|
||||
Each snapshot stores the fixed basket symbols, hash, and freeze date.
|
||||
Reconstructed history before that freeze date is retrospective/exploratory.
|
||||
@@ -321,39 +533,185 @@ today's number. The quadrant dividers rendered in Path view come from
|
||||
|
||||
## Warning study
|
||||
|
||||
The study calls the outcome a **10% correction**, not a regime break. The first
|
||||
70% of sessions freezes the 80th-percentile warning threshold; alarm episodes are
|
||||
measured on the final 30%. Because v3 dropped fundamentals from the score, the
|
||||
study now measures exactly the live Warning score rather than a technical-only
|
||||
approximation of it, and both are computed from one shared sensor definition
|
||||
(`warning_sensor_scores`) so they cannot drift apart.
|
||||
The study calls the outcome a **10% correction**, not a regime break. It measures
|
||||
two rules against that outcome, plus enough context to tell whether either number
|
||||
is any good.
|
||||
|
||||
A cached report is discarded when its methodology no longer matches, so the panel
|
||||
reverts to "not run yet" after a bump rather than showing stale numbers. **Re-run
|
||||
the Event Study job after cutting over to v4.**
|
||||
A cached report is discarded when its methodology no longer matches *or* when
|
||||
`STUDY_SCHEMA` moves, so the panel reverts to "not run yet" rather than showing
|
||||
stale numbers or a report missing half its blocks. **Re-run the Event Study job
|
||||
after a methodology cutover or a schema bump.**
|
||||
|
||||
### The headline is the rule that actually fires
|
||||
|
||||
Until 2026-08-12 the study measured a bare rising-edge crossing of an
|
||||
80th-percentile threshold fitted on the first 70% of sessions. **Nothing consumes
|
||||
that rule.** What reaches Telegram is `_collect_regime_quadrant`: a quadrant
|
||||
change with State ≥ 50 and Warning ≥ 40 as fixed dividers, a ±5 hysteresis
|
||||
deadband, a two-session confirmation, a 3-day cooldown, and a 75% coverage gate
|
||||
on both axes. The two differ on every one of those axes, including the threshold
|
||||
itself (a fitted ~32 against a shipped 40).
|
||||
|
||||
`replay_quadrant_changes` replays the shipped state machine over the whole
|
||||
sample. Three details are reproduced rather than cleaned up, because a state
|
||||
machine written from first principles gets each of them wrong:
|
||||
|
||||
- the prior session is classified against the **current baseline**, not against
|
||||
its own predecessor, so confirmation asks "did yesterday already look like this
|
||||
change" rather than "did yesterday change too";
|
||||
- the baseline advances only when an alert actually fires, so a change blocked by
|
||||
confirmation or cooldown is re-evaluated against the old quadrant next session;
|
||||
- one cooldown is shared by every quadrant change, so a 3→4 alert can swallow a
|
||||
4→2 alert three days later.
|
||||
|
||||
Two consequences worth stating. The alarm is dated at the **confirmation**, not
|
||||
at the first crossing, which costs one session of lead by construction. And the
|
||||
rule alerts on changes in both directions, so the replay's exits are recorded but
|
||||
filtered out by `entry_alarms` — only entering a Warning-high quadrant is a
|
||||
warning about anything.
|
||||
|
||||
The replay reuses `_compute_index` rather than re-deriving the axes. That is the
|
||||
same anti-drift argument that produced `warning_sensor_scores`: the v2 study
|
||||
re-derived Warning by hand and would have kept measuring the old construct
|
||||
through a scoring change. State has no equivalent shared helper, so the snapshot
|
||||
builder itself is the shared definition.
|
||||
|
||||
**Nothing is fitted, so nothing needs protecting from a training set.** There is
|
||||
no split, and every detected correction is evaluable instead of the four that
|
||||
happen to land in the last 30%. The `underpowered` and "threshold frozen on a
|
||||
different construct" caveats do not apply to this variant.
|
||||
|
||||
### Reading the result
|
||||
|
||||
The report carries a `reliability` block and the UI renders its warnings, because
|
||||
the headline numbers invite over-reading in two specific ways.
|
||||
A bare "2 of 4" is unreadable in either direction, so the report scores four more
|
||||
rules through the same `evaluate_alarms` harness over the same events and
|
||||
sessions, and adds a null. All use fixed thresholds — a threshold fitted on the
|
||||
full sample would have lookahead the shipped rule does not, and one fitted on a
|
||||
split could only be scored on the holdout events.
|
||||
|
||||
| kind | rules | the question |
|
||||
|---|---|---|
|
||||
| ablation | Warning ≥ 40 bare, State ≥ 50 bare | does the quadrant machinery earn its place? |
|
||||
| baseline | leader below its 50-DMA, VIX ≥ 20 | does the score earn its complexity? |
|
||||
| null | K random alarms at the observed firing rate | is any of this better than chance? |
|
||||
|
||||
The two kinds must not be read as one list. If a baseline matches the score, the
|
||||
composite is not earning its complexity and that is the finding — it does not
|
||||
mean the monitor is worthless, since State and Warning exist to be *read*, but it
|
||||
caps how much further calibration is justified. If the bare Warning crossing
|
||||
beats the shipped rule, the machinery (not the sensor) is what is costing recall.
|
||||
|
||||
The null draws only from sessions a rule could actually have fired on. Over the
|
||||
whole sample it would be diluted by warm-up sessions and would understate what
|
||||
chance achieves — which matters, because with ~11 events and a 20-session horizon
|
||||
roughly a sixth of the sample already sits inside a hit window. It is seeded, so
|
||||
a re-run cannot move the report. Corrections cluster and uniform placement does
|
||||
not, so it is the **floor, not the bar**: an alarm process that clustered would
|
||||
beat it for reasons unrelated to foresight.
|
||||
|
||||
### First result (2026-08-12): the shipped rule is not distinguishable from chance
|
||||
|
||||
Replayed over 2021-07-14 → 2026-08-12. The 200-DMA warm-up means the baseline
|
||||
only seeds on 2022-05-26, so 1056 of 1276 sessions are evaluable and 10 of the 11
|
||||
detected corrections fall inside them.
|
||||
|
||||
| rule | kind | warned | FA/yr | median lead |
|
||||
|---|---|---|---|---|
|
||||
| **Quadrant alert (shipped)** | | **1/10** | **0.9** | 19d |
|
||||
| Quadrant alert, both axes high | ablation | 0/10 | 0.9 | — |
|
||||
| Warning ≥ 40, bare crossing | ablation | 3/10 | 4.8 | 20d |
|
||||
| State ≥ 50, bare crossing | ablation | 0/10 | 0.7 | — |
|
||||
| SMH below its 50-DMA | baseline | 7/10 | 6.7 | 8d |
|
||||
| VIX ≥ 20 | baseline | 4/10 | 7.2 | 9.5d |
|
||||
| Random alarms, same firing rate | null | 0.9 ± 0.8 | — | — |
|
||||
|
||||
**P(chance ≥ 1/10) = 0.65.** Alarms scattered at random over the same sessions at
|
||||
the rule's own firing rate match or beat it two times in three. Whatever the
|
||||
score knows, this rule is not transmitting it.
|
||||
|
||||
Three readings, in order of how much they should change:
|
||||
|
||||
**The machinery costs more than it protects.** The bare Warning crossing catches
|
||||
3 with a 20-session lead; wrapping it in the quadrant rule drops that to 1. The
|
||||
State condition is the largest single cost — requiring both axes high catches
|
||||
nothing at all, which is what a coincident axis gating a leading one predicts.
|
||||
Hysteresis, the two-session confirmation and the shared cooldown between them
|
||||
take the rest, and the cooldown is shared across *every* quadrant change, so
|
||||
exits consume the budget that entries need. Only 5 of the 15 replayed changes are
|
||||
Warning-high entries.
|
||||
|
||||
**The crude baselines beat everything on recall, at a price.** SMH below its
|
||||
50-DMA catches 7 of 10 — but at 6.7 false alarms a year against the shipped
|
||||
rule's 0.9. That is a 7× recall improvement for 7× the noise, so it is not a
|
||||
clean dominance and this table cannot settle it; the missing axis is what a false
|
||||
alarm actually costs, which nothing here measures. What it does settle is that
|
||||
the composite is not buying recall the 50-DMA does not already have.
|
||||
|
||||
**The 0.9 false alarms/year is not the achievement it looks like.** A rule that
|
||||
almost never fires has few false alarms by construction. Read the two columns
|
||||
together or not at all.
|
||||
|
||||
Recorded from an offline replay (live Alpaca + FRED, no database, breadth
|
||||
computed from the same Alpaca closes rather than the stored universe). The job in
|
||||
Admin → Jobs is the canonical path and reads breadth from the DB, so re-run it to
|
||||
confirm these figures before treating them as the record.
|
||||
|
||||
**This is a verdict on the market channels only.** The fundamental and confluence
|
||||
rows in the same table are marked `measurable: false` and print "not measurable"
|
||||
rather than a ratio: with an empty observation series they never fire, and a 0/10
|
||||
sitting in a comparison column would read as tested-and-failed. `false` here means
|
||||
the input does not exist yet, not that the rule lost.
|
||||
|
||||
(The figures above were also produced under a briefly-built weighted modifier and
|
||||
came back bit-identical, which is what confirmed the modifier was inert over the
|
||||
whole window — the numbers depend on the technical sensors alone either way.)
|
||||
|
||||
**Not acted on.** Nothing in the alert path was changed on the strength of this.
|
||||
The obvious candidates — dropping the State condition from the entry test,
|
||||
separating the entry and exit cooldowns, or lowering the Warning divider — are
|
||||
threshold changes to a live alerting rule and want their own decision.
|
||||
|
||||
### The coverage gap relocates, it does not close
|
||||
|
||||
Dropping the fitted threshold makes the whole sample evaluable, but most of the
|
||||
extra events predate 2023-08. W3 does not exist there, so Warning renormalises to
|
||||
`(W1×45 + W2×30)/75` and the fixed 40 divider is applied to a different construct
|
||||
than it was reasoned about. The report therefore splits shipped-rule metrics at
|
||||
the credit sensor's first session and the panel states both, because replacing
|
||||
one misleading headline with a differently misleading one would be no gain.
|
||||
|
||||
Convenient side effect: the pre-credit era *is* the "Warning without W3"
|
||||
ablation, measured on real sessions rather than simulated ones, so that ablation
|
||||
is not run separately.
|
||||
|
||||
Alarms and events are assigned to eras by index, so an alarm days before the
|
||||
boundary matching an event days after it lands in the earlier era. With the eras
|
||||
years long and the events sparse, that costs nothing.
|
||||
|
||||
### The fitted variant, kept for continuity
|
||||
|
||||
The 70/30 percentile study is still computed and still reported, collapsed, with
|
||||
its `reliability` block intact — it is a genuinely different question, and it is
|
||||
what earlier revisions of this document report. Its caveats stand:
|
||||
|
||||
**The holdout is thin.** The study detects 11 corrections across 5 years but the
|
||||
70/30 split leaves only 4 in the test period. Recall is therefore one event away
|
||||
from a materially different headline, and in practice the event that flips is
|
||||
decided by where the frozen threshold happens to land rather than by whether the
|
||||
score saw anything. The v3 cutover run illustrates it: v3 scored 2/4 against v2's
|
||||
3/4, but "v3 without the credit sensor" scores 3/4 at a *higher* threshold
|
||||
(35.5) than shipped v3 misses it at (32.3) — because the alarm rule needs a
|
||||
rising edge, and a lower threshold can mean the alarm already fired outside the
|
||||
20-session horizon and never reset below. Below `MIN_EVENTS_FOR_CONFIDENCE`
|
||||
holdout events the report says so explicitly.
|
||||
70/30 split leaves only 4 in the test period. Recall is one event away from a
|
||||
materially different headline, and in practice the event that flips is decided by
|
||||
where the frozen threshold happens to land rather than by whether the score saw
|
||||
anything. The v3 cutover run illustrates it: v3 scored 2/4 against v2's 3/4, but
|
||||
"v3 without the credit sensor" scores 3/4 at a *higher* threshold (35.5) than
|
||||
shipped v3 misses it at (32.3) — because the alarm rule needs a rising edge, and a
|
||||
lower threshold can mean the alarm already fired outside the 20-session horizon
|
||||
and never reset below. Below `MIN_EVENTS_FOR_CONFIDENCE` holdout events the
|
||||
report says so explicitly.
|
||||
|
||||
Some events carry no information at all for comparison: in that run every
|
||||
variant caught 2026-03-06, every variant missed 2026-06-05, and every variant
|
||||
"caught" 2025-11-20 with a 1-session lead, which is coincident rather than a
|
||||
warning.
|
||||
warning. The headline recall does not currently discount those; a minimum-lead
|
||||
rule is the obvious next change and has not been made.
|
||||
|
||||
**Sensor coverage can straddle the split.** The score renormalises over available
|
||||
**Sensor coverage straddles the split.** The score renormalises over available
|
||||
sensors, so a training window predating a sensor's history freezes the threshold
|
||||
on a different construct than the holdout is measured against. At the v3 cutover
|
||||
only 39% of training sessions had all three Warning sensors versus 100% of the
|
||||
@@ -362,9 +720,22 @@ test period, because credit history begins 2023-07-25.
|
||||
Restricting the threshold to sensor-matched training sessions was tried and is
|
||||
*not* the fix: those sessions are a calm recent stretch, so the threshold drops
|
||||
from 32.3 to 22.5 and false alarms rise from 3.3 to 8.6 per year. It trades a
|
||||
coverage bias for a regime-selection bias. The honest position is that the
|
||||
threshold is hypersensitive to window choice at this sample size; the report
|
||||
states its limits rather than pretending to a precision it does not have.
|
||||
coverage bias for a regime-selection bias. The honest position is that a fitted
|
||||
threshold is hypersensitive to window choice at this sample size — which is the
|
||||
strongest argument for making the unfitted shipped rule the headline.
|
||||
|
||||
### Considered and not done
|
||||
|
||||
**An ETF credit proxy (HYG/IEF) to extend W3 back over the whole sample.** It
|
||||
would trade "two sensors versus three" for "proxy sensor versus real sensor" —
|
||||
still a construct straddle, but no longer flagged by the coverage split. This is
|
||||
the same objection that rejected `BAA10Y` as a percentile reference. If ever
|
||||
revisited, check the impulse correlation on the three years of real-OAS overlap
|
||||
first and report it as a sensitivity, never as the headline.
|
||||
|
||||
**A depth sweep (5%/7%/15% corrections) for more events.** `EVENT_COOLDOWN_DAYS`
|
||||
is 40, so at shallower thresholds re-triggers inside a single decline merge or
|
||||
drop and the denominator moves for cooldown reasons rather than market ones.
|
||||
|
||||
## Resolved in v4 (raised 2026-08-07, shipped 2026-08-08)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user