The Warning study measured a fitted percentile crossing that nothing consumes.
What reaches Telegram is a quadrant change: fixed 50/40 dividers, hysteresis,
two-session confirmation, 3-day cooldown. Those thresholds are constants, not
fits, so there is no training set to protect and all 11 detected corrections are
evaluable instead of the 4 that fell in a holdout.
Replaying it: 1/10 corrections, 0.9 false alarms/year. Random alarms at the same
firing rate match or beat that in 65% of draws. The panel now carries ablations
(does the quadrant machinery earn its place?), external baselines (does the score
earn its complexity?), and that null, because a bare "2 of 4" was unreadable in
either direction. Nothing in the alert path was retuned on the strength of it.
Fundamentals become a third channel rather than a term in either score. v3 cut
them arguing 12+8 of 100 points "could not change any published conclusion" --
true only when every technical sensor reads zero; weighted they moved the bar for
the 40 divider from 40 to 25. But no fusion weight is measurable either: with ~10
events and no fundamental history, any weight is a policy preference presented as
a measurement. So the read is a categorical state (supportive/neutral/adverse/
unknown) with an evidence grade, derived by fixed rules from stored facts, read
by confluence. The LLM extracts and explains; it does not score.
Absence stays absence throughout. `unknown` is unreachable by averaging, a stale
or empty observation may display but never confirm, extraction failures map to
`unknown` rather than `mixed`, and the study rows are coverage-matched and marked
not-measurable until enough corrections are covered -- otherwise a fortnight of
observations renders as 0/10 and reads as a failed test.
Observations become a real time series (migration 033); they lived in a single
overwritten settings slot, so no history existed to replay. Pre-rename snapshots
are adapted rather than discarded. METHODOLOGY stays v4 -- no score changed --
so no reseed; STUDY_SCHEMA moves to 3 and discards the cached report.
Post-deploy: re-run Event Study from Admin -> Jobs. The panel reads "not run yet"
until then.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two review findings, plus a lost-edit repair.
v2_reconstruction is now a required variant. It carries every published figure
the reproduction rests on (avg, p80, max, P3-pegged, W1-live), so a run without
it could emit a confident, non-provisional recommendation having checked nothing
against v2 at all -- while the methodology doc claims v2 and v3 are reproduced
first. The default invocation is now derived from REQUIRED_VARIANTS so the two
cannot drift, and a test asserts the default satisfies its own requirement.
The doc and the P1_TREND_BREAK_ANCHORS comment still justified skipping the
P1_SCORE_CAP with 17/408 = 4.2%, which is the all-session share and does not
evaluate the rule. The rule names sessions with State >= 40: 47 of them, P1 sole
argmax on 17 = 36.2%, against P2's 16 and P3's 14. Conclusion unchanged -- well
under the 80% trigger -- but the published rationale now states the metric that
actually decided it.
Root cause of that survival: the earlier correction WAS made, but in a script
that applied several substitutions and wrote the file once at the end. A later
substitution raised, so the successful edits were discarded with it. The
"Unlike P3 and V1 ... P3's do not" fix was lost the same way and is restored.
Also adds tests for the refusal paths themselves -- missing required variant,
unknown variant, custom window with no calendar anchor. They were verified by
hand last round but left unpinned, which is the same shape of problem as the
optional gates they exist to enforce. All return before any network call.
Deliberately not done, as not load-bearing: recording the oas400 variant's
missing-credit session count (the truncation conclusion rests on the
distribution mismatch, which is already recorded), and generalising
_pipeline_gates for arbitrary --end/--sessions windows.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Review of the v4 evidence path. The shipped sensors, bands, methodology bump and
categorical allowlist were found sound; these are gaps in the harness that
produced the evidence for them.
The recommendation gates were optional, so they were not gates. The calendar
anchor lived behind --expected-first-session, which defaulted to None -- so the
committed artifact had no first-date check at all, leaving only a session COUNT
that is tautological (the harness slices the tail of the price series to whatever
was asked for). And the state_v4 <= state_v3 invariant was appended only when
both variants were present, so `--methodology v3` alone could still emit a v4
recommendation having never evaluated v4. The anchor is now a published constant
asserted unconditionally, required explicitly whenever --end/--sessions are
overridden, and v3+v4 are mandatory. Both refusals exit 2.
The P1_SCORE_CAP decision was taken on the wrong population. The agreed rule was
"sole price argmax on >80% of sessions with State >= 40"; the harness reported
only all-session counts and the doc concluded from 17/408 = 4.2%. Measured on the
actual population: 47 qualifying sessions, P1 sole argmax on 17 = **36.2%** (P2
16, P3 14). Still well under 80, so the conclusion holds -- but it was reached
from a denominator that did not test the rule, and 36.2% is a materially
different number to have on the page.
Provenance did not identify the code that produced the artifact. It recorded
git_rev c3ae5ad while the live v4 variant depended on app changes that were still
uncommitted, so checking out that revision would not reproduce it. Now records
git_dirty plus sha256 of regime_monitor_service, breadth_service and the script
itself, and this artifact is regenerated from a clean tree.
The 400- vs 700-day OAS question was described as settled but was not
reproducible: the artifact carried only oas_fetch_days 4748, and
v2_reconstruction patches the per-session window to 3653 regardless, so
--oas-window-days 400 could not simulate it. Patching a window cannot stand in
for data that was simply absent, so v2_reconstruction_oas400 truncates the OAS
SOURCE series instead: avg 26.54, p80 42.52, max 100.00 against published
22.6 / 35.1 / 91.2. Full coverage reproduces all three, so the published figures
predate the truncation. Now recorded in the doc.
v4-vix-only and v4-p1-only had become no-ops: after the cutover the shipped
sensors ARE v4, so patching one candidate in left the other shipped and both
variants evaluated full v4. Each now restores the other sensor to its v3 formula,
and they separate properly (v3 18.13, v4-vix-only 16.64, v4-p1-only 16.28,
v4 14.78 -- each fix contributing about half the move).
Docs: the copy-paste invocation was mangled by a backslash-escaping bug and is
now a fenced, forward-slash command; "Unlike P3 and V1 ... P3's do not" corrected
to "Unlike P1 and V1"; the point-in-time section updated from 400 sessions to the
672-calendar-day / ~464-session window production actually replays; the exercised
52.33 VIX print recorded so the top anchors are not merely asserted.
Tests: band_for now pinned at 64.9/65 from both sides so a silent revert to 80
cannot pass, and the categorical carry-forward test stores locked=True and
asserts it survives -- losing it is half the failure mode, since
update_regime_monitor only auto-refreshes when locked is false.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two sensors saturated in exactly the range where resolution matters, and the top
State band had no headroom. Calibrated with scripts/run_regime_monitor_calibration.py
over the 408 sessions ending 2026-07-24; the shipped code reproduces that run's
band shares exactly (78.9 / 13.0 / 4.7 / 3.4).
V1 read VIX 30, 50 and 82 as an identical 100 — the same defect v3 had just
removed from P3, left in place one sensor over. In the window it flattened five
distinct April-2025 prints (52.33, 46.98, 45.31, 40.72, 38.57) into one value.
Now an anchor table reaching full scale at 55, not at 2020's ~82: anchoring the
top at a once-in-a-generation print would make VIX 50 read only ~70. Pegged on
14 of 408 sessions before; none now.
_under_200 returned a bare 0/100, so P1 printed 100 the moment SMH and QQQ were
both under their average — and since the price pillar takes max(P1, P2, P3),
that pinned the pillar and stopped P3's ladder resolving for the whole of a
selloff. Now graded by depth below the 200-DMA, with a deliberate floor of 20 at
the crossing: the break is a genuine binary event, only its depth is graded.
Pegged on 46 of 408 sessions before; none now. A 2% break reads ~30, not 100.
max() was KEPT — the defect was the step function feeding it, not the vote, and
v3's "one capped vote for correlated reads" rationale still holds. P1 is the sole
price argmax on 17 of 408 sessions (4.2%), so the P1_SCORE_CAP fallback drafted
during design was measured as unnecessary and not shipped.
STATE_BANDS breaking 80 -> 65, and only that threshold. Credit returns 0.0 (not
None) when calm, so it holds its 20 points pinned at zero and price + breadth +
volatility at literal maximum summed to exactly 80.0 — v3's threshold to the
decimal, with nothing above it. The sensor is deliberately unchanged: a
calm-credit selloff genuinely is less stressed. What was stale is the band, fit
on v2 while credit's since-removed percentile leg still contributed. A
2022-style AI/tech drawdown with calm credit computes to 70.3 (no death cross) or
74.0 (with one); 70 would have left 0.33 points of headroom, reproducing the
defect. Chosen by scenario arithmetic, and the realized breaking share then lands
on 3.4% — the same as v3's, arrived at independently.
"v4" added to CATEGORICAL_FUNDAMENTAL_METHODOLOGIES in this same commit, which is
load-bearing: that set is checked against the STORED blob, so bumping without it
discards the collected observation on first write, leaving fetched_at null and
locked false — and update_regime_monitor then fires a paid LLM refresh on every
run, forever. Now guarded by a test parametrised over v2 and v3 stored blobs.
SENSOR_REVISION deliberately stays 2: a METHODOLOGY change already forces a full
reseed via _parse_snapshot, and bumping both would imply the reseed was
revision-driven.
QUADRANT_STATE_DIVIDER stays 50 because only breaking moved, so alert_service,
RegimeChart and the quadrant tests need no change. A new test enforces
divider == band boundary on both axes, which nothing did before.
Doc renamed to regime-monitor-v4.md with a tombstone at the old path (commit
messages cite it), the three open questions converted to resolved with the
reasoning that closed them, and indexed in docs/research/README.md for the first
time. The P2 limit is stated honestly: _death_cross pegs at a -5% MA gap, so a
deep selloff still reaches 100 via P2 — v4 repairs the shallow-to-moderate break,
not "the price pillar no longer pegs".
DEPLOY: the first run reseeds ~464 sessions. Expect one phantom quadrant alert
(the dedup key carries basket_hash, not methodology) and re-run the Event Study
manually — its cached report self-invalidates but does not self-regenerate.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
v3 was calibrated by replaying the series offline, but that harness was never
committed -- so its published numbers could not be re-derived, and a v4 cut would
have had to choose anchors by argument rather than measurement. This is that
harness, and it reproduces the published figures.
scripts/run_regime_monitor_calibration.py replays State/Warning session by
session from the same inputs the live job uses (Alpaca for all 33 symbols, FRED
for VIX and HY OAS), with no database: breadth and divergence come from
breadth_service's pure helpers. It never reimplements an unchanged live sensor --
_compute_index, _score_pillars, P2, P4 and the Warning sensors are imported and
called. Only candidate formulas (proposed v4) and retired ones (v2, gone from the
codebase) are defined here and patched onto the module for a variant's duration.
Reproduction of the 408 sessions ending 2026-07-24, against the figures in
docs/research/regime-monitor-v3.md:
v2 State avg 22.6 -> 22.68
v2 State p80 35.1 -> 35.1 exact
v2 State max 91.2 -> 91.2 exact
v2 P3 pegged 39 -> 39 exact
v2 W1 live 108 -> 108 exact
v3 State max 87.4 -> 87.4 exact
v3 band shares 73.3/15.0/8.3/3.4 -> 73.0/15.4/8.1/3.4
Three things the harness had to get right to reach that, each of which was
initially wrong and caught by a gate rather than by inspection:
- "W1 live 108" counts NONZERO sessions, not non-null ones. v2's divergence
gate returned 0.0 during any decline (v3 tapers instead), so the retired
divergence formula had to be reconstructed too.
- v2 sliced HY_OAS_REFERENCE_YEARS = 10.0 per session, not v3's 700 days. The
percentile leg ranks against that window, so replaying it short shifted the
middle of the distribution while leaving the max exact.
- The published v2 numbers correspond to FULL OAS coverage. Replaying v2 with
the 400-calendar-day fetch it shipped with yields max 100.0 and 133
credit-less sessions -- so that truncation was not in force when the figures
were taken. Recorded rather than assumed.
The script refuses to emit a band recommendation unless every hard gate passes
(33 symbols fetched, per-symbol warm-up and final bar, full basket on every
session, calendar anchors, 100% coverage, and a row-wise state_v4 <= state_v3
invariant), and exits non-zero. It is meant to be structurally impossible to read
a calibration result out of a run whose pipeline did not validate. No v4 code
ships in this commit.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CI only lints app/, so 11 findings had accumulated in tests/ and scripts/.
Mechanical and behaviour-neutral, but two were not auto-fixable and needed a
judgement call rather than `ruff --fix`:
- E741 in run_fip_breadth_diagnostics: `l` is the OHLCV low and is genuinely
used, so this was a naming fix (`l` -> `lo`), not a deletion.
- F841 in the same file: `vol_ix`/`momr_ix` are assigned from a pure local
`_index()` and never read, so removing them cannot change any output. Their
upstream `vol_weeks`/`momr_weeks` maps *are* used further down and stay; the
comment above `_index` was corrected to say so.
The rest are unused imports and f-strings without placeholders (literal
markdown table headers, so identical output).
Verified beyond the linter, since py_compile does not catch a removed-but-used
import: every removed symbol has zero remaining references, all scripts compile,
and the full unit suite passes (852 passed, 1 skipped).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The A5 cutover has been on and observed in production, so SEC Company Facts +
DoltHub earnings are already the live source for `fundamental_data`. This
removes everything the legacy path still occupied.
Gone: the three providers and their config/env keys; the weekly
`fundamental_collector` job; the cutover toggle (SEC + Dolt is now the
unconditional path, so `off` can no longer silently freeze scoring inputs); the
A5 parity report, whose deltas became structurally zero once the candidate
builder started writing the table it compared against; and the FMP tier of
universe bootstrap.
Two behavioral notes:
- Disabling **SEC Fundamentals Import** now stops the SEC network fetch only.
The local cache refresh moved outside the job-enable check, because candidates
also derive from daily closes and earnings events — freezing those on an
ingestion pause would stale scoring with no fallback left to recover from.
- `/ingestion/fetch?sources=fundamentals` still accepts the key and reports
`skipped`; there is no per-ticker fetch any more.
Migration 029 does not blanket-delete the leftover settings rows. Migrations run
before the service restart, and pre-A6 code reads an absent `job_*_enabled` row
as *enabled* — so the two behavior-bearing keys become tombstones pinned to safe
values (hidden in Admin) and only the inert three are deleted. Removing the
provider keys from the production `.env` is the matching rollout step.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Brings the durable artifacts of research/portfolio-capacity-rebalancing onto
main so the rationale for raising the count cap lives with the code that cites
it. The matrix runner, the research simulator hooks and the study's unit tests
are deliberately left behind; they remain at tag research/portfolio-capacity-final.
Corrects conclusions that were reached on EV per trade and are now superseded:
the findings doc's decisions 1 (keep cap 10) and 4 (run the risk-floor A/B) are
struck through and answered in a new correction section, and the research README
and phase-A matrix entries are updated to match. The frozen specification itself
is untouched -- its recorded SHA-256 f1e37783 still verifies.
effective-risk-floor-ab.md is retained but marked CLOSED/NEGATIVE: the study it
proposes is already answered by cap15 vs cash_unbounded (-0.753pp CAGR while
EV/trade rises), and its EV-based pass rule would have shipped it.
scripts/research_rankings.py replaces a fourth copy of the historical rank-map
helper; run_research_matrix, run_execution_recovery_matrix and
run_daily_reentry_matrix now share it. The shared version adds a duplicate
observation guard and a deterministic symbol tie-break the copies lacked.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Operational plumbing to land the parser fixes and to make silent resolution
failures visible.
- Reparse: SecFundamentalsImporter(reparse=True) restages every accession with
the current parser and rewrites the ones that now reconstruct differently,
writing the full column set so a row is never half old-parse. Snapshots stay
immutable with respect to SEC; the stored row is our reconstruction, and after
a parser fix keeping it is a stale cache, not history. run_import(force=True)
bypasses the unchanged-revision no-op, since the staleness is on our side, not
the source's. Exposed as scripts/reparse_fundamentals.py, dry-run by default.
- CIK overrides: sec_universe reads a {symbol: cik} pin from
SystemSetting['sec_cik_overrides'], applied ahead of company_tickers.json, for
when SEC maps a ticker to a successor shell with no filings (XOM -> a zero-
filing "ExxonMobil Holdings Corp" while every 10-K/Q is under CIK 34088).
- Resolution validation: a tracked issuer resolving to a registrant with no XBRL
filings now records no_xbrl_filings and raises a warning naming the CIKs and
the override setting, instead of silently yielding nothing on every run.
- _diff_fields compares datetime instants, not representations: accepted_at
round-trips naive from SQLite but tz-aware from Postgres, which otherwise made
a reparse of identical data report every row as changed (and false-positived
the pre-existing discrepancy warning).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Earnings backfill sourced from the public DoltHub earnings repo at a
pinned commit rather than the FMP API: reproducible for anyone re-running
the study, and it burns no request quota. 12,414 events, 98.6% of symbols
with >=8 announcements, 99.2% paired actual/estimate, no keyed duplicates.
2a earnings-gap diagnostic: INFORMATIONAL, no filter shipped. The
pre-earnings cohort's right tail was better, so the registered
avoid-earnings condition failed. Note the raw 23/266 vs 115/574 incidence
gap is largely a duration confound -- severe losses stop out fast and have
less time to span an announcement -- so it is not evidence that holding
through earnings is safe.
2b SUE: FAIL against the pre-registered +0.03 bar (unconditional IC
+0.0151 over 56 reliable windows, momentum-conditional +0.0213). Signs
stable across eras, so this is a clean null rather than an ambiguous one,
consistent with post-earnings drift having decayed in large caps.
Closes the Tier-1 arc: Task 1 dead on deep evidence, Task 2 dead here,
Task 3 complete as diagnostic. No in-sample research thread remains open.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Drop intermediate history-depth reports, sector-residual runners/map/code hooks
(evidence stays in final reports + docs), and slim MacBook helper to ssl/earnings/
prod-book-matrix only. SSL bootstrap and archived research conclusions retained.
Pre-register A-D (4y/2016 x 505/505+liquid) with unchanged production knobs.
Runner caches full GTL candidates then re-ranks per arm; MacBook entry via
run_tier1_macbook.sh --prod-book-matrix.
SANITY-FAIL report showed megacaps/ETFs already at empirical 2016-01-04 floor
(2649 bars) after deepen; check wrongly required ~2013. Pass when megacaps leave
the old 2021 two-tier floor and match SPY; XLC listing exception retained.
Terminal follow-up for Task 1: detect/refetch shallow two-tier symbols and sector
ETFs at 5000d, regenerate manifest, run ONE liquid-1500 harness with era split, grade
mom_12_1_sector_resid mechanically. Bundled as run_tier1_macbook.sh --sector-resid-deep.
Extract app/ssl_bootstrap.py (shared with FastAPI main), wire it into research
scripts, and teach run_tier1_macbook.sh to locate combined-ca-bundle.pem, certifi,
optional USE_CORP_PROXY, plus --ssl-check diagnostics.
Tier-1 alpha research (local only, no production deploy):
Sector residual momentum: two-factor SPY+sector residual and sector demean signals, IC harness + A/B. Sector resid clears pre-registered bars narrowly (PROMOTE for human wire design only). Sector demean fails t vs market resid.
Earnings: earnings_events backfill (FMP bulk paid; FMP/AV per-symbol), 2a gap diagnostic report-only, 2b SUE IC (PARK; incomplete 48/506 coverage).
History-depth: pre-registered doc + runner for MacBook deep rebuild/harness.
Do not ship production residual or filters from this branch.
Log the 21:14 orphan as a snapshot-build race, rewrite the context table to
authoritative ICs only, and soften the vol-tilt warning. Add extender completion
manifest + breadth refuse guard; strip intermediate/orphaned reports; park the
thread (no book sim, no deploy).
Add lagged/tier/prod-subset/mom-conditional checks on research.sqlite.
Log: unconditional sign is a winner/bleeder tug-of-war; mom-conditional
fip stays negative and reliable; warn on high-vol tilt if universe broadens.
Add research-only snapshot extender, PIT dollar-volume mask for signal IC,
rank-only harness path, fingerprint+breadth runner, and docs. Fingerprint
reproduced IC -0.045 / t -2.91 on prod.sqlite. No production gate/schedule changes.
Document Phase A (max-hold/vol/corr closed; next-open as decision baseline).
Add stale_close and next_open gap-cap fill modes plus a small matrix to test
whether near-close scheduling recovers overnight momentum drift.
Ship shared Sharpe SE/PSR diagnostics, next-open fill and equity-curve vol targeting in the portfolio simulator, re-derived fip_id, and a checkpointed offline matrix runner for Mac-side validation sweeps.
Finnhub profile2 reports marketCapitalization in millions; storing it
as dollars made mega-caps like SPCX show as micro (e.g. 1.8M). Normalize
on ingest, add unit tests, and include a one-shot SQL backfill script.
dennisthiessenandClaude Opus 5