Two ways the recommendation still disagreed with the page it sits on.
It preferred the "all" monitor row while the UI defaulted its selector to "3y",
so a default page load showed one set of returns in the tiles and a different
set in the recommendation. The row it used is now published as basis_lookback
and the page defaults to it, so the two cannot open on different windows. The
test fixture gains a second monitor row with different numbers — with only an
"all" row present, a lookback mix-up could not fail.
Robustness picked its basis between "the recommended Nd hold" and "the S/R
target exit", naming an exit the production book replaced as recommended. There
is no ATR-trail ex-top-5% figure in the report, so it now always reports the
gate-level grading and says that is what it is, rather than dressing a legacy
number as a verdict on the production book. time_exit_sweep is no longer read
here at all.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The recommendation's "Book vs SPY" line read from portfolio_sim — the hold/target
policy book — while the tiles directly above read from portfolio_monitor, the
production ATR-trail book. Same SPY figure, different portfolio return, on one
screen. It now reads the same production row the tiles do.
Also removed, for the same reason: the "legacy exit diagnostic" comparing hold
against the S/R target. Both are exits the production book replaced, so a
recommendation between them could not lead to an action. And the fallback
headline, which advised the fixed-hold exit whenever a report had no production
row — a report that cannot describe the production baseline now states none.
portfolio_sim stays in the report payload: scripts/run_backtest_snapshot.py and
reports/compare_reports.py read it, and it is no longer surfaced in the UI. The
test fixture now carries a production monitor whose numbers differ from its
policy sim, so re-sourcing that line from the old place fails rather than passes
unnoticed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three portfolio metrics computed where their inputs already live in
_simulate_portfolio: Sortino off the existing daily return series, Gain-to-Pain
off a monthly aggregation of the equity curve, profit factor off closed-trade
dollar P&L.
Gain-to-Pain follows Schwager — sum of ALL monthly returns over the absolute
sum of the negative ones. The profit-factor-shaped variant,
sum(positive)/|sum(negative)|, sits exactly 1.0 higher for every input since
sum(all) = sum(pos) - |sum(neg)|; the test asserts against both so the wrong one
cannot pass. Sortino divides by len(rets), the full-sample lower partial moment,
not by the count of down days, which would shrink the denominator and inflate
the ratio.
No MAR field: calmar is already CAGR / max drawdown, the same number under the
other name (docs/research/effective-risk-floor-ab.md).
All three keys are emitted unconditionally even when None — the UI reads an
absent key as "report predates these metrics", so presence is a contract.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
active_only tested delisted_on IS NULL, so a symbol dropped out of signals the
moment a Form 25 was detected — ten days before Rule 12d2-2 makes the removal
effective, while it was demonstrably still trading. A manual future-dated mark
behaved the same way. It now compares against the database's own date, so a
pending delisting stays live until the day it takes effect.
That exposes a second problem the fix would otherwise create. Trading typically
stops before the ten-day delay expires, so across that window the symbol is
correctly active yet produces no bars — and confirm_delisting returned None for
an already-marked row, which would have fired the staleness warning daily for
ten days, the exact noise this flow exists to remove. It now reports the known
effective date on every path where the delisting is established, so the caller
warns only about gaps that are still unexplained.
CURRENT_DATE renders identically on postgres and sqlite, and the OR is
parenthesized when callers chain further where clauses.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Detection could retire an actively traded symbol — silently, since it then
vanishes from every signal. Three causes:
- Form 25 is filed per security class. An issuer removing its notes, preferred
or warrants files one while the common keeps trading. The filing's own
descriptionClassSecurity distinguishes them, so the primary document is now
fetched and read; anything not recognisably common equity is rejected, as is
anything unreadable (pre-2009 filings have no primary_doc.xml). Fail closed.
- Form 15 ends a reporting obligation and is no evidence trading stopped. The
whole family is dropped.
- A historical filing for a long-gone class could retire a symbol whose bars ran
years later, stamping the old date. Filings before the last bar (less a 30-day
lead for the exchange) are now ignored.
Rule 12d2-2 makes removal effective ten days after filing, so delisted_on is the
effective date rather than the filing date.
bootstrap_universe(prune_missing=True) still ran a cascading delete over
delisted rows, undoing the retention this branch exists for; it now skips them
and reports kept_delisted so the count is explicable.
clear_delisted had no route, which made "safe to automate because it is
reversible" false — reversal needed SQL. POST/DELETE /tickers/{symbol}/delisting
now mark and un-mark, giving an operator a non-destructive alternative to the
cascading DELETE that was the only option.
Shared-CIK siblings (GOOG/GOOGL) stay safe by construction: the probe is
per-symbol and gated on that symbol's own staleness, so a class that still
trades is never probed.
Not addressed: pruning a symbol merely dropped from the index still destroys its
history — the same survivorship problem in a different costume, needing a
tracked/membership state separate from delisting.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
mark_delisted returned early on any already-delisted row, so the sequence an
operator actually hits — mark EA by hand today, Form 25-NSE surfaces three days
later dated 2026-08-04 — left the estimated date and "manual" reason in place
permanently. Form 25 carries the real effective date, so it now replaces an
operator's estimate; a confirmed row is never downgraded or re-probed.
Also cover _get_ohlcv_priority_tickers, the one place active_only wraps a
compound select rather than a bare one — the unit suite reached none of it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Retiring a symbol meant delete_ticker or bootstrap_universe(prune_missing),
both of which cascade through OHLCV, setups and scores. That destroys exactly
the history four research documents already apologise for: today's tracked
universe projected backward is survivorship-biased, and hard-deleting every
delisted name is what causes it. Keeping the rows preserves the option to fix
that — it does not fix it, which needs the replay to model a delisting as an
exit event.
tickers gains delisted_on / delisted_reason (migration 032). NULL means
actively traded.
The filter is opt-in via ticker_service.active_only rather than folded into a
shared getter: the registry and admin views deliberately keep delisted rows so
the delisting is visible, and a silent default would undo that. Applied to the
live path only — scanner, momentum ranking, scoring, breadth, fundamentals
candidates, SEC universe, earnings import, ingestion loops. run_backtest keeps
them on purpose.
Detection runs off OHLCV staleness, not off the SEC fundamentals import: that
importer stalls for days on unrelated Company-Facts gaps and would take
detection down with it. On a stale symbol the scheduler asks SEC for a Form
25/25-NSE/15 and retires it only on a hit, so a halt or a rename (SATS->ECHO)
keeps the existing warning. The probe waits 3 stale days so a market-data
outage cannot turn into one SEC request per symbol per run.
Safe to automate because it is reversible: clear_delisted un-retires a false
positive, where a delete had already taken the history.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
MISSING_XBRL_RETRY_DAYS bounds how long ONE filing blocks promotion. It does
not bound the import as a whole, and the two come apart because a blocking
filing is only queued by promote(), which a deferred run never reaches. During
a rolling supply of unresolvable filings — earnings season, when SEC's
Company-Facts aggregation lags furthest — each new arrival restarts the 3-day
clock before the previous one clears, and nothing is written at all: not the
good rows, not the gap rows that would stop those filings blocking again.
Add an aggregate ceiling. Once promotions have been stale for
PROMOTION_CEILING_DAYS (7), every unresolved filing is aged past the retry
window in place, so promote() queues them all through the path that already
exists, source_max_date advances, and _missing() keeps queued rows aged-out on
later runs. The import self-heals instead of compounding.
Deliberately not the alternative of queueing gap rows on a deferred run: that
would drop the grace period to a single run for every filing, including the
common case of a Company-Facts lag that resolves in a day, and it needs a write
on a run that failed validation.
The per-filing window is untouched, a never-promoted source never trips (that
is initial setup, not a wedge), and affected symbols stay barred from setups
either way since setup_blocked_ciks ignores the window. A forced promotion
raises promotion_ceiling_forced so the safety valve is never silent.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The first pass guarded the replay loop but not the portfolio-simulation block,
which re-fetches price columns and loads the benchmark and the live exit policy
from the same session much later. A failure in any of those swallows the
exception without clearing the transaction — the identical failure mode, with
the identical symptom: the report write is the first unguarded statement and
takes the blame.
The outer handler is the backstop for the price_columns loop, which has no
handler of its own; rolling back a session an inner handler already cleared is
a no-op.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every DB call in run_backtest is best-effort so one unreadable ticker cannot
abort the whole replay, but the handlers swallowed the exception without
clearing the transaction. asyncpg then reports "current transaction is
aborted" for every later statement, and the first unguarded one — the report
write — surfaced it as the job error, long after the real cause.
Add _rollback_quietly at the three swallowing sites (benchmark load, parallel
fetch, sequential replay), matching the guard price_service already uses.
Load plain symbols instead of Ticker instances: a rollback expires ORM objects
held across it, and touching an expired attribute afterwards triggers sync
lazy-loading, which raises on an AsyncSession. rr_scanner_service hit this
same trap. Only .symbol was ever used.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two review findings, plus a lost-edit repair.
v2_reconstruction is now a required variant. It carries every published figure
the reproduction rests on (avg, p80, max, P3-pegged, W1-live), so a run without
it could emit a confident, non-provisional recommendation having checked nothing
against v2 at all -- while the methodology doc claims v2 and v3 are reproduced
first. The default invocation is now derived from REQUIRED_VARIANTS so the two
cannot drift, and a test asserts the default satisfies its own requirement.
The doc and the P1_TREND_BREAK_ANCHORS comment still justified skipping the
P1_SCORE_CAP with 17/408 = 4.2%, which is the all-session share and does not
evaluate the rule. The rule names sessions with State >= 40: 47 of them, P1 sole
argmax on 17 = 36.2%, against P2's 16 and P3's 14. Conclusion unchanged -- well
under the 80% trigger -- but the published rationale now states the metric that
actually decided it.
Root cause of that survival: the earlier correction WAS made, but in a script
that applied several substitutions and wrote the file once at the end. A later
substitution raised, so the successful edits were discarded with it. The
"Unlike P3 and V1 ... P3's do not" fix was lost the same way and is restored.
Also adds tests for the refusal paths themselves -- missing required variant,
unknown variant, custom window with no calendar anchor. They were verified by
hand last round but left unpinned, which is the same shape of problem as the
optional gates they exist to enforce. All return before any network call.
Deliberately not done, as not load-bearing: recording the oas400 variant's
missing-credit session count (the truncation conclusion rests on the
distribution mismatch, which is already recorded), and generalising
_pipeline_gates for arbitrary --end/--sessions windows.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two sensors saturated in exactly the range where resolution matters, and the top
State band had no headroom. Calibrated with scripts/run_regime_monitor_calibration.py
over the 408 sessions ending 2026-07-24; the shipped code reproduces that run's
band shares exactly (78.9 / 13.0 / 4.7 / 3.4).
V1 read VIX 30, 50 and 82 as an identical 100 — the same defect v3 had just
removed from P3, left in place one sensor over. In the window it flattened five
distinct April-2025 prints (52.33, 46.98, 45.31, 40.72, 38.57) into one value.
Now an anchor table reaching full scale at 55, not at 2020's ~82: anchoring the
top at a once-in-a-generation print would make VIX 50 read only ~70. Pegged on
14 of 408 sessions before; none now.
_under_200 returned a bare 0/100, so P1 printed 100 the moment SMH and QQQ were
both under their average — and since the price pillar takes max(P1, P2, P3),
that pinned the pillar and stopped P3's ladder resolving for the whole of a
selloff. Now graded by depth below the 200-DMA, with a deliberate floor of 20 at
the crossing: the break is a genuine binary event, only its depth is graded.
Pegged on 46 of 408 sessions before; none now. A 2% break reads ~30, not 100.
max() was KEPT — the defect was the step function feeding it, not the vote, and
v3's "one capped vote for correlated reads" rationale still holds. P1 is the sole
price argmax on 17 of 408 sessions (4.2%), so the P1_SCORE_CAP fallback drafted
during design was measured as unnecessary and not shipped.
STATE_BANDS breaking 80 -> 65, and only that threshold. Credit returns 0.0 (not
None) when calm, so it holds its 20 points pinned at zero and price + breadth +
volatility at literal maximum summed to exactly 80.0 — v3's threshold to the
decimal, with nothing above it. The sensor is deliberately unchanged: a
calm-credit selloff genuinely is less stressed. What was stale is the band, fit
on v2 while credit's since-removed percentile leg still contributed. A
2022-style AI/tech drawdown with calm credit computes to 70.3 (no death cross) or
74.0 (with one); 70 would have left 0.33 points of headroom, reproducing the
defect. Chosen by scenario arithmetic, and the realized breaking share then lands
on 3.4% — the same as v3's, arrived at independently.
"v4" added to CATEGORICAL_FUNDAMENTAL_METHODOLOGIES in this same commit, which is
load-bearing: that set is checked against the STORED blob, so bumping without it
discards the collected observation on first write, leaving fetched_at null and
locked false — and update_regime_monitor then fires a paid LLM refresh on every
run, forever. Now guarded by a test parametrised over v2 and v3 stored blobs.
SENSOR_REVISION deliberately stays 2: a METHODOLOGY change already forces a full
reseed via _parse_snapshot, and bumping both would imply the reseed was
revision-driven.
QUADRANT_STATE_DIVIDER stays 50 because only breaking moved, so alert_service,
RegimeChart and the quadrant tests need no change. A new test enforces
divider == band boundary on both axes, which nothing did before.
Doc renamed to regime-monitor-v4.md with a tombstone at the old path (commit
messages cite it), the three open questions converted to resolved with the
reasoning that closed them, and indexed in docs/research/README.md for the first
time. The P2 limit is stated honestly: _death_cross pegs at a -5% MA gap, so a
deep selloff still reaches 100 via P2 — v4 repairs the shallow-to-moderate break,
not "the price pillar no longer pegs".
DEPLOY: the first run reseeds ~464 sessions. Expect one phantom quadrant alert
(the dedup key carries basket_hash, not methodology) and re-run the Event Study
manually — its cached report self-invalidates but does not self-regenerate.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Scan of every module and exported symbol, with each candidate verified by hand
rather than trusted from the scan.
Deleted outright:
frontend/src/lib/fundamentals.ts (112 lines, 12 exports) — imported by
nothing, including FundamentalsPanel, which reads backend values. It mirrors
scoring_service._compute_fundamental_score, so it is the same *kind* of
thing as lib/qualification.ts — but nothing consumes it, so it mirrored
nothing and could drift out of sync unnoticed.
Skeleton.SkeletonLine, paperTrades.getEquityCurve, regime.regimeColor
breadth_service.compute_breadth_today — self-described "thin wrapper, for
future live use"; that future did not arrive.
Kept, but unexported — used inside their own module, so the dead part was the
public surface, not the code: Button.Spinner, exitPlan.SETUP_STOP_ATR_MULTIPLIER,
client.ApiError.
Three things the scan flagged that are NOT dead, recorded so the next sweep does
not re-raise them:
RegimeChart.tsx — lazy(() => import(...)) in RegimePage, so it looks orphaned
to any importer-graph scan. Deleting it would break the risk page.
qualification.ts MIN_TARGET_PROBABILITY / liveRiskReward — that file is a live
mirror of app/services/qualification.py used in five places, and the
constant is exported to document the backend value it tracks.
ssl_bootstrap.ssl_status — called from an inline python snippet inside
scripts/run_tier1_macbook.sh, invisible to a .py-only search.
No orphaned backend modules across app/.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two review findings on 22bee28, both reproduced before fixing.
[1] Concurrent writes could lose or rewind a row. record_finish did
select-then-insert-or-update, and pipelines are separate scheduler jobs that can
overlap while sharing step ids -- data_collector belongs to all four. Reproduced
both halves: two sessions that SELECT before either INSERTs make the second
commit raise IntegrityError, which _persist_job_run swallows, so the run
silently vanishes; and a later write carrying an OLDER finished_at rewound the
row from 12:00 back to 09:00, dragging the status with it, so the panel would
report a stale outcome as the latest.
Now a single atomic INSERT ... ON CONFLICT (job_name) DO UPDATE, guarded by
WHERE job_run_state.finished_at < excluded.finished_at so an older completion
can never overwrite a newer one. Dialect-specific because prod is Postgres and
tests are SQLite; both support it (SQLite >= 3.24, ours is 3.45). updated_at is
set explicitly, as the model's onupdate hook does not fire for a core upsert,
and get_map now uses populate_existing since core writes leave any
previously-loaded ORM instance stale in the identity map.
[2] Shutdown could dispose the engine underneath a pending write.
scheduler.shutdown(wait=False) returns before APScheduler dispatches its
completion events, and those events are what create persist tasks -- so a single
snapshot of the task set missed writes still to be queued. flush_job_run_persists
now settles briefly for pending callbacks, then drains in a loop until the set
stays empty, with the deadline still bounding total shutdown time. Left
shutdown(wait=False) alone deliberately: waiting would block a deploy restart
behind a long-running scan.
Six regression tests: interleaved first writes, older-never-rewinds,
newer-still-wins, a task queued mid-drain, prompt return when idle, and giving
up rather than hanging shutdown.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Job outcomes lived only in scheduler._job_runtime, an in-memory dict. Every
deploy wiped it, so Admin -> Jobs could report "Active" with no indication a job
had ever run or how it ended -- which is the main thing that page is for.
New job_run_state table (migration 031): one row per job, upserted on job_name.
Deliberately not history -- system_events already grows unbounded with no
retention job, and a second append-only operational table would repeat that
debt. Adding history later is purely additive.
Written from two hooks, NOT from _runtime_finish. That looked cheapest (one
function, ~40 call sites) but unit tests invoke job coroutines directly, so it
would fire detached DB writes at the real session factory throughout the suite,
and there is no testing flag to guard on.
- An APScheduler EVENT_JOB_EXECUTED/ERROR listener covers everything the
scheduler fires, including manual triggers. Its detached task is held in a
module-level set (a bare create_task result can be collected mid-flight) and
drained in the app lifespan before engine.dispose().
- _run_pipeline persists directly, and must: pipeline steps are plain
coroutine calls that emit no scheduler events, so the listener cannot see
them. The step persist sits AFTER the except that swallows step errors --
inside it, exactly the failed runs worth seeing would be skipped. The
orchestrator persists in the finally, and the disabled early-return persists
too, or "skipped" is silently dropped.
_persist_job_run never raises: a persistence failure must not break an otherwise
successful pipeline.
The API reports this as last_run_* and leaves runtime_* meaning strictly live
in-memory state. Reusing runtime_status would have been a regression, not a
no-op: JobControls drives the status chip from it (a job that errored eight days
ago would read "Last run error" forever instead of "Active") and picks the
rate-limit banner from it (a week-old rate limit would pin the banner
permanently). Tests pin the split.
The table starts empty; each job fills its row the next time it finishes. No
backfill from system_events, which records only warning/error outcomes under a
different status vocabulary and would invent successes that never happened.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Groundwork for the Admin -> Jobs cleanup. Three sources of truth collapse into
app/job_catalog.py, which imports nothing from app so both the scheduler and
admin_service can import it at module level (admin_service otherwise has to
import the scheduler inside functions to dodge a cycle).
PIPELINE_MEMBERS is now DERIVED from the four pipeline step lists instead of
being a literal set in admin_service duplicating four lists in scheduler.py with
nothing asserting they agreed. A test pins that the derivation reproduces the
previous hand-maintained 9 names exactly, so this is behaviour-preserving.
Deletes the private _JOB_NAMES list, which held 16 of the 19 jobs:
benchmark_collector, outcome_evaluator and shadow_book had no runtime row, and
so no "last run" line in the panel, until their first run in a given process.
_job_runtime is now seeded from the catalog, and a test pins the invariant.
Next-run is decided by category rather than by reading a timestamp. A pipeline
step has no schedule of its own, so it reports its parent's ("next via Morning
Pipeline in 3h") instead of nothing; a manual job says manual_only rather than
rendering a date. This also fixes a real bug: triggering a paused job set
next_run_time=now, APScheduler re-armed the 520-week backstop behind it, and the
panel displayed "next run in ~87600h". Two independent guards -- the category
rule, plus _visible_next_run dropping anything past a year -- and an APScheduler
listener that re-pauses steps and manual jobs once their run finishes. The
listener is registered at module level because configure_scheduler is called
more than once and add_listener does not deduplicate.
Migrates backtest and ticker_universe_sync from interval to cron (Sun 03:00 ET
and 01:00 ET). configure_scheduler calls remove_all_jobs() on every startup, so
an interval countdown restarts each deploy -- a 168h backtest needed a week of
uninterrupted uptime to fire even once. The codebase already documented this
pitfall as the reason cron was adopted; these two were never migrated. Both are
now editable in Admin -> Schedule.
Also: list_jobs went from one settings query per job (19) to one for all of
them, and data_backfill is hidden from the listing while staying registered and
API-triggerable.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Follows 5ea0785, which renamed the user-visible labels. This finishes the pass
so code, docs and operator output use one vocabulary: README (pipeline list,
route table, FRED row), the methodology doc title, .env.example and config
comments, the snapshot model / event-study / service / test docstrings, the
scheduler section headers and morning-pipeline docstring, the TopBar status
text ("bullish regime" -> "bullish trend"), and the four "Regime monitor:" log
prefixes.
Deliberately NOT changed, because "market regime" is also a standard finance
term and most occurrences are not this job: the backtest caveat "~6 months is
roughly one market regime" in backtest_service, README, BacktestPanel and every
generated reports/*.json; "a regime shift" in TrackRecordPanel; and the
capacity-bracket findings doc. Renaming those would have made the text wrong.
Also unchanged, being persisted or externally linked rather than wording: the
regime_monitor / market_regime job ids, the regime_quadrant_enabled setting key,
the /regime route, METHODOLOGY and the snapshot fields, the service/test module
filenames, and docs/research/regime-monitor-v3.md's path (referenced from commit
messages). The doc now carries a one-line note recording the old name and why
those identifiers still use it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
"Market Regime" and "Regime Monitor" sat next to each other in Admin -> Jobs
(pipeline steps 4 and 5) reading as the same job. They are unrelated, and the
names had it backwards: "Market Regime" is the SPY 50/200 guard that drives the
TopBar trend dot and the counter-trend warning on setups, so it changes what a
setup shows; "Regime Monitor" is the observational AI/Tech thermometer that
explicitly feeds no trades. The more consequential job had the vaguer name.
market_regime "Market Regime" -> "Market Trend (SPY)"
regime_monitor "Regime Monitor" -> "AI/Tech Risk Monitor"
Display strings only. The job *ids* are persisted -- they key the pipeline step
list, cron config, runtime tracking and run history -- so they are untouched,
as is the /regime route, which keeps existing links working.
The label the admin UI renders comes from JOB_LABELS in admin_service (via
routers/jobs.py), not from the scheduler's APScheduler `name=`. Both are updated;
only the former is user-visible.
Carries the vocabulary through the rest of the surface so it does not half-land:
page title, nav ("Regime" -> "Risk"), the empty-state instruction that names the
job to run, the quadrant alert toggle, the morning-pipeline hint, and the
Telegram alert headline ("Regime quadrant change" -> "AI/Tech risk quadrant
change"). No test asserts any of these strings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two review findings on 46ace50.
[P1] Raising HY_OAS_WINDOW_DAYS to 700 only reached newly computed rows. A
routine run recomputes the latest trading date alone, and `rebuilding` was keyed
on "no v3 snapshot exists at all", which is false once the cutover has run --
so every row already written kept the credit gap the wider window exists to
close, indefinitely.
Adds SENSOR_REVISION: stamped into each snapshot, absent on pre-marker rows
(read as 1), and a stored revision below the current one triggers exactly one
reseed. Deliberately not METHODOLOGY, which would partition the history API and
discard the cached event study -- neither warranted, since the study recomputes
its Warning series from source rather than reading snapshots and so cannot be
staled by a reseed.
The reseed is bounded by REBUILD_LOOKBACK_DAYS in calendar days rather than a
session count, because the binding constraint is the OAS fetch: each replayed
row needs W3's 20-business-day lookback inside HY_OAS_WINDOW_DAYS. Replaying by
session count would have left the oldest stored rows unrepaired -- the exact
rows the fix targets. At 672 days the replay covers ~464 sessions, W3's oldest
requirement lands on the first fetched OAS day, and the ~400-session series the
cutover wrote is fully covered. A test asserts that relationship so the two
constants cannot drift back into recreating the gap.
[P2] With nothing ever collected, current_observation returned available=true
and the default placeholders -- "unknown" for every hyperscaler, "mixed" for the
reaction -- so the card announced a reading that never happened. Those are the
absence of an observation, not an observation of absence. Gated on `observed`
(non-null fetched_at, the one field every path writing real content stamps),
which blanks the content and drives a proper empty state naming where an admin
collects one. This was a regression from 46ace50; fundamental_overlay never had
it, since no observation means no effective date means pending.
Also renames the leftover v2 identifiers in the touched paths
(rewrite_existing_v2, latest_v2).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The page had twelve stacked blocks, several of them different views of the
same numbers. The quadrant plot and the score-history chart drew the same two
series from the same query key, which read as two datasets; they are now one
card with a Time | Path toggle. The two pillar disclosures become one grouped
table, and three prose blocks (data quality, basket, coverage) become one
provenance chip strip. Page text is now limited to what changes how the reader
interprets today's number; the rest moved to the methodology doc.
Removes three stale-threshold bugs of one class. The quadrant fell back to v2's
60/60 dividers when quadrant_config was absent -- the real values are 50/40 and
they feed alert_service, so the chart could disagree with what actually fires.
The gauge fell back to v2's 30/60/80 band ticks, and drew a divider line that
always landed on its own "elevated" tick. The time series' reference lines were
at 30/60/80, which correspond to nothing in v3; they are now per-axis dashed
lines read from the same quadrant_config. Rendering also surfaced a live
clipping bug inherited from the old chart: margin.left -18 against YAxis
width 28 left ~10px for a 3-digit label, so every Y tick was cut off.
HY_OAS_WINDOW_DAYS was 400 *calendar* days while a rebuild replays
REBUILD_SESSIONS = 400 *trading* sessions (~579 calendar days), so the oldest
~180 days of any rebuild got no OAS at all and both credit sensors returned
None. State then lands at 80% coverage and Warning at exactly MIN_COVERAGE, so
both still publish bands -- a series that looks homogeneous while its oldest
rows were scored without credit. Widened to 700. This needs no methodology
bump: C1 reads [-1] and W3 reads [-21], both from the end, so widening only
prepends and every live score is bit-identical. Sequenced deliberately, since
acting on the open findings below bumps METHODOLOGY and fires the rebuild.
A just-collected fundamental observation was hidden until its effective date --
one day, three over a weekend -- because the live reading called the
point-in-time function, so refreshing appeared to do nothing. That was the
opposite of what the doc claimed. fundamental_overlay stays the gated record
(it runs for every replayed date during a rebuild); current_observation is the
live reading and reports the effective date instead of blanking the content.
Nothing in the overlay is scored, so showing it early cannot reach a published
number.
Documents four calculation findings. Three are not implemented, since each
changes a published score and so requires a v4 cut: State's top band is a
credit-event band (credit returns 0.0 rather than None below the 3.5 anchor, so
it is pinned at zero at weight 20 -- with everything else pegged State computes
to exactly 80.0, the breaking threshold); V1 saturates at VIX 30; and the
deliberate max(P1,P2,P3) defeats P3's anchoring because P1 is binary.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three follow-ups from review of the A6 commits.
The cache-summary re-finalize raised a second durable system event on a failed
run: _runtime_finish emits for `error`/`rate_limited`, and the dedup key
includes the message, so "SEC unavailable" and "SEC unavailable · cache 511 · 2
score inputs changed" landed as two unacknowledged Admin events. Adds
`emit_event` so a re-finalize that only rewords an outcome stays silent, with a
regression test asserting exactly one event.
The rollback section still claimed disabling the SEC job freezes the cache — the
opposite of what the same page says two lines earlier, and of what the code now
does. Rewritten: there is no Admin cache-off switch, restoring `fundamental_data`
alone is temporary because the next run rebuilds it from the same snapshots and
code, and a real freeze means stopping the service.
Remaining "shadow" wording: the two import jobs have never been shadow since
activation, so `_run_shadow_import` -> `_run_source_import`, its section heading,
the deployment doc's job label, and the plan doc's "production switch remains"
handoff paragraph are all brought up to date.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The runtime message only appended the cache summary when the import itself
completed. On a deferred, failed or source-locked run Admin → Jobs showed just
the import outcome, so an operator had no signal that `fundamental_data` had
advanced — contradicting the claim now made in the docstring, the schedule hint
and the deployment doc.
The import status still varies and stays the headline; the cache summary is
appended to all of them. Adds a source-locked regression test.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The A5 cutover has been on and observed in production, so SEC Company Facts +
DoltHub earnings are already the live source for `fundamental_data`. This
removes everything the legacy path still occupied.
Gone: the three providers and their config/env keys; the weekly
`fundamental_collector` job; the cutover toggle (SEC + Dolt is now the
unconditional path, so `off` can no longer silently freeze scoring inputs); the
A5 parity report, whose deltas became structurally zero once the candidate
builder started writing the table it compared against; and the FMP tier of
universe bootstrap.
Two behavioral notes:
- Disabling **SEC Fundamentals Import** now stops the SEC network fetch only.
The local cache refresh moved outside the job-enable check, because candidates
also derive from daily closes and earnings events — freezing those on an
ingestion pause would stale scoring with no fallback left to recover from.
- `/ingestion/fetch?sources=fundamentals` still accepts the key and reports
`skipped`; there is no per-ticker fetch any more.
Migration 029 does not blanket-delete the leftover settings rows. Migrations run
before the service restart, and pre-A6 code reads an absent `job_*_enabled` row
as *enabled* — so the two behavior-bearing keys become tombstones pinned to safe
values (hidden in Admin) and only the inert three are deleted. Removing the
provider keys from the production `.env` is the matching rollout step.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The capacity bracket study (reports/portfolio-construction-prod505-capacity-
bracket-daily-v1) showed a book whose count cap never binds earns +1.1pp CAGR
over the old 10 -- 51 of 175 paired cohorts better, 2 worse -- at unchanged
drawdown (+0.007pp) and better Calmar in 51 of the 52 cohorts that moved.
The headline EV-per-trade delta is ~0 (+0.001), which is the trap: capacity
does not change trade quality, it changes trade COUNT. Flat EV/trade means the
blocked entries were just as good as the taken ones, so refusing them cost their
whole contribution to return. Judge capacity on CAGR, never on EV per trade.
15 is headroom, not a target. cap15 peaked at 12 positions with zero full-book
skips, so cash plus SIM_NOTIONAL_CAP is the real ceiling and 15/20/None are the
same experiment.
SIM_MAX_POSITIONS and the shadow book's DEFAULT_CAPACITY move together to keep
backtest and production in parity. Historical research arms pass max_positions
explicitly, so their labels and past results are unaffected.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A weekday with no published index is usually just a market holiday
(~10/yr), not a fault. The WARNING still earns its place as the guard
against inferring missing from an error code, but the comment should
not claim more than it can.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The fundamentals import has been dead since 2026-07-25, alerting
SecForbiddenError on form.20260725.idx with "set a real sec_user_agent
contact email". The User-Agent was never the problem.
www.sec.gov/Archives is served from an S3 bucket with no ListBucket
grant, so a MISSING key cannot answer 404 — it returns 403 with S3's
AccessDenied XML. SEC publishes a daily index for business days only, so
2026-07-25 (a Saturday) is simply absent. _get mapped every 403 to the
fatal SecForbiddenError, which made daily_index's `except
SecNotFoundError` unreachable for the exact case it was written for:
the first weekend an incremental walk crossed killed the run, and
last_processed never advanced past Friday.
Latent until activation, not a change at SEC: with no promoted run the
importer takes the backfill path and makes zero daily_index calls, so
the walk was first exercised by the first incremental run.
Verified live 2026-07-30: Sat/Sun 403 with AccessDenied XML while Fri
(51 rows) and Mon (26 rows) return 200 on the same UA; a genuine
rejection is instead the WAF's text/html "Undeclared Automated Tool"
page, served even for files that exist. So the downgrade to "missing" is
gated on all three: the /Archives/ prefix, an XML content type, and
S3's own error code. Every other 403 still alerts and stops.
Missing weekday indexes now log at WARNING — if a rejection page were
ever misread as absent, the importer must not advance past real filings
quietly.
No state to reset: _last_processed_index_date reads promoted runs only,
so the next run walks 2026-07-25..29, skipping the weekend.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The fundamentals import had been failing for three days on two tracked
filings the index listed but Company Facts appeared not to have. They were
not lagging: SEC filed the XBRL of NEE's and DOW's 2026-07-24 combined
parent/subsidiary 10-Qs under the co-registrant's CIK (Florida Power &
Light, Dow Chemical), so the ticker-carrying filer's own facts file never
receives that accession. This does not self-correct - an NEE filing
misattributed the same way in 2014 is still misfiled.
Because source_max_date advances only on a promoted run, the failure was
self-perpetuating: every later run re-walked the same index day and re-hit
the same two filings.
- Recover from the co-registrant file. The daily index lists every
co-registrant of an accession, which is the only pointer to where the
facts actually landed. Rows are re-stamped to the real filer, since
parse_snapshots stamps the CIK of the payload it read.
- Guard the recovery with a share-count continuity check against the
issuer's own history, so a subsidiary's standalone facts can never be
stored as the parent's. No history, no recovery.
- Bound the blocking: a filing still unresolvable after
MISSING_XBRL_RETRY_DAYS promotes with a named unresolved_filing warning
instead of wedging every later import.
- Name the offending filings in the alert and record them in
validation_json, separating not_in_companyfacts from not_in_submissions.
The gate previously reported a count and discarded the accessions.
Verified against live SEC data: both filings recover with the correct CIK
and the guard rejects a mismatched reference.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A conditional clause in the summary sentence carried an f prefix with no
placeholders, failing `ruff check app/` and blocking the deploy. Literal only;
the rendered text is unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The v3 cutover run scored 2/4 corrections warned against v2's 3/4, which reads
like a regression and is not one. Only 4 of the 11 detected corrections fall in
the holdout, so recall is one event from a different headline -- and the event
that flips is decided by threshold placement, not by what the score saw. "v3
without the credit sensor" catches 2025-02-21 at a *higher* threshold (35.5)
than shipped v3 misses it at (32.3), because the alarm rule needs a rising edge
and a lower threshold can fire outside the horizon then never reset below.
Two caveats are now computed and surfaced rather than left for the reader to
infer:
- Holdout event count against MIN_EVENTS_FOR_CONFIDENCE. The summary sentence
states how many of the detected corrections actually fall in the test period.
- Warning-sensor coverage across the split. The score renormalises over what is
available, so a training window predating a sensor's history freezes the
threshold on a different construct than the holdout is measured against. At
the cutover that is 39% of training sessions with all three sensors versus
100% of the test period, credit history beginning 2023-07-25.
Restricting the threshold to sensor-matched training sessions was tested and
rejected: those sessions are a calm recent stretch, so the threshold falls from
32.3 to 22.5 and false alarms rise from 3.3 to 8.6/yr. It swaps a coverage bias
for a regime-selection bias. The report states its limits instead.
_warning_series now returns per-session sensor counts alongside the scores.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The LLM-sourced capex/earnings observations carried 12+8 of 100 Warning points,
so both pegged at 100 produced a Warning of 20.0 -- below the event study's 25.3
alarm threshold and still inside the "stable" band. The reading was
arithmetically incapable of changing anything on screen, which is why refreshing
it appeared to do nothing. They are now a qualitative overlay reported beside
the scores rather than diluted into them.
Calibrated against the 408 v2 sessions to 2026-07-24, reproduced offline from
Alpaca + FRED; the harness matched the stored prod distribution exactly before
any parameter was changed.
State:
- P3 used dd_pct * 5, reaching 100 at a 20% drawdown -- the 90th percentile of
the observed distribution -- so 39/408 sessions sat at exactly 100 with no
resolution left during the part of a selloff that matters most. Replaced with
anchored breakpoints keeping headroom past the observed 36% maximum, blended
2:1 like P1/P2 instead of max(). P3's realized share of State falls from 65%
to 40%, matching its nominal weight.
- Credit level is now anchors-only. ICE capped FRED's BAMLH0A0HYM2 at a rolling
3-year window in April 2026, silently turning the 10-year percentile leg into
a 3-year one that scored 20 points of stress at an OAS of 3.5 -- the level its
own anchors call "mild". The anchors already encode the long-run distribution.
Warning:
- Added HY OAS 20-session widening (25%). The level is pinned at zero below the
3.5 anchor; its rate of change is not.
- Divergence tapers to a 0.35 floor instead of a hard price_ret >= 0 gate, which
zeroed the sensor through every decline: on 2026-07-24 the basket shed 10
points of participation in 20 sessions and Warning printed exactly 0.
- The event study and the live monitor now share one sensor definition, so they
cannot silently drift apart.
Bands are per axis (State 20/50/80, Warning 20/40/60) with quadrant dividers at
50/40; v2 Warning never exceeded 64.9 against a shared 60, leaving that half of
the quadrant unreachable. Realized shares: State 73/15/8/3%, Warning 69/20/8/3%.
Snapshots now record credit_history_days and vix_history_days -- the percentile
defect went unnoticed for months because nothing asserted the window the code
claimed.
Cutover: the first run rebuilds 400 sessions automatically; the Event Study job
must be re-run, as its cached report self-invalidates on the methodology check.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Regression found by the post-reparse collision check. _period_identity trusted
submissions.fiscalYearEnd, which is not reliable: Franklin Resources (BEN)
declares 1231 while every one of its 10-Ks ends 09-30.
The effect was data loss, not just a bad label. BEN's real fiscal Q1 (Dec 31)
sat 0 days from the claimed year end, matching no quarter band, so it fell back
to SEC's fy/fp; its fiscal Q2 (Mar 31) computed 275 days out and was labelled
Q1. Both landed on the same key, the collision discarded one, and BEN lost TTM
EPS and revenue growth entirely — values it had before this branch.
A 10-K's reportDate IS the fiscal year end by definition, so resolve_fiscal_
year_end() now prefers the issuer's most recent annual filing and treats the
declared value as a fallback for issuers with no 10-K in the set.
Scanned the full tracked universe: 2 of 506 issuers declare a year end more
than 21 days from their own 10-K — BEN (91d, broken) and DELL (29d, mislabelled
but functionally correct). Both now derive correctly and match the legacy
provider: BEN revenue growth 3.8243 vs 3.82, DELL 38.5735 vs 38.57. Controls
(AAPL, COST, PEP, DPZ, IRM, JPM, CRM, STX, AVY) byte-identical.
826 unit tests pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Found in review. _merge_amendments rebuilds a period from _MERGED_FIELDS +
_CARRIED_FIELDS alone, so a column in neither list is absent from the merged
row, not just stale — and callers read it with getattr(..., None), which
silently yields None. weighted_avg_diluted_shares was never added when the
market-cap fallback landed (_SNAPSHOT_COLS in the importer was updated, its
counterpart in the derivation was not).
The failure needed both of this branch's fixes at once: a multi-class issuer
with a partial amendment on its latest period (META with a Part-III-only
10-K/A) would silently lose market cap and FCF yield again.
Adds the field, a regression test for that case, and a guard test asserting
the merge/carry lists cover every SnapshotRow field, so the next column added
fails loudly rather than losing data quietly. Confirmed the guard catches the
original bug.
Also from review:
- Expose pe_caveat in the valuation payload, so a P/E suppressed by split
contamination says why instead of looking like missing data (the caveat was
set but never read).
- no_xbrl_filings now names both causes; the old text advised pinning a CIK
override, which is wrong for a genuine new registrant that simply has not
filed yet and clears itself.
- Document that fiscalYearEnd is the issuer's current calendar, so a fiscal-
year-end change degrades old periods (fallback/newest-wins), not current ones.
- Parser-level tests for _select_weighted_avg_shares (shortest-span-wins and
concept priority), which only had derivation-level coverage.
823 unit tests pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Operational plumbing to land the parser fixes and to make silent resolution
failures visible.
- Reparse: SecFundamentalsImporter(reparse=True) restages every accession with
the current parser and rewrites the ones that now reconstruct differently,
writing the full column set so a row is never half old-parse. Snapshots stay
immutable with respect to SEC; the stored row is our reconstruction, and after
a parser fix keeping it is a stale cache, not history. run_import(force=True)
bypasses the unchanged-revision no-op, since the staleness is on our side, not
the source's. Exposed as scripts/reparse_fundamentals.py, dry-run by default.
- CIK overrides: sec_universe reads a {symbol: cik} pin from
SystemSetting['sec_cik_overrides'], applied ahead of company_tickers.json, for
when SEC maps a ticker to a successor shell with no filings (XOM -> a zero-
filing "ExxonMobil Holdings Corp" while every 10-K/Q is under CIK 34088).
- Resolution validation: a tracked issuer resolving to a registrant with no XBRL
filings now records no_xbrl_filings and raises a warning naming the CIKs and
the override setting, instead of silently yielding nothing on every run.
- _diff_fields compares datetime instants, not representations: accepted_at
round-trips naive from SQLite but tz-aware from Postgres, which otherwise made
a reparse of identical data report every row as changed (and false-positived
the pre-existing discrepancy warning).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The A5 parity report surfaced coverage gaps and wrong values that all traced
to the SEC facts parser and read-time derivation rather than to bad source
data. Fixes, each validated by replaying the production parser + derivation
against live company facts:
- Period identity is derived from period_end against the issuer's fiscal
calendar, not SEC's fy/fp fields, which collide (two period ends on one key,
one silently discarded) and invert (a period sorting before one that precedes
it) often enough to break the quarter chain. Recovers BXP, CRM, CRWD, FRT,
MTD, NTAP, PPL, STX, WDAY. Fixed labels are internal ordering keys only (not
in any API schema), so a filer whose year ends in early January shifting by
one is harmless.
- Revenue concept list gains RevenuesNetOfInterestExpense (banks) and the
IncludingAssessedTax variant (REITs/consumer); EPS gains the continuing-ops
variant (REG/FCX) and, last, basic EPS for a period tagging no diluted
variant at all (PPL). All appended, so any issuer that already resolved keeps
its concept.
- YTD span tolerance 20 -> 25 days, covering 4-4-5 retail calendars whose
36-week YTD-Q3 (251-252d) previously missed by ~2 (COST, PEP, DPZ).
- Amendment resolution is per field: a partial 10-K/A (Part III only, no
financial facts) no longer blanks the period (DVN).
- TTM diluted EPS is suppressed when a split contaminates the trailing window
(BKNG's mixed-unit sum produced a P/E of 1.10 that clamped to a perfect
fundamental sub-score). A post-filing split with no share-count evidence
(KLAC) remains undetectable from this data.
- Multi-class share fallback: weighted_avg_diluted_shares is captured and used
for market cap when the cover-page count is absent (dimensional, so missing
from company facts for META/CMCSA/CHTR/FOXA/NWSA/LEN). Within ~0.6% of the
true count on controls; flagged shares_estimated in the API. BRK-B has no
weighted-average fact either and stays unavailable.
820 unit tests pass; new tests confirmed to fail against the pre-fix code.
Effect is inert until existing rows are reparsed (see reparse path).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
