Retiring a symbol meant delete_ticker or bootstrap_universe(prune_missing),
both of which cascade through OHLCV, setups and scores. That destroys exactly
the history four research documents already apologise for: today's tracked
universe projected backward is survivorship-biased, and hard-deleting every
delisted name is what causes it. Keeping the rows preserves the option to fix
that — it does not fix it, which needs the replay to model a delisting as an
exit event.
tickers gains delisted_on / delisted_reason (migration 032). NULL means
actively traded.
The filter is opt-in via ticker_service.active_only rather than folded into a
shared getter: the registry and admin views deliberately keep delisted rows so
the delisting is visible, and a silent default would undo that. Applied to the
live path only — scanner, momentum ranking, scoring, breadth, fundamentals
candidates, SEC universe, earnings import, ingestion loops. run_backtest keeps
them on purpose.
Detection runs off OHLCV staleness, not off the SEC fundamentals import: that
importer stalls for days on unrelated Company-Facts gaps and would take
detection down with it. On a stale symbol the scheduler asks SEC for a Form
25/25-NSE/15 and retires it only on a hit, so a halt or a rename (SATS->ECHO)
keeps the existing warning. The probe waits 3 stale days so a market-data
outage cannot turn into one SEC request per symbol per run.
Safe to automate because it is reversible: clear_delisted un-retires a false
positive, where a delete had already taken the history.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Addresses the A2 review:
1. Every dolt subprocess is now bounded by a hard timeout
(dolt_command_timeout_seconds, default 600s); on expiry the process is killed
and DoltError raised — a hung pull/sql can no longer pin the import
connection and advisory lock indefinitely. Tested (timeout + non-zero exit).
2. Initial-load validate is stronger: besides zero-future, an initial load now
requires a real forward horizon (>= 21d, under the ~35d observed on the
clone) AND universe coverage >= 50% (a broken symbol join can't seed a hollow
calendar). Subsequent runs keep the 50% collapse gate.
3. Revision uses DOLT_HASHOF('HEAD') — formally HEAD, not dolt_log-by-timestamp.
4. Free-disk floor raised 2 GB -> 5 GB (safe headroom over the ~1.7 GB clone).
Full suite 702 passed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A SourceImporter that ingests post-no-preference/earnings into earnings_events
for the tracked universe. Shadow by construction (nothing reads earnings_events
until A4).
- earnings_alignment.py: pure calendar<->EPS-history min-cost monotonic DP,
reused from scripts/import_dolthub_earnings.py with identical constants (not
extending that one-off script); symbol/session normalization; unit-tested
against the pinned constants.
- dolt_client.py: async dolt CLI wrapper (pull / current_commit / query_csv via
asyncio.create_subprocess_exec — never blocks the shared event loop) + disk
guard before pull.
- dolt_earnings_importer.py: detect_revision = pull + HEAD hash; stage = query
earnings_calendar + eps_history, dedup, align, map act_symbol->ticker_id
(normalize both sides so dotted BRK.B joins); promote is destructive
(delete future dolt_earnings rows + upsert; past never deleted) so validate is
FAIL-CLOSED — blocks when the staged forward calendar is empty or has collapsed
below 50% of what's loaded (the forward calendar is the acceptance gate).
- NOTICE: CC BY-SA 4.0 attribution; config: DOLT_BINARY / DOLT_DATA_DIR / etc.
Verified end-to-end against the real 1.68 GB clone (5 tickers: 133 events, 128
paired, forward calendar to 2026-08-26, BRK.B joined). Tests: 9 alignment + 7
importer + 1 skip-guarded real-clone smoke. Full suite 699 passed.
Remaining for A2: wire the daily ~02:30 ET shadow cron — deferred to pair with
the deploy-time dolt install + DOLT_DATA_DIR provisioning.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
dennisthiessenandClaude Opus 5