signal-platform/docs/data-reliability.md

Data reliability

Three changes that make the platform's data trustworthy rather than faster or more profitable. They came out of the integrity experiments, which turned up two problems worth fixing on their own:

• Changing the stock universe materially changes results — Arm A 2.09 Sharpe / 51.6% CAGR against Arm B 1.57 / 38.1% over the same window, from universe composition alone (survivorship audit).
• Missing data quietly produces a seemingly successful backtest. Found while running Step 3a: a snapshot whose schema predated migrations 026/032 made every _fetch_columns call raise no such column: tickers.cik, both dispatch paths caught the per-ticker exception and continued, and the run wrote a complete report with zero candidates. Nothing in the report or the UI said the run had lost its entire universe.

#	Change	Status
1	Make incomplete backtests visibly incomplete	done — 2026-09-07
2	Preserve history when a stock leaves the tracked universe	done — 2026-09-07
3	Record universe membership by effective date	done — 2026-09-07

---

1. Make incomplete backtests visibly incomplete — done

The replay loop is deliberately fault-tolerant: one unreadable ticker must not abort a 500-symbol run. That is right for the run and wrong for its report, which had no way to say how much of the universe it actually covered. A run that lost every ticker and a run that honestly found no setups produced the same report.

What the report now carries

run_backtest builds a _DataQuality ledger (app/services/backtest_service.py) and emits it as report["data_quality"]:

{
  "status": "failed",
  "complete": false,
  "reasons": ["no symbol was evaluated successfully", "506 symbols failed to load"],
  "expected": 506, "loaded": 0, "evaluated": 0,
  "no_data": 0, "no_data_symbols": [], "short_history": 0,
  "failed": 506,
  "failures_by_stage": {"fetch": 506},
  "failures": [{"symbol": "A", "stage": "fetch",
                "error": "OperationalError: (sqlite3.OperationalError) no such column: tickers.cik"}],
  "failures_truncated": true,
  "benchmark": {"symbol": "SPY", "sessions": 1516,
                "first": "2020-07-06", "last": "2026-07-17",
                "momentum_basis": "residual", "error": null}
}

"Complete" means the processing and coverage checks passed — every expected symbol loaded and evaluated, benchmark present. It does not mean the underlying prices are proven correct. A complete run over silently wrong bars is still complete; splits, adjustment drift and bad vendor prints are a different problem, and this ledger must not be read as clearing them. The card says so on the page, not only here.

The counters are self-checking. Every universe symbol ends in exactly one of no_data, failures or evaluated, and as_report verifies no_data + failed + evaluated == expected. A ledger that stops balancing — because a counter was missed when a code path was added — reports itself as a failure rather than quietly under-counting. short_history is a subset of evaluated (loaded, but fewer than MIN_LOOKBACK + HORIZON bars, so no outcome can resolve), not a fourth bucket.

Benchmark availability is a separate axis. Without SPY closes, residual 12-1 momentum silently falls back to raw 12-1 — the gate stops being the production gate. The block records the basis actually used, so the degradation is on the report instead of only in a log line.

Three statuses, because severity differs

Status	Meaning	UI
ok	Every symbol evaluated, benchmark present	Normal
degraded	The strategy is intact but ran on less than the whole universe	Tiles shown, banner above
failed	The headline can't be read as the strategy it claims to be	Tiles replaced by the callout

failed means one of: nothing evaluated, an empty universe, an unbalanced ledger, or a missing benchmark. Blanket suppression on a single missing ticker would train the reader to ignore the banner, which is the failure mode this is supposed to fix.

The callout replaces the performance tiles rather than sitting above them. A warning over ordinary-looking numbers is precisely how the zero-candidate run would still have read as a quiet week.

Where it shows

• BacktestDataQualityCard in the backtest panel, with the failure list behind a disclosure.
• The "Last run" line now reads 505/506 tickers — evaluated over expected, not the universe size. "506 tickers" on a run that evaluated none was the misleading part.
• productionBaseRates returns null for a failed run, so per-ticker base rates quoted next to a live setup — which carry no caveat of their own — disappear rather than being sourced from a broken replay.
• EvaluationPanel drops a failed run's expectancy as a drift baseline instead of reporting "tracking" or "drift" against it.
• A report with no data_quality block reads as unknown, not clean. Old cached reports predate the check; defaulting them to complete would reproduce the exact bug being fixed.

Acceptance evidence

Both runs used scripts/run_backtest_snapshot.py.

Snapshot	Result
survivorship-a.sqlite (healthy)	degraded · 505/506 evaluated · 202,765 candidates · one bar-less row (RHM)
the same file with tickers.cik dropped	failed · 0/506 evaluated · 506 fetch failures · 0 candidates

The broken-snapshot run is the one that matters: before this change it wrote a report that looked like an ordinary run with no setups.

The healthy run is not ok, and that turned out to be correct. It flagged no_data: 1 (RHM) and short_history: 1 (SPCX, 14 bars), which an independent bar-count query confirms. Neither is new — Step 2 already recorded RHM as inert in tr-rank-agreement.md, which is why Step 3a generated candidates over 505 of 506 tickers. The ledger derives it independently, from the run rather than from a hand-written note, which is the point: the same 506-vs-505 gap now appears on the report instead of only in a research write-up someone remembered to make.

prod.sqlite has the same bar-less RHM row, so production runs will report degraded too until that universe row is cleared — which is item 2's territory. That is deliberate: silencing a bar-less universe row would let a wholesale import failure read as clean. The reason names the symbols so it is a one-line fix rather than a standing nag.

Tests: tests/unit/test_backtest_data_quality.py (10, covering each status, the ledger identity, the failure cap, symbol naming and the benchmark axis) and four end-to-end cases in tests/unit/test_backtest_service.py, including every-ticker-fails and missing-benchmark.

---

2. Preserve history when a stock leaves the tracked universe — done

bootstrap_universe(prune_missing=True) ran DELETE FROM tickers, cascading OHLCV, setups, scores and S/R levels away for any symbol that had merely dropped out of the index. Migration 032's own comment predicted it: "Pruning a symbol that is merely no longer an index constituent still destroys history; that needs a tracked/membership state separate from delisting."

The two states, kept apart

Migration 035 adds Ticker.untracked_on, the sibling of delisted_on:

Column	Meaning
delisted_on	The security stopped trading
untracked_on	It still trades; it left the index and we stopped scanning it

Neither deletes anything. NULL in both means "in today's live path". active_only — the single opt-in choke point already applied to scanning, ranking, scoring, breadth and ingestion — now filters on both, and honours as_of for both, so a historical query can still ask what was in the scanner on a given date.

prune_missing=True now marks instead of deleting, and clears the mark for symbols that have rejoined the universe. That last part is the bug this refactor could have introduced: a returning name is already tracked, so it is neither added nor untracked, and without an explicit clear it would keep the flag from its earlier removal and stay invisible to the scanner forever.

The deleted return key is now always 0 rather than repurposed — a caller reading deleted: 12 must not be told rows were destroyed when they were not. untracked, untracked_symbols and retracked carry the real counts, and the admin toast reads "untracked N" instead of "deleted N".

Historical eligibility stays explicit

The backtest's universe query is now _load_backtest_symbols, a named function whose docstring states that it deliberately does not use active_only. Today's membership flag must not erase yesterday's candidates: a replay grades what the strategy could have selected at each past date, so it must see symbols that have since been delisted or dropped out. "Fixing the inconsistency" by adding active_only there is exactly how survivorship bias gets reintroduced, which is why a test asserts the opposite.

The one path that still really deletes is DELETE /tickers/{symbol} — an explicit operator action asking for exactly that. Pruning is not.

Untracking one symbol

POST /tickers/{symbol}/tracking with {"tracked": false} excludes a single symbol from the live path, mirroring the delisting pair (both are API-only; no UI control exists for either). It exists because prune_missing=True can only act on the whole universe, and a manually added symbol is by definition not in the fetched constituent list — so a prune sweeps every manual addition along with the genuine leavers. Reversible with {"tracked": true}.

The case it was built for: RHM. Added by hand on 2026-06-28 out of interest in the stock, it has produced zero rows in every ticker-linked table in the ten weeks since — no bars, no ingestion progress, no scores — and the Alpaca name backfill never resolved it. It is the only symbol in the universe that has produced nothing at all (BK, CTRA and SATS also lack Alpaca names but carry full history, so a missing name alone proves nothing). Rheinmetall is XETRA-listed, so a plain US RHM is not something the OHLCV provider can serve. An attempt to confirm this against Alpaca's asset API returned 401 for the AAPL control too, so the identity is unverified — the justification is the persistent absence of usable data, not a confirmed listing status or a delisting.

Untracking it does not clear the backtest's degraded flag, and must not. The backtest deliberately includes untracked symbols, so RHM stays in the universe query, still returns no bars, and still counts in no_data. What untracking buys is narrower and still worth having: RHM leaves the scanner, ingestion and scoring, and the decision is recorded explicitly rather than sitting as an unexplained gap. Making the warning go away would mean changing historical eligibility to silence a real coverage gap — guarded by test_untracking_does_not_clear_a_data_quality_warning.

Note what is not solved: a manual addition is still indistinguishable from a universe member, so a prune untracks it (non-destructively, and visibly via untracked_symbols, but silently in the sense that nobody asked). Recording why a symbol is tracked is item 3's job.

Tests

tests/unit/test_universe_membership.py (12) covers the three behaviours together:

1. Pruning preserves the symbol's bars and its linked records (S/R levels are checked too, so the assertion is about the cascade, not just one table).
2. The pruned symbol leaves active_only while staying visible in the registry.
3. _load_backtest_symbols still returns it.

Plus re-tracking, the mark not being re-stamped on a second prune (which would erase when the symbol actually left), and as_of working for both columns. a row carrying both marks (it left the index, then delisted), and a compilation guard asserting every active_only call site still receives both predicates ANDed — including scheduler.py's, which chains group_by/order_by onto the result from outside. tests/unit/test_migration_035.py exercises the migration's upgrade and downgrade directly, including that existing rows default to NULL — an upgrade must not drop the whole universe out of the live path.

One detail worth keeping: untracked_on is stamped by the database (func.current_date()), not date.today(). active_only compares it against func.current_date(), so a process in a timezone ahead of the database would otherwise write a date that reads as "effective tomorrow" and leave the symbol in the scanner for a day. Same clock on both sides by construction.

3. Record universe membership by effective date — done

Migration 036. Three things that were still conflated are now separate:

Concept	Where	Written by
Index membership, observed history	universe_memberships intervals	every live sync
Excluded from scanning by a prune	Ticker.untracked_on	prune only
Operator switched it off	Ticker.scan_disabled_on	set_tracked only
Tracking ownership	Ticker.tracking_source	add_ticker / bootstrap

Why the operator flag is its own column. With one column serving both writers, a prune -> re-entry cycle would clear an explicit operator "off", because re-entry cannot tell who set it. Re-entry clears untracked_on and never touches scan_disabled_on.

active_only now tests three columns and stays join-free — it is on ~8 live call sites, and the compilation guard asserts all three predicates land, ANDed, including at the call site that chains group_by/order_by on from outside.

untracked_on is not current membership

It records exclusion by pruning. The scheduled sync runs prune_missing=False, so a symbol can have left the index and still be scanned; the two states legitimately differ, and nothing asserts they agree.

Tracking ownership is sticky towards manual

tracking_source is index or manual. Prune touches only index rows, so manual additions survive index pruning — the RHM case.

A bootstrap never converts manual to index. That direction looks harmless and is not: a hand-added symbol that happens to join the index would lose its protection the moment it left again, which is exactly when it is needed. Only new rows a bootstrap creates are stamped index; an explicit add_ticker on an existing index-owned symbol claims it as manual.

POST /tickers therefore has three responses:

Case	Response
New symbol	Created as manually tracked
Existing, index-owned	200 with the existing row, now manually tracked
Existing, already manual	409 Duplicate

Claiming is not a 409 because the operator is saying "keep this stock", and refusing while leaving it prunable would defeat the protection they just asked for. It deliberately does not touch scan_disabled_on or untracked_on — wanting a stock kept is not the same as asking for it to be scanned, and silently re-enabling something switched off by hand would undo a decision nobody revisited. POST /tickers/{symbol}/tracking is the explicit control.

Membership is observed, never asserted

first_seen_on is the first date a live fetch actually listed the symbol — not an index effective date. Anything earlier is unknown and is not guessed; the hand-reconstructed scripts/data/sp500_removals.json stays a research artifact.

Intervals are append-only: departure sets absent_since and retains the row, and a re-entry inserts a new one. A symbol that left in March and returned in August has two intervals, so a point-in-time query for May correctly answers "not a member" — reopening the original row would erase that.

Boundaries, stated once. The column is named absent_since, not last_seen_on, because it stores the first observation that found the symbol gone — on that date it was already absent:

member on D  <=>  first_seen_on <= D < absent_since  (or absent_since IS NULL)

Lower bound inclusive, upper bound exclusive. The real departure lies somewhere in (previous observation, absent_since] — a day with a daily sync, a week if a week was missed. The column records when we looked, which is the only thing actually known. members_on(db, universe, as_of) implements exactly this, so the rule lives in one place rather than being re-derived per call site.

A date before the first successful live sync returns an empty list. That means "we were not looking", not "the index was empty", and a caller has to know the difference — answering with today's set would be precisely the survivorship error this exists to end.

Recording runs on every live sync, independent of pruning and of who owns the row: the scheduled job uses prune_missing=False, so membership recorded only on manual prunes would be full of holes. Manually owned symbols get intervals like any other — membership and tracking ownership are independent axes. Switching off scanning does not close a membership; the index still contains the symbol, we simply stopped looking at it.

A failed fetch drives nothing at all. fetch_universe_symbols falls back to a cached snapshot and then to a built-in seed list of ~44 symbols. Nothing that acts on "this symbol is or isn't in the index" may run on that data: not recording membership, not pruning, and not clearing a prune flag. Skipping only the membership write was half a safeguard — a seed fallback would still have untracked the rest of the universe, which a regression test caught (2 of 3 symbols untracked before the fix). Adding is the one exception: it destroys nothing and is what lets a cold start bootstrap from seeds. The summary reports membership_observed: false so a caller can tell "the index did not change" from "we never looked".

Migration decisions

• Existing rows become manual. Provenance cannot be recovered from created_at, and manual is fail-safe: a wrong guess costs an un-pruned symbol rather than one swept out of the scanner. Bootstraps claim the ones they own as they create them.
• Existing "off" states are carried into scan_disabled_on. Whoever wrote an existing untracked_on is ambiguous, and re-entry clears that column — so the value is copied to the one no automated run touches. A symbol somebody deliberately switched off cannot be reactivated by a sync.
• No history is backfilled. The table starts empty; history begins at the first observation.

Tests

test_universe_membership.py (19) and test_universe_membership_intervals.py (14): manual additions survive a prune; a bootstrap never claims a manual row, including across a join-then-leave cycle; an explicit add claims an index-owned symbol without touching its switches; re-entry clears the prune flag but never the operator's; a live sync opens intervals without pruning; departure closes and re-entry appends; a cache/seed fallback records nothing, untracks nothing and clears no prune flag; membership is recorded for manual rows; switching off scanning does not close a membership; and the boundary rules — lower inclusive, upper exclusive, gaps are not membership, per-universe scoping, and pre-recording dates returning empty. test_migration_036.py (4) pins both migration judgement calls and the downgrade.

The fixtures pass a live source label deliberately: a made-up one would now skip pruning entirely and make those tests vacuous.

Backtest eligibility is unchanged — _load_backtest_symbols still returns every symbol with stored history, and untracking still cannot silence a coverage gap.