# Data reliability Three changes that make the platform's data trustworthy rather than faster or more profitable. They came out of the [integrity experiments](data-integrity-plan.md), which turned up two problems worth fixing on their own: - Changing the stock universe **materially changes results** — Arm A 2.09 Sharpe / 51.6% CAGR against Arm B 1.57 / 38.1% over the same window, from universe composition alone ([survivorship audit](research/survivorship-audit.md)). - **Missing data quietly produces a seemingly successful backtest.** Found while running Step 3a: a snapshot whose schema predated migrations 026/032 made every `_fetch_columns` call raise `no such column: tickers.cik`, both dispatch paths caught the per-ticker exception and continued, and the run wrote a complete report with zero candidates. Nothing in the report or the UI said the run had lost its entire universe. | # | Change | Status | |---|---|---| | 1 | Make incomplete backtests visibly incomplete | **done** — 2026-09-07 | | 2 | Preserve history when a stock leaves the tracked universe | **done** — 2026-09-07 | | 3 | Record universe membership by effective date | **done** — 2026-09-07 | --- ## 1. Make incomplete backtests visibly incomplete — done The replay loop is deliberately fault-tolerant: one unreadable ticker must not abort a 500-symbol run. That is right for the run and wrong for its report, which had no way to say how much of the universe it actually covered. A run that lost every ticker and a run that honestly found no setups produced the same report. ### What the report now carries `run_backtest` builds a `_DataQuality` ledger (`app/services/backtest_service.py`) and emits it as `report["data_quality"]`: ```json { "status": "failed", "complete": false, "reasons": ["no symbol was evaluated successfully", "506 symbols failed to load"], "expected": 506, "loaded": 0, "evaluated": 0, "no_data": 0, "no_data_symbols": [], "short_history": 0, "failed": 506, "failures_by_stage": {"fetch": 506}, "failures": [{"symbol": "A", "stage": "fetch", "error": "OperationalError: (sqlite3.OperationalError) no such column: tickers.cik"}], "failures_truncated": true, "benchmark": {"symbol": "SPY", "sessions": 1516, "first": "2020-07-06", "last": "2026-07-17", "momentum_basis": "residual", "error": null} } ``` **"Complete" means the processing and coverage checks passed** — every expected symbol loaded and evaluated, benchmark present. It does **not** mean the underlying prices are proven correct. A complete run over silently wrong bars is still complete; splits, adjustment drift and bad vendor prints are a different problem, and this ledger must not be read as clearing them. The card says so on the page, not only here. **The counters are self-checking.** Every universe symbol ends in exactly one of `no_data`, `failures` or `evaluated`, and `as_report` verifies `no_data + failed + evaluated == expected`. A ledger that stops balancing — because a counter was missed when a code path was added — reports itself as a failure rather than quietly under-counting. `short_history` is a *subset* of `evaluated` (loaded, but fewer than `MIN_LOOKBACK + HORIZON` bars, so no outcome can resolve), not a fourth bucket. **Benchmark availability is a separate axis.** Without SPY closes, residual 12-1 momentum silently falls back to raw 12-1 — the gate stops being the production gate. The block records the basis actually used, so the degradation is on the report instead of only in a log line. ### Three statuses, because severity differs | Status | Meaning | UI | |---|---|---| | `ok` | Every symbol evaluated, benchmark present | Normal | | `degraded` | The strategy is intact but ran on less than the whole universe | Tiles shown, banner above | | `failed` | The headline can't be read as the strategy it claims to be | **Tiles replaced by the callout** | `failed` means one of: nothing evaluated, an empty universe, an unbalanced ledger, or a missing benchmark. Blanket suppression on a single missing ticker would train the reader to ignore the banner, which is the failure mode this is supposed to fix. The callout **replaces** the performance tiles rather than sitting above them. A warning over ordinary-looking numbers is precisely how the zero-candidate run would still have read as a quiet week. ### Where it shows - `BacktestDataQualityCard` in the backtest panel, with the failure list behind a disclosure. - The "Last run" line now reads `505/506 tickers` — evaluated over expected, not the universe size. "506 tickers" on a run that evaluated none was the misleading part. - `productionBaseRates` returns `null` for a failed run, so per-ticker base rates quoted next to a live setup — which carry no caveat of their own — disappear rather than being sourced from a broken replay. - `EvaluationPanel` drops a failed run's expectancy as a drift baseline instead of reporting "tracking" or "drift" against it. - **A report with no `data_quality` block reads as unknown, not clean.** Old cached reports predate the check; defaulting them to complete would reproduce the exact bug being fixed. ### Acceptance evidence Both runs used `scripts/run_backtest_snapshot.py`. | Snapshot | Result | |---|---| | `survivorship-a.sqlite` (healthy) | `degraded` · 505/506 evaluated · 202,765 candidates · one bar-less row (RHM) | | the same file with `tickers.cik` dropped | `failed` · 0/506 evaluated · 506 fetch failures · 0 candidates | The broken-snapshot run is the one that matters: before this change it wrote a report that looked like an ordinary run with no setups. The healthy run is not `ok`, and that turned out to be correct. It flagged `no_data: 1` (RHM) and `short_history: 1` (SPCX, 14 bars), which an independent bar-count query confirms. Neither is new — Step 2 already recorded RHM as inert in `tr-rank-agreement.md`, which is why Step 3a generated candidates over 505 of 506 tickers. The ledger derives it independently, from the run rather than from a hand-written note, which is the point: the same 506-vs-505 gap now appears on the report instead of only in a research write-up someone remembered to make. `prod.sqlite` has the same bar-less RHM row, so **production runs will report `degraded` too** until that universe row is cleared — which is item 2's territory. That is deliberate: silencing a bar-less universe row would let a wholesale import failure read as clean. The reason names the symbols so it is a one-line fix rather than a standing nag. Tests: `tests/unit/test_backtest_data_quality.py` (10, covering each status, the ledger identity, the failure cap, symbol naming and the benchmark axis) and four end-to-end cases in `tests/unit/test_backtest_service.py`, including every-ticker-fails and missing-benchmark. --- ## 2. Preserve history when a stock leaves the tracked universe — done `bootstrap_universe(prune_missing=True)` ran `DELETE FROM tickers`, cascading OHLCV, setups, scores and S/R levels away for any symbol that had merely dropped out of the index. Migration 032's own comment predicted it: *"Pruning a symbol that is merely no longer an index constituent still destroys history; that needs a tracked/membership state separate from delisting."* ### The two states, kept apart Migration 035 adds `Ticker.untracked_on`, the sibling of `delisted_on`: | Column | Meaning | |---|---| | `delisted_on` | The security stopped trading | | `untracked_on` | It still trades; it left the index and we stopped scanning it | Neither deletes anything. `NULL` in both means "in today's live path". `active_only` — the single opt-in choke point already applied to scanning, ranking, scoring, breadth and ingestion — now filters on both, and honours `as_of` for both, so a historical query can still ask what was in the scanner on a given date. `prune_missing=True` now marks instead of deleting, and **clears the mark for symbols that have rejoined the universe**. That last part is the bug this refactor could have introduced: a returning name is already tracked, so it is neither added nor untracked, and without an explicit clear it would keep the flag from its earlier removal and stay invisible to the scanner forever. The `deleted` return key is now always `0` rather than repurposed — a caller reading `deleted: 12` must not be told rows were destroyed when they were not. `untracked`, `untracked_symbols` and `retracked` carry the real counts, and the admin toast reads "untracked N" instead of "deleted N". ### Historical eligibility stays explicit The backtest's universe query is now `_load_backtest_symbols`, a named function whose docstring states that it deliberately does **not** use `active_only`. Today's membership flag must not erase yesterday's candidates: a replay grades what the strategy could have selected at each past date, so it must see symbols that have since been delisted or dropped out. "Fixing the inconsistency" by adding `active_only` there is exactly how survivorship bias gets reintroduced, which is why a test asserts the opposite. The one path that still really deletes is `DELETE /tickers/{symbol}` — an explicit operator action asking for exactly that. Pruning is not. ### Untracking one symbol `POST /tickers/{symbol}/tracking` with `{"tracked": false}` excludes a single symbol from the live path, mirroring the delisting pair (both are API-only; no UI control exists for either). It exists because `prune_missing=True` can only act on the whole universe, and a manually added symbol is by definition *not* in the fetched constituent list — so a prune sweeps every manual addition along with the genuine leavers. Reversible with `{"tracked": true}`. The case it was built for: **RHM**. Added by hand on 2026-06-28 out of interest in the stock, it has produced zero rows in *every* ticker-linked table in the ten weeks since — no bars, no ingestion progress, no scores — and the Alpaca name backfill never resolved it. It is the only symbol in the universe that has produced nothing at all (BK, CTRA and SATS also lack Alpaca names but carry full history, so a missing name alone proves nothing). Rheinmetall is XETRA-listed, so a plain US `RHM` is not something the OHLCV provider can serve. An attempt to confirm this against Alpaca's asset API returned 401 for the AAPL control too, so the *identity* is unverified — the justification is the persistent absence of usable data, not a confirmed listing status or a delisting. **Untracking it does not clear the backtest's `degraded` flag, and must not.** The backtest deliberately includes untracked symbols, so RHM stays in the universe query, still returns no bars, and still counts in `no_data`. What untracking buys is narrower and still worth having: RHM leaves the scanner, ingestion and scoring, and the decision is recorded explicitly rather than sitting as an unexplained gap. Making the warning go away would mean changing historical eligibility to silence a real coverage gap — guarded by `test_untracking_does_not_clear_a_data_quality_warning`. Note what is *not* solved: a manual addition is still indistinguishable from a universe member, so a prune untracks it (non-destructively, and visibly via `untracked_symbols`, but silently in the sense that nobody asked). Recording *why* a symbol is tracked is item 3's job. ### Tests `tests/unit/test_universe_membership.py` (12) covers the three behaviours together: 1. Pruning preserves the symbol's bars **and** its linked records (S/R levels are checked too, so the assertion is about the cascade, not just one table). 2. The pruned symbol leaves `active_only` while staying visible in the registry. 3. `_load_backtest_symbols` still returns it. Plus re-tracking, the mark not being re-stamped on a second prune (which would erase when the symbol actually left), and `as_of` working for both columns. a row carrying *both* marks (it left the index, then delisted), and a compilation guard asserting every `active_only` call site still receives both predicates ANDed — including `scheduler.py`'s, which chains `group_by`/`order_by` onto the result from outside. `tests/unit/test_migration_035.py` exercises the migration's upgrade and downgrade directly, including that existing rows default to NULL — an upgrade must not drop the whole universe out of the live path. One detail worth keeping: `untracked_on` is stamped by the **database** (`func.current_date()`), not `date.today()`. `active_only` compares it against `func.current_date()`, so a process in a timezone ahead of the database would otherwise write a date that reads as "effective tomorrow" and leave the symbol in the scanner for a day. Same clock on both sides by construction. ## 3. Record universe membership by effective date — done Migration 036. Three things that were still conflated are now separate: | Concept | Where | Written by | |---|---|---| | Index membership, observed history | `universe_memberships` intervals | every **live** sync | | Excluded from scanning by a prune | `Ticker.untracked_on` | prune only | | Operator switched it off | `Ticker.scan_disabled_on` | `set_tracked` only | | Tracking ownership | `Ticker.tracking_source` | `add_ticker` / bootstrap | **Why the operator flag is its own column.** With one column serving both writers, a prune -> re-entry cycle would clear an explicit operator "off", because re-entry cannot tell who set it. Re-entry clears `untracked_on` and never touches `scan_disabled_on`. `active_only` now tests three columns and **stays join-free** — it is on ~8 live call sites, and the compilation guard asserts all three predicates land, ANDed, including at the call site that chains `group_by`/`order_by` on from outside. ### `untracked_on` is not current membership It records *exclusion by pruning*. The scheduled sync runs `prune_missing=False`, so a symbol can have left the index and still be scanned; the two states legitimately differ, and nothing asserts they agree. ### Tracking ownership is sticky towards manual `tracking_source` is `index` or `manual`. Prune touches only `index` rows, so **manual additions survive index pruning** — the RHM case. A bootstrap **never** converts `manual` to `index`. That direction looks harmless and is not: a hand-added symbol that happens to join the index would lose its protection the moment it left again, which is exactly when it is needed. Only new rows a bootstrap creates are stamped `index`; an explicit `add_ticker` on an existing index-owned symbol claims it as `manual`. `POST /tickers` therefore has three responses: | Case | Response | |---|---| | New symbol | Created as manually tracked | | Existing, index-owned | **200** with the existing row, now manually tracked | | Existing, already manual | 409 Duplicate | Claiming is not a 409 because the operator is saying "keep this stock", and refusing while leaving it prunable would defeat the protection they just asked for. It deliberately does not touch `scan_disabled_on` or `untracked_on` — wanting a stock kept is not the same as asking for it to be scanned, and silently re-enabling something switched off by hand would undo a decision nobody revisited. `POST /tickers/{symbol}/tracking` is the explicit control. ### Membership is observed, never asserted `first_seen_on` is the first date a live fetch actually listed the symbol — not an index effective date. Anything earlier is unknown and is not guessed; the hand-reconstructed `scripts/data/sp500_removals.json` stays a research artifact. Intervals are append-only: departure sets `absent_since` and **retains the row**, and a re-entry inserts a new one. A symbol that left in March and returned in August has two intervals, so a point-in-time query for May correctly answers "not a member" — reopening the original row would erase that. **Boundaries, stated once.** The column is named `absent_since`, not `last_seen_on`, because it stores the first observation that found the symbol *gone* — on that date it was already absent: ``` member on D <=> first_seen_on <= D < absent_since (or absent_since IS NULL) ``` Lower bound inclusive, upper bound exclusive. The real departure lies somewhere in `(previous observation, absent_since]` — a day with a daily sync, a week if a week was missed. The column records when we *looked*, which is the only thing actually known. `members_on(db, universe, as_of)` implements exactly this, so the rule lives in one place rather than being re-derived per call site. **A date before the first successful live sync returns an empty list.** That means "we were not looking", not "the index was empty", and a caller has to know the difference — answering with today's set would be precisely the survivorship error this exists to end. Recording runs on **every live sync**, independent of pruning and of who owns the row: the scheduled job uses `prune_missing=False`, so membership recorded only on manual prunes would be full of holes. Manually owned symbols get intervals like any other — membership and tracking ownership are independent axes. Switching off scanning does not close a membership; the index still contains the symbol, we simply stopped looking at it. **A failed fetch drives nothing at all.** `fetch_universe_symbols` falls back to a cached snapshot and then to a built-in seed list of ~44 symbols. Nothing that acts on "this symbol is or isn't in the index" may run on that data: not recording membership, **not pruning, and not clearing a prune flag**. Skipping only the membership write was half a safeguard — a seed fallback would still have untracked the rest of the universe, which a regression test caught (2 of 3 symbols untracked before the fix). Adding is the one exception: it destroys nothing and is what lets a cold start bootstrap from seeds. The summary reports `membership_observed: false` so a caller can tell "the index did not change" from "we never looked". ### Migration decisions - **Existing rows become `manual`.** Provenance cannot be recovered from `created_at`, and manual is fail-safe: a wrong guess costs an un-pruned symbol rather than one swept out of the scanner. Bootstraps claim the ones they own as they create them. - **Existing "off" states are carried into `scan_disabled_on`.** Whoever wrote an existing `untracked_on` is ambiguous, and re-entry clears that column — so the value is copied to the one no automated run touches. A symbol somebody deliberately switched off cannot be reactivated by a sync. - **No history is backfilled.** The table starts empty; history begins at the first observation. ### Tests `test_universe_membership.py` (19) and `test_universe_membership_intervals.py` (14): manual additions survive a prune; a bootstrap never claims a manual row, including across a join-then-leave cycle; an explicit add claims an index-owned symbol without touching its switches; re-entry clears the prune flag but never the operator's; a live sync opens intervals without pruning; departure closes and re-entry appends; a cache/seed fallback records nothing, untracks nothing and clears no prune flag; membership is recorded for manual rows; switching off scanning does not close a membership; and the boundary rules — lower inclusive, upper exclusive, gaps are not membership, per-universe scoping, and pre-recording dates returning empty. `test_migration_036.py` (4) pins both migration judgement calls and the downgrade. The fixtures pass a **live** source label deliberately: a made-up one would now skip pruning entirely and make those tests vacuous. Backtest eligibility is unchanged — `_load_backtest_symbols` still returns every symbol with stored history, and untracking still cannot silence a coverage gap.