feat(universe): separate scanner membership from retained history

bootstrap_universe(prune_missing=True) ran DELETE FROM tickers, cascading OHLCV,
setups, scores and S/R levels away for any symbol that had merely dropped out of
the index. Migration 032's own comment predicted this. The survivorship audit
measured what it costs: 2.09 Sharpe against 1.57 on universe composition alone.

Migration 035 adds untracked_on, the sibling of delisted_on — delisted means the
security stopped trading, untracked means a prune took it out of the scanner.
Nothing is deleted either way.

Migration 036 separates three more things that were still conflated:

  universe_memberships       what the index actually contained, when
  tickers.untracked_on       excluded from scanning by a prune
  tickers.scan_disabled_on   the operator's own "stop scanning this"
  tickers.tracking_source    who owns the row: a bootstrap, or an operator

The operator flag is its own column because with one column serving both
writers, a prune -> re-entry cycle would clear an explicit "off" — re-entry
cannot tell who set it. Re-entry clears untracked_on and never touches
scan_disabled_on.

untracked_on is NOT current index membership. The scheduled sync runs
prune_missing=False, so a symbol can have left the index and still be scanned;
the two legitimately differ and nothing asserts they agree.

Tracking ownership is sticky towards manual: prune touches index-owned rows
only, and a bootstrap never converts manual to index — a hand-added symbol that
joins the index would otherwise lose its protection the moment it left again,
which is exactly when it is needed. An explicit add of an index-owned symbol
claims it as manual without touching either off-switch.

Membership is observed, never asserted. first_seen_on is the first live
sighting; absent_since is the first observation that found the symbol GONE, so
the interval is [first_seen_on, absent_since) — named for what it stores rather
than "last_seen_on", which would have claimed the opposite. Intervals are
append-only, so a symbol that leaves and returns has two and a point-in-time
query for the gap correctly answers "not a member". members_on() implements the
boundary rule once. A date before the first live sync returns empty: that means
"we were not looking", not "the index was empty".

A failed fetch drives nothing. The cache and seed fallbacks mean the live fetch
failed, and the seed lists are ~44 symbols against a ~500-symbol universe — a
regression test caught a seed fallback untracking 2 of 3 tickers. Recording,
pruning and clearing prune flags all require a live source; only adding still
runs, so a cold start can still bootstrap from seeds.

Migration decisions: existing rows become manual (fail-safe — a wrong guess
costs an un-pruned symbol, not one swept out of the scanner), and existing
untracked_on values are copied into scan_disabled_on, since the original writer
is ambiguous and re-entry clears the prune column.

Backtest eligibility is unchanged. _load_backtest_symbols deliberately does not
use active_only, so today's scanner state cannot erase yesterday's candidates,
and untracking a symbol still cannot silence a coverage gap in the report.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HAH7vVtn7qbcn1EVheecHQ

docs/data-reliability.md

@@ -0,0 +1,380 @@
+# Data reliability
+
+Three changes that make the platform's data trustworthy rather than faster or
+more profitable. They came out of the [integrity
+experiments](data-integrity-plan.md), which turned up two problems worth fixing
+on their own:
+
+- Changing the stock universe **materially changes results** — Arm A 2.09 Sharpe
+  / 51.6% CAGR against Arm B 1.57 / 38.1% over the same window, from universe
+  composition alone ([survivorship audit](research/survivorship-audit.md)).
+- **Missing data quietly produces a seemingly successful backtest.** Found while
+  running Step 3a: a snapshot whose schema predated migrations 026/032 made every
+  `_fetch_columns` call raise `no such column: tickers.cik`, both dispatch paths
+  caught the per-ticker exception and continued, and the run wrote a complete
+  report with zero candidates. Nothing in the report or the UI said the run had
+  lost its entire universe.
+
+| # | Change | Status |
+|---|---|---|
+| 1 | Make incomplete backtests visibly incomplete | **done** — 2026-09-07 |
+| 2 | Preserve history when a stock leaves the tracked universe | **done** — 2026-09-07 |
+| 3 | Record universe membership by effective date | **done** — 2026-09-07 |
+
+---
+
+## 1. Make incomplete backtests visibly incomplete — done
+
+The replay loop is deliberately fault-tolerant: one unreadable ticker must not
+abort a 500-symbol run. That is right for the run and wrong for its report,
+which had no way to say how much of the universe it actually covered. A run that
+lost every ticker and a run that honestly found no setups produced the same
+report.
+
+### What the report now carries
+
+`run_backtest` builds a `_DataQuality` ledger
+(`app/services/backtest_service.py`) and emits it as `report["data_quality"]`:
+
+```json
+{
+  "status": "failed",
+  "complete": false,
+  "reasons": ["no symbol was evaluated successfully", "506 symbols failed to load"],
+  "expected": 506, "loaded": 0, "evaluated": 0,
+  "no_data": 0, "no_data_symbols": [], "short_history": 0,
+  "failed": 506,
+  "failures_by_stage": {"fetch": 506},
+  "failures": [{"symbol": "A", "stage": "fetch",
+                "error": "OperationalError: (sqlite3.OperationalError) no such column: tickers.cik"}],
+  "failures_truncated": true,
+  "benchmark": {"symbol": "SPY", "sessions": 1516,
+                "first": "2020-07-06", "last": "2026-07-17",
+                "momentum_basis": "residual", "error": null}
+}
+```
+
+**"Complete" means the processing and coverage checks passed** — every expected
+symbol loaded and evaluated, benchmark present. It does **not** mean the
+underlying prices are proven correct. A complete run over silently wrong bars is
+still complete; splits, adjustment drift and bad vendor prints are a different
+problem, and this ledger must not be read as clearing them. The card says so on
+the page, not only here.
+
+**The counters are self-checking.** Every universe symbol ends in exactly one of
+`no_data`, `failures` or `evaluated`, and `as_report` verifies
+`no_data + failed + evaluated == expected`. A ledger that stops balancing —
+because a counter was missed when a code path was added — reports itself as a
+failure rather than quietly under-counting. `short_history` is a *subset* of
+`evaluated` (loaded, but fewer than `MIN_LOOKBACK + HORIZON` bars, so no outcome
+can resolve), not a fourth bucket.
+
+**Benchmark availability is a separate axis.** Without SPY closes, residual 12-1
+momentum silently falls back to raw 12-1 — the gate stops being the production
+gate. The block records the basis actually used, so the degradation is on the
+report instead of only in a log line.
+
+### Three statuses, because severity differs
+
+| Status | Meaning | UI |
+|---|---|---|
+| `ok` | Every symbol evaluated, benchmark present | Normal |
+| `degraded` | The strategy is intact but ran on less than the whole universe | Tiles shown, banner above |
+| `failed` | The headline can't be read as the strategy it claims to be | **Tiles replaced by the callout** |
+
+`failed` means one of: nothing evaluated, an empty universe, an unbalanced
+ledger, or a missing benchmark. Blanket suppression on a single missing ticker
+would train the reader to ignore the banner, which is the failure mode this is
+supposed to fix.
+
+The callout **replaces** the performance tiles rather than sitting above them.
+A warning over ordinary-looking numbers is precisely how the zero-candidate run
+would still have read as a quiet week.
+
+### Where it shows
+
+- `BacktestDataQualityCard` in the backtest panel, with the failure list behind a
+  disclosure.
+- The "Last run" line now reads `505/506 tickers` — evaluated over expected, not
+  the universe size. "506 tickers" on a run that evaluated none was the
+  misleading part.
+- `productionBaseRates` returns `null` for a failed run, so per-ticker base rates
+  quoted next to a live setup — which carry no caveat of their own — disappear
+  rather than being sourced from a broken replay.
+- `EvaluationPanel` drops a failed run's expectancy as a drift baseline instead
+  of reporting "tracking" or "drift" against it.
+- **A report with no `data_quality` block reads as unknown, not clean.** Old
+  cached reports predate the check; defaulting them to complete would reproduce
+  the exact bug being fixed.
+
+### Acceptance evidence
+
+Both runs used `scripts/run_backtest_snapshot.py`.
+
+| Snapshot | Result |
+|---|---|
+| `survivorship-a.sqlite` (healthy) | `degraded` · 505/506 evaluated · 202,765 candidates · one bar-less row (RHM) |
+| the same file with `tickers.cik` dropped | `failed` · 0/506 evaluated · 506 fetch failures · 0 candidates |
+
+The broken-snapshot run is the one that matters: before this change it wrote a
+report that looked like an ordinary run with no setups.
+
+The healthy run is not `ok`, and that turned out to be correct. It flagged
+`no_data: 1` (RHM) and `short_history: 1` (SPCX, 14 bars), which an independent
+bar-count query confirms. Neither is new — Step 2 already recorded RHM as inert
+in `tr-rank-agreement.md`, which is why Step 3a generated candidates over 505 of
+506 tickers. The ledger derives it independently, from the run rather than from
+a hand-written note, which is the point: the same 506-vs-505 gap now appears on
+the report instead of only in a research write-up someone remembered to make.
+
+`prod.sqlite` has the same bar-less RHM row, so **production runs will report
+`degraded` too** until that universe row is cleared — which is item 2's
+territory. That is deliberate: silencing a bar-less universe row would let a
+wholesale import failure read as clean. The reason names the symbols so it is a
+one-line fix rather than a standing nag.
+
+Tests: `tests/unit/test_backtest_data_quality.py` (10, covering each status, the
+ledger identity, the failure cap, symbol naming and the benchmark axis) and four end-to-end
+cases in `tests/unit/test_backtest_service.py`, including every-ticker-fails and
+missing-benchmark.
+
+---
+
+## 2. Preserve history when a stock leaves the tracked universe — done
+
+`bootstrap_universe(prune_missing=True)` ran `DELETE FROM tickers`, cascading
+OHLCV, setups, scores and S/R levels away for any symbol that had merely dropped
+out of the index. Migration 032's own comment predicted it: *"Pruning a symbol
+that is merely no longer an index constituent still destroys history; that needs
+a tracked/membership state separate from delisting."*
+
+### The two states, kept apart
+
+Migration 035 adds `Ticker.untracked_on`, the sibling of `delisted_on`:
+
+| Column | Meaning |
+|---|---|
+| `delisted_on` | The security stopped trading |
+| `untracked_on` | It still trades; it left the index and we stopped scanning it |
+
+Neither deletes anything. `NULL` in both means "in today's live path".
+`active_only` — the single opt-in choke point already applied to scanning,
+ranking, scoring, breadth and ingestion — now filters on both, and honours
+`as_of` for both, so a historical query can still ask what was in the scanner on
+a given date.
+
+`prune_missing=True` now marks instead of deleting, and **clears the mark for
+symbols that have rejoined the universe**. That last part is the bug this
+refactor could have introduced: a returning name is already tracked, so it is
+neither added nor untracked, and without an explicit clear it would keep the
+flag from its earlier removal and stay invisible to the scanner forever.
+
+The `deleted` return key is now always `0` rather than repurposed — a caller
+reading `deleted: 12` must not be told rows were destroyed when they were not.
+`untracked`, `untracked_symbols` and `retracked` carry the real counts, and the
+admin toast reads "untracked N" instead of "deleted N".
+
+### Historical eligibility stays explicit
+
+The backtest's universe query is now `_load_backtest_symbols`, a named function
+whose docstring states that it deliberately does **not** use `active_only`.
+Today's membership flag must not erase yesterday's candidates: a replay grades
+what the strategy could have selected at each past date, so it must see symbols
+that have since been delisted or dropped out. "Fixing the inconsistency" by
+adding `active_only` there is exactly how survivorship bias gets reintroduced,
+which is why a test asserts the opposite.
+
+The one path that still really deletes is `DELETE /tickers/{symbol}` — an
+explicit operator action asking for exactly that. Pruning is not.
+
+### Untracking one symbol
+
+`POST /tickers/{symbol}/tracking` with `{"tracked": false}` excludes a single
+symbol from the live path, mirroring the delisting pair (both are API-only; no
+UI control exists for either). It exists because `prune_missing=True` can only
+act on the whole universe, and a manually added symbol is by definition *not* in
+the fetched constituent list — so a prune sweeps every manual addition along
+with the genuine leavers. Reversible with `{"tracked": true}`.
+
+The case it was built for: **RHM**. Added by hand on 2026-06-28 out of interest
+in the stock, it has produced zero rows in *every* ticker-linked table in the ten
+weeks since — no bars, no ingestion progress, no scores — and the Alpaca name
+backfill never resolved it. It is the only symbol in the universe that has
+produced nothing at all (BK, CTRA and SATS also lack Alpaca names but carry full
+history, so a missing name alone proves nothing). Rheinmetall is XETRA-listed, so
+a plain US `RHM` is not something the OHLCV provider can serve. An attempt to
+confirm this against Alpaca's asset API returned 401 for the AAPL control too,
+so the *identity* is unverified — the justification is the persistent absence of
+usable data, not a confirmed listing status or a delisting.
+
+**Untracking it does not clear the backtest's `degraded` flag, and must not.**
+The backtest deliberately includes untracked symbols, so RHM stays in the
+universe query, still returns no bars, and still counts in `no_data`. What
+untracking buys is narrower and still worth having: RHM leaves the scanner,
+ingestion and scoring, and the decision is recorded explicitly rather than
+sitting as an unexplained gap. Making the warning go away would mean changing
+historical eligibility to silence a real coverage gap — guarded by
+`test_untracking_does_not_clear_a_data_quality_warning`.
+
+Note what is *not* solved: a manual addition is still indistinguishable from a
+universe member, so a prune untracks it (non-destructively, and visibly via
+`untracked_symbols`, but silently in the sense that nobody asked). Recording
+*why* a symbol is tracked is item 3's job.
+
+### Tests
+
+`tests/unit/test_universe_membership.py` (12) covers the three behaviours
+together:
+
+1. Pruning preserves the symbol's bars **and** its linked records (S/R levels
+   are checked too, so the assertion is about the cascade, not just one table).
+2. The pruned symbol leaves `active_only` while staying visible in the registry.
+3. `_load_backtest_symbols` still returns it.
+
+Plus re-tracking, the mark not being re-stamped on a second prune (which would
+erase when the symbol actually left), and `as_of` working for both columns.
+a row carrying *both* marks (it left the index, then delisted), and a
+compilation guard asserting every `active_only` call site still receives both
+predicates ANDed — including `scheduler.py`'s, which chains `group_by`/`order_by`
+onto the result from outside. `tests/unit/test_migration_035.py` exercises the
+migration's upgrade and downgrade directly, including that existing rows default
+to NULL — an upgrade must not drop the whole universe out of the live path.
+
+One detail worth keeping: `untracked_on` is stamped by the **database**
+(`func.current_date()`), not `date.today()`. `active_only` compares it against
+`func.current_date()`, so a process in a timezone ahead of the database would
+otherwise write a date that reads as "effective tomorrow" and leave the symbol
+in the scanner for a day. Same clock on both sides by construction.
+
+## 3. Record universe membership by effective date — done
+
+Migration 036. Three things that were still conflated are now separate:
+
+| Concept | Where | Written by |
+|---|---|---|
+| Index membership, observed history | `universe_memberships` intervals | every **live** sync |
+| Excluded from scanning by a prune | `Ticker.untracked_on` | prune only |
+| Operator switched it off | `Ticker.scan_disabled_on` | `set_tracked` only |
+| Tracking ownership | `Ticker.tracking_source` | `add_ticker` / bootstrap |
+
+**Why the operator flag is its own column.** With one column serving both
+writers, a prune -> re-entry cycle would clear an explicit operator "off",
+because re-entry cannot tell who set it. Re-entry clears `untracked_on` and
+never touches `scan_disabled_on`.
+
+`active_only` now tests three columns and **stays join-free** — it is on ~8 live
+call sites, and the compilation guard asserts all three predicates land, ANDed,
+including at the call site that chains `group_by`/`order_by` on from outside.
+
+### `untracked_on` is not current membership
+
+It records *exclusion by pruning*. The scheduled sync runs
+`prune_missing=False`, so a symbol can have left the index and still be scanned;
+the two states legitimately differ, and nothing asserts they agree.
+
+### Tracking ownership is sticky towards manual
+
+`tracking_source` is `index` or `manual`. Prune touches only `index` rows, so
+**manual additions survive index pruning** — the RHM case.
+
+A bootstrap **never** converts `manual` to `index`. That direction looks harmless
+and is not: a hand-added symbol that happens to join the index would lose its
+protection the moment it left again, which is exactly when it is needed. Only
+new rows a bootstrap creates are stamped `index`; an explicit `add_ticker` on an
+existing index-owned symbol claims it as `manual`.
+
+`POST /tickers` therefore has three responses:
+
+| Case | Response |
+|---|---|
+| New symbol | Created as manually tracked |
+| Existing, index-owned | **200** with the existing row, now manually tracked |
+| Existing, already manual | 409 Duplicate |
+
+Claiming is not a 409 because the operator is saying "keep this stock", and
+refusing while leaving it prunable would defeat the protection they just asked
+for. It deliberately does not touch `scan_disabled_on` or `untracked_on` —
+wanting a stock kept is not the same as asking for it to be scanned, and
+silently re-enabling something switched off by hand would undo a decision nobody
+revisited. `POST /tickers/{symbol}/tracking` is the explicit control.
+
+### Membership is observed, never asserted
+
+`first_seen_on` is the first date a live fetch actually listed the symbol — not
+an index effective date. Anything earlier is unknown and is not guessed; the
+hand-reconstructed `scripts/data/sp500_removals.json` stays a research artifact.
+
+Intervals are append-only: departure sets `absent_since` and **retains the row**,
+and a re-entry inserts a new one. A symbol that left in March and returned in
+August has two intervals, so a point-in-time query for May correctly answers
+"not a member" — reopening the original row would erase that.
+
+**Boundaries, stated once.** The column is named `absent_since`, not
+`last_seen_on`, because it stores the first observation that found the symbol
+*gone* — on that date it was already absent:
+
+```
+member on D  <=>  first_seen_on <= D < absent_since  (or absent_since IS NULL)
+```
+
+Lower bound inclusive, upper bound exclusive. The real departure lies somewhere
+in `(previous observation, absent_since]` — a day with a daily sync, a week if a
+week was missed. The column records when we *looked*, which is the only thing
+actually known. `members_on(db, universe, as_of)` implements exactly this, so
+the rule lives in one place rather than being re-derived per call site.
+
+**A date before the first successful live sync returns an empty list.** That
+means "we were not looking", not "the index was empty", and a caller has to know
+the difference — answering with today's set would be precisely the survivorship
+error this exists to end.
+
+Recording runs on **every live sync**, independent of pruning and of who owns
+the row: the scheduled job uses `prune_missing=False`, so membership recorded
+only on manual prunes would be full of holes. Manually owned symbols get
+intervals like any other — membership and tracking ownership are independent
+axes. Switching off scanning does not close a membership; the index still
+contains the symbol, we simply stopped looking at it.
+
+**A failed fetch drives nothing at all.** `fetch_universe_symbols` falls back to
+a cached snapshot and then to a built-in seed list of ~44 symbols. Nothing that
+acts on "this symbol is or isn't in the index" may run on that data: not
+recording membership, **not pruning, and not clearing a prune flag**. Skipping
+only the membership write was half a safeguard — a seed fallback would still
+have untracked the rest of the universe, which a regression test caught
+(2 of 3 symbols untracked before the fix). Adding is the one exception: it
+destroys nothing and is what lets a cold start bootstrap from seeds. The summary
+reports `membership_observed: false` so a caller can tell "the index did not
+change" from "we never looked".
+
+### Migration decisions
+
+- **Existing rows become `manual`.** Provenance cannot be recovered from
+  `created_at`, and manual is fail-safe: a wrong guess costs an un-pruned symbol
+  rather than one swept out of the scanner. Bootstraps claim the ones they own
+  as they create them.
+- **Existing "off" states are carried into `scan_disabled_on`.** Whoever wrote an
+  existing `untracked_on` is ambiguous, and re-entry clears that column — so the
+  value is copied to the one no automated run touches. A symbol somebody
+  deliberately switched off cannot be reactivated by a sync.
+- **No history is backfilled.** The table starts empty; history begins at the
+  first observation.
+
+### Tests
+
+`test_universe_membership.py` (19) and `test_universe_membership_intervals.py`
+(14): manual additions survive a prune; a bootstrap never claims a manual row,
+including across a join-then-leave cycle; an explicit add claims an index-owned
+symbol without touching its switches; re-entry clears the prune flag but never
+the operator's; a live sync opens intervals without pruning; departure closes and
+re-entry appends; a cache/seed fallback records nothing, untracks nothing and
+clears no prune flag; membership is recorded for manual rows; switching off
+scanning does not close a membership; and the boundary rules — lower inclusive,
+upper exclusive, gaps are not membership, per-universe scoping, and pre-recording
+dates returning empty. `test_migration_036.py` (4) pins both migration judgement
+calls and the downgrade.
+
+The fixtures pass a **live** source label deliberately: a made-up one would now
+skip pruning entirely and make those tests vacuous.
+
+Backtest eligibility is unchanged — `_load_backtest_symbols` still returns every
+symbol with stored history, and untracking still cannot silence a coverage gap.