signal-platform/docs/data-reliability.md

# Data reliability

Three changes that make the platform's data trustworthy rather than faster or
more profitable. They came out of the [integrity
experiments](data-integrity-plan.md), which turned up two problems worth fixing
on their own:

- Changing the stock universe **materially changes results** — Arm A 2.09 Sharpe
  / 51.6% CAGR against Arm B 1.57 / 38.1% over the same window, from universe
  composition alone ([survivorship audit](research/survivorship-audit.md)).
- **Missing data quietly produces a seemingly successful backtest.** Found while
  running Step 3a: a snapshot whose schema predated migrations 026/032 made every
  `_fetch_columns` call raise `no such column: tickers.cik`, both dispatch paths
  caught the per-ticker exception and continued, and the run wrote a complete
  report with zero candidates. Nothing in the report or the UI said the run had
  lost its entire universe.

| # | Change | Status |
|---|---|---|
| 1 | Make incomplete backtests visibly incomplete | **done** — 2026-09-07 |
| 2 | Preserve history when a stock leaves the tracked universe | **done** — 2026-09-07 |
| 3 | Record universe membership by effective date | **done** — 2026-09-07 |

---

## 1. Make incomplete backtests visibly incomplete — done

The replay loop is deliberately fault-tolerant: one unreadable ticker must not
abort a 500-symbol run. That is right for the run and wrong for its report,
which had no way to say how much of the universe it actually covered. A run that
lost every ticker and a run that honestly found no setups produced the same
report.

### What the report now carries

`run_backtest` builds a `_DataQuality` ledger
(`app/services/backtest_service.py`) and emits it as `report["data_quality"]`:

```json
{
  "status": "failed",
  "complete": false,
  "reasons": ["no symbol was evaluated successfully", "506 symbols failed to load"],
  "expected": 506, "loaded": 0, "evaluated": 0,
  "no_data": 0, "no_data_symbols": [], "short_history": 0,
  "failed": 506,
  "failures_by_stage": {"fetch": 506},
  "failures": [{"symbol": "A", "stage": "fetch",
                "error": "OperationalError: (sqlite3.OperationalError) no such column: tickers.cik"}],
  "failures_truncated": true,
  "benchmark": {"symbol": "SPY", "sessions": 1516,
                "first": "2020-07-06", "last": "2026-07-17",
                "momentum_basis": "residual", "error": null}
}
```

**"Complete" means the processing and coverage checks passed** — every expected
symbol loaded and evaluated, benchmark present. It does **not** mean the
underlying prices are proven correct. A complete run over silently wrong bars is
still complete; splits, adjustment drift and bad vendor prints are a different
problem, and this ledger must not be read as clearing them. The card says so on
the page, not only here.

**The counters are self-checking.** Every universe symbol ends in exactly one of
`no_data`, `failures` or `evaluated`, and `as_report` verifies
`no_data + failed + evaluated == expected`. A ledger that stops balancing —
because a counter was missed when a code path was added — reports itself as a
failure rather than quietly under-counting. `short_history` is a *subset* of
`evaluated` (loaded, but fewer than `MIN_LOOKBACK + HORIZON` bars, so no outcome
can resolve), not a fourth bucket.

**Benchmark availability is a separate axis.** Without SPY closes, residual 12-1
momentum silently falls back to raw 12-1 — the gate stops being the production
gate. The block records the basis actually used, so the degradation is on the
report instead of only in a log line.

### Three statuses, because severity differs

| Status | Meaning | UI |
|---|---|---|
| `ok` | Every symbol evaluated, benchmark present | Normal |
| `degraded` | The strategy is intact but ran on less than the whole universe | Tiles shown, banner above |
| `failed` | The headline can't be read as the strategy it claims to be | **Tiles replaced by the callout** |

`failed` means one of: nothing evaluated, an empty universe, an unbalanced
ledger, or a missing benchmark. Blanket suppression on a single missing ticker
would train the reader to ignore the banner, which is the failure mode this is
supposed to fix.

The callout **replaces** the performance tiles rather than sitting above them.
A warning over ordinary-looking numbers is precisely how the zero-candidate run
would still have read as a quiet week.

### Where it shows

- `BacktestDataQualityCard` in the backtest panel, with the failure list behind a
  disclosure.
- The "Last run" line now reads `505/506 tickers` — evaluated over expected, not
  the universe size. "506 tickers" on a run that evaluated none was the
  misleading part.
- `productionBaseRates` returns `null` for a failed run, so per-ticker base rates
  quoted next to a live setup — which carry no caveat of their own — disappear
  rather than being sourced from a broken replay.
- `EvaluationPanel` drops a failed run's expectancy as a drift baseline instead
  of reporting "tracking" or "drift" against it.
- **A report with no `data_quality` block reads as unknown, not clean.** Old
  cached reports predate the check; defaulting them to complete would reproduce
  the exact bug being fixed.

### Acceptance evidence

Both runs used `scripts/run_backtest_snapshot.py`.

| Snapshot | Result |
|---|---|
| `survivorship-a.sqlite` (healthy) | `degraded` · 505/506 evaluated · 202,765 candidates · one bar-less row (RHM) |
| the same file with `tickers.cik` dropped | `failed` · 0/506 evaluated · 506 fetch failures · 0 candidates |

The broken-snapshot run is the one that matters: before this change it wrote a
report that looked like an ordinary run with no setups.

The healthy run is not `ok`, and that turned out to be correct. It flagged
`no_data: 1` (RHM) and `short_history: 1` (SPCX, 14 bars), which an independent
bar-count query confirms. Neither is new — Step 2 already recorded RHM as inert
in `tr-rank-agreement.md`, which is why Step 3a generated candidates over 505 of
506 tickers. The ledger derives it independently, from the run rather than from
a hand-written note, which is the point: the same 506-vs-505 gap now appears on
the report instead of only in a research write-up someone remembered to make.

`prod.sqlite` has the same bar-less RHM row, so **production runs will report
`degraded` too** until that universe row is cleared — which is item 2's
territory. That is deliberate: silencing a bar-less universe row would let a
wholesale import failure read as clean. The reason names the symbols so it is a
one-line fix rather than a standing nag.

Tests: `tests/unit/test_backtest_data_quality.py` (10, covering each status, the
ledger identity, the failure cap, symbol naming and the benchmark axis) and four end-to-end
cases in `tests/unit/test_backtest_service.py`, including every-ticker-fails and
missing-benchmark.

---

## 2. Preserve history when a stock leaves the tracked universe — done

`bootstrap_universe(prune_missing=True)` ran `DELETE FROM tickers`, cascading
OHLCV, setups, scores and S/R levels away for any symbol that had merely dropped
out of the index. Migration 032's own comment predicted it: *"Pruning a symbol
that is merely no longer an index constituent still destroys history; that needs
a tracked/membership state separate from delisting."*

### The two states, kept apart

Migration 035 adds `Ticker.untracked_on`, the sibling of `delisted_on`:

| Column | Meaning |
|---|---|
| `delisted_on` | The security stopped trading |
| `untracked_on` | It still trades; it left the index and we stopped scanning it |

Neither deletes anything. `NULL` in both means "in today's live path".
`active_only` — the single opt-in choke point already applied to scanning,
ranking, scoring, breadth and ingestion — now filters on both, and honours
`as_of` for both, so a historical query can still ask what was in the scanner on
a given date.

`prune_missing=True` now marks instead of deleting, and **clears the mark for
symbols that have rejoined the universe**. That last part is the bug this
refactor could have introduced: a returning name is already tracked, so it is
neither added nor untracked, and without an explicit clear it would keep the
flag from its earlier removal and stay invisible to the scanner forever.

The `deleted` return key is now always `0` rather than repurposed — a caller
reading `deleted: 12` must not be told rows were destroyed when they were not.
`untracked`, `untracked_symbols` and `retracked` carry the real counts, and the
admin toast reads "untracked N" instead of "deleted N".

### Historical eligibility stays explicit

The backtest's universe query is now `_load_backtest_symbols`, a named function
whose docstring states that it deliberately does **not** use `active_only`.
Today's membership flag must not erase yesterday's candidates: a replay grades
what the strategy could have selected at each past date, so it must see symbols
that have since been delisted or dropped out. "Fixing the inconsistency" by
adding `active_only` there is exactly how survivorship bias gets reintroduced,
which is why a test asserts the opposite.

The one path that still really deletes is `DELETE /tickers/{symbol}` — an
explicit operator action asking for exactly that. Pruning is not.

### Untracking one symbol

`POST /tickers/{symbol}/tracking` with `{"tracked": false}` excludes a single
symbol from the live path, mirroring the delisting pair (both are API-only; no
UI control exists for either). It exists because `prune_missing=True` can only
act on the whole universe, and a manually added symbol is by definition *not* in
the fetched constituent list — so a prune sweeps every manual addition along
with the genuine leavers. Reversible with `{"tracked": true}`.

The case it was built for: **RHM**. Added by hand on 2026-06-28 out of interest
in the stock, it has produced zero rows in *every* ticker-linked table in the ten
weeks since — no bars, no ingestion progress, no scores — and the Alpaca name
backfill never resolved it. It is the only symbol in the universe that has
produced nothing at all (BK, CTRA and SATS also lack Alpaca names but carry full
history, so a missing name alone proves nothing). Rheinmetall is XETRA-listed, so
a plain US `RHM` is not something the OHLCV provider can serve. An attempt to
confirm this against Alpaca's asset API returned 401 for the AAPL control too,
so the *identity* is unverified — the justification is the persistent absence of
usable data, not a confirmed listing status or a delisting.

**Untracking it does not clear the backtest's `degraded` flag, and must not.**
The backtest deliberately includes untracked symbols, so RHM stays in the
universe query, still returns no bars, and still counts in `no_data`. What
untracking buys is narrower and still worth having: RHM leaves the scanner,
ingestion and scoring, and the decision is recorded explicitly rather than
sitting as an unexplained gap. Making the warning go away would mean changing
historical eligibility to silence a real coverage gap — guarded by
`test_untracking_does_not_clear_a_data_quality_warning`.

Note what is *not* solved: a manual addition is still indistinguishable from a
universe member, so a prune untracks it (non-destructively, and visibly via
`untracked_symbols`, but silently in the sense that nobody asked). Recording
*why* a symbol is tracked is item 3's job.

### Tests

`tests/unit/test_universe_membership.py` (12) covers the three behaviours
together:

1. Pruning preserves the symbol's bars **and** its linked records (S/R levels
   are checked too, so the assertion is about the cascade, not just one table).
2. The pruned symbol leaves `active_only` while staying visible in the registry.
3. `_load_backtest_symbols` still returns it.

Plus re-tracking, the mark not being re-stamped on a second prune (which would
erase when the symbol actually left), and `as_of` working for both columns.
a row carrying *both* marks (it left the index, then delisted), and a
compilation guard asserting every `active_only` call site still receives both
predicates ANDed — including `scheduler.py`'s, which chains `group_by`/`order_by`
onto the result from outside. `tests/unit/test_migration_035.py` exercises the
migration's upgrade and downgrade directly, including that existing rows default
to NULL — an upgrade must not drop the whole universe out of the live path.

One detail worth keeping: `untracked_on` is stamped by the **database**
(`func.current_date()`), not `date.today()`. `active_only` compares it against
`func.current_date()`, so a process in a timezone ahead of the database would
otherwise write a date that reads as "effective tomorrow" and leave the symbol
in the scanner for a day. Same clock on both sides by construction.

## 3. Record universe membership by effective date — done

Migration 036. Three things that were still conflated are now separate:

| Concept | Where | Written by |
|---|---|---|
| Index membership, observed history | `universe_memberships` intervals | every **live** sync |
| Excluded from scanning by a prune | `Ticker.untracked_on` | prune only |
| Operator switched it off | `Ticker.scan_disabled_on` | `set_tracked` only |
| Tracking ownership | `Ticker.tracking_source` | `add_ticker` / bootstrap |

**Why the operator flag is its own column.** With one column serving both
writers, a prune -> re-entry cycle would clear an explicit operator "off",
because re-entry cannot tell who set it. Re-entry clears `untracked_on` and
never touches `scan_disabled_on`.

`active_only` now tests three columns and **stays join-free** — it is on ~8 live
call sites, and the compilation guard asserts all three predicates land, ANDed,
including at the call site that chains `group_by`/`order_by` on from outside.

### `untracked_on` is not current membership

It records *exclusion by pruning*. The scheduled sync runs
`prune_missing=False`, so a symbol can have left the index and still be scanned;
the two states legitimately differ, and nothing asserts they agree.

### Tracking ownership is sticky towards manual

`tracking_source` is `index` or `manual`. Prune touches only `index` rows, so
**manual additions survive index pruning** — the RHM case.

A bootstrap **never** converts `manual` to `index`. That direction looks harmless
and is not: a hand-added symbol that happens to join the index would lose its
protection the moment it left again, which is exactly when it is needed. Only
new rows a bootstrap creates are stamped `index`; an explicit `add_ticker` on an
existing index-owned symbol claims it as `manual`.

`POST /tickers` therefore has three responses:

| Case | Response |
|---|---|
| New symbol | Created as manually tracked |
| Existing, index-owned | **200** with the existing row, now manually tracked |
| Existing, already manual | 409 Duplicate |

Claiming is not a 409 because the operator is saying "keep this stock", and
refusing while leaving it prunable would defeat the protection they just asked
for. It deliberately does not touch `scan_disabled_on` or `untracked_on` —
wanting a stock kept is not the same as asking for it to be scanned, and
silently re-enabling something switched off by hand would undo a decision nobody
revisited. `POST /tickers/{symbol}/tracking` is the explicit control.

### Membership is observed, never asserted

`first_seen_on` is the first date a live fetch actually listed the symbol — not
an index effective date. Anything earlier is unknown and is not guessed; the
hand-reconstructed `scripts/data/sp500_removals.json` stays a research artifact.

Intervals are append-only: departure sets `absent_since` and **retains the row**,
and a re-entry inserts a new one. A symbol that left in March and returned in
August has two intervals, so a point-in-time query for May correctly answers
"not a member" — reopening the original row would erase that.

**Boundaries, stated once.** The column is named `absent_since`, not
`last_seen_on`, because it stores the first observation that found the symbol
*gone* — on that date it was already absent:

```
member on D  <=>  first_seen_on <= D < absent_since  (or absent_since IS NULL)
```

Lower bound inclusive, upper bound exclusive. The real departure lies somewhere
in `(previous observation, absent_since]` — a day with a daily sync, a week if a
week was missed. The column records when we *looked*, which is the only thing
actually known. `members_on(db, universe, as_of)` implements exactly this, so
the rule lives in one place rather than being re-derived per call site.

**A date before the first successful live sync returns an empty list.** That
means "we were not looking", not "the index was empty", and a caller has to know
the difference — answering with today's set would be precisely the survivorship
error this exists to end.

Recording runs on **every live sync**, independent of pruning and of who owns
the row: the scheduled job uses `prune_missing=False`, so membership recorded
only on manual prunes would be full of holes. Manually owned symbols get
intervals like any other — membership and tracking ownership are independent
axes. Switching off scanning does not close a membership; the index still
contains the symbol, we simply stopped looking at it.

**A failed fetch drives nothing at all.** `fetch_universe_symbols` falls back to
a cached snapshot and then to a built-in seed list of ~44 symbols. Nothing that
acts on "this symbol is or isn't in the index" may run on that data: not
recording membership, **not pruning, and not clearing a prune flag**. Skipping
only the membership write was half a safeguard — a seed fallback would still
have untracked the rest of the universe, which a regression test caught
(2 of 3 symbols untracked before the fix). Adding is the one exception: it
destroys nothing and is what lets a cold start bootstrap from seeds. The summary
reports `membership_observed: false` so a caller can tell "the index did not
change" from "we never looked".

### Migration decisions

- **Existing rows become `manual`.** Provenance cannot be recovered from
  `created_at`, and manual is fail-safe: a wrong guess costs an un-pruned symbol
  rather than one swept out of the scanner. Bootstraps claim the ones they own
  as they create them.
- **Existing "off" states are carried into `scan_disabled_on`.** Whoever wrote an
  existing `untracked_on` is ambiguous, and re-entry clears that column — so the
  value is copied to the one no automated run touches. A symbol somebody
  deliberately switched off cannot be reactivated by a sync.
- **No history is backfilled.** The table starts empty; history begins at the
  first observation.

### Tests

`test_universe_membership.py` (19) and `test_universe_membership_intervals.py`
(14): manual additions survive a prune; a bootstrap never claims a manual row,
including across a join-then-leave cycle; an explicit add claims an index-owned
symbol without touching its switches; re-entry clears the prune flag but never
the operator's; a live sync opens intervals without pruning; departure closes and
re-entry appends; a cache/seed fallback records nothing, untracks nothing and
clears no prune flag; membership is recorded for manual rows; switching off
scanning does not close a membership; and the boundary rules — lower inclusive,
upper exclusive, gaps are not membership, per-universe scoping, and pre-recording
dates returning empty. `test_migration_036.py` (4) pins both migration judgement
calls and the downgrade.

The fixtures pass a **live** source label deliberately: a made-up one would now
skip pruning entirely and make those tests vacuous.

Backtest eligibility is unchanged — `_load_backtest_symbols` still returns every
symbol with stored history, and untracking still cannot silence a coverage gap.