signal-platform/docs/data-integrity-plan.md

# Data & signal integrity — implementation plan

Status: **closed, 2026-09-07.** Steps 1, 2 and 3a ran; Step 3b was not started
because 3a did not clear its bar. Findings are in
[`research/survivorship-audit.md`](research/survivorship-audit.md),
[`research/tr-rank-agreement.md`](research/tr-rank-agreement.md) and
[`research/tr-total-return-ab.md`](research/tr-total-return-ab.md). The follow-on
work this raised is in [`data-reliability.md`](data-reliability.md).

Two questions, answered in order, with the production work made conditional on
the answers.

## Objective

1. Measure how the backtest changes when removed index constituents are restored.
   This is a survivorship sensitivity check, not a complete bias estimate.
2. Check whether total-return momentum changes the daily admission or ordering
   of names. Build the production path only if this screening test finds changes.

**Guiding principle: KISS — measure before building.** Every earlier draft of this
plan started with a schema change (`adj_close`, then `tr_factor`), an ingestion
maintenance path, a snapshot migration and a benchmark column, in order to deliver
a signal change whose effect nobody has measured. Both questions above can be
investigated by research scripts against a snapshot copy. Step 1 needs one small,
explicit replay option to include entries immediately before a terminal event.
No production schema or ingestion changes are needed for either research step.
Step 3 is not started until Step 2 finds a ranking or admission difference.

Why the research scripts need none of that machinery: the whole forward-anchored
factor design exists because *production* ingest is incremental, so back-adjusted
prices drift between fetches. A research script fetches each ticker's whole history
in **one** request, and within a single response the adjustment is internally
consistent. No column, no anchor, no maintenance.

---

## Step 1 — How sensitive is the backtest to omitted constituents?

Arm A uses the universe already present in the base snapshot.
Arm B adds the names that were removed from the S&P 500 since the snapshot's start
— including the ones that failed — and lets the strategy trade them.
Use the same configuration, cadence and evaluation dates in both arms. This is
deliberately an expanded-universe experiment, without historical membership gating.

### Files

- **[NEW] `scripts/data/sp500_removals.json`** — flat list, hand-curated from
  Wikipedia's S&P 500 changes table, checked into git:
  `[{"symbol": "SIVB", "removed_on": "2023-03-15", "reason": "receivership"}, ...]`
  Reasons: `receivership` | `bankruptcy` | `acquisition` | `market_cap`.
  No additions, no rename map, no membership intervals — see *Not doing*.
- **[NEW] `scripts/prep_survivorship_snapshot.py`** — makes both arm files from a
  base snapshot, leaving the base untouched:
  1. Copy base → `survivorship-a.sqlite` and `survivorship-b.sqlite`.
  2. **Schema sync** on both: add every column present in `Base.metadata` but
     missing from the file, nullable. The snapshots on disk predate migrations 026
     and 032, so `select(Ticker)` against them fails today with
     `no such column: tickers.cik` — and `_get_ticker` (`price_service.py:22`) is a
     full-entity select that `query_ohlcv` → `backtest_service._fetch_columns`
     reaches. Derive the column list from the metadata, not a hardcoded list, so
     migration 036 doesn't break it again.
  3. Into arm B only: for each removed symbol with no bars, fetch daily bars from
     Alpaca (it still serves delisted symbols — verified for SIVB, FRC, ATVI, TWTR)
     and insert the ticker row. Restrict downloaded bars to the base snapshot's
     date range and use the same price adjustment as its existing bars. Log fetched,
     already-present and unavailable symbols; an unavailable failure must be called
     out as missing coverage, not silently treated as having no strategy exposure.
     `removed_on` is an index-removal date, not necessarily a delisting date.
     **Nothing is inferred from missing bars**: `delisted_on` / `delisted_reason`
     are written only where the dataset carries a confirmed terminal event. A
     series that merely ends early leaves them NULL and is reported as an
     unresolved *ending status* — as built, that is CMA, SEE, HBI and DISH.
     Series are shortened only at a **reviewed cutoff** listed in the dataset
     (SBNY 2023-03-10, INFO 2022-02-25). Elsewhere a run of ≥5 zero-volume
     sessions is a tripwire, not a rule: the symbol is left out of the arm and
     reported for review rather than silently truncated.
  4. **Terminal mark**, arm B only: for a confirmed terminal event, append one
     synthetic bar on the next SPY trading session after the last real bar, with
     `o=h=l=c=0.01, volume=0`. A mark is written only when the retained final bar
     equals the confirmation's reviewed `last_real_bar`; a disagreement writes no
     mark and is reported. Write a sidecar JSON mapping symbol to terminal-bar
     date for the runner.
     The portfolio sim already fills a gap through the stop at the open
     (`backtest_service.py` ~2280), so the position resolves at ~-100% with no
     change to its stop-fill logic. Treat 0.01 as a near-total-loss assumption,
     not an observed execution. For acquisitions, retain the last available close
     as an approximation; this experiment does not model cash/stock consideration
     or release of capital on the acquisition date.
     This matters: failures halt mid-collapse (SIVB's last exchange bar is 106.04
     on the way to zero), so without it arm B spares exactly the trades it exists
     to measure. **Appending the bar alone is insufficient:** `_replay_ticker`
     currently stops at `n - HORIZON`, where `HORIZON = 30`. That excludes signals
     in the last 29 real bars even after the synthetic bar is added. Implement the
     explicit replay option below before interpreting the audit. Only positions
     still open at the halt should lose nearly everything; positions stopped or
     timed out earlier should retain their normal exits.
     `volume=0` is safe — `compute_volume_profile` guards its only divisor
     (`indicator_service.py:298-300`).
- **[NEW] `docs/research/survivorship-audit.md`** + an entry in
  `docs/research/README.md`.

### Small replay change required

- Add optional `--terminal-events <json>` to `run_backtest_snapshot.py`. Pass the
  map explicitly through `run_backtest` and `_replay_and_signals` to
  `_replay_ticker`, including the multiprocessing path. Default to no events.
  Do not infer terminal events from low prices or zero volume.
- Validate the whole map **before dispatch** and fail the run on any mismatch.
  `_replay_ticker` rejects a bad date, but both dispatch paths swallow per-ticker
  exceptions so a long replay survives one unreadable name — which would turn a
  stale events file into a quietly incomplete report. `_validate_terminal_events`
  raises `TerminalEventError` before any work starts, so no report is written.
- For a mapped ticker, validate that the terminal date matches its final bar.
  Keep the existing cadence and starting index, but allow signal dates through
  the last real bar instead of stopping 30 bars early. Never generate a setup on
  the synthetic bar. Unmapped tickers retain the existing replay bounds.
- Use the existing forward bars and outcome evaluation. For the long-only audit,
  a setup with a positive stop resolves no later than the terminal bar. Do not
  pad the history with 30 invented trading bars or turn unresolved shortened
  histories into normal 30-day outcomes.
- Keep terminal prices out of later signal generation; their role is resolving
  existing positions. Preserve normal stop, target and time exits before them.

### Focused validation

Add regression tests alongside the existing backtest tests:

1. A qualifying long setup five sessions before a terminal event is emitted with
   the option enabled and exits at 0.01 if still held. Use a controlled setup
   fixture so the test does not depend on whether SIVB happens to qualify.
2. A position that exits before the terminal event keeps that earlier exit.
3. No setup is emitted on the terminal bar; an ordinary ticker and a run without
   the option retain their existing replay behavior.

Before the full runs, smoke-test the prepared files with `_fetch_columns` for one
existing and one restored ticker. Confirm matching base-symbol bars between arms,
matching configuration and date bounds, and no modification of the base file.

### Running it

```powershell
python scripts/prep_survivorship_snapshot.py --base backtest_snapshots/prod.sqlite
python scripts/run_backtest_snapshot.py backtest_snapshots/survivorship-a.sqlite --out reports/backtest-YYYYMMDD-survivorship-a.json
python scripts/run_backtest_snapshot.py backtest_snapshots/survivorship-b.sqlite --terminal-events backtest_snapshots/survivorship-b-terminal-events.json --out reports/backtest-YYYYMMDD-survivorship-b.json
```

No new runner: `run_backtest_snapshot.py` already runs the production sim offline
and reports CAGR, max drawdown, Sharpe and total return. Compare the same production
portfolio-monitor strategy and `all` lookback in both reports, not whichever
variant performs best. Record B minus A for these metrics, trade counts, evaluated
dates, restored-symbol coverage and terminal-event assumptions. A failed ticker
replay or missing production report row makes the run incomplete.

### What it does and doesn't measure

An **expanded-universe sensitivity analysis**, not a lower bound. It restores
omitted names but does not remove addition lookahead. Still-trading former members
remain eligible after removal: `run_backtest` does not gate on `delisted_on`, and
an index-removal date would not be the right delisting field anyway.

Changing the universe changes percentile ranks and competing admissions. Either
correction can raise or lower this strategy's realized returns; do not claim that
deletions are the larger source of bias or that the true drag must exceed B minus A.
A small delta does not establish that survivorship bias is small. Report it as
the sensitivity observed under these assumptions. A complete point-in-time audit
remains separate work, not a requirement for finishing this experiment.

### Base snapshot

- `prod.sqlite`: 2021-06-24 → 2026-07-02, 90MB, the ~505 production names.
- `research.sqlite`: 2016-01-04 → 2026-07-17, 678MB, 4,650 names.

Use `prod.sqlite`: the highest-signal events (SIVB, FRC, SBNY, ATVI — all
2023) are inside its window, the copies are cheap, and it has no completion
manifest to keep in sync. The pre-registration then states a 2021–2026 window, not
a ten-year one. Verify the actual file's bounds and record its hash before running.
Do not silently substitute `research.sqlite`: its much broader universe answers a
different comparison. Keep the first experiment on the production snapshot.

---

## Step 2 — Would total return change any decision?

Use a conservative ranking screen, not a portfolio simulation. Compare the full
ordered momentum-admitted list on every daily decision date. Do not call a top-15
list the traded set: production also applies setup qualification, current holdings,
available slots and other admission constraints. A difference in this screen is
a reason to investigate actual trades in Step 3, not proof of changed P&L.

### File

- **[NEW] `scripts/research_tr_rank_agreement.py`** — use arm A's universe and dates:
  1. One `Adjustment.ALL` fetch and one `Adjustment.SPLIT` fetch per symbol, plus
     SPY, over the full history. Single fetch each, so each series is internally
     consistent. Align by date, never zip by index. Cache the responses locally
     for repeatable runs. Require matching dates in both adjustments; report and
     resolve missing downloads before applying the gate. Insufficient initial
     history is expected and should be recorded, with equal eligibility in both
     arms. Use only bars at or before each decision date.
  2. At each SPY trading date with sufficient history, compute residual 12-1
     momentum both ways with the existing
     `momentum_service.compute_residual_12_1_momentum` (price-return series against
     nominal SPY; total-return series against total-return SPY), percentile them
     with `momentum_service._percentiles`, and take the admitted set at the
     production `min_momentum_percentile`.
  3. Compute `compute_realized_vol_6m` and its percentiles from the SPLIT series
     once per date and hold that leg fixed in both arms. This isolates the change
     to momentum. Order the entire admitted lists by `blend_strategy_rank`, with
     symbol as a deterministic tie-breaker. Do not truncate to `max_positions`:
     a lower-ranked name may be the first actionable setup when others fail gates
     or are already held. Iterate symbols in the same sorted order in both arms
     before calling `_percentiles` so tied inputs do not create artificial changes.
  4. Save per-date results and an overall/per-year summary: number of evaluated
     dates, fraction with different admitted sets, fraction with different ordered
     lists, and mean and maximum set disagreement. Define set disagreement as
     `len(A symmetric_difference B) / len(A union B)`, or zero for two empty sets.
     Report Spearman correlation as a diagnostic only. Include the first few
     changed dates with the changed names to make the result inspectable.

Validation: identical input series must produce no differences. A fixture where
the admitted members stay the same but two names swap order must register a
changed ordered list. A difference on only one date must still trigger the gate.
Write results to `docs/research/tr-rank-agreement.md` and index it in the research
README, including data coverage, parameters and the exact report/cache paths.

### Pre-registered decision rule

- **No ordered-list differences on any evaluated daily date → stop.** Record
  "no admission or ordering differences observed in this window under the tested
  configuration." This is historical evidence, not a guarantee about future
  decisions. No schema or ingestion changes are needed.
- **Any ordered-list difference → Step 3.** This intentionally conservative rule
  can flag names that would never actually trade. Step 3's paired portfolio replay
  determines whether the difference reaches actual admissions and P&L.
- **Incomplete downloads or no evaluable dates → inconclusive.** Fix coverage
  before deciding. Never turn missing data into agreement.

There is no median-based tolerance: a zero median can hide differences on nearly
half the dates. Checking daily full lists avoids adding a second portfolio engine
to this screening script while retaining changes below the nominal top 15.

---

## Step 3 — Production total-return ranking (conditional)

**Gate cleared 2026-09-07 — Step 3 is now live work.** The screen found ordered
admitted lists differing on 1,009 of 1,009 evaluated dates, with the shift
correlating +0.79 with trailing dividend yield
([results](research/tr-rank-agreement.md)). That is a ranking difference, not a
P&L difference; Step 3's job is to find out whether it reaches trades.

### Step 3a — Paired portfolio replay (do this first, no schema)

Different ranks are not different trades. Before any schema or ingestion work,
replay the production book twice and measure actual admissions and P&L. If the
books are identical, the basis cannot matter and Step 3b never starts.

**[NEW] `scripts/run_total_return_ab.py`**, sourcing everything from artifacts
that already exist — `backtest_snapshots/survivorship-a.sqlite` for bars and
`backtest_snapshots/tr-rank-cache.sqlite` for the two adjusted series.

1. **Generate setup candidates once.** One replay pass produces the candidate set;
   both arms reuse it. Only momentum and what depends on it — the momentum
   percentile, qualification against `min_momentum_percentile`, and
   `blend_strategy_rank` ordering — are recomputed per arm. Setup geometry
   (entry, stop, target, R:R) is generated once and never differs.
2. **Both arms read the same cache.** Control takes SPLIT stock series against
   SPLIT SPY; treatment takes ALL stock against ALL SPY. Neither arm reads a
   freshly fetched series, and neither mixes cache with snapshot closes for the
   momentum input — a refreshed-vs-snapshot difference would confound the basis
   difference being measured.
3. **Everything else is pinned.** Execution uses the snapshot's nominal bars in
   both arms; the volatility leg, the non-momentum gates, capacity
   (`max_positions`), risk per trade and the exit policy are identical. Each book
   then carries its **own** resulting holdings — a divergent admission changes
   which slots are occupied on later dates, and that path dependence is part of
   what is being measured.
4. **Fixed up front:** universe (arm A's 505 symbols with bars), window
   (2021-06-24 → 2026-07-02), cadence, and the production config as read from the
   snapshot. Only bars at or before each signal date are used, in both arms.
5. **Saved to `reports/tr-total-return-ab.json`:** per-arm trade count, CAGR, max
   drawdown, Sharpe, total return, win rate; annual returns for both arms and the
   **count of years treatment beats control**; and the entry-level diff — dates
   and symbols entered by one arm and not the other, in both directions.
6. **Tests** (`tests/unit/test_total_return_ab.py`): an identical-input parity
   test — feeding the same series to both arms must produce byte-identical books,
   so any observed difference is attributable to the basis and not to replay
   nondeterminism; and a controlled case where a hand-built momentum difference
   flips one name across the cutoff and the resulting admission actually changes,
   proving the harness can see what it is looking for.

**Reading the result.** Different trades establish that the basis is
production-relevant. They do **not** authorize a rollout: that is the
non-inferiority decision below, taken on the paired numbers, and Step 3b starts
only after the paired results have been reported.

**Step 3a result (2026-09-07):** the basis reaches trades — ~30% of entries
differ. Which basis is better is unresolved: a paired block bootstrap on the
date-aligned daily return streams (5,000 replicates, 21- and 63-session blocks,
seed fixed beforehand, both reported) puts ΔCAGR at −3.6/−3.8pp with 95%
intervals spanning zero, ΔSharpe ≈ −0.11, and ΔMaxDD −1.0/−1.8pp in treatment's
favour. **Production stays unchanged; 3b not started.**
[Results](research/tr-total-return-ab.md).

### Step 3b — Production storage and ingestion (only if 3a shows real differences)

The design points already settled, worth carrying forward:

- `ohlcv_records.tr_factor`, `Float`, nullable, **no default and no server_default**
  — a default of 1.0 makes "never backfilled" indistinguishable from "no dividends"
  and would silently stamp a price-return ranking as total return.
- Forward-anchored: `f = 1.0` at `max(2016-01-04, ticker's first bar)`, compounded
  as `f_t = f_{t-1} * (R_t / R_{t-1})` where `R = all_close / split_close`. The
  ratio is anchor-invariant **only within one API response** — never persist `R` and
  compare across days.
- `upsert_ohlcv` must not clobber a stored factor:
  `coalesce(excluded.tr_factor, ohlcv_records.tr_factor)`. Daily ingest re-fetches
  the boundary bars, so overlapping writes are routine.
- Daily ingest pulls a ~5-bar overlap under both adjustments to extend the factor;
  a clean-ratio mismatch on the overlap bars means a split, which needs a
  full-history refetch (the current incremental ingest never restates history).
- One basis for the whole cross-section, never mixed, stamped in the rank log.
- Beta leg: if the stock series goes total return, SPY must too, in the live path
  and the backtest, or beta picks up SPY's ~1%/yr yield.
- A/B through the existing matrix on CAGR and max drawdown with per-year win/loss
  counts. Reuse the same setup candidates, nominal execution bars, core gates,
  volatility leg, capacity and exit policy; change only momentum and its dependent
  admission/ranking fields. Replay each book with its own resulting holdings and
  compare actual entry dates/symbols as well as P&L. Do not substitute top-15 lists
  for the existing portfolio admission logic. Non-inferiority, not superiority:
  total return is the definition 12-1
  momentum is built on, so an in-sample shortfall shouldn't lock in the basis we
  believe is measured wrong.

---

## Not doing

- **Chart total-return toggle.** The candlestick chart is an execution chart:
  stops, S/R levels, entry and paper-trade markers are all nominal. Rescaling the
  candles draws prices that never traded next to a stop that did.
- **`[D]` dividend markers.** Cosmetic, and they need `tr_factor` to exist. Revisit
  after Step 3, if Step 3 ever happens.
- **Point-in-time addition gating** — interval map, reverse-chronological replay,
  rename map and removal gating are deferred from this sensitivity experiment.
  Their relative impact is unknown; a small Step 1 delta does not rule them out.
  Note if a point-in-time audit is
  ever done: the ranking cross-section must be filtered *before*
  `_assign_*_percentiles` (`backtest_service.py:4198-4200`), not only at admission
  in `_simulate_portfolio` (4257), or the percentile cutoff is still computed
  against a non-PIT universe.
- **Prune protection and the `add_ticker` regex.** Only needed if delisted names
  are written to the production database. The audit works entirely in snapshot
  copies. (`bootstrap_universe(prune_missing=True)` cascading away a still-trading
  ex-constituent's history is a real latent risk — a separate ticket, not this one.)
- **`benchmark_prices.tr_close`.** Only needed for Step 3. Note for then: store a
  forward factor like the OHLCV side, not a back-adjusted close —
  `refresh_benchmark_prices` writes a row only when `close` changed and is called
  with variable spans, which would leave different anchors either side of the last
  refreshed window.
- **Migrating the snapshots in place.** The prep script syncs its own copies.
  `research.sqlite` stays byte-identical, so its completion manifest and the reports
  that cite it stay valid.