signal-platform/docs/research/survivorship-audit.md

Survivorship sensitivity — restoring removed S&P 500 constituents

Run date: 2026-09-07 · Status: measured, no strategy change proposed Reports: reports/backtest-20260907-survivorship-a.json, reports/backtest-20260907-survivorship-b.json, reports/backtest-20260907-survivorship-b-nomarks.json

Question

The backtest universe is the ~505 names tracked today. Every constituent that left the index between 2021 and 2026 is missing from it, including the ones that failed. How different does the production strategy look when they are put back?

This is an expanded-universe sensitivity analysis, not a bias estimate. It is pre-registered as a measurement: there is no pass/fail rule and no promotion attached to the outcome.

Method

Base: backtest_snapshots/prod.sqlite, sha256 7241f0ff0f6187a0…, window 2021-06-24 → 2026-07-02 (~5 years, not the full Alpaca history). Never modified — both arms are copies, and the hash is re-checked after the build.

Arm	Universe
A — control	506 tickers, the snapshot as it stands
B — treatment	591 tickers: the same 506 plus 85 restored removals

Both arms ran the same binary, config, cadence (weekly) and date bounds, and are compared on the same report row: portfolio_monitor.runs where is_production is true and lookback is all (residual80_highvol80_20_atr3).

Removals come from scripts/data/sp500_removals.json — 92 rows curated from Wikipedia's Historical components of the S&P 500 changes table, each carrying the verbatim source text plus independently measured Alpaca evidence. Build with scripts/prep_survivorship_snapshot.py.

Three data rules, all learned from the price data rather than assumed:

1. Nothing is inferred from missing bars. delisted_on / delisted_reason are written only for confirmed terminal events. A series that merely stops early leaves them NULL and is reported as an unresolved ending.
2. Series are shortened only at a reviewed cutoff. Alpaca does not end a halted symbol's series — it repeats the last price at zero volume. SBNY sits at 70.00 for 509 sessions after the FDIC closed it; INFO sits at 108.61 for 1,090 sessions, after which a different company reuses the ticker with no date gap. Both have explicit reviewed cutoffs (2023-03-10, 2022-02-25). Elsewhere a run of ≥5 zero-volume sessions is a tripwire: the symbol is left out and reported, never silently cut.
3. Terminal marks are explicit assumptions. A synthetic 0.01 bar is written on the next session after the last real bar for SIVB, SBNY and FRC only — each with an FDIC source and only where the retained final bar matches the reviewed last_real_bar. The FDIC sources establish the closure event; they do not establish what holders recovered. The mark is a near-total-loss assumption, not an observed closing-day execution.

Replay change. _replay_ticker normally stops HORIZON = 30 bars before the end so every setup has a full resolution window — which would hide every signal in the final 29 real bars, exactly the ones a collapse resolves. A new --terminal-events map opts the three marked symbols into walking through their last real bar; no setup is ever generated on the synthetic bar, and unmapped symbols are untouched. The map is validated against the snapshot's actual final bars before any replay is dispatched (_validate_terminal_events), because both dispatch paths swallow per-ticker exceptions and would otherwise turn a stale map into a quietly incomplete report.

Result

Source: portfolio_monitor.runs where is_production is true and lookback is all (strategy residual80_highvol80_20_atr3), from reports/backtest-20260907-survivorship-a.json and …-survivorship-b.json. Both runs: weekly cadence, 6 workers, --allow-spawn, gate cutoff 80.0, full period.

Metric	Arm A	Arm B	B − A
Total return %	424.60	264.40	−160.20
CAGR %	51.60	38.10	−13.50
Max drawdown %	21.40	23.60	+2.20
Sharpe	2.09	1.57	−0.52
Trades	322	346	+24
Win rate %	37.0	35.0	−2.0
Final equity	52,459	36,439	−16,020

Context: 202,765 → 229,642 candidates, 1,086 → 1,315 qualified. Gate cutoff 80.0 in both arms.

What actually caused it

Not the failures. worst_trade_r is −3.26 in both arms, and best_trade_r is 12.87 in both. A position held into a terminal mark gaps to $0.01 and fills at the open, which would land far below −3.26 — at candidate level those trades score −3.13R (SIVB) and −5.99R (SBNY). No such trade appears in either book, so none of the three failed banks reached the production portfolio.

What the control does and does not show. Arm B replayed without --terminal-events (backtest-20260907-survivorship-b-nomarks.json, same snapshot, same config) produces a production row identical to arm B with them in every field — 346 trades, 38.1% CAGR, 1.57 Sharpe, 23.6% drawdown, 264.4% total return, −3.26 worst trade. Candidates rise 229,617 → 229,642, the 25 extra signals the option exposes in the final 29 bars of the three symbols, while qualified is 1,315 either way.

So the control establishes precisely this: those 25 additional replay signals changed no portfolio decision. It does not independently establish that the terminal marks had no effect — both runs read survivorship-b.sqlite, which contains the synthetic bars either way, so a setup generated earlier could still resolve against one in both runs. Isolating the marks themselves would need a third snapshot built without them, which has not been run.

The separate evidence that no marked position reached the book is the trade extremes: a position resolving against a $0.01 bar fills at that bar's open and scores far worse than −3.26 (−3.13R for SIVB and −5.99R for SBNY at candidate level), yet arm B's worst_trade_r equals arm A's exactly. That is inference from the extremes, not a controlled measurement.

On that basis the −13.5pp CAGR is universe recomposition: 85 additional names (+17% of the universe) shift the cross-sectional percentile ranks and compete for the same 15 position slots, displacing trades arm A took. The strategy is markedly sensitive to which universe it ranks over — which is the finding, and it is a different claim from "survivorship bias costs 13.5pp".

Caveats

• This is not a bias estimate and not a lower bound. Restoring names is only one of the two corrections. Index-addition lookahead — today's members being tradeable before they joined — is not removed. Either correction can move returns in either direction, so B − A is the sensitivity observed under these assumptions, nothing more. A small delta would not have shown the bias to be small, and this large one does not establish its size either.
• Arm B is not a point-in-time universe. Still-trading former members remain eligible after their removal date; run_backtest does not gate on delisted_on, and an index-removal date would be the wrong field for it anyway. A true PIT audit is separate work.
• Four unresolved endings. CMA (ends 2026-01-30), SEE (2026-04-08), HBI (2025-11-28) and DISH (2023-12-29) stop trading before the window closes for reasons the source row does not explain. Their bars are real and their delisting fields are NULL; the ending status is simply unknown.
• Three near-total-loss assumptions (SIVB, SBNY, FRC), as described above. The control shows they did not affect this result, but they remain assumptions and would matter in any run where such a name did clear the gate.
• Acquisitions are approximated by the last available close. Cash or stock consideration and the release of capital on the deal date are not modelled.
• FBIN carries a corporate action. Fortune Brands distributed MasterBrand on 2022-12-14 and began trading as FBIN on 2022-12-15. Alpaca's split adjustment does not adjust for spin-offs, so the distribution appears as a real −6.91% close-to-close move (63.22 → 61.90 → 57.62). A holder did not lose that value, but any signal or stop crossing that date reads it as a decline. MBC enters the dataset separately with its first bar the same day.
• Window. 2021-06 to 2026-07, because that is what prod.sqlite holds. The 2023 bank failures are inside it; the 2016–2021 period is not.
• Snapshot schema. The snapshot files on disk predate migrations 026 and 032, so select(Ticker) against them fails with no such column: tickers.cik — the path backtest_service._fetch_columns takes. Per-ticker exceptions are caught and logged, so a run against an unsynced snapshot completes and writes a report with zero candidates rather than failing. The prep script syncs both copies against Base.metadata; anything else reading these snapshots needs the same.

Reproduce

python scripts/prep_survivorship_snapshot.py --base backtest_snapshots/prod.sqlite
python scripts/run_backtest_snapshot.py backtest_snapshots/survivorship-a.sqlite `
    --out reports/backtest-YYYYMMDD-survivorship-a.json --allow-spawn --workers 6
python scripts/run_backtest_snapshot.py backtest_snapshots/survivorship-b.sqlite `
    --terminal-events backtest_snapshots/survivorship-b-terminal-events.json `
    --out reports/backtest-YYYYMMDD-survivorship-b.json --allow-spawn --workers 6

survivorship-b-prep-manifest.json records the base hash, per-symbol coverage, the terminal events written and every unresolved item for the run.