research: survivorship sensitivity and total-return momentum investigation

Three questions answered against snapshot copies, with no production change.

Step 1 — how sensitive is the backtest to the ranking universe? Restoring 85
S&P 500 constituents removed between 2021-06 and 2026-07 moves the same window
from 2.09 Sharpe / 51.6% CAGR to 1.57 / 38.1%. The failed banks never reached
either book (identical best/worst trade R), so this measures universe
recomposition, not survivorship bias. The removals list is curated by hand with
per-symbol sources; only three explicitly confirmed terminal events get marks,
and four unresolved endings are reported as caveats rather than inferred.

Step 2 — does a total-return basis change the ranking? Ordered lists differ on
1009 of 1009 evaluated dates, 4.56 replacements per date. The shift correlates
with trailing implied yield (mean Spearman +0.79), which is strong evidence
consistent with a dividend effect but does not isolate own-yield, since both
the stock and benchmark legs change.

Step 3a — does that reach P&L? A paired replay reusing one shared candidate set
(control reproduces the production row exactly) changes ~30% of entries.
Treatment is -3.7pp CAGR and -1.9pp drawdown, with the entire return gap in
2025. A paired moving-block bootstrap (seed 20260907, 5000 replicates, 21/63
blocks, both reported) puts every interval across zero.

Decision: production stays unchanged. No schema, no ingestion, no wider sweep.
The observed window does not resolve which basis is better, and choosing a
threshold now — with -3.7pp on the table — would be fitting the rule to the
answer.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HAH7vVtn7qbcn1EVheecHQ

docs/research/survivorship-audit.md

@@ -0,0 +1,173 @@
+# Survivorship sensitivity — restoring removed S&P 500 constituents
+
+**Run date:** 2026-09-07 · **Status:** measured, no strategy change proposed
+**Reports:** `reports/backtest-20260907-survivorship-a.json`,
+`reports/backtest-20260907-survivorship-b.json`,
+`reports/backtest-20260907-survivorship-b-nomarks.json`
+
+## Question
+
+The backtest universe is the ~505 names tracked *today*. Every constituent that
+left the index between 2021 and 2026 is missing from it, including the ones that
+failed. How different does the production strategy look when they are put back?
+
+This is an **expanded-universe sensitivity analysis**, not a bias estimate. It is
+pre-registered as a measurement: there is no pass/fail rule and no promotion
+attached to the outcome.
+
+## Method
+
+**Base:** `backtest_snapshots/prod.sqlite`, sha256 `7241f0ff0f6187a0…`, window
+**2021-06-24 → 2026-07-02** (~5 years, not the full Alpaca history). Never
+modified — both arms are copies, and the hash is re-checked after the build.
+
+| Arm | Universe |
+|---|---|
+| **A** — control | 506 tickers, the snapshot as it stands |
+| **B** — treatment | 591 tickers: the same 506 plus 85 restored removals |
+
+Both arms ran the same binary, config, cadence (weekly) and date bounds, and are
+compared on the same report row: `portfolio_monitor.runs` where
+`is_production` is true and `lookback` is `all`
+(`residual80_highvol80_20_atr3`).
+
+Removals come from `scripts/data/sp500_removals.json` — 92 rows curated from
+Wikipedia's *Historical components of the S&P 500* changes table, each carrying
+the verbatim source text plus independently measured Alpaca evidence. Build with
+`scripts/prep_survivorship_snapshot.py`.
+
+Three data rules, all learned from the price data rather than assumed:
+
+1. **Nothing is inferred from missing bars.** `delisted_on` / `delisted_reason`
+   are written only for confirmed terminal events. A series that merely stops
+   early leaves them NULL and is reported as an unresolved ending.
+2. **Series are shortened only at a reviewed cutoff.** Alpaca does not end a
+   halted symbol's series — it repeats the last price at zero volume. SBNY sits
+   at 70.00 for 509 sessions after the FDIC closed it; INFO sits at 108.61 for
+   1,090 sessions, after which a *different company* reuses the ticker with no
+   date gap. Both have explicit reviewed cutoffs (2023-03-10, 2022-02-25).
+   Elsewhere a run of ≥5 zero-volume sessions is a tripwire: the symbol is left
+   out and reported, never silently cut.
+3. **Terminal marks are explicit assumptions.** A synthetic `0.01` bar is written
+   on the next session after the last real bar for SIVB, SBNY and FRC only —
+   each with an FDIC source and only where the retained final bar matches the
+   reviewed `last_real_bar`. The FDIC sources establish the closure *event*; they
+   do not establish what holders recovered. The mark is a near-total-loss
+   assumption, not an observed closing-day execution.
+
+**Replay change.** `_replay_ticker` normally stops `HORIZON = 30` bars before the
+end so every setup has a full resolution window — which would hide every signal
+in the final 29 real bars, exactly the ones a collapse resolves. A new
+`--terminal-events` map opts the three marked symbols into walking through their
+last real bar; no setup is ever generated on the synthetic bar, and unmapped
+symbols are untouched. The map is validated against the snapshot's actual final
+bars *before* any replay is dispatched (`_validate_terminal_events`), because both
+dispatch paths swallow per-ticker exceptions and would otherwise turn a stale map
+into a quietly incomplete report.
+
+## Result
+
+Source: `portfolio_monitor.runs` where `is_production` is true and `lookback` is
+`all` (strategy `residual80_highvol80_20_atr3`), from
+`reports/backtest-20260907-survivorship-a.json` and
+`…-survivorship-b.json`. Both runs: weekly cadence, 6 workers,
+`--allow-spawn`, gate cutoff 80.0, full period.
+
+| Metric | Arm A | Arm B | B − A |
+|---|---:|---:|---:|
+| Total return % | 424.60 | 264.40 | **−160.20** |
+| CAGR % | 51.60 | 38.10 | **−13.50** |
+| Max drawdown % | 21.40 | 23.60 | +2.20 |
+| Sharpe | 2.09 | 1.57 | **−0.52** |
+| Trades | 322 | 346 | +24 |
+| Win rate % | 37.0 | 35.0 | −2.0 |
+| Final equity | 52,459 | 36,439 | −16,020 |
+
+Context: 202,765 → 229,642 candidates, 1,086 → 1,315 qualified. Gate cutoff 80.0
+in both arms.
+
+## What actually caused it
+
+**Not the failures.** `worst_trade_r` is **−3.26 in both arms**, and
+`best_trade_r` is 12.87 in both. A position held into a terminal mark gaps to
+$0.01 and fills at the open, which would land far below −3.26 — at candidate
+level those trades score −3.13R (SIVB) and −5.99R (SBNY). No such trade appears
+in either book, so none of the three failed banks reached the production
+portfolio.
+
+**What the control does and does not show.** Arm B replayed *without*
+`--terminal-events` (`backtest-20260907-survivorship-b-nomarks.json`, same
+snapshot, same config) produces a production row identical to arm B with them in
+every field — 346 trades, 38.1% CAGR, 1.57 Sharpe, 23.6% drawdown, 264.4% total
+return, −3.26 worst trade. Candidates rise 229,617 → 229,642, the 25 extra
+signals the option exposes in the final 29 bars of the three symbols, while
+`qualified` is 1,315 either way.
+
+So the control establishes precisely this: **those 25 additional replay signals
+changed no portfolio decision.** It does *not* independently establish that the
+terminal marks had no effect — both runs read `survivorship-b.sqlite`, which
+contains the synthetic bars either way, so a setup generated earlier could still
+resolve against one in both runs. Isolating the marks themselves would need a
+third snapshot built without them, which has not been run.
+
+The separate evidence that no marked position reached the book is the trade
+extremes: a position resolving against a $0.01 bar fills at that bar's open and
+scores far worse than −3.26 (−3.13R for SIVB and −5.99R for SBNY at candidate
+level), yet arm B's `worst_trade_r` equals arm A's exactly. That is inference
+from the extremes, not a controlled measurement.
+
+On that basis the −13.5pp CAGR is **universe recomposition**: 85 additional names
+(+17% of the universe) shift the cross-sectional percentile ranks and compete for
+the same 15 position slots, displacing trades arm A took. The strategy is
+markedly sensitive to which universe it ranks over — which is the finding, and it
+is a different claim from "survivorship bias costs 13.5pp".
+
+## Caveats
+
+- **This is not a bias estimate and not a lower bound.** Restoring names is only
+  one of the two corrections. Index-*addition* lookahead — today's members being
+  tradeable before they joined — is not removed. Either correction can move
+  returns in either direction, so B − A is the sensitivity observed under these
+  assumptions, nothing more. A small delta would not have shown the bias to be
+  small, and this large one does not establish its size either.
+- **Arm B is not a point-in-time universe.** Still-trading former members remain
+  eligible after their removal date; `run_backtest` does not gate on
+  `delisted_on`, and an index-removal date would be the wrong field for it
+  anyway. A true PIT audit is separate work.
+- **Four unresolved endings.** CMA (ends 2026-01-30), SEE (2026-04-08), HBI
+  (2025-11-28) and DISH (2023-12-29) stop trading before the window closes for
+  reasons the source row does not explain. Their bars are real and their
+  delisting fields are NULL; the ending status is simply unknown.
+- **Three near-total-loss assumptions** (SIVB, SBNY, FRC), as described above.
+  The control shows they did not affect this result, but they remain assumptions
+  and would matter in any run where such a name did clear the gate.
+- **Acquisitions are approximated** by the last available close. Cash or stock
+  consideration and the release of capital on the deal date are not modelled.
+- **FBIN carries a corporate action.** Fortune Brands distributed MasterBrand on
+  2022-12-14 and began trading as FBIN on 2022-12-15. Alpaca's split adjustment
+  does not adjust for spin-offs, so the distribution appears as a real −6.91%
+  close-to-close move (63.22 → 61.90 → 57.62). A holder did not lose that value,
+  but any signal or stop crossing that date reads it as a decline. MBC enters the
+  dataset separately with its first bar the same day.
+- **Window.** 2021-06 to 2026-07, because that is what `prod.sqlite` holds. The
+  2023 bank failures are inside it; the 2016–2021 period is not.
+- **Snapshot schema.** The snapshot files on disk predate migrations 026 and 032,
+  so `select(Ticker)` against them fails with `no such column: tickers.cik` — the
+  path `backtest_service._fetch_columns` takes. Per-ticker exceptions are caught
+  and logged, so a run against an unsynced snapshot completes and writes a report
+  with **zero candidates** rather than failing. The prep script syncs both copies
+  against `Base.metadata`; anything else reading these snapshots needs the same.
+
+## Reproduce
+
+```powershell
+python scripts/prep_survivorship_snapshot.py --base backtest_snapshots/prod.sqlite
+python scripts/run_backtest_snapshot.py backtest_snapshots/survivorship-a.sqlite `
+    --out reports/backtest-YYYYMMDD-survivorship-a.json --allow-spawn --workers 6
+python scripts/run_backtest_snapshot.py backtest_snapshots/survivorship-b.sqlite `
+    --terminal-events backtest_snapshots/survivorship-b-terminal-events.json `
+    --out reports/backtest-YYYYMMDD-survivorship-b.json --allow-spawn --workers 6
+```
+
+`survivorship-b-prep-manifest.json` records the base hash, per-symbol coverage,
+the terminal events written and every unresolved item for the run.