dennisthiessenandClaude Opus 5
ebe9047403
Three questions answered against snapshot copies, with no production change. Step 1 — how sensitive is the backtest to the ranking universe? Restoring 85 S&P 500 constituents removed between 2021-06 and 2026-07 moves the same window from 2.09 Sharpe / 51.6% CAGR to 1.57 / 38.1%. The failed banks never reached either book (identical best/worst trade R), so this measures universe recomposition, not survivorship bias. The removals list is curated by hand with per-symbol sources; only three explicitly confirmed terminal events get marks, and four unresolved endings are reported as caveats rather than inferred. Step 2 — does a total-return basis change the ranking? Ordered lists differ on 1009 of 1009 evaluated dates, 4.56 replacements per date. The shift correlates with trailing implied yield (mean Spearman +0.79), which is strong evidence consistent with a dividend effect but does not isolate own-yield, since both the stock and benchmark legs change. Step 3a — does that reach P&L? A paired replay reusing one shared candidate set (control reproduces the production row exactly) changes ~30% of entries. Treatment is -3.7pp CAGR and -1.9pp drawdown, with the entire return gap in 2025. A paired moving-block bootstrap (seed 20260907, 5000 replicates, 21/63 blocks, both reported) puts every interval across zero. Decision: production stays unchanged. No schema, no ingestion, no wider sweep. The observed window does not resolve which basis is better, and choosing a threshold now — with -3.7pp on the table — would be fitting the rule to the answer. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HAH7vVtn7qbcn1EVheecHQ
174 lines
9.5 KiB
Markdown
174 lines
9.5 KiB
Markdown
# Survivorship sensitivity — restoring removed S&P 500 constituents
|
||
|
||
**Run date:** 2026-09-07 · **Status:** measured, no strategy change proposed
|
||
**Reports:** `reports/backtest-20260907-survivorship-a.json`,
|
||
`reports/backtest-20260907-survivorship-b.json`,
|
||
`reports/backtest-20260907-survivorship-b-nomarks.json`
|
||
|
||
## Question
|
||
|
||
The backtest universe is the ~505 names tracked *today*. Every constituent that
|
||
left the index between 2021 and 2026 is missing from it, including the ones that
|
||
failed. How different does the production strategy look when they are put back?
|
||
|
||
This is an **expanded-universe sensitivity analysis**, not a bias estimate. It is
|
||
pre-registered as a measurement: there is no pass/fail rule and no promotion
|
||
attached to the outcome.
|
||
|
||
## Method
|
||
|
||
**Base:** `backtest_snapshots/prod.sqlite`, sha256 `7241f0ff0f6187a0…`, window
|
||
**2021-06-24 → 2026-07-02** (~5 years, not the full Alpaca history). Never
|
||
modified — both arms are copies, and the hash is re-checked after the build.
|
||
|
||
| Arm | Universe |
|
||
|---|---|
|
||
| **A** — control | 506 tickers, the snapshot as it stands |
|
||
| **B** — treatment | 591 tickers: the same 506 plus 85 restored removals |
|
||
|
||
Both arms ran the same binary, config, cadence (weekly) and date bounds, and are
|
||
compared on the same report row: `portfolio_monitor.runs` where
|
||
`is_production` is true and `lookback` is `all`
|
||
(`residual80_highvol80_20_atr3`).
|
||
|
||
Removals come from `scripts/data/sp500_removals.json` — 92 rows curated from
|
||
Wikipedia's *Historical components of the S&P 500* changes table, each carrying
|
||
the verbatim source text plus independently measured Alpaca evidence. Build with
|
||
`scripts/prep_survivorship_snapshot.py`.
|
||
|
||
Three data rules, all learned from the price data rather than assumed:
|
||
|
||
1. **Nothing is inferred from missing bars.** `delisted_on` / `delisted_reason`
|
||
are written only for confirmed terminal events. A series that merely stops
|
||
early leaves them NULL and is reported as an unresolved ending.
|
||
2. **Series are shortened only at a reviewed cutoff.** Alpaca does not end a
|
||
halted symbol's series — it repeats the last price at zero volume. SBNY sits
|
||
at 70.00 for 509 sessions after the FDIC closed it; INFO sits at 108.61 for
|
||
1,090 sessions, after which a *different company* reuses the ticker with no
|
||
date gap. Both have explicit reviewed cutoffs (2023-03-10, 2022-02-25).
|
||
Elsewhere a run of ≥5 zero-volume sessions is a tripwire: the symbol is left
|
||
out and reported, never silently cut.
|
||
3. **Terminal marks are explicit assumptions.** A synthetic `0.01` bar is written
|
||
on the next session after the last real bar for SIVB, SBNY and FRC only —
|
||
each with an FDIC source and only where the retained final bar matches the
|
||
reviewed `last_real_bar`. The FDIC sources establish the closure *event*; they
|
||
do not establish what holders recovered. The mark is a near-total-loss
|
||
assumption, not an observed closing-day execution.
|
||
|
||
**Replay change.** `_replay_ticker` normally stops `HORIZON = 30` bars before the
|
||
end so every setup has a full resolution window — which would hide every signal
|
||
in the final 29 real bars, exactly the ones a collapse resolves. A new
|
||
`--terminal-events` map opts the three marked symbols into walking through their
|
||
last real bar; no setup is ever generated on the synthetic bar, and unmapped
|
||
symbols are untouched. The map is validated against the snapshot's actual final
|
||
bars *before* any replay is dispatched (`_validate_terminal_events`), because both
|
||
dispatch paths swallow per-ticker exceptions and would otherwise turn a stale map
|
||
into a quietly incomplete report.
|
||
|
||
## Result
|
||
|
||
Source: `portfolio_monitor.runs` where `is_production` is true and `lookback` is
|
||
`all` (strategy `residual80_highvol80_20_atr3`), from
|
||
`reports/backtest-20260907-survivorship-a.json` and
|
||
`…-survivorship-b.json`. Both runs: weekly cadence, 6 workers,
|
||
`--allow-spawn`, gate cutoff 80.0, full period.
|
||
|
||
| Metric | Arm A | Arm B | B − A |
|
||
|---|---:|---:|---:|
|
||
| Total return % | 424.60 | 264.40 | **−160.20** |
|
||
| CAGR % | 51.60 | 38.10 | **−13.50** |
|
||
| Max drawdown % | 21.40 | 23.60 | +2.20 |
|
||
| Sharpe | 2.09 | 1.57 | **−0.52** |
|
||
| Trades | 322 | 346 | +24 |
|
||
| Win rate % | 37.0 | 35.0 | −2.0 |
|
||
| Final equity | 52,459 | 36,439 | −16,020 |
|
||
|
||
Context: 202,765 → 229,642 candidates, 1,086 → 1,315 qualified. Gate cutoff 80.0
|
||
in both arms.
|
||
|
||
## What actually caused it
|
||
|
||
**Not the failures.** `worst_trade_r` is **−3.26 in both arms**, and
|
||
`best_trade_r` is 12.87 in both. A position held into a terminal mark gaps to
|
||
$0.01 and fills at the open, which would land far below −3.26 — at candidate
|
||
level those trades score −3.13R (SIVB) and −5.99R (SBNY). No such trade appears
|
||
in either book, so none of the three failed banks reached the production
|
||
portfolio.
|
||
|
||
**What the control does and does not show.** Arm B replayed *without*
|
||
`--terminal-events` (`backtest-20260907-survivorship-b-nomarks.json`, same
|
||
snapshot, same config) produces a production row identical to arm B with them in
|
||
every field — 346 trades, 38.1% CAGR, 1.57 Sharpe, 23.6% drawdown, 264.4% total
|
||
return, −3.26 worst trade. Candidates rise 229,617 → 229,642, the 25 extra
|
||
signals the option exposes in the final 29 bars of the three symbols, while
|
||
`qualified` is 1,315 either way.
|
||
|
||
So the control establishes precisely this: **those 25 additional replay signals
|
||
changed no portfolio decision.** It does *not* independently establish that the
|
||
terminal marks had no effect — both runs read `survivorship-b.sqlite`, which
|
||
contains the synthetic bars either way, so a setup generated earlier could still
|
||
resolve against one in both runs. Isolating the marks themselves would need a
|
||
third snapshot built without them, which has not been run.
|
||
|
||
The separate evidence that no marked position reached the book is the trade
|
||
extremes: a position resolving against a $0.01 bar fills at that bar's open and
|
||
scores far worse than −3.26 (−3.13R for SIVB and −5.99R for SBNY at candidate
|
||
level), yet arm B's `worst_trade_r` equals arm A's exactly. That is inference
|
||
from the extremes, not a controlled measurement.
|
||
|
||
On that basis the −13.5pp CAGR is **universe recomposition**: 85 additional names
|
||
(+17% of the universe) shift the cross-sectional percentile ranks and compete for
|
||
the same 15 position slots, displacing trades arm A took. The strategy is
|
||
markedly sensitive to which universe it ranks over — which is the finding, and it
|
||
is a different claim from "survivorship bias costs 13.5pp".
|
||
|
||
## Caveats
|
||
|
||
- **This is not a bias estimate and not a lower bound.** Restoring names is only
|
||
one of the two corrections. Index-*addition* lookahead — today's members being
|
||
tradeable before they joined — is not removed. Either correction can move
|
||
returns in either direction, so B − A is the sensitivity observed under these
|
||
assumptions, nothing more. A small delta would not have shown the bias to be
|
||
small, and this large one does not establish its size either.
|
||
- **Arm B is not a point-in-time universe.** Still-trading former members remain
|
||
eligible after their removal date; `run_backtest` does not gate on
|
||
`delisted_on`, and an index-removal date would be the wrong field for it
|
||
anyway. A true PIT audit is separate work.
|
||
- **Four unresolved endings.** CMA (ends 2026-01-30), SEE (2026-04-08), HBI
|
||
(2025-11-28) and DISH (2023-12-29) stop trading before the window closes for
|
||
reasons the source row does not explain. Their bars are real and their
|
||
delisting fields are NULL; the ending status is simply unknown.
|
||
- **Three near-total-loss assumptions** (SIVB, SBNY, FRC), as described above.
|
||
The control shows they did not affect this result, but they remain assumptions
|
||
and would matter in any run where such a name did clear the gate.
|
||
- **Acquisitions are approximated** by the last available close. Cash or stock
|
||
consideration and the release of capital on the deal date are not modelled.
|
||
- **FBIN carries a corporate action.** Fortune Brands distributed MasterBrand on
|
||
2022-12-14 and began trading as FBIN on 2022-12-15. Alpaca's split adjustment
|
||
does not adjust for spin-offs, so the distribution appears as a real −6.91%
|
||
close-to-close move (63.22 → 61.90 → 57.62). A holder did not lose that value,
|
||
but any signal or stop crossing that date reads it as a decline. MBC enters the
|
||
dataset separately with its first bar the same day.
|
||
- **Window.** 2021-06 to 2026-07, because that is what `prod.sqlite` holds. The
|
||
2023 bank failures are inside it; the 2016–2021 period is not.
|
||
- **Snapshot schema.** The snapshot files on disk predate migrations 026 and 032,
|
||
so `select(Ticker)` against them fails with `no such column: tickers.cik` — the
|
||
path `backtest_service._fetch_columns` takes. Per-ticker exceptions are caught
|
||
and logged, so a run against an unsynced snapshot completes and writes a report
|
||
with **zero candidates** rather than failing. The prep script syncs both copies
|
||
against `Base.metadata`; anything else reading these snapshots needs the same.
|
||
|
||
## Reproduce
|
||
|
||
```powershell
|
||
python scripts/prep_survivorship_snapshot.py --base backtest_snapshots/prod.sqlite
|
||
python scripts/run_backtest_snapshot.py backtest_snapshots/survivorship-a.sqlite `
|
||
--out reports/backtest-YYYYMMDD-survivorship-a.json --allow-spawn --workers 6
|
||
python scripts/run_backtest_snapshot.py backtest_snapshots/survivorship-b.sqlite `
|
||
--terminal-events backtest_snapshots/survivorship-b-terminal-events.json `
|
||
--out reports/backtest-YYYYMMDD-survivorship-b.json --allow-spawn --workers 6
|
||
```
|
||
|
||
`survivorship-b-prep-manifest.json` records the base hash, per-symbol coverage,
|
||
the terminal events written and every unresolved item for the run.
|