research: survivorship sensitivity and total-return momentum investigation
Three questions answered against snapshot copies, with no production change. Step 1 — how sensitive is the backtest to the ranking universe? Restoring 85 S&P 500 constituents removed between 2021-06 and 2026-07 moves the same window from 2.09 Sharpe / 51.6% CAGR to 1.57 / 38.1%. The failed banks never reached either book (identical best/worst trade R), so this measures universe recomposition, not survivorship bias. The removals list is curated by hand with per-symbol sources; only three explicitly confirmed terminal events get marks, and four unresolved endings are reported as caveats rather than inferred. Step 2 — does a total-return basis change the ranking? Ordered lists differ on 1009 of 1009 evaluated dates, 4.56 replacements per date. The shift correlates with trailing implied yield (mean Spearman +0.79), which is strong evidence consistent with a dividend effect but does not isolate own-yield, since both the stock and benchmark legs change. Step 3a — does that reach P&L? A paired replay reusing one shared candidate set (control reproduces the production row exactly) changes ~30% of entries. Treatment is -3.7pp CAGR and -1.9pp drawdown, with the entire return gap in 2025. A paired moving-block bootstrap (seed 20260907, 5000 replicates, 21/63 blocks, both reported) puts every interval across zero. Decision: production stays unchanged. No schema, no ingestion, no wider sweep. The observed window does not resolve which basis is better, and choosing a threshold now — with -3.7pp on the table — would be fitting the rule to the answer. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HAH7vVtn7qbcn1EVheecHQ
This commit is contained in:
dennisthiessenandClaude Opus 5 committed
1 parent
8453b87290
commit
ebe9047403
16 files changed
+442522
No files matched your search
@@ -145,6 +145,8 @@ knobs.
|
||||
| **`fip_id` / liquid breadth** | Fingerprint −0.045 / t −2.91; liquid unconditional **−0.017 / t −1.85** (not green); mom-conditional **−0.088 / t −4.58** | **Parked.** Orphan +0.0575 died (snapshot race). Breadth did not strengthen resid-mom t-stat. Optional reopen = pre-registered two-arm liquid-1500 book first. See [fip-breadth-ic.md](fip-breadth-ic.md) |
|
||||
| **Broader universe** | Composition changes factor signs (fip tug-of-war); vol-tilt on breadth is only a **directional hypothesis** (auth. −0.048 / t −1.36) | Any prod broaden must re-validate 80/20 tilt; offline research only; research.sqlite requires completion manifest |
|
||||
| **Forward paper-trade record** | The only true out-of-sample evidence the snapshot cannot give | Time; mark entries at actual near-close fill once ops ships |
|
||||
| **Total-return momentum basis** | Ranks change every day (screen: **1,009 of 1,009** dates, ~4.6 names replaced per date, shift correlates **+0.79** with trailing yield) **and it reaches trades** — paired replay differs on ~30% of entries (51 control-only, 63 treatment-only, 271 shared). Which basis is better is unresolved: paired block bootstrap (5,000 reps, 21- and 63-session blocks, seed fixed beforehand) gives ΔCAGR −3.6/−3.8pp with 95% intervals spanning zero ([−14.8, +6.4] and [−14.7, +5.4]), ΔSharpe ≈ −0.11, ΔMaxDD −1.0/−1.8pp in treatment's favour; ~75% of replicates on the point-estimate side. All of the CAGR shortfall sits in 2025 | **Production unchanged.** Needs a threshold pre-registered before it is applied, plus out-of-sample evidence; a capacity/date/cadence matrix would be sensitivity checks, not independent samples. Schema/ingestion deferred. [Screen](tr-rank-agreement.md) · [Paired replay](tr-total-return-ab.md) · [plan](../data-integrity-plan.md) |
|
||||
| **Universe sensitivity / point-in-time membership** | Restoring 85 removed constituents (2021-06→2026-07, prod snapshot) moves the production row from **Sharpe 2.09 / CAGR 51.6%** to **1.57 / 38.1%** — all of it universe recomposition; no failed-bank position reached either book (trade extremes identical, and a control shows the extra replay signals changed no portfolio decision). Says the ranks are highly sensitive to the universe, not that bias is 13.5pp | Not a bias estimate: addition lookahead is still unremoved and arm B is not point-in-time. A real PIT audit is separate work. [Survivorship sensitivity](survivorship-audit.md) |
|
||||
| **Better target model for clear-air names** | The return is demonstrably there (#2 wins on raw CAGR in *both* train and test); it's the *flat* 3× ATR target that makes it too expensive in risk | Needs a per-name model, not a constant k×ATR |
|
||||
| **Minimum effective-risk floor** | ⛔ CLOSED NEGATIVE, not run. The floor lifts EV/trade (+0.032) and PF (+0.073) *by deleting trades* — 11.4 fewer per path, never one more — and costs **−0.753pp CAGR**, −0.047 Sharpe, −0.051 Calmar | Do not run the A/B; its EV-based pass rule would have shipped it. [Withdrawn specification](effective-risk-floor-ab.md) / [findings](portfolio-capacity-bracket-findings.md#correction-2026-08-05-ev-per-trade-was-the-wrong-lens) |
|
||||
|
||||
@@ -176,6 +178,14 @@ knobs.
|
||||
`research.sqlite`. Extender now writes a completion manifest; breadth mode
|
||||
refuses without a match. Same class of protection as calendar-truncation
|
||||
asserts — do not re-mythologize numbers computed on half a universe.
|
||||
8. **A big delta is not the mechanism you assumed.** Restoring 85 removed index
|
||||
constituents cost 13.5pp of CAGR — and none of it came from the failed banks
|
||||
the experiment was built around. The driver was the universe the ranks are
|
||||
computed over. Measure which part of a change did the work before naming the
|
||||
effect, and state what the control actually isolates: ours compared replay
|
||||
*bounds* on one snapshot, so it measured the extra signals, not the synthetic
|
||||
bars those signals resolve against.
|
||||
[Survivorship sensitivity](survivorship-audit.md)
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -0,0 +1,173 @@
|
||||
# Survivorship sensitivity — restoring removed S&P 500 constituents
|
||||
|
||||
**Run date:** 2026-09-07 · **Status:** measured, no strategy change proposed
|
||||
**Reports:** `reports/backtest-20260907-survivorship-a.json`,
|
||||
`reports/backtest-20260907-survivorship-b.json`,
|
||||
`reports/backtest-20260907-survivorship-b-nomarks.json`
|
||||
|
||||
## Question
|
||||
|
||||
The backtest universe is the ~505 names tracked *today*. Every constituent that
|
||||
left the index between 2021 and 2026 is missing from it, including the ones that
|
||||
failed. How different does the production strategy look when they are put back?
|
||||
|
||||
This is an **expanded-universe sensitivity analysis**, not a bias estimate. It is
|
||||
pre-registered as a measurement: there is no pass/fail rule and no promotion
|
||||
attached to the outcome.
|
||||
|
||||
## Method
|
||||
|
||||
**Base:** `backtest_snapshots/prod.sqlite`, sha256 `7241f0ff0f6187a0…`, window
|
||||
**2021-06-24 → 2026-07-02** (~5 years, not the full Alpaca history). Never
|
||||
modified — both arms are copies, and the hash is re-checked after the build.
|
||||
|
||||
| Arm | Universe |
|
||||
|---|---|
|
||||
| **A** — control | 506 tickers, the snapshot as it stands |
|
||||
| **B** — treatment | 591 tickers: the same 506 plus 85 restored removals |
|
||||
|
||||
Both arms ran the same binary, config, cadence (weekly) and date bounds, and are
|
||||
compared on the same report row: `portfolio_monitor.runs` where
|
||||
`is_production` is true and `lookback` is `all`
|
||||
(`residual80_highvol80_20_atr3`).
|
||||
|
||||
Removals come from `scripts/data/sp500_removals.json` — 92 rows curated from
|
||||
Wikipedia's *Historical components of the S&P 500* changes table, each carrying
|
||||
the verbatim source text plus independently measured Alpaca evidence. Build with
|
||||
`scripts/prep_survivorship_snapshot.py`.
|
||||
|
||||
Three data rules, all learned from the price data rather than assumed:
|
||||
|
||||
1. **Nothing is inferred from missing bars.** `delisted_on` / `delisted_reason`
|
||||
are written only for confirmed terminal events. A series that merely stops
|
||||
early leaves them NULL and is reported as an unresolved ending.
|
||||
2. **Series are shortened only at a reviewed cutoff.** Alpaca does not end a
|
||||
halted symbol's series — it repeats the last price at zero volume. SBNY sits
|
||||
at 70.00 for 509 sessions after the FDIC closed it; INFO sits at 108.61 for
|
||||
1,090 sessions, after which a *different company* reuses the ticker with no
|
||||
date gap. Both have explicit reviewed cutoffs (2023-03-10, 2022-02-25).
|
||||
Elsewhere a run of ≥5 zero-volume sessions is a tripwire: the symbol is left
|
||||
out and reported, never silently cut.
|
||||
3. **Terminal marks are explicit assumptions.** A synthetic `0.01` bar is written
|
||||
on the next session after the last real bar for SIVB, SBNY and FRC only —
|
||||
each with an FDIC source and only where the retained final bar matches the
|
||||
reviewed `last_real_bar`. The FDIC sources establish the closure *event*; they
|
||||
do not establish what holders recovered. The mark is a near-total-loss
|
||||
assumption, not an observed closing-day execution.
|
||||
|
||||
**Replay change.** `_replay_ticker` normally stops `HORIZON = 30` bars before the
|
||||
end so every setup has a full resolution window — which would hide every signal
|
||||
in the final 29 real bars, exactly the ones a collapse resolves. A new
|
||||
`--terminal-events` map opts the three marked symbols into walking through their
|
||||
last real bar; no setup is ever generated on the synthetic bar, and unmapped
|
||||
symbols are untouched. The map is validated against the snapshot's actual final
|
||||
bars *before* any replay is dispatched (`_validate_terminal_events`), because both
|
||||
dispatch paths swallow per-ticker exceptions and would otherwise turn a stale map
|
||||
into a quietly incomplete report.
|
||||
|
||||
## Result
|
||||
|
||||
Source: `portfolio_monitor.runs` where `is_production` is true and `lookback` is
|
||||
`all` (strategy `residual80_highvol80_20_atr3`), from
|
||||
`reports/backtest-20260907-survivorship-a.json` and
|
||||
`…-survivorship-b.json`. Both runs: weekly cadence, 6 workers,
|
||||
`--allow-spawn`, gate cutoff 80.0, full period.
|
||||
|
||||
| Metric | Arm A | Arm B | B − A |
|
||||
|---|---:|---:|---:|
|
||||
| Total return % | 424.60 | 264.40 | **−160.20** |
|
||||
| CAGR % | 51.60 | 38.10 | **−13.50** |
|
||||
| Max drawdown % | 21.40 | 23.60 | +2.20 |
|
||||
| Sharpe | 2.09 | 1.57 | **−0.52** |
|
||||
| Trades | 322 | 346 | +24 |
|
||||
| Win rate % | 37.0 | 35.0 | −2.0 |
|
||||
| Final equity | 52,459 | 36,439 | −16,020 |
|
||||
|
||||
Context: 202,765 → 229,642 candidates, 1,086 → 1,315 qualified. Gate cutoff 80.0
|
||||
in both arms.
|
||||
|
||||
## What actually caused it
|
||||
|
||||
**Not the failures.** `worst_trade_r` is **−3.26 in both arms**, and
|
||||
`best_trade_r` is 12.87 in both. A position held into a terminal mark gaps to
|
||||
$0.01 and fills at the open, which would land far below −3.26 — at candidate
|
||||
level those trades score −3.13R (SIVB) and −5.99R (SBNY). No such trade appears
|
||||
in either book, so none of the three failed banks reached the production
|
||||
portfolio.
|
||||
|
||||
**What the control does and does not show.** Arm B replayed *without*
|
||||
`--terminal-events` (`backtest-20260907-survivorship-b-nomarks.json`, same
|
||||
snapshot, same config) produces a production row identical to arm B with them in
|
||||
every field — 346 trades, 38.1% CAGR, 1.57 Sharpe, 23.6% drawdown, 264.4% total
|
||||
return, −3.26 worst trade. Candidates rise 229,617 → 229,642, the 25 extra
|
||||
signals the option exposes in the final 29 bars of the three symbols, while
|
||||
`qualified` is 1,315 either way.
|
||||
|
||||
So the control establishes precisely this: **those 25 additional replay signals
|
||||
changed no portfolio decision.** It does *not* independently establish that the
|
||||
terminal marks had no effect — both runs read `survivorship-b.sqlite`, which
|
||||
contains the synthetic bars either way, so a setup generated earlier could still
|
||||
resolve against one in both runs. Isolating the marks themselves would need a
|
||||
third snapshot built without them, which has not been run.
|
||||
|
||||
The separate evidence that no marked position reached the book is the trade
|
||||
extremes: a position resolving against a $0.01 bar fills at that bar's open and
|
||||
scores far worse than −3.26 (−3.13R for SIVB and −5.99R for SBNY at candidate
|
||||
level), yet arm B's `worst_trade_r` equals arm A's exactly. That is inference
|
||||
from the extremes, not a controlled measurement.
|
||||
|
||||
On that basis the −13.5pp CAGR is **universe recomposition**: 85 additional names
|
||||
(+17% of the universe) shift the cross-sectional percentile ranks and compete for
|
||||
the same 15 position slots, displacing trades arm A took. The strategy is
|
||||
markedly sensitive to which universe it ranks over — which is the finding, and it
|
||||
is a different claim from "survivorship bias costs 13.5pp".
|
||||
|
||||
## Caveats
|
||||
|
||||
- **This is not a bias estimate and not a lower bound.** Restoring names is only
|
||||
one of the two corrections. Index-*addition* lookahead — today's members being
|
||||
tradeable before they joined — is not removed. Either correction can move
|
||||
returns in either direction, so B − A is the sensitivity observed under these
|
||||
assumptions, nothing more. A small delta would not have shown the bias to be
|
||||
small, and this large one does not establish its size either.
|
||||
- **Arm B is not a point-in-time universe.** Still-trading former members remain
|
||||
eligible after their removal date; `run_backtest` does not gate on
|
||||
`delisted_on`, and an index-removal date would be the wrong field for it
|
||||
anyway. A true PIT audit is separate work.
|
||||
- **Four unresolved endings.** CMA (ends 2026-01-30), SEE (2026-04-08), HBI
|
||||
(2025-11-28) and DISH (2023-12-29) stop trading before the window closes for
|
||||
reasons the source row does not explain. Their bars are real and their
|
||||
delisting fields are NULL; the ending status is simply unknown.
|
||||
- **Three near-total-loss assumptions** (SIVB, SBNY, FRC), as described above.
|
||||
The control shows they did not affect this result, but they remain assumptions
|
||||
and would matter in any run where such a name did clear the gate.
|
||||
- **Acquisitions are approximated** by the last available close. Cash or stock
|
||||
consideration and the release of capital on the deal date are not modelled.
|
||||
- **FBIN carries a corporate action.** Fortune Brands distributed MasterBrand on
|
||||
2022-12-14 and began trading as FBIN on 2022-12-15. Alpaca's split adjustment
|
||||
does not adjust for spin-offs, so the distribution appears as a real −6.91%
|
||||
close-to-close move (63.22 → 61.90 → 57.62). A holder did not lose that value,
|
||||
but any signal or stop crossing that date reads it as a decline. MBC enters the
|
||||
dataset separately with its first bar the same day.
|
||||
- **Window.** 2021-06 to 2026-07, because that is what `prod.sqlite` holds. The
|
||||
2023 bank failures are inside it; the 2016–2021 period is not.
|
||||
- **Snapshot schema.** The snapshot files on disk predate migrations 026 and 032,
|
||||
so `select(Ticker)` against them fails with `no such column: tickers.cik` — the
|
||||
path `backtest_service._fetch_columns` takes. Per-ticker exceptions are caught
|
||||
and logged, so a run against an unsynced snapshot completes and writes a report
|
||||
with **zero candidates** rather than failing. The prep script syncs both copies
|
||||
against `Base.metadata`; anything else reading these snapshots needs the same.
|
||||
|
||||
## Reproduce
|
||||
|
||||
```powershell
|
||||
python scripts/prep_survivorship_snapshot.py --base backtest_snapshots/prod.sqlite
|
||||
python scripts/run_backtest_snapshot.py backtest_snapshots/survivorship-a.sqlite `
|
||||
--out reports/backtest-YYYYMMDD-survivorship-a.json --allow-spawn --workers 6
|
||||
python scripts/run_backtest_snapshot.py backtest_snapshots/survivorship-b.sqlite `
|
||||
--terminal-events backtest_snapshots/survivorship-b-terminal-events.json `
|
||||
--out reports/backtest-YYYYMMDD-survivorship-b.json --allow-spawn --workers 6
|
||||
```
|
||||
|
||||
`survivorship-b-prep-manifest.json` records the base hash, per-symbol coverage,
|
||||
the terminal events written and every unresolved item for the run.
|
||||
@@ -0,0 +1,118 @@
|
||||
# Total-return vs price-return momentum — ranking screen
|
||||
|
||||
**Run date:** 2026-09-07 · **Verdict: proceed to Step 3** (pre-registered rule fired)
|
||||
**Reports:** `reports/tr-rank-agreement.json`, `reports/tr-rank-agreement-mechanism.json`
|
||||
**Script:** `scripts/research_tr_rank_agreement.py`
|
||||
**Cache:** `backtest_snapshots/tr-rank-cache.sqlite` (507 symbols, 630,524 aligned bars)
|
||||
|
||||
## Question
|
||||
|
||||
12-1 momentum is defined on total returns; production computes it on
|
||||
split-adjusted prices, which ignore dividends. Before building the production
|
||||
`tr_factor` machinery (Step 3 of [the plan](../data-integrity-plan.md)), does the
|
||||
basis actually change what the gate admits and how it orders?
|
||||
|
||||
Deliberately a screen, not a portfolio simulation. A difference here is a reason
|
||||
to measure trades, **not** evidence that P&L changes.
|
||||
|
||||
## Configuration
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Universe | `backtest_snapshots/survivorship-a.sqlite`, 505 symbols with bars (RHM has a ticker row but no bars in the snapshot, so it is inert in both arms) |
|
||||
| Window | 2021-06-24 → 2026-07-02, clipped to the snapshot |
|
||||
| Decision dates | every SPY session; 1,009 had enough history to evaluate |
|
||||
| Momentum | production `compute_residual_12_1_momentum`, unchanged; price series against nominal SPY, total-return series against total-return SPY |
|
||||
| Volatility leg | production `compute_realized_vol_6m` on the SPLIT series, computed once per date and **held fixed in both arms** so only momentum varies |
|
||||
| Gate | production `min_momentum_percentile` = 80.0, via production `_percentiles` |
|
||||
| Ordering | production `blend_strategy_rank`, symbol as deterministic tie-break, symbols fed in sorted order in both arms |
|
||||
| Comparison | the full admitted list, never truncated to `max_positions` |
|
||||
| Runtime | 146 s with the production function — no NumPy rewrite needed, so no equivalence test was required |
|
||||
|
||||
**Coverage is complete.** One `Adjustment.SPLIT` and one `Adjustment.ALL` request
|
||||
per symbol; 507 symbols cached, **zero** dates present in one adjustment but not
|
||||
the other. No symbol with snapshot bars is missing from the cache — the script
|
||||
refuses to run in that case rather than reporting the gap as agreement.
|
||||
|
||||
## Result
|
||||
|
||||
| | |
|
||||
|---|---:|
|
||||
| Evaluated dates | 1,009 |
|
||||
| Dates with a **different ordered list** | **1,009 (100%)** |
|
||||
| Dates with a different admitted set | 1,007 (99.8%) |
|
||||
| Mean set disagreement, `\|A △ B\| / \|A ∪ B\|` | 0.0867 |
|
||||
| Max set disagreement | 0.2143 |
|
||||
| Mean admitted names per arm | 100.09 |
|
||||
| Mean **replacements** per date (one out paired with one in) | **4.56** |
|
||||
| Max replacements on a single date | 12 |
|
||||
| Mean symmetric difference (names counted on both sides) | 9.13 |
|
||||
|
||||
Per year, every evaluated date differs:
|
||||
|
||||
| Year | Dates | Different set | Different order |
|
||||
|---|---:|---:|---:|
|
||||
| 2022 | 132 | 132 | 132 |
|
||||
| 2023 | 250 | 250 | 250 |
|
||||
| 2024 | 252 | 252 | 252 |
|
||||
| 2025 | 250 | 248 | 250 |
|
||||
| 2026 | 125 | 125 | 125 |
|
||||
|
||||
Counted directly from the daily set differences rather than derived from the
|
||||
Jaccard ratio: **4.56 names are replaced on an average date** — 4.56 leave and
|
||||
4.56 arrive, so 9.13 names appear on one side or the other, out of about 100
|
||||
admitted. A swap moves two names, which makes the symmetric difference twice the
|
||||
replacement count and the two easy to conflate.
|
||||
|
||||
## Strong evidence of a dividend effect
|
||||
|
||||
First changed date, 2022-06-24 — 99 names admitted in each arm, nine swapped:
|
||||
|
||||
- **Only on price return:** CDNS, MAR, FIX, COST, LIN, AON, STE, CI, AJG
|
||||
- **Only on total return:** EXC, PSX, WY, AMCR, DTE, GIS, FE, BMY, ES
|
||||
|
||||
Growth and low-yield names out; utilities, energy, staples and other high payers
|
||||
in. Nine tickers are not evidence, so it was measured (`--mechanism 25`, saved to
|
||||
`reports/tr-rank-agreement-mechanism.json` with its sampled dates and per-date
|
||||
values): across 40 sampled dates, the Spearman correlation between a name's
|
||||
trailing 12-month implied yield — from the ALL/SPLIT ratio over the formation
|
||||
window — and its momentum-percentile shift between arms is **+0.79** (range
|
||||
+0.71 to +0.85, every sampled date positive).
|
||||
|
||||
**What that does and does not pin down.** The reordering is systematic and
|
||||
directional rather than numerical noise, and it moves with yield. It is *not* a
|
||||
clean isolation of the stock's own yield: both the stock series and the benchmark
|
||||
leg change between arms, so the measured shift also contains the beta-weighted
|
||||
benchmark yield. Read it as strong evidence consistent with a dividend effect.
|
||||
Isolating own-yield would need a third variant — total-return stock series
|
||||
against a nominal SPY leg — which has not been run.
|
||||
|
||||
## Verdict
|
||||
|
||||
The pre-registered rule was "no ordered-list difference on any evaluated date →
|
||||
stop". Differences occur on **every** evaluated date, so **Step 3 proceeds**.
|
||||
|
||||
What this does and does not establish:
|
||||
|
||||
- **Does:** the momentum basis materially changes gate admission and ordering,
|
||||
every day, in a yield-driven direction.
|
||||
- **Does not:** show that realised P&L changes. Production also applies setup
|
||||
qualification, existing holdings, available slots and the rest of the admission
|
||||
path. A name that swaps into the admitted list may never produce a setup. Step
|
||||
3's paired portfolio replay is what decides whether any of this reaches trades.
|
||||
- **Worth noting:** the tuning recorded as closed in the research log was carried
|
||||
out on a price-return basis throughout. This screen does not invalidate it —
|
||||
it says the input those conclusions were computed on is not the one the factor
|
||||
is defined on.
|
||||
|
||||
## Reproduce
|
||||
|
||||
```powershell
|
||||
python scripts/research_tr_rank_agreement.py --fetch # 1,014 requests, cached
|
||||
python scripts/research_tr_rank_agreement.py --time-batch 20 # runtime projection
|
||||
python scripts/research_tr_rank_agreement.py # full screen
|
||||
python scripts/research_tr_rank_agreement.py --mechanism 25 # yield vs percentile shift
|
||||
```
|
||||
|
||||
The cache makes re-runs free and the screen deterministic; delete
|
||||
`backtest_snapshots/tr-rank-cache.sqlite` to refetch.
|
||||
@@ -0,0 +1,160 @@
|
||||
# Total-return momentum — paired portfolio replay (Step 3a)
|
||||
|
||||
**Run date:** 2026-09-07 · **Status:** measured; **no rollout decision taken**
|
||||
**Report:** `reports/tr-total-return-ab.json` · **Script:** `scripts/run_total_return_ab.py`
|
||||
**Inputs:** `backtest_snapshots/survivorship-a.sqlite` (bars),
|
||||
`backtest_snapshots/tr-rank-cache.sqlite` (both adjusted series),
|
||||
`backtest_snapshots/tr-ab-candidates.pkl` (202,765 shared candidates)
|
||||
|
||||
## Question
|
||||
|
||||
[The ranking screen](tr-rank-agreement.md) showed the momentum basis changes the
|
||||
admitted list on every evaluated date. Does that reach actual admissions and P&L?
|
||||
|
||||
## What was held identical
|
||||
|
||||
Setup candidates were generated **once** (202,765 over 505 symbols, matching the
|
||||
count in the standalone backtest report) and reused by both books. Only momentum
|
||||
and its dependents were recomputed per arm: residual and activation percentiles,
|
||||
`qualified`, and the 80/20 blend used for ordering.
|
||||
|
||||
| Held fixed | Varied |
|
||||
|---|---|
|
||||
| Setup geometry — entry, stop, target, R:R | Momentum series: SPLIT stock vs SPLIT SPY (control), ALL vs ALL SPY (treatment) |
|
||||
| Execution on the snapshot's nominal bars | The percentiles, `qualified` and ordering that follow from it |
|
||||
| `vol_6m` volatility leg, from candidate generation | |
|
||||
| Non-momentum gates, `max_positions`, risk per trade, ATR-trail exit, gate-reset re-entry | |
|
||||
|
||||
Both arms read the same cache, so a refreshed-vs-snapshot difference cannot be
|
||||
mistaken for a basis difference. Each book then keeps its own holdings, so a
|
||||
divergent admission changes later slot occupancy — that path dependence is part
|
||||
of what is measured.
|
||||
|
||||
**Harness parity.** The control arm reproduces the production row of
|
||||
`reports/backtest-20260907-survivorship-a.json` exactly — 322 trades, 424.6%
|
||||
total return, 51.6% CAGR, 21.4% drawdown, 2.09 Sharpe. Any difference below is
|
||||
therefore attributable to the basis, not to a re-implemented book.
|
||||
|
||||
## Result
|
||||
|
||||
| Metric | Control (price return) | Treatment (total return) | Δ |
|
||||
|---|---:|---:|---:|
|
||||
| Trades | 322 | 334 | +12 |
|
||||
| Total return % | 424.6 | 376.5 | −48.1 |
|
||||
| CAGR % | 51.6 | 47.9 | **−3.7** |
|
||||
| Max drawdown % | 21.4 | 19.5 | **−1.9** (better) |
|
||||
| Sharpe | 2.09 | 1.98 | −0.11 |
|
||||
| Win rate % | 37.0 | 36.8 | −0.2 |
|
||||
| Profit factor | 2.48 | 2.33 | −0.15 |
|
||||
|
||||
**The books really do differ.** 271 entries are shared; **51 are taken only by
|
||||
control and 63 only by treatment** — about 30% of the union of entries. The
|
||||
composition follows the Step 2 mechanism: control takes MAR, AAPL, BIIB, LVS;
|
||||
treatment takes BG, OKE, DE, ARES. Some differences are timing rather than
|
||||
selection — ABBV is entered on 2022-09-15 by control and 2022-09-22 by treatment.
|
||||
|
||||
## Where the difference sits
|
||||
|
||||
**One year carries all of it.** Per-year returns:
|
||||
|
||||
| Year | Control % | Treatment % | Δpp |
|
||||
|---|---:|---:|---:|
|
||||
| 2022 | 8.1 | 9.9 | +1.8 |
|
||||
| 2023 | 42.0 | 41.9 | −0.1 |
|
||||
| 2024 | 36.4 | 38.4 | +2.0 |
|
||||
| 2025 | 80.2 | 58.1 | **−22.1** |
|
||||
| 2026 | 39.2 | 39.6 | +0.4 |
|
||||
|
||||
Treatment wins three years and loses two, and four of the five differ by under
|
||||
2.1pp. The entire −3.7pp CAGR gap is 2025. That is a description of *where* the
|
||||
shortfall sits — a useful thing to know when reading it — and nothing more. It is
|
||||
not grounds for calling the shortfall noise or the estimate unstable.
|
||||
|
||||
**Uncertainty is not yet quantified.** The per-arm `sharpe_se` values (0.497 and
|
||||
0.498) are the standard errors of each Sharpe *individually*. They say nothing
|
||||
about the uncertainty of the **difference**: the two books share dates and most
|
||||
of their positions, so the arms are strongly correlated and the standard error of
|
||||
their difference depends on that covariance, not on the two marginal errors. An
|
||||
earlier version of this document compared the 0.11 Sharpe gap to a 0.50 marginal
|
||||
SE and concluded the books were "statistically indistinguishable" — that
|
||||
inference does not follow, and the same applies to the drawdown difference, for
|
||||
which no error estimate was reported at all. Both claims are withdrawn.
|
||||
|
||||
## Paired block bootstrap on the difference
|
||||
|
||||
Pre-specified before running — seed `20260907`, **5,000** replicates, block
|
||||
lengths **21 and 63** sessions, both reported rather than whichever reads better.
|
||||
The two arms' daily return streams are date-aligned (the script aborts if they
|
||||
are not), and **the same block indices are drawn for both arms**, so the pairing
|
||||
and whatever cross-arm correlation exists on those days are preserved. Moving
|
||||
blocks retain some of the serial dependence an i.i.d. resample would destroy.
|
||||
|
||||
Treatment − control, 95% interval:
|
||||
|
||||
| Block | Metric | Mean | 2.5% | 97.5% | Share of replicates < 0 |
|
||||
|---|---|---:|---:|---:|---:|
|
||||
| 21 | ΔCAGR pp | −3.61 | −14.76 | +6.45 | 0.75 |
|
||||
| 21 | ΔSharpe | −0.109 | −0.433 | +0.198 | 0.75 |
|
||||
| 21 | ΔMaxDD pp | −1.00 | −7.50 | +4.34 | 0.63 |
|
||||
| 63 | ΔCAGR pp | −3.78 | −14.67 | +5.40 | 0.76 |
|
||||
| 63 | ΔSharpe | −0.113 | −0.428 | +0.182 | 0.76 |
|
||||
| 63 | ΔMaxDD pp | −1.79 | −8.22 | +2.68 | 0.78 |
|
||||
|
||||
The two block lengths agree closely, which is reassuring about the resampling
|
||||
itself. Every interval spans zero, and roughly three-quarters of replicates fall
|
||||
below zero for CAGR and Sharpe (treatment worse) and for drawdown (treatment
|
||||
better, since a negative ΔMaxDD is a shallower drawdown). So the observed window
|
||||
leans toward control on return and treatment on drawdown without resolving either
|
||||
sign confidently.
|
||||
|
||||
**What this diagnostic is not.** It resamples the *realised* daily return streams;
|
||||
it does not re-run the strategy, so it cannot capture how a reordered history
|
||||
would have changed admissions or holdings. It is an exploratory uncertainty
|
||||
estimate from the observed window — not out-of-sample validation, and not a
|
||||
rollout gate.
|
||||
|
||||
## Verdict
|
||||
|
||||
**Decision: production stays unchanged.** No schema, no ingestion, no wider
|
||||
parameter sweep.
|
||||
|
||||
- **Established:** the basis is production-relevant. It changes roughly 30% of
|
||||
entries, in the yield-driven direction the screen predicted.
|
||||
- **Not established:** which basis performs better. Point estimates favour control
|
||||
on return and Sharpe and treatment on drawdown; every bootstrap interval spans
|
||||
zero, with about three-quarters of the mass on the point-estimate side. The
|
||||
observed window does not resolve the sign.
|
||||
- No acceptable return sacrifice is being chosen from these numbers, and no
|
||||
threshold is being fitted to them after the fact.
|
||||
|
||||
What a decision would still need:
|
||||
|
||||
1. **A pre-registered numeric rule, set before it is applied.** The plan commits
|
||||
to non-inferiority in principle — total return is the basis 12-1 momentum is
|
||||
defined on, so an in-sample shortfall should not automatically lock in the
|
||||
basis we believe is measured wrong — but carries no threshold. An earlier draft
|
||||
used ΔCAGR ≥ −0.50pp at ΔMaxDD ≤ +1.50pp. Choosing a threshold now, with
|
||||
−3.7pp on the table, would be fitting the rule to the answer.
|
||||
2. **Out-of-sample evidence.** One five-year snapshot, in-sample throughout.
|
||||
3. Note what would *not* help: a matrix over capacity, start date or cadence
|
||||
produces **sensitivity checks, not independent samples**. Fifty-three
|
||||
overlapping paths from one price history are not fifty-three pieces of
|
||||
evidence, and treating them that way would overstate what a sweep buys.
|
||||
|
||||
## Reproduce
|
||||
|
||||
```powershell
|
||||
python scripts/run_total_return_ab.py --build-candidates # ~9 min, cached
|
||||
python scripts/run_total_return_ab.py # both arms + bootstrap, ~40 s
|
||||
```
|
||||
|
||||
The bootstrap parameters are constants in the script (`BOOTSTRAP_SEED`,
|
||||
`BOOTSTRAP_REPLICATES`, `BOOTSTRAP_BLOCKS`) rather than flags, so a run cannot
|
||||
quietly pick a different seed or block length.
|
||||
|
||||
Guards in `tests/unit/test_total_return_ab.py`: identical input series must
|
||||
produce identical books (so an observed difference is the basis, not replay
|
||||
nondeterminism); a controlled momentum change must move a name across the cutoff
|
||||
(so the harness can see what it is looking for); the volatility leg must be
|
||||
untouched between arms; and a name with no computable momentum in an arm must not
|
||||
qualify on a value carried over from candidate generation.
|
||||
Reference in new issue
Block a user