research: survivorship sensitivity and total-return momentum investigation
Three questions answered against snapshot copies, with no production change. Step 1 — how sensitive is the backtest to the ranking universe? Restoring 85 S&P 500 constituents removed between 2021-06 and 2026-07 moves the same window from 2.09 Sharpe / 51.6% CAGR to 1.57 / 38.1%. The failed banks never reached either book (identical best/worst trade R), so this measures universe recomposition, not survivorship bias. The removals list is curated by hand with per-symbol sources; only three explicitly confirmed terminal events get marks, and four unresolved endings are reported as caveats rather than inferred. Step 2 — does a total-return basis change the ranking? Ordered lists differ on 1009 of 1009 evaluated dates, 4.56 replacements per date. The shift correlates with trailing implied yield (mean Spearman +0.79), which is strong evidence consistent with a dividend effect but does not isolate own-yield, since both the stock and benchmark legs change. Step 3a — does that reach P&L? A paired replay reusing one shared candidate set (control reproduces the production row exactly) changes ~30% of entries. Treatment is -3.7pp CAGR and -1.9pp drawdown, with the entire return gap in 2025. A paired moving-block bootstrap (seed 20260907, 5000 replicates, 21/63 blocks, both reported) puts every interval across zero. Decision: production stays unchanged. No schema, no ingestion, no wider sweep. The observed window does not resolve which basis is better, and choosing a threshold now — with -3.7pp on the table — would be fitting the rule to the answer. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HAH7vVtn7qbcn1EVheecHQ
This commit is contained in:
1 parent
8453b87290
commit
ebe9047403
16 files changed
+442522
No files matched your search
@@ -145,6 +145,8 @@ knobs.
|
||||
| **`fip_id` / liquid breadth** | Fingerprint −0.045 / t −2.91; liquid unconditional **−0.017 / t −1.85** (not green); mom-conditional **−0.088 / t −4.58** | **Parked.** Orphan +0.0575 died (snapshot race). Breadth did not strengthen resid-mom t-stat. Optional reopen = pre-registered two-arm liquid-1500 book first. See [fip-breadth-ic.md](fip-breadth-ic.md) |
|
||||
| **Broader universe** | Composition changes factor signs (fip tug-of-war); vol-tilt on breadth is only a **directional hypothesis** (auth. −0.048 / t −1.36) | Any prod broaden must re-validate 80/20 tilt; offline research only; research.sqlite requires completion manifest |
|
||||
| **Forward paper-trade record** | The only true out-of-sample evidence the snapshot cannot give | Time; mark entries at actual near-close fill once ops ships |
|
||||
| **Total-return momentum basis** | Ranks change every day (screen: **1,009 of 1,009** dates, ~4.6 names replaced per date, shift correlates **+0.79** with trailing yield) **and it reaches trades** — paired replay differs on ~30% of entries (51 control-only, 63 treatment-only, 271 shared). Which basis is better is unresolved: paired block bootstrap (5,000 reps, 21- and 63-session blocks, seed fixed beforehand) gives ΔCAGR −3.6/−3.8pp with 95% intervals spanning zero ([−14.8, +6.4] and [−14.7, +5.4]), ΔSharpe ≈ −0.11, ΔMaxDD −1.0/−1.8pp in treatment's favour; ~75% of replicates on the point-estimate side. All of the CAGR shortfall sits in 2025 | **Production unchanged.** Needs a threshold pre-registered before it is applied, plus out-of-sample evidence; a capacity/date/cadence matrix would be sensitivity checks, not independent samples. Schema/ingestion deferred. [Screen](tr-rank-agreement.md) · [Paired replay](tr-total-return-ab.md) · [plan](../data-integrity-plan.md) |
|
||||
| **Universe sensitivity / point-in-time membership** | Restoring 85 removed constituents (2021-06→2026-07, prod snapshot) moves the production row from **Sharpe 2.09 / CAGR 51.6%** to **1.57 / 38.1%** — all of it universe recomposition; no failed-bank position reached either book (trade extremes identical, and a control shows the extra replay signals changed no portfolio decision). Says the ranks are highly sensitive to the universe, not that bias is 13.5pp | Not a bias estimate: addition lookahead is still unremoved and arm B is not point-in-time. A real PIT audit is separate work. [Survivorship sensitivity](survivorship-audit.md) |
|
||||
| **Better target model for clear-air names** | The return is demonstrably there (#2 wins on raw CAGR in *both* train and test); it's the *flat* 3× ATR target that makes it too expensive in risk | Needs a per-name model, not a constant k×ATR |
|
||||
| **Minimum effective-risk floor** | ⛔ CLOSED NEGATIVE, not run. The floor lifts EV/trade (+0.032) and PF (+0.073) *by deleting trades* — 11.4 fewer per path, never one more — and costs **−0.753pp CAGR**, −0.047 Sharpe, −0.051 Calmar | Do not run the A/B; its EV-based pass rule would have shipped it. [Withdrawn specification](effective-risk-floor-ab.md) / [findings](portfolio-capacity-bracket-findings.md#correction-2026-08-05-ev-per-trade-was-the-wrong-lens) |
|
||||
|
||||
@@ -176,6 +178,14 @@ knobs.
|
||||
`research.sqlite`. Extender now writes a completion manifest; breadth mode
|
||||
refuses without a match. Same class of protection as calendar-truncation
|
||||
asserts — do not re-mythologize numbers computed on half a universe.
|
||||
8. **A big delta is not the mechanism you assumed.** Restoring 85 removed index
|
||||
constituents cost 13.5pp of CAGR — and none of it came from the failed banks
|
||||
the experiment was built around. The driver was the universe the ranks are
|
||||
computed over. Measure which part of a change did the work before naming the
|
||||
effect, and state what the control actually isolates: ours compared replay
|
||||
*bounds* on one snapshot, so it measured the extra signals, not the synthetic
|
||||
bars those signals resolve against.
|
||||
[Survivorship sensitivity](survivorship-audit.md)
|
||||
|
||||
---
|
||||
|
||||
|
||||
Reference in new issue
Block a user