dennisthiessenandClaude Opus 5
ebe9047403
Three questions answered against snapshot copies, with no production change. Step 1 — how sensitive is the backtest to the ranking universe? Restoring 85 S&P 500 constituents removed between 2021-06 and 2026-07 moves the same window from 2.09 Sharpe / 51.6% CAGR to 1.57 / 38.1%. The failed banks never reached either book (identical best/worst trade R), so this measures universe recomposition, not survivorship bias. The removals list is curated by hand with per-symbol sources; only three explicitly confirmed terminal events get marks, and four unresolved endings are reported as caveats rather than inferred. Step 2 — does a total-return basis change the ranking? Ordered lists differ on 1009 of 1009 evaluated dates, 4.56 replacements per date. The shift correlates with trailing implied yield (mean Spearman +0.79), which is strong evidence consistent with a dividend effect but does not isolate own-yield, since both the stock and benchmark legs change. Step 3a — does that reach P&L? A paired replay reusing one shared candidate set (control reproduces the production row exactly) changes ~30% of entries. Treatment is -3.7pp CAGR and -1.9pp drawdown, with the entire return gap in 2025. A paired moving-block bootstrap (seed 20260907, 5000 replicates, 21/63 blocks, both reported) puts every interval across zero. Decision: production stays unchanged. No schema, no ingestion, no wider sweep. The observed window does not resolve which basis is better, and choosing a threshold now — with -3.7pp on the table — would be fitting the rule to the answer. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HAH7vVtn7qbcn1EVheecHQ
119 lines
5.9 KiB
Markdown
119 lines
5.9 KiB
Markdown
# Total-return vs price-return momentum — ranking screen
|
||
|
||
**Run date:** 2026-09-07 · **Verdict: proceed to Step 3** (pre-registered rule fired)
|
||
**Reports:** `reports/tr-rank-agreement.json`, `reports/tr-rank-agreement-mechanism.json`
|
||
**Script:** `scripts/research_tr_rank_agreement.py`
|
||
**Cache:** `backtest_snapshots/tr-rank-cache.sqlite` (507 symbols, 630,524 aligned bars)
|
||
|
||
## Question
|
||
|
||
12-1 momentum is defined on total returns; production computes it on
|
||
split-adjusted prices, which ignore dividends. Before building the production
|
||
`tr_factor` machinery (Step 3 of [the plan](../data-integrity-plan.md)), does the
|
||
basis actually change what the gate admits and how it orders?
|
||
|
||
Deliberately a screen, not a portfolio simulation. A difference here is a reason
|
||
to measure trades, **not** evidence that P&L changes.
|
||
|
||
## Configuration
|
||
|
||
| | |
|
||
|---|---|
|
||
| Universe | `backtest_snapshots/survivorship-a.sqlite`, 505 symbols with bars (RHM has a ticker row but no bars in the snapshot, so it is inert in both arms) |
|
||
| Window | 2021-06-24 → 2026-07-02, clipped to the snapshot |
|
||
| Decision dates | every SPY session; 1,009 had enough history to evaluate |
|
||
| Momentum | production `compute_residual_12_1_momentum`, unchanged; price series against nominal SPY, total-return series against total-return SPY |
|
||
| Volatility leg | production `compute_realized_vol_6m` on the SPLIT series, computed once per date and **held fixed in both arms** so only momentum varies |
|
||
| Gate | production `min_momentum_percentile` = 80.0, via production `_percentiles` |
|
||
| Ordering | production `blend_strategy_rank`, symbol as deterministic tie-break, symbols fed in sorted order in both arms |
|
||
| Comparison | the full admitted list, never truncated to `max_positions` |
|
||
| Runtime | 146 s with the production function — no NumPy rewrite needed, so no equivalence test was required |
|
||
|
||
**Coverage is complete.** One `Adjustment.SPLIT` and one `Adjustment.ALL` request
|
||
per symbol; 507 symbols cached, **zero** dates present in one adjustment but not
|
||
the other. No symbol with snapshot bars is missing from the cache — the script
|
||
refuses to run in that case rather than reporting the gap as agreement.
|
||
|
||
## Result
|
||
|
||
| | |
|
||
|---|---:|
|
||
| Evaluated dates | 1,009 |
|
||
| Dates with a **different ordered list** | **1,009 (100%)** |
|
||
| Dates with a different admitted set | 1,007 (99.8%) |
|
||
| Mean set disagreement, `\|A △ B\| / \|A ∪ B\|` | 0.0867 |
|
||
| Max set disagreement | 0.2143 |
|
||
| Mean admitted names per arm | 100.09 |
|
||
| Mean **replacements** per date (one out paired with one in) | **4.56** |
|
||
| Max replacements on a single date | 12 |
|
||
| Mean symmetric difference (names counted on both sides) | 9.13 |
|
||
|
||
Per year, every evaluated date differs:
|
||
|
||
| Year | Dates | Different set | Different order |
|
||
|---|---:|---:|---:|
|
||
| 2022 | 132 | 132 | 132 |
|
||
| 2023 | 250 | 250 | 250 |
|
||
| 2024 | 252 | 252 | 252 |
|
||
| 2025 | 250 | 248 | 250 |
|
||
| 2026 | 125 | 125 | 125 |
|
||
|
||
Counted directly from the daily set differences rather than derived from the
|
||
Jaccard ratio: **4.56 names are replaced on an average date** — 4.56 leave and
|
||
4.56 arrive, so 9.13 names appear on one side or the other, out of about 100
|
||
admitted. A swap moves two names, which makes the symmetric difference twice the
|
||
replacement count and the two easy to conflate.
|
||
|
||
## Strong evidence of a dividend effect
|
||
|
||
First changed date, 2022-06-24 — 99 names admitted in each arm, nine swapped:
|
||
|
||
- **Only on price return:** CDNS, MAR, FIX, COST, LIN, AON, STE, CI, AJG
|
||
- **Only on total return:** EXC, PSX, WY, AMCR, DTE, GIS, FE, BMY, ES
|
||
|
||
Growth and low-yield names out; utilities, energy, staples and other high payers
|
||
in. Nine tickers are not evidence, so it was measured (`--mechanism 25`, saved to
|
||
`reports/tr-rank-agreement-mechanism.json` with its sampled dates and per-date
|
||
values): across 40 sampled dates, the Spearman correlation between a name's
|
||
trailing 12-month implied yield — from the ALL/SPLIT ratio over the formation
|
||
window — and its momentum-percentile shift between arms is **+0.79** (range
|
||
+0.71 to +0.85, every sampled date positive).
|
||
|
||
**What that does and does not pin down.** The reordering is systematic and
|
||
directional rather than numerical noise, and it moves with yield. It is *not* a
|
||
clean isolation of the stock's own yield: both the stock series and the benchmark
|
||
leg change between arms, so the measured shift also contains the beta-weighted
|
||
benchmark yield. Read it as strong evidence consistent with a dividend effect.
|
||
Isolating own-yield would need a third variant — total-return stock series
|
||
against a nominal SPY leg — which has not been run.
|
||
|
||
## Verdict
|
||
|
||
The pre-registered rule was "no ordered-list difference on any evaluated date →
|
||
stop". Differences occur on **every** evaluated date, so **Step 3 proceeds**.
|
||
|
||
What this does and does not establish:
|
||
|
||
- **Does:** the momentum basis materially changes gate admission and ordering,
|
||
every day, in a yield-driven direction.
|
||
- **Does not:** show that realised P&L changes. Production also applies setup
|
||
qualification, existing holdings, available slots and the rest of the admission
|
||
path. A name that swaps into the admitted list may never produce a setup. Step
|
||
3's paired portfolio replay is what decides whether any of this reaches trades.
|
||
- **Worth noting:** the tuning recorded as closed in the research log was carried
|
||
out on a price-return basis throughout. This screen does not invalidate it —
|
||
it says the input those conclusions were computed on is not the one the factor
|
||
is defined on.
|
||
|
||
## Reproduce
|
||
|
||
```powershell
|
||
python scripts/research_tr_rank_agreement.py --fetch # 1,014 requests, cached
|
||
python scripts/research_tr_rank_agreement.py --time-batch 20 # runtime projection
|
||
python scripts/research_tr_rank_agreement.py # full screen
|
||
python scripts/research_tr_rank_agreement.py --mechanism 25 # yield vs percentile shift
|
||
```
|
||
|
||
The cache makes re-runs free and the screen deterministic; delete
|
||
`backtest_snapshots/tr-rank-cache.sqlite` to refetch.
|