signal-platform/docs/research/tr-total-return-ab.md

Total-return momentum — paired portfolio replay (Step 3a)

Run date: 2026-09-07 · Status: measured; no rollout decision taken Report: reports/tr-total-return-ab.json · Script: scripts/run_total_return_ab.py Inputs: backtest_snapshots/survivorship-a.sqlite (bars), backtest_snapshots/tr-rank-cache.sqlite (both adjusted series), backtest_snapshots/tr-ab-candidates.pkl (202,765 shared candidates)

Question

The ranking screen showed the momentum basis changes the admitted list on every evaluated date. Does that reach actual admissions and P&L?

What was held identical

Setup candidates were generated once (202,765 over 505 symbols, matching the count in the standalone backtest report) and reused by both books. Only momentum and its dependents were recomputed per arm: residual and activation percentiles, qualified, and the 80/20 blend used for ordering.

Held fixed	Varied
Setup geometry — entry, stop, target, R:R	Momentum series: SPLIT stock vs SPLIT SPY (control), ALL vs ALL SPY (treatment)
Execution on the snapshot's nominal bars	The percentiles, qualified and ordering that follow from it
vol_6m volatility leg, from candidate generation
Non-momentum gates, max_positions, risk per trade, ATR-trail exit, gate-reset re-entry

Both arms read the same cache, so a refreshed-vs-snapshot difference cannot be mistaken for a basis difference. Each book then keeps its own holdings, so a divergent admission changes later slot occupancy — that path dependence is part of what is measured.

Harness parity. The control arm reproduces the production row of reports/backtest-20260907-survivorship-a.json exactly — 322 trades, 424.6% total return, 51.6% CAGR, 21.4% drawdown, 2.09 Sharpe. Any difference below is therefore attributable to the basis, not to a re-implemented book.

Result

Metric	Control (price return)	Treatment (total return)	Δ
Trades	322	334	+12
Total return %	424.6	376.5	−48.1
CAGR %	51.6	47.9	−3.7
Max drawdown %	21.4	19.5	−1.9 (better)
Sharpe	2.09	1.98	−0.11
Win rate %	37.0	36.8	−0.2
Profit factor	2.48	2.33	−0.15

The books really do differ. 271 entries are shared; 51 are taken only by control and 63 only by treatment — about 30% of the union of entries. The composition follows the Step 2 mechanism: control takes MAR, AAPL, BIIB, LVS; treatment takes BG, OKE, DE, ARES. Some differences are timing rather than selection — ABBV is entered on 2022-09-15 by control and 2022-09-22 by treatment.

Where the difference sits

One year carries all of it. Per-year returns:

Year	Control %	Treatment %	Δpp
2022	8.1	9.9	+1.8
2023	42.0	41.9	−0.1
2024	36.4	38.4	+2.0
2025	80.2	58.1	−22.1
2026	39.2	39.6	+0.4

Treatment wins three years and loses two, and four of the five differ by under 2.1pp. The entire −3.7pp CAGR gap is 2025. That is a description of where the shortfall sits — a useful thing to know when reading it — and nothing more. It is not grounds for calling the shortfall noise or the estimate unstable.

Uncertainty is not yet quantified. The per-arm sharpe_se values (0.497 and 0.498) are the standard errors of each Sharpe individually. They say nothing about the uncertainty of the difference: the two books share dates and most of their positions, so the arms are strongly correlated and the standard error of their difference depends on that covariance, not on the two marginal errors. An earlier version of this document compared the 0.11 Sharpe gap to a 0.50 marginal SE and concluded the books were "statistically indistinguishable" — that inference does not follow, and the same applies to the drawdown difference, for which no error estimate was reported at all. Both claims are withdrawn.

Paired block bootstrap on the difference

Pre-specified before running — seed 20260907, 5,000 replicates, block lengths 21 and 63 sessions, both reported rather than whichever reads better. The two arms' daily return streams are date-aligned (the script aborts if they are not), and the same block indices are drawn for both arms, so the pairing and whatever cross-arm correlation exists on those days are preserved. Moving blocks retain some of the serial dependence an i.i.d. resample would destroy.

Treatment − control, 95% interval:

Block	Metric	Mean	2.5%	97.5%	Share of replicates < 0
21	ΔCAGR pp	−3.61	−14.76	+6.45	0.75
21	ΔSharpe	−0.109	−0.433	+0.198	0.75
21	ΔMaxDD pp	−1.00	−7.50	+4.34	0.63
63	ΔCAGR pp	−3.78	−14.67	+5.40	0.76
63	ΔSharpe	−0.113	−0.428	+0.182	0.76
63	ΔMaxDD pp	−1.79	−8.22	+2.68	0.78

The two block lengths agree closely, which is reassuring about the resampling itself. Every interval spans zero, and roughly three-quarters of replicates fall below zero for CAGR and Sharpe (treatment worse) and for drawdown (treatment better, since a negative ΔMaxDD is a shallower drawdown). So the observed window leans toward control on return and treatment on drawdown without resolving either sign confidently.

What this diagnostic is not. It resamples the realised daily return streams; it does not re-run the strategy, so it cannot capture how a reordered history would have changed admissions or holdings. It is an exploratory uncertainty estimate from the observed window — not out-of-sample validation, and not a rollout gate.

Verdict

Decision: production stays unchanged. No schema, no ingestion, no wider parameter sweep.

• Established: the basis is production-relevant. It changes roughly 30% of entries, in the yield-driven direction the screen predicted.
• Not established: which basis performs better. Point estimates favour control on return and Sharpe and treatment on drawdown; every bootstrap interval spans zero, with about three-quarters of the mass on the point-estimate side. The observed window does not resolve the sign.
• No acceptable return sacrifice is being chosen from these numbers, and no threshold is being fitted to them after the fact.

What a decision would still need:

1. A pre-registered numeric rule, set before it is applied. The plan commits to non-inferiority in principle — total return is the basis 12-1 momentum is defined on, so an in-sample shortfall should not automatically lock in the basis we believe is measured wrong — but carries no threshold. An earlier draft used ΔCAGR ≥ −0.50pp at ΔMaxDD ≤ +1.50pp. Choosing a threshold now, with −3.7pp on the table, would be fitting the rule to the answer.
2. Out-of-sample evidence. One five-year snapshot, in-sample throughout.
3. Note what would not help: a matrix over capacity, start date or cadence produces sensitivity checks, not independent samples. Fifty-three overlapping paths from one price history are not fifty-three pieces of evidence, and treating them that way would overstate what a sweep buys.

Reproduce

python scripts/run_total_return_ab.py --build-candidates   # ~9 min, cached
python scripts/run_total_return_ab.py                      # both arms + bootstrap, ~40 s

The bootstrap parameters are constants in the script (BOOTSTRAP_SEED, BOOTSTRAP_REPLICATES, BOOTSTRAP_BLOCKS) rather than flags, so a run cannot quietly pick a different seed or block length.

Guards in tests/unit/test_total_return_ab.py: identical input series must produce identical books (so an observed difference is the basis, not replay nondeterminism); a controlled momentum change must move a name across the cutoff (so the harness can see what it is looking for); the volatility leg must be untouched between arms; and a name with no computable momentum in an arm must not qualify on a value carried over from candidate generation.