research: survivorship sensitivity and total-return momentum investigation

Three questions answered against snapshot copies, with no production change.

Step 1 — how sensitive is the backtest to the ranking universe? Restoring 85
S&P 500 constituents removed between 2021-06 and 2026-07 moves the same window
from 2.09 Sharpe / 51.6% CAGR to 1.57 / 38.1%. The failed banks never reached
either book (identical best/worst trade R), so this measures universe
recomposition, not survivorship bias. The removals list is curated by hand with
per-symbol sources; only three explicitly confirmed terminal events get marks,
and four unresolved endings are reported as caveats rather than inferred.

Step 2 — does a total-return basis change the ranking? Ordered lists differ on
1009 of 1009 evaluated dates, 4.56 replacements per date. The shift correlates
with trailing implied yield (mean Spearman +0.79), which is strong evidence
consistent with a dividend effect but does not isolate own-yield, since both
the stock and benchmark legs change.

Step 3a — does that reach P&L? A paired replay reusing one shared candidate set
(control reproduces the production row exactly) changes ~30% of entries.
Treatment is -3.7pp CAGR and -1.9pp drawdown, with the entire return gap in
2025. A paired moving-block bootstrap (seed 20260907, 5000 replicates, 21/63
blocks, both reported) puts every interval across zero.

Decision: production stays unchanged. No schema, no ingestion, no wider sweep.
The observed window does not resolve which basis is better, and choosing a
threshold now — with -3.7pp on the table — would be fitting the rule to the
answer.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HAH7vVtn7qbcn1EVheecHQ

tests/unit/test_total_return_ab.py

@@ -0,0 +1,186 @@
+"""Guards for the paired total-return A/B replay.
+
+Two things have to hold before any difference the A/B reports means anything:
+
+1. **Identical inputs must produce identical books.** If the harness were
+   nondeterministic — dict ordering, tie-breaks, path-dependent holdings — a
+   difference between the arms could come from the replay rather than the
+   momentum basis. Feeding both arms the same series must give byte-identical
+   results.
+2. **A real momentum difference must be able to change an admission.** A harness
+   that holds everything so tightly it can never diverge would report agreement
+   no matter what. One name pushed across the cutoff must actually change who
+   gets admitted.
+"""
+
+from __future__ import annotations
+
+import importlib.util
+from datetime import date, timedelta
+from pathlib import Path
+
+import pytest
+
+# Imported at module scope like the other backtest tests: loading the service
+# lazily inside the first test leaves conftest's in-memory engine without its
+# schema for every test after it.
+from app.services import backtest_service as _bt  # noqa: F401
+
+ROOT = Path(__file__).resolve().parents[2]
+_spec = importlib.util.spec_from_file_location(
+    "run_total_return_ab", ROOT / "scripts" / "run_total_return_ab.py"
+)
+ab = importlib.util.module_from_spec(_spec)
+_spec.loader.exec_module(ab)
+
+
+def _sessions(n: int, start: date = date(2021, 1, 4)) -> list[date]:
+    days, day = [], start
+    while len(days) < n:
+        if day.weekday() < 5:
+            days.append(day)
+        day += timedelta(days=1)
+    return days
+
+
+def _series(n: int = 400, drift: float = 0.30) -> dict:
+    """A cache-shaped series set: three names plus SPY, all with clean history."""
+    days = _sessions(n)
+    out = {}
+    for i, symbol in enumerate(("AAA", "BBB", "CCC")):
+        splits = [100.0 * (1.0 + (drift + i * 0.10) * k / n) for k in range(n)]
+        out[symbol] = (days, splits, list(splits))  # ALL == SPLIT: no dividends
+    spy = [100.0 * (1.0 + 0.10 * k / n) for k in range(n)]
+    out[ab.BENCHMARK] = (days, spy, list(spy))
+    return out
+
+
+def _candidates(days: list[date], signal_index: int) -> list[dict]:
+    """One qualifying long setup per symbol on a single signal date."""
+    as_of = days[signal_index]
+    out = []
+    for symbol in ("AAA", "BBB", "CCC"):
+        out.append({
+            "symbol": symbol,
+            "date": as_of.isoformat(),
+            # Real candidates carry both; the percentile pass groups on them.
+            "ranking_period": as_of.isocalendar()[:2],
+            "iso_week": as_of.isocalendar()[:2],
+            "direction": "long",
+            "entry": 100.0,
+            "stop": 95.0,
+            "target": 110.0,
+            "rr": 2.0,
+            "vol_6m": 0.02,
+            "residual_momentum": 0.0,
+            "momentum": 0.0,
+            "meets_core": True,
+            "qualified": True,
+        })
+    return out
+
+
+def test_identical_series_give_identical_momentum_and_ordering():
+    """ALL == SPLIT everywhere, so the two arms must agree exactly — any
+    difference the real run reports is then attributable to the basis."""
+    series = _series()
+    days = series["AAA"][0]
+    signal_dates = {s: {days[-1]} for s in ("AAA", "BBB", "CCC")}
+
+    control = ab.momentum_by_arm(series, signal_dates, total_return=False)
+    treatment = ab.momentum_by_arm(series, signal_dates, total_return=True)
+
+    assert control == treatment
+    assert control, "fixture must produce momentum values, else the test is vacuous"
+
+    cands = _candidates(days, len(days) - 1)
+    a = ab.apply_arm(cands, control, 80.0)
+    b = ab.apply_arm(cands, treatment, 80.0)
+    assert a == b
+
+
+def test_entry_diff_is_empty_for_identical_books():
+    sim = {"trade_details": [
+        {"entry_date": "2022-01-04", "symbol": "AAA"},
+        {"entry_date": "2022-02-01", "symbol": "BBB"},
+    ]}
+    diff = ab.entry_diff(sim, dict(sim))
+    assert diff["only_control_count"] == 0
+    assert diff["only_treatment_count"] == 0
+    assert diff["shared_entries"] == 2
+
+
+def test_a_momentum_change_moves_a_name_across_the_cutoff():
+    """The harness must be able to see a real difference: lifting one name's
+    momentum above its peers changes which names clear the gate."""
+    series = _series()
+    days = series["AAA"][0]
+    cands = _candidates(days, len(days) - 1)
+    as_of = days[-1]
+
+    # CCC ranks top on the control momentum, AAA bottom.
+    control_mom = {("AAA", as_of): 0.10, ("BBB", as_of): 0.20, ("CCC", as_of): 0.30}
+    # Treatment reverses it — as a dividend-heavy name overtaking a growth name would.
+    treatment_mom = {("AAA", as_of): 0.30, ("BBB", as_of): 0.20, ("CCC", as_of): 0.10}
+
+    a = ab.apply_arm(cands, control_mom, 80.0)
+    b = ab.apply_arm(cands, treatment_mom, 80.0)
+
+    admitted_a = {c["symbol"] for c in a if c["qualified"]}
+    admitted_b = {c["symbol"] for c in b if c["qualified"]}
+
+    assert admitted_a == {"CCC"}
+    assert admitted_b == {"AAA"}
+    assert admitted_a != admitted_b
+
+
+def test_the_volatility_leg_is_untouched_by_the_arm():
+    """Only momentum may vary; vol_6m carries through both arms unchanged, so the
+    80/20 blend differs solely through its momentum leg."""
+    series = _series()
+    days = series["AAA"][0]
+    as_of = days[-1]
+    cands = _candidates(days, len(days) - 1)
+    for index, cand in enumerate(cands):
+        cand["vol_6m"] = 0.01 * (index + 1)
+
+    a = ab.apply_arm(cands, {("AAA", as_of): 0.1, ("BBB", as_of): 0.2, ("CCC", as_of): 0.3}, 80.0)
+    b = ab.apply_arm(cands, {("AAA", as_of): 0.3, ("BBB", as_of): 0.2, ("CCC", as_of): 0.1}, 80.0)
+
+    from app.services.backtest_service import VOL_PERCENTILE_KEY
+
+    assert [c[VOL_PERCENTILE_KEY] for c in a] == [c[VOL_PERCENTILE_KEY] for c in b]
+    assert [c["vol_6m"] for c in a] == [c["vol_6m"] for c in b]
+
+
+def test_missing_momentum_leaves_a_candidate_unqualified():
+    """A name with no computable momentum in an arm must not sneak through the
+    gate on a stale value carried over from candidate generation."""
+    series = _series()
+    days = series["AAA"][0]
+    as_of = days[-1]
+    cands = _candidates(days, len(days) - 1)
+
+    arm = ab.apply_arm(cands, {("AAA", as_of): 0.5}, 80.0)
+    by_symbol = {c["symbol"]: c for c in arm}
+
+    assert by_symbol["BBB"]["residual_momentum"] is None
+    assert by_symbol["CCC"]["residual_momentum"] is None
+
+
+def test_yearly_comparison_counts_years_treatment_wins():
+    control = {"yearly_returns": [
+        {"year": 2022, "return_pct": 10.0},
+        {"year": 2023, "return_pct": 20.0},
+        {"year": 2024, "return_pct": 5.0},
+    ]}
+    treatment = {"yearly_returns": [
+        {"year": 2022, "return_pct": 12.0},
+        {"year": 2023, "return_pct": 18.0},
+        {"year": 2024, "return_pct": 5.0},
+    ]}
+    out = ab.yearly_comparison(control, treatment)
+    assert out["years_treatment_better"] == 1
+    assert out["years_treatment_worse"] == 1
+    assert out["years_equal"] == 1
+    assert out["rows"][0]["delta_pp"] == pytest.approx(2.0)