research: survivorship sensitivity and total-return momentum investigation

Three questions answered against snapshot copies, with no production change.

Step 1 — how sensitive is the backtest to the ranking universe? Restoring 85
S&P 500 constituents removed between 2021-06 and 2026-07 moves the same window
from 2.09 Sharpe / 51.6% CAGR to 1.57 / 38.1%. The failed banks never reached
either book (identical best/worst trade R), so this measures universe
recomposition, not survivorship bias. The removals list is curated by hand with
per-symbol sources; only three explicitly confirmed terminal events get marks,
and four unresolved endings are reported as caveats rather than inferred.

Step 2 — does a total-return basis change the ranking? Ordered lists differ on
1009 of 1009 evaluated dates, 4.56 replacements per date. The shift correlates
with trailing implied yield (mean Spearman +0.79), which is strong evidence
consistent with a dividend effect but does not isolate own-yield, since both
the stock and benchmark legs change.

Step 3a — does that reach P&L? A paired replay reusing one shared candidate set
(control reproduces the production row exactly) changes ~30% of entries.
Treatment is -3.7pp CAGR and -1.9pp drawdown, with the entire return gap in
2025. A paired moving-block bootstrap (seed 20260907, 5000 replicates, 21/63
blocks, both reported) puts every interval across zero.

Decision: production stays unchanged. No schema, no ingestion, no wider sweep.
The observed window does not resolve which basis is better, and choosing a
threshold now — with -3.7pp on the table — would be fitting the rule to the
answer.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HAH7vVtn7qbcn1EVheecHQ

docs/data-integrity-plan.md

@@ -0,0 +1,363 @@
+# Data & signal integrity — implementation plan
+
+Status: **closed, 2026-09-07.** Steps 1, 2 and 3a ran; Step 3b was not started
+because 3a did not clear its bar. Findings are in
+[`research/survivorship-audit.md`](research/survivorship-audit.md),
+[`research/tr-rank-agreement.md`](research/tr-rank-agreement.md) and
+[`research/tr-total-return-ab.md`](research/tr-total-return-ab.md). The follow-on
+work this raised is in [`data-reliability.md`](data-reliability.md).
+
+Two questions, answered in order, with the production work made conditional on
+the answers.
+
+## Objective
+
+1. Measure how the backtest changes when removed index constituents are restored.
+   This is a survivorship sensitivity check, not a complete bias estimate.
+2. Check whether total-return momentum changes the daily admission or ordering
+   of names. Build the production path only if this screening test finds changes.
+
+**Guiding principle: KISS — measure before building.** Every earlier draft of this
+plan started with a schema change (`adj_close`, then `tr_factor`), an ingestion
+maintenance path, a snapshot migration and a benchmark column, in order to deliver
+a signal change whose effect nobody has measured. Both questions above can be
+investigated by research scripts against a snapshot copy. Step 1 needs one small,
+explicit replay option to include entries immediately before a terminal event.
+No production schema or ingestion changes are needed for either research step.
+Step 3 is not started until Step 2 finds a ranking or admission difference.
+
+Why the research scripts need none of that machinery: the whole forward-anchored
+factor design exists because *production* ingest is incremental, so back-adjusted
+prices drift between fetches. A research script fetches each ticker's whole history
+in **one** request, and within a single response the adjustment is internally
+consistent. No column, no anchor, no maintenance.
+
+---
+
+## Step 1 — How sensitive is the backtest to omitted constituents?
+
+Arm A uses the universe already present in the base snapshot.
+Arm B adds the names that were removed from the S&P 500 since the snapshot's start
+— including the ones that failed — and lets the strategy trade them.
+Use the same configuration, cadence and evaluation dates in both arms. This is
+deliberately an expanded-universe experiment, without historical membership gating.
+
+### Files
+
+- **[NEW] `scripts/data/sp500_removals.json`** — flat list, hand-curated from
+  Wikipedia's S&P 500 changes table, checked into git:
+  `[{"symbol": "SIVB", "removed_on": "2023-03-15", "reason": "receivership"}, ...]`
+  Reasons: `receivership` | `bankruptcy` | `acquisition` | `market_cap`.
+  No additions, no rename map, no membership intervals — see *Not doing*.
+- **[NEW] `scripts/prep_survivorship_snapshot.py`** — makes both arm files from a
+  base snapshot, leaving the base untouched:
+  1. Copy base → `survivorship-a.sqlite` and `survivorship-b.sqlite`.
+  2. **Schema sync** on both: add every column present in `Base.metadata` but
+     missing from the file, nullable. The snapshots on disk predate migrations 026
+     and 032, so `select(Ticker)` against them fails today with
+     `no such column: tickers.cik` — and `_get_ticker` (`price_service.py:22`) is a
+     full-entity select that `query_ohlcv` → `backtest_service._fetch_columns`
+     reaches. Derive the column list from the metadata, not a hardcoded list, so
+     migration 036 doesn't break it again.
+  3. Into arm B only: for each removed symbol with no bars, fetch daily bars from
+     Alpaca (it still serves delisted symbols — verified for SIVB, FRC, ATVI, TWTR)
+     and insert the ticker row. Restrict downloaded bars to the base snapshot's
+     date range and use the same price adjustment as its existing bars. Log fetched,
+     already-present and unavailable symbols; an unavailable failure must be called
+     out as missing coverage, not silently treated as having no strategy exposure.
+     `removed_on` is an index-removal date, not necessarily a delisting date.
+     **Nothing is inferred from missing bars**: `delisted_on` / `delisted_reason`
+     are written only where the dataset carries a confirmed terminal event. A
+     series that merely ends early leaves them NULL and is reported as an
+     unresolved *ending status* — as built, that is CMA, SEE, HBI and DISH.
+     Series are shortened only at a **reviewed cutoff** listed in the dataset
+     (SBNY 2023-03-10, INFO 2022-02-25). Elsewhere a run of ≥5 zero-volume
+     sessions is a tripwire, not a rule: the symbol is left out of the arm and
+     reported for review rather than silently truncated.
+  4. **Terminal mark**, arm B only: for a confirmed terminal event, append one
+     synthetic bar on the next SPY trading session after the last real bar, with
+     `o=h=l=c=0.01, volume=0`. A mark is written only when the retained final bar
+     equals the confirmation's reviewed `last_real_bar`; a disagreement writes no
+     mark and is reported. Write a sidecar JSON mapping symbol to terminal-bar
+     date for the runner.
+     The portfolio sim already fills a gap through the stop at the open
+     (`backtest_service.py` ~2280), so the position resolves at ~-100% with no
+     change to its stop-fill logic. Treat 0.01 as a near-total-loss assumption,
+     not an observed execution. For acquisitions, retain the last available close
+     as an approximation; this experiment does not model cash/stock consideration
+     or release of capital on the acquisition date.
+     This matters: failures halt mid-collapse (SIVB's last exchange bar is 106.04
+     on the way to zero), so without it arm B spares exactly the trades it exists
+     to measure. **Appending the bar alone is insufficient:** `_replay_ticker`
+     currently stops at `n - HORIZON`, where `HORIZON = 30`. That excludes signals
+     in the last 29 real bars even after the synthetic bar is added. Implement the
+     explicit replay option below before interpreting the audit. Only positions
+     still open at the halt should lose nearly everything; positions stopped or
+     timed out earlier should retain their normal exits.
+     `volume=0` is safe — `compute_volume_profile` guards its only divisor
+     (`indicator_service.py:298-300`).
+- **[NEW] `docs/research/survivorship-audit.md`** + an entry in
+  `docs/research/README.md`.
+
+### Small replay change required
+
+- Add optional `--terminal-events <json>` to `run_backtest_snapshot.py`. Pass the
+  map explicitly through `run_backtest` and `_replay_and_signals` to
+  `_replay_ticker`, including the multiprocessing path. Default to no events.
+  Do not infer terminal events from low prices or zero volume.
+- Validate the whole map **before dispatch** and fail the run on any mismatch.
+  `_replay_ticker` rejects a bad date, but both dispatch paths swallow per-ticker
+  exceptions so a long replay survives one unreadable name — which would turn a
+  stale events file into a quietly incomplete report. `_validate_terminal_events`
+  raises `TerminalEventError` before any work starts, so no report is written.
+- For a mapped ticker, validate that the terminal date matches its final bar.
+  Keep the existing cadence and starting index, but allow signal dates through
+  the last real bar instead of stopping 30 bars early. Never generate a setup on
+  the synthetic bar. Unmapped tickers retain the existing replay bounds.
+- Use the existing forward bars and outcome evaluation. For the long-only audit,
+  a setup with a positive stop resolves no later than the terminal bar. Do not
+  pad the history with 30 invented trading bars or turn unresolved shortened
+  histories into normal 30-day outcomes.
+- Keep terminal prices out of later signal generation; their role is resolving
+  existing positions. Preserve normal stop, target and time exits before them.
+
+### Focused validation
+
+Add regression tests alongside the existing backtest tests:
+
+1. A qualifying long setup five sessions before a terminal event is emitted with
+   the option enabled and exits at 0.01 if still held. Use a controlled setup
+   fixture so the test does not depend on whether SIVB happens to qualify.
+2. A position that exits before the terminal event keeps that earlier exit.
+3. No setup is emitted on the terminal bar; an ordinary ticker and a run without
+   the option retain their existing replay behavior.
+
+Before the full runs, smoke-test the prepared files with `_fetch_columns` for one
+existing and one restored ticker. Confirm matching base-symbol bars between arms,
+matching configuration and date bounds, and no modification of the base file.
+
+### Running it
+
+```powershell
+python scripts/prep_survivorship_snapshot.py --base backtest_snapshots/prod.sqlite
+python scripts/run_backtest_snapshot.py backtest_snapshots/survivorship-a.sqlite --out reports/backtest-YYYYMMDD-survivorship-a.json
+python scripts/run_backtest_snapshot.py backtest_snapshots/survivorship-b.sqlite --terminal-events backtest_snapshots/survivorship-b-terminal-events.json --out reports/backtest-YYYYMMDD-survivorship-b.json
+```
+
+No new runner: `run_backtest_snapshot.py` already runs the production sim offline
+and reports CAGR, max drawdown, Sharpe and total return. Compare the same production
+portfolio-monitor strategy and `all` lookback in both reports, not whichever
+variant performs best. Record B minus A for these metrics, trade counts, evaluated
+dates, restored-symbol coverage and terminal-event assumptions. A failed ticker
+replay or missing production report row makes the run incomplete.
+
+### What it does and doesn't measure
+
+An **expanded-universe sensitivity analysis**, not a lower bound. It restores
+omitted names but does not remove addition lookahead. Still-trading former members
+remain eligible after removal: `run_backtest` does not gate on `delisted_on`, and
+an index-removal date would not be the right delisting field anyway.
+
+Changing the universe changes percentile ranks and competing admissions. Either
+correction can raise or lower this strategy's realized returns; do not claim that
+deletions are the larger source of bias or that the true drag must exceed B minus A.
+A small delta does not establish that survivorship bias is small. Report it as
+the sensitivity observed under these assumptions. A complete point-in-time audit
+remains separate work, not a requirement for finishing this experiment.
+
+### Base snapshot
+
+- `prod.sqlite`: 2021-06-24 → 2026-07-02, 90MB, the ~505 production names.
+- `research.sqlite`: 2016-01-04 → 2026-07-17, 678MB, 4,650 names.
+
+Use `prod.sqlite`: the highest-signal events (SIVB, FRC, SBNY, ATVI — all
+2023) are inside its window, the copies are cheap, and it has no completion
+manifest to keep in sync. The pre-registration then states a 2021–2026 window, not
+a ten-year one. Verify the actual file's bounds and record its hash before running.
+Do not silently substitute `research.sqlite`: its much broader universe answers a
+different comparison. Keep the first experiment on the production snapshot.
+
+---
+
+## Step 2 — Would total return change any decision?
+
+Use a conservative ranking screen, not a portfolio simulation. Compare the full
+ordered momentum-admitted list on every daily decision date. Do not call a top-15
+list the traded set: production also applies setup qualification, current holdings,
+available slots and other admission constraints. A difference in this screen is
+a reason to investigate actual trades in Step 3, not proof of changed P&L.
+
+### File
+
+- **[NEW] `scripts/research_tr_rank_agreement.py`** — use arm A's universe and dates:
+  1. One `Adjustment.ALL` fetch and one `Adjustment.SPLIT` fetch per symbol, plus
+     SPY, over the full history. Single fetch each, so each series is internally
+     consistent. Align by date, never zip by index. Cache the responses locally
+     for repeatable runs. Require matching dates in both adjustments; report and
+     resolve missing downloads before applying the gate. Insufficient initial
+     history is expected and should be recorded, with equal eligibility in both
+     arms. Use only bars at or before each decision date.
+  2. At each SPY trading date with sufficient history, compute residual 12-1
+     momentum both ways with the existing
+     `momentum_service.compute_residual_12_1_momentum` (price-return series against
+     nominal SPY; total-return series against total-return SPY), percentile them
+     with `momentum_service._percentiles`, and take the admitted set at the
+     production `min_momentum_percentile`.
+  3. Compute `compute_realized_vol_6m` and its percentiles from the SPLIT series
+     once per date and hold that leg fixed in both arms. This isolates the change
+     to momentum. Order the entire admitted lists by `blend_strategy_rank`, with
+     symbol as a deterministic tie-breaker. Do not truncate to `max_positions`:
+     a lower-ranked name may be the first actionable setup when others fail gates
+     or are already held. Iterate symbols in the same sorted order in both arms
+     before calling `_percentiles` so tied inputs do not create artificial changes.
+  4. Save per-date results and an overall/per-year summary: number of evaluated
+     dates, fraction with different admitted sets, fraction with different ordered
+     lists, and mean and maximum set disagreement. Define set disagreement as
+     `len(A symmetric_difference B) / len(A union B)`, or zero for two empty sets.
+     Report Spearman correlation as a diagnostic only. Include the first few
+     changed dates with the changed names to make the result inspectable.
+
+Validation: identical input series must produce no differences. A fixture where
+the admitted members stay the same but two names swap order must register a
+changed ordered list. A difference on only one date must still trigger the gate.
+Write results to `docs/research/tr-rank-agreement.md` and index it in the research
+README, including data coverage, parameters and the exact report/cache paths.
+
+### Pre-registered decision rule
+
+- **No ordered-list differences on any evaluated daily date → stop.** Record
+  "no admission or ordering differences observed in this window under the tested
+  configuration." This is historical evidence, not a guarantee about future
+  decisions. No schema or ingestion changes are needed.
+- **Any ordered-list difference → Step 3.** This intentionally conservative rule
+  can flag names that would never actually trade. Step 3's paired portfolio replay
+  determines whether the difference reaches actual admissions and P&L.
+- **Incomplete downloads or no evaluable dates → inconclusive.** Fix coverage
+  before deciding. Never turn missing data into agreement.
+
+There is no median-based tolerance: a zero median can hide differences on nearly
+half the dates. Checking daily full lists avoids adding a second portfolio engine
+to this screening script while retaining changes below the nominal top 15.
+
+---
+
+## Step 3 — Production total-return ranking (conditional)
+
+**Gate cleared 2026-09-07 — Step 3 is now live work.** The screen found ordered
+admitted lists differing on 1,009 of 1,009 evaluated dates, with the shift
+correlating +0.79 with trailing dividend yield
+([results](research/tr-rank-agreement.md)). That is a ranking difference, not a
+P&L difference; Step 3's job is to find out whether it reaches trades.
+
+### Step 3a — Paired portfolio replay (do this first, no schema)
+
+Different ranks are not different trades. Before any schema or ingestion work,
+replay the production book twice and measure actual admissions and P&L. If the
+books are identical, the basis cannot matter and Step 3b never starts.
+
+**[NEW] `scripts/run_total_return_ab.py`**, sourcing everything from artifacts
+that already exist — `backtest_snapshots/survivorship-a.sqlite` for bars and
+`backtest_snapshots/tr-rank-cache.sqlite` for the two adjusted series.
+
+1. **Generate setup candidates once.** One replay pass produces the candidate set;
+   both arms reuse it. Only momentum and what depends on it — the momentum
+   percentile, qualification against `min_momentum_percentile`, and
+   `blend_strategy_rank` ordering — are recomputed per arm. Setup geometry
+   (entry, stop, target, R:R) is generated once and never differs.
+2. **Both arms read the same cache.** Control takes SPLIT stock series against
+   SPLIT SPY; treatment takes ALL stock against ALL SPY. Neither arm reads a
+   freshly fetched series, and neither mixes cache with snapshot closes for the
+   momentum input — a refreshed-vs-snapshot difference would confound the basis
+   difference being measured.
+3. **Everything else is pinned.** Execution uses the snapshot's nominal bars in
+   both arms; the volatility leg, the non-momentum gates, capacity
+   (`max_positions`), risk per trade and the exit policy are identical. Each book
+   then carries its **own** resulting holdings — a divergent admission changes
+   which slots are occupied on later dates, and that path dependence is part of
+   what is being measured.
+4. **Fixed up front:** universe (arm A's 505 symbols with bars), window
+   (2021-06-24 → 2026-07-02), cadence, and the production config as read from the
+   snapshot. Only bars at or before each signal date are used, in both arms.
+5. **Saved to `reports/tr-total-return-ab.json`:** per-arm trade count, CAGR, max
+   drawdown, Sharpe, total return, win rate; annual returns for both arms and the
+   **count of years treatment beats control**; and the entry-level diff — dates
+   and symbols entered by one arm and not the other, in both directions.
+6. **Tests** (`tests/unit/test_total_return_ab.py`): an identical-input parity
+   test — feeding the same series to both arms must produce byte-identical books,
+   so any observed difference is attributable to the basis and not to replay
+   nondeterminism; and a controlled case where a hand-built momentum difference
+   flips one name across the cutoff and the resulting admission actually changes,
+   proving the harness can see what it is looking for.
+
+**Reading the result.** Different trades establish that the basis is
+production-relevant. They do **not** authorize a rollout: that is the
+non-inferiority decision below, taken on the paired numbers, and Step 3b starts
+only after the paired results have been reported.
+
+**Step 3a result (2026-09-07):** the basis reaches trades — ~30% of entries
+differ. Which basis is better is unresolved: a paired block bootstrap on the
+date-aligned daily return streams (5,000 replicates, 21- and 63-session blocks,
+seed fixed beforehand, both reported) puts ΔCAGR at −3.6/−3.8pp with 95%
+intervals spanning zero, ΔSharpe ≈ −0.11, and ΔMaxDD −1.0/−1.8pp in treatment's
+favour. **Production stays unchanged; 3b not started.**
+[Results](research/tr-total-return-ab.md).
+
+### Step 3b — Production storage and ingestion (only if 3a shows real differences)
+
+The design points already settled, worth carrying forward:
+
+- `ohlcv_records.tr_factor`, `Float`, nullable, **no default and no server_default**
+  — a default of 1.0 makes "never backfilled" indistinguishable from "no dividends"
+  and would silently stamp a price-return ranking as total return.
+- Forward-anchored: `f = 1.0` at `max(2016-01-04, ticker's first bar)`, compounded
+  as `f_t = f_{t-1} * (R_t / R_{t-1})` where `R = all_close / split_close`. The
+  ratio is anchor-invariant **only within one API response** — never persist `R` and
+  compare across days.
+- `upsert_ohlcv` must not clobber a stored factor:
+  `coalesce(excluded.tr_factor, ohlcv_records.tr_factor)`. Daily ingest re-fetches
+  the boundary bars, so overlapping writes are routine.
+- Daily ingest pulls a ~5-bar overlap under both adjustments to extend the factor;
+  a clean-ratio mismatch on the overlap bars means a split, which needs a
+  full-history refetch (the current incremental ingest never restates history).
+- One basis for the whole cross-section, never mixed, stamped in the rank log.
+- Beta leg: if the stock series goes total return, SPY must too, in the live path
+  and the backtest, or beta picks up SPY's ~1%/yr yield.
+- A/B through the existing matrix on CAGR and max drawdown with per-year win/loss
+  counts. Reuse the same setup candidates, nominal execution bars, core gates,
+  volatility leg, capacity and exit policy; change only momentum and its dependent
+  admission/ranking fields. Replay each book with its own resulting holdings and
+  compare actual entry dates/symbols as well as P&L. Do not substitute top-15 lists
+  for the existing portfolio admission logic. Non-inferiority, not superiority:
+  total return is the definition 12-1
+  momentum is built on, so an in-sample shortfall shouldn't lock in the basis we
+  believe is measured wrong.
+
+---
+
+## Not doing
+
+- **Chart total-return toggle.** The candlestick chart is an execution chart:
+  stops, S/R levels, entry and paper-trade markers are all nominal. Rescaling the
+  candles draws prices that never traded next to a stop that did.
+- **`[D]` dividend markers.** Cosmetic, and they need `tr_factor` to exist. Revisit
+  after Step 3, if Step 3 ever happens.
+- **Point-in-time addition gating** — interval map, reverse-chronological replay,
+  rename map and removal gating are deferred from this sensitivity experiment.
+  Their relative impact is unknown; a small Step 1 delta does not rule them out.
+  Note if a point-in-time audit is
+  ever done: the ranking cross-section must be filtered *before*
+  `_assign_*_percentiles` (`backtest_service.py:4198-4200`), not only at admission
+  in `_simulate_portfolio` (4257), or the percentile cutoff is still computed
+  against a non-PIT universe.
+- **Prune protection and the `add_ticker` regex.** Only needed if delisted names
+  are written to the production database. The audit works entirely in snapshot
+  copies. (`bootstrap_universe(prune_missing=True)` cascading away a still-trading
+  ex-constituent's history is a real latent risk — a separate ticket, not this one.)
+- **`benchmark_prices.tr_close`.** Only needed for Step 3. Note for then: store a
+  forward factor like the OHLCV side, not a back-adjusted close —
+  `refresh_benchmark_prices` writes a row only when `close` changed and is called
+  with variable spans, which would leave different anchors either side of the last
+  refreshed window.
+- **Migrating the snapshots in place.** The prep script syncs its own copies.
+  `research.sqlite` stays byte-identical, so its completion manifest and the reports
+  that cite it stay valid.

docs/research/README.md

@@ -145,6 +145,8 @@ knobs.
 | **`fip_id` / liquid breadth** | Fingerprint −0.045 / t −2.91; liquid unconditional **−0.017 / t −1.85** (not green); mom-conditional **−0.088 / t −4.58** | **Parked.** Orphan +0.0575 died (snapshot race). Breadth did not strengthen resid-mom t-stat. Optional reopen = pre-registered two-arm liquid-1500 book first. See [fip-breadth-ic.md](fip-breadth-ic.md) |
 | **Broader universe** | Composition changes factor signs (fip tug-of-war); vol-tilt on breadth is only a **directional hypothesis** (auth. −0.048 / t −1.36) | Any prod broaden must re-validate 80/20 tilt; offline research only; research.sqlite requires completion manifest |
 | **Forward paper-trade record** | The only true out-of-sample evidence the snapshot cannot give | Time; mark entries at actual near-close fill once ops ships |
+| **Total-return momentum basis** | Ranks change every day (screen: **1,009 of 1,009** dates, ~4.6 names replaced per date, shift correlates **+0.79** with trailing yield) **and it reaches trades** — paired replay differs on ~30% of entries (51 control-only, 63 treatment-only, 271 shared). Which basis is better is unresolved: paired block bootstrap (5,000 reps, 21- and 63-session blocks, seed fixed beforehand) gives ΔCAGR −3.6/−3.8pp with 95% intervals spanning zero ([−14.8, +6.4] and [−14.7, +5.4]), ΔSharpe ≈ −0.11, ΔMaxDD −1.0/−1.8pp in treatment's favour; ~75% of replicates on the point-estimate side. All of the CAGR shortfall sits in 2025 | **Production unchanged.** Needs a threshold pre-registered before it is applied, plus out-of-sample evidence; a capacity/date/cadence matrix would be sensitivity checks, not independent samples. Schema/ingestion deferred. [Screen](tr-rank-agreement.md) · [Paired replay](tr-total-return-ab.md) · [plan](../data-integrity-plan.md) |
+| **Universe sensitivity / point-in-time membership** | Restoring 85 removed constituents (2021-06→2026-07, prod snapshot) moves the production row from **Sharpe 2.09 / CAGR 51.6%** to **1.57 / 38.1%** — all of it universe recomposition; no failed-bank position reached either book (trade extremes identical, and a control shows the extra replay signals changed no portfolio decision). Says the ranks are highly sensitive to the universe, not that bias is 13.5pp | Not a bias estimate: addition lookahead is still unremoved and arm B is not point-in-time. A real PIT audit is separate work. [Survivorship sensitivity](survivorship-audit.md) |
 | **Better target model for clear-air names** | The return is demonstrably there (#2 wins on raw CAGR in *both* train and test); it's the *flat* 3× ATR target that makes it too expensive in risk | Needs a per-name model, not a constant k×ATR |
 | **Minimum effective-risk floor** | ⛔ CLOSED NEGATIVE, not run. The floor lifts EV/trade (+0.032) and PF (+0.073) *by deleting trades* — 11.4 fewer per path, never one more — and costs **−0.753pp CAGR**, −0.047 Sharpe, −0.051 Calmar | Do not run the A/B; its EV-based pass rule would have shipped it. [Withdrawn specification](effective-risk-floor-ab.md) / [findings](portfolio-capacity-bracket-findings.md#correction-2026-08-05-ev-per-trade-was-the-wrong-lens) |
 
@@ -176,6 +178,14 @@ knobs.
    `research.sqlite`. Extender now writes a completion manifest; breadth mode
    refuses without a match. Same class of protection as calendar-truncation
    asserts — do not re-mythologize numbers computed on half a universe.
+8. **A big delta is not the mechanism you assumed.** Restoring 85 removed index
+   constituents cost 13.5pp of CAGR — and none of it came from the failed banks
+   the experiment was built around. The driver was the universe the ranks are
+   computed over. Measure which part of a change did the work before naming the
+   effect, and state what the control actually isolates: ours compared replay
+   *bounds* on one snapshot, so it measured the extra signals, not the synthetic
+   bars those signals resolve against.
+   [Survivorship sensitivity](survivorship-audit.md)
 
 ---
 

docs/research/survivorship-audit.md

@@ -0,0 +1,173 @@
+# Survivorship sensitivity — restoring removed S&P 500 constituents
+
+**Run date:** 2026-09-07 · **Status:** measured, no strategy change proposed
+**Reports:** `reports/backtest-20260907-survivorship-a.json`,
+`reports/backtest-20260907-survivorship-b.json`,
+`reports/backtest-20260907-survivorship-b-nomarks.json`
+
+## Question
+
+The backtest universe is the ~505 names tracked *today*. Every constituent that
+left the index between 2021 and 2026 is missing from it, including the ones that
+failed. How different does the production strategy look when they are put back?
+
+This is an **expanded-universe sensitivity analysis**, not a bias estimate. It is
+pre-registered as a measurement: there is no pass/fail rule and no promotion
+attached to the outcome.
+
+## Method
+
+**Base:** `backtest_snapshots/prod.sqlite`, sha256 `7241f0ff0f6187a0…`, window
+**2021-06-24 → 2026-07-02** (~5 years, not the full Alpaca history). Never
+modified — both arms are copies, and the hash is re-checked after the build.
+
+| Arm | Universe |
+|---|---|
+| **A** — control | 506 tickers, the snapshot as it stands |
+| **B** — treatment | 591 tickers: the same 506 plus 85 restored removals |
+
+Both arms ran the same binary, config, cadence (weekly) and date bounds, and are
+compared on the same report row: `portfolio_monitor.runs` where
+`is_production` is true and `lookback` is `all`
+(`residual80_highvol80_20_atr3`).
+
+Removals come from `scripts/data/sp500_removals.json` — 92 rows curated from
+Wikipedia's *Historical components of the S&P 500* changes table, each carrying
+the verbatim source text plus independently measured Alpaca evidence. Build with
+`scripts/prep_survivorship_snapshot.py`.
+
+Three data rules, all learned from the price data rather than assumed:
+
+1. **Nothing is inferred from missing bars.** `delisted_on` / `delisted_reason`
+   are written only for confirmed terminal events. A series that merely stops
+   early leaves them NULL and is reported as an unresolved ending.
+2. **Series are shortened only at a reviewed cutoff.** Alpaca does not end a
+   halted symbol's series — it repeats the last price at zero volume. SBNY sits
+   at 70.00 for 509 sessions after the FDIC closed it; INFO sits at 108.61 for
+   1,090 sessions, after which a *different company* reuses the ticker with no
+   date gap. Both have explicit reviewed cutoffs (2023-03-10, 2022-02-25).
+   Elsewhere a run of ≥5 zero-volume sessions is a tripwire: the symbol is left
+   out and reported, never silently cut.
+3. **Terminal marks are explicit assumptions.** A synthetic `0.01` bar is written
+   on the next session after the last real bar for SIVB, SBNY and FRC only —
+   each with an FDIC source and only where the retained final bar matches the
+   reviewed `last_real_bar`. The FDIC sources establish the closure *event*; they
+   do not establish what holders recovered. The mark is a near-total-loss
+   assumption, not an observed closing-day execution.
+
+**Replay change.** `_replay_ticker` normally stops `HORIZON = 30` bars before the
+end so every setup has a full resolution window — which would hide every signal
+in the final 29 real bars, exactly the ones a collapse resolves. A new
+`--terminal-events` map opts the three marked symbols into walking through their
+last real bar; no setup is ever generated on the synthetic bar, and unmapped
+symbols are untouched. The map is validated against the snapshot's actual final
+bars *before* any replay is dispatched (`_validate_terminal_events`), because both
+dispatch paths swallow per-ticker exceptions and would otherwise turn a stale map
+into a quietly incomplete report.
+
+## Result
+
+Source: `portfolio_monitor.runs` where `is_production` is true and `lookback` is
+`all` (strategy `residual80_highvol80_20_atr3`), from
+`reports/backtest-20260907-survivorship-a.json` and
+`…-survivorship-b.json`. Both runs: weekly cadence, 6 workers,
+`--allow-spawn`, gate cutoff 80.0, full period.
+
+| Metric | Arm A | Arm B | B − A |
+|---|---:|---:|---:|
+| Total return % | 424.60 | 264.40 | **−160.20** |
+| CAGR % | 51.60 | 38.10 | **−13.50** |
+| Max drawdown % | 21.40 | 23.60 | +2.20 |
+| Sharpe | 2.09 | 1.57 | **−0.52** |
+| Trades | 322 | 346 | +24 |
+| Win rate % | 37.0 | 35.0 | −2.0 |
+| Final equity | 52,459 | 36,439 | −16,020 |
+
+Context: 202,765 → 229,642 candidates, 1,086 → 1,315 qualified. Gate cutoff 80.0
+in both arms.
+
+## What actually caused it
+
+**Not the failures.** `worst_trade_r` is **−3.26 in both arms**, and
+`best_trade_r` is 12.87 in both. A position held into a terminal mark gaps to
+$0.01 and fills at the open, which would land far below −3.26 — at candidate
+level those trades score −3.13R (SIVB) and −5.99R (SBNY). No such trade appears
+in either book, so none of the three failed banks reached the production
+portfolio.
+
+**What the control does and does not show.** Arm B replayed *without*
+`--terminal-events` (`backtest-20260907-survivorship-b-nomarks.json`, same
+snapshot, same config) produces a production row identical to arm B with them in
+every field — 346 trades, 38.1% CAGR, 1.57 Sharpe, 23.6% drawdown, 264.4% total
+return, −3.26 worst trade. Candidates rise 229,617 → 229,642, the 25 extra
+signals the option exposes in the final 29 bars of the three symbols, while
+`qualified` is 1,315 either way.
+
+So the control establishes precisely this: **those 25 additional replay signals
+changed no portfolio decision.** It does *not* independently establish that the
+terminal marks had no effect — both runs read `survivorship-b.sqlite`, which
+contains the synthetic bars either way, so a setup generated earlier could still
+resolve against one in both runs. Isolating the marks themselves would need a
+third snapshot built without them, which has not been run.
+
+The separate evidence that no marked position reached the book is the trade
+extremes: a position resolving against a $0.01 bar fills at that bar's open and
+scores far worse than −3.26 (−3.13R for SIVB and −5.99R for SBNY at candidate
+level), yet arm B's `worst_trade_r` equals arm A's exactly. That is inference
+from the extremes, not a controlled measurement.
+
+On that basis the −13.5pp CAGR is **universe recomposition**: 85 additional names
+(+17% of the universe) shift the cross-sectional percentile ranks and compete for
+the same 15 position slots, displacing trades arm A took. The strategy is
+markedly sensitive to which universe it ranks over — which is the finding, and it
+is a different claim from "survivorship bias costs 13.5pp".
+
+## Caveats
+
+- **This is not a bias estimate and not a lower bound.** Restoring names is only
+  one of the two corrections. Index-*addition* lookahead — today's members being
+  tradeable before they joined — is not removed. Either correction can move
+  returns in either direction, so B − A is the sensitivity observed under these
+  assumptions, nothing more. A small delta would not have shown the bias to be
+  small, and this large one does not establish its size either.
+- **Arm B is not a point-in-time universe.** Still-trading former members remain
+  eligible after their removal date; `run_backtest` does not gate on
+  `delisted_on`, and an index-removal date would be the wrong field for it
+  anyway. A true PIT audit is separate work.
+- **Four unresolved endings.** CMA (ends 2026-01-30), SEE (2026-04-08), HBI
+  (2025-11-28) and DISH (2023-12-29) stop trading before the window closes for
+  reasons the source row does not explain. Their bars are real and their
+  delisting fields are NULL; the ending status is simply unknown.
+- **Three near-total-loss assumptions** (SIVB, SBNY, FRC), as described above.
+  The control shows they did not affect this result, but they remain assumptions
+  and would matter in any run where such a name did clear the gate.
+- **Acquisitions are approximated** by the last available close. Cash or stock
+  consideration and the release of capital on the deal date are not modelled.
+- **FBIN carries a corporate action.** Fortune Brands distributed MasterBrand on
+  2022-12-14 and began trading as FBIN on 2022-12-15. Alpaca's split adjustment
+  does not adjust for spin-offs, so the distribution appears as a real −6.91%
+  close-to-close move (63.22 → 61.90 → 57.62). A holder did not lose that value,
+  but any signal or stop crossing that date reads it as a decline. MBC enters the
+  dataset separately with its first bar the same day.
+- **Window.** 2021-06 to 2026-07, because that is what `prod.sqlite` holds. The
+  2023 bank failures are inside it; the 2016–2021 period is not.
+- **Snapshot schema.** The snapshot files on disk predate migrations 026 and 032,
+  so `select(Ticker)` against them fails with `no such column: tickers.cik` — the
+  path `backtest_service._fetch_columns` takes. Per-ticker exceptions are caught
+  and logged, so a run against an unsynced snapshot completes and writes a report
+  with **zero candidates** rather than failing. The prep script syncs both copies
+  against `Base.metadata`; anything else reading these snapshots needs the same.
+
+## Reproduce
+
+```powershell
+python scripts/prep_survivorship_snapshot.py --base backtest_snapshots/prod.sqlite
+python scripts/run_backtest_snapshot.py backtest_snapshots/survivorship-a.sqlite `
+    --out reports/backtest-YYYYMMDD-survivorship-a.json --allow-spawn --workers 6
+python scripts/run_backtest_snapshot.py backtest_snapshots/survivorship-b.sqlite `
+    --terminal-events backtest_snapshots/survivorship-b-terminal-events.json `
+    --out reports/backtest-YYYYMMDD-survivorship-b.json --allow-spawn --workers 6
+```
+
+`survivorship-b-prep-manifest.json` records the base hash, per-symbol coverage,
+the terminal events written and every unresolved item for the run.

docs/research/tr-rank-agreement.md

@@ -0,0 +1,118 @@
+# Total-return vs price-return momentum — ranking screen
+
+**Run date:** 2026-09-07 · **Verdict: proceed to Step 3** (pre-registered rule fired)
+**Reports:** `reports/tr-rank-agreement.json`, `reports/tr-rank-agreement-mechanism.json`
+**Script:** `scripts/research_tr_rank_agreement.py`
+**Cache:** `backtest_snapshots/tr-rank-cache.sqlite` (507 symbols, 630,524 aligned bars)
+
+## Question
+
+12-1 momentum is defined on total returns; production computes it on
+split-adjusted prices, which ignore dividends. Before building the production
+`tr_factor` machinery (Step 3 of [the plan](../data-integrity-plan.md)), does the
+basis actually change what the gate admits and how it orders?
+
+Deliberately a screen, not a portfolio simulation. A difference here is a reason
+to measure trades, **not** evidence that P&L changes.
+
+## Configuration
+
+| | |
+|---|---|
+| Universe | `backtest_snapshots/survivorship-a.sqlite`, 505 symbols with bars (RHM has a ticker row but no bars in the snapshot, so it is inert in both arms) |
+| Window | 2021-06-24 → 2026-07-02, clipped to the snapshot |
+| Decision dates | every SPY session; 1,009 had enough history to evaluate |
+| Momentum | production `compute_residual_12_1_momentum`, unchanged; price series against nominal SPY, total-return series against total-return SPY |
+| Volatility leg | production `compute_realized_vol_6m` on the SPLIT series, computed once per date and **held fixed in both arms** so only momentum varies |
+| Gate | production `min_momentum_percentile` = 80.0, via production `_percentiles` |
+| Ordering | production `blend_strategy_rank`, symbol as deterministic tie-break, symbols fed in sorted order in both arms |
+| Comparison | the full admitted list, never truncated to `max_positions` |
+| Runtime | 146 s with the production function — no NumPy rewrite needed, so no equivalence test was required |
+
+**Coverage is complete.** One `Adjustment.SPLIT` and one `Adjustment.ALL` request
+per symbol; 507 symbols cached, **zero** dates present in one adjustment but not
+the other. No symbol with snapshot bars is missing from the cache — the script
+refuses to run in that case rather than reporting the gap as agreement.
+
+## Result
+
+| | |
+|---|---:|
+| Evaluated dates | 1,009 |
+| Dates with a **different ordered list** | **1,009 (100%)** |
+| Dates with a different admitted set | 1,007 (99.8%) |
+| Mean set disagreement, `\|A △ B\| / \|A ∪ B\|` | 0.0867 |
+| Max set disagreement | 0.2143 |
+| Mean admitted names per arm | 100.09 |
+| Mean **replacements** per date (one out paired with one in) | **4.56** |
+| Max replacements on a single date | 12 |
+| Mean symmetric difference (names counted on both sides) | 9.13 |
+
+Per year, every evaluated date differs:
+
+| Year | Dates | Different set | Different order |
+|---|---:|---:|---:|
+| 2022 | 132 | 132 | 132 |
+| 2023 | 250 | 250 | 250 |
+| 2024 | 252 | 252 | 252 |
+| 2025 | 250 | 248 | 250 |
+| 2026 | 125 | 125 | 125 |
+
+Counted directly from the daily set differences rather than derived from the
+Jaccard ratio: **4.56 names are replaced on an average date** — 4.56 leave and
+4.56 arrive, so 9.13 names appear on one side or the other, out of about 100
+admitted. A swap moves two names, which makes the symmetric difference twice the
+replacement count and the two easy to conflate.
+
+## Strong evidence of a dividend effect
+
+First changed date, 2022-06-24 — 99 names admitted in each arm, nine swapped:
+
+- **Only on price return:** CDNS, MAR, FIX, COST, LIN, AON, STE, CI, AJG
+- **Only on total return:** EXC, PSX, WY, AMCR, DTE, GIS, FE, BMY, ES
+
+Growth and low-yield names out; utilities, energy, staples and other high payers
+in. Nine tickers are not evidence, so it was measured (`--mechanism 25`, saved to
+`reports/tr-rank-agreement-mechanism.json` with its sampled dates and per-date
+values): across 40 sampled dates, the Spearman correlation between a name's
+trailing 12-month implied yield — from the ALL/SPLIT ratio over the formation
+window — and its momentum-percentile shift between arms is **+0.79** (range
++0.71 to +0.85, every sampled date positive).
+
+**What that does and does not pin down.** The reordering is systematic and
+directional rather than numerical noise, and it moves with yield. It is *not* a
+clean isolation of the stock's own yield: both the stock series and the benchmark
+leg change between arms, so the measured shift also contains the beta-weighted
+benchmark yield. Read it as strong evidence consistent with a dividend effect.
+Isolating own-yield would need a third variant — total-return stock series
+against a nominal SPY leg — which has not been run.
+
+## Verdict
+
+The pre-registered rule was "no ordered-list difference on any evaluated date →
+stop". Differences occur on **every** evaluated date, so **Step 3 proceeds**.
+
+What this does and does not establish:
+
+- **Does:** the momentum basis materially changes gate admission and ordering,
+  every day, in a yield-driven direction.
+- **Does not:** show that realised P&L changes. Production also applies setup
+  qualification, existing holdings, available slots and the rest of the admission
+  path. A name that swaps into the admitted list may never produce a setup. Step
+  3's paired portfolio replay is what decides whether any of this reaches trades.
+- **Worth noting:** the tuning recorded as closed in the research log was carried
+  out on a price-return basis throughout. This screen does not invalidate it —
+  it says the input those conclusions were computed on is not the one the factor
+  is defined on.
+
+## Reproduce
+
+```powershell
+python scripts/research_tr_rank_agreement.py --fetch          # 1,014 requests, cached
+python scripts/research_tr_rank_agreement.py --time-batch 20  # runtime projection
+python scripts/research_tr_rank_agreement.py                  # full screen
+python scripts/research_tr_rank_agreement.py --mechanism 25   # yield vs percentile shift
+```
+
+The cache makes re-runs free and the screen deterministic; delete
+`backtest_snapshots/tr-rank-cache.sqlite` to refetch.

docs/research/tr-total-return-ab.md

@@ -0,0 +1,160 @@
+# Total-return momentum — paired portfolio replay (Step 3a)
+
+**Run date:** 2026-09-07 · **Status:** measured; **no rollout decision taken**
+**Report:** `reports/tr-total-return-ab.json` · **Script:** `scripts/run_total_return_ab.py`
+**Inputs:** `backtest_snapshots/survivorship-a.sqlite` (bars),
+`backtest_snapshots/tr-rank-cache.sqlite` (both adjusted series),
+`backtest_snapshots/tr-ab-candidates.pkl` (202,765 shared candidates)
+
+## Question
+
+[The ranking screen](tr-rank-agreement.md) showed the momentum basis changes the
+admitted list on every evaluated date. Does that reach actual admissions and P&L?
+
+## What was held identical
+
+Setup candidates were generated **once** (202,765 over 505 symbols, matching the
+count in the standalone backtest report) and reused by both books. Only momentum
+and its dependents were recomputed per arm: residual and activation percentiles,
+`qualified`, and the 80/20 blend used for ordering.
+
+| Held fixed | Varied |
+|---|---|
+| Setup geometry — entry, stop, target, R:R | Momentum series: SPLIT stock vs SPLIT SPY (control), ALL vs ALL SPY (treatment) |
+| Execution on the snapshot's nominal bars | The percentiles, `qualified` and ordering that follow from it |
+| `vol_6m` volatility leg, from candidate generation | |
+| Non-momentum gates, `max_positions`, risk per trade, ATR-trail exit, gate-reset re-entry | |
+
+Both arms read the same cache, so a refreshed-vs-snapshot difference cannot be
+mistaken for a basis difference. Each book then keeps its own holdings, so a
+divergent admission changes later slot occupancy — that path dependence is part
+of what is measured.
+
+**Harness parity.** The control arm reproduces the production row of
+`reports/backtest-20260907-survivorship-a.json` exactly — 322 trades, 424.6%
+total return, 51.6% CAGR, 21.4% drawdown, 2.09 Sharpe. Any difference below is
+therefore attributable to the basis, not to a re-implemented book.
+
+## Result
+
+| Metric | Control (price return) | Treatment (total return) | Δ |
+|---|---:|---:|---:|
+| Trades | 322 | 334 | +12 |
+| Total return % | 424.6 | 376.5 | −48.1 |
+| CAGR % | 51.6 | 47.9 | **−3.7** |
+| Max drawdown % | 21.4 | 19.5 | **−1.9** (better) |
+| Sharpe | 2.09 | 1.98 | −0.11 |
+| Win rate % | 37.0 | 36.8 | −0.2 |
+| Profit factor | 2.48 | 2.33 | −0.15 |
+
+**The books really do differ.** 271 entries are shared; **51 are taken only by
+control and 63 only by treatment** — about 30% of the union of entries. The
+composition follows the Step 2 mechanism: control takes MAR, AAPL, BIIB, LVS;
+treatment takes BG, OKE, DE, ARES. Some differences are timing rather than
+selection — ABBV is entered on 2022-09-15 by control and 2022-09-22 by treatment.
+
+## Where the difference sits
+
+**One year carries all of it.** Per-year returns:
+
+| Year | Control % | Treatment % | Δpp |
+|---|---:|---:|---:|
+| 2022 | 8.1 | 9.9 | +1.8 |
+| 2023 | 42.0 | 41.9 | −0.1 |
+| 2024 | 36.4 | 38.4 | +2.0 |
+| 2025 | 80.2 | 58.1 | **−22.1** |
+| 2026 | 39.2 | 39.6 | +0.4 |
+
+Treatment wins three years and loses two, and four of the five differ by under
+2.1pp. The entire −3.7pp CAGR gap is 2025. That is a description of *where* the
+shortfall sits — a useful thing to know when reading it — and nothing more. It is
+not grounds for calling the shortfall noise or the estimate unstable.
+
+**Uncertainty is not yet quantified.** The per-arm `sharpe_se` values (0.497 and
+0.498) are the standard errors of each Sharpe *individually*. They say nothing
+about the uncertainty of the **difference**: the two books share dates and most
+of their positions, so the arms are strongly correlated and the standard error of
+their difference depends on that covariance, not on the two marginal errors. An
+earlier version of this document compared the 0.11 Sharpe gap to a 0.50 marginal
+SE and concluded the books were "statistically indistinguishable" — that
+inference does not follow, and the same applies to the drawdown difference, for
+which no error estimate was reported at all. Both claims are withdrawn.
+
+## Paired block bootstrap on the difference
+
+Pre-specified before running — seed `20260907`, **5,000** replicates, block
+lengths **21 and 63** sessions, both reported rather than whichever reads better.
+The two arms' daily return streams are date-aligned (the script aborts if they
+are not), and **the same block indices are drawn for both arms**, so the pairing
+and whatever cross-arm correlation exists on those days are preserved. Moving
+blocks retain some of the serial dependence an i.i.d. resample would destroy.
+
+Treatment − control, 95% interval:
+
+| Block | Metric | Mean | 2.5% | 97.5% | Share of replicates < 0 |
+|---|---|---:|---:|---:|---:|
+| 21 | ΔCAGR pp | −3.61 | −14.76 | +6.45 | 0.75 |
+| 21 | ΔSharpe | −0.109 | −0.433 | +0.198 | 0.75 |
+| 21 | ΔMaxDD pp | −1.00 | −7.50 | +4.34 | 0.63 |
+| 63 | ΔCAGR pp | −3.78 | −14.67 | +5.40 | 0.76 |
+| 63 | ΔSharpe | −0.113 | −0.428 | +0.182 | 0.76 |
+| 63 | ΔMaxDD pp | −1.79 | −8.22 | +2.68 | 0.78 |
+
+The two block lengths agree closely, which is reassuring about the resampling
+itself. Every interval spans zero, and roughly three-quarters of replicates fall
+below zero for CAGR and Sharpe (treatment worse) and for drawdown (treatment
+better, since a negative ΔMaxDD is a shallower drawdown). So the observed window
+leans toward control on return and treatment on drawdown without resolving either
+sign confidently.
+
+**What this diagnostic is not.** It resamples the *realised* daily return streams;
+it does not re-run the strategy, so it cannot capture how a reordered history
+would have changed admissions or holdings. It is an exploratory uncertainty
+estimate from the observed window — not out-of-sample validation, and not a
+rollout gate.
+
+## Verdict
+
+**Decision: production stays unchanged.** No schema, no ingestion, no wider
+parameter sweep.
+
+- **Established:** the basis is production-relevant. It changes roughly 30% of
+  entries, in the yield-driven direction the screen predicted.
+- **Not established:** which basis performs better. Point estimates favour control
+  on return and Sharpe and treatment on drawdown; every bootstrap interval spans
+  zero, with about three-quarters of the mass on the point-estimate side. The
+  observed window does not resolve the sign.
+- No acceptable return sacrifice is being chosen from these numbers, and no
+  threshold is being fitted to them after the fact.
+
+What a decision would still need:
+
+1. **A pre-registered numeric rule, set before it is applied.** The plan commits
+   to non-inferiority in principle — total return is the basis 12-1 momentum is
+   defined on, so an in-sample shortfall should not automatically lock in the
+   basis we believe is measured wrong — but carries no threshold. An earlier draft
+   used ΔCAGR ≥ −0.50pp at ΔMaxDD ≤ +1.50pp. Choosing a threshold now, with
+   −3.7pp on the table, would be fitting the rule to the answer.
+2. **Out-of-sample evidence.** One five-year snapshot, in-sample throughout.
+3. Note what would *not* help: a matrix over capacity, start date or cadence
+   produces **sensitivity checks, not independent samples**. Fifty-three
+   overlapping paths from one price history are not fifty-three pieces of
+   evidence, and treating them that way would overstate what a sweep buys.
+
+## Reproduce
+
+```powershell
+python scripts/run_total_return_ab.py --build-candidates   # ~9 min, cached
+python scripts/run_total_return_ab.py                      # both arms + bootstrap, ~40 s
+```
+
+The bootstrap parameters are constants in the script (`BOOTSTRAP_SEED`,
+`BOOTSTRAP_REPLICATES`, `BOOTSTRAP_BLOCKS`) rather than flags, so a run cannot
+quietly pick a different seed or block length.
+
+Guards in `tests/unit/test_total_return_ab.py`: identical input series must
+produce identical books (so an observed difference is the basis, not replay
+nondeterminism); a controlled momentum change must move a name across the cutoff
+(so the harness can see what it is looking for); the volatility leg must be
+untouched between arms; and a name with no computable momentum in an arm must not
+qualify on a value carried over from candidate generation.

reports/tr-rank-agreement-mechanism.json

@@ -0,0 +1,105 @@
+{
+  "generated_at": "2026-09-07T10:55:05.693160+00:00",
+  "snapshot": "C:\\Workspace\\signal-platform\\backtest_snapshots\\survivorship-a.sqlite",
+  "cache": "C:\\Workspace\\signal-platform\\backtest_snapshots\\tr-rank-cache.sqlite",
+  "window": {
+    "start": "2021-06-24",
+    "end": "2026-07-02"
+  },
+  "universe_size": 505,
+  "sample_every_n_sessions": 25,
+  "result": {
+    "what_it_measures": "Spearman correlation, per sampled date, between a name's trailing 12-month implied yield (from the ALL/SPLIT ratio over the formation window) and the change in its momentum percentile between the two arms.",
+    "caveat": "Both the stock series and the benchmark leg change between arms, so this is strong evidence consistent with a dividend effect - it does not isolate the stock's own yield from the beta-weighted benchmark yield.",
+    "formation_window_bars": 252,
+    "min_symbols_per_date": 30,
+    "sampled_dates": 40,
+    "sample_date_list": [
+      "2022-07-28",
+      "2022-09-01",
+      "2022-10-07",
+      "2022-11-11",
+      "2022-12-19",
+      "2023-01-26",
+      "2023-03-03",
+      "2023-04-10",
+      "2023-05-15",
+      "2023-06-21",
+      "2023-07-27",
+      "2023-08-31",
+      "2023-10-06",
+      "2023-11-10",
+      "2023-12-18",
+      "2024-01-25",
+      "2024-03-01",
+      "2024-04-08",
+      "2024-05-13",
+      "2024-06-18",
+      "2024-07-25",
+      "2024-08-29",
+      "2024-10-04",
+      "2024-11-08",
+      "2024-12-16",
+      "2025-01-24",
+      "2025-03-03",
+      "2025-04-07",
+      "2025-05-13",
+      "2025-06-18",
+      "2025-07-25",
+      "2025-08-29",
+      "2025-10-06",
+      "2025-11-10",
+      "2025-12-16",
+      "2026-01-23",
+      "2026-03-02",
+      "2026-04-07",
+      "2026-05-12",
+      "2026-06-17"
+    ],
+    "mean_spearman_yield_vs_percentile_shift": 0.7924,
+    "min": 0.7138,
+    "max": 0.8459,
+    "per_date_spearman": [
+      0.7814,
+      0.7933,
+      0.8216,
+      0.8325,
+      0.8108,
+      0.8147,
+      0.8074,
+      0.8421,
+      0.7867,
+      0.7907,
+      0.7682,
+      0.7568,
+      0.7849,
+      0.7536,
+      0.7138,
+      0.7385,
+      0.7414,
+      0.7697,
+      0.8078,
+      0.8325,
+      0.8132,
+      0.8459,
+      0.8397,
+      0.7966,
+      0.7808,
+      0.8285,
+      0.8068,
+      0.8376,
+      0.7974,
+      0.8256,
+      0.8046,
+      0.7762,
+      0.8008,
+      0.8046,
+      0.7793,
+      0.7978,
+      0.7678,
+      0.7549,
+      0.7501,
+      0.7384
+    ]
+  }
+}

reports/tr-rank-agreement.json

@@ -0,0 +1,540 @@
+{
+  "generated_at": "2026-09-07T10:57:26.602851+00:00",
+  "snapshot": "C:\\Workspace\\signal-platform\\backtest_snapshots\\survivorship-a.sqlite",
+  "cache": "C:\\Workspace\\signal-platform\\backtest_snapshots\\tr-rank-cache.sqlite",
+  "window": {
+    "start": "2021-06-24",
+    "end": "2026-07-02"
+  },
+  "universe_size": 505,
+  "inert_snapshot_symbols": [
+    "RHM"
+  ],
+  "cutoff": 80.0,
+  "verdict": "proceed to Step 3 - ordered lists differ",
+  "result": {
+    "evaluated_dates": 1009,
+    "skipped_dates_no_history": 252,
+    "dates_with_different_admitted_set": 1007,
+    "dates_with_different_ordered_list": 1009,
+    "mean_set_disagreement": 0.08668279921946223,
+    "max_set_disagreement": 0.21428571428571427,
+    "mean_admitted_per_arm": 100.09,
+    "mean_names_only_in_price_return": 4.56,
+    "mean_names_only_in_total_return": 4.56,
+    "mean_replacements_per_date": 4.56,
+    "max_replacements_per_date": 12,
+    "mean_symmetric_difference": 9.13,
+    "per_year": {
+      "2022": {
+        "dates": 132,
+        "changed_sets": 132,
+        "changed_orders": 132
+      },
+      "2023": {
+        "dates": 250,
+        "changed_sets": 250,
+        "changed_orders": 250
+      },
+      "2024": {
+        "dates": 252,
+        "changed_sets": 252,
+        "changed_orders": 252
+      },
+      "2025": {
+        "dates": 250,
+        "changed_sets": 248,
+        "changed_orders": 250
+      },
+      "2026": {
+        "dates": 125,
+        "changed_sets": 125,
+        "changed_orders": 125
+      }
+    },
+    "examples": [
+      {
+        "date": "2022-06-24",
+        "admitted_price_return": 99,
+        "admitted_total_return": 99,
+        "only_in_price_return": [
+          "CDNS",
+          "MAR",
+          "FIX",
+          "COST",
+          "LIN",
+          "AON",
+          "STE",
+          "CI",
+          "AJG"
+        ],
+        "only_in_total_return": [
+          "EXC",
+          "PSX",
+          "WY",
+          "AMCR",
+          "DTE",
+          "GIS",
+          "FE",
+          "BMY",
+          "ES"
+        ],
+        "order_changed_without_membership_change": false,
+        "first_10_price_return": [
+          "OXY",
+          "EQT",
+          "APA",
+          "DVN",
+          "MOS",
+          "CTRA",
+          "ON",
+          "ALB",
+          "CF",
+          "SMCI"
+        ],
+        "first_10_total_return": [
+          "OXY",
+          "EQT",
+          "APA",
+          "DVN",
+          "MOS",
+          "CTRA",
+          "ON",
+          "CF",
+          "ALB",
+          "SMCI"
+        ]
+      },
+      {
+        "date": "2022-06-27",
+        "admitted_price_return": 99,
+        "admitted_total_return": 99,
+        "only_in_price_return": [
+          "FCX",
+          "COST",
+          "FIX",
+          "CI",
+          "PAYX",
+          "HSIC",
+          "L",
+          "AIZ"
+        ],
+        "only_in_total_return": [
+          "EXC",
+          "MO",
+          "AMCR",
+          "GIS",
+          "DTE",
+          "CMS",
+          "FE",
+          "BMY"
+        ],
+        "order_changed_without_membership_change": false,
+        "first_10_price_return": [
+          "APA",
+          "OXY",
+          "DVN",
+          "EQT",
+          "MOS",
+          "ON",
+          "CTRA",
+          "ALB",
+          "CF",
+          "SMCI"
+        ],
+        "first_10_total_return": [
+          "APA",
+          "OXY",
+          "DVN",
+          "EQT",
+          "MOS",
+          "ON",
+          "CTRA",
+          "CF",
+          "SMCI",
+          "ALB"
+        ]
+      },
+      {
+        "date": "2022-06-28",
+        "admitted_price_return": 99,
+        "admitted_total_return": 99,
+        "only_in_price_return": [
+          "FCX",
+          "LYV",
+          "ULTA",
+          "CDNS",
+          "AMP",
+          "FIX",
+          "AJG"
+        ],
+        "only_in_total_return": [
+          "EXC",
+          "MO",
+          "AMCR",
+          "ETR",
+          "DTE",
+          "CMS",
+          "BMY"
+        ],
+        "order_changed_without_membership_change": false,
+        "first_10_price_return": [
+          "OXY",
+          "APA",
+          "DVN",
+          "MOS",
+          "EQT",
+          "CTRA",
+          "ON",
+          "CF",
+          "ALB",
+          "HAL"
+        ],
+        "first_10_total_return": [
+          "OXY",
+          "APA",
+          "DVN",
+          "EQT",
+          "MOS",
+          "CTRA",
+          "ON",
+          "CF",
+          "EXE",
+          "HAL"
+        ]
+      },
+      {
+        "date": "2022-06-29",
+        "admitted_price_return": 99,
+        "admitted_total_return": 99,
+        "only_in_price_return": [
+          "CDNS",
+          "FIX",
+          "COST",
+          "WAB",
+          "AJG",
+          "TRV"
+        ],
+        "only_in_total_return": [
+          "EXC",
+          "LYB",
+          "MO",
+          "AMCR",
+          "DTE",
+          "WEC"
+        ],
+        "order_changed_without_membership_change": false,
+        "first_10_price_return": [
+          "OXY",
+          "APA",
+          "DVN",
+          "EQT",
+          "MOS",
+          "CTRA",
+          "ALB",
+          "ON",
+          "CF",
+          "EXE"
+        ],
+        "first_10_total_return": [
+          "OXY",
+          "APA",
+          "DVN",
+          "EQT",
+          "MOS",
+          "CTRA",
+          "CF",
+          "ALB",
+          "EXE",
+          "ON"
+        ]
+      },
+      {
+        "date": "2022-06-30",
+        "admitted_price_return": 99,
+        "admitted_total_return": 99,
+        "only_in_price_return": [
+          "LYV",
+          "CDNS",
+          "RJF",
+          "WAB",
+          "FIX",
+          "AON",
+          "TKO"
+        ],
+        "only_in_total_return": [
+          "EXC",
+          "NRG",
+          "MO",
+          "LMT",
+          "DTE",
+          "WEC",
+          "AEE"
+        ],
+        "order_changed_without_membership_change": false,
+        "first_10_price_return": [
+          "OXY",
+          "APA",
+          "DVN",
+          "EQT",
+          "MOS",
+          "CTRA",
+          "ON",
+          "CF",
+          "ALB",
+          "HAL"
+        ],
+        "first_10_total_return": [
+          "OXY",
+          "APA",
+          "DVN",
+          "EQT",
+          "MOS",
+          "CTRA",
+          "ON",
+          "CF",
+          "EXE",
+          "HAL"
+        ]
+      },
+      {
+        "date": "2022-07-01",
+        "admitted_price_return": 99,
+        "admitted_total_return": 99,
+        "only_in_price_return": [
+          "DDOG",
+          "LYV",
+          "HLT",
+          "RJF",
+          "FIX",
+          "TKO",
+          "AON",
+          "AJG",
+          "ACGL"
+        ],
+        "only_in_total_return": [
+          "EXC",
+          "WY",
+          "NRG",
+          "AMCR",
+          "MO",
+          "PKG",
+          "LMT",
+          "CMS",
+          "KO"
+        ],
+        "order_changed_without_membership_change": false,
+        "first_10_price_return": [
+          "EQT",
+          "OXY",
+          "APA",
+          "DVN",
+          "MOS",
+          "CTRA",
+          "ON",
+          "SMCI",
+          "CF",
+          "EXE"
+        ],
+        "first_10_total_return": [
+          "EQT",
+          "OXY",
+          "APA",
+          "DVN",
+          "MOS",
+          "CTRA",
+          "ON",
+          "CF",
+          "SMCI",
+          "EXE"
+        ]
+      },
+      {
+        "date": "2022-07-05",
+        "admitted_price_return": 99,
+        "admitted_total_return": 99,
+        "only_in_price_return": [
+          "HLT",
+          "AON",
+          "TKO",
+          "L",
+          "TRV"
+        ],
+        "only_in_total_return": [
+          "EXC",
+          "WY",
+          "MO",
+          "CMS",
+          "DTE"
+        ],
+        "order_changed_without_membership_change": false,
+        "first_10_price_return": [
+          "APA",
+          "OXY",
+          "DVN",
+          "EQT",
+          "MOS",
+          "ON",
+          "SMCI",
+          "CTRA",
+          "CF",
+          "HAL"
+        ],
+        "first_10_total_return": [
+          "EQT",
+          "OXY",
+          "APA",
+          "DVN",
+          "MOS",
+          "ON",
+          "CTRA",
+          "SMCI",
+          "CF",
+          "EXE"
+        ]
+      },
+      {
+        "date": "2022-07-06",
+        "admitted_price_return": 99,
+        "admitted_total_return": 99,
+        "only_in_price_return": [
+          "TPL",
+          "ANET",
+          "HLT",
+          "LIN",
+          "UNH",
+          "CVS"
+        ],
+        "only_in_total_return": [
+          "EXC",
+          "RF",
+          "LYB",
+          "WY",
+          "MO",
+          "ETR"
+        ],
+        "order_changed_without_membership_change": false,
+        "first_10_price_return": [
+          "APA",
+          "OXY",
+          "DVN",
+          "EQT",
+          "MOS",
+          "ON",
+          "CTRA",
+          "SMCI",
+          "CF",
+          "HAL"
+        ],
+        "first_10_total_return": [
+          "APA",
+          "OXY",
+          "DVN",
+          "EQT",
+          "MOS",
+          "CTRA",
+          "ON",
+          "SMCI",
+          "CF",
+          "HAL"
+        ]
+      },
+      {
+        "date": "2022-07-07",
+        "admitted_price_return": 99,
+        "admitted_total_return": 99,
+        "only_in_price_return": [
+          "RJF",
+          "LIN",
+          "PKG",
+          "L"
+        ],
+        "only_in_total_return": [
+          "EXC",
+          "LYB",
+          "WY",
+          "ETR"
+        ],
+        "order_changed_without_membership_change": false,
+        "first_10_price_return": [
+          "APA",
+          "OXY",
+          "DVN",
+          "EQT",
+          "ON",
+          "MOS",
+          "SMCI",
+          "CTRA",
+          "HAL",
+          "CF"
+        ],
+        "first_10_total_return": [
+          "APA",
+          "EQT",
+          "OXY",
+          "DVN",
+          "MOS",
+          "ON",
+          "CTRA",
+          "SMCI",
+          "HAL",
+          "EXE"
+        ]
+      },
+      {
+        "date": "2022-07-08",
+        "admitted_price_return": 99,
+        "admitted_total_return": 99,
+        "only_in_price_return": [
+          "ANET",
+          "HLT",
+          "RJF",
+          "WAB",
+          "RSG",
+          "AIZ",
+          "HSY"
+        ],
+        "only_in_total_return": [
+          "EXC",
+          "LYB",
+          "AMCR",
+          "MO",
+          "ETR",
+          "AEP",
+          "ATO"
+        ],
+        "order_changed_without_membership_change": false,
+        "first_10_price_return": [
+          "APA",
+          "OXY",
+          "DVN",
+          "EQT",
+          "ON",
+          "MOS",
+          "CTRA",
+          "SMCI",
+          "HAL",
+          "VLO"
+        ],
+        "first_10_total_return": [
+          "APA",
+          "OXY",
+          "DVN",
+          "EQT",
+          "CTRA",
+          "MOS",
+          "ON",
+          "SMCI",
+          "HAL",
+          "EXE"
+        ]
+      }
+    ],
+    "elapsed_seconds": 132.0
+  },
+  "coverage": {
+    "symbols_cached": 507,
+    "symbols_with_unmatched_dates": 0,
+    "total_unmatched_dates": 0
+  }
+}

scripts/prep_survivorship_snapshot.py

@@ -0,0 +1,477 @@
+"""Build the two survivorship-audit snapshots from a base snapshot.
+
+Arm A is the universe the backtest already runs on. Arm B is the same universe
+plus the S&P 500 constituents removed while the snapshot window was open, so the
+strategy can trade the names that later left the index — including the ones that
+failed.
+
+The base file is never modified: both arms are copies, and the base hash is
+re-checked at the end.
+
+Three things this script has to get right, all of them learned from the data:
+
+* **Schema drift.** The snapshot files on disk predate migrations 026 and 032,
+  so ``select(Ticker)`` against them fails with ``no such column: tickers.cik``
+  — and that is the path ``backtest_service._fetch_columns`` takes. Both copies
+  are synced against ``Base.metadata`` before anything else touches them.
+* **Frozen tails.** Alpaca does not end a halted symbol's series; it repeats the
+  last price at zero volume. SBNY sits at 70.00 for 509 sessions after the FDIC
+  closed it, and INFO sits at 108.61 for 1,090 sessions before a different
+  company reuses the ticker. Only the symbols with a *reviewed cutoff* in the
+  dataset are truncated. Everywhere else a long zero-volume run is a tripwire:
+  the symbol is left out and reported for review rather than silently cut.
+* **Terminal marks.** A failed bank stops trading mid-collapse, so the exchange
+  bars never show the loss. A mark is written only where the dataset carries an
+  explicit confirmation *and* the retained final bar matches the reviewed
+  ``last_real_bar``. The mark is an assumption, not an observed execution.
+
+Nothing is inferred. ``delisted_on`` / ``delisted_reason`` are written only for
+confirmed terminal events; a series that merely ends early leaves them NULL and
+is reported as an unresolved ending status.
+
+Usage::
+
+    python scripts/prep_survivorship_snapshot.py
+    python scripts/prep_survivorship_snapshot.py --base backtest_snapshots/prod.sqlite --force
+"""
+
+from __future__ import annotations
+
+import argparse
+import asyncio
+import hashlib
+import json
+import shutil
+import sqlite3
+import sys
+from datetime import date, datetime, time, timedelta, timezone
+from pathlib import Path
+from typing import Any
+
+ROOT = Path(__file__).resolve().parents[1]
+if str(ROOT) not in sys.path:
+    sys.path.insert(0, str(ROOT))
+
+# Bars are cut at the first run of this many consecutive zero-volume sessions.
+ZERO_VOLUME_RUN = 5
+# Close assigned to a confirmed terminal event. Not zero: a zero price divides
+# through several return calculations, and 1 cent is a near-total loss either way.
+TERMINAL_CLOSE = 0.01
+
+DEFAULT_BASE = ROOT / "backtest_snapshots" / "prod.sqlite"
+DEFAULT_REMOVALS = ROOT / "scripts" / "data" / "sp500_removals.json"
+
+
+# ---------------------------------------------------------------------------
+# schema sync
+# ---------------------------------------------------------------------------
+
+def sync_schema(path: Path) -> dict[str, list[str]]:
+    """Add every column the models declare that the snapshot file lacks.
+
+    Nullable and untyped-default only: this brings an old snapshot up to the
+    current models without inventing data. Tables the snapshot does not have at
+    all are left alone — a snapshot is a subset of production by design.
+    """
+    from sqlalchemy.dialects import sqlite as sqlite_dialect
+
+    from app.database import Base
+    import app.models  # noqa: F401  (registers every model on Base.metadata)
+
+    dialect = sqlite_dialect.dialect()
+    added: dict[str, list[str]] = {}
+    con = sqlite3.connect(path)
+    try:
+        present = {r[0] for r in con.execute("SELECT name FROM sqlite_master WHERE type='table'")}
+        for table in Base.metadata.sorted_tables:
+            if table.name not in present:
+                continue
+            have = {r[1] for r in con.execute(f'PRAGMA table_info("{table.name}")')}
+            for column in table.columns:
+                if column.name in have:
+                    continue
+                type_sql = column.type.compile(dialect)
+                con.execute(f'ALTER TABLE "{table.name}" ADD COLUMN "{column.name}" {type_sql}')
+                added.setdefault(table.name, []).append(column.name)
+        con.commit()
+    finally:
+        con.close()
+    return added
+
+
+# ---------------------------------------------------------------------------
+# price data
+# ---------------------------------------------------------------------------
+
+def frozen_run_start(bars: list[Any]) -> int | None:
+    """Index where the first run of >= ZERO_VOLUME_RUN zero-volume bars begins.
+
+    A detector, not a cutter: the caller decides what to do with the answer, and
+    only a reviewed cutoff is allowed to shorten a series.
+    """
+    volumes = [int(b.volume) for b in bars]
+    i = 0
+    while i < len(volumes):
+        if volumes[i] == 0:
+            j = i
+            while j < len(volumes) and volumes[j] == 0:
+                j += 1
+            if j - i >= ZERO_VOLUME_RUN:
+                return i
+            i = j
+        else:
+            i += 1
+    return None
+
+
+def apply_reviewed_cutoff(bars: list[Any], cutoff: date) -> tuple[list[Any], int]:
+    """Keep bars up to and including *cutoff*. Returns (kept, dropped)."""
+    kept = [b for b in bars if b.timestamp.date() <= cutoff]
+    return kept, len(bars) - len(kept)
+
+
+async def fetch_bars(symbol: str, start: date, end: date) -> list[Any]:
+    """Daily SPLIT-adjusted bars, matching how the snapshot's existing bars were ingested."""
+    from app.config import settings
+    from app.providers.alpaca import AlpacaOHLCVProvider
+
+    if not settings.alpaca_api_key or not settings.alpaca_api_secret:
+        raise SystemExit("Alpaca credentials are not configured; cannot restore price history.")
+
+    from alpaca.data.enums import Adjustment
+    from alpaca.data.requests import StockBarsRequest
+    from alpaca.data.timeframe import TimeFrame
+
+    provider = AlpacaOHLCVProvider(settings.alpaca_api_key, settings.alpaca_api_secret)
+    alpaca_symbol = provider._to_alpaca_symbol(symbol)
+    request = StockBarsRequest(
+        symbol_or_symbols=alpaca_symbol,
+        timeframe=TimeFrame.Day,
+        start=datetime.combine(start, time.min, tzinfo=timezone.utc),
+        end=datetime.combine(end + timedelta(days=1), time.min, tzinfo=timezone.utc),
+        adjustment=Adjustment.SPLIT,
+    )
+    # 85 sequential requests are enough for Alpaca to drop one connection, and a
+    # half-built arm is worse than a slow one.
+    last_error: Exception | None = None
+    for attempt in range(4):
+        try:
+            bars = await asyncio.to_thread(provider._client.get_stock_bars, request)
+            return list(bars.data.get(alpaca_symbol, []))
+        except Exception as exc:  # noqa: BLE001 - retry any transport failure
+            last_error = exc
+            if attempt < 3:
+                await asyncio.sleep(2 * (attempt + 1))
+    raise SystemExit(f"Alpaca fetch failed for {symbol} after 4 attempts: {last_error}")
+
+
+# ---------------------------------------------------------------------------
+# snapshot writes
+# ---------------------------------------------------------------------------
+
+def snapshot_window(path: Path) -> tuple[date, date]:
+    con = sqlite3.connect(path)
+    try:
+        lo, hi = con.execute("SELECT min(date), max(date) FROM ohlcv_records").fetchone()
+    finally:
+        con.close()
+    return date.fromisoformat(lo), date.fromisoformat(hi)
+
+
+def trading_calendar(path: Path) -> list[date]:
+    """SPY sessions from the snapshot's own benchmark series."""
+    con = sqlite3.connect(path)
+    try:
+        rows = con.execute(
+            "SELECT date FROM benchmark_prices WHERE symbol = 'SPY' ORDER BY date"
+        ).fetchall()
+    finally:
+        con.close()
+    return [date.fromisoformat(r[0]) for r in rows]
+
+
+def next_session(calendar: list[date], after: date) -> date | None:
+    for d in calendar:
+        if d > after:
+            return d
+    return None
+
+
+def insert_symbol(
+    con: sqlite3.Connection,
+    symbol: str,
+    name: str,
+    bars: list[Any],
+    *,
+    delisted_on: str | None,
+    delisted_reason: str | None,
+    terminal_date: date | None,
+) -> int:
+    """Insert one restored ticker and its bars. Returns the bar count written."""
+    now = datetime.utcnow().isoformat(sep=" ")
+    cur = con.execute(
+        "INSERT INTO tickers (symbol, name, created_at, delisted_on, delisted_reason) "
+        "VALUES (?, ?, ?, ?, ?)",
+        (symbol, name, now, delisted_on, delisted_reason),
+    )
+    ticker_id = int(cur.lastrowid)
+
+    rows = [
+        (
+            ticker_id,
+            b.timestamp.date().isoformat(),
+            float(b.open),
+            float(b.high),
+            float(b.low),
+            float(b.close),
+            int(b.volume),
+            now,
+        )
+        for b in bars
+    ]
+    if terminal_date is not None:
+        rows.append((
+            ticker_id, terminal_date.isoformat(),
+            TERMINAL_CLOSE, TERMINAL_CLOSE, TERMINAL_CLOSE, TERMINAL_CLOSE, 0, now,
+        ))
+    con.executemany(
+        "INSERT INTO ohlcv_records "
+        "(ticker_id, date, open, high, low, close, volume, created_at) "
+        "VALUES (?, ?, ?, ?, ?, ?, ?, ?)",
+        rows,
+    )
+    return len(rows)
+
+
+# ---------------------------------------------------------------------------
+# driver
+# ---------------------------------------------------------------------------
+
+def file_hash(path: Path) -> str:
+    h = hashlib.sha256()
+    with path.open("rb") as fh:
+        for chunk in iter(lambda: fh.read(1 << 20), b""):
+            h.update(chunk)
+    return h.hexdigest()
+
+
+async def build(args: argparse.Namespace) -> int:
+    base = Path(args.base).resolve()
+    if not base.exists():
+        raise SystemExit(f"Base snapshot not found: {base}")
+
+    out_dir = Path(args.out_dir).resolve()
+    arm_a = out_dir / args.arm_a_name
+    arm_b = out_dir / args.arm_b_name
+    events_path = arm_b.with_name(arm_b.stem + "-terminal-events.json")
+
+    doc = json.loads(Path(args.removals).read_text(encoding="utf-8"))
+    removals = doc["removals"]
+    confirmations = {c["symbol"]: c for c in doc["terminal_confirmations"] if c.get("confirmed")}
+    cutoffs = {
+        c["symbol"]: date.fromisoformat(c["last_real_bar"])
+        for c in doc.get("reviewed_cutoffs", {}).get("cutoffs", [])
+    }
+    # A corporate action ends a listing; a market-cap removal does not, so a
+    # series that stops early without one is an ending we cannot account for.
+    ENDING_EXPLAINED = {"acquisition", "receivership", "bankruptcy", "ticker_change"}
+
+    base_hash_before = file_hash(base)
+    window_start, window_end = snapshot_window(base)
+    print(f"base     : {base}")
+    print(f"window   : {window_start} .. {window_end}")
+    print(f"removals : {len(removals)} rows, {sum(1 for r in removals if r['restore_symbol'])} restorable")
+
+    for target in (arm_a, arm_b):
+        if target.exists() and not args.force:
+            raise SystemExit(f"{target} already exists (use --force to rebuild)")
+        if target.exists():
+            target.unlink()
+        shutil.copy2(base, target)
+        added = sync_schema(target)
+        detail = ", ".join(f"{t}: {', '.join(c)}" for t, c in added.items()) or "nothing missing"
+        print(f"copied   : {target.name}  (schema sync -> {detail})")
+
+    calendar = trading_calendar(base)
+    con = sqlite3.connect(arm_b)
+    existing = {r[0].upper() for r in con.execute("SELECT symbol FROM tickers")}
+
+    restored: list[dict] = []
+    unresolved: list[dict] = []
+    terminal_events: dict[str, str] = {}
+
+    try:
+        for row in removals:
+            symbol = row["restore_symbol"]
+            if not symbol:
+                continue
+            if symbol.upper() in existing:
+                unresolved.append({"kind": "skipped", "symbol": symbol, "why": "already present in the base snapshot"})
+                continue
+
+            bars = await fetch_bars(symbol, window_start, window_end)
+            if not bars:
+                unresolved.append({"kind": "coverage", "symbol": symbol, "why": "Alpaca returned no bars in the window"})
+                print(f"  {symbol:<6} NO DATA  (unresolved coverage)")
+                continue
+
+            cutoff = cutoffs.get(symbol) or cutoffs.get(row["symbol"])
+            frozen_at = frozen_run_start(bars)
+            if cutoff is not None:
+                kept, dropped = apply_reviewed_cutoff(bars, cutoff)
+                if not kept or kept[-1].timestamp.date() != cutoff:
+                    unresolved.append({
+                        "symbol": symbol,
+                        "kind": "coverage",
+                        "why": f"reviewed cutoff {cutoff} is not a bar in the downloaded series",
+                    })
+                    print(f"  {symbol:<6} CUTOFF MISSING {cutoff}  (unresolved coverage)")
+                    continue
+            elif frozen_at is not None:
+                # Detector only. An unreviewed frozen run means the data changed
+                # under us; restoring either half would be a guess, so report it.
+                unresolved.append({
+                    "kind": "coverage",
+                    "symbol": symbol,
+                    "why": (
+                        f"unreviewed frozen run of >= {ZERO_VOLUME_RUN} zero-volume sessions "
+                        f"from {bars[frozen_at].timestamp.date()}; needs a reviewed cutoff "
+                        "before this symbol can be restored"
+                    ),
+                })
+                print(f"  {symbol:<6} FROZEN RUN at {bars[frozen_at].timestamp.date()}  (unresolved, not restored)")
+                continue
+            else:
+                kept, dropped = bars, 0
+
+            effective_last = kept[-1].timestamp.date()
+            confirmation = confirmations.get(row["symbol"]) or confirmations.get(symbol)
+            terminal_date = None
+            if confirmation is not None:
+                confirmed_last = date.fromisoformat(confirmation["last_real_bar"])
+                if effective_last != confirmed_last:
+                    unresolved.append({
+                        "symbol": symbol,
+                        "kind": "terminal_mark",
+                        "why": (
+                            f"retained final bar {effective_last} does not match the reviewed "
+                            f"last_real_bar {confirmed_last}; no terminal mark written"
+                        ),
+                    })
+                    print(f"  {symbol:<6} MARK SKIPPED  final bar {effective_last} != confirmed {confirmed_last}")
+                else:
+                    terminal_date = next_session(calendar, effective_last)
+                    if terminal_date is None or terminal_date > window_end:
+                        terminal_date = None
+                        unresolved.append({
+                            "symbol": symbol,
+                            "kind": "terminal_mark",
+                            "why": "confirmed terminal event has no trading session left inside the window",
+                        })
+                    else:
+                        terminal_events[symbol] = terminal_date.isoformat()
+
+            # Documentary only, and only where the event is confirmed. A series
+            # that merely ends early is an unresolved ending, not a delisting.
+            delisted_on = None
+            delisted_reason = None
+            if terminal_date is not None:
+                delisted_on = terminal_date.isoformat()
+                delisted_reason = row["reason"]
+            elif effective_last < window_end and row["reason"] not in ENDING_EXPLAINED:
+                unresolved.append({
+                    "kind": "ending_status",
+                    "symbol": symbol,
+                    "why": (
+                        f"price history ends {effective_last}, before the window closes on "
+                        f"{window_end}, and the source row ({row['reason']}) does not explain "
+                        "an ending; ending status unresolved and delisting fields left null"
+                    ),
+                })
+
+            written = insert_symbol(
+                con, symbol, row["name"], kept,
+                delisted_on=delisted_on, delisted_reason=delisted_reason,
+                terminal_date=terminal_date,
+            )
+            restored.append({
+                "symbol": symbol,
+                "removed_on": row["removed_on"],
+                "reason": row["reason"],
+                "bars": written,
+                "first_bar": kept[0].timestamp.date().isoformat(),
+                "effective_last_bar": effective_last.isoformat(),
+                "frozen_bars_dropped": dropped,
+                "terminal_mark": terminal_events.get(symbol),
+            })
+            mark = f"  terminal={terminal_events[symbol]} @ {TERMINAL_CLOSE}" if symbol in terminal_events else ""
+            drop = f"  dropped={dropped}" if dropped else ""
+            print(f"  {symbol:<6} bars={written:<5} {kept[0].timestamp.date()}..{effective_last}{drop}{mark}")
+
+        con.commit()
+    finally:
+        con.close()
+
+    events_path.write_text(json.dumps(terminal_events, indent=2) + "\n", encoding="utf-8")
+
+    if file_hash(base) != base_hash_before:
+        raise SystemExit(f"BASE SNAPSHOT WAS MODIFIED: {base}")
+
+    con_a = sqlite3.connect(arm_a)
+    con_b = sqlite3.connect(arm_b)
+    counts = {
+        "arm_a_tickers": con_a.execute("SELECT count(*) FROM tickers").fetchone()[0],
+        "arm_a_bars": con_a.execute("SELECT count(*) FROM ohlcv_records").fetchone()[0],
+        "arm_b_tickers": con_b.execute("SELECT count(*) FROM tickers").fetchone()[0],
+        "arm_b_bars": con_b.execute("SELECT count(*) FROM ohlcv_records").fetchone()[0],
+    }
+    con_a.close()
+    con_b.close()
+
+    by_kind: dict[str, list[dict]] = {}
+    for row in unresolved:
+        by_kind.setdefault(row.get("kind", "other"), []).append(row)
+
+    print("\nsummary")
+    print(f"  restored           : {len(restored)}")
+    print(f"  unresolved coverage: {len(by_kind.get('coverage', []))}  (symbols not restored)")
+    print(f"  unresolved endings : {len(by_kind.get('ending_status', []))}  (restored, ending status unknown)")
+    print(f"  terminal marks     : {len(terminal_events)} -> {events_path.name}")
+    print(f"  arm A              : {counts['arm_a_tickers']} tickers, {counts['arm_a_bars']} bars")
+    print(f"  arm B              : {counts['arm_b_tickers']} tickers, {counts['arm_b_bars']} bars")
+    print(f"  base unchanged     : sha256 {base_hash_before[:16]}...")
+    for kind in ("coverage", "ending_status", "terminal_mark"):
+        for row in by_kind.get(kind, []):
+            print(f"  UNRESOLVED[{kind}] {row['symbol']:<6} {row['why']}")
+
+    manifest = {
+        "generated_at": datetime.now(timezone.utc).isoformat(),
+        "base": str(base),
+        "base_sha256": base_hash_before,
+        "window": {"start": window_start.isoformat(), "end": window_end.isoformat()},
+        "zero_volume_run": ZERO_VOLUME_RUN,
+        "terminal_close": TERMINAL_CLOSE,
+        "counts": counts,
+        "restored": restored,
+        "unresolved": unresolved,
+        "terminal_events": terminal_events,
+    }
+    manifest_path = arm_b.with_name(arm_b.stem + "-prep-manifest.json")
+    manifest_path.write_text(json.dumps(manifest, indent=2) + "\n", encoding="utf-8")
+    print(f"  manifest           : {manifest_path.name}")
+    return 0
+
+
+def parse_args() -> argparse.Namespace:
+    p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+    p.add_argument("--base", default=str(DEFAULT_BASE), help="Base snapshot to copy (never modified).")
+    p.add_argument("--out-dir", default=str(DEFAULT_BASE.parent), help="Where the arm copies are written.")
+    p.add_argument("--arm-a-name", default="survivorship-a.sqlite")
+    p.add_argument("--arm-b-name", default="survivorship-b.sqlite")
+    p.add_argument("--removals", default=str(DEFAULT_REMOVALS))
+    p.add_argument("--force", action="store_true", help="Rebuild the arm files if they already exist.")
+    return p.parse_args()
+
+
+if __name__ == "__main__":
+    raise SystemExit(asyncio.run(build(parse_args())))

scripts/research_tr_rank_agreement.py

@@ -0,0 +1,596 @@
+"""Does total-return momentum change any admission or ordering decision?
+
+Step 2 of docs/data-integrity-plan.md. A screening test, deliberately cheap: if
+ranking on total return picks the same names in the same order as ranking on
+price return, nothing downstream can change and the production `tr_factor` work
+(Step 3) is unnecessary.
+
+Why this needs no schema and no ingestion change: the forward-anchored factor
+design exists because *production* ingest is incremental, so a back-adjusted
+series drifts between fetches. Here each symbol's whole history is fetched in one
+request, and within a single response the adjustment is internally consistent.
+
+Method
+------
+* One `Adjustment.SPLIT` and one `Adjustment.ALL` request per symbol plus SPY,
+  cached to sqlite so re-runs are free and reproducible. Aligned by date, never
+  zipped by index; a date present in only one adjustment is dropped and counted.
+* At every SPY session with enough history, residual 12-1 momentum is computed
+  twice with the *same* production function: the price-return series against
+  nominal SPY, and the total-return series against total-return SPY. Only bars at
+  or before the decision date are used.
+* The volatility leg is computed once per date from the SPLIT series and held
+  fixed in both arms, so the only thing that varies is momentum.
+* Both arms are percentiled with the production `_percentiles`, gated at the
+  production `min_momentum_percentile`, then ordered by `blend_strategy_rank`
+  with the symbol as a deterministic tie-break. The full admitted list is
+  compared - never truncated to `max_positions`, because a lower-ranked name can
+  be the first actionable setup when others fail gates or are already held.
+
+Decision rule (pre-registered, docs/data-integrity-plan.md)
+----------------------------------------------------------
+* no ordered-list difference on any evaluated date -> stop, Step 3 unnecessary
+* any ordered-list difference                      -> Step 3 investigates trades
+* incomplete downloads or no evaluable dates       -> inconclusive, fix coverage
+
+Usage::
+
+    python scripts/research_tr_rank_agreement.py --fetch      # populate the cache
+    python scripts/research_tr_rank_agreement.py --time-batch # project the runtime
+    python scripts/research_tr_rank_agreement.py              # full screen
+"""
+
+from __future__ import annotations
+
+import argparse
+import asyncio
+import json
+import sqlite3
+import sys
+import time
+from datetime import date, datetime, time as dtime, timedelta, timezone
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[1]
+if str(ROOT) not in sys.path:
+    sys.path.insert(0, str(ROOT))
+
+DEFAULT_SNAPSHOT = ROOT / "backtest_snapshots" / "survivorship-a.sqlite"
+DEFAULT_CACHE = ROOT / "backtest_snapshots" / "tr-rank-cache.sqlite"
+BENCHMARK = "SPY"
+
+
+# ---------------------------------------------------------------------------
+# cache
+# ---------------------------------------------------------------------------
+
+def open_cache(path: Path) -> sqlite3.Connection:
+    con = sqlite3.connect(path)
+    con.execute(
+        "CREATE TABLE IF NOT EXISTS bars ("
+        " symbol TEXT NOT NULL, date TEXT NOT NULL,"
+        " split_close REAL, all_close REAL,"
+        " PRIMARY KEY (symbol, date))"
+    )
+    con.execute(
+        "CREATE TABLE IF NOT EXISTS fetched ("
+        " symbol TEXT PRIMARY KEY, fetched_at TEXT NOT NULL,"
+        " split_bars INTEGER, all_bars INTEGER, aligned_bars INTEGER,"
+        " dropped_unmatched INTEGER)"
+    )
+    con.commit()
+    return con
+
+
+async def fetch_pair(symbol: str, start: date, end: date) -> tuple[dict, dict]:
+    """(split_closes, all_closes) keyed by date, each from a single request."""
+    from alpaca.data.enums import Adjustment
+    from alpaca.data.historical import StockHistoricalDataClient
+    from alpaca.data.requests import StockBarsRequest
+    from alpaca.data.timeframe import TimeFrame
+
+    from app.config import settings
+
+    if not settings.alpaca_api_key or not settings.alpaca_api_secret:
+        raise SystemExit("Alpaca credentials are not configured.")
+
+    client = StockHistoricalDataClient(settings.alpaca_api_key, settings.alpaca_api_secret)
+    alpaca_symbol = symbol.replace("-", ".")
+    window = (
+        datetime.combine(start, dtime.min, tzinfo=timezone.utc),
+        datetime.combine(end + timedelta(days=1), dtime.min, tzinfo=timezone.utc),
+    )
+
+    async def one(adjustment) -> dict[date, float]:
+        request = StockBarsRequest(
+            symbol_or_symbols=alpaca_symbol, timeframe=TimeFrame.Day,
+            start=window[0], end=window[1], adjustment=adjustment,
+        )
+        last: Exception | None = None
+        for attempt in range(4):
+            try:
+                bars = await asyncio.to_thread(client.get_stock_bars, request)
+                return {
+                    b.timestamp.date(): float(b.close)
+                    for b in bars.data.get(alpaca_symbol, [])
+                }
+            except Exception as exc:  # noqa: BLE001 - retry any transport failure
+                last = exc
+                if attempt < 3:
+                    await asyncio.sleep(2 * (attempt + 1))
+        raise SystemExit(f"Alpaca fetch failed for {symbol}: {last}")
+
+    return await one(Adjustment.SPLIT), await one(Adjustment.ALL)
+
+
+async def populate_cache(symbols: list[str], start: date, end: date, cache: Path) -> dict:
+    con = open_cache(cache)
+    done = {r[0] for r in con.execute("SELECT symbol FROM fetched")}
+    todo = [s for s in symbols if s not in done]
+    print(f"cache {cache.name}: {len(done)} cached, {len(todo)} to fetch")
+
+    coverage: dict[str, dict] = {}
+    for index, symbol in enumerate(todo, 1):
+        split, adjusted = await fetch_pair(symbol, start, end)
+        # Align by date. A date in only one adjustment cannot be compared and is
+        # dropped rather than paired with its neighbour.
+        common = sorted(set(split) & set(adjusted))
+        dropped = len(set(split) ^ set(adjusted))
+        con.executemany(
+            "INSERT OR REPLACE INTO bars (symbol, date, split_close, all_close) VALUES (?,?,?,?)",
+            [(symbol, d.isoformat(), split[d], adjusted[d]) for d in common],
+        )
+        con.execute(
+            "INSERT OR REPLACE INTO fetched VALUES (?,?,?,?,?,?)",
+            (symbol, datetime.now(timezone.utc).isoformat(), len(split), len(adjusted),
+             len(common), dropped),
+        )
+        con.commit()
+        if dropped:
+            print(f"  {symbol:<6} aligned={len(common)} dropped_unmatched={dropped}")
+        if index % 50 == 0:
+            print(f"  … {index}/{len(todo)}")
+
+    for row in con.execute("SELECT symbol, split_bars, all_bars, aligned_bars, dropped_unmatched FROM fetched"):
+        coverage[row[0]] = {
+            "split_bars": row[1], "all_bars": row[2],
+            "aligned_bars": row[3], "dropped_unmatched": row[4],
+        }
+    con.close()
+    return coverage
+
+
+def load_series(cache: Path, symbols: list[str]) -> dict[str, tuple[list[date], list[float], list[float]]]:
+    con = open_cache(cache)
+    out: dict[str, tuple[list[date], list[float], list[float]]] = {}
+    for symbol in symbols:
+        rows = con.execute(
+            "SELECT date, split_close, all_close FROM bars WHERE symbol = ? ORDER BY date",
+            (symbol,),
+        ).fetchall()
+        if not rows:
+            continue
+        out[symbol] = (
+            [date.fromisoformat(r[0]) for r in rows],
+            [float(r[1]) for r in rows],
+            [float(r[2]) for r in rows],
+        )
+    con.close()
+    return out
+
+
+# ---------------------------------------------------------------------------
+# screen
+# ---------------------------------------------------------------------------
+
+def snapshot_symbols(snapshot: Path) -> tuple[list[str], list[str], date, date]:
+    """(symbols with bars, symbols with none, window start, window end).
+
+    A ticker row with no bars in the snapshot cannot rank in either arm, so it is
+    inert rather than missing. Keeping the two apart is what stops a failed
+    download from being quietly reported as agreement.
+    """
+    con = sqlite3.connect(snapshot)
+    try:
+        rows = con.execute(
+            "SELECT t.symbol, count(o.id) FROM tickers t "
+            "LEFT JOIN ohlcv_records o ON o.ticker_id = t.id GROUP BY t.symbol"
+        ).fetchall()
+        lo, hi = con.execute("SELECT min(date), max(date) FROM ohlcv_records").fetchone()
+    finally:
+        con.close()
+    with_bars = sorted(r[0].upper() for r in rows if r[1])
+    without_bars = sorted(r[0].upper() for r in rows if not r[1])
+    return with_bars, without_bars, date.fromisoformat(lo), date.fromisoformat(hi)
+
+
+def ordered_list(
+    momentum: dict[str, float],
+    vol_percentiles: dict[str, float],
+    cutoff: float,
+) -> list[str]:
+    """Admitted set at the production cutoff, ordered by the production blend.
+
+    Symbols are fed to ``_percentiles`` in sorted order in both arms so that ties
+    resolve identically and cannot manufacture a difference.
+    """
+    from app.services.momentum_service import _percentiles, blend_strategy_rank
+
+    percentiles = _percentiles({s: momentum[s] for s in sorted(momentum)})
+    admitted = [s for s, p in percentiles.items() if p >= cutoff]
+    scored = {
+        s: blend_strategy_rank(percentiles[s], vol_percentiles.get(s))
+        for s in admitted
+    }
+    return sorted(admitted, key=lambda s: (-(scored[s] or 0.0), s))
+
+
+def mechanism_check(
+    series: dict[str, tuple[list[date], list[float], list[float]]],
+    sample_dates: list[date],
+) -> dict:
+    """Is the rank shift explained by dividend yield?
+
+    For each sampled decision date, correlate a name's trailing 12-month implied
+    yield — read straight off the ALL/SPLIT ratio, which is what separates the two
+    series — against how far its momentum percentile moves between the arms. A
+    strong positive rank correlation means the difference *is* the dividend, not
+    numerical noise.
+    """
+    from app.services.momentum_service import _MOM_LOOKBACK, _percentiles, compute_residual_12_1_momentum
+
+    bench_dates, bench_split, bench_all = series[BENCHMARK]
+    bench_price = dict(zip(bench_dates, bench_split))
+    bench_tr = dict(zip(bench_dates, bench_all))
+    symbols = sorted(s for s in series if s != BENCHMARK)
+    index_of = {s: {d: i for i, d in enumerate(series[s][0])} for s in symbols}
+
+    correlations: list[float] = []
+    used_dates: list[str] = []
+    for as_of in sample_dates:
+        price_mom: dict[str, float] = {}
+        tr_mom: dict[str, float] = {}
+        yields: dict[str, float] = {}
+        for symbol in symbols:
+            idx = index_of[symbol].get(as_of)
+            if idx is None or idx < _MOM_LOOKBACK:
+                continue
+            dates, split_closes, all_closes = series[symbol]
+            wd, ws, wa = dates[: idx + 1], split_closes[: idx + 1], all_closes[: idx + 1]
+            p = compute_residual_12_1_momentum(wd, ws, bench_price)
+            t = compute_residual_12_1_momentum(wd, wa, bench_tr)
+            if p is None or t is None:
+                continue
+            price_mom[symbol] = p
+            tr_mom[symbol] = t
+            # Ratio drift over the formation window is the accumulated dividend.
+            r_now = wa[-1] / ws[-1] if ws[-1] else None
+            r_then = wa[-_MOM_LOOKBACK] / ws[-_MOM_LOOKBACK] if ws[-_MOM_LOOKBACK] else None
+            if r_now and r_then:
+                yields[symbol] = r_now / r_then - 1.0
+
+        common = sorted(set(price_mom) & set(yields))
+        if len(common) < 30:
+            continue
+        pp = _percentiles({s: price_mom[s] for s in common})
+        tp = _percentiles({s: tr_mom[s] for s in common})
+        shift = {s: tp[s] - pp[s] for s in common}
+        correlations.append(_spearman([yields[s] for s in common], [shift[s] for s in common]))
+        used_dates.append(as_of.isoformat())
+
+    return {
+        "what_it_measures": (
+            "Spearman correlation, per sampled date, between a name's trailing 12-month "
+            "implied yield (from the ALL/SPLIT ratio over the formation window) and the "
+            "change in its momentum percentile between the two arms."
+        ),
+        "caveat": (
+            "Both the stock series and the benchmark leg change between arms, so this is "
+            "strong evidence consistent with a dividend effect - it does not isolate the "
+            "stock's own yield from the beta-weighted benchmark yield."
+        ),
+        "formation_window_bars": _MOM_LOOKBACK,
+        "min_symbols_per_date": 30,
+        "sampled_dates": len(correlations),
+        "sample_date_list": used_dates,
+        "mean_spearman_yield_vs_percentile_shift": (
+            round(sum(correlations) / len(correlations), 4) if correlations else None
+        ),
+        "min": round(min(correlations), 4) if correlations else None,
+        "max": round(max(correlations), 4) if correlations else None,
+        "per_date_spearman": [round(c, 4) for c in correlations],
+    }
+
+
+def _spearman(x: list[float], y: list[float]) -> float:
+    def rank(v: list[float]) -> list[float]:
+        order = sorted(range(len(v)), key=lambda i: v[i])
+        out = [0.0] * len(v)
+        for position, i in enumerate(order):
+            out[i] = float(position)
+        return out
+
+    rx, ry = rank(x), rank(y)
+    n = len(rx)
+    mx, my = sum(rx) / n, sum(ry) / n
+    num = sum((rx[i] - mx) * (ry[i] - my) for i in range(n))
+    dx = sum((rx[i] - mx) ** 2 for i in range(n)) ** 0.5
+    dy = sum((ry[i] - my) ** 2 for i in range(n)) ** 0.5
+    return num / (dx * dy) if dx and dy else 0.0
+
+
+def disagreement(a: list[str], b: list[str]) -> float:
+    sa, sb = set(a), set(b)
+    union = sa | sb
+    if not union:
+        return 0.0
+    return len(sa ^ sb) / len(union)
+
+
+def run_screen(
+    series: dict[str, tuple[list[date], list[float], list[float]]],
+    cutoff: float,
+    *,
+    limit_dates: int | None = None,
+    verbose: bool = True,
+) -> dict:
+    from app.services.momentum_service import (
+        _MOM_LOOKBACK,
+        compute_realized_vol_6m,
+        compute_residual_12_1_momentum,
+        _percentiles,
+    )
+
+    bench_dates, bench_split, bench_all = series[BENCHMARK]
+    bench_price = dict(zip(bench_dates, bench_split))
+    bench_tr = dict(zip(bench_dates, bench_all))
+
+    symbols = sorted(s for s in series if s != BENCHMARK)
+    index_of: dict[str, dict[date, int]] = {
+        s: {d: i for i, d in enumerate(series[s][0])} for s in symbols
+    }
+
+    evaluated = 0
+    skipped_no_history = 0
+    changed_sets = 0
+    changed_orders = 0
+    disagreements: list[float] = []
+    # Counted directly rather than derived from the Jaccard ratio: a swap moves
+    # two names (one out, one in), so the symmetric difference is twice the
+    # replacement count and the two are easy to conflate.
+    replacements: list[int] = []
+    only_price_counts: list[int] = []
+    only_tr_counts: list[int] = []
+    admitted_sizes: list[int] = []
+    per_year: dict[int, dict[str, int]] = {}
+    examples: list[dict] = []
+
+    decision_dates = [d for d in bench_dates]
+    if limit_dates:
+        decision_dates = decision_dates[-limit_dates:]
+
+    for as_of in decision_dates:
+        price_mom: dict[str, float] = {}
+        tr_mom: dict[str, float] = {}
+        vol_values: dict[str, float] = {}
+
+        for symbol in symbols:
+            idx = index_of[symbol].get(as_of)
+            if idx is None or idx < _MOM_LOOKBACK:
+                continue
+            dates, split_closes, all_closes = series[symbol]
+            window_dates = dates[: idx + 1]
+            window_split = split_closes[: idx + 1]
+            window_all = all_closes[: idx + 1]
+
+            price_value = compute_residual_12_1_momentum(window_dates, window_split, bench_price)
+            tr_value = compute_residual_12_1_momentum(window_dates, window_all, bench_tr)
+            # Equal eligibility: a name enters both arms or neither.
+            if price_value is None or tr_value is None:
+                continue
+            price_mom[symbol] = price_value
+            tr_mom[symbol] = tr_value
+
+            vol = compute_realized_vol_6m(window_split)
+            if vol is not None:
+                vol_values[symbol] = vol
+
+        if not price_mom:
+            skipped_no_history += 1
+            continue
+
+        evaluated += 1
+        vol_percentiles = _percentiles({s: vol_values[s] for s in sorted(vol_values)})
+        price_list = ordered_list(price_mom, vol_percentiles, cutoff)
+        tr_list = ordered_list(tr_mom, vol_percentiles, cutoff)
+
+        year = per_year.setdefault(as_of.year, {"dates": 0, "changed_sets": 0, "changed_orders": 0})
+        year["dates"] += 1
+
+        set_price, set_tr = set(price_list), set(tr_list)
+        only_price = len(set_price - set_tr)
+        only_tr = len(set_tr - set_price)
+        only_price_counts.append(only_price)
+        only_tr_counts.append(only_tr)
+        admitted_sizes.append(len(price_list))
+        # A replacement is one name leaving paired with one arriving; when the
+        # two arms admit different counts, the surplus is not a replacement.
+        replacements.append(min(only_price, only_tr))
+
+        same_set = set_price == set_tr
+        same_order = price_list == tr_list
+        if not same_set:
+            changed_sets += 1
+            year["changed_sets"] += 1
+        if not same_order:
+            changed_orders += 1
+            year["changed_orders"] += 1
+            if len(examples) < 10:
+                only_price = [s for s in price_list if s not in set(tr_list)]
+                only_tr = [s for s in tr_list if s not in set(price_list)]
+                examples.append({
+                    "date": as_of.isoformat(),
+                    "admitted_price_return": len(price_list),
+                    "admitted_total_return": len(tr_list),
+                    "only_in_price_return": only_price[:10],
+                    "only_in_total_return": only_tr[:10],
+                    "order_changed_without_membership_change": same_set,
+                    "first_10_price_return": price_list[:10],
+                    "first_10_total_return": tr_list[:10],
+                })
+        disagreements.append(disagreement(price_list, tr_list))
+
+        if verbose and evaluated % 100 == 0:
+            print(f"  … {evaluated} dates evaluated, {changed_orders} with a different ordered list")
+
+    return {
+        "evaluated_dates": evaluated,
+        "skipped_dates_no_history": skipped_no_history,
+        "dates_with_different_admitted_set": changed_sets,
+        "dates_with_different_ordered_list": changed_orders,
+        "mean_set_disagreement": (sum(disagreements) / len(disagreements)) if disagreements else 0.0,
+        "max_set_disagreement": max(disagreements) if disagreements else 0.0,
+        "mean_admitted_per_arm": round(sum(admitted_sizes) / len(admitted_sizes), 2) if admitted_sizes else 0.0,
+        "mean_names_only_in_price_return": round(sum(only_price_counts) / len(only_price_counts), 2) if only_price_counts else 0.0,
+        "mean_names_only_in_total_return": round(sum(only_tr_counts) / len(only_tr_counts), 2) if only_tr_counts else 0.0,
+        "mean_replacements_per_date": round(sum(replacements) / len(replacements), 2) if replacements else 0.0,
+        "max_replacements_per_date": max(replacements) if replacements else 0,
+        "mean_symmetric_difference": round(
+            (sum(only_price_counts) + sum(only_tr_counts)) / len(only_price_counts), 2
+        ) if only_price_counts else 0.0,
+        "per_year": {
+            str(y): v for y, v in sorted(per_year.items())
+        },
+        "examples": examples,
+    }
+
+
+# ---------------------------------------------------------------------------
+# driver
+# ---------------------------------------------------------------------------
+
+async def main(args: argparse.Namespace) -> int:
+    snapshot = Path(args.snapshot)
+    cache = Path(args.cache)
+    symbols, inert, window_start, window_end = snapshot_symbols(snapshot)
+    universe = symbols + [BENCHMARK]
+
+    print(f"snapshot : {snapshot.name}  ({len(symbols)} symbols with bars"
+          + (f", {len(inert)} inert: {', '.join(inert)}" if inert else "") + ")")
+    print(f"window   : {window_start} .. {window_end}  (clipped to the snapshot)")
+
+    if args.fetch:
+        await populate_cache(universe, window_start, window_end, cache)
+        return 0
+
+    series = load_series(cache, universe)
+    missing = [s for s in universe if s not in series]
+    if missing:
+        raise SystemExit(
+            f"{len(missing)} symbols have bars in the snapshot but none in the cache "
+            f"({', '.join(missing[:8])}). Run with --fetch first; an incomplete download "
+            "makes the screen inconclusive and must never be read as agreement."
+        )
+
+    from app.services.admin_service import ACTIVATION_DEFAULTS
+
+    cutoff = float(ACTIVATION_DEFAULTS["min_momentum_percentile"])
+    print(f"cutoff   : {cutoff} (production min_momentum_percentile)")
+
+    if args.mechanism:
+        bench_dates = load_series(cache, [BENCHMARK])[BENCHMARK][0]
+        sample = bench_dates[::args.mechanism]
+        print(f"mechanism check over {len(sample)} sampled dates ...")
+        payload = {
+            "generated_at": datetime.now(timezone.utc).isoformat(),
+            "snapshot": str(snapshot),
+            "cache": str(cache),
+            "window": {"start": window_start.isoformat(), "end": window_end.isoformat()},
+            "universe_size": len(symbols),
+            "sample_every_n_sessions": args.mechanism,
+            "result": mechanism_check(series, sample),
+        }
+        out = Path(args.mechanism_out)
+        out.parent.mkdir(parents=True, exist_ok=True)
+        out.write_text(json.dumps(payload, indent=2) + "\n", encoding="utf-8")
+        r = payload["result"]
+        print(f"  sampled dates : {r['sampled_dates']}")
+        print(f"  mean Spearman : {r['mean_spearman_yield_vs_percentile_shift']} "
+              f"(min {r['min']}, max {r['max']})")
+        print(f"  report        : {out}")
+        return 0
+
+    if args.time_batch:
+        n = args.time_batch
+        started = time.perf_counter()
+        result = run_screen(series, cutoff, limit_dates=n, verbose=False)
+        elapsed = time.perf_counter() - started
+        total_dates = len(series[BENCHMARK][0])
+        per_date = elapsed / max(result["evaluated_dates"], 1)
+        print(f"\ntimed {result['evaluated_dates']} evaluated dates in {elapsed:.1f}s "
+              f"({per_date:.2f}s/date)")
+        print(f"projected full run over {total_dates} sessions: "
+              f"{per_date * total_dates / 60:.1f} min")
+        return 0
+
+    started = time.perf_counter()
+    result = run_screen(series, cutoff)
+    result["elapsed_seconds"] = round(time.perf_counter() - started, 1)
+
+    con = open_cache(cache)
+    coverage = {
+        r[0]: {"split_bars": r[1], "all_bars": r[2], "aligned_bars": r[3], "dropped_unmatched": r[4]}
+        for r in con.execute("SELECT symbol, split_bars, all_bars, aligned_bars, dropped_unmatched FROM fetched")
+    }
+    con.close()
+
+    verdict = (
+        "stop - no admission or ordering difference in this window"
+        if result["dates_with_different_ordered_list"] == 0
+        else "proceed to Step 3 - ordered lists differ"
+    )
+    payload = {
+        "generated_at": datetime.now(timezone.utc).isoformat(),
+        "snapshot": str(snapshot),
+        "cache": str(cache),
+        "window": {"start": window_start.isoformat(), "end": window_end.isoformat()},
+        "universe_size": len(symbols),
+        "inert_snapshot_symbols": inert,
+        "cutoff": cutoff,
+        "verdict": verdict,
+        "result": result,
+        "coverage": {
+            "symbols_cached": len(coverage),
+            "symbols_with_unmatched_dates": sum(1 for c in coverage.values() if c["dropped_unmatched"]),
+            "total_unmatched_dates": sum(c["dropped_unmatched"] for c in coverage.values()),
+        },
+    }
+    out = Path(args.out)
+    out.parent.mkdir(parents=True, exist_ok=True)
+    out.write_text(json.dumps(payload, indent=2) + "\n", encoding="utf-8")
+
+    print(f"\nevaluated dates                  : {result['evaluated_dates']}")
+    print(f"dates with different admitted set: {result['dates_with_different_admitted_set']}")
+    print(f"dates with different ordered list: {result['dates_with_different_ordered_list']}")
+    print(f"mean / max set disagreement      : {result['mean_set_disagreement']:.4f} / {result['max_set_disagreement']:.4f}")
+    print(f"verdict                          : {verdict}")
+    print(f"report                           : {out}")
+    return 0
+
+
+def parse_args() -> argparse.Namespace:
+    p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+    p.add_argument("--snapshot", default=str(DEFAULT_SNAPSHOT), help="Universe and date bounds come from this snapshot.")
+    p.add_argument("--cache", default=str(DEFAULT_CACHE))
+    p.add_argument("--fetch", action="store_true", help="Populate the cache and exit.")
+    p.add_argument("--time-batch", type=int, default=None, metavar="N",
+                   help="Time the screen over the last N sessions and project the full runtime.")
+    p.add_argument("--mechanism", type=int, default=None, metavar="EVERY_N",
+                   help="Correlate trailing dividend yield against the percentile shift on every Nth session.")
+    p.add_argument("--out", default="reports/tr-rank-agreement.json")
+    p.add_argument("--mechanism-out", default="reports/tr-rank-agreement-mechanism.json")
+    return p.parse_args()
+
+
+if __name__ == "__main__":
+    raise SystemExit(asyncio.run(main(parse_args())))

scripts/run_total_return_ab.py

@@ -0,0 +1,544 @@
+"""Paired portfolio replay: does the momentum basis change actual trades?
+
+Step 3a of docs/data-integrity-plan.md. The Step 2 screen showed the ranking
+changes every day; this measures whether that reaches admissions and P&L. No
+schema, no ingestion — bars come from the survivorship-a snapshot and the two
+adjusted series from the Step 2 cache.
+
+What is held identical between the arms
+---------------------------------------
+* **Setup candidates are generated once** and reused by both books. Entry, stop,
+  target and R:R never differ. Only momentum and what depends on it —
+  the residual/activation percentiles, `qualified`, and the 80/20 blend used for
+  ordering — are recomputed per arm.
+* **Both arms read the same cache**: control uses SPLIT stock against SPLIT SPY,
+  treatment uses ALL stock against ALL SPY. Neither reads a freshly fetched
+  series, so a refreshed-vs-snapshot difference cannot be mistaken for a basis
+  difference.
+* **Execution, volatility, gates, capacity, exits**: snapshot nominal bars in both
+  arms; the 6-month volatility leg comes from candidate generation and is never
+  recomputed; non-momentum floors, `max_positions`, risk per trade and the exit
+  policy are the production values for both.
+
+Each book then keeps its **own** holdings: a divergent admission changes which
+slots are occupied later, and that path dependence is part of the measurement.
+
+Reading the output
+------------------
+Different trades establish that the basis is production-relevant. They do not
+authorise a rollout — that is a separate non-inferiority decision taken on the
+paired numbers.
+
+Usage::
+
+    python scripts/run_total_return_ab.py --build-candidates   # one-time, cached
+    python scripts/run_total_return_ab.py
+"""
+
+from __future__ import annotations
+
+import argparse
+import asyncio
+import json
+import pickle
+import sqlite3
+import sys
+import time
+from concurrent.futures import ProcessPoolExecutor
+from datetime import date, datetime, timezone
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[1]
+if str(ROOT) not in sys.path:
+    sys.path.insert(0, str(ROOT))
+
+DEFAULT_SNAPSHOT = ROOT / "backtest_snapshots" / "survivorship-a.sqlite"
+DEFAULT_CACHE = ROOT / "backtest_snapshots" / "tr-rank-cache.sqlite"
+DEFAULT_CANDIDATES = ROOT / "backtest_snapshots" / "tr-ab-candidates.pkl"
+BENCHMARK = "SPY"
+
+
+def _sqlite_url(path: Path) -> str:
+    return f"sqlite+aiosqlite:///{path.resolve().as_posix()}"
+
+
+# ---------------------------------------------------------------------------
+# candidates — generated once, reused by both arms
+# ---------------------------------------------------------------------------
+
+async def build_candidates(snapshot: Path, out: Path, cadence: str, workers: int) -> None:
+    from sqlalchemy import select
+    from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker, create_async_engine
+
+    from app.models.ticker import Ticker
+    from app.services import backtest_service as bt
+    from app.services.admin_service import get_activation_config
+    from app.services.recommendation_service import get_recommendation_config
+
+    engine = create_async_engine(_sqlite_url(snapshot), pool_pre_ping=True)
+    Session = async_sessionmaker(engine, class_=AsyncSession, expire_on_commit=False)
+
+    candidates: list[dict] = []
+    prices: dict[str, tuple] = {}
+    async with Session() as db:
+        config = await get_recommendation_config(db)
+        activation = await get_activation_config(db)
+        symbols = list((await db.execute(select(Ticker.symbol).order_by(Ticker.symbol))).scalars())
+        benchmark = await bt._load_benchmark_closes_for_backtest(db, refresh=False)
+        print(f"generating candidates for {len(symbols)} symbols (cadence={cadence}) …")
+
+        columns_by_symbol: dict[str, tuple] = {}
+        for symbol in symbols:
+            cols = await bt._fetch_columns(db, symbol)
+            if cols is not None:
+                columns_by_symbol[symbol] = cols
+    await engine.dispose()
+
+    started = time.perf_counter()
+    ctx = bt._mp_context()
+    pool = None
+    if ctx is not None and workers > 1:
+        try:
+            pool = ProcessPoolExecutor(max_workers=workers, mp_context=ctx)
+        except Exception:  # noqa: BLE001 - fall back to in-process replay
+            pool = None
+
+    items = list(columns_by_symbol.items())
+    if pool is not None:
+        with pool:
+            futures = {
+                symbol: pool.submit(
+                    bt._replay_and_signals, symbol, cols, config, activation,
+                    benchmark, bt.PRODUCTION_GTL_TARGET_MODEL, cadence, False, None,
+                )
+                for symbol, cols in items
+            }
+            for index, (symbol, future) in enumerate(futures.items(), 1):
+                got, _ = future.result()
+                candidates.extend(got)
+                prices[symbol] = columns_by_symbol[symbol]
+                if index % 100 == 0:
+                    print(f"  … {index}/{len(items)}")
+    else:
+        for index, (symbol, cols) in enumerate(items, 1):
+            got, _ = bt._replay_and_signals(
+                symbol, cols, config, activation, benchmark,
+                bt.PRODUCTION_GTL_TARGET_MODEL, cadence, False, None,
+            )
+            candidates.extend(got)
+            prices[symbol] = cols
+            if index % 100 == 0:
+                print(f"  … {index}/{len(items)}")
+
+    payload = {
+        "generated_at": datetime.now(timezone.utc).isoformat(),
+        "snapshot": str(snapshot),
+        "cadence": cadence,
+        "config": config,
+        "activation": activation,
+        "candidates": candidates,
+        "prices": prices,
+        "benchmark": benchmark,
+    }
+    out.write_bytes(pickle.dumps(payload))
+    print(f"{len(candidates)} candidates over {len(prices)} symbols "
+          f"in {time.perf_counter() - started:.0f}s -> {out.name}")
+
+
+# ---------------------------------------------------------------------------
+# momentum, recomputed per arm from the cache
+# ---------------------------------------------------------------------------
+
+def load_cache_series(cache: Path) -> dict[str, tuple[list[date], list[float], list[float]]]:
+    con = sqlite3.connect(cache)
+    try:
+        rows = con.execute("SELECT symbol, date, split_close, all_close FROM bars ORDER BY symbol, date")
+        out: dict[str, tuple[list, list, list]] = {}
+        for symbol, day, split_close, all_close in rows:
+            dates, splits, alls = out.setdefault(symbol, ([], [], []))
+            dates.append(date.fromisoformat(day))
+            splits.append(float(split_close))
+            alls.append(float(all_close))
+    finally:
+        con.close()
+    return out
+
+
+def momentum_by_arm(
+    series: dict[str, tuple[list[date], list[float], list[float]]],
+    signal_dates: dict[str, set[date]],
+    *,
+    total_return: bool,
+) -> dict[tuple[str, date], float]:
+    """Residual 12-1 momentum at every (symbol, signal date) for one arm.
+
+    Control reads the SPLIT series against SPLIT SPY; treatment reads ALL against
+    ALL SPY. Both use the unmodified production function and only bars at or
+    before the signal date.
+    """
+    from app.services.momentum_service import _MOM_LOOKBACK, compute_residual_12_1_momentum
+
+    bench_dates, bench_split, bench_all = series[BENCHMARK]
+    benchmark = dict(zip(bench_dates, bench_all if total_return else bench_split))
+
+    out: dict[tuple[str, date], float] = {}
+    for symbol, wanted in signal_dates.items():
+        entry = series.get(symbol)
+        if entry is None:
+            continue
+        dates, splits, alls = entry
+        closes = alls if total_return else splits
+        index_of = {d: i for i, d in enumerate(dates)}
+        for as_of in wanted:
+            idx = index_of.get(as_of)
+            if idx is None or idx < _MOM_LOOKBACK:
+                continue
+            value = compute_residual_12_1_momentum(
+                dates[: idx + 1], closes[: idx + 1], benchmark
+            )
+            if value is not None:
+                out[(symbol, as_of)] = value
+    return out
+
+
+def apply_arm(candidates: list[dict], momentum: dict[tuple[str, date], float], min_pct: float) -> list[dict]:
+    """Rebuild only the momentum-dependent fields, using the production pipeline."""
+    from app.services import backtest_service as bt
+
+    rebuilt = []
+    for cand in candidates:
+        clone = dict(cand)
+        key = (clone["symbol"], date.fromisoformat(clone["date"]))
+        clone["residual_momentum"] = momentum.get(key)
+        rebuilt.append(clone)
+
+    bt._assign_momentum_percentiles(rebuilt)
+    bt._assign_residual_momentum_percentiles(rebuilt)
+    bt._assign_low_volatility_percentiles(rebuilt)   # vol_6m untouched, so identical in both arms
+    bt._assign_activation_momentum_percentiles(rebuilt)
+    bt._assign_residual_low_vol_blend(rebuilt)
+    bt._assign_residual_high_vol_blend(rebuilt)
+    for cand in rebuilt:
+        cand["qualified"] = bt._momentum_qualifies(cand, min_pct)
+    return rebuilt
+
+
+def simulate(candidates: list[dict], prices: dict, benchmark: dict, cadence: str) -> dict:
+    """Run the production book: live gate, 80/20 ordering, ATR trail, gate reset."""
+    from app.services import backtest_service as bt
+
+    strategy = next(s for s in bt.PORTFOLIO_MONITOR_STRATEGIES if s.get("is_production"))
+    entry_cfg = bt._entry_variant_config(str(strategy["entry_variant"]))
+    ranking_key = str(strategy.get("ranking_key") or entry_cfg.get("ranking_key"))
+
+    reentry_fn = bt._make_gate_reset_reentry_fn(
+        candidates, prices, cadence=cadence, qualified_fn=None, ranking_key=ranking_key,
+    )
+    return bt._simulate_portfolio(
+        candidates,
+        prices,
+        benchmark,
+        str(strategy["exit_policy"]),
+        max(bt.TIME_EXIT_DAYS),
+        qualified_fn=None,                       # live qualification, as production
+        ranking_key=ranking_key,
+        max_positions=int(entry_cfg["max_positions"]),
+        risk_per_trade=float(entry_cfg["risk_per_trade"]),
+        atr_trail_multiplier=bt.ATR_TRAIL_MULTIPLIER,
+        post_stop_reentry_fn=reentry_fn,
+        include_curve=True,
+        include_trades=True,
+    )
+
+
+# ---------------------------------------------------------------------------
+# comparison
+# ---------------------------------------------------------------------------
+
+def entry_diff(control: dict, treatment: dict) -> dict:
+    def entries(sim: dict) -> set[tuple[str, str]]:
+        return {(t["entry_date"], t["symbol"]) for t in sim.get("trade_details", [])}
+
+    a, b = entries(control), entries(treatment)
+    only_a = sorted(a - b)
+    only_b = sorted(b - a)
+    return {
+        "shared_entries": len(a & b),
+        "only_control": [{"entry_date": d, "symbol": s} for d, s in only_a],
+        "only_treatment": [{"entry_date": d, "symbol": s} for d, s in only_b],
+        "only_control_count": len(only_a),
+        "only_treatment_count": len(only_b),
+    }
+
+
+def yearly_comparison(control: dict, treatment: dict) -> dict:
+    ca = {row["year"]: row for row in (control.get("yearly_returns") or [])}
+    tb = {row["year"]: row for row in (treatment.get("yearly_returns") or [])}
+    years = sorted(set(ca) | set(tb))
+    rows = []
+    better = worse = same = 0
+    for year in years:
+        c = (ca.get(year) or {}).get("return_pct")
+        t = (tb.get(year) or {}).get("return_pct")
+        delta = None
+        if c is not None and t is not None:
+            delta = round(t - c, 2)
+            if delta > 0:
+                better += 1
+            elif delta < 0:
+                worse += 1
+            else:
+                same += 1
+        rows.append({"year": year, "control_pct": c, "treatment_pct": t, "delta_pp": delta})
+    return {
+        "rows": rows,
+        "years_treatment_better": better,
+        "years_treatment_worse": worse,
+        "years_equal": same,
+    }
+
+
+HEADLINE = ("trades", "total_return_pct", "cagr_pct", "max_drawdown_pct", "sharpe", "win_rate")
+
+# Pre-specified before the diagnostic was run, and reported for BOTH block
+# lengths rather than whichever reads better.
+BOOTSTRAP_SEED = 20260907
+BOOTSTRAP_REPLICATES = 5000
+BOOTSTRAP_BLOCKS = (21, 63)
+TRADING_DAYS = 252
+
+
+def daily_returns(curve: list[dict]) -> tuple[list[str], list[float]]:
+    """Daily returns from an equity curve, with their dates."""
+    dates, rets = [], []
+    for previous, current in zip(curve, curve[1:]):
+        prior = float(previous["equity"])
+        if prior > 0:
+            dates.append(current["date"])
+            rets.append(float(current["equity"]) / prior - 1.0)
+    return dates, rets
+
+
+def _path_stats(returns) -> tuple[float, float, float]:
+    """(CAGR %, Sharpe, max drawdown %) for one resampled return path."""
+    import math
+
+    n = len(returns)
+    equity = 1.0
+    peak = 1.0
+    max_dd = 0.0
+    total = 0.0
+    total_sq = 0.0
+    for r in returns:
+        equity *= 1.0 + r
+        peak = max(peak, equity)
+        if peak > 0:
+            max_dd = max(max_dd, 1.0 - equity / peak)
+        total += r
+        total_sq += r * r
+    mean = total / n
+    var = max(total_sq / n - mean * mean, 0.0)
+    sd = math.sqrt(var)
+    cagr = (equity ** (TRADING_DAYS / n) - 1.0) * 100.0 if equity > 0 else -100.0
+    sharpe = (mean / sd * math.sqrt(TRADING_DAYS)) if sd > 0 else 0.0
+    return cagr, sharpe, max_dd * 100.0
+
+
+def paired_block_bootstrap(
+    control_returns: list[float],
+    treatment_returns: list[float],
+    *,
+    block: int,
+    replicates: int,
+    seed: int,
+) -> dict:
+    """Resample identical date blocks for both arms and difference the results.
+
+    The arms share dates and most positions, so their difference is what needs an
+    interval — not each arm separately. Drawing the *same* block indices for both
+    preserves the pairing and whatever cross-arm correlation exists on those days.
+    Moving blocks retain some of the serial dependence that an i.i.d. resample
+    would destroy; they do not make this out-of-sample.
+    """
+    import random
+    import statistics
+
+    rng = random.Random(seed)
+    n = len(control_returns)
+    n_blocks = -(-n // block)          # ceil
+    max_start = n - block
+
+    deltas = {"cagr_pct": [], "sharpe": [], "max_drawdown_pct": []}
+    for _ in range(replicates):
+        starts = [rng.randint(0, max_start) for _ in range(n_blocks)]
+        idx = [i for s in starts for i in range(s, s + block)][:n]
+        c = _path_stats([control_returns[i] for i in idx])
+        t = _path_stats([treatment_returns[i] for i in idx])
+        deltas["cagr_pct"].append(t[0] - c[0])
+        deltas["sharpe"].append(t[1] - c[1])
+        deltas["max_drawdown_pct"].append(t[2] - c[2])
+
+    def interval(values: list[float]) -> dict:
+        ordered = sorted(values)
+        def q(p: float) -> float:
+            return ordered[min(len(ordered) - 1, max(0, int(round(p * (len(ordered) - 1)))))]
+        return {
+            "mean": round(statistics.fmean(ordered), 3),
+            "median": round(q(0.50), 3),
+            "ci_2.5": round(q(0.025), 3),
+            "ci_97.5": round(q(0.975), 3),
+            "share_below_zero": round(sum(1 for v in ordered if v < 0) / len(ordered), 4),
+        }
+
+    return {metric: interval(values) for metric, values in deltas.items()}
+
+
+async def main(args: argparse.Namespace) -> int:
+    snapshot = Path(args.snapshot)
+    cache = Path(args.cache)
+    store = Path(args.candidates)
+
+    if args.build_candidates or not store.exists():
+        if not args.build_candidates:
+            print(f"{store.name} missing — building it first")
+        await build_candidates(snapshot, store, args.cadence, args.workers)
+        if args.build_candidates:
+            return 0
+
+    payload = pickle.loads(store.read_bytes())
+    candidates = payload["candidates"]
+    prices = payload["prices"]
+    benchmark = payload["benchmark"]
+    min_pct = float(payload["activation"].get("min_momentum_percentile", 80.0))
+    cadence = payload["cadence"]
+
+    signal_dates: dict[str, set[date]] = {}
+    for cand in candidates:
+        signal_dates.setdefault(cand["symbol"], set()).add(date.fromisoformat(cand["date"]))
+
+    series = load_cache_series(cache)
+    missing = sorted({s for s in signal_dates if s not in series})
+    if missing:
+        raise SystemExit(
+            f"{len(missing)} symbols have candidates but no cached series "
+            f"({', '.join(missing[:8])}). Run research_tr_rank_agreement.py --fetch."
+        )
+
+    print(f"candidates: {len(candidates)} over {len(signal_dates)} symbols, cutoff {min_pct}")
+    results = {}
+    for label, total_return in (("control_price_return", False), ("treatment_total_return", True)):
+        started = time.perf_counter()
+        momentum = momentum_by_arm(series, signal_dates, total_return=total_return)
+        arm = apply_arm(candidates, momentum, min_pct)
+        sim = simulate(arm, prices, benchmark, cadence)
+        results[label] = sim
+        print(f"  {label:<24} trades={sim['trades']:<5} CAGR={sim['cagr_pct']}% "
+              f"Sharpe={sim['sharpe']} DD={sim['max_drawdown_pct']}%  "
+              f"({time.perf_counter() - started:.0f}s)")
+
+    control = results["control_price_return"]
+    treatment = results["treatment_total_return"]
+    diff = entry_diff(control, treatment)
+    yearly = yearly_comparison(control, treatment)
+
+    # Paired uncertainty on the DIFFERENCE. Per-arm standard errors describe each
+    # book alone; these arms share dates and most positions, so only a paired
+    # resample speaks to the gap between them.
+    control_dates, control_rets = daily_returns(control.get("equity_curve") or [])
+    treatment_dates, treatment_rets = daily_returns(treatment.get("equity_curve") or [])
+    if control_dates != treatment_dates:
+        raise SystemExit(
+            "Arm return streams are not date-aligned; a paired bootstrap would "
+            "resample mismatched days."
+        )
+    bootstrap = {
+        "method": (
+            "paired moving-block bootstrap on date-aligned daily returns; the same "
+            "block indices are drawn for both arms so the pairing and cross-arm "
+            "correlation are preserved"
+        ),
+        "seed": BOOTSTRAP_SEED,
+        "replicates": BOOTSTRAP_REPLICATES,
+        "sessions": len(control_rets),
+        "pre_specified": (
+            "seed, replicate count and both block lengths were fixed before the run; "
+            "both are reported"
+        ),
+        "scope": (
+            "exploratory uncertainty estimate from the observed window - not "
+            "out-of-sample validation and not a rollout gate"
+        ),
+        "blocks": {},
+    }
+    for block in BOOTSTRAP_BLOCKS:
+        started = time.perf_counter()
+        bootstrap["blocks"][f"{block}_sessions"] = paired_block_bootstrap(
+            control_rets, treatment_rets,
+            block=block, replicates=BOOTSTRAP_REPLICATES, seed=BOOTSTRAP_SEED,
+        )
+        print(f"  bootstrap block={block:<3} {BOOTSTRAP_REPLICATES} reps "
+              f"({time.perf_counter() - started:.0f}s)")
+
+    out = Path(args.out)
+    out.parent.mkdir(parents=True, exist_ok=True)
+    out.write_text(json.dumps({
+        "generated_at": datetime.now(timezone.utc).isoformat(),
+        "snapshot": str(snapshot),
+        "cache": str(cache),
+        "candidates_store": str(store),
+        "cadence": cadence,
+        "min_momentum_percentile": min_pct,
+        "candidate_count": len(candidates),
+        "held_fixed": [
+            "setup candidates (entry/stop/target/rr) generated once and shared",
+            "execution bars: snapshot nominal closes in both arms",
+            "volatility leg: vol_6m from candidate generation, never recomputed",
+            "non-momentum gates, max_positions, risk_per_trade, exit policy, re-entry policy",
+        ],
+        "headline": {
+            label: {k: sim.get(k) for k in HEADLINE} for label, sim in results.items()
+        },
+        "deltas": {
+            k: (
+                round(treatment[k] - control[k], 2)
+                if isinstance(treatment.get(k), (int, float)) and isinstance(control.get(k), (int, float))
+                else None
+            )
+            for k in HEADLINE
+        },
+        "entry_diff": diff,
+        "yearly": yearly,
+        "paired_bootstrap": bootstrap,
+        "arms": {label: {k: v for k, v in sim.items() if k not in {"equity_curve", "benchmark_curve"}} for label, sim in results.items()},
+    }, indent=2, default=str) + "\n", encoding="utf-8")
+
+    print(f"\nentries only in control  : {diff['only_control_count']}")
+    print(f"entries only in treatment: {diff['only_treatment_count']}")
+    print(f"shared entries           : {diff['shared_entries']}")
+    print(f"years treatment better   : {yearly['years_treatment_better']} "
+          f"/ worse {yearly['years_treatment_worse']} / equal {yearly['years_equal']}")
+    print("\npaired block bootstrap, treatment - control (95% interval):")
+    for name, res in bootstrap["blocks"].items():
+        print(f"  block {name}")
+        for metric, stats in res.items():
+            print(f"    {metric:<20} mean {stats['mean']:>8}  "
+                  f"[{stats['ci_2.5']:>8}, {stats['ci_97.5']:>8}]  "
+                  f"P(<0)={stats['share_below_zero']}")
+    print(f"report                   : {out}")
+    return 0
+
+
+def parse_args() -> argparse.Namespace:
+    p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+    p.add_argument("--snapshot", default=str(DEFAULT_SNAPSHOT))
+    p.add_argument("--cache", default=str(DEFAULT_CACHE))
+    p.add_argument("--candidates", default=str(DEFAULT_CANDIDATES))
+    p.add_argument("--cadence", default="weekly", choices=("weekly", "daily"))
+    p.add_argument("--workers", type=int, default=6)
+    p.add_argument("--build-candidates", action="store_true", help="Build the candidate store and exit.")
+    p.add_argument("--out", default="reports/tr-total-return-ab.json")
+    return p.parse_args()
+
+
+if __name__ == "__main__":
+    raise SystemExit(asyncio.run(main(parse_args())))

tests/unit/test_total_return_ab.py

@@ -0,0 +1,186 @@
+"""Guards for the paired total-return A/B replay.
+
+Two things have to hold before any difference the A/B reports means anything:
+
+1. **Identical inputs must produce identical books.** If the harness were
+   nondeterministic — dict ordering, tie-breaks, path-dependent holdings — a
+   difference between the arms could come from the replay rather than the
+   momentum basis. Feeding both arms the same series must give byte-identical
+   results.
+2. **A real momentum difference must be able to change an admission.** A harness
+   that holds everything so tightly it can never diverge would report agreement
+   no matter what. One name pushed across the cutoff must actually change who
+   gets admitted.
+"""
+
+from __future__ import annotations
+
+import importlib.util
+from datetime import date, timedelta
+from pathlib import Path
+
+import pytest
+
+# Imported at module scope like the other backtest tests: loading the service
+# lazily inside the first test leaves conftest's in-memory engine without its
+# schema for every test after it.
+from app.services import backtest_service as _bt  # noqa: F401
+
+ROOT = Path(__file__).resolve().parents[2]
+_spec = importlib.util.spec_from_file_location(
+    "run_total_return_ab", ROOT / "scripts" / "run_total_return_ab.py"
+)
+ab = importlib.util.module_from_spec(_spec)
+_spec.loader.exec_module(ab)
+
+
+def _sessions(n: int, start: date = date(2021, 1, 4)) -> list[date]:
+    days, day = [], start
+    while len(days) < n:
+        if day.weekday() < 5:
+            days.append(day)
+        day += timedelta(days=1)
+    return days
+
+
+def _series(n: int = 400, drift: float = 0.30) -> dict:
+    """A cache-shaped series set: three names plus SPY, all with clean history."""
+    days = _sessions(n)
+    out = {}
+    for i, symbol in enumerate(("AAA", "BBB", "CCC")):
+        splits = [100.0 * (1.0 + (drift + i * 0.10) * k / n) for k in range(n)]
+        out[symbol] = (days, splits, list(splits))  # ALL == SPLIT: no dividends
+    spy = [100.0 * (1.0 + 0.10 * k / n) for k in range(n)]
+    out[ab.BENCHMARK] = (days, spy, list(spy))
+    return out
+
+
+def _candidates(days: list[date], signal_index: int) -> list[dict]:
+    """One qualifying long setup per symbol on a single signal date."""
+    as_of = days[signal_index]
+    out = []
+    for symbol in ("AAA", "BBB", "CCC"):
+        out.append({
+            "symbol": symbol,
+            "date": as_of.isoformat(),
+            # Real candidates carry both; the percentile pass groups on them.
+            "ranking_period": as_of.isocalendar()[:2],
+            "iso_week": as_of.isocalendar()[:2],
+            "direction": "long",
+            "entry": 100.0,
+            "stop": 95.0,
+            "target": 110.0,
+            "rr": 2.0,
+            "vol_6m": 0.02,
+            "residual_momentum": 0.0,
+            "momentum": 0.0,
+            "meets_core": True,
+            "qualified": True,
+        })
+    return out
+
+
+def test_identical_series_give_identical_momentum_and_ordering():
+    """ALL == SPLIT everywhere, so the two arms must agree exactly — any
+    difference the real run reports is then attributable to the basis."""
+    series = _series()
+    days = series["AAA"][0]
+    signal_dates = {s: {days[-1]} for s in ("AAA", "BBB", "CCC")}
+
+    control = ab.momentum_by_arm(series, signal_dates, total_return=False)
+    treatment = ab.momentum_by_arm(series, signal_dates, total_return=True)
+
+    assert control == treatment
+    assert control, "fixture must produce momentum values, else the test is vacuous"
+
+    cands = _candidates(days, len(days) - 1)
+    a = ab.apply_arm(cands, control, 80.0)
+    b = ab.apply_arm(cands, treatment, 80.0)
+    assert a == b
+
+
+def test_entry_diff_is_empty_for_identical_books():
+    sim = {"trade_details": [
+        {"entry_date": "2022-01-04", "symbol": "AAA"},
+        {"entry_date": "2022-02-01", "symbol": "BBB"},
+    ]}
+    diff = ab.entry_diff(sim, dict(sim))
+    assert diff["only_control_count"] == 0
+    assert diff["only_treatment_count"] == 0
+    assert diff["shared_entries"] == 2
+
+
+def test_a_momentum_change_moves_a_name_across_the_cutoff():
+    """The harness must be able to see a real difference: lifting one name's
+    momentum above its peers changes which names clear the gate."""
+    series = _series()
+    days = series["AAA"][0]
+    cands = _candidates(days, len(days) - 1)
+    as_of = days[-1]
+
+    # CCC ranks top on the control momentum, AAA bottom.
+    control_mom = {("AAA", as_of): 0.10, ("BBB", as_of): 0.20, ("CCC", as_of): 0.30}
+    # Treatment reverses it — as a dividend-heavy name overtaking a growth name would.
+    treatment_mom = {("AAA", as_of): 0.30, ("BBB", as_of): 0.20, ("CCC", as_of): 0.10}
+
+    a = ab.apply_arm(cands, control_mom, 80.0)
+    b = ab.apply_arm(cands, treatment_mom, 80.0)
+
+    admitted_a = {c["symbol"] for c in a if c["qualified"]}
+    admitted_b = {c["symbol"] for c in b if c["qualified"]}
+
+    assert admitted_a == {"CCC"}
+    assert admitted_b == {"AAA"}
+    assert admitted_a != admitted_b
+
+
+def test_the_volatility_leg_is_untouched_by_the_arm():
+    """Only momentum may vary; vol_6m carries through both arms unchanged, so the
+    80/20 blend differs solely through its momentum leg."""
+    series = _series()
+    days = series["AAA"][0]
+    as_of = days[-1]
+    cands = _candidates(days, len(days) - 1)
+    for index, cand in enumerate(cands):
+        cand["vol_6m"] = 0.01 * (index + 1)
+
+    a = ab.apply_arm(cands, {("AAA", as_of): 0.1, ("BBB", as_of): 0.2, ("CCC", as_of): 0.3}, 80.0)
+    b = ab.apply_arm(cands, {("AAA", as_of): 0.3, ("BBB", as_of): 0.2, ("CCC", as_of): 0.1}, 80.0)
+
+    from app.services.backtest_service import VOL_PERCENTILE_KEY
+
+    assert [c[VOL_PERCENTILE_KEY] for c in a] == [c[VOL_PERCENTILE_KEY] for c in b]
+    assert [c["vol_6m"] for c in a] == [c["vol_6m"] for c in b]
+
+
+def test_missing_momentum_leaves_a_candidate_unqualified():
+    """A name with no computable momentum in an arm must not sneak through the
+    gate on a stale value carried over from candidate generation."""
+    series = _series()
+    days = series["AAA"][0]
+    as_of = days[-1]
+    cands = _candidates(days, len(days) - 1)
+
+    arm = ab.apply_arm(cands, {("AAA", as_of): 0.5}, 80.0)
+    by_symbol = {c["symbol"]: c for c in arm}
+
+    assert by_symbol["BBB"]["residual_momentum"] is None
+    assert by_symbol["CCC"]["residual_momentum"] is None
+
+
+def test_yearly_comparison_counts_years_treatment_wins():
+    control = {"yearly_returns": [
+        {"year": 2022, "return_pct": 10.0},
+        {"year": 2023, "return_pct": 20.0},
+        {"year": 2024, "return_pct": 5.0},
+    ]}
+    treatment = {"yearly_returns": [
+        {"year": 2022, "return_pct": 12.0},
+        {"year": 2023, "return_pct": 18.0},
+        {"year": 2024, "return_pct": 5.0},
+    ]}
+    out = ab.yearly_comparison(control, treatment)
+    assert out["years_treatment_better"] == 1
+    assert out["years_treatment_worse"] == 1
+    assert out["years_equal"] == 1
+    assert out["rows"][0]["delta_pp"] == pytest.approx(2.0)