dennisthiessen/signal-platform
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

research: survivorship sensitivity and total-return momentum investigation

Browse Source 
Three questions answered against snapshot copies, with no production change.

Step 1 — how sensitive is the backtest to the ranking universe? Restoring 85
S&P 500 constituents removed between 2021-06 and 2026-07 moves the same window
from 2.09 Sharpe / 51.6% CAGR to 1.57 / 38.1%. The failed banks never reached
either book (identical best/worst trade R), so this measures universe
recomposition, not survivorship bias. The removals list is curated by hand with
per-symbol sources; only three explicitly confirmed terminal events get marks,
and four unresolved endings are reported as caveats rather than inferred.

Step 2 — does a total-return basis change the ranking? Ordered lists differ on
1009 of 1009 evaluated dates, 4.56 replacements per date. The shift correlates
with trailing implied yield (mean Spearman +0.79), which is strong evidence
consistent with a dividend effect but does not isolate own-yield, since both
the stock and benchmark legs change.

Step 3a — does that reach P&L? A paired replay reusing one shared candidate set
(control reproduces the production row exactly) changes ~30% of entries.
Treatment is -3.7pp CAGR and -1.9pp drawdown, with the entire return gap in
2025. A paired moving-block bootstrap (seed 20260907, 5000 replicates, 21/63
blocks, both reported) puts every interval across zero.

Decision: production stays unchanged. No schema, no ingestion, no wider sweep.
The observed window does not resolve which basis is better, and choosing a
threshold now — with -3.7pp on the table — would be fitting the rule to the
answer.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HAH7vVtn7qbcn1EVheecHQ
This commit is contained in:
dennisthiessenandClaude Opus 5 committed 2026-09-07 19:29:34 +02:00
1 parent 8453b87290
commit ebe9047403
16 files changed
+442522
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/data-integrity-plan.md
+363
View File
@@ -0,0 +1,363 @@
# Data & signal integrity — implementation plan
Status: **closed, 2026-09-07.** Steps 1, 2 and 3a ran; Step 3b was not started
because 3a did not clear its bar. Findings are in
[`research/survivorship-audit.md`](research/survivorship-audit.md),
[`research/tr-rank-agreement.md`](research/tr-rank-agreement.md) and
[`research/tr-total-return-ab.md`](research/tr-total-return-ab.md). The follow-on
work this raised is in [`data-reliability.md`](data-reliability.md).
Two questions, answered in order, with the production work made conditional on
the answers.
## Objective
1. Measure how the backtest changes when removed index constituents are restored.
This is a survivorship sensitivity check, not a complete bias estimate.
2. Check whether total-return momentum changes the daily admission or ordering
of names. Build the production path only if this screening test finds changes.
**Guiding principle: KISS — measure before building.** Every earlier draft of this
plan started with a schema change (`adj_close`, then `tr_factor`), an ingestion
maintenance path, a snapshot migration and a benchmark column, in order to deliver
a signal change whose effect nobody has measured. Both questions above can be
investigated by research scripts against a snapshot copy. Step 1 needs one small,
explicit replay option to include entries immediately before a terminal event.
No production schema or ingestion changes are needed for either research step.
Step 3 is not started until Step 2 finds a ranking or admission difference.
Why the research scripts need none of that machinery: the whole forward-anchored
factor design exists because *production* ingest is incremental, so back-adjusted
prices drift between fetches. A research script fetches each ticker's whole history
in **one** request, and within a single response the adjustment is internally
consistent. No column, no anchor, no maintenance.
---
## Step 1 — How sensitive is the backtest to omitted constituents?
Arm A uses the universe already present in the base snapshot.
Arm B adds the names that were removed from the S&P 500 since the snapshot's start
— including the ones that failed — and lets the strategy trade them.
Use the same configuration, cadence and evaluation dates in both arms. This is
deliberately an expanded-universe experiment, without historical membership gating.
### Files
- **[NEW] `scripts/data/sp500_removals.json`** — flat list, hand-curated from
Wikipedia's S&P 500 changes table, checked into git:
`[{"symbol": "SIVB", "removed_on": "2023-03-15", "reason": "receivership"}, ...]`
Reasons: `receivership` | `bankruptcy` | `acquisition` | `market_cap`.
No additions, no rename map, no membership intervals — see *Not doing*.
- **[NEW] `scripts/prep_survivorship_snapshot.py`** — makes both arm files from a
base snapshot, leaving the base untouched:
1. Copy base → `survivorship-a.sqlite` and `survivorship-b.sqlite`.
2. **Schema sync** on both: add every column present in `Base.metadata` but
missing from the file, nullable. The snapshots on disk predate migrations 026
and 032, so `select(Ticker)` against them fails today with
`no such column: tickers.cik` — and `_get_ticker` (`price_service.py:22`) is a
full-entity select that `query_ohlcv` → `backtest_service._fetch_columns`
reaches. Derive the column list from the metadata, not a hardcoded list, so
migration 036 doesn't break it again.
3. Into arm B only: for each removed symbol with no bars, fetch daily bars from
Alpaca (it still serves delisted symbols — verified for SIVB, FRC, ATVI, TWTR)
and insert the ticker row. Restrict downloaded bars to the base snapshot's
date range and use the same price adjustment as its existing bars. Log fetched,
already-present and unavailable symbols; an unavailable failure must be called
out as missing coverage, not silently treated as having no strategy exposure.
`removed_on` is an index-removal date, not necessarily a delisting date.
**Nothing is inferred from missing bars**: `delisted_on` / `delisted_reason`
are written only where the dataset carries a confirmed terminal event. A
series that merely ends early leaves them NULL and is reported as an
unresolved *ending status* — as built, that is CMA, SEE, HBI and DISH.
Series are shortened only at a **reviewed cutoff** listed in the dataset
(SBNY 2023-03-10, INFO 2022-02-25). Elsewhere a run of ≥5 zero-volume
sessions is a tripwire, not a rule: the symbol is left out of the arm and
reported for review rather than silently truncated.
4. **Terminal mark**, arm B only: for a confirmed terminal event, append one
synthetic bar on the next SPY trading session after the last real bar, with
`o=h=l=c=0.01, volume=0`. A mark is written only when the retained final bar
equals the confirmation's reviewed `last_real_bar`; a disagreement writes no
mark and is reported. Write a sidecar JSON mapping symbol to terminal-bar
date for the runner.
The portfolio sim already fills a gap through the stop at the open
(`backtest_service.py` ~2280), so the position resolves at ~-100% with no
change to its stop-fill logic. Treat 0.01 as a near-total-loss assumption,
not an observed execution. For acquisitions, retain the last available close
as an approximation; this experiment does not model cash/stock consideration
or release of capital on the acquisition date.
This matters: failures halt mid-collapse (SIVB's last exchange bar is 106.04
on the way to zero), so without it arm B spares exactly the trades it exists
to measure. **Appending the bar alone is insufficient:** `_replay_ticker`
currently stops at `n - HORIZON`, where `HORIZON = 30`. That excludes signals
in the last 29 real bars even after the synthetic bar is added. Implement the
explicit replay option below before interpreting the audit. Only positions
still open at the halt should lose nearly everything; positions stopped or
timed out earlier should retain their normal exits.
`volume=0` is safe — `compute_volume_profile` guards its only divisor
(`indicator_service.py:298-300`).
- **[NEW] `docs/research/survivorship-audit.md`** + an entry in
`docs/research/README.md`.
### Small replay change required
- Add optional `--terminal-events <json>` to `run_backtest_snapshot.py`. Pass the
map explicitly through `run_backtest` and `_replay_and_signals` to
`_replay_ticker`, including the multiprocessing path. Default to no events.
Do not infer terminal events from low prices or zero volume.
- Validate the whole map **before dispatch** and fail the run on any mismatch.
`_replay_ticker` rejects a bad date, but both dispatch paths swallow per-ticker
exceptions so a long replay survives one unreadable name — which would turn a
stale events file into a quietly incomplete report. `_validate_terminal_events`
raises `TerminalEventError` before any work starts, so no report is written.
- For a mapped ticker, validate that the terminal date matches its final bar.
Keep the existing cadence and starting index, but allow signal dates through
the last real bar instead of stopping 30 bars early. Never generate a setup on
the synthetic bar. Unmapped tickers retain the existing replay bounds.
- Use the existing forward bars and outcome evaluation. For the long-only audit,
a setup with a positive stop resolves no later than the terminal bar. Do not
pad the history with 30 invented trading bars or turn unresolved shortened
histories into normal 30-day outcomes.
- Keep terminal prices out of later signal generation; their role is resolving
existing positions. Preserve normal stop, target and time exits before them.
### Focused validation
Add regression tests alongside the existing backtest tests:
1. A qualifying long setup five sessions before a terminal event is emitted with
the option enabled and exits at 0.01 if still held. Use a controlled setup
fixture so the test does not depend on whether SIVB happens to qualify.
2. A position that exits before the terminal event keeps that earlier exit.
3. No setup is emitted on the terminal bar; an ordinary ticker and a run without
the option retain their existing replay behavior.
Before the full runs, smoke-test the prepared files with `_fetch_columns` for one
existing and one restored ticker. Confirm matching base-symbol bars between arms,
matching configuration and date bounds, and no modification of the base file.
### Running it
```powershell
python scripts/prep_survivorship_snapshot.py --base backtest_snapshots/prod.sqlite
python scripts/run_backtest_snapshot.py backtest_snapshots/survivorship-a.sqlite --out reports/backtest-YYYYMMDD-survivorship-a.json
python scripts/run_backtest_snapshot.py backtest_snapshots/survivorship-b.sqlite --terminal-events backtest_snapshots/survivorship-b-terminal-events.json --out reports/backtest-YYYYMMDD-survivorship-b.json
```
No new runner: `run_backtest_snapshot.py` already runs the production sim offline
and reports CAGR, max drawdown, Sharpe and total return. Compare the same production
portfolio-monitor strategy and `all` lookback in both reports, not whichever
variant performs best. Record B minus A for these metrics, trade counts, evaluated
dates, restored-symbol coverage and terminal-event assumptions. A failed ticker
replay or missing production report row makes the run incomplete.
### What it does and doesn't measure
An **expanded-universe sensitivity analysis**, not a lower bound. It restores
omitted names but does not remove addition lookahead. Still-trading former members
remain eligible after removal: `run_backtest` does not gate on `delisted_on`, and
an index-removal date would not be the right delisting field anyway.
Changing the universe changes percentile ranks and competing admissions. Either
correction can raise or lower this strategy's realized returns; do not claim that
deletions are the larger source of bias or that the true drag must exceed B minus A.
A small delta does not establish that survivorship bias is small. Report it as
the sensitivity observed under these assumptions. A complete point-in-time audit
remains separate work, not a requirement for finishing this experiment.
### Base snapshot
- `prod.sqlite`: 2021-06-24 → 2026-07-02, 90MB, the ~505 production names.
- `research.sqlite`: 2016-01-04 → 2026-07-17, 678MB, 4,650 names.
Use `prod.sqlite`: the highest-signal events (SIVB, FRC, SBNY, ATVI — all
2023) are inside its window, the copies are cheap, and it has no completion
manifest to keep in sync. The pre-registration then states a 2021–2026 window, not
a ten-year one. Verify the actual file's bounds and record its hash before running.
Do not silently substitute `research.sqlite`: its much broader universe answers a
different comparison. Keep the first experiment on the production snapshot.
---
## Step 2 — Would total return change any decision?
Use a conservative ranking screen, not a portfolio simulation. Compare the full
ordered momentum-admitted list on every daily decision date. Do not call a top-15
list the traded set: production also applies setup qualification, current holdings,
available slots and other admission constraints. A difference in this screen is
a reason to investigate actual trades in Step 3, not proof of changed P&L.
### File
- **[NEW] `scripts/research_tr_rank_agreement.py`** — use arm A's universe and dates:
1. One `Adjustment.ALL` fetch and one `Adjustment.SPLIT` fetch per symbol, plus
SPY, over the full history. Single fetch each, so each series is internally
consistent. Align by date, never zip by index. Cache the responses locally
for repeatable runs. Require matching dates in both adjustments; report and
resolve missing downloads before applying the gate. Insufficient initial
history is expected and should be recorded, with equal eligibility in both
arms. Use only bars at or before each decision date.
2. At each SPY trading date with sufficient history, compute residual 12-1
momentum both ways with the existing
`momentum_service.compute_residual_12_1_momentum` (price-return series against
nominal SPY; total-return series against total-return SPY), percentile them
with `momentum_service._percentiles`, and take the admitted set at the
production `min_momentum_percentile`.
3. Compute `compute_realized_vol_6m` and its percentiles from the SPLIT series
once per date and hold that leg fixed in both arms. This isolates the change
to momentum. Order the entire admitted lists by `blend_strategy_rank`, with
symbol as a deterministic tie-breaker. Do not truncate to `max_positions`:
a lower-ranked name may be the first actionable setup when others fail gates
or are already held. Iterate symbols in the same sorted order in both arms
before calling `_percentiles` so tied inputs do not create artificial changes.
4. Save per-date results and an overall/per-year summary: number of evaluated
dates, fraction with different admitted sets, fraction with different ordered
lists, and mean and maximum set disagreement. Define set disagreement as
`len(A symmetric_difference B) / len(A union B)`, or zero for two empty sets.
Report Spearman correlation as a diagnostic only. Include the first few
changed dates with the changed names to make the result inspectable.
Validation: identical input series must produce no differences. A fixture where
the admitted members stay the same but two names swap order must register a
changed ordered list. A difference on only one date must still trigger the gate.
Write results to `docs/research/tr-rank-agreement.md` and index it in the research
README, including data coverage, parameters and the exact report/cache paths.
### Pre-registered decision rule
- **No ordered-list differences on any evaluated daily date → stop.** Record
"no admission or ordering differences observed in this window under the tested
configuration." This is historical evidence, not a guarantee about future
decisions. No schema or ingestion changes are needed.
- **Any ordered-list difference → Step 3.** This intentionally conservative rule
can flag names that would never actually trade. Step 3's paired portfolio replay
determines whether the difference reaches actual admissions and P&L.
- **Incomplete downloads or no evaluable dates → inconclusive.** Fix coverage
before deciding. Never turn missing data into agreement.
There is no median-based tolerance: a zero median can hide differences on nearly
half the dates. Checking daily full lists avoids adding a second portfolio engine
to this screening script while retaining changes below the nominal top 15.
---
## Step 3 — Production total-return ranking (conditional)
**Gate cleared 2026-09-07 — Step 3 is now live work.** The screen found ordered
admitted lists differing on 1,009 of 1,009 evaluated dates, with the shift
correlating +0.79 with trailing dividend yield
([results](research/tr-rank-agreement.md)). That is a ranking difference, not a
P&L difference; Step 3's job is to find out whether it reaches trades.
### Step 3a — Paired portfolio replay (do this first, no schema)
Different ranks are not different trades. Before any schema or ingestion work,
replay the production book twice and measure actual admissions and P&L. If the
books are identical, the basis cannot matter and Step 3b never starts.
**[NEW] `scripts/run_total_return_ab.py`**, sourcing everything from artifacts
that already exist — `backtest_snapshots/survivorship-a.sqlite` for bars and
`backtest_snapshots/tr-rank-cache.sqlite` for the two adjusted series.
1. **Generate setup candidates once.** One replay pass produces the candidate set;
both arms reuse it. Only momentum and what depends on it — the momentum
percentile, qualification against `min_momentum_percentile`, and
`blend_strategy_rank` ordering — are recomputed per arm. Setup geometry
(entry, stop, target, R:R) is generated once and never differs.
2. **Both arms read the same cache.** Control takes SPLIT stock series against
SPLIT SPY; treatment takes ALL stock against ALL SPY. Neither arm reads a
freshly fetched series, and neither mixes cache with snapshot closes for the
momentum input — a refreshed-vs-snapshot difference would confound the basis
difference being measured.
3. **Everything else is pinned.** Execution uses the snapshot's nominal bars in
both arms; the volatility leg, the non-momentum gates, capacity
(`max_positions`), risk per trade and the exit policy are identical. Each book
then carries its **own** resulting holdings — a divergent admission changes
which slots are occupied on later dates, and that path dependence is part of
what is being measured.
4. **Fixed up front:** universe (arm A's 505 symbols with bars), window
(2021-06-24 → 2026-07-02), cadence, and the production config as read from the
snapshot. Only bars at or before each signal date are used, in both arms.
5. **Saved to `reports/tr-total-return-ab.json`:** per-arm trade count, CAGR, max
drawdown, Sharpe, total return, win rate; annual returns for both arms and the
**count of years treatment beats control**; and the entry-level diff — dates
and symbols entered by one arm and not the other, in both directions.
6. **Tests** (`tests/unit/test_total_return_ab.py`): an identical-input parity
test — feeding the same series to both arms must produce byte-identical books,
so any observed difference is attributable to the basis and not to replay
nondeterminism; and a controlled case where a hand-built momentum difference
flips one name across the cutoff and the resulting admission actually changes,
proving the harness can see what it is looking for.
**Reading the result.** Different trades establish that the basis is
production-relevant. They do **not** authorize a rollout: that is the
non-inferiority decision below, taken on the paired numbers, and Step 3b starts
only after the paired results have been reported.
**Step 3a result (2026-09-07):** the basis reaches trades — ~30% of entries
differ. Which basis is better is unresolved: a paired block bootstrap on the
date-aligned daily return streams (5,000 replicates, 21- and 63-session blocks,
seed fixed beforehand, both reported) puts ΔCAGR at −3.6/−3.8pp with 95%
intervals spanning zero, ΔSharpe ≈ −0.11, and ΔMaxDD −1.0/−1.8pp in treatment's
favour. **Production stays unchanged; 3b not started.**
[Results](research/tr-total-return-ab.md).
### Step 3b — Production storage and ingestion (only if 3a shows real differences)
The design points already settled, worth carrying forward:
- `ohlcv_records.tr_factor`, `Float`, nullable, **no default and no server_default**
— a default of 1.0 makes "never backfilled" indistinguishable from "no dividends"
and would silently stamp a price-return ranking as total return.
- Forward-anchored: `f = 1.0` at `max(2016-01-04, ticker's first bar)`, compounded
as `f_t = f_{t-1} * (R_t / R_{t-1})` where `R = all_close / split_close`. The
ratio is anchor-invariant **only within one API response** — never persist `R` and
compare across days.
- `upsert_ohlcv` must not clobber a stored factor:
`coalesce(excluded.tr_factor, ohlcv_records.tr_factor)`. Daily ingest re-fetches
the boundary bars, so overlapping writes are routine.
- Daily ingest pulls a ~5-bar overlap under both adjustments to extend the factor;
a clean-ratio mismatch on the overlap bars means a split, which needs a
full-history refetch (the current incremental ingest never restates history).
- One basis for the whole cross-section, never mixed, stamped in the rank log.
- Beta leg: if the stock series goes total return, SPY must too, in the live path
and the backtest, or beta picks up SPY's ~1%/yr yield.
- A/B through the existing matrix on CAGR and max drawdown with per-year win/loss
counts. Reuse the same setup candidates, nominal execution bars, core gates,
volatility leg, capacity and exit policy; change only momentum and its dependent
admission/ranking fields. Replay each book with its own resulting holdings and
compare actual entry dates/symbols as well as P&L. Do not substitute top-15 lists
for the existing portfolio admission logic. Non-inferiority, not superiority:
total return is the definition 12-1
momentum is built on, so an in-sample shortfall shouldn't lock in the basis we
believe is measured wrong.
---
## Not doing
- **Chart total-return toggle.** The candlestick chart is an execution chart:
stops, S/R levels, entry and paper-trade markers are all nominal. Rescaling the
candles draws prices that never traded next to a stop that did.
- **`[D]` dividend markers.** Cosmetic, and they need `tr_factor` to exist. Revisit
after Step 3, if Step 3 ever happens.
- **Point-in-time addition gating** — interval map, reverse-chronological replay,
rename map and removal gating are deferred from this sensitivity experiment.
Their relative impact is unknown; a small Step 1 delta does not rule them out.
Note if a point-in-time audit is
ever done: the ranking cross-section must be filtered *before*
`_assign_*_percentiles` (`backtest_service.py:4198-4200`), not only at admission
in `_simulate_portfolio` (4257), or the percentile cutoff is still computed
against a non-PIT universe.
- **Prune protection and the `add_ticker` regex.** Only needed if delisted names
are written to the production database. The audit works entirely in snapshot
copies. (`bootstrap_universe(prune_missing=True)` cascading away a still-trading
ex-constituent's history is a real latent risk — a separate ticket, not this one.)
- **`benchmark_prices.tr_close`.** Only needed for Step 3. Note for then: store a
forward factor like the OHLCV side, not a back-adjusted close —
`refresh_benchmark_prices` writes a row only when `close` changed and is called
with variable spans, which would leave different anchors either side of the last
refreshed window.
- **Migrating the snapshots in place.** The prep script syncs its own copies.
`research.sqlite` stays byte-identical, so its completion manifest and the reports
that cite it stay valid.
docs/research/README.md
+10
View File
@@ -145,6 +145,8 @@ knobs.
| **`fip_id` / liquid breadth** | Fingerprint −0.045 / t −2.91; liquid unconditional **−0.017 / t −1.85** (not green); mom-conditional **−0.088 / t −4.58** | **Parked.** Orphan +0.0575 died (snapshot race). Breadth did not strengthen resid-mom t-stat. Optional reopen = pre-registered two-arm liquid-1500 book first. See [fip-breadth-ic.md](fip-breadth-ic.md) |
| **Broader universe** | Composition changes factor signs (fip tug-of-war); vol-tilt on breadth is only a **directional hypothesis** (auth. −0.048 / t −1.36) | Any prod broaden must re-validate 80/20 tilt; offline research only; research.sqlite requires completion manifest |
| **Forward paper-trade record** | The only true out-of-sample evidence the snapshot cannot give | Time; mark entries at actual near-close fill once ops ships |
| **Total-return momentum basis** | Ranks change every day (screen: **1,009 of 1,009** dates, ~4.6 names replaced per date, shift correlates **+0.79** with trailing yield) **and it reaches trades** — paired replay differs on ~30% of entries (51 control-only, 63 treatment-only, 271 shared). Which basis is better is unresolved: paired block bootstrap (5,000 reps, 21- and 63-session blocks, seed fixed beforehand) gives ΔCAGR −3.6/−3.8pp with 95% intervals spanning zero ([−14.8, +6.4] and [−14.7, +5.4]), ΔSharpe ≈ −0.11, ΔMaxDD −1.0/−1.8pp in treatment's favour; ~75% of replicates on the point-estimate side. All of the CAGR shortfall sits in 2025 | **Production unchanged.** Needs a threshold pre-registered before it is applied, plus out-of-sample evidence; a capacity/date/cadence matrix would be sensitivity checks, not independent samples. Schema/ingestion deferred. [Screen](tr-rank-agreement.md) · [Paired replay](tr-total-return-ab.md) · [plan](../data-integrity-plan.md) |
| **Universe sensitivity / point-in-time membership** | Restoring 85 removed constituents (2021-06→2026-07, prod snapshot) moves the production row from **Sharpe 2.09 / CAGR 51.6%** to **1.57 / 38.1%** — all of it universe recomposition; no failed-bank position reached either book (trade extremes identical, and a control shows the extra replay signals changed no portfolio decision). Says the ranks are highly sensitive to the universe, not that bias is 13.5pp | Not a bias estimate: addition lookahead is still unremoved and arm B is not point-in-time. A real PIT audit is separate work. [Survivorship sensitivity](survivorship-audit.md) |
| **Better target model for clear-air names** | The return is demonstrably there (#2 wins on raw CAGR in *both* train and test); it's the *flat* 3× ATR target that makes it too expensive in risk | Needs a per-name model, not a constant k×ATR |
| **Minimum effective-risk floor** | ⛔ CLOSED NEGATIVE, not run. The floor lifts EV/trade (+0.032) and PF (+0.073) *by deleting trades* — 11.4 fewer per path, never one more — and costs **−0.753pp CAGR**, −0.047 Sharpe, −0.051 Calmar | Do not run the A/B; its EV-based pass rule would have shipped it. [Withdrawn specification](effective-risk-floor-ab.md) / [findings](portfolio-capacity-bracket-findings.md#correction-2026-08-05-ev-per-trade-was-the-wrong-lens) |
@@ -176,6 +178,14 @@ knobs.
`research.sqlite`. Extender now writes a completion manifest; breadth mode
refuses without a match. Same class of protection as calendar-truncation
asserts — do not re-mythologize numbers computed on half a universe.
8. **A big delta is not the mechanism you assumed.** Restoring 85 removed index
constituents cost 13.5pp of CAGR — and none of it came from the failed banks
the experiment was built around. The driver was the universe the ranks are
computed over. Measure which part of a change did the work before naming the
effect, and state what the control actually isolates: ours compared replay
*bounds* on one snapshot, so it measured the extra signals, not the synthetic
bars those signals resolve against.
[Survivorship sensitivity](survivorship-audit.md)
---
docs/research/survivorship-audit.md
+173
View File
@@ -0,0 +1,173 @@
# Survivorship sensitivity — restoring removed S&P 500 constituents
**Run date:** 2026-09-07 · **Status:** measured, no strategy change proposed
**Reports:** `reports/backtest-20260907-survivorship-a.json`,
`reports/backtest-20260907-survivorship-b.json`,
`reports/backtest-20260907-survivorship-b-nomarks.json`
## Question
The backtest universe is the ~505 names tracked *today*. Every constituent that
left the index between 2021 and 2026 is missing from it, including the ones that
failed. How different does the production strategy look when they are put back?
This is an **expanded-universe sensitivity analysis**, not a bias estimate. It is
pre-registered as a measurement: there is no pass/fail rule and no promotion
attached to the outcome.
## Method
**Base:** `backtest_snapshots/prod.sqlite`, sha256 `7241f0ff0f6187a0…`, window
**2021-06-24 → 2026-07-02** (~5 years, not the full Alpaca history). Never
modified — both arms are copies, and the hash is re-checked after the build.
| Arm | Universe |
|---|---|
| **A** — control | 506 tickers, the snapshot as it stands |
| **B** — treatment | 591 tickers: the same 506 plus 85 restored removals |
Both arms ran the same binary, config, cadence (weekly) and date bounds, and are
compared on the same report row: `portfolio_monitor.runs` where
`is_production` is true and `lookback` is `all`
(`residual80_highvol80_20_atr3`).
Removals come from `scripts/data/sp500_removals.json` — 92 rows curated from
Wikipedia's *Historical components of the S&P 500* changes table, each carrying
the verbatim source text plus independently measured Alpaca evidence. Build with
`scripts/prep_survivorship_snapshot.py`.
Three data rules, all learned from the price data rather than assumed:
1. **Nothing is inferred from missing bars.** `delisted_on` / `delisted_reason`
are written only for confirmed terminal events. A series that merely stops
early leaves them NULL and is reported as an unresolved ending.
2. **Series are shortened only at a reviewed cutoff.** Alpaca does not end a
halted symbol's series — it repeats the last price at zero volume. SBNY sits
at 70.00 for 509 sessions after the FDIC closed it; INFO sits at 108.61 for
1,090 sessions, after which a *different company* reuses the ticker with no
date gap. Both have explicit reviewed cutoffs (2023-03-10, 2022-02-25).
Elsewhere a run of ≥5 zero-volume sessions is a tripwire: the symbol is left
out and reported, never silently cut.
3. **Terminal marks are explicit assumptions.** A synthetic `0.01` bar is written
on the next session after the last real bar for SIVB, SBNY and FRC only —
each with an FDIC source and only where the retained final bar matches the
reviewed `last_real_bar`. The FDIC sources establish the closure *event*; they
do not establish what holders recovered. The mark is a near-total-loss
assumption, not an observed closing-day execution.
**Replay change.** `_replay_ticker` normally stops `HORIZON = 30` bars before the
end so every setup has a full resolution window — which would hide every signal
in the final 29 real bars, exactly the ones a collapse resolves. A new
`--terminal-events` map opts the three marked symbols into walking through their
last real bar; no setup is ever generated on the synthetic bar, and unmapped
symbols are untouched. The map is validated against the snapshot's actual final
bars *before* any replay is dispatched (`_validate_terminal_events`), because both
dispatch paths swallow per-ticker exceptions and would otherwise turn a stale map
into a quietly incomplete report.
## Result
Source: `portfolio_monitor.runs` where `is_production` is true and `lookback` is
`all` (strategy `residual80_highvol80_20_atr3`), from
`reports/backtest-20260907-survivorship-a.json` and
`…-survivorship-b.json`. Both runs: weekly cadence, 6 workers,
`--allow-spawn`, gate cutoff 80.0, full period.
| Metric | Arm A | Arm B | B − A |
|---|---:|---:|---:|
| Total return % | 424.60 | 264.40 | **−160.20** |
| CAGR % | 51.60 | 38.10 | **−13.50** |
| Max drawdown % | 21.40 | 23.60 | +2.20 |
| Sharpe | 2.09 | 1.57 | **−0.52** |
| Trades | 322 | 346 | +24 |
| Win rate % | 37.0 | 35.0 | −2.0 |
| Final equity | 52,459 | 36,439 | −16,020 |
Context: 202,765 → 229,642 candidates, 1,086 → 1,315 qualified. Gate cutoff 80.0
in both arms.
## What actually caused it
**Not the failures.** `worst_trade_r` is **−3.26 in both arms**, and
`best_trade_r` is 12.87 in both. A position held into a terminal mark gaps to
$0.01 and fills at the open, which would land far below −3.26 — at candidate
level those trades score −3.13R (SIVB) and −5.99R (SBNY). No such trade appears
in either book, so none of the three failed banks reached the production
portfolio.
**What the control does and does not show.** Arm B replayed *without*
`--terminal-events` (`backtest-20260907-survivorship-b-nomarks.json`, same
snapshot, same config) produces a production row identical to arm B with them in
every field — 346 trades, 38.1% CAGR, 1.57 Sharpe, 23.6% drawdown, 264.4% total
return, −3.26 worst trade. Candidates rise 229,617 → 229,642, the 25 extra
signals the option exposes in the final 29 bars of the three symbols, while
`qualified` is 1,315 either way.
So the control establishes precisely this: **those 25 additional replay signals
changed no portfolio decision.** It does *not* independently establish that the
terminal marks had no effect — both runs read `survivorship-b.sqlite`, which
contains the synthetic bars either way, so a setup generated earlier could still
resolve against one in both runs. Isolating the marks themselves would need a
third snapshot built without them, which has not been run.
The separate evidence that no marked position reached the book is the trade
extremes: a position resolving against a $0.01 bar fills at that bar's open and
scores far worse than −3.26 (−3.13R for SIVB and −5.99R for SBNY at candidate
level), yet arm B's `worst_trade_r` equals arm A's exactly. That is inference
from the extremes, not a controlled measurement.
On that basis the −13.5pp CAGR is **universe recomposition**: 85 additional names
(+17% of the universe) shift the cross-sectional percentile ranks and compete for
the same 15 position slots, displacing trades arm A took. The strategy is
markedly sensitive to which universe it ranks over — which is the finding, and it
is a different claim from "survivorship bias costs 13.5pp".
## Caveats
- **This is not a bias estimate and not a lower bound.** Restoring names is only
one of the two corrections. Index-*addition* lookahead — today's members being
tradeable before they joined — is not removed. Either correction can move
returns in either direction, so B − A is the sensitivity observed under these
assumptions, nothing more. A small delta would not have shown the bias to be
small, and this large one does not establish its size either.
- **Arm B is not a point-in-time universe.** Still-trading former members remain
eligible after their removal date; `run_backtest` does not gate on
`delisted_on`, and an index-removal date would be the wrong field for it
anyway. A true PIT audit is separate work.
- **Four unresolved endings.** CMA (ends 2026-01-30), SEE (2026-04-08), HBI
(2025-11-28) and DISH (2023-12-29) stop trading before the window closes for
reasons the source row does not explain. Their bars are real and their
delisting fields are NULL; the ending status is simply unknown.
- **Three near-total-loss assumptions** (SIVB, SBNY, FRC), as described above.
The control shows they did not affect this result, but they remain assumptions
and would matter in any run where such a name did clear the gate.
- **Acquisitions are approximated** by the last available close. Cash or stock
consideration and the release of capital on the deal date are not modelled.
- **FBIN carries a corporate action.** Fortune Brands distributed MasterBrand on
2022-12-14 and began trading as FBIN on 2022-12-15. Alpaca's split adjustment
does not adjust for spin-offs, so the distribution appears as a real −6.91%
close-to-close move (63.22 → 61.90 → 57.62). A holder did not lose that value,
but any signal or stop crossing that date reads it as a decline. MBC enters the
dataset separately with its first bar the same day.
- **Window.** 2021-06 to 2026-07, because that is what `prod.sqlite` holds. The
2023 bank failures are inside it; the 2016–2021 period is not.
- **Snapshot schema.** The snapshot files on disk predate migrations 026 and 032,
so `select(Ticker)` against them fails with `no such column: tickers.cik` — the
path `backtest_service._fetch_columns` takes. Per-ticker exceptions are caught
and logged, so a run against an unsynced snapshot completes and writes a report
with **zero candidates** rather than failing. The prep script syncs both copies
against `Base.metadata`; anything else reading these snapshots needs the same.
## Reproduce
```powershell
python scripts/prep_survivorship_snapshot.py --base backtest_snapshots/prod.sqlite
python scripts/run_backtest_snapshot.py backtest_snapshots/survivorship-a.sqlite `
--out reports/backtest-YYYYMMDD-survivorship-a.json --allow-spawn --workers 6
python scripts/run_backtest_snapshot.py backtest_snapshots/survivorship-b.sqlite `
--terminal-events backtest_snapshots/survivorship-b-terminal-events.json `
--out reports/backtest-YYYYMMDD-survivorship-b.json --allow-spawn --workers 6
```
`survivorship-b-prep-manifest.json` records the base hash, per-symbol coverage,
the terminal events written and every unresolved item for the run.
docs/research/tr-rank-agreement.md
+118
View File
@@ -0,0 +1,118 @@
# Total-return vs price-return momentum — ranking screen
**Run date:** 2026-09-07 · **Verdict: proceed to Step 3** (pre-registered rule fired)
**Reports:** `reports/tr-rank-agreement.json`, `reports/tr-rank-agreement-mechanism.json`
**Script:** `scripts/research_tr_rank_agreement.py`
**Cache:** `backtest_snapshots/tr-rank-cache.sqlite` (507 symbols, 630,524 aligned bars)
## Question
12-1 momentum is defined on total returns; production computes it on
split-adjusted prices, which ignore dividends. Before building the production
`tr_factor` machinery (Step 3 of [the plan](../data-integrity-plan.md)), does the
basis actually change what the gate admits and how it orders?
Deliberately a screen, not a portfolio simulation. A difference here is a reason
to measure trades, **not** evidence that P&L changes.
## Configuration
| | |
|---|---|
| Universe | `backtest_snapshots/survivorship-a.sqlite`, 505 symbols with bars (RHM has a ticker row but no bars in the snapshot, so it is inert in both arms) |
| Window | 2021-06-24 → 2026-07-02, clipped to the snapshot |
| Decision dates | every SPY session; 1,009 had enough history to evaluate |
| Momentum | production `compute_residual_12_1_momentum`, unchanged; price series against nominal SPY, total-return series against total-return SPY |
| Volatility leg | production `compute_realized_vol_6m` on the SPLIT series, computed once per date and **held fixed in both arms** so only momentum varies |
| Gate | production `min_momentum_percentile` = 80.0, via production `_percentiles` |
| Ordering | production `blend_strategy_rank`, symbol as deterministic tie-break, symbols fed in sorted order in both arms |
| Comparison | the full admitted list, never truncated to `max_positions` |
| Runtime | 146 s with the production function — no NumPy rewrite needed, so no equivalence test was required |
**Coverage is complete.** One `Adjustment.SPLIT` and one `Adjustment.ALL` request
per symbol; 507 symbols cached, **zero** dates present in one adjustment but not
the other. No symbol with snapshot bars is missing from the cache — the script
refuses to run in that case rather than reporting the gap as agreement.
## Result
| | |
|---|---:|
| Evaluated dates | 1,009 |
| Dates with a **different ordered list** | **1,009 (100%)** |
| Dates with a different admitted set | 1,007 (99.8%) |
| Mean set disagreement, `\|A △ B\| / \|A ∪ B\|` | 0.0867 |
| Max set disagreement | 0.2143 |
| Mean admitted names per arm | 100.09 |
| Mean **replacements** per date (one out paired with one in) | **4.56** |
| Max replacements on a single date | 12 |
| Mean symmetric difference (names counted on both sides) | 9.13 |
Per year, every evaluated date differs:
| Year | Dates | Different set | Different order |
|---|---:|---:|---:|
| 2022 | 132 | 132 | 132 |
| 2023 | 250 | 250 | 250 |
| 2024 | 252 | 252 | 252 |
| 2025 | 250 | 248 | 250 |
| 2026 | 125 | 125 | 125 |
Counted directly from the daily set differences rather than derived from the
Jaccard ratio: **4.56 names are replaced on an average date** — 4.56 leave and
4.56 arrive, so 9.13 names appear on one side or the other, out of about 100
admitted. A swap moves two names, which makes the symmetric difference twice the
replacement count and the two easy to conflate.
## Strong evidence of a dividend effect
First changed date, 2022-06-24 — 99 names admitted in each arm, nine swapped:
- **Only on price return:** CDNS, MAR, FIX, COST, LIN, AON, STE, CI, AJG
- **Only on total return:** EXC, PSX, WY, AMCR, DTE, GIS, FE, BMY, ES
Growth and low-yield names out; utilities, energy, staples and other high payers
in. Nine tickers are not evidence, so it was measured (`--mechanism 25`, saved to
`reports/tr-rank-agreement-mechanism.json` with its sampled dates and per-date
values): across 40 sampled dates, the Spearman correlation between a name's
trailing 12-month implied yield — from the ALL/SPLIT ratio over the formation
window — and its momentum-percentile shift between arms is **+0.79** (range
+0.71 to +0.85, every sampled date positive).
**What that does and does not pin down.** The reordering is systematic and
directional rather than numerical noise, and it moves with yield. It is *not* a
clean isolation of the stock's own yield: both the stock series and the benchmark
leg change between arms, so the measured shift also contains the beta-weighted
benchmark yield. Read it as strong evidence consistent with a dividend effect.
Isolating own-yield would need a third variant — total-return stock series
against a nominal SPY leg — which has not been run.
## Verdict
The pre-registered rule was "no ordered-list difference on any evaluated date →
stop". Differences occur on **every** evaluated date, so **Step 3 proceeds**.
What this does and does not establish:
- **Does:** the momentum basis materially changes gate admission and ordering,
every day, in a yield-driven direction.
- **Does not:** show that realised P&L changes. Production also applies setup
qualification, existing holdings, available slots and the rest of the admission
path. A name that swaps into the admitted list may never produce a setup. Step
3's paired portfolio replay is what decides whether any of this reaches trades.
- **Worth noting:** the tuning recorded as closed in the research log was carried
out on a price-return basis throughout. This screen does not invalidate it —
it says the input those conclusions were computed on is not the one the factor
is defined on.
## Reproduce
```powershell
python scripts/research_tr_rank_agreement.py --fetch # 1,014 requests, cached
python scripts/research_tr_rank_agreement.py --time-batch 20 # runtime projection
python scripts/research_tr_rank_agreement.py # full screen
python scripts/research_tr_rank_agreement.py --mechanism 25 # yield vs percentile shift
```
The cache makes re-runs free and the screen deterministic; delete
`backtest_snapshots/tr-rank-cache.sqlite` to refetch.
docs/research/tr-total-return-ab.md
+160
View File
@@ -0,0 +1,160 @@
# Total-return momentum — paired portfolio replay (Step 3a)
**Run date:** 2026-09-07 · **Status:** measured; **no rollout decision taken**
**Report:** `reports/tr-total-return-ab.json` · **Script:** `scripts/run_total_return_ab.py`
**Inputs:** `backtest_snapshots/survivorship-a.sqlite` (bars),
`backtest_snapshots/tr-rank-cache.sqlite` (both adjusted series),
`backtest_snapshots/tr-ab-candidates.pkl` (202,765 shared candidates)
## Question
[The ranking screen](tr-rank-agreement.md) showed the momentum basis changes the
admitted list on every evaluated date. Does that reach actual admissions and P&L?
## What was held identical
Setup candidates were generated **once** (202,765 over 505 symbols, matching the
count in the standalone backtest report) and reused by both books. Only momentum
and its dependents were recomputed per arm: residual and activation percentiles,
`qualified`, and the 80/20 blend used for ordering.
| Held fixed | Varied |
|---|---|
| Setup geometry — entry, stop, target, R:R | Momentum series: SPLIT stock vs SPLIT SPY (control), ALL vs ALL SPY (treatment) |
| Execution on the snapshot's nominal bars | The percentiles, `qualified` and ordering that follow from it |
| `vol_6m` volatility leg, from candidate generation | |
| Non-momentum gates, `max_positions`, risk per trade, ATR-trail exit, gate-reset re-entry | |
Both arms read the same cache, so a refreshed-vs-snapshot difference cannot be
mistaken for a basis difference. Each book then keeps its own holdings, so a
divergent admission changes later slot occupancy — that path dependence is part
of what is measured.
**Harness parity.** The control arm reproduces the production row of
`reports/backtest-20260907-survivorship-a.json` exactly — 322 trades, 424.6%
total return, 51.6% CAGR, 21.4% drawdown, 2.09 Sharpe. Any difference below is
therefore attributable to the basis, not to a re-implemented book.
## Result
| Metric | Control (price return) | Treatment (total return) | Δ |
|---|---:|---:|---:|
| Trades | 322 | 334 | +12 |
| Total return % | 424.6 | 376.5 | −48.1 |
| CAGR % | 51.6 | 47.9 | **−3.7** |
| Max drawdown % | 21.4 | 19.5 | **−1.9** (better) |
| Sharpe | 2.09 | 1.98 | −0.11 |
| Win rate % | 37.0 | 36.8 | −0.2 |
| Profit factor | 2.48 | 2.33 | −0.15 |
**The books really do differ.** 271 entries are shared; **51 are taken only by
control and 63 only by treatment** — about 30% of the union of entries. The
composition follows the Step 2 mechanism: control takes MAR, AAPL, BIIB, LVS;
treatment takes BG, OKE, DE, ARES. Some differences are timing rather than
selection — ABBV is entered on 2022-09-15 by control and 2022-09-22 by treatment.
## Where the difference sits
**One year carries all of it.** Per-year returns:
| Year | Control % | Treatment % | Δpp |
|---|---:|---:|---:|
| 2022 | 8.1 | 9.9 | +1.8 |
| 2023 | 42.0 | 41.9 | −0.1 |
| 2024 | 36.4 | 38.4 | +2.0 |
| 2025 | 80.2 | 58.1 | **−22.1** |
| 2026 | 39.2 | 39.6 | +0.4 |
Treatment wins three years and loses two, and four of the five differ by under
2.1pp. The entire −3.7pp CAGR gap is 2025. That is a description of *where* the
shortfall sits — a useful thing to know when reading it — and nothing more. It is
not grounds for calling the shortfall noise or the estimate unstable.
**Uncertainty is not yet quantified.** The per-arm `sharpe_se` values (0.497 and
0.498) are the standard errors of each Sharpe *individually*. They say nothing
about the uncertainty of the **difference**: the two books share dates and most
of their positions, so the arms are strongly correlated and the standard error of
their difference depends on that covariance, not on the two marginal errors. An
earlier version of this document compared the 0.11 Sharpe gap to a 0.50 marginal
SE and concluded the books were "statistically indistinguishable" — that
inference does not follow, and the same applies to the drawdown difference, for
which no error estimate was reported at all. Both claims are withdrawn.
## Paired block bootstrap on the difference
Pre-specified before running — seed `20260907`, **5,000** replicates, block
lengths **21 and 63** sessions, both reported rather than whichever reads better.
The two arms' daily return streams are date-aligned (the script aborts if they
are not), and **the same block indices are drawn for both arms**, so the pairing
and whatever cross-arm correlation exists on those days are preserved. Moving
blocks retain some of the serial dependence an i.i.d. resample would destroy.
Treatment − control, 95% interval:
| Block | Metric | Mean | 2.5% | 97.5% | Share of replicates < 0 |
|---|---|---:|---:|---:|---:|
| 21 | ΔCAGR pp | −3.61 | −14.76 | +6.45 | 0.75 |
| 21 | ΔSharpe | −0.109 | −0.433 | +0.198 | 0.75 |
| 21 | ΔMaxDD pp | −1.00 | −7.50 | +4.34 | 0.63 |
| 63 | ΔCAGR pp | −3.78 | −14.67 | +5.40 | 0.76 |
| 63 | ΔSharpe | −0.113 | −0.428 | +0.182 | 0.76 |
| 63 | ΔMaxDD pp | −1.79 | −8.22 | +2.68 | 0.78 |
The two block lengths agree closely, which is reassuring about the resampling
itself. Every interval spans zero, and roughly three-quarters of replicates fall
below zero for CAGR and Sharpe (treatment worse) and for drawdown (treatment
better, since a negative ΔMaxDD is a shallower drawdown). So the observed window
leans toward control on return and treatment on drawdown without resolving either
sign confidently.
**What this diagnostic is not.** It resamples the *realised* daily return streams;
it does not re-run the strategy, so it cannot capture how a reordered history
would have changed admissions or holdings. It is an exploratory uncertainty
estimate from the observed window — not out-of-sample validation, and not a
rollout gate.
## Verdict
**Decision: production stays unchanged.** No schema, no ingestion, no wider
parameter sweep.
- **Established:** the basis is production-relevant. It changes roughly 30% of
entries, in the yield-driven direction the screen predicted.
- **Not established:** which basis performs better. Point estimates favour control
on return and Sharpe and treatment on drawdown; every bootstrap interval spans
zero, with about three-quarters of the mass on the point-estimate side. The
observed window does not resolve the sign.
- No acceptable return sacrifice is being chosen from these numbers, and no
threshold is being fitted to them after the fact.
What a decision would still need:
1. **A pre-registered numeric rule, set before it is applied.** The plan commits
to non-inferiority in principle — total return is the basis 12-1 momentum is
defined on, so an in-sample shortfall should not automatically lock in the
basis we believe is measured wrong — but carries no threshold. An earlier draft
used ΔCAGR ≥ −0.50pp at ΔMaxDD ≤ +1.50pp. Choosing a threshold now, with
−3.7pp on the table, would be fitting the rule to the answer.
2. **Out-of-sample evidence.** One five-year snapshot, in-sample throughout.
3. Note what would *not* help: a matrix over capacity, start date or cadence
produces **sensitivity checks, not independent samples**. Fifty-three
overlapping paths from one price history are not fifty-three pieces of
evidence, and treating them that way would overstate what a sweep buys.
## Reproduce
```powershell
python scripts/run_total_return_ab.py --build-candidates # ~9 min, cached
python scripts/run_total_return_ab.py # both arms + bootstrap, ~40 s
```
The bootstrap parameters are constants in the script (`BOOTSTRAP_SEED`,
`BOOTSTRAP_REPLICATES`, `BOOTSTRAP_BLOCKS`) rather than flags, so a run cannot
quietly pick a different seed or block length.
Guards in `tests/unit/test_total_return_ab.py`: identical input series must
produce identical books (so an observed difference is the basis, not replay
nondeterminism); a controlled momentum change must move a name across the cutoff
(so the harness can see what it is looking for); the volatility leg must be
untouched between arms; and a name with no computable momentum in an arm must not
qualify on a value carried over from candidate generation.
reports/backtest-20260907-survivorship-a.json
+141185
View File
File diff suppressed because it is too large. Load diff
reports/backtest-20260907-survivorship-b-nomarks.json
+141644
View File
File diff suppressed because it is too large. Load diff
reports/backtest-20260907-survivorship-b.json
+141644
View File
File diff suppressed because it is too large. Load diff
reports/tr-rank-agreement-mechanism.json
+105
View File
@@ -0,0 +1,105 @@
{
"generated_at": "2026-09-07T10:55:05.693160+00:00",
"snapshot": "C:\\Workspace\\signal-platform\\backtest_snapshots\\survivorship-a.sqlite",
"cache": "C:\\Workspace\\signal-platform\\backtest_snapshots\\tr-rank-cache.sqlite",
"window": {
"start": "2021-06-24",
"end": "2026-07-02"
},
"universe_size": 505,
"sample_every_n_sessions": 25,
"result": {
"what_it_measures": "Spearman correlation, per sampled date, between a name's trailing 12-month implied yield (from the ALL/SPLIT ratio over the formation window) and the change in its momentum percentile between the two arms.",
"caveat": "Both the stock series and the benchmark leg change between arms, so this is strong evidence consistent with a dividend effect - it does not isolate the stock's own yield from the beta-weighted benchmark yield.",
"formation_window_bars": 252,
"min_symbols_per_date": 30,
"sampled_dates": 40,
"sample_date_list": [
"2022-07-28",
"2022-09-01",
"2022-10-07",
"2022-11-11",
"2022-12-19",
"2023-01-26",
"2023-03-03",
"2023-04-10",
"2023-05-15",
"2023-06-21",
"2023-07-27",
"2023-08-31",
"2023-10-06",
"2023-11-10",
"2023-12-18",
"2024-01-25",
"2024-03-01",
"2024-04-08",
"2024-05-13",
"2024-06-18",
"2024-07-25",
"2024-08-29",
"2024-10-04",
"2024-11-08",
"2024-12-16",
"2025-01-24",
"2025-03-03",
"2025-04-07",
"2025-05-13",
"2025-06-18",
"2025-07-25",
"2025-08-29",
"2025-10-06",
"2025-11-10",
"2025-12-16",
"2026-01-23",
"2026-03-02",
"2026-04-07",
"2026-05-12",
"2026-06-17"
],
"mean_spearman_yield_vs_percentile_shift": 0.7924,
"min": 0.7138,
"max": 0.8459,
"per_date_spearman": [
0.7814,
0.7933,
0.8216,
0.8325,
0.8108,
0.8147,
0.8074,
0.8421,
0.7867,
0.7907,
0.7682,
0.7568,
0.7849,
0.7536,
0.7138,
0.7385,
0.7414,
0.7697,
0.8078,
0.8325,
0.8132,
0.8459,
0.8397,
0.7966,
0.7808,
0.8285,
0.8068,
0.8376,
0.7974,
0.8256,
0.8046,
0.7762,
0.8008,
0.8046,
0.7793,
0.7978,
0.7678,
0.7549,
0.7501,
0.7384
]
}
}
reports/tr-rank-agreement.json
+540
View File
@@ -0,0 +1,540 @@
{
"generated_at": "2026-09-07T10:57:26.602851+00:00",
"snapshot": "C:\\Workspace\\signal-platform\\backtest_snapshots\\survivorship-a.sqlite",
"cache": "C:\\Workspace\\signal-platform\\backtest_snapshots\\tr-rank-cache.sqlite",
"window": {
"start": "2021-06-24",
"end": "2026-07-02"
},
"universe_size": 505,
"inert_snapshot_symbols": [
"RHM"
],
"cutoff": 80.0,
"verdict": "proceed to Step 3 - ordered lists differ",
"result": {
"evaluated_dates": 1009,
"skipped_dates_no_history": 252,
"dates_with_different_admitted_set": 1007,
"dates_with_different_ordered_list": 1009,
"mean_set_disagreement": 0.08668279921946223,
"max_set_disagreement": 0.21428571428571427,
"mean_admitted_per_arm": 100.09,
"mean_names_only_in_price_return": 4.56,
"mean_names_only_in_total_return": 4.56,
"mean_replacements_per_date": 4.56,
"max_replacements_per_date": 12,
"mean_symmetric_difference": 9.13,
"per_year": {
"2022": {
"dates": 132,
"changed_sets": 132,
"changed_orders": 132
},
"2023": {
"dates": 250,
"changed_sets": 250,
"changed_orders": 250
},
"2024": {
"dates": 252,
"changed_sets": 252,
"changed_orders": 252
},
"2025": {
"dates": 250,
"changed_sets": 248,
"changed_orders": 250
},
"2026": {
"dates": 125,
"changed_sets": 125,
"changed_orders": 125
}
},
"examples": [
{
"date": "2022-06-24",
"admitted_price_return": 99,
"admitted_total_return": 99,
"only_in_price_return": [
"CDNS",
"MAR",
"FIX",
"COST",
"LIN",
"AON",
"STE",
"CI",
"AJG"
],
"only_in_total_return": [
"EXC",
"PSX",
"WY",
"AMCR",
"DTE",
"GIS",
"FE",
"BMY",
"ES"
],
"order_changed_without_membership_change": false,
"first_10_price_return": [
"OXY",
"EQT",
"APA",
"DVN",
"MOS",
"CTRA",
"ON",
"ALB",
"CF",
"SMCI"
],
"first_10_total_return": [
"OXY",
"EQT",
"APA",
"DVN",
"MOS",
"CTRA",
"ON",
"CF",
"ALB",
"SMCI"
]
},
{
"date": "2022-06-27",
"admitted_price_return": 99,
"admitted_total_return": 99,
"only_in_price_return": [
"FCX",
"COST",
"FIX",
"CI",
"PAYX",
"HSIC",
"L",
"AIZ"
],
"only_in_total_return": [
"EXC",
"MO",
"AMCR",
"GIS",
"DTE",
"CMS",
"FE",
"BMY"
],
"order_changed_without_membership_change": false,
"first_10_price_return": [
"APA",
"OXY",
"DVN",
"EQT",
"MOS",
"ON",
"CTRA",
"ALB",
"CF",
"SMCI"
],
"first_10_total_return": [
"APA",
"OXY",
"DVN",
"EQT",
"MOS",
"ON",
"CTRA",
"CF",
"SMCI",
"ALB"
]
},
{
"date": "2022-06-28",
"admitted_price_return": 99,
"admitted_total_return": 99,
"only_in_price_return": [
"FCX",
"LYV",
"ULTA",
"CDNS",
"AMP",
"FIX",
"AJG"
],
"only_in_total_return": [
"EXC",
"MO",
"AMCR",
"ETR",
"DTE",
"CMS",
"BMY"
],
"order_changed_without_membership_change": false,
"first_10_price_return": [
"OXY",
"APA",
"DVN",
"MOS",
"EQT",
"CTRA",
"ON",
"CF",
"ALB",
"HAL"
],
"first_10_total_return": [
"OXY",
"APA",
"DVN",
"EQT",
"MOS",
"CTRA",
"ON",
"CF",
"EXE",
"HAL"
]
},
{
"date": "2022-06-29",
"admitted_price_return": 99,
"admitted_total_return": 99,
"only_in_price_return": [
"CDNS",
"FIX",
"COST",
"WAB",
"AJG",
"TRV"
],
"only_in_total_return": [
"EXC",
"LYB",
"MO",
"AMCR",
"DTE",
"WEC"
],
"order_changed_without_membership_change": false,
"first_10_price_return": [
"OXY",
"APA",
"DVN",
"EQT",
"MOS",
"CTRA",
"ALB",
"ON",
"CF",
"EXE"
],
"first_10_total_return": [
"OXY",
"APA",
"DVN",
"EQT",
"MOS",
"CTRA",
"CF",
"ALB",
"EXE",
"ON"
]
},
{
"date": "2022-06-30",
"admitted_price_return": 99,
"admitted_total_return": 99,
"only_in_price_return": [
"LYV",
"CDNS",
"RJF",
"WAB",
"FIX",
"AON",
"TKO"
],
"only_in_total_return": [
"EXC",
"NRG",
"MO",
"LMT",
"DTE",
"WEC",
"AEE"
],
"order_changed_without_membership_change": false,
"first_10_price_return": [
"OXY",
"APA",
"DVN",
"EQT",
"MOS",
"CTRA",
"ON",
"CF",
"ALB",
"HAL"
],
"first_10_total_return": [
"OXY",
"APA",
"DVN",
"EQT",
"MOS",
"CTRA",
"ON",
"CF",
"EXE",
"HAL"
]
},
{
"date": "2022-07-01",
"admitted_price_return": 99,
"admitted_total_return": 99,
"only_in_price_return": [
"DDOG",
"LYV",
"HLT",
"RJF",
"FIX",
"TKO",
"AON",
"AJG",
"ACGL"
],
"only_in_total_return": [
"EXC",
"WY",
"NRG",
"AMCR",
"MO",
"PKG",
"LMT",
"CMS",
"KO"
],
"order_changed_without_membership_change": false,
"first_10_price_return": [
"EQT",
"OXY",
"APA",
"DVN",
"MOS",
"CTRA",
"ON",
"SMCI",
"CF",
"EXE"
],
"first_10_total_return": [
"EQT",
"OXY",
"APA",
"DVN",
"MOS",
"CTRA",
"ON",
"CF",
"SMCI",
"EXE"
]
},
{
"date": "2022-07-05",
"admitted_price_return": 99,
"admitted_total_return": 99,
"only_in_price_return": [
"HLT",
"AON",
"TKO",
"L",
"TRV"
],
"only_in_total_return": [
"EXC",
"WY",
"MO",
"CMS",
"DTE"
],
"order_changed_without_membership_change": false,
"first_10_price_return": [
"APA",
"OXY",
"DVN",
"EQT",
"MOS",
"ON",
"SMCI",
"CTRA",
"CF",
"HAL"
],
"first_10_total_return": [
"EQT",
"OXY",
"APA",
"DVN",
"MOS",
"ON",
"CTRA",
"SMCI",
"CF",
"EXE"
]
},
{
"date": "2022-07-06",
"admitted_price_return": 99,
"admitted_total_return": 99,
"only_in_price_return": [
"TPL",
"ANET",
"HLT",
"LIN",
"UNH",
"CVS"
],
"only_in_total_return": [
"EXC",
"RF",
"LYB",
"WY",
"MO",
"ETR"
],
"order_changed_without_membership_change": false,
"first_10_price_return": [
"APA",
"OXY",
"DVN",
"EQT",
"MOS",
"ON",
"CTRA",
"SMCI",
"CF",
"HAL"
],
"first_10_total_return": [
"APA",
"OXY",
"DVN",
"EQT",
"MOS",
"CTRA",
"ON",
"SMCI",
"CF",
"HAL"
]
},
{
"date": "2022-07-07",
"admitted_price_return": 99,
"admitted_total_return": 99,
"only_in_price_return": [
"RJF",
"LIN",
"PKG",
"L"
],
"only_in_total_return": [
"EXC",
"LYB",
"WY",
"ETR"
],
"order_changed_without_membership_change": false,
"first_10_price_return": [
"APA",
"OXY",
"DVN",
"EQT",
"ON",
"MOS",
"SMCI",
"CTRA",
"HAL",
"CF"
],
"first_10_total_return": [
"APA",
"EQT",
"OXY",
"DVN",
"MOS",
"ON",
"CTRA",
"SMCI",
"HAL",
"EXE"
]
},
{
"date": "2022-07-08",
"admitted_price_return": 99,
"admitted_total_return": 99,
"only_in_price_return": [
"ANET",
"HLT",
"RJF",
"WAB",
"RSG",
"AIZ",
"HSY"
],
"only_in_total_return": [
"EXC",
"LYB",
"AMCR",
"MO",
"ETR",
"AEP",
"ATO"
],
"order_changed_without_membership_change": false,
"first_10_price_return": [
"APA",
"OXY",
"DVN",
"EQT",
"ON",
"MOS",
"CTRA",
"SMCI",
"HAL",
"VLO"
],
"first_10_total_return": [
"APA",
"OXY",
"DVN",
"EQT",
"CTRA",
"MOS",
"ON",
"SMCI",
"HAL",
"EXE"
]
}
],
"elapsed_seconds": 132.0
},
"coverage": {
"symbols_cached": 507,
"symbols_with_unmatched_dates": 0,
"total_unmatched_dates": 0
}
}
reports/tr-total-return-ab.json
+13003
View File
File diff suppressed because it is too large. Load diff
scripts/data/sp500_removals.json
+1774
View File
File diff suppressed because it is too large. Load diff
scripts/prep_survivorship_snapshot.py
+477
View File
@@ -0,0 +1,477 @@
"""Build the two survivorship-audit snapshots from a base snapshot.
Arm A is the universe the backtest already runs on. Arm B is the same universe
plus the S&P 500 constituents removed while the snapshot window was open, so the
strategy can trade the names that later left the index — including the ones that
failed.
The base file is never modified: both arms are copies, and the base hash is
re-checked at the end.
Three things this script has to get right, all of them learned from the data:
* **Schema drift.** The snapshot files on disk predate migrations 026 and 032,
so ``select(Ticker)`` against them fails with ``no such column: tickers.cik``
— and that is the path ``backtest_service._fetch_columns`` takes. Both copies
are synced against ``Base.metadata`` before anything else touches them.
* **Frozen tails.** Alpaca does not end a halted symbol's series; it repeats the
last price at zero volume. SBNY sits at 70.00 for 509 sessions after the FDIC
closed it, and INFO sits at 108.61 for 1,090 sessions before a different
company reuses the ticker. Only the symbols with a *reviewed cutoff* in the
dataset are truncated. Everywhere else a long zero-volume run is a tripwire:
the symbol is left out and reported for review rather than silently cut.
* **Terminal marks.** A failed bank stops trading mid-collapse, so the exchange
bars never show the loss. A mark is written only where the dataset carries an
explicit confirmation *and* the retained final bar matches the reviewed
``last_real_bar``. The mark is an assumption, not an observed execution.
Nothing is inferred. ``delisted_on`` / ``delisted_reason`` are written only for
confirmed terminal events; a series that merely ends early leaves them NULL and
is reported as an unresolved ending status.
Usage::
python scripts/prep_survivorship_snapshot.py
python scripts/prep_survivorship_snapshot.py --base backtest_snapshots/prod.sqlite --force
"""
from __future__ import annotations
import argparse
import asyncio
import hashlib
import json
import shutil
import sqlite3
import sys
from datetime import date, datetime, time, timedelta, timezone
from pathlib import Path
from typing import Any
ROOT = Path(__file__).resolve().parents[1]
if str(ROOT) not in sys.path:
sys.path.insert(0, str(ROOT))
# Bars are cut at the first run of this many consecutive zero-volume sessions.
ZERO_VOLUME_RUN = 5
# Close assigned to a confirmed terminal event. Not zero: a zero price divides
# through several return calculations, and 1 cent is a near-total loss either way.
TERMINAL_CLOSE = 0.01
DEFAULT_BASE = ROOT / "backtest_snapshots" / "prod.sqlite"
DEFAULT_REMOVALS = ROOT / "scripts" / "data" / "sp500_removals.json"
# ---------------------------------------------------------------------------
# schema sync
# ---------------------------------------------------------------------------
def sync_schema(path: Path) -> dict[str, list[str]]:
"""Add every column the models declare that the snapshot file lacks.
Nullable and untyped-default only: this brings an old snapshot up to the
current models without inventing data. Tables the snapshot does not have at
all are left alone — a snapshot is a subset of production by design.
"""
from sqlalchemy.dialects import sqlite as sqlite_dialect
from app.database import Base
import app.models # noqa: F401 (registers every model on Base.metadata)
dialect = sqlite_dialect.dialect()
added: dict[str, list[str]] = {}
con = sqlite3.connect(path)
try:
present = {r[0] for r in con.execute("SELECT name FROM sqlite_master WHERE type='table'")}
for table in Base.metadata.sorted_tables:
if table.name not in present:
continue
have = {r[1] for r in con.execute(f'PRAGMA table_info("{table.name}")')}
for column in table.columns:
if column.name in have:
continue
type_sql = column.type.compile(dialect)
con.execute(f'ALTER TABLE "{table.name}" ADD COLUMN "{column.name}" {type_sql}')
added.setdefault(table.name, []).append(column.name)
con.commit()
finally:
con.close()
return added
# ---------------------------------------------------------------------------
# price data
# ---------------------------------------------------------------------------
def frozen_run_start(bars: list[Any]) -> int | None:
"""Index where the first run of >= ZERO_VOLUME_RUN zero-volume bars begins.
A detector, not a cutter: the caller decides what to do with the answer, and
only a reviewed cutoff is allowed to shorten a series.
"""
volumes = [int(b.volume) for b in bars]
i = 0
while i < len(volumes):
if volumes[i] == 0:
j = i
while j < len(volumes) and volumes[j] == 0:
j += 1
if j - i >= ZERO_VOLUME_RUN:
return i
i = j
else:
i += 1
return None
def apply_reviewed_cutoff(bars: list[Any], cutoff: date) -> tuple[list[Any], int]:
"""Keep bars up to and including *cutoff*. Returns (kept, dropped)."""
kept = [b for b in bars if b.timestamp.date() <= cutoff]
return kept, len(bars) - len(kept)
async def fetch_bars(symbol: str, start: date, end: date) -> list[Any]:
"""Daily SPLIT-adjusted bars, matching how the snapshot's existing bars were ingested."""
from app.config import settings
from app.providers.alpaca import AlpacaOHLCVProvider
if not settings.alpaca_api_key or not settings.alpaca_api_secret:
raise SystemExit("Alpaca credentials are not configured; cannot restore price history.")
from alpaca.data.enums import Adjustment
from alpaca.data.requests import StockBarsRequest
from alpaca.data.timeframe import TimeFrame
provider = AlpacaOHLCVProvider(settings.alpaca_api_key, settings.alpaca_api_secret)
alpaca_symbol = provider._to_alpaca_symbol(symbol)
request = StockBarsRequest(
symbol_or_symbols=alpaca_symbol,
timeframe=TimeFrame.Day,
start=datetime.combine(start, time.min, tzinfo=timezone.utc),
end=datetime.combine(end + timedelta(days=1), time.min, tzinfo=timezone.utc),
adjustment=Adjustment.SPLIT,
)
# 85 sequential requests are enough for Alpaca to drop one connection, and a
# half-built arm is worse than a slow one.
last_error: Exception | None = None
for attempt in range(4):
try:
bars = await asyncio.to_thread(provider._client.get_stock_bars, request)
return list(bars.data.get(alpaca_symbol, []))
except Exception as exc: # noqa: BLE001 - retry any transport failure
last_error = exc
if attempt < 3:
await asyncio.sleep(2 * (attempt + 1))
raise SystemExit(f"Alpaca fetch failed for {symbol} after 4 attempts: {last_error}")
# ---------------------------------------------------------------------------
# snapshot writes
# ---------------------------------------------------------------------------
def snapshot_window(path: Path) -> tuple[date, date]:
con = sqlite3.connect(path)
try:
lo, hi = con.execute("SELECT min(date), max(date) FROM ohlcv_records").fetchone()
finally:
con.close()
return date.fromisoformat(lo), date.fromisoformat(hi)
def trading_calendar(path: Path) -> list[date]:
"""SPY sessions from the snapshot's own benchmark series."""
con = sqlite3.connect(path)
try:
rows = con.execute(
"SELECT date FROM benchmark_prices WHERE symbol = 'SPY' ORDER BY date"
).fetchall()
finally:
con.close()
return [date.fromisoformat(r[0]) for r in rows]
def next_session(calendar: list[date], after: date) -> date | None:
for d in calendar:
if d > after:
return d
return None
def insert_symbol(
con: sqlite3.Connection,
symbol: str,
name: str,
bars: list[Any],
*,
delisted_on: str | None,
delisted_reason: str | None,
terminal_date: date | None,
) -> int:
"""Insert one restored ticker and its bars. Returns the bar count written."""
now = datetime.utcnow().isoformat(sep=" ")
cur = con.execute(
"INSERT INTO tickers (symbol, name, created_at, delisted_on, delisted_reason) "
"VALUES (?, ?, ?, ?, ?)",
(symbol, name, now, delisted_on, delisted_reason),
)
ticker_id = int(cur.lastrowid)
rows = [
(
ticker_id,
b.timestamp.date().isoformat(),
float(b.open),
float(b.high),
float(b.low),
float(b.close),
int(b.volume),
now,
)
for b in bars
]
if terminal_date is not None:
rows.append((
ticker_id, terminal_date.isoformat(),
TERMINAL_CLOSE, TERMINAL_CLOSE, TERMINAL_CLOSE, TERMINAL_CLOSE, 0, now,
))
con.executemany(
"INSERT INTO ohlcv_records "
"(ticker_id, date, open, high, low, close, volume, created_at) "
"VALUES (?, ?, ?, ?, ?, ?, ?, ?)",
rows,
)
return len(rows)
# ---------------------------------------------------------------------------
# driver
# ---------------------------------------------------------------------------
def file_hash(path: Path) -> str:
h = hashlib.sha256()
with path.open("rb") as fh:
for chunk in iter(lambda: fh.read(1 << 20), b""):
h.update(chunk)
return h.hexdigest()
async def build(args: argparse.Namespace) -> int:
base = Path(args.base).resolve()
if not base.exists():
raise SystemExit(f"Base snapshot not found: {base}")
out_dir = Path(args.out_dir).resolve()
arm_a = out_dir / args.arm_a_name
arm_b = out_dir / args.arm_b_name
events_path = arm_b.with_name(arm_b.stem + "-terminal-events.json")
doc = json.loads(Path(args.removals).read_text(encoding="utf-8"))
removals = doc["removals"]
confirmations = {c["symbol"]: c for c in doc["terminal_confirmations"] if c.get("confirmed")}
cutoffs = {
c["symbol"]: date.fromisoformat(c["last_real_bar"])
for c in doc.get("reviewed_cutoffs", {}).get("cutoffs", [])
}
# A corporate action ends a listing; a market-cap removal does not, so a
# series that stops early without one is an ending we cannot account for.
ENDING_EXPLAINED = {"acquisition", "receivership", "bankruptcy", "ticker_change"}
base_hash_before = file_hash(base)
window_start, window_end = snapshot_window(base)
print(f"base : {base}")
print(f"window : {window_start} .. {window_end}")
print(f"removals : {len(removals)} rows, {sum(1 for r in removals if r['restore_symbol'])} restorable")
for target in (arm_a, arm_b):
if target.exists() and not args.force:
raise SystemExit(f"{target} already exists (use --force to rebuild)")
if target.exists():
target.unlink()
shutil.copy2(base, target)
added = sync_schema(target)
detail = ", ".join(f"{t}: {', '.join(c)}" for t, c in added.items()) or "nothing missing"
print(f"copied : {target.name} (schema sync -> {detail})")
calendar = trading_calendar(base)
con = sqlite3.connect(arm_b)
existing = {r[0].upper() for r in con.execute("SELECT symbol FROM tickers")}
restored: list[dict] = []
unresolved: list[dict] = []
terminal_events: dict[str, str] = {}
try:
for row in removals:
symbol = row["restore_symbol"]
if not symbol:
continue
if symbol.upper() in existing:
unresolved.append({"kind": "skipped", "symbol": symbol, "why": "already present in the base snapshot"})
continue
bars = await fetch_bars(symbol, window_start, window_end)
if not bars:
unresolved.append({"kind": "coverage", "symbol": symbol, "why": "Alpaca returned no bars in the window"})
print(f" {symbol:<6} NO DATA (unresolved coverage)")
continue
cutoff = cutoffs.get(symbol) or cutoffs.get(row["symbol"])
frozen_at = frozen_run_start(bars)
if cutoff is not None:
kept, dropped = apply_reviewed_cutoff(bars, cutoff)
if not kept or kept[-1].timestamp.date() != cutoff:
unresolved.append({
"symbol": symbol,
"kind": "coverage",
"why": f"reviewed cutoff {cutoff} is not a bar in the downloaded series",
})
print(f" {symbol:<6} CUTOFF MISSING {cutoff} (unresolved coverage)")
continue
elif frozen_at is not None:
# Detector only. An unreviewed frozen run means the data changed
# under us; restoring either half would be a guess, so report it.
unresolved.append({
"kind": "coverage",
"symbol": symbol,
"why": (
f"unreviewed frozen run of >= {ZERO_VOLUME_RUN} zero-volume sessions "
f"from {bars[frozen_at].timestamp.date()}; needs a reviewed cutoff "
"before this symbol can be restored"
),
})
print(f" {symbol:<6} FROZEN RUN at {bars[frozen_at].timestamp.date()} (unresolved, not restored)")
continue
else:
kept, dropped = bars, 0
effective_last = kept[-1].timestamp.date()
confirmation = confirmations.get(row["symbol"]) or confirmations.get(symbol)
terminal_date = None
if confirmation is not None:
confirmed_last = date.fromisoformat(confirmation["last_real_bar"])
if effective_last != confirmed_last:
unresolved.append({
"symbol": symbol,
"kind": "terminal_mark",
"why": (
f"retained final bar {effective_last} does not match the reviewed "
f"last_real_bar {confirmed_last}; no terminal mark written"
),
})
print(f" {symbol:<6} MARK SKIPPED final bar {effective_last} != confirmed {confirmed_last}")
else:
terminal_date = next_session(calendar, effective_last)
if terminal_date is None or terminal_date > window_end:
terminal_date = None
unresolved.append({
"symbol": symbol,
"kind": "terminal_mark",
"why": "confirmed terminal event has no trading session left inside the window",
})
else:
terminal_events[symbol] = terminal_date.isoformat()
# Documentary only, and only where the event is confirmed. A series
# that merely ends early is an unresolved ending, not a delisting.
delisted_on = None
delisted_reason = None
if terminal_date is not None:
delisted_on = terminal_date.isoformat()
delisted_reason = row["reason"]
elif effective_last < window_end and row["reason"] not in ENDING_EXPLAINED:
unresolved.append({
"kind": "ending_status",
"symbol": symbol,
"why": (
f"price history ends {effective_last}, before the window closes on "
f"{window_end}, and the source row ({row['reason']}) does not explain "
"an ending; ending status unresolved and delisting fields left null"
),
})
written = insert_symbol(
con, symbol, row["name"], kept,
delisted_on=delisted_on, delisted_reason=delisted_reason,
terminal_date=terminal_date,
)
restored.append({
"symbol": symbol,
"removed_on": row["removed_on"],
"reason": row["reason"],
"bars": written,
"first_bar": kept[0].timestamp.date().isoformat(),
"effective_last_bar": effective_last.isoformat(),
"frozen_bars_dropped": dropped,
"terminal_mark": terminal_events.get(symbol),
})
mark = f" terminal={terminal_events[symbol]} @ {TERMINAL_CLOSE}" if symbol in terminal_events else ""
drop = f" dropped={dropped}" if dropped else ""
print(f" {symbol:<6} bars={written:<5} {kept[0].timestamp.date()}..{effective_last}{drop}{mark}")
con.commit()
finally:
con.close()
events_path.write_text(json.dumps(terminal_events, indent=2) + "\n", encoding="utf-8")
if file_hash(base) != base_hash_before:
raise SystemExit(f"BASE SNAPSHOT WAS MODIFIED: {base}")
con_a = sqlite3.connect(arm_a)
con_b = sqlite3.connect(arm_b)
counts = {
"arm_a_tickers": con_a.execute("SELECT count(*) FROM tickers").fetchone()[0],
"arm_a_bars": con_a.execute("SELECT count(*) FROM ohlcv_records").fetchone()[0],
"arm_b_tickers": con_b.execute("SELECT count(*) FROM tickers").fetchone()[0],
"arm_b_bars": con_b.execute("SELECT count(*) FROM ohlcv_records").fetchone()[0],
}
con_a.close()
con_b.close()
by_kind: dict[str, list[dict]] = {}
for row in unresolved:
by_kind.setdefault(row.get("kind", "other"), []).append(row)
print("\nsummary")
print(f" restored : {len(restored)}")
print(f" unresolved coverage: {len(by_kind.get('coverage', []))} (symbols not restored)")
print(f" unresolved endings : {len(by_kind.get('ending_status', []))} (restored, ending status unknown)")
print(f" terminal marks : {len(terminal_events)} -> {events_path.name}")
print(f" arm A : {counts['arm_a_tickers']} tickers, {counts['arm_a_bars']} bars")
print(f" arm B : {counts['arm_b_tickers']} tickers, {counts['arm_b_bars']} bars")
print(f" base unchanged : sha256 {base_hash_before[:16]}...")
for kind in ("coverage", "ending_status", "terminal_mark"):
for row in by_kind.get(kind, []):
print(f" UNRESOLVED[{kind}] {row['symbol']:<6} {row['why']}")
manifest = {
"generated_at": datetime.now(timezone.utc).isoformat(),
"base": str(base),
"base_sha256": base_hash_before,
"window": {"start": window_start.isoformat(), "end": window_end.isoformat()},
"zero_volume_run": ZERO_VOLUME_RUN,
"terminal_close": TERMINAL_CLOSE,
"counts": counts,
"restored": restored,
"unresolved": unresolved,
"terminal_events": terminal_events,
}
manifest_path = arm_b.with_name(arm_b.stem + "-prep-manifest.json")
manifest_path.write_text(json.dumps(manifest, indent=2) + "\n", encoding="utf-8")
print(f" manifest : {manifest_path.name}")
return 0
def parse_args() -> argparse.Namespace:
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
p.add_argument("--base", default=str(DEFAULT_BASE), help="Base snapshot to copy (never modified).")
p.add_argument("--out-dir", default=str(DEFAULT_BASE.parent), help="Where the arm copies are written.")
p.add_argument("--arm-a-name", default="survivorship-a.sqlite")
p.add_argument("--arm-b-name", default="survivorship-b.sqlite")
p.add_argument("--removals", default=str(DEFAULT_REMOVALS))
p.add_argument("--force", action="store_true", help="Rebuild the arm files if they already exist.")
return p.parse_args()
if __name__ == "__main__":
raise SystemExit(asyncio.run(build(parse_args())))
scripts/research_tr_rank_agreement.py
+596
View File
@@ -0,0 +1,596 @@
"""Does total-return momentum change any admission or ordering decision?
Step 2 of docs/data-integrity-plan.md. A screening test, deliberately cheap: if
ranking on total return picks the same names in the same order as ranking on
price return, nothing downstream can change and the production `tr_factor` work
(Step 3) is unnecessary.
Why this needs no schema and no ingestion change: the forward-anchored factor
design exists because *production* ingest is incremental, so a back-adjusted
series drifts between fetches. Here each symbol's whole history is fetched in one
request, and within a single response the adjustment is internally consistent.
Method
------
* One `Adjustment.SPLIT` and one `Adjustment.ALL` request per symbol plus SPY,
cached to sqlite so re-runs are free and reproducible. Aligned by date, never
zipped by index; a date present in only one adjustment is dropped and counted.
* At every SPY session with enough history, residual 12-1 momentum is computed
twice with the *same* production function: the price-return series against
nominal SPY, and the total-return series against total-return SPY. Only bars at
or before the decision date are used.
* The volatility leg is computed once per date from the SPLIT series and held
fixed in both arms, so the only thing that varies is momentum.
* Both arms are percentiled with the production `_percentiles`, gated at the
production `min_momentum_percentile`, then ordered by `blend_strategy_rank`
with the symbol as a deterministic tie-break. The full admitted list is
compared - never truncated to `max_positions`, because a lower-ranked name can
be the first actionable setup when others fail gates or are already held.
Decision rule (pre-registered, docs/data-integrity-plan.md)
----------------------------------------------------------
* no ordered-list difference on any evaluated date -> stop, Step 3 unnecessary
* any ordered-list difference -> Step 3 investigates trades
* incomplete downloads or no evaluable dates -> inconclusive, fix coverage
Usage::
python scripts/research_tr_rank_agreement.py --fetch # populate the cache
python scripts/research_tr_rank_agreement.py --time-batch # project the runtime
python scripts/research_tr_rank_agreement.py # full screen
"""
from __future__ import annotations
import argparse
import asyncio
import json
import sqlite3
import sys
import time
from datetime import date, datetime, time as dtime, timedelta, timezone
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
if str(ROOT) not in sys.path:
sys.path.insert(0, str(ROOT))
DEFAULT_SNAPSHOT = ROOT / "backtest_snapshots" / "survivorship-a.sqlite"
DEFAULT_CACHE = ROOT / "backtest_snapshots" / "tr-rank-cache.sqlite"
BENCHMARK = "SPY"
# ---------------------------------------------------------------------------
# cache
# ---------------------------------------------------------------------------
def open_cache(path: Path) -> sqlite3.Connection:
con = sqlite3.connect(path)
con.execute(
"CREATE TABLE IF NOT EXISTS bars ("
" symbol TEXT NOT NULL, date TEXT NOT NULL,"
" split_close REAL, all_close REAL,"
" PRIMARY KEY (symbol, date))"
)
con.execute(
"CREATE TABLE IF NOT EXISTS fetched ("
" symbol TEXT PRIMARY KEY, fetched_at TEXT NOT NULL,"
" split_bars INTEGER, all_bars INTEGER, aligned_bars INTEGER,"
" dropped_unmatched INTEGER)"
)
con.commit()
return con
async def fetch_pair(symbol: str, start: date, end: date) -> tuple[dict, dict]:
"""(split_closes, all_closes) keyed by date, each from a single request."""
from alpaca.data.enums import Adjustment
from alpaca.data.historical import StockHistoricalDataClient
from alpaca.data.requests import StockBarsRequest
from alpaca.data.timeframe import TimeFrame
from app.config import settings
if not settings.alpaca_api_key or not settings.alpaca_api_secret:
raise SystemExit("Alpaca credentials are not configured.")
client = StockHistoricalDataClient(settings.alpaca_api_key, settings.alpaca_api_secret)
alpaca_symbol = symbol.replace("-", ".")
window = (
datetime.combine(start, dtime.min, tzinfo=timezone.utc),
datetime.combine(end + timedelta(days=1), dtime.min, tzinfo=timezone.utc),
)
async def one(adjustment) -> dict[date, float]:
request = StockBarsRequest(
symbol_or_symbols=alpaca_symbol, timeframe=TimeFrame.Day,
start=window[0], end=window[1], adjustment=adjustment,
)
last: Exception | None = None
for attempt in range(4):
try:
bars = await asyncio.to_thread(client.get_stock_bars, request)
return {
b.timestamp.date(): float(b.close)
for b in bars.data.get(alpaca_symbol, [])
}
except Exception as exc: # noqa: BLE001 - retry any transport failure
last = exc
if attempt < 3:
await asyncio.sleep(2 * (attempt + 1))
raise SystemExit(f"Alpaca fetch failed for {symbol}: {last}")
return await one(Adjustment.SPLIT), await one(Adjustment.ALL)
async def populate_cache(symbols: list[str], start: date, end: date, cache: Path) -> dict:
con = open_cache(cache)
done = {r[0] for r in con.execute("SELECT symbol FROM fetched")}
todo = [s for s in symbols if s not in done]
print(f"cache {cache.name}: {len(done)} cached, {len(todo)} to fetch")
coverage: dict[str, dict] = {}
for index, symbol in enumerate(todo, 1):
split, adjusted = await fetch_pair(symbol, start, end)
# Align by date. A date in only one adjustment cannot be compared and is
# dropped rather than paired with its neighbour.
common = sorted(set(split) & set(adjusted))
dropped = len(set(split) ^ set(adjusted))
con.executemany(
"INSERT OR REPLACE INTO bars (symbol, date, split_close, all_close) VALUES (?,?,?,?)",
[(symbol, d.isoformat(), split[d], adjusted[d]) for d in common],
)
con.execute(
"INSERT OR REPLACE INTO fetched VALUES (?,?,?,?,?,?)",
(symbol, datetime.now(timezone.utc).isoformat(), len(split), len(adjusted),
len(common), dropped),
)
con.commit()
if dropped:
print(f" {symbol:<6} aligned={len(common)} dropped_unmatched={dropped}")
if index % 50 == 0:
print(f" … {index}/{len(todo)}")
for row in con.execute("SELECT symbol, split_bars, all_bars, aligned_bars, dropped_unmatched FROM fetched"):
coverage[row[0]] = {
"split_bars": row[1], "all_bars": row[2],
"aligned_bars": row[3], "dropped_unmatched": row[4],
}
con.close()
return coverage
def load_series(cache: Path, symbols: list[str]) -> dict[str, tuple[list[date], list[float], list[float]]]:
con = open_cache(cache)
out: dict[str, tuple[list[date], list[float], list[float]]] = {}
for symbol in symbols:
rows = con.execute(
"SELECT date, split_close, all_close FROM bars WHERE symbol = ? ORDER BY date",
(symbol,),
).fetchall()
if not rows:
continue
out[symbol] = (
[date.fromisoformat(r[0]) for r in rows],
[float(r[1]) for r in rows],
[float(r[2]) for r in rows],
)
con.close()
return out
# ---------------------------------------------------------------------------
# screen
# ---------------------------------------------------------------------------
def snapshot_symbols(snapshot: Path) -> tuple[list[str], list[str], date, date]:
"""(symbols with bars, symbols with none, window start, window end).
A ticker row with no bars in the snapshot cannot rank in either arm, so it is
inert rather than missing. Keeping the two apart is what stops a failed
download from being quietly reported as agreement.
"""
con = sqlite3.connect(snapshot)
try:
rows = con.execute(
"SELECT t.symbol, count(o.id) FROM tickers t "
"LEFT JOIN ohlcv_records o ON o.ticker_id = t.id GROUP BY t.symbol"
).fetchall()
lo, hi = con.execute("SELECT min(date), max(date) FROM ohlcv_records").fetchone()
finally:
con.close()
with_bars = sorted(r[0].upper() for r in rows if r[1])
without_bars = sorted(r[0].upper() for r in rows if not r[1])
return with_bars, without_bars, date.fromisoformat(lo), date.fromisoformat(hi)
def ordered_list(
momentum: dict[str, float],
vol_percentiles: dict[str, float],
cutoff: float,
) -> list[str]:
"""Admitted set at the production cutoff, ordered by the production blend.
Symbols are fed to ``_percentiles`` in sorted order in both arms so that ties
resolve identically and cannot manufacture a difference.
"""
from app.services.momentum_service import _percentiles, blend_strategy_rank
percentiles = _percentiles({s: momentum[s] for s in sorted(momentum)})
admitted = [s for s, p in percentiles.items() if p >= cutoff]
scored = {
s: blend_strategy_rank(percentiles[s], vol_percentiles.get(s))
for s in admitted
}
return sorted(admitted, key=lambda s: (-(scored[s] or 0.0), s))
def mechanism_check(
series: dict[str, tuple[list[date], list[float], list[float]]],
sample_dates: list[date],
) -> dict:
"""Is the rank shift explained by dividend yield?
For each sampled decision date, correlate a name's trailing 12-month implied
yield — read straight off the ALL/SPLIT ratio, which is what separates the two
series — against how far its momentum percentile moves between the arms. A
strong positive rank correlation means the difference *is* the dividend, not
numerical noise.
"""
from app.services.momentum_service import _MOM_LOOKBACK, _percentiles, compute_residual_12_1_momentum
bench_dates, bench_split, bench_all = series[BENCHMARK]
bench_price = dict(zip(bench_dates, bench_split))
bench_tr = dict(zip(bench_dates, bench_all))
symbols = sorted(s for s in series if s != BENCHMARK)
index_of = {s: {d: i for i, d in enumerate(series[s][0])} for s in symbols}
correlations: list[float] = []
used_dates: list[str] = []
for as_of in sample_dates:
price_mom: dict[str, float] = {}
tr_mom: dict[str, float] = {}
yields: dict[str, float] = {}
for symbol in symbols:
idx = index_of[symbol].get(as_of)
if idx is None or idx < _MOM_LOOKBACK:
continue
dates, split_closes, all_closes = series[symbol]
wd, ws, wa = dates[: idx + 1], split_closes[: idx + 1], all_closes[: idx + 1]
p = compute_residual_12_1_momentum(wd, ws, bench_price)
t = compute_residual_12_1_momentum(wd, wa, bench_tr)
if p is None or t is None:
continue
price_mom[symbol] = p
tr_mom[symbol] = t
# Ratio drift over the formation window is the accumulated dividend.
r_now = wa[-1] / ws[-1] if ws[-1] else None
r_then = wa[-_MOM_LOOKBACK] / ws[-_MOM_LOOKBACK] if ws[-_MOM_LOOKBACK] else None
if r_now and r_then:
yields[symbol] = r_now / r_then - 1.0
common = sorted(set(price_mom) & set(yields))
if len(common) < 30:
continue
pp = _percentiles({s: price_mom[s] for s in common})
tp = _percentiles({s: tr_mom[s] for s in common})
shift = {s: tp[s] - pp[s] for s in common}
correlations.append(_spearman([yields[s] for s in common], [shift[s] for s in common]))
used_dates.append(as_of.isoformat())
return {
"what_it_measures": (
"Spearman correlation, per sampled date, between a name's trailing 12-month "
"implied yield (from the ALL/SPLIT ratio over the formation window) and the "
"change in its momentum percentile between the two arms."
),
"caveat": (
"Both the stock series and the benchmark leg change between arms, so this is "
"strong evidence consistent with a dividend effect - it does not isolate the "
"stock's own yield from the beta-weighted benchmark yield."
),
"formation_window_bars": _MOM_LOOKBACK,
"min_symbols_per_date": 30,
"sampled_dates": len(correlations),
"sample_date_list": used_dates,
"mean_spearman_yield_vs_percentile_shift": (
round(sum(correlations) / len(correlations), 4) if correlations else None
),
"min": round(min(correlations), 4) if correlations else None,
"max": round(max(correlations), 4) if correlations else None,
"per_date_spearman": [round(c, 4) for c in correlations],
}
def _spearman(x: list[float], y: list[float]) -> float:
def rank(v: list[float]) -> list[float]:
order = sorted(range(len(v)), key=lambda i: v[i])
out = [0.0] * len(v)
for position, i in enumerate(order):
out[i] = float(position)
return out
rx, ry = rank(x), rank(y)
n = len(rx)
mx, my = sum(rx) / n, sum(ry) / n
num = sum((rx[i] - mx) * (ry[i] - my) for i in range(n))
dx = sum((rx[i] - mx) ** 2 for i in range(n)) ** 0.5
dy = sum((ry[i] - my) ** 2 for i in range(n)) ** 0.5
return num / (dx * dy) if dx and dy else 0.0
def disagreement(a: list[str], b: list[str]) -> float:
sa, sb = set(a), set(b)
union = sa | sb
if not union:
return 0.0
return len(sa ^ sb) / len(union)
def run_screen(
series: dict[str, tuple[list[date], list[float], list[float]]],
cutoff: float,
*,
limit_dates: int | None = None,
verbose: bool = True,
) -> dict:
from app.services.momentum_service import (
_MOM_LOOKBACK,
compute_realized_vol_6m,
compute_residual_12_1_momentum,
_percentiles,
)
bench_dates, bench_split, bench_all = series[BENCHMARK]
bench_price = dict(zip(bench_dates, bench_split))
bench_tr = dict(zip(bench_dates, bench_all))
symbols = sorted(s for s in series if s != BENCHMARK)
index_of: dict[str, dict[date, int]] = {
s: {d: i for i, d in enumerate(series[s][0])} for s in symbols
}
evaluated = 0
skipped_no_history = 0
changed_sets = 0
changed_orders = 0
disagreements: list[float] = []
# Counted directly rather than derived from the Jaccard ratio: a swap moves
# two names (one out, one in), so the symmetric difference is twice the
# replacement count and the two are easy to conflate.
replacements: list[int] = []
only_price_counts: list[int] = []
only_tr_counts: list[int] = []
admitted_sizes: list[int] = []
per_year: dict[int, dict[str, int]] = {}
examples: list[dict] = []
decision_dates = [d for d in bench_dates]
if limit_dates:
decision_dates = decision_dates[-limit_dates:]
for as_of in decision_dates:
price_mom: dict[str, float] = {}
tr_mom: dict[str, float] = {}
vol_values: dict[str, float] = {}
for symbol in symbols:
idx = index_of[symbol].get(as_of)
if idx is None or idx < _MOM_LOOKBACK:
continue
dates, split_closes, all_closes = series[symbol]
window_dates = dates[: idx + 1]
window_split = split_closes[: idx + 1]
window_all = all_closes[: idx + 1]
price_value = compute_residual_12_1_momentum(window_dates, window_split, bench_price)
tr_value = compute_residual_12_1_momentum(window_dates, window_all, bench_tr)
# Equal eligibility: a name enters both arms or neither.
if price_value is None or tr_value is None:
continue
price_mom[symbol] = price_value
tr_mom[symbol] = tr_value
vol = compute_realized_vol_6m(window_split)
if vol is not None:
vol_values[symbol] = vol
if not price_mom:
skipped_no_history += 1
continue
evaluated += 1
vol_percentiles = _percentiles({s: vol_values[s] for s in sorted(vol_values)})
price_list = ordered_list(price_mom, vol_percentiles, cutoff)
tr_list = ordered_list(tr_mom, vol_percentiles, cutoff)
year = per_year.setdefault(as_of.year, {"dates": 0, "changed_sets": 0, "changed_orders": 0})
year["dates"] += 1
set_price, set_tr = set(price_list), set(tr_list)
only_price = len(set_price - set_tr)
only_tr = len(set_tr - set_price)
only_price_counts.append(only_price)
only_tr_counts.append(only_tr)
admitted_sizes.append(len(price_list))
# A replacement is one name leaving paired with one arriving; when the
# two arms admit different counts, the surplus is not a replacement.
replacements.append(min(only_price, only_tr))
same_set = set_price == set_tr
same_order = price_list == tr_list
if not same_set:
changed_sets += 1
year["changed_sets"] += 1
if not same_order:
changed_orders += 1
year["changed_orders"] += 1
if len(examples) < 10:
only_price = [s for s in price_list if s not in set(tr_list)]
only_tr = [s for s in tr_list if s not in set(price_list)]
examples.append({
"date": as_of.isoformat(),
"admitted_price_return": len(price_list),
"admitted_total_return": len(tr_list),
"only_in_price_return": only_price[:10],
"only_in_total_return": only_tr[:10],
"order_changed_without_membership_change": same_set,
"first_10_price_return": price_list[:10],
"first_10_total_return": tr_list[:10],
})
disagreements.append(disagreement(price_list, tr_list))
if verbose and evaluated % 100 == 0:
print(f" … {evaluated} dates evaluated, {changed_orders} with a different ordered list")
return {
"evaluated_dates": evaluated,
"skipped_dates_no_history": skipped_no_history,
"dates_with_different_admitted_set": changed_sets,
"dates_with_different_ordered_list": changed_orders,
"mean_set_disagreement": (sum(disagreements) / len(disagreements)) if disagreements else 0.0,
"max_set_disagreement": max(disagreements) if disagreements else 0.0,
"mean_admitted_per_arm": round(sum(admitted_sizes) / len(admitted_sizes), 2) if admitted_sizes else 0.0,
"mean_names_only_in_price_return": round(sum(only_price_counts) / len(only_price_counts), 2) if only_price_counts else 0.0,
"mean_names_only_in_total_return": round(sum(only_tr_counts) / len(only_tr_counts), 2) if only_tr_counts else 0.0,
"mean_replacements_per_date": round(sum(replacements) / len(replacements), 2) if replacements else 0.0,
"max_replacements_per_date": max(replacements) if replacements else 0,
"mean_symmetric_difference": round(
(sum(only_price_counts) + sum(only_tr_counts)) / len(only_price_counts), 2
) if only_price_counts else 0.0,
"per_year": {
str(y): v for y, v in sorted(per_year.items())
},
"examples": examples,
}
# ---------------------------------------------------------------------------
# driver
# ---------------------------------------------------------------------------
async def main(args: argparse.Namespace) -> int:
snapshot = Path(args.snapshot)
cache = Path(args.cache)
symbols, inert, window_start, window_end = snapshot_symbols(snapshot)
universe = symbols + [BENCHMARK]
print(f"snapshot : {snapshot.name} ({len(symbols)} symbols with bars"
+ (f", {len(inert)} inert: {', '.join(inert)}" if inert else "") + ")")
print(f"window : {window_start} .. {window_end} (clipped to the snapshot)")
if args.fetch:
await populate_cache(universe, window_start, window_end, cache)
return 0
series = load_series(cache, universe)
missing = [s for s in universe if s not in series]
if missing:
raise SystemExit(
f"{len(missing)} symbols have bars in the snapshot but none in the cache "
f"({', '.join(missing[:8])}). Run with --fetch first; an incomplete download "
"makes the screen inconclusive and must never be read as agreement."
)
from app.services.admin_service import ACTIVATION_DEFAULTS
cutoff = float(ACTIVATION_DEFAULTS["min_momentum_percentile"])
print(f"cutoff : {cutoff} (production min_momentum_percentile)")
if args.mechanism:
bench_dates = load_series(cache, [BENCHMARK])[BENCHMARK][0]
sample = bench_dates[::args.mechanism]
print(f"mechanism check over {len(sample)} sampled dates ...")
payload = {
"generated_at": datetime.now(timezone.utc).isoformat(),
"snapshot": str(snapshot),
"cache": str(cache),
"window": {"start": window_start.isoformat(), "end": window_end.isoformat()},
"universe_size": len(symbols),
"sample_every_n_sessions": args.mechanism,
"result": mechanism_check(series, sample),
}
out = Path(args.mechanism_out)
out.parent.mkdir(parents=True, exist_ok=True)
out.write_text(json.dumps(payload, indent=2) + "\n", encoding="utf-8")
r = payload["result"]
print(f" sampled dates : {r['sampled_dates']}")
print(f" mean Spearman : {r['mean_spearman_yield_vs_percentile_shift']} "
f"(min {r['min']}, max {r['max']})")
print(f" report : {out}")
return 0
if args.time_batch:
n = args.time_batch
started = time.perf_counter()
result = run_screen(series, cutoff, limit_dates=n, verbose=False)
elapsed = time.perf_counter() - started
total_dates = len(series[BENCHMARK][0])
per_date = elapsed / max(result["evaluated_dates"], 1)
print(f"\ntimed {result['evaluated_dates']} evaluated dates in {elapsed:.1f}s "
f"({per_date:.2f}s/date)")
print(f"projected full run over {total_dates} sessions: "
f"{per_date * total_dates / 60:.1f} min")
return 0
started = time.perf_counter()
result = run_screen(series, cutoff)
result["elapsed_seconds"] = round(time.perf_counter() - started, 1)
con = open_cache(cache)
coverage = {
r[0]: {"split_bars": r[1], "all_bars": r[2], "aligned_bars": r[3], "dropped_unmatched": r[4]}
for r in con.execute("SELECT symbol, split_bars, all_bars, aligned_bars, dropped_unmatched FROM fetched")
}
con.close()
verdict = (
"stop - no admission or ordering difference in this window"
if result["dates_with_different_ordered_list"] == 0
else "proceed to Step 3 - ordered lists differ"
)
payload = {
"generated_at": datetime.now(timezone.utc).isoformat(),
"snapshot": str(snapshot),
"cache": str(cache),
"window": {"start": window_start.isoformat(), "end": window_end.isoformat()},
"universe_size": len(symbols),
"inert_snapshot_symbols": inert,
"cutoff": cutoff,
"verdict": verdict,
"result": result,
"coverage": {
"symbols_cached": len(coverage),
"symbols_with_unmatched_dates": sum(1 for c in coverage.values() if c["dropped_unmatched"]),
"total_unmatched_dates": sum(c["dropped_unmatched"] for c in coverage.values()),
},
}
out = Path(args.out)
out.parent.mkdir(parents=True, exist_ok=True)
out.write_text(json.dumps(payload, indent=2) + "\n", encoding="utf-8")
print(f"\nevaluated dates : {result['evaluated_dates']}")
print(f"dates with different admitted set: {result['dates_with_different_admitted_set']}")
print(f"dates with different ordered list: {result['dates_with_different_ordered_list']}")
print(f"mean / max set disagreement : {result['mean_set_disagreement']:.4f} / {result['max_set_disagreement']:.4f}")
print(f"verdict : {verdict}")
print(f"report : {out}")
return 0
def parse_args() -> argparse.Namespace:
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
p.add_argument("--snapshot", default=str(DEFAULT_SNAPSHOT), help="Universe and date bounds come from this snapshot.")
p.add_argument("--cache", default=str(DEFAULT_CACHE))
p.add_argument("--fetch", action="store_true", help="Populate the cache and exit.")
p.add_argument("--time-batch", type=int, default=None, metavar="N",
help="Time the screen over the last N sessions and project the full runtime.")
p.add_argument("--mechanism", type=int, default=None, metavar="EVERY_N",
help="Correlate trailing dividend yield against the percentile shift on every Nth session.")
p.add_argument("--out", default="reports/tr-rank-agreement.json")
p.add_argument("--mechanism-out", default="reports/tr-rank-agreement-mechanism.json")
return p.parse_args()
if __name__ == "__main__":
raise SystemExit(asyncio.run(main(parse_args())))
scripts/run_total_return_ab.py
+544
View File
@@ -0,0 +1,544 @@
"""Paired portfolio replay: does the momentum basis change actual trades?
Step 3a of docs/data-integrity-plan.md. The Step 2 screen showed the ranking
changes every day; this measures whether that reaches admissions and P&L. No
schema, no ingestion — bars come from the survivorship-a snapshot and the two
adjusted series from the Step 2 cache.
What is held identical between the arms
---------------------------------------
* **Setup candidates are generated once** and reused by both books. Entry, stop,
target and R:R never differ. Only momentum and what depends on it —
the residual/activation percentiles, `qualified`, and the 80/20 blend used for
ordering — are recomputed per arm.
* **Both arms read the same cache**: control uses SPLIT stock against SPLIT SPY,
treatment uses ALL stock against ALL SPY. Neither reads a freshly fetched
series, so a refreshed-vs-snapshot difference cannot be mistaken for a basis
difference.
* **Execution, volatility, gates, capacity, exits**: snapshot nominal bars in both
arms; the 6-month volatility leg comes from candidate generation and is never
recomputed; non-momentum floors, `max_positions`, risk per trade and the exit
policy are the production values for both.
Each book then keeps its **own** holdings: a divergent admission changes which
slots are occupied later, and that path dependence is part of the measurement.
Reading the output
------------------
Different trades establish that the basis is production-relevant. They do not
authorise a rollout — that is a separate non-inferiority decision taken on the
paired numbers.
Usage::
python scripts/run_total_return_ab.py --build-candidates # one-time, cached
python scripts/run_total_return_ab.py
"""
from __future__ import annotations
import argparse
import asyncio
import json
import pickle
import sqlite3
import sys
import time
from concurrent.futures import ProcessPoolExecutor
from datetime import date, datetime, timezone
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
if str(ROOT) not in sys.path:
sys.path.insert(0, str(ROOT))
DEFAULT_SNAPSHOT = ROOT / "backtest_snapshots" / "survivorship-a.sqlite"
DEFAULT_CACHE = ROOT / "backtest_snapshots" / "tr-rank-cache.sqlite"
DEFAULT_CANDIDATES = ROOT / "backtest_snapshots" / "tr-ab-candidates.pkl"
BENCHMARK = "SPY"
def _sqlite_url(path: Path) -> str:
return f"sqlite+aiosqlite:///{path.resolve().as_posix()}"
# ---------------------------------------------------------------------------
# candidates — generated once, reused by both arms
# ---------------------------------------------------------------------------
async def build_candidates(snapshot: Path, out: Path, cadence: str, workers: int) -> None:
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker, create_async_engine
from app.models.ticker import Ticker
from app.services import backtest_service as bt
from app.services.admin_service import get_activation_config
from app.services.recommendation_service import get_recommendation_config
engine = create_async_engine(_sqlite_url(snapshot), pool_pre_ping=True)
Session = async_sessionmaker(engine, class_=AsyncSession, expire_on_commit=False)
candidates: list[dict] = []
prices: dict[str, tuple] = {}
async with Session() as db:
config = await get_recommendation_config(db)
activation = await get_activation_config(db)
symbols = list((await db.execute(select(Ticker.symbol).order_by(Ticker.symbol))).scalars())
benchmark = await bt._load_benchmark_closes_for_backtest(db, refresh=False)
print(f"generating candidates for {len(symbols)} symbols (cadence={cadence}) …")
columns_by_symbol: dict[str, tuple] = {}
for symbol in symbols:
cols = await bt._fetch_columns(db, symbol)
if cols is not None:
columns_by_symbol[symbol] = cols
await engine.dispose()
started = time.perf_counter()
ctx = bt._mp_context()
pool = None
if ctx is not None and workers > 1:
try:
pool = ProcessPoolExecutor(max_workers=workers, mp_context=ctx)
except Exception: # noqa: BLE001 - fall back to in-process replay
pool = None
items = list(columns_by_symbol.items())
if pool is not None:
with pool:
futures = {
symbol: pool.submit(
bt._replay_and_signals, symbol, cols, config, activation,
benchmark, bt.PRODUCTION_GTL_TARGET_MODEL, cadence, False, None,
)
for symbol, cols in items
}
for index, (symbol, future) in enumerate(futures.items(), 1):
got, _ = future.result()
candidates.extend(got)
prices[symbol] = columns_by_symbol[symbol]
if index % 100 == 0:
print(f" … {index}/{len(items)}")
else:
for index, (symbol, cols) in enumerate(items, 1):
got, _ = bt._replay_and_signals(
symbol, cols, config, activation, benchmark,
bt.PRODUCTION_GTL_TARGET_MODEL, cadence, False, None,
)
candidates.extend(got)
prices[symbol] = cols
if index % 100 == 0:
print(f" … {index}/{len(items)}")
payload = {
"generated_at": datetime.now(timezone.utc).isoformat(),
"snapshot": str(snapshot),
"cadence": cadence,
"config": config,
"activation": activation,
"candidates": candidates,
"prices": prices,
"benchmark": benchmark,
}
out.write_bytes(pickle.dumps(payload))
print(f"{len(candidates)} candidates over {len(prices)} symbols "
f"in {time.perf_counter() - started:.0f}s -> {out.name}")
# ---------------------------------------------------------------------------
# momentum, recomputed per arm from the cache
# ---------------------------------------------------------------------------
def load_cache_series(cache: Path) -> dict[str, tuple[list[date], list[float], list[float]]]:
con = sqlite3.connect(cache)
try:
rows = con.execute("SELECT symbol, date, split_close, all_close FROM bars ORDER BY symbol, date")
out: dict[str, tuple[list, list, list]] = {}
for symbol, day, split_close, all_close in rows:
dates, splits, alls = out.setdefault(symbol, ([], [], []))
dates.append(date.fromisoformat(day))
splits.append(float(split_close))
alls.append(float(all_close))
finally:
con.close()
return out
def momentum_by_arm(
series: dict[str, tuple[list[date], list[float], list[float]]],
signal_dates: dict[str, set[date]],
*,
total_return: bool,
) -> dict[tuple[str, date], float]:
"""Residual 12-1 momentum at every (symbol, signal date) for one arm.
Control reads the SPLIT series against SPLIT SPY; treatment reads ALL against
ALL SPY. Both use the unmodified production function and only bars at or
before the signal date.
"""
from app.services.momentum_service import _MOM_LOOKBACK, compute_residual_12_1_momentum
bench_dates, bench_split, bench_all = series[BENCHMARK]
benchmark = dict(zip(bench_dates, bench_all if total_return else bench_split))
out: dict[tuple[str, date], float] = {}
for symbol, wanted in signal_dates.items():
entry = series.get(symbol)
if entry is None:
continue
dates, splits, alls = entry
closes = alls if total_return else splits
index_of = {d: i for i, d in enumerate(dates)}
for as_of in wanted:
idx = index_of.get(as_of)
if idx is None or idx < _MOM_LOOKBACK:
continue
value = compute_residual_12_1_momentum(
dates[: idx + 1], closes[: idx + 1], benchmark
)
if value is not None:
out[(symbol, as_of)] = value
return out
def apply_arm(candidates: list[dict], momentum: dict[tuple[str, date], float], min_pct: float) -> list[dict]:
"""Rebuild only the momentum-dependent fields, using the production pipeline."""
from app.services import backtest_service as bt
rebuilt = []
for cand in candidates:
clone = dict(cand)
key = (clone["symbol"], date.fromisoformat(clone["date"]))
clone["residual_momentum"] = momentum.get(key)
rebuilt.append(clone)
bt._assign_momentum_percentiles(rebuilt)
bt._assign_residual_momentum_percentiles(rebuilt)
bt._assign_low_volatility_percentiles(rebuilt) # vol_6m untouched, so identical in both arms
bt._assign_activation_momentum_percentiles(rebuilt)
bt._assign_residual_low_vol_blend(rebuilt)
bt._assign_residual_high_vol_blend(rebuilt)
for cand in rebuilt:
cand["qualified"] = bt._momentum_qualifies(cand, min_pct)
return rebuilt
def simulate(candidates: list[dict], prices: dict, benchmark: dict, cadence: str) -> dict:
"""Run the production book: live gate, 80/20 ordering, ATR trail, gate reset."""
from app.services import backtest_service as bt
strategy = next(s for s in bt.PORTFOLIO_MONITOR_STRATEGIES if s.get("is_production"))
entry_cfg = bt._entry_variant_config(str(strategy["entry_variant"]))
ranking_key = str(strategy.get("ranking_key") or entry_cfg.get("ranking_key"))
reentry_fn = bt._make_gate_reset_reentry_fn(
candidates, prices, cadence=cadence, qualified_fn=None, ranking_key=ranking_key,
)
return bt._simulate_portfolio(
candidates,
prices,
benchmark,
str(strategy["exit_policy"]),
max(bt.TIME_EXIT_DAYS),
qualified_fn=None, # live qualification, as production
ranking_key=ranking_key,
max_positions=int(entry_cfg["max_positions"]),
risk_per_trade=float(entry_cfg["risk_per_trade"]),
atr_trail_multiplier=bt.ATR_TRAIL_MULTIPLIER,
post_stop_reentry_fn=reentry_fn,
include_curve=True,
include_trades=True,
)
# ---------------------------------------------------------------------------
# comparison
# ---------------------------------------------------------------------------
def entry_diff(control: dict, treatment: dict) -> dict:
def entries(sim: dict) -> set[tuple[str, str]]:
return {(t["entry_date"], t["symbol"]) for t in sim.get("trade_details", [])}
a, b = entries(control), entries(treatment)
only_a = sorted(a - b)
only_b = sorted(b - a)
return {
"shared_entries": len(a & b),
"only_control": [{"entry_date": d, "symbol": s} for d, s in only_a],
"only_treatment": [{"entry_date": d, "symbol": s} for d, s in only_b],
"only_control_count": len(only_a),
"only_treatment_count": len(only_b),
}
def yearly_comparison(control: dict, treatment: dict) -> dict:
ca = {row["year"]: row for row in (control.get("yearly_returns") or [])}
tb = {row["year"]: row for row in (treatment.get("yearly_returns") or [])}
years = sorted(set(ca) | set(tb))
rows = []
better = worse = same = 0
for year in years:
c = (ca.get(year) or {}).get("return_pct")
t = (tb.get(year) or {}).get("return_pct")
delta = None
if c is not None and t is not None:
delta = round(t - c, 2)
if delta > 0:
better += 1
elif delta < 0:
worse += 1
else:
same += 1
rows.append({"year": year, "control_pct": c, "treatment_pct": t, "delta_pp": delta})
return {
"rows": rows,
"years_treatment_better": better,
"years_treatment_worse": worse,
"years_equal": same,
}
HEADLINE = ("trades", "total_return_pct", "cagr_pct", "max_drawdown_pct", "sharpe", "win_rate")
# Pre-specified before the diagnostic was run, and reported for BOTH block
# lengths rather than whichever reads better.
BOOTSTRAP_SEED = 20260907
BOOTSTRAP_REPLICATES = 5000
BOOTSTRAP_BLOCKS = (21, 63)
TRADING_DAYS = 252
def daily_returns(curve: list[dict]) -> tuple[list[str], list[float]]:
"""Daily returns from an equity curve, with their dates."""
dates, rets = [], []
for previous, current in zip(curve, curve[1:]):
prior = float(previous["equity"])
if prior > 0:
dates.append(current["date"])
rets.append(float(current["equity"]) / prior - 1.0)
return dates, rets
def _path_stats(returns) -> tuple[float, float, float]:
"""(CAGR %, Sharpe, max drawdown %) for one resampled return path."""
import math
n = len(returns)
equity = 1.0
peak = 1.0
max_dd = 0.0
total = 0.0
total_sq = 0.0
for r in returns:
equity *= 1.0 + r
peak = max(peak, equity)
if peak > 0:
max_dd = max(max_dd, 1.0 - equity / peak)
total += r
total_sq += r * r
mean = total / n
var = max(total_sq / n - mean * mean, 0.0)
sd = math.sqrt(var)
cagr = (equity ** (TRADING_DAYS / n) - 1.0) * 100.0 if equity > 0 else -100.0
sharpe = (mean / sd * math.sqrt(TRADING_DAYS)) if sd > 0 else 0.0
return cagr, sharpe, max_dd * 100.0
def paired_block_bootstrap(
control_returns: list[float],
treatment_returns: list[float],
*,
block: int,
replicates: int,
seed: int,
) -> dict:
"""Resample identical date blocks for both arms and difference the results.
The arms share dates and most positions, so their difference is what needs an
interval — not each arm separately. Drawing the *same* block indices for both
preserves the pairing and whatever cross-arm correlation exists on those days.
Moving blocks retain some of the serial dependence that an i.i.d. resample
would destroy; they do not make this out-of-sample.
"""
import random
import statistics
rng = random.Random(seed)
n = len(control_returns)
n_blocks = -(-n // block) # ceil
max_start = n - block
deltas = {"cagr_pct": [], "sharpe": [], "max_drawdown_pct": []}
for _ in range(replicates):
starts = [rng.randint(0, max_start) for _ in range(n_blocks)]
idx = [i for s in starts for i in range(s, s + block)][:n]
c = _path_stats([control_returns[i] for i in idx])
t = _path_stats([treatment_returns[i] for i in idx])
deltas["cagr_pct"].append(t[0] - c[0])
deltas["sharpe"].append(t[1] - c[1])
deltas["max_drawdown_pct"].append(t[2] - c[2])
def interval(values: list[float]) -> dict:
ordered = sorted(values)
def q(p: float) -> float:
return ordered[min(len(ordered) - 1, max(0, int(round(p * (len(ordered) - 1)))))]
return {
"mean": round(statistics.fmean(ordered), 3),
"median": round(q(0.50), 3),
"ci_2.5": round(q(0.025), 3),
"ci_97.5": round(q(0.975), 3),
"share_below_zero": round(sum(1 for v in ordered if v < 0) / len(ordered), 4),
}
return {metric: interval(values) for metric, values in deltas.items()}
async def main(args: argparse.Namespace) -> int:
snapshot = Path(args.snapshot)
cache = Path(args.cache)
store = Path(args.candidates)
if args.build_candidates or not store.exists():
if not args.build_candidates:
print(f"{store.name} missing — building it first")
await build_candidates(snapshot, store, args.cadence, args.workers)
if args.build_candidates:
return 0
payload = pickle.loads(store.read_bytes())
candidates = payload["candidates"]
prices = payload["prices"]
benchmark = payload["benchmark"]
min_pct = float(payload["activation"].get("min_momentum_percentile", 80.0))
cadence = payload["cadence"]
signal_dates: dict[str, set[date]] = {}
for cand in candidates:
signal_dates.setdefault(cand["symbol"], set()).add(date.fromisoformat(cand["date"]))
series = load_cache_series(cache)
missing = sorted({s for s in signal_dates if s not in series})
if missing:
raise SystemExit(
f"{len(missing)} symbols have candidates but no cached series "
f"({', '.join(missing[:8])}). Run research_tr_rank_agreement.py --fetch."
)
print(f"candidates: {len(candidates)} over {len(signal_dates)} symbols, cutoff {min_pct}")
results = {}
for label, total_return in (("control_price_return", False), ("treatment_total_return", True)):
started = time.perf_counter()
momentum = momentum_by_arm(series, signal_dates, total_return=total_return)
arm = apply_arm(candidates, momentum, min_pct)
sim = simulate(arm, prices, benchmark, cadence)
results[label] = sim
print(f" {label:<24} trades={sim['trades']:<5} CAGR={sim['cagr_pct']}% "
f"Sharpe={sim['sharpe']} DD={sim['max_drawdown_pct']}% "
f"({time.perf_counter() - started:.0f}s)")
control = results["control_price_return"]
treatment = results["treatment_total_return"]
diff = entry_diff(control, treatment)
yearly = yearly_comparison(control, treatment)
# Paired uncertainty on the DIFFERENCE. Per-arm standard errors describe each
# book alone; these arms share dates and most positions, so only a paired
# resample speaks to the gap between them.
control_dates, control_rets = daily_returns(control.get("equity_curve") or [])
treatment_dates, treatment_rets = daily_returns(treatment.get("equity_curve") or [])
if control_dates != treatment_dates:
raise SystemExit(
"Arm return streams are not date-aligned; a paired bootstrap would "
"resample mismatched days."
)
bootstrap = {
"method": (
"paired moving-block bootstrap on date-aligned daily returns; the same "
"block indices are drawn for both arms so the pairing and cross-arm "
"correlation are preserved"
),
"seed": BOOTSTRAP_SEED,
"replicates": BOOTSTRAP_REPLICATES,
"sessions": len(control_rets),
"pre_specified": (
"seed, replicate count and both block lengths were fixed before the run; "
"both are reported"
),
"scope": (
"exploratory uncertainty estimate from the observed window - not "
"out-of-sample validation and not a rollout gate"
),
"blocks": {},
}
for block in BOOTSTRAP_BLOCKS:
started = time.perf_counter()
bootstrap["blocks"][f"{block}_sessions"] = paired_block_bootstrap(
control_rets, treatment_rets,
block=block, replicates=BOOTSTRAP_REPLICATES, seed=BOOTSTRAP_SEED,
)
print(f" bootstrap block={block:<3} {BOOTSTRAP_REPLICATES} reps "
f"({time.perf_counter() - started:.0f}s)")
out = Path(args.out)
out.parent.mkdir(parents=True, exist_ok=True)
out.write_text(json.dumps({
"generated_at": datetime.now(timezone.utc).isoformat(),
"snapshot": str(snapshot),
"cache": str(cache),
"candidates_store": str(store),
"cadence": cadence,
"min_momentum_percentile": min_pct,
"candidate_count": len(candidates),
"held_fixed": [
"setup candidates (entry/stop/target/rr) generated once and shared",
"execution bars: snapshot nominal closes in both arms",
"volatility leg: vol_6m from candidate generation, never recomputed",
"non-momentum gates, max_positions, risk_per_trade, exit policy, re-entry policy",
],
"headline": {
label: {k: sim.get(k) for k in HEADLINE} for label, sim in results.items()
},
"deltas": {
k: (
round(treatment[k] - control[k], 2)
if isinstance(treatment.get(k), (int, float)) and isinstance(control.get(k), (int, float))
else None
)
for k in HEADLINE
},
"entry_diff": diff,
"yearly": yearly,
"paired_bootstrap": bootstrap,
"arms": {label: {k: v for k, v in sim.items() if k not in {"equity_curve", "benchmark_curve"}} for label, sim in results.items()},
}, indent=2, default=str) + "\n", encoding="utf-8")
print(f"\nentries only in control : {diff['only_control_count']}")
print(f"entries only in treatment: {diff['only_treatment_count']}")
print(f"shared entries : {diff['shared_entries']}")
print(f"years treatment better : {yearly['years_treatment_better']} "
f"/ worse {yearly['years_treatment_worse']} / equal {yearly['years_equal']}")
print("\npaired block bootstrap, treatment - control (95% interval):")
for name, res in bootstrap["blocks"].items():
print(f" block {name}")
for metric, stats in res.items():
print(f" {metric:<20} mean {stats['mean']:>8} "
f"[{stats['ci_2.5']:>8}, {stats['ci_97.5']:>8}] "
f"P(<0)={stats['share_below_zero']}")
print(f"report : {out}")
return 0
def parse_args() -> argparse.Namespace:
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
p.add_argument("--snapshot", default=str(DEFAULT_SNAPSHOT))
p.add_argument("--cache", default=str(DEFAULT_CACHE))
p.add_argument("--candidates", default=str(DEFAULT_CANDIDATES))
p.add_argument("--cadence", default="weekly", choices=("weekly", "daily"))
p.add_argument("--workers", type=int, default=6)
p.add_argument("--build-candidates", action="store_true", help="Build the candidate store and exit.")
p.add_argument("--out", default="reports/tr-total-return-ab.json")
return p.parse_args()
if __name__ == "__main__":
raise SystemExit(asyncio.run(main(parse_args())))
tests/unit/test_total_return_ab.py
+186
View File
@@ -0,0 +1,186 @@
"""Guards for the paired total-return A/B replay.
Two things have to hold before any difference the A/B reports means anything:
1. **Identical inputs must produce identical books.** If the harness were
nondeterministic — dict ordering, tie-breaks, path-dependent holdings — a
difference between the arms could come from the replay rather than the
momentum basis. Feeding both arms the same series must give byte-identical
results.
2. **A real momentum difference must be able to change an admission.** A harness
that holds everything so tightly it can never diverge would report agreement
no matter what. One name pushed across the cutoff must actually change who
gets admitted.
"""
from __future__ import annotations
import importlib.util
from datetime import date, timedelta
from pathlib import Path
import pytest
# Imported at module scope like the other backtest tests: loading the service
# lazily inside the first test leaves conftest's in-memory engine without its
# schema for every test after it.
from app.services import backtest_service as _bt # noqa: F401
ROOT = Path(__file__).resolve().parents[2]
_spec = importlib.util.spec_from_file_location(
"run_total_return_ab", ROOT / "scripts" / "run_total_return_ab.py"
)
ab = importlib.util.module_from_spec(_spec)
_spec.loader.exec_module(ab)
def _sessions(n: int, start: date = date(2021, 1, 4)) -> list[date]:
days, day = [], start
while len(days) < n:
if day.weekday() < 5:
days.append(day)
day += timedelta(days=1)
return days
def _series(n: int = 400, drift: float = 0.30) -> dict:
"""A cache-shaped series set: three names plus SPY, all with clean history."""
days = _sessions(n)
out = {}
for i, symbol in enumerate(("AAA", "BBB", "CCC")):
splits = [100.0 * (1.0 + (drift + i * 0.10) * k / n) for k in range(n)]
out[symbol] = (days, splits, list(splits)) # ALL == SPLIT: no dividends
spy = [100.0 * (1.0 + 0.10 * k / n) for k in range(n)]
out[ab.BENCHMARK] = (days, spy, list(spy))
return out
def _candidates(days: list[date], signal_index: int) -> list[dict]:
"""One qualifying long setup per symbol on a single signal date."""
as_of = days[signal_index]
out = []
for symbol in ("AAA", "BBB", "CCC"):
out.append({
"symbol": symbol,
"date": as_of.isoformat(),
# Real candidates carry both; the percentile pass groups on them.
"ranking_period": as_of.isocalendar()[:2],
"iso_week": as_of.isocalendar()[:2],
"direction": "long",
"entry": 100.0,
"stop": 95.0,
"target": 110.0,
"rr": 2.0,
"vol_6m": 0.02,
"residual_momentum": 0.0,
"momentum": 0.0,
"meets_core": True,
"qualified": True,
})
return out
def test_identical_series_give_identical_momentum_and_ordering():
"""ALL == SPLIT everywhere, so the two arms must agree exactly — any
difference the real run reports is then attributable to the basis."""
series = _series()
days = series["AAA"][0]
signal_dates = {s: {days[-1]} for s in ("AAA", "BBB", "CCC")}
control = ab.momentum_by_arm(series, signal_dates, total_return=False)
treatment = ab.momentum_by_arm(series, signal_dates, total_return=True)
assert control == treatment
assert control, "fixture must produce momentum values, else the test is vacuous"
cands = _candidates(days, len(days) - 1)
a = ab.apply_arm(cands, control, 80.0)
b = ab.apply_arm(cands, treatment, 80.0)
assert a == b
def test_entry_diff_is_empty_for_identical_books():
sim = {"trade_details": [
{"entry_date": "2022-01-04", "symbol": "AAA"},
{"entry_date": "2022-02-01", "symbol": "BBB"},
]}
diff = ab.entry_diff(sim, dict(sim))
assert diff["only_control_count"] == 0
assert diff["only_treatment_count"] == 0
assert diff["shared_entries"] == 2
def test_a_momentum_change_moves_a_name_across_the_cutoff():
"""The harness must be able to see a real difference: lifting one name's
momentum above its peers changes which names clear the gate."""
series = _series()
days = series["AAA"][0]
cands = _candidates(days, len(days) - 1)
as_of = days[-1]
# CCC ranks top on the control momentum, AAA bottom.
control_mom = {("AAA", as_of): 0.10, ("BBB", as_of): 0.20, ("CCC", as_of): 0.30}
# Treatment reverses it — as a dividend-heavy name overtaking a growth name would.
treatment_mom = {("AAA", as_of): 0.30, ("BBB", as_of): 0.20, ("CCC", as_of): 0.10}
a = ab.apply_arm(cands, control_mom, 80.0)
b = ab.apply_arm(cands, treatment_mom, 80.0)
admitted_a = {c["symbol"] for c in a if c["qualified"]}
admitted_b = {c["symbol"] for c in b if c["qualified"]}
assert admitted_a == {"CCC"}
assert admitted_b == {"AAA"}
assert admitted_a != admitted_b
def test_the_volatility_leg_is_untouched_by_the_arm():
"""Only momentum may vary; vol_6m carries through both arms unchanged, so the
80/20 blend differs solely through its momentum leg."""
series = _series()
days = series["AAA"][0]
as_of = days[-1]
cands = _candidates(days, len(days) - 1)
for index, cand in enumerate(cands):
cand["vol_6m"] = 0.01 * (index + 1)
a = ab.apply_arm(cands, {("AAA", as_of): 0.1, ("BBB", as_of): 0.2, ("CCC", as_of): 0.3}, 80.0)
b = ab.apply_arm(cands, {("AAA", as_of): 0.3, ("BBB", as_of): 0.2, ("CCC", as_of): 0.1}, 80.0)
from app.services.backtest_service import VOL_PERCENTILE_KEY
assert [c[VOL_PERCENTILE_KEY] for c in a] == [c[VOL_PERCENTILE_KEY] for c in b]
assert [c["vol_6m"] for c in a] == [c["vol_6m"] for c in b]
def test_missing_momentum_leaves_a_candidate_unqualified():
"""A name with no computable momentum in an arm must not sneak through the
gate on a stale value carried over from candidate generation."""
series = _series()
days = series["AAA"][0]
as_of = days[-1]
cands = _candidates(days, len(days) - 1)
arm = ab.apply_arm(cands, {("AAA", as_of): 0.5}, 80.0)
by_symbol = {c["symbol"]: c for c in arm}
assert by_symbol["BBB"]["residual_momentum"] is None
assert by_symbol["CCC"]["residual_momentum"] is None
def test_yearly_comparison_counts_years_treatment_wins():
control = {"yearly_returns": [
{"year": 2022, "return_pct": 10.0},
{"year": 2023, "return_pct": 20.0},
{"year": 2024, "return_pct": 5.0},
]}
treatment = {"yearly_returns": [
{"year": 2022, "return_pct": 12.0},
{"year": 2023, "return_pct": 18.0},
{"year": 2024, "return_pct": 5.0},
]}
out = ab.yearly_comparison(control, treatment)
assert out["years_treatment_better"] == 1
assert out["years_treatment_worse"] == 1
assert out["years_equal"] == 1
assert out["rows"][0]["delta_pp"] == pytest.approx(2.0)