dennisthiessen/signal-platform
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

research: survivorship sensitivity and total-return momentum investigation

Browse Source 
Three questions answered against snapshot copies, with no production change.

Step 1 — how sensitive is the backtest to the ranking universe? Restoring 85
S&P 500 constituents removed between 2021-06 and 2026-07 moves the same window
from 2.09 Sharpe / 51.6% CAGR to 1.57 / 38.1%. The failed banks never reached
either book (identical best/worst trade R), so this measures universe
recomposition, not survivorship bias. The removals list is curated by hand with
per-symbol sources; only three explicitly confirmed terminal events get marks,
and four unresolved endings are reported as caveats rather than inferred.

Step 2 — does a total-return basis change the ranking? Ordered lists differ on
1009 of 1009 evaluated dates, 4.56 replacements per date. The shift correlates
with trailing implied yield (mean Spearman +0.79), which is strong evidence
consistent with a dividend effect but does not isolate own-yield, since both
the stock and benchmark legs change.

Step 3a — does that reach P&L? A paired replay reusing one shared candidate set
(control reproduces the production row exactly) changes ~30% of entries.
Treatment is -3.7pp CAGR and -1.9pp drawdown, with the entire return gap in
2025. A paired moving-block bootstrap (seed 20260907, 5000 replicates, 21/63
blocks, both reported) puts every interval across zero.

Decision: production stays unchanged. No schema, no ingestion, no wider sweep.
The observed window does not resolve which basis is better, and choosing a
threshold now — with -3.7pp on the table — would be fitting the rule to the
answer.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HAH7vVtn7qbcn1EVheecHQ
This commit is contained in:
dennisthiessenandClaude Opus 5 committed 2026-09-07 19:29:34 +02:00
1 parent 8453b87290
commit ebe9047403
16 files changed
+442522
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

scripts/data/sp500_removals.json
+1774
View File
File diff suppressed because it is too large. Load diff
scripts/prep_survivorship_snapshot.py
+477
View File
@@ -0,0 +1,477 @@
"""Build the two survivorship-audit snapshots from a base snapshot.
Arm A is the universe the backtest already runs on. Arm B is the same universe
plus the S&P 500 constituents removed while the snapshot window was open, so the
strategy can trade the names that later left the index — including the ones that
failed.
The base file is never modified: both arms are copies, and the base hash is
re-checked at the end.
Three things this script has to get right, all of them learned from the data:
* **Schema drift.** The snapshot files on disk predate migrations 026 and 032,
so ``select(Ticker)`` against them fails with ``no such column: tickers.cik``
— and that is the path ``backtest_service._fetch_columns`` takes. Both copies
are synced against ``Base.metadata`` before anything else touches them.
* **Frozen tails.** Alpaca does not end a halted symbol's series; it repeats the
last price at zero volume. SBNY sits at 70.00 for 509 sessions after the FDIC
closed it, and INFO sits at 108.61 for 1,090 sessions before a different
company reuses the ticker. Only the symbols with a *reviewed cutoff* in the
dataset are truncated. Everywhere else a long zero-volume run is a tripwire:
the symbol is left out and reported for review rather than silently cut.
* **Terminal marks.** A failed bank stops trading mid-collapse, so the exchange
bars never show the loss. A mark is written only where the dataset carries an
explicit confirmation *and* the retained final bar matches the reviewed
``last_real_bar``. The mark is an assumption, not an observed execution.
Nothing is inferred. ``delisted_on`` / ``delisted_reason`` are written only for
confirmed terminal events; a series that merely ends early leaves them NULL and
is reported as an unresolved ending status.
Usage::
python scripts/prep_survivorship_snapshot.py
python scripts/prep_survivorship_snapshot.py --base backtest_snapshots/prod.sqlite --force
"""
from __future__ import annotations
import argparse
import asyncio
import hashlib
import json
import shutil
import sqlite3
import sys
from datetime import date, datetime, time, timedelta, timezone
from pathlib import Path
from typing import Any
ROOT = Path(__file__).resolve().parents[1]
if str(ROOT) not in sys.path:
sys.path.insert(0, str(ROOT))
# Bars are cut at the first run of this many consecutive zero-volume sessions.
ZERO_VOLUME_RUN = 5
# Close assigned to a confirmed terminal event. Not zero: a zero price divides
# through several return calculations, and 1 cent is a near-total loss either way.
TERMINAL_CLOSE = 0.01
DEFAULT_BASE = ROOT / "backtest_snapshots" / "prod.sqlite"
DEFAULT_REMOVALS = ROOT / "scripts" / "data" / "sp500_removals.json"
# ---------------------------------------------------------------------------
# schema sync
# ---------------------------------------------------------------------------
def sync_schema(path: Path) -> dict[str, list[str]]:
"""Add every column the models declare that the snapshot file lacks.
Nullable and untyped-default only: this brings an old snapshot up to the
current models without inventing data. Tables the snapshot does not have at
all are left alone — a snapshot is a subset of production by design.
"""
from sqlalchemy.dialects import sqlite as sqlite_dialect
from app.database import Base
import app.models # noqa: F401 (registers every model on Base.metadata)
dialect = sqlite_dialect.dialect()
added: dict[str, list[str]] = {}
con = sqlite3.connect(path)
try:
present = {r[0] for r in con.execute("SELECT name FROM sqlite_master WHERE type='table'")}
for table in Base.metadata.sorted_tables:
if table.name not in present:
continue
have = {r[1] for r in con.execute(f'PRAGMA table_info("{table.name}")')}
for column in table.columns:
if column.name in have:
continue
type_sql = column.type.compile(dialect)
con.execute(f'ALTER TABLE "{table.name}" ADD COLUMN "{column.name}" {type_sql}')
added.setdefault(table.name, []).append(column.name)
con.commit()
finally:
con.close()
return added
# ---------------------------------------------------------------------------
# price data
# ---------------------------------------------------------------------------
def frozen_run_start(bars: list[Any]) -> int | None:
"""Index where the first run of >= ZERO_VOLUME_RUN zero-volume bars begins.
A detector, not a cutter: the caller decides what to do with the answer, and
only a reviewed cutoff is allowed to shorten a series.
"""
volumes = [int(b.volume) for b in bars]
i = 0
while i < len(volumes):
if volumes[i] == 0:
j = i
while j < len(volumes) and volumes[j] == 0:
j += 1
if j - i >= ZERO_VOLUME_RUN:
return i
i = j
else:
i += 1
return None
def apply_reviewed_cutoff(bars: list[Any], cutoff: date) -> tuple[list[Any], int]:
"""Keep bars up to and including *cutoff*. Returns (kept, dropped)."""
kept = [b for b in bars if b.timestamp.date() <= cutoff]
return kept, len(bars) - len(kept)
async def fetch_bars(symbol: str, start: date, end: date) -> list[Any]:
"""Daily SPLIT-adjusted bars, matching how the snapshot's existing bars were ingested."""
from app.config import settings
from app.providers.alpaca import AlpacaOHLCVProvider
if not settings.alpaca_api_key or not settings.alpaca_api_secret:
raise SystemExit("Alpaca credentials are not configured; cannot restore price history.")
from alpaca.data.enums import Adjustment
from alpaca.data.requests import StockBarsRequest
from alpaca.data.timeframe import TimeFrame
provider = AlpacaOHLCVProvider(settings.alpaca_api_key, settings.alpaca_api_secret)
alpaca_symbol = provider._to_alpaca_symbol(symbol)
request = StockBarsRequest(
symbol_or_symbols=alpaca_symbol,
timeframe=TimeFrame.Day,
start=datetime.combine(start, time.min, tzinfo=timezone.utc),
end=datetime.combine(end + timedelta(days=1), time.min, tzinfo=timezone.utc),
adjustment=Adjustment.SPLIT,
)
# 85 sequential requests are enough for Alpaca to drop one connection, and a
# half-built arm is worse than a slow one.
last_error: Exception | None = None
for attempt in range(4):
try:
bars = await asyncio.to_thread(provider._client.get_stock_bars, request)
return list(bars.data.get(alpaca_symbol, []))
except Exception as exc: # noqa: BLE001 - retry any transport failure
last_error = exc
if attempt < 3:
await asyncio.sleep(2 * (attempt + 1))
raise SystemExit(f"Alpaca fetch failed for {symbol} after 4 attempts: {last_error}")
# ---------------------------------------------------------------------------
# snapshot writes
# ---------------------------------------------------------------------------
def snapshot_window(path: Path) -> tuple[date, date]:
con = sqlite3.connect(path)
try:
lo, hi = con.execute("SELECT min(date), max(date) FROM ohlcv_records").fetchone()
finally:
con.close()
return date.fromisoformat(lo), date.fromisoformat(hi)
def trading_calendar(path: Path) -> list[date]:
"""SPY sessions from the snapshot's own benchmark series."""
con = sqlite3.connect(path)
try:
rows = con.execute(
"SELECT date FROM benchmark_prices WHERE symbol = 'SPY' ORDER BY date"
).fetchall()
finally:
con.close()
return [date.fromisoformat(r[0]) for r in rows]
def next_session(calendar: list[date], after: date) -> date | None:
for d in calendar:
if d > after:
return d
return None
def insert_symbol(
con: sqlite3.Connection,
symbol: str,
name: str,
bars: list[Any],
*,
delisted_on: str | None,
delisted_reason: str | None,
terminal_date: date | None,
) -> int:
"""Insert one restored ticker and its bars. Returns the bar count written."""
now = datetime.utcnow().isoformat(sep=" ")
cur = con.execute(
"INSERT INTO tickers (symbol, name, created_at, delisted_on, delisted_reason) "
"VALUES (?, ?, ?, ?, ?)",
(symbol, name, now, delisted_on, delisted_reason),
)
ticker_id = int(cur.lastrowid)
rows = [
(
ticker_id,
b.timestamp.date().isoformat(),
float(b.open),
float(b.high),
float(b.low),
float(b.close),
int(b.volume),
now,
)
for b in bars
]
if terminal_date is not None:
rows.append((
ticker_id, terminal_date.isoformat(),
TERMINAL_CLOSE, TERMINAL_CLOSE, TERMINAL_CLOSE, TERMINAL_CLOSE, 0, now,
))
con.executemany(
"INSERT INTO ohlcv_records "
"(ticker_id, date, open, high, low, close, volume, created_at) "
"VALUES (?, ?, ?, ?, ?, ?, ?, ?)",
rows,
)
return len(rows)
# ---------------------------------------------------------------------------
# driver
# ---------------------------------------------------------------------------
def file_hash(path: Path) -> str:
h = hashlib.sha256()
with path.open("rb") as fh:
for chunk in iter(lambda: fh.read(1 << 20), b""):
h.update(chunk)
return h.hexdigest()
async def build(args: argparse.Namespace) -> int:
base = Path(args.base).resolve()
if not base.exists():
raise SystemExit(f"Base snapshot not found: {base}")
out_dir = Path(args.out_dir).resolve()
arm_a = out_dir / args.arm_a_name
arm_b = out_dir / args.arm_b_name
events_path = arm_b.with_name(arm_b.stem + "-terminal-events.json")
doc = json.loads(Path(args.removals).read_text(encoding="utf-8"))
removals = doc["removals"]
confirmations = {c["symbol"]: c for c in doc["terminal_confirmations"] if c.get("confirmed")}
cutoffs = {
c["symbol"]: date.fromisoformat(c["last_real_bar"])
for c in doc.get("reviewed_cutoffs", {}).get("cutoffs", [])
}
# A corporate action ends a listing; a market-cap removal does not, so a
# series that stops early without one is an ending we cannot account for.
ENDING_EXPLAINED = {"acquisition", "receivership", "bankruptcy", "ticker_change"}
base_hash_before = file_hash(base)
window_start, window_end = snapshot_window(base)
print(f"base : {base}")
print(f"window : {window_start} .. {window_end}")
print(f"removals : {len(removals)} rows, {sum(1 for r in removals if r['restore_symbol'])} restorable")
for target in (arm_a, arm_b):
if target.exists() and not args.force:
raise SystemExit(f"{target} already exists (use --force to rebuild)")
if target.exists():
target.unlink()
shutil.copy2(base, target)
added = sync_schema(target)
detail = ", ".join(f"{t}: {', '.join(c)}" for t, c in added.items()) or "nothing missing"
print(f"copied : {target.name} (schema sync -> {detail})")
calendar = trading_calendar(base)
con = sqlite3.connect(arm_b)
existing = {r[0].upper() for r in con.execute("SELECT symbol FROM tickers")}
restored: list[dict] = []
unresolved: list[dict] = []
terminal_events: dict[str, str] = {}
try:
for row in removals:
symbol = row["restore_symbol"]
if not symbol:
continue
if symbol.upper() in existing:
unresolved.append({"kind": "skipped", "symbol": symbol, "why": "already present in the base snapshot"})
continue
bars = await fetch_bars(symbol, window_start, window_end)
if not bars:
unresolved.append({"kind": "coverage", "symbol": symbol, "why": "Alpaca returned no bars in the window"})
print(f" {symbol:<6} NO DATA (unresolved coverage)")
continue
cutoff = cutoffs.get(symbol) or cutoffs.get(row["symbol"])
frozen_at = frozen_run_start(bars)
if cutoff is not None:
kept, dropped = apply_reviewed_cutoff(bars, cutoff)
if not kept or kept[-1].timestamp.date() != cutoff:
unresolved.append({
"symbol": symbol,
"kind": "coverage",
"why": f"reviewed cutoff {cutoff} is not a bar in the downloaded series",
})
print(f" {symbol:<6} CUTOFF MISSING {cutoff} (unresolved coverage)")
continue
elif frozen_at is not None:
# Detector only. An unreviewed frozen run means the data changed
# under us; restoring either half would be a guess, so report it.
unresolved.append({
"kind": "coverage",
"symbol": symbol,
"why": (
f"unreviewed frozen run of >= {ZERO_VOLUME_RUN} zero-volume sessions "
f"from {bars[frozen_at].timestamp.date()}; needs a reviewed cutoff "
"before this symbol can be restored"
),
})
print(f" {symbol:<6} FROZEN RUN at {bars[frozen_at].timestamp.date()} (unresolved, not restored)")
continue
else:
kept, dropped = bars, 0
effective_last = kept[-1].timestamp.date()
confirmation = confirmations.get(row["symbol"]) or confirmations.get(symbol)
terminal_date = None
if confirmation is not None:
confirmed_last = date.fromisoformat(confirmation["last_real_bar"])
if effective_last != confirmed_last:
unresolved.append({
"symbol": symbol,
"kind": "terminal_mark",
"why": (
f"retained final bar {effective_last} does not match the reviewed "
f"last_real_bar {confirmed_last}; no terminal mark written"
),
})
print(f" {symbol:<6} MARK SKIPPED final bar {effective_last} != confirmed {confirmed_last}")
else:
terminal_date = next_session(calendar, effective_last)
if terminal_date is None or terminal_date > window_end:
terminal_date = None
unresolved.append({
"symbol": symbol,
"kind": "terminal_mark",
"why": "confirmed terminal event has no trading session left inside the window",
})
else:
terminal_events[symbol] = terminal_date.isoformat()
# Documentary only, and only where the event is confirmed. A series
# that merely ends early is an unresolved ending, not a delisting.
delisted_on = None
delisted_reason = None
if terminal_date is not None:
delisted_on = terminal_date.isoformat()
delisted_reason = row["reason"]
elif effective_last < window_end and row["reason"] not in ENDING_EXPLAINED:
unresolved.append({
"kind": "ending_status",
"symbol": symbol,
"why": (
f"price history ends {effective_last}, before the window closes on "
f"{window_end}, and the source row ({row['reason']}) does not explain "
"an ending; ending status unresolved and delisting fields left null"
),
})
written = insert_symbol(
con, symbol, row["name"], kept,
delisted_on=delisted_on, delisted_reason=delisted_reason,
terminal_date=terminal_date,
)
restored.append({
"symbol": symbol,
"removed_on": row["removed_on"],
"reason": row["reason"],
"bars": written,
"first_bar": kept[0].timestamp.date().isoformat(),
"effective_last_bar": effective_last.isoformat(),
"frozen_bars_dropped": dropped,
"terminal_mark": terminal_events.get(symbol),
})
mark = f" terminal={terminal_events[symbol]} @ {TERMINAL_CLOSE}" if symbol in terminal_events else ""
drop = f" dropped={dropped}" if dropped else ""
print(f" {symbol:<6} bars={written:<5} {kept[0].timestamp.date()}..{effective_last}{drop}{mark}")
con.commit()
finally:
con.close()
events_path.write_text(json.dumps(terminal_events, indent=2) + "\n", encoding="utf-8")
if file_hash(base) != base_hash_before:
raise SystemExit(f"BASE SNAPSHOT WAS MODIFIED: {base}")
con_a = sqlite3.connect(arm_a)
con_b = sqlite3.connect(arm_b)
counts = {
"arm_a_tickers": con_a.execute("SELECT count(*) FROM tickers").fetchone()[0],
"arm_a_bars": con_a.execute("SELECT count(*) FROM ohlcv_records").fetchone()[0],
"arm_b_tickers": con_b.execute("SELECT count(*) FROM tickers").fetchone()[0],
"arm_b_bars": con_b.execute("SELECT count(*) FROM ohlcv_records").fetchone()[0],
}
con_a.close()
con_b.close()
by_kind: dict[str, list[dict]] = {}
for row in unresolved:
by_kind.setdefault(row.get("kind", "other"), []).append(row)
print("\nsummary")
print(f" restored : {len(restored)}")
print(f" unresolved coverage: {len(by_kind.get('coverage', []))} (symbols not restored)")
print(f" unresolved endings : {len(by_kind.get('ending_status', []))} (restored, ending status unknown)")
print(f" terminal marks : {len(terminal_events)} -> {events_path.name}")
print(f" arm A : {counts['arm_a_tickers']} tickers, {counts['arm_a_bars']} bars")
print(f" arm B : {counts['arm_b_tickers']} tickers, {counts['arm_b_bars']} bars")
print(f" base unchanged : sha256 {base_hash_before[:16]}...")
for kind in ("coverage", "ending_status", "terminal_mark"):
for row in by_kind.get(kind, []):
print(f" UNRESOLVED[{kind}] {row['symbol']:<6} {row['why']}")
manifest = {
"generated_at": datetime.now(timezone.utc).isoformat(),
"base": str(base),
"base_sha256": base_hash_before,
"window": {"start": window_start.isoformat(), "end": window_end.isoformat()},
"zero_volume_run": ZERO_VOLUME_RUN,
"terminal_close": TERMINAL_CLOSE,
"counts": counts,
"restored": restored,
"unresolved": unresolved,
"terminal_events": terminal_events,
}
manifest_path = arm_b.with_name(arm_b.stem + "-prep-manifest.json")
manifest_path.write_text(json.dumps(manifest, indent=2) + "\n", encoding="utf-8")
print(f" manifest : {manifest_path.name}")
return 0
def parse_args() -> argparse.Namespace:
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
p.add_argument("--base", default=str(DEFAULT_BASE), help="Base snapshot to copy (never modified).")
p.add_argument("--out-dir", default=str(DEFAULT_BASE.parent), help="Where the arm copies are written.")
p.add_argument("--arm-a-name", default="survivorship-a.sqlite")
p.add_argument("--arm-b-name", default="survivorship-b.sqlite")
p.add_argument("--removals", default=str(DEFAULT_REMOVALS))
p.add_argument("--force", action="store_true", help="Rebuild the arm files if they already exist.")
return p.parse_args()
if __name__ == "__main__":
raise SystemExit(asyncio.run(build(parse_args())))
scripts/research_tr_rank_agreement.py
+596
View File
@@ -0,0 +1,596 @@
"""Does total-return momentum change any admission or ordering decision?
Step 2 of docs/data-integrity-plan.md. A screening test, deliberately cheap: if
ranking on total return picks the same names in the same order as ranking on
price return, nothing downstream can change and the production `tr_factor` work
(Step 3) is unnecessary.
Why this needs no schema and no ingestion change: the forward-anchored factor
design exists because *production* ingest is incremental, so a back-adjusted
series drifts between fetches. Here each symbol's whole history is fetched in one
request, and within a single response the adjustment is internally consistent.
Method
------
* One `Adjustment.SPLIT` and one `Adjustment.ALL` request per symbol plus SPY,
cached to sqlite so re-runs are free and reproducible. Aligned by date, never
zipped by index; a date present in only one adjustment is dropped and counted.
* At every SPY session with enough history, residual 12-1 momentum is computed
twice with the *same* production function: the price-return series against
nominal SPY, and the total-return series against total-return SPY. Only bars at
or before the decision date are used.
* The volatility leg is computed once per date from the SPLIT series and held
fixed in both arms, so the only thing that varies is momentum.
* Both arms are percentiled with the production `_percentiles`, gated at the
production `min_momentum_percentile`, then ordered by `blend_strategy_rank`
with the symbol as a deterministic tie-break. The full admitted list is
compared - never truncated to `max_positions`, because a lower-ranked name can
be the first actionable setup when others fail gates or are already held.
Decision rule (pre-registered, docs/data-integrity-plan.md)
----------------------------------------------------------
* no ordered-list difference on any evaluated date -> stop, Step 3 unnecessary
* any ordered-list difference -> Step 3 investigates trades
* incomplete downloads or no evaluable dates -> inconclusive, fix coverage
Usage::
python scripts/research_tr_rank_agreement.py --fetch # populate the cache
python scripts/research_tr_rank_agreement.py --time-batch # project the runtime
python scripts/research_tr_rank_agreement.py # full screen
"""
from __future__ import annotations
import argparse
import asyncio
import json
import sqlite3
import sys
import time
from datetime import date, datetime, time as dtime, timedelta, timezone
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
if str(ROOT) not in sys.path:
sys.path.insert(0, str(ROOT))
DEFAULT_SNAPSHOT = ROOT / "backtest_snapshots" / "survivorship-a.sqlite"
DEFAULT_CACHE = ROOT / "backtest_snapshots" / "tr-rank-cache.sqlite"
BENCHMARK = "SPY"
# ---------------------------------------------------------------------------
# cache
# ---------------------------------------------------------------------------
def open_cache(path: Path) -> sqlite3.Connection:
con = sqlite3.connect(path)
con.execute(
"CREATE TABLE IF NOT EXISTS bars ("
" symbol TEXT NOT NULL, date TEXT NOT NULL,"
" split_close REAL, all_close REAL,"
" PRIMARY KEY (symbol, date))"
)
con.execute(
"CREATE TABLE IF NOT EXISTS fetched ("
" symbol TEXT PRIMARY KEY, fetched_at TEXT NOT NULL,"
" split_bars INTEGER, all_bars INTEGER, aligned_bars INTEGER,"
" dropped_unmatched INTEGER)"
)
con.commit()
return con
async def fetch_pair(symbol: str, start: date, end: date) -> tuple[dict, dict]:
"""(split_closes, all_closes) keyed by date, each from a single request."""
from alpaca.data.enums import Adjustment
from alpaca.data.historical import StockHistoricalDataClient
from alpaca.data.requests import StockBarsRequest
from alpaca.data.timeframe import TimeFrame
from app.config import settings
if not settings.alpaca_api_key or not settings.alpaca_api_secret:
raise SystemExit("Alpaca credentials are not configured.")
client = StockHistoricalDataClient(settings.alpaca_api_key, settings.alpaca_api_secret)
alpaca_symbol = symbol.replace("-", ".")
window = (
datetime.combine(start, dtime.min, tzinfo=timezone.utc),
datetime.combine(end + timedelta(days=1), dtime.min, tzinfo=timezone.utc),
)
async def one(adjustment) -> dict[date, float]:
request = StockBarsRequest(
symbol_or_symbols=alpaca_symbol, timeframe=TimeFrame.Day,
start=window[0], end=window[1], adjustment=adjustment,
)
last: Exception | None = None
for attempt in range(4):
try:
bars = await asyncio.to_thread(client.get_stock_bars, request)
return {
b.timestamp.date(): float(b.close)
for b in bars.data.get(alpaca_symbol, [])
}
except Exception as exc: # noqa: BLE001 - retry any transport failure
last = exc
if attempt < 3:
await asyncio.sleep(2 * (attempt + 1))
raise SystemExit(f"Alpaca fetch failed for {symbol}: {last}")
return await one(Adjustment.SPLIT), await one(Adjustment.ALL)
async def populate_cache(symbols: list[str], start: date, end: date, cache: Path) -> dict:
con = open_cache(cache)
done = {r[0] for r in con.execute("SELECT symbol FROM fetched")}
todo = [s for s in symbols if s not in done]
print(f"cache {cache.name}: {len(done)} cached, {len(todo)} to fetch")
coverage: dict[str, dict] = {}
for index, symbol in enumerate(todo, 1):
split, adjusted = await fetch_pair(symbol, start, end)
# Align by date. A date in only one adjustment cannot be compared and is
# dropped rather than paired with its neighbour.
common = sorted(set(split) & set(adjusted))
dropped = len(set(split) ^ set(adjusted))
con.executemany(
"INSERT OR REPLACE INTO bars (symbol, date, split_close, all_close) VALUES (?,?,?,?)",
[(symbol, d.isoformat(), split[d], adjusted[d]) for d in common],
)
con.execute(
"INSERT OR REPLACE INTO fetched VALUES (?,?,?,?,?,?)",
(symbol, datetime.now(timezone.utc).isoformat(), len(split), len(adjusted),
len(common), dropped),
)
con.commit()
if dropped:
print(f" {symbol:<6} aligned={len(common)} dropped_unmatched={dropped}")
if index % 50 == 0:
print(f" … {index}/{len(todo)}")
for row in con.execute("SELECT symbol, split_bars, all_bars, aligned_bars, dropped_unmatched FROM fetched"):
coverage[row[0]] = {
"split_bars": row[1], "all_bars": row[2],
"aligned_bars": row[3], "dropped_unmatched": row[4],
}
con.close()
return coverage
def load_series(cache: Path, symbols: list[str]) -> dict[str, tuple[list[date], list[float], list[float]]]:
con = open_cache(cache)
out: dict[str, tuple[list[date], list[float], list[float]]] = {}
for symbol in symbols:
rows = con.execute(
"SELECT date, split_close, all_close FROM bars WHERE symbol = ? ORDER BY date",
(symbol,),
).fetchall()
if not rows:
continue
out[symbol] = (
[date.fromisoformat(r[0]) for r in rows],
[float(r[1]) for r in rows],
[float(r[2]) for r in rows],
)
con.close()
return out
# ---------------------------------------------------------------------------
# screen
# ---------------------------------------------------------------------------
def snapshot_symbols(snapshot: Path) -> tuple[list[str], list[str], date, date]:
"""(symbols with bars, symbols with none, window start, window end).
A ticker row with no bars in the snapshot cannot rank in either arm, so it is
inert rather than missing. Keeping the two apart is what stops a failed
download from being quietly reported as agreement.
"""
con = sqlite3.connect(snapshot)
try:
rows = con.execute(
"SELECT t.symbol, count(o.id) FROM tickers t "
"LEFT JOIN ohlcv_records o ON o.ticker_id = t.id GROUP BY t.symbol"
).fetchall()
lo, hi = con.execute("SELECT min(date), max(date) FROM ohlcv_records").fetchone()
finally:
con.close()
with_bars = sorted(r[0].upper() for r in rows if r[1])
without_bars = sorted(r[0].upper() for r in rows if not r[1])
return with_bars, without_bars, date.fromisoformat(lo), date.fromisoformat(hi)
def ordered_list(
momentum: dict[str, float],
vol_percentiles: dict[str, float],
cutoff: float,
) -> list[str]:
"""Admitted set at the production cutoff, ordered by the production blend.
Symbols are fed to ``_percentiles`` in sorted order in both arms so that ties
resolve identically and cannot manufacture a difference.
"""
from app.services.momentum_service import _percentiles, blend_strategy_rank
percentiles = _percentiles({s: momentum[s] for s in sorted(momentum)})
admitted = [s for s, p in percentiles.items() if p >= cutoff]
scored = {
s: blend_strategy_rank(percentiles[s], vol_percentiles.get(s))
for s in admitted
}
return sorted(admitted, key=lambda s: (-(scored[s] or 0.0), s))
def mechanism_check(
series: dict[str, tuple[list[date], list[float], list[float]]],
sample_dates: list[date],
) -> dict:
"""Is the rank shift explained by dividend yield?
For each sampled decision date, correlate a name's trailing 12-month implied
yield — read straight off the ALL/SPLIT ratio, which is what separates the two
series — against how far its momentum percentile moves between the arms. A
strong positive rank correlation means the difference *is* the dividend, not
numerical noise.
"""
from app.services.momentum_service import _MOM_LOOKBACK, _percentiles, compute_residual_12_1_momentum
bench_dates, bench_split, bench_all = series[BENCHMARK]
bench_price = dict(zip(bench_dates, bench_split))
bench_tr = dict(zip(bench_dates, bench_all))
symbols = sorted(s for s in series if s != BENCHMARK)
index_of = {s: {d: i for i, d in enumerate(series[s][0])} for s in symbols}
correlations: list[float] = []
used_dates: list[str] = []
for as_of in sample_dates:
price_mom: dict[str, float] = {}
tr_mom: dict[str, float] = {}
yields: dict[str, float] = {}
for symbol in symbols:
idx = index_of[symbol].get(as_of)
if idx is None or idx < _MOM_LOOKBACK:
continue
dates, split_closes, all_closes = series[symbol]
wd, ws, wa = dates[: idx + 1], split_closes[: idx + 1], all_closes[: idx + 1]
p = compute_residual_12_1_momentum(wd, ws, bench_price)
t = compute_residual_12_1_momentum(wd, wa, bench_tr)
if p is None or t is None:
continue
price_mom[symbol] = p
tr_mom[symbol] = t
# Ratio drift over the formation window is the accumulated dividend.
r_now = wa[-1] / ws[-1] if ws[-1] else None
r_then = wa[-_MOM_LOOKBACK] / ws[-_MOM_LOOKBACK] if ws[-_MOM_LOOKBACK] else None
if r_now and r_then:
yields[symbol] = r_now / r_then - 1.0
common = sorted(set(price_mom) & set(yields))
if len(common) < 30:
continue
pp = _percentiles({s: price_mom[s] for s in common})
tp = _percentiles({s: tr_mom[s] for s in common})
shift = {s: tp[s] - pp[s] for s in common}
correlations.append(_spearman([yields[s] for s in common], [shift[s] for s in common]))
used_dates.append(as_of.isoformat())
return {
"what_it_measures": (
"Spearman correlation, per sampled date, between a name's trailing 12-month "
"implied yield (from the ALL/SPLIT ratio over the formation window) and the "
"change in its momentum percentile between the two arms."
),
"caveat": (
"Both the stock series and the benchmark leg change between arms, so this is "
"strong evidence consistent with a dividend effect - it does not isolate the "
"stock's own yield from the beta-weighted benchmark yield."
),
"formation_window_bars": _MOM_LOOKBACK,
"min_symbols_per_date": 30,
"sampled_dates": len(correlations),
"sample_date_list": used_dates,
"mean_spearman_yield_vs_percentile_shift": (
round(sum(correlations) / len(correlations), 4) if correlations else None
),
"min": round(min(correlations), 4) if correlations else None,
"max": round(max(correlations), 4) if correlations else None,
"per_date_spearman": [round(c, 4) for c in correlations],
}
def _spearman(x: list[float], y: list[float]) -> float:
def rank(v: list[float]) -> list[float]:
order = sorted(range(len(v)), key=lambda i: v[i])
out = [0.0] * len(v)
for position, i in enumerate(order):
out[i] = float(position)
return out
rx, ry = rank(x), rank(y)
n = len(rx)
mx, my = sum(rx) / n, sum(ry) / n
num = sum((rx[i] - mx) * (ry[i] - my) for i in range(n))
dx = sum((rx[i] - mx) ** 2 for i in range(n)) ** 0.5
dy = sum((ry[i] - my) ** 2 for i in range(n)) ** 0.5
return num / (dx * dy) if dx and dy else 0.0
def disagreement(a: list[str], b: list[str]) -> float:
sa, sb = set(a), set(b)
union = sa | sb
if not union:
return 0.0
return len(sa ^ sb) / len(union)
def run_screen(
series: dict[str, tuple[list[date], list[float], list[float]]],
cutoff: float,
*,
limit_dates: int | None = None,
verbose: bool = True,
) -> dict:
from app.services.momentum_service import (
_MOM_LOOKBACK,
compute_realized_vol_6m,
compute_residual_12_1_momentum,
_percentiles,
)
bench_dates, bench_split, bench_all = series[BENCHMARK]
bench_price = dict(zip(bench_dates, bench_split))
bench_tr = dict(zip(bench_dates, bench_all))
symbols = sorted(s for s in series if s != BENCHMARK)
index_of: dict[str, dict[date, int]] = {
s: {d: i for i, d in enumerate(series[s][0])} for s in symbols
}
evaluated = 0
skipped_no_history = 0
changed_sets = 0
changed_orders = 0
disagreements: list[float] = []
# Counted directly rather than derived from the Jaccard ratio: a swap moves
# two names (one out, one in), so the symmetric difference is twice the
# replacement count and the two are easy to conflate.
replacements: list[int] = []
only_price_counts: list[int] = []
only_tr_counts: list[int] = []
admitted_sizes: list[int] = []
per_year: dict[int, dict[str, int]] = {}
examples: list[dict] = []
decision_dates = [d for d in bench_dates]
if limit_dates:
decision_dates = decision_dates[-limit_dates:]
for as_of in decision_dates:
price_mom: dict[str, float] = {}
tr_mom: dict[str, float] = {}
vol_values: dict[str, float] = {}
for symbol in symbols:
idx = index_of[symbol].get(as_of)
if idx is None or idx < _MOM_LOOKBACK:
continue
dates, split_closes, all_closes = series[symbol]
window_dates = dates[: idx + 1]
window_split = split_closes[: idx + 1]
window_all = all_closes[: idx + 1]
price_value = compute_residual_12_1_momentum(window_dates, window_split, bench_price)
tr_value = compute_residual_12_1_momentum(window_dates, window_all, bench_tr)
# Equal eligibility: a name enters both arms or neither.
if price_value is None or tr_value is None:
continue
price_mom[symbol] = price_value
tr_mom[symbol] = tr_value
vol = compute_realized_vol_6m(window_split)
if vol is not None:
vol_values[symbol] = vol
if not price_mom:
skipped_no_history += 1
continue
evaluated += 1
vol_percentiles = _percentiles({s: vol_values[s] for s in sorted(vol_values)})
price_list = ordered_list(price_mom, vol_percentiles, cutoff)
tr_list = ordered_list(tr_mom, vol_percentiles, cutoff)
year = per_year.setdefault(as_of.year, {"dates": 0, "changed_sets": 0, "changed_orders": 0})
year["dates"] += 1
set_price, set_tr = set(price_list), set(tr_list)
only_price = len(set_price - set_tr)
only_tr = len(set_tr - set_price)
only_price_counts.append(only_price)
only_tr_counts.append(only_tr)
admitted_sizes.append(len(price_list))
# A replacement is one name leaving paired with one arriving; when the
# two arms admit different counts, the surplus is not a replacement.
replacements.append(min(only_price, only_tr))
same_set = set_price == set_tr
same_order = price_list == tr_list
if not same_set:
changed_sets += 1
year["changed_sets"] += 1
if not same_order:
changed_orders += 1
year["changed_orders"] += 1
if len(examples) < 10:
only_price = [s for s in price_list if s not in set(tr_list)]
only_tr = [s for s in tr_list if s not in set(price_list)]
examples.append({
"date": as_of.isoformat(),
"admitted_price_return": len(price_list),
"admitted_total_return": len(tr_list),
"only_in_price_return": only_price[:10],
"only_in_total_return": only_tr[:10],
"order_changed_without_membership_change": same_set,
"first_10_price_return": price_list[:10],
"first_10_total_return": tr_list[:10],
})
disagreements.append(disagreement(price_list, tr_list))
if verbose and evaluated % 100 == 0:
print(f" … {evaluated} dates evaluated, {changed_orders} with a different ordered list")
return {
"evaluated_dates": evaluated,
"skipped_dates_no_history": skipped_no_history,
"dates_with_different_admitted_set": changed_sets,
"dates_with_different_ordered_list": changed_orders,
"mean_set_disagreement": (sum(disagreements) / len(disagreements)) if disagreements else 0.0,
"max_set_disagreement": max(disagreements) if disagreements else 0.0,
"mean_admitted_per_arm": round(sum(admitted_sizes) / len(admitted_sizes), 2) if admitted_sizes else 0.0,
"mean_names_only_in_price_return": round(sum(only_price_counts) / len(only_price_counts), 2) if only_price_counts else 0.0,
"mean_names_only_in_total_return": round(sum(only_tr_counts) / len(only_tr_counts), 2) if only_tr_counts else 0.0,
"mean_replacements_per_date": round(sum(replacements) / len(replacements), 2) if replacements else 0.0,
"max_replacements_per_date": max(replacements) if replacements else 0,
"mean_symmetric_difference": round(
(sum(only_price_counts) + sum(only_tr_counts)) / len(only_price_counts), 2
) if only_price_counts else 0.0,
"per_year": {
str(y): v for y, v in sorted(per_year.items())
},
"examples": examples,
}
# ---------------------------------------------------------------------------
# driver
# ---------------------------------------------------------------------------
async def main(args: argparse.Namespace) -> int:
snapshot = Path(args.snapshot)
cache = Path(args.cache)
symbols, inert, window_start, window_end = snapshot_symbols(snapshot)
universe = symbols + [BENCHMARK]
print(f"snapshot : {snapshot.name} ({len(symbols)} symbols with bars"
+ (f", {len(inert)} inert: {', '.join(inert)}" if inert else "") + ")")
print(f"window : {window_start} .. {window_end} (clipped to the snapshot)")
if args.fetch:
await populate_cache(universe, window_start, window_end, cache)
return 0
series = load_series(cache, universe)
missing = [s for s in universe if s not in series]
if missing:
raise SystemExit(
f"{len(missing)} symbols have bars in the snapshot but none in the cache "
f"({', '.join(missing[:8])}). Run with --fetch first; an incomplete download "
"makes the screen inconclusive and must never be read as agreement."
)
from app.services.admin_service import ACTIVATION_DEFAULTS
cutoff = float(ACTIVATION_DEFAULTS["min_momentum_percentile"])
print(f"cutoff : {cutoff} (production min_momentum_percentile)")
if args.mechanism:
bench_dates = load_series(cache, [BENCHMARK])[BENCHMARK][0]
sample = bench_dates[::args.mechanism]
print(f"mechanism check over {len(sample)} sampled dates ...")
payload = {
"generated_at": datetime.now(timezone.utc).isoformat(),
"snapshot": str(snapshot),
"cache": str(cache),
"window": {"start": window_start.isoformat(), "end": window_end.isoformat()},
"universe_size": len(symbols),
"sample_every_n_sessions": args.mechanism,
"result": mechanism_check(series, sample),
}
out = Path(args.mechanism_out)
out.parent.mkdir(parents=True, exist_ok=True)
out.write_text(json.dumps(payload, indent=2) + "\n", encoding="utf-8")
r = payload["result"]
print(f" sampled dates : {r['sampled_dates']}")
print(f" mean Spearman : {r['mean_spearman_yield_vs_percentile_shift']} "
f"(min {r['min']}, max {r['max']})")
print(f" report : {out}")
return 0
if args.time_batch:
n = args.time_batch
started = time.perf_counter()
result = run_screen(series, cutoff, limit_dates=n, verbose=False)
elapsed = time.perf_counter() - started
total_dates = len(series[BENCHMARK][0])
per_date = elapsed / max(result["evaluated_dates"], 1)
print(f"\ntimed {result['evaluated_dates']} evaluated dates in {elapsed:.1f}s "
f"({per_date:.2f}s/date)")
print(f"projected full run over {total_dates} sessions: "
f"{per_date * total_dates / 60:.1f} min")
return 0
started = time.perf_counter()
result = run_screen(series, cutoff)
result["elapsed_seconds"] = round(time.perf_counter() - started, 1)
con = open_cache(cache)
coverage = {
r[0]: {"split_bars": r[1], "all_bars": r[2], "aligned_bars": r[3], "dropped_unmatched": r[4]}
for r in con.execute("SELECT symbol, split_bars, all_bars, aligned_bars, dropped_unmatched FROM fetched")
}
con.close()
verdict = (
"stop - no admission or ordering difference in this window"
if result["dates_with_different_ordered_list"] == 0
else "proceed to Step 3 - ordered lists differ"
)
payload = {
"generated_at": datetime.now(timezone.utc).isoformat(),
"snapshot": str(snapshot),
"cache": str(cache),
"window": {"start": window_start.isoformat(), "end": window_end.isoformat()},
"universe_size": len(symbols),
"inert_snapshot_symbols": inert,
"cutoff": cutoff,
"verdict": verdict,
"result": result,
"coverage": {
"symbols_cached": len(coverage),
"symbols_with_unmatched_dates": sum(1 for c in coverage.values() if c["dropped_unmatched"]),
"total_unmatched_dates": sum(c["dropped_unmatched"] for c in coverage.values()),
},
}
out = Path(args.out)
out.parent.mkdir(parents=True, exist_ok=True)
out.write_text(json.dumps(payload, indent=2) + "\n", encoding="utf-8")
print(f"\nevaluated dates : {result['evaluated_dates']}")
print(f"dates with different admitted set: {result['dates_with_different_admitted_set']}")
print(f"dates with different ordered list: {result['dates_with_different_ordered_list']}")
print(f"mean / max set disagreement : {result['mean_set_disagreement']:.4f} / {result['max_set_disagreement']:.4f}")
print(f"verdict : {verdict}")
print(f"report : {out}")
return 0
def parse_args() -> argparse.Namespace:
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
p.add_argument("--snapshot", default=str(DEFAULT_SNAPSHOT), help="Universe and date bounds come from this snapshot.")
p.add_argument("--cache", default=str(DEFAULT_CACHE))
p.add_argument("--fetch", action="store_true", help="Populate the cache and exit.")
p.add_argument("--time-batch", type=int, default=None, metavar="N",
help="Time the screen over the last N sessions and project the full runtime.")
p.add_argument("--mechanism", type=int, default=None, metavar="EVERY_N",
help="Correlate trailing dividend yield against the percentile shift on every Nth session.")
p.add_argument("--out", default="reports/tr-rank-agreement.json")
p.add_argument("--mechanism-out", default="reports/tr-rank-agreement-mechanism.json")
return p.parse_args()
if __name__ == "__main__":
raise SystemExit(asyncio.run(main(parse_args())))
scripts/run_total_return_ab.py
+544
View File
@@ -0,0 +1,544 @@
"""Paired portfolio replay: does the momentum basis change actual trades?
Step 3a of docs/data-integrity-plan.md. The Step 2 screen showed the ranking
changes every day; this measures whether that reaches admissions and P&L. No
schema, no ingestion — bars come from the survivorship-a snapshot and the two
adjusted series from the Step 2 cache.
What is held identical between the arms
---------------------------------------
* **Setup candidates are generated once** and reused by both books. Entry, stop,
target and R:R never differ. Only momentum and what depends on it —
the residual/activation percentiles, `qualified`, and the 80/20 blend used for
ordering — are recomputed per arm.
* **Both arms read the same cache**: control uses SPLIT stock against SPLIT SPY,
treatment uses ALL stock against ALL SPY. Neither reads a freshly fetched
series, so a refreshed-vs-snapshot difference cannot be mistaken for a basis
difference.
* **Execution, volatility, gates, capacity, exits**: snapshot nominal bars in both
arms; the 6-month volatility leg comes from candidate generation and is never
recomputed; non-momentum floors, `max_positions`, risk per trade and the exit
policy are the production values for both.
Each book then keeps its **own** holdings: a divergent admission changes which
slots are occupied later, and that path dependence is part of the measurement.
Reading the output
------------------
Different trades establish that the basis is production-relevant. They do not
authorise a rollout — that is a separate non-inferiority decision taken on the
paired numbers.
Usage::
python scripts/run_total_return_ab.py --build-candidates # one-time, cached
python scripts/run_total_return_ab.py
"""
from __future__ import annotations
import argparse
import asyncio
import json
import pickle
import sqlite3
import sys
import time
from concurrent.futures import ProcessPoolExecutor
from datetime import date, datetime, timezone
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
if str(ROOT) not in sys.path:
sys.path.insert(0, str(ROOT))
DEFAULT_SNAPSHOT = ROOT / "backtest_snapshots" / "survivorship-a.sqlite"
DEFAULT_CACHE = ROOT / "backtest_snapshots" / "tr-rank-cache.sqlite"
DEFAULT_CANDIDATES = ROOT / "backtest_snapshots" / "tr-ab-candidates.pkl"
BENCHMARK = "SPY"
def _sqlite_url(path: Path) -> str:
return f"sqlite+aiosqlite:///{path.resolve().as_posix()}"
# ---------------------------------------------------------------------------
# candidates — generated once, reused by both arms
# ---------------------------------------------------------------------------
async def build_candidates(snapshot: Path, out: Path, cadence: str, workers: int) -> None:
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker, create_async_engine
from app.models.ticker import Ticker
from app.services import backtest_service as bt
from app.services.admin_service import get_activation_config
from app.services.recommendation_service import get_recommendation_config
engine = create_async_engine(_sqlite_url(snapshot), pool_pre_ping=True)
Session = async_sessionmaker(engine, class_=AsyncSession, expire_on_commit=False)
candidates: list[dict] = []
prices: dict[str, tuple] = {}
async with Session() as db:
config = await get_recommendation_config(db)
activation = await get_activation_config(db)
symbols = list((await db.execute(select(Ticker.symbol).order_by(Ticker.symbol))).scalars())
benchmark = await bt._load_benchmark_closes_for_backtest(db, refresh=False)
print(f"generating candidates for {len(symbols)} symbols (cadence={cadence}) …")
columns_by_symbol: dict[str, tuple] = {}
for symbol in symbols:
cols = await bt._fetch_columns(db, symbol)
if cols is not None:
columns_by_symbol[symbol] = cols
await engine.dispose()
started = time.perf_counter()
ctx = bt._mp_context()
pool = None
if ctx is not None and workers > 1:
try:
pool = ProcessPoolExecutor(max_workers=workers, mp_context=ctx)
except Exception: # noqa: BLE001 - fall back to in-process replay
pool = None
items = list(columns_by_symbol.items())
if pool is not None:
with pool:
futures = {
symbol: pool.submit(
bt._replay_and_signals, symbol, cols, config, activation,
benchmark, bt.PRODUCTION_GTL_TARGET_MODEL, cadence, False, None,
)
for symbol, cols in items
}
for index, (symbol, future) in enumerate(futures.items(), 1):
got, _ = future.result()
candidates.extend(got)
prices[symbol] = columns_by_symbol[symbol]
if index % 100 == 0:
print(f" … {index}/{len(items)}")
else:
for index, (symbol, cols) in enumerate(items, 1):
got, _ = bt._replay_and_signals(
symbol, cols, config, activation, benchmark,
bt.PRODUCTION_GTL_TARGET_MODEL, cadence, False, None,
)
candidates.extend(got)
prices[symbol] = cols
if index % 100 == 0:
print(f" … {index}/{len(items)}")
payload = {
"generated_at": datetime.now(timezone.utc).isoformat(),
"snapshot": str(snapshot),
"cadence": cadence,
"config": config,
"activation": activation,
"candidates": candidates,
"prices": prices,
"benchmark": benchmark,
}
out.write_bytes(pickle.dumps(payload))
print(f"{len(candidates)} candidates over {len(prices)} symbols "
f"in {time.perf_counter() - started:.0f}s -> {out.name}")
# ---------------------------------------------------------------------------
# momentum, recomputed per arm from the cache
# ---------------------------------------------------------------------------
def load_cache_series(cache: Path) -> dict[str, tuple[list[date], list[float], list[float]]]:
con = sqlite3.connect(cache)
try:
rows = con.execute("SELECT symbol, date, split_close, all_close FROM bars ORDER BY symbol, date")
out: dict[str, tuple[list, list, list]] = {}
for symbol, day, split_close, all_close in rows:
dates, splits, alls = out.setdefault(symbol, ([], [], []))
dates.append(date.fromisoformat(day))
splits.append(float(split_close))
alls.append(float(all_close))
finally:
con.close()
return out
def momentum_by_arm(
series: dict[str, tuple[list[date], list[float], list[float]]],
signal_dates: dict[str, set[date]],
*,
total_return: bool,
) -> dict[tuple[str, date], float]:
"""Residual 12-1 momentum at every (symbol, signal date) for one arm.
Control reads the SPLIT series against SPLIT SPY; treatment reads ALL against
ALL SPY. Both use the unmodified production function and only bars at or
before the signal date.
"""
from app.services.momentum_service import _MOM_LOOKBACK, compute_residual_12_1_momentum
bench_dates, bench_split, bench_all = series[BENCHMARK]
benchmark = dict(zip(bench_dates, bench_all if total_return else bench_split))
out: dict[tuple[str, date], float] = {}
for symbol, wanted in signal_dates.items():
entry = series.get(symbol)
if entry is None:
continue
dates, splits, alls = entry
closes = alls if total_return else splits
index_of = {d: i for i, d in enumerate(dates)}
for as_of in wanted:
idx = index_of.get(as_of)
if idx is None or idx < _MOM_LOOKBACK:
continue
value = compute_residual_12_1_momentum(
dates[: idx + 1], closes[: idx + 1], benchmark
)
if value is not None:
out[(symbol, as_of)] = value
return out
def apply_arm(candidates: list[dict], momentum: dict[tuple[str, date], float], min_pct: float) -> list[dict]:
"""Rebuild only the momentum-dependent fields, using the production pipeline."""
from app.services import backtest_service as bt
rebuilt = []
for cand in candidates:
clone = dict(cand)
key = (clone["symbol"], date.fromisoformat(clone["date"]))
clone["residual_momentum"] = momentum.get(key)
rebuilt.append(clone)
bt._assign_momentum_percentiles(rebuilt)
bt._assign_residual_momentum_percentiles(rebuilt)
bt._assign_low_volatility_percentiles(rebuilt) # vol_6m untouched, so identical in both arms
bt._assign_activation_momentum_percentiles(rebuilt)
bt._assign_residual_low_vol_blend(rebuilt)
bt._assign_residual_high_vol_blend(rebuilt)
for cand in rebuilt:
cand["qualified"] = bt._momentum_qualifies(cand, min_pct)
return rebuilt
def simulate(candidates: list[dict], prices: dict, benchmark: dict, cadence: str) -> dict:
"""Run the production book: live gate, 80/20 ordering, ATR trail, gate reset."""
from app.services import backtest_service as bt
strategy = next(s for s in bt.PORTFOLIO_MONITOR_STRATEGIES if s.get("is_production"))
entry_cfg = bt._entry_variant_config(str(strategy["entry_variant"]))
ranking_key = str(strategy.get("ranking_key") or entry_cfg.get("ranking_key"))
reentry_fn = bt._make_gate_reset_reentry_fn(
candidates, prices, cadence=cadence, qualified_fn=None, ranking_key=ranking_key,
)
return bt._simulate_portfolio(
candidates,
prices,
benchmark,
str(strategy["exit_policy"]),
max(bt.TIME_EXIT_DAYS),
qualified_fn=None, # live qualification, as production
ranking_key=ranking_key,
max_positions=int(entry_cfg["max_positions"]),
risk_per_trade=float(entry_cfg["risk_per_trade"]),
atr_trail_multiplier=bt.ATR_TRAIL_MULTIPLIER,
post_stop_reentry_fn=reentry_fn,
include_curve=True,
include_trades=True,
)
# ---------------------------------------------------------------------------
# comparison
# ---------------------------------------------------------------------------
def entry_diff(control: dict, treatment: dict) -> dict:
def entries(sim: dict) -> set[tuple[str, str]]:
return {(t["entry_date"], t["symbol"]) for t in sim.get("trade_details", [])}
a, b = entries(control), entries(treatment)
only_a = sorted(a - b)
only_b = sorted(b - a)
return {
"shared_entries": len(a & b),
"only_control": [{"entry_date": d, "symbol": s} for d, s in only_a],
"only_treatment": [{"entry_date": d, "symbol": s} for d, s in only_b],
"only_control_count": len(only_a),
"only_treatment_count": len(only_b),
}
def yearly_comparison(control: dict, treatment: dict) -> dict:
ca = {row["year"]: row for row in (control.get("yearly_returns") or [])}
tb = {row["year"]: row for row in (treatment.get("yearly_returns") or [])}
years = sorted(set(ca) | set(tb))
rows = []
better = worse = same = 0
for year in years:
c = (ca.get(year) or {}).get("return_pct")
t = (tb.get(year) or {}).get("return_pct")
delta = None
if c is not None and t is not None:
delta = round(t - c, 2)
if delta > 0:
better += 1
elif delta < 0:
worse += 1
else:
same += 1
rows.append({"year": year, "control_pct": c, "treatment_pct": t, "delta_pp": delta})
return {
"rows": rows,
"years_treatment_better": better,
"years_treatment_worse": worse,
"years_equal": same,
}
HEADLINE = ("trades", "total_return_pct", "cagr_pct", "max_drawdown_pct", "sharpe", "win_rate")
# Pre-specified before the diagnostic was run, and reported for BOTH block
# lengths rather than whichever reads better.
BOOTSTRAP_SEED = 20260907
BOOTSTRAP_REPLICATES = 5000
BOOTSTRAP_BLOCKS = (21, 63)
TRADING_DAYS = 252
def daily_returns(curve: list[dict]) -> tuple[list[str], list[float]]:
"""Daily returns from an equity curve, with their dates."""
dates, rets = [], []
for previous, current in zip(curve, curve[1:]):
prior = float(previous["equity"])
if prior > 0:
dates.append(current["date"])
rets.append(float(current["equity"]) / prior - 1.0)
return dates, rets
def _path_stats(returns) -> tuple[float, float, float]:
"""(CAGR %, Sharpe, max drawdown %) for one resampled return path."""
import math
n = len(returns)
equity = 1.0
peak = 1.0
max_dd = 0.0
total = 0.0
total_sq = 0.0
for r in returns:
equity *= 1.0 + r
peak = max(peak, equity)
if peak > 0:
max_dd = max(max_dd, 1.0 - equity / peak)
total += r
total_sq += r * r
mean = total / n
var = max(total_sq / n - mean * mean, 0.0)
sd = math.sqrt(var)
cagr = (equity ** (TRADING_DAYS / n) - 1.0) * 100.0 if equity > 0 else -100.0
sharpe = (mean / sd * math.sqrt(TRADING_DAYS)) if sd > 0 else 0.0
return cagr, sharpe, max_dd * 100.0
def paired_block_bootstrap(
control_returns: list[float],
treatment_returns: list[float],
*,
block: int,
replicates: int,
seed: int,
) -> dict:
"""Resample identical date blocks for both arms and difference the results.
The arms share dates and most positions, so their difference is what needs an
interval — not each arm separately. Drawing the *same* block indices for both
preserves the pairing and whatever cross-arm correlation exists on those days.
Moving blocks retain some of the serial dependence that an i.i.d. resample
would destroy; they do not make this out-of-sample.
"""
import random
import statistics
rng = random.Random(seed)
n = len(control_returns)
n_blocks = -(-n // block) # ceil
max_start = n - block
deltas = {"cagr_pct": [], "sharpe": [], "max_drawdown_pct": []}
for _ in range(replicates):
starts = [rng.randint(0, max_start) for _ in range(n_blocks)]
idx = [i for s in starts for i in range(s, s + block)][:n]
c = _path_stats([control_returns[i] for i in idx])
t = _path_stats([treatment_returns[i] for i in idx])
deltas["cagr_pct"].append(t[0] - c[0])
deltas["sharpe"].append(t[1] - c[1])
deltas["max_drawdown_pct"].append(t[2] - c[2])
def interval(values: list[float]) -> dict:
ordered = sorted(values)
def q(p: float) -> float:
return ordered[min(len(ordered) - 1, max(0, int(round(p * (len(ordered) - 1)))))]
return {
"mean": round(statistics.fmean(ordered), 3),
"median": round(q(0.50), 3),
"ci_2.5": round(q(0.025), 3),
"ci_97.5": round(q(0.975), 3),
"share_below_zero": round(sum(1 for v in ordered if v < 0) / len(ordered), 4),
}
return {metric: interval(values) for metric, values in deltas.items()}
async def main(args: argparse.Namespace) -> int:
snapshot = Path(args.snapshot)
cache = Path(args.cache)
store = Path(args.candidates)
if args.build_candidates or not store.exists():
if not args.build_candidates:
print(f"{store.name} missing — building it first")
await build_candidates(snapshot, store, args.cadence, args.workers)
if args.build_candidates:
return 0
payload = pickle.loads(store.read_bytes())
candidates = payload["candidates"]
prices = payload["prices"]
benchmark = payload["benchmark"]
min_pct = float(payload["activation"].get("min_momentum_percentile", 80.0))
cadence = payload["cadence"]
signal_dates: dict[str, set[date]] = {}
for cand in candidates:
signal_dates.setdefault(cand["symbol"], set()).add(date.fromisoformat(cand["date"]))
series = load_cache_series(cache)
missing = sorted({s for s in signal_dates if s not in series})
if missing:
raise SystemExit(
f"{len(missing)} symbols have candidates but no cached series "
f"({', '.join(missing[:8])}). Run research_tr_rank_agreement.py --fetch."
)
print(f"candidates: {len(candidates)} over {len(signal_dates)} symbols, cutoff {min_pct}")
results = {}
for label, total_return in (("control_price_return", False), ("treatment_total_return", True)):
started = time.perf_counter()
momentum = momentum_by_arm(series, signal_dates, total_return=total_return)
arm = apply_arm(candidates, momentum, min_pct)
sim = simulate(arm, prices, benchmark, cadence)
results[label] = sim
print(f" {label:<24} trades={sim['trades']:<5} CAGR={sim['cagr_pct']}% "
f"Sharpe={sim['sharpe']} DD={sim['max_drawdown_pct']}% "
f"({time.perf_counter() - started:.0f}s)")
control = results["control_price_return"]
treatment = results["treatment_total_return"]
diff = entry_diff(control, treatment)
yearly = yearly_comparison(control, treatment)
# Paired uncertainty on the DIFFERENCE. Per-arm standard errors describe each
# book alone; these arms share dates and most positions, so only a paired
# resample speaks to the gap between them.
control_dates, control_rets = daily_returns(control.get("equity_curve") or [])
treatment_dates, treatment_rets = daily_returns(treatment.get("equity_curve") or [])
if control_dates != treatment_dates:
raise SystemExit(
"Arm return streams are not date-aligned; a paired bootstrap would "
"resample mismatched days."
)
bootstrap = {
"method": (
"paired moving-block bootstrap on date-aligned daily returns; the same "
"block indices are drawn for both arms so the pairing and cross-arm "
"correlation are preserved"
),
"seed": BOOTSTRAP_SEED,
"replicates": BOOTSTRAP_REPLICATES,
"sessions": len(control_rets),
"pre_specified": (
"seed, replicate count and both block lengths were fixed before the run; "
"both are reported"
),
"scope": (
"exploratory uncertainty estimate from the observed window - not "
"out-of-sample validation and not a rollout gate"
),
"blocks": {},
}
for block in BOOTSTRAP_BLOCKS:
started = time.perf_counter()
bootstrap["blocks"][f"{block}_sessions"] = paired_block_bootstrap(
control_rets, treatment_rets,
block=block, replicates=BOOTSTRAP_REPLICATES, seed=BOOTSTRAP_SEED,
)
print(f" bootstrap block={block:<3} {BOOTSTRAP_REPLICATES} reps "
f"({time.perf_counter() - started:.0f}s)")
out = Path(args.out)
out.parent.mkdir(parents=True, exist_ok=True)
out.write_text(json.dumps({
"generated_at": datetime.now(timezone.utc).isoformat(),
"snapshot": str(snapshot),
"cache": str(cache),
"candidates_store": str(store),
"cadence": cadence,
"min_momentum_percentile": min_pct,
"candidate_count": len(candidates),
"held_fixed": [
"setup candidates (entry/stop/target/rr) generated once and shared",
"execution bars: snapshot nominal closes in both arms",
"volatility leg: vol_6m from candidate generation, never recomputed",
"non-momentum gates, max_positions, risk_per_trade, exit policy, re-entry policy",
],
"headline": {
label: {k: sim.get(k) for k in HEADLINE} for label, sim in results.items()
},
"deltas": {
k: (
round(treatment[k] - control[k], 2)
if isinstance(treatment.get(k), (int, float)) and isinstance(control.get(k), (int, float))
else None
)
for k in HEADLINE
},
"entry_diff": diff,
"yearly": yearly,
"paired_bootstrap": bootstrap,
"arms": {label: {k: v for k, v in sim.items() if k not in {"equity_curve", "benchmark_curve"}} for label, sim in results.items()},
}, indent=2, default=str) + "\n", encoding="utf-8")
print(f"\nentries only in control : {diff['only_control_count']}")
print(f"entries only in treatment: {diff['only_treatment_count']}")
print(f"shared entries : {diff['shared_entries']}")
print(f"years treatment better : {yearly['years_treatment_better']} "
f"/ worse {yearly['years_treatment_worse']} / equal {yearly['years_equal']}")
print("\npaired block bootstrap, treatment - control (95% interval):")
for name, res in bootstrap["blocks"].items():
print(f" block {name}")
for metric, stats in res.items():
print(f" {metric:<20} mean {stats['mean']:>8} "
f"[{stats['ci_2.5']:>8}, {stats['ci_97.5']:>8}] "
f"P(<0)={stats['share_below_zero']}")
print(f"report : {out}")
return 0
def parse_args() -> argparse.Namespace:
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
p.add_argument("--snapshot", default=str(DEFAULT_SNAPSHOT))
p.add_argument("--cache", default=str(DEFAULT_CACHE))
p.add_argument("--candidates", default=str(DEFAULT_CANDIDATES))
p.add_argument("--cadence", default="weekly", choices=("weekly", "daily"))
p.add_argument("--workers", type=int, default=6)
p.add_argument("--build-candidates", action="store_true", help="Build the candidate store and exit.")
p.add_argument("--out", default="reports/tr-total-return-ab.json")
return p.parse_args()
if __name__ == "__main__":
raise SystemExit(asyncio.run(main(parse_args())))