feat(universe): separate scanner membership from retained history

bootstrap_universe(prune_missing=True) ran DELETE FROM tickers, cascading OHLCV,
setups, scores and S/R levels away for any symbol that had merely dropped out of
the index. Migration 032's own comment predicted this. The survivorship audit
measured what it costs: 2.09 Sharpe against 1.57 on universe composition alone.

Migration 035 adds untracked_on, the sibling of delisted_on — delisted means the
security stopped trading, untracked means a prune took it out of the scanner.
Nothing is deleted either way.

Migration 036 separates three more things that were still conflated:

  universe_memberships       what the index actually contained, when
  tickers.untracked_on       excluded from scanning by a prune
  tickers.scan_disabled_on   the operator's own "stop scanning this"
  tickers.tracking_source    who owns the row: a bootstrap, or an operator

The operator flag is its own column because with one column serving both
writers, a prune -> re-entry cycle would clear an explicit "off" — re-entry
cannot tell who set it. Re-entry clears untracked_on and never touches
scan_disabled_on.

untracked_on is NOT current index membership. The scheduled sync runs
prune_missing=False, so a symbol can have left the index and still be scanned;
the two legitimately differ and nothing asserts they agree.

Tracking ownership is sticky towards manual: prune touches index-owned rows
only, and a bootstrap never converts manual to index — a hand-added symbol that
joins the index would otherwise lose its protection the moment it left again,
which is exactly when it is needed. An explicit add of an index-owned symbol
claims it as manual without touching either off-switch.

Membership is observed, never asserted. first_seen_on is the first live
sighting; absent_since is the first observation that found the symbol GONE, so
the interval is [first_seen_on, absent_since) — named for what it stores rather
than "last_seen_on", which would have claimed the opposite. Intervals are
append-only, so a symbol that leaves and returns has two and a point-in-time
query for the gap correctly answers "not a member". members_on() implements the
boundary rule once. A date before the first live sync returns empty: that means
"we were not looking", not "the index was empty".

A failed fetch drives nothing. The cache and seed fallbacks mean the live fetch
failed, and the seed lists are ~44 symbols against a ~500-symbol universe — a
regression test caught a seed fallback untracking 2 of 3 tickers. Recording,
pruning and clearing prune flags all require a live source; only adding still
runs, so a cold start can still bootstrap from seeds.

Migration decisions: existing rows become manual (fail-safe — a wrong guess
costs an un-pruned symbol, not one swept out of the scanner), and existing
untracked_on values are copied into scan_disabled_on, since the original writer
is ambiguous and re-entry clears the prune column.

Backtest eligibility is unchanged. _load_backtest_symbols deliberately does not
use active_only, so today's scanner state cannot erase yesterday's candidates,
and untracking a symbol still cannot silence a coverage gap in the report.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HAH7vVtn7qbcn1EVheecHQ

alembic/versions/035_ticker_untracked.py

@@ -0,0 +1,56 @@
+"""Separate scanner membership from retained history
+
+Revision ID: 035
+Revises: 034
+Create Date: 2026-09-07 00:00:00.000000
+
+Migration 032 stopped ``delete_ticker`` and the delisting path from destroying
+history, but left the other half of the problem open — its own comment said so:
+"Pruning a symbol that is merely no longer an index constituent still destroys
+history; that needs a tracked/membership state separate from delisting."
+
+``bootstrap_universe(prune_missing=True)`` still ran ``DELETE FROM tickers``,
+cascading OHLCV, setups and scores away for any symbol that had merely dropped
+out of the index. The survivorship audit measured what that costs: restoring 85
+removed constituents moved the same window from 2.09 Sharpe / 51.6% CAGR to
+1.57 / 38.1%. A prune that deletes is how a universe silently becomes the
+survivors.
+
+``untracked_on`` is the sibling of ``delisted_on``, and the distinction is the
+point:
+
+  delisted_on   the security stopped trading
+  untracked_on  it still trades; we stopped scanning it
+
+Both retain every row. NULL in both means "in the live path" — ``active_only``
+filters on both, so scanning, ranking, scoring, breadth and ingestion skip an
+untracked name, while the backtest's universe query deliberately does not, so
+today's membership cannot erase yesterday's candidates.
+
+Nullable and reversible: a symbol that returns to the index has ``untracked_on``
+cleared by the next bootstrap and rejoins the live path with its history intact.
+"""
+from typing import Sequence, Union
+
+from alembic import op
+import sqlalchemy as sa
+
+
+revision: str = "035"
+down_revision: Union[str, None] = "034"
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    op.add_column("tickers", sa.Column("untracked_on", sa.Date(), nullable=True))
+    # Same reasoning as ix_tickers_delisted_on: the live path tests this
+    # predicate on every universe scan, and untracked rows accumulate.
+    op.create_index("ix_tickers_untracked_on", "tickers", ["untracked_on"])
+
+
+def downgrade() -> None:
+    # Dropping the column re-admits every untracked symbol to the live path.
+    # The rows and their history survive either way; that is the whole point.
+    op.drop_index("ix_tickers_untracked_on", table_name="tickers")
+    op.drop_column("tickers", "untracked_on")

alembic/versions/036_universe_membership.py

@@ -0,0 +1,116 @@
+"""Record universe membership, tracking ownership and operator preference
+
+Revision ID: 036
+Revises: 035
+Create Date: 2026-09-07 00:00:00.000000
+
+Migration 035 separated "stopped trading" from "stopped scanning". This
+separates three more things that were still conflated:
+
+  universe_memberships   which symbols the index actually contained, when
+  tickers.tracking_source who owns the row: a bootstrap, or an operator
+  tickers.scan_disabled_on the operator's own "stop scanning this"
+
+**Why the operator flag is a separate column.** ``untracked_on`` was written by
+both the prune and ``set_tracked``. With one column a prune -> re-entry cycle
+would clear an operator's explicit "off", because re-entry cannot tell which
+writer set it. Splitting them makes the two survivable independently.
+
+**Existing "off" states are carried over, not reinterpreted.** Any row that
+already has ``untracked_on`` set was written by one of those two callers and
+there is no way to tell which. Copying the date into ``scan_disabled_on`` is the
+conservative reading: a later sync clears ``untracked_on`` on re-entry but never
+touches ``scan_disabled_on``, so a symbol someone deliberately switched off
+cannot be reactivated by an automated run.
+
+**Existing rows default to ``manual``.** Provenance cannot be recovered from
+``created_at``, and ``manual`` is the fail-safe value: manual rows are never
+pruned, so a bad guess costs an un-pruned symbol rather than a symbol swept out
+of the scanner. New rows created by a bootstrap are stamped ``index``
+explicitly. Nothing ever converts ``manual`` to ``index`` — a hand-added symbol
+that later joins the index keeps its protection.
+
+Membership history starts at first observation. ``first_seen_on`` is the first
+date a live fetch actually listed the symbol, never a reconstructed index
+effective date; earlier membership is simply unknown.
+"""
+from typing import Sequence, Union
+
+from alembic import op
+import sqlalchemy as sa
+
+
+revision: str = "036"
+down_revision: Union[str, None] = "035"
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    op.add_column("tickers", sa.Column("scan_disabled_on", sa.Date(), nullable=True))
+    op.create_index("ix_tickers_scan_disabled_on", "tickers", ["scan_disabled_on"])
+    op.add_column(
+        "tickers",
+        sa.Column(
+            "tracking_source",
+            sa.String(length=16),
+            nullable=False,
+            server_default="manual",
+        ),
+    )
+
+    # Every pre-existing row is treated as manually tracked: unprunable until a
+    # bootstrap has had a chance to claim the ones it owns.
+    op.execute("UPDATE tickers SET tracking_source = 'manual'")
+    # Preserve any existing "off" so an automated sync cannot reactivate it.
+    op.execute(
+        "UPDATE tickers SET scan_disabled_on = untracked_on "
+        "WHERE untracked_on IS NOT NULL"
+    )
+
+    op.create_table(
+        "universe_memberships",
+        sa.Column("id", sa.Integer(), primary_key=True),
+        sa.Column(
+            "ticker_id",
+            sa.Integer(),
+            sa.ForeignKey("tickers.id", ondelete="CASCADE"),
+            nullable=False,
+        ),
+        sa.Column("universe", sa.String(length=20), nullable=False),
+        sa.Column("first_seen_on", sa.Date(), nullable=False),
+        # NULL == still observed. Closing sets this and retains the row; a
+        # re-entry inserts a new one.
+        sa.Column("absent_since", sa.Date(), nullable=True),
+        sa.Column(
+            "created_at",
+            sa.DateTime(timezone=True),
+            server_default=sa.func.now(),
+            nullable=False,
+        ),
+    )
+    op.create_index(
+        "ix_universe_memberships_ticker_universe",
+        "universe_memberships",
+        ["ticker_id", "universe"],
+    )
+    op.create_index(
+        "ix_universe_memberships_universe_seen",
+        "universe_memberships",
+        ["universe", "absent_since"],
+    )
+
+
+def downgrade() -> None:
+    # Membership history is observation-only and cannot be reconstructed once
+    # dropped; the ticker rows and their price history are untouched either way.
+    op.drop_index(
+        "ix_universe_memberships_universe_seen", table_name="universe_memberships"
+    )
+    op.drop_index(
+        "ix_universe_memberships_ticker_universe", table_name="universe_memberships"
+    )
+    op.drop_table("universe_memberships")
+    op.drop_column("tickers", "tracking_source")
+    op.drop_index("ix_tickers_scan_disabled_on", table_name="tickers")
+    op.drop_column("tickers", "scan_disabled_on")

app/models/__init__.py

@@ -18,6 +18,7 @@ from app.models.regime_fundamental_observation import RegimeFundamentalObservati
 from app.models.benchmark_price import BenchmarkPrice
 from app.models.signal_context_snapshot import SignalContextSnapshot
 from app.models.system_event import SystemEvent
+from app.models.universe_membership import UniverseMembership
 from app.models.sec_filing_gap import SecFilingGap
 from app.models.job_run_state import JobRunState
 
@@ -46,4 +47,5 @@ __all__ = [
     "SystemEvent",
     "SecFilingGap",
     "JobRunState",
+    "UniverseMembership",
 ]

app/models/ticker.py

@@ -28,6 +28,32 @@ class Ticker(Base):
     delisted_on: Mapped[date | None] = mapped_column(Date, nullable=True, index=True)
     # How we learned: "form_25" (SEC confirmed), "manual" (operator).
     delisted_reason: Mapped[str | None] = mapped_column(String(32), nullable=True)
+    # The sibling of delisted_on, and the distinction matters: delisted means the
+    # security stopped trading; untracked means it still trades and a prune took
+    # it out of the scanner. Both retain every row — the delete this replaced
+    # cascaded away exactly the history a survivorship-honest backtest needs.
+    #
+    # This records *exclusion by pruning*, NOT current index membership. The
+    # scheduled sync runs prune_missing=False, so a symbol can have left the
+    # index and still be scanned; the two states legitimately differ. Membership
+    # itself lives in universe_memberships.
+    #
+    # active_only filters on it; the backtest's universe query deliberately does
+    # not, so today's scanner state cannot erase yesterday's candidates.
+    untracked_on: Mapped[date | None] = mapped_column(Date, nullable=True, index=True)
+    # The operator's own "stop scanning this", kept apart from the prune flag on
+    # purpose. If one column served both, a prune -> re-entry cycle would clear
+    # an explicit operator "off", because re-entry cannot tell which writer set
+    # it. Only set_tracked writes this; no bootstrap ever clears it.
+    scan_disabled_on: Mapped[date | None] = mapped_column(Date, nullable=True, index=True)
+    # Who owns this row's tracking: "index" (created by a universe bootstrap) or
+    # "manual" (an operator added it). Pruning only ever touches index-owned
+    # rows. Ownership is sticky towards manual — a bootstrap must never convert
+    # manual to index, or a hand-added symbol that happens to join the index
+    # would lose its protection when it later leaves.
+    tracking_source: Mapped[str] = mapped_column(
+        String(16), nullable=False, server_default="manual", default="manual"
+    )
     created_at: Mapped[datetime] = mapped_column(
         DateTime(timezone=True), default=datetime.utcnow, nullable=False
     )
@@ -43,3 +69,4 @@ class Ticker(Base):
     watchlist_entries = relationship("WatchlistEntry", back_populates="ticker", cascade="all, delete-orphan")
     ingestion_progress = relationship("IngestionProgress", back_populates="ticker", cascade="all, delete-orphan", uselist=False)
     earnings_events = relationship("EarningsEvent", back_populates="ticker", cascade="all, delete-orphan")
+    universe_memberships = relationship("UniverseMembership", back_populates="ticker", cascade="all, delete-orphan")

app/models/universe_membership.py

@@ -0,0 +1,59 @@
+from datetime import date, datetime
+
+from sqlalchemy import Date, DateTime, ForeignKey, Index, String
+from sqlalchemy.orm import Mapped, mapped_column, relationship
+
+from app.database import Base
+
+
+class UniverseMembership(Base):
+    """One observed interval of a symbol's membership in an index universe.
+
+    Append-only. A symbol that leaves and returns has two rows, so a
+    point-in-time query for the gap between them correctly answers "not a
+    member" — reopening the original row would erase that.
+
+    **Observed, not asserted.** ``first_seen_on`` is the first date a live
+    universe fetch actually listed the symbol, not the date it joined the index.
+    Anything earlier is unknown, and this table never guesses: the removals list
+    reconstructed by hand in ``scripts/data/sp500_removals.json`` stays a
+    research artifact rather than a seed.
+
+    Membership is independent of *tracking ownership* (``Ticker.tracking_source``)
+    and of whether the symbol is scanned. A manually added symbol that happens to
+    be an index constituent gets intervals like any other; being untracked or
+    scan-disabled does not close one.
+    """
+
+    __tablename__ = "universe_memberships"
+    __table_args__ = (
+        Index("ix_universe_memberships_ticker_universe", "ticker_id", "universe"),
+        # "Who was a member of U at date D" — the point-in-time query.
+        Index("ix_universe_memberships_universe_seen", "universe", "absent_since"),
+    )
+
+    id: Mapped[int] = mapped_column(primary_key=True)
+    ticker_id: Mapped[int] = mapped_column(
+        ForeignKey("tickers.id", ondelete="CASCADE"), nullable=False
+    )
+    # "sp500" | "nasdaq100" | "nasdaq_all" — a symbol can belong to several.
+    universe: Mapped[str] = mapped_column(String(20), nullable=False)
+    # The first observation that listed this symbol. INCLUSIVE lower bound. Not
+    # the date it joined the index — that is unknown and never guessed.
+    first_seen_on: Mapped[date] = mapped_column(Date, nullable=False)
+    # The first observation that found the symbol ABSENT — hence the name. It is
+    # *not* "last seen present": the symbol was present at every observation
+    # before this one and absent at this one, so it is an EXCLUSIVE upper bound.
+    # NULL means still present at the most recent observation.
+    #
+    #   member on D  <=>  first_seen_on <= D < absent_since (or absent_since IS NULL)
+    #
+    # The real departure lies somewhere in (previous observation, absent_since];
+    # with a daily sync that is a day, and with a missed week it is a week. The
+    # column records when we LOOKED, which is the only thing actually known.
+    absent_since: Mapped[date | None] = mapped_column(Date, nullable=True)
+    created_at: Mapped[datetime] = mapped_column(
+        DateTime(timezone=True), default=datetime.utcnow, nullable=False
+    )
+
+    ticker = relationship("Ticker", back_populates="universe_memberships")

app/routers/tickers.py

@@ -6,7 +6,12 @@ from sqlalchemy.ext.asyncio import AsyncSession
 from app.dependencies import get_db, require_access
 from app.models.user import User
 from app.schemas.common import APIEnvelope
-from app.schemas.ticker import TickerCreate, TickerDelistingUpdate, TickerResponse
+from app.schemas.ticker import (
+    TickerCreate,
+    TickerDelistingUpdate,
+    TickerResponse,
+    TickerTrackingUpdate,
+)
 from app.services import ticker_service
 
 router = APIRouter(tags=["tickers"])
@@ -18,7 +23,23 @@ async def create_ticker(
     _user: User = Depends(require_access),
     db: AsyncSession = Depends(get_db),
 ):
-    """Add a new ticker to the registry."""
+    """Add a ticker to the registry, or claim an existing index-owned one.
+
+    Responses:
+
+    - **New symbol** — created as manually tracked and returned.
+    - **Existing, index-owned** — 200 with the existing row, now marked manually
+      tracked. Not a 409: the operator is saying "keep this stock", and refusing
+      while leaving it prunable would defeat the protection they just asked for.
+      Ownership only ever moves towards manual, so a bootstrap cannot take it
+      back when the symbol next joins the index.
+    - **Existing, already manual** — 409 Duplicate. Nothing left to change.
+
+    Claiming never clears ``scan_disabled_on`` or ``untracked_on``: wanting a
+    stock kept is not the same as asking for it to be scanned, and silently
+    re-enabling something switched off by hand would undo a decision nobody
+    revisited. ``POST /tickers/{symbol}/tracking`` is the explicit control.
+    """
     ticker = await ticker_service.add_ticker(db, body.symbol)
     return APIEnvelope(
         status="success",
@@ -70,6 +91,24 @@ async def mark_ticker_delisted(
     return APIEnvelope(status="success", data={"changed": changed})
 
 
+@router.post("/tickers/{symbol}/tracking", response_model=APIEnvelope)
+async def set_ticker_tracking(
+    symbol: str,
+    body: TickerTrackingUpdate,
+    _user: User = Depends(require_access),
+    db: AsyncSession = Depends(get_db),
+):
+    """Include or exclude one symbol from scanning, keeping its history.
+
+    Untracking is not delisting — the security still trades, we just stop
+    scanning it. The case this exists for: a manually added symbol no provider
+    can serve, which would otherwise sit in the universe forever producing no
+    bars and flagging every backtest as incomplete.
+    """
+    changed = await ticker_service.set_tracked(db, symbol, tracked=body.tracked)
+    return APIEnvelope(status="success", data={"changed": changed})
+
+
 @router.delete("/tickers/{symbol}/delisting", response_model=APIEnvelope)
 async def clear_ticker_delisting(
     symbol: str,

app/schemas/ticker.py

@@ -27,3 +27,13 @@ class TickerDelistingUpdate(BaseModel):
     delisted_on: date = Field(
         ..., description="Effective date the symbol stopped trading"
     )
+
+
+class TickerTrackingUpdate(BaseModel):
+    tracked: bool = Field(
+        ...,
+        description=(
+            "True to scan this symbol, False to exclude it from the live path. "
+            "History is retained either way — this is not a delisting."
+        ),
+    )

app/services/backtest_service.py

@@ -50,14 +50,16 @@ import statistics
 from collections import defaultdict
 from collections.abc import Callable
 from concurrent.futures import ProcessPoolExecutor
+from dataclasses import dataclass, field
 from datetime import date, datetime, timezone
 from types import SimpleNamespace
 from typing import Any
 
-from sqlalchemy import select
+from sqlalchemy import func, select
 from sqlalchemy.ext.asyncio import AsyncSession
 
 from app.config import settings
+from app.models.ohlcv import OHLCVRecord
 from app.models.ticker import Ticker
 from app.services import settings_store
 from app.services.admin_service import get_activation_config, update_setting
@@ -497,16 +499,40 @@ def _replay_ticker(
     benchmark_closes: dict[date, float] | None = None,
     target_model: str = PRODUCTION_GTL_TARGET_MODEL,
     cadence: str = DEFAULT_BACKTEST_CADENCE,
+    terminal_date: date | None = None,
 ) -> list[dict]:
-    """Walk one ticker at the selected cadence and resolve each setup outcome."""
+    """Walk one ticker at the selected cadence and resolve each setup outcome.
+
+    ``terminal_date`` marks a symbol whose final bar is a research terminal mark
+    (a confirmed near-total loss written by ``prep_survivorship_snapshot.py``).
+    The normal loop stops ``HORIZON`` bars early so every setup has a full
+    resolution window, which would hide every signal in the last 29 real bars —
+    exactly the ones a collapse resolves. For a marked symbol the walk instead
+    runs through the last *real* bar and never proposes a setup on the synthetic
+    one. Nothing else changes: outcomes still come from the real forward bars,
+    and a setup the shortened history cannot resolve is skipped rather than
+    counted as an expiry.
+    """
     cadence = validate_backtest_cadence(cadence)
     step_sessions = backtest_step_sessions(cadence)
     candidates: list[dict] = []
     n = len(records)
+
+    if terminal_date is not None:
+        if not records or records[-1].date != terminal_date:
+            raise ValueError(
+                f"{symbol}: terminal event {terminal_date} does not match the final bar "
+                f"{records[-1].date if records else None}"
+            )
+        if n < MIN_LOOKBACK + 1:
+            return candidates
+        stop_index = n - 1  # exclusive: the last signal date is the last real bar
+    else:
         if n < MIN_LOOKBACK + HORIZON:
             return candidates
+        stop_index = n - HORIZON
 
-    for i in range(MIN_LOOKBACK - 1, n - HORIZON, step_sessions):
+    for i in range(MIN_LOOKBACK - 1, stop_index, step_sessions):
         window = records[: i + 1]
         forward = records[i + 1 :]
         forward_bars = [Bar(date=r.date, high=r.high, low=r.low) for r in forward]
@@ -1278,6 +1304,7 @@ def _replay_and_signals(
     target_model: str = PRODUCTION_GTL_TARGET_MODEL,
     cadence: str = DEFAULT_BACKTEST_CADENCE,
     signal_only: bool = False,
+    terminal_date: date | None = None,
 ) -> tuple[list[dict], dict]:
     """The CPU-bound per-ticker work, as a top-level (picklable) function so it can
     run in a worker process. Takes primitive column arrays (cheap to pickle),
@@ -1303,6 +1330,7 @@ def _replay_and_signals(
             benchmark_closes,
             target_model,
             cadence,
+            terminal_date,
         )
     return (
         candidates,
@@ -1475,6 +1503,217 @@ async def _rollback_quietly(db: AsyncSession, context: str) -> None:
         logger.exception("Session rollback after %s also failed", context)
 
 
+# Failures are listed individually up to this many; beyond it the report keeps
+# the counts and says the list was truncated.
+DATA_QUALITY_FAILURE_SAMPLE = 25
+# Bar-less universe rows are named rather than counted while the list is short.
+NO_DATA_SAMPLE = 5
+
+
+def _has_short_history(columns: tuple) -> int:
+    """1 when a loaded ticker has too few bars for ``_replay_ticker`` to resolve
+    a single outcome — a recent listing, not an error, but worth counting since
+    it looks identical to "found no setups" in the report."""
+    return int(len(columns[0]) < MIN_LOOKBACK + HORIZON)
+
+
+@dataclass
+class _DataQuality:
+    """Completeness ledger for one backtest run.
+
+    Both dispatch paths swallow per-ticker exceptions so a single unreadable
+    name can't abort a 500-symbol replay. That is right for a long run and wrong
+    for its report: a snapshot whose schema predates a migration fails *every*
+    ``_fetch_columns`` call and still writes a report that reads like an
+    ordinary one that simply found no setups. This records what actually
+    happened so the report — and the UI — can say so.
+
+    Every universe symbol ends in exactly one of ``no_data``, ``failures`` or
+    ``evaluated``. ``short_history`` is a subset of ``evaluated`` (loaded, but
+    too few bars to produce a setup), not a fourth bucket. :meth:`as_report`
+    checks the three against ``expected`` and treats an unbalanced ledger as a
+    failure in its own right — a counter that stopped being maintained should
+    surface as loudly as a missing ticker.
+
+    **What ``complete`` does and does not claim.** It means the run's processing
+    and coverage checks passed: every expected symbol was loaded and evaluated,
+    and the benchmark was there. It says nothing about whether the underlying
+    prices are *correct* — a complete run over silently wrong bars is still
+    complete. Splits, adjustment drift and bad vendor prints are a different
+    problem, and this ledger must not be read as clearing them.
+    """
+
+    expected: int = 0
+    evaluated: int = 0
+    short_history: int = 0
+    no_data_symbols: list[str] = field(default_factory=list)
+    failures: list[dict] = field(default_factory=list)
+    benchmark_sessions: int = 0
+    benchmark_error: str | None = None
+    benchmark_first: date | None = None
+    benchmark_last: date | None = None
+
+    def record_failure(self, symbol: str, stage: str, exc: BaseException) -> None:
+        """Stage is ``fetch`` (reading bars) or ``replay`` (evaluating them).
+
+        Only the first line is kept: a SQLAlchemy error appends the full SQL and
+        its parameters, which buries the one useful clause ("no such column:
+        tickers.cik") under 300 characters of SELECT. The full traceback is
+        already in the run log.
+        """
+        first_line = str(exc).strip().splitlines()[0] if str(exc).strip() else ""
+        self.failures.append({
+            "symbol": symbol,
+            "stage": stage,
+            "error": f"{type(exc).__name__}: {first_line}"[:300],
+        })
+
+    def record_benchmark(self, closes: dict[date, float], error: BaseException | None) -> None:
+        self.benchmark_sessions = len(closes)
+        self.benchmark_error = f"{type(error).__name__}: {error}"[:300] if error else None
+        if closes:
+            self.benchmark_first = min(closes)
+            self.benchmark_last = max(closes)
+
+    def as_report(self) -> dict:
+        # Lazily imported like ``_load_benchmark_closes_for_backtest`` does, so
+        # building a report never drags in the provider stack.
+        from app.services.benchmark_service import BENCHMARK_SYMBOL
+
+        by_stage: dict[str, int] = defaultdict(int)
+        for entry in self.failures:
+            by_stage[entry["stage"]] += 1
+        no_data = len(self.no_data_symbols)
+        accounted = no_data + len(self.failures) + self.evaluated
+        balanced = accounted == self.expected
+
+        # A run is "failed" when its headline numbers can't be read as the
+        # strategy the report claims to describe; "degraded" when the strategy
+        # is intact but ran on less than the whole universe.
+        reasons: list[str] = []
+        if self.expected == 0:
+            reasons.append("no symbols in the backtest universe")
+        elif self.evaluated == 0:
+            reasons.append("no symbol was evaluated successfully")
+        if not balanced:
+            reasons.append(
+                f"ledger does not balance: {accounted} symbols accounted for, "
+                f"{self.expected} expected"
+            )
+        if self.benchmark_sessions == 0:
+            # Residual 12-1 momentum silently degrades to raw 12-1 without a
+            # benchmark stream, so the gate is not the production gate.
+            reasons.append(
+                "benchmark unavailable — residual momentum fell back to raw momentum"
+                + (f" ({self.benchmark_error})" if self.benchmark_error else "")
+            )
+        failed = bool(reasons)
+
+        # These strings are shown verbatim in the UI, so they read as sentences.
+        def _symbols(n: int) -> str:
+            return f"{n} symbol" if n == 1 else f"{n} symbols"
+
+        if by_stage.get("fetch"):
+            reasons.append(f"{_symbols(by_stage['fetch'])} failed to load")
+        if by_stage.get("replay"):
+            reasons.append(f"{_symbols(by_stage['replay'])} failed during replay")
+        if no_data:
+            # Named while the list is short, because a universe row with no bars
+            # is a standing wart that recurs on every run until someone clears
+            # it — an unnamed count would just nag.
+            named = ", ".join(sorted(self.no_data_symbols)[:NO_DATA_SAMPLE])
+            if no_data > NO_DATA_SAMPLE:
+                named += ", …"
+            reasons.append(f"{_symbols(no_data)} had no stored bars ({named})")
+
+        return {
+            "status": "failed" if failed else ("degraded" if reasons else "ok"),
+            "complete": not reasons,
+            "reasons": reasons,
+            "expected": self.expected,
+            "loaded": self.expected - no_data - by_stage.get("fetch", 0),
+            "evaluated": self.evaluated,
+            "no_data": no_data,
+            "no_data_symbols": sorted(self.no_data_symbols)[:NO_DATA_SAMPLE],
+            "short_history": self.short_history,
+            "failed": len(self.failures),
+            "failures_by_stage": dict(by_stage),
+            "failures": self.failures[:DATA_QUALITY_FAILURE_SAMPLE],
+            "failures_truncated": len(self.failures) > DATA_QUALITY_FAILURE_SAMPLE,
+            "benchmark": {
+                "symbol": BENCHMARK_SYMBOL,
+                "sessions": self.benchmark_sessions,
+                "first": self.benchmark_first.isoformat() if self.benchmark_first else None,
+                "last": self.benchmark_last.isoformat() if self.benchmark_last else None,
+                "momentum_basis": "residual" if self.benchmark_sessions else "raw_fallback",
+                "error": self.benchmark_error,
+            },
+        }
+
+
+async def _load_backtest_symbols(db: AsyncSession) -> list[str]:
+    """Every ticker with stored history — deliberately *not* ``active_only``.
+
+    A replay grades what the strategy could have selected at each past date, so
+    it must see symbols that have since been delisted or dropped out of the
+    index. Filtering on today's membership here is precisely how a universe
+    becomes the survivors: the audit in ``docs/research/survivorship-audit.md``
+    measured 2.09 Sharpe against 1.57 for the same window on that difference
+    alone. Guarded by
+    ``tests/unit/test_universe_membership.py::test_todays_membership_does_not_erase_yesterdays_candidates``.
+
+    Plain strings, not Ticker instances: the rollbacks in the replay loop expire
+    any ORM objects held across them, and touching an expired attribute
+    afterwards triggers sync lazy-loading, which raises on an AsyncSession.
+    """
+    result = await db.execute(select(Ticker.symbol).order_by(Ticker.symbol))
+    return list(result.scalars().all())
+
+
+class TerminalEventError(RuntimeError):
+    """A terminal-event map does not describe the snapshot it was given."""
+
+
+async def _validate_terminal_events(
+    db: AsyncSession, terminal_events: dict[str, date], symbols: list[str]
+) -> None:
+    """Check every mapped symbol before any replay is dispatched.
+
+    ``_replay_ticker`` also rejects a mismatch, but both dispatch paths swallow
+    per-ticker exceptions so the replay survives one unreadable name — which
+    would turn a stale events file into a quietly incomplete report instead of a
+    failure. Validating up front makes the whole run fail, before any work.
+    """
+    tracked = set(symbols)
+    missing = sorted(set(terminal_events) - tracked)
+
+    rows = await db.execute(
+        select(Ticker.symbol, func.max(OHLCVRecord.date))
+        .join(OHLCVRecord, OHLCVRecord.ticker_id == Ticker.id)
+        .where(Ticker.symbol.in_(list(terminal_events)))
+        .group_by(Ticker.symbol)
+    )
+    final_bar = dict(rows.all())
+
+    mismatched = []
+    for symbol, expected in sorted(terminal_events.items()):
+        if symbol in missing:
+            continue
+        actual = final_bar.get(symbol)
+        if actual != expected:
+            mismatched.append(f"{symbol}: events file says {expected}, final bar is {actual}")
+
+    problems = []
+    if missing:
+        problems.append(f"symbols not in this snapshot: {', '.join(missing)}")
+    if mismatched:
+        problems.append("; ".join(mismatched))
+    if problems:
+        raise TerminalEventError(
+            "Terminal-event map does not match the snapshot — " + " | ".join(problems)
+        )
+
+
 async def _fetch_columns(db: AsyncSession, symbol: str) -> tuple | None:
     """Read one ticker's OHLCV and detach it to primitive column arrays in the
     event loop (safe ORM access), ready to hand to a worker. None if no data."""
@@ -4066,19 +4305,25 @@ async def run_backtest(
     *,
     target_model: str = PRODUCTION_GTL_TARGET_MODEL,
     cadence: str = DEFAULT_BACKTEST_CADENCE,
+    terminal_events: dict[str, date] | None = None,
 ) -> dict:
-    """Replay every ticker and aggregate the Phase-1 reports for the current config."""
+    """Replay every ticker and aggregate the Phase-1 reports for the current config.
+
+    ``terminal_events`` maps a symbol to the date of a research terminal mark on
+    its final bar (see ``scripts/prep_survivorship_snapshot.py``). Unmapped
+    symbols keep the ordinary replay bounds, so an empty or omitted map leaves
+    every existing run byte-identical.
+    """
     target_model = validate_backtest_target_model(target_model)
     cadence = validate_backtest_cadence(cadence)
+    terminal_events = terminal_events or {}
     config = await get_recommendation_config(db)
     activation = await get_activation_config(db)
 
-    # Plain strings, not Ticker instances: the rollbacks below expire any ORM
-    # objects held across them, and touching an expired attribute afterwards
-    # triggers sync lazy-loading, which raises on an AsyncSession.
-    result = await db.execute(select(Ticker.symbol).order_by(Ticker.symbol))
-    symbols = list(result.scalars().all())
+    symbols = await _load_backtest_symbols(db)
     total = len(symbols)
+    if terminal_events:
+        await _validate_terminal_events(db, terminal_events, symbols)
     rank_only_symbols = await _load_research_rank_only_symbols(db)
     if rank_only_symbols:
         logger.info(json.dumps({
@@ -4086,6 +4331,7 @@ async def run_backtest(
             "count": len(rank_only_symbols),
         }))
 
+    quality = _DataQuality(expected=total)
     candidates: list[dict] = []
     # Signal IC remains a weekly, non-overlapping diagnostic regardless of the
     # entry cadence. Production activation ranks are assigned from candidates
@@ -4096,13 +4342,16 @@ async def run_backtest(
     # if SPY benchmark data is unavailable, the residual signal simply won't be
     # emitted and the rest of the report remains valid.
     benchmark_closes: dict[date, float] = {}
+    benchmark_error: Exception | None = None
     try:
         benchmark_closes = await _load_benchmark_closes_for_backtest(
             db, days=settings.ohlcv_history_days + 365
         )
-    except Exception:
+    except Exception as exc:
+        benchmark_error = exc
         logger.exception("Benchmark load for residual momentum failed")
         await _rollback_quietly(db, "benchmark load")
+    quality.record_benchmark(benchmark_closes, benchmark_error)
 
     def _merge(result: tuple[list[dict], dict]) -> None:
         cands, series = result
@@ -4135,16 +4384,23 @@ async def run_backtest(
         with pool:
             for start in range(0, total, chunk):
                 batch = symbols[start : start + chunk]
-                futures = []
+                # Symbol travels with its future: ``gather`` returns bare
+                # exceptions, so without the pairing a worker failure can't name
+                # the ticker it lost.
+                dispatched: list[tuple[str, Any]] = []
                 for symbol in batch:
                     try:
                         columns = await _fetch_columns(db, symbol)
-                    except Exception:
+                    except Exception as exc:
+                        quality.record_failure(symbol, "fetch", exc)
                         logger.exception("Backtest fetch failed for %s", symbol)
                         await _rollback_quietly(db, f"fetch for {symbol}")
                         continue
-                    if columns is not None:
-                        futures.append(loop.run_in_executor(
+                    if columns is None:
+                        quality.no_data_symbols.append(symbol)
+                        continue
+                    quality.short_history += _has_short_history(columns)
+                    dispatched.append((symbol, loop.run_in_executor(
                         pool,
                         _replay_and_signals,
                         symbol,
@@ -4155,11 +4411,21 @@ async def run_backtest(
                         target_model,
                         cadence,
                         symbol in rank_only_symbols,
-                        ))
-                for result in await asyncio.gather(*futures, return_exceptions=True):
-                    if isinstance(result, Exception):
-                        logger.error(json.dumps({"event": "backtest_worker_error", "message": str(result)}))
+                        terminal_events.get(symbol),
+                    )))
+                results = await asyncio.gather(
+                    *(fut for _, fut in dispatched), return_exceptions=True
+                )
+                for (symbol, _), result in zip(dispatched, results):
+                    if isinstance(result, BaseException):
+                        quality.record_failure(symbol, "replay", result)
+                        logger.error(json.dumps({
+                            "event": "backtest_worker_error",
+                            "symbol": symbol,
+                            "message": str(result),
+                        }))
                     else:
+                        quality.evaluated += 1
                         _merge(result)
                 done += len(batch)
                 if progress_cb is not None:
@@ -4172,7 +4438,16 @@ async def run_backtest(
                 progress_cb(index, total, symbol)
             try:
                 columns = await _fetch_columns(db, symbol)
-                if columns is not None:
+            except Exception as exc:
+                quality.record_failure(symbol, "fetch", exc)
+                logger.exception("Backtest fetch failed for %s", symbol)
+                await _rollback_quietly(db, f"fetch for {symbol}")
+                continue
+            if columns is None:
+                quality.no_data_symbols.append(symbol)
+                continue
+            quality.short_history += _has_short_history(columns)
+            try:
                 _merge(await asyncio.to_thread(
                     _replay_and_signals,
                     symbol,
@@ -4183,10 +4458,14 @@ async def run_backtest(
                     target_model,
                     cadence,
                     symbol in rank_only_symbols,
+                    terminal_events.get(symbol),
                 ))
-            except Exception:
+            except Exception as exc:
+                quality.record_failure(symbol, "replay", exc)
                 logger.exception("Backtest replay failed for %s", symbol)
                 await _rollback_quietly(db, f"replay for {symbol}")
+                continue
+            quality.evaluated += 1
 
     if progress_cb is not None and total:
         progress_cb(total, total, "")
@@ -4298,12 +4577,21 @@ async def run_backtest(
             # backstop for whichever DB call actually failed.
             await _rollback_quietly(db, "portfolio simulation")
 
+    data_quality = quality.as_report()
+    if not data_quality["complete"]:
+        logger.warning(json.dumps({
+            "event": "backtest_incomplete",
+            "status": data_quality["status"],
+            "reasons": data_quality["reasons"],
+        }))
+
     report = {
         "generated_at": datetime.now(timezone.utc).isoformat(),
         "tickers": total,
         "rank_only_tickers": len(rank_only_symbols),
         "candidates": len(candidates),
         "qualified": len(qualified),
+        "data_quality": data_quality,
         "params": {
             # Keep step_days for old report consumers; the value counts stored
             # market sessions rather than calendar days.

app/services/ticker_service.py

@@ -16,6 +16,13 @@ logger = logging.getLogger(__name__)
 REASON_FORM_25 = "form_25"   # SEC Form 25/25-NSE/15 confirmed the exchange exit
 REASON_MANUAL = "manual"     # an operator decided
 
+# Ticker.tracking_source — who owns this row's presence in the tracked set.
+# Ownership is sticky towards manual: a bootstrap creates ``index`` rows and an
+# operator's add claims a row as ``manual``, but nothing ever converts manual
+# back. Only index-owned rows are prunable.
+SOURCE_INDEX = "index"
+SOURCE_MANUAL = "manual"
+
 # How long a symbol must be without bars before we spend an SEC request asking
 # whether it delisted. Guards against a market-data outage probing the whole
 # universe at once; a real delisting is still stale days later.
@@ -43,21 +50,42 @@ def _sec_client_factory():
 
 
 def active_only(stmt, *, as_of: date | None = None):
-    """Restrict a Ticker query to symbols that still trade.
+    """Restrict a Ticker query to symbols in today's live signal path.
+
+    Three independent ways to leave it, and they mean different things:
+
+    ``delisted_on``       the security stopped trading
+    ``untracked_on``      a prune excluded it from scanning
+    ``scan_disabled_on``  an operator switched it off by hand
+
+    Any one of them excludes a symbol here. None deletes anything — the rows and
+    their history stay, which is what a survivorship-honest backtest needs.
+
+    ``untracked_on`` is not "no longer an index constituent": the scheduled sync
+    runs ``prune_missing=False``, so a symbol can have left the index and still
+    be scanned. Membership lives in ``universe_memberships`` and is not consulted
+    here — this stays a pure column filter so it costs nothing on the ~8 live
+    call sites that use it.
 
     Opt-in on purpose rather than folded into a shared getter: list and admin
-    views deliberately keep delisted rows so the delisting is *visible*, which a
+    views deliberately keep both kinds of row so the state is *visible*, which a
     silent default would undo. Apply this on the live signal path — scanning,
-    ranking, scoring, breadth, ingestion — and nowhere else.
+    ranking, scoring, breadth, ingestion — and nowhere else. In particular the
+    backtest's universe query must NOT use it: today's membership would silently
+    erase yesterday's candidates.
 
     ``delisted_on`` is an *effective* date, and a Form 25 is known ten days
     before it takes effect, so a future date must not drop the symbol yet — it
-    is still trading and still worth scanning and ingesting. Compared in SQL
-    against the database's own date; ``as_of`` overrides it for tests.
+    is still trading and still worth scanning and ingesting. The other two read
+    the same way, so ``as_of`` answers "what was in the scanner then?" for all
+    three rather than only one. Compared in SQL against the database's own date;
+    ``as_of`` overrides it for tests.
     """
     cutoff = func.current_date() if as_of is None else as_of
     return stmt.where(
-        or_(Ticker.delisted_on.is_(None), Ticker.delisted_on > cutoff)
+        or_(Ticker.delisted_on.is_(None), Ticker.delisted_on > cutoff),
+        or_(Ticker.untracked_on.is_(None), Ticker.untracked_on > cutoff),
+        or_(Ticker.scan_disabled_on.is_(None), Ticker.scan_disabled_on > cutoff),
     )
 
 
@@ -78,10 +106,27 @@ async def add_ticker(db: AsyncSession, symbol: str) -> Ticker:
         )
 
     result = await db.execute(select(Ticker).where(Ticker.symbol == normalised))
-    if result.scalar_one_or_none() is not None:
+    existing = result.scalar_one_or_none()
+    if existing is not None:
+        if existing.tracking_source == SOURCE_MANUAL:
             raise DuplicateError(f"Ticker already exists: {normalised}")
+        # An explicit manual add of an index-owned symbol claims it: the operator
+        # wants this stock kept regardless of the index, so it must stop being
+        # prunable. Ownership only ever moves towards manual — a bootstrap never
+        # moves it back, or the protection would evaporate the moment the symbol
+        # rejoined the index.
+        #
+        # Deliberately does not touch scan_disabled_on or untracked_on: claiming
+        # ownership is not the same as asking for it to be scanned, and silently
+        # re-enabling something switched off by hand would undo a decision
+        # nobody revisited. ``set_tracked`` is the explicit control for that.
+        existing.tracking_source = SOURCE_MANUAL
+        await db.commit()
+        await db.refresh(existing)
+        logger.info("ticker %s claimed as manually tracked", normalised)
+        return existing
 
-    ticker = Ticker(symbol=normalised)
+    ticker = Ticker(symbol=normalised, tracking_source=SOURCE_MANUAL)
     db.add(ticker)
     await db.commit()
     await db.refresh(ticker)
@@ -229,6 +274,45 @@ async def confirm_delisting(
     return known
 
 
+async def set_tracked(db: AsyncSession, symbol: str, *, tracked: bool) -> bool:
+    """Include or exclude one symbol from the live path. True if it changed.
+
+    The per-symbol counterpart to ``bootstrap_universe(prune_missing=True)``,
+    which can only act on the whole universe at once — and would sweep every
+    manually added symbol along with it, since a manual add is by definition not
+    in the fetched constituent list.
+
+    Switching off is not delisting: the security still trades, we just stop
+    scanning it. Use this for a symbol that cannot produce data (no provider
+    coverage) or that an operator simply does not want scanned. The rows and
+    their history stay either way, so a backtest still sees them — untracking is
+    not a way to make a coverage gap disappear from the completeness report.
+
+    Writes ``scan_disabled_on``, never ``untracked_on``: the operator's decision
+    has to survive a prune -> re-entry cycle, which clears the prune flag.
+    """
+    normalised = symbol.strip().upper()
+    result = await db.execute(select(Ticker).where(Ticker.symbol == normalised))
+    ticker = result.scalar_one_or_none()
+    if ticker is None:
+        raise NotFoundError(f"Ticker not found: {normalised}")
+    if (ticker.scan_disabled_on is None) == tracked:
+        return False
+
+    await db.execute(
+        update(Ticker)
+        .where(Ticker.id == ticker.id)
+        # Stamped by the database for the same reason bootstrap_universe does:
+        # active_only compares it against func.current_date().
+        .values(scan_disabled_on=None if tracked else func.current_date())
+    )
+    await db.commit()
+    logger.info(
+        "ticker %s scanning %s", normalised, "enabled" if tracked else "disabled"
+    )
+    return True
+
+
 async def clear_delisted(db: AsyncSession, symbol: str) -> bool:
     """Un-retire a symbol. True if it had been marked.
 

app/services/ticker_universe_service.py

@@ -12,17 +12,18 @@ import logging
 import os
 import re
 from collections.abc import Iterable
-from datetime import datetime, timezone
+from datetime import date, datetime, timezone
 from pathlib import Path
 
 import httpx
-from sqlalchemy import delete, select
+from sqlalchemy import func, or_, select, update
 from sqlalchemy.ext.asyncio import AsyncSession
 
 from app.config import settings
 from app.exceptions import ProviderError, ValidationError
 from app.models.ticker import Ticker
-from app.services import settings_store
+from app.models.universe_membership import UniverseMembership
+from app.services import settings_store, ticker_service
 
 logger = logging.getLogger(__name__)
 
@@ -333,6 +334,114 @@ async def backfill_ticker_names(db: AsyncSession, *, only_missing: bool = True)
     return {"updated": updated, "checked": len(targets), "unmatched": len(targets) - updated}
 
 
+# Sources that represent an actual look at the index today. A ``cache`` or
+# ``seed`` result means the live fetch FAILED — the seed lists are ~44 symbols,
+# so treating one as an observation would close ~460 memberships on a network
+# blip. Membership is only ever recorded from a live read.
+LIVE_UNIVERSE_SOURCES = frozenset({
+    "wikipedia_sp500", "wikipedia_nasdaq100", "nasdaq_trader", "public",
+})
+
+
+async def members_on(db: AsyncSession, universe: str, as_of: date) -> list[str]:
+    """Which symbols were observed in *universe* on *as_of*, alphabetically.
+
+    The point-in-time question item 3 exists to answer, with the interval
+    boundaries written down once as code rather than re-derived at each call
+    site:
+
+        member on D  <=>  first_seen_on <= D < absent_since (or absent_since IS NULL)
+
+    Lower bound inclusive, upper bound exclusive. ``absent_since`` is the first
+    observation that found the symbol *gone*, so on that date it was already
+    absent.
+
+    **This is observation history, not index history.** A date before the first
+    successful live sync returns an empty list — that is "we were not looking",
+    not "the index was empty", and the caller has to know the difference. The
+    reconstructed removals in ``scripts/data/sp500_removals.json`` are research
+    data and deliberately do not seed this table.
+    """
+    rows = await db.execute(
+        select(Ticker.symbol)
+        .join(UniverseMembership, UniverseMembership.ticker_id == Ticker.id)
+        .where(
+            UniverseMembership.universe == universe,
+            UniverseMembership.first_seen_on <= as_of,
+            or_(
+                UniverseMembership.absent_since.is_(None),
+                UniverseMembership.absent_since > as_of,
+            ),
+        )
+        .order_by(Ticker.symbol)
+    )
+    return list(rows.scalars().all())
+
+
+async def _observe_membership(
+    db: AsyncSession, universe: str, present: set[str]
+) -> dict[str, int]:
+    """Record what a live universe fetch just saw. Returns opened/closed counts.
+
+    Runs on every sync, not only when pruning: the scheduled job uses
+    ``prune_missing=False``, so membership recorded only on manual prunes would
+    be full of holes. Recording membership and deciding what to scan are
+    separate concerns, and this is the recording one — it never touches
+    ``untracked_on``, ``scan_disabled_on`` or ``tracking_source``, and it applies
+    to manually tracked symbols exactly as it does to index-owned ones.
+
+    Intervals are append-only. A symbol that leaves and returns gets a second
+    row, so a point-in-time query for the gap answers "not a member"; reopening
+    the first row would erase that.
+
+    Dates come from the database (``current_date``), matching every other date
+    this module writes. ``first_seen_on`` is the first *observation*, not an
+    index effective date — anything earlier is unknown and is not guessed.
+    """
+    rows = (await db.execute(
+        select(Ticker.id, Ticker.symbol).where(Ticker.symbol.in_(present))
+    )).all()
+    ticker_ids = {symbol: tid for tid, symbol in rows}
+
+    open_rows = (await db.execute(
+        select(UniverseMembership.ticker_id, Ticker.symbol)
+        .join(Ticker, Ticker.id == UniverseMembership.ticker_id)
+        .where(
+            UniverseMembership.universe == universe,
+            UniverseMembership.absent_since.is_(None),
+        )
+    )).all()
+    open_symbols = {symbol for _, symbol in open_rows}
+
+    opened = 0
+    for symbol in sorted(present - open_symbols):
+        ticker_id = ticker_ids.get(symbol)
+        if ticker_id is None:
+            continue
+        db.add(UniverseMembership(
+            ticker_id=ticker_id,
+            universe=universe,
+            first_seen_on=func.current_date(),
+        ))
+        opened += 1
+
+    departed_ids = [tid for tid, symbol in open_rows if symbol not in present]
+    closed = 0
+    if departed_ids:
+        result = await db.execute(
+            update(UniverseMembership)
+            .where(
+                UniverseMembership.universe == universe,
+                UniverseMembership.absent_since.is_(None),
+                UniverseMembership.ticker_id.in_(departed_ids),
+            )
+            .values(absent_since=func.current_date())
+        )
+        closed = int(result.rowcount or 0)
+
+    return {"opened": opened, "closed": closed}
+
+
 async def bootstrap_universe(
     db: AsyncSession,
     universe: str,
@@ -341,42 +450,98 @@ async def bootstrap_universe(
 ) -> dict[str, int | str]:
     """Upsert ticker universe into tracked tickers.
 
-    Returns summary counts for added/existing/deleted symbols.
+    ``prune_missing`` marks symbols no longer in the universe as untracked; it
+    never deletes them. "Not in today's scanner" and "not stored for historical
+    analysis" are different states, and conflating them is what makes a universe
+    quietly become the survivors. Symbols that have rejoined the universe have
+    the mark cleared and return to the live path with their history intact.
+
+    Returns summary counts for added / already-tracked / untracked / retracked.
     """
     normalised_universe = _validate_universe(universe)
     symbols, source = await fetch_universe_symbols(db, normalised_universe)
 
+    # A cache or seed result means the live fetch FAILED. The seed lists are ~44
+    # symbols against a ~500-symbol universe, so treating one as the truth would
+    # untrack the entire index on a single network blip. Nothing that acts on
+    # "this symbol is/isn't in the index" may run on fallback data: not pruning,
+    # not clearing a prune flag, not recording membership. Adding is the one
+    # exception — it destroys nothing and is what lets a cold start bootstrap
+    # from seeds.
+    observed = source in LIVE_UNIVERSE_SOURCES
+
     existing_rows = await db.execute(select(Ticker.symbol))
     existing_symbols = set(existing_rows.scalars().all())
     target_symbols = set(symbols)
 
     symbols_to_add = sorted(target_symbols - existing_symbols)
-    symbols_to_delete = sorted(existing_symbols - target_symbols) if prune_missing else []
 
-    for symbol in symbols_to_add:
-        db.add(Ticker(symbol=symbol))
-
-    deleted_count = 0
-    skipped_delisted: list[str] = []
-    if symbols_to_delete:
-        # A delisted row was retained on purpose — its price history is exactly
-        # what a survivorship-honest backtest needs, and the delete cascades it
-        # away. Pruning must not undo that. (Pruning a symbol that is merely no
-        # longer an index constituent still destroys history; that needs a
-        # tracked/membership state separate from delisting.)
-        protected = (
+    # Only index-owned rows are prunable. A manual addition is by definition not
+    # in the constituent list, so without this filter every hand-added symbol
+    # would be swept by any prune.
+    prunable = set() if not (prune_missing and observed) else set((
         await db.execute(
             select(Ticker.symbol).where(
-                    Ticker.symbol.in_(symbols_to_delete),
-                    Ticker.delisted_on.is_not(None),
+                Ticker.symbol.in_(existing_symbols - target_symbols),
+                Ticker.tracking_source == ticker_service.SOURCE_INDEX,
             )
         )
-        ).scalars().all()
-        skipped_delisted = sorted(protected)
-        deletable = [s for s in symbols_to_delete if s not in set(protected)]
-        if deletable:
-            result = await db.execute(delete(Ticker).where(Ticker.symbol.in_(deletable)))
-            deleted_count = int(result.rowcount or 0)
+    ).scalars().all())
+    symbols_to_untrack = sorted(prunable)
+    kept_manual = (
+        sorted((existing_symbols - target_symbols) - prunable)
+        if (prune_missing and observed) else []
+    )
+
+    for symbol in symbols_to_add:
+        db.add(Ticker(symbol=symbol, tracking_source=ticker_service.SOURCE_INDEX))
+
+    # A symbol that has returned to the index must rejoin the live path. It is
+    # already in ``existing_symbols``, so it is neither added nor untracked here
+    # — without this it would keep the flag from its earlier removal and stay
+    # invisible to the scanner forever, with nothing to show why.
+    #
+    # Clears the prune flag ONLY. ``scan_disabled_on`` is the operator's own
+    # decision and no automated run may undo it: a symbol someone switched off
+    # by hand must stay off across a prune -> re-entry cycle. Nor does this touch
+    # ``tracking_source`` — a manual row that joins the index stays manual, or it
+    # would lose its protection the next time it left.
+    retracked_count = 0
+    if observed:
+        retracked = await db.execute(
+            update(Ticker)
+            .where(Ticker.symbol.in_(target_symbols), Ticker.untracked_on.is_not(None))
+            .values(untracked_on=None)
+        )
+        retracked_count = int(retracked.rowcount or 0)
+
+    untracked_count = 0
+    if symbols_to_untrack:
+        # Marked, never deleted. The delete this replaced cascaded through
+        # OHLCV, setups and scores, which is precisely the history a
+        # survivorship-honest backtest needs — see migration 035. Rows already
+        # carrying a mark keep their original date rather than being re-stamped
+        # on every bootstrap.
+        #
+        # Stamped by the database, not date.today(): active_only compares this
+        # against func.current_date(), and a process in a timezone ahead of the
+        # database would otherwise write a date that reads as "effective
+        # tomorrow" and leave the symbol in the scanner for a day. Same clock on
+        # both sides by construction.
+        result = await db.execute(
+            update(Ticker)
+            .where(Ticker.symbol.in_(symbols_to_untrack), Ticker.untracked_on.is_(None))
+            .values(untracked_on=func.current_date())
+        )
+        untracked_count = int(result.rowcount or 0)
+
+    # Membership is observed on every live sync, independent of pruning and of
+    # who owns the row. A failed fetch must never close a membership.
+    membership = (
+        await _observe_membership(db, normalised_universe, target_symbols)
+        if observed
+        else {"opened": 0, "closed": 0}
+    )
 
     await db.commit()
 
@@ -393,10 +558,29 @@ async def bootstrap_universe(
         "total_universe_symbols": len(symbols),
         "added": len(symbols_to_add),
         "already_tracked": len(target_symbols & existing_symbols),
-        "deleted": deleted_count,
+        # Always 0. Kept so a caller reading this key is told plainly that
+        # pruning no longer destroys rows, rather than seeing the old key vanish
+        # and assuming nothing happened — or worse, seeing a count under
+        # "deleted" that now means something else entirely.
+        "deleted": 0,
+        "untracked": untracked_count,
+        "untracked_symbols": symbols_to_untrack[:50],
+        # Symbols that rejoined the index and were returned to the live path.
+        "retracked": retracked_count,
+        # Absent from the universe but manually owned, so a prune left them
+        # alone. The operator asked for these by hand.
+        "kept_manual": kept_manual[:50],
+        "kept_manual_count": len(kept_manual),
         "added_symbols": symbols_to_add[:50],
-        # Delisted rows a prune declined to destroy, so the caller can see the
-        # count did not match what they asked to remove.
-        "kept_delisted": skipped_delisted[:50],
-        "kept_delisted_count": len(skipped_delisted),
+        # Membership observation is skipped entirely on a cache/seed fallback,
+        # so the caller can tell "nothing changed" from "we never looked".
+        "membership_observed": observed,
+        "membership_opened": membership["opened"],
+        "membership_closed": membership["closed"],
+        "prune_note": (
+            "Pruning marks untracked_on and never deletes: the rows keep their "
+            "bars, setups and scores so historical backtests stay honest. "
+            "Untracked symbols leave the scanner, rankings and ingestion. "
+            "Manually added symbols are never pruned."
+        ),
     }

docs/data-integrity-plan.md

@@ -0,0 +1,363 @@
+# Data & signal integrity — implementation plan
+
+Status: **closed, 2026-09-07.** Steps 1, 2 and 3a ran; Step 3b was not started
+because 3a did not clear its bar. Findings are in
+[`research/survivorship-audit.md`](research/survivorship-audit.md),
+[`research/tr-rank-agreement.md`](research/tr-rank-agreement.md) and
+[`research/tr-total-return-ab.md`](research/tr-total-return-ab.md). The follow-on
+work this raised is in [`data-reliability.md`](data-reliability.md).
+
+Two questions, answered in order, with the production work made conditional on
+the answers.
+
+## Objective
+
+1. Measure how the backtest changes when removed index constituents are restored.
+   This is a survivorship sensitivity check, not a complete bias estimate.
+2. Check whether total-return momentum changes the daily admission or ordering
+   of names. Build the production path only if this screening test finds changes.
+
+**Guiding principle: KISS — measure before building.** Every earlier draft of this
+plan started with a schema change (`adj_close`, then `tr_factor`), an ingestion
+maintenance path, a snapshot migration and a benchmark column, in order to deliver
+a signal change whose effect nobody has measured. Both questions above can be
+investigated by research scripts against a snapshot copy. Step 1 needs one small,
+explicit replay option to include entries immediately before a terminal event.
+No production schema or ingestion changes are needed for either research step.
+Step 3 is not started until Step 2 finds a ranking or admission difference.
+
+Why the research scripts need none of that machinery: the whole forward-anchored
+factor design exists because *production* ingest is incremental, so back-adjusted
+prices drift between fetches. A research script fetches each ticker's whole history
+in **one** request, and within a single response the adjustment is internally
+consistent. No column, no anchor, no maintenance.
+
+---
+
+## Step 1 — How sensitive is the backtest to omitted constituents?
+
+Arm A uses the universe already present in the base snapshot.
+Arm B adds the names that were removed from the S&P 500 since the snapshot's start
+— including the ones that failed — and lets the strategy trade them.
+Use the same configuration, cadence and evaluation dates in both arms. This is
+deliberately an expanded-universe experiment, without historical membership gating.
+
+### Files
+
+- **[NEW] `scripts/data/sp500_removals.json`** — flat list, hand-curated from
+  Wikipedia's S&P 500 changes table, checked into git:
+  `[{"symbol": "SIVB", "removed_on": "2023-03-15", "reason": "receivership"}, ...]`
+  Reasons: `receivership` | `bankruptcy` | `acquisition` | `market_cap`.
+  No additions, no rename map, no membership intervals — see *Not doing*.
+- **[NEW] `scripts/prep_survivorship_snapshot.py`** — makes both arm files from a
+  base snapshot, leaving the base untouched:
+  1. Copy base → `survivorship-a.sqlite` and `survivorship-b.sqlite`.
+  2. **Schema sync** on both: add every column present in `Base.metadata` but
+     missing from the file, nullable. The snapshots on disk predate migrations 026
+     and 032, so `select(Ticker)` against them fails today with
+     `no such column: tickers.cik` — and `_get_ticker` (`price_service.py:22`) is a
+     full-entity select that `query_ohlcv` → `backtest_service._fetch_columns`
+     reaches. Derive the column list from the metadata, not a hardcoded list, so
+     migration 036 doesn't break it again.
+  3. Into arm B only: for each removed symbol with no bars, fetch daily bars from
+     Alpaca (it still serves delisted symbols — verified for SIVB, FRC, ATVI, TWTR)
+     and insert the ticker row. Restrict downloaded bars to the base snapshot's
+     date range and use the same price adjustment as its existing bars. Log fetched,
+     already-present and unavailable symbols; an unavailable failure must be called
+     out as missing coverage, not silently treated as having no strategy exposure.
+     `removed_on` is an index-removal date, not necessarily a delisting date.
+     **Nothing is inferred from missing bars**: `delisted_on` / `delisted_reason`
+     are written only where the dataset carries a confirmed terminal event. A
+     series that merely ends early leaves them NULL and is reported as an
+     unresolved *ending status* — as built, that is CMA, SEE, HBI and DISH.
+     Series are shortened only at a **reviewed cutoff** listed in the dataset
+     (SBNY 2023-03-10, INFO 2022-02-25). Elsewhere a run of ≥5 zero-volume
+     sessions is a tripwire, not a rule: the symbol is left out of the arm and
+     reported for review rather than silently truncated.
+  4. **Terminal mark**, arm B only: for a confirmed terminal event, append one
+     synthetic bar on the next SPY trading session after the last real bar, with
+     `o=h=l=c=0.01, volume=0`. A mark is written only when the retained final bar
+     equals the confirmation's reviewed `last_real_bar`; a disagreement writes no
+     mark and is reported. Write a sidecar JSON mapping symbol to terminal-bar
+     date for the runner.
+     The portfolio sim already fills a gap through the stop at the open
+     (`backtest_service.py` ~2280), so the position resolves at ~-100% with no
+     change to its stop-fill logic. Treat 0.01 as a near-total-loss assumption,
+     not an observed execution. For acquisitions, retain the last available close
+     as an approximation; this experiment does not model cash/stock consideration
+     or release of capital on the acquisition date.
+     This matters: failures halt mid-collapse (SIVB's last exchange bar is 106.04
+     on the way to zero), so without it arm B spares exactly the trades it exists
+     to measure. **Appending the bar alone is insufficient:** `_replay_ticker`
+     currently stops at `n - HORIZON`, where `HORIZON = 30`. That excludes signals
+     in the last 29 real bars even after the synthetic bar is added. Implement the
+     explicit replay option below before interpreting the audit. Only positions
+     still open at the halt should lose nearly everything; positions stopped or
+     timed out earlier should retain their normal exits.
+     `volume=0` is safe — `compute_volume_profile` guards its only divisor
+     (`indicator_service.py:298-300`).
+- **[NEW] `docs/research/survivorship-audit.md`** + an entry in
+  `docs/research/README.md`.
+
+### Small replay change required
+
+- Add optional `--terminal-events <json>` to `run_backtest_snapshot.py`. Pass the
+  map explicitly through `run_backtest` and `_replay_and_signals` to
+  `_replay_ticker`, including the multiprocessing path. Default to no events.
+  Do not infer terminal events from low prices or zero volume.
+- Validate the whole map **before dispatch** and fail the run on any mismatch.
+  `_replay_ticker` rejects a bad date, but both dispatch paths swallow per-ticker
+  exceptions so a long replay survives one unreadable name — which would turn a
+  stale events file into a quietly incomplete report. `_validate_terminal_events`
+  raises `TerminalEventError` before any work starts, so no report is written.
+- For a mapped ticker, validate that the terminal date matches its final bar.
+  Keep the existing cadence and starting index, but allow signal dates through
+  the last real bar instead of stopping 30 bars early. Never generate a setup on
+  the synthetic bar. Unmapped tickers retain the existing replay bounds.
+- Use the existing forward bars and outcome evaluation. For the long-only audit,
+  a setup with a positive stop resolves no later than the terminal bar. Do not
+  pad the history with 30 invented trading bars or turn unresolved shortened
+  histories into normal 30-day outcomes.
+- Keep terminal prices out of later signal generation; their role is resolving
+  existing positions. Preserve normal stop, target and time exits before them.
+
+### Focused validation
+
+Add regression tests alongside the existing backtest tests:
+
+1. A qualifying long setup five sessions before a terminal event is emitted with
+   the option enabled and exits at 0.01 if still held. Use a controlled setup
+   fixture so the test does not depend on whether SIVB happens to qualify.
+2. A position that exits before the terminal event keeps that earlier exit.
+3. No setup is emitted on the terminal bar; an ordinary ticker and a run without
+   the option retain their existing replay behavior.
+
+Before the full runs, smoke-test the prepared files with `_fetch_columns` for one
+existing and one restored ticker. Confirm matching base-symbol bars between arms,
+matching configuration and date bounds, and no modification of the base file.
+
+### Running it
+
+```powershell
+python scripts/prep_survivorship_snapshot.py --base backtest_snapshots/prod.sqlite
+python scripts/run_backtest_snapshot.py backtest_snapshots/survivorship-a.sqlite --out reports/backtest-YYYYMMDD-survivorship-a.json
+python scripts/run_backtest_snapshot.py backtest_snapshots/survivorship-b.sqlite --terminal-events backtest_snapshots/survivorship-b-terminal-events.json --out reports/backtest-YYYYMMDD-survivorship-b.json
+```
+
+No new runner: `run_backtest_snapshot.py` already runs the production sim offline
+and reports CAGR, max drawdown, Sharpe and total return. Compare the same production
+portfolio-monitor strategy and `all` lookback in both reports, not whichever
+variant performs best. Record B minus A for these metrics, trade counts, evaluated
+dates, restored-symbol coverage and terminal-event assumptions. A failed ticker
+replay or missing production report row makes the run incomplete.
+
+### What it does and doesn't measure
+
+An **expanded-universe sensitivity analysis**, not a lower bound. It restores
+omitted names but does not remove addition lookahead. Still-trading former members
+remain eligible after removal: `run_backtest` does not gate on `delisted_on`, and
+an index-removal date would not be the right delisting field anyway.
+
+Changing the universe changes percentile ranks and competing admissions. Either
+correction can raise or lower this strategy's realized returns; do not claim that
+deletions are the larger source of bias or that the true drag must exceed B minus A.
+A small delta does not establish that survivorship bias is small. Report it as
+the sensitivity observed under these assumptions. A complete point-in-time audit
+remains separate work, not a requirement for finishing this experiment.
+
+### Base snapshot
+
+- `prod.sqlite`: 2021-06-24 → 2026-07-02, 90MB, the ~505 production names.
+- `research.sqlite`: 2016-01-04 → 2026-07-17, 678MB, 4,650 names.
+
+Use `prod.sqlite`: the highest-signal events (SIVB, FRC, SBNY, ATVI — all
+2023) are inside its window, the copies are cheap, and it has no completion
+manifest to keep in sync. The pre-registration then states a 2021–2026 window, not
+a ten-year one. Verify the actual file's bounds and record its hash before running.
+Do not silently substitute `research.sqlite`: its much broader universe answers a
+different comparison. Keep the first experiment on the production snapshot.
+
+---
+
+## Step 2 — Would total return change any decision?
+
+Use a conservative ranking screen, not a portfolio simulation. Compare the full
+ordered momentum-admitted list on every daily decision date. Do not call a top-15
+list the traded set: production also applies setup qualification, current holdings,
+available slots and other admission constraints. A difference in this screen is
+a reason to investigate actual trades in Step 3, not proof of changed P&L.
+
+### File
+
+- **[NEW] `scripts/research_tr_rank_agreement.py`** — use arm A's universe and dates:
+  1. One `Adjustment.ALL` fetch and one `Adjustment.SPLIT` fetch per symbol, plus
+     SPY, over the full history. Single fetch each, so each series is internally
+     consistent. Align by date, never zip by index. Cache the responses locally
+     for repeatable runs. Require matching dates in both adjustments; report and
+     resolve missing downloads before applying the gate. Insufficient initial
+     history is expected and should be recorded, with equal eligibility in both
+     arms. Use only bars at or before each decision date.
+  2. At each SPY trading date with sufficient history, compute residual 12-1
+     momentum both ways with the existing
+     `momentum_service.compute_residual_12_1_momentum` (price-return series against
+     nominal SPY; total-return series against total-return SPY), percentile them
+     with `momentum_service._percentiles`, and take the admitted set at the
+     production `min_momentum_percentile`.
+  3. Compute `compute_realized_vol_6m` and its percentiles from the SPLIT series
+     once per date and hold that leg fixed in both arms. This isolates the change
+     to momentum. Order the entire admitted lists by `blend_strategy_rank`, with
+     symbol as a deterministic tie-breaker. Do not truncate to `max_positions`:
+     a lower-ranked name may be the first actionable setup when others fail gates
+     or are already held. Iterate symbols in the same sorted order in both arms
+     before calling `_percentiles` so tied inputs do not create artificial changes.
+  4. Save per-date results and an overall/per-year summary: number of evaluated
+     dates, fraction with different admitted sets, fraction with different ordered
+     lists, and mean and maximum set disagreement. Define set disagreement as
+     `len(A symmetric_difference B) / len(A union B)`, or zero for two empty sets.
+     Report Spearman correlation as a diagnostic only. Include the first few
+     changed dates with the changed names to make the result inspectable.
+
+Validation: identical input series must produce no differences. A fixture where
+the admitted members stay the same but two names swap order must register a
+changed ordered list. A difference on only one date must still trigger the gate.
+Write results to `docs/research/tr-rank-agreement.md` and index it in the research
+README, including data coverage, parameters and the exact report/cache paths.
+
+### Pre-registered decision rule
+
+- **No ordered-list differences on any evaluated daily date → stop.** Record
+  "no admission or ordering differences observed in this window under the tested
+  configuration." This is historical evidence, not a guarantee about future
+  decisions. No schema or ingestion changes are needed.
+- **Any ordered-list difference → Step 3.** This intentionally conservative rule
+  can flag names that would never actually trade. Step 3's paired portfolio replay
+  determines whether the difference reaches actual admissions and P&L.
+- **Incomplete downloads or no evaluable dates → inconclusive.** Fix coverage
+  before deciding. Never turn missing data into agreement.
+
+There is no median-based tolerance: a zero median can hide differences on nearly
+half the dates. Checking daily full lists avoids adding a second portfolio engine
+to this screening script while retaining changes below the nominal top 15.
+
+---
+
+## Step 3 — Production total-return ranking (conditional)
+
+**Gate cleared 2026-09-07 — Step 3 is now live work.** The screen found ordered
+admitted lists differing on 1,009 of 1,009 evaluated dates, with the shift
+correlating +0.79 with trailing dividend yield
+([results](research/tr-rank-agreement.md)). That is a ranking difference, not a
+P&L difference; Step 3's job is to find out whether it reaches trades.
+
+### Step 3a — Paired portfolio replay (do this first, no schema)
+
+Different ranks are not different trades. Before any schema or ingestion work,
+replay the production book twice and measure actual admissions and P&L. If the
+books are identical, the basis cannot matter and Step 3b never starts.
+
+**[NEW] `scripts/run_total_return_ab.py`**, sourcing everything from artifacts
+that already exist — `backtest_snapshots/survivorship-a.sqlite` for bars and
+`backtest_snapshots/tr-rank-cache.sqlite` for the two adjusted series.
+
+1. **Generate setup candidates once.** One replay pass produces the candidate set;
+   both arms reuse it. Only momentum and what depends on it — the momentum
+   percentile, qualification against `min_momentum_percentile`, and
+   `blend_strategy_rank` ordering — are recomputed per arm. Setup geometry
+   (entry, stop, target, R:R) is generated once and never differs.
+2. **Both arms read the same cache.** Control takes SPLIT stock series against
+   SPLIT SPY; treatment takes ALL stock against ALL SPY. Neither arm reads a
+   freshly fetched series, and neither mixes cache with snapshot closes for the
+   momentum input — a refreshed-vs-snapshot difference would confound the basis
+   difference being measured.
+3. **Everything else is pinned.** Execution uses the snapshot's nominal bars in
+   both arms; the volatility leg, the non-momentum gates, capacity
+   (`max_positions`), risk per trade and the exit policy are identical. Each book
+   then carries its **own** resulting holdings — a divergent admission changes
+   which slots are occupied on later dates, and that path dependence is part of
+   what is being measured.
+4. **Fixed up front:** universe (arm A's 505 symbols with bars), window
+   (2021-06-24 → 2026-07-02), cadence, and the production config as read from the
+   snapshot. Only bars at or before each signal date are used, in both arms.
+5. **Saved to `reports/tr-total-return-ab.json`:** per-arm trade count, CAGR, max
+   drawdown, Sharpe, total return, win rate; annual returns for both arms and the
+   **count of years treatment beats control**; and the entry-level diff — dates
+   and symbols entered by one arm and not the other, in both directions.
+6. **Tests** (`tests/unit/test_total_return_ab.py`): an identical-input parity
+   test — feeding the same series to both arms must produce byte-identical books,
+   so any observed difference is attributable to the basis and not to replay
+   nondeterminism; and a controlled case where a hand-built momentum difference
+   flips one name across the cutoff and the resulting admission actually changes,
+   proving the harness can see what it is looking for.
+
+**Reading the result.** Different trades establish that the basis is
+production-relevant. They do **not** authorize a rollout: that is the
+non-inferiority decision below, taken on the paired numbers, and Step 3b starts
+only after the paired results have been reported.
+
+**Step 3a result (2026-09-07):** the basis reaches trades — ~30% of entries
+differ. Which basis is better is unresolved: a paired block bootstrap on the
+date-aligned daily return streams (5,000 replicates, 21- and 63-session blocks,
+seed fixed beforehand, both reported) puts ΔCAGR at −3.6/−3.8pp with 95%
+intervals spanning zero, ΔSharpe ≈ −0.11, and ΔMaxDD −1.0/−1.8pp in treatment's
+favour. **Production stays unchanged; 3b not started.**
+[Results](research/tr-total-return-ab.md).
+
+### Step 3b — Production storage and ingestion (only if 3a shows real differences)
+
+The design points already settled, worth carrying forward:
+
+- `ohlcv_records.tr_factor`, `Float`, nullable, **no default and no server_default**
+  — a default of 1.0 makes "never backfilled" indistinguishable from "no dividends"
+  and would silently stamp a price-return ranking as total return.
+- Forward-anchored: `f = 1.0` at `max(2016-01-04, ticker's first bar)`, compounded
+  as `f_t = f_{t-1} * (R_t / R_{t-1})` where `R = all_close / split_close`. The
+  ratio is anchor-invariant **only within one API response** — never persist `R` and
+  compare across days.
+- `upsert_ohlcv` must not clobber a stored factor:
+  `coalesce(excluded.tr_factor, ohlcv_records.tr_factor)`. Daily ingest re-fetches
+  the boundary bars, so overlapping writes are routine.
+- Daily ingest pulls a ~5-bar overlap under both adjustments to extend the factor;
+  a clean-ratio mismatch on the overlap bars means a split, which needs a
+  full-history refetch (the current incremental ingest never restates history).
+- One basis for the whole cross-section, never mixed, stamped in the rank log.
+- Beta leg: if the stock series goes total return, SPY must too, in the live path
+  and the backtest, or beta picks up SPY's ~1%/yr yield.
+- A/B through the existing matrix on CAGR and max drawdown with per-year win/loss
+  counts. Reuse the same setup candidates, nominal execution bars, core gates,
+  volatility leg, capacity and exit policy; change only momentum and its dependent
+  admission/ranking fields. Replay each book with its own resulting holdings and
+  compare actual entry dates/symbols as well as P&L. Do not substitute top-15 lists
+  for the existing portfolio admission logic. Non-inferiority, not superiority:
+  total return is the definition 12-1
+  momentum is built on, so an in-sample shortfall shouldn't lock in the basis we
+  believe is measured wrong.
+
+---
+
+## Not doing
+
+- **Chart total-return toggle.** The candlestick chart is an execution chart:
+  stops, S/R levels, entry and paper-trade markers are all nominal. Rescaling the
+  candles draws prices that never traded next to a stop that did.
+- **`[D]` dividend markers.** Cosmetic, and they need `tr_factor` to exist. Revisit
+  after Step 3, if Step 3 ever happens.
+- **Point-in-time addition gating** — interval map, reverse-chronological replay,
+  rename map and removal gating are deferred from this sensitivity experiment.
+  Their relative impact is unknown; a small Step 1 delta does not rule them out.
+  Note if a point-in-time audit is
+  ever done: the ranking cross-section must be filtered *before*
+  `_assign_*_percentiles` (`backtest_service.py:4198-4200`), not only at admission
+  in `_simulate_portfolio` (4257), or the percentile cutoff is still computed
+  against a non-PIT universe.
+- **Prune protection and the `add_ticker` regex.** Only needed if delisted names
+  are written to the production database. The audit works entirely in snapshot
+  copies. (`bootstrap_universe(prune_missing=True)` cascading away a still-trading
+  ex-constituent's history is a real latent risk — a separate ticket, not this one.)
+- **`benchmark_prices.tr_close`.** Only needed for Step 3. Note for then: store a
+  forward factor like the OHLCV side, not a back-adjusted close —
+  `refresh_benchmark_prices` writes a row only when `close` changed and is called
+  with variable spans, which would leave different anchors either side of the last
+  refreshed window.
+- **Migrating the snapshots in place.** The prep script syncs its own copies.
+  `research.sqlite` stays byte-identical, so its completion manifest and the reports
+  that cite it stay valid.

docs/data-reliability.md

@@ -0,0 +1,380 @@
+# Data reliability
+
+Three changes that make the platform's data trustworthy rather than faster or
+more profitable. They came out of the [integrity
+experiments](data-integrity-plan.md), which turned up two problems worth fixing
+on their own:
+
+- Changing the stock universe **materially changes results** — Arm A 2.09 Sharpe
+  / 51.6% CAGR against Arm B 1.57 / 38.1% over the same window, from universe
+  composition alone ([survivorship audit](research/survivorship-audit.md)).
+- **Missing data quietly produces a seemingly successful backtest.** Found while
+  running Step 3a: a snapshot whose schema predated migrations 026/032 made every
+  `_fetch_columns` call raise `no such column: tickers.cik`, both dispatch paths
+  caught the per-ticker exception and continued, and the run wrote a complete
+  report with zero candidates. Nothing in the report or the UI said the run had
+  lost its entire universe.
+
+| # | Change | Status |
+|---|---|---|
+| 1 | Make incomplete backtests visibly incomplete | **done** — 2026-09-07 |
+| 2 | Preserve history when a stock leaves the tracked universe | **done** — 2026-09-07 |
+| 3 | Record universe membership by effective date | **done** — 2026-09-07 |
+
+---
+
+## 1. Make incomplete backtests visibly incomplete — done
+
+The replay loop is deliberately fault-tolerant: one unreadable ticker must not
+abort a 500-symbol run. That is right for the run and wrong for its report,
+which had no way to say how much of the universe it actually covered. A run that
+lost every ticker and a run that honestly found no setups produced the same
+report.
+
+### What the report now carries
+
+`run_backtest` builds a `_DataQuality` ledger
+(`app/services/backtest_service.py`) and emits it as `report["data_quality"]`:
+
+```json
+{
+  "status": "failed",
+  "complete": false,
+  "reasons": ["no symbol was evaluated successfully", "506 symbols failed to load"],
+  "expected": 506, "loaded": 0, "evaluated": 0,
+  "no_data": 0, "no_data_symbols": [], "short_history": 0,
+  "failed": 506,
+  "failures_by_stage": {"fetch": 506},
+  "failures": [{"symbol": "A", "stage": "fetch",
+                "error": "OperationalError: (sqlite3.OperationalError) no such column: tickers.cik"}],
+  "failures_truncated": true,
+  "benchmark": {"symbol": "SPY", "sessions": 1516,
+                "first": "2020-07-06", "last": "2026-07-17",
+                "momentum_basis": "residual", "error": null}
+}
+```
+
+**"Complete" means the processing and coverage checks passed** — every expected
+symbol loaded and evaluated, benchmark present. It does **not** mean the
+underlying prices are proven correct. A complete run over silently wrong bars is
+still complete; splits, adjustment drift and bad vendor prints are a different
+problem, and this ledger must not be read as clearing them. The card says so on
+the page, not only here.
+
+**The counters are self-checking.** Every universe symbol ends in exactly one of
+`no_data`, `failures` or `evaluated`, and `as_report` verifies
+`no_data + failed + evaluated == expected`. A ledger that stops balancing —
+because a counter was missed when a code path was added — reports itself as a
+failure rather than quietly under-counting. `short_history` is a *subset* of
+`evaluated` (loaded, but fewer than `MIN_LOOKBACK + HORIZON` bars, so no outcome
+can resolve), not a fourth bucket.
+
+**Benchmark availability is a separate axis.** Without SPY closes, residual 12-1
+momentum silently falls back to raw 12-1 — the gate stops being the production
+gate. The block records the basis actually used, so the degradation is on the
+report instead of only in a log line.
+
+### Three statuses, because severity differs
+
+| Status | Meaning | UI |
+|---|---|---|
+| `ok` | Every symbol evaluated, benchmark present | Normal |
+| `degraded` | The strategy is intact but ran on less than the whole universe | Tiles shown, banner above |
+| `failed` | The headline can't be read as the strategy it claims to be | **Tiles replaced by the callout** |
+
+`failed` means one of: nothing evaluated, an empty universe, an unbalanced
+ledger, or a missing benchmark. Blanket suppression on a single missing ticker
+would train the reader to ignore the banner, which is the failure mode this is
+supposed to fix.
+
+The callout **replaces** the performance tiles rather than sitting above them.
+A warning over ordinary-looking numbers is precisely how the zero-candidate run
+would still have read as a quiet week.
+
+### Where it shows
+
+- `BacktestDataQualityCard` in the backtest panel, with the failure list behind a
+  disclosure.
+- The "Last run" line now reads `505/506 tickers` — evaluated over expected, not
+  the universe size. "506 tickers" on a run that evaluated none was the
+  misleading part.
+- `productionBaseRates` returns `null` for a failed run, so per-ticker base rates
+  quoted next to a live setup — which carry no caveat of their own — disappear
+  rather than being sourced from a broken replay.
+- `EvaluationPanel` drops a failed run's expectancy as a drift baseline instead
+  of reporting "tracking" or "drift" against it.
+- **A report with no `data_quality` block reads as unknown, not clean.** Old
+  cached reports predate the check; defaulting them to complete would reproduce
+  the exact bug being fixed.
+
+### Acceptance evidence
+
+Both runs used `scripts/run_backtest_snapshot.py`.
+
+| Snapshot | Result |
+|---|---|
+| `survivorship-a.sqlite` (healthy) | `degraded` · 505/506 evaluated · 202,765 candidates · one bar-less row (RHM) |
+| the same file with `tickers.cik` dropped | `failed` · 0/506 evaluated · 506 fetch failures · 0 candidates |
+
+The broken-snapshot run is the one that matters: before this change it wrote a
+report that looked like an ordinary run with no setups.
+
+The healthy run is not `ok`, and that turned out to be correct. It flagged
+`no_data: 1` (RHM) and `short_history: 1` (SPCX, 14 bars), which an independent
+bar-count query confirms. Neither is new — Step 2 already recorded RHM as inert
+in `tr-rank-agreement.md`, which is why Step 3a generated candidates over 505 of
+506 tickers. The ledger derives it independently, from the run rather than from
+a hand-written note, which is the point: the same 506-vs-505 gap now appears on
+the report instead of only in a research write-up someone remembered to make.
+
+`prod.sqlite` has the same bar-less RHM row, so **production runs will report
+`degraded` too** until that universe row is cleared — which is item 2's
+territory. That is deliberate: silencing a bar-less universe row would let a
+wholesale import failure read as clean. The reason names the symbols so it is a
+one-line fix rather than a standing nag.
+
+Tests: `tests/unit/test_backtest_data_quality.py` (10, covering each status, the
+ledger identity, the failure cap, symbol naming and the benchmark axis) and four end-to-end
+cases in `tests/unit/test_backtest_service.py`, including every-ticker-fails and
+missing-benchmark.
+
+---
+
+## 2. Preserve history when a stock leaves the tracked universe — done
+
+`bootstrap_universe(prune_missing=True)` ran `DELETE FROM tickers`, cascading
+OHLCV, setups, scores and S/R levels away for any symbol that had merely dropped
+out of the index. Migration 032's own comment predicted it: *"Pruning a symbol
+that is merely no longer an index constituent still destroys history; that needs
+a tracked/membership state separate from delisting."*
+
+### The two states, kept apart
+
+Migration 035 adds `Ticker.untracked_on`, the sibling of `delisted_on`:
+
+| Column | Meaning |
+|---|---|
+| `delisted_on` | The security stopped trading |
+| `untracked_on` | It still trades; it left the index and we stopped scanning it |
+
+Neither deletes anything. `NULL` in both means "in today's live path".
+`active_only` — the single opt-in choke point already applied to scanning,
+ranking, scoring, breadth and ingestion — now filters on both, and honours
+`as_of` for both, so a historical query can still ask what was in the scanner on
+a given date.
+
+`prune_missing=True` now marks instead of deleting, and **clears the mark for
+symbols that have rejoined the universe**. That last part is the bug this
+refactor could have introduced: a returning name is already tracked, so it is
+neither added nor untracked, and without an explicit clear it would keep the
+flag from its earlier removal and stay invisible to the scanner forever.
+
+The `deleted` return key is now always `0` rather than repurposed — a caller
+reading `deleted: 12` must not be told rows were destroyed when they were not.
+`untracked`, `untracked_symbols` and `retracked` carry the real counts, and the
+admin toast reads "untracked N" instead of "deleted N".
+
+### Historical eligibility stays explicit
+
+The backtest's universe query is now `_load_backtest_symbols`, a named function
+whose docstring states that it deliberately does **not** use `active_only`.
+Today's membership flag must not erase yesterday's candidates: a replay grades
+what the strategy could have selected at each past date, so it must see symbols
+that have since been delisted or dropped out. "Fixing the inconsistency" by
+adding `active_only` there is exactly how survivorship bias gets reintroduced,
+which is why a test asserts the opposite.
+
+The one path that still really deletes is `DELETE /tickers/{symbol}` — an
+explicit operator action asking for exactly that. Pruning is not.
+
+### Untracking one symbol
+
+`POST /tickers/{symbol}/tracking` with `{"tracked": false}` excludes a single
+symbol from the live path, mirroring the delisting pair (both are API-only; no
+UI control exists for either). It exists because `prune_missing=True` can only
+act on the whole universe, and a manually added symbol is by definition *not* in
+the fetched constituent list — so a prune sweeps every manual addition along
+with the genuine leavers. Reversible with `{"tracked": true}`.
+
+The case it was built for: **RHM**. Added by hand on 2026-06-28 out of interest
+in the stock, it has produced zero rows in *every* ticker-linked table in the ten
+weeks since — no bars, no ingestion progress, no scores — and the Alpaca name
+backfill never resolved it. It is the only symbol in the universe that has
+produced nothing at all (BK, CTRA and SATS also lack Alpaca names but carry full
+history, so a missing name alone proves nothing). Rheinmetall is XETRA-listed, so
+a plain US `RHM` is not something the OHLCV provider can serve. An attempt to
+confirm this against Alpaca's asset API returned 401 for the AAPL control too,
+so the *identity* is unverified — the justification is the persistent absence of
+usable data, not a confirmed listing status or a delisting.
+
+**Untracking it does not clear the backtest's `degraded` flag, and must not.**
+The backtest deliberately includes untracked symbols, so RHM stays in the
+universe query, still returns no bars, and still counts in `no_data`. What
+untracking buys is narrower and still worth having: RHM leaves the scanner,
+ingestion and scoring, and the decision is recorded explicitly rather than
+sitting as an unexplained gap. Making the warning go away would mean changing
+historical eligibility to silence a real coverage gap — guarded by
+`test_untracking_does_not_clear_a_data_quality_warning`.
+
+Note what is *not* solved: a manual addition is still indistinguishable from a
+universe member, so a prune untracks it (non-destructively, and visibly via
+`untracked_symbols`, but silently in the sense that nobody asked). Recording
+*why* a symbol is tracked is item 3's job.
+
+### Tests
+
+`tests/unit/test_universe_membership.py` (12) covers the three behaviours
+together:
+
+1. Pruning preserves the symbol's bars **and** its linked records (S/R levels
+   are checked too, so the assertion is about the cascade, not just one table).
+2. The pruned symbol leaves `active_only` while staying visible in the registry.
+3. `_load_backtest_symbols` still returns it.
+
+Plus re-tracking, the mark not being re-stamped on a second prune (which would
+erase when the symbol actually left), and `as_of` working for both columns.
+a row carrying *both* marks (it left the index, then delisted), and a
+compilation guard asserting every `active_only` call site still receives both
+predicates ANDed — including `scheduler.py`'s, which chains `group_by`/`order_by`
+onto the result from outside. `tests/unit/test_migration_035.py` exercises the
+migration's upgrade and downgrade directly, including that existing rows default
+to NULL — an upgrade must not drop the whole universe out of the live path.
+
+One detail worth keeping: `untracked_on` is stamped by the **database**
+(`func.current_date()`), not `date.today()`. `active_only` compares it against
+`func.current_date()`, so a process in a timezone ahead of the database would
+otherwise write a date that reads as "effective tomorrow" and leave the symbol
+in the scanner for a day. Same clock on both sides by construction.
+
+## 3. Record universe membership by effective date — done
+
+Migration 036. Three things that were still conflated are now separate:
+
+| Concept | Where | Written by |
+|---|---|---|
+| Index membership, observed history | `universe_memberships` intervals | every **live** sync |
+| Excluded from scanning by a prune | `Ticker.untracked_on` | prune only |
+| Operator switched it off | `Ticker.scan_disabled_on` | `set_tracked` only |
+| Tracking ownership | `Ticker.tracking_source` | `add_ticker` / bootstrap |
+
+**Why the operator flag is its own column.** With one column serving both
+writers, a prune -> re-entry cycle would clear an explicit operator "off",
+because re-entry cannot tell who set it. Re-entry clears `untracked_on` and
+never touches `scan_disabled_on`.
+
+`active_only` now tests three columns and **stays join-free** — it is on ~8 live
+call sites, and the compilation guard asserts all three predicates land, ANDed,
+including at the call site that chains `group_by`/`order_by` on from outside.
+
+### `untracked_on` is not current membership
+
+It records *exclusion by pruning*. The scheduled sync runs
+`prune_missing=False`, so a symbol can have left the index and still be scanned;
+the two states legitimately differ, and nothing asserts they agree.
+
+### Tracking ownership is sticky towards manual
+
+`tracking_source` is `index` or `manual`. Prune touches only `index` rows, so
+**manual additions survive index pruning** — the RHM case.
+
+A bootstrap **never** converts `manual` to `index`. That direction looks harmless
+and is not: a hand-added symbol that happens to join the index would lose its
+protection the moment it left again, which is exactly when it is needed. Only
+new rows a bootstrap creates are stamped `index`; an explicit `add_ticker` on an
+existing index-owned symbol claims it as `manual`.
+
+`POST /tickers` therefore has three responses:
+
+| Case | Response |
+|---|---|
+| New symbol | Created as manually tracked |
+| Existing, index-owned | **200** with the existing row, now manually tracked |
+| Existing, already manual | 409 Duplicate |
+
+Claiming is not a 409 because the operator is saying "keep this stock", and
+refusing while leaving it prunable would defeat the protection they just asked
+for. It deliberately does not touch `scan_disabled_on` or `untracked_on` —
+wanting a stock kept is not the same as asking for it to be scanned, and
+silently re-enabling something switched off by hand would undo a decision nobody
+revisited. `POST /tickers/{symbol}/tracking` is the explicit control.
+
+### Membership is observed, never asserted
+
+`first_seen_on` is the first date a live fetch actually listed the symbol — not
+an index effective date. Anything earlier is unknown and is not guessed; the
+hand-reconstructed `scripts/data/sp500_removals.json` stays a research artifact.
+
+Intervals are append-only: departure sets `absent_since` and **retains the row**,
+and a re-entry inserts a new one. A symbol that left in March and returned in
+August has two intervals, so a point-in-time query for May correctly answers
+"not a member" — reopening the original row would erase that.
+
+**Boundaries, stated once.** The column is named `absent_since`, not
+`last_seen_on`, because it stores the first observation that found the symbol
+*gone* — on that date it was already absent:
+
+```
+member on D  <=>  first_seen_on <= D < absent_since  (or absent_since IS NULL)
+```
+
+Lower bound inclusive, upper bound exclusive. The real departure lies somewhere
+in `(previous observation, absent_since]` — a day with a daily sync, a week if a
+week was missed. The column records when we *looked*, which is the only thing
+actually known. `members_on(db, universe, as_of)` implements exactly this, so
+the rule lives in one place rather than being re-derived per call site.
+
+**A date before the first successful live sync returns an empty list.** That
+means "we were not looking", not "the index was empty", and a caller has to know
+the difference — answering with today's set would be precisely the survivorship
+error this exists to end.
+
+Recording runs on **every live sync**, independent of pruning and of who owns
+the row: the scheduled job uses `prune_missing=False`, so membership recorded
+only on manual prunes would be full of holes. Manually owned symbols get
+intervals like any other — membership and tracking ownership are independent
+axes. Switching off scanning does not close a membership; the index still
+contains the symbol, we simply stopped looking at it.
+
+**A failed fetch drives nothing at all.** `fetch_universe_symbols` falls back to
+a cached snapshot and then to a built-in seed list of ~44 symbols. Nothing that
+acts on "this symbol is or isn't in the index" may run on that data: not
+recording membership, **not pruning, and not clearing a prune flag**. Skipping
+only the membership write was half a safeguard — a seed fallback would still
+have untracked the rest of the universe, which a regression test caught
+(2 of 3 symbols untracked before the fix). Adding is the one exception: it
+destroys nothing and is what lets a cold start bootstrap from seeds. The summary
+reports `membership_observed: false` so a caller can tell "the index did not
+change" from "we never looked".
+
+### Migration decisions
+
+- **Existing rows become `manual`.** Provenance cannot be recovered from
+  `created_at`, and manual is fail-safe: a wrong guess costs an un-pruned symbol
+  rather than one swept out of the scanner. Bootstraps claim the ones they own
+  as they create them.
+- **Existing "off" states are carried into `scan_disabled_on`.** Whoever wrote an
+  existing `untracked_on` is ambiguous, and re-entry clears that column — so the
+  value is copied to the one no automated run touches. A symbol somebody
+  deliberately switched off cannot be reactivated by a sync.
+- **No history is backfilled.** The table starts empty; history begins at the
+  first observation.
+
+### Tests
+
+`test_universe_membership.py` (19) and `test_universe_membership_intervals.py`
+(14): manual additions survive a prune; a bootstrap never claims a manual row,
+including across a join-then-leave cycle; an explicit add claims an index-owned
+symbol without touching its switches; re-entry clears the prune flag but never
+the operator's; a live sync opens intervals without pruning; departure closes and
+re-entry appends; a cache/seed fallback records nothing, untracks nothing and
+clears no prune flag; membership is recorded for manual rows; switching off
+scanning does not close a membership; and the boundary rules — lower inclusive,
+upper exclusive, gaps are not membership, per-universe scoping, and pre-recording
+dates returning empty. `test_migration_036.py` (4) pins both migration judgement
+calls and the downgrade.
+
+The fixtures pass a **live** source label deliberately: a made-up one would now
+skip pruning entirely and make those tests vacuous.
+
+Backtest eligibility is unchanged — `_load_backtest_symbols` still returns every
+symbol with stored history, and untracking still cannot silence a coverage gap.

docs/research/README.md

@@ -145,6 +145,8 @@ knobs.
 | **`fip_id` / liquid breadth** | Fingerprint −0.045 / t −2.91; liquid unconditional **−0.017 / t −1.85** (not green); mom-conditional **−0.088 / t −4.58** | **Parked.** Orphan +0.0575 died (snapshot race). Breadth did not strengthen resid-mom t-stat. Optional reopen = pre-registered two-arm liquid-1500 book first. See [fip-breadth-ic.md](fip-breadth-ic.md) |
 | **Broader universe** | Composition changes factor signs (fip tug-of-war); vol-tilt on breadth is only a **directional hypothesis** (auth. −0.048 / t −1.36) | Any prod broaden must re-validate 80/20 tilt; offline research only; research.sqlite requires completion manifest |
 | **Forward paper-trade record** | The only true out-of-sample evidence the snapshot cannot give | Time; mark entries at actual near-close fill once ops ships |
+| **Total-return momentum basis** | Ranks change every day (screen: **1,009 of 1,009** dates, ~4.6 names replaced per date, shift correlates **+0.79** with trailing yield) **and it reaches trades** — paired replay differs on ~30% of entries (51 control-only, 63 treatment-only, 271 shared). Which basis is better is unresolved: paired block bootstrap (5,000 reps, 21- and 63-session blocks, seed fixed beforehand) gives ΔCAGR −3.6/−3.8pp with 95% intervals spanning zero ([−14.8, +6.4] and [−14.7, +5.4]), ΔSharpe ≈ −0.11, ΔMaxDD −1.0/−1.8pp in treatment's favour; ~75% of replicates on the point-estimate side. All of the CAGR shortfall sits in 2025 | **Production unchanged.** Needs a threshold pre-registered before it is applied, plus out-of-sample evidence; a capacity/date/cadence matrix would be sensitivity checks, not independent samples. Schema/ingestion deferred. [Screen](tr-rank-agreement.md) · [Paired replay](tr-total-return-ab.md) · [plan](../data-integrity-plan.md) |
+| **Universe sensitivity / point-in-time membership** | Restoring 85 removed constituents (2021-06→2026-07, prod snapshot) moves the production row from **Sharpe 2.09 / CAGR 51.6%** to **1.57 / 38.1%** — all of it universe recomposition; no failed-bank position reached either book (trade extremes identical, and a control shows the extra replay signals changed no portfolio decision). Says the ranks are highly sensitive to the universe, not that bias is 13.5pp | Not a bias estimate: addition lookahead is still unremoved and arm B is not point-in-time. A real PIT audit is separate work. [Survivorship sensitivity](survivorship-audit.md) |
 | **Better target model for clear-air names** | The return is demonstrably there (#2 wins on raw CAGR in *both* train and test); it's the *flat* 3× ATR target that makes it too expensive in risk | Needs a per-name model, not a constant k×ATR |
 | **Minimum effective-risk floor** | ⛔ CLOSED NEGATIVE, not run. The floor lifts EV/trade (+0.032) and PF (+0.073) *by deleting trades* — 11.4 fewer per path, never one more — and costs **−0.753pp CAGR**, −0.047 Sharpe, −0.051 Calmar | Do not run the A/B; its EV-based pass rule would have shipped it. [Withdrawn specification](effective-risk-floor-ab.md) / [findings](portfolio-capacity-bracket-findings.md#correction-2026-08-05-ev-per-trade-was-the-wrong-lens) |
 
@@ -176,6 +178,14 @@ knobs.
    `research.sqlite`. Extender now writes a completion manifest; breadth mode
    refuses without a match. Same class of protection as calendar-truncation
    asserts — do not re-mythologize numbers computed on half a universe.
+8. **A big delta is not the mechanism you assumed.** Restoring 85 removed index
+   constituents cost 13.5pp of CAGR — and none of it came from the failed banks
+   the experiment was built around. The driver was the universe the ranks are
+   computed over. Measure which part of a change did the work before naming the
+   effect, and state what the control actually isolates: ours compared replay
+   *bounds* on one snapshot, so it measured the extra signals, not the synthetic
+   bars those signals resolve against.
+   [Survivorship sensitivity](survivorship-audit.md)
 
 ---
 

docs/research/survivorship-audit.md

@@ -0,0 +1,173 @@
+# Survivorship sensitivity — restoring removed S&P 500 constituents
+
+**Run date:** 2026-09-07 · **Status:** measured, no strategy change proposed
+**Reports:** `reports/backtest-20260907-survivorship-a.json`,
+`reports/backtest-20260907-survivorship-b.json`,
+`reports/backtest-20260907-survivorship-b-nomarks.json`
+
+## Question
+
+The backtest universe is the ~505 names tracked *today*. Every constituent that
+left the index between 2021 and 2026 is missing from it, including the ones that
+failed. How different does the production strategy look when they are put back?
+
+This is an **expanded-universe sensitivity analysis**, not a bias estimate. It is
+pre-registered as a measurement: there is no pass/fail rule and no promotion
+attached to the outcome.
+
+## Method
+
+**Base:** `backtest_snapshots/prod.sqlite`, sha256 `7241f0ff0f6187a0…`, window
+**2021-06-24 → 2026-07-02** (~5 years, not the full Alpaca history). Never
+modified — both arms are copies, and the hash is re-checked after the build.
+
+| Arm | Universe |
+|---|---|
+| **A** — control | 506 tickers, the snapshot as it stands |
+| **B** — treatment | 591 tickers: the same 506 plus 85 restored removals |
+
+Both arms ran the same binary, config, cadence (weekly) and date bounds, and are
+compared on the same report row: `portfolio_monitor.runs` where
+`is_production` is true and `lookback` is `all`
+(`residual80_highvol80_20_atr3`).
+
+Removals come from `scripts/data/sp500_removals.json` — 92 rows curated from
+Wikipedia's *Historical components of the S&P 500* changes table, each carrying
+the verbatim source text plus independently measured Alpaca evidence. Build with
+`scripts/prep_survivorship_snapshot.py`.
+
+Three data rules, all learned from the price data rather than assumed:
+
+1. **Nothing is inferred from missing bars.** `delisted_on` / `delisted_reason`
+   are written only for confirmed terminal events. A series that merely stops
+   early leaves them NULL and is reported as an unresolved ending.
+2. **Series are shortened only at a reviewed cutoff.** Alpaca does not end a
+   halted symbol's series — it repeats the last price at zero volume. SBNY sits
+   at 70.00 for 509 sessions after the FDIC closed it; INFO sits at 108.61 for
+   1,090 sessions, after which a *different company* reuses the ticker with no
+   date gap. Both have explicit reviewed cutoffs (2023-03-10, 2022-02-25).
+   Elsewhere a run of ≥5 zero-volume sessions is a tripwire: the symbol is left
+   out and reported, never silently cut.
+3. **Terminal marks are explicit assumptions.** A synthetic `0.01` bar is written
+   on the next session after the last real bar for SIVB, SBNY and FRC only —
+   each with an FDIC source and only where the retained final bar matches the
+   reviewed `last_real_bar`. The FDIC sources establish the closure *event*; they
+   do not establish what holders recovered. The mark is a near-total-loss
+   assumption, not an observed closing-day execution.
+
+**Replay change.** `_replay_ticker` normally stops `HORIZON = 30` bars before the
+end so every setup has a full resolution window — which would hide every signal
+in the final 29 real bars, exactly the ones a collapse resolves. A new
+`--terminal-events` map opts the three marked symbols into walking through their
+last real bar; no setup is ever generated on the synthetic bar, and unmapped
+symbols are untouched. The map is validated against the snapshot's actual final
+bars *before* any replay is dispatched (`_validate_terminal_events`), because both
+dispatch paths swallow per-ticker exceptions and would otherwise turn a stale map
+into a quietly incomplete report.
+
+## Result
+
+Source: `portfolio_monitor.runs` where `is_production` is true and `lookback` is
+`all` (strategy `residual80_highvol80_20_atr3`), from
+`reports/backtest-20260907-survivorship-a.json` and
+`…-survivorship-b.json`. Both runs: weekly cadence, 6 workers,
+`--allow-spawn`, gate cutoff 80.0, full period.
+
+| Metric | Arm A | Arm B | B − A |
+|---|---:|---:|---:|
+| Total return % | 424.60 | 264.40 | **−160.20** |
+| CAGR % | 51.60 | 38.10 | **−13.50** |
+| Max drawdown % | 21.40 | 23.60 | +2.20 |
+| Sharpe | 2.09 | 1.57 | **−0.52** |
+| Trades | 322 | 346 | +24 |
+| Win rate % | 37.0 | 35.0 | −2.0 |
+| Final equity | 52,459 | 36,439 | −16,020 |
+
+Context: 202,765 → 229,642 candidates, 1,086 → 1,315 qualified. Gate cutoff 80.0
+in both arms.
+
+## What actually caused it
+
+**Not the failures.** `worst_trade_r` is **−3.26 in both arms**, and
+`best_trade_r` is 12.87 in both. A position held into a terminal mark gaps to
+$0.01 and fills at the open, which would land far below −3.26 — at candidate
+level those trades score −3.13R (SIVB) and −5.99R (SBNY). No such trade appears
+in either book, so none of the three failed banks reached the production
+portfolio.
+
+**What the control does and does not show.** Arm B replayed *without*
+`--terminal-events` (`backtest-20260907-survivorship-b-nomarks.json`, same
+snapshot, same config) produces a production row identical to arm B with them in
+every field — 346 trades, 38.1% CAGR, 1.57 Sharpe, 23.6% drawdown, 264.4% total
+return, −3.26 worst trade. Candidates rise 229,617 → 229,642, the 25 extra
+signals the option exposes in the final 29 bars of the three symbols, while
+`qualified` is 1,315 either way.
+
+So the control establishes precisely this: **those 25 additional replay signals
+changed no portfolio decision.** It does *not* independently establish that the
+terminal marks had no effect — both runs read `survivorship-b.sqlite`, which
+contains the synthetic bars either way, so a setup generated earlier could still
+resolve against one in both runs. Isolating the marks themselves would need a
+third snapshot built without them, which has not been run.
+
+The separate evidence that no marked position reached the book is the trade
+extremes: a position resolving against a $0.01 bar fills at that bar's open and
+scores far worse than −3.26 (−3.13R for SIVB and −5.99R for SBNY at candidate
+level), yet arm B's `worst_trade_r` equals arm A's exactly. That is inference
+from the extremes, not a controlled measurement.
+
+On that basis the −13.5pp CAGR is **universe recomposition**: 85 additional names
+(+17% of the universe) shift the cross-sectional percentile ranks and compete for
+the same 15 position slots, displacing trades arm A took. The strategy is
+markedly sensitive to which universe it ranks over — which is the finding, and it
+is a different claim from "survivorship bias costs 13.5pp".
+
+## Caveats
+
+- **This is not a bias estimate and not a lower bound.** Restoring names is only
+  one of the two corrections. Index-*addition* lookahead — today's members being
+  tradeable before they joined — is not removed. Either correction can move
+  returns in either direction, so B − A is the sensitivity observed under these
+  assumptions, nothing more. A small delta would not have shown the bias to be
+  small, and this large one does not establish its size either.
+- **Arm B is not a point-in-time universe.** Still-trading former members remain
+  eligible after their removal date; `run_backtest` does not gate on
+  `delisted_on`, and an index-removal date would be the wrong field for it
+  anyway. A true PIT audit is separate work.
+- **Four unresolved endings.** CMA (ends 2026-01-30), SEE (2026-04-08), HBI
+  (2025-11-28) and DISH (2023-12-29) stop trading before the window closes for
+  reasons the source row does not explain. Their bars are real and their
+  delisting fields are NULL; the ending status is simply unknown.
+- **Three near-total-loss assumptions** (SIVB, SBNY, FRC), as described above.
+  The control shows they did not affect this result, but they remain assumptions
+  and would matter in any run where such a name did clear the gate.
+- **Acquisitions are approximated** by the last available close. Cash or stock
+  consideration and the release of capital on the deal date are not modelled.
+- **FBIN carries a corporate action.** Fortune Brands distributed MasterBrand on
+  2022-12-14 and began trading as FBIN on 2022-12-15. Alpaca's split adjustment
+  does not adjust for spin-offs, so the distribution appears as a real −6.91%
+  close-to-close move (63.22 → 61.90 → 57.62). A holder did not lose that value,
+  but any signal or stop crossing that date reads it as a decline. MBC enters the
+  dataset separately with its first bar the same day.
+- **Window.** 2021-06 to 2026-07, because that is what `prod.sqlite` holds. The
+  2023 bank failures are inside it; the 2016–2021 period is not.
+- **Snapshot schema.** The snapshot files on disk predate migrations 026 and 032,
+  so `select(Ticker)` against them fails with `no such column: tickers.cik` — the
+  path `backtest_service._fetch_columns` takes. Per-ticker exceptions are caught
+  and logged, so a run against an unsynced snapshot completes and writes a report
+  with **zero candidates** rather than failing. The prep script syncs both copies
+  against `Base.metadata`; anything else reading these snapshots needs the same.
+
+## Reproduce
+
+```powershell
+python scripts/prep_survivorship_snapshot.py --base backtest_snapshots/prod.sqlite
+python scripts/run_backtest_snapshot.py backtest_snapshots/survivorship-a.sqlite `
+    --out reports/backtest-YYYYMMDD-survivorship-a.json --allow-spawn --workers 6
+python scripts/run_backtest_snapshot.py backtest_snapshots/survivorship-b.sqlite `
+    --terminal-events backtest_snapshots/survivorship-b-terminal-events.json `
+    --out reports/backtest-YYYYMMDD-survivorship-b.json --allow-spawn --workers 6
+```
+
+`survivorship-b-prep-manifest.json` records the base hash, per-symbol coverage,
+the terminal events written and every unresolved item for the run.

docs/research/tr-rank-agreement.md

@@ -0,0 +1,118 @@
+# Total-return vs price-return momentum — ranking screen
+
+**Run date:** 2026-09-07 · **Verdict: proceed to Step 3** (pre-registered rule fired)
+**Reports:** `reports/tr-rank-agreement.json`, `reports/tr-rank-agreement-mechanism.json`
+**Script:** `scripts/research_tr_rank_agreement.py`
+**Cache:** `backtest_snapshots/tr-rank-cache.sqlite` (507 symbols, 630,524 aligned bars)
+
+## Question
+
+12-1 momentum is defined on total returns; production computes it on
+split-adjusted prices, which ignore dividends. Before building the production
+`tr_factor` machinery (Step 3 of [the plan](../data-integrity-plan.md)), does the
+basis actually change what the gate admits and how it orders?
+
+Deliberately a screen, not a portfolio simulation. A difference here is a reason
+to measure trades, **not** evidence that P&L changes.
+
+## Configuration
+
+| | |
+|---|---|
+| Universe | `backtest_snapshots/survivorship-a.sqlite`, 505 symbols with bars (RHM has a ticker row but no bars in the snapshot, so it is inert in both arms) |
+| Window | 2021-06-24 → 2026-07-02, clipped to the snapshot |
+| Decision dates | every SPY session; 1,009 had enough history to evaluate |
+| Momentum | production `compute_residual_12_1_momentum`, unchanged; price series against nominal SPY, total-return series against total-return SPY |
+| Volatility leg | production `compute_realized_vol_6m` on the SPLIT series, computed once per date and **held fixed in both arms** so only momentum varies |
+| Gate | production `min_momentum_percentile` = 80.0, via production `_percentiles` |
+| Ordering | production `blend_strategy_rank`, symbol as deterministic tie-break, symbols fed in sorted order in both arms |
+| Comparison | the full admitted list, never truncated to `max_positions` |
+| Runtime | 146 s with the production function — no NumPy rewrite needed, so no equivalence test was required |
+
+**Coverage is complete.** One `Adjustment.SPLIT` and one `Adjustment.ALL` request
+per symbol; 507 symbols cached, **zero** dates present in one adjustment but not
+the other. No symbol with snapshot bars is missing from the cache — the script
+refuses to run in that case rather than reporting the gap as agreement.
+
+## Result
+
+| | |
+|---|---:|
+| Evaluated dates | 1,009 |
+| Dates with a **different ordered list** | **1,009 (100%)** |
+| Dates with a different admitted set | 1,007 (99.8%) |
+| Mean set disagreement, `\|A △ B\| / \|A ∪ B\|` | 0.0867 |
+| Max set disagreement | 0.2143 |
+| Mean admitted names per arm | 100.09 |
+| Mean **replacements** per date (one out paired with one in) | **4.56** |
+| Max replacements on a single date | 12 |
+| Mean symmetric difference (names counted on both sides) | 9.13 |
+
+Per year, every evaluated date differs:
+
+| Year | Dates | Different set | Different order |
+|---|---:|---:|---:|
+| 2022 | 132 | 132 | 132 |
+| 2023 | 250 | 250 | 250 |
+| 2024 | 252 | 252 | 252 |
+| 2025 | 250 | 248 | 250 |
+| 2026 | 125 | 125 | 125 |
+
+Counted directly from the daily set differences rather than derived from the
+Jaccard ratio: **4.56 names are replaced on an average date** — 4.56 leave and
+4.56 arrive, so 9.13 names appear on one side or the other, out of about 100
+admitted. A swap moves two names, which makes the symmetric difference twice the
+replacement count and the two easy to conflate.
+
+## Strong evidence of a dividend effect
+
+First changed date, 2022-06-24 — 99 names admitted in each arm, nine swapped:
+
+- **Only on price return:** CDNS, MAR, FIX, COST, LIN, AON, STE, CI, AJG
+- **Only on total return:** EXC, PSX, WY, AMCR, DTE, GIS, FE, BMY, ES
+
+Growth and low-yield names out; utilities, energy, staples and other high payers
+in. Nine tickers are not evidence, so it was measured (`--mechanism 25`, saved to
+`reports/tr-rank-agreement-mechanism.json` with its sampled dates and per-date
+values): across 40 sampled dates, the Spearman correlation between a name's
+trailing 12-month implied yield — from the ALL/SPLIT ratio over the formation
+window — and its momentum-percentile shift between arms is **+0.79** (range
++0.71 to +0.85, every sampled date positive).
+
+**What that does and does not pin down.** The reordering is systematic and
+directional rather than numerical noise, and it moves with yield. It is *not* a
+clean isolation of the stock's own yield: both the stock series and the benchmark
+leg change between arms, so the measured shift also contains the beta-weighted
+benchmark yield. Read it as strong evidence consistent with a dividend effect.
+Isolating own-yield would need a third variant — total-return stock series
+against a nominal SPY leg — which has not been run.
+
+## Verdict
+
+The pre-registered rule was "no ordered-list difference on any evaluated date →
+stop". Differences occur on **every** evaluated date, so **Step 3 proceeds**.
+
+What this does and does not establish:
+
+- **Does:** the momentum basis materially changes gate admission and ordering,
+  every day, in a yield-driven direction.
+- **Does not:** show that realised P&L changes. Production also applies setup
+  qualification, existing holdings, available slots and the rest of the admission
+  path. A name that swaps into the admitted list may never produce a setup. Step
+  3's paired portfolio replay is what decides whether any of this reaches trades.
+- **Worth noting:** the tuning recorded as closed in the research log was carried
+  out on a price-return basis throughout. This screen does not invalidate it —
+  it says the input those conclusions were computed on is not the one the factor
+  is defined on.
+
+## Reproduce
+
+```powershell
+python scripts/research_tr_rank_agreement.py --fetch          # 1,014 requests, cached
+python scripts/research_tr_rank_agreement.py --time-batch 20  # runtime projection
+python scripts/research_tr_rank_agreement.py                  # full screen
+python scripts/research_tr_rank_agreement.py --mechanism 25   # yield vs percentile shift
+```
+
+The cache makes re-runs free and the screen deterministic; delete
+`backtest_snapshots/tr-rank-cache.sqlite` to refetch.

docs/research/tr-total-return-ab.md

@@ -0,0 +1,160 @@
+# Total-return momentum — paired portfolio replay (Step 3a)
+
+**Run date:** 2026-09-07 · **Status:** measured; **no rollout decision taken**
+**Report:** `reports/tr-total-return-ab.json` · **Script:** `scripts/run_total_return_ab.py`
+**Inputs:** `backtest_snapshots/survivorship-a.sqlite` (bars),
+`backtest_snapshots/tr-rank-cache.sqlite` (both adjusted series),
+`backtest_snapshots/tr-ab-candidates.pkl` (202,765 shared candidates)
+
+## Question
+
+[The ranking screen](tr-rank-agreement.md) showed the momentum basis changes the
+admitted list on every evaluated date. Does that reach actual admissions and P&L?
+
+## What was held identical
+
+Setup candidates were generated **once** (202,765 over 505 symbols, matching the
+count in the standalone backtest report) and reused by both books. Only momentum
+and its dependents were recomputed per arm: residual and activation percentiles,
+`qualified`, and the 80/20 blend used for ordering.
+
+| Held fixed | Varied |
+|---|---|
+| Setup geometry — entry, stop, target, R:R | Momentum series: SPLIT stock vs SPLIT SPY (control), ALL vs ALL SPY (treatment) |
+| Execution on the snapshot's nominal bars | The percentiles, `qualified` and ordering that follow from it |
+| `vol_6m` volatility leg, from candidate generation | |
+| Non-momentum gates, `max_positions`, risk per trade, ATR-trail exit, gate-reset re-entry | |
+
+Both arms read the same cache, so a refreshed-vs-snapshot difference cannot be
+mistaken for a basis difference. Each book then keeps its own holdings, so a
+divergent admission changes later slot occupancy — that path dependence is part
+of what is measured.
+
+**Harness parity.** The control arm reproduces the production row of
+`reports/backtest-20260907-survivorship-a.json` exactly — 322 trades, 424.6%
+total return, 51.6% CAGR, 21.4% drawdown, 2.09 Sharpe. Any difference below is
+therefore attributable to the basis, not to a re-implemented book.
+
+## Result
+
+| Metric | Control (price return) | Treatment (total return) | Δ |
+|---|---:|---:|---:|
+| Trades | 322 | 334 | +12 |
+| Total return % | 424.6 | 376.5 | −48.1 |
+| CAGR % | 51.6 | 47.9 | **−3.7** |
+| Max drawdown % | 21.4 | 19.5 | **−1.9** (better) |
+| Sharpe | 2.09 | 1.98 | −0.11 |
+| Win rate % | 37.0 | 36.8 | −0.2 |
+| Profit factor | 2.48 | 2.33 | −0.15 |
+
+**The books really do differ.** 271 entries are shared; **51 are taken only by
+control and 63 only by treatment** — about 30% of the union of entries. The
+composition follows the Step 2 mechanism: control takes MAR, AAPL, BIIB, LVS;
+treatment takes BG, OKE, DE, ARES. Some differences are timing rather than
+selection — ABBV is entered on 2022-09-15 by control and 2022-09-22 by treatment.
+
+## Where the difference sits
+
+**One year carries all of it.** Per-year returns:
+
+| Year | Control % | Treatment % | Δpp |
+|---|---:|---:|---:|
+| 2022 | 8.1 | 9.9 | +1.8 |
+| 2023 | 42.0 | 41.9 | −0.1 |
+| 2024 | 36.4 | 38.4 | +2.0 |
+| 2025 | 80.2 | 58.1 | **−22.1** |
+| 2026 | 39.2 | 39.6 | +0.4 |
+
+Treatment wins three years and loses two, and four of the five differ by under
+2.1pp. The entire −3.7pp CAGR gap is 2025. That is a description of *where* the
+shortfall sits — a useful thing to know when reading it — and nothing more. It is
+not grounds for calling the shortfall noise or the estimate unstable.
+
+**Uncertainty is not yet quantified.** The per-arm `sharpe_se` values (0.497 and
+0.498) are the standard errors of each Sharpe *individually*. They say nothing
+about the uncertainty of the **difference**: the two books share dates and most
+of their positions, so the arms are strongly correlated and the standard error of
+their difference depends on that covariance, not on the two marginal errors. An
+earlier version of this document compared the 0.11 Sharpe gap to a 0.50 marginal
+SE and concluded the books were "statistically indistinguishable" — that
+inference does not follow, and the same applies to the drawdown difference, for
+which no error estimate was reported at all. Both claims are withdrawn.
+
+## Paired block bootstrap on the difference
+
+Pre-specified before running — seed `20260907`, **5,000** replicates, block
+lengths **21 and 63** sessions, both reported rather than whichever reads better.
+The two arms' daily return streams are date-aligned (the script aborts if they
+are not), and **the same block indices are drawn for both arms**, so the pairing
+and whatever cross-arm correlation exists on those days are preserved. Moving
+blocks retain some of the serial dependence an i.i.d. resample would destroy.
+
+Treatment − control, 95% interval:
+
+| Block | Metric | Mean | 2.5% | 97.5% | Share of replicates < 0 |
+|---|---|---:|---:|---:|---:|
+| 21 | ΔCAGR pp | −3.61 | −14.76 | +6.45 | 0.75 |
+| 21 | ΔSharpe | −0.109 | −0.433 | +0.198 | 0.75 |
+| 21 | ΔMaxDD pp | −1.00 | −7.50 | +4.34 | 0.63 |
+| 63 | ΔCAGR pp | −3.78 | −14.67 | +5.40 | 0.76 |
+| 63 | ΔSharpe | −0.113 | −0.428 | +0.182 | 0.76 |
+| 63 | ΔMaxDD pp | −1.79 | −8.22 | +2.68 | 0.78 |
+
+The two block lengths agree closely, which is reassuring about the resampling
+itself. Every interval spans zero, and roughly three-quarters of replicates fall
+below zero for CAGR and Sharpe (treatment worse) and for drawdown (treatment
+better, since a negative ΔMaxDD is a shallower drawdown). So the observed window
+leans toward control on return and treatment on drawdown without resolving either
+sign confidently.
+
+**What this diagnostic is not.** It resamples the *realised* daily return streams;
+it does not re-run the strategy, so it cannot capture how a reordered history
+would have changed admissions or holdings. It is an exploratory uncertainty
+estimate from the observed window — not out-of-sample validation, and not a
+rollout gate.
+
+## Verdict
+
+**Decision: production stays unchanged.** No schema, no ingestion, no wider
+parameter sweep.
+
+- **Established:** the basis is production-relevant. It changes roughly 30% of
+  entries, in the yield-driven direction the screen predicted.
+- **Not established:** which basis performs better. Point estimates favour control
+  on return and Sharpe and treatment on drawdown; every bootstrap interval spans
+  zero, with about three-quarters of the mass on the point-estimate side. The
+  observed window does not resolve the sign.
+- No acceptable return sacrifice is being chosen from these numbers, and no
+  threshold is being fitted to them after the fact.
+
+What a decision would still need:
+
+1. **A pre-registered numeric rule, set before it is applied.** The plan commits
+   to non-inferiority in principle — total return is the basis 12-1 momentum is
+   defined on, so an in-sample shortfall should not automatically lock in the
+   basis we believe is measured wrong — but carries no threshold. An earlier draft
+   used ΔCAGR ≥ −0.50pp at ΔMaxDD ≤ +1.50pp. Choosing a threshold now, with
+   −3.7pp on the table, would be fitting the rule to the answer.
+2. **Out-of-sample evidence.** One five-year snapshot, in-sample throughout.
+3. Note what would *not* help: a matrix over capacity, start date or cadence
+   produces **sensitivity checks, not independent samples**. Fifty-three
+   overlapping paths from one price history are not fifty-three pieces of
+   evidence, and treating them that way would overstate what a sweep buys.
+
+## Reproduce
+
+```powershell
+python scripts/run_total_return_ab.py --build-candidates   # ~9 min, cached
+python scripts/run_total_return_ab.py                      # both arms + bootstrap, ~40 s
+```
+
+The bootstrap parameters are constants in the script (`BOOTSTRAP_SEED`,
+`BOOTSTRAP_REPLICATES`, `BOOTSTRAP_BLOCKS`) rather than flags, so a run cannot
+quietly pick a different seed or block length.
+
+Guards in `tests/unit/test_total_return_ab.py`: identical input series must
+produce identical books (so an observed difference is the basis, not replay
+nondeterminism); a controlled momentum change must move a name across the cutoff
+(so the harness can see what it is looking for); the volatility leg must be
+untouched between arms; and a name with no computable momentum in an arm must not
+qualify on a value carried over from candidate generation.

frontend/src/components/signals/BacktestDataQualityCard.tsx

@@ -0,0 +1,77 @@
+import { Callout } from '../ui/Callout';
+import { Disclosure } from '../ui/Disclosure';
+import type { BacktestDataQuality } from '../../lib/types';
+
+/** How many stocks were expected, loaded and actually evaluated.
+ *
+ *  A run that lost every ticker still writes a report — the replay loop catches
+ *  per-ticker errors so one bad name can't abort a 500-symbol run — and that
+ *  report used to be indistinguishable from an honest one that found no setups.
+ *  When the status is `failed` the panel shows this instead of the performance
+ *  tiles, because a headline computed from a broken run is worse than none.
+ */
+export function BacktestDataQualityCard({ quality }: { quality: BacktestDataQuality }) {
+  const failed = quality.status === 'failed';
+  const { benchmark } = quality;
+
+  return (
+    <Callout variant={failed ? 'error' : 'warning'}>
+      <p className="font-medium">
+        {failed ? 'This run is not usable' : 'This run is incomplete'}
+      </p>
+      <ul className="mt-2 list-disc space-y-1 pl-4 text-xs opacity-90">
+        {quality.reasons.map((reason) => (
+          <li key={reason}>{reason}</li>
+        ))}
+      </ul>
+      <p className="mt-2 text-xs opacity-80">
+        {quality.evaluated} of {quality.expected} tickers evaluated
+        {quality.loaded !== quality.evaluated && <> · {quality.loaded} loaded</>}
+        {quality.short_history > 0 && (
+          <> · {quality.short_history} too short to trade</>
+        )}
+        {' · '}
+        {benchmark.momentum_basis === 'residual' ? (
+          <>
+            {benchmark.symbol} {benchmark.sessions} sessions
+            {benchmark.first && benchmark.last && (
+              <> ({benchmark.first} → {benchmark.last})</>
+            )}
+          </>
+        ) : (
+          <>no {benchmark.symbol} benchmark — momentum ran raw, not residual</>
+        )}
+      </p>
+      {failed && (
+        <p className="mt-2 text-xs opacity-80">
+          Performance figures are hidden for this run. Re-run the backtest once the
+          underlying data is readable.
+        </p>
+      )}
+      {/* Says what the check covers, so a clean run isn't read as a price audit. */}
+      <p className="mt-2 text-[11px] opacity-70">
+        Processing and coverage checks only — this does not verify that the underlying
+        prices are correct.
+      </p>
+      {quality.failures.length > 0 && (
+        <div className="mt-2">
+          <Disclosure summary={`Failures (${quality.failed})`}>
+            <ul className="space-y-1 font-mono text-[11px] opacity-80">
+              {quality.failures.map((f) => (
+                <li key={`${f.symbol}-${f.stage}`}>
+                  {f.symbol} · {f.stage} · {f.error}
+                </li>
+              ))}
+            </ul>
+            {quality.failures_truncated && (
+              <p className="mt-1 text-[11px] opacity-70">
+                Showing the first {quality.failures.length} of {quality.failed} — the
+                rest are in the run log.
+              </p>
+            )}
+          </Disclosure>
+        </div>
+      )}
+    </Callout>
+  );
+}

frontend/src/components/signals/BacktestPanel.tsx

@@ -9,6 +9,7 @@ import { Disclosure } from '../ui/Disclosure';
 import { Dropdown } from '../ui/Dropdown';
 import { Section } from '../ui/Section';
 import { useToast } from '../ui/Toast';
+import { BacktestDataQualityCard } from './BacktestDataQualityCard';
 import { BacktestRecommendationCard } from './BacktestRecommendationCard';
 import { PortfolioMonitorPanel } from './PortfolioMonitorPanel';
 
@@ -39,6 +40,8 @@ export function BacktestPanel() {
   const [targetModel, setTargetModel] = useState<BacktestTargetModel>('production_gtl');
   const [cadence, setCadence] = useState<BacktestCadence>('weekly');
 
+  const quality = report?.data_quality ?? null;
+  const runIsUnusable = quality?.status === 'failed';
   const monitor = report?.portfolio_monitor ?? null;
   const activeStrategy =
     selectedStrategy || monitor?.production_strategy || monitor?.strategies[0]?.strategy || '';
@@ -89,8 +92,15 @@ export function BacktestPanel() {
             <p className="section-index">Last run</p>
             {report ? (
               <p className="mt-1 text-xs text-gray-400">
-                {timeAgo(report.generated_at)} · {report.tickers} tickers ·{' '}
-                {report.candidates} setups ({report.qualified} qualified) ·{' '}
+                {timeAgo(report.generated_at)} ·{' '}
+                {/* Evaluated over expected, not the universe size: "506 tickers"
+                    on a run that evaluated none was the misleading part. */}
+                <span className={quality && quality.status !== 'ok' ? 'text-amber-300' : undefined}>
+                  {quality
+                    ? `${quality.evaluated}/${quality.expected} tickers`
+                    : `${report.tickers} tickers`}
+                </span>{' '}
+                · {report.candidates} setups ({report.qualified} qualified) ·{' '}
                 {report.params.entry_cadence ?? 'weekly'},{' '}
                 {report.params.horizon_days}d horizon
                 {report.params.cost_per_side_pct != null && (
@@ -177,7 +187,20 @@ export function BacktestPanel() {
           </Callout>
         )}
 
-        {report && (
+        {/* Completeness gates the headline. `failed` replaces the tiles rather
+            than warning above them — a banner over ordinary-looking numbers is
+            exactly how a zero-candidate run reads as a quiet week. */}
+        {report && quality && quality.status !== 'ok' && (
+          <BacktestDataQualityCard quality={quality} />
+        )}
+
+        {report && !quality && (
+          <p className="text-[11px] text-gray-500">
+            Completeness was not recorded for this run — it predates the check.
+          </p>
+        )}
+
+        {report && !runIsUnusable && (
           <>
             <PortfolioMonitorPanel
               monitor={monitor}

frontend/src/components/signals/EvaluationPanel.tsx

@@ -38,14 +38,18 @@ export function EvaluationPanel() {
   const { data: report } = useBacktestReport();
   const liveAvgR = perf?.overall.avg_r ?? null;
   const liveN = perf?.overall.total ?? 0;
-  const btAvgR = report?.overall_qualified.avg_r ?? null;
+  // A failed run's expectancy is not a baseline — comparing live results to it
+  // would report "drift" or "tracking" off numbers from a broken replay.
+  const btAvgR =
+    report?.data_quality?.status === 'failed' ? null : (report?.overall_qualified.avg_r ?? null);
   let status: PipelineStatus = 'building';
   if (liveAvgR != null && liveN >= MIN_MATURED) {
     status = btAvgR == null ? 'no-backtest' : liveAvgR >= btAvgR - DRIFT_TOLERANCE_R ? 'tracking' : 'drift';
   }
   const statusNote: Record<PipelineStatus, string> = {
     building: `Fewer than ~${MIN_MATURED} matured setups so far — too few to compare.`,
-    'no-backtest': 'Run the backtest to get a target/stop baseline to check against.',
+    'no-backtest':
+      'No usable backtest baseline to check against — run the backtest, or fix the data the last run reported as incomplete.',
     tracking:
       "Live setup outcomes are resolving in line with the backtest's target/stop model — the outcome-evaluation pipeline shows no look-ahead, config or data drift. (Checks the setup-grading pipeline, not the ATR-trail production book above.)",
     drift:

frontend/src/hooks/useAdmin.ts

@@ -274,7 +274,16 @@ export function useBootstrapTickers() {
       qc.invalidateQueries({ queryKey: ['admin', 'ticker-universe'] });
       addToast(
         'success',
-        `Bootstrap done (${result.source ?? 'unknown'}): +${result.added}, existing ${result.already_tracked}, deleted ${result.deleted}`
+        // "untracked", not "deleted": pruning leaves the rows and their history
+        // in place and only removes the symbol from the scanner.
+        `Bootstrap done (${result.source ?? 'unknown'}): +${result.added}, existing ${result.already_tracked}, untracked ${result.untracked ?? 0}`
+        + (result.retracked ? `, back in scanner ${result.retracked}` : '')
+        + (result.kept_manual_count ? `, kept manual ${result.kept_manual_count}` : '')
+        // A cache/seed fallback means the live fetch failed: say we never looked
+        // rather than letting it read as "the index did not change".
+        + (result.membership_observed === false
+          ? ' · membership not recorded (live fetch failed)'
+          : '')
         + (result.added_symbols?.length
           ? ` · added ${result.added_symbols.slice(0, 8).join(', ')}${result.added > 8 ? '…' : ''}`
           : ''),

frontend/src/lib/baseRates.ts

@@ -44,9 +44,15 @@ const EXIT_LABELS: Record<string, string> = {
 
 /**
  * Pull the production strategy's full-history row out of a backtest report.
- * Returns null when the report hasn't run or has no production row.
+ * Returns null when the report hasn't run, failed its completeness check, or
+ * has no production row.
  */
 export function productionBaseRates(report: unknown): BaseRates | null {
+  // Base rates quoted next to a live setup carry no caveat, so a run that lost
+  // its universe must yield none at all rather than plausible-looking ones.
+  if ((report as { data_quality?: { status?: string } })?.data_quality?.status === 'failed') {
+    return null;
+  }
   const monitor = (report as { portfolio_monitor?: { production_strategy?: string; runs?: MonitorRun[] } })
     ?.portfolio_monitor;
   if (!monitor?.runs?.length) return null;

frontend/src/lib/types.ts

@@ -427,11 +427,39 @@ export interface BacktestSignalEvalRow {
   reliable: boolean;
 }
 
+export interface BacktestDataQuality {
+  /** `failed` = the headline can't be read as the strategy it claims to be. */
+  status: 'ok' | 'degraded' | 'failed';
+  complete: boolean;
+  reasons: string[];
+  expected: number;
+  loaded: number;
+  evaluated: number;
+  no_data: number;
+  no_data_symbols: string[];
+  short_history: number;
+  failed: number;
+  failures_by_stage: Record<string, number>;
+  failures: { symbol: string; stage: string; error: string }[];
+  failures_truncated: boolean;
+  benchmark: {
+    symbol: string;
+    sessions: number;
+    first: string | null;
+    last: string | null;
+    momentum_basis: 'residual' | 'raw_fallback';
+    error: string | null;
+  };
+}
+
 export interface BacktestReport {
   generated_at: string;
   tickers: number;
   candidates: number;
   qualified: number;
+  /** Absent on reports generated before completeness tracking — treat as
+   *  unknown rather than clean; that assumption is the bug this block fixes. */
+  data_quality?: BacktestDataQuality;
   params: {
     step_days: number;
     step_sessions?: number;
@@ -1010,9 +1038,23 @@ export interface TickerUniverseBootstrapResult {
   total_universe_symbols: number;
   added: number;
   already_tracked: number;
+  /** Always 0 — pruning marks symbols untracked and never deletes rows. */
   deleted: number;
+  /** Symbols dropped from the scanner; their bars and linked records remain. */
+  untracked?: number;
+  untracked_symbols?: string[];
+  /** Symbols that rejoined the universe and were returned to the live path. */
+  retracked?: number;
+  /** Absent from the universe but manually owned, so a prune left them alone. */
+  kept_manual?: string[];
+  kept_manual_count?: number;
+  /** False when the live fetch failed (cache/seed fallback) — nothing was recorded. */
+  membership_observed?: boolean;
+  membership_opened?: number;
+  membership_closed?: number;
   /** Sample of newly added symbols (capped server-side). */
   added_symbols?: string[];
+  prune_note?: string;
 }
 
 export interface PipelineReadiness {

reports/tr-rank-agreement-mechanism.json

@@ -0,0 +1,105 @@
+{
+  "generated_at": "2026-09-07T10:55:05.693160+00:00",
+  "snapshot": "C:\\Workspace\\signal-platform\\backtest_snapshots\\survivorship-a.sqlite",
+  "cache": "C:\\Workspace\\signal-platform\\backtest_snapshots\\tr-rank-cache.sqlite",
+  "window": {
+    "start": "2021-06-24",
+    "end": "2026-07-02"
+  },
+  "universe_size": 505,
+  "sample_every_n_sessions": 25,
+  "result": {
+    "what_it_measures": "Spearman correlation, per sampled date, between a name's trailing 12-month implied yield (from the ALL/SPLIT ratio over the formation window) and the change in its momentum percentile between the two arms.",
+    "caveat": "Both the stock series and the benchmark leg change between arms, so this is strong evidence consistent with a dividend effect - it does not isolate the stock's own yield from the beta-weighted benchmark yield.",
+    "formation_window_bars": 252,
+    "min_symbols_per_date": 30,
+    "sampled_dates": 40,
+    "sample_date_list": [
+      "2022-07-28",
+      "2022-09-01",
+      "2022-10-07",
+      "2022-11-11",
+      "2022-12-19",
+      "2023-01-26",
+      "2023-03-03",
+      "2023-04-10",
+      "2023-05-15",
+      "2023-06-21",
+      "2023-07-27",
+      "2023-08-31",
+      "2023-10-06",
+      "2023-11-10",
+      "2023-12-18",
+      "2024-01-25",
+      "2024-03-01",
+      "2024-04-08",
+      "2024-05-13",
+      "2024-06-18",
+      "2024-07-25",
+      "2024-08-29",
+      "2024-10-04",
+      "2024-11-08",
+      "2024-12-16",
+      "2025-01-24",
+      "2025-03-03",
+      "2025-04-07",
+      "2025-05-13",
+      "2025-06-18",
+      "2025-07-25",
+      "2025-08-29",
+      "2025-10-06",
+      "2025-11-10",
+      "2025-12-16",
+      "2026-01-23",
+      "2026-03-02",
+      "2026-04-07",
+      "2026-05-12",
+      "2026-06-17"
+    ],
+    "mean_spearman_yield_vs_percentile_shift": 0.7924,
+    "min": 0.7138,
+    "max": 0.8459,
+    "per_date_spearman": [
+      0.7814,
+      0.7933,
+      0.8216,
+      0.8325,
+      0.8108,
+      0.8147,
+      0.8074,
+      0.8421,
+      0.7867,
+      0.7907,
+      0.7682,
+      0.7568,
+      0.7849,
+      0.7536,
+      0.7138,
+      0.7385,
+      0.7414,
+      0.7697,
+      0.8078,
+      0.8325,
+      0.8132,
+      0.8459,
+      0.8397,
+      0.7966,
+      0.7808,
+      0.8285,
+      0.8068,
+      0.8376,
+      0.7974,
+      0.8256,
+      0.8046,
+      0.7762,
+      0.8008,
+      0.8046,
+      0.7793,
+      0.7978,
+      0.7678,
+      0.7549,
+      0.7501,
+      0.7384
+    ]
+  }
+}

reports/tr-rank-agreement.json

@@ -0,0 +1,540 @@
+{
+  "generated_at": "2026-09-07T10:57:26.602851+00:00",
+  "snapshot": "C:\\Workspace\\signal-platform\\backtest_snapshots\\survivorship-a.sqlite",
+  "cache": "C:\\Workspace\\signal-platform\\backtest_snapshots\\tr-rank-cache.sqlite",
+  "window": {
+    "start": "2021-06-24",
+    "end": "2026-07-02"
+  },
+  "universe_size": 505,
+  "inert_snapshot_symbols": [
+    "RHM"
+  ],
+  "cutoff": 80.0,
+  "verdict": "proceed to Step 3 - ordered lists differ",
+  "result": {
+    "evaluated_dates": 1009,
+    "skipped_dates_no_history": 252,
+    "dates_with_different_admitted_set": 1007,
+    "dates_with_different_ordered_list": 1009,
+    "mean_set_disagreement": 0.08668279921946223,
+    "max_set_disagreement": 0.21428571428571427,
+    "mean_admitted_per_arm": 100.09,
+    "mean_names_only_in_price_return": 4.56,
+    "mean_names_only_in_total_return": 4.56,
+    "mean_replacements_per_date": 4.56,
+    "max_replacements_per_date": 12,
+    "mean_symmetric_difference": 9.13,
+    "per_year": {
+      "2022": {
+        "dates": 132,
+        "changed_sets": 132,
+        "changed_orders": 132
+      },
+      "2023": {
+        "dates": 250,
+        "changed_sets": 250,
+        "changed_orders": 250
+      },
+      "2024": {
+        "dates": 252,
+        "changed_sets": 252,
+        "changed_orders": 252
+      },
+      "2025": {
+        "dates": 250,
+        "changed_sets": 248,
+        "changed_orders": 250
+      },
+      "2026": {
+        "dates": 125,
+        "changed_sets": 125,
+        "changed_orders": 125
+      }
+    },
+    "examples": [
+      {
+        "date": "2022-06-24",
+        "admitted_price_return": 99,
+        "admitted_total_return": 99,
+        "only_in_price_return": [
+          "CDNS",
+          "MAR",
+          "FIX",
+          "COST",
+          "LIN",
+          "AON",
+          "STE",
+          "CI",
+          "AJG"
+        ],
+        "only_in_total_return": [
+          "EXC",
+          "PSX",
+          "WY",
+          "AMCR",
+          "DTE",
+          "GIS",
+          "FE",
+          "BMY",
+          "ES"
+        ],
+        "order_changed_without_membership_change": false,
+        "first_10_price_return": [
+          "OXY",
+          "EQT",
+          "APA",
+          "DVN",
+          "MOS",
+          "CTRA",
+          "ON",
+          "ALB",
+          "CF",
+          "SMCI"
+        ],
+        "first_10_total_return": [
+          "OXY",
+          "EQT",
+          "APA",
+          "DVN",
+          "MOS",
+          "CTRA",
+          "ON",
+          "CF",
+          "ALB",
+          "SMCI"
+        ]
+      },
+      {
+        "date": "2022-06-27",
+        "admitted_price_return": 99,
+        "admitted_total_return": 99,
+        "only_in_price_return": [
+          "FCX",
+          "COST",
+          "FIX",
+          "CI",
+          "PAYX",
+          "HSIC",
+          "L",
+          "AIZ"
+        ],
+        "only_in_total_return": [
+          "EXC",
+          "MO",
+          "AMCR",
+          "GIS",
+          "DTE",
+          "CMS",
+          "FE",
+          "BMY"
+        ],
+        "order_changed_without_membership_change": false,
+        "first_10_price_return": [
+          "APA",
+          "OXY",
+          "DVN",
+          "EQT",
+          "MOS",
+          "ON",
+          "CTRA",
+          "ALB",
+          "CF",
+          "SMCI"
+        ],
+        "first_10_total_return": [
+          "APA",
+          "OXY",
+          "DVN",
+          "EQT",
+          "MOS",
+          "ON",
+          "CTRA",
+          "CF",
+          "SMCI",
+          "ALB"
+        ]
+      },
+      {
+        "date": "2022-06-28",
+        "admitted_price_return": 99,
+        "admitted_total_return": 99,
+        "only_in_price_return": [
+          "FCX",
+          "LYV",
+          "ULTA",
+          "CDNS",
+          "AMP",
+          "FIX",
+          "AJG"
+        ],
+        "only_in_total_return": [
+          "EXC",
+          "MO",
+          "AMCR",
+          "ETR",
+          "DTE",
+          "CMS",
+          "BMY"
+        ],
+        "order_changed_without_membership_change": false,
+        "first_10_price_return": [
+          "OXY",
+          "APA",
+          "DVN",
+          "MOS",
+          "EQT",
+          "CTRA",
+          "ON",
+          "CF",
+          "ALB",
+          "HAL"
+        ],
+        "first_10_total_return": [
+          "OXY",
+          "APA",
+          "DVN",
+          "EQT",
+          "MOS",
+          "CTRA",
+          "ON",
+          "CF",
+          "EXE",
+          "HAL"
+        ]
+      },
+      {
+        "date": "2022-06-29",
+        "admitted_price_return": 99,
+        "admitted_total_return": 99,
+        "only_in_price_return": [
+          "CDNS",
+          "FIX",
+          "COST",
+          "WAB",
+          "AJG",
+          "TRV"
+        ],
+        "only_in_total_return": [
+          "EXC",
+          "LYB",
+          "MO",
+          "AMCR",
+          "DTE",
+          "WEC"
+        ],
+        "order_changed_without_membership_change": false,
+        "first_10_price_return": [
+          "OXY",
+          "APA",
+          "DVN",
+          "EQT",
+          "MOS",
+          "CTRA",
+          "ALB",
+          "ON",
+          "CF",
+          "EXE"
+        ],
+        "first_10_total_return": [
+          "OXY",
+          "APA",
+          "DVN",
+          "EQT",
+          "MOS",
+          "CTRA",
+          "CF",
+          "ALB",
+          "EXE",
+          "ON"
+        ]
+      },
+      {
+        "date": "2022-06-30",
+        "admitted_price_return": 99,
+        "admitted_total_return": 99,
+        "only_in_price_return": [
+          "LYV",
+          "CDNS",
+          "RJF",
+          "WAB",
+          "FIX",
+          "AON",
+          "TKO"
+        ],
+        "only_in_total_return": [
+          "EXC",
+          "NRG",
+          "MO",
+          "LMT",
+          "DTE",
+          "WEC",
+          "AEE"
+        ],
+        "order_changed_without_membership_change": false,
+        "first_10_price_return": [
+          "OXY",
+          "APA",
+          "DVN",
+          "EQT",
+          "MOS",
+          "CTRA",
+          "ON",
+          "CF",
+          "ALB",
+          "HAL"
+        ],
+        "first_10_total_return": [
+          "OXY",
+          "APA",
+          "DVN",
+          "EQT",
+          "MOS",
+          "CTRA",
+          "ON",
+          "CF",
+          "EXE",
+          "HAL"
+        ]
+      },
+      {
+        "date": "2022-07-01",
+        "admitted_price_return": 99,
+        "admitted_total_return": 99,
+        "only_in_price_return": [
+          "DDOG",
+          "LYV",
+          "HLT",
+          "RJF",
+          "FIX",
+          "TKO",
+          "AON",
+          "AJG",
+          "ACGL"
+        ],
+        "only_in_total_return": [
+          "EXC",
+          "WY",
+          "NRG",
+          "AMCR",
+          "MO",
+          "PKG",
+          "LMT",
+          "CMS",
+          "KO"
+        ],
+        "order_changed_without_membership_change": false,
+        "first_10_price_return": [
+          "EQT",
+          "OXY",
+          "APA",
+          "DVN",
+          "MOS",
+          "CTRA",
+          "ON",
+          "SMCI",
+          "CF",
+          "EXE"
+        ],
+        "first_10_total_return": [
+          "EQT",
+          "OXY",
+          "APA",
+          "DVN",
+          "MOS",
+          "CTRA",
+          "ON",
+          "CF",
+          "SMCI",
+          "EXE"
+        ]
+      },
+      {
+        "date": "2022-07-05",
+        "admitted_price_return": 99,
+        "admitted_total_return": 99,
+        "only_in_price_return": [
+          "HLT",
+          "AON",
+          "TKO",
+          "L",
+          "TRV"
+        ],
+        "only_in_total_return": [
+          "EXC",
+          "WY",
+          "MO",
+          "CMS",
+          "DTE"
+        ],
+        "order_changed_without_membership_change": false,
+        "first_10_price_return": [
+          "APA",
+          "OXY",
+          "DVN",
+          "EQT",
+          "MOS",
+          "ON",
+          "SMCI",
+          "CTRA",
+          "CF",
+          "HAL"
+        ],
+        "first_10_total_return": [
+          "EQT",
+          "OXY",
+          "APA",
+          "DVN",
+          "MOS",
+          "ON",
+          "CTRA",
+          "SMCI",
+          "CF",
+          "EXE"
+        ]
+      },
+      {
+        "date": "2022-07-06",
+        "admitted_price_return": 99,
+        "admitted_total_return": 99,
+        "only_in_price_return": [
+          "TPL",
+          "ANET",
+          "HLT",
+          "LIN",
+          "UNH",
+          "CVS"
+        ],
+        "only_in_total_return": [
+          "EXC",
+          "RF",
+          "LYB",
+          "WY",
+          "MO",
+          "ETR"
+        ],
+        "order_changed_without_membership_change": false,
+        "first_10_price_return": [
+          "APA",
+          "OXY",
+          "DVN",
+          "EQT",
+          "MOS",
+          "ON",
+          "CTRA",
+          "SMCI",
+          "CF",
+          "HAL"
+        ],
+        "first_10_total_return": [
+          "APA",
+          "OXY",
+          "DVN",
+          "EQT",
+          "MOS",
+          "CTRA",
+          "ON",
+          "SMCI",
+          "CF",
+          "HAL"
+        ]
+      },
+      {
+        "date": "2022-07-07",
+        "admitted_price_return": 99,
+        "admitted_total_return": 99,
+        "only_in_price_return": [
+          "RJF",
+          "LIN",
+          "PKG",
+          "L"
+        ],
+        "only_in_total_return": [
+          "EXC",
+          "LYB",
+          "WY",
+          "ETR"
+        ],
+        "order_changed_without_membership_change": false,
+        "first_10_price_return": [
+          "APA",
+          "OXY",
+          "DVN",
+          "EQT",
+          "ON",
+          "MOS",
+          "SMCI",
+          "CTRA",
+          "HAL",
+          "CF"
+        ],
+        "first_10_total_return": [
+          "APA",
+          "EQT",
+          "OXY",
+          "DVN",
+          "MOS",
+          "ON",
+          "CTRA",
+          "SMCI",
+          "HAL",
+          "EXE"
+        ]
+      },
+      {
+        "date": "2022-07-08",
+        "admitted_price_return": 99,
+        "admitted_total_return": 99,
+        "only_in_price_return": [
+          "ANET",
+          "HLT",
+          "RJF",
+          "WAB",
+          "RSG",
+          "AIZ",
+          "HSY"
+        ],
+        "only_in_total_return": [
+          "EXC",
+          "LYB",
+          "AMCR",
+          "MO",
+          "ETR",
+          "AEP",
+          "ATO"
+        ],
+        "order_changed_without_membership_change": false,
+        "first_10_price_return": [
+          "APA",
+          "OXY",
+          "DVN",
+          "EQT",
+          "ON",
+          "MOS",
+          "CTRA",
+          "SMCI",
+          "HAL",
+          "VLO"
+        ],
+        "first_10_total_return": [
+          "APA",
+          "OXY",
+          "DVN",
+          "EQT",
+          "CTRA",
+          "MOS",
+          "ON",
+          "SMCI",
+          "HAL",
+          "EXE"
+        ]
+      }
+    ],
+    "elapsed_seconds": 132.0
+  },
+  "coverage": {
+    "symbols_cached": 507,
+    "symbols_with_unmatched_dates": 0,
+    "total_unmatched_dates": 0
+  }
+}

scripts/prep_survivorship_snapshot.py

@@ -0,0 +1,477 @@
+"""Build the two survivorship-audit snapshots from a base snapshot.
+
+Arm A is the universe the backtest already runs on. Arm B is the same universe
+plus the S&P 500 constituents removed while the snapshot window was open, so the
+strategy can trade the names that later left the index — including the ones that
+failed.
+
+The base file is never modified: both arms are copies, and the base hash is
+re-checked at the end.
+
+Three things this script has to get right, all of them learned from the data:
+
+* **Schema drift.** The snapshot files on disk predate migrations 026 and 032,
+  so ``select(Ticker)`` against them fails with ``no such column: tickers.cik``
+  — and that is the path ``backtest_service._fetch_columns`` takes. Both copies
+  are synced against ``Base.metadata`` before anything else touches them.
+* **Frozen tails.** Alpaca does not end a halted symbol's series; it repeats the
+  last price at zero volume. SBNY sits at 70.00 for 509 sessions after the FDIC
+  closed it, and INFO sits at 108.61 for 1,090 sessions before a different
+  company reuses the ticker. Only the symbols with a *reviewed cutoff* in the
+  dataset are truncated. Everywhere else a long zero-volume run is a tripwire:
+  the symbol is left out and reported for review rather than silently cut.
+* **Terminal marks.** A failed bank stops trading mid-collapse, so the exchange
+  bars never show the loss. A mark is written only where the dataset carries an
+  explicit confirmation *and* the retained final bar matches the reviewed
+  ``last_real_bar``. The mark is an assumption, not an observed execution.
+
+Nothing is inferred. ``delisted_on`` / ``delisted_reason`` are written only for
+confirmed terminal events; a series that merely ends early leaves them NULL and
+is reported as an unresolved ending status.
+
+Usage::
+
+    python scripts/prep_survivorship_snapshot.py
+    python scripts/prep_survivorship_snapshot.py --base backtest_snapshots/prod.sqlite --force
+"""
+
+from __future__ import annotations
+
+import argparse
+import asyncio
+import hashlib
+import json
+import shutil
+import sqlite3
+import sys
+from datetime import date, datetime, time, timedelta, timezone
+from pathlib import Path
+from typing import Any
+
+ROOT = Path(__file__).resolve().parents[1]
+if str(ROOT) not in sys.path:
+    sys.path.insert(0, str(ROOT))
+
+# Bars are cut at the first run of this many consecutive zero-volume sessions.
+ZERO_VOLUME_RUN = 5
+# Close assigned to a confirmed terminal event. Not zero: a zero price divides
+# through several return calculations, and 1 cent is a near-total loss either way.
+TERMINAL_CLOSE = 0.01
+
+DEFAULT_BASE = ROOT / "backtest_snapshots" / "prod.sqlite"
+DEFAULT_REMOVALS = ROOT / "scripts" / "data" / "sp500_removals.json"
+
+
+# ---------------------------------------------------------------------------
+# schema sync
+# ---------------------------------------------------------------------------
+
+def sync_schema(path: Path) -> dict[str, list[str]]:
+    """Add every column the models declare that the snapshot file lacks.
+
+    Nullable and untyped-default only: this brings an old snapshot up to the
+    current models without inventing data. Tables the snapshot does not have at
+    all are left alone — a snapshot is a subset of production by design.
+    """
+    from sqlalchemy.dialects import sqlite as sqlite_dialect
+
+    from app.database import Base
+    import app.models  # noqa: F401  (registers every model on Base.metadata)
+
+    dialect = sqlite_dialect.dialect()
+    added: dict[str, list[str]] = {}
+    con = sqlite3.connect(path)
+    try:
+        present = {r[0] for r in con.execute("SELECT name FROM sqlite_master WHERE type='table'")}
+        for table in Base.metadata.sorted_tables:
+            if table.name not in present:
+                continue
+            have = {r[1] for r in con.execute(f'PRAGMA table_info("{table.name}")')}
+            for column in table.columns:
+                if column.name in have:
+                    continue
+                type_sql = column.type.compile(dialect)
+                con.execute(f'ALTER TABLE "{table.name}" ADD COLUMN "{column.name}" {type_sql}')
+                added.setdefault(table.name, []).append(column.name)
+        con.commit()
+    finally:
+        con.close()
+    return added
+
+
+# ---------------------------------------------------------------------------
+# price data
+# ---------------------------------------------------------------------------
+
+def frozen_run_start(bars: list[Any]) -> int | None:
+    """Index where the first run of >= ZERO_VOLUME_RUN zero-volume bars begins.
+
+    A detector, not a cutter: the caller decides what to do with the answer, and
+    only a reviewed cutoff is allowed to shorten a series.
+    """
+    volumes = [int(b.volume) for b in bars]
+    i = 0
+    while i < len(volumes):
+        if volumes[i] == 0:
+            j = i
+            while j < len(volumes) and volumes[j] == 0:
+                j += 1
+            if j - i >= ZERO_VOLUME_RUN:
+                return i
+            i = j
+        else:
+            i += 1
+    return None
+
+
+def apply_reviewed_cutoff(bars: list[Any], cutoff: date) -> tuple[list[Any], int]:
+    """Keep bars up to and including *cutoff*. Returns (kept, dropped)."""
+    kept = [b for b in bars if b.timestamp.date() <= cutoff]
+    return kept, len(bars) - len(kept)
+
+
+async def fetch_bars(symbol: str, start: date, end: date) -> list[Any]:
+    """Daily SPLIT-adjusted bars, matching how the snapshot's existing bars were ingested."""
+    from app.config import settings
+    from app.providers.alpaca import AlpacaOHLCVProvider
+
+    if not settings.alpaca_api_key or not settings.alpaca_api_secret:
+        raise SystemExit("Alpaca credentials are not configured; cannot restore price history.")
+
+    from alpaca.data.enums import Adjustment
+    from alpaca.data.requests import StockBarsRequest
+    from alpaca.data.timeframe import TimeFrame
+
+    provider = AlpacaOHLCVProvider(settings.alpaca_api_key, settings.alpaca_api_secret)
+    alpaca_symbol = provider._to_alpaca_symbol(symbol)
+    request = StockBarsRequest(
+        symbol_or_symbols=alpaca_symbol,
+        timeframe=TimeFrame.Day,
+        start=datetime.combine(start, time.min, tzinfo=timezone.utc),
+        end=datetime.combine(end + timedelta(days=1), time.min, tzinfo=timezone.utc),
+        adjustment=Adjustment.SPLIT,
+    )
+    # 85 sequential requests are enough for Alpaca to drop one connection, and a
+    # half-built arm is worse than a slow one.
+    last_error: Exception | None = None
+    for attempt in range(4):
+        try:
+            bars = await asyncio.to_thread(provider._client.get_stock_bars, request)
+            return list(bars.data.get(alpaca_symbol, []))
+        except Exception as exc:  # noqa: BLE001 - retry any transport failure
+            last_error = exc
+            if attempt < 3:
+                await asyncio.sleep(2 * (attempt + 1))
+    raise SystemExit(f"Alpaca fetch failed for {symbol} after 4 attempts: {last_error}")
+
+
+# ---------------------------------------------------------------------------
+# snapshot writes
+# ---------------------------------------------------------------------------
+
+def snapshot_window(path: Path) -> tuple[date, date]:
+    con = sqlite3.connect(path)
+    try:
+        lo, hi = con.execute("SELECT min(date), max(date) FROM ohlcv_records").fetchone()
+    finally:
+        con.close()
+    return date.fromisoformat(lo), date.fromisoformat(hi)
+
+
+def trading_calendar(path: Path) -> list[date]:
+    """SPY sessions from the snapshot's own benchmark series."""
+    con = sqlite3.connect(path)
+    try:
+        rows = con.execute(
+            "SELECT date FROM benchmark_prices WHERE symbol = 'SPY' ORDER BY date"
+        ).fetchall()
+    finally:
+        con.close()
+    return [date.fromisoformat(r[0]) for r in rows]
+
+
+def next_session(calendar: list[date], after: date) -> date | None:
+    for d in calendar:
+        if d > after:
+            return d
+    return None
+
+
+def insert_symbol(
+    con: sqlite3.Connection,
+    symbol: str,
+    name: str,
+    bars: list[Any],
+    *,
+    delisted_on: str | None,
+    delisted_reason: str | None,
+    terminal_date: date | None,
+) -> int:
+    """Insert one restored ticker and its bars. Returns the bar count written."""
+    now = datetime.utcnow().isoformat(sep=" ")
+    cur = con.execute(
+        "INSERT INTO tickers (symbol, name, created_at, delisted_on, delisted_reason) "
+        "VALUES (?, ?, ?, ?, ?)",
+        (symbol, name, now, delisted_on, delisted_reason),
+    )
+    ticker_id = int(cur.lastrowid)
+
+    rows = [
+        (
+            ticker_id,
+            b.timestamp.date().isoformat(),
+            float(b.open),
+            float(b.high),
+            float(b.low),
+            float(b.close),
+            int(b.volume),
+            now,
+        )
+        for b in bars
+    ]
+    if terminal_date is not None:
+        rows.append((
+            ticker_id, terminal_date.isoformat(),
+            TERMINAL_CLOSE, TERMINAL_CLOSE, TERMINAL_CLOSE, TERMINAL_CLOSE, 0, now,
+        ))
+    con.executemany(
+        "INSERT INTO ohlcv_records "
+        "(ticker_id, date, open, high, low, close, volume, created_at) "
+        "VALUES (?, ?, ?, ?, ?, ?, ?, ?)",
+        rows,
+    )
+    return len(rows)
+
+
+# ---------------------------------------------------------------------------
+# driver
+# ---------------------------------------------------------------------------
+
+def file_hash(path: Path) -> str:
+    h = hashlib.sha256()
+    with path.open("rb") as fh:
+        for chunk in iter(lambda: fh.read(1 << 20), b""):
+            h.update(chunk)
+    return h.hexdigest()
+
+
+async def build(args: argparse.Namespace) -> int:
+    base = Path(args.base).resolve()
+    if not base.exists():
+        raise SystemExit(f"Base snapshot not found: {base}")
+
+    out_dir = Path(args.out_dir).resolve()
+    arm_a = out_dir / args.arm_a_name
+    arm_b = out_dir / args.arm_b_name
+    events_path = arm_b.with_name(arm_b.stem + "-terminal-events.json")
+
+    doc = json.loads(Path(args.removals).read_text(encoding="utf-8"))
+    removals = doc["removals"]
+    confirmations = {c["symbol"]: c for c in doc["terminal_confirmations"] if c.get("confirmed")}
+    cutoffs = {
+        c["symbol"]: date.fromisoformat(c["last_real_bar"])
+        for c in doc.get("reviewed_cutoffs", {}).get("cutoffs", [])
+    }
+    # A corporate action ends a listing; a market-cap removal does not, so a
+    # series that stops early without one is an ending we cannot account for.
+    ENDING_EXPLAINED = {"acquisition", "receivership", "bankruptcy", "ticker_change"}
+
+    base_hash_before = file_hash(base)
+    window_start, window_end = snapshot_window(base)
+    print(f"base     : {base}")
+    print(f"window   : {window_start} .. {window_end}")
+    print(f"removals : {len(removals)} rows, {sum(1 for r in removals if r['restore_symbol'])} restorable")
+
+    for target in (arm_a, arm_b):
+        if target.exists() and not args.force:
+            raise SystemExit(f"{target} already exists (use --force to rebuild)")
+        if target.exists():
+            target.unlink()
+        shutil.copy2(base, target)
+        added = sync_schema(target)
+        detail = ", ".join(f"{t}: {', '.join(c)}" for t, c in added.items()) or "nothing missing"
+        print(f"copied   : {target.name}  (schema sync -> {detail})")
+
+    calendar = trading_calendar(base)
+    con = sqlite3.connect(arm_b)
+    existing = {r[0].upper() for r in con.execute("SELECT symbol FROM tickers")}
+
+    restored: list[dict] = []
+    unresolved: list[dict] = []
+    terminal_events: dict[str, str] = {}
+
+    try:
+        for row in removals:
+            symbol = row["restore_symbol"]
+            if not symbol:
+                continue
+            if symbol.upper() in existing:
+                unresolved.append({"kind": "skipped", "symbol": symbol, "why": "already present in the base snapshot"})
+                continue
+
+            bars = await fetch_bars(symbol, window_start, window_end)
+            if not bars:
+                unresolved.append({"kind": "coverage", "symbol": symbol, "why": "Alpaca returned no bars in the window"})
+                print(f"  {symbol:<6} NO DATA  (unresolved coverage)")
+                continue
+
+            cutoff = cutoffs.get(symbol) or cutoffs.get(row["symbol"])
+            frozen_at = frozen_run_start(bars)
+            if cutoff is not None:
+                kept, dropped = apply_reviewed_cutoff(bars, cutoff)
+                if not kept or kept[-1].timestamp.date() != cutoff:
+                    unresolved.append({
+                        "symbol": symbol,
+                        "kind": "coverage",
+                        "why": f"reviewed cutoff {cutoff} is not a bar in the downloaded series",
+                    })
+                    print(f"  {symbol:<6} CUTOFF MISSING {cutoff}  (unresolved coverage)")
+                    continue
+            elif frozen_at is not None:
+                # Detector only. An unreviewed frozen run means the data changed
+                # under us; restoring either half would be a guess, so report it.
+                unresolved.append({
+                    "kind": "coverage",
+                    "symbol": symbol,
+                    "why": (
+                        f"unreviewed frozen run of >= {ZERO_VOLUME_RUN} zero-volume sessions "
+                        f"from {bars[frozen_at].timestamp.date()}; needs a reviewed cutoff "
+                        "before this symbol can be restored"
+                    ),
+                })
+                print(f"  {symbol:<6} FROZEN RUN at {bars[frozen_at].timestamp.date()}  (unresolved, not restored)")
+                continue
+            else:
+                kept, dropped = bars, 0
+
+            effective_last = kept[-1].timestamp.date()
+            confirmation = confirmations.get(row["symbol"]) or confirmations.get(symbol)
+            terminal_date = None
+            if confirmation is not None:
+                confirmed_last = date.fromisoformat(confirmation["last_real_bar"])
+                if effective_last != confirmed_last:
+                    unresolved.append({
+                        "symbol": symbol,
+                        "kind": "terminal_mark",
+                        "why": (
+                            f"retained final bar {effective_last} does not match the reviewed "
+                            f"last_real_bar {confirmed_last}; no terminal mark written"
+                        ),
+                    })
+                    print(f"  {symbol:<6} MARK SKIPPED  final bar {effective_last} != confirmed {confirmed_last}")
+                else:
+                    terminal_date = next_session(calendar, effective_last)
+                    if terminal_date is None or terminal_date > window_end:
+                        terminal_date = None
+                        unresolved.append({
+                            "symbol": symbol,
+                            "kind": "terminal_mark",
+                            "why": "confirmed terminal event has no trading session left inside the window",
+                        })
+                    else:
+                        terminal_events[symbol] = terminal_date.isoformat()
+
+            # Documentary only, and only where the event is confirmed. A series
+            # that merely ends early is an unresolved ending, not a delisting.
+            delisted_on = None
+            delisted_reason = None
+            if terminal_date is not None:
+                delisted_on = terminal_date.isoformat()
+                delisted_reason = row["reason"]
+            elif effective_last < window_end and row["reason"] not in ENDING_EXPLAINED:
+                unresolved.append({
+                    "kind": "ending_status",
+                    "symbol": symbol,
+                    "why": (
+                        f"price history ends {effective_last}, before the window closes on "
+                        f"{window_end}, and the source row ({row['reason']}) does not explain "
+                        "an ending; ending status unresolved and delisting fields left null"
+                    ),
+                })
+
+            written = insert_symbol(
+                con, symbol, row["name"], kept,
+                delisted_on=delisted_on, delisted_reason=delisted_reason,
+                terminal_date=terminal_date,
+            )
+            restored.append({
+                "symbol": symbol,
+                "removed_on": row["removed_on"],
+                "reason": row["reason"],
+                "bars": written,
+                "first_bar": kept[0].timestamp.date().isoformat(),
+                "effective_last_bar": effective_last.isoformat(),
+                "frozen_bars_dropped": dropped,
+                "terminal_mark": terminal_events.get(symbol),
+            })
+            mark = f"  terminal={terminal_events[symbol]} @ {TERMINAL_CLOSE}" if symbol in terminal_events else ""
+            drop = f"  dropped={dropped}" if dropped else ""
+            print(f"  {symbol:<6} bars={written:<5} {kept[0].timestamp.date()}..{effective_last}{drop}{mark}")
+
+        con.commit()
+    finally:
+        con.close()
+
+    events_path.write_text(json.dumps(terminal_events, indent=2) + "\n", encoding="utf-8")
+
+    if file_hash(base) != base_hash_before:
+        raise SystemExit(f"BASE SNAPSHOT WAS MODIFIED: {base}")
+
+    con_a = sqlite3.connect(arm_a)
+    con_b = sqlite3.connect(arm_b)
+    counts = {
+        "arm_a_tickers": con_a.execute("SELECT count(*) FROM tickers").fetchone()[0],
+        "arm_a_bars": con_a.execute("SELECT count(*) FROM ohlcv_records").fetchone()[0],
+        "arm_b_tickers": con_b.execute("SELECT count(*) FROM tickers").fetchone()[0],
+        "arm_b_bars": con_b.execute("SELECT count(*) FROM ohlcv_records").fetchone()[0],
+    }
+    con_a.close()
+    con_b.close()
+
+    by_kind: dict[str, list[dict]] = {}
+    for row in unresolved:
+        by_kind.setdefault(row.get("kind", "other"), []).append(row)
+
+    print("\nsummary")
+    print(f"  restored           : {len(restored)}")
+    print(f"  unresolved coverage: {len(by_kind.get('coverage', []))}  (symbols not restored)")
+    print(f"  unresolved endings : {len(by_kind.get('ending_status', []))}  (restored, ending status unknown)")
+    print(f"  terminal marks     : {len(terminal_events)} -> {events_path.name}")
+    print(f"  arm A              : {counts['arm_a_tickers']} tickers, {counts['arm_a_bars']} bars")
+    print(f"  arm B              : {counts['arm_b_tickers']} tickers, {counts['arm_b_bars']} bars")
+    print(f"  base unchanged     : sha256 {base_hash_before[:16]}...")
+    for kind in ("coverage", "ending_status", "terminal_mark"):
+        for row in by_kind.get(kind, []):
+            print(f"  UNRESOLVED[{kind}] {row['symbol']:<6} {row['why']}")
+
+    manifest = {
+        "generated_at": datetime.now(timezone.utc).isoformat(),
+        "base": str(base),
+        "base_sha256": base_hash_before,
+        "window": {"start": window_start.isoformat(), "end": window_end.isoformat()},
+        "zero_volume_run": ZERO_VOLUME_RUN,
+        "terminal_close": TERMINAL_CLOSE,
+        "counts": counts,
+        "restored": restored,
+        "unresolved": unresolved,
+        "terminal_events": terminal_events,
+    }
+    manifest_path = arm_b.with_name(arm_b.stem + "-prep-manifest.json")
+    manifest_path.write_text(json.dumps(manifest, indent=2) + "\n", encoding="utf-8")
+    print(f"  manifest           : {manifest_path.name}")
+    return 0
+
+
+def parse_args() -> argparse.Namespace:
+    p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+    p.add_argument("--base", default=str(DEFAULT_BASE), help="Base snapshot to copy (never modified).")
+    p.add_argument("--out-dir", default=str(DEFAULT_BASE.parent), help="Where the arm copies are written.")
+    p.add_argument("--arm-a-name", default="survivorship-a.sqlite")
+    p.add_argument("--arm-b-name", default="survivorship-b.sqlite")
+    p.add_argument("--removals", default=str(DEFAULT_REMOVALS))
+    p.add_argument("--force", action="store_true", help="Rebuild the arm files if they already exist.")
+    return p.parse_args()
+
+
+if __name__ == "__main__":
+    raise SystemExit(asyncio.run(build(parse_args())))

scripts/research_tr_rank_agreement.py

@@ -0,0 +1,596 @@
+"""Does total-return momentum change any admission or ordering decision?
+
+Step 2 of docs/data-integrity-plan.md. A screening test, deliberately cheap: if
+ranking on total return picks the same names in the same order as ranking on
+price return, nothing downstream can change and the production `tr_factor` work
+(Step 3) is unnecessary.
+
+Why this needs no schema and no ingestion change: the forward-anchored factor
+design exists because *production* ingest is incremental, so a back-adjusted
+series drifts between fetches. Here each symbol's whole history is fetched in one
+request, and within a single response the adjustment is internally consistent.
+
+Method
+------
+* One `Adjustment.SPLIT` and one `Adjustment.ALL` request per symbol plus SPY,
+  cached to sqlite so re-runs are free and reproducible. Aligned by date, never
+  zipped by index; a date present in only one adjustment is dropped and counted.
+* At every SPY session with enough history, residual 12-1 momentum is computed
+  twice with the *same* production function: the price-return series against
+  nominal SPY, and the total-return series against total-return SPY. Only bars at
+  or before the decision date are used.
+* The volatility leg is computed once per date from the SPLIT series and held
+  fixed in both arms, so the only thing that varies is momentum.
+* Both arms are percentiled with the production `_percentiles`, gated at the
+  production `min_momentum_percentile`, then ordered by `blend_strategy_rank`
+  with the symbol as a deterministic tie-break. The full admitted list is
+  compared - never truncated to `max_positions`, because a lower-ranked name can
+  be the first actionable setup when others fail gates or are already held.
+
+Decision rule (pre-registered, docs/data-integrity-plan.md)
+----------------------------------------------------------
+* no ordered-list difference on any evaluated date -> stop, Step 3 unnecessary
+* any ordered-list difference                      -> Step 3 investigates trades
+* incomplete downloads or no evaluable dates       -> inconclusive, fix coverage
+
+Usage::
+
+    python scripts/research_tr_rank_agreement.py --fetch      # populate the cache
+    python scripts/research_tr_rank_agreement.py --time-batch # project the runtime
+    python scripts/research_tr_rank_agreement.py              # full screen
+"""
+
+from __future__ import annotations
+
+import argparse
+import asyncio
+import json
+import sqlite3
+import sys
+import time
+from datetime import date, datetime, time as dtime, timedelta, timezone
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[1]
+if str(ROOT) not in sys.path:
+    sys.path.insert(0, str(ROOT))
+
+DEFAULT_SNAPSHOT = ROOT / "backtest_snapshots" / "survivorship-a.sqlite"
+DEFAULT_CACHE = ROOT / "backtest_snapshots" / "tr-rank-cache.sqlite"
+BENCHMARK = "SPY"
+
+
+# ---------------------------------------------------------------------------
+# cache
+# ---------------------------------------------------------------------------
+
+def open_cache(path: Path) -> sqlite3.Connection:
+    con = sqlite3.connect(path)
+    con.execute(
+        "CREATE TABLE IF NOT EXISTS bars ("
+        " symbol TEXT NOT NULL, date TEXT NOT NULL,"
+        " split_close REAL, all_close REAL,"
+        " PRIMARY KEY (symbol, date))"
+    )
+    con.execute(
+        "CREATE TABLE IF NOT EXISTS fetched ("
+        " symbol TEXT PRIMARY KEY, fetched_at TEXT NOT NULL,"
+        " split_bars INTEGER, all_bars INTEGER, aligned_bars INTEGER,"
+        " dropped_unmatched INTEGER)"
+    )
+    con.commit()
+    return con
+
+
+async def fetch_pair(symbol: str, start: date, end: date) -> tuple[dict, dict]:
+    """(split_closes, all_closes) keyed by date, each from a single request."""
+    from alpaca.data.enums import Adjustment
+    from alpaca.data.historical import StockHistoricalDataClient
+    from alpaca.data.requests import StockBarsRequest
+    from alpaca.data.timeframe import TimeFrame
+
+    from app.config import settings
+
+    if not settings.alpaca_api_key or not settings.alpaca_api_secret:
+        raise SystemExit("Alpaca credentials are not configured.")
+
+    client = StockHistoricalDataClient(settings.alpaca_api_key, settings.alpaca_api_secret)
+    alpaca_symbol = symbol.replace("-", ".")
+    window = (
+        datetime.combine(start, dtime.min, tzinfo=timezone.utc),
+        datetime.combine(end + timedelta(days=1), dtime.min, tzinfo=timezone.utc),
+    )
+
+    async def one(adjustment) -> dict[date, float]:
+        request = StockBarsRequest(
+            symbol_or_symbols=alpaca_symbol, timeframe=TimeFrame.Day,
+            start=window[0], end=window[1], adjustment=adjustment,
+        )
+        last: Exception | None = None
+        for attempt in range(4):
+            try:
+                bars = await asyncio.to_thread(client.get_stock_bars, request)
+                return {
+                    b.timestamp.date(): float(b.close)
+                    for b in bars.data.get(alpaca_symbol, [])
+                }
+            except Exception as exc:  # noqa: BLE001 - retry any transport failure
+                last = exc
+                if attempt < 3:
+                    await asyncio.sleep(2 * (attempt + 1))
+        raise SystemExit(f"Alpaca fetch failed for {symbol}: {last}")
+
+    return await one(Adjustment.SPLIT), await one(Adjustment.ALL)
+
+
+async def populate_cache(symbols: list[str], start: date, end: date, cache: Path) -> dict:
+    con = open_cache(cache)
+    done = {r[0] for r in con.execute("SELECT symbol FROM fetched")}
+    todo = [s for s in symbols if s not in done]
+    print(f"cache {cache.name}: {len(done)} cached, {len(todo)} to fetch")
+
+    coverage: dict[str, dict] = {}
+    for index, symbol in enumerate(todo, 1):
+        split, adjusted = await fetch_pair(symbol, start, end)
+        # Align by date. A date in only one adjustment cannot be compared and is
+        # dropped rather than paired with its neighbour.
+        common = sorted(set(split) & set(adjusted))
+        dropped = len(set(split) ^ set(adjusted))
+        con.executemany(
+            "INSERT OR REPLACE INTO bars (symbol, date, split_close, all_close) VALUES (?,?,?,?)",
+            [(symbol, d.isoformat(), split[d], adjusted[d]) for d in common],
+        )
+        con.execute(
+            "INSERT OR REPLACE INTO fetched VALUES (?,?,?,?,?,?)",
+            (symbol, datetime.now(timezone.utc).isoformat(), len(split), len(adjusted),
+             len(common), dropped),
+        )
+        con.commit()
+        if dropped:
+            print(f"  {symbol:<6} aligned={len(common)} dropped_unmatched={dropped}")
+        if index % 50 == 0:
+            print(f"  … {index}/{len(todo)}")
+
+    for row in con.execute("SELECT symbol, split_bars, all_bars, aligned_bars, dropped_unmatched FROM fetched"):
+        coverage[row[0]] = {
+            "split_bars": row[1], "all_bars": row[2],
+            "aligned_bars": row[3], "dropped_unmatched": row[4],
+        }
+    con.close()
+    return coverage
+
+
+def load_series(cache: Path, symbols: list[str]) -> dict[str, tuple[list[date], list[float], list[float]]]:
+    con = open_cache(cache)
+    out: dict[str, tuple[list[date], list[float], list[float]]] = {}
+    for symbol in symbols:
+        rows = con.execute(
+            "SELECT date, split_close, all_close FROM bars WHERE symbol = ? ORDER BY date",
+            (symbol,),
+        ).fetchall()
+        if not rows:
+            continue
+        out[symbol] = (
+            [date.fromisoformat(r[0]) for r in rows],
+            [float(r[1]) for r in rows],
+            [float(r[2]) for r in rows],
+        )
+    con.close()
+    return out
+
+
+# ---------------------------------------------------------------------------
+# screen
+# ---------------------------------------------------------------------------
+
+def snapshot_symbols(snapshot: Path) -> tuple[list[str], list[str], date, date]:
+    """(symbols with bars, symbols with none, window start, window end).
+
+    A ticker row with no bars in the snapshot cannot rank in either arm, so it is
+    inert rather than missing. Keeping the two apart is what stops a failed
+    download from being quietly reported as agreement.
+    """
+    con = sqlite3.connect(snapshot)
+    try:
+        rows = con.execute(
+            "SELECT t.symbol, count(o.id) FROM tickers t "
+            "LEFT JOIN ohlcv_records o ON o.ticker_id = t.id GROUP BY t.symbol"
+        ).fetchall()
+        lo, hi = con.execute("SELECT min(date), max(date) FROM ohlcv_records").fetchone()
+    finally:
+        con.close()
+    with_bars = sorted(r[0].upper() for r in rows if r[1])
+    without_bars = sorted(r[0].upper() for r in rows if not r[1])
+    return with_bars, without_bars, date.fromisoformat(lo), date.fromisoformat(hi)
+
+
+def ordered_list(
+    momentum: dict[str, float],
+    vol_percentiles: dict[str, float],
+    cutoff: float,
+) -> list[str]:
+    """Admitted set at the production cutoff, ordered by the production blend.
+
+    Symbols are fed to ``_percentiles`` in sorted order in both arms so that ties
+    resolve identically and cannot manufacture a difference.
+    """
+    from app.services.momentum_service import _percentiles, blend_strategy_rank
+
+    percentiles = _percentiles({s: momentum[s] for s in sorted(momentum)})
+    admitted = [s for s, p in percentiles.items() if p >= cutoff]
+    scored = {
+        s: blend_strategy_rank(percentiles[s], vol_percentiles.get(s))
+        for s in admitted
+    }
+    return sorted(admitted, key=lambda s: (-(scored[s] or 0.0), s))
+
+
+def mechanism_check(
+    series: dict[str, tuple[list[date], list[float], list[float]]],
+    sample_dates: list[date],
+) -> dict:
+    """Is the rank shift explained by dividend yield?
+
+    For each sampled decision date, correlate a name's trailing 12-month implied
+    yield — read straight off the ALL/SPLIT ratio, which is what separates the two
+    series — against how far its momentum percentile moves between the arms. A
+    strong positive rank correlation means the difference *is* the dividend, not
+    numerical noise.
+    """
+    from app.services.momentum_service import _MOM_LOOKBACK, _percentiles, compute_residual_12_1_momentum
+
+    bench_dates, bench_split, bench_all = series[BENCHMARK]
+    bench_price = dict(zip(bench_dates, bench_split))
+    bench_tr = dict(zip(bench_dates, bench_all))
+    symbols = sorted(s for s in series if s != BENCHMARK)
+    index_of = {s: {d: i for i, d in enumerate(series[s][0])} for s in symbols}
+
+    correlations: list[float] = []
+    used_dates: list[str] = []
+    for as_of in sample_dates:
+        price_mom: dict[str, float] = {}
+        tr_mom: dict[str, float] = {}
+        yields: dict[str, float] = {}
+        for symbol in symbols:
+            idx = index_of[symbol].get(as_of)
+            if idx is None or idx < _MOM_LOOKBACK:
+                continue
+            dates, split_closes, all_closes = series[symbol]
+            wd, ws, wa = dates[: idx + 1], split_closes[: idx + 1], all_closes[: idx + 1]
+            p = compute_residual_12_1_momentum(wd, ws, bench_price)
+            t = compute_residual_12_1_momentum(wd, wa, bench_tr)
+            if p is None or t is None:
+                continue
+            price_mom[symbol] = p
+            tr_mom[symbol] = t
+            # Ratio drift over the formation window is the accumulated dividend.
+            r_now = wa[-1] / ws[-1] if ws[-1] else None
+            r_then = wa[-_MOM_LOOKBACK] / ws[-_MOM_LOOKBACK] if ws[-_MOM_LOOKBACK] else None
+            if r_now and r_then:
+                yields[symbol] = r_now / r_then - 1.0
+
+        common = sorted(set(price_mom) & set(yields))
+        if len(common) < 30:
+            continue
+        pp = _percentiles({s: price_mom[s] for s in common})
+        tp = _percentiles({s: tr_mom[s] for s in common})
+        shift = {s: tp[s] - pp[s] for s in common}
+        correlations.append(_spearman([yields[s] for s in common], [shift[s] for s in common]))
+        used_dates.append(as_of.isoformat())
+
+    return {
+        "what_it_measures": (
+            "Spearman correlation, per sampled date, between a name's trailing 12-month "
+            "implied yield (from the ALL/SPLIT ratio over the formation window) and the "
+            "change in its momentum percentile between the two arms."
+        ),
+        "caveat": (
+            "Both the stock series and the benchmark leg change between arms, so this is "
+            "strong evidence consistent with a dividend effect - it does not isolate the "
+            "stock's own yield from the beta-weighted benchmark yield."
+        ),
+        "formation_window_bars": _MOM_LOOKBACK,
+        "min_symbols_per_date": 30,
+        "sampled_dates": len(correlations),
+        "sample_date_list": used_dates,
+        "mean_spearman_yield_vs_percentile_shift": (
+            round(sum(correlations) / len(correlations), 4) if correlations else None
+        ),
+        "min": round(min(correlations), 4) if correlations else None,
+        "max": round(max(correlations), 4) if correlations else None,
+        "per_date_spearman": [round(c, 4) for c in correlations],
+    }
+
+
+def _spearman(x: list[float], y: list[float]) -> float:
+    def rank(v: list[float]) -> list[float]:
+        order = sorted(range(len(v)), key=lambda i: v[i])
+        out = [0.0] * len(v)
+        for position, i in enumerate(order):
+            out[i] = float(position)
+        return out
+
+    rx, ry = rank(x), rank(y)
+    n = len(rx)
+    mx, my = sum(rx) / n, sum(ry) / n
+    num = sum((rx[i] - mx) * (ry[i] - my) for i in range(n))
+    dx = sum((rx[i] - mx) ** 2 for i in range(n)) ** 0.5
+    dy = sum((ry[i] - my) ** 2 for i in range(n)) ** 0.5
+    return num / (dx * dy) if dx and dy else 0.0
+
+
+def disagreement(a: list[str], b: list[str]) -> float:
+    sa, sb = set(a), set(b)
+    union = sa | sb
+    if not union:
+        return 0.0
+    return len(sa ^ sb) / len(union)
+
+
+def run_screen(
+    series: dict[str, tuple[list[date], list[float], list[float]]],
+    cutoff: float,
+    *,
+    limit_dates: int | None = None,
+    verbose: bool = True,
+) -> dict:
+    from app.services.momentum_service import (
+        _MOM_LOOKBACK,
+        compute_realized_vol_6m,
+        compute_residual_12_1_momentum,
+        _percentiles,
+    )
+
+    bench_dates, bench_split, bench_all = series[BENCHMARK]
+    bench_price = dict(zip(bench_dates, bench_split))
+    bench_tr = dict(zip(bench_dates, bench_all))
+
+    symbols = sorted(s for s in series if s != BENCHMARK)
+    index_of: dict[str, dict[date, int]] = {
+        s: {d: i for i, d in enumerate(series[s][0])} for s in symbols
+    }
+
+    evaluated = 0
+    skipped_no_history = 0
+    changed_sets = 0
+    changed_orders = 0
+    disagreements: list[float] = []
+    # Counted directly rather than derived from the Jaccard ratio: a swap moves
+    # two names (one out, one in), so the symmetric difference is twice the
+    # replacement count and the two are easy to conflate.
+    replacements: list[int] = []
+    only_price_counts: list[int] = []
+    only_tr_counts: list[int] = []
+    admitted_sizes: list[int] = []
+    per_year: dict[int, dict[str, int]] = {}
+    examples: list[dict] = []
+
+    decision_dates = [d for d in bench_dates]
+    if limit_dates:
+        decision_dates = decision_dates[-limit_dates:]
+
+    for as_of in decision_dates:
+        price_mom: dict[str, float] = {}
+        tr_mom: dict[str, float] = {}
+        vol_values: dict[str, float] = {}
+
+        for symbol in symbols:
+            idx = index_of[symbol].get(as_of)
+            if idx is None or idx < _MOM_LOOKBACK:
+                continue
+            dates, split_closes, all_closes = series[symbol]
+            window_dates = dates[: idx + 1]
+            window_split = split_closes[: idx + 1]
+            window_all = all_closes[: idx + 1]
+
+            price_value = compute_residual_12_1_momentum(window_dates, window_split, bench_price)
+            tr_value = compute_residual_12_1_momentum(window_dates, window_all, bench_tr)
+            # Equal eligibility: a name enters both arms or neither.
+            if price_value is None or tr_value is None:
+                continue
+            price_mom[symbol] = price_value
+            tr_mom[symbol] = tr_value
+
+            vol = compute_realized_vol_6m(window_split)
+            if vol is not None:
+                vol_values[symbol] = vol
+
+        if not price_mom:
+            skipped_no_history += 1
+            continue
+
+        evaluated += 1
+        vol_percentiles = _percentiles({s: vol_values[s] for s in sorted(vol_values)})
+        price_list = ordered_list(price_mom, vol_percentiles, cutoff)
+        tr_list = ordered_list(tr_mom, vol_percentiles, cutoff)
+
+        year = per_year.setdefault(as_of.year, {"dates": 0, "changed_sets": 0, "changed_orders": 0})
+        year["dates"] += 1
+
+        set_price, set_tr = set(price_list), set(tr_list)
+        only_price = len(set_price - set_tr)
+        only_tr = len(set_tr - set_price)
+        only_price_counts.append(only_price)
+        only_tr_counts.append(only_tr)
+        admitted_sizes.append(len(price_list))
+        # A replacement is one name leaving paired with one arriving; when the
+        # two arms admit different counts, the surplus is not a replacement.
+        replacements.append(min(only_price, only_tr))
+
+        same_set = set_price == set_tr
+        same_order = price_list == tr_list
+        if not same_set:
+            changed_sets += 1
+            year["changed_sets"] += 1
+        if not same_order:
+            changed_orders += 1
+            year["changed_orders"] += 1
+            if len(examples) < 10:
+                only_price = [s for s in price_list if s not in set(tr_list)]
+                only_tr = [s for s in tr_list if s not in set(price_list)]
+                examples.append({
+                    "date": as_of.isoformat(),
+                    "admitted_price_return": len(price_list),
+                    "admitted_total_return": len(tr_list),
+                    "only_in_price_return": only_price[:10],
+                    "only_in_total_return": only_tr[:10],
+                    "order_changed_without_membership_change": same_set,
+                    "first_10_price_return": price_list[:10],
+                    "first_10_total_return": tr_list[:10],
+                })
+        disagreements.append(disagreement(price_list, tr_list))
+
+        if verbose and evaluated % 100 == 0:
+            print(f"  … {evaluated} dates evaluated, {changed_orders} with a different ordered list")
+
+    return {
+        "evaluated_dates": evaluated,
+        "skipped_dates_no_history": skipped_no_history,
+        "dates_with_different_admitted_set": changed_sets,
+        "dates_with_different_ordered_list": changed_orders,
+        "mean_set_disagreement": (sum(disagreements) / len(disagreements)) if disagreements else 0.0,
+        "max_set_disagreement": max(disagreements) if disagreements else 0.0,
+        "mean_admitted_per_arm": round(sum(admitted_sizes) / len(admitted_sizes), 2) if admitted_sizes else 0.0,
+        "mean_names_only_in_price_return": round(sum(only_price_counts) / len(only_price_counts), 2) if only_price_counts else 0.0,
+        "mean_names_only_in_total_return": round(sum(only_tr_counts) / len(only_tr_counts), 2) if only_tr_counts else 0.0,
+        "mean_replacements_per_date": round(sum(replacements) / len(replacements), 2) if replacements else 0.0,
+        "max_replacements_per_date": max(replacements) if replacements else 0,
+        "mean_symmetric_difference": round(
+            (sum(only_price_counts) + sum(only_tr_counts)) / len(only_price_counts), 2
+        ) if only_price_counts else 0.0,
+        "per_year": {
+            str(y): v for y, v in sorted(per_year.items())
+        },
+        "examples": examples,
+    }
+
+
+# ---------------------------------------------------------------------------
+# driver
+# ---------------------------------------------------------------------------
+
+async def main(args: argparse.Namespace) -> int:
+    snapshot = Path(args.snapshot)
+    cache = Path(args.cache)
+    symbols, inert, window_start, window_end = snapshot_symbols(snapshot)
+    universe = symbols + [BENCHMARK]
+
+    print(f"snapshot : {snapshot.name}  ({len(symbols)} symbols with bars"
+          + (f", {len(inert)} inert: {', '.join(inert)}" if inert else "") + ")")
+    print(f"window   : {window_start} .. {window_end}  (clipped to the snapshot)")
+
+    if args.fetch:
+        await populate_cache(universe, window_start, window_end, cache)
+        return 0
+
+    series = load_series(cache, universe)
+    missing = [s for s in universe if s not in series]
+    if missing:
+        raise SystemExit(
+            f"{len(missing)} symbols have bars in the snapshot but none in the cache "
+            f"({', '.join(missing[:8])}). Run with --fetch first; an incomplete download "
+            "makes the screen inconclusive and must never be read as agreement."
+        )
+
+    from app.services.admin_service import ACTIVATION_DEFAULTS
+
+    cutoff = float(ACTIVATION_DEFAULTS["min_momentum_percentile"])
+    print(f"cutoff   : {cutoff} (production min_momentum_percentile)")
+
+    if args.mechanism:
+        bench_dates = load_series(cache, [BENCHMARK])[BENCHMARK][0]
+        sample = bench_dates[::args.mechanism]
+        print(f"mechanism check over {len(sample)} sampled dates ...")
+        payload = {
+            "generated_at": datetime.now(timezone.utc).isoformat(),
+            "snapshot": str(snapshot),
+            "cache": str(cache),
+            "window": {"start": window_start.isoformat(), "end": window_end.isoformat()},
+            "universe_size": len(symbols),
+            "sample_every_n_sessions": args.mechanism,
+            "result": mechanism_check(series, sample),
+        }
+        out = Path(args.mechanism_out)
+        out.parent.mkdir(parents=True, exist_ok=True)
+        out.write_text(json.dumps(payload, indent=2) + "\n", encoding="utf-8")
+        r = payload["result"]
+        print(f"  sampled dates : {r['sampled_dates']}")
+        print(f"  mean Spearman : {r['mean_spearman_yield_vs_percentile_shift']} "
+              f"(min {r['min']}, max {r['max']})")
+        print(f"  report        : {out}")
+        return 0
+
+    if args.time_batch:
+        n = args.time_batch
+        started = time.perf_counter()
+        result = run_screen(series, cutoff, limit_dates=n, verbose=False)
+        elapsed = time.perf_counter() - started
+        total_dates = len(series[BENCHMARK][0])
+        per_date = elapsed / max(result["evaluated_dates"], 1)
+        print(f"\ntimed {result['evaluated_dates']} evaluated dates in {elapsed:.1f}s "
+              f"({per_date:.2f}s/date)")
+        print(f"projected full run over {total_dates} sessions: "
+              f"{per_date * total_dates / 60:.1f} min")
+        return 0
+
+    started = time.perf_counter()
+    result = run_screen(series, cutoff)
+    result["elapsed_seconds"] = round(time.perf_counter() - started, 1)
+
+    con = open_cache(cache)
+    coverage = {
+        r[0]: {"split_bars": r[1], "all_bars": r[2], "aligned_bars": r[3], "dropped_unmatched": r[4]}
+        for r in con.execute("SELECT symbol, split_bars, all_bars, aligned_bars, dropped_unmatched FROM fetched")
+    }
+    con.close()
+
+    verdict = (
+        "stop - no admission or ordering difference in this window"
+        if result["dates_with_different_ordered_list"] == 0
+        else "proceed to Step 3 - ordered lists differ"
+    )
+    payload = {
+        "generated_at": datetime.now(timezone.utc).isoformat(),
+        "snapshot": str(snapshot),
+        "cache": str(cache),
+        "window": {"start": window_start.isoformat(), "end": window_end.isoformat()},
+        "universe_size": len(symbols),
+        "inert_snapshot_symbols": inert,
+        "cutoff": cutoff,
+        "verdict": verdict,
+        "result": result,
+        "coverage": {
+            "symbols_cached": len(coverage),
+            "symbols_with_unmatched_dates": sum(1 for c in coverage.values() if c["dropped_unmatched"]),
+            "total_unmatched_dates": sum(c["dropped_unmatched"] for c in coverage.values()),
+        },
+    }
+    out = Path(args.out)
+    out.parent.mkdir(parents=True, exist_ok=True)
+    out.write_text(json.dumps(payload, indent=2) + "\n", encoding="utf-8")
+
+    print(f"\nevaluated dates                  : {result['evaluated_dates']}")
+    print(f"dates with different admitted set: {result['dates_with_different_admitted_set']}")
+    print(f"dates with different ordered list: {result['dates_with_different_ordered_list']}")
+    print(f"mean / max set disagreement      : {result['mean_set_disagreement']:.4f} / {result['max_set_disagreement']:.4f}")
+    print(f"verdict                          : {verdict}")
+    print(f"report                           : {out}")
+    return 0
+
+
+def parse_args() -> argparse.Namespace:
+    p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+    p.add_argument("--snapshot", default=str(DEFAULT_SNAPSHOT), help="Universe and date bounds come from this snapshot.")
+    p.add_argument("--cache", default=str(DEFAULT_CACHE))
+    p.add_argument("--fetch", action="store_true", help="Populate the cache and exit.")
+    p.add_argument("--time-batch", type=int, default=None, metavar="N",
+                   help="Time the screen over the last N sessions and project the full runtime.")
+    p.add_argument("--mechanism", type=int, default=None, metavar="EVERY_N",
+                   help="Correlate trailing dividend yield against the percentile shift on every Nth session.")
+    p.add_argument("--out", default="reports/tr-rank-agreement.json")
+    p.add_argument("--mechanism-out", default="reports/tr-rank-agreement-mechanism.json")
+    return p.parse_args()
+
+
+if __name__ == "__main__":
+    raise SystemExit(asyncio.run(main(parse_args())))

scripts/run_backtest_snapshot.py

@@ -72,9 +72,29 @@ def _parse_args() -> argparse.Namespace:
         default=None,
         help="Add a disjoint train/test portfolio report split at YYYY-MM-DD.",
     )
+    parser.add_argument(
+        "--terminal-events",
+        default=None,
+        help=(
+            "JSON map of {symbol: YYYY-MM-DD} marking symbols whose final bar is a "
+            "research terminal mark, written by prep_survivorship_snapshot.py. Those "
+            "symbols are replayed through their last real bar instead of stopping 30 "
+            "bars early; every other symbol is unaffected."
+        ),
+    )
     return parser.parse_args()
 
 
+def _load_terminal_events(path: str | None) -> dict[str, date]:
+    """Parse the terminal-event map. Explicit input only — never inferred."""
+    if not path:
+        return {}
+    raw = json.loads(Path(path).read_text(encoding="utf-8"))
+    if not isinstance(raw, dict):
+        raise SystemExit(f"Terminal events file must be a JSON object: {path}")
+    return {str(sym).upper(): date.fromisoformat(str(value)) for sym, value in raw.items()}
+
+
 def _default_output_path(cadence: str) -> Path:
     stamp = datetime.now().strftime("%Y%m%d-%H%M%S")
     return Path("reports") / f"backtest-{cadence}-{stamp}.json"
@@ -190,6 +210,10 @@ async def _main() -> None:
             raise SystemExit("--holdout-split must use YYYY-MM-DD") from exc
         os.environ["BACKTEST_HOLDOUT_SPLIT"] = args.holdout_split
 
+    terminal_events = _load_terminal_events(args.terminal_events)
+    if terminal_events:
+        print(f"terminal events: {len(terminal_events)} symbols ({', '.join(sorted(terminal_events))})")
+
     from app.config import settings
     from app.services.backtest_service import run_backtest
 
@@ -222,6 +246,7 @@ async def _main() -> None:
                 progress_cb=progress,
                 target_model=args.target_model,
                 cadence=args.cadence,
+                terminal_events=terminal_events,
             )
     finally:
         await engine.dispose()

scripts/run_total_return_ab.py

@@ -0,0 +1,544 @@
+"""Paired portfolio replay: does the momentum basis change actual trades?
+
+Step 3a of docs/data-integrity-plan.md. The Step 2 screen showed the ranking
+changes every day; this measures whether that reaches admissions and P&L. No
+schema, no ingestion — bars come from the survivorship-a snapshot and the two
+adjusted series from the Step 2 cache.
+
+What is held identical between the arms
+---------------------------------------
+* **Setup candidates are generated once** and reused by both books. Entry, stop,
+  target and R:R never differ. Only momentum and what depends on it —
+  the residual/activation percentiles, `qualified`, and the 80/20 blend used for
+  ordering — are recomputed per arm.
+* **Both arms read the same cache**: control uses SPLIT stock against SPLIT SPY,
+  treatment uses ALL stock against ALL SPY. Neither reads a freshly fetched
+  series, so a refreshed-vs-snapshot difference cannot be mistaken for a basis
+  difference.
+* **Execution, volatility, gates, capacity, exits**: snapshot nominal bars in both
+  arms; the 6-month volatility leg comes from candidate generation and is never
+  recomputed; non-momentum floors, `max_positions`, risk per trade and the exit
+  policy are the production values for both.
+
+Each book then keeps its **own** holdings: a divergent admission changes which
+slots are occupied later, and that path dependence is part of the measurement.
+
+Reading the output
+------------------
+Different trades establish that the basis is production-relevant. They do not
+authorise a rollout — that is a separate non-inferiority decision taken on the
+paired numbers.
+
+Usage::
+
+    python scripts/run_total_return_ab.py --build-candidates   # one-time, cached
+    python scripts/run_total_return_ab.py
+"""
+
+from __future__ import annotations
+
+import argparse
+import asyncio
+import json
+import pickle
+import sqlite3
+import sys
+import time
+from concurrent.futures import ProcessPoolExecutor
+from datetime import date, datetime, timezone
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[1]
+if str(ROOT) not in sys.path:
+    sys.path.insert(0, str(ROOT))
+
+DEFAULT_SNAPSHOT = ROOT / "backtest_snapshots" / "survivorship-a.sqlite"
+DEFAULT_CACHE = ROOT / "backtest_snapshots" / "tr-rank-cache.sqlite"
+DEFAULT_CANDIDATES = ROOT / "backtest_snapshots" / "tr-ab-candidates.pkl"
+BENCHMARK = "SPY"
+
+
+def _sqlite_url(path: Path) -> str:
+    return f"sqlite+aiosqlite:///{path.resolve().as_posix()}"
+
+
+# ---------------------------------------------------------------------------
+# candidates — generated once, reused by both arms
+# ---------------------------------------------------------------------------
+
+async def build_candidates(snapshot: Path, out: Path, cadence: str, workers: int) -> None:
+    from sqlalchemy import select
+    from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker, create_async_engine
+
+    from app.models.ticker import Ticker
+    from app.services import backtest_service as bt
+    from app.services.admin_service import get_activation_config
+    from app.services.recommendation_service import get_recommendation_config
+
+    engine = create_async_engine(_sqlite_url(snapshot), pool_pre_ping=True)
+    Session = async_sessionmaker(engine, class_=AsyncSession, expire_on_commit=False)
+
+    candidates: list[dict] = []
+    prices: dict[str, tuple] = {}
+    async with Session() as db:
+        config = await get_recommendation_config(db)
+        activation = await get_activation_config(db)
+        symbols = list((await db.execute(select(Ticker.symbol).order_by(Ticker.symbol))).scalars())
+        benchmark = await bt._load_benchmark_closes_for_backtest(db, refresh=False)
+        print(f"generating candidates for {len(symbols)} symbols (cadence={cadence}) …")
+
+        columns_by_symbol: dict[str, tuple] = {}
+        for symbol in symbols:
+            cols = await bt._fetch_columns(db, symbol)
+            if cols is not None:
+                columns_by_symbol[symbol] = cols
+    await engine.dispose()
+
+    started = time.perf_counter()
+    ctx = bt._mp_context()
+    pool = None
+    if ctx is not None and workers > 1:
+        try:
+            pool = ProcessPoolExecutor(max_workers=workers, mp_context=ctx)
+        except Exception:  # noqa: BLE001 - fall back to in-process replay
+            pool = None
+
+    items = list(columns_by_symbol.items())
+    if pool is not None:
+        with pool:
+            futures = {
+                symbol: pool.submit(
+                    bt._replay_and_signals, symbol, cols, config, activation,
+                    benchmark, bt.PRODUCTION_GTL_TARGET_MODEL, cadence, False, None,
+                )
+                for symbol, cols in items
+            }
+            for index, (symbol, future) in enumerate(futures.items(), 1):
+                got, _ = future.result()
+                candidates.extend(got)
+                prices[symbol] = columns_by_symbol[symbol]
+                if index % 100 == 0:
+                    print(f"  … {index}/{len(items)}")
+    else:
+        for index, (symbol, cols) in enumerate(items, 1):
+            got, _ = bt._replay_and_signals(
+                symbol, cols, config, activation, benchmark,
+                bt.PRODUCTION_GTL_TARGET_MODEL, cadence, False, None,
+            )
+            candidates.extend(got)
+            prices[symbol] = cols
+            if index % 100 == 0:
+                print(f"  … {index}/{len(items)}")
+
+    payload = {
+        "generated_at": datetime.now(timezone.utc).isoformat(),
+        "snapshot": str(snapshot),
+        "cadence": cadence,
+        "config": config,
+        "activation": activation,
+        "candidates": candidates,
+        "prices": prices,
+        "benchmark": benchmark,
+    }
+    out.write_bytes(pickle.dumps(payload))
+    print(f"{len(candidates)} candidates over {len(prices)} symbols "
+          f"in {time.perf_counter() - started:.0f}s -> {out.name}")
+
+
+# ---------------------------------------------------------------------------
+# momentum, recomputed per arm from the cache
+# ---------------------------------------------------------------------------
+
+def load_cache_series(cache: Path) -> dict[str, tuple[list[date], list[float], list[float]]]:
+    con = sqlite3.connect(cache)
+    try:
+        rows = con.execute("SELECT symbol, date, split_close, all_close FROM bars ORDER BY symbol, date")
+        out: dict[str, tuple[list, list, list]] = {}
+        for symbol, day, split_close, all_close in rows:
+            dates, splits, alls = out.setdefault(symbol, ([], [], []))
+            dates.append(date.fromisoformat(day))
+            splits.append(float(split_close))
+            alls.append(float(all_close))
+    finally:
+        con.close()
+    return out
+
+
+def momentum_by_arm(
+    series: dict[str, tuple[list[date], list[float], list[float]]],
+    signal_dates: dict[str, set[date]],
+    *,
+    total_return: bool,
+) -> dict[tuple[str, date], float]:
+    """Residual 12-1 momentum at every (symbol, signal date) for one arm.
+
+    Control reads the SPLIT series against SPLIT SPY; treatment reads ALL against
+    ALL SPY. Both use the unmodified production function and only bars at or
+    before the signal date.
+    """
+    from app.services.momentum_service import _MOM_LOOKBACK, compute_residual_12_1_momentum
+
+    bench_dates, bench_split, bench_all = series[BENCHMARK]
+    benchmark = dict(zip(bench_dates, bench_all if total_return else bench_split))
+
+    out: dict[tuple[str, date], float] = {}
+    for symbol, wanted in signal_dates.items():
+        entry = series.get(symbol)
+        if entry is None:
+            continue
+        dates, splits, alls = entry
+        closes = alls if total_return else splits
+        index_of = {d: i for i, d in enumerate(dates)}
+        for as_of in wanted:
+            idx = index_of.get(as_of)
+            if idx is None or idx < _MOM_LOOKBACK:
+                continue
+            value = compute_residual_12_1_momentum(
+                dates[: idx + 1], closes[: idx + 1], benchmark
+            )
+            if value is not None:
+                out[(symbol, as_of)] = value
+    return out
+
+
+def apply_arm(candidates: list[dict], momentum: dict[tuple[str, date], float], min_pct: float) -> list[dict]:
+    """Rebuild only the momentum-dependent fields, using the production pipeline."""
+    from app.services import backtest_service as bt
+
+    rebuilt = []
+    for cand in candidates:
+        clone = dict(cand)
+        key = (clone["symbol"], date.fromisoformat(clone["date"]))
+        clone["residual_momentum"] = momentum.get(key)
+        rebuilt.append(clone)
+
+    bt._assign_momentum_percentiles(rebuilt)
+    bt._assign_residual_momentum_percentiles(rebuilt)
+    bt._assign_low_volatility_percentiles(rebuilt)   # vol_6m untouched, so identical in both arms
+    bt._assign_activation_momentum_percentiles(rebuilt)
+    bt._assign_residual_low_vol_blend(rebuilt)
+    bt._assign_residual_high_vol_blend(rebuilt)
+    for cand in rebuilt:
+        cand["qualified"] = bt._momentum_qualifies(cand, min_pct)
+    return rebuilt
+
+
+def simulate(candidates: list[dict], prices: dict, benchmark: dict, cadence: str) -> dict:
+    """Run the production book: live gate, 80/20 ordering, ATR trail, gate reset."""
+    from app.services import backtest_service as bt
+
+    strategy = next(s for s in bt.PORTFOLIO_MONITOR_STRATEGIES if s.get("is_production"))
+    entry_cfg = bt._entry_variant_config(str(strategy["entry_variant"]))
+    ranking_key = str(strategy.get("ranking_key") or entry_cfg.get("ranking_key"))
+
+    reentry_fn = bt._make_gate_reset_reentry_fn(
+        candidates, prices, cadence=cadence, qualified_fn=None, ranking_key=ranking_key,
+    )
+    return bt._simulate_portfolio(
+        candidates,
+        prices,
+        benchmark,
+        str(strategy["exit_policy"]),
+        max(bt.TIME_EXIT_DAYS),
+        qualified_fn=None,                       # live qualification, as production
+        ranking_key=ranking_key,
+        max_positions=int(entry_cfg["max_positions"]),
+        risk_per_trade=float(entry_cfg["risk_per_trade"]),
+        atr_trail_multiplier=bt.ATR_TRAIL_MULTIPLIER,
+        post_stop_reentry_fn=reentry_fn,
+        include_curve=True,
+        include_trades=True,
+    )
+
+
+# ---------------------------------------------------------------------------
+# comparison
+# ---------------------------------------------------------------------------
+
+def entry_diff(control: dict, treatment: dict) -> dict:
+    def entries(sim: dict) -> set[tuple[str, str]]:
+        return {(t["entry_date"], t["symbol"]) for t in sim.get("trade_details", [])}
+
+    a, b = entries(control), entries(treatment)
+    only_a = sorted(a - b)
+    only_b = sorted(b - a)
+    return {
+        "shared_entries": len(a & b),
+        "only_control": [{"entry_date": d, "symbol": s} for d, s in only_a],
+        "only_treatment": [{"entry_date": d, "symbol": s} for d, s in only_b],
+        "only_control_count": len(only_a),
+        "only_treatment_count": len(only_b),
+    }
+
+
+def yearly_comparison(control: dict, treatment: dict) -> dict:
+    ca = {row["year"]: row for row in (control.get("yearly_returns") or [])}
+    tb = {row["year"]: row for row in (treatment.get("yearly_returns") or [])}
+    years = sorted(set(ca) | set(tb))
+    rows = []
+    better = worse = same = 0
+    for year in years:
+        c = (ca.get(year) or {}).get("return_pct")
+        t = (tb.get(year) or {}).get("return_pct")
+        delta = None
+        if c is not None and t is not None:
+            delta = round(t - c, 2)
+            if delta > 0:
+                better += 1
+            elif delta < 0:
+                worse += 1
+            else:
+                same += 1
+        rows.append({"year": year, "control_pct": c, "treatment_pct": t, "delta_pp": delta})
+    return {
+        "rows": rows,
+        "years_treatment_better": better,
+        "years_treatment_worse": worse,
+        "years_equal": same,
+    }
+
+
+HEADLINE = ("trades", "total_return_pct", "cagr_pct", "max_drawdown_pct", "sharpe", "win_rate")
+
+# Pre-specified before the diagnostic was run, and reported for BOTH block
+# lengths rather than whichever reads better.
+BOOTSTRAP_SEED = 20260907
+BOOTSTRAP_REPLICATES = 5000
+BOOTSTRAP_BLOCKS = (21, 63)
+TRADING_DAYS = 252
+
+
+def daily_returns(curve: list[dict]) -> tuple[list[str], list[float]]:
+    """Daily returns from an equity curve, with their dates."""
+    dates, rets = [], []
+    for previous, current in zip(curve, curve[1:]):
+        prior = float(previous["equity"])
+        if prior > 0:
+            dates.append(current["date"])
+            rets.append(float(current["equity"]) / prior - 1.0)
+    return dates, rets
+
+
+def _path_stats(returns) -> tuple[float, float, float]:
+    """(CAGR %, Sharpe, max drawdown %) for one resampled return path."""
+    import math
+
+    n = len(returns)
+    equity = 1.0
+    peak = 1.0
+    max_dd = 0.0
+    total = 0.0
+    total_sq = 0.0
+    for r in returns:
+        equity *= 1.0 + r
+        peak = max(peak, equity)
+        if peak > 0:
+            max_dd = max(max_dd, 1.0 - equity / peak)
+        total += r
+        total_sq += r * r
+    mean = total / n
+    var = max(total_sq / n - mean * mean, 0.0)
+    sd = math.sqrt(var)
+    cagr = (equity ** (TRADING_DAYS / n) - 1.0) * 100.0 if equity > 0 else -100.0
+    sharpe = (mean / sd * math.sqrt(TRADING_DAYS)) if sd > 0 else 0.0
+    return cagr, sharpe, max_dd * 100.0
+
+
+def paired_block_bootstrap(
+    control_returns: list[float],
+    treatment_returns: list[float],
+    *,
+    block: int,
+    replicates: int,
+    seed: int,
+) -> dict:
+    """Resample identical date blocks for both arms and difference the results.
+
+    The arms share dates and most positions, so their difference is what needs an
+    interval — not each arm separately. Drawing the *same* block indices for both
+    preserves the pairing and whatever cross-arm correlation exists on those days.
+    Moving blocks retain some of the serial dependence that an i.i.d. resample
+    would destroy; they do not make this out-of-sample.
+    """
+    import random
+    import statistics
+
+    rng = random.Random(seed)
+    n = len(control_returns)
+    n_blocks = -(-n // block)          # ceil
+    max_start = n - block
+
+    deltas = {"cagr_pct": [], "sharpe": [], "max_drawdown_pct": []}
+    for _ in range(replicates):
+        starts = [rng.randint(0, max_start) for _ in range(n_blocks)]
+        idx = [i for s in starts for i in range(s, s + block)][:n]
+        c = _path_stats([control_returns[i] for i in idx])
+        t = _path_stats([treatment_returns[i] for i in idx])
+        deltas["cagr_pct"].append(t[0] - c[0])
+        deltas["sharpe"].append(t[1] - c[1])
+        deltas["max_drawdown_pct"].append(t[2] - c[2])
+
+    def interval(values: list[float]) -> dict:
+        ordered = sorted(values)
+        def q(p: float) -> float:
+            return ordered[min(len(ordered) - 1, max(0, int(round(p * (len(ordered) - 1)))))]
+        return {
+            "mean": round(statistics.fmean(ordered), 3),
+            "median": round(q(0.50), 3),
+            "ci_2.5": round(q(0.025), 3),
+            "ci_97.5": round(q(0.975), 3),
+            "share_below_zero": round(sum(1 for v in ordered if v < 0) / len(ordered), 4),
+        }
+
+    return {metric: interval(values) for metric, values in deltas.items()}
+
+
+async def main(args: argparse.Namespace) -> int:
+    snapshot = Path(args.snapshot)
+    cache = Path(args.cache)
+    store = Path(args.candidates)
+
+    if args.build_candidates or not store.exists():
+        if not args.build_candidates:
+            print(f"{store.name} missing — building it first")
+        await build_candidates(snapshot, store, args.cadence, args.workers)
+        if args.build_candidates:
+            return 0
+
+    payload = pickle.loads(store.read_bytes())
+    candidates = payload["candidates"]
+    prices = payload["prices"]
+    benchmark = payload["benchmark"]
+    min_pct = float(payload["activation"].get("min_momentum_percentile", 80.0))
+    cadence = payload["cadence"]
+
+    signal_dates: dict[str, set[date]] = {}
+    for cand in candidates:
+        signal_dates.setdefault(cand["symbol"], set()).add(date.fromisoformat(cand["date"]))
+
+    series = load_cache_series(cache)
+    missing = sorted({s for s in signal_dates if s not in series})
+    if missing:
+        raise SystemExit(
+            f"{len(missing)} symbols have candidates but no cached series "
+            f"({', '.join(missing[:8])}). Run research_tr_rank_agreement.py --fetch."
+        )
+
+    print(f"candidates: {len(candidates)} over {len(signal_dates)} symbols, cutoff {min_pct}")
+    results = {}
+    for label, total_return in (("control_price_return", False), ("treatment_total_return", True)):
+        started = time.perf_counter()
+        momentum = momentum_by_arm(series, signal_dates, total_return=total_return)
+        arm = apply_arm(candidates, momentum, min_pct)
+        sim = simulate(arm, prices, benchmark, cadence)
+        results[label] = sim
+        print(f"  {label:<24} trades={sim['trades']:<5} CAGR={sim['cagr_pct']}% "
+              f"Sharpe={sim['sharpe']} DD={sim['max_drawdown_pct']}%  "
+              f"({time.perf_counter() - started:.0f}s)")
+
+    control = results["control_price_return"]
+    treatment = results["treatment_total_return"]
+    diff = entry_diff(control, treatment)
+    yearly = yearly_comparison(control, treatment)
+
+    # Paired uncertainty on the DIFFERENCE. Per-arm standard errors describe each
+    # book alone; these arms share dates and most positions, so only a paired
+    # resample speaks to the gap between them.
+    control_dates, control_rets = daily_returns(control.get("equity_curve") or [])
+    treatment_dates, treatment_rets = daily_returns(treatment.get("equity_curve") or [])
+    if control_dates != treatment_dates:
+        raise SystemExit(
+            "Arm return streams are not date-aligned; a paired bootstrap would "
+            "resample mismatched days."
+        )
+    bootstrap = {
+        "method": (
+            "paired moving-block bootstrap on date-aligned daily returns; the same "
+            "block indices are drawn for both arms so the pairing and cross-arm "
+            "correlation are preserved"
+        ),
+        "seed": BOOTSTRAP_SEED,
+        "replicates": BOOTSTRAP_REPLICATES,
+        "sessions": len(control_rets),
+        "pre_specified": (
+            "seed, replicate count and both block lengths were fixed before the run; "
+            "both are reported"
+        ),
+        "scope": (
+            "exploratory uncertainty estimate from the observed window - not "
+            "out-of-sample validation and not a rollout gate"
+        ),
+        "blocks": {},
+    }
+    for block in BOOTSTRAP_BLOCKS:
+        started = time.perf_counter()
+        bootstrap["blocks"][f"{block}_sessions"] = paired_block_bootstrap(
+            control_rets, treatment_rets,
+            block=block, replicates=BOOTSTRAP_REPLICATES, seed=BOOTSTRAP_SEED,
+        )
+        print(f"  bootstrap block={block:<3} {BOOTSTRAP_REPLICATES} reps "
+              f"({time.perf_counter() - started:.0f}s)")
+
+    out = Path(args.out)
+    out.parent.mkdir(parents=True, exist_ok=True)
+    out.write_text(json.dumps({
+        "generated_at": datetime.now(timezone.utc).isoformat(),
+        "snapshot": str(snapshot),
+        "cache": str(cache),
+        "candidates_store": str(store),
+        "cadence": cadence,
+        "min_momentum_percentile": min_pct,
+        "candidate_count": len(candidates),
+        "held_fixed": [
+            "setup candidates (entry/stop/target/rr) generated once and shared",
+            "execution bars: snapshot nominal closes in both arms",
+            "volatility leg: vol_6m from candidate generation, never recomputed",
+            "non-momentum gates, max_positions, risk_per_trade, exit policy, re-entry policy",
+        ],
+        "headline": {
+            label: {k: sim.get(k) for k in HEADLINE} for label, sim in results.items()
+        },
+        "deltas": {
+            k: (
+                round(treatment[k] - control[k], 2)
+                if isinstance(treatment.get(k), (int, float)) and isinstance(control.get(k), (int, float))
+                else None
+            )
+            for k in HEADLINE
+        },
+        "entry_diff": diff,
+        "yearly": yearly,
+        "paired_bootstrap": bootstrap,
+        "arms": {label: {k: v for k, v in sim.items() if k not in {"equity_curve", "benchmark_curve"}} for label, sim in results.items()},
+    }, indent=2, default=str) + "\n", encoding="utf-8")
+
+    print(f"\nentries only in control  : {diff['only_control_count']}")
+    print(f"entries only in treatment: {diff['only_treatment_count']}")
+    print(f"shared entries           : {diff['shared_entries']}")
+    print(f"years treatment better   : {yearly['years_treatment_better']} "
+          f"/ worse {yearly['years_treatment_worse']} / equal {yearly['years_equal']}")
+    print("\npaired block bootstrap, treatment - control (95% interval):")
+    for name, res in bootstrap["blocks"].items():
+        print(f"  block {name}")
+        for metric, stats in res.items():
+            print(f"    {metric:<20} mean {stats['mean']:>8}  "
+                  f"[{stats['ci_2.5']:>8}, {stats['ci_97.5']:>8}]  "
+                  f"P(<0)={stats['share_below_zero']}")
+    print(f"report                   : {out}")
+    return 0
+
+
+def parse_args() -> argparse.Namespace:
+    p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+    p.add_argument("--snapshot", default=str(DEFAULT_SNAPSHOT))
+    p.add_argument("--cache", default=str(DEFAULT_CACHE))
+    p.add_argument("--candidates", default=str(DEFAULT_CANDIDATES))
+    p.add_argument("--cadence", default="weekly", choices=("weekly", "daily"))
+    p.add_argument("--workers", type=int, default=6)
+    p.add_argument("--build-candidates", action="store_true", help="Build the candidate store and exit.")
+    p.add_argument("--out", default="reports/tr-total-return-ab.json")
+    return p.parse_args()
+
+
+if __name__ == "__main__":
+    raise SystemExit(asyncio.run(main(parse_args())))

tests/unit/test_backtest_data_quality.py

@@ -0,0 +1,141 @@
+"""Guards for the backtest completeness ledger.
+
+The failure this exists to prevent is concrete: a snapshot whose schema predates
+a migration makes every ``_fetch_columns`` call raise, both dispatch paths catch
+per-ticker exceptions and continue, and the run writes a report with zero
+candidates that is indistinguishable from an honest run that found no setups.
+The ledger has to turn that into a report that says so.
+"""
+
+from __future__ import annotations
+
+from datetime import date
+
+from app.services.backtest_service import (
+    DATA_QUALITY_FAILURE_SAMPLE,
+    HORIZON,
+    MIN_LOOKBACK,
+    _DataQuality,
+    _has_short_history,
+)
+
+
+def _healthy(expected: int = 3, evaluated: int = 3) -> _DataQuality:
+    q = _DataQuality(expected=expected, evaluated=evaluated)
+    q.record_benchmark({date(2024, 1, 2): 470.0, date(2024, 1, 3): 471.0}, None)
+    return q
+
+
+def test_a_complete_run_is_ok():
+    report = _healthy().as_report()
+    assert report["status"] == "ok"
+    assert report["complete"] is True
+    assert report["reasons"] == []
+    assert report["loaded"] == report["evaluated"] == 3
+    assert report["benchmark"]["momentum_basis"] == "residual"
+    assert report["benchmark"]["first"] == "2024-01-02"
+    assert report["benchmark"]["last"] == "2024-01-03"
+
+
+def test_every_symbol_failing_to_load_reads_as_failed_not_as_no_setups():
+    """The unsynced-snapshot case: 3 expected, 3 fetch errors, 0 evaluated."""
+    q = _DataQuality(expected=3)
+    q.record_benchmark({date(2024, 1, 2): 470.0}, None)
+    for symbol in ("AAPL", "MSFT", "NVDA"):
+        q.record_failure(symbol, "fetch", RuntimeError("no such column: tickers.cik"))
+
+    report = q.as_report()
+    assert report["status"] == "failed"
+    assert report["complete"] is False
+    assert report["evaluated"] == 0
+    assert report["loaded"] == 0
+    assert report["failed"] == 3
+    assert report["failures_by_stage"] == {"fetch": 3}
+    assert "no symbol was evaluated successfully" in report["reasons"]
+    assert report["failures"][0]["symbol"] == "AAPL"
+    assert "tickers.cik" in report["failures"][0]["error"]
+
+
+def test_a_few_failures_degrade_but_do_not_fail_the_run():
+    q = _DataQuality(expected=10, evaluated=8, no_data_symbols=["RHM"])
+    q.record_benchmark({date(2024, 1, 2): 470.0}, None)
+    q.record_failure("HBI", "replay", ValueError("bad bar"))
+
+    report = q.as_report()
+    assert report["status"] == "degraded"
+    assert report["complete"] is False
+    assert report["loaded"] == 9  # no_data is not a load failure
+    assert report["reasons"] == [
+        "1 symbol failed during replay",
+        "1 symbol had no stored bars (RHM)",
+    ]
+
+
+def test_bar_less_symbols_are_named_so_the_reason_is_actionable():
+    """A universe row with no bars recurs on every run until someone clears it.
+    An unnamed count would just nag; the symbol makes it a one-line fix."""
+    q = _DataQuality(expected=9, evaluated=2, no_data_symbols=list("ABCDEFG"))
+    q.record_benchmark({date(2024, 1, 2): 470.0}, None)
+    q.evaluated = 2
+
+    report = q.as_report()
+    assert report["no_data"] == 7
+    assert report["no_data_symbols"] == ["A", "B", "C", "D", "E"]
+    assert report["reasons"] == ["7 symbols had no stored bars (A, B, C, D, E, …)"]
+
+
+def test_a_missing_benchmark_fails_the_run_because_the_signal_changed():
+    """Without SPY the gate ranks on raw 12-1 momentum, so the headline no
+    longer describes the production strategy even though every ticker loaded."""
+    q = _DataQuality(expected=3, evaluated=3)
+    q.record_benchmark({}, ConnectionError("alpaca unreachable"))
+
+    report = q.as_report()
+    assert report["status"] == "failed"
+    assert report["benchmark"]["momentum_basis"] == "raw_fallback"
+    assert report["benchmark"]["sessions"] == 0
+    assert "alpaca unreachable" in report["benchmark"]["error"]
+    assert any("fell back to raw momentum" in r for r in report["reasons"])
+
+
+def test_an_unbalanced_ledger_is_itself_a_failure():
+    """If a counter stops being maintained the block must not quietly report a
+    clean run — the arithmetic is the self-check."""
+    q = _healthy(expected=10, evaluated=3)  # 7 symbols unaccounted for
+    report = q.as_report()
+    assert report["status"] == "failed"
+    assert any("ledger does not balance" in r for r in report["reasons"])
+
+
+def test_an_empty_universe_is_a_failure():
+    q = _DataQuality(expected=0)
+    q.record_benchmark({date(2024, 1, 2): 470.0}, None)
+    assert q.as_report()["status"] == "failed"
+
+
+def test_the_failure_list_is_capped_but_the_count_is_not():
+    q = _DataQuality(expected=100)
+    q.record_benchmark({date(2024, 1, 2): 470.0}, None)
+    for i in range(60):
+        q.record_failure(f"SYM{i}", "fetch", RuntimeError("boom"))
+    q.evaluated = 40
+
+    report = q.as_report()
+    assert report["failed"] == 60
+    assert len(report["failures"]) == DATA_QUALITY_FAILURE_SAMPLE
+    assert report["failures_truncated"] is True
+
+
+def test_short_history_counts_a_recent_listing_without_failing_the_run():
+    q = _healthy()
+    q.short_history = 1
+    report = q.as_report()
+    assert report["short_history"] == 1
+    assert report["status"] == "ok", "a recent listing is not a data-quality failure"
+
+
+def test_has_short_history_uses_the_replay_minimum():
+    enough = ([0] * (MIN_LOOKBACK + HORIZON),) + ((),) * 5
+    one_short = ([0] * (MIN_LOOKBACK + HORIZON - 1),) + ((),) * 5
+    assert _has_short_history(enough) == 0
+    assert _has_short_history(one_short) == 1

tests/unit/test_backtest_service.py

@@ -1689,6 +1689,74 @@ async def test_run_backtest_rolls_back_a_failed_ticker_fetch(session, monkeypatc
     assert report["tickers"] == 2
     assert report["candidates"] >= 1
 
+    # ...and the report says one ticker was lost rather than presenting the
+    # remaining one as the whole universe.
+    quality = report["data_quality"]
+    assert quality["expected"] == 2
+    assert quality["evaluated"] == 1
+    assert quality["failed"] == 1
+    assert quality["failures"][0]["symbol"] == "AAA"
+    assert quality["failures"][0]["stage"] == "fetch"
+    assert "simulated OHLCV read failure" in quality["failures"][0]["error"]
+    assert quality["complete"] is False
+    assert quality["status"] == "degraded"
+
+
+async def test_run_backtest_reports_its_own_completeness(session):
+    """The healthy case: every ticker accounted for, so the report says so."""
+    await _seed_oscillating_ticker(session, "OSC")
+    report = await bt.run_backtest(session)
+
+    quality = report["data_quality"]
+    assert quality["expected"] == report["tickers"] == 1
+    assert quality["evaluated"] == 1
+    assert quality["failed"] == 0
+    assert quality["no_data"] == 0
+    # Ledger identity — the three terminal buckets must cover the universe.
+    assert quality["no_data"] + quality["failed"] + quality["evaluated"] == quality["expected"]
+
+
+async def test_run_backtest_marks_a_run_failed_when_every_ticker_fails(session, monkeypatch):
+    """The demonstrated failure: an unsynced snapshot raises on every read, the
+    replay loop swallows each one, and the run writes a report with no
+    candidates. It must come back "failed", not look like a quiet run.
+    """
+    await _seed_oscillating_ticker(session, "AAA")
+    await _seed_oscillating_ticker(session, "OSC")
+
+    async def always_failing_fetch(db, symbol):
+        raise RuntimeError("no such column: tickers.cik")
+
+    monkeypatch.setattr(bt, "_fetch_columns", always_failing_fetch)
+
+    report = await bt.run_backtest(session)
+
+    assert report["candidates"] == 0
+    quality = report["data_quality"]
+    assert quality["status"] == "failed"
+    assert quality["evaluated"] == 0
+    assert quality["loaded"] == 0
+    assert quality["failed"] == 2
+    assert "no symbol was evaluated successfully" in quality["reasons"]
+
+
+async def test_run_backtest_flags_a_missing_benchmark(session, monkeypatch):
+    """Residual momentum degrades to raw without SPY. The numbers then describe
+    a different gate than the one the report claims, so the run is not clean."""
+    await _seed_oscillating_ticker(session, "OSC")
+
+    async def no_benchmark(db, **kwargs):
+        return {}
+
+    monkeypatch.setattr(bt, "_load_benchmark_closes_for_backtest", no_benchmark)
+
+    report = await bt.run_backtest(session)
+    benchmark = report["data_quality"]["benchmark"]
+
+    assert benchmark["sessions"] == 0
+    assert benchmark["momentum_basis"] == "raw_fallback"
+    assert report["data_quality"]["status"] == "failed"
+
 
 async def test_run_backtest_rolls_back_a_failed_portfolio_sim_load(session, monkeypatch):
     """The portfolio-sim block loads the benchmark and the live exit policy from

tests/unit/test_backtest_terminal_events.py

@@ -0,0 +1,167 @@
+"""Replay bounds for a symbol carrying a research terminal mark.
+
+``_replay_ticker`` normally stops ``HORIZON`` bars before the end so every setup
+has a full resolution window. For a delisted name that is exactly wrong: the
+collapse happens in those final bars, so the signals that would have been caught
+by it are the ones the bound hides. ``terminal_date`` opts a single symbol into
+walking through its last real bar.
+
+The fixtures build a controlled setup rather than replaying a real ticker, so
+the tests do not depend on whether SIVB happened to qualify on a given day.
+"""
+
+from __future__ import annotations
+
+from datetime import date, timedelta
+from types import SimpleNamespace
+
+import pytest
+
+from app.services import backtest_service as bt
+
+
+def _bar(day: date, close: float, *, high: float | None = None, low: float | None = None):
+    return SimpleNamespace(
+        date=day,
+        open=close,
+        high=close * 1.01 if high is None else high,
+        low=close * 0.99 if low is None else low,
+        close=close,
+        volume=1_000_000,
+    )
+
+
+def _series(n: int, *, start: date = date(2021, 1, 4)) -> list:
+    """A steadily rising series long enough to clear MIN_LOOKBACK."""
+    bars = []
+    day = start
+    for i in range(n):
+        while day.weekday() >= 5:
+            day += timedelta(days=1)
+        bars.append(_bar(day, 100.0 + i * 0.25))
+        day += timedelta(days=1)
+    return bars
+
+
+def _terminal_bar(previous: date) -> SimpleNamespace:
+    day = previous + timedelta(days=1)
+    while day.weekday() >= 5:
+        day += timedelta(days=1)
+    return SimpleNamespace(
+        date=day, open=0.01, high=0.01, low=0.01, close=0.01, volume=0
+    )
+
+
+@pytest.fixture()
+def replay(monkeypatch):
+    """Force one long setup on every evaluated bar so signal dates are countable."""
+
+    def _fake_setups(window, config, activation, *, target_model):
+        close = float(window[-1].close)
+        return [{
+            "direction": "long",
+            "entry": close,
+            "stop": close * 0.95,
+            # Reachable within HORIZON on the rising fixture, but not within the
+            # four bars left before a terminal event.
+            "target": close * 1.02,
+            "rr": 2.0,
+            "confidence": 80,
+            "momentum": 0.1,
+            "best_prob": 0.5,
+            "primary_prob": 0.5,
+            "meets_core": True,
+            "action": "BUY",
+            "risk_level": "medium",
+            "target_model": target_model,
+            "primary_sources": [],
+            "primary_strength": 1.0,
+            "primary_rejection_count": 0,
+            "primary_last_rejection_age": None,
+            "primary_distance_atr": 1.0,
+            "raw_level_count": 1,
+            "gate_level_count": 1,
+        }]
+
+    monkeypatch.setattr(bt, "_window_setups", _fake_setups)
+
+    def _run(records, terminal_date=None):
+        return bt._replay_ticker(
+            "TEST", records, {}, {}, None,
+            bt.PRODUCTION_GTL_TARGET_MODEL, "daily", terminal_date,
+        )
+
+    return _run
+
+
+def test_setup_near_a_terminal_event_is_emitted_and_resolves_at_the_mark(replay):
+    """A setup five sessions before the halt is replayed, and if it is still open
+    at the terminal bar the near-zero price resolves it as a loss."""
+    real = _series(bt.MIN_LOOKBACK + 40)
+    records = real + [_terminal_bar(real[-1].date)]
+
+    without = replay(records)
+    with_mark = replay(records, terminal_date=records[-1].date)
+
+    last_real = real[-1].date
+    fifth_last = real[-5].date
+
+    without_dates = {c["date"] for c in without}
+    with_dates = {c["date"] for c in with_mark}
+
+    assert fifth_last.isoformat() not in without_dates, (
+        "baseline should hide signals inside the final HORIZON bars"
+    )
+    assert fifth_last.isoformat() in with_dates
+    assert last_real.isoformat() in with_dates
+
+    late = next(c for c in with_mark if c["date"] == fifth_last.isoformat())
+    assert late["outcome"] == bt.OUTCOME_STOP_HIT
+    assert late["realized_r"] < -1.0, "a gap to 0.01 must fill below the stop, not at it"
+
+
+def test_no_setup_is_generated_on_the_terminal_bar(replay):
+    real = _series(bt.MIN_LOOKBACK + 40)
+    terminal = _terminal_bar(real[-1].date)
+    records = real + [terminal]
+
+    candidates = replay(records, terminal_date=terminal.date)
+
+    assert terminal.date.isoformat() not in {c["date"] for c in candidates}
+    assert max(c["date"] for c in candidates) == real[-1].date.isoformat()
+
+
+def test_a_position_that_exits_before_the_event_keeps_its_own_exit(replay):
+    """An early setup resolves on its own bars; the terminal mark must not
+    retroactively turn it into a total loss."""
+    real = _series(bt.MIN_LOOKBACK + 40)
+    records = real + [_terminal_bar(real[-1].date)]
+
+    with_mark = replay(records, terminal_date=records[-1].date)
+    early_date = real[bt.MIN_LOOKBACK - 1].date.isoformat()
+
+    early = next(c for c in with_mark if c["date"] == early_date)
+    assert early["outcome"] == bt.OUTCOME_TARGET_HIT
+    assert early["realized_r"] == pytest.approx(2.0)
+
+
+def test_unmapped_symbols_and_runs_without_the_option_are_unchanged(replay):
+    """The ordinary replay bounds are untouched when no terminal date is given."""
+    records = _series(bt.MIN_LOOKBACK + 40)
+
+    baseline = replay(records)
+    again = replay(records, terminal_date=None)
+
+    assert [c["date"] for c in baseline] == [c["date"] for c in again]
+    assert max(c["date"] for c in baseline) < records[-1].date.isoformat(), (
+        "without a terminal mark the walk must still stop HORIZON bars early"
+    )
+
+
+def test_a_mismatched_terminal_date_is_rejected(replay):
+    """The date is validated against the final bar rather than trusted, so a
+    stale events file fails loudly instead of shifting the replay window."""
+    records = _series(bt.MIN_LOOKBACK + 40)
+
+    with pytest.raises(ValueError, match="does not match the final bar"):
+        replay(records, terminal_date=date(1999, 1, 1))

tests/unit/test_migration_035.py

@@ -0,0 +1,86 @@
+"""Migration 035 adds the untracked flag and stays reversible.
+
+The downgrade matters more than usual here: dropping the column re-admits every
+untracked symbol to the live path, and the rows survive either way. A migration
+that could not roll back cleanly would make the prune change hard to undo on a
+deploy that went wrong.
+"""
+
+from __future__ import annotations
+
+import importlib.util
+from pathlib import Path
+
+import sqlalchemy as sa
+from alembic.migration import MigrationContext
+from alembic.operations import Operations
+
+
+def _load_migration_module():
+    path = (
+        Path(__file__).resolve().parents[2]
+        / "alembic"
+        / "versions"
+        / "035_ticker_untracked.py"
+    )
+    spec = importlib.util.spec_from_file_location("migration_035", path)
+    assert spec is not None and spec.loader is not None
+    module = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(module)
+    return module
+
+
+def _tickers_table(connection: sa.Connection) -> None:
+    connection.execute(sa.text(
+        """
+        CREATE TABLE tickers (
+            id INTEGER PRIMARY KEY,
+            symbol VARCHAR(10) NOT NULL,
+            delisted_on DATE
+        )
+        """
+    ))
+    connection.execute(sa.text(
+        "INSERT INTO tickers (id, symbol, delisted_on) VALUES (1, 'AAPL', NULL)"
+    ))
+
+
+def _columns(connection: sa.Connection) -> set[str]:
+    return {row[1] for row in connection.execute(sa.text("PRAGMA table_info(tickers)"))}
+
+
+def test_upgrade_adds_a_nullable_untracked_column_and_keeps_existing_rows():
+    migration = _load_migration_module()
+    engine = sa.create_engine("sqlite://")
+
+    with engine.begin() as connection:
+        _tickers_table(connection)
+        with Operations.context(MigrationContext.configure(connection)):
+            migration.upgrade()
+
+        assert "untracked_on" in _columns(connection)
+        # Existing rows default to NULL, i.e. still in the scanner — an upgrade
+        # must not silently drop the whole universe out of the live path.
+        rows = connection.execute(
+            sa.text("SELECT symbol, untracked_on FROM tickers")
+        ).all()
+        assert rows == [("AAPL", None)]
+
+
+def test_downgrade_removes_the_column_without_touching_the_rows():
+    migration = _load_migration_module()
+    engine = sa.create_engine("sqlite://")
+
+    with engine.begin() as connection:
+        _tickers_table(connection)
+        with Operations.context(MigrationContext.configure(connection)):
+            migration.upgrade()
+        connection.execute(sa.text(
+            "UPDATE tickers SET untracked_on = '2026-09-07' WHERE symbol = 'AAPL'"
+        ))
+        with Operations.context(MigrationContext.configure(connection)):
+            migration.downgrade()
+
+        assert "untracked_on" not in _columns(connection)
+        # The ticker itself survives the rollback with its identity intact.
+        assert connection.execute(sa.text("SELECT symbol FROM tickers")).all() == [("AAPL",)]

tests/unit/test_migration_036.py

@@ -0,0 +1,132 @@
+"""Migration 036: provenance, the operator switch, and membership intervals.
+
+The migration makes two irreversible-feeling judgement calls, and both need to
+be pinned:
+
+1. **Existing rows become ``manual``.** Provenance cannot be recovered from
+   ``created_at``, and manual is the fail-safe guess — manual rows are never
+   pruned, so a wrong guess costs an un-pruned symbol rather than one swept out
+   of the scanner.
+2. **Existing "off" states are carried into ``scan_disabled_on``.** Whoever set
+   ``untracked_on`` is ambiguous, and re-entry clears the prune flag. Copying it
+   is what stops an automated sync reactivating something switched off by hand.
+"""
+
+from __future__ import annotations
+
+import importlib.util
+from pathlib import Path
+
+import sqlalchemy as sa
+from alembic.migration import MigrationContext
+from alembic.operations import Operations
+
+
+def _load_migration_module():
+    path = (
+        Path(__file__).resolve().parents[2]
+        / "alembic"
+        / "versions"
+        / "036_universe_membership.py"
+    )
+    spec = importlib.util.spec_from_file_location("migration_036", path)
+    assert spec is not None and spec.loader is not None
+    module = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(module)
+    return module
+
+
+def _tickers_table(connection: sa.Connection) -> None:
+    """The schema as migration 035 leaves it."""
+    connection.execute(sa.text(
+        """
+        CREATE TABLE tickers (
+            id INTEGER PRIMARY KEY,
+            symbol VARCHAR(10) NOT NULL,
+            delisted_on DATE,
+            untracked_on DATE
+        )
+        """
+    ))
+    connection.execute(sa.text(
+        "INSERT INTO tickers (id, symbol, delisted_on, untracked_on) VALUES "
+        "(1, 'AAPL', NULL, NULL), "
+        "(2, 'OFF', NULL, '2026-08-01')"
+    ))
+
+
+def _columns(connection: sa.Connection, table: str) -> set[str]:
+    return {row[1] for row in connection.execute(sa.text(f"PRAGMA table_info({table})"))}
+
+
+def test_upgrade_defaults_existing_rows_to_manual():
+    migration = _load_migration_module()
+    engine = sa.create_engine("sqlite://")
+
+    with engine.begin() as connection:
+        _tickers_table(connection)
+        with Operations.context(MigrationContext.configure(connection)):
+            migration.upgrade()
+
+        rows = dict(connection.execute(
+            sa.text("SELECT symbol, tracking_source FROM tickers")
+        ).all())
+        assert rows == {"AAPL": "manual", "OFF": "manual"}
+
+
+def test_upgrade_carries_an_existing_off_state_into_the_operator_switch():
+    migration = _load_migration_module()
+    engine = sa.create_engine("sqlite://")
+
+    with engine.begin() as connection:
+        _tickers_table(connection)
+        with Operations.context(MigrationContext.configure(connection)):
+            migration.upgrade()
+
+        rows = dict(connection.execute(
+            sa.text("SELECT symbol, scan_disabled_on FROM tickers")
+        ).all())
+        # The ambiguous existing "off" is preserved on the column no sync clears.
+        assert rows["OFF"] == "2026-08-01"
+        # A row that was never off stays on.
+        assert rows["AAPL"] is None
+
+
+def test_upgrade_creates_the_membership_table_empty():
+    migration = _load_migration_module()
+    engine = sa.create_engine("sqlite://")
+
+    with engine.begin() as connection:
+        _tickers_table(connection)
+        with Operations.context(MigrationContext.configure(connection)):
+            migration.upgrade()
+
+        assert {"ticker_id", "universe", "first_seen_on", "absent_since"} <= _columns(
+            connection, "universe_memberships"
+        )
+        # History starts at first observation — the migration invents none.
+        count = connection.execute(
+            sa.text("SELECT COUNT(*) FROM universe_memberships")
+        ).scalar_one()
+        assert count == 0
+
+
+def test_downgrade_removes_the_new_state_and_keeps_the_tickers():
+    migration = _load_migration_module()
+    engine = sa.create_engine("sqlite://")
+
+    with engine.begin() as connection:
+        _tickers_table(connection)
+        with Operations.context(MigrationContext.configure(connection)):
+            migration.upgrade()
+        with Operations.context(MigrationContext.configure(connection)):
+            migration.downgrade()
+
+        columns = _columns(connection, "tickers")
+        assert "tracking_source" not in columns
+        assert "scan_disabled_on" not in columns
+        # 035's column and the rows themselves are untouched.
+        assert "untracked_on" in columns
+        assert connection.execute(
+            sa.text("SELECT symbol FROM tickers ORDER BY symbol")
+        ).all() == [("AAPL",), ("OFF",)]

tests/unit/test_run_backtest_snapshot_terminal_events.py

@@ -0,0 +1,123 @@
+"""Runner-level guard: a stale terminal-event map must abort the whole run.
+
+``_replay_ticker`` rejects a mismatched date, but both dispatch paths in
+``run_backtest`` swallow per-ticker exceptions so one unreadable symbol cannot
+kill a long replay. That safety net would turn a wrong events file into a
+quietly incomplete report, so the map is validated before any replay starts —
+and the runner must therefore write no report at all.
+"""
+
+from __future__ import annotations
+
+import json
+from datetime import date, datetime, timedelta
+from pathlib import Path
+
+import pytest
+from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker, create_async_engine
+
+from app.database import Base
+from app.models.ohlcv import OHLCVRecord
+from app.models.ticker import Ticker
+from app.services.backtest_service import TerminalEventError, run_backtest
+
+
+async def _make_snapshot(path: Path, *, symbol: str = "DEAD", bars: int = 40) -> date:
+    """A minimal snapshot with one ticker. Returns its final bar date."""
+    engine = create_async_engine(f"sqlite+aiosqlite:///{path.as_posix()}")
+    async with engine.begin() as conn:
+        await conn.run_sync(Base.metadata.create_all)
+
+    Session = async_sessionmaker(engine, class_=AsyncSession, expire_on_commit=False)
+    day = date(2022, 1, 3)
+    last = day
+    async with Session() as db:
+        ticker = Ticker(symbol=symbol, name=symbol, created_at=datetime.utcnow())
+        db.add(ticker)
+        await db.flush()
+        for i in range(bars):
+            while day.weekday() >= 5:
+                day += timedelta(days=1)
+            price = 100.0 + i
+            db.add(OHLCVRecord(
+                ticker_id=ticker.id, date=day, open=price, high=price * 1.01,
+                low=price * 0.99, close=price, volume=1_000_000,
+                created_at=datetime.utcnow(),
+            ))
+            last = day
+            day += timedelta(days=1)
+        await db.commit()
+    await engine.dispose()
+    return last
+
+
+@pytest.mark.asyncio
+async def test_a_mismatched_terminal_date_aborts_the_run(tmp_path):
+    snapshot = tmp_path / "snap.sqlite"
+    final_bar = await _make_snapshot(snapshot)
+
+    engine = create_async_engine(f"sqlite+aiosqlite:///{snapshot.as_posix()}")
+    Session = async_sessionmaker(engine, class_=AsyncSession, expire_on_commit=False)
+    try:
+        async with Session() as db:
+            with pytest.raises(TerminalEventError, match="final bar"):
+                await run_backtest(db, terminal_events={"DEAD": final_bar + timedelta(days=5)})
+    finally:
+        await engine.dispose()
+
+
+@pytest.mark.asyncio
+async def test_an_unknown_symbol_aborts_the_run(tmp_path):
+    snapshot = tmp_path / "snap.sqlite"
+    await _make_snapshot(snapshot)
+
+    engine = create_async_engine(f"sqlite+aiosqlite:///{snapshot.as_posix()}")
+    Session = async_sessionmaker(engine, class_=AsyncSession, expire_on_commit=False)
+    try:
+        async with Session() as db:
+            with pytest.raises(TerminalEventError, match="not in this snapshot"):
+                await run_backtest(db, terminal_events={"NOPE": date(2022, 2, 1)})
+    finally:
+        await engine.dispose()
+
+
+@pytest.mark.asyncio
+async def test_the_runner_writes_no_report_when_validation_fails(tmp_path, monkeypatch):
+    """The report is written only after run_backtest returns, so an abort must
+    leave the output path untouched rather than a partial or empty report."""
+    snapshot = tmp_path / "snap.sqlite"
+    final_bar = await _make_snapshot(snapshot)
+
+    events = tmp_path / "events.json"
+    events.write_text(json.dumps({"DEAD": (final_bar + timedelta(days=5)).isoformat()}))
+    out = tmp_path / "report.json"
+
+    import scripts.run_backtest_snapshot as runner
+
+    monkeypatch.setattr(
+        runner.sys, "argv",
+        ["run_backtest_snapshot.py", str(snapshot), "--out", str(out),
+         "--terminal-events", str(events), "--quiet", "--workers", "1"],
+    )
+
+    with pytest.raises(TerminalEventError):
+        await runner._main()
+
+    assert not out.exists(), "a failed run must not leave a report behind"
+
+
+@pytest.mark.asyncio
+async def test_a_matching_map_is_accepted(tmp_path):
+    """The guard must not reject a correct map — the run proceeds normally."""
+    snapshot = tmp_path / "snap.sqlite"
+    final_bar = await _make_snapshot(snapshot)
+
+    engine = create_async_engine(f"sqlite+aiosqlite:///{snapshot.as_posix()}")
+    Session = async_sessionmaker(engine, class_=AsyncSession, expire_on_commit=False)
+    try:
+        async with Session() as db:
+            report = await run_backtest(db, terminal_events={"DEAD": final_bar})
+    finally:
+        await engine.dispose()
+
+    assert isinstance(report, dict)

tests/unit/test_ticker_delisting.py

@@ -347,26 +347,37 @@ def test_common_stock_classification(description: str, expected: bool):
     assert _is_common_stock(description) is expected
 
 
-async def test_prune_keeps_delisted_rows(session: AsyncSession, monkeypatch):
-    """A prune must not destroy rows the delisting flow deliberately retained —
-    their price history is the whole reason those rows still exist."""
+async def test_prune_keeps_every_row(session: AsyncSession, monkeypatch):
+    """A prune destroys nothing at all now — not the delisted row it always
+    protected, and not the ex-constituent it used to delete outright."""
     from app.services import ticker_universe_service as tus
 
-    session.add_all([Ticker(symbol="AAPL"), Ticker(symbol="EA"), Ticker(symbol="GONE")])
+    session.add_all([
+        Ticker(symbol=s, tracking_source=ticker_service.SOURCE_INDEX)
+        for s in ("AAPL", "EA", "GONE")
+    ])
     await session.commit()
     await ticker_service.mark_delisted(session, "EA", delisted_on=date(2026, 8, 14))
 
     async def fake_fetch(db, universe):
-        return ["AAPL"], "test"
+        # A live source label: pruning only acts on a successful fetch.
+        return ["AAPL"], "wikipedia_sp500"
 
     monkeypatch.setattr(tus, "fetch_universe_symbols", fake_fetch)
 
     summary = await tus.bootstrap_universe(session, "sp500", prune_missing=True)
 
     remaining = sorted(t.symbol for t in await ticker_service.list_tickers(session))
-    assert remaining == ["AAPL", "EA"]  # GONE pruned, EA protected
-    assert summary["deleted"] == 1
-    assert summary["kept_delisted"] == ["EA"]
+    assert remaining == ["AAPL", "EA", "GONE"]
+    assert summary["deleted"] == 0
+    assert summary["untracked"] == 2
+
+    # Both leave the live path, by different routes: EA stopped trading, GONE
+    # merely left the index.
+    live = await session.execute(
+        ticker_service.active_only(select(Ticker.symbol).order_by(Ticker.symbol))
+    )
+    assert list(live.scalars().all()) == ["AAPL"]
 
 
 async def test_a_future_effective_date_keeps_the_symbol_live(session: AsyncSession):

tests/unit/test_ticker_universe_service.py

@@ -12,7 +12,7 @@ from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker, create_asyn
 from app.database import Base
 from app.models.settings import SystemSetting
 from app.models.ticker import Ticker
-from app.services import ticker_universe_service
+from app.services import ticker_service, ticker_universe_service
 from app.services.ticker_universe_service import _extract_wiki_symbols, _normalise_symbols
 
 _engine = create_async_engine("sqlite+aiosqlite://", echo=False)
@@ -58,11 +58,17 @@ async def test_bootstrap_universe_adds_missing_symbols(session: AsyncSession, mo
 
 @pytest.mark.asyncio
 async def test_bootstrap_universe_prunes_missing_symbols(session: AsyncSession, monkeypatch: pytest.MonkeyPatch):
-    session.add_all([Ticker(symbol="AAPL"), Ticker(symbol="MSFT"), Ticker(symbol="TSLA")])
+    # Index-owned: a row defaults to manual, and manual rows are never pruned.
+    session.add_all([
+        Ticker(symbol=s, tracking_source=ticker_service.SOURCE_INDEX)
+        for s in ("AAPL", "MSFT", "TSLA")
+    ])
     await session.commit()
 
     async def _fake_fetch(_db: AsyncSession, _universe: str) -> tuple[list[str], str]:
-        return ["AAPL", "MSFT"], "test"
+        # A live source label: a fallback result must not drive pruning, so a
+        # made-up label would make this test vacuous.
+        return ["AAPL", "MSFT"], "wikipedia_sp500"
 
     monkeypatch.setattr(ticker_universe_service, "fetch_universe_symbols", _fake_fetch)
 
@@ -74,10 +80,18 @@ async def test_bootstrap_universe_prunes_missing_symbols(session: AsyncSession,
 
     assert result["added"] == 0
     assert result["already_tracked"] == 2
-    assert result["deleted"] == 1
+    assert result["untracked"] == 1
+    assert result["untracked_symbols"] == ["TSLA"]
+    # Nothing is destroyed any more, and the key says so rather than vanishing.
+    assert result["deleted"] == 0
 
+    # The row survives the prune; only its scanner membership changed.
     rows = await session.execute(select(Ticker.symbol).order_by(Ticker.symbol.asc()))
-    assert list(rows.scalars().all()) == ["AAPL", "MSFT"]
+    assert list(rows.scalars().all()) == ["AAPL", "MSFT", "TSLA"]
+    live = await session.execute(
+        ticker_service.active_only(select(Ticker.symbol).order_by(Ticker.symbol.asc()))
+    )
+    assert list(live.scalars().all()) == ["AAPL", "MSFT"]
 
 
 @pytest.mark.asyncio

tests/unit/test_total_return_ab.py

@@ -0,0 +1,186 @@
+"""Guards for the paired total-return A/B replay.
+
+Two things have to hold before any difference the A/B reports means anything:
+
+1. **Identical inputs must produce identical books.** If the harness were
+   nondeterministic — dict ordering, tie-breaks, path-dependent holdings — a
+   difference between the arms could come from the replay rather than the
+   momentum basis. Feeding both arms the same series must give byte-identical
+   results.
+2. **A real momentum difference must be able to change an admission.** A harness
+   that holds everything so tightly it can never diverge would report agreement
+   no matter what. One name pushed across the cutoff must actually change who
+   gets admitted.
+"""
+
+from __future__ import annotations
+
+import importlib.util
+from datetime import date, timedelta
+from pathlib import Path
+
+import pytest
+
+# Imported at module scope like the other backtest tests: loading the service
+# lazily inside the first test leaves conftest's in-memory engine without its
+# schema for every test after it.
+from app.services import backtest_service as _bt  # noqa: F401
+
+ROOT = Path(__file__).resolve().parents[2]
+_spec = importlib.util.spec_from_file_location(
+    "run_total_return_ab", ROOT / "scripts" / "run_total_return_ab.py"
+)
+ab = importlib.util.module_from_spec(_spec)
+_spec.loader.exec_module(ab)
+
+
+def _sessions(n: int, start: date = date(2021, 1, 4)) -> list[date]:
+    days, day = [], start
+    while len(days) < n:
+        if day.weekday() < 5:
+            days.append(day)
+        day += timedelta(days=1)
+    return days
+
+
+def _series(n: int = 400, drift: float = 0.30) -> dict:
+    """A cache-shaped series set: three names plus SPY, all with clean history."""
+    days = _sessions(n)
+    out = {}
+    for i, symbol in enumerate(("AAA", "BBB", "CCC")):
+        splits = [100.0 * (1.0 + (drift + i * 0.10) * k / n) for k in range(n)]
+        out[symbol] = (days, splits, list(splits))  # ALL == SPLIT: no dividends
+    spy = [100.0 * (1.0 + 0.10 * k / n) for k in range(n)]
+    out[ab.BENCHMARK] = (days, spy, list(spy))
+    return out
+
+
+def _candidates(days: list[date], signal_index: int) -> list[dict]:
+    """One qualifying long setup per symbol on a single signal date."""
+    as_of = days[signal_index]
+    out = []
+    for symbol in ("AAA", "BBB", "CCC"):
+        out.append({
+            "symbol": symbol,
+            "date": as_of.isoformat(),
+            # Real candidates carry both; the percentile pass groups on them.
+            "ranking_period": as_of.isocalendar()[:2],
+            "iso_week": as_of.isocalendar()[:2],
+            "direction": "long",
+            "entry": 100.0,
+            "stop": 95.0,
+            "target": 110.0,
+            "rr": 2.0,
+            "vol_6m": 0.02,
+            "residual_momentum": 0.0,
+            "momentum": 0.0,
+            "meets_core": True,
+            "qualified": True,
+        })
+    return out
+
+
+def test_identical_series_give_identical_momentum_and_ordering():
+    """ALL == SPLIT everywhere, so the two arms must agree exactly — any
+    difference the real run reports is then attributable to the basis."""
+    series = _series()
+    days = series["AAA"][0]
+    signal_dates = {s: {days[-1]} for s in ("AAA", "BBB", "CCC")}
+
+    control = ab.momentum_by_arm(series, signal_dates, total_return=False)
+    treatment = ab.momentum_by_arm(series, signal_dates, total_return=True)
+
+    assert control == treatment
+    assert control, "fixture must produce momentum values, else the test is vacuous"
+
+    cands = _candidates(days, len(days) - 1)
+    a = ab.apply_arm(cands, control, 80.0)
+    b = ab.apply_arm(cands, treatment, 80.0)
+    assert a == b
+
+
+def test_entry_diff_is_empty_for_identical_books():
+    sim = {"trade_details": [
+        {"entry_date": "2022-01-04", "symbol": "AAA"},
+        {"entry_date": "2022-02-01", "symbol": "BBB"},
+    ]}
+    diff = ab.entry_diff(sim, dict(sim))
+    assert diff["only_control_count"] == 0
+    assert diff["only_treatment_count"] == 0
+    assert diff["shared_entries"] == 2
+
+
+def test_a_momentum_change_moves_a_name_across_the_cutoff():
+    """The harness must be able to see a real difference: lifting one name's
+    momentum above its peers changes which names clear the gate."""
+    series = _series()
+    days = series["AAA"][0]
+    cands = _candidates(days, len(days) - 1)
+    as_of = days[-1]
+
+    # CCC ranks top on the control momentum, AAA bottom.
+    control_mom = {("AAA", as_of): 0.10, ("BBB", as_of): 0.20, ("CCC", as_of): 0.30}
+    # Treatment reverses it — as a dividend-heavy name overtaking a growth name would.
+    treatment_mom = {("AAA", as_of): 0.30, ("BBB", as_of): 0.20, ("CCC", as_of): 0.10}
+
+    a = ab.apply_arm(cands, control_mom, 80.0)
+    b = ab.apply_arm(cands, treatment_mom, 80.0)
+
+    admitted_a = {c["symbol"] for c in a if c["qualified"]}
+    admitted_b = {c["symbol"] for c in b if c["qualified"]}
+
+    assert admitted_a == {"CCC"}
+    assert admitted_b == {"AAA"}
+    assert admitted_a != admitted_b
+
+
+def test_the_volatility_leg_is_untouched_by_the_arm():
+    """Only momentum may vary; vol_6m carries through both arms unchanged, so the
+    80/20 blend differs solely through its momentum leg."""
+    series = _series()
+    days = series["AAA"][0]
+    as_of = days[-1]
+    cands = _candidates(days, len(days) - 1)
+    for index, cand in enumerate(cands):
+        cand["vol_6m"] = 0.01 * (index + 1)
+
+    a = ab.apply_arm(cands, {("AAA", as_of): 0.1, ("BBB", as_of): 0.2, ("CCC", as_of): 0.3}, 80.0)
+    b = ab.apply_arm(cands, {("AAA", as_of): 0.3, ("BBB", as_of): 0.2, ("CCC", as_of): 0.1}, 80.0)
+
+    from app.services.backtest_service import VOL_PERCENTILE_KEY
+
+    assert [c[VOL_PERCENTILE_KEY] for c in a] == [c[VOL_PERCENTILE_KEY] for c in b]
+    assert [c["vol_6m"] for c in a] == [c["vol_6m"] for c in b]
+
+
+def test_missing_momentum_leaves_a_candidate_unqualified():
+    """A name with no computable momentum in an arm must not sneak through the
+    gate on a stale value carried over from candidate generation."""
+    series = _series()
+    days = series["AAA"][0]
+    as_of = days[-1]
+    cands = _candidates(days, len(days) - 1)
+
+    arm = ab.apply_arm(cands, {("AAA", as_of): 0.5}, 80.0)
+    by_symbol = {c["symbol"]: c for c in arm}
+
+    assert by_symbol["BBB"]["residual_momentum"] is None
+    assert by_symbol["CCC"]["residual_momentum"] is None
+
+
+def test_yearly_comparison_counts_years_treatment_wins():
+    control = {"yearly_returns": [
+        {"year": 2022, "return_pct": 10.0},
+        {"year": 2023, "return_pct": 20.0},
+        {"year": 2024, "return_pct": 5.0},
+    ]}
+    treatment = {"yearly_returns": [
+        {"year": 2022, "return_pct": 12.0},
+        {"year": 2023, "return_pct": 18.0},
+        {"year": 2024, "return_pct": 5.0},
+    ]}
+    out = ab.yearly_comparison(control, treatment)
+    assert out["years_treatment_better"] == 1
+    assert out["years_treatment_worse"] == 1
+    assert out["years_equal"] == 1
+    assert out["rows"][0]["delta_pp"] == pytest.approx(2.0)

tests/unit/test_universe_membership.py

@@ -0,0 +1,477 @@
+"""Scanner membership and retained history are separate states.
+
+`bootstrap_universe(prune_missing=True)` used to run `DELETE FROM tickers`,
+cascading OHLCV, setups and scores away for any symbol that had merely dropped
+out of the index. The survivorship audit measured the cost of that: restoring 85
+removed constituents moved the same window from 2.09 Sharpe / 51.6% CAGR to 1.57
+/ 38.1%.
+
+Three properties have to hold together, and the third is the one nobody would
+think to check:
+
+1. Pruning preserves the symbol's bars and its linked records.
+2. The pruned symbol leaves today's scanner and rankings.
+3. The backtest's universe query still returns it — today's membership flag must
+   not silently erase yesterday's candidates.
+"""
+
+from __future__ import annotations
+
+from collections.abc import AsyncGenerator
+from datetime import date, timedelta
+
+import pytest
+from sqlalchemy import select
+from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker, create_async_engine
+
+from app.database import Base
+from app.models.ohlcv import OHLCVRecord
+from app.models.sr_level import SRLevel
+from app.models.ticker import Ticker
+from app.services import ticker_service
+from app.services import ticker_universe_service as tus
+
+# Own engine, like the sibling delisting tests: these exercise schema-level
+# behaviour (a new column, a cascade that must NOT fire) and want a clean slate.
+_engine = create_async_engine("sqlite+aiosqlite://", echo=False)
+_session_factory = async_sessionmaker(_engine, class_=AsyncSession, expire_on_commit=False)
+
+
+@pytest.fixture(autouse=True)
+async def _setup_tables() -> AsyncGenerator[None, None]:
+    async with _engine.begin() as conn:
+        await conn.run_sync(Base.metadata.create_all)
+    yield
+    async with _engine.begin() as conn:
+        await conn.run_sync(Base.metadata.drop_all)
+
+
+@pytest.fixture
+async def session() -> AsyncGenerator[AsyncSession, None]:
+    async with _session_factory() as s:
+        yield s
+
+
+async def _seed(session: AsyncSession) -> None:
+    """Two index-owned constituents, one about to leave with history.
+
+    ``tracking_source`` is explicit: a row defaults to ``manual``, and manual
+    rows are never pruned, so a fixture that omitted it would silently test
+    nothing.
+    """
+    session.add_all([
+        Ticker(symbol="AAPL", tracking_source=ticker_service.SOURCE_INDEX),
+        Ticker(symbol="LEAVER", tracking_source=ticker_service.SOURCE_INDEX),
+    ])
+    await session.flush()
+
+    leaver = (
+        await session.execute(select(Ticker).where(Ticker.symbol == "LEAVER"))
+    ).scalar_one()
+    start = date(2026, 1, 5)
+    for i in range(10):
+        session.add(OHLCVRecord(
+            ticker_id=leaver.id,
+            date=start + timedelta(days=i),
+            open=100.0, high=101.0, low=99.0, close=100.5, volume=1_000_000,
+        ))
+    session.add(SRLevel(
+        ticker_id=leaver.id, price_level=95.0, type="support", strength=70,
+        detection_method="swing",
+    ))
+    await session.commit()
+
+
+async def _prune_to(session: AsyncSession, monkeypatch, keep: list[str]) -> dict:
+    async def fake_fetch(db, universe):
+        # A LIVE source label: pruning and re-entry act only on a successful
+        # fetch, so a made-up label would silently make these tests vacuous.
+        return keep, "wikipedia_sp500"
+
+    monkeypatch.setattr(tus, "fetch_universe_symbols", fake_fetch)
+    return await tus.bootstrap_universe(session, "sp500", prune_missing=True)
+
+
+async def test_pruning_preserves_bars_and_linked_records(session: AsyncSession, monkeypatch):
+    await _seed(session)
+    summary = await _prune_to(session, monkeypatch, ["AAPL"])
+
+    assert summary["untracked"] == 1
+    assert summary["untracked_symbols"] == ["LEAVER"]
+    assert summary["deleted"] == 0
+
+    leaver = (
+        await session.execute(select(Ticker).where(Ticker.symbol == "LEAVER"))
+    ).scalar_one_or_none()
+    assert leaver is not None, "the row itself must survive the prune"
+    # The database stamps this (see bootstrap_universe), and SQLite's
+    # CURRENT_DATE is UTC while date.today() is local — so allow the one-day
+    # window rather than making this fail around midnight.
+    assert leaver.untracked_on is not None
+    assert abs((leaver.untracked_on - date.today()).days) <= 1
+    # Untracked is not delisted: the security still trades.
+    assert leaver.delisted_on is None
+
+    bars = (await session.execute(
+        select(OHLCVRecord).where(OHLCVRecord.ticker_id == leaver.id)
+    )).scalars().all()
+    assert len(bars) == 10, "the delete used to cascade these away"
+
+    levels = (await session.execute(
+        select(SRLevel).where(SRLevel.ticker_id == leaver.id)
+    )).scalars().all()
+    assert len(levels) == 1, "every cascade relationship must survive, not just bars"
+
+
+async def test_a_pruned_symbol_leaves_the_live_path(session: AsyncSession, monkeypatch):
+    await _seed(session)
+    await _prune_to(session, monkeypatch, ["AAPL"])
+
+    live = await session.execute(
+        ticker_service.active_only(select(Ticker.symbol).order_by(Ticker.symbol))
+    )
+    assert list(live.scalars().all()) == ["AAPL"]
+
+    # ...while the registry still shows it, so the state is visible rather than
+    # silently absent.
+    everything = await session.execute(select(Ticker.symbol).order_by(Ticker.symbol))
+    assert list(everything.scalars().all()) == ["AAPL", "LEAVER"]
+
+
+async def test_todays_membership_does_not_erase_yesterdays_candidates(
+    session: AsyncSession, monkeypatch
+):
+    """The backtest replays stored history, so it must see every symbol that has
+    bars — including ones that have since left the index. This guards
+    ``backtest_service``'s bare universe query: "fixing the inconsistency" by
+    adding ``active_only`` there is exactly how survivorship bias gets
+    reintroduced.
+    """
+    await _seed(session)
+    await _prune_to(session, monkeypatch, ["AAPL"])
+
+    from app.services import backtest_service as bt
+
+    symbols = await bt._load_backtest_symbols(session)
+    assert "LEAVER" in symbols, (
+        "the backtest universe must include untracked symbols — their bars are "
+        "exactly the history that keeps a replay survivorship-honest"
+    )
+    assert symbols == ["AAPL", "LEAVER"]
+
+
+async def test_a_symbol_returning_to_the_index_rejoins_the_scanner(
+    session: AsyncSession, monkeypatch
+):
+    """Re-tracking is the failure this refactor could introduce: a returning
+    symbol is already tracked, so it is neither added nor untracked, and without
+    an explicit clear it would keep its old mark and stay invisible forever."""
+    await _seed(session)
+    await _prune_to(session, monkeypatch, ["AAPL"])
+
+    summary = await _prune_to(session, monkeypatch, ["AAPL", "LEAVER"])
+    assert summary["retracked"] == 1
+    assert summary["added"] == 0, "the row was never deleted, so there is nothing to add"
+
+    live = await session.execute(
+        ticker_service.active_only(select(Ticker.symbol).order_by(Ticker.symbol))
+    )
+    assert list(live.scalars().all()) == ["AAPL", "LEAVER"]
+
+    leaver = (
+        await session.execute(select(Ticker).where(Ticker.symbol == "LEAVER"))
+    ).scalar_one()
+    assert leaver.untracked_on is None
+    bars = (await session.execute(
+        select(OHLCVRecord).where(OHLCVRecord.ticker_id == leaver.id)
+    )).scalars().all()
+    assert len(bars) == 10, "it rejoins with its history, not as a blank new row"
+
+
+async def test_a_second_prune_does_not_restamp_the_date(session: AsyncSession, monkeypatch):
+    """The mark records when the symbol left, so a daily bootstrap must not keep
+    moving it forward — that would erase the only record of when it happened."""
+    await _seed(session)
+    await _prune_to(session, monkeypatch, ["AAPL"])
+
+    leaver = (
+        await session.execute(select(Ticker).where(Ticker.symbol == "LEAVER"))
+    ).scalar_one()
+    leaver.untracked_on = date(2026, 1, 15)
+    await session.commit()
+
+    summary = await _prune_to(session, monkeypatch, ["AAPL"])
+    assert summary["untracked"] == 0, "already-marked rows are not re-marked"
+
+    await session.refresh(leaver)
+    assert leaver.untracked_on == date(2026, 1, 15)
+
+
+@pytest.mark.parametrize("column", ["delisted_on", "untracked_on", "scan_disabled_on"])
+async def test_as_of_answers_what_was_live_then_for_both_columns(
+    session: AsyncSession, column: str
+):
+    """``as_of`` exists so a historical query can ask what was live at a date.
+    An untracked clause that ignored it would break that for the new column."""
+    session.add(Ticker(symbol="AAPL"))
+    session.add(Ticker(symbol="LEFT", **{column: date(2026, 6, 1)}))
+    await session.commit()
+
+    before = await session.execute(
+        ticker_service.active_only(select(Ticker.symbol), as_of=date(2026, 5, 1))
+    )
+    assert sorted(before.scalars().all()) == ["AAPL", "LEFT"]
+
+    after = await session.execute(
+        ticker_service.active_only(select(Ticker.symbol), as_of=date(2026, 7, 1))
+    )
+    assert sorted(after.scalars().all()) == ["AAPL"]
+
+
+async def test_a_symbol_can_be_both_untracked_and_delisted(session: AsyncSession):
+    """The state a name reaches when it leaves the index and later stops
+    trading. The two clauses are independent, so this must not double-count or
+    resurrect it."""
+    session.add(Ticker(
+        symbol="BOTH", untracked_on=date(2026, 3, 1), delisted_on=date(2026, 6, 1)
+    ))
+    session.add(Ticker(symbol="AAPL"))
+    await session.commit()
+
+    live = await session.execute(
+        ticker_service.active_only(select(Ticker.symbol), as_of=date(2026, 7, 1))
+    )
+    assert sorted(live.scalars().all()) == ["AAPL"]
+
+    # Before either date it was still in the live path.
+    early = await session.execute(
+        ticker_service.active_only(select(Ticker.symbol), as_of=date(2026, 1, 1))
+    )
+    assert sorted(early.scalars().all()) == ["AAPL", "BOTH"]
+
+    # Between them: out of the scanner already, but not yet delisted.
+    between = await session.execute(
+        ticker_service.active_only(select(Ticker.symbol), as_of=date(2026, 4, 1))
+    )
+    assert sorted(between.scalars().all()) == ["AAPL"]
+
+
+def test_every_active_only_call_site_carries_both_predicates():
+    """``active_only`` grew from one ``.where()`` to two. Call sites that chain
+    ``group_by``/``order_by`` onto its result, or pass multi-column selects,
+    must still get both clauses ANDed — a silently dropped predicate would put
+    untracked symbols back in the live path with nothing to show for it.
+    """
+    from sqlalchemy import case, func
+    from sqlalchemy.dialects import postgresql
+
+    from app.models.ohlcv import OHLCVRecord
+
+    latest = func.max(OHLCVRecord.date)
+    call_sites = {
+        # scheduler.py _get_all_tickers
+        "plain": select(Ticker.symbol).order_by(Ticker.symbol),
+        # sec_universe.py — multi-column select
+        "multi_column": select(Ticker.id, Ticker.symbol, Ticker.cik),
+        # rr_scanner_service.py — whole-entity select
+        "entity": select(Ticker).order_by(Ticker.symbol),
+    }
+    compiled = {
+        name: " ".join(str(
+            ticker_service.active_only(stmt).compile(dialect=postgresql.dialect())
+        ).split())
+        for name, stmt in call_sites.items()
+    }
+    # scheduler.py _get_ohlcv_priority_tickers chains OUTSIDE the call.
+    compiled["chained"] = " ".join(str(
+        ticker_service.active_only(
+            select(Ticker.symbol).outerjoin(OHLCVRecord, OHLCVRecord.ticker_id == Ticker.id)
+        )
+        .group_by(Ticker.id, Ticker.symbol)
+        .order_by(case((latest.is_(None), 0), else_=1).asc(), Ticker.symbol.asc())
+        .compile(dialect=postgresql.dialect())
+    ).split())
+
+    for name, sql in compiled.items():
+        assert "tickers.delisted_on IS NULL" in sql, name
+        assert "tickers.untracked_on IS NULL" in sql, name
+        assert "tickers.scan_disabled_on IS NULL" in sql, name
+        # ANDed together, not collapsed into one OR that would let either alone
+        # re-admit the symbol.
+        assert ") AND (" in sql[sql.index("WHERE"):], name
+
+
+async def test_a_symbol_can_be_untracked_individually(session: AsyncSession):
+    """The case RHM is: a manually added symbol no provider can serve. Without a
+    per-symbol control the only way to untrack it is a whole-universe prune,
+    which would sweep every other manual addition with it."""
+    await _seed(session)
+
+    changed = await ticker_service.set_tracked(session, "LEAVER", tracked=False)
+    assert changed is True
+
+    live = await session.execute(
+        ticker_service.active_only(select(Ticker.symbol).order_by(Ticker.symbol))
+    )
+    assert list(live.scalars().all()) == ["AAPL"]
+
+    leaver = (
+        await session.execute(select(Ticker).where(Ticker.symbol == "LEAVER"))
+    ).scalar_one()
+    # The operator's switch, not the prune flag: it has to survive a
+    # prune -> re-entry cycle, which clears untracked_on.
+    assert leaver.scan_disabled_on is not None
+    assert leaver.untracked_on is None
+    # Switching off is not delisting; the security still trades.
+    assert leaver.delisted_on is None
+    bars = (await session.execute(
+        select(OHLCVRecord).where(OHLCVRecord.ticker_id == leaver.id)
+    )).scalars().all()
+    assert len(bars) == 10
+
+
+async def test_setting_tracking_is_idempotent_and_reversible(session: AsyncSession):
+    await _seed(session)
+
+    assert await ticker_service.set_tracked(session, "LEAVER", tracked=False) is True
+    assert await ticker_service.set_tracked(session, "LEAVER", tracked=False) is False
+
+    assert await ticker_service.set_tracked(session, "LEAVER", tracked=True) is True
+    live = await session.execute(
+        ticker_service.active_only(select(Ticker.symbol).order_by(Ticker.symbol))
+    )
+    assert list(live.scalars().all()) == ["AAPL", "LEAVER"]
+
+
+async def test_untracking_an_unknown_symbol_raises(session: AsyncSession):
+    from app.exceptions import NotFoundError
+
+    with pytest.raises(NotFoundError):
+        await ticker_service.set_tracked(session, "NOPE", tracked=False)
+
+
+async def test_untracking_does_not_clear_a_data_quality_warning(session: AsyncSession):
+    """Untracking is not a way to silence the completeness check.
+
+    The backtest deliberately includes untracked symbols, so an untracked ticker
+    with no bars still counts in ``no_data`` and still reports ``degraded``. If
+    it did not, "untrack it" would become a way to make coverage gaps disappear
+    from the report — which is the exact failure the ledger exists to prevent.
+    """
+    from app.services import backtest_service as bt
+
+    session.add_all([Ticker(symbol="AAPL"), Ticker(symbol="NOBARS")])
+    await session.commit()
+    await ticker_service.set_tracked(session, "NOBARS", tracked=False)
+
+    # Still in the backtest universe...
+    assert await bt._load_backtest_symbols(session) == ["AAPL", "NOBARS"]
+    # ...and still unreadable, so still a coverage gap.
+    assert await bt._fetch_columns(session, "NOBARS") is None
+
+    quality = bt._DataQuality(expected=2, evaluated=1)
+    quality.no_data_symbols.append("NOBARS")
+    quality.record_benchmark({date(2026, 1, 5): 470.0}, None)
+    report = quality.as_report()
+
+    assert report["status"] == "degraded"
+    assert report["no_data_symbols"] == ["NOBARS"]
+    assert "1 symbol had no stored bars (NOBARS)" in report["reasons"]
+
+
+# ---------------------------------------------------------------------------
+# Tracking ownership (item 3)
+# ---------------------------------------------------------------------------
+
+
+async def test_a_manual_addition_survives_index_pruning(session: AsyncSession, monkeypatch):
+    """The RHM case. A hand-added symbol is by definition absent from the
+    constituent list, so without the ownership filter every prune would sweep
+    it."""
+    await _seed(session)
+    session.add(Ticker(symbol="MINE", tracking_source=ticker_service.SOURCE_MANUAL))
+    await session.commit()
+
+    summary = await _prune_to(session, monkeypatch, ["AAPL"])
+
+    assert summary["untracked_symbols"] == ["LEAVER"]
+    assert summary["kept_manual"] == ["MINE"]
+    assert summary["kept_manual_count"] == 1
+
+    live = await session.execute(
+        ticker_service.active_only(select(Ticker.symbol).order_by(Ticker.symbol))
+    )
+    assert list(live.scalars().all()) == ["AAPL", "MINE"]
+
+
+async def test_a_bootstrap_never_converts_a_manual_row_to_index_owned(
+    session: AsyncSession, monkeypatch
+):
+    """Ownership is sticky towards manual. If a bootstrap could claim a manual
+    row just because the symbol joined the index, the protection would vanish
+    the moment it left again — which is exactly when it is needed."""
+    await _seed(session)
+    session.add(Ticker(symbol="MINE", tracking_source=ticker_service.SOURCE_MANUAL))
+    await session.commit()
+
+    # MINE joins the index.
+    await _prune_to(session, monkeypatch, ["AAPL", "MINE"])
+    mine = (
+        await session.execute(select(Ticker).where(Ticker.symbol == "MINE"))
+    ).scalar_one()
+    assert mine.tracking_source == ticker_service.SOURCE_MANUAL
+
+    # ...and later leaves it again. Still protected.
+    summary = await _prune_to(session, monkeypatch, ["AAPL"])
+    assert summary["kept_manual"] == ["MINE"]
+    live = await session.execute(
+        ticker_service.active_only(select(Ticker.symbol).order_by(Ticker.symbol))
+    )
+    assert list(live.scalars().all()) == ["AAPL", "MINE"]
+
+
+async def test_a_manual_add_claims_an_index_owned_symbol(session: AsyncSession):
+    """An explicit "I want this stock" has to stop it being prunable."""
+    await _seed(session)
+
+    claimed = await ticker_service.add_ticker(session, "leaver")
+    assert claimed.tracking_source == ticker_service.SOURCE_MANUAL
+
+    # Claiming ownership is not the same as asking for it to be scanned, so the
+    # off-switches are left exactly as they were.
+    assert claimed.scan_disabled_on is None
+    assert claimed.untracked_on is None
+
+
+async def test_adding_an_already_manual_symbol_still_raises(session: AsyncSession):
+    from app.exceptions import DuplicateError
+
+    await _seed(session)
+    await ticker_service.add_ticker(session, "LEAVER")
+    with pytest.raises(DuplicateError):
+        await ticker_service.add_ticker(session, "LEAVER")
+
+
+async def test_re_entry_clears_the_prune_flag_but_never_the_operator_switch(
+    session: AsyncSession, monkeypatch
+):
+    """The conflation bug the extra column exists to prevent: one column serving
+    both writers means a prune -> re-entry cycle silently re-enables a symbol
+    somebody deliberately switched off."""
+    await _seed(session)
+    await ticker_service.set_tracked(session, "LEAVER", tracked=False)
+
+    await _prune_to(session, monkeypatch, ["AAPL"])          # leaves the index
+    summary = await _prune_to(session, monkeypatch, ["AAPL", "LEAVER"])  # returns
+
+    assert summary["retracked"] == 1
+    leaver = (
+        await session.execute(select(Ticker).where(Ticker.symbol == "LEAVER"))
+    ).scalar_one()
+    assert leaver.untracked_on is None, "the prune flag is cleared by re-entry"
+    assert leaver.scan_disabled_on is not None, "the operator's switch is not"
+
+    live = await session.execute(
+        ticker_service.active_only(select(Ticker.symbol).order_by(Ticker.symbol))
+    )
+    assert list(live.scalars().all()) == ["AAPL"]

tests/unit/test_universe_membership_intervals.py

@@ -0,0 +1,284 @@
+"""Observed membership intervals.
+
+The table answers "what could the strategy actually have selected on date D?"
+Three properties make that answer trustworthy:
+
+1. History starts at the first *observation*. Nothing invents an index effective
+   date for a symbol that was already tracked when recording began.
+2. Intervals are append-only. Leaving and returning produces two rows, so a
+   point-in-time query for the gap correctly answers "not a member".
+3. A failed universe fetch records nothing. The seed fallback is ~44 symbols, so
+   treating it as an observation would close ~460 memberships on a network blip.
+"""
+
+from __future__ import annotations
+
+from collections.abc import AsyncGenerator
+from datetime import date
+
+import pytest
+from sqlalchemy import select
+from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker, create_async_engine
+
+from app.database import Base
+from app.models.ticker import Ticker
+from app.models.universe_membership import UniverseMembership
+from app.services import ticker_service
+from app.services import ticker_universe_service as tus
+
+_engine = create_async_engine("sqlite+aiosqlite://", echo=False)
+_session_factory = async_sessionmaker(_engine, class_=AsyncSession, expire_on_commit=False)
+
+
+@pytest.fixture(autouse=True)
+async def _setup_tables() -> AsyncGenerator[None, None]:
+    async with _engine.begin() as conn:
+        await conn.run_sync(Base.metadata.create_all)
+    yield
+    async with _engine.begin() as conn:
+        await conn.run_sync(Base.metadata.drop_all)
+
+
+@pytest.fixture
+async def session() -> AsyncGenerator[AsyncSession, None]:
+    async with _session_factory() as s:
+        yield s
+
+
+async def _sync(session, monkeypatch, symbols: list[str], *, source: str = "wikipedia_sp500",
+                prune: bool = False) -> dict:
+    async def fake_fetch(db, universe):
+        return symbols, source
+
+    monkeypatch.setattr(tus, "fetch_universe_symbols", fake_fetch)
+    return await tus.bootstrap_universe(session, "sp500", prune_missing=prune)
+
+
+async def _intervals(session, symbol: str) -> list[tuple]:
+    rows = (await session.execute(
+        select(UniverseMembership.first_seen_on, UniverseMembership.absent_since)
+        .join(Ticker, Ticker.id == UniverseMembership.ticker_id)
+        .where(Ticker.symbol == symbol)
+        .order_by(UniverseMembership.id)
+    )).all()
+    return [tuple(r) for r in rows]
+
+
+async def test_a_live_sync_opens_intervals_without_pruning(session: AsyncSession, monkeypatch):
+    """The scheduled job runs prune_missing=False. If membership were only
+    recorded on prunes the history would be full of holes."""
+    summary = await _sync(session, monkeypatch, ["AAPL", "MSFT"])
+
+    assert summary["membership_observed"] is True
+    assert summary["membership_opened"] == 2
+    assert summary["membership_closed"] == 0
+    assert summary["untracked"] == 0, "recording membership is not pruning"
+
+    intervals = await _intervals(session, "AAPL")
+    assert len(intervals) == 1
+    first_seen, last_seen = intervals[0]
+    assert last_seen is None, "an open interval means still observed"
+    assert abs((first_seen - date.today()).days) <= 1
+
+
+async def test_a_second_sync_does_not_duplicate_an_open_interval(
+    session: AsyncSession, monkeypatch
+):
+    await _sync(session, monkeypatch, ["AAPL"])
+    summary = await _sync(session, monkeypatch, ["AAPL"])
+
+    assert summary["membership_opened"] == 0
+    assert len(await _intervals(session, "AAPL")) == 1
+
+
+async def test_leaving_closes_the_interval_and_returning_opens_a_new_one(
+    session: AsyncSession, monkeypatch
+):
+    await _sync(session, monkeypatch, ["AAPL", "LEAVER"])
+    closing = await _sync(session, monkeypatch, ["AAPL"])
+    assert closing["membership_closed"] == 1
+
+    intervals = await _intervals(session, "LEAVER")
+    assert len(intervals) == 1
+    assert intervals[0][1] is not None, "departure closes the interval"
+
+    returning = await _sync(session, monkeypatch, ["AAPL", "LEAVER"])
+    assert returning["membership_opened"] == 1
+
+    intervals = await _intervals(session, "LEAVER")
+    assert len(intervals) == 2, "re-entry appends; it does not reopen"
+    assert intervals[0][1] is not None, "the earlier interval keeps its end date"
+    assert intervals[1][1] is None
+
+
+async def test_a_failed_fetch_records_nothing(session: AsyncSession, monkeypatch):
+    """A cache or seed result means the live fetch failed. The seed list is a
+    fraction of the real universe — closing memberships from it would rewrite
+    history on a network blip."""
+    await _sync(session, monkeypatch, ["AAPL", "MSFT"])
+
+    for failed_source in ("cache", "seed"):
+        summary = await _sync(session, monkeypatch, ["AAPL"], source=failed_source)
+        assert summary["membership_observed"] is False
+        assert summary["membership_closed"] == 0
+        assert summary["membership_opened"] == 0
+
+    # MSFT's membership is untouched: we never looked, so we know nothing new.
+    msft = await _intervals(session, "MSFT")
+    assert len(msft) == 1
+    assert msft[0][1] is None, "still open — a failed fetch is not a departure"
+
+
+async def test_membership_is_recorded_for_manual_rows_too(session: AsyncSession, monkeypatch):
+    """Membership and tracking ownership are independent axes: a manually owned
+    symbol that is genuinely in the index still gets intervals."""
+    session.add(Ticker(symbol="MINE", tracking_source=ticker_service.SOURCE_MANUAL))
+    await session.commit()
+
+    await _sync(session, monkeypatch, ["AAPL", "MINE"])
+
+    mine = (
+        await session.execute(select(Ticker).where(Ticker.symbol == "MINE"))
+    ).scalar_one()
+    assert mine.tracking_source == ticker_service.SOURCE_MANUAL
+    assert len(await _intervals(session, "MINE")) == 1
+
+
+async def test_switching_off_scanning_does_not_close_a_membership(
+    session: AsyncSession, monkeypatch
+):
+    """The index still contains the symbol; we simply stopped looking at it.
+    Conflating those would corrupt the point-in-time answer."""
+    await _sync(session, monkeypatch, ["AAPL", "MSFT"])
+    await ticker_service.set_tracked(session, "MSFT", tracked=False)
+
+    summary = await _sync(session, monkeypatch, ["AAPL", "MSFT"])
+    assert summary["membership_closed"] == 0
+    assert (await _intervals(session, "MSFT"))[0][1] is None
+
+
+async def test_pruning_does_not_close_a_membership_by_itself(
+    session: AsyncSession, monkeypatch
+):
+    """Closing happens because the symbol was absent from a live list, not
+    because a prune ran — the two just usually coincide."""
+    await _sync(session, monkeypatch, ["AAPL", "LEAVER"])
+    session.add(Ticker(symbol="MINE", tracking_source=ticker_service.SOURCE_MANUAL))
+    await session.commit()
+
+    summary = await _sync(session, monkeypatch, ["AAPL"], prune=True)
+    # LEAVER closed because it was absent; MINE never had an interval to close.
+    assert summary["membership_closed"] == 1
+    assert await _intervals(session, "MINE") == []
+
+
+async def test_a_failed_fetch_does_not_untrack_anything(session: AsyncSession, monkeypatch):
+    """The seed fallback is ~44 symbols against a ~500-symbol universe. Pruning
+    from it would untrack the entire index on a single network blip — skipping
+    membership recording is only half the safeguard."""
+    session.add_all([
+        Ticker(symbol=s, tracking_source=ticker_service.SOURCE_INDEX)
+        for s in ("AAPL", "MSFT", "NVDA")
+    ])
+    await session.commit()
+
+    summary = await _sync(session, monkeypatch, ["AAPL"], source="seed", prune=True)
+
+    assert summary["untracked"] == 0, "a failed fetch must not drive pruning"
+    live = await session.execute(
+        ticker_service.active_only(select(Ticker.symbol).order_by(Ticker.symbol))
+    )
+    assert list(live.scalars().all()) == ["AAPL", "MSFT", "NVDA"]
+
+
+async def test_a_failed_fetch_does_not_clear_a_prune_flag(session: AsyncSession, monkeypatch):
+    """Re-entry is a claim about the index. A cache fallback is not evidence the
+    symbol rejoined it."""
+    session.add_all([
+        Ticker(symbol=s, tracking_source=ticker_service.SOURCE_INDEX)
+        for s in ("AAPL", "LEAVER")
+    ])
+    await session.commit()
+    await _sync(session, monkeypatch, ["AAPL"], prune=True)  # LEAVER untracked
+
+    summary = await _sync(session, monkeypatch, ["AAPL", "LEAVER"], source="cache")
+    assert summary["retracked"] == 0, "stale data must not re-admit a symbol"
+
+    leaver = (
+        await session.execute(select(Ticker).where(Ticker.symbol == "LEAVER"))
+    ).scalar_one()
+    assert leaver.untracked_on is not None
+
+
+# ---------------------------------------------------------------------------
+# Interval boundaries, stated as executable rules
+# ---------------------------------------------------------------------------
+
+
+async def _interval(session, symbol: str, universe: str, first: date, absent: date | None):
+    """Write one interval directly, so boundary tests don't depend on today."""
+    ticker = (
+        await session.execute(select(Ticker).where(Ticker.symbol == symbol))
+    ).scalar_one_or_none()
+    if ticker is None:
+        ticker = Ticker(symbol=symbol, tracking_source=ticker_service.SOURCE_INDEX)
+        session.add(ticker)
+        await session.flush()
+    session.add(UniverseMembership(
+        ticker_id=ticker.id, universe=universe,
+        first_seen_on=first, absent_since=absent,
+    ))
+    await session.commit()
+
+
+async def test_membership_bounds_are_lower_inclusive_upper_exclusive(session: AsyncSession):
+    """``absent_since`` is the first observation that found the symbol GONE, so
+    on that date it is already absent. ``first_seen_on`` is the first that found
+    it present, so on that date it is already a member."""
+    await _interval(session, "EDGE", "sp500", date(2026, 3, 10), date(2026, 8, 20))
+
+    async def members(d: date) -> list[str]:
+        return await tus.members_on(session, "sp500", d)
+
+    assert await members(date(2026, 3, 9)) == [], "the day before first observation"
+    assert await members(date(2026, 3, 10)) == ["EDGE"], "lower bound is inclusive"
+    assert await members(date(2026, 8, 19)) == ["EDGE"], "still present the day before"
+    assert await members(date(2026, 8, 20)) == [], "upper bound is exclusive"
+
+
+async def test_an_open_interval_has_no_upper_bound(session: AsyncSession):
+    await _interval(session, "OPEN", "sp500", date(2026, 3, 10), None)
+    assert await tus.members_on(session, "sp500", date(2030, 1, 1)) == ["OPEN"]
+
+
+async def test_a_gap_between_two_intervals_is_not_membership(session: AsyncSession):
+    """The reason re-entry appends instead of reopening."""
+    await _interval(session, "GAP", "sp500", date(2026, 1, 5), date(2026, 3, 2))
+    await _interval(session, "GAP", "sp500", date(2026, 8, 3), None)
+
+    assert await tus.members_on(session, "sp500", date(2026, 2, 1)) == ["GAP"]
+    assert await tus.members_on(session, "sp500", date(2026, 5, 1)) == []
+    assert await tus.members_on(session, "sp500", date(2026, 9, 1)) == ["GAP"]
+
+
+async def test_membership_is_scoped_per_universe(session: AsyncSession):
+    await _interval(session, "BOTH", "sp500", date(2026, 1, 5), None)
+    await _interval(session, "BOTH", "nasdaq100", date(2026, 6, 1), None)
+    await _interval(session, "SPONLY", "sp500", date(2026, 1, 5), None)
+
+    assert await tus.members_on(session, "sp500", date(2026, 7, 1)) == ["BOTH", "SPONLY"]
+    assert await tus.members_on(session, "nasdaq100", date(2026, 7, 1)) == ["BOTH"]
+    # Before it joined the Nasdaq-100 it was only an S&P name.
+    assert await tus.members_on(session, "nasdaq100", date(2026, 5, 1)) == []
+
+
+async def test_before_recording_began_returns_empty_not_the_current_set(
+    session: AsyncSession, monkeypatch
+):
+    """The honest limit of this table: a date before the first successful live
+    sync means "we were not looking", not "the index was empty". Answering with
+    today's set would be exactly the survivorship error item 3 exists to end."""
+    await _sync(session, monkeypatch, ["AAPL", "MSFT"])
+
+    assert await tus.members_on(session, "sp500", date(2020, 1, 2)) == []
+    assert await tus.members_on(session, "sp500", date.today()) == ["AAPL", "MSFT"]