feat(universe): separate scanner membership from retained history

bootstrap_universe(prune_missing=True) ran DELETE FROM tickers, cascading OHLCV,
setups, scores and S/R levels away for any symbol that had merely dropped out of
the index. Migration 032's own comment predicted this. The survivorship audit
measured what it costs: 2.09 Sharpe against 1.57 on universe composition alone.

Migration 035 adds untracked_on, the sibling of delisted_on — delisted means the
security stopped trading, untracked means a prune took it out of the scanner.
Nothing is deleted either way.

Migration 036 separates three more things that were still conflated:

  universe_memberships       what the index actually contained, when
  tickers.untracked_on       excluded from scanning by a prune
  tickers.scan_disabled_on   the operator's own "stop scanning this"
  tickers.tracking_source    who owns the row: a bootstrap, or an operator

The operator flag is its own column because with one column serving both
writers, a prune -> re-entry cycle would clear an explicit "off" — re-entry
cannot tell who set it. Re-entry clears untracked_on and never touches
scan_disabled_on.

untracked_on is NOT current index membership. The scheduled sync runs
prune_missing=False, so a symbol can have left the index and still be scanned;
the two legitimately differ and nothing asserts they agree.

Tracking ownership is sticky towards manual: prune touches index-owned rows
only, and a bootstrap never converts manual to index — a hand-added symbol that
joins the index would otherwise lose its protection the moment it left again,
which is exactly when it is needed. An explicit add of an index-owned symbol
claims it as manual without touching either off-switch.

Membership is observed, never asserted. first_seen_on is the first live
sighting; absent_since is the first observation that found the symbol GONE, so
the interval is [first_seen_on, absent_since) — named for what it stores rather
than "last_seen_on", which would have claimed the opposite. Intervals are
append-only, so a symbol that leaves and returns has two and a point-in-time
query for the gap correctly answers "not a member". members_on() implements the
boundary rule once. A date before the first live sync returns empty: that means
"we were not looking", not "the index was empty".

A failed fetch drives nothing. The cache and seed fallbacks mean the live fetch
failed, and the seed lists are ~44 symbols against a ~500-symbol universe — a
regression test caught a seed fallback untracking 2 of 3 tickers. Recording,
pruning and clearing prune flags all require a live source; only adding still
runs, so a cold start can still bootstrap from seeds.

Migration decisions: existing rows become manual (fail-safe — a wrong guess
costs an un-pruned symbol, not one swept out of the scanner), and existing
untracked_on values are copied into scan_disabled_on, since the original writer
is ambiguous and re-entry clears the prune column.

Backtest eligibility is unchanged. _load_backtest_symbols deliberately does not
use active_only, so today's scanner state cannot erase yesterday's candidates,
and untracking a symbol still cannot silence a coverage gap in the report.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HAH7vVtn7qbcn1EVheecHQ

app/services/ticker_universe_service.py

@@ -12,17 +12,18 @@ import logging
 import os
 import re
 from collections.abc import Iterable
-from datetime import datetime, timezone
+from datetime import date, datetime, timezone
 from pathlib import Path
 
 import httpx
-from sqlalchemy import delete, select
+from sqlalchemy import func, or_, select, update
 from sqlalchemy.ext.asyncio import AsyncSession
 
 from app.config import settings
 from app.exceptions import ProviderError, ValidationError
 from app.models.ticker import Ticker
-from app.services import settings_store
+from app.models.universe_membership import UniverseMembership
+from app.services import settings_store, ticker_service
 
 logger = logging.getLogger(__name__)
 
@@ -333,6 +334,114 @@ async def backfill_ticker_names(db: AsyncSession, *, only_missing: bool = True)
     return {"updated": updated, "checked": len(targets), "unmatched": len(targets) - updated}
 
 
+# Sources that represent an actual look at the index today. A ``cache`` or
+# ``seed`` result means the live fetch FAILED — the seed lists are ~44 symbols,
+# so treating one as an observation would close ~460 memberships on a network
+# blip. Membership is only ever recorded from a live read.
+LIVE_UNIVERSE_SOURCES = frozenset({
+    "wikipedia_sp500", "wikipedia_nasdaq100", "nasdaq_trader", "public",
+})
+
+
+async def members_on(db: AsyncSession, universe: str, as_of: date) -> list[str]:
+    """Which symbols were observed in *universe* on *as_of*, alphabetically.
+
+    The point-in-time question item 3 exists to answer, with the interval
+    boundaries written down once as code rather than re-derived at each call
+    site:
+
+        member on D  <=>  first_seen_on <= D < absent_since (or absent_since IS NULL)
+
+    Lower bound inclusive, upper bound exclusive. ``absent_since`` is the first
+    observation that found the symbol *gone*, so on that date it was already
+    absent.
+
+    **This is observation history, not index history.** A date before the first
+    successful live sync returns an empty list — that is "we were not looking",
+    not "the index was empty", and the caller has to know the difference. The
+    reconstructed removals in ``scripts/data/sp500_removals.json`` are research
+    data and deliberately do not seed this table.
+    """
+    rows = await db.execute(
+        select(Ticker.symbol)
+        .join(UniverseMembership, UniverseMembership.ticker_id == Ticker.id)
+        .where(
+            UniverseMembership.universe == universe,
+            UniverseMembership.first_seen_on <= as_of,
+            or_(
+                UniverseMembership.absent_since.is_(None),
+                UniverseMembership.absent_since > as_of,
+            ),
+        )
+        .order_by(Ticker.symbol)
+    )
+    return list(rows.scalars().all())
+
+
+async def _observe_membership(
+    db: AsyncSession, universe: str, present: set[str]
+) -> dict[str, int]:
+    """Record what a live universe fetch just saw. Returns opened/closed counts.
+
+    Runs on every sync, not only when pruning: the scheduled job uses
+    ``prune_missing=False``, so membership recorded only on manual prunes would
+    be full of holes. Recording membership and deciding what to scan are
+    separate concerns, and this is the recording one — it never touches
+    ``untracked_on``, ``scan_disabled_on`` or ``tracking_source``, and it applies
+    to manually tracked symbols exactly as it does to index-owned ones.
+
+    Intervals are append-only. A symbol that leaves and returns gets a second
+    row, so a point-in-time query for the gap answers "not a member"; reopening
+    the first row would erase that.
+
+    Dates come from the database (``current_date``), matching every other date
+    this module writes. ``first_seen_on`` is the first *observation*, not an
+    index effective date — anything earlier is unknown and is not guessed.
+    """
+    rows = (await db.execute(
+        select(Ticker.id, Ticker.symbol).where(Ticker.symbol.in_(present))
+    )).all()
+    ticker_ids = {symbol: tid for tid, symbol in rows}
+
+    open_rows = (await db.execute(
+        select(UniverseMembership.ticker_id, Ticker.symbol)
+        .join(Ticker, Ticker.id == UniverseMembership.ticker_id)
+        .where(
+            UniverseMembership.universe == universe,
+            UniverseMembership.absent_since.is_(None),
+        )
+    )).all()
+    open_symbols = {symbol for _, symbol in open_rows}
+
+    opened = 0
+    for symbol in sorted(present - open_symbols):
+        ticker_id = ticker_ids.get(symbol)
+        if ticker_id is None:
+            continue
+        db.add(UniverseMembership(
+            ticker_id=ticker_id,
+            universe=universe,
+            first_seen_on=func.current_date(),
+        ))
+        opened += 1
+
+    departed_ids = [tid for tid, symbol in open_rows if symbol not in present]
+    closed = 0
+    if departed_ids:
+        result = await db.execute(
+            update(UniverseMembership)
+            .where(
+                UniverseMembership.universe == universe,
+                UniverseMembership.absent_since.is_(None),
+                UniverseMembership.ticker_id.in_(departed_ids),
+            )
+            .values(absent_since=func.current_date())
+        )
+        closed = int(result.rowcount or 0)
+
+    return {"opened": opened, "closed": closed}
+
+
 async def bootstrap_universe(
     db: AsyncSession,
     universe: str,
@@ -341,42 +450,98 @@ async def bootstrap_universe(
 ) -> dict[str, int | str]:
     """Upsert ticker universe into tracked tickers.
 
-    Returns summary counts for added/existing/deleted symbols.
+    ``prune_missing`` marks symbols no longer in the universe as untracked; it
+    never deletes them. "Not in today's scanner" and "not stored for historical
+    analysis" are different states, and conflating them is what makes a universe
+    quietly become the survivors. Symbols that have rejoined the universe have
+    the mark cleared and return to the live path with their history intact.
+
+    Returns summary counts for added / already-tracked / untracked / retracked.
     """
     normalised_universe = _validate_universe(universe)
     symbols, source = await fetch_universe_symbols(db, normalised_universe)
 
+    # A cache or seed result means the live fetch FAILED. The seed lists are ~44
+    # symbols against a ~500-symbol universe, so treating one as the truth would
+    # untrack the entire index on a single network blip. Nothing that acts on
+    # "this symbol is/isn't in the index" may run on fallback data: not pruning,
+    # not clearing a prune flag, not recording membership. Adding is the one
+    # exception — it destroys nothing and is what lets a cold start bootstrap
+    # from seeds.
+    observed = source in LIVE_UNIVERSE_SOURCES
+
     existing_rows = await db.execute(select(Ticker.symbol))
     existing_symbols = set(existing_rows.scalars().all())
     target_symbols = set(symbols)
 
     symbols_to_add = sorted(target_symbols - existing_symbols)
-    symbols_to_delete = sorted(existing_symbols - target_symbols) if prune_missing else []
+
+    # Only index-owned rows are prunable. A manual addition is by definition not
+    # in the constituent list, so without this filter every hand-added symbol
+    # would be swept by any prune.
+    prunable = set() if not (prune_missing and observed) else set((
+        await db.execute(
+            select(Ticker.symbol).where(
+                Ticker.symbol.in_(existing_symbols - target_symbols),
+                Ticker.tracking_source == ticker_service.SOURCE_INDEX,
+            )
+        )
+    ).scalars().all())
+    symbols_to_untrack = sorted(prunable)
+    kept_manual = (
+        sorted((existing_symbols - target_symbols) - prunable)
+        if (prune_missing and observed) else []
+    )
 
     for symbol in symbols_to_add:
-        db.add(Ticker(symbol=symbol))
+        db.add(Ticker(symbol=symbol, tracking_source=ticker_service.SOURCE_INDEX))
 
-    deleted_count = 0
-    skipped_delisted: list[str] = []
-    if symbols_to_delete:
-        # A delisted row was retained on purpose — its price history is exactly
-        # what a survivorship-honest backtest needs, and the delete cascades it
-        # away. Pruning must not undo that. (Pruning a symbol that is merely no
-        # longer an index constituent still destroys history; that needs a
-        # tracked/membership state separate from delisting.)
-        protected = (
-            await db.execute(
-                select(Ticker.symbol).where(
-                    Ticker.symbol.in_(symbols_to_delete),
-                    Ticker.delisted_on.is_not(None),
-                )
-            )
-        ).scalars().all()
-        skipped_delisted = sorted(protected)
-        deletable = [s for s in symbols_to_delete if s not in set(protected)]
-        if deletable:
-            result = await db.execute(delete(Ticker).where(Ticker.symbol.in_(deletable)))
-            deleted_count = int(result.rowcount or 0)
+    # A symbol that has returned to the index must rejoin the live path. It is
+    # already in ``existing_symbols``, so it is neither added nor untracked here
+    # — without this it would keep the flag from its earlier removal and stay
+    # invisible to the scanner forever, with nothing to show why.
+    #
+    # Clears the prune flag ONLY. ``scan_disabled_on`` is the operator's own
+    # decision and no automated run may undo it: a symbol someone switched off
+    # by hand must stay off across a prune -> re-entry cycle. Nor does this touch
+    # ``tracking_source`` — a manual row that joins the index stays manual, or it
+    # would lose its protection the next time it left.
+    retracked_count = 0
+    if observed:
+        retracked = await db.execute(
+            update(Ticker)
+            .where(Ticker.symbol.in_(target_symbols), Ticker.untracked_on.is_not(None))
+            .values(untracked_on=None)
+        )
+        retracked_count = int(retracked.rowcount or 0)
+
+    untracked_count = 0
+    if symbols_to_untrack:
+        # Marked, never deleted. The delete this replaced cascaded through
+        # OHLCV, setups and scores, which is precisely the history a
+        # survivorship-honest backtest needs — see migration 035. Rows already
+        # carrying a mark keep their original date rather than being re-stamped
+        # on every bootstrap.
+        #
+        # Stamped by the database, not date.today(): active_only compares this
+        # against func.current_date(), and a process in a timezone ahead of the
+        # database would otherwise write a date that reads as "effective
+        # tomorrow" and leave the symbol in the scanner for a day. Same clock on
+        # both sides by construction.
+        result = await db.execute(
+            update(Ticker)
+            .where(Ticker.symbol.in_(symbols_to_untrack), Ticker.untracked_on.is_(None))
+            .values(untracked_on=func.current_date())
+        )
+        untracked_count = int(result.rowcount or 0)
+
+    # Membership is observed on every live sync, independent of pruning and of
+    # who owns the row. A failed fetch must never close a membership.
+    membership = (
+        await _observe_membership(db, normalised_universe, target_symbols)
+        if observed
+        else {"opened": 0, "closed": 0}
+    )
 
     await db.commit()
 
@@ -393,10 +558,29 @@ async def bootstrap_universe(
         "total_universe_symbols": len(symbols),
         "added": len(symbols_to_add),
         "already_tracked": len(target_symbols & existing_symbols),
-        "deleted": deleted_count,
+        # Always 0. Kept so a caller reading this key is told plainly that
+        # pruning no longer destroys rows, rather than seeing the old key vanish
+        # and assuming nothing happened — or worse, seeing a count under
+        # "deleted" that now means something else entirely.
+        "deleted": 0,
+        "untracked": untracked_count,
+        "untracked_symbols": symbols_to_untrack[:50],
+        # Symbols that rejoined the index and were returned to the live path.
+        "retracked": retracked_count,
+        # Absent from the universe but manually owned, so a prune left them
+        # alone. The operator asked for these by hand.
+        "kept_manual": kept_manual[:50],
+        "kept_manual_count": len(kept_manual),
         "added_symbols": symbols_to_add[:50],
-        # Delisted rows a prune declined to destroy, so the caller can see the
-        # count did not match what they asked to remove.
-        "kept_delisted": skipped_delisted[:50],
-        "kept_delisted_count": len(skipped_delisted),
+        # Membership observation is skipped entirely on a cache/seed fallback,
+        # so the caller can tell "nothing changed" from "we never looked".
+        "membership_observed": observed,
+        "membership_opened": membership["opened"],
+        "membership_closed": membership["closed"],
+        "prune_note": (
+            "Pruning marks untracked_on and never deletes: the rows keep their "
+            "bars, setups and scores so historical backtests stay honest. "
+            "Untracked symbols leave the scanner, rankings and ingestion. "
+            "Manually added symbols are never pruned."
+        ),
     }