feat(universe): separate scanner membership from retained history

bootstrap_universe(prune_missing=True) ran DELETE FROM tickers, cascading OHLCV,
setups, scores and S/R levels away for any symbol that had merely dropped out of
the index. Migration 032's own comment predicted this. The survivorship audit
measured what it costs: 2.09 Sharpe against 1.57 on universe composition alone.

Migration 035 adds untracked_on, the sibling of delisted_on — delisted means the
security stopped trading, untracked means a prune took it out of the scanner.
Nothing is deleted either way.

Migration 036 separates three more things that were still conflated:

  universe_memberships       what the index actually contained, when
  tickers.untracked_on       excluded from scanning by a prune
  tickers.scan_disabled_on   the operator's own "stop scanning this"
  tickers.tracking_source    who owns the row: a bootstrap, or an operator

The operator flag is its own column because with one column serving both
writers, a prune -> re-entry cycle would clear an explicit "off" — re-entry
cannot tell who set it. Re-entry clears untracked_on and never touches
scan_disabled_on.

untracked_on is NOT current index membership. The scheduled sync runs
prune_missing=False, so a symbol can have left the index and still be scanned;
the two legitimately differ and nothing asserts they agree.

Tracking ownership is sticky towards manual: prune touches index-owned rows
only, and a bootstrap never converts manual to index — a hand-added symbol that
joins the index would otherwise lose its protection the moment it left again,
which is exactly when it is needed. An explicit add of an index-owned symbol
claims it as manual without touching either off-switch.

Membership is observed, never asserted. first_seen_on is the first live
sighting; absent_since is the first observation that found the symbol GONE, so
the interval is [first_seen_on, absent_since) — named for what it stores rather
than "last_seen_on", which would have claimed the opposite. Intervals are
append-only, so a symbol that leaves and returns has two and a point-in-time
query for the gap correctly answers "not a member". members_on() implements the
boundary rule once. A date before the first live sync returns empty: that means
"we were not looking", not "the index was empty".

A failed fetch drives nothing. The cache and seed fallbacks mean the live fetch
failed, and the seed lists are ~44 symbols against a ~500-symbol universe — a
regression test caught a seed fallback untracking 2 of 3 tickers. Recording,
pruning and clearing prune flags all require a live source; only adding still
runs, so a cold start can still bootstrap from seeds.

Migration decisions: existing rows become manual (fail-safe — a wrong guess
costs an un-pruned symbol, not one swept out of the scanner), and existing
untracked_on values are copied into scan_disabled_on, since the original writer
is ambiguous and re-entry clears the prune column.

Backtest eligibility is unchanged. _load_backtest_symbols deliberately does not
use active_only, so today's scanner state cannot erase yesterday's candidates,
and untracking a symbol still cannot silence a coverage gap in the report.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HAH7vVtn7qbcn1EVheecHQ

app/services/ticker_service.py

@@ -16,6 +16,13 @@ logger = logging.getLogger(__name__)
 REASON_FORM_25 = "form_25"   # SEC Form 25/25-NSE/15 confirmed the exchange exit
 REASON_MANUAL = "manual"     # an operator decided
 
+# Ticker.tracking_source — who owns this row's presence in the tracked set.
+# Ownership is sticky towards manual: a bootstrap creates ``index`` rows and an
+# operator's add claims a row as ``manual``, but nothing ever converts manual
+# back. Only index-owned rows are prunable.
+SOURCE_INDEX = "index"
+SOURCE_MANUAL = "manual"
+
 # How long a symbol must be without bars before we spend an SEC request asking
 # whether it delisted. Guards against a market-data outage probing the whole
 # universe at once; a real delisting is still stale days later.
@@ -43,21 +50,42 @@ def _sec_client_factory():
 
 
 def active_only(stmt, *, as_of: date | None = None):
-    """Restrict a Ticker query to symbols that still trade.
+    """Restrict a Ticker query to symbols in today's live signal path.
+
+    Three independent ways to leave it, and they mean different things:
+
+    ``delisted_on``       the security stopped trading
+    ``untracked_on``      a prune excluded it from scanning
+    ``scan_disabled_on``  an operator switched it off by hand
+
+    Any one of them excludes a symbol here. None deletes anything — the rows and
+    their history stay, which is what a survivorship-honest backtest needs.
+
+    ``untracked_on`` is not "no longer an index constituent": the scheduled sync
+    runs ``prune_missing=False``, so a symbol can have left the index and still
+    be scanned. Membership lives in ``universe_memberships`` and is not consulted
+    here — this stays a pure column filter so it costs nothing on the ~8 live
+    call sites that use it.
 
     Opt-in on purpose rather than folded into a shared getter: list and admin
-    views deliberately keep delisted rows so the delisting is *visible*, which a
+    views deliberately keep both kinds of row so the state is *visible*, which a
     silent default would undo. Apply this on the live signal path — scanning,
-    ranking, scoring, breadth, ingestion — and nowhere else.
+    ranking, scoring, breadth, ingestion — and nowhere else. In particular the
+    backtest's universe query must NOT use it: today's membership would silently
+    erase yesterday's candidates.
 
     ``delisted_on`` is an *effective* date, and a Form 25 is known ten days
     before it takes effect, so a future date must not drop the symbol yet — it
-    is still trading and still worth scanning and ingesting. Compared in SQL
-    against the database's own date; ``as_of`` overrides it for tests.
+    is still trading and still worth scanning and ingesting. The other two read
+    the same way, so ``as_of`` answers "what was in the scanner then?" for all
+    three rather than only one. Compared in SQL against the database's own date;
+    ``as_of`` overrides it for tests.
     """
     cutoff = func.current_date() if as_of is None else as_of
     return stmt.where(
-        or_(Ticker.delisted_on.is_(None), Ticker.delisted_on > cutoff)
+        or_(Ticker.delisted_on.is_(None), Ticker.delisted_on > cutoff),
+        or_(Ticker.untracked_on.is_(None), Ticker.untracked_on > cutoff),
+        or_(Ticker.scan_disabled_on.is_(None), Ticker.scan_disabled_on > cutoff),
     )
 
 
@@ -78,10 +106,27 @@ async def add_ticker(db: AsyncSession, symbol: str) -> Ticker:
         )
 
     result = await db.execute(select(Ticker).where(Ticker.symbol == normalised))
-    if result.scalar_one_or_none() is not None:
-        raise DuplicateError(f"Ticker already exists: {normalised}")
+    existing = result.scalar_one_or_none()
+    if existing is not None:
+        if existing.tracking_source == SOURCE_MANUAL:
+            raise DuplicateError(f"Ticker already exists: {normalised}")
+        # An explicit manual add of an index-owned symbol claims it: the operator
+        # wants this stock kept regardless of the index, so it must stop being
+        # prunable. Ownership only ever moves towards manual — a bootstrap never
+        # moves it back, or the protection would evaporate the moment the symbol
+        # rejoined the index.
+        #
+        # Deliberately does not touch scan_disabled_on or untracked_on: claiming
+        # ownership is not the same as asking for it to be scanned, and silently
+        # re-enabling something switched off by hand would undo a decision
+        # nobody revisited. ``set_tracked`` is the explicit control for that.
+        existing.tracking_source = SOURCE_MANUAL
+        await db.commit()
+        await db.refresh(existing)
+        logger.info("ticker %s claimed as manually tracked", normalised)
+        return existing
 
-    ticker = Ticker(symbol=normalised)
+    ticker = Ticker(symbol=normalised, tracking_source=SOURCE_MANUAL)
     db.add(ticker)
     await db.commit()
     await db.refresh(ticker)
@@ -229,6 +274,45 @@ async def confirm_delisting(
     return known
 
 
+async def set_tracked(db: AsyncSession, symbol: str, *, tracked: bool) -> bool:
+    """Include or exclude one symbol from the live path. True if it changed.
+
+    The per-symbol counterpart to ``bootstrap_universe(prune_missing=True)``,
+    which can only act on the whole universe at once — and would sweep every
+    manually added symbol along with it, since a manual add is by definition not
+    in the fetched constituent list.
+
+    Switching off is not delisting: the security still trades, we just stop
+    scanning it. Use this for a symbol that cannot produce data (no provider
+    coverage) or that an operator simply does not want scanned. The rows and
+    their history stay either way, so a backtest still sees them — untracking is
+    not a way to make a coverage gap disappear from the completeness report.
+
+    Writes ``scan_disabled_on``, never ``untracked_on``: the operator's decision
+    has to survive a prune -> re-entry cycle, which clears the prune flag.
+    """
+    normalised = symbol.strip().upper()
+    result = await db.execute(select(Ticker).where(Ticker.symbol == normalised))
+    ticker = result.scalar_one_or_none()
+    if ticker is None:
+        raise NotFoundError(f"Ticker not found: {normalised}")
+    if (ticker.scan_disabled_on is None) == tracked:
+        return False
+
+    await db.execute(
+        update(Ticker)
+        .where(Ticker.id == ticker.id)
+        # Stamped by the database for the same reason bootstrap_universe does:
+        # active_only compares it against func.current_date().
+        .values(scan_disabled_on=None if tracked else func.current_date())
+    )
+    await db.commit()
+    logger.info(
+        "ticker %s scanning %s", normalised, "enabled" if tracked else "disabled"
+    )
+    return True
+
+
 async def clear_delisted(db: AsyncSession, symbol: str) -> bool:
     """Un-retire a symbol. True if it had been marked.