feat(universe): separate scanner membership from retained history

bootstrap_universe(prune_missing=True) ran DELETE FROM tickers, cascading OHLCV,
setups, scores and S/R levels away for any symbol that had merely dropped out of
the index. Migration 032's own comment predicted this. The survivorship audit
measured what it costs: 2.09 Sharpe against 1.57 on universe composition alone.

Migration 035 adds untracked_on, the sibling of delisted_on — delisted means the
security stopped trading, untracked means a prune took it out of the scanner.
Nothing is deleted either way.

Migration 036 separates three more things that were still conflated:

  universe_memberships       what the index actually contained, when
  tickers.untracked_on       excluded from scanning by a prune
  tickers.scan_disabled_on   the operator's own "stop scanning this"
  tickers.tracking_source    who owns the row: a bootstrap, or an operator

The operator flag is its own column because with one column serving both
writers, a prune -> re-entry cycle would clear an explicit "off" — re-entry
cannot tell who set it. Re-entry clears untracked_on and never touches
scan_disabled_on.

untracked_on is NOT current index membership. The scheduled sync runs
prune_missing=False, so a symbol can have left the index and still be scanned;
the two legitimately differ and nothing asserts they agree.

Tracking ownership is sticky towards manual: prune touches index-owned rows
only, and a bootstrap never converts manual to index — a hand-added symbol that
joins the index would otherwise lose its protection the moment it left again,
which is exactly when it is needed. An explicit add of an index-owned symbol
claims it as manual without touching either off-switch.

Membership is observed, never asserted. first_seen_on is the first live
sighting; absent_since is the first observation that found the symbol GONE, so
the interval is [first_seen_on, absent_since) — named for what it stores rather
than "last_seen_on", which would have claimed the opposite. Intervals are
append-only, so a symbol that leaves and returns has two and a point-in-time
query for the gap correctly answers "not a member". members_on() implements the
boundary rule once. A date before the first live sync returns empty: that means
"we were not looking", not "the index was empty".

A failed fetch drives nothing. The cache and seed fallbacks mean the live fetch
failed, and the seed lists are ~44 symbols against a ~500-symbol universe — a
regression test caught a seed fallback untracking 2 of 3 tickers. Recording,
pruning and clearing prune flags all require a live source; only adding still
runs, so a cold start can still bootstrap from seeds.

Migration decisions: existing rows become manual (fail-safe — a wrong guess
costs an un-pruned symbol, not one swept out of the scanner), and existing
untracked_on values are copied into scan_disabled_on, since the original writer
is ambiguous and re-entry clears the prune column.

Backtest eligibility is unchanged. _load_backtest_symbols deliberately does not
use active_only, so today's scanner state cannot erase yesterday's candidates,
and untracking a symbol still cannot silence a coverage gap in the report.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HAH7vVtn7qbcn1EVheecHQ

tests/unit/test_ticker_universe_service.py

@@ -12,7 +12,7 @@ from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker, create_asyn
 from app.database import Base
 from app.models.settings import SystemSetting
 from app.models.ticker import Ticker
-from app.services import ticker_universe_service
+from app.services import ticker_service, ticker_universe_service
 from app.services.ticker_universe_service import _extract_wiki_symbols, _normalise_symbols
 
 _engine = create_async_engine("sqlite+aiosqlite://", echo=False)
@@ -58,11 +58,17 @@ async def test_bootstrap_universe_adds_missing_symbols(session: AsyncSession, mo
 
 @pytest.mark.asyncio
 async def test_bootstrap_universe_prunes_missing_symbols(session: AsyncSession, monkeypatch: pytest.MonkeyPatch):
-    session.add_all([Ticker(symbol="AAPL"), Ticker(symbol="MSFT"), Ticker(symbol="TSLA")])
+    # Index-owned: a row defaults to manual, and manual rows are never pruned.
+    session.add_all([
+        Ticker(symbol=s, tracking_source=ticker_service.SOURCE_INDEX)
+        for s in ("AAPL", "MSFT", "TSLA")
+    ])
     await session.commit()
 
     async def _fake_fetch(_db: AsyncSession, _universe: str) -> tuple[list[str], str]:
-        return ["AAPL", "MSFT"], "test"
+        # A live source label: a fallback result must not drive pruning, so a
+        # made-up label would make this test vacuous.
+        return ["AAPL", "MSFT"], "wikipedia_sp500"
 
     monkeypatch.setattr(ticker_universe_service, "fetch_universe_symbols", _fake_fetch)
 
@@ -74,10 +80,18 @@ async def test_bootstrap_universe_prunes_missing_symbols(session: AsyncSession,
 
     assert result["added"] == 0
     assert result["already_tracked"] == 2
-    assert result["deleted"] == 1
+    assert result["untracked"] == 1
+    assert result["untracked_symbols"] == ["TSLA"]
+    # Nothing is destroyed any more, and the key says so rather than vanishing.
+    assert result["deleted"] == 0
 
+    # The row survives the prune; only its scanner membership changed.
     rows = await session.execute(select(Ticker.symbol).order_by(Ticker.symbol.asc()))
-    assert list(rows.scalars().all()) == ["AAPL", "MSFT"]
+    assert list(rows.scalars().all()) == ["AAPL", "MSFT", "TSLA"]
+    live = await session.execute(
+        ticker_service.active_only(select(Ticker.symbol).order_by(Ticker.symbol.asc()))
+    )
+    assert list(live.scalars().all()) == ["AAPL", "MSFT"]
 
 
 @pytest.mark.asyncio