feat(universe): separate scanner membership from retained history

bootstrap_universe(prune_missing=True) ran DELETE FROM tickers, cascading OHLCV,
setups, scores and S/R levels away for any symbol that had merely dropped out of
the index. Migration 032's own comment predicted this. The survivorship audit
measured what it costs: 2.09 Sharpe against 1.57 on universe composition alone.

Migration 035 adds untracked_on, the sibling of delisted_on — delisted means the
security stopped trading, untracked means a prune took it out of the scanner.
Nothing is deleted either way.

Migration 036 separates three more things that were still conflated:

  universe_memberships       what the index actually contained, when
  tickers.untracked_on       excluded from scanning by a prune
  tickers.scan_disabled_on   the operator's own "stop scanning this"
  tickers.tracking_source    who owns the row: a bootstrap, or an operator

The operator flag is its own column because with one column serving both
writers, a prune -> re-entry cycle would clear an explicit "off" — re-entry
cannot tell who set it. Re-entry clears untracked_on and never touches
scan_disabled_on.

untracked_on is NOT current index membership. The scheduled sync runs
prune_missing=False, so a symbol can have left the index and still be scanned;
the two legitimately differ and nothing asserts they agree.

Tracking ownership is sticky towards manual: prune touches index-owned rows
only, and a bootstrap never converts manual to index — a hand-added symbol that
joins the index would otherwise lose its protection the moment it left again,
which is exactly when it is needed. An explicit add of an index-owned symbol
claims it as manual without touching either off-switch.

Membership is observed, never asserted. first_seen_on is the first live
sighting; absent_since is the first observation that found the symbol GONE, so
the interval is [first_seen_on, absent_since) — named for what it stores rather
than "last_seen_on", which would have claimed the opposite. Intervals are
append-only, so a symbol that leaves and returns has two and a point-in-time
query for the gap correctly answers "not a member". members_on() implements the
boundary rule once. A date before the first live sync returns empty: that means
"we were not looking", not "the index was empty".

A failed fetch drives nothing. The cache and seed fallbacks mean the live fetch
failed, and the seed lists are ~44 symbols against a ~500-symbol universe — a
regression test caught a seed fallback untracking 2 of 3 tickers. Recording,
pruning and clearing prune flags all require a live source; only adding still
runs, so a cold start can still bootstrap from seeds.

Migration decisions: existing rows become manual (fail-safe — a wrong guess
costs an un-pruned symbol, not one swept out of the scanner), and existing
untracked_on values are copied into scan_disabled_on, since the original writer
is ambiguous and re-entry clears the prune column.

Backtest eligibility is unchanged. _load_backtest_symbols deliberately does not
use active_only, so today's scanner state cannot erase yesterday's candidates,
and untracking a symbol still cannot silence a coverage gap in the report.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HAH7vVtn7qbcn1EVheecHQ

tests/unit/test_migration_036.py

@@ -0,0 +1,132 @@
+"""Migration 036: provenance, the operator switch, and membership intervals.
+
+The migration makes two irreversible-feeling judgement calls, and both need to
+be pinned:
+
+1. **Existing rows become ``manual``.** Provenance cannot be recovered from
+   ``created_at``, and manual is the fail-safe guess — manual rows are never
+   pruned, so a wrong guess costs an un-pruned symbol rather than one swept out
+   of the scanner.
+2. **Existing "off" states are carried into ``scan_disabled_on``.** Whoever set
+   ``untracked_on`` is ambiguous, and re-entry clears the prune flag. Copying it
+   is what stops an automated sync reactivating something switched off by hand.
+"""
+
+from __future__ import annotations
+
+import importlib.util
+from pathlib import Path
+
+import sqlalchemy as sa
+from alembic.migration import MigrationContext
+from alembic.operations import Operations
+
+
+def _load_migration_module():
+    path = (
+        Path(__file__).resolve().parents[2]
+        / "alembic"
+        / "versions"
+        / "036_universe_membership.py"
+    )
+    spec = importlib.util.spec_from_file_location("migration_036", path)
+    assert spec is not None and spec.loader is not None
+    module = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(module)
+    return module
+
+
+def _tickers_table(connection: sa.Connection) -> None:
+    """The schema as migration 035 leaves it."""
+    connection.execute(sa.text(
+        """
+        CREATE TABLE tickers (
+            id INTEGER PRIMARY KEY,
+            symbol VARCHAR(10) NOT NULL,
+            delisted_on DATE,
+            untracked_on DATE
+        )
+        """
+    ))
+    connection.execute(sa.text(
+        "INSERT INTO tickers (id, symbol, delisted_on, untracked_on) VALUES "
+        "(1, 'AAPL', NULL, NULL), "
+        "(2, 'OFF', NULL, '2026-08-01')"
+    ))
+
+
+def _columns(connection: sa.Connection, table: str) -> set[str]:
+    return {row[1] for row in connection.execute(sa.text(f"PRAGMA table_info({table})"))}
+
+
+def test_upgrade_defaults_existing_rows_to_manual():
+    migration = _load_migration_module()
+    engine = sa.create_engine("sqlite://")
+
+    with engine.begin() as connection:
+        _tickers_table(connection)
+        with Operations.context(MigrationContext.configure(connection)):
+            migration.upgrade()
+
+        rows = dict(connection.execute(
+            sa.text("SELECT symbol, tracking_source FROM tickers")
+        ).all())
+        assert rows == {"AAPL": "manual", "OFF": "manual"}
+
+
+def test_upgrade_carries_an_existing_off_state_into_the_operator_switch():
+    migration = _load_migration_module()
+    engine = sa.create_engine("sqlite://")
+
+    with engine.begin() as connection:
+        _tickers_table(connection)
+        with Operations.context(MigrationContext.configure(connection)):
+            migration.upgrade()
+
+        rows = dict(connection.execute(
+            sa.text("SELECT symbol, scan_disabled_on FROM tickers")
+        ).all())
+        # The ambiguous existing "off" is preserved on the column no sync clears.
+        assert rows["OFF"] == "2026-08-01"
+        # A row that was never off stays on.
+        assert rows["AAPL"] is None
+
+
+def test_upgrade_creates_the_membership_table_empty():
+    migration = _load_migration_module()
+    engine = sa.create_engine("sqlite://")
+
+    with engine.begin() as connection:
+        _tickers_table(connection)
+        with Operations.context(MigrationContext.configure(connection)):
+            migration.upgrade()
+
+        assert {"ticker_id", "universe", "first_seen_on", "absent_since"} <= _columns(
+            connection, "universe_memberships"
+        )
+        # History starts at first observation — the migration invents none.
+        count = connection.execute(
+            sa.text("SELECT COUNT(*) FROM universe_memberships")
+        ).scalar_one()
+        assert count == 0
+
+
+def test_downgrade_removes_the_new_state_and_keeps_the_tickers():
+    migration = _load_migration_module()
+    engine = sa.create_engine("sqlite://")
+
+    with engine.begin() as connection:
+        _tickers_table(connection)
+        with Operations.context(MigrationContext.configure(connection)):
+            migration.upgrade()
+        with Operations.context(MigrationContext.configure(connection)):
+            migration.downgrade()
+
+        columns = _columns(connection, "tickers")
+        assert "tracking_source" not in columns
+        assert "scan_disabled_on" not in columns
+        # 035's column and the rows themselves are untouched.
+        assert "untracked_on" in columns
+        assert connection.execute(
+            sa.text("SELECT symbol FROM tickers ORDER BY symbol")
+        ).all() == [("AAPL",), ("OFF",)]