dennisthiessenandClaude Opus 5
ad3e47dc50
bootstrap_universe(prune_missing=True) ran DELETE FROM tickers, cascading OHLCV, setups, scores and S/R levels away for any symbol that had merely dropped out of the index. Migration 032's own comment predicted this. The survivorship audit measured what it costs: 2.09 Sharpe against 1.57 on universe composition alone. Migration 035 adds untracked_on, the sibling of delisted_on — delisted means the security stopped trading, untracked means a prune took it out of the scanner. Nothing is deleted either way. Migration 036 separates three more things that were still conflated: universe_memberships what the index actually contained, when tickers.untracked_on excluded from scanning by a prune tickers.scan_disabled_on the operator's own "stop scanning this" tickers.tracking_source who owns the row: a bootstrap, or an operator The operator flag is its own column because with one column serving both writers, a prune -> re-entry cycle would clear an explicit "off" — re-entry cannot tell who set it. Re-entry clears untracked_on and never touches scan_disabled_on. untracked_on is NOT current index membership. The scheduled sync runs prune_missing=False, so a symbol can have left the index and still be scanned; the two legitimately differ and nothing asserts they agree. Tracking ownership is sticky towards manual: prune touches index-owned rows only, and a bootstrap never converts manual to index — a hand-added symbol that joins the index would otherwise lose its protection the moment it left again, which is exactly when it is needed. An explicit add of an index-owned symbol claims it as manual without touching either off-switch. Membership is observed, never asserted. first_seen_on is the first live sighting; absent_since is the first observation that found the symbol GONE, so the interval is [first_seen_on, absent_since) — named for what it stores rather than "last_seen_on", which would have claimed the opposite. Intervals are append-only, so a symbol that leaves and returns has two and a point-in-time query for the gap correctly answers "not a member". members_on() implements the boundary rule once. A date before the first live sync returns empty: that means "we were not looking", not "the index was empty". A failed fetch drives nothing. The cache and seed fallbacks mean the live fetch failed, and the seed lists are ~44 symbols against a ~500-symbol universe — a regression test caught a seed fallback untracking 2 of 3 tickers. Recording, pruning and clearing prune flags all require a live source; only adding still runs, so a cold start can still bootstrap from seeds. Migration decisions: existing rows become manual (fail-safe — a wrong guess costs an un-pruned symbol, not one swept out of the scanner), and existing untracked_on values are copied into scan_disabled_on, since the original writer is ambiguous and re-entry clears the prune column. Backtest eligibility is unchanged. _load_backtest_symbols deliberately does not use active_only, so today's scanner state cannot erase yesterday's candidates, and untracking a symbol still cannot silence a coverage gap in the report. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HAH7vVtn7qbcn1EVheecHQ
57 lines
2.3 KiB
Python
57 lines
2.3 KiB
Python
"""Separate scanner membership from retained history
|
|
|
|
Revision ID: 035
|
|
Revises: 034
|
|
Create Date: 2026-09-07 00:00:00.000000
|
|
|
|
Migration 032 stopped ``delete_ticker`` and the delisting path from destroying
|
|
history, but left the other half of the problem open — its own comment said so:
|
|
"Pruning a symbol that is merely no longer an index constituent still destroys
|
|
history; that needs a tracked/membership state separate from delisting."
|
|
|
|
``bootstrap_universe(prune_missing=True)`` still ran ``DELETE FROM tickers``,
|
|
cascading OHLCV, setups and scores away for any symbol that had merely dropped
|
|
out of the index. The survivorship audit measured what that costs: restoring 85
|
|
removed constituents moved the same window from 2.09 Sharpe / 51.6% CAGR to
|
|
1.57 / 38.1%. A prune that deletes is how a universe silently becomes the
|
|
survivors.
|
|
|
|
``untracked_on`` is the sibling of ``delisted_on``, and the distinction is the
|
|
point:
|
|
|
|
delisted_on the security stopped trading
|
|
untracked_on it still trades; we stopped scanning it
|
|
|
|
Both retain every row. NULL in both means "in the live path" — ``active_only``
|
|
filters on both, so scanning, ranking, scoring, breadth and ingestion skip an
|
|
untracked name, while the backtest's universe query deliberately does not, so
|
|
today's membership cannot erase yesterday's candidates.
|
|
|
|
Nullable and reversible: a symbol that returns to the index has ``untracked_on``
|
|
cleared by the next bootstrap and rejoins the live path with its history intact.
|
|
"""
|
|
from typing import Sequence, Union
|
|
|
|
from alembic import op
|
|
import sqlalchemy as sa
|
|
|
|
|
|
revision: str = "035"
|
|
down_revision: Union[str, None] = "034"
|
|
branch_labels: Union[str, Sequence[str], None] = None
|
|
depends_on: Union[str, Sequence[str], None] = None
|
|
|
|
|
|
def upgrade() -> None:
|
|
op.add_column("tickers", sa.Column("untracked_on", sa.Date(), nullable=True))
|
|
# Same reasoning as ix_tickers_delisted_on: the live path tests this
|
|
# predicate on every universe scan, and untracked rows accumulate.
|
|
op.create_index("ix_tickers_untracked_on", "tickers", ["untracked_on"])
|
|
|
|
|
|
def downgrade() -> None:
|
|
# Dropping the column re-admits every untracked symbol to the live path.
|
|
# The rows and their history survive either way; that is the whole point.
|
|
op.drop_index("ix_tickers_untracked_on", table_name="tickers")
|
|
op.drop_column("tickers", "untracked_on")
|