dennisthiessenandClaude Opus 5
ad3e47dc50
bootstrap_universe(prune_missing=True) ran DELETE FROM tickers, cascading OHLCV, setups, scores and S/R levels away for any symbol that had merely dropped out of the index. Migration 032's own comment predicted this. The survivorship audit measured what it costs: 2.09 Sharpe against 1.57 on universe composition alone. Migration 035 adds untracked_on, the sibling of delisted_on — delisted means the security stopped trading, untracked means a prune took it out of the scanner. Nothing is deleted either way. Migration 036 separates three more things that were still conflated: universe_memberships what the index actually contained, when tickers.untracked_on excluded from scanning by a prune tickers.scan_disabled_on the operator's own "stop scanning this" tickers.tracking_source who owns the row: a bootstrap, or an operator The operator flag is its own column because with one column serving both writers, a prune -> re-entry cycle would clear an explicit "off" — re-entry cannot tell who set it. Re-entry clears untracked_on and never touches scan_disabled_on. untracked_on is NOT current index membership. The scheduled sync runs prune_missing=False, so a symbol can have left the index and still be scanned; the two legitimately differ and nothing asserts they agree. Tracking ownership is sticky towards manual: prune touches index-owned rows only, and a bootstrap never converts manual to index — a hand-added symbol that joins the index would otherwise lose its protection the moment it left again, which is exactly when it is needed. An explicit add of an index-owned symbol claims it as manual without touching either off-switch. Membership is observed, never asserted. first_seen_on is the first live sighting; absent_since is the first observation that found the symbol GONE, so the interval is [first_seen_on, absent_since) — named for what it stores rather than "last_seen_on", which would have claimed the opposite. Intervals are append-only, so a symbol that leaves and returns has two and a point-in-time query for the gap correctly answers "not a member". members_on() implements the boundary rule once. A date before the first live sync returns empty: that means "we were not looking", not "the index was empty". A failed fetch drives nothing. The cache and seed fallbacks mean the live fetch failed, and the seed lists are ~44 symbols against a ~500-symbol universe — a regression test caught a seed fallback untracking 2 of 3 tickers. Recording, pruning and clearing prune flags all require a live source; only adding still runs, so a cold start can still bootstrap from seeds. Migration decisions: existing rows become manual (fail-safe — a wrong guess costs an un-pruned symbol, not one swept out of the scanner), and existing untracked_on values are copied into scan_disabled_on, since the original writer is ambiguous and re-entry clears the prune column. Backtest eligibility is unchanged. _load_backtest_symbols deliberately does not use active_only, so today's scanner state cannot erase yesterday's candidates, and untracking a symbol still cannot silence a coverage gap in the report. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HAH7vVtn7qbcn1EVheecHQ
87 lines
2.9 KiB
Python
87 lines
2.9 KiB
Python
"""Migration 035 adds the untracked flag and stays reversible.
|
|
|
|
The downgrade matters more than usual here: dropping the column re-admits every
|
|
untracked symbol to the live path, and the rows survive either way. A migration
|
|
that could not roll back cleanly would make the prune change hard to undo on a
|
|
deploy that went wrong.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import importlib.util
|
|
from pathlib import Path
|
|
|
|
import sqlalchemy as sa
|
|
from alembic.migration import MigrationContext
|
|
from alembic.operations import Operations
|
|
|
|
|
|
def _load_migration_module():
|
|
path = (
|
|
Path(__file__).resolve().parents[2]
|
|
/ "alembic"
|
|
/ "versions"
|
|
/ "035_ticker_untracked.py"
|
|
)
|
|
spec = importlib.util.spec_from_file_location("migration_035", path)
|
|
assert spec is not None and spec.loader is not None
|
|
module = importlib.util.module_from_spec(spec)
|
|
spec.loader.exec_module(module)
|
|
return module
|
|
|
|
|
|
def _tickers_table(connection: sa.Connection) -> None:
|
|
connection.execute(sa.text(
|
|
"""
|
|
CREATE TABLE tickers (
|
|
id INTEGER PRIMARY KEY,
|
|
symbol VARCHAR(10) NOT NULL,
|
|
delisted_on DATE
|
|
)
|
|
"""
|
|
))
|
|
connection.execute(sa.text(
|
|
"INSERT INTO tickers (id, symbol, delisted_on) VALUES (1, 'AAPL', NULL)"
|
|
))
|
|
|
|
|
|
def _columns(connection: sa.Connection) -> set[str]:
|
|
return {row[1] for row in connection.execute(sa.text("PRAGMA table_info(tickers)"))}
|
|
|
|
|
|
def test_upgrade_adds_a_nullable_untracked_column_and_keeps_existing_rows():
|
|
migration = _load_migration_module()
|
|
engine = sa.create_engine("sqlite://")
|
|
|
|
with engine.begin() as connection:
|
|
_tickers_table(connection)
|
|
with Operations.context(MigrationContext.configure(connection)):
|
|
migration.upgrade()
|
|
|
|
assert "untracked_on" in _columns(connection)
|
|
# Existing rows default to NULL, i.e. still in the scanner — an upgrade
|
|
# must not silently drop the whole universe out of the live path.
|
|
rows = connection.execute(
|
|
sa.text("SELECT symbol, untracked_on FROM tickers")
|
|
).all()
|
|
assert rows == [("AAPL", None)]
|
|
|
|
|
|
def test_downgrade_removes_the_column_without_touching_the_rows():
|
|
migration = _load_migration_module()
|
|
engine = sa.create_engine("sqlite://")
|
|
|
|
with engine.begin() as connection:
|
|
_tickers_table(connection)
|
|
with Operations.context(MigrationContext.configure(connection)):
|
|
migration.upgrade()
|
|
connection.execute(sa.text(
|
|
"UPDATE tickers SET untracked_on = '2026-09-07' WHERE symbol = 'AAPL'"
|
|
))
|
|
with Operations.context(MigrationContext.configure(connection)):
|
|
migration.downgrade()
|
|
|
|
assert "untracked_on" not in _columns(connection)
|
|
# The ticker itself survives the rollback with its identity intact.
|
|
assert connection.execute(sa.text("SELECT symbol FROM tickers")).all() == [("AAPL",)]
|