dennisthiessenandClaude Opus 5
ad3e47dc50
bootstrap_universe(prune_missing=True) ran DELETE FROM tickers, cascading OHLCV, setups, scores and S/R levels away for any symbol that had merely dropped out of the index. Migration 032's own comment predicted this. The survivorship audit measured what it costs: 2.09 Sharpe against 1.57 on universe composition alone. Migration 035 adds untracked_on, the sibling of delisted_on — delisted means the security stopped trading, untracked means a prune took it out of the scanner. Nothing is deleted either way. Migration 036 separates three more things that were still conflated: universe_memberships what the index actually contained, when tickers.untracked_on excluded from scanning by a prune tickers.scan_disabled_on the operator's own "stop scanning this" tickers.tracking_source who owns the row: a bootstrap, or an operator The operator flag is its own column because with one column serving both writers, a prune -> re-entry cycle would clear an explicit "off" — re-entry cannot tell who set it. Re-entry clears untracked_on and never touches scan_disabled_on. untracked_on is NOT current index membership. The scheduled sync runs prune_missing=False, so a symbol can have left the index and still be scanned; the two legitimately differ and nothing asserts they agree. Tracking ownership is sticky towards manual: prune touches index-owned rows only, and a bootstrap never converts manual to index — a hand-added symbol that joins the index would otherwise lose its protection the moment it left again, which is exactly when it is needed. An explicit add of an index-owned symbol claims it as manual without touching either off-switch. Membership is observed, never asserted. first_seen_on is the first live sighting; absent_since is the first observation that found the symbol GONE, so the interval is [first_seen_on, absent_since) — named for what it stores rather than "last_seen_on", which would have claimed the opposite. Intervals are append-only, so a symbol that leaves and returns has two and a point-in-time query for the gap correctly answers "not a member". members_on() implements the boundary rule once. A date before the first live sync returns empty: that means "we were not looking", not "the index was empty". A failed fetch drives nothing. The cache and seed fallbacks mean the live fetch failed, and the seed lists are ~44 symbols against a ~500-symbol universe — a regression test caught a seed fallback untracking 2 of 3 tickers. Recording, pruning and clearing prune flags all require a live source; only adding still runs, so a cold start can still bootstrap from seeds. Migration decisions: existing rows become manual (fail-safe — a wrong guess costs an un-pruned symbol, not one swept out of the scanner), and existing untracked_on values are copied into scan_disabled_on, since the original writer is ambiguous and re-entry clears the prune column. Backtest eligibility is unchanged. _load_backtest_symbols deliberately does not use active_only, so today's scanner state cannot erase yesterday's candidates, and untracking a symbol still cannot silence a coverage gap in the report. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HAH7vVtn7qbcn1EVheecHQ
73 lines
4.7 KiB
Python
73 lines
4.7 KiB
Python
from datetime import date, datetime
|
|
|
|
from sqlalchemy import Date, String, DateTime
|
|
from sqlalchemy.orm import Mapped, mapped_column, relationship
|
|
|
|
from app.database import Base
|
|
|
|
|
|
class Ticker(Base):
|
|
__tablename__ = "tickers"
|
|
|
|
id: Mapped[int] = mapped_column(primary_key=True)
|
|
symbol: Mapped[str] = mapped_column(String(10), unique=True, nullable=False)
|
|
# Company name (e.g. "Biogen Inc."); backfilled from Alpaca, nullable for
|
|
# symbols Alpaca doesn't know.
|
|
name: Mapped[str | None] = mapped_column(String(120), nullable=True)
|
|
# SEC issuer identity, refreshed by the SEC fundamentals import from
|
|
# company_tickers.json / submissions. The only ticker<->issuer join point;
|
|
# multi-class tickers (GOOG/GOOGL) share these values. Nullable: not every
|
|
# symbol resolves to a CIK (e.g. ADRs, foreign issuers not in SEC data).
|
|
cik: Mapped[str | None] = mapped_column(String(10), nullable=True)
|
|
sic: Mapped[str | None] = mapped_column(String(4), nullable=True)
|
|
sic_description: Mapped[str | None] = mapped_column(String(160), nullable=True)
|
|
# Delisting is recorded, never deleted: the rows carry the price history that
|
|
# makes a backtest less survivorship-biased, and a delete cascades it away.
|
|
# NULL == actively traded. The live signal path filters on this (see
|
|
# ticker_service.active_only); list/admin views keep the row and show it.
|
|
delisted_on: Mapped[date | None] = mapped_column(Date, nullable=True, index=True)
|
|
# How we learned: "form_25" (SEC confirmed), "manual" (operator).
|
|
delisted_reason: Mapped[str | None] = mapped_column(String(32), nullable=True)
|
|
# The sibling of delisted_on, and the distinction matters: delisted means the
|
|
# security stopped trading; untracked means it still trades and a prune took
|
|
# it out of the scanner. Both retain every row — the delete this replaced
|
|
# cascaded away exactly the history a survivorship-honest backtest needs.
|
|
#
|
|
# This records *exclusion by pruning*, NOT current index membership. The
|
|
# scheduled sync runs prune_missing=False, so a symbol can have left the
|
|
# index and still be scanned; the two states legitimately differ. Membership
|
|
# itself lives in universe_memberships.
|
|
#
|
|
# active_only filters on it; the backtest's universe query deliberately does
|
|
# not, so today's scanner state cannot erase yesterday's candidates.
|
|
untracked_on: Mapped[date | None] = mapped_column(Date, nullable=True, index=True)
|
|
# The operator's own "stop scanning this", kept apart from the prune flag on
|
|
# purpose. If one column served both, a prune -> re-entry cycle would clear
|
|
# an explicit operator "off", because re-entry cannot tell which writer set
|
|
# it. Only set_tracked writes this; no bootstrap ever clears it.
|
|
scan_disabled_on: Mapped[date | None] = mapped_column(Date, nullable=True, index=True)
|
|
# Who owns this row's tracking: "index" (created by a universe bootstrap) or
|
|
# "manual" (an operator added it). Pruning only ever touches index-owned
|
|
# rows. Ownership is sticky towards manual — a bootstrap must never convert
|
|
# manual to index, or a hand-added symbol that happens to join the index
|
|
# would lose its protection when it later leaves.
|
|
tracking_source: Mapped[str] = mapped_column(
|
|
String(16), nullable=False, server_default="manual", default="manual"
|
|
)
|
|
created_at: Mapped[datetime] = mapped_column(
|
|
DateTime(timezone=True), default=datetime.utcnow, nullable=False
|
|
)
|
|
|
|
# Relationships (cascade deletes)
|
|
ohlcv_records = relationship("OHLCVRecord", back_populates="ticker", cascade="all, delete-orphan")
|
|
sentiment_scores = relationship("SentimentScore", back_populates="ticker", cascade="all, delete-orphan")
|
|
fundamental_data = relationship("FundamentalData", back_populates="ticker", cascade="all, delete-orphan")
|
|
sr_levels = relationship("SRLevel", back_populates="ticker", cascade="all, delete-orphan")
|
|
dimension_scores = relationship("DimensionScore", back_populates="ticker", cascade="all, delete-orphan")
|
|
composite_scores = relationship("CompositeScore", back_populates="ticker", cascade="all, delete-orphan")
|
|
trade_setups = relationship("TradeSetup", back_populates="ticker", cascade="all, delete-orphan")
|
|
watchlist_entries = relationship("WatchlistEntry", back_populates="ticker", cascade="all, delete-orphan")
|
|
ingestion_progress = relationship("IngestionProgress", back_populates="ticker", cascade="all, delete-orphan", uselist=False)
|
|
earnings_events = relationship("EarningsEvent", back_populates="ticker", cascade="all, delete-orphan")
|
|
universe_memberships = relationship("UniverseMembership", back_populates="ticker", cascade="all, delete-orphan")
|