dennisthiessen/signal-platform
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(universe): separate scanner membership from retained history
Deploy / lint (push) Successful in 43s
Details
Deploy / test (push) Failing after 2m9s
Details
Deploy / deploy (push) Skipped
Details

Browse Source 
bootstrap_universe(prune_missing=True) ran DELETE FROM tickers, cascading OHLCV,
setups, scores and S/R levels away for any symbol that had merely dropped out of
the index. Migration 032's own comment predicted this. The survivorship audit
measured what it costs: 2.09 Sharpe against 1.57 on universe composition alone.

Migration 035 adds untracked_on, the sibling of delisted_on — delisted means the
security stopped trading, untracked means a prune took it out of the scanner.
Nothing is deleted either way.

Migration 036 separates three more things that were still conflated:

  universe_memberships       what the index actually contained, when
  tickers.untracked_on       excluded from scanning by a prune
  tickers.scan_disabled_on   the operator's own "stop scanning this"
  tickers.tracking_source    who owns the row: a bootstrap, or an operator

The operator flag is its own column because with one column serving both
writers, a prune -> re-entry cycle would clear an explicit "off" — re-entry
cannot tell who set it. Re-entry clears untracked_on and never touches
scan_disabled_on.

untracked_on is NOT current index membership. The scheduled sync runs
prune_missing=False, so a symbol can have left the index and still be scanned;
the two legitimately differ and nothing asserts they agree.

Tracking ownership is sticky towards manual: prune touches index-owned rows
only, and a bootstrap never converts manual to index — a hand-added symbol that
joins the index would otherwise lose its protection the moment it left again,
which is exactly when it is needed. An explicit add of an index-owned symbol
claims it as manual without touching either off-switch.

Membership is observed, never asserted. first_seen_on is the first live
sighting; absent_since is the first observation that found the symbol GONE, so
the interval is [first_seen_on, absent_since) — named for what it stores rather
than "last_seen_on", which would have claimed the opposite. Intervals are
append-only, so a symbol that leaves and returns has two and a point-in-time
query for the gap correctly answers "not a member". members_on() implements the
boundary rule once. A date before the first live sync returns empty: that means
"we were not looking", not "the index was empty".

A failed fetch drives nothing. The cache and seed fallbacks mean the live fetch
failed, and the seed lists are ~44 symbols against a ~500-symbol universe — a
regression test caught a seed fallback untracking 2 of 3 tickers. Recording,
pruning and clearing prune flags all require a live source; only adding still
runs, so a cold start can still bootstrap from seeds.

Migration decisions: existing rows become manual (fail-safe — a wrong guess
costs an un-pruned symbol, not one swept out of the scanner), and existing
untracked_on values are copied into scan_disabled_on, since the original writer
is ambiguous and re-entry clears the prune column.

Backtest eligibility is unchanged. _load_backtest_symbols deliberately does not
use active_only, so today's scanner state cannot erase yesterday's candidates,
and untracking a symbol still cannot silence a coverage gap in the report.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HAH7vVtn7qbcn1EVheecHQ
This commit is contained in:
dennisthiessenandClaude Opus 5 committed 2026-09-07 19:30:05 +02:00
1 parent f2ef8e5438
commit ad3e47dc50
17 files changed
+2027 -57
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

app/services/ticker_service.py
+93 -9
View File
@@ -16,6 +16,13 @@ logger = logging.getLogger(__name__)
REASON_FORM_25 = "form_25" # SEC Form 25/25-NSE/15 confirmed the exchange exit
REASON_MANUAL = "manual" # an operator decided
# Ticker.tracking_source — who owns this row's presence in the tracked set.
# Ownership is sticky towards manual: a bootstrap creates ``index`` rows and an
# operator's add claims a row as ``manual``, but nothing ever converts manual
# back. Only index-owned rows are prunable.
SOURCE_INDEX = "index"
SOURCE_MANUAL = "manual"
# How long a symbol must be without bars before we spend an SEC request asking
# whether it delisted. Guards against a market-data outage probing the whole
# universe at once; a real delisting is still stale days later.
@@ -43,21 +50,42 @@ def _sec_client_factory():
def active_only(stmt, *, as_of: date | None = None):
"""Restrict a Ticker query to symbols that still trade.
"""Restrict a Ticker query to symbols in today's live signal path.
Three independent ways to leave it, and they mean different things:
``delisted_on`` the security stopped trading
``untracked_on`` a prune excluded it from scanning
``scan_disabled_on`` an operator switched it off by hand
Any one of them excludes a symbol here. None deletes anything — the rows and
their history stay, which is what a survivorship-honest backtest needs.
``untracked_on`` is not "no longer an index constituent": the scheduled sync
runs ``prune_missing=False``, so a symbol can have left the index and still
be scanned. Membership lives in ``universe_memberships`` and is not consulted
here — this stays a pure column filter so it costs nothing on the ~8 live
call sites that use it.
Opt-in on purpose rather than folded into a shared getter: list and admin
views deliberately keep delisted rows so the delisting is *visible*, which a
views deliberately keep both kinds of row so the state is *visible*, which a
silent default would undo. Apply this on the live signal path — scanning,
ranking, scoring, breadth, ingestion — and nowhere else.
ranking, scoring, breadth, ingestion — and nowhere else. In particular the
backtest's universe query must NOT use it: today's membership would silently
erase yesterday's candidates.
``delisted_on`` is an *effective* date, and a Form 25 is known ten days
before it takes effect, so a future date must not drop the symbol yet — it
is still trading and still worth scanning and ingesting. Compared in SQL
against the database's own date; ``as_of`` overrides it for tests.
is still trading and still worth scanning and ingesting. The other two read
the same way, so ``as_of`` answers "what was in the scanner then?" for all
three rather than only one. Compared in SQL against the database's own date;
``as_of`` overrides it for tests.
"""
cutoff = func.current_date() if as_of is None else as_of
return stmt.where(
or_(Ticker.delisted_on.is_(None), Ticker.delisted_on > cutoff)
or_(Ticker.delisted_on.is_(None), Ticker.delisted_on > cutoff),
or_(Ticker.untracked_on.is_(None), Ticker.untracked_on > cutoff),
or_(Ticker.scan_disabled_on.is_(None), Ticker.scan_disabled_on > cutoff),
)
@@ -78,10 +106,27 @@ async def add_ticker(db: AsyncSession, symbol: str) -> Ticker:
)
result = await db.execute(select(Ticker).where(Ticker.symbol == normalised))
if result.scalar_one_or_none() is not None:
raise DuplicateError(f"Ticker already exists: {normalised}")
existing = result.scalar_one_or_none()
if existing is not None:
if existing.tracking_source == SOURCE_MANUAL:
raise DuplicateError(f"Ticker already exists: {normalised}")
# An explicit manual add of an index-owned symbol claims it: the operator
# wants this stock kept regardless of the index, so it must stop being
# prunable. Ownership only ever moves towards manual — a bootstrap never
# moves it back, or the protection would evaporate the moment the symbol
# rejoined the index.
#
# Deliberately does not touch scan_disabled_on or untracked_on: claiming
# ownership is not the same as asking for it to be scanned, and silently
# re-enabling something switched off by hand would undo a decision
# nobody revisited. ``set_tracked`` is the explicit control for that.
existing.tracking_source = SOURCE_MANUAL
await db.commit()
await db.refresh(existing)
logger.info("ticker %s claimed as manually tracked", normalised)
return existing
ticker = Ticker(symbol=normalised)
ticker = Ticker(symbol=normalised, tracking_source=SOURCE_MANUAL)
db.add(ticker)
await db.commit()
await db.refresh(ticker)
@@ -229,6 +274,45 @@ async def confirm_delisting(
return known
async def set_tracked(db: AsyncSession, symbol: str, *, tracked: bool) -> bool:
"""Include or exclude one symbol from the live path. True if it changed.
The per-symbol counterpart to ``bootstrap_universe(prune_missing=True)``,
which can only act on the whole universe at once — and would sweep every
manually added symbol along with it, since a manual add is by definition not
in the fetched constituent list.
Switching off is not delisting: the security still trades, we just stop
scanning it. Use this for a symbol that cannot produce data (no provider
coverage) or that an operator simply does not want scanned. The rows and
their history stay either way, so a backtest still sees them — untracking is
not a way to make a coverage gap disappear from the completeness report.
Writes ``scan_disabled_on``, never ``untracked_on``: the operator's decision
has to survive a prune -> re-entry cycle, which clears the prune flag.
"""
normalised = symbol.strip().upper()
result = await db.execute(select(Ticker).where(Ticker.symbol == normalised))
ticker = result.scalar_one_or_none()
if ticker is None:
raise NotFoundError(f"Ticker not found: {normalised}")
if (ticker.scan_disabled_on is None) == tracked:
return False
await db.execute(
update(Ticker)
.where(Ticker.id == ticker.id)
# Stamped by the database for the same reason bootstrap_universe does:
# active_only compares it against func.current_date().
.values(scan_disabled_on=None if tracked else func.current_date())
)
await db.commit()
logger.info(
"ticker %s scanning %s", normalised, "enabled" if tracked else "disabled"
)
return True
async def clear_delisted(db: AsyncSession, symbol: str) -> bool:
"""Un-retire a symbol. True if it had been marked.
app/services/ticker_universe_service.py
+216 -32
View File
@@ -12,17 +12,18 @@ import logging
import os
import re
from collections.abc import Iterable
from datetime import datetime, timezone
from datetime import date, datetime, timezone
from pathlib import Path
import httpx
from sqlalchemy import delete, select
from sqlalchemy import func, or_, select, update
from sqlalchemy.ext.asyncio import AsyncSession
from app.config import settings
from app.exceptions import ProviderError, ValidationError
from app.models.ticker import Ticker
from app.services import settings_store
from app.models.universe_membership import UniverseMembership
from app.services import settings_store, ticker_service
logger = logging.getLogger(__name__)
@@ -333,6 +334,114 @@ async def backfill_ticker_names(db: AsyncSession, *, only_missing: bool = True)
return {"updated": updated, "checked": len(targets), "unmatched": len(targets) - updated}
# Sources that represent an actual look at the index today. A ``cache`` or
# ``seed`` result means the live fetch FAILED — the seed lists are ~44 symbols,
# so treating one as an observation would close ~460 memberships on a network
# blip. Membership is only ever recorded from a live read.
LIVE_UNIVERSE_SOURCES = frozenset({
"wikipedia_sp500", "wikipedia_nasdaq100", "nasdaq_trader", "public",
})
async def members_on(db: AsyncSession, universe: str, as_of: date) -> list[str]:
"""Which symbols were observed in *universe* on *as_of*, alphabetically.
The point-in-time question item 3 exists to answer, with the interval
boundaries written down once as code rather than re-derived at each call
site:
member on D <=> first_seen_on <= D < absent_since (or absent_since IS NULL)
Lower bound inclusive, upper bound exclusive. ``absent_since`` is the first
observation that found the symbol *gone*, so on that date it was already
absent.
**This is observation history, not index history.** A date before the first
successful live sync returns an empty list — that is "we were not looking",
not "the index was empty", and the caller has to know the difference. The
reconstructed removals in ``scripts/data/sp500_removals.json`` are research
data and deliberately do not seed this table.
"""
rows = await db.execute(
select(Ticker.symbol)
.join(UniverseMembership, UniverseMembership.ticker_id == Ticker.id)
.where(
UniverseMembership.universe == universe,
UniverseMembership.first_seen_on <= as_of,
or_(
UniverseMembership.absent_since.is_(None),
UniverseMembership.absent_since > as_of,
),
)
.order_by(Ticker.symbol)
)
return list(rows.scalars().all())
async def _observe_membership(
db: AsyncSession, universe: str, present: set[str]
) -> dict[str, int]:
"""Record what a live universe fetch just saw. Returns opened/closed counts.
Runs on every sync, not only when pruning: the scheduled job uses
``prune_missing=False``, so membership recorded only on manual prunes would
be full of holes. Recording membership and deciding what to scan are
separate concerns, and this is the recording one — it never touches
``untracked_on``, ``scan_disabled_on`` or ``tracking_source``, and it applies
to manually tracked symbols exactly as it does to index-owned ones.
Intervals are append-only. A symbol that leaves and returns gets a second
row, so a point-in-time query for the gap answers "not a member"; reopening
the first row would erase that.
Dates come from the database (``current_date``), matching every other date
this module writes. ``first_seen_on`` is the first *observation*, not an
index effective date — anything earlier is unknown and is not guessed.
"""
rows = (await db.execute(
select(Ticker.id, Ticker.symbol).where(Ticker.symbol.in_(present))
)).all()
ticker_ids = {symbol: tid for tid, symbol in rows}
open_rows = (await db.execute(
select(UniverseMembership.ticker_id, Ticker.symbol)
.join(Ticker, Ticker.id == UniverseMembership.ticker_id)
.where(
UniverseMembership.universe == universe,
UniverseMembership.absent_since.is_(None),
)
)).all()
open_symbols = {symbol for _, symbol in open_rows}
opened = 0
for symbol in sorted(present - open_symbols):
ticker_id = ticker_ids.get(symbol)
if ticker_id is None:
continue
db.add(UniverseMembership(
ticker_id=ticker_id,
universe=universe,
first_seen_on=func.current_date(),
))
opened += 1
departed_ids = [tid for tid, symbol in open_rows if symbol not in present]
closed = 0
if departed_ids:
result = await db.execute(
update(UniverseMembership)
.where(
UniverseMembership.universe == universe,
UniverseMembership.absent_since.is_(None),
UniverseMembership.ticker_id.in_(departed_ids),
)
.values(absent_since=func.current_date())
)
closed = int(result.rowcount or 0)
return {"opened": opened, "closed": closed}
async def bootstrap_universe(
db: AsyncSession,
universe: str,
@@ -341,42 +450,98 @@ async def bootstrap_universe(
) -> dict[str, int | str]:
"""Upsert ticker universe into tracked tickers.
Returns summary counts for added/existing/deleted symbols.
``prune_missing`` marks symbols no longer in the universe as untracked; it
never deletes them. "Not in today's scanner" and "not stored for historical
analysis" are different states, and conflating them is what makes a universe
quietly become the survivors. Symbols that have rejoined the universe have
the mark cleared and return to the live path with their history intact.
Returns summary counts for added / already-tracked / untracked / retracked.
"""
normalised_universe = _validate_universe(universe)
symbols, source = await fetch_universe_symbols(db, normalised_universe)
# A cache or seed result means the live fetch FAILED. The seed lists are ~44
# symbols against a ~500-symbol universe, so treating one as the truth would
# untrack the entire index on a single network blip. Nothing that acts on
# "this symbol is/isn't in the index" may run on fallback data: not pruning,
# not clearing a prune flag, not recording membership. Adding is the one
# exception — it destroys nothing and is what lets a cold start bootstrap
# from seeds.
observed = source in LIVE_UNIVERSE_SOURCES
existing_rows = await db.execute(select(Ticker.symbol))
existing_symbols = set(existing_rows.scalars().all())
target_symbols = set(symbols)
symbols_to_add = sorted(target_symbols - existing_symbols)
symbols_to_delete = sorted(existing_symbols - target_symbols) if prune_missing else []
# Only index-owned rows are prunable. A manual addition is by definition not
# in the constituent list, so without this filter every hand-added symbol
# would be swept by any prune.
prunable = set() if not (prune_missing and observed) else set((
await db.execute(
select(Ticker.symbol).where(
Ticker.symbol.in_(existing_symbols - target_symbols),
Ticker.tracking_source == ticker_service.SOURCE_INDEX,
)
)
).scalars().all())
symbols_to_untrack = sorted(prunable)
kept_manual = (
sorted((existing_symbols - target_symbols) - prunable)
if (prune_missing and observed) else []
)
for symbol in symbols_to_add:
db.add(Ticker(symbol=symbol))
db.add(Ticker(symbol=symbol, tracking_source=ticker_service.SOURCE_INDEX))
deleted_count = 0
skipped_delisted: list[str] = []
if symbols_to_delete:
# A delisted row was retained on purpose — its price history is exactly
# what a survivorship-honest backtest needs, and the delete cascades it
# away. Pruning must not undo that. (Pruning a symbol that is merely no
# longer an index constituent still destroys history; that needs a
# tracked/membership state separate from delisting.)
protected = (
await db.execute(
select(Ticker.symbol).where(
Ticker.symbol.in_(symbols_to_delete),
Ticker.delisted_on.is_not(None),
)
)
).scalars().all()
skipped_delisted = sorted(protected)
deletable = [s for s in symbols_to_delete if s not in set(protected)]
if deletable:
result = await db.execute(delete(Ticker).where(Ticker.symbol.in_(deletable)))
deleted_count = int(result.rowcount or 0)
# A symbol that has returned to the index must rejoin the live path. It is
# already in ``existing_symbols``, so it is neither added nor untracked here
# — without this it would keep the flag from its earlier removal and stay
# invisible to the scanner forever, with nothing to show why.
#
# Clears the prune flag ONLY. ``scan_disabled_on`` is the operator's own
# decision and no automated run may undo it: a symbol someone switched off
# by hand must stay off across a prune -> re-entry cycle. Nor does this touch
# ``tracking_source`` — a manual row that joins the index stays manual, or it
# would lose its protection the next time it left.
retracked_count = 0
if observed:
retracked = await db.execute(
update(Ticker)
.where(Ticker.symbol.in_(target_symbols), Ticker.untracked_on.is_not(None))
.values(untracked_on=None)
)
retracked_count = int(retracked.rowcount or 0)
untracked_count = 0
if symbols_to_untrack:
# Marked, never deleted. The delete this replaced cascaded through
# OHLCV, setups and scores, which is precisely the history a
# survivorship-honest backtest needs — see migration 035. Rows already
# carrying a mark keep their original date rather than being re-stamped
# on every bootstrap.
#
# Stamped by the database, not date.today(): active_only compares this
# against func.current_date(), and a process in a timezone ahead of the
# database would otherwise write a date that reads as "effective
# tomorrow" and leave the symbol in the scanner for a day. Same clock on
# both sides by construction.
result = await db.execute(
update(Ticker)
.where(Ticker.symbol.in_(symbols_to_untrack), Ticker.untracked_on.is_(None))
.values(untracked_on=func.current_date())
)
untracked_count = int(result.rowcount or 0)
# Membership is observed on every live sync, independent of pruning and of
# who owns the row. A failed fetch must never close a membership.
membership = (
await _observe_membership(db, normalised_universe, target_symbols)
if observed
else {"opened": 0, "closed": 0}
)
await db.commit()
@@ -393,10 +558,29 @@ async def bootstrap_universe(
"total_universe_symbols": len(symbols),
"added": len(symbols_to_add),
"already_tracked": len(target_symbols & existing_symbols),
"deleted": deleted_count,
# Always 0. Kept so a caller reading this key is told plainly that
# pruning no longer destroys rows, rather than seeing the old key vanish
# and assuming nothing happened — or worse, seeing a count under
# "deleted" that now means something else entirely.
"deleted": 0,
"untracked": untracked_count,
"untracked_symbols": symbols_to_untrack[:50],
# Symbols that rejoined the index and were returned to the live path.
"retracked": retracked_count,
# Absent from the universe but manually owned, so a prune left them
# alone. The operator asked for these by hand.
"kept_manual": kept_manual[:50],
"kept_manual_count": len(kept_manual),
"added_symbols": symbols_to_add[:50],
# Delisted rows a prune declined to destroy, so the caller can see the
# count did not match what they asked to remove.
"kept_delisted": skipped_delisted[:50],
"kept_delisted_count": len(skipped_delisted),
# Membership observation is skipped entirely on a cache/seed fallback,
# so the caller can tell "nothing changed" from "we never looked".
"membership_observed": observed,
"membership_opened": membership["opened"],
"membership_closed": membership["closed"],
"prune_note": (
"Pruning marks untracked_on and never deletes: the rows keep their "
"bars, setups and scores so historical backtests stay honest. "
"Untracked symbols leave the scanner, rankings and ingestion. "
"Manually added symbols are never pruned."
),
}