A manually triggered rr_scanner and the scheduled near-close pipeline are
separate APScheduler jobs; max_instances=1 serialises a job only against
itself, so they can overlap. A manual scan starting just before the
pipeline can finish just after it began and overwrite the scan markers.
Its completion timestamp is then later than the pipeline start, so the
previous 'completed >= pipeline_start' check accepted its batch as though
it were the pipeline's own -- exactly when the pipeline's scan may have
failed.
Replace the timestamp comparison with an exact run-id match. A new
pipeline_run module holds a per-task run-id contextvar (separate module so
the scanner and scheduler import it without a cycle). _run_pipeline binds a
fresh id per invocation; scan_all_tickers stamps that id -- or a fresh one
when run standalone -- into the scan markers, written with started/completed
in a single commit. The shadow step requires the stored run id to equal its
pipeline's id exactly, so a concurrent manual scan (its own id) or a failed
pipeline scan (a prior run's id) can never be mistaken for it. Direct Admin
triggers have no pipeline context and keep the freshness fallback.
Known residual: the id match governs whether shadow proceeds; setup
selection remains detected_at >= scan start, so a fully per-run setup
isolation would need a run_id column on trade_setups (not required here).
Tests cover the reported race (manual scan finishing last is refused), a
failed pipeline scan, the id-match accept path, and contextvar propagation
and non-leakage across tasks.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A 6-hour freshness window proves only that some scan ran recently, which a
manual mid-day scan satisfies. Scenario: a manual scan succeeds at 13:00;
the 15:30 near-close pipeline's scan step is disabled or fails; at 15:30
the 13:00 completion is still 'fresh', so the shadow step trades that
earlier batch despite no successful scan in the current pipeline.
_run_pipeline now records its start in a per-task contextvar, visible to
the steps it awaits. run_shadow_book reads it and requires the scan
completion marker to be at/after the pipeline start, so a scan that failed
or was disabled in this pass (marker left at a prior run, before the
pipeline began) cannot be substituted by an earlier manual scan. A direct
Admin trigger has no pipeline context and falls back to the freshness
window -- an explicit operator action, not an automated one.
Tests pin the reported case: a fresh manual scan predating the pipeline
start is refused; the pipeline's own post-start scan is accepted.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Second review round on the shadow book; all three findings were real.
- Scan freshness is now proven, not assumed. Pipeline steps run and fail
independently, so a disabled or failed scan step still let the shadow
step run on the newest *stored* setups -- a prior session's picks at
stale prices. scan_all_tickers now records a run boundary
(last_scan_run_started_at / _completed_at) only on successful
completion; the shadow book refuses to trade unless COMPLETED is fresh
and selects only setups with detected_at >= the run start. Deduplication
to the latest row per ticker now happens BEFORE qualification, so a newer
unqualified row suppresses an older qualified one rather than the reverse.
- Shadow selection is hard long-only. setup_qualifies only enforces
long-only when min_momentum_percentile > 0, but 0 is a legal admin
setting, and the cash accounting assumes long positions -- so the
constraint is enforced in shadow selection regardless of gate config.
- The personal setup list excludes only the caller's own open positions.
get_trade_setups gained exclude_open_trade_user_id; the trades route
passes the authenticated user, while the Telegram broadcast stays global
since it has no single owner.
New tests cover stale/absent scan markers, prior-run exclusion, newer
unqualified suppressing older qualified, long-only under a disabled gate,
and both sides of the user-scoped exclusion.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Review of the shadow book found seven ways the two books could leak into
each other; all are fixed here. The most serious silently invalidated the
comparison the shadow book exists to make.
- Shadow holdings no longer suppress the manual candidate list. The
open-trade exclusion filtered on any book, so shadow taking the
top-ranked names removed exactly those from the user's list and alerts,
confining the discretionary book to leftovers. Scoped to the manual
book. Closed-trade alerts and paper-book equity were leaking the same
way and are likewise scoped.
- Shadow sizing now matches _simulate_portfolio: min(1% risk, 20% notional
cap, available cash) from marked equity, plus the sub- dust guard.
Previously risk-only from realized equity, so a tight stop produced a
multiples-of-equity leveraged position the strategy would never take.
- Shadow only trades setups from the scan that just ran (<6h old) with one
setup per ticker. A failed or disabled scan step could otherwise open
positions from a prior session at stale prices.
- Gate-reset transitions are observed for both books, so a shadow stop-out
completes fail -> requalify instead of staying locked forever.
- Manual list/close endpoints default to the manual book and reject
hand-closing shadow trades; the performance endpoint is scoped to the
caller so 'your picks' is not every user's book.
- run_shadow_book is registered as a paused job so Admin can trigger it.
Also anchors three pre-existing paper-trade tests (and the new alpaca
window test) on the UTC date. They build fixtures from the local date but
the service stamps opened_at in UTC, so they failed only between 00:00 and
02:00 in a UTC+hh timezone -- latent on ba2df8b, exposed by the clock.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The manual paper book only contains trades taken by hand, inside a 20
minute window, on days someone was available. The backtest that validated
this strategy auto-takes the top-ranked qualified setups up to capacity
every session. The forward record was therefore measuring strategy plus
discretion plus availability -- and degrading silently on busy days.
The shadow book closes that gap: it mirrors the backtest's selection rule
(top strategy_rank qualified, up to capacity, 1% fixed-fractional risk)
and shares the manual book's exit policy, so the only difference between
the two books is which setups get taken. Selection ordering reuses the
strategy_rank the scanner already stores rather than recomputing it, so
the two cannot drift apart. It runs as a near-close pipeline step right
after the scan, marking entries at the same prices a human would see.
Gate-reset re-entry state is now scoped per book -- the books diverge as
soon as their entries differ, and each must see only its own stops.
Performance view rewritten around the comparison:
- three series (shadow, manual, SPY) from a new endpoint
- SPY changes from a per-trade cost-basis counterfactual to plain
buy-and-hold %, since one line has to serve two books
- headline stats are R-multiples, not currency: the books size
differently, so only R compares across them
- configurable start date, because the strategy has been revised
repeatedly and pre-cutover trades ran under rules that no longer
exist
Migration 024 also repairs the numeric weekday crons written by 023,
rewriting only rows still holding the broken form so hand-corrected
settings survive. Its literals are inlined because bound parameters
render as NULL under 'alembic upgrade --sql'.
The shadow book is opt-in and writes nothing until enabled. Verify its
first selections match a backtest of that day's cross-section before
trusting any point on the curve.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
dennisthiessenandClaude Fable 5