Second review round on the shadow book; all three findings were real.
- Scan freshness is now proven, not assumed. Pipeline steps run and fail
independently, so a disabled or failed scan step still let the shadow
step run on the newest *stored* setups -- a prior session's picks at
stale prices. scan_all_tickers now records a run boundary
(last_scan_run_started_at / _completed_at) only on successful
completion; the shadow book refuses to trade unless COMPLETED is fresh
and selects only setups with detected_at >= the run start. Deduplication
to the latest row per ticker now happens BEFORE qualification, so a newer
unqualified row suppresses an older qualified one rather than the reverse.
- Shadow selection is hard long-only. setup_qualifies only enforces
long-only when min_momentum_percentile > 0, but 0 is a legal admin
setting, and the cash accounting assumes long positions -- so the
constraint is enforced in shadow selection regardless of gate config.
- The personal setup list excludes only the caller's own open positions.
get_trade_setups gained exclude_open_trade_user_id; the trades route
passes the authenticated user, while the Telegram broadcast stays global
since it has no single owner.
New tests cover stale/absent scan markers, prior-run exclusion, newer
unqualified suppressing older qualified, long-only under a disabled gate,
and both sides of the user-scoped exclusion.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Review of the shadow book found seven ways the two books could leak into
each other; all are fixed here. The most serious silently invalidated the
comparison the shadow book exists to make.
- Shadow holdings no longer suppress the manual candidate list. The
open-trade exclusion filtered on any book, so shadow taking the
top-ranked names removed exactly those from the user's list and alerts,
confining the discretionary book to leftovers. Scoped to the manual
book. Closed-trade alerts and paper-book equity were leaking the same
way and are likewise scoped.
- Shadow sizing now matches _simulate_portfolio: min(1% risk, 20% notional
cap, available cash) from marked equity, plus the sub- dust guard.
Previously risk-only from realized equity, so a tight stop produced a
multiples-of-equity leveraged position the strategy would never take.
- Shadow only trades setups from the scan that just ran (<6h old) with one
setup per ticker. A failed or disabled scan step could otherwise open
positions from a prior session at stale prices.
- Gate-reset transitions are observed for both books, so a shadow stop-out
completes fail -> requalify instead of staying locked forever.
- Manual list/close endpoints default to the manual book and reject
hand-closing shadow trades; the performance endpoint is scoped to the
caller so 'your picks' is not every user's book.
- run_shadow_book is registered as a paused job so Admin can trigger it.
Also anchors three pre-existing paper-trade tests (and the new alpaca
window test) on the UTC date. They build fixtures from the local date but
the service stamps opened_at in UTC, so they failed only between 00:00 and
02:00 in a UTC+hh timezone -- latent on ba2df8b, exposed by the clock.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The manual paper book only contains trades taken by hand, inside a 20
minute window, on days someone was available. The backtest that validated
this strategy auto-takes the top-ranked qualified setups up to capacity
every session. The forward record was therefore measuring strategy plus
discretion plus availability -- and degrading silently on busy days.
The shadow book closes that gap: it mirrors the backtest's selection rule
(top strategy_rank qualified, up to capacity, 1% fixed-fractional risk)
and shares the manual book's exit policy, so the only difference between
the two books is which setups get taken. Selection ordering reuses the
strategy_rank the scanner already stores rather than recomputing it, so
the two cannot drift apart. It runs as a near-close pipeline step right
after the scan, marking entries at the same prices a human would see.
Gate-reset re-entry state is now scoped per book -- the books diverge as
soon as their entries differ, and each must see only its own stops.
Performance view rewritten around the comparison:
- three series (shadow, manual, SPY) from a new endpoint
- SPY changes from a per-trade cost-basis counterfactual to plain
buy-and-hold %, since one line has to serve two books
- headline stats are R-multiples, not currency: the books size
differently, so only R compares across them
- configurable start date, because the strategy has been revised
repeatedly and pre-cutover trades ran under rules that no longer
exist
Migration 024 also repairs the numeric weekday crons written by 023,
rewriting only rows still holding the broken form so hand-corrected
settings survive. Its literals are inlined because bound parameters
render as NULL under 'alembic upgrade --sql'.
The shadow book is opt-in and writes nothing until enabled. Verify its
first selections match a backtest of that day's cross-section before
trusting any point on the curve.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Earnings backfill sourced from the public DoltHub earnings repo at a
pinned commit rather than the FMP API: reproducible for anyone re-running
the study, and it burns no request quota. 12,414 events, 98.6% of symbols
with >=8 announcements, 99.2% paired actual/estimate, no keyed duplicates.
2a earnings-gap diagnostic: INFORMATIONAL, no filter shipped. The
pre-earnings cohort's right tail was better, so the registered
avoid-earnings condition failed. Note the raw 23/266 vs 115/574 incidence
gap is largely a duration confound -- severe losses stop out fast and have
less time to span an announcement -- so it is not evidence that holding
through earnings is safe.
2b SUE: FAIL against the pre-registered +0.03 bar (unconditional IC
+0.0151 over 56 reliable windows, momentum-conditional +0.0213). Signs
stable across eras, so this is a clean null rather than an ambiguous one,
consistent with post-earnings drift having decayed in large caps.
Closes the Tier-1 arc: Task 1 dead on deep evidence, Task 2 dead here,
Task 3 complete as diagnostic. No in-sample research thread remains open.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two independent bugs left the near-close scan running on the previous
session's close, silently degrading live execution to the stale_close
floor (~1.57 Sharpe) instead of the intended ~1.77 close-fill case.
1. OHLCV window never covered the current day. Daily bars are stamped at
session start (04:00Z under EDT), so an end of midnight-on-end_date
landed before that day's bar and dropped it. Widening the window alone
fails the whole request with 'subscription does not permit querying
recent SIP data', so end is also clamped to now-20min. Today's bar is
now returned, roughly 20 minutes behind live -- within the staleness
the near-close design already assumed.
Intraday runs therefore store a partial bar and ingestion progress
reaches today, which made incremental resume skip the after-close
refresh entirely. collect_ohlcv_final() re-pulls the last sessions so
the consolidated bar overwrites the partial one before outcome eval.
2. APScheduler's from_crontab() passes day-of-week to its own field where
0=Monday, so '1-5' meant Tue-Sat: every Monday was skipped and the
scanner ran Saturdays on stale data. Weekday schedules now use names.
Stored settings already corrected via Admin; this fixes the defaults.
Tests cover both: today's bar inside the window, the delayed-data clamp,
historical windows untruncated, and a week of fire times asserting Monday
is present and weekends are not.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Drop intermediate history-depth reports, sector-residual runners/map/code hooks
(evidence stays in final reports + docs), and slim MacBook helper to ssl/earnings/
prod-book-matrix only. SSL bootstrap and archived research conclusions retained.
Four-arm results: 505 stays positive (softer on deep history); liquid breadth
destroys book under current knobs. Stop tracking 1.3GB pkl cache under reports/.cache.
Pre-register A-D (4y/2016 x 505/505+liquid) with unchanged production knobs.
Runner caches full GTL candidates then re-ranks per arm; MacBook entry via
run_tier1_macbook.sh --prod-book-matrix.
Deep masked retest failed the iron IC bar (0.027 < 0.03). Log as rejected #13 in
research README; close sector-residual and history-depth docs. Production market
residual unchanged.
SANITY-FAIL report showed megacaps/ETFs already at empirical 2016-01-04 floor
(2649 bars) after deepen; check wrongly required ~2013. Pass when megacaps leave
the old 2021 two-tier floor and match SPY; XLC listing exception retained.
Terminal follow-up for Task 1: detect/refetch shallow two-tier symbols and sector
ETFs at 5000d, regenerate manifest, run ONE liquid-1500 harness with era split, grade
mom_12_1_sector_resid mechanically. Bundled as run_tier1_macbook.sh --sector-resid-deep.
Authoritative report history-depth-20260719-103315: race guard pass on deep
research.sqlite. Sector residual still short-window only (no pre-2021); fip sign
flips on broad deep sample; no production retune.
Extract app/ssl_bootstrap.py (shared with FastAPI main), wire it into research
scripts, and teach run_tier1_macbook.sh to locate combined-ca-bundle.pem, certifi,
optional USE_CORP_PROXY, plus --ssl-check diagnostics.
Tier-1 alpha research (local only, no production deploy):
Sector residual momentum: two-factor SPY+sector residual and sector demean signals, IC harness + A/B. Sector resid clears pre-registered bars narrowly (PROMOTE for human wire design only). Sector demean fails t vs market resid.
Earnings: earnings_events backfill (FMP bulk paid; FMP/AV per-symbol), 2a gap diagnostic report-only, 2b SUE IC (PARK; incomplete 48/506 coverage).
History-depth: pre-registered doc + runner for MacBook deep rebuild/harness.
Do not ship production residual or filters from this branch.
Brings env-gated liquid-breadth harness hooks, research tooling, compact
evidence, and the completion-manifest race guard. No production behavior
change when liquid env vars are unset. Nothing to deploy.
Log the 21:14 orphan as a snapshot-build race, rewrite the context table to
authoritative ICs only, and soften the vol-tilt warning. Add extender completion
manifest + breadth refuse guard; strip intermediate/orphaned reports; park the
thread (no book sim, no deploy).
Add lagged/tier/prod-subset/mom-conditional checks on research.sqlite.
Log: unconditional sign is a winner/bleeder tug-of-war; mom-conditional
fip stays negative and reliable; warn on high-vol tilt if universe broadens.
Add research-only snapshot extender, PIT dollar-volume mask for signal IC,
rank-only harness path, fingerprint+breadth runner, and docs. Fingerprint
reproduced IC -0.045 / t -2.91 on prod.sqlite. No production gate/schedule changes.
Display-only Da/Gurun/Warachka information discreteness on the ticker
indicator panel. Shared compute with the backtest harness; not wired into
gate or rank.
Regime is computed at 02:00 ET; without a morning alert pass, quadrant
changes waited until 15:30. Dispatcher is change-driven so quiet days stay
quiet. Drop unused alerts_frequency config.
Move the only qualifying R:R scan to 15:30 ET with chained Telegram alerts,
put outcome eval after a final-bar OHLCV fetch, enforce NY trading-day
requalify semantics, stamp paper trades fill_mode=near_close, and migrate
stored schedule_* keys to America/New_York.
Record monotone fill-timing gradient, live [1.57,1.77] bracket, recover-flag
interpretation, gap-cap as third tail-trim, and pre-scheduler ops checklist.
Document Phase A (max-hold/vol/corr closed; next-open as decision baseline).
Add stale_close and next_open gap-cap fill modes plus a small matrix to test
whether near-close scheduling recovers overnight momentum drift.
Ship shared Sharpe SE/PSR diagnostics, next-open fill and equity-curve vol targeting in the portfolio simulator, re-derived fip_id, and a checkpointed offline matrix runner for Mac-side validation sweeps.
Honor custom S/R tolerance as a transient detect, refresh levels after OHLCV
mutations without failing committed price writes, report per-ticker S/R
rebuild failures from admin cleanup, and warn in the admin UI when refresh is partial.
Ship greenfield min_rr=2.0 and conf=0, read-only Structural S/R, indicator
cache invalidation, and UI/gate language that treats GTL as screening not exit.
Align strategy_rank missing-vol fallback live vs backtest, single-source
PRIMARY_TARGET_MIN_RR, expand prod parity tests, and drop dead FE clients.
Finnhub profile2 reports marketCapitalization in millions; storing it
as dollars made mega-caps like SPCX show as micro (e.g. 1.8M). Normalize
on ingest, add unit tests, and include a one-shot SQL backfill script.
Wikipedia no longer uses plain symbol table cells; parse exchange links and NyseSymbol templates, surface the list source in bootstrap results, and keep legacy cell parsing as a fallback.
Persist job and ingestion warnings/errors for 7 days, surface a dismissible top-nav badge, treat stale OHLCV as a warning (e.g. ticker renames), and show market bar age on the ticker freshness chip.
Reconstruct the ATR/% trail bar-by-bar so the mini chart shows a ratchet path instead of a flat current level, and de-emphasize screening targets under trailing exit modes.
Keep support/resistance visible even near trade lines, tighten the how-this-exits spine so steps connect, unify step styling, and add space above the panel.
De-clutter setup labels (left roles, right prices; hide Entry near Now; S/R in-plot), and draw green/red arrows for paper trade entry and exit on the matching sessions.
Single-ticker fetch now attaches residual-momentum ranks so setups do not silently fail the activation gate. Exit plan is a timeline, chart labels move left of the price scale, and missing ranks surface explicitly.
- Remove unused _gate_eligible_levels filtering logic and its tests (research-only)
- Add prominent RESEARCH/DIAGNOSTIC markers and docs to clear-air/ATR fallback helpers
- Document production vs research BACKTEST_* environment variables in backtest_service
- Minor cleanups: update legacy report text, improve outdated function docstring
Browse, drill, and compare all report sections (not just four tables), overlay equity curves, and sort reports by generated_at so the newest run is always on top.
min_rr = 2.0 was hand-set in Admin (2026-06-24) and never swept — the gate
ablation only tested the floor on-vs-off, never its level. It was the last
un-swept knob in the live gate.
Swept against portfolio Sharpe under the real exit, with a parity self-check
(reproduces_production_gate: the row at the live floor must rebuild production's
exact 1,089-setup qualified set — it does).
min_rr qualified in-sample Sh/CAGR OOS Sh/CAGR (entries >= 2024-07)
0.0 6636 1.98 / 58.5% 2.02 / 66.2%
1.2 3897 1.34 / 33.9% 1.12 / 28.8%
1.5 3127 1.20 / 29.6% 1.12 / 28.8%
1.75 1974 1.64 / 44.5% 1.15 / 27.4%
2.0 (live) 1089 2.04 / 50.4% 2.78 / 73.3%
2.25 577 1.64 / 31.8% 1.71 / 31.9%
2.5 286 1.67 / 29.0% 0.68 / 8.7%
KEEP 2.0. It is the optimum in both windows, and a peak that reproduces in data
it was never fitted to is real evidence. But treat it as fragile: unlike the ATR
trail (a plateau), this is a spike with a trough beside it — +/-0.25 costs ~0.4
Sharpe in-sample and ~1.6 out-of-sample — and the curve is bimodal (floor-off is
good, 1.2-1.75 is bad, 2.0 is good). The hand-set value landed on the peak by
luck, not by tuning. Do not nudge it.
Worth knowing: turning the floor OFF entirely is the second-best row in both
windows, with substantially higher CAGR (58.5% / 66.2%) and more trades. If CAGR
ever outranks Sharpe here, "no R:R floor" is a live option — and it would sever
the gate's last dependency on the weak S/R detector.
Also fixes a metric artifact in the holdout harness. The train book's equity curve
ran to the end of the data while its entries stopped at the split, so it sat in
flat cash for two years and deflated its own CAGR/Sharpe (reported 0.95 / 14.6%;
actually 1.31 / 29.6%). _simulate_portfolio now truncates the calendar to
hold_days after the last entry when end_date is set — it only triggers on the
holdout train window, so no other number moves. The clear-air OOS verdict is
unaffected: it rests on the test row, whose entries and curve both start at the
split and were always clean. Both holdout reports regenerated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The README opened with "find the path of least resistance, key S/R zones, and
asymmetric R:R setups" — a description of a strategy we do not run. What we run
is a long-only cross-sectional momentum book with a trailing exit. The S/R
engine, the composite score, sentiment and fundamentals are screening and
display; none has a measured edge.
- Rewrites the intro/philosophy around the real strategy, and says plainly what
is NOT the edge.
- Adds a mermaid decision graph, universe -> qualified -> ranked -> opened ->
closed, with the real exit distribution on the terminal nodes: initial stop 45%,
trailing stop 31%, max hold 24%, S/R target 0%. Validated against the mermaid
parser, not eyeballed.
- Documents that the R:R and touch-probability are GATE INPUTS, not forecasts of
the trade — the single easiest way to misread this app.
- Adds win rate, best/worst R and the exit-reason split to the production
baseline table.
- New docs/research/README.md: every strategy tested, the result, the decision,
and why we stay with the current one. 12 rejected ideas (take-profit exits,
clear-air gate relaxation, EV gate, regime overlay, inverse-vol sizing, shorts,
standalone vol, FIP, ...), the confirmed tuning knobs, the open leads, and the
method rules we learned the hard way (nested lookbacks are not out-of-sample; a
rising win rate is a warning, not a win).
- Documents the research flags and the holdout harness, and warns that the
portfolio_monitor lookbacks are nested windows, NOT a holdout.
- Notes the snapshot must copy paper_% settings or it silently diverges from prod.
All baseline numbers re-verified against reports/backtest-20260711-prod-baseline.json
(506 tickers, 1,089 qualified, CAGR 50.4%, +413.8% vs SPY +95.7%, DD -21.4%,
Sharpe 2.04, 320 trades, 15.3d avg hold, and all five promotion contenders). No
corrections were needed — the numbers were right, the framing was not.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The UI told a swing-trade story (entry -> target -> stop) while the engine runs
a momentum portfolio (buy strength, trail out, re-rank). The selection was
honest; everything around it was borrowed from a strategy we don't run.
The target is never an exit under `atr_trailing`: `_atr_trailing_close()` does
not even take it as a parameter. It exists only to compute the R:R and touch
odds that admit a setup through the activation gate. Backtested exit reasons for
the production strategy: 144 initial stop, 98 trailing stop, 78 max hold —
target 0. See docs/research/sr-levels-and-exits.md.
What changed:
- New ExitPlanPanel on every setup card states the rules that actually close the
trade: initial stop (1R), the price at which the 3x ATR trail takes over from
it, the trail width in R, and the max hold. Derived in lib/exitPlan.ts from the
live exit policy, so it follows Admin rather than hardcoding the default.
- New BaseRatesPanel replaces per-target "probability" as the answer to "what
usually happens": win rate, average hold, best/worst R, and how trades actually
ended — measured under the real exit, from the backtest report.
- "Target"/"target probability" relabelled to "level"/"touch odds" and grouped as
gate metrics, with the R:R. On the dashboard focus card, residual momentum
(the actual signal) takes the headline stat those two used to occupy.
- The take-trade dialog no longer offers a target dropdown whose value the exit
ignores; it states the trailing plan instead. The picker returns only when the
live policy is mode='target', where the choice is real. The stored target is
now the setup's own, not whichever row was last clicked while exploring.
- "Played out" is gone. A setup was declared dead once price reached the target —
backwards under a trailing exit, where reaching a level is the good case and
the trade keeps running. Only the stop invalidates a setup now; running past
the entry is an "extended" warning, measured in R (you'd be chasing).
The levels ladder, the price rail and the chart overlay all stay fully
explorable — clicking a level still drives them. It is framed as overhead
structure, which is what it is, rather than a menu of exits.
Adds a parity guard: the UI recovers ATR as |entry - stop| / 1.5, so the test
fails if the scanner's stop width ever moves.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Investigated whether our support/resistance detection follows best practice
and whether we actually use it that way. Three findings, all backed by runs
against the prod snapshot and written up in docs/research/sr-levels-and-exits.md:
- The S/R target must NOT become an exit. Honoring it as a take-profit on top
of the 3x ATR trail drops Sharpe 2.04 -> 1.47 and halves CAGR. Win rate rises
(37.5% -> 40.0%), which is the tell: it truncates the right tail where
momentum's edge lives.
- The clear-air fallback (synthesize a 3xATR target where no resistance exists,
so 52-week-high breakouts stop being vetoed) looked strictly better in-sample
(Sharpe 2.04 -> 2.07, CAGR 50.4% -> 62.3%, DD 21.4% -> 20.1%) but FAILED a
real out-of-sample holdout: on entries after 2024-07-01 it is worse on Sharpe
(2.78 -> 2.45) and Calmar, better only on raw CAGR. Not shipped.
- The detector itself is weak vs best practice (POC/VAH/VAL computed then
discarded, HVN = any above-mean bin, 1.48x volume double-counting, "touch"
counts pass-throughs, no round numbers), but its only causal path to P&L is
the entry gate. Fix it for the displayed levels, not for returns.
Method note: nested lookback windows are NOT out-of-sample. The in-sample result
was clean, large, and consistent across five windows, and still did not survive
a proper entry-date split.
All research paths are off by default and the default report is unchanged:
BACKTEST_RESEARCH_EXITS=1 take-profit exit rows
BACKTEST_ATR_TARGET_FALLBACK=k synthetic k*ATR target when S/R offers none
BACKTEST_FALLBACK_CLEAR_AIR_ONLY=1 restrict that to genuinely clear air
BACKTEST_HOLDOUT_SPLIT=YYYY-MM-DD train/test split by entry date
Also fixes two reproducibility holes found while reconciling our local baseline
against the live report:
- create_backtest_snapshot.py now copies paper_% settings. The production
monitor row replays the runtime exit policy via get_exit_policy(); without
those keys a snapshot silently falls back to code defaults, so a live-tuned
exit would never be reflected.
- Migration 020 drops activation_min_expected_value and
activation_min_target_probability. Both are orphans of the June EV-gate
redesign, read by no code path, but prod carries min_target_probability = 50.0
which implies a probability floor that is not enforced (the real floor is the
20% constant in qualification.py).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The reports are the evidence behind the production baseline, so they belong
next to the README that quotes them rather than living only on one machine.
Un-ignores reports/*.json (~2.6 MB compressed for all 11); the snapshot DBs
they run against stay ignored.
Renames the reports to a single dated scheme so they sort chronologically and
say what they measured. Each name is derived from the report's own contents
(the atr_trail_sweep / regime_overlay / blue_sky_projected / sizing_test
sections, and the qualified counts that identify the A/B arms), not from the
ad-hoc slugs they carried before. The run the README quotes is now
backtest-20260711-prod-baseline.json.
reports/compare_reports.py loads every report into one sortable table
(portfolio monitor, entry variants, exit policies, portfolio sim), filters by
report and lookback, and highlights the best row for a chosen metric — max
drawdown correctly ranking lowest-as-best. Stdlib tkinter, no dependencies.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The baseline table and promotion evidence still carried pre-primary-target-floor
figures. Re-derived every number from the 2026-07-11 run, the first baseline
measured after the 20% probability floor pruned lottery targets (1,428 -> 1,089
qualified).
The promotion evidence table also claimed the promoted book beat legacy on
"CAGR, Sharpe, and drawdown". That no longer holds: legacy residual 80 + hold
now has the shallowest drawdown (-15.8% vs -21.4%). Production still wins on
Sharpe, so the promotion stands, but the text now says so honestly rather than
implying a clean sweep.
Also documents the primary-target reach-probability floor in the gate
description, which shipped in c7a198b/8f41143 but never reached the README.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
dennisthiessenandClaude Fable 5