diff --git a/README.md b/README.md
index e1bb589..a104d3f 100644
--- a/README.md
+++ b/README.md
@@ -2,7 +2,7 @@
Investing-signal platform for US equities. It runs one strategy, and it is a boring one:
-> **A long-only cross-sectional momentum book.** Buy the top quintile by beta-adjusted 12-1 month momentum, tilt toward higher volatility, hold at most 10 names, cut at 1.5× ATR, then trail at 3× ATR for up to 30 trading days. After an initial-stop exit, re-enter only after the gate has failed and subsequently qualified again.
+> **A long-only cross-sectional momentum book.** Buy the top quintile by beta-adjusted 12-1 month momentum, tilt toward higher volatility, hold at most 15 names, cut at 1.5× ATR, then trail at 3× ATR for up to 30 trading days. After an initial-stop exit, re-enter only after the gate has failed and subsequently qualified again.
**Philosophy:** don't predict price — rank it. The edge is *relative* strength across the universe, and the discipline is in the exit: cut losers fast, let winners run until the trail catches them.
@@ -31,7 +31,7 @@ flowchart TD
Q -->|no| SKIP
Q -->|yes| RANK["Rank by production score
80% momentum %ile
+ 20% volatility %ile"]
- RANK --> BOOK{"Room in the book?
max 10 positions"}
+ RANK --> BOOK{"Room in the book?
max 15 positions"}
BOOK -->|no| WAIT["Wait for a slot"]
BOOK -->|yes| OPEN["OPEN — size at 1% account risk"]
@@ -131,7 +131,7 @@ indicators.
**Morning** (~02:00 ET) — data and display only, **no** qualifying R:R scan:
-1. **OHLCV** — latest daily bars (Alpaca); new tickers backfill ~5 years.
+1. **OHLCV** — latest daily bars (Alpaca) plus the SPY benchmark; new tickers backfill ~5 years. A symbol whose bars have been stale for 3 days is probed against SEC for a Form 25/25-NSE/15 and **retired** on a hit (history kept — see *Delisting*).
2. **Sentiment** — stale names that matter (top-pick feeders, watchlist, open paper, discovery net). Display context only; the activation gate is price-only.
3. **Market Trend (SPY)** + **AI/Tech Risk Monitor** — the SPY trend guard and the v4 risk thermometer; feed no trades.
4. **Telegram alerts** — change-driven (risk-quadrant etc.); quiet days stay quiet. Setup alerts still fire on the near-close pipeline after the scan.
@@ -140,7 +140,8 @@ indicators.
1. **OHLCV fetch** — refresh the in-progress day-t bar (same path as intraday).
2. **R:R Scan** — Structural S/R, scores, Gate Target Ladder setups, residual 12‑1 + 80/20 rank. Advances post-stop gate-reset transitions; failed scans never count.
-3. **Telegram alerts** — chained immediately so manual MOC fills can still hit ~15:50/15:55.
+3. **Shadow book** — opt-in automated book; opens top-ranked qualified setups up to capacity at the same near-close prices. Only accepts a scan from this same pipeline run.
+4. **Telegram alerts** — chained immediately so manual MOC fills can still hit ~15:50/15:55.
**After close** (~16:45 ET Mon–Fri):
@@ -157,6 +158,38 @@ Hourly mid-session (Mon–Fri ~10:00–15:00 ET): only **OHLCV → Outcome Eval*
Dolt earnings import (daily 02:30 ET) · SEC fundamentals import (daily 04:00 ET, also refreshes the fundamentals cache scoring reads) · Backtest (weekly) · Ticker-universe sync (daily). Alerts auto-fire only via the near-close pipeline (still manually triggerable). Deep history backfill and event study are manual-only (Admin → Jobs).
+The SEC import defers a run rather than writing partial data when a filing's XBRL
+hasn't landed. Two bounds keep that from compounding: `MISSING_XBRL_RETRY_DAYS`
+caps how long *one* filing blocks promotion, and `PROMOTION_CEILING_DAYS` (7)
+caps how long the import as a whole can stay deferred — past the ceiling every
+unresolved filing is aged out in place so `promote()` queues it as a gap row,
+`source_max_date` advances, and the import self-heals. A `deferred_stale` alert
+inside that window is normal and clears on its own; check `source_max_date` in
+`data_import_runs` before diagnosing a wedge.
+
+### Delisting, not deletion
+
+Retiring a symbol used to mean `delete_ticker` or a pruning universe bootstrap,
+both of which cascade through OHLCV, setups and scores. That destroys exactly the
+history four research documents apologise for: today's tracked universe projected
+backward is survivorship-biased, and hard-deleting every delisted name is what
+causes it. Keeping the rows preserves the option to fix that later (it does not
+fix it — the replay still has to model a delisting as an exit event).
+
+`tickers` therefore carries `delisted_on` / `delisted_reason` (migration 032);
+`NULL` means actively traded. The filter is **opt-in** via
+`ticker_service.active_only`, applied to the live path only — scanner, momentum
+ranking, scoring, breadth, fundamentals candidates, SEC universe, earnings import,
+ingestion. The registry and admin views deliberately keep delisted rows visible,
+and `run_backtest` keeps them on purpose. Detection runs off OHLCV staleness
+(not the SEC fundamentals import, which stalls for days on unrelated Company-Facts
+gaps) and retires only on a Form 25/25-NSE/15 hit, so a halt or a rename keeps the
+existing warning instead. `delisted_on` is the *effective* date — Rule 12d2-2
+makes a Form 25 removal take effect ten days after filing, so a symbol filed today
+keeps trading (and keeps qualifying) until that date. It is safe to automate
+because it is reversible: `clear_delisted` un-retires a false positive, where a
+delete had already taken the history.
+
### From score to "top pick"
1. **Composite score** — technical, S/R-quality, sentiment, fundamental and momentum sub-scores (0–100) combine into a weighted composite (weights configurable; missing dimensions re-normalize). **Display and ranking only — it does not select trades.**
@@ -166,6 +199,33 @@ Dolt earnings import (daily 02:30 ET) · SEC fundamentals import (daily 04:00 ET
**What the R:R and reach-probability in step 3 actually are.** They are *gate inputs*, computed from a Gate Target Ladder proposal the trade will never exit at — they exist to filter setups, not to forecast the trade you're about to take. A setup with "R:R 2.4:1, 34% reach probability" is not a claim that you'll make 2.4R with 34% probability; it's a claim that this setup cleared the screen. What actually happens to a trade is in the exit box of the diagram above, and on the "what usually happens" panel in the UI. Conflating the two is the single easiest way to misread this app.
+### Two books: shadow (automated) and discretionary (manual)
+
+The platform keeps **two** paper books, and the difference between them is the
+whole point.
+
+| Book | Who selects | What it measures |
+|---|---|---|
+| **Shadow book** (`app/services/shadow_book_service.py`) | The machine — top-ranked qualified setups up to capacity, every near-close scan | The **strategy**, faithfully |
+| **Discretionary book** | You, by clicking "paper trade" on a setup | The strategy **plus** your discretion and availability |
+
+The manual book only ever contains trades the user chose to take, inside a ~20
+minute window, on days they were around. The backtest that validated this
+strategy does none of that, which makes the manual record unusable on its own as
+out-of-sample evidence. The shadow book closes that gap: it mirrors
+`_simulate_portfolio`'s selection rule exactly, orders on the *stored*
+`strategy_rank` the scanner already wrote (so the two cannot drift apart) and
+shares the manual book's exit policy — the only difference between the books is
+*which* qualified setups get taken.
+
+It runs as a step of the near-close pipeline, straight after the scan so entries
+mark at the same near-close prices, and it only accepts a scan from the same
+pipeline run. It is **opt-in** (`shadow_book_enabled`, with capacity, risk % and
+starting equity under **Admin → Settings → Performance & Shadow Book**) because it
+writes live trades. The **Dashboard**'s performance chart plots shadow vs
+discretionary vs SPY; *Signals → Paper Trades* still shows the discretionary book
+only.
+
## Strategy Status — What's Validated and What Isn't
**Read this before touching scoring, gating, or setup logic.** The platform measures itself — a weekly-replay backtest plus a factor rank-IC harness (`app/services/backtest_service.py`) — and the verdicts below come from those reports (latest run July 2026, ~5 years of OHLCV), not from opinion.
@@ -176,7 +236,8 @@ Dolt earnings import (daily 02:30 ET) · SEC fundamentals import (daily 04:00 ET
|---|---|---|
| **Residual 12-1 cross-sectional momentum** (the activation gate, long-only) | **Production gate — in-sample edge** | Promoted July 2026 after the portfolio variant beat raw 80 on CAGR, Sharpe and drawdown. Raw 12-1 remains a fallback only when benchmark data is unavailable |
| **3× ATR trailing exit** (+ 1.5× ATR initial stop, 30-day max hold) | **Production exit — best Sharpe of every exit tested** | Beat hold / SMA50 / 20-day-low / technical-40 and both take-profit variants (July 2026) |
-| **Post-stop gate reset** | **Production re-entry policy** | The initial stop always closes; the ticker must later fail the daily gate and subsequently qualify again. At the production capacity of 10: Sharpe 1.67 → 1.77, CAGR 45.2% → 48.3%, DD 24.3% → 21.6% versus immediate re-entry. [Full study](docs/research/post-stop-reentry.md) |
+| **Post-stop gate reset** | **Production re-entry policy** | The initial stop always closes; the ticker must later fail the daily gate and subsequently qualify again. At the then-production capacity of 10: Sharpe 1.67 → 1.77, CAGR 45.2% → 48.3%, DD 24.3% → 21.6% versus immediate re-entry. Capacity has since been raised to 15 — see the open question under the re-entry section. [Full study](docs/research/post-stop-reentry.md) |
+| **Book capacity 15** (raised from 10, 2026-08-05) | **Production sizing** | The focused daily capacity bracket found the count cap was binding and cost real compounding: +1.075pp CAGR paired, 51 paths better / 2 worse, drawdown unchanged. Cash plus the 20% notional cap saturates the book near 12, so the cap no longer binds. [Findings](docs/research/portfolio-capacity-bracket-findings.md#correction-2026-08-05-ev-per-trade-was-the-wrong-lens) |
| **Structural S/R** | **Human-facing context only — not a gate and not an exit** | Clean, capped zones are persisted for charts and alerts. The scanner deliberately does not read them. |
| **Gate Target Ladder** | **Gate input only — not market structure and not an exit** | Volume-free range grid + pivots preserves the useful legacy screening behavior exactly: 1,086/1,086 qualified setups retained and identical Sharpe 2.03 / CAGR 50.0% / DD 21.4% / 321 trades. The exit never reads its target. [Full write-up](docs/research/sr-levels-and-exits.md#explicit-gate-target-ladder) |
| Composite score + 5 dimensions | **Display/ranking only** | Sub-scores are hand-built heuristics; none has a measured IC. Note: the "momentum" *dimension* is 5/20-day ROC — NOT the validated 12-1 factor (that lives in `momentum_service`) |
@@ -187,7 +248,7 @@ Dolt earnings import (daily 02:30 ET) · SEC fundamentals import (daily 04:00 ET
| Gate target as a take-profit (tested July 2026) | **Rejected** | Sharpe 2.04 → 1.47, CAGR halved. Win rate *rose* — it truncates the right tail where the edge lives |
| "Clear-air" gate relaxation (tested July 2026) | **Rejected — failed out-of-sample** | Strictly better in-sample (Sharpe 2.07 / CAGR 62.3% / DD 20.1%), then lost on a real train/test split (Sharpe 2.78 → 2.45). A cautionary tale: nested lookbacks are not OOS |
-Caveats on the momentum result: in-sample, roughly one market regime, costs/slippage approximated at 0.1% per side, and residual momentum still needs SPY benchmark history to compute. The **out-of-sample proof is the forward paper-trade record**: Signals → Track Record compares live qualified expectancy against the backtest.
+Caveats on the momentum result: in-sample, roughly one market regime, costs/slippage approximated at 0.1% per side, and residual momentum still needs SPY benchmark history to compute. The **out-of-sample proof is the forward record of the shadow book** — the automated twin that takes every top-ranked qualified setup, with no discretion or availability mixed in. The Dashboard chart tracks it against the discretionary book and SPY; *Signals → Backtest* is what it is being compared against.
### Daily post-stop re-entry decision (2026-07-17)
@@ -200,7 +261,9 @@ The production policy is **normal gate reset**, evaluated with daily setup oppor
| Strict gate reset (live timing analogue) | 342.7% | 44.8% | -23.4% | 1.68 | 471 |
| Fixed five-session cooldown | 250.8% | 36.6% | -22.2% | 1.47 | 473 |
-In the disjoint 2025+ book, gate reset also beat immediate re-entry (Sharpe 1.66 vs 1.55; CAGR 41.8% vs 39.3%) and the fixed five-session rule (Sharpe 1.43; CAGR 32.7%). Its lead over both survived costs of 0.2% and 0.3% per side. The result is capacity-specific: cooldown 5 won at capacity 5, while immediate had slightly higher return and Sharpe at capacity 15. Production uses capacity 10, so that is the portfolio for which this decision is valid.
+In the disjoint 2025+ book, gate reset also beat immediate re-entry (Sharpe 1.66 vs 1.55; CAGR 41.8% vs 39.3%) and the fixed five-session rule (Sharpe 1.43; CAGR 32.7%). Its lead over both survived costs of 0.2% and 0.3% per side. The result is capacity-specific: cooldown 5 won at capacity 5, while immediate had slightly higher return and Sharpe at capacity 15.
+
+> **Open question (since 2026-08-05).** This study was run — and gate reset promoted — at capacity 10. Production capacity was subsequently raised to 15, which is the one capacity in the matrix where *immediate* re-entry edged ahead. The re-entry policy is therefore currently running outside the portfolio it was validated on. Nothing else changed, and the two arms differed only modestly, but the matrix should be rerun at capacity 15 before treating gate reset as settled. Until then, keep gate reset (the incumbent) rather than switching on an untested read.
Those promotion numbers belong to the selected normal-reset study arm. Under the **pre-cutover** morning-scan scheduler (scan always before any outcome eval), live first-observation timing matched the stricter `strict_gate_reset` analogue (full-period Sharpe 1.68 / CAGR 44.8% / DD 23.4%). After the **near-close cutover** (2026-07), stops closed by earlier same-day intraday evals can receive a same-day fail observation at ~15:30 ET — moving live behavior **toward** the promoted `gate_reset` arm. Requalification still requires a later America/New_York trading date than the failure (`trade_policy` distinct-day guard). Full definitions and all nine policy arms: [docs/research/post-stop-reentry.md](docs/research/post-stop-reentry.md); execution evidence: [docs/research/execution-recovery.md](docs/research/execution-recovery.md).
@@ -208,7 +271,7 @@ Those promotion numbers belong to the selected normal-reset study arm. Under the
### Historical weekly production baseline (pre gate-reset)
-Use this as the historical ranking/exit regression guardrail, not as a return promise or the current re-entry-policy result. This run predates the post-stop gate reset and uses weekly entry replay, so its portfolio headline is not directly comparable with the daily matrix above. Backtest run: local production SQLite snapshot, 506 tickers, weekly cadence, 30-trading-day horizon, 2022-06-28 → 2026-07-02, 0.1% per-side costs, price-only SPY benchmark. Numbers below are the 2026-07-11 run (`reports/backtest-20260711-prod-baseline.json`) — measured *after* the primary-target probability floor shipped, which pruned lottery-target setups (1,428 → 1,089 qualified) and lifted Sharpe on all three promotion contenders.
+Use this as the historical ranking/exit regression guardrail, not as a return promise or the current re-entry-policy result. This run predates the post-stop gate reset **and the 2026-08-05 capacity raise to 15**, and uses weekly entry replay, so its portfolio headline is not directly comparable with the daily matrix above. Backtest run: local production SQLite snapshot, 506 tickers, weekly cadence, 30-trading-day horizon, 2022-06-28 → 2026-07-02, 0.1% per-side costs, price-only SPY benchmark. Numbers below are the 2026-07-11 run (`reports/backtest-20260711-prod-baseline.json`) — measured *after* the primary-target probability floor shipped, which pruned lottery-target setups (1,428 → 1,089 qualified) and lifted Sharpe on all three promotion contenders.
| Item | Historical weekly baseline |
|---|---|
@@ -248,16 +311,16 @@ Parity guard (July 2026): the portfolio monitor's **Production** row replays the
### Tuned and confirmed — do not retest without new data (July 2026)
-A systematic single-variable sweep (offline prod snapshot, production gate/rank/exit, 2022-06 → 2026-07 plus disjoint 2022–23 / 2024–26 folds) confirmed **every** production setting. Retesting these against the same ~4-year snapshot is wasted compute and invites overfitting; revisit only with meaningfully new data (longer history or broader universe).
+A systematic single-variable sweep (offline prod snapshot, production gate/rank/exit, 2022-06 → 2026-07 plus disjoint 2022–23 / 2024–26 folds) confirmed every production setting **except book size**, which a later focused bracket reversed (see the row below). Retesting these against the same ~4-year snapshot is wasted compute and invites overfitting; revisit only with meaningfully new data (longer history or broader universe) — or, as with capacity, a demonstrably better measurement lens.
| Knob tested | Verdict | Evidence |
|---|---|---|
| ATR trail multiple {1.5–4.0} | **Keep 3.0** | Return+Sharpe peak; ≤2.0 whipsaws out the momentum right tail; ≥2.5 is a plateau |
| SPY 200d-MA regime overlay (block entries / go flat) | **Reject** | Halves return (315%→138%) with zero drawdown benefit — the ATR trail already manages downside, and the filter blocks the recovery-phase entries that make the money |
| Momentum lookback: 6-1, 3-1, 12-7 (Novy-Marx), composites | **Keep residual 12-1** | 6-1/3-1 rank-IC ≈ 0; 12-7 IC 0.045 / t 1.58 — weaker than residual 12-1 (0.055 / t 1.98) |
-| Selection cutoff {70, 75, 85, 90} × book size {10, 15, 20} | **Keep 80 × 10** | Monotonically worse in both directions from 80; the 10-slot cap never binds (<10 concurrent) |
+| Selection cutoff {70, 75, 85, 90} × book size {10, 15, 20} | **Keep cutoff 80; book size raised to 15 (2026-08-05)** | The cutoff is monotonically worse in both directions from 80. The book-size half of this row was **reversed**: the weekly replay's "the 10-slot cap never binds" read came from EV per trade, which is the wrong lens for anything that changes trade *count*. The focused daily bracket found cap 10 *was* binding and cost +1.075pp CAGR; at 15 the cap never bound in any cell (max observed 12 concurrent, zero full-book skips) |
| Position sizing: equal-weight, inverse-vol, risk-% sweep | **Keep 1% fixed-fractional** | See the inverse-vol warning below |
-| Post-stop re-entry: immediate, fixed 2–5 sessions, gate resets, confirmation filters | **Keep normal gate reset for the 10-position production book** | Sharpe 1.77 vs 1.67 immediate and 1.47 cooldown 5; rerun before changing portfolio capacity |
+| Post-stop re-entry: immediate, fixed 2–5 sessions, gate resets, confirmation filters | **Keep normal gate reset** — but measured at capacity 10, and capacity is now 15 | Sharpe 1.77 vs 1.67 immediate and 1.47 cooldown 5. The "rerun before changing portfolio capacity" caveat is now outstanding — see the open question above |
| FIP path-smoothness as an in-book tie-breaker/filter | **Reject** (but see the lead below) | Non-monotonic across FIP quintiles within the qualified set; either half of a median split underperforms the full book — thinning the entry stream costs more compounding than the tilt returns |
Two findings future sessions must not re-litigate:
@@ -270,23 +333,24 @@ Two findings future sessions must not re-litigate:
A signal earns its way into selection **only** through the factor harness:
1. Add it as a point-in-time function of past bars in `_signal_values()` (`backtest_service.py`).
-2. Run the backtest (Admin → Jobs, or the weekly run) and read the **Signal edge** table (Signals → Track Record).
+2. Run the backtest (Admin → Jobs, or the weekly run) and read the report's `signal_eval` section. This one is **local-report only** — the deployed Backtest tab does not render it (see *Reading a local backtest report* below).
3. Wire it into the gate or ranking **only if** |mean IC| ≳ 0.03 with a consistent sign and `reliable: true` (≥ 12 non-overlapping windows).
Corollaries: never let an unvalidated score gate setups; the outcome evaluator must keep scoring **all** setups (unqualified ones are the control group); LLM output stays display-only in the quant path.
### Highest-value next experiments (in order)
-> Check **[docs/research/](docs/research/README.md)** first — 12 strategy ideas have already been tested and rejected, including the obvious ones (take-profit exits, regime overlays, inverse-vol sizing, shorts).
+> Check **[docs/research/](docs/research/README.md)** first — 13 strategy ideas have already been tested and rejected, including the obvious ones (take-profit exits, regime overlays, inverse-vol sizing, shorts, sector-residual momentum).
-1. **Forward monitor the promoted strategy** — the production UI now behaves like a portfolio monitor for the current strategy, with selectable lookbacks and SPY comparison. Forward paper-trade months are the only evidence the snapshot cannot provide; the July 2026 tuning pass closed every in-sample lead. (Trailing-stop sensitivity and the max-15 capacity check are done — see the tuning table above.)
+1. **Forward monitor the promoted strategy** — *Signals → Backtest* behaves like a portfolio monitor for the current strategy, with selectable lookbacks and SPY comparison, and the Dashboard chart carries the forward record. Forward months of the **shadow book** are the only evidence the snapshot cannot provide; the July 2026 tuning pass closed every in-sample lead. (Trailing-stop sensitivity and the capacity bracket are done — capacity was raised to 15.)
2. **Signal context snapshots** — accumulate point-in-time composite/sentiment/fundamental context for every new setup so the discretionary overlay can be tested forward-only.
3. **Breadth is no longer free leverage** — Phase B found residual-mom t-stat *fell* on liquid-1500 vs the 505-name fingerprint (0.055/1.98 → 0.029/1.33). Any breadth book must clear a pre-registered baseline arm before fip tilts mean anything. (Deeper history was considered and declined.)
## Key Use Cases
- **Find today's best long setup.** On the **Dashboard**, the *Top Setups* table lists residual-gated qualified setups ranked by the production 80/20 residual/high-vol score, with the #1 flagged "Top pick". Each row opens the ticker page for its chart, Structural S/R, Gate Target Ladder targets and entry/stop.
-- **Track a trade you took.** Mark a setup as a **paper trade**: it's marked-to-market against the latest close, auto-closed by the active exit policy (default: 3x ATR trail with a 30-trading-day max hold), and its sentiment stays fresh while open. *Signals → Track Record* shows the realized edge.
+- **Track a trade you took.** Mark a setup as a **paper trade**: it's marked-to-market against the latest close, auto-closed by the active exit policy (default: 3x ATR trail with a 30-trading-day max hold), and its sentiment stays fresh while open. *Signals → Paper Trades* shows the realized edge of your discretionary book; the Dashboard chart puts it next to the automated shadow book and SPY.
+- **Ask whether the strategy is worth trading at all.** *Signals → Backtest* replays the promoted strategy over history — portfolio monitor vs SPY over selectable lookbacks, headline risk-adjusted metrics (Sharpe, Sortino, Gain-to-Pain, dollar profit factor) and the report's own recommendation — with the live-outcome evaluation panel underneath it.
## Stack
@@ -306,7 +370,7 @@ Corollaries: never let an unvalidated score gate setups; the outcome evaluator m
## Features
### Backend
-- Ticker registry with full cascade delete
+- Ticker registry with reversible delisting (history preserved) plus an explicit cascade delete
- Universe bootstrap for `sp500`, `nasdaq100`, `nasdaq_all` via admin endpoint — free public sources (Wikipedia / NASDAQ Trader), then the cached snapshot, then a built-in seed list. The seeds are representative, not complete, so a *fresh* install bootstrapped while the public source is unreachable gets a partial universe; a warm instance falls through to its cache.
- OHLCV price storage with upsert and validation
- Technical indicators: ADX, EMA, RSI, ATR, Volume Profile, Pivot Points, EMA Cross
@@ -319,6 +383,8 @@ Corollaries: never let an unvalidated score gate setups; the outcome evaluator m
- Activation gate — qualifies setups on a residual-momentum percentile floor (the actual selection), a headline gate-target R:R floor (prod: 2.0) and a 20% primary-target reach-probability floor (validated long-only edge)
- Recommendation layer — directional confidence, conflict detection, per-target reach-probability
- Paper trading — take a setup, mark-to-market vs. latest close, auto-close per the exit policy (default: 3x ATR trail with a 30-trading-day max hold; time / percent-trailing / target-stop selectable), realized track record + outcome evaluation
+- Shadow book — opt-in automated twin of the backtest's selection rule (top-ranked qualified setups up to capacity, every near-close scan), sharing the manual book's exit policy; the honest forward out-of-sample record
+- System events — structured job/import/data warnings with acknowledgement, surfaced in Admin and deduplicated for alerting
- Market-regime guard + observational State/Warning monitor (fixed-basket breadth, VIX, credit level + impulse) with a manual chronological correction study
- Telegram alerts (e.g. regime-quadrant changes)
- User-curated watchlist (cap: 20), enriched with composite score, R:R and S/R summary
@@ -337,7 +403,10 @@ Corollaries: never let an unvalidated score gate setups; the outcome evaluator m
- Ticker detail page: chart, scores, sentiment breakdown, fundamentals, technical indicators, S/R table
- Rankings table with configurable dimension weights
- Trade scanner showing detected R:R setups
-- Admin page: user management, job status with live indicators, enable/disable toggles, data cleanup, system settings
+- Backtest tab: portfolio monitor vs SPY over selectable lookbacks, headline risk-adjusted tiles (Sharpe, Sortino, Gain-to-Pain, dollar profit factor), the report's recommendation card, and a live-outcome evaluation panel
+- Dashboard performance chart: cumulative shadow book vs discretionary book vs SPY since the configured start date
+- Paper Trades tab: open/closed discretionary trades with realized R and P&L tiles
+- Admin page: user management, job status with live indicators, enable/disable toggles, pipeline readiness, system-event log, ticker management, data cleanup, system settings
- Protected routes with JWT auth, admin-only sections
- Responsive layout with mobile navigation
- Toast notifications for async operations
@@ -348,14 +417,14 @@ Corollaries: never let an unvalidated score gate setups; the outcome evaluator m
|---|---|---|
| `/login` | Login | Public |
| `/register` | Register | Public (when enabled) |
-| `/` | Dashboard — top setups, open trades, regime (default) | Authenticated |
+| `/` | Dashboard — top setups, open trades, regime, shadow-vs-manual-vs-SPY performance chart (default) | Authenticated |
| `/market` | Market — watchlist + rankings tabs | Authenticated |
-| `/signals` | Signals — scanner + track record tabs | Authenticated |
+| `/signals` | Signals — Setups / Paper Trades / Backtest tabs | Authenticated |
| `/regime` | AI/Tech Risk Monitor | Authenticated |
| `/ticker/:symbol` | Ticker Detail | Authenticated |
| `/admin` | Admin Panel | Admin only |
-Legacy routes redirect: `/watchlist` → `/market`, `/rankings` → `/market?tab=rankings`, `/scanner` → `/signals`, `/performance` → `/signals?tab=track`.
+Legacy routes redirect: `/watchlist` → `/market`, `/rankings` → `/market?tab=rankings`, `/scanner` → `/signals`, `/performance` → `/signals?tab=track` (the Paper Trades tab — `track` stays its slug so the old link keeps working).
## API Endpoints
@@ -365,7 +434,7 @@ All under `/api/v1/`. Interactive docs at `/docs` (Swagger) and `/redoc`.
|---|---|
| Health | `GET /health` |
| Auth | `POST /auth/register`, `POST /auth/login` |
-| Tickers | `POST /tickers`, `GET /tickers`, `DELETE /tickers/{symbol}` |
+| Tickers | `POST /tickers`, `GET /tickers`, `DELETE /tickers/{symbol}`, `POST /tickers/{symbol}/delisting`, `DELETE /tickers/{symbol}/delisting` |
| OHLCV | `POST /ohlcv`, `GET /ohlcv/{symbol}` |
| Ingestion | `POST /ingestion/fetch/{symbol}` |
| Indicators | `GET /indicators/{symbol}/{type}`, `GET /indicators/{symbol}/ema-cross` |
@@ -375,11 +444,11 @@ All under `/api/v1/`. Interactive docs at `/docs` (Swagger) and `/redoc`.
| Fundamentals | `GET /fundamentals/{symbol}` |
| Scores | `GET /scores/{symbol}`, `GET /rankings`, `PUT /scores/weights` |
| Trades | `GET /trades`, `GET /trades/{symbol}`, `GET /trades/{symbol}/history`, `GET /trades/activation`, `GET /trades/performance` |
-| Paper Trades | `GET /paper-trades`, `POST /paper-trades`, `POST /paper-trades/{id}/close` |
-| Market / Regime | `GET /market/regime`, `GET /regime/monitor`, `GET/PUT /regime/config`, `GET /regime/history`, `GET /regime/event-study`, `GET/PUT /regime/fundamentals`, `GET /backtest/report` |
+| Paper Trades | `GET /paper-trades`, `POST /paper-trades`, `POST /paper-trades/{id}/close`, `GET /paper-trades/equity-curve`, `GET /paper-trades/performance` (shadow vs manual vs SPY), `GET/PUT /paper-trades/exit-policy` |
+| Market / Regime | `GET /market/regime`, `GET /regime/monitor`, `GET/PUT /regime/config`, `GET /regime/history`, `GET /regime/event-study`, `GET/PUT /regime/fundamentals`, `POST /regime/fundamentals/refresh`, `GET /backtest/report` |
| Jobs | `GET /jobs/running` |
| Watchlist | `GET /watchlist`, `POST /watchlist/{symbol}`, `DELETE /watchlist/{symbol}` |
-| Admin | `GET /admin/users`, `POST /admin/users`, `PUT /admin/users/{id}/access`, `PUT /admin/users/{id}/password`, `PUT /admin/settings/registration`, `GET /admin/settings`, `PUT /admin/settings/{key}`, `GET/PUT /admin/settings/recommendations`, `GET/PUT /admin/settings/ticker-universe`, `POST /admin/tickers/bootstrap`, `POST /admin/data/cleanup`, `GET /admin/jobs`, `POST /admin/jobs/{name}/trigger`, `PUT /admin/jobs/{name}/toggle`, `GET /admin/pipeline/readiness` |
+| Admin | `GET /admin/users`, `POST /admin/users`, `PUT /admin/users/{id}/access`, `PUT /admin/users/{id}/password`, `PUT /admin/settings/registration`, `GET /admin/settings`, `PUT /admin/settings/{key}`, `GET/PUT /admin/settings/{recommendations,activation,schedule,performance,shadow-book,sentiment,alerts,ticker-universe}`, `POST /admin/settings/{sentiment,alerts}/test`, `POST /admin/tickers/bootstrap`, `POST /admin/tickers/backfill-names`, `POST /admin/data/cleanup`, `POST /admin/track-record/reset`, `GET /admin/jobs`, `POST /admin/jobs/{name}/trigger`, `PUT /admin/jobs/{name}/toggle`, `GET /admin/pipeline/readiness`, `GET /admin/system-events`, `GET /admin/system-events/summary`, `POST /admin/system-events/acknowledge` |
## Development Setup
@@ -439,8 +508,8 @@ npm run preview # Preview the production build locally
# Backend tests (in-memory SQLite — no PostgreSQL needed)
pytest tests/ -v
-# Frontend: there is no test suite — `npm test` calls vitest, which is not
-# installed. The frontend check is the full TypeScript build:
+# Frontend: there is no test suite and no `test` script at all. The frontend
+# check is the full TypeScript build:
cd frontend
npm run build
```
@@ -524,10 +593,10 @@ the [full research record](docs/research/sr-levels-and-exits.md#gtl-tuning-matri
### Reading a local backtest report
-The deployed **Signals → Track Record** page is deliberately trimmed to validation
-(portfolio monitor vs SPY, realized paper trades) and how-to-trade. The
-strategy-tuning tables that used to live there now live **only** in the local
-report — inspect these `reports/backtest-.json` sections and produce the
+The deployed **Signals → Backtest** tab is deliberately trimmed to validation
+(portfolio monitor vs SPY, headline metrics, the report's recommendation, and the
+live-outcome evaluation panel). The strategy-tuning tables that used to live there
+now live **only** in the local report — inspect these `reports/backtest-.json` sections and produce the
matching decision. Every change still goes through the factor harness first (see
**The iron rule for strategy changes** above).
@@ -564,8 +633,11 @@ Research-only flags, all off by default (the default report is byte-identical to
| `BACKTEST_ATR_TARGET_FALLBACK=k` | Synthesizes a k×ATR target where S/R offers none |
| `BACKTEST_FALLBACK_CLEAR_AIR_ONLY=1` | Restricts that fallback to setups with genuinely no structure ahead |
-`recommendation` is the one section surfaced on the deployed page ("What this
-backtest recommends"); everything else in this table is intentionally local-only.
+`portfolio_monitor` and `recommendation` are the sections surfaced on the deployed
+Backtest tab (the monitor chart/tiles and "What this backtest recommends"; the
+recommendation is rebuilt on read, so it always matches the lookback on screen and
+flags one it was not computed on). Everything else in this table is intentionally
+local-only.
## Environment Variables
@@ -583,6 +655,14 @@ Configure in `.env` (copy from `.env.example`):
| `OPENAI_API_KEY` | For sentiment (OpenAI path) | — | OpenAI API key |
| `OPENAI_MODEL` | No | `gpt-4o-mini` | OpenAI model name |
| `OPENAI_SENTIMENT_BATCH_SIZE` | No | `5` | Micro-batch size for sentiment collector |
+| `DEEPSEEK_API_KEY` / `XAI_API_KEY` | For sentiment (those paths) | — | Alternative pluggable sentiment providers |
+| `SEC_USER_AGENT` | **For fundamentals** | placeholder | SEC EDGAR requires a real `name (contact: email)` UA — the shipped default is a placeholder and SEC will throttle/refuse it |
+| `SEC_REQUEST_SPACING_SECONDS` | No | `0.2` | Politeness delay between SEC requests |
+| `SEC_MAX_RETRIES` / `SEC_REQUEST_TIMEOUT_SECONDS` | No | `4` / `30` | SEC client retry and timeout budget |
+| `DOLT_BINARY` | For earnings import | `dolt` | Path to the `dolt` executable |
+| `DOLT_DATA_DIR` / `DOLT_EARNINGS_SUBDIR` | No | `dolt-data` / `earnings` | Local Dolt clone location |
+| `DOLT_MIN_FREE_DISK_GB` | No | `5.0` | Refuse to clone/pull below this free space |
+| `DOLT_COMMAND_TIMEOUT_SECONDS` | No | `600` | Per-command Dolt timeout |
| `FRED_API_KEY` | Optional (risk monitor) | — | FRED key for the AI/Tech risk monitor (VIX, credit spreads) |
| `TELEGRAM_BOT_TOKEN` | Optional (alerts) | — | Telegram bot token for alerts (can also be set in Admin) |
| `TELEGRAM_CHAT_ID` | Optional (alerts) | — | Telegram chat id for alerts |
@@ -591,6 +671,9 @@ Configure in `.env` (copy from `.env.example`):
| `RR_SCAN_FREQUENCY` | No | `daily` | R:R scanner schedule |
| `DEFAULT_WATCHLIST_AUTO_SIZE` | No | `10` | Auto-watchlist size |
| `DEFAULT_RR_THRESHOLD` | No | `1.5` | Minimum R:R ratio for setups |
+| `OHLCV_HISTORY_DAYS` | No | `1825` | Backfill depth for new tickers (~5 years) |
+| `OUTCOME_EVALUATION_MAX_BARS` | No | `30` | Bars the outcome evaluator resolves a setup over |
+| `BACKTEST_WORKERS` | No | `4` | Worker processes for the scheduled backtest |
| `DB_POOL_SIZE` | No | `5` | Database connection pool size |
| `LOG_LEVEL` | No | `INFO` | Logging level |
@@ -683,7 +766,9 @@ app/
├── exceptions.py # Exception hierarchy
├── middleware.py # Global error handler → JSON envelope
├── cache.py # LRU cache with per-ticker invalidation
+├── ssl_bootstrap.py # TLS trust-store bootstrap for outbound calls
├── scheduler.py # APScheduler job definitions
+├── job_catalog.py # Single source of truth for job names + pipeline step lists
├── models/ # SQLAlchemy ORM models
├── schemas/ # Pydantic request/response schemas
├── services/ # Business logic layer
@@ -703,9 +788,11 @@ frontend/
│ ├── admin/ # User table, job controls, settings, data cleanup
│ ├── auth/ # Protected route wrapper
│ ├── charts/ # Canvas candlestick chart
+ │ ├── dashboard/ # Top setups, open trades, shadow-vs-manual performance chart
│ ├── layout/ # App shell, sidebar, mobile nav
│ ├── rankings/ # Rankings table, weights form
│ ├── scanner/ # Trade table
+ │ ├── signals/ # Setups / Paper Trades / Backtest panels
│ ├── ticker/ # Sentiment panel, fundamentals, indicators, S/R overlay
│ ├── ui/ # Badge, toast, skeleton, score card, confirm dialog
│ └── watchlist/ # Watchlist table, add ticker form
@@ -716,16 +803,26 @@ frontend/
└── styles/ # Global CSS with glassmorphism classes
docs/
+├── dolt-integration-plan.md # Design record for the Dolt/SEC fundamentals workstream
+├── dolt-sec-a3-design.md
+├── fundamentals-deployment.md
└── research/ # Experiment log: what was tested, the result, the decision
├── README.md # Overview — start here before proposing a strategy change
- └── sr-levels-and-exits.md
+ ├── sr-levels-and-exits.md
+ ├── post-stop-reentry.md
+ ├── portfolio-capacity-bracket*.md
+ ├── execution-recovery.md
+ ├── fip-breadth-ic.md
+ ├── regime-monitor-v3.md / -v4.md
+ └── … # 16 documents total
reports/ # Committed backtest reports (JSON) + compare_reports.py
deploy/
├── nginx.conf # Reverse proxy + static file serving
├── setup_db.sh # Idempotent DB setup script
-└── stock-data-backend.service # systemd unit
+├── provision_fundamentals.sh # Server-side Dolt/SEC fundamentals provisioning
+└── signalplatform.service # systemd unit
tests/
├── conftest.py # Fixtures, strategies, test DB
@@ -743,9 +840,11 @@ Context for whoever — human or AI — continues this work. The owner pushes st
- **Live scan and backtest share the same pure functions.** The backtest replays production logic through DB-free functions (`compute_technical_from_arrays`, `compute_momentum_from_closes`, `detect_sr_levels`, `detect_gate_target_ladder`, the recommendation helpers). New strategy logic must stay in pure functions consumed by both paths, or the backtest stops measuring what production actually does.
- **Keep the two price-level models separate.** `detect_sr_levels` produces persisted Structural S/R for charts and alerts. `detect_gate_target_ladder` produces transient screening proposals and must never be persisted or presented as market structure. The scanner must not read `SRLevel` rows for target generation.
- **The Gate Target Ladder target is a gate input, never an exit.** `_atr_trailing_close()` does not take it as a parameter, and it must stay that way — take-profit exits were tested and halve CAGR. Any UI or alert that implies the trade exits at the target is a bug ([research](docs/research/sr-levels-and-exits.md#explicit-gate-target-ladder)).
-- **The outcome evaluator evaluates ALL setups**, not just qualified ones — unqualified setups are the control group that makes the Track Record meaningful.
+- **The outcome evaluator evaluates ALL setups**, not just qualified ones — unqualified setups are the control group that makes the realized-outcome record meaningful.
- **`SystemSetting` access goes through `app/services/settings_store.py`** — don't query the model directly.
- **Time-series data gets a real table** (see `benchmark_prices`, `regime_snapshots`); `SystemSetting` JSON is only for config and cached reports.
+- **The shadow book must stay parity-clean.** It orders on the *stored* `strategy_rank` the scanner wrote and mirrors `_simulate_portfolio`'s selection rule; it accepts only a scan from its own pipeline run. Recomputing its ranking, or letting it consume a stale/manual scan, turns the forward OOS record back into an approximation.
+- **Delisted tickers are retired, never deleted.** Live paths opt into `ticker_service.active_only`; the registry, admin views and `run_backtest` deliberately still see them. Deleting a symbol takes the history that a survivorship-bias fix would need.
- **Discretionary overlay data is forward-only.** `signal_context_snapshots` captures composite/dimension/sentiment/fundamental context for new setups. Do not approximate historical sentiment/fundamental snapshots from today's data.
- Style: surgical changes, minimal new files; extend existing services rather than adding parallel ones.
@@ -762,11 +861,14 @@ Context for whoever — human or AI — continues this work. The owner pushes st
| Backtest + factor rank-IC harness ("Signal edge") | `app/services/backtest_service.py` |
| Outcome resolution (target/stop/expired/ambiguous) | `app/services/outcome_service.py` |
| Paper trades + time/trailing/target auto-exit | `app/services/paper_trade_service.py` |
+| Shadow book (automated twin of the backtest's selection) | `app/services/shadow_book_service.py` |
+| Re-entry locks / distinct-day guard / book identities | `app/services/trade_policy.py` |
+| Ticker registry, delisting + `active_only` filter | `app/services/ticker_service.py` |
| Point-in-time setup context snapshots | `app/models/signal_context_snapshot.py` + `app/services/rr_scanner_service.py` |
| Structural S/R detection, Gate Target Ladder & zone clustering | `app/services/sr_service.py` |
| **Research log — what's been tested and rejected** | **`docs/research/`** |
| SPY benchmark for residual momentum + paper-trade alpha | `app/services/benchmark_service.py` |
-| Pipelines & job registration | `app/scheduler.py` |
+| Pipelines & job registration | `app/scheduler.py` (step lists and job names in `app/job_catalog.py`) |
### Verifying changes
@@ -775,7 +877,7 @@ pytest tests/ -q # backend; in-memory SQLite, no Postgres needed
cd frontend && npm run build # full tsc check — this IS the frontend "test"
```
-- `npm test` in `frontend/` is dead (vitest isn't installed; there are no frontend test files). Use `npm run build`.
+- There is no `npm test` in `frontend/` — no test script, no test files. `npm run build` (`tsc -b && vite build`) is the frontend check.
- Backend tests that exercise services which `commit()` need a plain session fixture, not the rolling-back `db_session` — copy the pattern in `tests/unit/test_rr_scanner_integration.py`.
- `ruff` reports ~11 pre-existing errors in old test files; those are not regressions.
@@ -792,6 +894,6 @@ Practical consequences:
### Roadmap (agreed June 2026)
-1. **Forward paper-test the momentum book** — the out-of-sample proof the backtest can't give. Watch Signals → Track Record (live vs backtest).
+1. **Forward paper-test the momentum book** — the out-of-sample proof the backtest can't give. Watch the Dashboard chart (shadow book vs discretionary vs SPY) against Signals → Backtest.
2. **Full IBKR integration** — read real positions, overlay entries/stops on charts, alert on holdings' score deterioration. (Paper trading, the lighter alternative, is done.)
3. Strategy experiments in the order listed under **Strategy Status** above — each one goes through the factor harness first.