research: prepare effective risk floor ab

TradeChart shows a fixed 21-bar window. Once a trade is older than that,
its entry bar precedes the window and entryIdx goes negative, so the price
and trail paths index past the start of series/stopPath and emit NaN
coordinates -- the browser then drops both paths entirely, leaving only the
horizontal level lines. Clamp the index to the left edge and drop the entry
marker when the entry bar is outside the window; the full-width entry line
already carries it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

.env.example

@@ -27,6 +27,35 @@ FINNHUB_API_KEY=
 # Fundamentals Provider — Alpha Vantage (optional fallback)
 ALPHA_VANTAGE_API_KEY=
 
+# Dolt bulk data — local clone of post-no-preference/earnings (workstream A).
+# DOLT_BINARY: path to the dolt CLI (set the full path in dev if it's not on PATH,
+#   e.g. Windows: C:\Program Files\Dolt\bin\dolt.exe). DOLT_DATA_DIR holds the
+#   clones; in PRODUCTION it MUST be outside the deploy tree (deploy is
+#   rsync --delete) — e.g. /var/lib/signal-platform/dolt. The earnings clone lives
+#   at <DOLT_DATA_DIR>/<DOLT_EARNINGS_SUBDIR>. Production setup is automated by
+#   deploy/provision_fundamentals.sh; see docs/fundamentals-deployment.md.
+DOLT_BINARY=dolt
+DOLT_DATA_DIR=dolt-data
+DOLT_EARNINGS_SUBDIR=earnings
+# Free-space floor checked before a pull (clone is ~1.7 GB and grows). 5 GB is a
+# safe production default; lower only on a space-constrained dev box.
+DOLT_MIN_FREE_DISK_GB=5.0
+# Hard timeout (s) on each dolt subprocess so a hung pull/sql can't pin the
+# import connection + advisory lock.
+DOLT_COMMAND_TIMEOUT_SECONDS=600.0
+
+# SEC EDGAR (fundamentals, workstream A). SEC fair-access REQUIRES an identifying
+# User-Agent with a REAL contact email — set it, or requests get 403'd. Stay well
+# under 10 req/s (spacing below).
+SEC_USER_AGENT=signal-platform/1.0 (contact: you@example.com)
+SEC_REQUEST_SPACING_SECONDS=0.2
+SEC_MAX_RETRIES=4
+SEC_REQUEST_TIMEOUT_SECONDS=30.0
+
+# A5 read-only parity report archive. In production keep this outside the
+# rsync deployment tree, e.g. /var/lib/signal-platform/reports/fundamentals-parity.
+FUNDAMENTALS_PARITY_REPORT_DIR=reports/fundamentals-parity
+
 # Regime Monitor — FRED (VIX + HY credit spreads). Free key: https://fred.stlouisfed.org/docs/api/api_key.html
 # Optional: without it the volatility (V1) and credit (C1) pillars show as n/a.
 FRED_API_KEY=

.gitignore

@@ -39,7 +39,18 @@ alembic/versions/__pycache__/
 # Generated SSL bundle
 combined-ca-bundle.pem
 
+# Dolt local dev clones. Production keeps clones in DOLT_DATA_DIR OUTSIDE the
+# repo tree (deploy is rsync --delete of the tree); this dir is dev-only.
+dolt-data/
+
 # Local research artifacts
 # Backtest reports in reports/ are tracked: they are the evidence behind the
 # production baseline in the README. The snapshot DBs they run against are not.
 backtest_snapshots/
+# Rebuildable pickle caches are local accelerators, not decision evidence.
+reports/*.pkl
+reports/*.pk1
+reports/.cache/
+# Runtime A5 parity bundles are generated on the production server. Research
+# conclusions belong in docs/research, not as an ever-growing artifact archive.
+reports/fundamentals-parity/

NOTICE

@@ -0,0 +1,24 @@
+Third-party data attribution
+============================
+
+Earnings calendar and EPS history
+---------------------------------
+This application ingests the earnings calendar and EPS surprise history from the
+public DoltHub repository:
+
+    post-no-preference/earnings
+    https://www.dolthub.com/repositories/post-no-preference/earnings
+
+Licensed under Creative Commons Attribution-ShareAlike 4.0 International
+(CC BY-SA 4.0): https://creativecommons.org/licenses/by-sa/4.0/
+
+Use in this project: private, internal ingestion only. The data is normalized
+into PostgreSQL (`earnings_events`) — the announcement calendar is aligned to the
+EPS history via a minimum-cost monotonic pairing, symbols are normalized, and the
+session field is mapped to bmo/amc/unknown. No public API, bulk export, or
+redistribution of the data is provided. This attribution and the upstream license
+are preserved per the CC BY-SA 4.0 terms. Re-review licensing before any public
+or commercial access.
+
+The post-no-preference/stocks repository (workstream B) is not used at this time
+and would be reviewed separately.

README.md

@@ -2,13 +2,13 @@
 
 Investing-signal platform for US equities. It runs one strategy, and it is a boring one:
 
-> **A long-only cross-sectional momentum book.** Buy the top quintile by beta-adjusted 12-1 month momentum, tilt toward higher volatility, hold at most 10 names, cut at 1.5× ATR, then trail at 3× ATR for up to 30 trading days.
+> **A long-only cross-sectional momentum book.** Buy the top quintile by beta-adjusted 12-1 month momentum, tilt toward higher volatility, hold at most 10 names, cut at 1.5× ATR, then trail at 3× ATR for up to 30 trading days. After an initial-stop exit, re-enter only after the gate has failed and subsequently qualified again.
 
 **Philosophy:** don't predict price — rank it. The edge is *relative* strength across the universe, and the discipline is in the exit: cut losers fast, let winners run until the trail catches them.
 
 **What is NOT the edge — read this before trusting a number on screen.** The composite score, the 5 dimensions, sentiment, fundamentals, and Structural S/R are **display context**, not validated predictors. The Gate Target Ladder is screening machinery that preserves the production setup population; it is not a claim about true market structure. In particular:
 
-- **The headline "target" is not an exit.** It comes from the internal **Gate Target Ladder** and exists only to compute the R:R and reach-probability used by the activation gate. Human-facing chart S/R is a separate model. The live exit reads neither. Across 320 backtested production trades the exit reasons were **144 initial stop, 98 trailing stop, 78 max hold — and 0 targets.** Honoring the target as a take-profit was tested and *halves CAGR* ([research](docs/research/sr-levels-and-exits.md)).
+- **The headline "target" is not an exit.** It comes from the internal **Gate Target Ladder** and exists only to compute the R:R and reach-probability used by the activation gate. Human-facing chart S/R is a separate model. The live exit reads neither. Across 472 trades in the current daily gate-reset replay, the exit reasons were **229 initial stop, 147 trailing stop, 96 max hold — and 0 targets.** Honoring the target as a take-profit was tested and *halves CAGR* ([research](docs/research/sr-levels-and-exits.md)).
 - **The composite score does not select trades.** Residual momentum does.
 
 Full experiment log — everything tested, kept, and rejected: **[docs/research/](docs/research/README.md)**.
@@ -36,23 +36,35 @@ flowchart TD
     BOOK -->|yes| OPEN["OPEN — size at 1% account risk"]
 
     OPEN --> EXIT{"Exit — whichever comes first"}
-    EXIT --> E1["Initial stop hit<br/>entry − 1.5 × ATR → −1R<br/><b>45% of trades</b>"]
+    EXIT --> E1["Initial stop hit<br/>entry − 1.5 × ATR → −1R<br/><b>49% of trades</b>"]
     EXIT --> E2["Trailing stop hit<br/>highest close − 3 × ATR<br/><i>only binds once price is ~1R up</i><br/><b>31% of trades</b>"]
-    EXIT --> E3["Max hold reached<br/>30 trading days<br/><b>24% of trades</b>"]
+    EXIT --> E3["Max hold reached<br/>30 trading days<br/><b>20% of trades</b>"]
     EXIT -.->|"NEVER"| E4["Gate Target Ladder target<br/><b>0% of trades</b>"]
 
+    E1 --> LOCK["Re-entry locked"]
+    LOCK --> GF{"Later daily scan<br/>fails the gate?"}
+    GF -->|no| LOCK
+    GF -->|yes| GQ{"A subsequent daily scan<br/>qualifies again?"}
+    GQ -->|no| GQ
+    GQ -->|yes| RANK
+
     style M fill:#1e3a5f,color:#fff
     style OPEN fill:#1e4d2b,color:#fff
     style E4 fill:#2a2a2a,color:#888
     style E1 fill:#4a1f1f,color:#fff
     style E2 fill:#1e4d2b,color:#fff
+    style LOCK fill:#4a351f,color:#fff
 ```
 
-**How to read the exit box.** The initial stop is tight (1.5× ATR) and the trail is wide (3× ATR), so the trail sits *below* the initial stop at entry and only takes over once price has advanced roughly 1R. Cut fast when wrong; give room once right. That asymmetry is what produces the right-tailed return profile the strategy depends on — most trades lose a little (win rate ~37.5%), a few win big (best trade +12.9R), and *that is why there is no take-profit*.
+**How to read the exit box.** The initial stop is tight (1.5× ATR) and the trail is wide (3× ATR), so the trail sits *below* the initial stop at entry and only takes over once price has advanced roughly 1R. Cut fast when wrong; give room once right. That asymmetry is what produces the right-tailed return profile the strategy depends on — most trades lose a little (win rate 36.2%), a few win big (best trade +12.0R), and *that is why there is no take-profit*.
+
+**What happens after an initial stop.** The stop always closes the trade and realizes its costs. The ticker is then locked until a successful daily full-universe scan first observes it outside the production gate and a later scan observes a fresh qualification. A continuously qualified ticker therefore cannot generate an immediate duplicate entry. Other exit reasons do not start this reset. See the [daily post-stop re-entry study](docs/research/post-stop-reentry.md).
+
+**Live timing matters.** The **only** full-universe R:R scan runs near the US close (~15:30 ET), then Telegram alerts fire immediately so manual fills can still hit MOC. Outcome eval runs later (~16:45 ET) after a fresh OHLCV fetch of the final bar. Morning jobs refresh data/sentiment/regime without scanning. Stops closed by earlier same-day intraday outcome evals can get a **same-day** fail observation at the near-close scan — closer to the promoted research `gate_reset` arm than the old morning-scan `strict_gate_reset` analogue. Stops after the bell still need a later day. Same-day fail+qualify cannot unlock: `trade_policy` requires the failure to fall on an earlier America/New_York trading date.
 
 ## How It Works
 
-Scheduled pipelines turn raw prices into a ranked, gated list of tradeable setups. Everything downstream of OHLCV is recomputed from stored data, so each refresh is cheap and idempotent. Job timing is cron-based and configurable in **Admin → Jobs** (default timezone Europe/Berlin).
+Scheduled pipelines turn raw prices into a ranked, gated list of tradeable setups. Everything downstream of OHLCV is recomputed from stored data, so each refresh is cheap and idempotent. Job timing is cron-based and configurable in **Admin → Jobs** (default timezone **America/New_York** so the near-close scan tracks the cash close through DST).
 
 ### Price-level architecture: two different jobs
 
@@ -115,26 +127,35 @@ percentile rails show each input. Only momentum carries the live activation-
 gate marker. These are cross-sectional scan percentiles, not historical chart
 indicators.
 
-### Daily Load — the full refresh
+### Pipelines (America/New_York)
 
-Once a day (default 07:00). Steps run **in dependency order**, each consuming the previous step's fresh output:
+**Morning** (~02:00 ET) — data and display only, **no** qualifying R:R scan:
 
-1. **OHLCV** — fetch the latest daily bars for every tracked ticker (Alpaca); new tickers backfill ~5 years.
-2. **Sentiment** — fetch sentiment for the names that matter and are stale (> 5 days): top-pick feeders (residual-momentum leaders with a tradeable long setup), the watchlist, and open paper trades, plus a top-N-by-composite discovery net. Runs *before* the scan so the scan sees fresh sentiment.
-3. **R:R Scan** — persist clean Structural S/R for charts/alerts, recompute the 5-dimension scores, and build long/short setups from a transient Gate Target Ladder (ATR stops and nominal gate targets) for every ticker. Attach each ticker's residual 12‑1 momentum activation percentile plus the promoted 80/20 production rank.
-4. **Outcome Eval** — resolve setups that hit target/stop or expired (default 30 trading days) and auto-close paper trades per the exit policy (default: 3x ATR trail with a 30-trading-day max hold).
-5. **Market Regime** — recompute the regime index (breadth/trend).
-6. **Regime Monitor** — separate v2 State/Warning risk thermometer with fixed-basket breadth, VIX, credit, and point-in-time fundamentals; feeds no trades.
+1. **OHLCV** — latest daily bars (Alpaca); new tickers backfill ~5 years.
+2. **Sentiment** — stale names that matter (top-pick feeders, watchlist, open paper, discovery net). Display context only; the activation gate is price-only.
+3. **Market Regime** + **Regime Monitor** — breadth/trend and the v3 risk thermometer; feed no trades.
+4. **Telegram alerts** — change-driven (regime-quadrant etc.); quiet days stay quiet. Setup alerts still fire on the near-close pipeline after the scan.
 
-A failing step is logged; the pipeline continues with the next.
+**Near-close** (~15:30 ET Mon–Fri) — the only full-universe qualifying observation:
+
+1. **OHLCV fetch** — refresh the in-progress day-t bar (same path as intraday).
+2. **R:R Scan** — Structural S/R, scores, Gate Target Ladder setups, residual 12‑1 + 80/20 rank. Advances post-stop gate-reset transitions; failed scans never count.
+3. **Telegram alerts** — chained immediately so manual MOC fills can still hit ~15:50/15:55.
+
+**After close** (~16:45 ET Mon–Fri):
+
+1. **OHLCV fetch** — final bar (not the partial near-close bar).
+2. **Outcome Eval** — resolve setups and auto-close paper trades (default 3× ATR trail, 30-day max hold).
+
+A failing step is logged; the pipeline continues with the next. Near-close duration is logged; warn if &gt; 10 minutes.
 
 ### Intraday — light refresh
 
-Hourly across the US session (Mon–Fri): only **OHLCV → Outcome Eval**, to keep prices current and close paper trades intraday. No scan/sentiment — the dashboard recomputes live R:R from the latest price, so fresh prices are enough.
+Hourly mid-session (Mon–Fri ~10:00–15:00 ET): only **OHLCV → Outcome Eval**, to keep prices current and close paper trades intraday. No scan/sentiment — the dashboard recomputes live R:R from the latest price.
 
 ### Other jobs
 
-Fundamentals (weekly, early Monday) · Alerts (hourly, Telegram) · Backtest (weekly) · Ticker-universe sync (daily). Deep history backfill and event study are manual-only (Admin → Jobs).
+Fundamentals (weekly, early Monday ET) · Backtest (weekly) · Ticker-universe sync (daily). Alerts auto-fire only via the near-close pipeline (still manually triggerable). Deep history backfill and event study are manual-only (Admin → Jobs).
 
 ### From score to "top pick"
 
@@ -155,6 +176,7 @@ Fundamentals (weekly, early Monday) · Alerts (hourly, Telegram) · Backtest (we
 |---|---|---|
 | **Residual 12-1 cross-sectional momentum** (the activation gate, long-only) | **Production gate — in-sample edge** | Promoted July 2026 after the portfolio variant beat raw 80 on CAGR, Sharpe and drawdown. Raw 12-1 remains a fallback only when benchmark data is unavailable |
 | **3× ATR trailing exit** (+ 1.5× ATR initial stop, 30-day max hold) | **Production exit — best Sharpe of every exit tested** | Beat hold / SMA50 / 20-day-low / technical-40 and both take-profit variants (July 2026) |
+| **Post-stop gate reset** | **Production re-entry policy** | The initial stop always closes; the ticker must later fail the daily gate and subsequently qualify again. At the production capacity of 10: Sharpe 1.67 → 1.77, CAGR 45.2% → 48.3%, DD 24.3% → 21.6% versus immediate re-entry. [Full study](docs/research/post-stop-reentry.md) |
 | **Structural S/R** | **Human-facing context only — not a gate and not an exit** | Clean, capped zones are persisted for charts and alerts. The scanner deliberately does not read them. |
 | **Gate Target Ladder** | **Gate input only — not market structure and not an exit** | Volume-free range grid + pivots preserves the useful legacy screening behavior exactly: 1,086/1,086 qualified setups retained and identical Sharpe 2.03 / CAGR 50.0% / DD 21.4% / 321 trades. The exit never reads its target. [Full write-up](docs/research/sr-levels-and-exits.md#explicit-gate-target-ladder) |
 | Composite score + 5 dimensions | **Display/ranking only** | Sub-scores are hand-built heuristics; none has a measured IC. Note: the "momentum" *dimension* is 5/20-day ROC — NOT the validated 12-1 factor (that lives in `momentum_service`) |
@@ -167,14 +189,31 @@ Fundamentals (weekly, early Monday) · Alerts (hourly, Telegram) · Backtest (we
 
 Caveats on the momentum result: in-sample, roughly one market regime, costs/slippage approximated at 0.1% per side, and residual momentum still needs SPY benchmark history to compute. The **out-of-sample proof is the forward paper-trade record**: Signals → Track Record compares live qualified expectancy against the backtest.
 
-### Current production baseline
+### Daily post-stop re-entry decision (2026-07-17)
 
-Use this as a regression guardrail for future strategy changes, not as a return promise. Backtest run: local production SQLite snapshot, 506 tickers, weekly cadence, 30-trading-day horizon, 2022-06-28 → 2026-07-02, 0.1% per-side costs, price-only SPY benchmark. Numbers below are the 2026-07-11 run (`reports/backtest-20260711-prod-baseline.json`) — measured *after* the primary-target probability floor shipped, which pruned lottery-target setups (1,428 → 1,089 qualified) and lifted Sharpe on all three promotion contenders.
+The production policy is **normal gate reset**, evaluated with daily setup opportunities and live-like full-universe ranking. An initial stop always closes. Re-entry unlocks only after a later successful daily scan observes the ticker failing the gate and a subsequent scan observes it qualifying again. The study replayed 1,011,248 point-in-time candidate observations across 505 tickers from 2022-06-24 through 2026-07-02, with the production GTL gate, 80/20 rank, exit, fees, sizing, and 10-position capacity.
 
-| Item | Current baseline |
+| Re-entry policy | Total return | CAGR | Max DD | Sharpe | Trades |
+|---|---:|---:|---:|---:|---:|
+| Immediate | 348.4% | 45.2% | -24.3% | 1.67 | 489 |
+| **Gate reset (selected study arm)** | **388.1%** | **48.3%** | **-21.6%** | **1.77** | **472** |
+| Strict gate reset (live timing analogue) | 342.7% | 44.8% | -23.4% | 1.68 | 471 |
+| Fixed five-session cooldown | 250.8% | 36.6% | -22.2% | 1.47 | 473 |
+
+In the disjoint 2025+ book, gate reset also beat immediate re-entry (Sharpe 1.66 vs 1.55; CAGR 41.8% vs 39.3%) and the fixed five-session rule (Sharpe 1.43; CAGR 32.7%). Its lead over both survived costs of 0.2% and 0.3% per side. The result is capacity-specific: cooldown 5 won at capacity 5, while immediate had slightly higher return and Sharpe at capacity 15. Production uses capacity 10, so that is the portfolio for which this decision is valid.
+
+Those promotion numbers belong to the selected normal-reset study arm. Under the **pre-cutover** morning-scan scheduler (scan always before any outcome eval), live first-observation timing matched the stricter `strict_gate_reset` analogue (full-period Sharpe 1.68 / CAGR 44.8% / DD 23.4%). After the **near-close cutover** (2026-07), stops closed by earlier same-day intraday evals can receive a same-day fail observation at ~15:30 ET — moving live behavior **toward** the promoted `gate_reset` arm. Requalification still requires a later America/New_York trading date than the failure (`trade_policy` distinct-day guard). Full definitions and all nine policy arms: [docs/research/post-stop-reentry.md](docs/research/post-stop-reentry.md); execution evidence: [docs/research/execution-recovery.md](docs/research/execution-recovery.md).
+
+`gate_reset` and a simple `next_session` block happened to produce the same executed live-universe portfolio in this sample. Their rules are still different: this establishes that same-day re-entry was harmful here, but does not isolate a separate historical return premium from the reset condition. Gate reset was promoted because it represents a genuinely new signal episode and did not sacrifice results in the production book. Full definitions, all nine policy arms, cost/capacity sensitivity, and legacy-rank results are in [docs/research/post-stop-reentry.md](docs/research/post-stop-reentry.md); source report: [`reports/daily_reentry_matrix.json`](reports/daily_reentry_matrix.json).
+
+### Historical weekly production baseline (pre gate-reset)
+
+Use this as the historical ranking/exit regression guardrail, not as a return promise or the current re-entry-policy result. This run predates the post-stop gate reset and uses weekly entry replay, so its portfolio headline is not directly comparable with the daily matrix above. Backtest run: local production SQLite snapshot, 506 tickers, weekly cadence, 30-trading-day horizon, 2022-06-28 → 2026-07-02, 0.1% per-side costs, price-only SPY benchmark. Numbers below are the 2026-07-11 run (`reports/backtest-20260711-prod-baseline.json`) — measured *after* the primary-target probability floor shipped, which pruned lottery-target setups (1,428 → 1,089 qualified) and lifted Sharpe on all three promotion contenders.
+
+| Item | Historical weekly baseline |
 |---|---|
 | Strategy version | `residual_highvol_80_20_atr_trail3_v1` |
-| Production gate | Long-only, residual 12-1 momentum percentile >= 80, headline gate-target R:R >= 2.0 (live `activation_min_rr`; the code default is 1.2), primary-target reach-probability >= 20%, NEUTRAL excluded, confidence floor off (0) |
+| Production gate | Long-only, residual 12-1 momentum percentile >= 80, headline gate-target R:R >= 2.0 (live `activation_min_rr`; code default 2.0), primary-target reach-probability >= 20%, NEUTRAL excluded, confidence floor off (0) |
 | Production rank | 80% residual momentum percentile + 20% 6-month realized-volatility percentile |
 | Exit | Initial ATR stop plus 3x ATR trailing stop, max 30 trading days |
 | Portfolio CAGR | +50.4% |
@@ -216,14 +255,22 @@ A systematic single-variable sweep (offline prod snapshot, production gate/rank/
 | ATR trail multiple {1.5–4.0} | **Keep 3.0** | Return+Sharpe peak; ≤2.0 whipsaws out the momentum right tail; ≥2.5 is a plateau |
 | SPY 200d-MA regime overlay (block entries / go flat) | **Reject** | Halves return (315%→138%) with zero drawdown benefit — the ATR trail already manages downside, and the filter blocks the recovery-phase entries that make the money |
 | Momentum lookback: 6-1, 3-1, 12-7 (Novy-Marx), composites | **Keep residual 12-1** | 6-1/3-1 rank-IC ≈ 0; 12-7 IC 0.045 / t 1.58 — weaker than residual 12-1 (0.055 / t 1.98) |
-| Selection cutoff {70, 75, 85, 90} × book size {10, 15, 20} | **Keep 80 × 10** | Monotonically worse in both directions from 80; the 10-slot cap never binds (<10 concurrent) |
+| Selection cutoff {70, 75, 85, 90} × book size {10, 15, 20} | **Keep cutoff 80; capacity reopened** | The older weekly replay favored 80 × 10, but its no-cap-pressure conclusion is superseded by 519 book-full rejections versus 472 trades under the current daily gate-reset control |
 | Position sizing: equal-weight, inverse-vol, risk-% sweep | **Keep 1% fixed-fractional** | See the inverse-vol warning below |
+| Post-stop re-entry: immediate, fixed 2–5 sessions, gate resets, confirmation filters | **Keep normal gate reset for the 10-position production book** | Sharpe 1.77 vs 1.67 immediate and 1.47 cooldown 5; rerun before changing portfolio capacity |
 | FIP path-smoothness as an in-book tie-breaker/filter | **Reject** (but see the lead below) | Non-monotonic across FIP quintiles within the qualified set; either half of a median split underperforms the full book — thinning the entry stream costs more compounding than the tilt returns |
 
+> **Capacity correction (2026-08-05):** the table's older weekly conclusion
+> that the ten-slot cap never binds is superseded. Under the current daily
+> gate-reset Phase A control, 472 trades were admitted and 519 qualified entries
+> were rejected because the book was full (52.4% of admitted+blocked
+> opportunities). Cutoff 80 remains the signal setting; portfolio capacity is
+> reopened in the focused capacity-bracket study.
+
 Two findings future sessions must not re-litigate:
 
 - **The "inverse-vol sizing win" (July 2026) was mis-attributed — do not resurrect.** The diagnostic sized `notional = equity × 1% / vol_6m`, and the 20% notional cap bound on 95% of entries, so it actually measured "~5 positions × 20% notional each" — a concentration/risk-appetite bump economically equivalent to raising risk to 1.5%, not vol-managed sizing. Genuine inverse-vol sizing (risk budget × median-vol/vol) cuts max drawdown to −18.2% but costs ~58pp total return at flat Sharpe: a risk-preference trade, not edge.
-- **`fip_id` — Da/Gurun/Warachka information discreteness over the 12-1 formation window — is the strongest cross-sectional signal measured on this universe: IC −0.045, t = −2.91, correct sign (continuous-information winners outperform).** It clears the iron-rule bar in isolation but does not improve this book (the momentum gate already captures the effect in-sample). It is the prime ranking/gate candidate **if the universe broadens** (e.g. `nasdaq_all`).
+- **`fip_id` — Da/Gurun/Warachka information discreteness over the 12-1 formation window — is the strongest cross-sectional signal on the *production* universe: IC −0.045, t = −2.91, correct sign (continuous-information winners outperform).** It clears the iron-rule bar in isolation but does not improve this book (the momentum gate already captures the effect in-sample). **Phase B (liquid-1500, research branch only):** unconditional fip fails iron rule (−0.017 / t −1.85); mom-conditional fip (−0.088 / t −4.58) is a *book-tilt candidate only* after a baseline breadth mom book is proven. Do **not** cite the orphaned 21:14 row (+0.0575) — it raced a partial `research.sqlite`. See `docs/research/fip-breadth-ic.md`.
 
 ### The iron rule for strategy changes
 
@@ -241,7 +288,7 @@ Corollaries: never let an unvalidated score gate setups; the outcome evaluator m
 
 1. **Forward monitor the promoted strategy** — the production UI now behaves like a portfolio monitor for the current strategy, with selectable lookbacks and SPY comparison. Forward paper-trade months are the only evidence the snapshot cannot provide; the July 2026 tuning pass closed every in-sample lead. (Trailing-stop sensitivity and the max-15 capacity check are done — see the tuning table above.)
 2. **Signal context snapshots** — accumulate point-in-time composite/sentiment/fundamental context for every new setup so the discretionary overlay can be tested forward-only.
-3. **More breadth, not more history** — widening the ranked universe (e.g. `nasdaq_all`) strengthens each week's cross-section and the IC t-stat, even if only the top slice is traded. Now doubly motivated: it is also where the strong `fip_id` signal (see tuning findings) could become tradeable. (Deeper history was considered and declined.)
+3. **Breadth is no longer free leverage** — Phase B found residual-mom t-stat *fell* on liquid-1500 vs the 505-name fingerprint (0.055/1.98 → 0.029/1.33). Any breadth book must clear a pre-registered baseline arm before fip tilts mean anything. (Deeper history was considered and declined.)
 
 ## Key Use Cases
 
@@ -279,7 +326,7 @@ Corollaries: never let an unvalidated score gate setups; the outcome evaluator m
 - Activation gate — qualifies setups on a residual-momentum percentile floor (the actual selection), a headline gate-target R:R floor (prod: 2.0) and a 20% primary-target reach-probability floor (validated long-only edge)
 - Recommendation layer — directional confidence, conflict detection, per-target reach-probability
 - Paper trading — take a setup, mark-to-market vs. latest close, auto-close per the exit policy (default: 3x ATR trail with a 30-trading-day max hold; time / percent-trailing / target-stop selectable), realized track record + outcome evaluation
-- Market-regime guard + observational State/Warning monitor (fixed-basket breadth, VIX, credit, PIT fundamentals) with a manual chronological correction study
+- Market-regime guard + observational State/Warning monitor (fixed-basket breadth, VIX, credit level + impulse) with a manual chronological correction study
 - Telegram alerts (e.g. regime-quadrant changes)
 - User-curated watchlist (cap: 20), enriched with composite score, R:R and S/R summary
 - JWT auth with admin role, configurable registration, user access control
@@ -450,6 +497,14 @@ python scripts/run_backtest_snapshot.py backtest_snapshots/prod.sqlite --workers
 .venv\Scripts\python.exe scripts\run_backtest_snapshot.py backtest_snapshots\prod.sqlite --workers 6 --allow-spawn
 ```
 
+Weekly remains the resource-safe default. Add `--cadence daily` for live-like daily entry opportunities; this performs roughly five times as many setup evaluations. To generate the complete weekly/daily × immediate/gate-reset comparison in one invocation, use:
+
+```bash
+python scripts/run_backtest_cadence_comparison.py backtest_snapshots/prod.sqlite --workers 7
+```
+
+On Windows, add `--allow-spawn`. The comparison runner writes the two full cadence reports plus one compact four-arm report. For the larger nine-policy daily research matrix used in the post-stop decision, see `scripts/run_daily_reentry_matrix.py` and the [research record](docs/research/post-stop-reentry.md).
+
 On an 8-thread machine, `--workers 6` is a good starting point: it leaves a
 couple of threads for Windows, the shell, and browser/UI work while still using
 most of the CPU.
@@ -490,6 +545,7 @@ matching decision. Every change still goes through the factor harness first (see
 | `gate_ablation` | Net expectancy with each floor removed | Drop a floor only if removing it doesn't hurt net expectancy |
 | `time_exit_sweep` | Net avg R / net R-per-day by hold length | Whether a fixed time exit beats the promoted ATR trail |
 | `portfolio_monitor`, `portfolio_sim`, `strategy_variants` | CAGR, Sharpe, max drawdown, per-year returns | Promote a strategy only if it beats the current baseline on CAGR/Sharpe/DD |
+| `production_cadence_comparison` | Immediate vs production gate reset at the selected weekly or daily cadence | Isolates the re-entry rule while keeping gate, rank, exit, fees, sizing, and capacity fixed |
 | `signal_eval` | Mean IC, t-stat, IC>0 %, `reliable` | Iron rule: wire a new factor in only if \|IC\| ≳ 0.03 with a consistent sign and `reliable: true` |
 | `holdout` (opt-in) | Train vs test books, split by entry date | **The only honest OOS read.** Set `BACKTEST_HOLDOUT_SPLIT=YYYY-MM-DD` |
 | `recommendation`, `research_recommendation` | The report's own headline read | A starting point, not a substitute for the sections above |

alembic/versions/022_add_paper_trade_reentry_gate_reset.py

@@ -0,0 +1,53 @@
+"""add persistent post-stop gate-reset observation
+
+Revision ID: 022
+Revises: 021
+Create Date: 2026-07-17 00:00:00.000000
+
+"""
+from typing import Sequence, Union
+
+from alembic import op
+import sqlalchemy as sa
+
+
+revision: str = "022"
+down_revision: Union[str, None] = "021"
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    op.add_column(
+        "paper_trades",
+        sa.Column("reentry_gate_failed_at", sa.DateTime(timezone=True), nullable=True),
+    )
+    op.add_column(
+        "paper_trades",
+        sa.Column(
+            "reentry_gate_requalified_at",
+            sa.DateTime(timezone=True),
+            nullable=True,
+        ),
+    )
+    # The policy starts at this deployment. Historical NULL values mean the
+    # scanner never recorded reset observations, not that those old episodes
+    # are still active. Mark both transitions complete so only stops created
+    # after the migration can open a re-entry lock.
+    op.execute(
+        sa.text(
+            """
+            UPDATE paper_trades
+            SET reentry_gate_failed_at = closed_at,
+                reentry_gate_requalified_at = closed_at
+            WHERE status = 'closed'
+              AND close_reason = 'stop'
+              AND closed_at IS NOT NULL
+            """
+        )
+    )
+
+
+def downgrade() -> None:
+    op.drop_column("paper_trades", "reentry_gate_requalified_at")
+    op.drop_column("paper_trades", "reentry_gate_failed_at")

alembic/versions/023_near_close_schedule_and_fill_mode.py

@@ -0,0 +1,73 @@
+"""near-close schedule cutover + paper trade fill_mode era tag
+
+Revision ID: 023
+Revises: 022
+Create Date: 2026-07-18 00:00:00.000000
+
+"""
+from typing import Sequence, Union
+
+from alembic import op
+import sqlalchemy as sa
+
+
+revision: str = "023"
+down_revision: Union[str, None] = "022"
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+# Deliberate schedule rewrite (not a soft defaults refresh). Old stored values
+# are logged then replaced so prod does not keep scanning at 07:00 Berlin.
+_SCHEDULE_REWRITE: dict[str, str] = {
+    "schedule_timezone": "America/New_York",
+    "schedule_daily_pipeline_cron": "0 2 * * *",
+    "schedule_near_close_pipeline_cron": "30 15 * * 1-5",
+    "schedule_after_close_pipeline_cron": "45 16 * * 1-5",
+    "schedule_intraday_pipeline_cron": "0 10-15 * * 1-5",
+    "schedule_fundamentals_cron": "0 1 * * 1",
+}
+
+
+def upgrade() -> None:
+    op.add_column(
+        "paper_trades",
+        sa.Column("fill_mode", sa.String(length=20), nullable=True),
+    )
+
+    conn = op.get_bind()
+    settings = sa.table(
+        "system_settings",
+        sa.column("id", sa.Integer),
+        sa.column("key", sa.String),
+        sa.column("value", sa.Text),
+        sa.column("updated_at", sa.DateTime(timezone=True)),
+    )
+    now = sa.func.now()
+
+    for key, new_value in _SCHEDULE_REWRITE.items():
+        row = conn.execute(
+            sa.select(settings.c.value).where(settings.c.key == key)
+        ).fetchone()
+        old_value = row[0] if row is not None else None
+        # Always log so ops can recover the pre-cutover schedule from migration output.
+        print(
+            f"schedule_cutover {key}: {old_value!r} -> {new_value!r}",
+            flush=True,
+        )
+        if row is None:
+            conn.execute(
+                sa.insert(settings).values(
+                    key=key, value=new_value, updated_at=now
+                )
+            )
+        else:
+            conn.execute(
+                sa.update(settings)
+                .where(settings.c.key == key)
+                .values(value=new_value, updated_at=now)
+            )
+
+
+def downgrade() -> None:
+    op.drop_column("paper_trades", "fill_mode")
+    # Do not restore old crons — unknown prior values; leave stored schedule as-is.

alembic/versions/024_paper_trade_book.py

@@ -0,0 +1,59 @@
+"""paper trade book tag (manual vs shadow) + weekday cron repair
+
+Revision ID: 024
+Revises: 023
+Create Date: 2026-07-20 00:00:00.000000
+
+Two things ship together because both are corrections to 023's stored state.
+
+1. ``paper_trades.book`` separates the discretionary book from the automatic
+   shadow book. Everything that exists today was opened by hand, so the
+   backfill value is "manual".
+
+2. 023 wrote weekday crons with a numeric day-of-week. APScheduler's
+   from_crontab() feeds field 5 to its own day_of_week where 0=Monday, so
+   "1-5" resolved to Tue-Sat: every Monday was skipped and the scanner ran on
+   Saturdays against stale data. Rewrite only the rows that still hold the
+   broken numeric form, so a hand-corrected setting is never clobbered.
+"""
+from typing import Sequence, Union
+
+from alembic import op
+import sqlalchemy as sa
+
+
+revision: str = "024"
+down_revision: Union[str, None] = "023"
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+# key -> (broken numeric form written by 023, corrected named form)
+_CRON_REPAIR: dict[str, tuple[str, str]] = {
+    "schedule_near_close_pipeline_cron": ("30 15 * * 1-5", "30 15 * * mon-fri"),
+    "schedule_after_close_pipeline_cron": ("45 16 * * 1-5", "45 16 * * mon-fri"),
+    "schedule_intraday_pipeline_cron": ("0 10-15 * * 1-5", "0 10-15 * * mon-fri"),
+    "schedule_fundamentals_cron": ("0 1 * * 1", "0 1 * * mon"),
+}
+
+
+def upgrade() -> None:
+    # server_default backfills existing rows, so no separate UPDATE is needed.
+    op.add_column(
+        "paper_trades",
+        sa.Column("book", sa.String(length=10), nullable=False, server_default="manual"),
+    )
+
+    # Literals are inlined rather than bound because bound parameters render as
+    # NULL under `alembic upgrade --sql`, which would silently produce a script
+    # that matches nothing. Every value here is a constant defined above.
+    for key, (broken, fixed) in _CRON_REPAIR.items():
+        op.execute(
+            f"UPDATE system_settings SET value = '{fixed}' "  # noqa: S608
+            f"WHERE key = '{key}' AND value = '{broken}'"
+        )
+
+
+def downgrade() -> None:
+    op.drop_column("paper_trades", "book")
+    # Crons are deliberately left corrected — restoring the numeric form would
+    # reintroduce the skipped-Monday bug.

alembic/versions/025_trade_setup_scan_run_id.py

@@ -0,0 +1,38 @@
+"""trade_setup scan_run_id — identity of the producing scan run
+
+Revision ID: 025
+Revises: 024
+Create Date: 2026-07-21 00:00:00.000000
+
+The shadow book must select the exact batch produced by its pipeline's scan.
+Matching the scan-completion marker's run id proves which scan wrote last, but
+setup selection was still a detected_at window that a concurrent manual scan
+could write rows into. Stamping each row with its scan's run id lets the shadow
+book select by identity instead. Existing rows are null (they predate the
+column and are never traded by the shadow book).
+"""
+from typing import Sequence, Union
+
+from alembic import op
+import sqlalchemy as sa
+
+
+revision: str = "025"
+down_revision: Union[str, None] = "024"
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    op.add_column(
+        "trade_setups",
+        sa.Column("scan_run_id", sa.String(length=32), nullable=True),
+    )
+    op.create_index(
+        "ix_trade_setups_scan_run_id", "trade_setups", ["scan_run_id"]
+    )
+
+
+def downgrade() -> None:
+    op.drop_index("ix_trade_setups_scan_run_id", table_name="trade_setups")
+    op.drop_column("trade_setups", "scan_run_id")

alembic/versions/026_dolt_fundamentals_schema.py

@@ -0,0 +1,145 @@
+"""Dolt/SEC fundamentals schema — workstream A
+
+Revision ID: 026
+Revises: 025
+Create Date: 2026-07-21 00:00:00.000000
+
+Foundational schema for the Dolt bulk-data integration (workstream A): the
+batch import-run audit table, the SEC-sourced immutable fundamental snapshots
+(CIK-keyed, one row per accession), the Dolt earnings calendar/history, and the
+SEC issuer identity columns on ``tickers``. No data is populated here — the
+importers land in a later phase. ``fundamental_data`` is left untouched; its
+cutover is gated separately (phase A5). ``data_import_runs`` is created first
+because the other two tables carry an ``import_run_id`` FK to it.
+"""
+from typing import Sequence, Union
+
+from alembic import op
+import sqlalchemy as sa
+
+
+revision: str = "026"
+down_revision: Union[str, None] = "025"
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    op.create_table(
+        "data_import_runs",
+        sa.Column("id", sa.Integer(), primary_key=True),
+        sa.Column("source", sa.String(length=32), nullable=False),
+        sa.Column("revision", sa.String(length=64), nullable=True),
+        sa.Column("status", sa.String(length=16), nullable=False),
+        sa.Column("source_max_date", sa.Date(), nullable=True),
+        sa.Column("row_counts_json", sa.Text(), nullable=True),
+        sa.Column("validation_json", sa.Text(), nullable=True),
+        sa.Column("started_at", sa.DateTime(timezone=True), nullable=False),
+        sa.Column("completed_at", sa.DateTime(timezone=True), nullable=True),
+        sa.Column("error_details", sa.Text(), nullable=True),
+    )
+    op.create_index(
+        "ix_data_import_runs_source_started", "data_import_runs", ["source", "started_at"]
+    )
+
+    op.create_table(
+        "fundamental_snapshots",
+        sa.Column("id", sa.Integer(), primary_key=True),
+        sa.Column("cik", sa.String(length=10), nullable=False),
+        sa.Column("accession", sa.String(length=25), nullable=False),
+        sa.Column("form", sa.String(length=12), nullable=False),
+        sa.Column("filed_date", sa.Date(), nullable=False),
+        sa.Column("accepted_at", sa.DateTime(timezone=True), nullable=False),
+        sa.Column("period_start", sa.Date(), nullable=True),
+        sa.Column("period_end", sa.Date(), nullable=False),
+        sa.Column("fiscal_year", sa.Integer(), nullable=False),
+        sa.Column("fiscal_period", sa.String(length=4), nullable=False),
+        # duration facts — cumulative YTD/FY
+        sa.Column("revenue", sa.Float(), nullable=True),
+        sa.Column("net_income", sa.Float(), nullable=True),
+        sa.Column("operating_income", sa.Float(), nullable=True),
+        sa.Column("diluted_eps", sa.Float(), nullable=True),
+        sa.Column("cfo", sa.Float(), nullable=True),
+        sa.Column("capex", sa.Float(), nullable=True),
+        sa.Column("depreciation_amortization", sa.Float(), nullable=True),
+        # balance-sheet facts — period-end
+        sa.Column("cash_and_st_investments", sa.Float(), nullable=True),
+        sa.Column("total_debt", sa.Float(), nullable=True),
+        sa.Column("shares_outstanding", sa.Float(), nullable=True),
+        sa.Column("shares_outstanding_date", sa.Date(), nullable=True),
+        sa.Column(
+            "import_run_id",
+            sa.Integer(),
+            sa.ForeignKey("data_import_runs.id", ondelete="SET NULL"),
+            nullable=True,
+        ),
+        sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
+        sa.UniqueConstraint("accession", name="uq_fundamental_snapshots_accession"),
+    )
+    op.create_index(
+        "ix_fundamental_snapshots_cik_period",
+        "fundamental_snapshots",
+        ["cik", "fiscal_year", "fiscal_period"],
+    )
+    op.create_index(
+        "ix_fundamental_snapshots_cik_period_end",
+        "fundamental_snapshots",
+        ["cik", "period_end"],
+    )
+
+    op.create_table(
+        "earnings_events",
+        sa.Column("id", sa.Integer(), primary_key=True),
+        sa.Column(
+            "ticker_id",
+            sa.Integer(),
+            sa.ForeignKey("tickers.id", ondelete="CASCADE"),
+            nullable=False,
+        ),
+        sa.Column("announce_date", sa.Date(), nullable=False),
+        sa.Column("session", sa.String(length=10), nullable=False),
+        sa.Column("period_end", sa.Date(), nullable=True),
+        sa.Column("eps_estimate", sa.Float(), nullable=True),
+        sa.Column("eps_actual", sa.Float(), nullable=True),
+        sa.Column("source", sa.String(length=32), nullable=False),
+        sa.Column(
+            "import_run_id",
+            sa.Integer(),
+            sa.ForeignKey("data_import_runs.id", ondelete="SET NULL"),
+            nullable=True,
+        ),
+        sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
+        sa.UniqueConstraint("ticker_id", "announce_date", name="uq_earnings_ticker_announce"),
+    )
+    op.create_index(
+        "ix_earnings_events_announce_date", "earnings_events", ["announce_date"]
+    )
+
+    # SEC issuer identity on tickers (nullable; the only ticker<->issuer join point).
+    op.add_column("tickers", sa.Column("cik", sa.String(length=10), nullable=True))
+    op.add_column("tickers", sa.Column("sic", sa.String(length=4), nullable=True))
+    op.add_column(
+        "tickers", sa.Column("sic_description", sa.String(length=160), nullable=True)
+    )
+
+
+def downgrade() -> None:
+    op.drop_column("tickers", "sic_description")
+    op.drop_column("tickers", "sic")
+    op.drop_column("tickers", "cik")
+
+    op.drop_index("ix_earnings_events_announce_date", table_name="earnings_events")
+    op.drop_table("earnings_events")
+
+    op.drop_index(
+        "ix_fundamental_snapshots_cik_period_end", table_name="fundamental_snapshots"
+    )
+    op.drop_index(
+        "ix_fundamental_snapshots_cik_period", table_name="fundamental_snapshots"
+    )
+    op.drop_table("fundamental_snapshots")
+
+    op.drop_index(
+        "ix_data_import_runs_source_started", table_name="data_import_runs"
+    )
+    op.drop_table("data_import_runs")

alembic/versions/027_snapshot_weighted_avg_diluted_shares.py

@@ -0,0 +1,43 @@
+"""fundamental_snapshots.weighted_avg_diluted_shares — market-cap fallback
+
+Revision ID: 027
+Revises: 026
+Create Date: 2026-07-24 00:00:00.000000
+
+Multi-class issuers report the cover-page share count per share class. That is a
+dimensional fact and Company Facts is non-dimensional, so it is absent entirely:
+META has never tagged it, CMCSA stops in 2009, BRK-B in 2011, CHTR in 2016 (when
+the Time Warner Cable deal made it multi-class). `shares_outstanding` is
+therefore null for a large slice of the mega-cap universe, which silently removes
+both `market_cap_est` and `fcf_yield`.
+
+The weighted-average diluted count is always present (EPS requires it) and is
+consolidated across classes. Measured against issuers where the true
+point-in-time count IS available, it lands within ~0.6%: GOOGL 0.9936, MRNA
+1.0045, AAPL 0.9974, MSFT 0.9978.
+
+Stored as its own column rather than backfilled into `shares_outstanding`, so the
+point-in-time column keeps its strict meaning and the fallback stays an explicit,
+labelled read-time decision. Existing rows are null until a reparse.
+"""
+from typing import Sequence, Union
+
+from alembic import op
+import sqlalchemy as sa
+
+
+revision: str = "027"
+down_revision: Union[str, None] = "026"
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    op.add_column(
+        "fundamental_snapshots",
+        sa.Column("weighted_avg_diluted_shares", sa.Float(), nullable=True),
+    )
+
+
+def downgrade() -> None:
+    op.drop_column("fundamental_snapshots", "weighted_avg_diluted_shares")

alembic/versions/028_sec_filing_retry_queue.py

@@ -0,0 +1,147 @@
+"""SEC filing retry queue and setup-quality gate
+
+Revision ID: 028
+Revises: 027
+Create Date: 2026-08-03 00:00:00.000000
+"""
+from datetime import date, datetime, timezone
+import json
+from typing import Sequence, Union
+
+from alembic import op
+import sqlalchemy as sa
+
+
+revision: str = "028"
+down_revision: Union[str, None] = "027"
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    op.create_table(
+        "sec_filing_gaps",
+        sa.Column("id", sa.Integer(), primary_key=True),
+        sa.Column("cik", sa.String(length=10), nullable=False),
+        sa.Column("accession", sa.String(length=25), nullable=False),
+        sa.Column("form", sa.String(length=12), nullable=True),
+        sa.Column("index_date", sa.Date(), nullable=True),
+        sa.Column("reason", sa.String(length=64), nullable=False),
+        sa.Column("coregistrant_ciks_json", sa.Text(), nullable=True),
+        sa.Column("first_seen_at", sa.DateTime(timezone=True), nullable=False),
+        sa.Column("last_attempted_at", sa.DateTime(timezone=True), nullable=False),
+        sa.Column("escalated_at", sa.DateTime(timezone=True), nullable=True),
+        sa.UniqueConstraint("accession", name="uq_sec_filing_gaps_accession"),
+    )
+    op.create_index("ix_sec_filing_gaps_cik", "sec_filing_gaps", ["cik"])
+    _backfill_retry_queue()
+
+
+def downgrade() -> None:
+    op.drop_index("ix_sec_filing_gaps_cik", table_name="sec_filing_gaps")
+    op.drop_table("sec_filing_gaps")
+
+
+def _as_date(value) -> date | None:
+    if isinstance(value, datetime):
+        return value.date()
+    if isinstance(value, date):
+        return value
+    if isinstance(value, str):
+        try:
+            return date.fromisoformat(value)
+        except ValueError:
+            return None
+    return None
+
+
+def _backfill_retry_queue() -> None:
+    """Materialize pre-queue promoted gaps once; runtime never scans history."""
+    bind = op.get_bind()
+    runs = sa.table(
+        "data_import_runs",
+        sa.column("source", sa.String()),
+        sa.column("status", sa.String()),
+        sa.column("validation_json", sa.Text()),
+        sa.column("source_max_date", sa.Date()),
+        sa.column("started_at", sa.DateTime(timezone=True)),
+    )
+    snapshots = sa.table(
+        "fundamental_snapshots",
+        sa.column("cik", sa.String()),
+        sa.column("accession", sa.String()),
+        sa.column("filed_date", sa.Date()),
+    )
+    gaps = sa.table(
+        "sec_filing_gaps",
+        sa.column("cik", sa.String()),
+        sa.column("accession", sa.String()),
+        sa.column("form", sa.String()),
+        sa.column("index_date", sa.Date()),
+        sa.column("reason", sa.String()),
+        sa.column("coregistrant_ciks_json", sa.Text()),
+        sa.column("first_seen_at", sa.DateTime(timezone=True)),
+        sa.column("last_attempted_at", sa.DateTime(timezone=True)),
+        sa.column("escalated_at", sa.DateTime(timezone=True)),
+    )
+
+    snapshot_rows = bind.execute(
+        sa.select(snapshots.c.cik, snapshots.c.accession, snapshots.c.filed_date)
+    ).all()
+    resolved_accessions = {row.accession for row in snapshot_rows}
+    latest_filed_by_cik: dict[str, date] = {}
+    for row in snapshot_rows:
+        if row.filed_date is not None:
+            current = latest_filed_by_cik.get(row.cik)
+            if current is None or row.filed_date > current:
+                latest_filed_by_cik[row.cik] = row.filed_date
+
+    audit_rows = bind.execute(
+        sa.select(
+            runs.c.validation_json,
+            runs.c.source_max_date,
+            runs.c.started_at,
+        ).where(
+            runs.c.source == "sec_facts",
+            runs.c.status == "promoted",
+            runs.c.validation_json.is_not(None),
+        )
+    ).all()
+    now = datetime.now(timezone.utc)
+    candidates: dict[str, dict] = {}
+    for audit in audit_rows:
+        try:
+            summary = json.loads(audit.validation_json)
+        except (TypeError, ValueError):
+            continue
+        if not isinstance(summary, dict):
+            continue
+        for item in summary.get("missing_xbrl") or []:
+            accession = item.get("accession")
+            raw_cik = item.get("cik")
+            if not accession or raw_cik is None or accession in resolved_accessions:
+                continue
+            cik = str(raw_cik).zfill(10)
+            index_date = _as_date(item.get("index_date")) or _as_date(
+                audit.source_max_date
+            )
+            later_filed = latest_filed_by_cik.get(cik)
+            if index_date is not None and later_filed is not None and later_filed > index_date:
+                continue
+            first_seen = audit.started_at or now
+            existing = candidates.get(accession)
+            if existing is not None and existing["first_seen_at"] <= first_seen:
+                continue
+            candidates[accession] = {
+                "cik": cik,
+                "accession": accession,
+                "form": item.get("form"),
+                "index_date": index_date,
+                "reason": item.get("reason") or "not_in_companyfacts",
+                "coregistrant_ciks_json": json.dumps(item.get("coregistrants") or []),
+                "first_seen_at": first_seen,
+                "last_attempted_at": first_seen,
+                "escalated_at": None,
+            }
+    if candidates:
+        op.bulk_insert(gaps, list(candidates.values()))

app/config.py

@@ -37,6 +37,34 @@ class Settings(BaseSettings):
     # Fundamentals Provider — Alpha Vantage (optional fallback)
     alpha_vantage_api_key: str = ""
 
+    # Dolt bulk-data — local clone of post-no-preference/earnings (workstream A).
+    # dolt_binary: full path when not on PATH (dev/Windows install). dolt_data_dir
+    # holds the clones; in production it MUST be outside the deploy tree (deploy is
+    # rsync --delete) — set DOLT_DATA_DIR to a persistent path. The earnings clone
+    # lives at <dolt_data_dir>/<dolt_earnings_subdir>.
+    dolt_binary: str = "dolt"
+    dolt_data_dir: str = "dolt-data"
+    dolt_earnings_subdir: str = "earnings"
+    # Headroom above the ~1.7 GB earnings clone (grows with pulls); 5 GB is a safe
+    # production floor — override lower only in a space-constrained dev box.
+    dolt_min_free_disk_gb: float = 5.0
+    # Bound every dolt subprocess so a hung pull/sql can't pin the import's
+    # connection + advisory lock indefinitely.
+    dolt_command_timeout_seconds: float = 600.0
+
+    # SEC EDGAR (workstream A, fundamentals). Fair-access policy REQUIRES an
+    # identifying User-Agent with a contact email — set a real one. Stay well
+    # under 10 req/s (spacing below); 403 means the UA/pattern is wrong → the
+    # client alerts and stops rather than retry-looping.
+    sec_user_agent: str = "signal-platform/1.0 (contact: set-a-real-email@example.com)"
+    sec_request_spacing_seconds: float = 0.2
+    sec_max_retries: int = 4
+    sec_request_timeout_seconds: float = 30.0
+
+    # A5 read-only comparison artifacts. Production must keep this outside the
+    # rsync deployment tree so the 5-7 day review window survives deploys.
+    fundamentals_parity_report_dir: str = "reports/fundamentals-parity"
+
     # Regime Monitor — FRED (VIX level + HY credit spreads). Optional: without it
     # the volatility (P5) and credit-spread (F2) signals are reported as n/a.
     fred_api_key: str = ""
@@ -59,8 +87,8 @@ class Settings(BaseSettings):
     sentiment_fresh_hours: int = 120
     sentiment_top_composite: int = 30
     fundamental_fetch_frequency: str = "weekly"  # quarterly-ish data; weekly conserves API quota
-    rr_scan_frequency: str = "daily"
-    alerts_frequency: str = "hourly"
+    rr_scan_frequency: str = "daily"  # legacy label; qualifying scan is cron near-close
+    # alerts_frequency removed: alerts fire only via morning + near-close pipelines
     fundamental_rate_limit_retries: int = 3
     fundamental_rate_limit_backoff_seconds: int = 15
     # Pause between tickers in the bulk fundamentals job. Free tiers throttle

app/main.py

@@ -3,56 +3,9 @@
 # ---------------------------------------------------------------------------
 # SSL + proxy injection — MUST happen before any HTTP client imports
 # ---------------------------------------------------------------------------
-import os as _os
-import ssl as _ssl
-from pathlib import Path as _Path
+from app.ssl_bootstrap import bootstrap_ssl
 
-_COMBINED_CERT = _Path(__file__).resolve().parent.parent / "combined-ca-bundle.pem"
-
-if _COMBINED_CERT.exists():
-    _cert_path = str(_COMBINED_CERT)
-    # Env vars for libraries that respect them (requests, urllib3)
-    _os.environ["SSL_CERT_FILE"] = _cert_path
-    _os.environ["REQUESTS_CA_BUNDLE"] = _cert_path
-    _os.environ["CURL_CA_BUNDLE"] = _cert_path
-
-    # Monkey-patch ssl.create_default_context so that ALL libraries
-    # (aiohttp, httpx, google-genai, alpaca-py, etc.) automatically
-    # use our combined CA bundle that includes the corporate root cert.
-    _original_create_default_context = _ssl.create_default_context
-
-    def _patched_create_default_context(
-        purpose=_ssl.Purpose.SERVER_AUTH, *, cafile=None, capath=None, cadata=None
-    ):
-        ctx = _original_create_default_context(
-            purpose, cafile=cafile, capath=capath, cadata=cadata
-        )
-        # Always load our combined bundle on top of whatever was loaded
-        ctx.load_verify_locations(cafile=_cert_path)
-        return ctx
-
-    _ssl.create_default_context = _patched_create_default_context
-
-    # Also patch aiohttp's cached SSL context objects directly, since
-    # aiohttp creates them at import time and may have already cached
-    # a context without our corporate CA bundle.
-    try:
-        import aiohttp.connector as _aio_conn
-        if hasattr(_aio_conn, '_SSL_CONTEXT_VERIFIED') and _aio_conn._SSL_CONTEXT_VERIFIED is not None:
-            _aio_conn._SSL_CONTEXT_VERIFIED.load_verify_locations(cafile=_cert_path)
-        if hasattr(_aio_conn, '_SSL_CONTEXT_UNVERIFIED') and _aio_conn._SSL_CONTEXT_UNVERIFIED is not None:
-            _aio_conn._SSL_CONTEXT_UNVERIFIED.load_verify_locations(cafile=_cert_path)
-    except ImportError:
-        pass
-
-# Corporate proxy — needed when Kiro spawns the process (no .zshrc sourced)
-# Only enable this if explicitly requested via environment variable.
-if _os.environ.get("USE_CORP_PROXY", "0") == "1":
-    _PROXY = "http://aproxy.corproot.net:8080"
-    _NO_PROXY = "corproot.net,sharedtcs.net,127.0.0.1,localhost,bix.swisscom.com,swisscom.com"
-    _os.environ.setdefault("HTTP_PROXY", _PROXY)
-    _os.environ.setdefault("HTTPS_PROXY", _PROXY)
-    _os.environ.setdefault("NO_PROXY", _NO_PROXY)
+bootstrap_ssl()
 
 import logging
 import sys

app/models/__init__.py

@@ -3,6 +3,9 @@ from app.models.ohlcv import OHLCVRecord
 from app.models.user import User
 from app.models.sentiment import SentimentScore
 from app.models.fundamental import FundamentalData
+from app.models.fundamental_snapshot import FundamentalSnapshot
+from app.models.earnings_event import EarningsEvent
+from app.models.data_import_run import DataImportRun
 from app.models.score import DimensionScore, CompositeScore
 from app.models.sr_level import SRLevel
 from app.models.trade_setup import TradeSetup
@@ -14,6 +17,7 @@ from app.models.regime_snapshot import RegimeSnapshot
 from app.models.benchmark_price import BenchmarkPrice
 from app.models.signal_context_snapshot import SignalContextSnapshot
 from app.models.system_event import SystemEvent
+from app.models.sec_filing_gap import SecFilingGap
 
 __all__ = [
     "Ticker",
@@ -21,6 +25,9 @@ __all__ = [
     "User",
     "SentimentScore",
     "FundamentalData",
+    "FundamentalSnapshot",
+    "EarningsEvent",
+    "DataImportRun",
     "DimensionScore",
     "CompositeScore",
     "SRLevel",
@@ -34,4 +41,5 @@ __all__ = [
     "BenchmarkPrice",
     "SignalContextSnapshot",
     "SystemEvent",
+    "SecFilingGap",
 ]

app/models/data_import_run.py

@@ -0,0 +1,44 @@
+from datetime import date, datetime
+
+from sqlalchemy import Date, DateTime, Index, String, Text
+from sqlalchemy.orm import Mapped, mapped_column
+
+from app.database import Base
+
+
+class DataImportRun(Base):
+    """One row per bulk-import attempt (SEC facts / Dolt earnings / Dolt stocks).
+
+    Lean audit record for the batch import framework: every attempt is logged,
+    whether it promoted, was a ``no_op`` (unchanged revision), was ``deferred``
+    for an expected retry, or ``failed``.
+    ``row_counts`` and ``validation`` hold JSON strings (repo convention — see
+    ``fundamental_data.unavailable_fields_json``), not JSONB; the validation
+    blob carries reconciliation/discrepancy summaries so no separate conflicts
+    table is needed. One run per source at a time is enforced at write time by a
+    Postgres advisory lock keyed by ``source``.
+    """
+
+    __tablename__ = "data_import_runs"
+    __table_args__ = (
+        Index("ix_data_import_runs_source_started", "source", "started_at"),
+    )
+
+    id: Mapped[int] = mapped_column(primary_key=True)
+    # sec_facts | dolt_earnings | dolt_stocks
+    source: Mapped[str] = mapped_column(String(32), nullable=False)
+    # Dolt commit hash, or SEC archive SHA-256. Null until known.
+    revision: Mapped[str | None] = mapped_column(String(64), nullable=True)
+    # running | validated | promoted | no_op | deferred | failed
+    status: Mapped[str] = mapped_column(String(16), nullable=False)
+    source_max_date: Mapped[date | None] = mapped_column(Date, nullable=True)
+    row_counts_json: Mapped[str | None] = mapped_column(Text, nullable=True)
+    validation_json: Mapped[str | None] = mapped_column(Text, nullable=True)
+    started_at: Mapped[datetime] = mapped_column(
+        DateTime(timezone=True), default=datetime.utcnow, nullable=False
+    )
+    completed_at: Mapped[datetime | None] = mapped_column(
+        DateTime(timezone=True), nullable=True
+    )
+    # Failure detail, or the non-error reason when status is deferred.
+    error_details: Mapped[str | None] = mapped_column(Text, nullable=True)

app/models/earnings_event.py

@@ -0,0 +1,42 @@
+from datetime import date, datetime
+
+from sqlalchemy import Date, DateTime, Float, ForeignKey, Index, String, UniqueConstraint
+from sqlalchemy.orm import Mapped, mapped_column, relationship
+
+from app.database import Base
+
+
+class EarningsEvent(Base):
+    """Earnings calendar + surprise history, sourced from the DoltHub earnings repo.
+
+    Forward rows (``announce_date`` > today) are the calendar; past rows are
+    results. Rescheduling is handled in the importer's promotion transaction:
+    this source's future-dated rows are deleted and re-inserted from the new
+    snapshot so moved/cancelled dates never linger; past rows are never deleted.
+    """
+
+    __tablename__ = "earnings_events"
+    __table_args__ = (
+        UniqueConstraint("ticker_id", "announce_date", name="uq_earnings_ticker_announce"),
+        Index("ix_earnings_events_announce_date", "announce_date"),
+    )
+
+    id: Mapped[int] = mapped_column(primary_key=True)
+    ticker_id: Mapped[int] = mapped_column(
+        ForeignKey("tickers.id", ondelete="CASCADE"), nullable=False
+    )
+    announce_date: Mapped[date] = mapped_column(Date, nullable=False)
+    # bmo | amc | unknown (source coverage is partial)
+    session: Mapped[str] = mapped_column(String(10), nullable=False, default="unknown")
+    period_end: Mapped[date | None] = mapped_column(Date, nullable=True)
+    eps_estimate: Mapped[float | None] = mapped_column(Float, nullable=True)
+    eps_actual: Mapped[float | None] = mapped_column(Float, nullable=True)
+    source: Mapped[str] = mapped_column(String(32), nullable=False)
+    import_run_id: Mapped[int | None] = mapped_column(
+        ForeignKey("data_import_runs.id", ondelete="SET NULL"), nullable=True
+    )
+    created_at: Mapped[datetime] = mapped_column(
+        DateTime(timezone=True), default=datetime.utcnow, nullable=False
+    )
+
+    ticker = relationship("Ticker", back_populates="earnings_events")

app/models/fundamental_snapshot.py

@@ -0,0 +1,87 @@
+from datetime import date, datetime
+
+from sqlalchemy import Date, DateTime, Float, ForeignKey, Index, String, UniqueConstraint
+from sqlalchemy.orm import Mapped, mapped_column
+
+from app.database import Base
+
+
+class FundamentalSnapshot(Base):
+    """CIK-keyed, one immutable row per SEC accession.
+
+    Keyed by issuer (CIK), not ticker — multi-class issuers (GOOG/GOOGL) share
+    one CIK and one set of fundamentals; the ``tickers.cik`` column is the only
+    join point. Amendments are retained: every accession is a distinct immutable
+    row, and readers resolve (cik, fiscal_year, fiscal_period) at read time by
+    taking the newest ``accepted_at`` **per field**, falling back to the newest
+    accession that actually reports one — a partial amendment (a 10-K/A adding
+    Part III reports no financial facts) must not blank the period — no flags, no mutation.
+
+    **Facts are stored as the filing reports them, never as derived quarters.**
+    Duration facts (revenue, net_income, operating_income, diluted_eps, cfo,
+    capex, depreciation_amortization) hold the filing's normalized **cumulative
+    YTD/FY** value over (period_start -> period_end). Balance-sheet facts
+    (cash_and_st_investments, total_debt, shares_outstanding) are **period-end**
+    values. ``shares_outstanding`` is a single consolidated point-in-time count —
+    the ``dei:EntityCommonStockSharesOutstanding`` cover-page fact, or
+    ``us-gaap:CommonStockSharesOutstanding`` at period end when no dei fact exists
+    (e.g. Alphabet). It is never a class sum (companyfacts is non-dimensional) nor
+    the weighted-average diluted count, since both consumers (estimated market cap,
+    YoY dilution read) want a point-in-time value. Discrete quarters (10-Q YTD
+    deltas, Q4 = FY - Q1..Q3), TTM, YoY and
+    the quarter tape are all derived at read time — so non-calendar fiscal years
+    resolve correctly and a later amendment never leaves a stale frozen quarter.
+    """
+
+    __tablename__ = "fundamental_snapshots"
+    __table_args__ = (
+        UniqueConstraint("accession", name="uq_fundamental_snapshots_accession"),
+        Index("ix_fundamental_snapshots_cik_period", "cik", "fiscal_year", "fiscal_period"),
+        Index("ix_fundamental_snapshots_cik_period_end", "cik", "period_end"),
+    )
+
+    id: Mapped[int] = mapped_column(primary_key=True)
+    cik: Mapped[str] = mapped_column(String(10), nullable=False)
+    accession: Mapped[str] = mapped_column(String(25), nullable=False)
+    form: Mapped[str] = mapped_column(String(12), nullable=False)  # 10-Q, 10-K, 10-K/A ...
+    filed_date: Mapped[date] = mapped_column(Date, nullable=False)
+    # Kept although PIT enforcement is deferred (one timestamp now vs painful retrofit).
+    accepted_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
+
+    # Period identity — required to align non-calendar fiscal years and to derive
+    # discrete quarters from cumulative facts.
+    period_start: Mapped[date | None] = mapped_column(Date, nullable=True)
+    period_end: Mapped[date] = mapped_column(Date, nullable=False)
+    fiscal_year: Mapped[int] = mapped_column(nullable=False)
+    fiscal_period: Mapped[str] = mapped_column(String(4), nullable=False)  # Q1|Q2|Q3|Q4|FY
+
+    # Duration facts — cumulative YTD/FY over (period_start -> period_end).
+    revenue: Mapped[float | None] = mapped_column(Float, nullable=True)
+    net_income: Mapped[float | None] = mapped_column(Float, nullable=True)
+    operating_income: Mapped[float | None] = mapped_column(Float, nullable=True)
+    diluted_eps: Mapped[float | None] = mapped_column(Float, nullable=True)
+    cfo: Mapped[float | None] = mapped_column(Float, nullable=True)  # cash flow from operations
+    capex: Mapped[float | None] = mapped_column(Float, nullable=True)
+    depreciation_amortization: Mapped[float | None] = mapped_column(Float, nullable=True)
+
+    # Balance-sheet facts — period-end values.
+    cash_and_st_investments: Mapped[float | None] = mapped_column(Float, nullable=True)
+    total_debt: Mapped[float | None] = mapped_column(Float, nullable=True)
+    shares_outstanding: Mapped[float | None] = mapped_column(Float, nullable=True)
+    # The cover-page share count (dei:EntityCommonStockSharesOutstanding) is
+    # reported "as of" its own date, which can differ from period_end — store it
+    # so market cap uses the right point-in-time count.
+    shares_outstanding_date: Mapped[date | None] = mapped_column(Date, nullable=True)
+    # Weighted-average diluted count for the filing's most recent quarter — the
+    # market-cap fallback when the cover-page count is absent, which it always is
+    # for multi-class issuers (per-class facts are dimensional, and companyfacts
+    # is not). An average is not cumulative, so unlike the duration facts above
+    # this is NOT a YTD value: it is the shortest-span fact ending at period_end.
+    weighted_avg_diluted_shares: Mapped[float | None] = mapped_column(Float, nullable=True)
+
+    import_run_id: Mapped[int | None] = mapped_column(
+        ForeignKey("data_import_runs.id", ondelete="SET NULL"), nullable=True
+    )
+    created_at: Mapped[datetime] = mapped_column(
+        DateTime(timezone=True), default=datetime.utcnow, nullable=False
+    )

app/models/paper_trade.py

@@ -36,3 +36,25 @@ class PaperTrade(Base):
     closed_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
     # How the trade was closed: "time" | "trailing" | "stop" | "target" | "manual".
     close_reason: Mapped[str | None] = mapped_column(String(10), nullable=True)
+    # A trade stopped at its initial stop starts a re-entry gate-reset episode.
+    # The daily full-universe scanner records both state transitions: the first
+    # failed gate observation and a later fresh qualification. Re-entry remains
+    # non-actionable until both timestamps exist.
+    reentry_gate_failed_at: Mapped[datetime | None] = mapped_column(
+        DateTime(timezone=True), nullable=True
+    )
+    reentry_gate_requalified_at: Mapped[datetime | None] = mapped_column(
+        DateTime(timezone=True), nullable=True
+    )
+    # Execution era for forward vs backtest comparison:
+    # null/legacy = pre-cutover morning-scan, "near_close" = post near-close cutover.
+    fill_mode: Mapped[str | None] = mapped_column(String(20), nullable=True)
+    # Which book this trade belongs to:
+    #   "manual" — discretionary, opened by the user from a qualified setup
+    #   "shadow" — opened automatically by the validated strategy (top-ranked
+    #              qualified up to capacity, 1% risk). The shadow book is the
+    #              faithful live twin of the backtest; the two books share the
+    #              same exit policy so the only difference is *selection*.
+    # Gate-reset re-entry state is tracked per book — the books diverge as soon
+    # as their entries differ, and each must see its own trade history.
+    book: Mapped[str] = mapped_column(String(10), nullable=False, default="manual")

app/models/sec_filing_gap.py

@@ -0,0 +1,32 @@
+from datetime import date, datetime
+
+from sqlalchemy import Date, DateTime, Index, String, Text, UniqueConstraint
+from sqlalchemy.orm import Mapped, mapped_column
+
+from app.database import Base
+
+
+class SecFilingGap(Base):
+    """Active SEC filing that could not yet be reconstructed.
+
+    Rows form a small retry queue. Successful snapshot ingestion deletes the
+    matching row; a later valid filing supersedes it. While a current row remains,
+    tickers mapped to its CIK are not eligible for actionable trade setups.
+    """
+
+    __tablename__ = "sec_filing_gaps"
+    __table_args__ = (
+        UniqueConstraint("accession", name="uq_sec_filing_gaps_accession"),
+        Index("ix_sec_filing_gaps_cik", "cik"),
+    )
+
+    id: Mapped[int] = mapped_column(primary_key=True)
+    cik: Mapped[str] = mapped_column(String(10), nullable=False)
+    accession: Mapped[str] = mapped_column(String(25), nullable=False)
+    form: Mapped[str | None] = mapped_column(String(12), nullable=True)
+    index_date: Mapped[date | None] = mapped_column(Date, nullable=True)
+    reason: Mapped[str] = mapped_column(String(64), nullable=False)
+    coregistrant_ciks_json: Mapped[str | None] = mapped_column(Text, nullable=True)
+    first_seen_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
+    last_attempted_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
+    escalated_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)

app/models/ticker.py

@@ -14,6 +14,13 @@ class Ticker(Base):
     # Company name (e.g. "Biogen Inc."); backfilled from Alpaca, nullable for
     # symbols Alpaca doesn't know.
     name: Mapped[str | None] = mapped_column(String(120), nullable=True)
+    # SEC issuer identity, refreshed by the SEC fundamentals import from
+    # company_tickers.json / submissions. The only ticker<->issuer join point;
+    # multi-class tickers (GOOG/GOOGL) share these values. Nullable: not every
+    # symbol resolves to a CIK (e.g. ADRs, foreign issuers not in SEC data).
+    cik: Mapped[str | None] = mapped_column(String(10), nullable=True)
+    sic: Mapped[str | None] = mapped_column(String(4), nullable=True)
+    sic_description: Mapped[str | None] = mapped_column(String(160), nullable=True)
     created_at: Mapped[datetime] = mapped_column(
         DateTime(timezone=True), default=datetime.utcnow, nullable=False
     )
@@ -28,3 +35,4 @@ class Ticker(Base):
     trade_setups = relationship("TradeSetup", back_populates="ticker", cascade="all, delete-orphan")
     watchlist_entries = relationship("WatchlistEntry", back_populates="ticker", cascade="all, delete-orphan")
     ingestion_progress = relationship("IngestionProgress", back_populates="ticker", cascade="all, delete-orphan", uselist=False)
+    earnings_events = relationship("EarningsEvent", back_populates="ticker", cascade="all, delete-orphan")

app/models/trade_setup.py

@@ -45,6 +45,11 @@ class TradeSetup(Base):
         DateTime(timezone=True), nullable=True
     )
     outcome_date: Mapped[date | None] = mapped_column(Date, nullable=True)
+    # Identity of the scan run that produced this row. The shadow book selects
+    # its batch by this id, not by a detected_at window, so a concurrent manual
+    # scan writing rows in the same time window is excluded by identity. Null on
+    # rows predating the column and on any non-scan creator.
+    scan_run_id: Mapped[str | None] = mapped_column(String(32), nullable=True)
 
     ticker = relationship("Ticker", back_populates="trade_setups")
 

app/providers/alpaca.py

@@ -4,7 +4,7 @@ from __future__ import annotations
 
 import asyncio
 import logging
-from datetime import date
+from datetime import date, datetime, time, timedelta, timezone
 
 from alpaca.data.historical import StockHistoricalDataClient
 from alpaca.data.requests import StockBarsRequest
@@ -16,6 +16,11 @@ from app.providers.protocol import OHLCVData
 
 logger = logging.getLogger(__name__)
 
+# Free plans may not query data from the most recent ~15 minutes, and a window
+# reaching into it fails the *entire* request — which would silently leave the
+# near-close scan on yesterday's close. Margin over the documented boundary.
+_RECENT_DATA_CUTOFF = timedelta(minutes=20)
+
 
 class AlpacaOHLCVProvider:
     """Fetches daily OHLCV bars from Alpaca Markets Data API."""
@@ -25,6 +30,26 @@ class AlpacaOHLCVProvider:
             raise ProviderError("Alpaca API key and secret are required")
         self._client = StockHistoricalDataClient(api_key, api_secret)
 
+    @staticmethod
+    def _resolve_window(start_date: date, end_date: date) -> tuple[datetime, datetime]:
+        """Return the instants covering ``start_date``..``end_date`` inclusive.
+
+        Two boundaries have to be right or today's bar disappears:
+
+        * Daily bars are stamped at the session start in UTC (04:00Z under EDT),
+          so an ``end`` of midnight on ``end_date`` lands *before* that day's bar
+          and silently drops it — extend to the following midnight instead.
+        * The window must stay out of the delayed-data period, otherwise the
+          request is rejected outright with "subscription does not permit
+          querying recent SIP data". Clamping keeps today's in-progress bar
+          available, roughly 20 minutes behind live.
+        """
+        start = datetime.combine(start_date, time.min, tzinfo=timezone.utc)
+        end = datetime.combine(
+            end_date + timedelta(days=1), time.min, tzinfo=timezone.utc
+        )
+        return start, min(end, datetime.now(timezone.utc) - _RECENT_DATA_CUTOFF)
+
     @staticmethod
     def _to_alpaca_symbol(symbol: str) -> str:
         """Convert internal symbol format (BRK-B) to Alpaca format (BRK.B)."""
@@ -40,12 +65,15 @@ class AlpacaOHLCVProvider:
     ) -> list[OHLCVData]:
         """Fetch daily OHLCV bars for *ticker* between *start_date* and *end_date*."""
         alpaca_symbol = self._to_alpaca_symbol(ticker)
+        start, end = self._resolve_window(start_date, end_date)
+        if end <= start:
+            return []
         try:
             request = StockBarsRequest(
                 symbol_or_symbols=alpaca_symbol,
                 timeframe=TimeFrame.Day,
-                start=start_date,
-                end=end_date,
+                start=start,
+                end=end,
                 adjustment=Adjustment.SPLIT,
             )
 

app/providers/fundamentals_chain.py

@@ -101,7 +101,10 @@ class FinnhubFundamentalProvider:
         earnings_payload = earnings_resp.json() if earnings_resp.text else []
 
         metrics = metric_payload.get("metric", {}) if isinstance(metric_payload, dict) else {}
-        market_cap = _safe_float((profile_payload or {}).get("marketCapitalization"))
+        # Finnhub profile2 marketCapitalization is in millions of USD.
+        # Normalize to absolute dollars so cap bands / formatters match FMP & Alpha Vantage.
+        market_cap_millions = _safe_float((profile_payload or {}).get("marketCapitalization"))
+        market_cap = market_cap_millions * 1_000_000.0 if market_cap_millions is not None else None
         pe_ratio = _safe_float(metrics.get("peTTM") or metrics.get("peNormalizedAnnual"))
         revenue_growth = _safe_float(metrics.get("revenueGrowthTTMYoy") or metrics.get("revenueGrowth5Y"))
 

app/routers/admin.py

@@ -13,11 +13,14 @@ from app.schemas.admin import (
     AlertConfigUpdate,
     CreateUserRequest,
     DataCleanupRequest,
+    FundamentalsCutoverConfigUpdate,
     JobTriggerRequest,
     JobToggle,
     RecommendationConfigUpdate,
+    PerformanceConfigUpdate,
     ScheduleConfigUpdate,
     SentimentConfigUpdate,
+    ShadowBookConfigUpdate,
     SentimentTestRequest,
     PasswordReset,
     RegistrationToggle,
@@ -135,6 +138,27 @@ async def list_settings(
     )
 
 
+@router.get("/admin/settings/fundamentals-cutover", response_model=APIEnvelope)
+async def get_fundamentals_cutover_settings(
+    _admin: User = Depends(require_admin),
+    db: AsyncSession = Depends(get_db),
+):
+    config = await admin_service.get_fundamentals_cutover_config(db)
+    return APIEnvelope(status="success", data=config)
+
+
+@router.put("/admin/settings/fundamentals-cutover", response_model=APIEnvelope)
+async def update_fundamentals_cutover_settings(
+    body: FundamentalsCutoverConfigUpdate,
+    _admin: User = Depends(require_admin),
+    db: AsyncSession = Depends(get_db),
+):
+    config = await admin_service.update_fundamentals_cutover_config(
+        db, body.enabled
+    )
+    return APIEnvelope(status="success", data=config)
+
+
 @router.get("/admin/settings/recommendations", response_model=APIEnvelope)
 async def get_recommendation_settings(
     _admin: User = Depends(require_admin),
@@ -201,6 +225,50 @@ async def update_schedule_settings(
     return APIEnvelope(status="success", data=updated)
 
 
+@router.get("/admin/settings/performance", response_model=APIEnvelope)
+async def get_performance_settings(
+    _admin: User = Depends(require_admin),
+    db: AsyncSession = Depends(get_db),
+):
+    return APIEnvelope(
+        status="success", data=await admin_service.get_performance_config(db)
+    )
+
+
+@router.put("/admin/settings/performance", response_model=APIEnvelope)
+async def update_performance_settings(
+    body: PerformanceConfigUpdate,
+    _admin: User = Depends(require_admin),
+    db: AsyncSession = Depends(get_db),
+):
+    updated = await admin_service.update_performance_config(
+        db, body.model_dump(exclude_unset=True)
+    )
+    return APIEnvelope(status="success", data=updated)
+
+
+@router.get("/admin/settings/shadow-book", response_model=APIEnvelope)
+async def get_shadow_book_settings(
+    _admin: User = Depends(require_admin),
+    db: AsyncSession = Depends(get_db),
+):
+    return APIEnvelope(
+        status="success", data=await admin_service.get_shadow_book_config(db)
+    )
+
+
+@router.put("/admin/settings/shadow-book", response_model=APIEnvelope)
+async def update_shadow_book_settings(
+    body: ShadowBookConfigUpdate,
+    _admin: User = Depends(require_admin),
+    db: AsyncSession = Depends(get_db),
+):
+    updated = await admin_service.update_shadow_book_config(
+        db, body.model_dump(exclude_unset=True, exclude_none=True)
+    )
+    return APIEnvelope(status="success", data=updated)
+
+
 @router.get("/admin/settings/sentiment", response_model=APIEnvelope)
 async def get_sentiment_settings(
     _admin: User = Depends(require_admin),
@@ -407,6 +475,36 @@ async def toggle_job(
     )
 
 
+@router.get("/admin/fundamentals-parity", response_model=APIEnvelope)
+async def get_fundamentals_parity_report(
+    _admin: User = Depends(require_admin),
+):
+    """Latest read-only A5 source/score comparison, or null before first run."""
+    return APIEnvelope(
+        status="success", data=admin_service.get_fundamentals_parity_report()
+    )
+
+
+@router.get("/admin/fundamentals-parity/csv", response_model=APIEnvelope)
+async def get_fundamentals_parity_csv(
+    _admin: User = Depends(require_admin),
+):
+    """Latest flattened A5 report for an authenticated browser download."""
+    artifact = admin_service.get_fundamentals_parity_csv()
+    data = None if artifact is None else {"filename": artifact[0], "content": artifact[1]}
+    return APIEnvelope(status="success", data=data)
+
+
+@router.get("/admin/fundamentals-parity/json", response_model=APIEnvelope)
+async def get_fundamentals_parity_json(
+    _admin: User = Depends(require_admin),
+):
+    """Canonical A5 JSON artifact for an authenticated browser download."""
+    artifact = admin_service.get_fundamentals_parity_json()
+    data = None if artifact is None else {"filename": artifact[0], "content": artifact[1]}
+    return APIEnvelope(status="success", data=data)
+
+
 # ---------------------------------------------------------------------------
 # System events (operational warnings / errors)
 # ---------------------------------------------------------------------------

app/routers/fundamentals.py

@@ -9,6 +9,8 @@ from app.dependencies import get_db, require_access
 from app.schemas.common import APIEnvelope
 from app.schemas.fundamental import FundamentalResponse
 from app.services.fundamental_service import get_fundamental
+from app.services.fundamentals_api_service import build_fundamentals_v1
+from app.services import fundamentals_quality_service
 
 router = APIRouter(tags=["fundamentals"])
 
@@ -30,14 +32,14 @@ async def read_fundamentals(
     _user=Depends(require_access),
     db: AsyncSession = Depends(get_db),
 ) -> APIEnvelope:
-    """Get latest fundamental data for a symbol."""
+    """Get latest fundamental data for a symbol (legacy fields + additive v1)."""
     record = await get_fundamental(db, symbol)
+    v1 = await build_fundamentals_v1(db, symbol)
+    quality = await fundamentals_quality_service.ticker_quality(db, symbol)
 
-    if record is None:
-        data = FundamentalResponse(symbol=symbol.strip().upper())
-    else:
-        data = FundamentalResponse(
-            symbol=symbol.strip().upper(),
+    legacy: dict = {}
+    if record is not None:
+        legacy = dict(
             pe_ratio=record.pe_ratio,
             revenue_growth=record.revenue_growth,
             earnings_surprise=record.earnings_surprise,
@@ -47,4 +49,12 @@ async def read_fundamentals(
             unavailable_fields=_parse_unavailable_fields(record.unavailable_fields_json),
         )
 
+    data = FundamentalResponse(
+        symbol=symbol.strip().upper(),
+        setup_eligible=quality.eligible,
+        setup_block_code=quality.code,
+        setup_block_reason=quality.message,
+        **legacy,
+        **v1,
+    )
     return APIEnvelope(status="success", data=data.model_dump())

app/routers/ingestion.py

@@ -19,6 +19,7 @@ from app.dependencies import get_db, require_access
 from app.exceptions import ProviderError
 from app.models.ohlcv import OHLCVRecord
 from app.models.settings import IngestionProgress
+from app.models.sr_level import SRLevel
 from app.models.ticker import Ticker
 from app.models.user import User
 from app.providers.alpaca import AlpacaOHLCVProvider
@@ -105,8 +106,13 @@ async def fetch_symbol(
                 await db.execute(
                     delete(IngestionProgress).where(IngestionProgress.ticker_id == ticker_obj.id)
                 )
+                # Drop Structural S/R with the bars; a failed re-fetch must not
+                # leave zones computed from deleted history.
+                await db.execute(
+                    delete(SRLevel).where(SRLevel.ticker_id == ticker_obj.id)
+                )
                 await db.commit()
-                logger.info("force_refetch: cleared OHLCV and progress for %s", symbol_upper)
+                logger.info("force_refetch: cleared OHLCV, S/R, and progress for %s", symbol_upper)
         except Exception as exc:
             logger.error("force_refetch cleanup failed for %s: %s", symbol_upper, exc)
 

app/routers/paper_trades.py

@@ -65,6 +65,18 @@ async def paper_trade_equity_curve(
     )
 
 
+@router.get("/paper-trades/performance", response_model=APIEnvelope)
+async def paper_trade_performance(
+    user: User = Depends(require_access),
+    db: AsyncSession = Depends(get_db),
+) -> APIEnvelope:
+    """Shadow book vs discretionary book vs SPY since the configured start date."""
+    return APIEnvelope(
+        status="success",
+        data=await paper_trade_service.performance_summary(db, user.id),
+    )
+
+
 @router.put("/paper-trades/exit-policy", response_model=APIEnvelope)
 async def write_exit_policy(
     body: ExitPolicyUpdate,

app/routers/sr_levels.py

@@ -74,7 +74,11 @@ async def read_sr_levels(
         None,
         ge=0,
         le=0.1,
-        description="Merge tolerance as fraction of price; omit for ATR-adaptive default",
+        description=(
+            "Merge tolerance as fraction of price. Omit to return persisted levels "
+            "(ATR-adaptive at last recalculation). When set, returns a transient "
+            "detect with this tolerance (not written to the DB)."
+        ),
     ),
     max_zones: int = Query(6, ge=0, description="Max S/R zones to return (default 6)"),
     _user=Depends(require_access),

app/routers/trades.py

@@ -25,7 +25,7 @@ async def list_trade_setups(
         None,
         description="Filter by action: LONG_HIGH, LONG_MODERATE, SHORT_HIGH, SHORT_MODERATE, NEUTRAL",
     ),
-    _user=Depends(require_access),
+    user=Depends(require_access),
     db: AsyncSession = Depends(get_db),
 ) -> APIEnvelope:
     """Get latest trade setups with recommendation data."""
@@ -36,7 +36,8 @@ async def list_trade_setups(
         recommended_action=recommended_action,
         live_recommendation=True,
         exclude_open_trade_tickers=True,
-        exclude_reentry_lockdown_tickers=True,
+        exclude_open_trade_user_id=user.id,
+        exclude_reentry_gate_locked_tickers=True,
     )
 
     data = []
@@ -76,13 +77,13 @@ async def get_trade_performance(
     _user=Depends(require_access),
     db: AsyncSession = Depends(get_db),
 ) -> APIEnvelope:
-    """Aggregate outcome statistics over evaluated trade setups.
+    """Aggregate setup-outcome statistics (gate barrier diagnostic).
 
-    Outcomes are written by the nightly outcome_evaluator job (win = target
-    hit first, loss = stop hit first, expired = neither within the window).
-    With qualified_only, the overall/direction/action breakdowns cover only
-    setups clearing the activation gate; the confidence breakdown always
-    covers all setups so the gate can be validated against it.
+    Outcomes come from the nightly outcome_evaluator: win = gate target first,
+    loss = stop first, expired = neither in the window. This is **not** the
+    production ATR-trail book; it checks setup grading plumbing only.
+    With qualified_only, overall/direction/action cover only gate-clearing
+    setups; the confidence breakdown always covers all setups.
     """
     config = await admin_service.get_activation_config(db) if qualified_only else None
     stats = await get_performance_stats(db, config=config)
@@ -99,7 +100,7 @@ async def get_ticker_trade_setups(
         db,
         symbol=symbol,
         live_recommendation=True,
-        include_reentry_lockdown=True,
+        include_reentry_gate_lock=True,
     )
     data = []
     for row in rows:

app/scheduler.py

@@ -33,7 +33,24 @@ from app.exceptions import ProviderError
 from app.providers.alpaca import AlpacaOHLCVProvider
 from app.providers.fundamentals_chain import build_fundamental_provider_chain
 from app.providers.protocol import SentimentData
-from app.services import fundamental_service, ingestion_service, sentiment_service, settings_store
+from app.services import (
+    fundamental_service,
+    ingestion_service,
+    pipeline_run,
+    sentiment_service,
+    settings_store,
+    shadow_book_service,
+    fundamentals_parity_service,
+    fundamental_data_refresh_service,
+)
+from app.services.data_import import (
+    STATUS_DEFERRED,
+    STATUS_FAILED,
+    SourceImporter,
+    run_import,
+)
+from app.services.dolt_earnings_importer import DoltEarningsImporter
+from app.services.sec_fundamentals_importer import SecFundamentalsImporter
 from app.services.alert_service import dispatch_alerts
 from app.services.backtest_service import (
     BACKTEST_TARGET_MODELS,
@@ -86,6 +103,9 @@ _JOB_NAMES = [
     "data_backfill",
     "sentiment_collector",
     "fundamental_collector",
+    "dolt_earnings_import",
+    "sec_fundamentals_import",
+    "fundamentals_parity_report",
     "rr_scanner",
     "ticker_universe_sync",
     "alerts",
@@ -93,7 +113,9 @@ _JOB_NAMES = [
     "regime_monitor",
     "event_study",
     "backtest",
-    "daily_pipeline",
+    "daily_pipeline",  # morning: OHLCV/sentiment/regime — no qualifying scan
+    "near_close_pipeline",  # OHLCV fetch → R:R scan → Telegram alerts
+    "after_close_pipeline",  # OHLCV fetch → outcome eval (final bar)
     "intraday_pipeline",
 ]
 
@@ -485,19 +507,29 @@ def _chunked(symbols: list[str], chunk_size: int) -> list[list[str]]:
 # ---------------------------------------------------------------------------
 
 
-async def collect_ohlcv(full_backfill: bool = False, job_name: str = "data_collector") -> None:
+async def collect_ohlcv(
+    full_backfill: bool = False,
+    job_name: str = "data_collector",
+    *,
+    refetch_days: int = 0,
+    refresh_sr: bool = True,
+) -> None:
     """Fetch latest daily OHLCV for all tracked tickers.
 
     Uses AlpacaOHLCVProvider. Processes each ticker independently.
     On rate limit, records last successful ticker for resume.
         Start date is resolved by ingestion progress:
-            - existing ticker: resume from last_ingested_date + 1
+            - existing ticker: overlap last_ingested_date so partial bars refresh
             - new ticker: backfill the configured history window
 
     ``full_backfill`` forces every ticker to re-fetch the full
     ``settings.ohlcv_history_days`` window (ignoring incremental resume) — used by
     the manual data_backfill job to deepen shallow histories. ``job_name`` lets the
     backfill report its own runtime/resume state separate from data_collector.
+
+    ``refetch_days`` re-pulls the last N days regardless of ingestion progress —
+    the after-close run uses it to overwrite the day's partial intraday bar, which
+    resume logic would otherwise skip as "already up to date".
     """
     _log_event(logging.INFO, "job_start", job=job_name)
     _runtime_start(job_name)
@@ -534,11 +566,14 @@ async def collect_ohlcv(full_backfill: bool = False, job_name: str = "data_colle
             return
 
         end_date = date.today()
-        # Full backfill: pass an explicit start_date so fetch_and_ingest re-pulls
-        # the whole window instead of resuming from the last stored bar.
-        backfill_start = (
-            end_date - timedelta(days=settings.ohlcv_history_days) if full_backfill else None
-        )
+        # An explicit start_date makes fetch_and_ingest re-pull that window instead
+        # of resuming from the last stored bar (upsert overwrites, so this is safe).
+        if full_backfill:
+            backfill_start = end_date - timedelta(days=settings.ohlcv_history_days)
+        elif refetch_days:
+            backfill_start = end_date - timedelta(days=refetch_days)
+        else:
+            backfill_start = None
 
         for symbol in symbols:
             _runtime_progress(job_name, processed=processed, total=total, current_ticker=symbol)
@@ -546,6 +581,7 @@ async def collect_ohlcv(full_backfill: bool = False, job_name: str = "data_colle
                 try:
                     result = await ingestion_service.fetch_and_ingest(
                         db, provider, symbol, start_date=backfill_start, end_date=end_date,
+                        refresh_sr=refresh_sr,
                     )
                     _last_successful[job_name] = symbol
                     processed += 1
@@ -585,6 +621,11 @@ async def collect_ohlcv(full_backfill: bool = False, job_name: str = "data_colle
         _runtime_finish(job_name, "error", processed=processed, total=total, message=str(exc))
 
 
+async def collect_ohlcv_for_scan() -> None:
+    """Near-close fetch; the scanner immediately rebuilds S/R per ticker."""
+    await collect_ohlcv(refresh_sr=False)
+
+
 async def backfill_ohlcv() -> None:
     """Deep historical backfill: re-fetch the full ``settings.ohlcv_history_days``
     window for every ticker, ignoring incremental resume.
@@ -596,6 +637,76 @@ async def backfill_ohlcv() -> None:
     await collect_ohlcv(full_backfill=True, job_name="data_backfill")
 
 
+async def run_shadow_book() -> None:
+    """Open the strategy's own positions from the latest qualifying scan.
+
+    The shadow book is the faithful live twin of the backtest: top-ranked
+    qualified setups, up to capacity, 1% risk, no human input. It runs straight
+    after the near-close scan so its entries are marked at the same near-close
+    prices the discretionary book sees, leaving *selection* as the only
+    difference between the two books.
+
+    When run as a pipeline step it acts only on the scan that stamped *this
+    pipeline's* run id (``expected_run_id``): if the pipeline's own scan was
+    disabled or failed, the stored run id is some other scan's — including a
+    manual scan that overlapped and finished last — and shadow refuses.
+    Triggered directly from Admin (no pipeline context) it falls back to the
+    scan-freshness window — an explicit operator action.
+
+    Opt-in (``shadow_book_enabled``) because it writes live trades.
+    """
+    job_name = "shadow_book"
+    expected_run_id = pipeline_run.current()
+    _log_event(logging.INFO, "job_start", job=job_name)
+    _runtime_start(job_name, total=1)
+
+    try:
+        async with async_session_factory() as db:
+            if not await _is_job_enabled(db, job_name):
+                _log_event(logging.INFO, "job_skipped", job=job_name, reason="disabled")
+                _runtime_finish(job_name, "skipped", processed=0, total=1, message="Disabled")
+                return False
+            if not await shadow_book_service.is_enabled(db):
+                _log_event(logging.INFO, "job_skipped", job=job_name, reason="not enabled in settings")
+                _runtime_finish(job_name, "skipped", processed=0, total=1, message="Not enabled")
+                return
+
+            from app.services.admin_service import get_activation_config
+
+            activation_config = await get_activation_config(db)
+            summary = await shadow_book_service.open_shadow_positions(
+                db,
+                activation_config=activation_config,
+                expected_run_id=expected_run_id,
+            )
+            symbols = await shadow_book_service.symbols_for(db, summary["symbols"])
+
+        _runtime_progress(job_name, processed=1, total=1)
+        _runtime_finish(
+            job_name, "completed", processed=1, total=1,
+            message=(
+                f"Opened {summary['opened']} ({', '.join(symbols) if symbols else 'none'}); "
+                f"held {summary['skipped_held']}, gate-locked {summary['skipped_locked']}"
+            ),
+        )
+        _log_event(logging.INFO, "job_complete", job=job_name, opened=summary["opened"], symbols=symbols)
+    except Exception as exc:
+        _runtime_finish(job_name, "error", processed=0, total=1, message=str(exc))
+        _log_event(logging.ERROR, "job_error", job=job_name, error_type=type(exc).__name__, message=str(exc))
+
+
+async def collect_ohlcv_final() -> None:
+    """After-close OHLCV refresh that replaces the day's partial bar.
+
+    Intraday runs store today's bar while the session is still open, so ingestion
+    progress already reads "today" and incremental resume would skip the day
+    entirely — leaving a partial bar as the permanent record. ``refetch_days``
+    forces the last few sessions to be re-pulled so outcome evaluation and
+    fill-quality checks grade against the real close.
+    """
+    await collect_ohlcv(refetch_days=_FINAL_REFETCH_DAYS)
+
+
 # ---------------------------------------------------------------------------
 # Job: Sentiment Collector
 # ---------------------------------------------------------------------------
@@ -727,6 +838,22 @@ async def collect_fundamentals() -> None:
                 _log_event(logging.INFO, "job_skipped", job=job_name, reason="disabled")
                 _runtime_finish(job_name, "skipped", processed=0, total=0, message="Disabled")
                 return
+            if await fundamental_data_refresh_service.is_enabled(db):
+                message = "SEC + Dolt fundamentals cutover is active"
+                _log_event(
+                    logging.INFO,
+                    "job_skipped",
+                    job=job_name,
+                    reason="sec_dolt_cutover_active",
+                )
+                _runtime_finish(
+                    job_name,
+                    "skipped",
+                    processed=0,
+                    total=0,
+                    message=message,
+                )
+                return
 
             symbols = await _get_fundamental_priority_tickers(db)
             if not symbols:
@@ -821,6 +948,184 @@ async def collect_fundamentals() -> None:
         _runtime_finish(job_name, "error", processed=processed, total=total, message=str(exc))
 
 
+# ---------------------------------------------------------------------------
+# Jobs: shadow fundamentals sources
+# ---------------------------------------------------------------------------
+
+
+async def _run_shadow_import(job_name: str, importer: SourceImporter) -> bool:
+    """Run an importer and return whether its scheduled job was enabled.
+
+    The SEC wrapper uses the return value to run its activated local cache step
+    after deferred, failed, no-op, promoted, or source-locked attempts while honoring
+    the job-level disable switch.
+    """
+    _log_event(logging.INFO, "job_start", job=job_name)
+    _runtime_start(job_name, total=1)
+
+    try:
+        async with async_session_factory() as db:
+            if not await _is_job_enabled(db, job_name):
+                _log_event(logging.INFO, "job_skipped", job=job_name, reason="disabled")
+                _runtime_finish(job_name, "skipped", processed=0, total=1, message="Disabled")
+                return
+
+        run = await run_import(importer)
+        if run is None:
+            message = "Another import for this source is already running"
+            _log_event(logging.INFO, "job_skipped", job=job_name, reason="source_locked")
+            _runtime_finish(job_name, "skipped", processed=0, total=1, message=message)
+            return True
+
+        revision = f" · {run.revision[:12]}" if run.revision else ""
+        message = f"{run.status}{revision}"
+        if run.status == STATUS_DEFERRED:
+            message = run.error_details or message
+            _log_event(logging.INFO, "job_deferred", job=job_name, message=message)
+            _runtime_finish(job_name, "deferred", processed=0, total=1, message=message)
+            return True
+        if run.status == STATUS_FAILED:
+            message = run.error_details or message
+            _log_event(logging.ERROR, "job_error", job=job_name, message=message)
+            _runtime_finish(job_name, "error", processed=0, total=1, message=message)
+            return True
+
+        _log_event(
+            logging.INFO,
+            "job_complete",
+            job=job_name,
+            import_status=run.status,
+            revision=run.revision,
+        )
+        _runtime_finish(job_name, "completed", processed=1, total=1, message=message)
+        return True
+    except asyncio.CancelledError:
+        _runtime_finish(job_name, "error", processed=0, total=1, message="Cancelled")
+        raise
+    except Exception as exc:
+        _log_event(
+            logging.ERROR,
+            "job_error",
+            job=job_name,
+            error_type=type(exc).__name__,
+            message=str(exc),
+        )
+        _runtime_finish(job_name, "error", processed=0, total=1, message=str(exc))
+        return True
+
+
+async def run_dolt_earnings_import() -> None:
+    """Pull and import the Dolt earnings calendar/results feed in shadow."""
+    await _run_shadow_import("dolt_earnings_import", DoltEarningsImporter())
+
+
+async def run_sec_fundamentals_import() -> None:
+    """Import SEC facts, then run the activated local compat-cache refresh.
+
+    The refresh is deliberately separate from the network import result. Once
+    activated it therefore still runs from stored snapshots/earnings/prices when
+    SEC is unavailable, unchanged, or another SEC import owns the source lock.
+    """
+    job_name = "sec_fundamentals_import"
+    job_enabled = await _run_shadow_import(job_name, SecFundamentalsImporter())
+    if not job_enabled:
+        return
+
+    try:
+        async with async_session_factory() as db:
+            summary = await fundamental_data_refresh_service.refresh_if_enabled(db)
+    except asyncio.CancelledError:
+        _runtime_finish(
+            job_name, "error", processed=0, total=1, message="Cancelled"
+        )
+        raise
+    except Exception as exc:
+        message = f"Local fundamental_data refresh failed: {exc}"
+        _log_event(
+            logging.ERROR,
+            "fundamental_data_refresh_error",
+            job=job_name,
+            error_type=type(exc).__name__,
+            message=str(exc),
+        )
+        _runtime_finish(job_name, "error", processed=0, total=1, message=message)
+        return
+
+    if not summary["enabled"]:
+        _log_event(
+            logging.INFO,
+            "fundamental_data_refresh_skipped",
+            job=job_name,
+            reason="cutover_disabled",
+            setting=fundamental_data_refresh_service.ACTIVATION_KEY,
+        )
+        return
+
+    _log_event(
+        logging.INFO,
+        "fundamental_data_refresh_complete",
+        job=job_name,
+        **summary,
+    )
+    runtime = get_job_runtime_snapshot(job_name)
+    if runtime.get("status") == "completed":
+        import_message = runtime.get("message") or "import completed"
+        cache_message = (
+            f"cache {summary['refreshed']} · "
+            f"{summary['score_inputs_changed']} score inputs changed"
+        )
+        _runtime_finish(
+            job_name,
+            "completed",
+            processed=1,
+            total=1,
+            message=f"{import_message} · {cache_message}",
+        )
+
+
+async def run_fundamentals_parity_report() -> None:
+    """Generate the A5 comparison bundle without mutating live fundamentals/scores."""
+    job_name = "fundamentals_parity_report"
+    _log_event(logging.INFO, "job_start", job=job_name)
+    _runtime_start(job_name, total=1)
+    try:
+        async with async_session_factory() as db:
+            if not await _is_job_enabled(db, job_name):
+                _runtime_finish(
+                    job_name, "skipped", processed=0, total=1, message="Disabled"
+                )
+                return
+            report, artifacts = await fundamentals_parity_service.generate_and_store(
+                db, settings.fundamentals_parity_report_dir
+            )
+        summary = report["summary"]
+        message = (
+            f"{summary['universe_count']} tickers · "
+            f"{summary['fundamental_score_material_changes']} material score changes"
+        )
+        _runtime_finish(job_name, "completed", processed=1, total=1, message=message)
+        _log_event(
+            logging.INFO,
+            "job_complete",
+            job=job_name,
+            generated_at=report["generated_at"],
+            json_path=artifacts["json"],
+            csv_path=artifacts["csv"],
+        )
+    except asyncio.CancelledError:
+        _runtime_finish(job_name, "error", processed=0, total=1, message="Cancelled")
+        raise
+    except Exception as exc:
+        _runtime_finish(job_name, "error", processed=0, total=1, message=str(exc))
+        _log_event(
+            logging.ERROR,
+            "job_error",
+            job=job_name,
+            error_type=type(exc).__name__,
+            message=str(exc),
+        )
+
+
 # ---------------------------------------------------------------------------
 # Job: R:R Scanner
 # ---------------------------------------------------------------------------
@@ -1177,15 +1482,52 @@ async def sync_ticker_universe() -> None:
 # updates its own runtime status while the pipeline runs.
 #
 # Daily (full): the complete data→signal refresh, once a day.
+# Morning (America/New_York ~02:00): refresh data + display context. No R:R scan
+# — the qualifying full-universe scan runs once near the US close so post-stop
+# gate-reset sees one observation per trading day (plus the trade_policy
+# distinct-day guard for manual re-scans).
+# Sessions re-pulled by the after-close fetch so the consolidated bar overwrites
+# the intraday partial one (covers a long weekend / holiday gap).
+_FINAL_REFETCH_DAYS = 5
+
 _DAILY_PIPELINE_STEPS = [
     ("data_collector", "collect_ohlcv"),
     ("benchmark_collector", "collect_benchmark"),
     ("sentiment_collector", "collect_sentiment"),
-    ("rr_scanner", "scan_rr"),
-    ("outcome_evaluator", "evaluate_outcomes"),
     ("market_regime", "compute_market_regime"),
-    # Observational only — runs here for scheduling; its output feeds nothing else.
+    # Observational only — display/alerts; not trade selection.
     ("regime_monitor", "compute_regime_monitor"),
+    # Alerts after regime so quadrant changes reach Telegram in the morning.
+    # Dispatcher is change-driven; quiet days stay quiet. Setup alerts still
+    # fire on the near-close pipeline after the qualifying scan.
+    ("alerts", "dispatch_alerts_job"),
+]
+
+# Near-close (~15:30 ET Mon–Fri): refresh in-progress day-t bars (incremental
+# ingestion overlaps the latest stored session), then the only daily
+# qualifying R:R scan, then Telegram immediately so manual fills can still hit
+# MOC cutoffs (~15:50/15:55). Under a 15-minute delayed SIP feed a 15:30 scan
+# may see ~15:15 prices — immaterial for a 12-1 momentum signal.
+#
+# US early-close days (~3/year, 13:00 ET close): this job runs post-close and
+# entries behave like stale_close (still acceptable per execution-recovery matrix).
+# No exchange calendar dependency.
+_NEAR_CLOSE_PIPELINE_STEPS = [
+    # Must land today's in-progress bar (~20 min behind live), or the scan falls
+    # back to the previous close and execution degrades to the stale_close floor.
+    ("data_collector", "collect_ohlcv_for_scan"),
+    ("rr_scanner", "scan_rr"),
+    # Straight after the scan so shadow entries mark at the same near-close
+    # prices the discretionary book is looking at.
+    ("shadow_book", "run_shadow_book"),
+    ("alerts", "dispatch_alerts_job"),
+]
+
+# After close (~16:45 ET Mon–Fri): fresh OHLCV fetch so outcomes resolve on the
+# final bar, not the near-close partial bar, then outcome/paper close.
+_AFTER_CLOSE_PIPELINE_STEPS = [
+    ("data_collector", "collect_ohlcv_final"),
+    ("outcome_evaluator", "evaluate_outcomes"),
 ]
 
 # Intraday (light): keep prices current and resolve outcomes through the day,
@@ -1197,12 +1539,21 @@ _INTRADAY_PIPELINE_STEPS = [
     ("outcome_evaluator", "evaluate_outcomes"),
 ]
 
+# Warn if near-close fetch+scan+alert drifts past this — entries leave the close
+# and the stale_close floor quietly becomes the ceiling.
+_NEAR_CLOSE_DURATION_WARN_SECONDS = 600
+
 
 async def _run_pipeline(job_name: str, steps: list[tuple[str, str]]) -> None:
     """Run an ordered list of (step_name, coroutine_name) steps.
 
     Each step respects its own enable flag and manages its own runtime status; a
     failing step is logged and the pipeline continues with the next one.
+
+    A unique run id is bound for the invocation and visible to every step via the
+    shared task context: the scan step stamps it into its completion markers and
+    the shadow step requires an exact match, so only a scan that ran inside this
+    pipeline can drive the shadow book.
     """
     _log_event(logging.INFO, "job_start", job=job_name)
     async with async_session_factory() as db:
@@ -1216,6 +1567,7 @@ async def _run_pipeline(job_name: str, steps: list[tuple[str, str]]) -> None:
 
     funcs = globals()
     done = 0
+    token = pipeline_run.bind(pipeline_run.new_run_id())
     try:
         for step_name, func_name in steps:
             _runtime_progress(job_name, processed=done, total=total, current_ticker=step_name)
@@ -1229,14 +1581,49 @@ async def _run_pipeline(job_name: str, steps: list[tuple[str, str]]) -> None:
     except Exception as exc:
         _runtime_finish(job_name, "error", processed=done, total=total, message=str(exc))
         _log_event(logging.ERROR, "job_error", job=job_name, error_type=type(exc).__name__, message=str(exc))
+    finally:
+        pipeline_run.release(token)
 
 
 async def run_daily_pipeline() -> None:
-    """Full daily flow: OHLCV → benchmark → sentiment → R:R scan → outcome eval
-    (+paper close) → market regime."""
+    """Morning flow: OHLCV → benchmark → sentiment → market regime (no scan)."""
     await _run_pipeline("daily_pipeline", _DAILY_PIPELINE_STEPS)
 
 
+async def run_near_close_pipeline() -> None:
+    """Near-close flow: OHLCV fetch → R:R scan → Telegram alerts.
+
+    Logs wall duration; warn if past 10 minutes so operators notice close drift.
+    """
+    import time
+
+    started = time.monotonic()
+    await _run_pipeline("near_close_pipeline", _NEAR_CLOSE_PIPELINE_STEPS)
+    elapsed = time.monotonic() - started
+    payload = {
+        "job": "near_close_pipeline",
+        "duration_seconds": round(elapsed, 1),
+    }
+    if elapsed > _NEAR_CLOSE_DURATION_WARN_SECONDS:
+        _log_event(
+            logging.WARNING,
+            "near_close_pipeline_slow",
+            **payload,
+            threshold_seconds=_NEAR_CLOSE_DURATION_WARN_SECONDS,
+            message=(
+                "Near-close pipeline exceeded 10 minutes — entries may drift from "
+                "the close toward the stale_close research floor"
+            ),
+        )
+    else:
+        _log_event(logging.INFO, "near_close_pipeline_duration", **payload)
+
+
+async def run_after_close_pipeline() -> None:
+    """After-close flow: OHLCV fetch (final bar) → outcome eval (+paper close)."""
+    await _run_pipeline("after_close_pipeline", _AFTER_CLOSE_PIPELINE_STEPS)
+
+
 async def run_intraday_pipeline() -> None:
     """Light intraday flow: refresh OHLCV → evaluate outcomes (+paper close)."""
     await _run_pipeline("intraday_pipeline", _INTRADAY_PIPELINE_STEPS)
@@ -1268,16 +1655,40 @@ def _parse_frequency(freq: str) -> dict[str, int]:
 # every process restart, so on a box that's redeployed often it can keep being
 # deferred and never fire. Cron fires at a fixed local time regardless.
 
+# All wall times are America/New_York after the near-close execution cutover.
+# Stored SystemSetting values shadow these defaults — deploy migration 023
+# rewrites schedule_* keys so prod does not keep scanning at 07:00 Berlin.
+# DAY-OF-WEEK MUST BE NAMES, NEVER NUMBERS. APScheduler's from_crontab() passes
+# field 5 straight to its own day_of_week, where 0=Monday — so "1-5" resolves to
+# Tue–Sat, silently skipping every Monday and scanning on Saturdays. Names are
+# unambiguous in both dialects.
 SCHEDULE_DEFAULTS: dict[str, str] = {
-    "schedule_timezone": "Europe/Berlin",
-    "schedule_daily_pipeline_cron": "0 7 * * *",        # full refresh, ready by ~8am
-    "schedule_intraday_pipeline_cron": "0 14-22 * * 1-5",  # hourly across the US session
-    "schedule_fundamentals_cron": "0 4 * * 1",          # weekly, early Monday (slow job)
+    "schedule_timezone": "America/New_York",
+    # Morning data/display refresh (no qualifying R:R scan).
+    "schedule_daily_pipeline_cron": "0 2 * * *",
+    # Bulk source imports. The SEC job writes the legacy compat cache only after
+    # the explicit, default-off A5 cutover setting is enabled.
+    "schedule_dolt_earnings_cron": "30 2 * * *",
+    "schedule_sec_fundamentals_cron": "0 4 * * *",
+    "schedule_fundamentals_parity_cron": "30 5 * * *",
+    # Fetch in-progress bars → scan → Telegram (manual MOC window).
+    "schedule_near_close_pipeline_cron": "30 15 * * mon-fri",
+    # Fetch final bars → outcome eval (must not run on the partial near-close bar).
+    "schedule_after_close_pipeline_cron": "45 16 * * mon-fri",
+    # Hourly mid-session price + outcome (10:00–15:00 ET Mon–Fri).
+    "schedule_intraday_pipeline_cron": "0 10-15 * * mon-fri",
+    # Weekly fundamentals early Monday NY.
+    "schedule_fundamentals_cron": "0 1 * * mon",
 }
 
 # job id -> schedule setting key
 _CRON_JOBS: dict[str, str] = {
     "daily_pipeline": "schedule_daily_pipeline_cron",
+    "dolt_earnings_import": "schedule_dolt_earnings_cron",
+    "sec_fundamentals_import": "schedule_sec_fundamentals_cron",
+    "fundamentals_parity_report": "schedule_fundamentals_parity_cron",
+    "near_close_pipeline": "schedule_near_close_pipeline_cron",
+    "after_close_pipeline": "schedule_after_close_pipeline_cron",
     "intraday_pipeline": "schedule_intraday_pipeline_cron",
     "fundamental_collector": "schedule_fundamentals_cron",
 }
@@ -1343,6 +1754,7 @@ def configure_scheduler(schedule_config: dict[str, str] | None = None) -> None:
         (collect_benchmark, "benchmark_collector", "Benchmark Collector"),
         (collect_sentiment, "sentiment_collector", "Sentiment Collector"),
         (scan_rr, "rr_scanner", "R:R Scanner"),
+        (run_shadow_book, "shadow_book", "Shadow Book (auto-traded strategy)"),
         (evaluate_outcomes, "outcome_evaluator", "Outcome Evaluator"),
         (compute_market_regime, "market_regime", "Market Regime"),
         (compute_regime_monitor, "regime_monitor", "Regime Monitor"),
@@ -1357,7 +1769,62 @@ def configure_scheduler(schedule_config: dict[str, str] | None = None) -> None:
     scheduler.add_job(
         run_daily_pipeline,
         _cron_trigger(cfg["schedule_daily_pipeline_cron"], tz, "schedule_daily_pipeline_cron"),
-        id="daily_pipeline", name="Daily Pipeline", replace_existing=True,
+        id="daily_pipeline", name="Morning Pipeline", replace_existing=True,
+    )
+    scheduler.add_job(
+        run_dolt_earnings_import,
+        _cron_trigger(
+            cfg["schedule_dolt_earnings_cron"],
+            tz,
+            "schedule_dolt_earnings_cron",
+        ),
+        id="dolt_earnings_import",
+        name="Dolt Earnings Import (shadow)",
+        replace_existing=True,
+    )
+    scheduler.add_job(
+        run_sec_fundamentals_import,
+        _cron_trigger(
+            cfg["schedule_sec_fundamentals_cron"],
+            tz,
+            "schedule_sec_fundamentals_cron",
+        ),
+        id="sec_fundamentals_import",
+        name="SEC Fundamentals Import",
+        replace_existing=True,
+    )
+    scheduler.add_job(
+        run_fundamentals_parity_report,
+        _cron_trigger(
+            cfg["schedule_fundamentals_parity_cron"],
+            tz,
+            "schedule_fundamentals_parity_cron",
+        ),
+        id="fundamentals_parity_report",
+        name="Fundamentals Parity Report (read-only)",
+        replace_existing=True,
+    )
+    scheduler.add_job(
+        run_near_close_pipeline,
+        _cron_trigger(
+            cfg["schedule_near_close_pipeline_cron"],
+            tz,
+            "schedule_near_close_pipeline_cron",
+        ),
+        id="near_close_pipeline",
+        name="Near-Close Pipeline (scan+alert)",
+        replace_existing=True,
+    )
+    scheduler.add_job(
+        run_after_close_pipeline,
+        _cron_trigger(
+            cfg["schedule_after_close_pipeline_cron"],
+            tz,
+            "schedule_after_close_pipeline_cron",
+        ),
+        id="after_close_pipeline",
+        name="After-Close Pipeline (outcome)",
+        replace_existing=True,
     )
     scheduler.add_job(
         run_intraday_pipeline,
@@ -1377,10 +1844,12 @@ def configure_scheduler(schedule_config: dict[str, str] | None = None) -> None:
         sync_ticker_universe, "interval", hours=24,
         id="ticker_universe_sync", name="Ticker Universe Sync", replace_existing=True,
     )
-    alerts_interval = _parse_frequency(settings.alerts_frequency)
+    # Alerts auto-fire only via near_close_pipeline (scan → alert before MOC).
+    # Keep the job registered for Admin manual trigger; no independent interval.
     scheduler.add_job(
-        dispatch_alerts_job, "interval", **alerts_interval,
-        id="alerts", name="Alerts Dispatcher", replace_existing=True,
+        dispatch_alerts_job, "interval", weeks=520,
+        id="alerts", name="Alerts Dispatcher",
+        replace_existing=True, next_run_time=None,
     )
     scheduler.add_job(
         run_backtest_job, "interval", hours=168,
@@ -1400,10 +1869,32 @@ def configure_scheduler(schedule_config: dict[str, str] | None = None) -> None:
         replace_existing=True, next_run_time=None,
     )
 
-    _log_event(logging.INFO, "scheduler_configured", timezone=tz, daily_pipeline={
+    _log_event(
+        logging.INFO,
+        "scheduler_configured",
+        timezone=tz,
+        daily_pipeline={
             "cron": cfg["schedule_daily_pipeline_cron"],
             "steps": [name for name, _ in _DAILY_PIPELINE_STEPS],
-            }, intraday_pipeline={
+        },
+        dolt_earnings_import={"cron": cfg["schedule_dolt_earnings_cron"]},
+        sec_fundamentals_import={"cron": cfg["schedule_sec_fundamentals_cron"]},
+        fundamentals_parity_report={
+            "cron": cfg["schedule_fundamentals_parity_cron"]
+        },
+        near_close_pipeline={
+            "cron": cfg["schedule_near_close_pipeline_cron"],
+            "steps": [name for name, _ in _NEAR_CLOSE_PIPELINE_STEPS],
+        },
+        after_close_pipeline={
+            "cron": cfg["schedule_after_close_pipeline_cron"],
+            "steps": [name for name, _ in _AFTER_CLOSE_PIPELINE_STEPS],
+        },
+        intraday_pipeline={
             "cron": cfg["schedule_intraday_pipeline_cron"],
             "steps": [name for name, _ in _INTRADAY_PIPELINE_STEPS],
-            }, fundamental_collector={"cron": cfg["schedule_fundamentals_cron"]}, independent=["ticker_universe_sync", "alerts", "backtest"])
+        },
+        fundamental_collector={"cron": cfg["schedule_fundamentals_cron"]},
+        independent=["ticker_universe_sync", "backtest"],
+        manual_only=["alerts", "data_backfill", "event_study"],
+    )

app/schemas/admin.py

@@ -73,15 +73,44 @@ class ActivationConfigUpdate(BaseModel):
     exclude_neutral: bool | None = None
 
 
+class FundamentalsCutoverConfigUpdate(BaseModel):
+    """Switch the legacy fundamentals cache from quota APIs to SEC/Dolt."""
+    enabled: bool
+
+
 class ScheduleConfigUpdate(BaseModel):
     """Cron schedule for the pipelines + fundamentals. Crons are 5-field
-    (min hour dom month dow); timezone is an IANA name (e.g. Europe/Berlin)."""
+    (min hour dom month dow); timezone is an IANA name (e.g. America/New_York)."""
     schedule_timezone: str | None = Field(default=None, max_length=64)
     schedule_daily_pipeline_cron: str | None = Field(default=None, max_length=120)
+    schedule_dolt_earnings_cron: str | None = Field(default=None, max_length=120)
+    schedule_sec_fundamentals_cron: str | None = Field(default=None, max_length=120)
+    schedule_fundamentals_parity_cron: str | None = Field(default=None, max_length=120)
+    schedule_near_close_pipeline_cron: str | None = Field(default=None, max_length=120)
+    schedule_after_close_pipeline_cron: str | None = Field(default=None, max_length=120)
     schedule_intraday_pipeline_cron: str | None = Field(default=None, max_length=120)
     schedule_fundamentals_cron: str | None = Field(default=None, max_length=120)
 
 
+class PerformanceConfigUpdate(BaseModel):
+    """Window for the Performance comparison.
+
+    ``start_date`` is an ISO date, or empty string to show all history. The
+    strategy has been revised repeatedly; pinning a start keeps the shadow-vs-
+    manual comparison inside one configuration instead of averaging across
+    rules that no longer exist.
+    """
+    start_date: str | None = Field(default=None, max_length=10)
+
+
+class ShadowBookConfigUpdate(BaseModel):
+    """Auto-traded shadow book: the validated strategy with no human input."""
+    enabled: bool | None = None
+    capacity: int | None = Field(default=None, ge=1, le=100)
+    risk_pct: float | None = Field(default=None, gt=0, le=10)
+    start_equity: float | None = Field(default=None, ge=1000)
+
+
 class SentimentConfigUpdate(BaseModel):
     """Runtime sentiment LLM config. api_key is write-only; omit/empty to keep
     the stored key."""

app/schemas/fundamental.py

@@ -7,8 +7,75 @@ from datetime import date, datetime
 from pydantic import BaseModel
 
 
+class MetricIndustry(BaseModel):
+    label: str
+    median: float
+    favorable_percentile: int  # 0-100, polarity-aware (higher = more favorable)
+    peer_count: int
+
+
+class MetricHistoryPoint(BaseModel):
+    period_end: str  # YYYY-MM-DD
+    value: float | None
+
+
+class MetricItem(BaseModel):
+    key: str
+    value: float | None = None
+    history: list[MetricHistoryPoint] = []
+    industry: MetricIndustry | None = None
+    period_end: str | None = None
+    filed_date: str | None = None
+    caveat: str | None = None
+    source: str = "sec"
+
+
+class EarningsNext(BaseModel):
+    date: str
+    session: str
+    days_until: int
+
+
+class EarningsRecent(BaseModel):
+    announce_date: str
+    period_end: str | None = None
+    eps_estimate: float | None = None
+    eps_actual: float | None = None
+    surprise_pct: float | None = None
+
+
+class EarningsObject(BaseModel):
+    next: EarningsNext | None = None
+    recent: list[EarningsRecent] = []
+
+
+class Valuation(BaseModel):
+    pe: float | None = None
+    fcf_yield: float | None = None
+    market_cap_est: float | None = None
+    pe_industry: MetricIndustry | None = None
+    fcf_yield_industry: MetricIndustry | None = None
+    price_date: str | None = None
+
+
+class FundamentalsReads(BaseModel):
+    """Deterministic text outputs, separate from the numeric metrics.
+
+    ``by_key`` is a fixed map over every metric key plus ``pe`` and ``fcf_yield``,
+    each a read string or null. ``header`` is null when there is no read at all."""
+
+    header: str | None = None
+    by_key: dict[str, str | None] = {}
+
+
 class FundamentalResponse(BaseModel):
-    """Envelope-ready fundamental data response."""
+    """Envelope-ready fundamental data response.
+
+    Legacy fields are preserved unchanged (they come from ``fundamental_data`` /
+    the legacy providers). The additive v1 objects — earnings, metrics, valuation,
+    reads — are SEC/Dolt-derived and independent; a null legacy field is never
+    mapped onto the new SEC metrics and vice-versa.
+    """
 
     symbol: str
     pe_ratio: float | None = None
@@ -18,3 +85,12 @@ class FundamentalResponse(BaseModel):
     next_earnings_date: date | None = None
     fetched_at: datetime | None = None
     unavailable_fields: dict[str, str] = {}
+
+    # --- additive v1 (always present; empty/null when unavailable) ---
+    earnings: EarningsObject | None = None
+    metrics: list[MetricItem] | None = None
+    valuation: Valuation | None = None
+    reads: FundamentalsReads | None = None
+    setup_eligible: bool = True
+    setup_block_code: str | None = None
+    setup_block_reason: str | None = None

app/schemas/paper_trade.py

@@ -47,7 +47,13 @@ class PaperTradeResponse(BaseModel):
     alpha_pct: float | None = None
     alpha_usd: float | None = None
     close_reason: str | None = None
+    # Execution era: null = pre-cutover / unknown; "near_close" = post schedule cutover.
+    fill_mode: str | None = None
     # Live trailing-stop level + how far price sits above it (% ), for open trades
     # when the trailing exit policy is active.
     trailing_stop: float | None = None
     trailing_distance_pct: float | None = None
+    # Trading sessions represented by post-entry OHLCV bars. These are populated
+    # only while the active exit policy has a max-hold rule.
+    sessions_held: int | None = None
+    sessions_remaining: int | None = None

app/schemas/trade_setup.py

@@ -59,6 +59,6 @@ class TradeSetupResponse(BaseModel):
     momentum_percentile: float | None = None
     strategy_rank: float | None = None
     volatility_percentile: float | None = None
-    reentry_lockdown_remaining_sessions: int | None = None
+    reentry_gate_reset_required: bool = False
     context_as_of: TradeSetupContextAsOfResponse | None = None
     recommendation_summary: RecommendationSummaryResponse | None = None

app/services/admin_service.py

@@ -17,7 +17,7 @@ from app.models.settings import SystemSetting
 from app.models.ticker import Ticker
 from app.models.trade_setup import TradeSetup
 from app.models.user import User
-from app.services import settings_store
+from app.services import fundamental_data_refresh_service, settings_store
 
 logger = logging.getLogger(__name__)
 
@@ -54,7 +54,9 @@ _ACTIVATION_BOOL_KEYS: dict[str, str] = {
 }
 ACTIVATION_DEFAULTS: dict[str, float | bool] = {
     "min_momentum_percentile": 80.0,
-    "min_rr": 1.2,
+    # Production floor from the 2026-07-12 min_rr sweep (in-sample and OOS peak).
+    # 1.2 was the old code default and the trough next to the spike — do not restore.
+    "min_rr": 2.0,
     # 0 = off. The July 2026 gate ablation showed the confidence floor added
     # nothing (identical net/trade with it removed, under both exit models)
     # while cutting ~25% of qualified trades.
@@ -157,6 +159,28 @@ async def update_setting(db: AsyncSession, key: str, value: str) -> SystemSettin
     return setting
 
 
+# ---------------------------------------------------------------------------
+# Fundamentals source cutover
+# ---------------------------------------------------------------------------
+
+async def get_fundamentals_cutover_config(db: AsyncSession) -> dict[str, bool]:
+    """Return the explicit A5 cache-cutover switch (default off)."""
+    return {"enabled": await fundamental_data_refresh_service.is_enabled(db)}
+
+
+async def update_fundamentals_cutover_config(
+    db: AsyncSession, enabled: bool
+) -> dict[str, bool]:
+    """Activate or pause SEC/Dolt writes to the legacy fundamentals cache."""
+    await settings_store.upsert_setting(
+        db,
+        fundamental_data_refresh_service.ACTIVATION_KEY,
+        "true" if enabled else "false",
+    )
+    await db.commit()
+    return await get_fundamentals_cutover_config(db)
+
+
 # ---------------------------------------------------------------------------
 # Activation thresholds
 # ---------------------------------------------------------------------------
@@ -202,6 +226,61 @@ async def update_activation_config(
     return await get_activation_config(db)
 
 
+# ---------------------------------------------------------------------------
+# Performance window + shadow book
+# ---------------------------------------------------------------------------
+
+async def get_performance_config(db: AsyncSession) -> dict:
+    """Start date for the Performance comparison ('' = all history)."""
+    from app.services.paper_trade_service import KEY_PERFORMANCE_START
+
+    return {"start_date": await settings_store.get_value(db, KEY_PERFORMANCE_START, "") or ""}
+
+
+async def update_performance_config(db: AsyncSession, updates: dict) -> dict:
+    """Set (or clear) the performance start date. Empty string means all history."""
+    from datetime import date as _date
+
+    from app.services.paper_trade_service import KEY_PERFORMANCE_START
+
+    if "start_date" in updates:
+        raw = (updates.get("start_date") or "").strip()
+        if raw:
+            try:
+                _date.fromisoformat(raw)
+            except ValueError as exc:
+                raise ValidationError("start_date must be an ISO date (YYYY-MM-DD)") from exc
+        await update_setting(db, KEY_PERFORMANCE_START, raw)
+    return await get_performance_config(db)
+
+
+async def get_shadow_book_config(db: AsyncSession) -> dict:
+    """Shadow book switch + sizing, with the validated defaults filled in."""
+    from app.services import shadow_book_service
+
+    config = await shadow_book_service.get_config(db)
+    config["enabled"] = await shadow_book_service.is_enabled(db)
+    return config
+
+
+async def update_shadow_book_config(db: AsyncSession, updates: dict) -> dict:
+    """Update the shadow book. Enabling it starts automatic live entries."""
+    from app.services import shadow_book_service
+
+    if "enabled" in updates:
+        await update_setting(
+            db, shadow_book_service.KEY_ENABLED, "true" if updates["enabled"] else "false"
+        )
+    for key, storage_key in (
+        ("capacity", shadow_book_service.KEY_CAPACITY),
+        ("risk_pct", shadow_book_service.KEY_RISK_PCT),
+        ("start_equity", shadow_book_service.KEY_START_EQUITY),
+    ):
+        if key in updates:
+            await update_setting(db, storage_key, str(updates[key]))
+    return await get_shadow_book_config(db)
+
+
 # ---------------------------------------------------------------------------
 # Pipeline schedule (cron)
 # ---------------------------------------------------------------------------
@@ -316,14 +395,18 @@ async def update_ticker_universe_default(db: AsyncSession, universe: str) -> dic
 # Data cleanup
 # ---------------------------------------------------------------------------
 
-async def cleanup_data(db: AsyncSession, older_than_days: int) -> dict[str, int]:
+async def cleanup_data(db: AsyncSession, older_than_days: int) -> dict:
     """Delete OHLCV, sentiment, and fundamental records older than N days.
 
-    Preserves tickers, users, and latest scores.
-    Returns a dict with counts of deleted records per table.
+    Preserves tickers, users, and latest scores. After OHLCV pruning, rebuilds
+    Structural S/R for every ticker so chart levels match the remaining history.
+
+    Returns deleted-row counts plus S/R refresh outcomes. A per-ticker S/R
+    failure rolls the session back (so later tickers still run) and is listed
+    in ``sr_refresh_failures`` rather than aborting the whole cleanup.
     """
     cutoff = datetime.now(timezone.utc) - timedelta(days=older_than_days)
-    counts: dict[str, int] = {}
+    counts: dict = {}
 
     # OHLCV — date column is a date, compare with cutoff date
     result = await db.execute(
@@ -344,6 +427,36 @@ async def cleanup_data(db: AsyncSession, older_than_days: int) -> dict[str, int]
     counts["fundamentals"] = result.rowcount  # type: ignore[assignment]
 
     await db.commit()
+
+    counts["sr_refresh_ok"] = 0
+    counts["sr_refresh_failed"] = 0
+    counts["sr_refresh_failures"] = []
+
+    # Structural S/R is derived from OHLCV; recompute after history shrinks.
+    if counts["ohlcv"]:
+        from app.services.sr_service import recalculate_sr_levels
+
+        symbols = list(
+            (await db.execute(select(Ticker.symbol).order_by(Ticker.symbol))).scalars().all()
+        )
+        for symbol in symbols:
+            try:
+                await recalculate_sr_levels(db, symbol)
+                counts["sr_refresh_ok"] += 1
+            except Exception as exc:
+                logger.exception("S/R refresh after cleanup failed for %s", symbol)
+                try:
+                    await db.rollback()
+                except Exception:
+                    logger.exception(
+                        "Session rollback after S/R cleanup failure also failed for %s",
+                        symbol,
+                    )
+                counts["sr_refresh_failed"] += 1
+                counts["sr_refresh_failures"].append(
+                    {"symbol": symbol, "error": f"{type(exc).__name__}: {exc}"}
+                )
+
     return counts
 
 
@@ -521,6 +634,9 @@ VALID_JOB_NAMES = {
     "benchmark_collector",
     "sentiment_collector",
     "fundamental_collector",
+    "dolt_earnings_import",
+    "sec_fundamentals_import",
+    "fundamentals_parity_report",
     "rr_scanner",
     "ticker_universe_sync",
     "outcome_evaluator",
@@ -530,7 +646,10 @@ VALID_JOB_NAMES = {
     "event_study",
     "backtest",
     "daily_pipeline",
+    "near_close_pipeline",
+    "after_close_pipeline",
     "intraday_pipeline",
+    "shadow_book",
 }
 
 JOB_LABELS = {
@@ -539,6 +658,9 @@ JOB_LABELS = {
     "benchmark_collector": "Benchmark Collector",
     "sentiment_collector": "Sentiment Collector",
     "fundamental_collector": "Fundamental Collector",
+    "dolt_earnings_import": "Dolt Earnings Import (shadow)",
+    "sec_fundamentals_import": "SEC Fundamentals Import",
+    "fundamentals_parity_report": "Fundamentals Parity Report (read-only)",
     "rr_scanner": "R:R Scanner",
     "ticker_universe_sync": "Ticker Universe Sync",
     "outcome_evaluator": "Outcome Evaluator",
@@ -547,19 +669,24 @@ JOB_LABELS = {
     "regime_monitor": "Regime Monitor",
     "event_study": "Event Study",
     "backtest": "Backtest",
-    "daily_pipeline": "Daily Pipeline",
+    "daily_pipeline": "Morning Pipeline",
+    "near_close_pipeline": "Near-Close Pipeline (scan+alert)",
+    "after_close_pipeline": "After-Close Pipeline (outcome)",
     "intraday_pipeline": "Intraday Pipeline",
+    "shadow_book": "Shadow Book (auto-traded strategy)",
 }
 
-# Jobs driven by the daily_pipeline (in order) rather than their own timer.
+# Jobs driven by a pipeline (in order) rather than their own auto timer.
 PIPELINE_MEMBERS = {
     "data_collector",
     "benchmark_collector",
     "sentiment_collector",
     "rr_scanner",
     "outcome_evaluator",
+    "alerts",
     "market_regime",
     "regime_monitor",
+    "shadow_book",
 }
 
 
@@ -672,3 +799,30 @@ async def toggle_job(db: AsyncSession, job_name: str, enabled: bool) -> SystemSe
 
     key = f"job_{job_name}_enabled"
     return await update_setting(db, key, str(enabled).lower())
+
+
+def get_fundamentals_parity_report() -> dict | None:
+    """Return the latest compact A5 summary, if the job has run."""
+    from app.config import settings
+    from app.services.fundamentals_parity_service import load_latest
+
+    report = load_latest(settings.fundamentals_parity_report_dir)
+    if report is not None:
+        report.pop("rows", None)  # full per-ticker data is download-only
+    return report
+
+
+def get_fundamentals_parity_csv() -> tuple[str, str] | None:
+    """Return the latest A5 CSV filename and content for authenticated download."""
+    from app.config import settings
+    from app.services.fundamentals_parity_service import load_latest_csv
+
+    return load_latest_csv(settings.fundamentals_parity_report_dir)
+
+
+def get_fundamentals_parity_json() -> tuple[str, str] | None:
+    """Return the canonical A5 JSON artifact for authenticated download."""
+    from app.config import settings
+    from app.services.fundamentals_parity_service import load_latest_json
+
+    return load_latest_json(settings.fundamentals_parity_report_dir)

app/services/alert_service.py

@@ -29,6 +29,7 @@ from app.config import settings
 from app.models.alert import AlertLog
 from app.models.ohlcv import OHLCVRecord
 from app.models.paper_trade import PaperTrade
+from app.services.trade_policy import MANUAL_BOOK
 from app.models.score import CompositeScore
 from app.models.sr_level import SRLevel
 from app.models.ticker import Ticker
@@ -96,8 +97,8 @@ SIGNAL_BUNDLE_MAX_CHARS = 3900  # Telegram limit is 4096; keep room for HTML par
 # Hysteresis (a deadband around each divider) stops a point sitting on a boundary
 # from flip-flopping; the cooldown caps how often a genuine change can re-alert.
 QUAD_TYPE = "regime_quadrant"
-QUAD_X_DIV = 60.0           # v2 State divider (backend response is authoritative)
-QUAD_Y_DIV = 60.0           # v2 Warning divider
+QUAD_X_DIV = 50.0           # v3 State divider (backend response is authoritative)
+QUAD_Y_DIV = 40.0           # v3 Warning divider; the axes have different ranges
 QUAD_MARGIN = 5.0           # half-width of the hysteresis deadband around each divider
 QUAD_COOLDOWN_DAYS = 3      # min days between quadrant-change alerts
 QUAD_LABELS = {
@@ -282,7 +283,7 @@ async def _qualified_setups(db: AsyncSession) -> list[dict]:
         db,
         live_recommendation=True,
         exclude_open_trade_tickers=True,
-        exclude_reentry_lockdown_tickers=True,
+        exclude_reentry_gate_locked_tickers=True,
     )
     config = await get_activation_config(db)
     return [s for s in setups if setup_qualifies(SimpleNamespace(**s), config)]
@@ -632,6 +633,10 @@ async def _collect_closed_trades(db: AsyncSession) -> list[ClosedTradeItem]:
             PaperTrade.closed_at.is_not(None),
             PaperTrade.closed_at > cutoff,
             PaperTrade.close_reason.in_(("trailing", "stop", "target", "time")),
+            # Your own positions only — shadow trades are a research record, not
+            # something you hold, and mixing them in unlabelled reads as if you
+            # were stopped out of a name you never took.
+            PaperTrade.book == MANUAL_BOOK,
         )
         .order_by(PaperTrade.closed_at.desc())
     )
@@ -642,8 +647,14 @@ async def _collect_closed_trades(db: AsyncSession) -> list[ClosedTradeItem]:
 
 
 async def _paper_book_value(db: AsyncSession) -> float:
-    """Paper-trade equity: fixed capital plus realized/unrealized P&L."""
-    result = await db.execute(select(PaperTrade))
+    """Paper-trade equity: fixed capital plus realized/unrealized P&L.
+
+    Discretionary book only — the shadow book runs on its own notional equity
+    and folding it in would report a number matching neither book.
+    """
+    result = await db.execute(
+        select(PaperTrade).where(PaperTrade.book == MANUAL_BOOK)
+    )
     trades = list(result.scalars().all())
     latest: dict[int, float | None] = {}
     for trade in trades:

app/services/breadth_service.py

@@ -72,15 +72,25 @@ def _breadth_from_closes(
     return _breadth_with_counts(closes_by_symbol, window, min_tickers)[0]
 
 
+# Breadth deterioration counts fully when price masks it (true divergence, the
+# dangerous pre-top case) and at CONFIRMED_FLOOR when price falls with it.
+# v2 used a hard ``price_ret >= 0`` cliff, which zeroed the sensor during every
+# decline -- so on 2026-07-24, with the basket shedding 10 percentage points
+# above their 200-DMA in 20 sessions, Warning read exactly 0.  Breadth *level*
+# lives in State but breadth *velocity* appears nowhere else, so partial credit
+# here is not double counting.
+DIVERGENCE_CONFIRMED_FLOOR = 0.35
+DIVERGENCE_TAPER_PCT = 3.0
+
+
 def compute_divergence_series(
     breadth: dict[date, float], benchmark_closes: Series, lookback: int = 20
 ) -> dict[date, float]:
     """Early-warning score (0-100, high = fragile) per date.
 
-    This is deliberately a pure divergence: it is positive only when benchmark
-    price holds/rises while breadth falls.  Absolute low breadth belongs in the
-    State score, so it is not counted again here.  A 20 percentage-point breadth
-    deterioration maps to 100.
+    A 20 percentage-point breadth deterioration maps to 100 when the benchmark
+    is flat or rising, tapering to ``DIVERGENCE_CONFIRMED_FLOOR`` of that once
+    the benchmark is down ``DIVERGENCE_TAPER_PCT`` or more over the window.
     """
     bench = {d: c for d, c in benchmark_closes}
     common = sorted(d for d in bench if d in breadth)
@@ -93,8 +103,9 @@ def compute_divergence_series(
         price_ret = (bench[d] / price_past - 1.0) * 100.0   # %
         breadth_chg = breadth[d] - breadth[d0]              # percentage points
         deterioration = max(0.0, -breadth_chg)
-        score = deterioration * 5.0 if price_ret >= 0 else 0.0
-        out[d] = max(0.0, min(100.0, round(score, 2)))
+        taper = max(0.0, min(1.0, (price_ret + DIVERGENCE_TAPER_PCT) / DIVERGENCE_TAPER_PCT))
+        gate = DIVERGENCE_CONFIRMED_FLOOR + (1.0 - DIVERGENCE_CONFIRMED_FLOOR) * taper
+        out[d] = max(0.0, min(100.0, round(deterioration * 5.0 * gate, 2)))
     return out
 
 

app/services/data_import.py

@@ -0,0 +1,348 @@
+"""Source-agnostic batch import framework (Dolt/SEC bulk data → PostgreSQL).
+
+Every bulk importer (SEC facts, Dolt earnings, later Dolt stocks) plugs into
+``run_import`` and gets, for free, the plan's non-negotiables:
+
+- **One run per source at a time** — a Postgres *session-level* advisory lock
+  keyed by source. It is held on a single pinned connection for the whole run,
+  so it survives the intermediate commits (the ``running`` row, then the
+  promotion) and only releases at the end. No-op on non-Postgres (tests).
+- **Idempotent per revision** — the cheap ``detect_revision`` probe is compared
+  against the last *promoted* run; an unchanged revision records a ``no_op``
+  with **zero row changes** (no expensive fetch, no writes).
+- **Staging then atomic promotion** — the importer stages into an in-memory
+  object (no physical staging tables), validation reads it, and only a passing
+  run calls ``promote`` whose writes commit together with the run-row flip to
+  ``promoted`` in a single transaction.
+- **Failure is inert** — a failed validation or a mid-run exception marks the
+  run ``failed``, alerts via the system-events path, and leaves the live tables
+  exactly as they were (nothing is written before ``promote``).
+
+Every attempt — promoted, no_op, or failed — is recorded in ``data_import_runs``.
+KISS: no conflicts table (summaries go in ``validation_json``), no revision
+table (idempotency queries the last run), no aggregate tables.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import hashlib
+import json
+import logging
+from dataclasses import dataclass, field
+from datetime import date, datetime, timedelta, timezone
+from typing import Any, Protocol, runtime_checkable
+
+from sqlalchemy import exists, select, text
+from sqlalchemy.engine import Engine  # noqa: F401  (typing only)
+from sqlalchemy.ext.asyncio import AsyncEngine, AsyncSession
+
+from app.database import engine as app_engine
+from app.models.data_import_run import DataImportRun
+from app.services import system_event_service
+
+logger = logging.getLogger(__name__)
+
+# data_import_runs.status values
+STATUS_RUNNING = "running"
+STATUS_VALIDATED = "validated"
+STATUS_PROMOTED = "promoted"
+STATUS_NO_OP = "no_op"
+STATUS_DEFERRED = "deferred"
+STATUS_FAILED = "failed"
+
+_MAX_ERROR_LEN = 4000
+
+
+@dataclass
+class ValidationResult:
+    """Outcome of an importer's validation gates.
+
+    ``summary`` is serialized into ``validation_json`` (reconciliation /
+    discrepancy details live here — no separate conflicts table). ``validate``
+    MUST be read-only: it reads the staged object and, if needed, live tables
+    for comparison, but writes nothing — that invariant is what makes a failed
+    run leave the dataset untouched.
+    """
+
+    ok: bool
+    summary: dict[str, Any] = field(default_factory=dict)
+    source_max_date: date | None = None
+    messages: list[str] = field(default_factory=list)
+    # Expected source-side lag: retry without an immediate error alert. Sources
+    # can bound the quiet period with deferred_alert_after_days. Only meaningful
+    # when ok=False.
+    retryable: bool = False
+    deferred_alert_after_days: int | None = None
+    deferred_alert_messages: list[str] = field(default_factory=list)
+
+
+@runtime_checkable
+class SourceImporter(Protocol):
+    """Interface a concrete bulk importer implements. All methods receive the
+    session bound to the lock-holding connection; ``stage`` and ``validate``
+    never write to live tables, only ``promote`` does."""
+
+    source: str  # sec_facts | dolt_earnings | dolt_stocks
+
+    async def detect_revision(self, db: AsyncSession) -> str | None:
+        """Cheap probe of the source revision (Dolt commit / SEC archive SHA).
+
+        Returns the revision id, or None when it can't be determined cheaply
+        (in which case idempotency is skipped and the run always stages)."""
+        ...
+
+    async def stage(self, db: AsyncSession) -> Any:
+        """Download/parse into an in-memory staged representation. No writes to
+        live tables."""
+        ...
+
+    async def validate(self, db: AsyncSession, staged: Any) -> ValidationResult:
+        """Run the source's validation gates against ``staged``. Read-only."""
+        ...
+
+    async def promote(self, db: AsyncSession, staged: Any, run_id: int) -> dict[str, int]:
+        """Apply ``staged`` to the live tables. Called inside the promotion
+        transaction; the caller commits. ``run_id`` is the current
+        ``data_import_runs.id`` so written rows can be stamped with their
+        ``import_run_id``. Returns row-count deltas."""
+        ...
+
+
+def _advisory_key(source: str) -> int:
+    """Deterministic signed 64-bit key for a source's advisory lock."""
+    digest = hashlib.blake2b(source.encode("utf-8"), digest_size=8).digest()
+    return int.from_bytes(digest, "big", signed=True)
+
+
+async def _last_promoted_revision(db: AsyncSession, source: str) -> str | None:
+    """Revision of the most recent *promoted* run for ``source`` (the revision
+    currently loaded), or None if none has promoted yet."""
+    row = await db.execute(
+        select(DataImportRun.revision)
+        .where(
+            DataImportRun.source == source,
+            DataImportRun.status == STATUS_PROMOTED,
+        )
+        .order_by(DataImportRun.id.desc())
+        .limit(1)
+    )
+    return row.scalar_one_or_none()
+
+
+async def _promotion_state_since(
+    db: AsyncSession, source: str, cutoff: datetime
+) -> str:
+    promoted = (
+        DataImportRun.source == source,
+        DataImportRun.status == STATUS_PROMOTED,
+    )
+    ever, recent = (
+        await db.execute(
+            select(
+                exists().where(*promoted),
+                exists().where(*promoted, DataImportRun.started_at >= cutoff),
+            )
+        )
+    ).one()
+    return "recent" if recent else "stale" if ever else "never"
+
+
+def _now() -> datetime:
+    return datetime.now(timezone.utc)
+
+
+async def _alert(
+    db: AsyncSession,
+    source: str,
+    code: str,
+    messages: list[str],
+    *,
+    severity: str = "error",
+    dedup_hours: int = 24,
+) -> None:
+    try:
+        await system_event_service.log_event(
+            db,
+            severity=severity,
+            source="data_import",
+            code=f"{source}_{code}",
+            message=(("; ".join(messages)) or code)[:_MAX_ERROR_LEN],
+            dedup_key=f"data_import:{source}:{code}",
+            dedup_hours=dedup_hours,
+        )
+    except Exception:  # noqa: BLE001 — alerting must never mask the real outcome
+        logger.exception("Failed to emit data_import alert %s/%s", source, code)
+
+
+async def run_import(
+    importer: SourceImporter,
+    *,
+    engine: AsyncEngine | None = None,
+    force: bool = False,
+) -> DataImportRun | None:
+    """Run one import for ``importer``.
+
+    Returns the recorded ``DataImportRun`` (promoted / no_op / deferred /
+    failed), or None
+    when the per-source advisory lock is already held (another run is active).
+
+    ``force`` runs even when the revision is unchanged. The revision tracks the
+    *source*, so a re-import driven by a change on our side — a parser fix that
+    makes stored rows stale — is a no_op under the normal gate. Manually invoked
+    only; scheduled jobs must leave it False so an unchanged source stays a no_op.
+    """
+    engine = engine or app_engine
+    source = importer.source
+    is_pg = engine.dialect.name == "postgresql"
+    key = _advisory_key(source)
+
+    async with engine.connect() as conn:
+        # Bind the session to this one connection so the session-level advisory
+        # lock persists across our commits. expire_on_commit must be set here —
+        # the app factory's setting doesn't carry to a directly-built session.
+        session = AsyncSession(bind=conn, expire_on_commit=False)
+        try:
+            if is_pg:
+                got = (
+                    await session.execute(
+                        text("SELECT pg_try_advisory_lock(:k)"), {"k": key}
+                    )
+                ).scalar()
+                await session.commit()
+                if not got:
+                    logger.info("data_import %s: lock held, skipping", source)
+                    return None
+
+            # Record the attempt FIRST — before the external revision probe, the
+            # most likely failure — so anything below is recorded and alerted and
+            # never escapes unrecorded. Revision is filled in once detected.
+            run = DataImportRun(
+                source=source,
+                status=STATUS_RUNNING,
+                started_at=_now(),
+            )
+            session.add(run)
+            await session.commit()
+            await session.refresh(run)
+
+            try:
+                revision = await importer.detect_revision(session)
+                run.revision = revision
+                last_rev = await _last_promoted_revision(session, source)
+                if not force and revision is not None and revision == last_rev:
+                    run.status = STATUS_NO_OP
+                    run.completed_at = _now()
+                    await session.commit()
+                    logger.info("data_import %s: no_op (revision %s)", source, revision)
+                    return run
+
+                staged = await importer.stage(session)
+                result = await importer.validate(session, staged)
+                run.source_max_date = result.source_max_date
+                run.validation_json = json.dumps(result.summary, default=str)
+
+                if not result.ok:
+                    run.error_details = ("; ".join(result.messages))[:_MAX_ERROR_LEN]
+                    run.completed_at = _now()
+                    if result.retryable:
+                        run.status = STATUS_DEFERRED
+                        await session.commit()
+                        alert_days = result.deferred_alert_after_days
+                        if alert_days is not None:
+                            alert_days = max(1, alert_days)
+                            cutoff = run.started_at - timedelta(days=alert_days)
+                            promotion_state = await _promotion_state_since(
+                                session, source, cutoff
+                            )
+                            if promotion_state != "recent":
+                                history = (
+                                    f"{source} import has never promoted successfully"
+                                    if promotion_state == "never"
+                                    else f"{source} import has not promoted successfully "
+                                    f"within {alert_days} day(s)"
+                                )
+                                await _alert(
+                                    session,
+                                    source,
+                                    "deferred_stale",
+                                    [
+                                        f"{history}; import remains deferred",
+                                        *result.deferred_alert_messages,
+                                        f"Current deferral: "
+                                        f"{run.error_details or 'validation deferred'}",
+                                    ],
+                                    severity="warning",
+                                    dedup_hours=alert_days * 24,
+                                )
+                        logger.info(
+                            "data_import %s: deferred for retry: %s",
+                            source,
+                            result.messages,
+                        )
+                        return run
+
+                    run.status = STATUS_FAILED
+                    await session.commit()
+                    await _alert(session, source, "validation_failed", result.messages)
+                    logger.warning(
+                        "data_import %s: validation failed: %s",
+                        source,
+                        result.messages,
+                    )
+                    return run
+
+                # Promotion: importer writes + run-row flip in one transaction.
+                row_counts = await importer.promote(session, staged, run.id)
+                run.status = STATUS_PROMOTED
+                run.row_counts_json = json.dumps(row_counts, default=str)
+                run.completed_at = _now()
+                await session.commit()
+                await session.refresh(run)
+                logger.info(
+                    "data_import %s: promoted (revision %s, rows %s)",
+                    source,
+                    revision,
+                    row_counts,
+                )
+                return run
+
+            except asyncio.CancelledError:
+                # Deploy / scheduler shutdown: best-effort mark failed so no
+                # ``running`` row lingers, then let the cancellation propagate —
+                # never swallow it.
+                try:
+                    await session.rollback()
+                    run.status = STATUS_FAILED
+                    run.error_details = "cancelled"
+                    run.completed_at = _now()
+                    await session.commit()
+                except BaseException:  # noqa: BLE001 — best-effort during teardown
+                    logger.warning(
+                        "data_import %s: could not record cancellation", source
+                    )
+                raise
+
+            except Exception as exc:  # noqa: BLE001 — record + alert, don't crash the job
+                await session.rollback()
+                run.status = STATUS_FAILED
+                run.error_details = repr(exc)[:_MAX_ERROR_LEN]
+                run.completed_at = _now()
+                try:
+                    await session.commit()
+                except Exception:  # noqa: BLE001
+                    logger.exception("data_import %s: failed to record failure", source)
+                await _alert(session, source, "import_error", [repr(exc)])
+                logger.exception("data_import %s: import error", source)
+                return run
+
+        finally:
+            if is_pg:
+                try:
+                    await session.execute(
+                        text("SELECT pg_advisory_unlock(:k)"), {"k": key}
+                    )
+                    await session.commit()
+                except Exception:  # noqa: BLE001
+                    logger.exception("data_import %s: failed to release lock", source)
+            await session.close()

app/services/dolt_client.py

@@ -0,0 +1,105 @@
+"""Minimal async client for a local Dolt clone.
+
+The application never runs a long-lived Dolt sql-server; it shells out to the
+`dolt` CLI against a persistent clone and reads results as CSV. Every call goes
+through ``asyncio.create_subprocess_exec`` because the scheduler shares one event
+loop with the API (`app/scheduler.py:73`) — a blocking `subprocess.run` here
+would stall request handling.
+
+Production keeps the clone in ``DOLT_DATA_DIR`` outside the deploy tree; the
+binary path and data dir are configured (see ``app/config.py``). Read via
+``dolt sql -r csv``; refresh with ``pull`` and record the resulting commit hash
+as the import revision.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import csv
+import io
+import logging
+import shutil
+from pathlib import Path
+
+logger = logging.getLogger(__name__)
+
+# Default subprocess timeout. A hung `dolt pull`/`sql` would otherwise pin the
+# import's connection and its advisory lock indefinitely, so every call is
+# bounded; callers may override per operation.
+DEFAULT_TIMEOUT = 600.0
+
+
+class DoltError(RuntimeError):
+    """A dolt subprocess failed, timed out, or exited non-zero."""
+
+
+async def _run(
+    binary: str, args: list[str], *, cwd: Path, timeout: float = DEFAULT_TIMEOUT
+) -> str:
+    proc = await asyncio.create_subprocess_exec(
+        binary,
+        *args,
+        cwd=str(cwd),
+        stdout=asyncio.subprocess.PIPE,
+        stderr=asyncio.subprocess.PIPE,
+    )
+    try:
+        stdout, stderr = await asyncio.wait_for(proc.communicate(), timeout=timeout)
+    except asyncio.TimeoutError:
+        proc.kill()
+        try:
+            await proc.wait()
+        except ProcessLookupError:
+            pass
+        raise DoltError(f"dolt {args[0] if args else ''} timed out after {timeout:.0f}s")
+    if proc.returncode != 0:
+        raise DoltError(
+            f"dolt {' '.join(args)} failed ({proc.returncode}): "
+            f"{stderr.decode('utf-8', 'replace').strip()[:500]}"
+        )
+    return stdout.decode("utf-8", "replace")
+
+
+def ensure_free_disk(path: Path, min_free_gb: float) -> None:
+    """Raise if free space at ``path`` is below the threshold (checked before a
+    pull that could grow the clone). Uses the nearest existing ancestor so it
+    works before the clone dir exists."""
+    probe = path
+    while not probe.exists() and probe.parent != probe:
+        probe = probe.parent
+    free_gb = shutil.disk_usage(probe).free / (1024**3)
+    if free_gb < min_free_gb:
+        raise DoltError(
+            f"insufficient disk for dolt at {path}: {free_gb:.1f} GB free "
+            f"< {min_free_gb:.1f} GB required"
+        )
+
+
+async def pull(repo_dir: Path, *, binary: str, timeout: float = DEFAULT_TIMEOUT) -> None:
+    """`dolt pull` the persistent clone to the latest upstream revision."""
+    await _run(binary, ["pull"], cwd=repo_dir, timeout=timeout)
+
+
+async def current_commit(
+    repo_dir: Path, *, binary: str, timeout: float = DEFAULT_TIMEOUT
+) -> str:
+    """The HEAD commit hash of the clone — used as the import revision.
+
+    Uses ``DOLT_HASHOF('HEAD')`` (which formally identifies HEAD) rather than
+    ordering ``dolt_log`` by timestamp."""
+    rows = await query_csv(
+        repo_dir, "SELECT DOLT_HASHOF('HEAD') AS commit_hash", binary=binary, timeout=timeout
+    )
+    if not rows or not rows[0].get("commit_hash"):
+        raise DoltError("could not read HEAD commit hash")
+    return rows[0]["commit_hash"]
+
+
+async def query_csv(
+    repo_dir: Path, sql: str, *, binary: str, timeout: float = DEFAULT_TIMEOUT
+) -> list[dict[str, str]]:
+    """Run a read query and parse the CSV result into a list of dict rows."""
+    out = await _run(binary, ["sql", "-q", sql, "-r", "csv"], cwd=repo_dir, timeout=timeout)
+    if not out.strip():
+        return []
+    return list(csv.DictReader(io.StringIO(out)))

app/services/dolt_earnings_importer.py

@@ -0,0 +1,357 @@
+"""Production importer for the DoltHub post-no-preference/earnings calendar.
+
+A ``SourceImporter`` (see ``app/services/data_import.py``) that pulls the local
+Dolt clone, aligns the announcement calendar to the EPS history with the pure DP
+in ``earnings_alignment`` (reused from the research script, not extending it),
+and writes ``earnings_events`` for the tracked universe.
+
+Shadow by construction: nothing reads ``earnings_events`` until the API/panel
+lands (A4), so writing it does not touch production behavior.
+
+**Promotion is destructive** — future-dated rows for this source are deleted and
+re-inserted every run so reschedules/cancellations never linger. The forward
+calendar is the project's acceptance gate, so ``validate`` is fail-closed: it
+blocks promotion when the staged future set is empty or has collapsed relative
+to what's already loaded.
+
+Attribution: the earnings data is CC BY-SA 4.0 from post-no-preference/earnings.
+See the repo ``NOTICE``. Internal use only — no redistribution.
+"""
+
+from __future__ import annotations
+
+import logging
+from dataclasses import dataclass, field
+from datetime import date, datetime, timezone
+from pathlib import Path
+from typing import Any
+
+from sqlalchemy import case, delete, func, select
+
+from app.config import settings
+from app.database import insert_for_session
+from app.models.earnings_event import EarningsEvent
+from app.models.ticker import Ticker
+from app.services import dolt_client, earnings_alignment
+from app.services.data_import import ValidationResult
+
+logger = logging.getLogger(__name__)
+
+SOURCE = "dolt_earnings"
+
+# Earliest announcement date to import (matches the research backfill window).
+WINDOW_START = date(2020, 1, 22)
+# Alignment tolerances (research defaults): an announcement may lead its period
+# end by up to 14 days or lag it by up to 90.
+MAX_LAG_DAYS = 90
+MAX_LEAD_DAYS = 14
+# Fail promotion if the staged forward calendar drops below this fraction of the
+# currently-loaded forward calendar (guards the destructive re-insert against a
+# partial parse / symbol-mapping regression).
+MIN_FUTURE_RATIO = 0.5
+# Initial-load gates (when nothing is loaded yet — the ratio gate has no baseline).
+# The source publishes a forward calendar; require a real horizon, not one stray
+# future row. 21 days is a conservative floor under the ~35d horizon observed on
+# the live clone.
+MIN_FORWARD_HORIZON_DAYS = 21
+# ...and require the symbol join to reach most of the tracked universe, so a
+# broken/normalization-dropped join can't seed a hollow calendar.
+MIN_INITIAL_COVERAGE = 0.5
+
+_CAL_SQL = (
+    "SELECT act_symbol, `date`, `when` FROM earnings_calendar "
+    f"WHERE `date` >= '{WINDOW_START.isoformat()}'"
+)
+_HIST_SQL = (
+    "SELECT act_symbol, period_end_date, reported, estimate FROM eps_history "
+    f"WHERE period_end_date >= '{(WINDOW_START.replace(year=WINDOW_START.year - 1)).isoformat()}'"
+)
+
+
+@dataclass
+class StagedEarnings:
+    rows: list[dict[str, Any]]
+    stats: dict[str, Any] = field(default_factory=dict)
+    future_count: int = 0
+    max_announce_date: date | None = None
+
+
+def _now() -> datetime:
+    return datetime.now(timezone.utc)
+
+
+class DoltEarningsImporter:
+    source = SOURCE
+
+    def __init__(
+        self,
+        *,
+        repo_dir: Path | str | None = None,
+        binary: str | None = None,
+        today: date | None = None,
+        do_pull: bool = True,
+        dolt: Any = dolt_client,
+    ) -> None:
+        self.repo_dir = Path(
+            repo_dir
+            or (Path(settings.dolt_data_dir) / settings.dolt_earnings_subdir)
+        )
+        self.binary = binary or settings.dolt_binary
+        self.today = today or _now().date()
+        self.do_pull = do_pull
+        self._dolt = dolt  # injectable for tests
+
+    # -- SourceImporter protocol -------------------------------------------
+
+    async def detect_revision(self, db) -> str | None:
+        timeout = settings.dolt_command_timeout_seconds
+        if self.do_pull:
+            dolt_client.ensure_free_disk(self.repo_dir, settings.dolt_min_free_disk_gb)
+            await self._dolt.pull(self.repo_dir, binary=self.binary, timeout=timeout)
+        return await self._dolt.current_commit(
+            self.repo_dir, binary=self.binary, timeout=timeout
+        )
+
+    async def stage(self, db) -> StagedEarnings:
+        universe = await self._load_universe(db)  # {normalised symbol: ticker_id}
+
+        timeout = settings.dolt_command_timeout_seconds
+        cal_raw = await self._dolt.query_csv(
+            self.repo_dir, _CAL_SQL, binary=self.binary, timeout=timeout
+        )
+        hist_raw = await self._dolt.query_csv(
+            self.repo_dir, _HIST_SQL, binary=self.binary, timeout=timeout
+        )
+        _require_columns(cal_raw, {"act_symbol", "date", "when"}, "earnings_calendar")
+        _require_columns(
+            hist_raw, {"act_symbol", "period_end_date", "reported", "estimate"}, "eps_history"
+        )
+
+        cal_parsed = _parse_calendar(cal_raw, universe)
+        hist_parsed = _parse_history(hist_raw, universe)
+        calendar, cal_stats = earnings_alignment.dedup_calendar(cal_parsed)
+        history, hist_stats = earnings_alignment.dedup_history(hist_parsed)
+
+        period_lower = WINDOW_START.replace(year=WINDOW_START.year - 1)
+        rows: list[dict[str, Any]] = []
+        matched = unmatched = 0
+        for symbol, events in calendar.items():
+            ticker_id = universe[symbol]
+            periods = [
+                p for p in history.get(symbol, []) if p["period_end_date"] >= period_lower
+            ]
+            matches, unmatched_events, _ = earnings_alignment.align_symbol(
+                events, periods, max_lag_days=MAX_LAG_DAYS, max_lead_days=MAX_LEAD_DAYS
+            )
+            matched += len(matches)
+            unmatched += len(unmatched_events)
+            matched_by_event = {e: p for e, p in matches}
+            for e_idx, event in enumerate(events):
+                p_idx = matched_by_event.get(e_idx)
+                period = periods[p_idx] if p_idx is not None else None
+                rows.append(
+                    {
+                        "ticker_id": ticker_id,
+                        "symbol": symbol,
+                        "announce_date": event["announce_date"],
+                        "session": event["session"],
+                        "period_end": period["period_end_date"] if period else None,
+                        "eps_estimate": period["eps_estimate"] if period else None,
+                        "eps_actual": period["eps_actual"] if period else None,
+                    }
+                )
+
+        future_rows = [r for r in rows if r["announce_date"] > self.today]
+        tickers_with_future = {r["ticker_id"] for r in future_rows}
+        stats = {
+            "calendar": cal_stats,
+            "eps_history": hist_stats,
+            "universe_size": len(universe),
+            "symbols_with_calendar": len(calendar),
+            "matched_events": matched,
+            "unmatched_events": unmatched,
+            "tracked_tickers_with_future_date": len(tickers_with_future),
+        }
+        return StagedEarnings(
+            rows=rows,
+            stats=stats,
+            future_count=len(future_rows),
+            max_announce_date=max((r["announce_date"] for r in rows), default=None),
+        )
+
+    async def validate(self, db, staged: StagedEarnings) -> ValidationResult:
+        # Promote deletes+reinserts the forward calendar, so this gate is
+        # fail-closed. The forward calendar is the project's acceptance gate.
+        messages: list[str] = []
+        current_future = await self._current_future_count(db)
+        universe_size = int(staged.stats.get("universe_size", 0) or 0)
+        coverage = (
+            staged.stats.get("symbols_with_calendar", 0) / universe_size
+            if universe_size
+            else 0.0
+        )
+        horizon_days = (
+            (staged.max_announce_date - self.today).days if staged.max_announce_date else 0
+        )
+
+        if staged.future_count == 0:
+            messages.append("no future-dated earnings rows staged")
+        elif current_future == 0:
+            # Initial load: no baseline for the ratio gate, so require a real
+            # forward horizon and broad universe coverage instead of one stray row.
+            if horizon_days < MIN_FORWARD_HORIZON_DAYS:
+                messages.append(
+                    f"forward horizon only {horizon_days}d < {MIN_FORWARD_HORIZON_DAYS}d "
+                    "on initial load"
+                )
+            if coverage < MIN_INITIAL_COVERAGE:
+                messages.append(
+                    f"initial universe coverage {coverage:.0%} "
+                    f"< {MIN_INITIAL_COVERAGE:.0%} — symbol join likely broken"
+                )
+        elif staged.future_count < current_future * MIN_FUTURE_RATIO:
+            messages.append(
+                f"forward calendar collapsed: staged {staged.future_count} future rows "
+                f"< {MIN_FUTURE_RATIO:.0%} of current {current_future}"
+            )
+
+        keys = [(r["ticker_id"], r["announce_date"]) for r in staged.rows]
+        if len(keys) != len(set(keys)):
+            messages.append("duplicate (ticker_id, announce_date) in staged set")
+
+        summary = {
+            **staged.stats,
+            "staged_rows": len(staged.rows),
+            "future_rows": staged.future_count,
+            "current_future_rows": current_future,
+            "forward_horizon_days": horizon_days,
+            "universe_coverage": round(coverage, 3),
+        }
+        return ValidationResult(
+            ok=not messages,
+            summary=summary,
+            source_max_date=staged.max_announce_date,
+            messages=messages,
+        )
+
+    async def promote(self, db, staged: StagedEarnings, run_id: int) -> dict[str, int]:
+        # Rescheduling: drop this source's future rows, then upsert the staged
+        # set. Past rows (results) are never deleted; moved/cancelled future
+        # dates simply don't reappear.
+        deleted = (
+            await db.execute(
+                delete(EarningsEvent).where(
+                    EarningsEvent.source == SOURCE,
+                    EarningsEvent.announce_date > self.today,
+                )
+            )
+        ).rowcount or 0
+
+        now = _now()
+        for r in staged.rows:
+            stmt = insert_for_session(db, EarningsEvent).values(
+                ticker_id=r["ticker_id"],
+                announce_date=r["announce_date"],
+                session=r["session"],
+                period_end=r["period_end"],
+                eps_estimate=r["eps_estimate"],
+                eps_actual=r["eps_actual"],
+                source=SOURCE,
+                import_run_id=run_id,
+                created_at=now,
+            )
+            # Preserve a non-null prior EPS/period-end if a re-pairing comes back
+            # null; prefer a known session over 'unknown'.
+            stmt = stmt.on_conflict_do_update(
+                index_elements=["ticker_id", "announce_date"],
+                set_={
+                    "session": case(
+                        (stmt.excluded.session != "unknown", stmt.excluded.session),
+                        else_=EarningsEvent.session,
+                    ),
+                    "period_end": func.coalesce(
+                        stmt.excluded.period_end, EarningsEvent.period_end
+                    ),
+                    "eps_estimate": func.coalesce(
+                        stmt.excluded.eps_estimate, EarningsEvent.eps_estimate
+                    ),
+                    "eps_actual": func.coalesce(
+                        stmt.excluded.eps_actual, EarningsEvent.eps_actual
+                    ),
+                    "source": stmt.excluded.source,
+                    "import_run_id": stmt.excluded.import_run_id,
+                },
+            )
+            await db.execute(stmt)
+
+        return {"deleted_future": int(deleted), "upserted": len(staged.rows)}
+
+    # -- helpers -----------------------------------------------------------
+
+    async def _load_universe(self, db) -> dict[str, int]:
+        rows = (await db.execute(select(Ticker.id, Ticker.symbol))).all()
+        return {
+            earnings_alignment.normalise_symbol(symbol): tid
+            for tid, symbol in rows
+            if symbol
+        }
+
+    async def _current_future_count(self, db) -> int:
+        return (
+            await db.execute(
+                select(func.count())
+                .select_from(EarningsEvent)
+                .where(
+                    EarningsEvent.source == SOURCE,
+                    EarningsEvent.announce_date > self.today,
+                )
+            )
+        ).scalar_one()
+
+
+def _require_columns(rows: list[dict[str, str]], required: set[str], table: str) -> None:
+    """Upstream schema-change gate: a missing column stops the run (→ failed)."""
+    if not rows:
+        return
+    present = set(rows[0].keys())
+    missing = required - present
+    if missing:
+        raise ValueError(f"{table}: upstream schema change, missing columns {sorted(missing)}")
+
+
+def _parse_calendar(raw: list[dict[str, str]], universe: dict[str, int]) -> list[dict[str, Any]]:
+    out: list[dict[str, Any]] = []
+    for row in raw:
+        symbol = earnings_alignment.normalise_symbol(row.get("act_symbol"))
+        raw_date = str(row.get("date") or "")[:10]
+        if symbol not in universe or not raw_date:
+            continue
+        announce_date = date.fromisoformat(raw_date)
+        if announce_date < WINDOW_START:
+            continue
+        out.append(
+            {
+                "symbol": symbol,
+                "announce_date": announce_date,
+                "session": earnings_alignment.normalise_session(row.get("when")),
+            }
+        )
+    return out
+
+
+def _parse_history(raw: list[dict[str, str]], universe: dict[str, int]) -> list[dict[str, Any]]:
+    out: list[dict[str, Any]] = []
+    for row in raw:
+        symbol = earnings_alignment.normalise_symbol(row.get("act_symbol"))
+        raw_date = str(row.get("period_end_date") or "")[:10]
+        if symbol not in universe or not raw_date:
+            continue
+        out.append(
+            {
+                "symbol": symbol,
+                "period_end_date": date.fromisoformat(raw_date),
+                "eps_actual": earnings_alignment.safe_number(row.get("reported")),
+                "eps_estimate": earnings_alignment.safe_number(row.get("estimate")),
+            }
+        )
+    return out

app/services/earnings_alignment.py

@@ -0,0 +1,207 @@
+"""Pure calendar<->EPS-history alignment for the DoltHub earnings source.
+
+The earnings repo keeps the announcement calendar (`earnings_calendar`) and the
+reported/estimate EPS history (`eps_history`) in separate tables with no shared
+key — the calendar has announce dates, the history has period-end dates. This
+module reproduces the research importer's **minimum-cost monotonic alignment**
+(`scripts/import_dolthub_earnings.py`) as pure, DB-free, unit-testable functions
+so the production importer can reuse it without extending that one-off script.
+
+Constants and cost function are kept identical to the research script; the DP is
+what pairs each announcement with the quarter it reported, tolerating gaps on
+either side. Do not tune these without re-validating surprise-history pairing.
+"""
+
+from __future__ import annotations
+
+import math
+from collections import defaultdict
+from datetime import date
+from typing import Any
+
+# Alignment costs — identical to scripts/import_dolthub_earnings.py.
+SKIP_EVENT_COST = 45.0
+SKIP_PERIOD_COST = 45.0
+_TYPICAL_ANNOUNCE_LAG_DAYS = 30  # announcements land ~a month after period end
+_MISSING_SESSION_PENALTY = 3.0
+
+# Session normalization → the three values the schema/API promise.
+_SESSION_ALIASES = {
+    "before market open": "bmo",
+    "before open": "bmo",
+    "bmo": "bmo",
+    "after market close": "amc",
+    "after close": "amc",
+    "amc": "amc",
+}
+
+
+def normalise_symbol(value: Any) -> str:
+    """Upper-case, trim, and map dots to dashes so the DoltHub `act_symbol`
+    (`BF.B`) and the app's `tickers.symbol` join after the same normalization."""
+    return str(value or "").strip().upper().replace(".", "-")
+
+
+def normalise_session(value: Any) -> str:
+    """Map the source `when` text to bmo | amc | unknown. Anything not clearly a
+    pre-open or post-close session (including 'during market hours' and blanks)
+    collapses to 'unknown' — the schema/API only promise those three."""
+    cleaned = str(value or "").strip().lower().replace("_", " ").replace("-", " ")
+    return _SESSION_ALIASES.get(cleaned, "unknown")
+
+
+def safe_number(value: Any) -> float | None:
+    if value is None or str(value).strip() == "":
+        return None
+    try:
+        result = float(value)
+    except (TypeError, ValueError):
+        return None
+    return result if math.isfinite(result) else None
+
+
+def dedup_calendar(
+    rows: list[dict[str, Any]],
+) -> tuple[dict[str, list[dict[str, Any]]], dict[str, int]]:
+    """Collapse to one row per (symbol, announce_date), preferring a known
+    session over 'unknown'. Rows must be pre-parsed:
+    {symbol, announce_date: date, session}. Returns {symbol: [events sorted by
+    date]} and dedup stats."""
+    by_key: dict[tuple[str, date], dict[str, Any]] = {}
+    duplicate_rows = 0
+    restated_rows = 0
+    for row in rows:
+        key = (row["symbol"], row["announce_date"])
+        previous = by_key.get(key)
+        if previous is None:
+            by_key[key] = row
+            continue
+        duplicate_rows += 1
+        prev_known = previous["session"] != "unknown"
+        new_known = row["session"] != "unknown"
+        if prev_known and new_known and previous["session"] != row["session"]:
+            restated_rows += 1
+        # Prefer a row that carries a known session.
+        if new_known:
+            by_key[key] = row
+    grouped: dict[str, list[dict[str, Any]]] = defaultdict(list)
+    for row in by_key.values():
+        grouped[row["symbol"]].append(row)
+    for events in grouped.values():
+        events.sort(key=lambda item: item["announce_date"])
+    return grouped, {
+        "deduped_rows": len(by_key),
+        "duplicate_rows": duplicate_rows,
+        "restated_rows": restated_rows,
+    }
+
+
+def dedup_history(
+    rows: list[dict[str, Any]],
+) -> tuple[dict[str, list[dict[str, Any]]], dict[str, int]]:
+    """Collapse to one row per (symbol, period_end_date), preferring the row with
+    more non-null EPS fields. Rows must be pre-parsed:
+    {symbol, period_end_date: date, eps_actual, eps_estimate}."""
+    fields = ("eps_actual", "eps_estimate")
+    by_key: dict[tuple[str, date], dict[str, Any]] = {}
+    duplicate_rows = 0
+    restated_rows = 0
+    for row in rows:
+        key = (row["symbol"], row["period_end_date"])
+        previous = by_key.get(key)
+        if previous is None:
+            by_key[key] = row
+            continue
+        duplicate_rows += 1
+        if any(
+            previous.get(f) is not None
+            and row.get(f) is not None
+            and previous[f] != row[f]
+            for f in fields
+        ):
+            restated_rows += 1
+        prev_score = sum(previous.get(f) is not None for f in fields)
+        new_score = sum(row.get(f) is not None for f in fields)
+        if new_score >= prev_score:
+            by_key[key] = row
+    grouped: dict[str, list[dict[str, Any]]] = defaultdict(list)
+    for row in by_key.values():
+        grouped[row["symbol"]].append(row)
+    for periods in grouped.values():
+        periods.sort(key=lambda item: item["period_end_date"])
+    return grouped, {
+        "deduped_rows": len(by_key),
+        "duplicate_rows": duplicate_rows,
+        "restated_rows": restated_rows,
+    }
+
+
+def match_cost(event: dict[str, Any], period: dict[str, Any]) -> float:
+    delta = (event["announce_date"] - period["period_end_date"]).days
+    penalty = _MISSING_SESSION_PENALTY if event.get("session") == "unknown" else 0.0
+    return float(abs(delta - _TYPICAL_ANNOUNCE_LAG_DAYS)) + penalty
+
+
+def align_symbol(
+    events: list[dict[str, Any]],
+    periods: list[dict[str, Any]],
+    *,
+    max_lag_days: int,
+    max_lead_days: int,
+) -> tuple[list[tuple[int, int]], list[int], list[int]]:
+    """Minimum-cost monotonic calendar-to-period alignment for one symbol.
+
+    Both lists must be sorted ascending (by announce_date / period_end_date). A
+    match is allowed only when ``-max_lead_days <= announce_date - period_end <=
+    max_lag_days``. Returns (matches, unmatched_event_indices,
+    unmatched_period_indices).
+    """
+    n_events = len(events)
+    n_periods = len(periods)
+    scores = [[0.0] * (n_periods + 1) for _ in range(n_events + 1)]
+    choices = [[""] * (n_periods + 1) for _ in range(n_events + 1)]
+    for e in range(n_events - 1, -1, -1):
+        scores[e][n_periods] = scores[e + 1][n_periods] + SKIP_EVENT_COST
+        choices[e][n_periods] = "event"
+    for p in range(n_periods - 1, -1, -1):
+        scores[n_events][p] = scores[n_events][p + 1] + SKIP_PERIOD_COST
+        choices[n_events][p] = "period"
+
+    for e in range(n_events - 1, -1, -1):
+        for p in range(n_periods - 1, -1, -1):
+            options = [
+                (scores[e + 1][p] + SKIP_EVENT_COST, 2, "event"),
+                (scores[e][p + 1] + SKIP_PERIOD_COST, 1, "period"),
+            ]
+            delta = (events[e]["announce_date"] - periods[p]["period_end_date"]).days
+            if -max_lead_days <= delta <= max_lag_days:
+                options.append(
+                    (scores[e + 1][p + 1] + match_cost(events[e], periods[p]), 0, "match")
+                )
+            score, _, choice = min(options)
+            scores[e][p] = score
+            choices[e][p] = choice
+
+    matches: list[tuple[int, int]] = []
+    unmatched_events: list[int] = []
+    unmatched_periods: list[int] = []
+    e = p = 0
+    while e < n_events or p < n_periods:
+        if e >= n_events:
+            unmatched_periods.extend(range(p, n_periods))
+            break
+        if p >= n_periods:
+            unmatched_events.extend(range(e, n_events))
+            break
+        choice = choices[e][p]
+        if choice == "match":
+            matches.append((e, p))
+            e += 1
+            p += 1
+        elif choice == "period":
+            unmatched_periods.append(p)
+            p += 1
+        else:
+            unmatched_events.append(e)
+            e += 1
+    return matches, unmatched_events, unmatched_periods

app/services/event_study_service.py

@@ -28,6 +28,10 @@ DRAWDOWN_LOOKBACK = 252
 HORIZON_DAYS = 20
 WARN_PERCENTILE = 80.0
 TRAIN_FRACTION = 0.70
+# Below this many holdout corrections, recall is one event away from a very
+# different headline and should not be read as a property of the score.
+MIN_EVENTS_FOR_CONFIDENCE = 8
+SENSOR_MISMATCH_TOLERANCE = 0.10
 
 
 def _median(values: list[float]) -> float | None:
@@ -149,30 +153,72 @@ def _warning_series(
     breadth_divergence: dict[date, float],
     dates: list[date],
     config: dict,
-) -> dict[date, float]:
-    """Technical Warning score used historically (fundamentals have no PIT history)."""
+    oas_series: rms.Series | None = None,
+) -> tuple[dict[date, float], dict[date, int]]:
+    """Warning score per session plus how many sensors backed it.
+
+    v2 re-derived this by hand from ``WARNING_WEIGHTS`` and so would have kept
+    measuring the old construct after a scoring change.  Since v3 dropped
+    fundamentals from the score, this is now exactly the live Warning score
+    rather than a technical-only approximation of it.
+
+    The sensor count matters because the score renormalises over whatever is
+    available: a session backed by two sensors is not drawn from the same
+    distribution as one backed by three, and the frozen threshold assumes it is.
+    """
     tickers = config["tickers"]
     smh_full = prices.get(tickers["leaders"][0], [])
     spy_full = prices.get(tickers["market"], [])
     out: dict[date, float] = {}
+    backing: dict[date, int] = {}
     for session in dates:
-        divergence = breadth_divergence.get(session)
-        relative = rms.p4_relative_strength(
+        sensors = rms.warning_sensor_scores(
+            breadth_divergence.get(session),
             rms._closes_asof(smh_full, session),
             rms._closes_asof(spy_full, session),
+            rms._window_asof(oas_series, session, rms.HY_OAS_WINDOW_DAYS),
         )
-        values: list[tuple[float, float]] = []
-        if divergence is not None:
-            values.append((divergence, rms.WARNING_WEIGHTS["breadth_divergence"]))
-        if relative is not None:
-            values.append((relative, rms.WARNING_WEIGHTS["relative_strength"]))
-        if values:
-            out[session] = round(
-                sum(value * weight for value, weight in values)
-                / sum(weight for _, weight in values),
-                2,
-            )
-    return out
+        score = rms.score_warning_sensors(sensors)
+        if score is not None:
+            out[session] = round(score, 2)
+            backing[session] = sum(1 for value in sensors.values() if value is not None)
+    return out, backing
+
+
+def _reliability(
+    dates: list[date],
+    split: int,
+    backing: dict[date, int],
+    events_detected: int,
+    events_in_holdout: int,
+) -> dict:
+    """How far the headline metrics can actually be trusted.
+
+    Two things repeatedly invite over-reading this report:
+
+    * The holdout carries only the corrections that fall in the last 30% of the
+      sample. A "2/4" is one event away from "3/4", and in practice the events
+      that flip are decided by where the frozen threshold happens to land rather
+      than by whether the score saw anything.
+    * The score renormalises over available sensors, so a training window that
+      predates a sensor's history freezes a threshold on a different construct
+      than the holdout is measured against.
+    """
+    expected = len(rms.WARNING_WEIGHTS)
+    train = [backing[d] for d in dates[:split] if d in backing]
+    holdout = [backing[d] for d in dates[split:] if d in backing]
+    train_full = sum(1 for n in train if n == expected) / len(train) if train else 0.0
+    holdout_full = sum(1 for n in holdout if n == expected) / len(holdout) if holdout else 0.0
+    return {
+        "events_detected": events_detected,
+        "events_in_holdout": events_in_holdout,
+        "minimum_events": MIN_EVENTS_FOR_CONFIDENCE,
+        "underpowered": events_in_holdout < MIN_EVENTS_FOR_CONFIDENCE,
+        "sensors_expected": expected,
+        "train_full_sensor_share": round(train_full * 100, 1),
+        "holdout_full_sensor_share": round(holdout_full * 100, 1),
+        "sensor_coverage_mismatch": abs(train_full - holdout_full) > SENSOR_MISMATCH_TOLERANCE,
+    }
 
 
 async def run_event_study(
@@ -195,7 +241,13 @@ async def run_event_study(
         db, config["breadth_basket"], window=200, min_tickers=20
     )
     divergence = breadth_service.compute_divergence_series(breadth, benchmark)
-    warning = _warning_series(prices, divergence, dates, config)
+    oas_series = await rms._fetch_fred_series("BAMLH0A0HYM2", start, end)
+    warning, backing = _warning_series(prices, divergence, dates, config, oas_series)
+    # The credit sensor cannot reach back as far as the price history does (the
+    # upstream series is capped at ~3 years), so the earlier part of the sample
+    # scores on W1+W2 alone via renormalisation. Report where W3 starts rather
+    # than letting the threshold quietly straddle two sensor sets.
+    credit_from = oas_series[0][0].isoformat() if oas_series else None
 
     split = max(1, min(len(dates) - 1, int(len(dates) * TRAIN_FRACTION)))
     train_values = [warning[d] for d in dates[:split] if d in warning]
@@ -212,6 +264,8 @@ async def run_event_study(
         metrics["false_alarms"] / (holdout_sessions / 252.0), 2
     )
 
+    reliability = _reliability(dates, split, backing, len(all_events), len(holdout_events))
+
     basket_asof = date.fromisoformat(config["basket_asof"])
     retrospective = dates[split] < basket_asof
     evaluation = "exploratory" if retrospective else "holdout"
@@ -224,7 +278,14 @@ async def run_event_study(
         f"{evaluation.capitalize()} chronological test: warning episodes preceded "
         f"{metrics['events_warned']}/{metrics['events']} 10% corrections; "
         f"{metrics['events_missed']} missed, {metrics['false_alarms_per_year']:.1f} "
-        f"false alarms/year, {lead_text}."
+        f"false alarms/year, {lead_text}. "
+        f"{metrics['events']} of {reliability['events_detected']} detected corrections "
+        f"fall in the test period"
+        + (
+            "; too few to read recall as a property of the score."
+            if reliability["underpowered"]
+            else "."
+        )
     )
     per_event = metrics.pop("per_event")
 
@@ -243,6 +304,7 @@ async def run_event_study(
             "train_fraction": TRAIN_FRACTION,
             "warn_percentile": WARN_PERCENTILE,
             "warn_threshold": round(warn_threshold, 1),
+            "credit_sensor_from": credit_from,
             "basket_hash": rms._basket_hash(config["breadth_basket"]),
             "basket_asof": config["basket_asof"],
         },
@@ -255,6 +317,7 @@ async def run_event_study(
             "holdout_sessions": holdout_sessions,
         },
         "metrics": metrics,
+        "reliability": reliability,
         "events": per_event,
         "recent_breadth": [
             {"date": d.isoformat(), "breadth": breadth[d], "warning": warning.get(d)}
@@ -266,8 +329,11 @@ async def run_event_study(
         "event": "regime_event_study_complete",
         "evaluation": evaluation,
         "events": metrics["events"],
+        "events_detected": reliability["events_detected"],
         "warned": metrics["events_warned"],
         "false_alarms_per_year": metrics["false_alarms_per_year"],
+        "underpowered": reliability["underpowered"],
+        "sensor_coverage_mismatch": reliability["sensor_coverage_mismatch"],
     }))
     return report
 

app/services/fundamental_data_refresh_service.py

@@ -0,0 +1,179 @@
+"""A5 activation: refresh the legacy fundamentals cache from local bulk data."""
+
+from __future__ import annotations
+
+import json
+from datetime import date, datetime, timezone
+from typing import Any
+
+from sqlalchemy import select, update
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.database import insert_for_session
+from app.models.fundamental import FundamentalData
+from app.models.score import CompositeScore, DimensionScore
+from app.services import fundamentals_candidate_service, settings_store
+
+
+# Absence is deliberately false. Production activation therefore requires one
+# explicit, durable SystemSetting change after the A5 evidence is approved.
+ACTIVATION_KEY = "fundamental_data_sec_dolt_cutover_enabled"
+_SCORE_FIELDS = ("pe_ratio", "revenue_growth", "earnings_surprise")
+
+
+async def is_enabled(db: AsyncSession) -> bool:
+    raw = await settings_store.get_value(db, ACTIVATION_KEY, "false")
+    return str(raw).strip().lower() == "true"
+
+
+async def refresh_if_enabled(
+    db: AsyncSession,
+    *,
+    now: datetime | None = None,
+    today: date | None = None,
+) -> dict[str, Any]:
+    """Refresh atomically when activated; otherwise perform no writes."""
+    if not await is_enabled(db):
+        return {
+            "enabled": False,
+            "refreshed": 0,
+            "score_inputs_changed": 0,
+            "dimension_scores_staled": 0,
+            "composite_scores_staled": 0,
+        }
+    return await refresh(db, now=now, today=today)
+
+
+async def refresh(
+    db: AsyncSession,
+    *,
+    now: datetime | None = None,
+    today: date | None = None,
+) -> dict[str, Any]:
+    """Replace every ticker's compat-cache row in one database transaction.
+
+    Candidate values are assembled before the first write and use only local
+    PostgreSQL tables. A failure rolls the whole refresh back. Only changes to
+    the three scoring inputs invalidate cached scores; market cap and the next
+    earnings date are display-only.
+    """
+    refreshed_at = now or datetime.now(timezone.utc)
+    candidates = await fundamentals_candidate_service.build_candidates(
+        db, today=today
+    )
+    ticker_ids = [candidate.ticker_id for candidate in candidates]
+    existing = await _existing_by_ticker(db, ticker_ids)
+    changed_ids = {
+        candidate.ticker_id
+        for candidate in candidates
+        if _score_inputs_changed(existing.get(candidate.ticker_id), candidate)
+    }
+
+    for candidate in candidates:
+        unavailable_json = json.dumps(
+            candidate.unavailable_fields, sort_keys=True
+        )
+        stmt = insert_for_session(db, FundamentalData).values(
+            ticker_id=candidate.ticker_id,
+            pe_ratio=candidate.pe_ratio,
+            revenue_growth=candidate.revenue_growth,
+            earnings_surprise=candidate.earnings_surprise,
+            market_cap=candidate.market_cap,
+            next_earnings_date=candidate.next_earnings_date,
+            fetched_at=refreshed_at,
+            unavailable_fields_json=unavailable_json,
+        )
+        await db.execute(
+            stmt.on_conflict_do_update(
+                index_elements=["ticker_id"],
+                set_={
+                    "pe_ratio": stmt.excluded.pe_ratio,
+                    "revenue_growth": stmt.excluded.revenue_growth,
+                    "earnings_surprise": stmt.excluded.earnings_surprise,
+                    "market_cap": stmt.excluded.market_cap,
+                    "next_earnings_date": stmt.excluded.next_earnings_date,
+                    "fetched_at": stmt.excluded.fetched_at,
+                    "unavailable_fields_json": (
+                        stmt.excluded.unavailable_fields_json
+                    ),
+                },
+            )
+        )
+
+    dimension_ids = await _fundamental_dimension_ids(db, changed_ids)
+    composite_ids = await _composite_ids(db, changed_ids)
+    if dimension_ids:
+        await db.execute(
+            update(DimensionScore)
+            .where(DimensionScore.ticker_id.in_(dimension_ids))
+            .values(is_stale=True)
+        )
+    if composite_ids:
+        await db.execute(
+            update(CompositeScore)
+            .where(CompositeScore.ticker_id.in_(composite_ids))
+            .values(is_stale=True)
+        )
+
+    await db.commit()
+    return {
+        "enabled": True,
+        "refreshed": len(candidates),
+        "score_inputs_changed": len(changed_ids),
+        "dimension_scores_staled": len(dimension_ids),
+        "composite_scores_staled": len(composite_ids),
+    }
+
+
+async def _existing_by_ticker(
+    db: AsyncSession, ticker_ids: list[int]
+) -> dict[int, FundamentalData]:
+    if not ticker_ids:
+        return {}
+    rows = (
+        await db.execute(
+            select(FundamentalData).where(
+                FundamentalData.ticker_id.in_(ticker_ids)
+            )
+        )
+    ).scalars()
+    return {row.ticker_id: row for row in rows}
+
+
+async def _fundamental_dimension_ids(
+    db: AsyncSession, ticker_ids: set[int]
+) -> set[int]:
+    if not ticker_ids:
+        return set()
+    rows = await db.execute(
+        select(DimensionScore.ticker_id).where(
+            DimensionScore.ticker_id.in_(ticker_ids),
+            DimensionScore.dimension == "fundamental",
+        )
+    )
+    return set(rows.scalars())
+
+
+async def _composite_ids(
+    db: AsyncSession, ticker_ids: set[int]
+) -> set[int]:
+    if not ticker_ids:
+        return set()
+    rows = await db.execute(
+        select(CompositeScore.ticker_id).where(
+            CompositeScore.ticker_id.in_(ticker_ids)
+        )
+    )
+    return set(rows.scalars())
+
+
+def _score_inputs_changed(
+    existing: FundamentalData | None,
+    candidate: fundamentals_candidate_service.CandidateFundamentals,
+) -> bool:
+    if existing is None:
+        return True
+    return any(
+        getattr(existing, field) != getattr(candidate, field)
+        for field in _SCORE_FIELDS
+    )

app/services/fundamentals_api_service.py

@@ -0,0 +1,344 @@
+"""Assemble the additive fundamentals API v1 objects (earnings, metrics,
+valuation, reads) from SEC snapshots + Dolt earnings + the latest price.
+
+Strictly additive: the router merges these into the existing FundamentalResponse
+without touching legacy fields. Valuation ratios are computed at REQUEST TIME from
+the stored snapshots + the latest ohlcv close (no stored valuation). Peer stats are
+batched and CIK-deduplicated; invalid valuation inputs are guarded to null.
+"""
+
+from __future__ import annotations
+
+import math
+from collections import defaultdict
+from datetime import date, datetime
+from typing import Any
+from zoneinfo import ZoneInfo
+
+from sqlalchemy import func, select
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.models.earnings_event import EarningsEvent
+from app.models.fundamental_snapshot import FundamentalSnapshot
+from app.models.ohlcv import OHLCVRecord
+from app.models.ticker import Ticker
+from app.services import fundamentals_derivation as deriv
+from app.services import fundamentals_peers as peers
+from app.services import fundamentals_reads as reads
+
+# The fixed metric row set — every key always present, value null when unavailable.
+METRIC_KEYS = (
+    "revenue_growth_yoy", "eps_growth_yoy", "operating_margin", "fcf_margin",
+    "net_debt", "net_debt_to_ebitda", "share_count_change_yoy",
+)
+
+
+async def build_fundamentals_v1(db: AsyncSession, symbol: str, *, today: date | None = None) -> dict[str, Any]:
+    today = today or _ny_today()
+    ticker = await _ticker_by_symbol(db, symbol)
+
+    earnings = await _build_earnings(db, ticker.id, today) if ticker else _empty_earnings()
+    if ticker is None or not ticker.cik:
+        # No SEC identity: metrics present but null, valuation null, empty reads.
+        return {"earnings": earnings, "metrics": _empty_metrics(), "valuation": None,
+                "reads": _empty_reads()}
+
+    subject_cik = ticker.cik
+    derived = deriv.derive((await _snapshots_for(db, [subject_cik])).get(subject_cik, []))
+
+    two = peers.two_digit_sic(ticker.sic)
+    peer_derived: dict[str, deriv.DerivedFundamentals] = {}
+    peer_price_by_cik: dict[str, tuple[float, date] | None] = {}
+    if two:
+        # Subject's representative is the REQUESTED ticker (so its price is used for
+        # the subject in the peer set); other issuers pick a deterministic-by-symbol rep.
+        group = await _peer_group(db, two, subject_cik, ticker.id)
+        peer_snaps = await _snapshots_for(db, list(group))
+        peer_derived = {cik: deriv.derive(rows) for cik, rows in peer_snaps.items()}
+        closes = await _latest_closes(db, set(group.values()))
+        peer_price_by_cik = {cik: closes.get(tid) for cik, tid in group.items()}
+
+    subject_price = await _latest_close(db, ticker.id)
+    metrics = _build_metrics(derived, peer_derived, two)
+    valuation = _build_valuation(derived, subject_price, peer_derived, peer_price_by_cik, two)
+    reads_obj = _build_reads(metrics, valuation)
+    return {"earnings": earnings, "metrics": metrics, "valuation": valuation, "reads": reads_obj}
+
+
+# -- earnings ----------------------------------------------------------------
+
+async def _build_earnings(db, ticker_id: int, today: date) -> dict[str, Any]:
+    rows = (await db.execute(
+        select(EarningsEvent).where(EarningsEvent.ticker_id == ticker_id)
+    )).scalars().all()
+    # Same-day earnings are UPCOMING (days_until 0); recent is strictly earlier.
+    upcoming = sorted((e for e in rows if e.announce_date >= today), key=lambda e: e.announce_date)
+    past = sorted((e for e in rows if e.announce_date < today), key=lambda e: e.announce_date, reverse=True)
+
+    nxt = None
+    if upcoming:
+        e = upcoming[0]
+        nxt = {"date": e.announce_date.isoformat(), "session": e.session,
+               "days_until": (e.announce_date - today).days}
+    recent = [{
+        "announce_date": e.announce_date.isoformat(),
+        "period_end": _iso(e.period_end),
+        "eps_estimate": e.eps_estimate,
+        "eps_actual": e.eps_actual,
+        "surprise_pct": _surprise_pct(e.eps_estimate, e.eps_actual),
+    } for e in past[:4]]
+    return {"next": nxt, "recent": recent}
+
+
+def _surprise_pct(estimate, actual):
+    if estimate is None or actual is None or estimate == 0:
+        return None
+    return round((actual - estimate) / abs(estimate) * 100.0, 2)
+
+
+# -- metrics -----------------------------------------------------------------
+
+def _build_metrics(derived, peer_derived, two: str | None) -> list[dict[str, Any]]:
+    out = []
+    for key in METRIC_KEYS:
+        series = derived.metrics.get(key)
+        value = series.value if series else None
+        history = [{"period_end": _iso(p.period_end), "value": p.value} for p in (series.history if series else [])]
+        industry = None
+        if two and peer_derived and key in peers.HIGHER_IS_BETTER:
+            group_values = [
+                (pd.metrics.get(key).value if pd.metrics.get(key) else None)
+                for pd in peer_derived.values()
+            ]
+            stat = peers.peer_stat_for(key, value, group_values)
+            if stat:
+                industry = {"label": f"SIC {two} peers", "median": round(stat.median, 4),
+                            "favorable_percentile": stat.favorable_percentile, "peer_count": stat.peer_count}
+        out.append({
+            "key": key,
+            "value": value,
+            "history": history,
+            "industry": industry,
+            "period_end": _iso(series.period_end) if series else None,
+            "filed_date": _iso(series.filed_date) if series else None,
+            "caveat": series.caveat if series else None,
+            "source": "sec",
+        })
+    return out
+
+
+# -- valuation (request-time) ------------------------------------------------
+
+def _build_valuation(derived, subject_price, peer_derived, peer_price_by_cik, two) -> dict[str, Any] | None:
+    if derived.latest_period_end is None:
+        return None  # no snapshots yet
+    price = subject_price[0] if subject_price else None
+    price_date = subject_price[1] if subject_price else None
+    if not _finite(price) or price <= 0:
+        return None  # no usable price -> valuation null (approved contract)
+
+    pe = _pe(price, derived.ttm_diluted_eps)
+    market_cap = _market_cap(price, derived.shares_outstanding)
+    fcf_yield = _fcf_yield(derived.ttm_fcf, market_cap)
+
+    pe_industry = fcf_yield_industry = None
+    if two and peer_derived:
+        pe_values = [_pe(_p(peer_price_by_cik.get(cik)), pd.ttm_diluted_eps) for cik, pd in peer_derived.items()]
+        fy_values = [
+            _fcf_yield(pd.ttm_fcf, _market_cap(_p(peer_price_by_cik.get(cik)), pd.shares_outstanding))
+            for cik, pd in peer_derived.items()
+        ]
+        pe_industry = _industry("pe", pe, pe_values, two)
+        fcf_yield_industry = _industry("fcf_yield", fcf_yield, fy_values, two)
+
+    return {
+        "pe": _round(pe, 2),
+        "fcf_yield": _round(fcf_yield, 2),
+        "market_cap_est": _round(market_cap, 0),
+        # market_cap_est and fcf_yield both rest on the share count. When it came
+        # from the weighted-average diluted fallback (multi-class issuers, whose
+        # per-class cover-page count is absent from companyfacts), say so rather
+        # than presenting a period average as a point-in-time count.
+        "shares_estimated": bool(
+            market_cap is not None and derived.shares_outstanding_estimated
+        ),
+        # A null P/E is ambiguous: no earnings data, or earnings we deliberately
+        # suppressed. Only the latter carries a caveat, so a split-contaminated
+        # TTM says why instead of looking like missing data.
+        "pe_caveat": derived.ttm_diluted_eps_caveat if pe is None else None,
+        "pe_industry": pe_industry,
+        "fcf_yield_industry": fcf_yield_industry,
+        "price_date": _iso(price_date),
+    }
+
+
+def _pe(price, ttm_eps):
+    if not _finite(price) or price <= 0 or not _finite(ttm_eps) or ttm_eps <= 0:
+        return None
+    return price / ttm_eps
+
+
+def _market_cap(price, shares):
+    if not _finite(price) or price <= 0 or not _finite(shares) or shares <= 0:
+        return None
+    return price * shares
+
+
+def _fcf_yield(ttm_fcf, market_cap):
+    if not _finite(ttm_fcf) or not _finite(market_cap) or market_cap <= 0:
+        return None
+    return ttm_fcf / market_cap * 100.0
+
+
+def _industry(key, subject, group_values, two):
+    stat = peers.peer_stat_for(key, subject, group_values)
+    if stat is None:
+        return None
+    return {"label": f"SIC {two} peers", "median": round(stat.median, 4),
+            "favorable_percentile": stat.favorable_percentile, "peer_count": stat.peer_count}
+
+
+# -- reads -------------------------------------------------------------------
+
+_READ_KEYS = METRIC_KEYS + ("pe", "fcf_yield")
+
+
+def _build_reads(metrics: list[dict], valuation: dict | None) -> dict[str, Any]:
+    by_metric = {m["key"]: m for m in metrics}
+
+    def hist(key):
+        return [_Pt(p["value"]) for p in by_metric.get(key, {}).get("history", [])]
+
+    growth = reads.growth_read(hist("revenue_growth_yoy"))
+    eps_growth = reads.growth_read(hist("eps_growth_yoy"))
+    op_margin = reads.margin_read(hist("operating_margin"))
+    fcf_margin = reads.margin_read(hist("fcf_margin"))
+    share = reads.share_count_read(by_metric.get("share_count_change_yoy", {}).get("value"))
+    leverage = reads.peer_read("net_debt_to_ebitda", _pct(by_metric.get("net_debt_to_ebitda", {}).get("industry")))
+    pe_read = reads.peer_read("pe", _pct(valuation.get("pe_industry"))) if valuation else None
+    fcf_yield_read = reads.peer_read("fcf_yield", _pct(valuation.get("fcf_yield_industry"))) if valuation else None
+
+    # Fixed by_key map over every metric + pe + fcf_yield (null where unavailable).
+    by_key: dict[str, str | None] = {k: None for k in _READ_KEYS}
+    by_key.update({
+        "revenue_growth_yoy": growth,
+        "eps_growth_yoy": eps_growth,
+        "operating_margin": op_margin,
+        "fcf_margin": fcf_margin,
+        "share_count_change_yoy": share,
+        "net_debt_to_ebitda": leverage,
+        "pe": pe_read,
+        "fcf_yield": fcf_yield_read,
+    })
+    header = reads.header_sentence(growth, op_margin, pe_read or fcf_yield_read) or None
+    return {"header": header, "by_key": by_key}
+
+
+def _empty_reads() -> dict[str, Any]:
+    return {"header": None, "by_key": {k: None for k in _READ_KEYS}}
+
+
+class _Pt:
+    __slots__ = ("value",)
+
+    def __init__(self, value):
+        self.value = value
+
+
+def _pct(industry: dict | None):
+    return industry.get("favorable_percentile") if industry else None
+
+
+# -- queries -----------------------------------------------------------------
+
+async def _ticker_by_symbol(db, symbol: str) -> Ticker | None:
+    return (await db.execute(
+        select(Ticker).where(Ticker.symbol == symbol.strip().upper())
+    )).scalar_one_or_none()
+
+
+async def _snapshots_for(db, ciks) -> dict[str, list]:
+    out: dict[str, list] = defaultdict(list)
+    if not ciks:
+        return out
+    rows = (await db.execute(
+        select(FundamentalSnapshot).where(FundamentalSnapshot.cik.in_(list(ciks)))
+    )).scalars().all()
+    for r in rows:
+        out[r.cik].append(r)
+    return out
+
+
+async def _peer_group(db, two: str, subject_cik: str, subject_tid: int) -> dict[str, int]:
+    """{cik: representative ticker_id} for tracked issuers in the 2-digit SIC group,
+    CIK-deduplicated. Each issuer's representative is its lexicographically-smallest
+    symbol (deterministic), EXCEPT the subject issuer, which uses the requested
+    ticker — so a multi-class subject (GOOGL) is priced by the requested class, not
+    an arbitrary sibling (GOOG)."""
+    rows = (await db.execute(
+        select(Ticker.cik, Ticker.id, Ticker.symbol)
+        .where(Ticker.cik.is_not(None), func.substr(Ticker.sic, 1, 2) == two)
+    )).all()
+    rep: dict[str, tuple[int, str]] = {}
+    for cik, tid, sym in rows:
+        key = sym or ""
+        if cik not in rep or key < rep[cik][1]:
+            rep[cik] = (tid, key)
+    group = {cik: tid for cik, (tid, _) in rep.items()}
+    if subject_cik in group:
+        group[subject_cik] = subject_tid  # requested ticker prices the subject
+    return group
+
+
+async def _latest_closes(db, ticker_ids: set[int]) -> dict[int, tuple[float, date]]:
+    if not ticker_ids:
+        return {}
+    latest = (
+        select(OHLCVRecord.ticker_id, func.max(OHLCVRecord.date).label("d"))
+        .where(OHLCVRecord.ticker_id.in_(list(ticker_ids)))
+        .group_by(OHLCVRecord.ticker_id)
+        .subquery()
+    )
+    rows = (await db.execute(
+        select(OHLCVRecord.ticker_id, OHLCVRecord.close, OHLCVRecord.date).join(
+            latest, (OHLCVRecord.ticker_id == latest.c.ticker_id) & (OHLCVRecord.date == latest.c.d)
+        )
+    )).all()
+    return {tid: (close, d) for tid, close, d in rows}
+
+
+async def _latest_close(db, ticker_id: int) -> tuple[float, date] | None:
+    return (await _latest_closes(db, {ticker_id})).get(ticker_id)
+
+
+# -- helpers -----------------------------------------------------------------
+
+def _empty_metrics() -> list[dict[str, Any]]:
+    return [{"key": k, "value": None, "history": [], "industry": None,
+             "period_end": None, "filed_date": None, "caveat": None,
+             "source": "sec"} for k in METRIC_KEYS]
+
+
+def _empty_earnings() -> dict[str, Any]:
+    return {"next": None, "recent": []}
+
+
+def _p(price_tuple):
+    return price_tuple[0] if price_tuple else None
+
+
+def _finite(v) -> bool:
+    return isinstance(v, (int, float)) and not isinstance(v, bool) and math.isfinite(v)
+
+
+def _round(v, ndigits):
+    return round(v, ndigits) if _finite(v) else None
+
+
+def _iso(d) -> str | None:
+    return d.isoformat() if d else None
+
+
+def _ny_today() -> date:
+    """Today's New York calendar date — the market's day, not the server's."""
+    return datetime.now(ZoneInfo("America/New_York")).date()

app/services/fundamentals_candidate_service.py

@@ -0,0 +1,288 @@
+"""Local SEC/Dolt candidate values for the legacy fundamentals cache.
+
+This is the single read path shared by the A5 parity report and the activated
+``fundamental_data`` refresh.  It never contacts SEC or Dolt: every input comes
+from PostgreSQL, so price- and earnings-driven values can still refresh when an
+upstream import is unchanged or unavailable.
+"""
+
+from __future__ import annotations
+
+import math
+from collections import defaultdict
+from dataclasses import dataclass, field
+from datetime import date, datetime
+from typing import Any
+from zoneinfo import ZoneInfo
+
+from sqlalchemy import func, select
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.models.earnings_event import EarningsEvent
+from app.models.fundamental_snapshot import FundamentalSnapshot
+from app.models.ohlcv import OHLCVRecord
+from app.models.ticker import Ticker
+from app.services import fundamentals_derivation as deriv
+
+
+@dataclass(frozen=True)
+class CandidateFundamentals:
+    ticker_id: int
+    symbol: str
+    cik: str | None
+    pe_ratio: float | None
+    revenue_growth: float | None
+    earnings_surprise: float | None
+    market_cap: float | None
+    next_earnings_date: date | None
+    price_date: date | None
+    unavailable_fields: dict[str, str] = field(default_factory=dict)
+
+
+async def build_candidates(
+    db: AsyncSession,
+    *,
+    today: date | None = None,
+) -> list[CandidateFundamentals]:
+    """Derive current cache candidates using only already-stored data."""
+    today = today or datetime.now(ZoneInfo("America/New_York")).date()
+    tickers = list(
+        (await db.execute(select(Ticker).order_by(Ticker.symbol))).scalars()
+    )
+    if not tickers:
+        return []
+
+    ticker_ids = [ticker.id for ticker in tickers]
+    ciks = sorted({ticker.cik for ticker in tickers if ticker.cik})
+    derived_by_cik = await _derived_by_cik(db, ciks)
+    closes_by_ticker = await _latest_closes(db, ticker_ids)
+    surprise_by_ticker, next_by_ticker = await _earnings_values(
+        db, ticker_ids, today
+    )
+
+    out: list[CandidateFundamentals] = []
+    for ticker in tickers:
+        derived = derived_by_cik.get(ticker.cik) if ticker.cik else None
+        close = closes_by_ticker.get(ticker.id)
+        price = close[0] if close is not None else None
+        price_date = close[1] if close is not None else None
+        growth_series = (
+            derived.metrics.get("revenue_growth_yoy")
+            if derived is not None
+            else None
+        )
+
+        pe_ratio = (
+            _pe(price, derived.ttm_diluted_eps)
+            if derived is not None
+            else None
+        )
+        revenue_growth = (
+            float(growth_series.value)
+            if growth_series is not None and _finite(growth_series.value)
+            else None
+        )
+        earnings_surprise = surprise_by_ticker.get(ticker.id)
+        market_cap = (
+            _market_cap(price, derived.shares_outstanding)
+            if derived is not None
+            else None
+        )
+        next_earnings_date = next_by_ticker.get(ticker.id)
+
+        out.append(
+            CandidateFundamentals(
+                ticker_id=ticker.id,
+                symbol=ticker.symbol,
+                cik=ticker.cik,
+                pe_ratio=pe_ratio,
+                revenue_growth=revenue_growth,
+                earnings_surprise=earnings_surprise,
+                market_cap=market_cap,
+                next_earnings_date=next_earnings_date,
+                price_date=price_date,
+                unavailable_fields=_availability_metadata(
+                    derived=derived,
+                    price=price,
+                    pe_ratio=pe_ratio,
+                    revenue_growth=revenue_growth,
+                    earnings_surprise=earnings_surprise,
+                    market_cap=market_cap,
+                    next_earnings_date=next_earnings_date,
+                ),
+            )
+        )
+    return out
+
+
+async def _derived_by_cik(
+    db: AsyncSession, ciks: list[str]
+) -> dict[str, deriv.DerivedFundamentals]:
+    if not ciks:
+        return {}
+    grouped: dict[str, list[FundamentalSnapshot]] = defaultdict(list)
+    rows = (
+        await db.execute(
+            select(FundamentalSnapshot).where(FundamentalSnapshot.cik.in_(ciks))
+        )
+    ).scalars()
+    for row in rows:
+        grouped[row.cik].append(row)
+    return {cik: deriv.derive(grouped.get(cik, [])) for cik in ciks}
+
+
+async def _latest_closes(
+    db: AsyncSession, ticker_ids: list[int]
+) -> dict[int, tuple[float, date]]:
+    latest = (
+        select(
+            OHLCVRecord.ticker_id,
+            func.max(OHLCVRecord.date).label("max_date"),
+        )
+        .where(OHLCVRecord.ticker_id.in_(ticker_ids))
+        .group_by(OHLCVRecord.ticker_id)
+        .subquery()
+    )
+    rows = (
+        await db.execute(
+            select(
+                OHLCVRecord.ticker_id,
+                OHLCVRecord.close,
+                OHLCVRecord.date,
+            ).join(
+                latest,
+                (OHLCVRecord.ticker_id == latest.c.ticker_id)
+                & (OHLCVRecord.date == latest.c.max_date),
+            )
+        )
+    ).all()
+    return {
+        ticker_id: (float(close), close_date)
+        for ticker_id, close, close_date in rows
+        if _finite(close)
+    }
+
+
+async def _earnings_values(
+    db: AsyncSession,
+    ticker_ids: list[int],
+    today: date,
+) -> tuple[dict[int, float], dict[int, date]]:
+    rows = (
+        await db.execute(
+            select(EarningsEvent)
+            .where(EarningsEvent.ticker_id.in_(ticker_ids))
+            .order_by(EarningsEvent.ticker_id, EarningsEvent.announce_date.desc())
+        )
+    ).scalars()
+    surprises: dict[int, float] = {}
+    upcoming: dict[int, date] = {}
+    for row in rows:
+        if row.announce_date >= today:
+            current = upcoming.get(row.ticker_id)
+            if current is None or row.announce_date < current:
+                upcoming[row.ticker_id] = row.announce_date
+            continue
+        if row.ticker_id in surprises:
+            continue
+        surprise = _surprise(row.eps_estimate, row.eps_actual)
+        if surprise is not None:
+            surprises[row.ticker_id] = surprise
+    return surprises, upcoming
+
+
+def _availability_metadata(
+    *,
+    derived: deriv.DerivedFundamentals | None,
+    price: float | None,
+    pe_ratio: float | None,
+    revenue_growth: float | None,
+    earnings_surprise: float | None,
+    market_cap: float | None,
+    next_earnings_date: date | None,
+) -> dict[str, str]:
+    metadata: dict[str, str] = {}
+
+    if pe_ratio is not None:
+        metadata["source_pe_ratio"] = "sec_facts+ohlcv_records"
+    elif derived is None or derived.latest_period_end is None:
+        metadata["pe_ratio"] = "no SEC fundamental snapshots"
+    elif not _finite(price) or price <= 0:
+        metadata["pe_ratio"] = "no usable PostgreSQL close"
+    elif derived.ttm_diluted_eps_caveat:
+        metadata["pe_ratio"] = derived.ttm_diluted_eps_caveat
+    else:
+        metadata["pe_ratio"] = "no positive SEC-derived TTM diluted EPS"
+
+    if revenue_growth is not None:
+        metadata["source_revenue_growth"] = "sec_facts"
+    else:
+        metadata["revenue_growth"] = "SEC-derived TTM revenue growth unavailable"
+
+    if earnings_surprise is not None:
+        metadata["source_earnings_surprise"] = "dolt_earnings"
+    else:
+        metadata["earnings_surprise"] = (
+            "no completed earnings event with actual and nonzero estimate"
+        )
+
+    if market_cap is not None:
+        metadata["source_market_cap"] = "sec_facts+ohlcv_records"
+        if derived is not None and derived.shares_outstanding_estimated:
+            metadata["market_cap_estimated"] = (
+                "shares use the SEC weighted-average diluted fallback"
+            )
+    elif derived is None or derived.latest_period_end is None:
+        metadata["market_cap"] = "no SEC fundamental snapshots"
+    elif not _finite(price) or price <= 0:
+        metadata["market_cap"] = "no usable PostgreSQL close"
+    else:
+        metadata["market_cap"] = "SEC-derived shares outstanding unavailable"
+
+    if next_earnings_date is not None:
+        metadata["source_next_earnings_date"] = "dolt_earnings"
+    else:
+        metadata["next_earnings_date"] = "no upcoming earnings event"
+    return metadata
+
+
+def _surprise(
+    estimate: float | None,
+    actual: float | None,
+) -> float | None:
+    if not _finite(estimate) or not _finite(actual) or estimate == 0:
+        return None
+    return (float(actual) - float(estimate)) / abs(float(estimate)) * 100.0
+
+
+def _pe(price: float | None, ttm_eps: float | None) -> float | None:
+    if (
+        not _finite(price)
+        or price <= 0
+        or not _finite(ttm_eps)
+        or ttm_eps <= 0
+    ):
+        return None
+    return float(price) / float(ttm_eps)
+
+
+def _market_cap(
+    price: float | None,
+    shares_outstanding: float | None,
+) -> float | None:
+    if (
+        not _finite(price)
+        or price <= 0
+        or not _finite(shares_outstanding)
+        or shares_outstanding <= 0
+    ):
+        return None
+    return float(price) * float(shares_outstanding)
+
+
+def _finite(value: Any) -> bool:
+    return (
+        isinstance(value, (int, float))
+        and not isinstance(value, bool)
+        and math.isfinite(value)
+    )

app/services/fundamentals_derivation.py

@@ -0,0 +1,396 @@
+"""Pure read-time derivation of fundamental metrics from stored snapshots.
+
+`fundamental_snapshots` stores one immutable row per accession with **cumulative
+YTD** duration facts and period-end balance-sheet instants (A3). This module
+derives everything the UI/API shows — discrete quarters, Q4, TTM, YoY growth,
+margins, leverage, dilution, and the quarter tape — at read time, per the plan's
+schema decision. No I/O, no DB: it takes an issuer's snapshot rows (ORM rows or
+any objects with the same attributes) and returns structured metrics.
+
+Rules:
+- **Amendment selection:** for each (fiscal_year, fiscal_period), the newest
+  `accepted_at` wins **per field**, falling back to the newest row that actually
+  reports one. A partial amendment (a 10-K/A adding Part III carries no financial
+  facts) must not blank the period.
+- **Discrete quarter** = YTD(Qn) − YTD(Qn−1); Q1 = YTD(Q1); **Q4 = YTD(FY) −
+  YTD(Q3)**. Any missing period → the derived value is null, never partial.
+- **TTM** = sum of the trailing four discrete quarters ending at a period.
+- Units follow app convention: percentages are percentage points (21.0 = 21%),
+  net-debt/EBITDA is a multiple, net debt is dollars.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from datetime import date
+from types import SimpleNamespace
+from typing import Any, Iterable
+
+_FP_TO_Q = {"Q1": 1, "Q2": 2, "Q3": 3, "FY": 4}
+_Q_TO_FP = {1: "Q1", 2: "Q2", 3: "Q3", 4: "FY"}
+_PREV_FP = {"Q2": "Q1", "Q3": "Q2", "FY": "Q3"}
+TAPE_LEN = 4  # quarter-tape length
+SPLIT_SUSPECT_SHARE_CHANGE_PCT = 25.0
+SPLIT_SENSITIVE_CAVEAT = (
+    "Not comparable: share count changed at least 25%; possible split or "
+    "corporate action."
+)
+
+# Duration (flow) fields differenced from YTD into discrete quarters + summed to TTM.
+_FLOW_FIELDS = (
+    "revenue", "net_income", "operating_income", "diluted_eps", "cfo", "capex",
+    "depreciation_amortization",
+)
+# Reported facts resolved independently across a period's accessions (see
+# _merge_amendments); period identity/provenance is taken from the newest one.
+_MERGED_FIELDS = (
+    *_FLOW_FIELDS,
+    "cash_and_st_investments", "total_debt", "shares_outstanding",
+    "shares_outstanding_date", "weighted_avg_diluted_shares",
+    # period_start is set alongside revenue by the parser, so it follows the same
+    # fallback: a bare amendment reports neither and must not blank it.
+    "period_start",
+)
+_CARRIED_FIELDS = (
+    "fiscal_year", "fiscal_period", "period_end", "filed_date",
+    "accepted_at", "form", "accession", "cik",
+)
+
+
+@dataclass
+class MetricPoint:
+    period_end: date
+    value: float | None
+
+
+@dataclass
+class MetricSeries:
+    value: float | None = None
+    history: list[MetricPoint] = field(default_factory=list)  # oldest -> newest, <= TAPE_LEN
+    period_end: date | None = None
+    filed_date: date | None = None
+    caveat: str | None = None
+
+
+@dataclass
+class DerivedFundamentals:
+    metrics: dict[str, MetricSeries] = field(default_factory=dict)
+    # request-time valuation inputs (ratios are computed in the API with price)
+    ttm_diluted_eps: float | None = None
+    # Set when ttm_diluted_eps was suppressed rather than simply unavailable.
+    ttm_diluted_eps_caveat: str | None = None
+    ttm_fcf: float | None = None
+    shares_outstanding: float | None = None
+    # True when shares_outstanding came from the weighted-average diluted count
+    # because the point-in-time cover-page count was absent (always so for
+    # multi-class issuers). Consumers must label anything derived from it as
+    # estimated — it is a period average, not a point-in-time count.
+    shares_outstanding_estimated: bool = False
+    latest_period_end: date | None = None
+    latest_filed_date: date | None = None
+
+
+def _prev_q(fy: int, q: int) -> tuple[int, int]:
+    return (fy, q - 1) if q > 1 else (fy - 1, 4)
+
+
+def derive(snapshots: Iterable[Any]) -> DerivedFundamentals:
+    selected = _select_latest_per_period(snapshots)
+    result = DerivedFundamentals()
+    if not selected:
+        return result
+
+    # Discrete quarter values per flow field: {field: {(fy, q): value}}.
+    discrete = {f: _discrete_quarters(selected, f) for f in _FLOW_FIELDS}
+    quarters = _ordered_quarters(selected)  # chronological (fy, q) with a row
+    latest = quarters[-1]
+    latest_row = selected[(latest[0], _Q_TO_FP[latest[1]])]
+
+    result.latest_period_end = latest_row.period_end
+    result.latest_filed_date = latest_row.filed_date
+    result.shares_outstanding = getattr(latest_row, "shares_outstanding", None)
+    if result.shares_outstanding is None:
+        # Multi-class issuers (META, CMCSA, BRK-B, CHTR, FOXA, NWSA, LEN) report
+        # the cover-page count per class, which is dimensional and so absent from
+        # companyfacts — leaving market cap and FCF yield silently unavailable for
+        # some of the largest names. The weighted-average diluted count is always
+        # present and within ~0.6% of the true count where both exist, so fall
+        # back to it and mark the result estimated rather than show nothing.
+        result.shares_outstanding = getattr(latest_row, "weighted_avg_diluted_shares", None)
+        result.shares_outstanding_estimated = result.shares_outstanding is not None
+    result.ttm_diluted_eps = _ttm(discrete["diluted_eps"], *latest)
+    ttm_cfo = _ttm(discrete["cfo"], *latest)
+    ttm_capex = _ttm(discrete["capex"], *latest)
+    result.ttm_fcf = None if ttm_cfo is None or ttm_capex is None else ttm_cfo - ttm_capex
+
+    # tape = the CONSECUTIVE run of up to TAPE_LEN quarters ending at the latest,
+    # stopping at a gap — so trend text never compares non-adjacent periods.
+    tape = _consecutive_suffix(quarters, TAPE_LEN)
+    result.metrics = {
+        "revenue_growth_yoy": _yoy_growth_series(discrete["revenue"], selected, tape),
+        "eps_growth_yoy": _yoy_growth_series(discrete["diluted_eps"], selected, tape),
+        "operating_margin": _margin_series(discrete["operating_income"], discrete["revenue"], selected, tape),
+        "fcf_margin": _fcf_margin_series(discrete, selected, tape),
+        "net_debt": _instant_series(selected, tape, _net_debt),
+        "net_debt_to_ebitda": _leverage_series(selected, discrete, tape),
+        "share_count_change_yoy": _share_change_series(selected, tape),
+    }
+    # TTM EPS sums four quarters of *per-share* values, so a split inside that
+    # window mixes pre- and post-split units — the same distortion the guard
+    # already catches for the series, and the one that produced BKNG's P/E of
+    # 1.10. Left unguarded it does not merely mislead: a nonsense-low P/E clamps
+    # to a perfect 100 fundamental sub-score, so it must null out like the rest.
+    if _guard_split_sensitive_metrics(result.metrics):
+        result.ttm_diluted_eps = None
+        result.ttm_diluted_eps_caveat = SPLIT_SENSITIVE_CAVEAT
+    for series in result.metrics.values():
+        series.period_end = latest_row.period_end
+        series.filed_date = latest_row.filed_date
+    return result
+
+
+# -- period selection --------------------------------------------------------
+
+def _select_latest_per_period(snapshots: Iterable[Any]) -> dict[tuple[int, str], Any]:
+    grouped: dict[tuple[int, str], list[Any]] = {}
+    for row in snapshots:
+        fp = getattr(row, "fiscal_period", None)
+        fy = getattr(row, "fiscal_year", None)
+        if fp not in _FP_TO_Q or fy is None:
+            continue
+        grouped.setdefault((fy, fp), []).append(row)
+    return {key: _merge_amendments(rows) for key, rows in grouped.items()}
+
+
+def _merge_amendments(rows: list[Any]) -> Any:
+    """Resolve one period from its accessions: newest wins, per field.
+
+    Amendments are frequently partial — a 10-K/A filed only to add Part III
+    reports no financial facts at all. Taking the newest accession wholesale
+    would blank every field it omits and null the period downstream (and with
+    it TTM and YoY, which need an unbroken quarter chain), so each field falls
+    back to the newest accession that actually reports it.
+
+    Only rows sharing the newest row's ``period_end`` are merged. A same-key row
+    covering a *different* period is a mislabelled filing, not an amendment, and
+    blending the two would silently mix fiscal years.
+    """
+    if len(rows) == 1:
+        return rows[0]
+    ordered = sorted(rows, key=_amendment_order, reverse=True)  # newest first
+    newest = ordered[0]
+    same_period = [
+        row
+        for row in ordered
+        if getattr(row, "period_end", None) == getattr(newest, "period_end", None)
+    ]
+    if len(same_period) == 1:
+        return newest
+    merged = SimpleNamespace(**{name: getattr(newest, name, None) for name in _CARRIED_FIELDS})
+    for name in _MERGED_FIELDS:
+        merged_value = None
+        for row in same_period:  # newest first
+            value = getattr(row, name, None)
+            if value is not None:
+                merged_value = value
+                break
+        setattr(merged, name, merged_value)
+    return merged
+
+
+def _amendment_order(row: Any) -> tuple[bool, Any]:
+    # (has-timestamp, timestamp) so a row without one sorts oldest instead of
+    # raising when compared against a row that has one.
+    accepted = _accepted(row)
+    return (accepted is not None, accepted)
+
+
+def _accepted(row: Any):
+    return getattr(row, "accepted_at", None) or getattr(row, "filed_date", None)
+
+
+def _ordered_quarters(selected: dict[tuple[int, str], Any]) -> list[tuple[int, int]]:
+    return sorted((fy, _FP_TO_Q[fp]) for (fy, fp) in selected)
+
+
+def _consecutive_suffix(quarters: list[tuple[int, int]], n: int) -> list[tuple[int, int]]:
+    """The run of up to n quarters ending at the latest, walking back only through
+    adjacent periods (stop at the first gap). Returned oldest -> newest."""
+    if not quarters:
+        return []
+    present = set(quarters)
+    run = [quarters[-1]]
+    cur = quarters[-1]
+    while len(run) < n:
+        prev = _prev_q(*cur)
+        if prev not in present:
+            break
+        run.append(prev)
+        cur = prev
+    run.reverse()
+    return run
+
+
+# -- discrete + TTM ----------------------------------------------------------
+
+def _discrete_quarters(selected: dict[tuple[int, str], Any], field_name: str) -> dict[tuple[int, int], float]:
+    out: dict[tuple[int, int], float] = {}
+    for (fy, fp), row in selected.items():
+        val = _discrete_value(selected, fy, fp, field_name)
+        if val is not None:
+            out[(fy, _FP_TO_Q[fp])] = val
+    return out
+
+
+def _discrete_value(selected, fy: int, fp: str, field_name: str) -> float | None:
+    cur = getattr(selected[(fy, fp)], field_name, None)
+    if cur is None:
+        return None
+    if fp == "Q1":
+        return cur
+    prev = selected.get((fy, _PREV_FP[fp]))
+    prev_val = getattr(prev, field_name, None) if prev is not None else None
+    if prev_val is None:
+        return None
+    return cur - prev_val
+
+
+def _ttm(dq: dict[tuple[int, int], float], fy: int, q: int) -> float | None:
+    keys = [(fy, q)]
+    k = (fy, q)
+    for _ in range(3):
+        k = _prev_q(*k)
+        keys.append(k)
+    vals = [dq.get(kk) for kk in keys]
+    if any(v is None for v in vals):
+        return None
+    return sum(vals)
+
+
+def _pct_change(cur: float | None, prior: float | None) -> float | None:
+    # A non-positive prior makes a YoY % meaningless (e.g. loss->profit), so null it.
+    if cur is None or prior is None or prior <= 0:
+        return None
+    return (cur / prior - 1.0) * 100.0
+
+
+# -- per-metric series (value at latest + tape history) ----------------------
+
+def _period_end(selected, fy: int, q: int) -> date | None:
+    row = selected.get((fy, _Q_TO_FP[q]))
+    return row.period_end if row is not None else None
+
+
+def _yoy_growth_series(dq, selected, tape) -> MetricSeries:
+    pts = []
+    for (fy, q) in tape:
+        cur, prior = _ttm(dq, fy, q), _ttm(dq, fy - 1, q)
+        pts.append(MetricPoint(_period_end(selected, fy, q), _pct_change(cur, prior)))
+    return _series(pts)
+
+
+def _margin_series(num_dq, den_dq, selected, tape) -> MetricSeries:
+    pts = []
+    for (fy, q) in tape:
+        num, den = _ttm(num_dq, fy, q), _ttm(den_dq, fy, q)
+        val = None if num is None or not den else num / den * 100.0
+        pts.append(MetricPoint(_period_end(selected, fy, q), val))
+    return _series(pts)
+
+
+def _fcf_margin_series(discrete, selected, tape) -> MetricSeries:
+    pts = []
+    for (fy, q) in tape:
+        cfo, capex, rev = _ttm(discrete["cfo"], fy, q), _ttm(discrete["capex"], fy, q), _ttm(discrete["revenue"], fy, q)
+        val = None if cfo is None or capex is None or not rev else (cfo - capex) / rev * 100.0
+        pts.append(MetricPoint(_period_end(selected, fy, q), val))
+    return _series(pts)
+
+
+def _instant_series(selected, tape, fn) -> MetricSeries:
+    pts = [MetricPoint(_period_end(selected, fy, q), fn(selected.get((fy, _Q_TO_FP[q])))) for (fy, q) in tape]
+    return _series(pts)
+
+
+def _leverage_series(selected, discrete, tape) -> MetricSeries:
+    pts = []
+    for (fy, q) in tape:
+        row = selected.get((fy, _Q_TO_FP[q]))
+        nd = _net_debt(row)
+        op, da = _ttm(discrete["operating_income"], fy, q), _ttm(discrete["depreciation_amortization"], fy, q)
+        ebitda = None if op is None or da is None else op + da
+        # Null when EBITDA <= 0: a negative denominator would flip polarity and a
+        # "lower is better" read would rank a distressed issuer as favorable.
+        val = None if nd is None or ebitda is None or ebitda <= 0 else nd / ebitda
+        pts.append(MetricPoint(_period_end(selected, fy, q), val))
+    return _series(pts)
+
+
+def _share_change_series(selected, tape) -> MetricSeries:
+    pts = []
+    for (fy, q) in tape:
+        cur = _shares(selected.get((fy, _Q_TO_FP[q])))
+        prior = _shares(selected.get((fy - 1, _Q_TO_FP[q])))
+        pts.append(MetricPoint(_period_end(selected, fy, q), _pct_change(cur, prior)))
+    return _series(pts)
+
+
+def _guard_split_sensitive_metrics(metrics: dict[str, MetricSeries]) -> bool:
+    """Suppress historical comparisons likely distorted by a corporate action.
+
+    Company Facts has no point-in-time split factors. A large YoY share-count
+    move can therefore make both the point-in-time share comparison and
+    per-share EPS growth non-comparable. Keep the raw facts in snapshots, but
+    expose nulls plus an explicit caveat in the user-facing derived series.
+
+    Returns True when the *latest* period is suspect, so callers can apply the
+    same suppression to per-share scalars derived from that window.
+    """
+    shares = metrics.get("share_count_change_yoy")
+    eps = metrics.get("eps_growth_yoy")
+    if shares is None or eps is None:
+        return False
+
+    suspect_periods = {
+        point.period_end
+        for point in shares.history
+        if point.value is not None
+        and abs(point.value) >= SPLIT_SUSPECT_SHARE_CHANGE_PCT
+    }
+    if not suspect_periods:
+        return False
+
+    latest_suspect = False
+    for series in (shares, eps):
+        latest_guarded = bool(
+            series.history and series.history[-1].period_end in suspect_periods
+        )
+        latest_suspect = latest_suspect or latest_guarded
+        for point in series.history:
+            if point.period_end in suspect_periods:
+                point.value = None
+        series.value = series.history[-1].value if series.history else None
+        if latest_guarded:
+            series.caveat = SPLIT_SENSITIVE_CAVEAT
+    return latest_suspect
+
+
+def _net_debt(row: Any) -> float | None:
+    if row is None:
+        return None
+    cash = getattr(row, "cash_and_st_investments", None)
+    debt = getattr(row, "total_debt", None)
+    # Require BOTH components — treating a missing side as zero would produce a
+    # partial, misleading value.
+    if cash is None or debt is None:
+        return None
+    return debt - cash  # positive = net debt
+
+
+def _shares(row: Any) -> float | None:
+    return getattr(row, "shares_outstanding", None) if row is not None else None
+
+
+def _series(points: list[MetricPoint]) -> MetricSeries:
+    value = points[-1].value if points else None
+    return MetricSeries(value=value, history=points)

app/services/fundamentals_parity_service.py

@@ -0,0 +1,498 @@
+"""Read-only A5 comparison of legacy and SEC/Dolt fundamental inputs.
+
+The report deliberately does not write ``fundamental_data`` or score tables.
+It reconstructs the current legacy and candidate fundamental scores, projects
+their composite-score/rank effect with the active weights, and archives a
+timestamped JSON + CSV bundle for explicit human approval.
+"""
+
+from __future__ import annotations
+
+import csv
+import io
+import json
+import math
+import os
+import statistics
+from datetime import date, datetime, timezone
+from pathlib import Path
+from typing import Any, Iterable
+from zoneinfo import ZoneInfo
+
+from sqlalchemy import select, text
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.models.data_import_run import DataImportRun
+from app.models.fundamental import FundamentalData
+from app.services import fundamentals_candidate_service as candidate_service
+
+REPORT_VERSION = 1
+APPROVAL_STATUS = "pending_explicit_approval"
+FIELD_KEYS = ("pe_ratio", "revenue_growth", "earnings_surprise")
+MIN_SCORE_METRICS = 2
+
+# Materiality is a review aid, never an automatic cutover verdict. Definition
+# changes remain visible even when a delta falls inside these bands.
+FIELD_TOLERANCES = {
+    "pe_ratio": {"absolute": 1.0, "relative_pct": 10.0},
+    "revenue_growth": {"absolute": 2.0, "relative_pct": None},
+    "earnings_surprise": {"absolute": 2.0, "relative_pct": None},
+}
+DEFINITION_NOTES = {
+    "pe_ratio": (
+        "Legacy provider P/E convention versus latest close divided by "
+        "SEC-derived TTM diluted EPS."
+    ),
+    "revenue_growth": (
+        "Legacy provider growth convention versus SEC-derived TTM revenue YoY."
+    ),
+    "earnings_surprise": (
+        "Legacy provider latest surprise versus latest completed Dolt earnings "
+        "event with actual and estimate."
+    ),
+}
+
+
+def fundamental_score(
+    pe_ratio: float | None,
+    revenue_growth: float | None,
+    earnings_surprise: float | None,
+) -> float | None:
+    """Match the production fundamental-dimension formula without persistence."""
+    scores: list[float] = []
+    if _finite(pe_ratio) and pe_ratio > 0:
+        scores.append(max(0.0, min(100.0, 100.0 - (pe_ratio - 15.0) * (100.0 / 30.0))))
+    if _finite(revenue_growth):
+        scores.append(max(0.0, min(100.0, 50.0 + revenue_growth * 2.5)))
+    if _finite(earnings_surprise):
+        scores.append(max(0.0, min(100.0, 50.0 + earnings_surprise * 5.0)))
+    return sum(scores) / len(scores) if len(scores) >= MIN_SCORE_METRICS else None
+
+
+async def build_report(
+    db: AsyncSession,
+    *,
+    generated_at: datetime | None = None,
+    today: date | None = None,
+) -> dict[str, Any]:
+    """Build a point-in-time parity report from one database session."""
+    generated_at = generated_at or datetime.now(timezone.utc)
+    today = today or datetime.now(ZoneInfo("America/New_York")).date()
+
+    # A report must not mix rows from before and after a concurrent import
+    # promotion. The scheduled job provides a fresh session, so establish the
+    # production snapshot before its first query and have Postgres enforce the
+    # no-write contract as well. SQLite tests retain their normal transaction.
+    if db.get_bind().dialect.name == "postgresql":
+        connection = await db.connection(
+            execution_options={"isolation_level": "REPEATABLE READ"}
+        )
+        await connection.execute(text("SET TRANSACTION READ ONLY"))
+
+    candidates = await candidate_service.build_candidates(db, today=today)
+    ticker_ids = [candidate.ticker_id for candidate in candidates]
+    legacy_by_ticker = await _legacy_values(db, ticker_ids)
+    source_runs = await _source_runs(db)
+
+    rows: list[dict[str, Any]] = []
+    for candidate in candidates:
+        legacy = legacy_by_ticker.get(candidate.ticker_id)
+        candidate_values = {
+            "pe_ratio": candidate.pe_ratio,
+            "revenue_growth": candidate.revenue_growth,
+            "earnings_surprise": candidate.earnings_surprise,
+        }
+        legacy_values = {
+            "pe_ratio": legacy.pe_ratio if legacy else None,
+            "revenue_growth": legacy.revenue_growth if legacy else None,
+            "earnings_surprise": legacy.earnings_surprise if legacy else None,
+        }
+        fields = {
+            key: _field_comparison(key, legacy_values[key], candidate_values[key])
+            for key in FIELD_KEYS
+        }
+        legacy_score = fundamental_score(**legacy_values)
+        candidate_score = fundamental_score(**candidate_values)
+        rows.append(
+            {
+                "symbol": candidate.symbol,
+                "cik": candidate.cik,
+                "legacy_fetched_at": _iso(legacy.fetched_at) if legacy else None,
+                "price_date": _iso(candidate.price_date),
+                "fields": fields,
+                "scores": {
+                    "legacy_fundamental": _round(legacy_score),
+                    "candidate_fundamental": _round(candidate_score),
+                    "fundamental_delta": _delta(legacy_score, candidate_score),
+                    "legacy_fundamental_rank": None,
+                    "candidate_fundamental_rank": None,
+                    "fundamental_rank_change": None,
+                },
+            }
+        )
+
+    _attach_ranks(rows, "legacy_fundamental", "legacy_fundamental_rank")
+    _attach_ranks(rows, "candidate_fundamental", "candidate_fundamental_rank")
+    for row in rows:
+        scores = row["scores"]
+        scores["fundamental_rank_change"] = _rank_change(
+            scores["legacy_fundamental_rank"], scores["candidate_fundamental_rank"]
+        )
+
+    return {
+        "report_version": REPORT_VERSION,
+        "generated_at": generated_at.isoformat(),
+        "as_of_date": today.isoformat(),
+        "approval_status": APPROVAL_STATUS,
+        "read_only": True,
+        "fundamental_score_formula": (
+            "Equal-weighted mean of 2+ available sub-scores: P/E = "
+            "clamp(100-(pe-15)*(100/30)); revenue growth = "
+            "clamp(50+growth*2.5); earnings surprise = "
+            "clamp(50+surprise*5)."
+        ),
+        "source_runs": source_runs,
+        "definition_notes": DEFINITION_NOTES,
+        "materiality_notes": {
+            "fields": FIELD_TOLERANCES,
+            "fundamental_score_absolute": 5.0,
+            "automatic_cutover": False,
+        },
+        "summary": _summary(rows),
+        "rows": rows,
+    }
+
+
+def store_report(report: dict[str, Any], report_dir: str | Path) -> dict[str, str]:
+    """Atomically archive JSON/CSV artifacts and update the latest manifest."""
+    directory = Path(report_dir).expanduser().resolve()
+    directory.mkdir(parents=True, exist_ok=True)
+    stamp = _artifact_stamp(report["generated_at"])
+    json_name = f"fundamentals-parity-{stamp}.json"
+    csv_name = f"fundamentals-parity-{stamp}.csv"
+    json_path = directory / json_name
+    csv_path = directory / csv_name
+
+    _atomic_write(json_path, json.dumps(report, indent=2, sort_keys=True) + "\n")
+    _atomic_write(csv_path, report_csv(report))
+    manifest = {
+        "generated_at": report["generated_at"],
+        "json_file": json_name,
+        "csv_file": csv_name,
+    }
+    _atomic_write(
+        directory / "latest.json",
+        json.dumps(manifest, indent=2, sort_keys=True) + "\n",
+    )
+    return {
+        "json": str(json_path),
+        "csv": str(csv_path),
+        "manifest": str(directory / "latest.json"),
+    }
+
+
+async def generate_and_store(
+    db: AsyncSession,
+    report_dir: str | Path,
+    *,
+    generated_at: datetime | None = None,
+    today: date | None = None,
+) -> tuple[dict[str, Any], dict[str, str]]:
+    report = await build_report(db, generated_at=generated_at, today=today)
+    return report, store_report(report, report_dir)
+
+
+def load_latest(report_dir: str | Path) -> dict[str, Any] | None:
+    manifest = _load_manifest(report_dir)
+    if manifest is None:
+        return None
+    try:
+        path = _manifest_artifact(report_dir, manifest, "json_file")
+        loaded = json.loads(path.read_text(encoding="utf-8"))
+    except (OSError, json.JSONDecodeError, TypeError, ValueError):
+        return None
+    return loaded if isinstance(loaded, dict) else None
+
+
+def load_latest_csv(report_dir: str | Path) -> tuple[str, str] | None:
+    return _load_latest_text_artifact(report_dir, "csv_file")
+
+
+def load_latest_json(report_dir: str | Path) -> tuple[str, str] | None:
+    return _load_latest_text_artifact(report_dir, "json_file")
+
+
+def _load_latest_text_artifact(
+    report_dir: str | Path, manifest_key: str
+) -> tuple[str, str] | None:
+    manifest = _load_manifest(report_dir)
+    if manifest is None:
+        return None
+    try:
+        path = _manifest_artifact(report_dir, manifest, manifest_key)
+        return path.name, path.read_text(encoding="utf-8")
+    except (OSError, TypeError, ValueError):
+        return None
+
+
+def report_csv(report: dict[str, Any]) -> str:
+    output = io.StringIO(newline="")
+    columns = [
+        "symbol",
+        "cik",
+        "legacy_fetched_at",
+        "price_date",
+        *(
+            f"{field}_{suffix}"
+            for field in FIELD_KEYS
+            for suffix in ("legacy", "candidate", "absolute_delta", "relative_delta_pct", "material")
+        ),
+        "legacy_fundamental",
+        "candidate_fundamental",
+        "fundamental_delta",
+        "legacy_fundamental_rank",
+        "candidate_fundamental_rank",
+        "fundamental_rank_change",
+    ]
+    writer = csv.DictWriter(output, fieldnames=columns)
+    writer.writeheader()
+    for row in report.get("rows", []):
+        flat = {
+            "symbol": row["symbol"],
+            "cik": row.get("cik"),
+            "legacy_fetched_at": row.get("legacy_fetched_at"),
+            "price_date": row.get("price_date"),
+            **row["scores"],
+        }
+        for field in FIELD_KEYS:
+            comparison = row["fields"][field]
+            for suffix in (
+                "legacy",
+                "candidate",
+                "absolute_delta",
+                "relative_delta_pct",
+                "material",
+            ):
+                flat[f"{field}_{suffix}"] = comparison.get(suffix)
+        writer.writerow(flat)
+    return output.getvalue()
+
+
+async def _legacy_values(
+    db: AsyncSession, ticker_ids: list[int]
+) -> dict[int, FundamentalData]:
+    if not ticker_ids:
+        return {}
+    rows = (
+        await db.execute(
+            select(FundamentalData).where(FundamentalData.ticker_id.in_(ticker_ids))
+        )
+    ).scalars()
+    return {row.ticker_id: row for row in rows}
+
+
+async def _source_runs(db: AsyncSession) -> dict[str, dict[str, Any] | None]:
+    sources = ("sec_facts", "dolt_earnings")
+    rows = (
+        await db.execute(
+            select(DataImportRun)
+            .where(
+                DataImportRun.source.in_(sources),
+                DataImportRun.status.in_(("promoted", "no_op")),
+            )
+            .order_by(DataImportRun.id.desc())
+        )
+    ).scalars()
+    latest: dict[str, dict[str, Any] | None] = {source: None for source in sources}
+    for row in rows:
+        if latest[row.source] is None:
+            latest[row.source] = {
+                "run_id": row.id,
+                "status": row.status,
+                "revision": row.revision,
+                "source_max_date": _iso(row.source_max_date),
+                "completed_at": _iso(row.completed_at),
+            }
+    return latest
+
+
+def _field_comparison(
+    key: str, legacy: float | None, candidate: float | None
+) -> dict[str, Any]:
+    legacy = float(legacy) if _finite(legacy) else None
+    candidate = float(candidate) if _finite(candidate) else None
+    absolute = _delta(legacy, candidate)
+    relative = (
+        None
+        if absolute is None or legacy in (None, 0)
+        else round(absolute / abs(legacy) * 100.0, 4)
+    )
+    tolerance = FIELD_TOLERANCES[key]
+    material = False
+    if absolute is not None:
+        material = abs(absolute) > tolerance["absolute"]
+        relative_limit = tolerance["relative_pct"]
+        if relative_limit is not None:
+            material = material and relative is not None and abs(relative) > relative_limit
+    return {
+        "legacy": _round(legacy),
+        "candidate": _round(candidate),
+        "absolute_delta": absolute,
+        "relative_delta_pct": relative,
+        "material": material,
+        "definition_changed": True,
+    }
+
+
+def _attach_ranks(rows: list[dict[str, Any]], value_key: str, rank_key: str) -> None:
+    values = [
+        row["scores"][value_key]
+        for row in rows
+        if _finite(row["scores"][value_key])
+    ]
+    for row in rows:
+        value = row["scores"][value_key]
+        row["scores"][rank_key] = (
+            1 + sum(other > value for other in values) if _finite(value) else None
+        )
+
+
+def _summary(rows: list[dict[str, Any]]) -> dict[str, Any]:
+    field_stats = {}
+    for key in FIELD_KEYS:
+        comparisons = [row["fields"][key] for row in rows]
+        deltas = [
+            abs(item["absolute_delta"])
+            for item in comparisons
+            if item["absolute_delta"] is not None
+        ]
+        field_stats[key] = {
+            "legacy_available": sum(item["legacy"] is not None for item in comparisons),
+            "candidate_available": sum(
+                item["candidate"] is not None for item in comparisons
+            ),
+            "both_available": len(deltas),
+            "material_differences": sum(item["material"] for item in comparisons),
+            "median_absolute_delta": _round(statistics.median(deltas) if deltas else None),
+            "p95_absolute_delta": _round(_percentile(deltas, 0.95)),
+            "max_absolute_delta": _round(max(deltas) if deltas else None),
+        }
+
+    fundamental_deltas = _score_deltas(rows, "fundamental_delta")
+    changed_rows = sorted(
+        (
+            {
+                "symbol": row["symbol"],
+                "fundamental_delta": row["scores"]["fundamental_delta"],
+                "fundamental_rank_change": row["scores"]["fundamental_rank_change"],
+            }
+            for row in rows
+            if row["scores"]["fundamental_delta"] is not None
+        ),
+        key=lambda item: (
+            abs(item["fundamental_delta"] or 0),
+        ),
+        reverse=True,
+    )[:20]
+    return {
+        "universe_count": len(rows),
+        "legacy_fundamental_score_available": _count_score(
+            rows, "legacy_fundamental"
+        ),
+        "candidate_fundamental_score_available": _count_score(
+            rows, "candidate_fundamental"
+        ),
+        "fundamental_scores_compared": len(fundamental_deltas),
+        "fundamental_score_material_changes": sum(
+            abs(delta) > 5.0 for delta in fundamental_deltas
+        ),
+        "fundamental_rank_changes": _rank_change_count(
+            rows, "fundamental_rank_change"
+        ),
+        "field_stats": field_stats,
+        "largest_changes": changed_rows,
+    }
+
+
+def _score_deltas(rows: Iterable[dict[str, Any]], key: str) -> list[float]:
+    return [
+        row["scores"][key]
+        for row in rows
+        if row["scores"][key] is not None
+    ]
+
+
+def _count_score(rows: Iterable[dict[str, Any]], key: str) -> int:
+    return sum(row["scores"][key] is not None for row in rows)
+
+
+def _rank_change_count(rows: Iterable[dict[str, Any]], key: str) -> int:
+    return sum(
+        row["scores"][key] not in (None, 0)
+        for row in rows
+    )
+
+
+def _rank_change(legacy: int | None, candidate: int | None) -> int | None:
+    # Positive means the candidate improved its rank.
+    return legacy - candidate if legacy is not None and candidate is not None else None
+
+
+def _delta(legacy: float | None, candidate: float | None) -> float | None:
+    if not _finite(legacy) or not _finite(candidate):
+        return None
+    return round(candidate - legacy, 4)
+
+
+def _round(value: float | None, digits: int = 4) -> float | None:
+    return round(float(value), digits) if _finite(value) else None
+
+
+def _percentile(values: list[float], quantile: float) -> float | None:
+    if not values:
+        return None
+    ordered = sorted(values)
+    index = max(0, math.ceil(quantile * len(ordered)) - 1)
+    return ordered[index]
+
+
+def _finite(value: Any) -> bool:
+    return (
+        isinstance(value, (int, float))
+        and not isinstance(value, bool)
+        and math.isfinite(value)
+    )
+
+
+def _iso(value: Any) -> str | None:
+    return value.isoformat() if value is not None else None
+
+
+def _artifact_stamp(raw: str) -> str:
+    parsed = datetime.fromisoformat(raw.replace("Z", "+00:00"))
+    return parsed.astimezone(timezone.utc).strftime("%Y%m%dT%H%M%S%fZ")
+
+
+def _atomic_write(path: Path, content: str) -> None:
+    temp = path.with_name(f".{path.name}.{os.getpid()}.tmp")
+    temp.write_text(content, encoding="utf-8", newline="")
+    os.replace(temp, path)
+
+
+def _load_manifest(report_dir: str | Path) -> dict[str, Any] | None:
+    path = Path(report_dir).expanduser().resolve() / "latest.json"
+    try:
+        loaded = json.loads(path.read_text(encoding="utf-8"))
+    except (OSError, json.JSONDecodeError, TypeError, ValueError):
+        return None
+    return loaded if isinstance(loaded, dict) else None
+
+
+def _manifest_artifact(
+    report_dir: str | Path, manifest: dict[str, Any], key: str
+) -> Path:
+    directory = Path(report_dir).expanduser().resolve()
+    name = Path(str(manifest.get(key, ""))).name
+    if not name:
+        raise ValueError(f"Latest parity manifest has no {key}")
+    return directory / name

app/services/fundamentals_peers.py

@@ -0,0 +1,107 @@
+"""Pure peer comparison for fundamentals (read-time).
+
+Peers are tracked-universe issuers sharing the **first two SIC digits**,
+deduplicated by CIK (GOOG/GOOGL are one issuer, one observation). This module is
+the pure statistics core: given a subject value and the peer group's values for a
+metric, it returns median + polarity-aware favorable percentile + peer_count, or
+None when there are fewer than the minimum valid peers (the caller then omits the
+industry object entirely rather than show a misleading comparison).
+
+Grouping (which issuers share a 2-digit SIC, CIK-dedup) is the API's job; this
+module only does the math. **Absolute net_debt is size-dependent and must not get
+a peer percentile** — leverage is compared via net_debt_to_ebitda.
+"""
+
+from __future__ import annotations
+
+import math
+import statistics
+from dataclasses import dataclass
+from typing import Any
+
+MIN_PEERS = 5
+
+
+def _finite(v: Any) -> bool:
+    """True for a finite number — excludes None, bool, NaN, ±inf (plan: null/invalid)."""
+    return isinstance(v, (int, float)) and not isinstance(v, bool) and math.isfinite(v)
+
+# Metric -> is a higher value more favorable? (Peer-eligible metrics only;
+# absolute net_debt is intentionally absent — size-dependent.)
+HIGHER_IS_BETTER: dict[str, bool] = {
+    "revenue_growth_yoy": True,
+    "eps_growth_yoy": True,
+    "operating_margin": True,
+    "fcf_margin": True,
+    "fcf_yield": True,
+    "net_debt_to_ebitda": False,   # lower leverage is better
+    "pe": False,                   # cheaper is better
+    "share_count_change_yoy": False,  # dilution is bad
+}
+
+
+@dataclass
+class PeerStat:
+    median: float
+    favorable_percentile: int  # 0-100, polarity-aware (higher = more favorable)
+    peer_count: int            # valid issuers in the group
+
+
+def peer_stat(
+    subject: float | None,
+    group_values: list[float | None],
+    *,
+    higher_is_better: bool,
+    min_peers: int = MIN_PEERS,
+) -> PeerStat | None:
+    """Median + favorable percentile for ``subject`` within its group.
+
+    ``group_values`` is every issuer's value for the metric (including the
+    subject), CIK-deduplicated by the caller. Null/invalid (non-finite) values are
+    excluded. Returns None when fewer than ``min_peers`` valid values exist, or
+    the subject is null/invalid.
+
+    The percentile is a **tie-aware rank against the other issuers** —
+    ``(worse + 0.5·tied) / (peers − 1)`` — so a whole group of equal values maps
+    to 50, not 100, and the median maps to 50.
+    """
+    valid = [v for v in group_values if _finite(v)]
+    if not _finite(subject) or len(valid) < min_peers:
+        return None
+    median = statistics.median(valid)
+
+    others = valid.copy()
+    try:
+        others.remove(subject)  # rank the subject against the OTHER issuers
+    except ValueError:
+        pass
+    denom = len(others)
+    if denom == 0:
+        return None
+    if higher_is_better:
+        worse = sum(1 for v in others if v < subject)
+    else:
+        worse = sum(1 for v in others if v > subject)
+    tied = sum(1 for v in others if v == subject)
+    percentile = round((worse + 0.5 * tied) / denom * 100)
+    return PeerStat(median=median, favorable_percentile=percentile, peer_count=len(valid))
+
+
+def peer_stat_for(
+    metric_key: str, subject: float | None, group_values: list[float | None], **kwargs
+) -> PeerStat | None:
+    """Convenience wrapper that looks up polarity by metric key. Returns None for
+    metrics not eligible for peer comparison (e.g. absolute net_debt)."""
+    if metric_key not in HIGHER_IS_BETTER:
+        return None
+    return peer_stat(
+        subject, group_values, higher_is_better=HIGHER_IS_BETTER[metric_key], **kwargs
+    )
+
+
+def two_digit_sic(sic: str | None) -> str | None:
+    """The 2-digit SIC prefix used for grouping, or None if unusable."""
+    if not sic:
+        return None
+    digits = str(sic).strip()
+    return digits[:2] if len(digits) >= 2 and digits[:2].isdigit() else None

app/services/fundamentals_quality_service.py

@@ -0,0 +1,164 @@
+"""Actionability gate for incomplete SEC fundamentals."""
+
+from __future__ import annotations
+
+import json
+from dataclasses import dataclass
+
+from sqlalchemy import exists, func, select
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.models.data_import_run import DataImportRun
+from app.models.fundamental_snapshot import FundamentalSnapshot
+from app.models.sec_filing_gap import SecFilingGap
+from app.models.ticker import Ticker
+from app.services import fundamental_data_refresh_service
+
+_SEC_FORMS = ("10-K", "10-Q", "10-K/A", "10-Q/A")
+
+
+@dataclass(frozen=True)
+class SetupQuality:
+    eligible: bool
+    code: str | None = None
+    message: str | None = None
+
+
+async def active_gaps(
+    db: AsyncSession,
+    ciks: set[str] | None = None,
+) -> list[SecFilingGap]:
+    """Unresolved gaps that have not been superseded by a later filing."""
+    matching_snapshot = exists().where(
+        FundamentalSnapshot.accession == SecFilingGap.accession
+    )
+    gap_date = func.coalesce(
+        SecFilingGap.index_date,
+        func.date(SecFilingGap.first_seen_at),
+    )
+    later_snapshot = exists().where(
+        FundamentalSnapshot.cik == SecFilingGap.cik,
+        FundamentalSnapshot.form.in_(_SEC_FORMS),
+        FundamentalSnapshot.filed_date > gap_date,
+    )
+    stmt = select(SecFilingGap).where(
+        ~matching_snapshot,
+        ~later_snapshot,
+    )
+    if ciks is not None:
+        if not ciks:
+            return []
+        stmt = stmt.where(SecFilingGap.cik.in_(ciks))
+    return list((await db.execute(stmt)).scalars().all())
+
+
+async def _latest_validation(db: AsyncSession) -> dict:
+    payload = (
+        await db.execute(
+            select(DataImportRun.validation_json)
+            .where(
+                DataImportRun.source == "sec_facts",
+                DataImportRun.validation_json.is_not(None),
+            )
+            .order_by(DataImportRun.id.desc())
+            .limit(1)
+        )
+    ).scalar_one_or_none()
+    if not payload:
+        return {}
+    try:
+        summary = json.loads(payload)
+    except (TypeError, ValueError):
+        return {}
+    return summary if isinstance(summary, dict) else {}
+
+
+async def blocked_reasons_by_cik(
+    db: AsyncSession,
+    ciks: set[str] | None = None,
+) -> dict[str, str]:
+    """Current SEC blocker code by CIK; no historical audit scan."""
+    if not await fundamental_data_refresh_service.is_enabled(db):
+        return {}
+    if ciks is not None and not ciks:
+        return {}
+
+    reasons = {
+        gap.cik: "sec_filing_gap" for gap in await active_gaps(db, ciks)
+    }
+    summary = await _latest_validation(db)
+
+    def wanted(cik: str) -> bool:
+        return ciks is None or cik in ciks
+
+    # New summaries carry the complete compact CIK set while the detailed lists
+    # stay capped for audit readability. Detailed entries supply the reason.
+    for cik in summary.get("setup_blocked_ciks") or []:
+        normalized = str(cik) if cik else ""
+        if normalized and wanted(normalized):
+            reasons.setdefault(normalized, "sec_filing_gap")
+    for item in summary.get("missing_xbrl") or []:
+        normalized = str(item.get("cik") or "")
+        if normalized and wanted(normalized):
+            reasons.setdefault(normalized, "sec_filing_gap")
+    for cik in summary.get("no_xbrl_ciks") or []:
+        normalized = str(cik) if cik else ""
+        if normalized and wanted(normalized):
+            reasons[normalized] = "no_xbrl_filings"
+    for item in summary.get("no_xbrl_filings") or []:
+        normalized = str(item.get("cik") or "")
+        if normalized and wanted(normalized):
+            reasons[normalized] = "no_xbrl_filings"
+    return reasons
+
+
+async def blocked_ciks(db: AsyncSession) -> set[str]:
+    return set(await blocked_reasons_by_cik(db))
+
+
+async def blocked_ticker_ids(db: AsyncSession) -> set[int]:
+    ciks = await blocked_ciks(db)
+    if not ciks:
+        return set()
+    rows = await db.execute(select(Ticker.id).where(Ticker.cik.in_(ciks)))
+    return {int(ticker_id) for ticker_id in rows.scalars()}
+
+
+async def ticker_quality(db: AsyncSession, symbol: str) -> SetupQuality:
+    ticker = (
+        await db.execute(
+            select(Ticker).where(Ticker.symbol == symbol.strip().upper())
+        )
+    ).scalar_one_or_none()
+    if ticker is None or not ticker.cik:
+        return SetupQuality(eligible=True)
+    reason = (await blocked_reasons_by_cik(db, {ticker.cik})).get(ticker.cik)
+    if reason == "no_xbrl_filings":
+        return SetupQuality(
+            eligible=False,
+            code=reason,
+            message=(
+                "No SEC 10-K/10-Q is available for this registrant, so new setups "
+                "are paused. New registrants clear automatically after their first "
+                "filing; a successor shell needs an SEC CIK override."
+            ),
+        )
+    if reason:
+        return SetupQuality(
+            eligible=False,
+            code=reason,
+            message=(
+                "A recent SEC filing is still being reconciled, so new setups are "
+                "paused. The scheduled fundamentals import retries it automatically."
+            ),
+        )
+    return SetupQuality(eligible=True)
+
+
+async def ticker_is_eligible(db: AsyncSession, ticker_id: int) -> bool:
+    cik = (
+        await db.execute(select(Ticker.cik).where(Ticker.id == ticker_id))
+    ).scalar_one_or_none()
+    if not cik:
+        return True
+    return cik not in await blocked_reasons_by_cik(db, {cik})

app/services/fundamentals_reads.py

@@ -0,0 +1,115 @@
+"""Deterministic text 'reads' for the fundamentals panel (pure, one rule set).
+
+The tape reads and the header sentence use identical outputs — no LLM, no new
+composite score. Thresholds are tunable named constants, not scattered literals
+(plan: ±2pp growth, ±1pp margins, ±1% dilution, 60/40 peer bands, ≥3 periods).
+
+Consumers pass metric series (value + dated history, from
+``fundamentals_derivation``) and peer percentiles; these functions return short
+strings or None (render "—", no read).
+"""
+
+from __future__ import annotations
+
+from statistics import mean
+from typing import Any
+
+MIN_PERIODS = 3
+GROWTH_ACCEL_PP = 2.0
+MARGIN_MOVE_PP = 1.0
+SHARE_DILUTION_PCT = 1.0
+PEER_FAVORABLE = 60
+PEER_ADVERSE = 40
+
+
+def _latest_run(history: list[Any]) -> list[float]:
+    """The consecutive non-null values ending at the latest point (oldest->newest).
+    A null latest, or an internal gap, truncates the run — so a read never reflects
+    a period whose displayed value is n/a."""
+    run: list[float] = []
+    for p in reversed(history):
+        if p.value is None:
+            break
+        run.append(p.value)
+    run.reverse()
+    return run
+
+
+def growth_read(history: list[Any]) -> str | None:
+    """Change in a YoY-growth series: latest − prior. Needs >= 3 consecutive
+    non-null values ending at the latest point."""
+    vals = _latest_run(history)
+    if len(vals) < MIN_PERIODS:
+        return None
+    delta = vals[-1] - vals[-2]
+    if delta >= GROWTH_ACCEL_PP:
+        return "accelerating"
+    if delta <= -GROWTH_ACCEL_PP:
+        return "decelerating"
+    return "steady"
+
+
+def margin_read(history: list[Any]) -> str | None:
+    """Latest margin vs the mean of prior periods (pp). Needs >= 3 consecutive
+    non-null values ending at the latest point."""
+    vals = _latest_run(history)
+    if len(vals) < MIN_PERIODS:
+        return None
+    delta = vals[-1] - mean(vals[:-1])
+    if delta >= MARGIN_MOVE_PP:
+        return "improving"
+    if delta <= -MARGIN_MOVE_PP:
+        return "deteriorating"
+    return "stable"
+
+
+def share_count_read(value: float | None) -> str | None:
+    """Share-count YoY %: >+1% dilution, <-1% buying back, else flat."""
+    if value is None:
+        return None
+    if value > SHARE_DILUTION_PCT:
+        return f"{value:.1f}% dilution"
+    if value < -SHARE_DILUTION_PCT:
+        return "buying back"
+    return "flat"
+
+
+def peer_read(metric_key: str, favorable_percentile: int | None) -> str | None:
+    """Peer-relative read for a metric, polarity already baked into the
+    percentile (higher = more favorable)."""
+    if favorable_percentile is None:
+        return None
+    if favorable_percentile >= PEER_FAVORABLE:
+        return _FAVORABLE.get(metric_key, "above peers")
+    if favorable_percentile <= PEER_ADVERSE:
+        return _ADVERSE.get(metric_key, "below peers")
+    return "in line"
+
+
+_FAVORABLE = {
+    "pe": "attractively valued",
+    "fcf_yield": "above peers",
+    "net_debt_to_ebitda": "conservative leverage",
+}
+_ADVERSE = {
+    "pe": "priced above peers",
+    "fcf_yield": "below peers",
+    "net_debt_to_ebitda": "elevated leverage",
+}
+
+
+def header_sentence(
+    growth: str | None, margin: str | None, valuation: str | None
+) -> str:
+    """Join the growth / margin / peer-valuation reads with ' · ', omitting
+    segments with no read. Segment sources are fixed by the caller (growth =
+    revenue-growth read, margin = operating-margin read, valuation = P/E peer
+    read falling back to FCF yield)."""
+    parts = []
+    if growth:
+        parts.append(f"growth {growth}")
+    if margin:
+        parts.append(f"margins {margin}")
+    if valuation:
+        parts.append(f"valuation {valuation}")
+    return " · ".join(parts)

app/services/indicator_service.py

@@ -28,6 +28,7 @@ MIN_BARS: dict[str, int] = {
     "atr": 15,
     "volume_profile": 20,
     "pivot_points": 5,
+    "fip_id": 253,  # 12-1 formation: need index i-252
 }
 
 DEFAULT_PERIODS: dict[str, int] = {
@@ -407,6 +408,88 @@ def compute_pivot_points(
     }
 
 
+# Path labels for display only. Calibrated on the ~505-name prod snapshot
+# (2026-07, n=502 with full history): empirical p25 ≈ −0.082, p75 ≈ +0.004,
+# mean ≈ −0.043. Paper-style |ID| ≳ 0.25 almost never appears in live equities
+# (only ~0.2% of names); real momentum winners cluster around −0.04…−0.12.
+# Thresholds are therefore ~quartile cutoffs, not ±0.25 textbook extremes.
+# Distribution is left-skewed (bullish sample → more "continuous" than "discrete"),
+# so the discrete band is not symmetric.
+FIP_PATH_CONTINUOUS_MAX = -0.08  # ~p25: smoother quartile
+FIP_PATH_DISCRETE_MIN = 0.00  # ~p75: less-continuous quartile
+
+
+def compute_fip_id(closes: list[float], as_of_index: int | None = None) -> dict[str, Any]:
+    """Da/Gurun/Warachka information discreteness over the 12-1 formation window.
+
+    Display / research context only — **not** used by the activation gate or
+    production rank. Same window as residual 12-1 momentum: cumulative return
+    from close[i-252] to close[i-21] (skip last month).
+
+        ID = sign(PRET) × (%neg − %pos)
+
+    Lower ID ⇒ smoother / more continuous path (for a winner: many small up days).
+    Higher ID ⇒ jumpy / discrete path (few large moves).
+
+    Zero-return days count in neither numerator but remain in the denominator
+    (paper definition). Quirk: a flat series with one big jump can still land
+    near zero ("mixed") because zeros dilute %pos/%neg — faithful to the paper
+    and to real equities (exact zero daily returns are rare). Synthetic jump
+    tests assert ordering vs a steady climber, not the discrete label itself.
+    """
+    i = len(closes) - 1 if as_of_index is None else as_of_index
+    if i < 252 or closes[i - 252] <= 0 or closes[i - 21] <= 0:
+        raise ValidationError(
+            f"FIP ID requires at least 253 bars with positive formation closes, "
+            f"got {len(closes)}"
+        )
+    pret = closes[i - 21] / closes[i - 252] - 1.0
+    rets: list[float] = []
+    for k in range(i - 251, i - 20):
+        prev = closes[k - 1]
+        if prev <= 0:
+            raise ValidationError("FIP ID requires positive closes in the formation window")
+        rets.append(closes[k] / prev - 1.0)
+    if len(rets) < 200:
+        raise ValidationError(
+            f"FIP ID requires ≥200 daily returns in formation, got {len(rets)}"
+        )
+    n = len(rets)
+    pct_pos = sum(1 for r in rets if r > 0) / n
+    pct_neg = sum(1 for r in rets if r < 0) / n
+    if pret > 0:
+        sign = 1.0
+    elif pret < 0:
+        sign = -1.0
+    else:
+        sign = 0.0
+    fip = sign * (pct_neg - pct_pos)
+    # Map observed ID range (~[-0.3, 0.15]) loosely to 0–100 for the card chrome;
+    # lower ID (smoother) → higher score. Display only.
+    score = max(0.0, min(100.0, 50.0 * (1.0 - fip)))
+    if fip <= FIP_PATH_CONTINUOUS_MAX:
+        path = "continuous"
+        path_label = "smooth grind (continuous information)"
+    elif fip >= FIP_PATH_DISCRETE_MIN:
+        path = "discrete"
+        path_label = "jumpy path (discrete information)"
+    else:
+        path = "mixed"
+        path_label = "mixed path"
+    return {
+        "fip_id": round(fip, 4),
+        "pret_12_1": round(pret, 4),
+        "pct_up_days": round(pct_pos * 100.0, 1),
+        "pct_down_days": round(pct_neg * 100.0, 1),
+        "formation_days": n,
+        "path": path,
+        "path_label": path_label,
+        "display_only": True,
+        "note": "Not used by the production gate or rank — context only.",
+        "score": round(score, 4),
+    }
+
+
 def compute_ema_cross(
     closes: list[float],
     short_period: int = 20,
@@ -451,7 +534,15 @@ def compute_ema_cross(
 # Supported indicator types
 # ---------------------------------------------------------------------------
 
-INDICATOR_TYPES = {"adx", "ema", "rsi", "atr", "volume_profile", "pivot_points"}
+INDICATOR_TYPES = {
+    "adx",
+    "ema",
+    "rsi",
+    "atr",
+    "volume_profile",
+    "pivot_points",
+    "fip_id",
+}
 
 
 # ---------------------------------------------------------------------------
@@ -514,6 +605,8 @@ async def get_indicator(
         result = compute_volume_profile(highs, lows, closes, volumes)
     elif indicator_type == "pivot_points":
         result = compute_pivot_points(highs, lows, closes)
+    elif indicator_type == "fip_id":
+        result = compute_fip_id(closes)
     else:
         raise ValidationError(f"Unknown indicator type: {indicator_type}")
 

app/services/ingestion_service.py

@@ -23,6 +23,15 @@ from app.services import price_service
 logger = logging.getLogger(__name__)
 
 
+async def _refresh_structural_sr(db: AsyncSession, symbol: str) -> None:
+    """Rebuild Structural S/R after batch OHLCV writes (best-effort).
+
+    Price bars are already committed; an S/R failure must not discard the
+    ingestion result. Shared with single-bar upsert via price_service.
+    """
+    await price_service._refresh_structural_sr_best_effort(db, symbol)
+
+
 @dataclass
 class IngestionResult:
     """Result of an ingestion run."""
@@ -91,6 +100,8 @@ async def fetch_and_ingest(
     symbol: str,
     start_date: date | None = None,
     end_date: date | None = None,
+    *,
+    refresh_sr: bool = True,
 ) -> IngestionResult:
     """Fetch OHLCV data from provider and upsert into Price Store.
 
@@ -120,7 +131,12 @@ async def fetch_and_ingest(
         if bar_count < minimum_backfill_bars:
             start_date = backfill_start
         elif progress is not None:
-            start_date = progress.last_ingested_date + timedelta(days=1)
+            # Re-fetch the latest stored session so an in-progress daily bar can
+            # be overwritten as the market moves. Starting one day later makes
+            # every subsequent intraday, near-close, and manual refresh skip
+            # today's bar once the first partial snapshot has been stored.
+            # The price-store upsert keeps this one-session overlap idempotent.
+            start_date = progress.last_ingested_date
         else:
             start_date = backfill_start
 
@@ -213,6 +229,8 @@ async def fetch_and_ingest(
                 low=record.low,
                 close=record.close,
                 volume=record.volume,
+                # One S/R rebuild at the end of the batch, not per bar.
+                refresh_sr=False,
             )
             ingested_count += 1
             last_ingested = record.date
@@ -221,12 +239,15 @@ async def fetch_and_ingest(
             await _update_progress(db, ticker.id, record.date)
 
         except RateLimitError:
-            # Mid-ingestion rate limit — return partial progress
+            # Mid-ingestion rate limit — return partial progress after
+            # refreshing S/R from whatever bars we already wrote.
             logger.warning(
                 "Rate limited during ingestion for %s after %d records",
                 ticker.symbol,
                 ingested_count,
             )
+            if ingested_count > 0 and refresh_sr:
+                await _refresh_structural_sr(db, ticker.symbol)
             return IngestionResult(
                 symbol=ticker.symbol,
                 records_ingested=ingested_count,
@@ -235,6 +256,28 @@ async def fetch_and_ingest(
                 message=f"Rate limited. Ingested {ingested_count} records. Resume available.",
             )
 
+    if ingested_count > 0 and refresh_sr:
+        await _refresh_structural_sr(db, ticker.symbol)
+
+    # Incremental fetches deliberately overlap the latest stored session so an
+    # in-progress bar can be updated. A halted/delisted symbol can therefore
+    # return one old bar forever; non-empty no longer means fresh. Judge stale
+    # state from the newest stored session after the upserts instead.
+    latest = await _get_latest_ohlcv_date(db, ticker.id)
+    gap_days = (end_date - latest).days if latest is not None else None
+    if gap_days is not None and gap_days > _STALE_OHLCV_GAP_DAYS:
+        return IngestionResult(
+            symbol=ticker.symbol,
+            records_ingested=ingested_count,
+            last_date=latest,
+            status="stale",
+            message=(
+                f"No new bars since {latest.isoformat()} ({gap_days}d gap). "
+                "The symbol may be halted, delisted, or renamed under a new ticker — "
+                "check the listing and add/fetch the current symbol if it changed."
+            ),
+        )
+
     return IngestionResult(
         symbol=ticker.symbol,
         records_ingested=ingested_count,

app/services/momentum_service.py

@@ -34,6 +34,28 @@ STRATEGY_RANK_MOMENTUM_WEIGHT = 0.8
 STRATEGY_RANK_VOL_WEIGHT = 1.0 - STRATEGY_RANK_MOMENTUM_WEIGHT
 
 
+def blend_strategy_rank(
+    momentum_percentile: float | None,
+    volatility_percentile: float | None,
+    *,
+    momentum_weight: float = STRATEGY_RANK_MOMENTUM_WEIGHT,
+) -> float | None:
+    """80/20 production rank with mom-only fallback when vol is missing.
+
+    Live and backtest must share this policy: missing vol must not send a
+    residual-qualified name to the bottom of the book (that was the old
+    backtest behaviour when either leg was None).
+    """
+    if momentum_percentile is not None and volatility_percentile is not None:
+        vol_weight = 1.0 - momentum_weight
+        return round(
+            float(momentum_percentile) * momentum_weight
+            + float(volatility_percentile) * vol_weight,
+            2,
+        )
+    return float(momentum_percentile) if momentum_percentile is not None else None
+
+
 def compute_12_1_momentum(closes: list[float]) -> float | None:
     """Return over the window ending ~1 month ago, starting ~12 months ago.
     None when there isn't a full year of history."""
@@ -100,41 +122,17 @@ async def _load_activation_benchmark(db: AsyncSession) -> dict[date, float]:
 
 
 async def compute_momentum_percentiles(db: AsyncSession) -> dict[str, float]:
-    """Compute each ticker's activation momentum rank.
+    """Momentum leg only — thin view of ``compute_activation_ranks``.
 
-    Production uses residual 12-1 momentum when benchmark data is available. If
-    SPY data is absent, fall back to raw 12-1 momentum rather than disabling the
-    scanner. Tickers without enough stock/benchmark history are absent.
+    Prefer ``compute_activation_ranks`` in new code (includes vol + strategy_rank).
+    Kept so tests/helpers that only need the residual/raw percentile map stay simple.
     """
-    result = await db.execute(select(Ticker).order_by(Ticker.symbol))
-    tickers = list(result.scalars().all())
-
-    benchmark_closes = await _load_activation_benchmark(db)
-    using_residual = len(benchmark_closes) >= _MOM_LOOKBACK
-
-    values: dict[str, float] = {}
-    for ticker in tickers:
-        try:
-            records = await query_ohlcv(db, ticker.symbol)
-        except Exception:
-            logger.exception("Momentum fetch failed for %s", ticker.symbol)
-            continue
-        closes = [float(r.close) for r in records]
-        value = (
-            compute_residual_12_1_momentum([r.date for r in records], closes, benchmark_closes)
-            if using_residual
-            else compute_12_1_momentum(closes)
-        )
-        if value is not None:
-            values[ticker.symbol] = value
-
-    percentiles = _percentiles(values)
-    logger.info(json.dumps({
-        "event": "momentum_ranked",
-        "signal": "residual_12_1" if using_residual else "raw_12_1_fallback",
-        "tickers": len(percentiles),
-    }))
-    return percentiles
+    ranks = await compute_activation_ranks(db)
+    return {
+        sym: float(row["momentum_percentile"])
+        for sym, row in ranks.items()
+        if row.get("momentum_percentile") is not None
+    }
 
 
 def compute_realized_vol_6m(closes: list[float]) -> float | None:
@@ -204,19 +202,10 @@ async def compute_activation_ranks(db: AsyncSession) -> dict[str, dict[str, floa
     for sym in symbols:
         momentum_pct = momentum_percentiles.get(sym)
         vol_pct = vol_percentiles.get(sym)
-        strategy_rank = (
-            round(
-                momentum_pct * STRATEGY_RANK_MOMENTUM_WEIGHT
-                + vol_pct * STRATEGY_RANK_VOL_WEIGHT,
-                2,
-            )
-            if momentum_pct is not None and vol_pct is not None
-            else momentum_pct
-        )
         ranks[sym] = {
             "momentum_percentile": momentum_pct,
             "volatility_percentile": vol_pct,
-            "strategy_rank": strategy_rank,
+            "strategy_rank": blend_strategy_rank(momentum_pct, vol_pct),
         }
 
     logger.info(json.dumps({

app/services/outcome_service.py

@@ -1,11 +1,15 @@
 """Trade setup outcome evaluation service.
 
-Closes the feedback loop on R:R scanner setups: walks daily OHLCV bars
-after detection and records whether the stop or the target was hit first.
+Diagnostic barrier resolution for scanner setups: walks daily OHLCV bars
+after detection and records whether the gate target or the stop was hit first.
+
+This is **not** the production exit model. Live paper trades and the portfolio
+monitor use ATR trail / max hold and never exit at the gate target. Track-record
+stats from this path measure gate-level plumbing, not ATR-trail book expectancy.
 
 Outcome semantics (entry is the close at detection time, i.e. market entry):
-  - target_hit: target reached before the stop
-  - stop_hit:   stop reached before the target
+  - target_hit: gate target reached before the stop
+  - stop_hit:   stop reached before the gate target
   - ambiguous:  stop AND target both within the same daily bar — with daily
                 granularity the order is unknowable, counted as a loss in stats
   - expired:    neither level hit within ``max_bars`` trading days

app/services/paper_trade_service.py

@@ -3,6 +3,7 @@
 from __future__ import annotations
 
 import bisect
+import logging
 from datetime import date, datetime, timezone
 
 from sqlalchemy import and_, func, select
@@ -20,7 +21,9 @@ from app.services.outcome_service import (
     Bar,
     evaluate_setup_against_bars,
 )
-from app.services.trade_policy import get_reentry_lockdowns
+from app.services.trade_policy import MANUAL_BOOK, SHADOW_BOOK, get_reentry_gate_locks
+
+logger = logging.getLogger(__name__)
 
 # Exit policy for OPEN paper trades (auto-close). Production defaults to the
 # July 2026 promoted strategy: initial stop + 3x ATR trailing stop, with a max
@@ -319,12 +322,9 @@ async def create_trade(
         raise ValidationError("shares and entry_price must be positive")
 
     ticker = await _get_ticker(db, symbol)
-    remaining_sessions = (await get_reentry_lockdowns(db)).get(ticker.id)
-    if remaining_sessions is not None:
-        suffix = "session" if remaining_sessions == 1 else "sessions"
+    if ticker.id in await get_reentry_gate_locks(db):
         raise ValidationError(
-            f"{ticker.symbol} is in a post-stop re-entry lockdown: "
-            f"{remaining_sessions} market {suffix} remaining"
+            f"{ticker.symbol} requires a post-stop gate reset before re-entry"
         )
     trade = PaperTrade(
         user_id=user_id,
@@ -336,6 +336,9 @@ async def create_trade(
         target=target,
         status="open",
         opened_at=datetime.now(timezone.utc),
+        # Near-close cutover era — Track Record must not mix with morning-scan
+        # fills or future broker-routed fills when comparing to backtests.
+        fill_mode="near_close",
     )
     db.add(trade)
     await db.commit()
@@ -349,6 +352,7 @@ def _to_dict(
     current_price: float | None,
     benchmark_closes: dict[date, float] | None = None,
     trailing: tuple[float, float | None] | None = None,
+    holding_sessions: tuple[int, int] | None = None,
 ) -> dict:
     # For open trades, mark to market; for closed, the realized exit price.
     ref = current_price if trade.status == "open" else trade.close_price
@@ -389,8 +393,11 @@ def _to_dict(
         "alpha_pct": alpha_pct,
         "alpha_usd": alpha_usd,
         "close_reason": trade.close_reason,
+        "fill_mode": trade.fill_mode,
         "trailing_stop": trailing[0] if trailing else None,
         "trailing_distance_pct": trailing[1] if trailing else None,
+        "sessions_held": holding_sessions[0] if holding_sessions else None,
+        "sessions_remaining": holding_sessions[1] if holding_sessions else None,
     }
 
 
@@ -398,7 +405,15 @@ async def list_trades(
     db: AsyncSession,
     user_id: int | None = None,
     status: str | None = None,
+    book: str | None = MANUAL_BOOK,
 ) -> list[dict]:
+    """Trades for the UI. Defaults to the discretionary book.
+
+    Shadow trades are attached to a user row for FK reasons only — they are not
+    that person's decisions. Listing them alongside manual trades would mix two
+    different books in one P&L and let the autonomous record be edited by hand.
+    Pass ``book=None`` to deliberately span both.
+    """
     stmt = (
         select(PaperTrade, Ticker.symbol)
         .join(Ticker, PaperTrade.ticker_id == Ticker.id)
@@ -407,6 +422,8 @@ async def list_trades(
         stmt = stmt.where(PaperTrade.user_id == user_id)
     if status is not None:
         stmt = stmt.where(PaperTrade.status == status)
+    if book is not None:
+        stmt = stmt.where(PaperTrade.book == book)
     stmt = stmt.order_by(PaperTrade.opened_at.desc())
 
     rows = (await db.execute(stmt)).all()
@@ -421,6 +438,35 @@ async def list_trades(
     # Current trailing-stop level + distance for open trades (when a trailing
     # policy is active).
     policy = await get_exit_policy(db)
+    holding_sessions: dict[int, tuple[int, int]] = {}
+    if policy["mode"] in ("time", "atr_trailing"):
+        hold_days = int(policy["hold_days"])
+        open_trades = [trade for trade, _ in rows if trade.status == "open"]
+        if open_trades:
+            ticker_ids = {trade.ticker_id for trade in open_trades}
+            earliest_opened = min(trade.opened_at.date() for trade in open_trades)
+            session_rows = (
+                await db.execute(
+                    select(OHLCVRecord.ticker_id, OHLCVRecord.date)
+                    .where(
+                        OHLCVRecord.ticker_id.in_(ticker_ids),
+                        OHLCVRecord.date > earliest_opened,
+                    )
+                    .order_by(OHLCVRecord.ticker_id, OHLCVRecord.date)
+                )
+            ).all()
+            dates_by_ticker: dict[int, list[date]] = {}
+            for ticker_id, session_date in session_rows:
+                dates_by_ticker.setdefault(int(ticker_id), []).append(session_date)
+            for trade in open_trades:
+                dates = dates_by_ticker.get(trade.ticker_id, [])
+                held = len(dates) - bisect.bisect_right(
+                    dates, trade.opened_at.date()
+                )
+                # Do not clamp: a policy shortened below the current holding
+                # period must remain visible as overdue until the exit pass runs.
+                holding_sessions[trade.id] = (held, hold_days - held)
+
     trailing_info: dict[int, tuple[float, float | None]] = {}
     if policy["mode"] == "trailing":
         trail_frac = policy["trailing_pct"] / 100.0
@@ -469,7 +515,14 @@ async def list_trades(
             trailing_info[t.id] = (level, dist)
 
     return [
-        _to_dict(t, sym, prices.get(t.ticker_id), benchmark_closes, trailing_info.get(t.id))
+        _to_dict(
+            t,
+            sym,
+            prices.get(t.ticker_id),
+            benchmark_closes,
+            trailing_info.get(t.id),
+            holding_sessions.get(t.id),
+        )
         for t, sym in rows
     ]
 
@@ -489,6 +542,13 @@ async def close_trade(
     trade = result.scalar_one_or_none()
     if trade is None:
         raise NotFoundError(f"Paper trade not found: {trade_id}")
+    if trade.book == SHADOW_BOOK:
+        # The shadow book's value is that no human touched it. A hand-closed
+        # position would make its record something other than what the strategy
+        # would have done; it exits only via the automatic exit policy.
+        raise ValidationError(
+            "Shadow book trades are closed by the exit policy, not by hand"
+        )
     if trade.status == "closed":
         raise ValidationError("Trade is already closed")
 
@@ -689,6 +749,66 @@ def build_equity_curve(
     return out
 
 
+KEY_PERFORMANCE_START = "performance_start_date"
+
+
+async def get_performance_start(db: AsyncSession) -> date | None:
+    """Date the performance view starts from, or None for 'all history'.
+
+    The strategy has been revised repeatedly, so early trades were taken under
+    rules that no longer exist. Pinning a start date keeps the comparison inside
+    one regime instead of averaging across configurations that were replaced.
+    """
+    raw = await settings_store.get_value(db, KEY_PERFORMANCE_START, "")
+    if not raw or not str(raw).strip():
+        return None
+    try:
+        return date.fromisoformat(str(raw).strip())
+    except ValueError:
+        logger.warning("invalid %s: %r", KEY_PERFORMANCE_START, raw)
+        return None
+
+
+def trade_r_multiple(trade, mark: float | None) -> float | None:
+    """Result in R — profit measured in units of the trade's own initial risk.
+
+    R is the only sizing-independent yardstick available here: the shadow book
+    sizes at a fixed 1% of equity while manual trades were sized by hand, so
+    currency P&L cannot compare them. Open trades are marked to ``mark``.
+    """
+    risk_per_share = abs(trade.entry_price - trade.stop_loss)
+    if risk_per_share <= 0:
+        return None
+    exit_price = trade.close_price if trade.status == "closed" else mark
+    if exit_price is None:
+        return None
+    per_share = (
+        exit_price - trade.entry_price
+        if trade.direction == "long"
+        else trade.entry_price - exit_price
+    )
+    return per_share / risk_per_share
+
+
+def book_stats(trades: list, marks: dict[int, float]) -> dict:
+    """Sizing-independent summary of one book: counts, win rate, R-multiples."""
+    rs = [
+        r
+        for r in (trade_r_multiple(t, marks.get(t.ticker_id)) for t in trades)
+        if r is not None
+    ]
+    closed = [t for t in trades if t.status == "closed"]
+    wins = [r for r in rs if r > 0]
+    return {
+        "trades": len(trades),
+        "closed": len(closed),
+        "open": len(trades) - len(closed),
+        "win_rate": round(100.0 * len(wins) / len(rs), 1) if rs else None,
+        "total_r": round(sum(rs), 2) if rs else 0.0,
+        "avg_r": round(sum(rs) / len(rs), 3) if rs else None,
+    }
+
+
 async def equity_curve(db: AsyncSession, user_id: int) -> list[dict]:
     """Equity-curve series for a user's paper book (empty without benchmark data)."""
     trades = (
@@ -713,3 +833,118 @@ async def equity_curve(db: AsyncSession, user_id: int) -> list[dict]:
     for tid, day, close in rows.all():
         ticker_closes.setdefault(tid, {})[day] = float(close)
     return build_equity_curve(list(trades), ticker_closes, benchmark_closes)
+
+
+def _cumulative_pnl(trades: list, ticker_closes: dict, days: list[date]) -> list[float]:
+    """Cumulative realized + mark-to-market P&L of one book on each day."""
+    sorted_dates = {tid: sorted(c) for tid, c in ticker_closes.items()}
+    out: list[float] = []
+    for d in days:
+        total = 0.0
+        for t in trades:
+            if t.opened_at.date() > d:
+                continue
+            closed_on = (
+                t.closed_at.date()
+                if (t.status == "closed" and t.closed_at is not None)
+                else None
+            )
+            if closed_on is not None and closed_on <= d and t.close_price is not None:
+                ref = float(t.close_price)
+            else:
+                ref = _value_on_or_before(
+                    sorted_dates.get(t.ticker_id) or [],
+                    ticker_closes.get(t.ticker_id) or {},
+                    d,
+                )
+                if ref is None:
+                    continue
+            per_share = (
+                ref - t.entry_price if t.direction == "long" else t.entry_price - ref
+            )
+            total += per_share * t.shares
+        out.append(round(total, 2))
+    return out
+
+
+async def performance_summary(db: AsyncSession, user_id: int | None = None) -> dict:
+    """Shadow book vs discretionary book vs SPY, from the configured start date.
+
+    Currency P&L is reported per book but is *not* the comparison — the books
+    size differently, so the honest read is the R-multiple stats. SPY is a plain
+    buy-and-hold reference over the same window rather than a per-trade
+    counterfactual, so one line serves both books.
+    """
+    start = await get_performance_start(db)
+    stmt = select(PaperTrade)
+    if start is not None:
+        stmt = stmt.where(func.date(PaperTrade.opened_at) >= start)
+    if user_id is not None:
+        # "Your picks" must be *yours*. The shadow book is a single autonomous
+        # book with no owner, so it is never scoped to a user.
+        stmt = stmt.where(
+            (PaperTrade.book == SHADOW_BOOK) | (PaperTrade.user_id == user_id)
+        )
+    trades = list((await db.execute(stmt)).scalars().all())
+
+    benchmark_closes = await benchmark_service.load_benchmark_closes(db)
+    empty = {
+        "start_date": start.isoformat() if start else None,
+        "series": [],
+        "stats": {},
+    }
+    if not trades or not benchmark_closes:
+        return empty
+
+    first = min(t.opened_at.date() for t in trades)
+    if start is not None:
+        first = max(first, start)
+    days = [d for d in sorted(benchmark_closes) if d >= first]
+    if not days:
+        return empty
+
+    ticker_ids = {t.ticker_id for t in trades}
+    rows = await db.execute(
+        select(OHLCVRecord.ticker_id, OHLCVRecord.date, OHLCVRecord.close).where(
+            OHLCVRecord.ticker_id.in_(ticker_ids), OHLCVRecord.date >= first
+        )
+    )
+    ticker_closes: dict[int, dict[date, float]] = {}
+    for tid, day, close in rows.all():
+        ticker_closes.setdefault(tid, {})[day] = float(close)
+
+    books = {
+        MANUAL_BOOK: [t for t in trades if (t.book or MANUAL_BOOK) == MANUAL_BOOK],
+        SHADOW_BOOK: [t for t in trades if t.book == SHADOW_BOOK],
+    }
+    pnl = {
+        name: _cumulative_pnl(book_trades, ticker_closes, days)
+        for name, book_trades in books.items()
+    }
+
+    bench_dates = sorted(benchmark_closes)
+    spy0 = _value_on_or_before(bench_dates, benchmark_closes, days[0])
+    spy_pct = [
+        round(100.0 * (benchmark_closes[d] / spy0 - 1.0), 2) if spy0 else 0.0
+        for d in days
+    ]
+
+    # Latest close per ticker, for marking open positions in the R stats.
+    marks = {
+        tid: closes[max(closes)] for tid, closes in ticker_closes.items() if closes
+    }
+    stats = {name: book_stats(bt, marks) for name, bt in books.items()}
+    for name in books:
+        stats[name]["pnl"] = pnl[name][-1] if pnl[name] else 0.0
+    stats["spy"] = {"pct": spy_pct[-1] if spy_pct else 0.0}
+
+    series = [
+        {
+            "date": d.isoformat(),
+            "manual_pnl": pnl[MANUAL_BOOK][i],
+            "shadow_pnl": pnl[SHADOW_BOOK][i],
+            "spy_pct": spy_pct[i],
+        }
+        for i, d in enumerate(days)
+    ]
+    return {"start_date": start.isoformat() if start else None, "series": series, "stats": stats}

app/services/pipeline_run.py

@@ -0,0 +1,47 @@
+"""Per-invocation identity for pipeline runs.
+
+A pipeline invocation stamps a unique run id into the task context. The scan it
+runs records that id alongside its completion markers, and the shadow book
+requires an *exact* match before acting on the scan's batch.
+
+This is what timestamp comparison cannot provide. A manually triggered scan and
+the scheduled near-close pipeline are separate APScheduler jobs, and
+``max_instances=1`` only serialises a job against itself — not two different
+jobs. So a manual scan can start just before the pipeline and finish just after
+it began, leaving a completion timestamp later than the pipeline's start even
+though its batch is unrelated. Matching on a run id generated by the pipeline,
+and stamped only by the scan running inside that pipeline, removes the ambiguity.
+
+Lives in its own module so the scheduler (which sets the id), the scanner (which
+stamps it), and the shadow book (which checks it) can all import it without an
+import cycle.
+"""
+
+from __future__ import annotations
+
+import contextvars
+import uuid
+
+_run_id: contextvars.ContextVar[str | None] = contextvars.ContextVar(
+    "pipeline_run_id", default=None
+)
+
+
+def new_run_id() -> str:
+    """A fresh, collision-free run id."""
+    return uuid.uuid4().hex
+
+
+def current() -> str | None:
+    """Run id of the pipeline invocation on the current task, if any."""
+    return _run_id.get()
+
+
+def bind(run_id: str) -> contextvars.Token:
+    """Set the current run id; pass the returned token to ``release``."""
+    return _run_id.set(run_id)
+
+
+def release(token: contextvars.Token) -> None:
+    """Restore the previous run id (call in a finally)."""
+    _run_id.reset(token)

app/services/price_service.py

@@ -1,5 +1,8 @@
 """Price Store service: upsert and query OHLCV records."""
 
+from __future__ import annotations
+
+import logging
 from datetime import date, datetime
 
 from sqlalchemy import select
@@ -10,6 +13,8 @@ from app.exceptions import NotFoundError, ValidationError
 from app.models.ohlcv import OHLCVRecord
 from app.models.ticker import Ticker
 
+logger = logging.getLogger(__name__)
+
 
 async def _get_ticker(db: AsyncSession, symbol: str) -> Ticker:
     """Look up a ticker by symbol. Raises NotFoundError if missing."""
@@ -44,11 +49,22 @@ async def upsert_ohlcv(
     low: float,
     close: float,
     volume: int,
+    *,
+    refresh_sr: bool = True,
 ) -> OHLCVRecord:
     """Insert or update an OHLCV record for (ticker, date).
 
     Validates business rules, resolves ticker, then uses
     ON CONFLICT DO UPDATE on the (ticker_id, date) unique constraint.
+
+    ``refresh_sr`` (default True) recalculates persisted Structural S/R after
+    the write so chart levels stay current. Batch ingestion passes
+    ``refresh_sr=False`` and refreshes once at the end of the ticker batch.
+
+    The OHLCV commit is authoritative: if S/R rebuild fails after a successful
+    price write, the error is logged, the session is rolled back to clear
+    poison, and the upsert still returns the persisted bar (caller can retry
+    S/R via the scanner/ingestion pipeline).
     """
     _validate_ohlcv(high, low, open_, close, volume, record_date)
     ticker = await _get_ticker(db, symbol)
@@ -80,12 +96,36 @@ async def upsert_ohlcv(
 
     record = result.scalar_one()
 
-    # TODO: Invalidate LRU cache entries for this ticker (Task 7.1)
-    # TODO: Mark composite score as stale for this ticker (Task 10.1)
+    from app.cache import indicator_cache
+
+    indicator_cache.invalidate_ticker(ticker.symbol)
+
+    if refresh_sr:
+        await _refresh_structural_sr_best_effort(db, ticker.symbol)
 
     return record
 
 
+async def _refresh_structural_sr_best_effort(db: AsyncSession, symbol: str) -> bool:
+    """Rebuild Structural S/R; never fail a successful OHLCV write.
+
+    Returns True on success. On failure rolls the session back so a later
+    operation on the same session is not poisoned by the failed unit of work.
+    """
+    from app.services.sr_service import recalculate_sr_levels
+
+    try:
+        await recalculate_sr_levels(db, symbol)
+        return True
+    except Exception:
+        logger.exception("Structural S/R refresh failed for %s after OHLCV write", symbol)
+        try:
+            await db.rollback()
+        except Exception:
+            logger.exception("Session rollback after S/R failure also failed for %s", symbol)
+        return False
+
+
 async def query_ohlcv(
     db: AsyncSession,
     symbol: str,

app/services/recommendation_service.py

@@ -618,6 +618,10 @@ def build_recommendation_snapshot(
 # agree on what counts as a probability-backed target.
 PRIMARY_TARGET_MIN_PROBABILITY = MIN_TARGET_PROBABILITY
 
+# Primary-target selector floor (independent of the live activation min_rr).
+# Live scanner and backtest setup replay must share this constant.
+PRIMARY_TARGET_MIN_RR = 1.5
+
 
 def _prune_floor_pinned_targets(targets: list[dict]) -> list[dict]:
     """Keep only the nearest target pinned at the probability clamp floor.

app/services/regime_monitor_service.py

@@ -1,16 +1,23 @@
-"""AI/Tech Regime Monitor v2.
+"""AI/Tech Regime Monitor v3.
 
 The monitor is a risk thermometer, not a probability or trading rule.  It keeps
 two deliberately separate outputs:
 
 * State: current structural stress (price, breadth, credit, volatility).
-* Warning: deterioration/divergence that may precede State (breadth, relative
-  strength, and sourced fundamental observations).
+* Warning: deterioration/divergence that may precede State (breadth divergence,
+  relative strength, credit impulse).
 
-Daily snapshots are the point-in-time record.  The first v2 run rewrites the
-latest ``REBUILD_SESSIONS`` trading sessions once; ordinary runs thereafter only
-upsert the latest trading date.  Fundamental observations are never replayed
-before their effective date.
+Both scores are quantitative and daily.  The sourced hyperscaler capex and
+earnings-reaction observations are a qualitative *overlay* in v3 rather than
+weighted sensors: at a combined 20 points they could not reach the event
+study's alarm threshold even when both pegged, so refreshing them appeared to
+do nothing.  They are reported next to the scores instead of inside them.
+
+Daily snapshots are the point-in-time record.  The first run under a new
+``METHODOLOGY`` rewrites the latest ``REBUILD_SESSIONS`` trading sessions once;
+ordinary runs thereafter only upsert the latest trading date.  The overlay is
+still gated by its effective date so a rebuild cannot stamp today's observation
+onto historical snapshots.
 """
 
 from __future__ import annotations
@@ -41,19 +48,51 @@ _CA_BUNDLE = os.environ.get("SSL_CERT_FILE", "")
 KEY_CONFIG = "regime_monitor_config"
 KEY_FUNDAMENTALS = "regime_fundamental_overrides"
 
-METHODOLOGY = "v2"
+METHODOLOGY = "v3"
+# Snapshots are reseeded on a methodology bump, but fundamental observations are
+# collected by hand/LLM and carried across it when the format is compatible.
+CATEGORICAL_FUNDAMENTAL_METHODOLOGIES = frozenset({"v2", "v3"})
 REBUILD_SESSIONS = 400
 MIN_COVERAGE = 75.0
 SOURCE_MAX_LAG_DAYS = 7
 
-QUADRANT_STATE_DIVIDER = 60.0
-QUADRANT_WARNING_DIVIDER = 60.0
+# Bands are per axis: the two scores have genuinely different realized ranges,
+# so one shared set made Warning's top bands unreachable (v2 Warning never
+# exceeded 64.9 in 408 sessions while State reached 91.2).  Thresholds are round
+# numbers chosen so each band covers a sane share of history, not percentile
+# fits -- percentile-derived bands would drift on every rebuild and silently
+# rewrite what past snapshots meant.  Realized shares over the 408 sessions to
+# 2026-07-24: State 73/15/8/3%, Warning 69/20/8/3%.
+STATE_BANDS = (20.0, 50.0, 80.0)
+WARNING_BANDS = (20.0, 40.0, 60.0)
+
+QUADRANT_STATE_DIVIDER = 50.0
+QUADRANT_WARNING_DIVIDER = 40.0
 QUADRANT_MARGIN = 5.0
 
 HY_OAS_MILD = 3.5
 HY_OAS_ELEVATED = 5.0
 HY_OAS_STRESSED = 7.0
-HY_OAS_REFERENCE_YEARS = 10.0
+# ICE restricted FRED to a rolling 3-year window for BAMLH0A0HYM2 in April 2026
+# ("Starting in April 2026, this series will only include 3 years of
+# observations"), so v2's 10-year reference window silently became 3.  Against 3
+# years of uniformly tight spreads (2.59-4.61 over the calibration window) the
+# blended upper-tail percentile saturated at an OAS of ~4.5 and scored 20 points
+# of stress at 3.5 -- the level these anchors call "mild".  The anchors already
+# encode the long-run distribution, so the credit *level* is now purely anchored
+# and credit *dynamics* live in W3 on the Warning axis where they belong.
+HY_OAS_WINDOW_DAYS = 400  # only W3's lookback plus slack is needed now
+W3_OAS_LOOKBACK = 20
+W3_OAS_FULL_SCALE_PCT = 35.0
+
+# Drawdown anchors (drawdown %, stress score).  v2 used a bare ``dd_pct * 5``,
+# which pegged at a 20% drawdown -- the 90th percentile of the observed
+# distribution -- so 39 of 408 sessions sat at exactly 100 with no resolution
+# left during the part of a selloff that matters most.  These anchors keep
+# headroom past the observed 36% maximum.
+P3_DRAWDOWN_ANCHORS = (
+    (0.0, 0.0), (4.0, 10.0), (8.0, 25.0), (16.0, 50.0), (28.0, 78.0), (40.0, 100.0),
+)
 
 STATE_WEIGHTS = {
     "price": 40.0,
@@ -61,11 +100,15 @@ STATE_WEIGHTS = {
     "credit": 20.0,
     "volatility": 15.0,
 }
+# Fundamentals left the score in v3.  At 12 + 8 points they could not reach the
+# event study's alarm threshold even when both pegged at 100, so the LLM read was
+# decorative; it is now a separate qualitative overlay.  Credit *impulse* takes
+# their place because the OAS level is pinned at zero below the 3.5 anchor while
+# its rate of change is not.
 WARNING_WEIGHTS = {
-    "breadth_divergence": 50.0,
+    "breadth_divergence": 45.0,
     "relative_strength": 30.0,
-    "capex": 12.0,
-    "earnings_reaction": 8.0,
+    "credit_impulse": 25.0,
 }
 
 # Fixed at the v2 launch.  These are liquid S&P 500/Nasdaq AI, semiconductor,
@@ -92,7 +135,10 @@ DEFAULT_CONFIG: dict = {
 
 CAPEX_STATES = ("raising", "holding", "cutting", "unknown")
 GNSD_STATES = ("yes", "no", "mixed")
-_CAPEX_STATE_SCORES = {"raising": 0.0, "holding": 0.0, "cutting": 100.0}
+# v2 scored raising and holding identically at 0, so in a capex boom the reading
+# was pinned at 0 and could not express the raising -> holding deceleration that
+# is the actual early warning. Display-only in v3, but it should still describe.
+_CAPEX_STATE_SCORES = {"raising": 0.0, "holding": 50.0, "cutting": 100.0}
 _GNSD_SCORES = {"yes": 100.0, "no": 0.0}
 
 Series = list[tuple[date, float]]
@@ -127,12 +173,23 @@ def _blend(leader: float | None, confirm: float | None, leader_weight: float = 2
     return sum(v * w for v, w in parts) / sum(w for _, w in parts)
 
 
-def band_for(score: float) -> str:
-    if score < 30:
+def _interpolate(x: float, anchors: tuple[tuple[float, float], ...]) -> float:
+    """Piecewise-linear lookup, flat outside the first and last anchor."""
+    if x <= anchors[0][0]:
+        return anchors[0][1]
+    for (x0, y0), (x1, y1) in zip(anchors, anchors[1:]):
+        if x <= x1:
+            return y0 + (y1 - y0) * (x - x0) / (x1 - x0)
+    return anchors[-1][1]
+
+
+def band_for(score: float, bands: tuple[float, float, float] = STATE_BANDS) -> str:
+    watch, elevated, breaking = bands
+    if score < watch:
         return "stable"
-    if score < 60:
+    if score < elevated:
         return "watch"
-    if score < 80:
+    if score < breaking:
         return "elevated"
     return "breaking"
 
@@ -167,19 +224,29 @@ def p2_death_cross(smh: list[float], qqq: list[float], leader_weight: float = 2.
     return _blend(_death_cross(smh), _death_cross(qqq), leader_weight)
 
 
-def _drawdown(closes: list[float]) -> float | None:
+def drawdown_pct(closes: list[float]) -> float | None:
+    """Percentage below the trailing 52-week closing high."""
     if len(closes) < 30:
         return None
     peak = max(closes[-252:])
     if peak <= 0:
         return None
-    dd_pct = (peak - closes[-1]) / peak * 100.0
-    return _clamp(dd_pct * 5.0)
+    return (peak - closes[-1]) / peak * 100.0
 
 
-def p3_drawdown(smh: list[float], qqq: list[float]) -> float | None:
-    vals = [v for v in (_drawdown(smh), _drawdown(qqq)) if v is not None]
-    return max(vals) if vals else None
+def _drawdown(closes: list[float]) -> float | None:
+    dd_pct = drawdown_pct(closes)
+    return None if dd_pct is None else _clamp(_interpolate(dd_pct, P3_DRAWDOWN_ANCHORS))
+
+
+def p3_drawdown(smh: list[float], qqq: list[float], leader_weight: float = 2.0) -> float | None:
+    """Anchored drawdown stress on the same 2:1 leader/confirm blend P1 and P2 use.
+
+    v2 took ``max()`` here, which meant the more volatile leader always won and
+    the price pillar reduced to this one sensor: its realized share of State was
+    65% against a nominal 40% weight.  Blending brings that back to 40%.
+    """
+    return _blend(_drawdown(smh), _drawdown(qqq), leader_weight)
 
 
 def p4_relative_strength(smh: list[float], spy: list[float], lookback: int = 60) -> float | None:
@@ -220,18 +287,68 @@ def _oas_absolute_score(value: float) -> float:
 
 
 def f2_credit_spreads(oas_values: list[float]) -> float | None:
-    """HY OAS stress: 70% named absolute anchors + 30% upper-tail percentile."""
+    """HY OAS level against named absolute anchors (3.5 mild / 5.0 / 7.0).
+
+    v2 blended 70% of this with a 30% upper-tail percentile over the available
+    history. That leg was always a second, noisier estimate of what the anchors
+    already encode -- and once the usable window shrank to 3 uniformly tight
+    years it saturated far below any real stress level. Removed rather than
+    repaired: see ``HY_OAS_WINDOW_DAYS``.
+    """
     if not oas_values:
         return None
-    latest = oas_values[-1]
-    absolute = _oas_absolute_score(latest)
-    if len(oas_values) < 30:
-        return round(absolute, 2)
-    less = sum(1 for v in oas_values if v < latest)
-    equal = sum(1 for v in oas_values if v == latest)
-    percentile = (less + 0.5 * equal) / len(oas_values) * 100.0
-    relative = _clamp((percentile - 50.0) / 45.0 * 100.0)
-    return round(absolute * 0.7 + relative * 0.3, 2)
+    return round(_oas_absolute_score(oas_values[-1]), 2)
+
+
+def w3_credit_impulse(
+    oas_values: list[float], lookback: int = W3_OAS_LOOKBACK
+) -> float | None:
+    """HY OAS rate of change: widening only, relative so it works at any level.
+
+    The credit *level* (C1) sits at zero for as long as spreads stay under the
+    3.5 mild anchor -- 2.77 as of the v3 cutover -- so it contributes nothing to
+    State in a calm tape.  The rate of change still does, and spread widening is
+    a classic lead, which is what Warning is for.  Relative rather than absolute
+    because +0.5pp means something very different at 2.7 than at 8.0.
+    """
+    if len(oas_values) < lookback + 1:
+        return None
+    past = oas_values[-lookback - 1]
+    if past <= 0:
+        return None
+    change_pct = (oas_values[-1] / past - 1.0) * 100.0
+    return _clamp(change_pct / W3_OAS_FULL_SCALE_PCT * 100.0)
+
+
+def warning_sensor_scores(
+    divergence: float | None,
+    leader_closes: list[float],
+    market_closes: list[float],
+    oas_window: list[float],
+) -> dict[str, float | None]:
+    """The three Warning sensors, by pillar id.
+
+    Single definition so the live monitor and the event study cannot drift apart
+    -- in v2 the study re-derived the score from ``WARNING_WEIGHTS`` by hand and
+    would have silently kept measuring the old construct through this change.
+    """
+    return {
+        "breadth_divergence": divergence,
+        "relative_strength": p4_relative_strength(leader_closes, market_closes),
+        "credit_impulse": w3_credit_impulse(oas_window),
+    }
+
+
+def score_warning_sensors(sensors: dict[str, float | None]) -> float | None:
+    """Weighted Warning score, renormalised over the sensors that are available."""
+    live = [
+        (float(score), float(WARNING_WEIGHTS[key]))
+        for key, score in sensors.items()
+        if score is not None and key in WARNING_WEIGHTS
+    ]
+    if not live:
+        return None
+    return sum(s * w for s, w in live) / sum(w for _, w in live)
 
 
 def _sensor(sensor_id: str, label: str, score: float | None, **details: object) -> dict:
@@ -244,7 +361,11 @@ def _sensor(sensor_id: str, label: str, score: float | None, **details: object)
     }
 
 
-def _score_pillars(pillars: list[dict], weights: dict[str, float]) -> dict:
+def _score_pillars(
+    pillars: list[dict],
+    weights: dict[str, float],
+    bands: tuple[float, float, float] = STATE_BANDS,
+) -> dict:
     expected = sum(max(0.0, float(w)) for w in weights.values())
     available_weight = sum(
         max(0.0, float(weights.get(p["id"], 0.0)))
@@ -276,7 +397,12 @@ def _score_pillars(pillars: list[dict], weights: dict[str, float]) -> dict:
     rounded = round(score, 1) if score is not None else None
     return {
         "score": rounded,
-        "band": band_for(rounded) if rounded is not None and coverage >= MIN_COVERAGE else None,
+        "band": (
+            band_for(rounded, bands)
+            if rounded is not None and coverage >= MIN_COVERAGE
+            else None
+        ),
+        "bands": {"watch": bands[0], "elevated": bands[1], "breaking": bands[2]},
         "coverage": round(coverage, 1),
         "minimum_coverage": MIN_COVERAGE,
         "available_pillars": [p["id"] for p in rows if p["available"]],
@@ -309,13 +435,24 @@ def _value_asof(series: Series | None, as_of: date) -> float | None:
     return item[1] if item else None
 
 
-def _window_asof(series: Series | None, as_of: date, years: float) -> list[float]:
+def _window_asof(series: Series | None, as_of: date, days: int) -> list[float]:
     if not series:
         return []
-    start = as_of - timedelta(days=int(365.25 * years))
+    start = as_of - timedelta(days=days)
     return [v for d, v in series if start <= d <= as_of]
 
 
+def _coverage_days(series: Series | None, as_of: date) -> int | None:
+    """Span of history actually available at ``as_of``.
+
+    Recorded in every snapshot because the v2 credit percentile degraded from a
+    10-year to a 3-year reference silently when the upstream licence changed --
+    nothing asserted the window it claimed, so nothing noticed for months.
+    """
+    dates = [d for d, _ in series or [] if d <= as_of]
+    return (as_of - dates[0]).days if dates else None
+
+
 def _next_weekday(d: date) -> date:
     candidate = d + timedelta(days=1)
     while candidate.weekday() >= 5:
@@ -340,19 +477,31 @@ def _fundamental_effective_date(overrides: dict) -> date | None:
     return _next_weekday(fetched) if fetched else None
 
 
-def _fundamental_scores_asof(overrides: dict, config: dict, as_of: date) -> tuple[float | None, float | None, dict]:
+def fundamental_overlay(overrides: dict, config: dict, as_of: date) -> dict:
+    """Point-in-time qualitative overlay. Never feeds State or Warning in v3.
+
+    The effective-date gate stays even though nothing is scored from this: the
+    400-session rebuild replays historical dates, and stamping today's LLM read
+    onto 2024 snapshots would be plain lookahead in the stored record.
+    """
     effective = _fundamental_effective_date(overrides)
-    if effective is None or as_of < effective:
-        return None, None, {"effective_date": effective.isoformat() if effective else None, "age_days": None}
-    age = (as_of - effective).days
-    stale = age > int(config.get("fundamental_staleness_days", 80))
-    f1 = overrides.get("f1_score")
-    f3 = overrides.get("f3_score")
-    return (
-        None if stale or f1 is None else _clamp(float(f1)),
-        None if stale or f3 is None else _clamp(float(f3)),
-        {"effective_date": effective.isoformat(), "age_days": age, "stale": stale},
-    )
+    pending = effective is None or as_of < effective
+    age = None if pending else (as_of - effective).days
+    stale = bool(age is not None and age > int(config.get("fundamental_staleness_days", 80)))
+    return {
+        "available": not pending and not stale,
+        "pending": pending,
+        "stale": stale,
+        "effective_date": effective.isoformat() if effective else None,
+        "age_days": age,
+        "capex": None if pending else overrides.get("capex"),
+        "good_news_stock_down": None if pending else overrides.get("good_news_stock_down"),
+        "capex_stress": None if pending else overrides.get("f1_score"),
+        "earnings_stress": None if pending else overrides.get("f3_score"),
+        "reasoning": None if pending else overrides.get("reasoning"),
+        "source": overrides.get("source"),
+        "fetched_at": overrides.get("fetched_at"),
+    }
 
 
 def _basket_hash(symbols: list[str]) -> str:
@@ -393,12 +542,14 @@ def _compute_index(
     vix_item = _item_asof(vix_series, as_of)
     vix_score = p5_volatility(vix_item[1] if vix_item else None)
     oas_item = _item_asof(oas_series, as_of)
-    oas_window = _window_asof(oas_series, as_of, HY_OAS_REFERENCE_YEARS)
+    oas_window = _window_asof(oas_series, as_of, HY_OAS_WINDOW_DAYS)
     credit_score = f2_credit_spreads(oas_window)
 
     divergence = _value_asof(divergence_series, as_of)
-    relative_strength = p4_relative_strength(smh, spy)
-    f1, f3, fundamental_meta = _fundamental_scores_asof(overrides, config, as_of)
+    sensors = warning_sensor_scores(divergence, smh, spy, oas_window)
+    relative_strength = sensors["relative_strength"]
+    credit_impulse = sensors["credit_impulse"]
+    overlay = fundamental_overlay(overrides, config, as_of)
 
     state_pillars = [
         {
@@ -444,21 +595,22 @@ def _compute_index(
             "sensors": [_sensor("W2", "60-session relative-strength deterioration", relative_strength)],
         },
         {
-            "id": "capex",
-            "label": "Hyperscaler capex revisions",
-            "score": round(f1, 1) if f1 is not None else None,
-            "sensors": [_sensor("F1", "Capex guidance cuts", f1)],
-        },
-        {
-            "id": "earnings_reaction",
-            "label": "Good news, stock down",
-            "score": round(f3, 1) if f3 is not None else None,
-            "sensors": [_sensor("F3", "Abnormal earnings reaction", f3)],
+            "id": "credit_impulse",
+            "label": "Credit impulse",
+            "score": round(credit_impulse, 1) if credit_impulse is not None else None,
+            "sensors": [
+                _sensor(
+                    "W3",
+                    f"HY OAS {W3_OAS_LOOKBACK}-session widening",
+                    credit_impulse,
+                    oas=oas_item[1] if oas_item else None,
+                )
+            ],
         },
     ]
 
-    state = _score_pillars(state_pillars, STATE_WEIGHTS)
-    warning = _score_pillars(warning_pillars, WARNING_WEIGHTS)
+    state = _score_pillars(state_pillars, STATE_WEIGHTS, STATE_BANDS)
+    warning = _score_pillars(warning_pillars, WARNING_WEIGHTS, WARNING_BANDS)
 
     price_item = _item_asof(prices.get(tickers["leaders"][0]), as_of)
     dated_sources = {
@@ -481,6 +633,7 @@ def _compute_index(
         "date": as_of.isoformat(),
         "state": state,
         "warning": warning,
+        "fundamental_overlay": overlay,
         "quadrant_config": {
             "state_divider": QUADRANT_STATE_DIVIDER,
             "warning_divider": QUADRANT_WARNING_DIVIDER,
@@ -502,14 +655,18 @@ def _compute_index(
             "breadth_pct_above_200": round(breadth_pct, 1) if breadth_pct is not None else None,
             "breadth_date": breadth_item[0].isoformat() if breadth_item else None,
             "fundamentals_fetched_at": overrides.get("fetched_at"),
-            "fundamentals_effective_date": fundamental_meta.get("effective_date"),
-            "fundamentals_age_days": fundamental_meta.get("age_days"),
+            "fundamentals_effective_date": overlay.get("effective_date"),
+            "fundamentals_age_days": overlay.get("age_days"),
         },
         "data_quality": {
             "minimum_coverage": MIN_COVERAGE,
             "oldest_market_input_age_days": max(source_ages.values()) if source_ages else None,
             "stale_inputs": stale_inputs,
             "inputs_fresh": not stale_inputs,
+            # Upstream history spans, so a provider silently truncating a series
+            # shows up in the record instead of quietly reshaping a sensor.
+            "credit_history_days": _coverage_days(oas_series, as_of),
+            "vix_history_days": _coverage_days(vix_series, as_of),
         },
     }
 
@@ -581,7 +738,11 @@ async def get_fundamental_overrides(db: AsyncSession) -> dict:
         stored = json.loads(raw)
     except (TypeError, ValueError):
         return default
-    if stored.get("methodology") != METHODOLOGY:
+    # The guard rejects pre-v2 blobs, where f1/f3 were arbitrary numbers with no
+    # categorical source. v2 and v3 share the categorical format and both derive
+    # f1/f3 from it below, so a methodology bump must not discard a live
+    # observation -- only the capex *scale* changed, and that is recomputed.
+    if stored.get("methodology") not in CATEGORICAL_FUNDAMENTAL_METHODOLOGIES:
         return default
     capex = _normalise_capex_states(stored.get("capex"), names)
     reaction = str(stored.get("good_news_stock_down", "mixed")).strip().lower()
@@ -745,7 +906,7 @@ async def _upsert_snapshot(
             created_at=datetime.now(timezone.utc),
         ))
     else:
-        existing_v2 = _parse_v2(row.breakdown_json)
+        existing_v2 = _parse_snapshot(row.breakdown_json)
         if existing_v2 is not None and not rewrite_existing_v2:
             return False, existing_v2
         row.total_score = float(state_score or 0.0)
@@ -754,7 +915,7 @@ async def _upsert_snapshot(
     return True, result
 
 
-def _parse_v2(raw: str) -> dict | None:
+def _parse_snapshot(raw: str) -> dict | None:
     try:
         parsed = json.loads(raw)
     except (TypeError, ValueError):
@@ -762,12 +923,12 @@ def _parse_v2(raw: str) -> dict | None:
     return parsed if parsed.get("methodology") == METHODOLOGY else None
 
 
-async def _latest_v2_row(db: AsyncSession) -> tuple[RegimeSnapshot, dict] | None:
+async def _latest_snapshot_row(db: AsyncSession) -> tuple[RegimeSnapshot, dict] | None:
     result = await db.execute(
         select(RegimeSnapshot).order_by(RegimeSnapshot.date.desc()).limit(1000)
     )
     for row in result.scalars().all():
-        parsed = _parse_v2(row.breakdown_json)
+        parsed = _parse_snapshot(row.breakdown_json)
         if parsed is not None:
             return row, parsed
     return None
@@ -791,8 +952,10 @@ async def update_regime_monitor(db: AsyncSession, rebuild_sessions: int = REBUIL
     latest_date = leader_series[-1][0]
 
     vix_series = await _fetch_fred_series("VIXCLS", end - timedelta(days=1200), end)
+    # Asking for 13 years was misleading once the licence capped the series at 3;
+    # the level needs the latest point and W3 needs its lookback, nothing more.
     oas_series = await _fetch_fred_series(
-        "BAMLH0A0HYM2", end - timedelta(days=int(365.25 * 13)), end
+        "BAMLH0A0HYM2", end - timedelta(days=HY_OAS_WINDOW_DAYS), end
     )
 
     basket = config["breadth_basket"]
@@ -805,7 +968,7 @@ async def update_regime_monitor(db: AsyncSession, rebuild_sessions: int = REBUIL
         logger.warning("Regime monitor: fixed-basket breadth skipped: %s", exc)
         breadth, breadth_counts, divergence = {}, {}, {}
 
-    latest_v2 = await _latest_v2_row(db)
+    latest_v2 = await _latest_snapshot_row(db)
     rebuilding = latest_v2 is None and bool(leader_series)
     if rebuilding:
         dates = [d for d, _ in leader_series[-max(1, rebuild_sessions):]]
@@ -860,7 +1023,7 @@ async def _result_at_or_before(
         .limit(1000)
     )
     for raw in result.scalars().all():
-        parsed = _parse_v2(raw)
+        parsed = _parse_snapshot(raw)
         parsed_hash = ((parsed or {}).get("basket") or {}).get("hash")
         if parsed is not None and (basket_hash is None or parsed_hash == basket_hash):
             return parsed
@@ -877,7 +1040,7 @@ def _delta(current: dict, previous: dict | None) -> float | None:
 
 
 async def get_regime_monitor(db: AsyncSession) -> dict:
-    latest = await _latest_v2_row(db)
+    latest = await _latest_snapshot_row(db)
     if latest is None:
         return {"available": False, "reason": "v2 not computed yet"}
     row, result = latest
@@ -902,6 +1065,15 @@ async def get_regime_monitor(db: AsyncSession) -> dict:
     quality["snapshot_age_days"] = snapshot_age
     quality["is_fresh"] = bool(quality.get("inputs_fresh")) and snapshot_age <= 4
     result["data_quality"] = quality
+
+    # The snapshot's overlay is the point-in-time record; the reader also wants
+    # the current observation even when it is not effective until the next
+    # session, because otherwise refreshing it looks like it did nothing.
+    config = await get_regime_config(db)
+    overrides = await get_fundamental_overrides(db)
+    live = fundamental_overlay(overrides, config, date.today())
+    live["observed_in_snapshot"] = bool((result.get("fundamental_overlay") or {}).get("available"))
+    result["fundamental_context"] = live
     result["available"] = True
     return result
 
@@ -915,7 +1087,7 @@ async def get_regime_history(db: AsyncSession, days: int = 800) -> list[dict]:
     )
     out: list[dict] = []
     for row in result.scalars().all():
-        data = _parse_v2(row.breakdown_json)
+        data = _parse_snapshot(row.breakdown_json)
         if data is None:
             continue
         state, warning = data.get("state") or {}, data.get("warning") or {}

app/services/rr_scanner_service.py

@@ -27,10 +27,19 @@ from app.models.signal_context_snapshot import SignalContextSnapshot
 from app.models.ticker import Ticker
 from app.models.trade_setup import TradeSetup
 from app.services.indicator_service import _extract_ohlcv, compute_atr
+from app.services import fundamentals_quality_service, system_event_service
 from app.services.price_service import query_ohlcv
+from app.services.qualification import setup_qualifies
 from app.services.sr_service import detect_gate_target_ladder
-from app.services.trade_policy import get_reentry_lockdowns
+from app.services import settings_store
+from app.services.trade_policy import (
+    MANUAL_BOOK,
+    SHADOW_BOOK,
+    get_reentry_gate_locks,
+    observe_reentry_gate_transitions,
+)
 from app.services.recommendation_service import (
+    PRIMARY_TARGET_MIN_RR,
     _risk_level_from_conflicts,
     build_recommendation_snapshot,
     enhance_trade_setup,
@@ -39,8 +48,15 @@ from app.services.recommendation_service import (
 
 logger = logging.getLogger(__name__)
 
+# Markers of the most recent *successful* scan, written together only when
+# scan_all_tickers completes. COMPLETED gives its freshness; RUN_ID identifies
+# the run — the same id stamped on every setup row it produced. The shadow book
+# matches RUN_ID exactly and then selects setups by that id, so neither a
+# concurrent manual scan nor a stale prior run can be mistaken for it.
+KEY_LAST_SCAN_COMPLETED = "last_scan_run_completed_at"
+KEY_LAST_SCAN_RUN_ID = "last_scan_run_id"
+
 STRATEGY_VERSION = "residual_highvol_80_20_atr_trail3_v1"
-PRIMARY_TARGET_MIN_RR = 1.5
 
 # A setup counts as live only while the daily scan keeps re-emitting it. The
 # scan runs every day (07:00 UTC cron), so anything older than this was NOT
@@ -510,6 +526,8 @@ async def scan_ticker(
     volatility_percentile: float | None = None,
     primary_min_rr: float | None = None,
     gate_levels_override: list[Any] | None = None,
+    scan_run_id: str | None = None,
+    fundamentals_eligible: bool | None = None,
 ) -> list[TradeSetup]:
     """Scan a single ticker for trade setups meeting the R:R threshold.
 
@@ -526,6 +544,17 @@ async def scan_ticker(
     """
     ticker = await _get_ticker(db, symbol)
 
+    if fundamentals_eligible is None:
+        fundamentals_eligible = await fundamentals_quality_service.ticker_is_eligible(
+            db, ticker.id
+        )
+    if not fundamentals_eligible:
+        logger.info(
+            "Skipping %s: unresolved or unavailable SEC fundamentals",
+            ticker.symbol,
+        )
+        return []
+
     if primary_min_rr is None:
         primary_min_rr = PRIMARY_TARGET_MIN_RR
 
@@ -676,6 +705,9 @@ async def scan_ticker(
             enhanced_setups.append(setup)
 
     for setup in enhanced_setups:
+        # Stamp identity after enhancement so it survives regardless of how the
+        # enhancer rebuilds the row; the shadow book selects its batch by this.
+        setup.scan_run_id = scan_run_id
         db.add(setup)
 
     await db.commit()
@@ -700,12 +732,47 @@ async def scan_all_tickers(
     ``progress_callback(processed, total, current_symbol)`` is invoked as each
     ticker is scanned so callers (e.g. the scheduler) can surface live progress.
     """
-    # Plain strings, not Ticker instances: the rollbacks below expire any ORM
-    # objects held across them, and touching an expired attribute afterwards
+    # Plain ids/strings, not Ticker instances: the rollbacks below expire any
+    # ORM objects held across them, and touching an expired attribute afterwards
     # triggers sync lazy-loading, which raises on an AsyncSession.
-    result = await db.execute(select(Ticker.symbol).order_by(Ticker.symbol))
-    symbols = list(result.scalars().all())
-    total = len(symbols)
+    result = await db.execute(select(Ticker.id, Ticker.symbol).order_by(Ticker.symbol))
+    ticker_rows = [(int(ticker_id), symbol) for ticker_id, symbol in result.all()]
+    total = len(ticker_rows)
+
+    # Data-quality failures are not weak signals: they make a ticker ineligible.
+    # Resolve once for the universe scan and pass the decision into scan_ticker.
+    try:
+        fundamentals_blocked_ids = (
+            await fundamentals_quality_service.blocked_ticker_ids(db)
+        )
+    except Exception:
+        await db.rollback()
+        logger.exception(
+            "Could not resolve fundamentals quality; blocking this scan closed"
+        )
+        await system_event_service.log_event_standalone(
+            severity="error",
+            source="rr_scanner",
+            code="fundamentals_quality_unavailable",
+            message=(
+                "The fundamentals quality gate could not be evaluated; the "
+                "universe scan was blocked to avoid issuing unchecked setups."
+            ),
+            dedup_key="rr_scanner:fundamentals_quality_unavailable",
+        )
+        fundamentals_blocked_ids = {ticker_id for ticker_id, _ in ticker_rows}
+
+    # Gate-reset observations must use the same runtime activation settings as
+    # the live setup list. If the config cannot be loaded, scan normally but do
+    # not mutate reset state from an evaluation whose rules are unknown.
+    activation: dict | None = None
+    try:
+        from app.services.admin_service import get_activation_config
+
+        activation = await get_activation_config(db)
+    except Exception:
+        await db.rollback()
+        logger.exception("Activation config load for re-entry gate reset failed")
 
     # Rank the universe up front so each new setup carries both the residual
     # activation gate percentile and the promoted production ordering score.
@@ -721,17 +788,34 @@ async def scan_all_tickers(
         ranks = {}
 
     all_setups: list[TradeSetup] = []
-    for index, symbol in enumerate(symbols):
+    evaluated_ticker_ids: set[int] = set()
+    qualified_ticker_ids: set[int] = set()
+    gate_observation_started_at = datetime.now(timezone.utc)
+    # One id for the whole run: stamped on every setup row and written to the
+    # completion marker, so the shadow book can select this run's batch by
+    # identity. From the pipeline when run as its scan step; a fresh id (never
+    # matching any pipeline's) when triggered standalone.
+    from app.services import pipeline_run
+
+    scan_run_id = pipeline_run.current() or pipeline_run.new_run_id()
+    for index, (ticker_id, symbol) in enumerate(ticker_rows):
         if progress_callback is not None:
             progress_callback(index, total, symbol)
-        # Refresh scores first so the scheduled scan works off current data.
-        # Nothing else marks scores stale, so without this they'd never update
-        # for tickers the user doesn't manually fetch. A refresh failure still
-        # scans the ticker: qualification re-gates on live scores at alert
-        # time, so a stale score is recoverable but a skipped scan is not.
+        if ticker_id in fundamentals_blocked_ids:
+            logger.info(
+                "Skipping %s: unresolved or unavailable SEC fundamentals",
+                symbol,
+            )
+            continue
+        # Refresh Structural S/R once, then scores. get_sr_levels is read-only;
+        # without this recalculate the score path would see yesterday's zones.
+        # A refresh failure still scans the ticker: qualification re-gates on
+        # live scores at alert time, so a stale score is recoverable but a
+        # skipped scan is not.
         try:
-            from app.services import scoring_service
+            from app.services import scoring_service, sr_service
 
+            await sr_service.recalculate_sr_levels(db, symbol)
             await scoring_service.compute_all_dimensions(db, symbol)
             await scoring_service.compute_composite_score(db, symbol)
             await db.commit()
@@ -752,15 +836,56 @@ async def scan_all_tickers(
                 strategy_rank=(ranks.get(symbol) or {}).get("strategy_rank"),
                 volatility_percentile=(ranks.get(symbol) or {}).get("volatility_percentile"),
                 primary_min_rr=PRIMARY_TARGET_MIN_RR,
+                scan_run_id=scan_run_id,
+                fundamentals_eligible=True,
             )
             all_setups.extend(setups)
+            if activation is not None:
+                try:
+                    if any(setup_qualifies(setup, activation) for setup in setups):
+                        qualified_ticker_ids.add(ticker_id)
+                    evaluated_ticker_ids.add(ticker_id)
+                except Exception:
+                    logger.exception(
+                        "Gate-reset qualification observation failed for %s", symbol
+                    )
         except Exception:
             await db.rollback()
             logger.exception("Error scanning ticker %s", symbol)
 
+    if activation is not None:
+        # Both books, from the same observation: gate-reset state is per book,
+        # so observing only the manual book would leave shadow stop-outs stuck
+        # with a fail timestamp that never requalifies — permanently ineligible.
+        transitioned_ticker_ids: set[int] = set()
+        for book in (MANUAL_BOOK, SHADOW_BOOK):
+            transitioned_ticker_ids |= await observe_reentry_gate_transitions(
+                db,
+                evaluated_ticker_ids=evaluated_ticker_ids,
+                qualified_ticker_ids=qualified_ticker_ids,
+                observed_at=gate_observation_started_at,
+                book=book,
+            )
+        await db.commit()
+        if transitioned_ticker_ids:
+            logger.info(
+                "Updated post-stop gate-reset state for %d ticker(s)",
+                len(transitioned_ticker_ids),
+            )
+
     if progress_callback is not None and total:
         progress_callback(total, total, "")
 
+    # Publish the run markers only now that the scan has completed: COMPLETED for
+    # freshness and RUN_ID (the same id stamped on this run's setup rows) for
+    # identity, in one commit. A hard failure above leaves the previous,
+    # now-superseded, markers in place — so the shadow book will not match.
+    await settings_store.upsert_setting(
+        db, KEY_LAST_SCAN_COMPLETED, datetime.now(timezone.utc).isoformat()
+    )
+    await settings_store.upsert_setting(db, KEY_LAST_SCAN_RUN_ID, scan_run_id)
+    await db.commit()
+
     return all_setups
 
 
@@ -772,8 +897,9 @@ async def get_trade_setups(
     symbol: str | None = None,
     live_recommendation: bool = False,
     exclude_open_trade_tickers: bool = False,
-    exclude_reentry_lockdown_tickers: bool = False,
-    include_reentry_lockdown: bool = False,
+    exclude_open_trade_user_id: int | None = None,
+    exclude_reentry_gate_locked_tickers: bool = False,
+    include_reentry_gate_lock: bool = False,
 ) -> list[dict]:
     """Get latest stored trade setups, optionally filtered.
 
@@ -798,20 +924,52 @@ async def get_trade_setups(
     if recommended_action is not None and not live_recommendation:
         stmt = stmt.where(TradeSetup.recommended_action == recommended_action)
     excluded_ticker_ids: set[int] = set()
-    reentry_lockdowns: dict[int, int] = {}
+    reentry_gate_locks: dict[int, datetime] = {}
+    try:
+        excluded_ticker_ids.update(
+            await fundamentals_quality_service.blocked_ticker_ids(db)
+        )
+    except Exception:
+        await db.rollback()
+        logger.exception(
+            "Could not resolve fundamentals quality; hiding actionable setups"
+        )
+        await system_event_service.log_event_standalone(
+            severity="error",
+            source="rr_scanner",
+            code="fundamentals_quality_unavailable",
+            message=(
+                "The fundamentals quality gate could not be evaluated; actionable "
+                "setups were hidden until the metadata check recovers."
+            ),
+            dedup_key="rr_scanner:fundamentals_quality_unavailable",
+        )
+        return []
     if exclude_open_trade_tickers:
-        open_trade_result = await db.execute(
+        # Manual book only. The shadow book holds the *top-ranked* names by
+        # construction, so letting its positions hide setups would leave the
+        # discretionary list picking over leftovers — and would bias the very
+        # shadow-vs-manual comparison the shadow book exists to measure.
+        open_trade_stmt = (
             select(PaperTrade.ticker_id)
-            .where(PaperTrade.status == "open")
+            .where(PaperTrade.status == "open", PaperTrade.book == MANUAL_BOOK)
             .distinct()
         )
+        # Scope to one user for the personal setup list (don't hide a name just
+        # because someone else holds it); leave it global for the Telegram
+        # broadcast, which has no single owner.
+        if exclude_open_trade_user_id is not None:
+            open_trade_stmt = open_trade_stmt.where(
+                PaperTrade.user_id == exclude_open_trade_user_id
+            )
+        open_trade_result = await db.execute(open_trade_stmt)
         excluded_ticker_ids.update(
             ticker_id for ticker_id, in open_trade_result.all()
         )
-    if exclude_reentry_lockdown_tickers or include_reentry_lockdown:
-        reentry_lockdowns = await get_reentry_lockdowns(db)
-    if exclude_reentry_lockdown_tickers:
-        excluded_ticker_ids.update(reentry_lockdowns)
+    if exclude_reentry_gate_locked_tickers or include_reentry_gate_lock:
+        reentry_gate_locks = await get_reentry_gate_locks(db)
+    if exclude_reentry_gate_locked_tickers:
+        excluded_ticker_ids.update(reentry_gate_locks)
     if excluded_ticker_ids:
         stmt = stmt.where(~TradeSetup.ticker_id.in_(excluded_ticker_ids))
 
@@ -866,14 +1024,14 @@ async def get_trade_setups(
             ),
             reverse=True,
         )
-    if include_reentry_lockdown:
+    if include_reentry_gate_lock:
         ticker_by_setup_id = {
             setup.id: setup.ticker_id for setup, _ in latest_rows
         }
         for row in rows_out:
             ticker_id = ticker_by_setup_id.get(row["id"])
-            row["reentry_lockdown_remaining_sessions"] = (
-                reentry_lockdowns.get(ticker_id) if ticker_id is not None else None
+            row["reentry_gate_reset_required"] = (
+                ticker_id in reentry_gate_locks if ticker_id is not None else False
             )
     return rows_out
 

app/services/sec_client.py

@@ -0,0 +1,398 @@
+"""Async SEC EDGAR client for the fundamentals importer (workstream A).
+
+All access is batch (never at request time). This wraps the three SEC products
+the A3 design uses — `company_tickers.json`, `submissions/`, `companyfacts/`, and
+the daily filing index — behind one client that honors SEC's fair-access policy:
+
+- an identifying ``User-Agent`` with a contact email on every request (config);
+- request spacing well under the 10 req/s limit;
+- exponential backoff + retry on 429;
+- **403 → alert and stop** (raise ``SecForbiddenError``), never a retry-loop — a
+  403 means the UA or request pattern is wrong and retrying won't fix it. The one
+  exception is S3's ``AccessDenied`` on an ``/Archives/`` path, which is how the
+  bucket reports an absent file (``_is_absent_archive_key``).
+
+Parsing lives here (index fixed-width, submissions pagination); DB writes and the
+snapshot mapping live in the importer. No conditional GETs — the companyfacts
+endpoint exposes no ETag/Last-Modified (verified), which is why the importer is
+daily-index driven rather than polling archives.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import logging
+import os
+import re
+from datetime import date, datetime
+from pathlib import Path
+from typing import Any
+
+import httpx
+
+from app.config import settings
+from app.exceptions import ProviderError
+from app.services.earnings_alignment import normalise_symbol
+
+logger = logging.getLogger(__name__)
+
+_WWW = "https://www.sec.gov"
+_DATA = "https://data.sec.gov"
+
+# Resolve CA bundle for explicit httpx verify (matches app/providers/fmp.py).
+_CA = os.environ.get("SSL_CERT_FILE", "")
+_CA_VERIFY: str | bool = _CA if _CA and Path(_CA).exists() else True
+
+_FORMS_10 = frozenset({"10-K", "10-Q", "10-K/A", "10-Q/A"})
+
+
+class SecError(ProviderError):
+    """SEC request failed (403, exhausted 429/5xx, timeout, transport, parse)."""
+
+
+class SecForbiddenError(SecError):
+    """SEC returned 403 — User-Agent/pattern rejected. Alert and stop."""
+
+
+class SecNotFoundError(SecError):
+    """The resource does not exist (e.g. no daily index published for a day).
+
+    The *only* error a caller may treat as 'missing' — every other SecError
+    (fair-access rejection, exhausted retries, 5xx, timeout) must propagate so a
+    fetch failure is never mistaken for an empty result.
+
+    Raised for a 404, and for the one 403 that also means "absent": see
+    ``_is_absent_archive_key``."""
+
+
+def _is_absent_archive_key(url: str, resp: httpx.Response) -> bool:
+    """True when a 403 means "this file does not exist", not "you are blocked".
+
+    ``www.sec.gov/Archives`` is served straight out of an S3 bucket that grants
+    no ``s3:ListBucket``, so a missing key cannot be answered with 404 — S3
+    returns **403 with its ``AccessDenied`` XML** instead. SEC publishes a daily
+    index only for business days, so every weekend and market holiday inside an
+    incremental walk lands on exactly this response (verified 2026-07-30:
+    ``form.20260725.idx``, a Saturday, 403s while the Friday and Monday files
+    return 200 on the same User-Agent).
+
+    A genuine fair-access rejection is distinguishable and must stay fatal: it is
+    SEC's WAF interstitial — ``text/html``, "Your Request Originates from an
+    Undeclared Automated Tool" — and it is returned for files that *do* exist,
+    on any path. Hence the narrow gate: the Archives prefix plus S3's own error
+    document. Nothing else may be downgraded to "missing"."""
+    try:
+        parsed = httpx.URL(url)
+    except (TypeError, ValueError):  # pragma: no cover — url comes from us
+        return False
+    if parsed.host != "www.sec.gov" or not parsed.path.startswith("/Archives/"):
+        return False
+    if "xml" not in resp.headers.get("Content-Type", "").lower():
+        return False
+    try:
+        return "<Code>AccessDenied</Code>" in resp.text
+    except (UnicodeDecodeError, httpx.HTTPError):  # pragma: no cover
+        return False
+
+
+def _looks_like_contact_email(ua: str) -> bool:
+    if "example.com" in ua.lower() or "set-a-real-email" in ua.lower():
+        return False
+    return re.search(r"[^@\s]+@[^@\s]+\.[^@\s]+", ua) is not None
+
+
+# SEC asks callers to stay well under 10 req/s; enforce a floor on real clients.
+_MIN_PROD_SPACING = 0.11
+
+
+def cik10(cik: int | str) -> str:
+    """Zero-pad a CIK to the 10-digit form SEC URLs use (320193 -> 0000320193)."""
+    return str(int(cik)).zfill(10)
+
+
+class SecClient:
+    """Fair-access SEC HTTP client. Use as ``async with SecClient() as c:``."""
+
+    def __init__(
+        self,
+        *,
+        user_agent: str | None = None,
+        spacing_seconds: float | None = None,
+        max_retries: int | None = None,
+        timeout: float | None = None,
+        transport: httpx.AsyncBaseTransport | None = None,
+    ) -> None:
+        self._ua = user_agent or settings.sec_user_agent
+        self._spacing = (
+            spacing_seconds if spacing_seconds is not None else settings.sec_request_spacing_seconds
+        )
+        self._max_retries = (
+            max_retries if max_retries is not None else settings.sec_max_retries
+        )
+        self._timeout = timeout if timeout is not None else settings.sec_request_timeout_seconds
+        self._transport = transport  # injectable for tests
+        self._client: httpx.AsyncClient | None = None
+        self._lock = asyncio.Lock()
+        self._last_request = 0.0
+
+    def _validate_fair_access(self) -> None:
+        """On a real (non-mocked) client, enforce SEC fair-access preconditions
+        so we can't accidentally hammer SEC or get 403'd: a genuine contact-email
+        User-Agent and a spacing floor. Mock transports skip this (tests use 0)."""
+        if not _looks_like_contact_email(self._ua):
+            raise SecError(
+                "sec_user_agent must contain a real contact email (got "
+                f"{self._ua!r}) — SEC fair-access requires it"
+            )
+        if self._spacing < _MIN_PROD_SPACING:
+            raise SecError(
+                f"sec_request_spacing_seconds {self._spacing} is below the "
+                f"{_MIN_PROD_SPACING}s fair-access floor"
+            )
+
+    async def __aenter__(self) -> "SecClient":
+        if self._transport is None:
+            self._validate_fair_access()
+        self._client = httpx.AsyncClient(
+            headers={"User-Agent": self._ua, "Accept-Encoding": "gzip, deflate"},
+            timeout=self._timeout,
+            verify=_CA_VERIFY,
+            transport=self._transport,
+        )
+        return self
+
+    async def __aexit__(self, *exc) -> None:
+        if self._client is not None:
+            await self._client.aclose()
+            self._client = None
+
+    async def _throttle(self) -> None:
+        async with self._lock:
+            now = asyncio.get_event_loop().time()
+            wait = self._spacing - (now - self._last_request)
+            if wait > 0:
+                await asyncio.sleep(wait)
+            self._last_request = asyncio.get_event_loop().time()
+
+    async def _get(self, url: str) -> httpx.Response:
+        assert self._client is not None, "use `async with SecClient()`"
+        attempt = 0
+        while True:
+            await self._throttle()
+            try:
+                resp = await self._client.get(url)
+            except (httpx.TimeoutException, httpx.TransportError) as exc:
+                attempt += 1
+                if attempt > self._max_retries:
+                    raise SecError(f"SEC network error for {url}: {exc}") from exc
+                await asyncio.sleep(min(2.0**attempt, 30.0))
+                continue
+
+            code = resp.status_code
+            if code == 403:
+                if _is_absent_archive_key(url, resp):
+                    raise SecNotFoundError(f"SEC 403/AccessDenied (absent) for {url}")
+                raise SecForbiddenError(
+                    f"SEC 403 for {url} — User-Agent/pattern rejected; set a real "
+                    "sec_user_agent contact email"
+                )
+            if code == 404:
+                raise SecNotFoundError(f"SEC 404 for {url}")
+            # 429 and 5xx are transient — retry with backoff, honoring Retry-After.
+            if code == 429 or 500 <= code < 600:
+                attempt += 1
+                if attempt > self._max_retries:
+                    raise SecError(f"SEC {code} after {self._max_retries} retries: {url}")
+                delay = _retry_after_seconds(resp) or min(2.0**attempt, 30.0)
+                logger.warning("SEC %d for %s — backoff %.1fs (attempt %d)", code, url, delay, attempt)
+                await asyncio.sleep(delay)
+                continue
+            if code >= 400:
+                raise SecError(f"SEC {code} for {url}")
+            return resp
+
+    async def get_json(self, url: str) -> Any:
+        return (await self._get(url)).json()
+
+    async def get_text(self, url: str) -> str:
+        return (await self._get(url)).text
+
+    # -- domain fetchers ---------------------------------------------------
+
+    async def company_tickers(self) -> dict[str, int]:
+        """Map normalised ticker -> CIK (int). Multi-class tickers share a CIK."""
+        data = await self.get_json(f"{_WWW}/files/company_tickers.json")
+        out: dict[str, int] = {}
+        for row in data.values():
+            sym = normalise_symbol(row.get("ticker"))
+            if sym:
+                out[sym] = int(row["cik_str"])
+        return out
+
+    async def submissions(self, cik: int | str, *, include_history: bool = False) -> dict[str, Any]:
+        """Issuer metadata + filing list.
+
+        ``filings.recent`` caps at 1000; older accessions live in
+        ``filings.files[]`` shards. Only ``include_history=True`` (the one-time
+        full backfill) fetches those shards — SIC refresh and incremental runs
+        use the recent list alone and make no extra requests.
+        """
+        base = await self.get_json(f"{_DATA}/submissions/CIK{cik10(cik)}.json")
+        filings = _rows_from_arrays(base["filings"]["recent"])
+        if include_history:
+            for shard in base["filings"].get("files") or []:
+                shard_data = await self.get_json(f"{_DATA}/submissions/{shard['name']}")
+                filings.extend(_rows_from_arrays(shard_data))
+        return {
+            "cik": int(base["cik"]),
+            "name": base.get("name"),
+            "sic": base.get("sic"),
+            "sic_description": base.get("sicDescription"),
+            "fiscal_year_end": base.get("fiscalYearEnd"),
+            "tickers": base.get("tickers") or [],
+            "filings": filings,
+        }
+
+    async def companyfacts(self, cik: int | str) -> dict[str, Any]:
+        """Raw companyfacts JSON ({cik, entityName, facts})."""
+        return await self.get_json(f"{_DATA}/api/xbrl/companyfacts/CIK{cik10(cik)}.json")
+
+    async def latest_index_date(self, today: date | None = None) -> date | None:
+        """The most recent published daily-index date (drives the revision). Checks
+        the current quarter, falling back to the previous one at a quarter boundary."""
+        today = today or date.today()
+        for year, qtr in _quarters_back(today, 2):
+            url = f"{_WWW}/Archives/edgar/daily-index/{year}/QTR{qtr}/index.json"
+            try:
+                idx = await self.get_json(url)
+            except SecNotFoundError:
+                continue  # quarter dir absent — only 404 is "missing"
+            dates = [
+                d
+                for item in idx.get("directory", {}).get("item", [])
+                if (d := _index_file_date(item.get("name", ""))) is not None
+                and d <= today
+            ]
+            if dates:
+                return max(dates)
+        return None
+
+    async def daily_index(self, day: date) -> list[dict[str, Any]]:
+        """Parse the daily form index into 10-K/10-Q(/A) rows for all issuers.
+
+        Returns [{form, cik, accession, company}]. The caller filters to the
+        tracked universe. A missing index (weekend/holiday/not-yet-published)
+        returns [] rather than raising.
+        """
+        qtr = (day.month - 1) // 3 + 1
+        url = f"{_WWW}/Archives/edgar/daily-index/{day.year}/QTR{qtr}/form.{day:%Y%m%d}.idx"
+        try:
+            text = await self.get_text(url)
+        except SecNotFoundError:
+            # Absent on a weekend is routine (SEC publishes business days only); on a
+            # weekday it is either a market holiday or something worth a look — a SEC
+            # hiccup, or a rejection page misread as absent, would otherwise let the
+            # importer advance past real filings silently. Log-level only, no alert:
+            # cheaper than carrying a holiday calendar just to stay quiet ~10 days/yr.
+            logger.log(
+                logging.INFO if day.weekday() >= 5 else logging.WARNING,
+                "no daily index published for %s (%s)",
+                day,
+                f"{day:%a}",
+            )
+            return []  # weekend/holiday/not-yet-published; other errors propagate
+        return _parse_form_index(text)
+
+
+def _rows_from_arrays(arrays: dict[str, list]) -> list[dict[str, Any]]:
+    """Turn SEC's parallel-array filing block into row dicts (keeping only 10-K/10-Q
+    family filings — the ones that carry XBRL fundamentals)."""
+    forms = arrays.get("form", [])
+    out: list[dict[str, Any]] = []
+    for i, form in enumerate(forms):
+        if form not in _FORMS_10:
+            continue
+        out.append(
+            {
+                "accession": arrays["accessionNumber"][i],
+                "form": form,
+                "report_date": arrays["reportDate"][i] or None,
+                "filing_date": arrays["filingDate"][i] or None,
+                "acceptance_datetime": arrays["acceptanceDateTime"][i] or None,
+                "is_xbrl": bool(arrays.get("isXBRL", [0] * len(forms))[i]),
+            }
+        )
+    return out
+
+
+def _parse_form_index(text: str) -> list[dict[str, Any]]:
+    """Parse a daily ``form.YYYYMMDD.idx`` (fixed columns: Form / Company / CIK /
+    Date Filed / File Name-with-accession)."""
+    rows: list[dict[str, Any]] = []
+    started = False
+    for line in text.splitlines():
+        if not started:
+            if set(line.strip()) == {"-"}:  # the dashed separator row
+                started = True
+            continue
+        parts = line.split()
+        if len(parts) < 5:
+            continue
+        form = parts[0]
+        if form not in _FORMS_10:
+            continue
+        path = parts[-1]  # edgar/data/<cik>/<accession>.txt
+        cik = _cik_from_path(path)
+        accession = _accession_from_path(path)
+        if cik is None or accession is None:
+            continue
+        rows.append({"form": form, "cik": cik, "accession": accession, "path": path})
+    return rows
+
+
+def _retry_after_seconds(resp: httpx.Response) -> float | None:
+    """Parse a numeric-seconds Retry-After header (SEC uses seconds), capped."""
+    raw = resp.headers.get("Retry-After")
+    if not raw:
+        return None
+    try:
+        return min(float(raw), 60.0)
+    except (TypeError, ValueError):
+        return None
+
+
+def _index_file_date(name: str) -> date | None:
+    if name.startswith("form.") and name.endswith(".idx"):
+        try:
+            return datetime.strptime(name[5:13], "%Y%m%d").date()
+        except ValueError:
+            return None
+    return None
+
+
+def _quarters_back(today: date, n: int) -> list[tuple[int, int]]:
+    """(year, quarter) for `today`'s quarter and the previous n-1, newest first."""
+    q = (today.month - 1) // 3 + 1
+    out = []
+    y = today.year
+    for _ in range(n):
+        out.append((y, q))
+        q -= 1
+        if q == 0:
+            q = 4
+            y -= 1
+    return out
+
+
+def _cik_from_path(path: str) -> int | None:
+    segs = path.split("/")
+    if len(segs) >= 3 and segs[2].isdigit():
+        return int(segs[2])
+    return None
+
+
+def _accession_from_path(path: str) -> str | None:
+    stem = path.rsplit("/", 1)[-1]
+    if stem.endswith(".txt"):
+        stem = stem[:-4]
+    return stem or None

app/services/sec_facts_parser.py

@@ -0,0 +1,557 @@
+"""Pure parser: SEC companyfacts -> fundamental_snapshots rows.
+
+Turns one issuer's `companyfacts` JSON (+ its submissions filing metadata) into
+per-accession snapshot rows for the filing's **primary period**, following the
+A3 design (docs/dolt-sec-a3-design.md). No I/O, no DB — unit-testable against a
+fixture and verifiable against a real companyfacts pull.
+
+The load-bearing rules (design Decision 2 + review):
+- Period identity comes from `end == submissions.reportDate`, never `fy/fp`
+  (fy/fp is the *filing's* context; comparatives inside a filing repeat it).
+  This applies to the stored `fiscal_year`/`fiscal_period` too: they are derived
+  from `reportDate` against the issuer's `fiscalYearEnd` (see `_period_identity`),
+  because SEC's fy/fp collide and invert often enough to break the quarter chain.
+- Duration facts are stored as **cumulative YTD**: pick the fact whose span
+  matches the fiscal-period-to-date length (Q1≈3mo … FY≈12mo) within tolerance.
+  If no YTD-length fact exists, store null — never a discrete masquerading as YTD.
+- Balance-sheet instants are taken at `end == reportDate`. `shares_outstanding`
+  is a single consolidated value: the cover-page `dei` fact (its own cover-date
+  `end` stored separately) if present, else `us-gaap:CommonStockSharesOutstanding`
+  at period end (e.g. Alphabet has no `dei` fact) — never a class sum or the
+  weighted-average/diluted count. Multi-class issuers report it per class, which
+  is dimensional and therefore absent from companyfacts entirely, so
+  `weighted_avg_diluted_shares` is stored alongside as an explicit fallback for
+  market cap — a separate column, never backfilled into `shares_outstanding`.
+- Cash and debt composites are aggregate-first and mutually exclusive (each
+  source tag counted at most once).
+
+`parse_snapshots` separates `skipped_filings` (no usable row produced) from
+`field_issues` (a row was produced but a field is null/ambiguous) — callers must
+not treat field issues as missing coverage.
+"""
+
+from __future__ import annotations
+
+import logging
+import math
+from dataclasses import dataclass, field
+from datetime import date, datetime, timedelta
+from typing import Any, NamedTuple
+
+logger = logging.getLogger(__name__)
+
+# Expected YTD span (days) per fiscal period; a duration fact must land within
+# tolerance of this to count as the period's cumulative value.
+_EXPECTED_YTD_DAYS = {"Q1": 91, "Q2": 182, "Q3": 273, "FY": 365}
+# Period identity (see _period_identity): how far a quarter end sits before its
+# fiscal-year end, and how far a fiscal-year end may drift from the nominal MMDD.
+# The quarter bands are 91 days apart, so ±35 stays unambiguous even for a 4-4-5
+# filer whose 16-week Q4 puts Q3 112 days out.
+_QUARTER_DAYS_TO_FY_END = {"Q1": 273, "Q2": 182, "Q3": 91}
+_QUARTER_TOLERANCE_DAYS = 35
+_FYE_DRIFT_TOLERANCE_DAYS = 21
+# Covers 52/53-week calendars *and* 4-4-5 retail ones (12/12/12/16 weeks), whose
+# YTD-Q3 is 36 weeks = 251-252 days and missed a 20-day tolerance by ~2 -- so
+# COST/PEP lost Q3 every year, breaking the quarter chain and nulling TTM + YoY.
+# Q1 84d, Q2 168d and FY 364d were always inside. Adjacent periods stay
+# unambiguous at 25 (66-116, 157-207, 248-298, 340-390).
+_YTD_TOLERANCE_DAYS = 25
+
+# us-gaap duration concepts (money), priority order; first present wins.
+_DURATION_USD = {
+    # Order is load-bearing (first present wins) and the tail entries are
+    # deliberately *appended*: every issuer that already resolved keeps the same
+    # concept, and only issuers that resolved to nothing gain a value.
+    #   - IncludingAssessedTax: REITs/consumer filers that tag only this variant
+    #     (e.g. ARE, KHC) reported no revenue at all.
+    #   - RevenuesNetOfInterestExpense: the banks' total-revenue tag. JPM/GS/WFC
+    #     tag it in every 10-Q and `Revenues` only (if at all) in the 10-K.
+    "revenue": [
+        "RevenueFromContractWithCustomerExcludingAssessedTax",
+        "Revenues",
+        "SalesRevenueNet",
+        "RevenueFromContractWithCustomerIncludingAssessedTax",
+        "RevenuesNetOfInterestExpense",
+    ],
+    "net_income": ["NetIncomeLoss"],
+    "operating_income": ["OperatingIncomeLoss"],
+    "cfo": [
+        "NetCashProvidedByUsedInOperatingActivities",
+        "NetCashProvidedByUsedInOperatingActivitiesContinuingOperations",
+    ],
+    "capex": [
+        "PaymentsToAcquirePropertyPlantAndEquipment",
+        "PaymentsToAcquireProductiveAssets",
+    ],
+    "depreciation_amortization": [
+        "DepreciationDepletionAndAmortization",
+        "DepreciationAmortizationAndAccretionNet",
+        "DepreciationAndAmortization",
+    ],
+}
+# unit USD/shares. Appended (not reordered) so any issuer that already resolved
+# keeps the same concept. REG tags only the continuing-operations variant on every
+# filing; FCX switches by form type -- EarningsPerShareDiluted in its 10-Qs, the
+# continuing-ops tag in its 10-K -- which nulled the FY row and killed Q4 + TTM.
+# The basic variants are a last resort for a period that tags no diluted EPS at
+# all (PPL's 2026 Q1). Basic ignores option/convert dilution so it slightly
+# overstates EPS (~1.2% for PPL), but only fires when diluted is entirely absent,
+# and high-dilution names always tag diluted -- so it never displaces a real one.
+_EPS_CONCEPTS = [
+    "EarningsPerShareDiluted",
+    "IncomeLossFromContinuingOperationsPerDilutedShare",
+    "EarningsPerShareBasic",
+    "IncomeLossFromContinuingOperationsPerBasicShare",
+]
+# Weighted-average diluted share count (unit "shares"), the market-cap fallback
+# for multi-class issuers whose cover-page count is dimensional and therefore
+# absent from companyfacts. Always present, since EPS is computed from it.
+_WEIGHTED_AVG_SHARE_CONCEPTS = [
+    "WeightedAverageNumberOfDilutedSharesOutstanding",
+    "WeightedAverageNumberOfSharesOutstandingBasicAndDiluted",
+]
+# us-gaap instant (balance-sheet) concepts, at end == reportDate.
+_CASH = ["CashAndCashEquivalentsAtCarryingValue"]
+_ST_INVESTMENTS = ["ShortTermInvestments", "MarketableSecuritiesCurrent"]  # pick one
+_LONG_TERM_DEBT_AGG = ["LongTermDebt"]
+_LONG_TERM_DEBT_PARTS = ["LongTermDebtNoncurrent", "LongTermDebtCurrent"]
+_SHORT_TERM_DEBT = ["ShortTermBorrowings", "CommercialPaper"]  # pick one
+
+
+class Fact(NamedTuple):
+    taxonomy: str
+    concept: str
+    unit: str
+    start: date | None  # None => instant
+    end: date
+    val: float
+    fy: int | None
+    fp: str | None
+
+
+@dataclass
+class SnapshotRow:
+    cik: str
+    accession: str
+    form: str
+    filed_date: date
+    accepted_at: datetime
+    period_end: date
+    fiscal_year: int
+    fiscal_period: str
+    period_start: date | None = None
+    revenue: float | None = None
+    net_income: float | None = None
+    operating_income: float | None = None
+    diluted_eps: float | None = None
+    cfo: float | None = None
+    capex: float | None = None
+    depreciation_amortization: float | None = None
+    cash_and_st_investments: float | None = None
+    total_debt: float | None = None
+    shares_outstanding: float | None = None
+    shares_outstanding_date: date | None = None
+    weighted_avg_diluted_shares: float | None = None
+
+
+@dataclass
+class FilingMeta:
+    report_date: date
+    filing_date: date
+    accepted_at: datetime
+    form: str
+
+
+@dataclass
+class ParseResult:
+    rows: list[SnapshotRow] = field(default_factory=list)
+    # accessions for which NO row was produced (no facts / no usable period).
+    skipped_filings: list[dict[str, str]] = field(default_factory=list)
+    # accessions with a row but a field-level warning (e.g. ambiguous shares).
+    field_issues: list[dict[str, str]] = field(default_factory=list)
+
+
+def parse_snapshots(
+    companyfacts: dict[str, Any],
+    filings: dict[str, FilingMeta],
+    accessions: set[str],
+    fiscal_year_end: str | None = None,
+) -> ParseResult:
+    """Build snapshot rows for ``accessions`` (those with facts + filing meta).
+
+    ``fiscal_year_end`` is the issuer's declared ``submissions.fiscalYearEnd``
+    (MMDD) and seeds period identity (see ``_period_identity``), making it
+    independent of SEC's unreliable fy/fp fields. It is only a hint: the issuer's
+    own 10-K period ends override it (see ``resolve_fiscal_year_end``). With
+    neither available, the old fy/fp behaviour is used.
+
+    ``skipped_filings`` = no row produced (missing facts/meta or no usable period
+    identity); ``field_issues`` = a row was produced but a field is null/ambiguous.
+    Callers must not use field issues as failed-row coverage.
+    """
+    cik = f"{int(companyfacts['cik']):010d}"
+    # The declared value is only a hint; the issuer's own 10-Ks are authoritative.
+    fiscal_year_end = resolve_fiscal_year_end(filings, fiscal_year_end)
+    by_accn = _index_by_accession(companyfacts)
+    result = ParseResult()
+    for accn in accessions:
+        meta = filings.get(accn)
+        facts = by_accn.get(accn)
+        if meta is None or not facts:
+            result.skipped_filings.append({"accession": accn, "reason": "no facts or filing metadata"})
+            continue
+        row, note = _parse_one(cik, accn, facts, meta, fiscal_year_end)
+        if row is None:
+            result.skipped_filings.append({"accession": accn, "reason": note or "unparseable"})
+            continue
+        result.rows.append(row)
+        if note:
+            result.field_issues.append({"accession": accn, "reason": note})
+    return result
+
+
+def companyfacts_accessions(companyfacts: dict[str, Any]) -> set[str]:
+    """Every accession that appears anywhere in a companyfacts payload — used by
+    the importer's index↔Company-Facts consistency gate."""
+    return set(_index_by_accession(companyfacts).keys())
+
+
+def _index_by_accession(companyfacts: dict[str, Any]) -> dict[str, list[Fact]]:
+    """One pass over companyfacts -> {accession: [Fact, ...]}."""
+    out: dict[str, list[Fact]] = {}
+    for taxonomy, concepts in companyfacts.get("facts", {}).items():
+        for concept, body in concepts.items():
+            for unit, facts in body.get("units", {}).items():
+                for f in facts:
+                    accn = f.get("accn")
+                    end = _d(f.get("end"))
+                    val = f.get("val")
+                    # Skip malformed facts so they can't be selected accidentally:
+                    # every usable fact needs an accession, an end date, and a
+                    # finite numeric value.
+                    if not accn or end is None or not _finite(val):
+                        continue
+                    out.setdefault(accn, []).append(
+                        Fact(
+                            taxonomy=taxonomy,
+                            concept=concept,
+                            unit=unit,
+                            start=_d(f.get("start")),
+                            end=end,
+                            val=val,
+                            fy=f.get("fy"),
+                            fp=f.get("fp"),
+                        )
+                    )
+    return out
+
+
+def _parse_one(
+    cik: str, accn: str, facts: list[Fact], meta: FilingMeta,
+    fiscal_year_end: str | None = None,
+) -> tuple[SnapshotRow | None, str | None]:
+    """Returns (row, note). row is None when there's no usable period identity;
+    note is a validation reason (row-skip reason when row is None, else a
+    field-level issue such as ambiguous shares)."""
+    fy, fp = _period_identity(meta, fiscal_year_end)
+    if fy is None or fp is None:
+        # No fiscal calendar, or a period the calendar cannot place (a transition
+        # period). Fall back to the filing's own context: an imperfect label still
+        # beats dropping the filing entirely.
+        fy, fp = _fiscal_context(facts, meta.report_date)
+    if fy is None or fp not in _EXPECTED_YTD_DAYS:
+        return None, "no usable period identity"
+
+    row = SnapshotRow(
+        cik=cik,
+        accession=accn,
+        form=meta.form,
+        filed_date=meta.filing_date,
+        accepted_at=meta.accepted_at,
+        period_end=meta.report_date,
+        fiscal_year=fy,
+        fiscal_period=fp,
+    )
+
+    # duration YTD facts (money) + EPS
+    for field_name, concepts in _DURATION_USD.items():
+        val, start = _select_ytd(facts, concepts, meta.report_date, fp, "USD")
+        setattr(row, field_name, val)
+        if field_name == "revenue" and start is not None:
+            row.period_start = start
+    eps, eps_start = _select_ytd(facts, _EPS_CONCEPTS, meta.report_date, fp, "USD/shares")
+    row.diluted_eps = eps
+    if row.period_start is None and eps_start is not None:
+        row.period_start = eps_start
+
+    # balance-sheet instants at reportDate
+    row.cash_and_st_investments = _compose_cash(facts, meta.report_date)
+    row.total_debt = _compose_debt(facts, meta.report_date)
+    shares, shares_date, ambiguous = _select_shares(facts, meta.report_date)
+    row.shares_outstanding = shares
+    row.shares_outstanding_date = shares_date
+    row.weighted_avg_diluted_shares = _select_weighted_avg_shares(facts, meta.report_date)
+
+    return row, ("ambiguous shares outstanding" if ambiguous else None)
+
+
+def resolve_fiscal_year_end(
+    filings: dict[str, FilingMeta], declared: str | None
+) -> str | None:
+    """The issuer's fiscal-year-end MMDD, preferring its own 10-K period ends.
+
+    ``submissions.fiscalYearEnd`` is *not* reliable: Franklin Resources (BEN)
+    declares 1231 while every one of its 10-Ks ends 09-30. Trusting it put BEN's
+    fiscal Q1 (Dec) 0 days from the claimed year end — matching no quarter band —
+    and labelled its fiscal Q2 (Mar) as Q1, colliding two periods on one key and
+    destroying the quarter chain.
+
+    A 10-K's reportDate **is** the fiscal year end by definition, so it wins
+    whenever one is available; the declared value is only a fallback for an issuer
+    with no annual filing in the set. The most recent 10-K is used, so an issuer
+    that changed its year end is measured against its current calendar.
+    """
+    annual = [m.report_date for m in filings.values() if m.form.startswith("10-K")]
+    if annual:
+        latest = max(annual)
+        return f"{latest.month:02d}{latest.day:02d}"
+    return declared
+
+
+def _period_identity(
+    meta: FilingMeta, fiscal_year_end: str | None
+) -> tuple[int | None, str | None]:
+    """(fiscal_year, fiscal_period) from the period end and the issuer's fiscal
+    calendar — never from the fy/fp fields.
+
+    SEC's fy/fp describe the *filing*, and they are unreliable as period identity:
+    observed in production, a 10-Q labelled ``FY`` (BXP), a year ending 2025-12-31
+    labelled 2024 (FRT, a December filer), a year ending 2025-06-27 labelled 2027
+    (STX), and four different period ends all labelled 2022 Q3 (PPL). Because
+    readers key on (fiscal_year, fiscal_period), colliding labels silently discard
+    a period and inverted ones scramble the quarter chain — nulling TTM and YoY.
+
+    ``period_end`` is authoritative, so identity is derived from it: the form
+    decides FY vs quarter, and distance to the fiscal-year end decides which
+    quarter. Labels need not match the issuer's own naming — a filer whose year
+    ends in early January (DPZ) shifts by one — they need to be unique, monotonic
+    and YoY-aligned, which is all the derivation asks of them. Nothing outside the
+    derivation reads these columns.
+
+    Known limitation: ``fiscalYearEnd`` is the issuer's *current* calendar, so a
+    company that has changed its fiscal year end gets its historical periods
+    measured against the new one. The quarter tolerance shunts most of those to
+    the fy/fp fallback, and a same-key collision resolves newest-wins, so the
+    failure mode is a degraded old year rather than a scrambled current one.
+    """
+    fy = _fiscal_year_of(meta.report_date, fiscal_year_end)
+    if fy is None:
+        return None, None
+    if meta.form.startswith("10-K"):
+        return fy, "FY"
+    nominal_end = _nominal_fy_end(fy, fiscal_year_end)
+    if nominal_end is None:
+        return None, None
+    remaining = (nominal_end - meta.report_date).days
+    best = min(
+        _QUARTER_DAYS_TO_FY_END,
+        key=lambda k: abs(_QUARTER_DAYS_TO_FY_END[k] - remaining),
+    )
+    if abs(_QUARTER_DAYS_TO_FY_END[best] - remaining) > _QUARTER_TOLERANCE_DAYS:
+        return None, None  # transition period or odd filing — let the caller fall back
+    return fy, best
+
+
+def _nominal_fy_end(year: int, fiscal_year_end: str | None) -> date | None:
+    """The issuer's nominal fiscal-year end in ``year`` from a MMDD string."""
+    if not fiscal_year_end or len(fiscal_year_end) != 4 or not fiscal_year_end.isdigit():
+        return None
+    month, day = int(fiscal_year_end[:2]), int(fiscal_year_end[2:])
+    if not 1 <= month <= 12 or not 1 <= day <= 31:
+        return None
+    while day > 28:  # 52/53-week ends land on 0229/0230/0231 in some filings
+        try:
+            return date(year, month, day)
+        except ValueError:
+            day -= 1
+    return date(year, month, day)
+
+
+def _fiscal_year_of(period_end: date, fiscal_year_end: str | None) -> int | None:
+    """Which fiscal year ``period_end`` belongs to.
+
+    A 52/53-week calendar's real year end drifts around the nominal MMDD (and can
+    cross the calendar year), so allow drift before rolling into the next year.
+    """
+    nominal = _nominal_fy_end(period_end.year, fiscal_year_end)
+    if nominal is None:
+        return None
+    return (
+        period_end.year
+        if period_end <= nominal + timedelta(days=_FYE_DRIFT_TOLERANCE_DAYS)
+        else period_end.year + 1
+    )
+
+
+def _fiscal_context(facts: list[Fact], report_date: date) -> tuple[int | None, str | None]:
+    """The filing's (fy, fp) taken as the majority context among the facts that
+    end at reportDate (the current-period facts, which share the filing's
+    context). Reject a tie so a conflicting context is never chosen arbitrarily."""
+    counts: dict[tuple[int, str], int] = {}
+    for f in facts:
+        if f.end == report_date and f.fy is not None and f.fp:
+            counts[(f.fy, f.fp)] = counts.get((f.fy, f.fp), 0) + 1
+    if not counts:
+        return None, None
+    ranked = sorted(counts.items(), key=lambda kv: kv[1], reverse=True)
+    if len(ranked) > 1 and ranked[0][1] == ranked[1][1]:
+        return None, None  # tie → conflicting contexts, reject
+    return ranked[0][0]
+
+
+def _select_ytd(
+    facts: list[Fact], concepts: list[str], report_date: date, fp: str, unit: str
+) -> tuple[float | None, date | None]:
+    """First present concept whose duration fact ends at reportDate and whose span
+    matches the fiscal-period-to-date length. Returns (val, period_start)."""
+    expected = _EXPECTED_YTD_DAYS[fp]
+    for concept in concepts:
+        best: Fact | None = None
+        best_diff: int | None = None
+        for f in facts:
+            if (
+                f.taxonomy != "us-gaap"
+                or f.concept != concept
+                or f.unit != unit
+                or f.start is None
+                or f.end != report_date
+            ):
+                continue
+            diff = abs((f.end - f.start).days - expected)
+            if diff <= _YTD_TOLERANCE_DAYS and (best_diff is None or diff < best_diff):
+                best, best_diff = f, diff
+        if best is not None:
+            return float(best.val), best.start
+    return None, None
+
+
+def _select_instant(facts: list[Fact], concepts: list[str], report_date: date) -> float | None:
+    """First present instant (balance-sheet) fact at end == reportDate, unit USD."""
+    for concept in concepts:
+        for f in facts:
+            if (
+                f.taxonomy == "us-gaap"
+                and f.concept == concept
+                and f.unit == "USD"
+                and f.start is None
+                and f.end == report_date
+            ):
+                return float(f.val)
+    return None
+
+
+def _compose_cash(facts: list[Fact], report_date: date) -> float | None:
+    cash = _select_instant(facts, _CASH, report_date)
+    st = _select_instant(facts, _ST_INVESTMENTS, report_date)  # first present of the two
+    if cash is None and st is None:
+        return None
+    return (cash or 0.0) + (st or 0.0)
+
+
+def _compose_debt(facts: list[Fact], report_date: date) -> float | None:
+    long_term = _select_instant(facts, _LONG_TERM_DEBT_AGG, report_date)
+    if long_term is None:
+        nc = _select_instant(facts, ["LongTermDebtNoncurrent"], report_date)
+        cur = _select_instant(facts, ["LongTermDebtCurrent"], report_date)
+        long_term = None if nc is None and cur is None else (nc or 0.0) + (cur or 0.0)
+    short_term = _select_instant(facts, _SHORT_TERM_DEBT, report_date)
+    if long_term is None and short_term is None:
+        return None
+    return (long_term or 0.0) + (short_term or 0.0)
+
+
+def _select_shares(
+    facts: list[Fact], report_date: date
+) -> tuple[float | None, date | None, bool]:
+    """Issuer-wide shares outstanding as a single consolidated value (never a
+    class sum — companyfacts is non-dimensional — and never weighted-average/
+    diluted). Returns (value, shares_date, ambiguous).
+
+    1. Prefer the `dei:EntityCommonStockSharesOutstanding` cover-page instant;
+       its own end is the shares date (cover date != period_end).
+    2. Else fall back to `us-gaap:CommonStockSharesOutstanding` at period end
+       (e.g. Alphabet has no dei fact); shares date = reportDate.
+    Conflicting values within the chosen source → (None, None, True) to be
+    counted in validation.
+    """
+    dei = [
+        f
+        for f in facts
+        if f.taxonomy == "dei"
+        and f.concept == "EntityCommonStockSharesOutstanding"
+        and f.unit == "shares"
+        and f.start is None
+    ]
+    if dei:
+        if len({f.val for f in dei}) > 1:
+            return None, None, True
+        best = max(dei, key=lambda f: f.end)
+        return float(best.val), best.end, False
+
+    gaap = [
+        f
+        for f in facts
+        if f.taxonomy == "us-gaap"
+        and f.concept == "CommonStockSharesOutstanding"
+        and f.unit == "shares"
+        and f.start is None
+        and f.end == report_date
+    ]
+    if gaap:
+        if len({f.val for f in gaap}) > 1:
+            return None, None, True
+        return float(gaap[0].val), report_date, False
+
+    return None, None, False  # simply absent — not a conflict
+
+
+def _select_weighted_avg_shares(facts: list[Fact], report_date: date) -> float | None:
+    """The most recent quarter's weighted-average diluted share count.
+
+    Deliberately the **shortest** duration ending at reportDate, not the YTD one:
+    the shorter the window the closer the average sits to the current count, which
+    is what a market cap wants. Measured against issuers where the true
+    point-in-time count is available, the quarter average is within ~0.6%.
+    """
+    best: tuple[int, float] | None = None
+    for concept in _WEIGHTED_AVG_SHARE_CONCEPTS:
+        for f in facts:
+            if (
+                f.taxonomy != "us-gaap"
+                or f.concept != concept
+                or f.unit != "shares"
+                or f.start is None
+                or f.end != report_date
+                or f.val <= 0
+            ):
+                continue
+            span = (f.end - f.start).days
+            if best is None or span < best[0]:
+                best = (span, float(f.val))
+        if best is not None:
+            return best[1]  # first present concept wins, as elsewhere
+    return None
+
+
+def _d(value: Any) -> date | None:
+    if not value:
+        return None
+    try:
+        return date.fromisoformat(str(value)[:10])
+    except ValueError:
+        return None
+
+
+def _finite(value: Any) -> bool:
+    """True for a finite numeric value (rejects None, bool, strings, NaN/inf)."""
+    return isinstance(value, (int, float)) and not isinstance(value, bool) and math.isfinite(value)

app/services/sec_fundamentals_importer.py

@@ -0,0 +1,991 @@
+"""SEC fundamentals importer (workstream A, phase A3).
+
+A ``SourceImporter`` (see ``app/services/data_import.py``) that populates the
+immutable ``fundamental_snapshots`` from SEC Company Facts and back-fills
+``tickers.cik/sic/sic_description``. EDGAR-daily-index driven: it fetches
+companyfacts only for tracked issuers that filed since the last run (full-history
+backfill on the first run / for newly-added issuers). Shadow only — nothing reads
+snapshots until A4.
+
+Guardrails (design + reviews):
+- ``detect_revision`` caches the resolved universe + the exact tracked index rows
+  and composes the revision from them; ``stage`` consumes those same cached inputs
+  (it does not refetch the index/universe) so promoted data always matches the
+  computed revision.
+- Resolution is read-only in ``stage`` (proposals only); ticker writes happen in
+  ``promote`` via ``apply_ticker_updates``.
+- ``validate`` runs the **index↔Company-Facts consistency gate** before any write:
+  a tracked XBRL index accession missing from Company Facts fails the run (the two
+  are separate SEC products that can lag) so we retry rather than record a
+  null/partial snapshot. Non-XBRL amendments are skipped with a recorded reason.
+  A failure here blocks every later run (``source_max_date`` only advances on a
+  promoted run), so it names the offending filings in the alert and separates the
+  causes — ``not_in_companyfacts`` (facts lag) vs ``not_in_submissions`` (the
+  index row is absent from the issuer's own filing list, which no retry fixes).
+- **Co-registrant recovery**, because "missing from Company Facts" is often not
+  missing at all: SEC files some combined parent/subsidiary filings' XBRL under
+  the co-registrant's CIK, so the ticker-carrying parent's own file never gets
+  that accession. The daily index lists every co-registrant of an accession, so
+  the facts are found there and re-stamped to the real filer — guarded by a
+  share-count continuity check so a subsidiary's standalone numbers can never be
+  stored as the parent's. Confirmed 2026-07-27 (NEE via FPL, DOW via Dow Chemical)
+  and it is not transient: an NEE filing misattributed in 2014 is still misfiled.
+- **Bounded blocking.** Anything still unresolvable after ``MISSING_XBRL_RETRY_DAYS``
+  stops failing the whole import and enters a durable retry queue. The scheduled
+  importer retries queued accessions automatically, while the affected issuer is
+  excluded from actionable setups until its filing is recovered.
+- ``promote`` inserts snapshots ``ON CONFLICT (accession) DO NOTHING`` (immutable),
+  reports differing existing accessions, and applies ticker updates in the same
+  transaction.
+- ``reparse=True`` is the one exception to immutability, and it is deliberate:
+  it restages every accession with the current parser and **rewrites** the rows
+  that now reconstruct differently. Immutability protects SEC's record (one row
+  per accession, amendments retained) — but the stored row is *our* reconstruction,
+  so after a parser fix, keeping it is preserving a stale cache, not history.
+  Manually invoked through ``scripts/reparse_fundamentals.py``; never scheduled.
+"""
+
+from __future__ import annotations
+
+import json
+import logging
+from collections import Counter, defaultdict
+from dataclasses import dataclass, field, replace
+from datetime import date, datetime, timedelta, timezone
+from typing import Any, Callable
+
+from sqlalchemy import delete, select, update
+
+from app.database import insert_for_session
+from app.models.data_import_run import DataImportRun
+from app.models.fundamental_snapshot import FundamentalSnapshot
+from app.models.sec_filing_gap import SecFilingGap
+from app.models.system_event import SystemEvent
+from app.services import fundamentals_quality_service
+from app.services import sec_facts_parser as parser
+from app.services import sec_universe
+from app.services.data_import import STATUS_PROMOTED, ValidationResult
+from app.services.sec_client import SecClient, SecError, cik10
+from app.services.sec_facts_parser import FilingMeta, SnapshotRow
+from app.services.sec_universe import ResolvedUniverse
+
+logger = logging.getLogger(__name__)
+
+SOURCE = "sec_facts"
+_XBRL_FORMS = {"10-K", "10-Q", "10-K/A", "10-Q/A"}
+# On the one-time backfill, require this fraction of tracked issuers to yield at
+# least one snapshot (guards a broken fetch/parse from promoting a hollow table).
+MIN_BACKFILL_COVERAGE = 0.5
+# How long an index accession may stay unresolvable before the run stops failing
+# on it. Genuine index↔facts lag clears within a day (a weekend stretches it to
+# three); past that it is misfiled, not late, and blocking forever costs more
+# than the missing filing does — see the unresolved-filing guardrail below.
+MISSING_XBRL_RETRY_DAYS = 3
+FILING_GAP_ESCALATE_DAYS = 14
+# Share-count band a co-registrant-recovered row must land in, relative to the
+# issuer's own last snapshot. Wide enough for buybacks/issuance, nowhere near
+# wide enough to let a subsidiary shell's token float through (see _shares_continuous).
+RECOVERY_SHARES_MIN = 0.5
+RECOVERY_SHARES_MAX = 2.0
+
+_SNAPSHOT_COLS = (
+    "cik", "accession", "form", "filed_date", "accepted_at", "period_start",
+    "period_end", "fiscal_year", "fiscal_period", "revenue", "net_income",
+    "operating_income", "diluted_eps", "cfo", "capex", "depreciation_amortization",
+    "cash_and_st_investments", "total_debt", "shares_outstanding",
+    "shares_outstanding_date", "weighted_avg_diluted_shares",
+)
+# Compare ALL source fields (every column except the accession key) to flag a
+# differing existing accession — immutable, so we report, never mutate.
+_COMPARE_COLS = tuple(c for c in _SNAPSHOT_COLS if c != "accession")
+
+
+@dataclass
+class StagedFundamentals:
+    resolved: ResolvedUniverse
+    sic_updates: list[tuple[int, str | None, str | None]] = field(default_factory=list)
+    rows: list[SnapshotRow] = field(default_factory=list)
+    skipped_filings: list[dict[str, str]] = field(default_factory=list)
+    field_issues: list[dict[str, str]] = field(default_factory=list)
+    skipped_non_xbrl: list[dict[str, str]] = field(default_factory=list)
+    # Index rows we could not resolve to Company Facts, with a per-row reason
+    # (not_in_companyfacts | not_in_submissions | ...) — see _missing().
+    missing_xbrl: list[dict[str, Any]] = field(default_factory=list)
+    # Accessions parsed out of a co-registrant's Company Facts file.
+    recovered: list[dict[str, Any]] = field(default_factory=list)
+    invalid_payloads: list[dict[str, str]] = field(default_factory=list)
+    existing_accessions: set[str] = field(default_factory=set)
+    # Tracked issuers whose registrant has NO XBRL 10-K/10-Q at all: they can
+    # never yield a snapshot, so this is a resolution problem (a ticker pointed
+    # at a successor shell), not missing data. See sec_universe.CIK_OVERRIDES_KEY.
+    no_xbrl_filings: list[dict[str, Any]] = field(default_factory=list)
+    discrepancies: list[dict[str, Any]] = field(default_factory=list)
+    backfill: bool = False
+    issuers_fetched: int = 0
+    issuers_with_rows: int = 0
+
+
+def _now() -> datetime:
+    return datetime.now(timezone.utc)
+
+
+class SecFundamentalsImporter:
+    source = SOURCE
+
+    def __init__(
+        self,
+        *,
+        client_factory: Callable[[], SecClient] | None = None,
+        today: date | None = None,
+        reparse: bool = False,
+    ) -> None:
+        self._client_factory = client_factory or (lambda: SecClient())
+        self.today = today or _now().date()
+        # Reparse: re-derive every stored accession with the CURRENT parser and
+        # rewrite the ones that now reconstruct differently. Snapshots are
+        # immutable with respect to SEC (one row per accession, amendments kept),
+        # but the stored row is *our reconstruction* — when a parser bug is fixed,
+        # leaving it stale is not immutability, it is a stale cache. Manually
+        # invoked via scripts/reparse_fundamentals.py; never scheduled.
+        self.reparse = reparse
+        # cached by detect_revision, consumed by stage:
+        self._resolved: ResolvedUniverse | None = None
+        self._index_rows: list[dict[str, Any]] = []
+        # accession -> the OTHER CIKs the daily index lists it under (co-registrants
+        # of a combined filing). Only populated for accessions a tracked issuer filed.
+        self._coregistrants: dict[str, list[int]] = {}
+        self._retry_rows: list[dict[str, Any]] = []
+        self._latest_index_date: date | None = None
+        self._backfill = False
+
+    # -- SourceImporter protocol -------------------------------------------
+
+    async def detect_revision(self, db) -> str | None:
+        async with self._client_factory() as client:
+            self._resolved = await sec_universe.resolve_ciks(db, client)
+            last_processed = await self._last_processed_index_date(db)
+            self._latest_index_date = await client.latest_index_date(self.today)
+            if self._latest_index_date is None:
+                raise SecError("no EDGAR daily index available")
+            # Reparse needs every accession restaged, not just those filed since
+            # the last run — the facts a fixed parser now accepts were never
+            # stored, so a reparse cannot be served from the database.
+            self._coregistrants = {}
+            if last_processed is None or self.reparse:
+                self._backfill = True
+                self._index_rows = []
+            else:
+                self._backfill = False
+                self._index_rows = await self._collect_index_rows(
+                    client, last_processed, self._latest_index_date
+                )
+        content = sec_universe.index_content_hash(self._index_rows)
+        revision = sec_universe.compose_revision(
+            self._latest_index_date, content, self._resolved.symbol_to_cik
+        )
+        self._retry_rows = []
+        if not self._backfill:
+            self._retry_rows = await self._retry_backlog(
+                db,
+                set(self._resolved.cik_to_ticker_ids),
+            )
+        # Company Facts can change while the daily index revision stays fixed.
+        # Returning None deliberately bypasses the framework's no-op gate so a
+        # scheduled run retries every active gap.
+        return None if self._retry_rows else revision
+
+    async def stage(self, db) -> StagedFundamentals:
+        assert self._resolved is not None, "detect_revision must run first"
+        resolved = self._resolved
+        staged = StagedFundamentals(resolved=resolved, backfill=self._backfill)
+
+        cik_to_tids = resolved.cik_to_ticker_ids
+        # Whole index rows (not bare accessions): form + index date are what make
+        # an unresolvable filing diagnosable without re-walking the index by hand.
+        filed_by_cik: dict[int, list[dict[str, Any]]] = defaultdict(list)
+        for r in self._index_rows:
+            if r["cik"] in cik_to_tids:
+                filed_by_cik[r["cik"]].append(r)
+
+        # Promoted-around filings live in a small durable retry queue, including
+        # the one-time migration backfill. Merge them into normal incremental
+        # work so the scheduled importer heals them without operator action.
+        if not self._backfill:
+            seen = {
+                (int(cik), row["accession"])
+                for cik, rows in filed_by_cik.items()
+                for row in rows
+            }
+            for row in self._retry_rows:
+                cik = int(row["cik"])
+                key = (cik, row["accession"])
+                if key in seen:
+                    continue
+                filed_by_cik[cik].append(row)
+                seen.add(key)
+                coregistrants = [int(value) for value in row.get("coregistrants") or []]
+                if coregistrants:
+                    self._coregistrants[row["accession"]] = coregistrants
+
+        existing = await self._ciks_with_snapshots(db, set(cik_to_tids))
+        if self._backfill:
+            backfill_ciks = set(cik_to_tids)
+        else:
+            # Newly added issuers (resolved but no snapshots yet) get a full-history
+            # backfill; issuers that already have history are handled incrementally.
+            backfill_ciks = {c for c in cik_to_tids if c not in existing}
+        incremental_ciks = set(filed_by_cik) - backfill_ciks
+
+        # Continuity reference for co-registrant recovery, read once up front.
+        last_shares = await self._last_shares_outstanding(db, set(cik_to_tids))
+
+        async with self._client_factory() as client:
+            for cik in sorted(backfill_ciks | incremental_ciks):
+                is_backfill = cik in backfill_ciks
+                await self._stage_issuer(
+                    client, cik, is_backfill, filed_by_cik, staged, last_shares
+                )
+
+        # Read-only discrepancy detection: an accession we reconstructed that is
+        # already stored, differing in ANY source field (immutable → report in
+        # validation, event on promote, never mutate). Also gives promote the
+        # existing set so its insert count is dialect-independent.
+        if staged.rows:
+            existing = await self._existing_by_accession(db, [r.accession for r in staged.rows])
+            staged.existing_accessions = set(existing)
+            for row in staged.rows:
+                old = existing.get(row.accession)
+                if old is not None:
+                    fields = _diff_fields(row, old)
+                    if fields:
+                        staged.discrepancies.append({"accession": row.accession, "fields": fields})
+        return staged
+
+    async def _stage_issuer(
+        self, client, cik, is_backfill, filed_by_cik, staged, last_shares
+    ) -> None:
+        cf = await client.companyfacts(cik)
+        bad = _companyfacts_structure_error(cf)
+        if bad is not None:
+            # Malformed payload (missing facts/units structure) — record separately
+            # and fail validation, rather than letting it degrade to skipped rows.
+            staged.invalid_payloads.append({"cik": cik10(cik), "reason": bad})
+            staged.issuers_fetched += 1
+            return
+        sub = await client.submissions(cik, include_history=is_backfill)
+        xbrl_meta, nonxbrl = _filing_meta(sub)
+        if not xbrl_meta:
+            staged.no_xbrl_filings.append(
+                {"cik": cik10(cik), "name": sub.get("name"), "tickers": sub.get("tickers")}
+            )
+
+        fiscal_year_end = sub.get("fiscal_year_end")
+        recovered_rows: list[SnapshotRow] = []
+        index_rows = {
+            row["accession"]: row for row in filed_by_cik.get(cik, [])
+        }
+        if is_backfill:
+            accns = set(xbrl_meta)
+        else:
+            present = parser.companyfacts_accessions(cf)
+            accns = set()
+            for index_row in filed_by_cik.get(cik, []):
+                accn = index_row["accession"]
+                if accn in nonxbrl:
+                    staged.skipped_non_xbrl.append({"cik": cik10(cik), "accession": accn})
+                elif accn not in xbrl_meta:
+                    # The daily index lists it but the issuer's own filing list does
+                    # not (submissions lagging the index, no usable period metadata,
+                    # or a co-registrant filing). NOT a Company-Facts lag — separate
+                    # cause, separate fix, so it gets its own reason.
+                    staged.missing_xbrl.append(
+                        _missing(
+                            cik,
+                            index_row,
+                            "not_in_submissions",
+                            self.today,
+                            self._coregistrants.get(accn),
+                        )
+                    )
+                elif accn in present:
+                    accns.add(accn)
+                else:
+                    # Filed, XBRL, but absent from this issuer's Company Facts. Try
+                    # the co-registrant file before treating it as missing data.
+                    row, source_cik = await self._recover_from_coregistrant(
+                        client, cik, accn, xbrl_meta, fiscal_year_end,
+                        last_shares.get(cik10(cik)), staged,
+                    )
+                    if row is not None:
+                        recovered_rows.append(row)
+                        staged.recovered.append({
+                            "cik": cik10(cik),
+                            "accession": accn,
+                            "source_cik": source_cik,
+                            "form": index_row.get("form"),
+                        })
+                    else:
+                        staged.missing_xbrl.append(_missing(
+                            cik, index_row,
+                            # Found, but it did not look like this issuer's own
+                            # numbers — say so; it is not the same as absent.
+                            "coregistrant_facts_rejected" if source_cik
+                            else "not_in_companyfacts",
+                            self.today,
+                            self._coregistrants.get(accn),
+                        ))
+
+        # fiscalYearEnd (MMDD) is what lets the parser derive period identity from
+        # reportDate instead of SEC's unreliable fy/fp fields.
+        result = parser.parse_snapshots(cf, xbrl_meta, accns, fiscal_year_end=fiscal_year_end)
+        for skipped in result.skipped_filings:
+            index_row = index_rows.get(skipped["accession"])
+            if index_row is not None:
+                # Facts are present but our parser cannot construct a snapshot.
+                # A new index row keeps the normal grace period before promotion;
+                # a row already read from the queue retains its _retry_queue marker
+                # so later imports promote and retry without wedging the index.
+                staged.missing_xbrl.append(_missing(
+                    cik,
+                    index_row,
+                    "parser_unusable",
+                    self.today,
+                    self._coregistrants.get(skipped["accession"]),
+                ))
+        staged.rows.extend(result.rows)
+        staged.rows.extend(recovered_rows)
+        staged.skipped_filings.extend(result.skipped_filings)
+        staged.field_issues.extend(result.field_issues)
+        staged.issuers_fetched += 1
+        if result.rows or recovered_rows:
+            staged.issuers_with_rows += 1
+
+        # SIC proposal for this issuer's tickers (read-only; applied in promote).
+        sic = str(sub["sic"]) if sub.get("sic") else None
+        desc = sub.get("sic_description")
+        for tid in staged.resolved.cik_to_ticker_ids.get(cik, []):
+            staged.sic_updates.append((tid, sic, desc))
+
+    async def _recover_from_coregistrant(
+        self, client, cik: int, accn: str, xbrl_meta, fiscal_year_end, reference, staged,
+    ) -> tuple[SnapshotRow | None, str | None]:
+        """Look for ``accn``'s facts in a co-registrant's Company Facts file.
+
+        SEC sometimes files a combined parent/subsidiary filing's XBRL under the
+        co-registrant's CIK rather than the filer's — the ticker-carrying parent's
+        own file simply never gets that accession. Verified 2026-07-27 for NEE
+        (facts under Florida Power & Light) and DOW (under Dow Chemical); an NEE
+        filing misattributed the same way in **2014** is still misattributed, so
+        this does not self-correct and no amount of retrying recovers it.
+
+        Returns ``(row, source_cik)`` on success, ``(None, source_cik)`` when the
+        facts were found but rejected by the continuity guard, ``(None, None)``
+        when no co-registrant has them.
+
+        Incremental path only: the co-registrant map comes from the daily index,
+        which a backfill/reparse does not walk. A reparse therefore recovers a
+        filing only once SEC re-files it under the filer's own CIK.
+        """
+        for co in self._coregistrants.get(accn, []):
+            try:
+                cf_co = await client.companyfacts(co)
+            except SecError:
+                continue  # a co-registrant shell often has no facts file at all
+            if _companyfacts_structure_error(cf_co) is not None:
+                continue
+            result = parser.parse_snapshots(
+                cf_co, xbrl_meta, {accn}, fiscal_year_end=fiscal_year_end
+            )
+            if not result.rows:
+                continue
+            row = result.rows[0]
+            if not _shares_continuous(row.shares_outstanding, reference):
+                return None, cik10(co)
+            # A recovered row is the one most worth flagging, so its parser caveats
+            # travel with it rather than being dropped on the way out.
+            staged.field_issues.extend(result.field_issues)
+            # parse_snapshots stamps the CIK of the payload it read — re-stamp to
+            # the issuer that actually filed, or the row lands under the shell.
+            return replace(row, cik=cik10(cik)), cik10(co)
+        return None, None
+
+    async def validate(self, db, staged: StagedFundamentals) -> ValidationResult:
+        messages: list[str] = []
+
+        # Consistency gate — before any write. Only filings still inside the retry
+        # window block: a failure here stops every later run too (source_max_date
+        # advances on promotion alone), so blocking forever on a filing SEC has
+        # misfiled would cost far more than the one filing it withholds. Older
+        # ones are carried by promote() as a warning instead. The message names
+        # the filings: "which ones" has to be in the alert itself, not merely
+        # reconstructible by re-walking the index.
+        blocking = _within_retry_window(staged.missing_xbrl)
+        aged_out = _past_retry_window(staged.missing_xbrl)
+        if blocking:
+            messages.append(
+                f"{len(blocking)} tracked XBRL filing(s) unresolved within the "
+                f"{MISSING_XBRL_RETRY_DAYS}-day retry window "
+                f"({_reason_counts(blocking)}) — retry: {_missing_detail(blocking)}"
+            )
+        # Malformed companyfacts payloads must fail, not degrade to skipped rows.
+        if staged.invalid_payloads:
+            messages.append(
+                f"{len(staged.invalid_payloads)} issuer(s) returned a malformed "
+                "companyfacts payload (missing facts structure)"
+            )
+
+        accns = [r.accession for r in staged.rows]
+        if len(accns) != len(set(accns)):
+            messages.append("duplicate accession in staged snapshots")
+
+        if staged.backfill:
+            n_issuers = len(staged.resolved.cik_to_ticker_ids)
+            coverage = staged.issuers_with_rows / n_issuers if n_issuers else 0.0
+            if coverage < MIN_BACKFILL_COVERAGE:
+                messages.append(
+                    f"backfill coverage {coverage:.0%} < {MIN_BACKFILL_COVERAGE:.0%}"
+                )
+
+        summary = {
+            "backfill": staged.backfill,
+            "issuers_fetched": staged.issuers_fetched,
+            "issuers_with_rows": staged.issuers_with_rows,
+            "snapshot_rows": len(staged.rows),
+            "skipped_filings": len(staged.skipped_filings),
+            "field_issues": len(staged.field_issues),
+            "skipped_non_xbrl": len(staged.skipped_non_xbrl),
+            "no_xbrl_filings": staged.no_xbrl_filings[:50],
+            "no_xbrl_filings_count": len(staged.no_xbrl_filings),
+            "no_xbrl_ciks": sorted({
+                str(item["cik"])
+                for item in staged.no_xbrl_filings
+                if item.get("cik")
+            }),
+            "missing_xbrl": staged.missing_xbrl[:50],
+            "missing_xbrl_count": len(staged.missing_xbrl),
+            "missing_xbrl_blocking": len(blocking),
+            "recovered_from_coregistrant": staged.recovered[:50],
+            "recovered_count": len(staged.recovered),
+            # Complete compact gate input; detailed audit lists above stay capped.
+            "setup_blocked_ciks": sorted({
+                str(item["cik"])
+                for item in [*staged.missing_xbrl, *staged.no_xbrl_filings]
+                if item.get("cik")
+            }),
+            "invalid_payloads": staged.invalid_payloads,
+            "cik_updates": len(staged.resolved.cik_updates),
+            # differing existing accessions (immutable — kept, reported here)
+            "discrepancies": staged.discrepancies[:50],
+            "discrepancy_count": len(staged.discrepancies),
+        }
+        return ValidationResult(
+            ok=not messages,
+            summary=summary,
+            source_max_date=self._latest_index_date,
+            messages=messages,
+            # Company-Facts absence is usually publication lag, but can also be a
+            # permanent co-registrant misfile that the daily index did not expose.
+            # Defer quietly at first; the framework warns if promotions stay stale.
+            retryable=(
+                len(messages) == 1
+                and bool(blocking)
+                and all(
+                    m.get("reason") in {"not_in_companyfacts", "parser_unusable"}
+                    for m in blocking
+                )
+            ),
+            deferred_alert_after_days=MISSING_XBRL_RETRY_DAYS,
+            deferred_alert_messages=(
+                [
+                    f"{len(aged_out)} tracked SEC filing(s) remain unresolved past "
+                    f"the {MISSING_XBRL_RETRY_DAYS}-day retry window. They will "
+                    f"enter automatic retry and block affected symbols from setups: "
+                    f"{_missing_detail(aged_out)}"
+                ]
+                if aged_out
+                else []
+            ),
+        )
+
+    async def promote(self, db, staged: StagedFundamentals, run_id: int) -> dict[str, int]:
+        inserted = 0
+        updated = 0
+        # Only accessions whose reconstruction actually changed are rewritten;
+        # an unchanged stored row is left completely alone.
+        changed = {d["accession"] for d in staged.discrepancies} if self.reparse else set()
+        for row in staged.rows:
+            if row.accession in staged.existing_accessions:
+                if row.accession in changed:
+                    # Write the FULL column set (_row_values covers _SNAPSHOT_COLS)
+                    # so a rewritten row is never half old-parse, half new-parse.
+                    # created_at stays at the original insert; import_run_id
+                    # attributes the rewrite.
+                    values = _row_values(row, run_id)
+                    values.pop("created_at", None)
+                    await db.execute(
+                        update(FundamentalSnapshot)
+                        .where(FundamentalSnapshot.accession == row.accession)
+                        .values(**values)
+                    )
+                    updated += 1
+                continue  # otherwise immutable — keep the original row
+            stmt = insert_for_session(db, FundamentalSnapshot).values(**_row_values(row, run_id))
+            stmt = stmt.on_conflict_do_nothing(index_elements=["accession"])  # race belt-and-suspenders
+            await db.execute(stmt)
+            inserted += 1
+
+        # Synchronize the retry queue in the snapshot-promotion transaction.
+        existing_gaps = (await db.execute(select(SecFilingGap))).scalars().all()
+        existing_gap_accessions = {gap.accession for gap in existing_gaps}
+        resolved_accessions = {row.accession for row in staged.rows}
+        # A filing now classified non-XBRL can never yield a snapshot and is no
+        # longer a fundamentals completeness gap.
+        resolved_accessions.update(
+            item["accession"] for item in staged.skipped_non_xbrl
+        )
+        queue_resolved = 0
+        if resolved_accessions:
+            result = await db.execute(
+                delete(SecFilingGap).where(
+                    SecFilingGap.accession.in_(resolved_accessions)
+                )
+            )
+            queue_resolved = int(result.rowcount or 0)
+
+        now = _now()
+        tolerated = _past_retry_window(staged.missing_xbrl)
+        for gap in tolerated:
+            stmt = insert_for_session(db, SecFilingGap).values(
+                cik=gap["cik"],
+                accession=gap["accession"],
+                form=gap.get("form"),
+                index_date=gap.get("index_date"),
+                reason=gap["reason"],
+                coregistrant_ciks_json=json.dumps(gap.get("coregistrants") or []),
+                first_seen_at=now,
+                last_attempted_at=now,
+            )
+            await db.execute(
+                stmt.on_conflict_do_update(
+                    index_elements=["accession"],
+                    set_={
+                        "cik": stmt.excluded.cik,
+                        "form": stmt.excluded.form,
+                        "index_date": stmt.excluded.index_date,
+                        "reason": stmt.excluded.reason,
+                        "coregistrant_ciks_json": stmt.excluded.coregistrant_ciks_json,
+                        "last_attempted_at": stmt.excluded.last_attempted_at,
+                    },
+                )
+            )
+
+        # Remove gaps made irrelevant by a later valid 10-K/10-Q. Quality reads
+        # already ignore them; physical cleanup keeps the queue small.
+        active_ids = {gap.id for gap in await fundamentals_quality_service.active_gaps(db)}
+        obsolete_ids = {
+            gap.id for gap in existing_gaps
+            if gap.id not in active_ids and gap.accession not in resolved_accessions
+        }
+        if obsolete_ids:
+            result = await db.execute(
+                delete(SecFilingGap).where(SecFilingGap.id.in_(obsolete_ids))
+            )
+            queue_resolved += int(result.rowcount or 0)
+
+        newly_queued = [
+            gap for gap in tolerated
+            if gap["accession"] not in existing_gap_accessions
+        ]
+
+        # Warn (in-transaction, so it commits atomically with the promotion) when
+        # any existing accession reconstructed differently — kept immutable.
+        if staged.discrepancies:
+            accns = ", ".join(d["accession"] for d in staged.discrepancies[:10])
+            disposition = (
+                f"REWRITTEN by reparse run {run_id}" if self.reparse else "kept immutable"
+            )
+            db.add(SystemEvent(
+                severity="warning",
+                source="sec_facts",
+                code="snapshot_reparse" if self.reparse else "snapshot_discrepancy",
+                message=(
+                    f"{len(staged.discrepancies)} stored accession(s) reconstructed "
+                    f"differently; {disposition}: {accns}"
+                )[:4000],
+                dedup_key=f"sec_facts:discrepancy:{run_id}",
+                created_at=_now(),
+            ))
+
+        # Persistent current gaps get one actionable escalation rather than a
+        # daily warning. The nullable marker makes this durable and noise-free.
+        escalation_cutoff = now - timedelta(days=FILING_GAP_ESCALATE_DAYS)
+        aged_gaps = (
+            await db.execute(
+                select(SecFilingGap).where(
+                    SecFilingGap.first_seen_at <= escalation_cutoff,
+                    SecFilingGap.escalated_at.is_(None),
+                )
+            )
+        ).scalars().all()
+        if aged_gaps:
+            named = ", ".join(
+                f"{gap.cik}/{gap.accession} ({gap.reason})"
+                for gap in aged_gaps[:10]
+            )
+            db.add(SystemEvent(
+                severity="warning",
+                source="sec_facts",
+                code="filing_gap_aged",
+                message=(
+                    f"{len(aged_gaps)} SEC filing gap(s) remain unresolved after "
+                    f"{FILING_GAP_ESCALATE_DAYS} days; affected setups remain paused. "
+                    f"Review the filing/CIK mapping or parser: {named}"
+                )[:4000],
+                dedup_key=f"sec_facts:filing_gap_aged:{run_id}",
+                created_at=now,
+            ))
+            await db.execute(
+                update(SecFilingGap)
+                .where(SecFilingGap.id.in_([gap.id for gap in aged_gaps]))
+                .values(escalated_at=now)
+            )
+
+        # Recovered rows are real data from an unexpected place — record where they
+        # came from, so a wrong recovery is auditable rather than invisible.
+        if staged.recovered:
+            named = ", ".join(
+                f"{r['accession']} <- CIK {r['source_cik']}" for r in staged.recovered[:10]
+            )
+            logger.info(
+                "sec_facts: recovered %d filing(s) from co-registrants: %s",
+                len(staged.recovered),
+                named,
+            )
+
+        # One warning when a gap first enters automatic retry. Repeating it every
+        # day adds noise; the queue remains the durable actionable state.
+        if newly_queued:
+            symbols_by_cik: dict[str, list[str]] = defaultdict(list)
+            for symbol, cik in staged.resolved.symbol_to_cik.items():
+                symbols_by_cik[cik10(cik)].append(symbol)
+            named = ", ".join(
+                f"{'/'.join(symbols_by_cik.get(gap['cik'], [])) or gap['cik']}"
+                f"/{gap['accession']}"
+                for gap in newly_queued[:10]
+            )
+            db.add(SystemEvent(
+                severity="warning",
+                source="sec_facts",
+                code="unresolved_filing",
+                message=(
+                    f"{len(newly_queued)} filing(s) entered automatic SEC retry. "
+                    f"Affected symbols are blocked from new actionable setups until "
+                    f"their filing is recovered: {named}"
+                )[:4000],
+                dedup_key=f"sec_facts:unresolved_filing:{run_id}",
+                created_at=_now(),
+            ))
+
+        # A new registrant may have no XBRL filing yet. Keep it out of actionable
+        # setups, but log it instead of raising a recurring operator warning.
+        if staged.no_xbrl_filings:
+            named = ", ".join(
+                f"{e['cik']} ({e.get('name') or '?'})" for e in staged.no_xbrl_filings[:10]
+            )
+            logger.info(
+                "sec_facts: %d registrant(s) have no XBRL history yet: %s",
+                len(staged.no_xbrl_filings),
+                named,
+            )
+
+        ticker_counts = await sec_universe.apply_ticker_updates(
+            db, staged.resolved, staged.sic_updates
+        )
+        return {
+            "inserted": inserted,
+            "updated": updated,
+            "existing_unchanged": len(staged.existing_accessions) - updated,
+            "discrepancies": len(staged.discrepancies),
+            "retry_queue_added": len(newly_queued),
+            "retry_queue_resolved": queue_resolved,
+            **ticker_counts,
+        }
+
+    # -- helpers -----------------------------------------------------------
+
+    async def _retry_backlog(
+        self,
+        db,
+        tracked_ciks: set[int],
+    ) -> list[dict[str, Any]]:
+        """Active typed gaps; migration 028 owns historical bootstrap."""
+        if not tracked_ciks:
+            return []
+        tracked = {cik10(cik) for cik in tracked_ciks}
+        candidates: dict[str, dict[str, Any]] = {}
+
+        queued = await fundamentals_quality_service.active_gaps(db, tracked)
+        for gap in queued:
+            try:
+                coregistrants = json.loads(gap.coregistrant_ciks_json or "[]")
+            except (TypeError, ValueError):
+                coregistrants = []
+            candidates[gap.accession] = {
+                "cik": gap.cik,
+                "accession": gap.accession,
+                "form": gap.form,
+                "index_date": gap.index_date,
+                "reason": gap.reason,
+                "coregistrants": coregistrants,
+                "_retry_queue": True,
+            }
+
+        if not candidates:
+            return []
+        resolved = set(
+            (
+                await db.execute(
+                    select(FundamentalSnapshot.accession).where(
+                        FundamentalSnapshot.accession.in_(list(candidates))
+                    )
+                )
+            ).scalars().all()
+        )
+        return [
+            item
+            for accession, item in candidates.items()
+            if accession not in resolved
+        ]
+
+    async def _last_processed_index_date(self, db) -> date | None:
+        return (
+            await db.execute(
+                select(DataImportRun.source_max_date)
+                .where(DataImportRun.source == SOURCE, DataImportRun.status == STATUS_PROMOTED)
+                .order_by(DataImportRun.id.desc())
+                .limit(1)
+            )
+        ).scalar_one_or_none()
+
+    async def _collect_index_rows(
+        self, client: SecClient, last_processed: date, latest: date
+    ) -> list[dict[str, Any]]:
+        # Walk EVERY unprocessed date. No cap — dropping the older part of a long
+        # outage while still advancing source_max_date would permanently lose
+        # those filings. A large gap is one-time cost, not silent data loss.
+        tracked = set(self._resolved.cik_to_ticker_ids) if self._resolved else set()
+        gap = (latest - last_processed).days
+        if gap > 60:
+            logger.warning("sec_facts: %d-day index gap since %s; walking all", gap, last_processed)
+        rows: list[dict[str, Any]] = []
+        day = last_processed + timedelta(days=1)
+        while day <= latest:
+            # Group the whole day first: a combined filing is listed once per
+            # co-registrant CIK, and those sibling CIKs are the only pointer to
+            # where SEC may have put the XBRL (see _recover_from_coregistrant).
+            by_accession: dict[str, list[dict[str, Any]]] = defaultdict(list)
+            for r in await client.daily_index(day):
+                if r["form"] in _XBRL_FORMS:
+                    by_accession[r["accession"]].append(r)
+            for accession, group in by_accession.items():
+                filers = {r["cik"] for r in group}
+                tracked_filers = filers & tracked
+                if not tracked_filers:
+                    continue
+                siblings = sorted(filers - tracked_filers)
+                if siblings:
+                    self._coregistrants[accession] = siblings
+                for r in group:
+                    if r["cik"] in tracked_filers:
+                        r["index_date"] = day  # not hashed (revision uses cik/accession)
+                        rows.append(r)
+            day += timedelta(days=1)
+        return rows
+
+    async def _ciks_with_snapshots(self, db, ciks: set[int]) -> set[int]:
+        if not ciks:
+            return set()
+        cik_strs = [cik10(c) for c in ciks]
+        found = (
+            await db.execute(
+                select(FundamentalSnapshot.cik)
+                .where(FundamentalSnapshot.cik.in_(cik_strs))
+                .distinct()
+            )
+        ).scalars().all()
+        return {int(c) for c in found}
+
+    async def _last_shares_outstanding(self, db, ciks: set[int]) -> dict[str, float]:
+        """Latest known shares outstanding per tracked issuer — the continuity
+        reference co-registrant recovery is checked against."""
+        if not ciks:
+            return {}
+        rows = (
+            await db.execute(
+                select(FundamentalSnapshot.cik, FundamentalSnapshot.shares_outstanding)
+                .where(
+                    FundamentalSnapshot.cik.in_([cik10(c) for c in ciks]),
+                    FundamentalSnapshot.shares_outstanding.is_not(None),
+                )
+                # Last write per cik wins, so the sort must be total: an amendment
+                # and its original share a period_end, and an undefined tie there
+                # would make recovery non-deterministic across runs and dialects.
+                .order_by(
+                    FundamentalSnapshot.period_end,
+                    FundamentalSnapshot.filed_date,
+                    FundamentalSnapshot.accession,
+                )
+            )
+        ).all()
+        return {cik: float(shares) for cik, shares in rows}
+
+    async def _existing_by_accession(self, db, accessions: list[str]) -> dict[str, FundamentalSnapshot]:
+        if not accessions:
+            return {}
+        rows = (
+            await db.execute(
+                select(FundamentalSnapshot).where(FundamentalSnapshot.accession.in_(accessions))
+            )
+        ).scalars().all()
+        return {r.accession: r for r in rows}
+
+
+def _companyfacts_structure_error(cf: Any) -> str | None:
+    """None if the payload is structurally sound, else a reason string. Checks the
+    top-level ``facts`` mapping AND that every concept carries a ``units`` mapping —
+    a missing/non-dict units would silently drop that concept's facts otherwise."""
+    if not isinstance(cf, dict) or not isinstance(cf.get("facts"), dict):
+        return "missing facts structure"
+    for concepts in cf["facts"].values():
+        if not isinstance(concepts, dict):
+            return "malformed taxonomy structure"
+        for body in concepts.values():
+            if not isinstance(body, dict) or not isinstance(body.get("units"), dict):
+                return "missing units structure"
+    return None
+
+
+def _missing(
+    cik: int,
+    row: dict[str, Any],
+    reason: str,
+    today: date,
+    coregistrants: list[int] | None = None,
+) -> dict[str, Any]:
+    """One unresolvable index row, carrying everything needed to look the filing
+    up by hand (EDGAR accession + the index date it was seen on) and to decide
+    whether it is still young enough to be worth blocking on."""
+    index_date = row.get("index_date")
+    age_days = (
+        (today - index_date).days if isinstance(index_date, date) else 0
+    )
+    if row.get("_retry_queue"):
+        age_days = max(age_days, MISSING_XBRL_RETRY_DAYS + 1)
+    return {
+        "cik": cik10(cik),
+        "accession": row["accession"],
+        "form": row.get("form"),
+        "index_date": index_date,
+        # A newly observed row without a date blocks safely. A durable queue row
+        # has already passed the bounded window and is forced aged-out above.
+        "age_days": age_days,
+        "reason": reason,
+        "coregistrants": list(coregistrants or []),
+    }
+
+
+def _within_retry_window(missing: list[dict[str, Any]]) -> list[dict[str, Any]]:
+    return [m for m in missing if m.get("age_days", 0) <= MISSING_XBRL_RETRY_DAYS]
+
+
+def _past_retry_window(missing: list[dict[str, Any]]) -> list[dict[str, Any]]:
+    return [m for m in missing if m.get("age_days", 0) > MISSING_XBRL_RETRY_DAYS]
+
+
+def _shares_continuous(shares: float | None, reference: float | None) -> bool:
+    """Does a co-registrant-recovered share count look like this issuer's own?
+
+    The failure worth preventing is storing a subsidiary's standalone facts as the
+    parent's. A co-registrant shell holds a token float — Florida Power & Light
+    against NextEra's 2.09bn shares — so any sane band separates them while still
+    tolerating buybacks and issuance. With no history to compare against (a newly
+    tracked issuer) or no share count at all, recovery is refused, not guessed.
+    """
+    if not shares or not reference:
+        return False
+    return RECOVERY_SHARES_MIN <= shares / reference <= RECOVERY_SHARES_MAX
+
+
+def _reason_counts(missing: list[dict[str, Any]]) -> str:
+    counts = Counter(m["reason"] for m in missing)
+    return ", ".join(f"{reason}={n}" for reason, n in sorted(counts.items()))
+
+
+def _missing_detail(missing: list[dict[str, Any]], limit: int = 10) -> str:
+    detail = ", ".join(
+        f"{m['cik']}/{m['accession']} {m.get('form') or '?'} "
+        f"[{m.get('index_date') or '?'}] {m['reason']}"
+        for m in missing[:limit]
+    )
+    if len(missing) > limit:
+        detail += f", +{len(missing) - limit} more"
+    return detail
+
+
+def _filing_meta(sub: dict[str, Any]) -> tuple[dict[str, FilingMeta], set[str]]:
+    """(xbrl_meta, nonxbrl_accessions) from a submissions payload. xbrl_meta only
+    includes 10-K/10-Q(/A) filings that are XBRL and have full period metadata."""
+    xbrl: dict[str, FilingMeta] = {}
+    nonxbrl: set[str] = set()
+    for f in sub.get("filings", []):
+        if f["form"] not in _XBRL_FORMS:
+            continue
+        if not f.get("is_xbrl"):
+            nonxbrl.add(f["accession"])
+            continue
+        if not (f.get("report_date") and f.get("filing_date") and f.get("acceptance_datetime")):
+            continue
+        xbrl[f["accession"]] = FilingMeta(
+            report_date=date.fromisoformat(f["report_date"]),
+            filing_date=date.fromisoformat(f["filing_date"]),
+            accepted_at=_parse_dt(f["acceptance_datetime"]),
+            form=f["form"],
+        )
+    return xbrl, nonxbrl
+
+
+def _parse_dt(value: str) -> datetime:
+    return datetime.fromisoformat(value.replace("Z", "+00:00"))
+
+
+def _row_values(row: SnapshotRow, run_id: int) -> dict[str, Any]:
+    values = {col: getattr(row, col) for col in _SNAPSHOT_COLS}
+    values["import_run_id"] = run_id
+    values["created_at"] = _now()
+    return values
+
+
+def _diff_fields(row: SnapshotRow, old: FundamentalSnapshot) -> list[str]:
+    """Source fields where a re-parsed row differs from the stored row."""
+    return [
+        col for col in _COMPARE_COLS
+        if not _same_value(getattr(row, col), getattr(old, col))
+    ]
+
+
+def _same_value(parsed: Any, stored: Any) -> bool:
+    """Compare a freshly parsed value against its stored round-trip.
+
+    Datetimes need care: every timestamp here is UTC by construction, but
+    ``DateTime(timezone=True)`` only preserves tzinfo on Postgres — SQLite hands
+    back a naive value. Comparing representations would report an unchanged row
+    as differing, which would both spam the discrepancy warning and make a
+    reparse rewrite every row it touched. Compare instants instead.
+    """
+    if isinstance(parsed, datetime) and isinstance(stored, datetime):
+        return _as_utc(parsed) == _as_utc(stored)
+    return parsed == stored
+
+
+def _as_utc(value: datetime) -> datetime:
+    return value if value.tzinfo is not None else value.replace(tzinfo=timezone.utc)

app/services/sec_universe.py

@@ -0,0 +1,159 @@
+"""Tracked-universe CIK/SIC resolution and the SEC importer's composite revision.
+
+Resolves the app's tracked tickers to SEC issuers (CIK) and prepares
+``tickers.cik/sic/sic_description`` back-fills. Also builds the **universe
+fingerprint** in the importer's composite revision, so adding a ticker changes
+the revision and forces a run instead of being ``no_op``'d away or starved
+waiting for its issuer to file (A3 design, Decision 1 review fix).
+
+**Transaction contract:** resolution is read-only — `resolve_ciks` and
+`fetch_sic_updates` compute *proposed* updates and mutate nothing. They run in
+the importer's `stage` (which must not write, or a failed validation would leak
+changes on the framework's failure commit). The proposals are applied only in
+`promote`, via `apply_ticker_updates`, atomically with the snapshot inserts.
+"""
+
+from __future__ import annotations
+
+import hashlib
+import json
+import logging
+from dataclasses import dataclass, field
+from typing import Iterable
+
+from sqlalchemy import select, update
+
+from app.models.ticker import Ticker
+from app.services import settings_store
+from app.services.earnings_alignment import normalise_symbol
+from app.services.sec_client import SecClient
+
+logger = logging.getLogger(__name__)
+
+# JSON {symbol: cik} pinning a ticker to a specific registrant, overriding
+# company_tickers.json. Needed when SEC maps a ticker to a successor entity that
+# has not filed: XOM points at CIK 2115436 "ExxonMobil Holdings Corp" (zero XBRL
+# filings) while every 10-K/10-Q — including one filed 2026-05-04 — is still under
+# CIK 34088. Which registrant is the real filer is a judgement about a corporate
+# event, so it is pinned explicitly rather than guessed. The importer's
+# `no_xbrl_filings` warning is what tells you a pin is needed.
+CIK_OVERRIDES_KEY = "sec_cik_overrides"
+
+
+@dataclass
+class ResolvedUniverse:
+    """Read-only result of CIK resolution. `cik_updates` are proposed writes
+    (ticker_id → new cik string) applied later in promote."""
+
+    symbol_to_cik: dict[str, int] = field(default_factory=dict)
+    cik_to_ticker_ids: dict[int, list[int]] = field(default_factory=dict)
+    cik_updates: list[tuple[int, str]] = field(default_factory=list)
+
+
+async def resolve_ciks(db, client: SecClient) -> ResolvedUniverse:
+    """Resolve tracked tickers to CIKs via company_tickers.json. **Read-only** —
+    returns the mapping + proposed `tickers.cik` writes; mutates nothing."""
+    ticker_to_cik = await client.company_tickers()
+    overrides = await cik_overrides(db)
+    rows = (await db.execute(select(Ticker.id, Ticker.symbol, Ticker.cik))).all()
+
+    result = ResolvedUniverse()
+    for tid, symbol, current_cik in rows:
+        if not symbol:
+            continue
+        sym = normalise_symbol(symbol)
+        cik = overrides.get(sym) or ticker_to_cik.get(sym)
+        if cik is None:
+            continue  # ADRs / non-SEC issuers — snapshots simply absent
+        result.symbol_to_cik[sym] = cik
+        result.cik_to_ticker_ids.setdefault(cik, []).append(tid)
+        if current_cik != f"{cik:010d}":
+            result.cik_updates.append((tid, f"{cik:010d}"))
+    logger.info(
+        "resolve_ciks: %d resolved, %d proposed cik updates",
+        len(result.symbol_to_cik),
+        len(result.cik_updates),
+    )
+    return result
+
+
+async def cik_overrides(db) -> dict[str, int]:
+    """Manual ``{symbol: cik}`` pins from ``SystemSetting[CIK_OVERRIDES_KEY]``.
+
+    A malformed setting must never take the importer down, so anything unparseable
+    is logged and ignored — the run then falls back to company_tickers.json.
+    """
+    raw = await settings_store.get_value(db, CIK_OVERRIDES_KEY)
+    if not raw:
+        return {}
+    try:
+        loaded = json.loads(raw)
+    except (TypeError, ValueError):
+        logger.warning("%s is not valid JSON — ignoring CIK overrides", CIK_OVERRIDES_KEY)
+        return {}
+    if not isinstance(loaded, dict):
+        logger.warning("%s must be a {symbol: cik} object — ignoring", CIK_OVERRIDES_KEY)
+        return {}
+    out: dict[str, int] = {}
+    for symbol, cik in loaded.items():
+        try:
+            out[normalise_symbol(str(symbol))] = int(cik)
+        except (TypeError, ValueError):
+            logger.warning("%s: bad entry %r -> %r — ignoring", CIK_OVERRIDES_KEY, symbol, cik)
+    if out:
+        logger.info("resolve_ciks: %d CIK override(s) applied: %s", len(out), sorted(out))
+    return out
+
+
+async def fetch_sic_updates(
+    client: SecClient, cik_to_ticker_ids: dict[int, Iterable[int]]
+) -> list[tuple[int, str | None, str | None]]:
+    """Fetch SIC for each CIK (recent-only submissions, no history shards) and
+    return proposed `(ticker_id, sic, sic_description)` writes. **Read-only** — no
+    DB mutation. Callers pass only the CIKs that need it (e.g. missing a SIC)."""
+    updates: list[tuple[int, str | None, str | None]] = []
+    for cik, ticker_ids in cik_to_ticker_ids.items():
+        sub = await client.submissions(cik, include_history=False)
+        sic = str(sub["sic"]) if sub.get("sic") else None
+        desc = sub.get("sic_description")
+        for tid in ticker_ids:
+            updates.append((tid, sic, desc))
+    return updates
+
+
+async def apply_ticker_updates(
+    db,
+    resolved: ResolvedUniverse,
+    sic_updates: list[tuple[int, str | None, str | None]] | None = None,
+) -> dict[str, int]:
+    """Apply the proposed cik / sic writes. **The only writer** — call inside
+    promote so it commits atomically with the snapshot inserts."""
+    for tid, cik in resolved.cik_updates:
+        await db.execute(update(Ticker).where(Ticker.id == tid).values(cik=cik))
+    for tid, sic, desc in sic_updates or []:
+        await db.execute(
+            update(Ticker).where(Ticker.id == tid).values(sic=sic, sic_description=desc)
+        )
+    return {"cik_updates": len(resolved.cik_updates), "sic_updates": len(sic_updates or [])}
+
+
+def universe_fingerprint(symbol_to_cik: dict[str, int]) -> str:
+    """Stable short hash of the tracked symbol->CIK set. Changes whenever a ticker
+    is added/removed or its CIK mapping changes."""
+    canonical = ";".join(f"{sym}:{cik}" for sym, cik in sorted(symbol_to_cik.items()))
+    return hashlib.blake2b(canonical.encode("utf-8"), digest_size=12).hexdigest()
+
+
+def index_content_hash(index_rows: Iterable[dict]) -> str:
+    """Order-independent hash of the tracked index accessions consumed this run."""
+    keys = sorted(f"{r['cik']}/{r['accession']}" for r in index_rows)
+    return hashlib.blake2b("|".join(keys).encode("utf-8"), digest_size=12).hexdigest()
+
+
+def compose_revision(index_date, content_hash: str, symbol_to_cik: dict[str, int]) -> str:
+    """Composite revision = processed index date + index-content hash + universe
+    fingerprint. Equal across runs ⇒ nothing new ⇒ no_op. Rejects a missing index
+    date rather than emitting a `None:...` revision that could false-match."""
+    if index_date is None:
+        raise ValueError("compose_revision requires a non-null index date")
+    return f"{index_date}:{content_hash}:{universe_fingerprint(symbol_to_cik)}"

app/services/shadow_book_service.py

@@ -0,0 +1,363 @@
+"""Shadow book — the validated strategy, traded automatically.
+
+The discretionary paper book only ever contains trades the user chose to take,
+inside a ~20 minute window, on days they were available. The backtest that
+validated this strategy does none of that: it takes the top-ranked qualified
+setups up to capacity, every session, with no human involved. That difference
+makes the manual book unusable as out-of-sample evidence — it measures the
+strategy *plus* discretion and availability.
+
+The shadow book closes that gap. It mirrors ``_simulate_portfolio``'s selection
+rule exactly and shares the manual book's exit policy, so the only difference
+between the two books is *which* qualified setups get taken.
+
+Parity is the load-bearing property here. Selection ordering comes from the
+stored ``strategy_rank`` the scanner already wrote (the same 80/20
+momentum/vol blend the backtest ranks on) rather than being recomputed, so the
+two cannot drift apart.
+"""
+
+from __future__ import annotations
+
+import logging
+from datetime import datetime, timedelta, timezone
+
+from sqlalchemy import select
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.models.paper_trade import PaperTrade
+from app.models.ticker import Ticker
+from app.models.trade_setup import TradeSetup
+from app.models.user import User
+from app.services import settings_store
+from app.services.qualification import setup_qualifies
+from app.services.trade_policy import SHADOW_BOOK, get_reentry_gate_locks
+
+logger = logging.getLogger(__name__)
+
+KEY_ENABLED = "shadow_book_enabled"
+KEY_CAPACITY = "shadow_book_capacity"
+KEY_RISK_PCT = "shadow_book_risk_pct"
+KEY_START_EQUITY = "shadow_book_start_equity"
+
+# Matches the validated configuration: 10-position book, 1% fixed-fractional
+# risk. Start equity is only a sizing base — comparisons are drawn in percent
+# and R-multiples, never in raw currency.
+DEFAULT_CAPACITY = 10
+DEFAULT_RISK_PCT = 1.0
+DEFAULT_START_EQUITY = 100_000.0
+
+# Mirrors ``_simulate_portfolio``'s SIM_NOTIONAL_CAP: no single position may
+# exceed this fraction of equity, and the book never uses margin. Without the
+# cap, a setup with a tight stop turns 1% risk into a position several times
+# equity — a leveraged trade the validated strategy would never have taken.
+NOTIONAL_CAP = 0.20
+
+# If the last successful scan completed longer ago than this, no scan ran in the
+# current pipeline pass (scans are daily, ~24h apart), so there is nothing fresh
+# to trade. Comfortably longer than a scan's own duration, far shorter than the
+# gap between scans.
+MAX_SCAN_AGE = timedelta(hours=6)
+
+
+async def get_config(db: AsyncSession) -> dict:
+    """Shadow book sizing/capacity config, falling back to validated defaults."""
+    raw = await settings_store.get_map(
+        db, [KEY_CAPACITY, KEY_RISK_PCT, KEY_START_EQUITY]
+    )
+
+    def _num(key: str, default: float, *, minimum: float, maximum: float) -> float:
+        try:
+            value = float(raw.get(key) or default)
+        except (TypeError, ValueError):
+            return default
+        return max(minimum, min(maximum, value))
+
+    return {
+        "capacity": int(_num(KEY_CAPACITY, DEFAULT_CAPACITY, minimum=1, maximum=100)),
+        "risk_pct": _num(KEY_RISK_PCT, DEFAULT_RISK_PCT, minimum=0.05, maximum=10.0),
+        "start_equity": _num(
+            KEY_START_EQUITY, DEFAULT_START_EQUITY, minimum=1000.0, maximum=1e9
+        ),
+    }
+
+
+async def is_enabled(db: AsyncSession) -> bool:
+    """Shadow book writes trades to the live book, so it is opt-in."""
+    value = await settings_store.get_value(db, KEY_ENABLED, "false")
+    return str(value).strip().lower() in {"1", "true", "yes", "on"}
+
+
+async def equity_and_cash(
+    db: AsyncSession, start_equity: float, positions: list[PaperTrade]
+) -> tuple[float, float]:
+    """Marked equity and free cash, matching ``_simulate_portfolio``.
+
+    The simulator sizes from *marked* equity — cash plus open positions at their
+    latest close — and spends from cash, so a book that is fully invested cannot
+    keep buying. Sizing from realized P&L alone would drift away from the
+    backtest as soon as positions were held across a scan.
+    """
+    from app.services.paper_trade_service import _latest_closes
+
+    result = await db.execute(
+        select(PaperTrade).where(
+            PaperTrade.book == SHADOW_BOOK,
+            PaperTrade.status == "closed",
+            PaperTrade.close_price.is_not(None),
+        )
+    )
+    realized = 0.0
+    for trade in result.scalars():
+        per_share = (
+            trade.close_price - trade.entry_price
+            if trade.direction == "long"
+            else trade.entry_price - trade.close_price
+        )
+        realized += per_share * trade.shares
+
+    open_cost = sum(p.entry_price * p.shares for p in positions)
+    marks = await _latest_closes(db, {p.ticker_id for p in positions})
+    open_value = sum(
+        (marks.get(p.ticker_id) or p.entry_price) * p.shares for p in positions
+    )
+
+    cash = start_equity + realized - open_cost
+    return cash + open_value, cash
+
+
+def position_shares(
+    equity: float,
+    risk_pct: float,
+    entry: float,
+    stop: float,
+    *,
+    cash_available: float | None = None,
+) -> float:
+    """Shares to buy, sized exactly as ``_simulate_portfolio`` sizes them.
+
+    Fixed-fractional risk first, then the two caps the simulator applies: no
+    position may exceed ``NOTIONAL_CAP`` of equity, and the book cannot spend
+    cash it does not have. Dropping either cap lets a tight stop produce a
+    leveraged position and breaks compounding parity with the backtest.
+    """
+    risk_per_share = abs(entry - stop)
+    if risk_per_share <= 0 or equity <= 0 or entry <= 0:
+        return 0.0
+
+    shares = (equity * risk_pct / 100.0) / risk_per_share
+    shares = min(shares, (equity * NOTIONAL_CAP) / entry)
+    if cash_available is not None:
+        shares = min(shares, max(0.0, cash_available) / entry)
+    # Dust guard, as in the simulator: sub-$1 positions are noise, not trades.
+    return shares if shares * entry >= 1.0 else 0.0
+
+
+async def _open_positions(db: AsyncSession) -> list[PaperTrade]:
+    result = await db.execute(
+        select(PaperTrade).where(
+            PaperTrade.book == SHADOW_BOOK, PaperTrade.status == "open"
+        )
+    )
+    return list(result.scalars().all())
+
+
+async def _shadow_user_id(db: AsyncSession) -> int | None:
+    """Shadow trades are not owned by a person; attach them to the first user."""
+    result = await db.execute(select(User.id).order_by(User.id.asc()).limit(1))
+    row = result.first()
+    return int(row[0]) if row else None
+
+
+async def _scan_run_to_trade(
+    db: AsyncSession,
+    *,
+    now: datetime,
+    expected_run_id: str | None = None,
+) -> str | None:
+    """The run id whose setups the shadow book may act on, or None.
+
+    * ``expected_run_id`` set (pipeline step): the stored run id must match it
+      exactly. This is the airtight guarantee — a scan that was disabled or
+      failed in *this* pipeline never stamped this id, and a concurrent manual
+      scan (a separate APScheduler job, not serialised against the pipeline)
+      stamps its own id even when it finishes last, so neither can be mistaken
+      for the pipeline's own scan. Timestamp order alone cannot tell them apart.
+    * ``expected_run_id`` None (direct Admin trigger): fall back to the freshness
+      window on the last scan's own id. There is no pipeline scan to bind to, so
+      acting on a recent scan is the operator's explicit choice.
+
+    Setups are then selected by ``scan_run_id`` equal to the returned id, so a
+    concurrent scan's rows in the same time window are excluded by identity.
+    """
+    from app.services import rr_scanner_service as rr
+
+    completed = _parse_dt(
+        await settings_store.get_value(db, rr.KEY_LAST_SCAN_COMPLETED)
+    )
+    run_id = await settings_store.get_value(db, rr.KEY_LAST_SCAN_RUN_ID)
+    if completed is None or not run_id:
+        return None
+    if expected_run_id is not None:
+        return run_id if run_id == expected_run_id else None
+    if now - completed > MAX_SCAN_AGE:
+        return None
+    return run_id
+
+
+def _parse_dt(raw: str | None) -> datetime | None:
+    if not raw:
+        return None
+    try:
+        return datetime.fromisoformat(raw)
+    except ValueError:
+        return None
+
+
+async def _todays_qualified_setups(
+    db: AsyncSession,
+    config: dict,
+    *,
+    now: datetime,
+    expected_run_id: str | None = None,
+) -> list[TradeSetup]:
+    """Long-only qualified setups from the scan we may act on, best rank first.
+
+    Order matters here, and matches the review's requirement:
+
+    1. Take only rows the matched scan produced (``scan_run_id == run id``). A
+       previous run, or a manual scan overlapping in time, carries a different
+       id and is excluded by identity — not by a time window it could write into.
+    2. Keep long only. The validated strategy is long-only, but the gate permits
+       shorts when ``min_momentum_percentile`` is 0 (a legal admin setting), and
+       the cash accounting assumes longs — so this is enforced here, not left to
+       the gate.
+    3. Deduplicate to the latest row per ticker *before* qualifying, so a newer
+       unqualified row correctly suppresses an older qualified one rather than
+       the reverse.
+    4. Qualify, then rank by ``strategy_rank`` (unranked sort last).
+    """
+    run_id = await _scan_run_to_trade(db, now=now, expected_run_id=expected_run_id)
+    if run_id is None:
+        return []
+
+    result = await db.execute(
+        select(TradeSetup).where(TradeSetup.scan_run_id == run_id)
+    )
+    rows = [s for s in result.scalars() if (s.direction or "long") == "long"]
+
+    latest: dict[int, TradeSetup] = {}
+    for setup in rows:
+        held = latest.get(setup.ticker_id)
+        if held is None or (setup.detected_at, setup.id) > (
+            held.detected_at,
+            held.id,
+        ):
+            latest[setup.ticker_id] = setup
+
+    qualified = [s for s in latest.values() if setup_qualifies(s, config)]
+    return sorted(
+        qualified,
+        key=lambda s: (
+            s.strategy_rank if s.strategy_rank is not None else float("-inf")
+        ),
+        reverse=True,
+    )
+
+
+async def open_shadow_positions(
+    db: AsyncSession,
+    *,
+    activation_config: dict,
+    opened_at: datetime | None = None,
+    expected_run_id: str | None = None,
+) -> dict:
+    """Fill free capacity with the top-ranked qualified setups.
+
+    Mirrors the backtest: rank the qualified cross-section, walk it top-down,
+    skip anything already held or locked out by post-stop gate-reset, and stop
+    at capacity. Returns a summary for the job log.
+
+    ``expected_run_id`` binds this run to the scan that stamped that exact id
+    (the pipeline's own scan), so a scan that failed in this pipeline — or a
+    concurrent manual scan that finished last — cannot substitute for it. See
+    ``_scan_run_to_trade``.
+    """
+    summary = {
+        "opened": 0,
+        "skipped_held": 0,
+        "skipped_locked": 0,
+        "skipped_no_cash": 0,
+        "symbols": [],
+    }
+    config = await get_config(db)
+
+    positions = await _open_positions(db)
+    held = {p.ticker_id for p in positions}
+    free_slots = config["capacity"] - len(positions)
+    if free_slots <= 0:
+        return summary
+
+    user_id = await _shadow_user_id(db)
+    if user_id is None:
+        logger.warning("shadow book skipped: no user to attach trades to")
+        return summary
+
+    locks = await get_reentry_gate_locks(db, book=SHADOW_BOOK)
+    equity, cash = await equity_and_cash(db, config["start_equity"], positions)
+    timestamp = opened_at or datetime.now(timezone.utc)
+
+    candidates = await _todays_qualified_setups(
+        db, activation_config, now=timestamp, expected_run_id=expected_run_id
+    )
+    for setup in candidates:
+        if free_slots <= 0:
+            break
+        if setup.ticker_id in held:
+            summary["skipped_held"] += 1
+            continue
+        if setup.ticker_id in locks:
+            summary["skipped_locked"] += 1
+            continue
+
+        entry = float(setup.entry_price or 0.0)
+        stop = float(setup.stop_loss or 0.0)
+        shares = position_shares(
+            equity, config["risk_pct"], entry, stop, cash_available=cash
+        )
+        if shares <= 0:
+            summary["skipped_no_cash"] += 1
+            continue
+        cash -= shares * entry
+
+        db.add(
+            PaperTrade(
+                user_id=user_id,
+                ticker_id=setup.ticker_id,
+                direction=setup.direction,
+                entry_price=entry,
+                shares=shares,
+                stop_loss=stop,
+                target=float(setup.target or 0.0),
+                status="open",
+                opened_at=timestamp,
+                fill_mode="near_close",
+                book=SHADOW_BOOK,
+            )
+        )
+        held.add(setup.ticker_id)
+        free_slots -= 1
+        summary["opened"] += 1
+        summary["symbols"].append(setup.ticker_id)
+
+    if summary["opened"]:
+        await db.commit()
+    return summary
+
+
+async def symbols_for(db: AsyncSession, ticker_ids: list[int]) -> list[str]:
+    """Resolve ticker ids to symbols for logging."""
+    if not ticker_ids:
+        return []
+    result = await db.execute(select(Ticker.symbol).where(Ticker.id.in_(ticker_ids)))
+    return [row[0] for row in result.all()]

app/services/sr_service.py

@@ -857,8 +857,42 @@ async def get_sr_levels(
     symbol: str,
     tolerance: float | None = None,
 ) -> list[SRLevel]:
-    """Get S/R levels for a ticker, recalculating on every request (MVP).
+    """Return Structural S/R for a ticker, strength descending.
 
-    Returns levels sorted by strength descending.
+    Default (``tolerance is None``): read persisted levels only — no rewrite.
+    Pipeline/ingestion call ``recalculate_sr_levels`` after OHLCV changes.
+
+    When ``tolerance`` is set: build a **transient** detect view with that merge
+    tolerance and do not persist it (custom merge for API clients). Transient
+    rows use negative ids so they cannot be confused with stored levels.
     """
-    return await recalculate_sr_levels(db, symbol, tolerance)
+    if tolerance is None:
+        ticker = await _get_ticker(db, symbol)
+        result = await db.execute(
+            select(SRLevel)
+            .where(SRLevel.ticker_id == ticker.id)
+            .order_by(SRLevel.strength.desc())
+        )
+        return list(result.scalars().all())
+
+    from types import SimpleNamespace
+
+    ticker = await _get_ticker(db, symbol)
+    records = await query_ohlcv(db, symbol)
+    if not records:
+        return []
+    _, highs, lows, closes, volumes = _extract_ohlcv(records)
+    detected = detect_sr_levels(highs, lows, closes, volumes, tolerance)
+    now = datetime.utcnow()
+    # Ephemeral objects with the SRLevel attribute shape the router expects.
+    return [
+        SimpleNamespace(  # type: ignore[return-value]
+            id=-(i + 1),
+            price_level=lvl["price_level"],
+            type=lvl["type"],
+            strength=lvl["strength"],
+            detection_method=lvl["detection_method"],
+            created_at=now,
+        )
+        for i, lvl in enumerate(detected)
+    ]

app/services/trade_policy.py

@@ -1,120 +1,132 @@
-"""Shared live/backtest trading-policy constants and availability checks."""
+"""Shared live trading-policy state and availability checks."""
 
 from __future__ import annotations
 
-from collections import defaultdict
+from collections.abc import Iterable
 from datetime import date, datetime, timezone
+from zoneinfo import ZoneInfo
 
 from sqlalchemy import func, select
 from sqlalchemy.ext.asyncio import AsyncSession
 
-from app.models.benchmark_price import BenchmarkPrice
-from app.models.ohlcv import OHLCVRecord
 from app.models.paper_trade import PaperTrade
-from app.services.benchmark_service import BENCHMARK_SYMBOL
 
-# A ticker stopped at its initial stop may qualify again immediately, but the
-# July 2026 event study showed that waiting five market sessions materially
-# improved the production book. The stop session is wait_session=0; the first
-# permitted re-entry is wait_session=5, provided the normal gate still passes.
-REENTRY_LOCKDOWN_SESSIONS = 5
+# Gate-reset "day" boundary matches US cash equities session calendar, not UTC.
+_REENTRY_DAY_TZ = ZoneInfo("America/New_York")
 
 
-async def get_reentry_lockdowns(
+def _ny_trading_date(moment: datetime) -> date:
+    """Calendar date in America/New_York for a gate-reset observation."""
+    if moment.tzinfo is None:
+        moment = moment.replace(tzinfo=timezone.utc)
+    return moment.astimezone(_REENTRY_DAY_TZ).date()
+
+
+MANUAL_BOOK = "manual"
+SHADOW_BOOK = "shadow"
+
+
+async def _latest_initial_stop_trades(
     db: AsyncSession,
     *,
-    as_of: date | None = None,
-    sessions: int = REENTRY_LOCKDOWN_SESSIONS,
-) -> dict[int, int]:
-    """Return ``{ticker_id: remaining_sessions}`` for active lockdowns.
+    closed_before: datetime | None = None,
+    book: str = MANUAL_BOOK,
+) -> dict[int, PaperTrade]:
+    """Return a ticker's latest closed trade only when it was an initial stop.
 
-    SPY is the canonical calendar for the platform's US-equity universe. When
-    the stored benchmark history does not reach an older stop, only that
-    ticker's own OHLCV dates are used as a conservative fallback. Unrelated
-    ticker dates can therefore never shorten a lockdown.
+    Scoped to one ``book``: the discretionary and shadow books diverge as soon
+    as their entries differ, so each must see only its own stop history when
+    deciding whether a ticker is locked out of re-entry.
     """
-    sessions = max(0, int(sessions))
-    if sessions == 0:
-        return {}
-
-    session_cutoff = as_of or datetime.now(timezone.utc).date()
-    stop_result = await db.execute(
+    ranked_stmt = (
         select(
-            PaperTrade.ticker_id,
-            func.max(PaperTrade.closed_at).label("last_stop_at"),
+            PaperTrade.id.label("trade_id"),
+            func.row_number()
+            .over(
+                partition_by=PaperTrade.ticker_id,
+                order_by=(PaperTrade.closed_at.desc(), PaperTrade.id.desc()),
+            )
+            .label("recency"),
         )
         .where(
             PaperTrade.status == "closed",
-            PaperTrade.close_reason == "stop",
             PaperTrade.closed_at.is_not(None),
+            PaperTrade.book == book,
         )
-        .group_by(PaperTrade.ticker_id)
     )
-    stop_dates = {
-        ticker_id: stopped_at.date()
-        for ticker_id, stopped_at in stop_result.all()
-        if stopped_at is not None and stopped_at.date() <= session_cutoff
+    if closed_before is not None:
+        ranked_stmt = ranked_stmt.where(PaperTrade.closed_at <= closed_before)
+    ranked = ranked_stmt.subquery()
+    stmt = (
+        select(PaperTrade)
+        .join(ranked, ranked.c.trade_id == PaperTrade.id)
+        .where(
+            ranked.c.recency == 1,
+            PaperTrade.close_reason == "stop",
+        )
+    )
+    result = await db.execute(stmt)
+    return {trade.ticker_id: trade for trade in result.scalars()}
+
+
+async def get_reentry_gate_locks(
+    db: AsyncSession, *, book: str = MANUAL_BOOK
+) -> dict[int, datetime]:
+    """Return tickers still waiting for a post-stop gate failure.
+
+    A later qualified setup is actionable only after the daily scanner has
+    observed an unqualified evaluation after the latest initial-stop exit and
+    then a fresh qualification. The returned timestamp is the stop time and is
+    useful for diagnostics; callers normally only need the keys.
+    """
+    latest = await _latest_initial_stop_trades(db, book=book)
+    return {
+        ticker_id: trade.closed_at
+        for ticker_id, trade in latest.items()
+        if trade.reentry_gate_requalified_at is None and trade.closed_at is not None
     }
-    if not stop_dates:
-        return {}
-
-    benchmark_result = await db.execute(
-        select(BenchmarkPrice.date)
-        .where(
-            BenchmarkPrice.symbol == BENCHMARK_SYMBOL,
-            BenchmarkPrice.date <= session_cutoff,
-        )
-        .order_by(BenchmarkPrice.date.asc())
-    )
-    benchmark_dates = [row[0] for row in benchmark_result.all()]
-
-    lockdowns: dict[int, int] = {}
-    fallback_stops: dict[int, date] = {}
-    first_benchmark_date = benchmark_dates[0] if benchmark_dates else None
-    for ticker_id, stop_date in stop_dates.items():
-        completed = sum(day > stop_date for day in benchmark_dates)
-        if completed >= sessions:
-            continue
-        if first_benchmark_date is not None and first_benchmark_date <= stop_date:
-            lockdowns[ticker_id] = sessions - completed
-        else:
-            # The benchmark table starts after this stop (or is empty), so it
-            # cannot prove how many sessions elapsed. Resolve only this ticker
-            # against its own bars instead of using universe-wide dates.
-            fallback_stops[ticker_id] = stop_date
-
-    if fallback_stops:
-        own_session_result = await db.execute(
-            select(OHLCVRecord.ticker_id, OHLCVRecord.date)
-            .where(
-                OHLCVRecord.ticker_id.in_(fallback_stops),
-                OHLCVRecord.date > min(fallback_stops.values()),
-                OHLCVRecord.date <= session_cutoff,
-            )
-            .distinct()
-        )
-        own_dates: dict[int, set[date]] = defaultdict(set)
-        for ticker_id, market_date in own_session_result.all():
-            own_dates[ticker_id].add(market_date)
-        for ticker_id, stop_date in fallback_stops.items():
-            completed = sum(day > stop_date for day in own_dates[ticker_id])
-            if completed < sessions:
-                lockdowns[ticker_id] = sessions - completed
-
-    return lockdowns
 
 
-async def get_reentry_lockdown_ticker_ids(
+async def observe_reentry_gate_transitions(
     db: AsyncSession,
     *,
-    as_of: date | None = None,
-    sessions: int = REENTRY_LOCKDOWN_SESSIONS,
+    evaluated_ticker_ids: Iterable[int],
+    qualified_ticker_ids: Iterable[int],
+    observed_at: datetime | None = None,
+    book: str = MANUAL_BOOK,
 ) -> set[int]:
-    """Compatibility wrapper for callers that only need blocked ticker ids."""
-    return set(
-        await get_reentry_lockdowns(
-            db,
-            as_of=as_of,
-            sessions=sessions,
-        )
-    )
+    """Persist gate-failure and later requalification observations.
+
+    Only tickers whose scan completed successfully belong in
+    ``evaluated_ticker_ids``. This prevents a scanner exception from being
+    mistaken for a real gate exit. The caller owns the transaction; this helper
+    flushes so the new state is immediately visible in that transaction.
+    """
+    evaluated = {int(ticker_id) for ticker_id in evaluated_ticker_ids}
+    if not evaluated:
+        return set()
+    qualified = {int(ticker_id) for ticker_id in qualified_ticker_ids}
+    timestamp = observed_at or datetime.now(timezone.utc)
+    latest = await _latest_initial_stop_trades(db, closed_before=timestamp, book=book)
+    updated: set[int] = set()
+    for ticker_id in evaluated:
+        trade = latest.get(ticker_id)
+        if trade is None or trade.reentry_gate_requalified_at is not None:
+            continue
+        if trade.reentry_gate_failed_at is None:
+            if ticker_id not in qualified:
+                trade.reentry_gate_failed_at = timestamp
+                updated.add(ticker_id)
+        elif ticker_id in qualified:
+            # Study semantics: requalify only on a *subsequent* daily observation.
+            # Same America/New_York calendar day as the failure does not unlock,
+            # even if multiple full-universe scans run (manual + near-close).
+            if _ny_trading_date(trade.reentry_gate_failed_at) < _ny_trading_date(
+                timestamp
+            ):
+                trade.reentry_gate_requalified_at = timestamp
+                updated.add(ticker_id)
+
+    if updated:
+        await db.flush()
+    return updated

app/ssl_bootstrap.py

@@ -0,0 +1,136 @@
+"""TLS / corporate-proxy bootstrap for CLI scripts and the API.
+
+Must run **before** httpx / alpaca / aiohttp open connections.
+
+Resolution order for the CA bundle:
+1. ``combined-ca-bundle.pem`` in the repo root (gitignored corporate bundle)
+2. ``$HOME/combined-ca-bundle.pem`` (MacBook path used by existing tooling)
+3. ``SSL_CERT_FILE`` / ``REQUESTS_CA_BUNDLE`` if already set and present
+4. ``certifi.where()`` when the package is installed
+5. System defaults (no patch)
+
+Optional corporate proxy (Swisscom-style) when ``USE_CORP_PROXY=1``.
+"""
+
+from __future__ import annotations
+
+import os
+import ssl
+from pathlib import Path
+
+_BOOTSTRAPPED = False
+
+
+def _candidate_ca_paths() -> list[Path]:
+    root = Path(__file__).resolve().parent.parent
+    home = Path.home()
+    env_paths = [
+        os.environ.get("SSL_CERT_FILE", ""),
+        os.environ.get("REQUESTS_CA_BUNDLE", ""),
+        os.environ.get("CURL_CA_BUNDLE", ""),
+    ]
+    paths = [
+        root / "combined-ca-bundle.pem",
+        home / "combined-ca-bundle.pem",
+        *[Path(p) for p in env_paths if p],
+    ]
+    try:
+        import certifi
+
+        paths.append(Path(certifi.where()))
+    except Exception:
+        pass
+    return paths
+
+
+def resolve_ca_bundle() -> str | None:
+    for path in _candidate_ca_paths():
+        try:
+            if path.is_file() and path.stat().st_size > 0:
+                return str(path.resolve())
+        except OSError:
+            continue
+    return None
+
+
+def apply_corp_proxy_if_requested() -> None:
+    if os.environ.get("USE_CORP_PROXY", "0") != "1":
+        return
+    proxy = os.environ.get("CORP_HTTP_PROXY", "http://aproxy.corproot.net:8080")
+    no_proxy = os.environ.get(
+        "CORP_NO_PROXY",
+        "corproot.net,sharedtcs.net,127.0.0.1,localhost,bix.swisscom.com,swisscom.com",
+    )
+    os.environ.setdefault("HTTP_PROXY", proxy)
+    os.environ.setdefault("HTTPS_PROXY", proxy)
+    os.environ.setdefault("NO_PROXY", no_proxy)
+    os.environ.setdefault("http_proxy", proxy)
+    os.environ.setdefault("https_proxy", proxy)
+    os.environ.setdefault("no_proxy", no_proxy)
+
+
+def bootstrap_ssl(*, force: bool = False) -> str | None:
+    """Install CA env vars + patch ``ssl.create_default_context``.
+
+    Returns the CA path used, or None if nothing was applied.
+    Safe to call multiple times.
+    """
+    global _BOOTSTRAPPED
+    if _BOOTSTRAPPED and not force:
+        return os.environ.get("SSL_CERT_FILE") or None
+
+    apply_corp_proxy_if_requested()
+
+    cert_path = resolve_ca_bundle()
+    if not cert_path:
+        _BOOTSTRAPPED = True
+        return None
+
+    os.environ["SSL_CERT_FILE"] = cert_path
+    os.environ["REQUESTS_CA_BUNDLE"] = cert_path
+    os.environ["CURL_CA_BUNDLE"] = cert_path
+
+    original = ssl.create_default_context
+
+    def _patched(
+        purpose=ssl.Purpose.SERVER_AUTH, *, cafile=None, capath=None, cadata=None
+    ):
+        ctx = original(purpose, cafile=cafile, capath=capath, cadata=cadata)
+        try:
+            ctx.load_verify_locations(cafile=cert_path)
+        except Exception:
+            pass
+        return ctx
+
+    ssl.create_default_context = _patched  # type: ignore[assignment]
+
+    # aiohttp may cache SSL contexts at import time.
+    try:
+        import aiohttp.connector as aio_conn
+
+        for attr in ("_SSL_CONTEXT_VERIFIED", "_SSL_CONTEXT_UNVERIFIED"):
+            ctx = getattr(aio_conn, attr, None)
+            if ctx is not None:
+                try:
+                    ctx.load_verify_locations(cafile=cert_path)
+                except Exception:
+                    pass
+    except ImportError:
+        pass
+
+    _BOOTSTRAPPED = True
+    return cert_path
+
+
+def ssl_status() -> dict:
+    """Diagnostic blob for research scripts / MacBook troubleshooting."""
+    ca = resolve_ca_bundle()
+    return {
+        "ca_bundle": ca,
+        "ssl_cert_file_env": os.environ.get("SSL_CERT_FILE"),
+        "use_corp_proxy": os.environ.get("USE_CORP_PROXY", "0"),
+        "http_proxy": os.environ.get("HTTPS_PROXY") or os.environ.get("HTTP_PROXY"),
+        "candidates_exist": {
+            str(p): p.is_file() for p in _candidate_ca_paths()[:4]
+        },
+    }

deploy/provision_fundamentals.sh

@@ -0,0 +1,153 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+# One-time production provisioning for the shadow fundamentals sources.
+# Run as root to install; run with --check as the deploy user for a read-only
+# preflight. Version upgrades are intentional code changes, never "latest".
+DOLT_VERSION="2.2.0"
+DOLT_BINARY="${DOLT_BINARY:-/usr/local/bin/dolt}"
+DOLT_DATA_DIR="${DOLT_DATA_DIR:-/var/lib/signal-platform/dolt}"
+DOLT_EARNINGS_SUBDIR="${DOLT_EARNINGS_SUBDIR:-earnings}"
+APP_USER="${APP_USER:-deploy}"
+APP_GROUP="${APP_GROUP:-deploy}"
+ENV_FILE="${ENV_FILE:-/opt/signalplatform/.env}"
+MIN_FREE_GB="${DOLT_MIN_FREE_DISK_GB:-5}"
+EARNINGS_DIR="${DOLT_DATA_DIR}/${DOLT_EARNINGS_SUBDIR}"
+DOLT_IDENTITY_NAME="${DOLT_IDENTITY_NAME:-Signal Platform}"
+DOLT_IDENTITY_EMAIL="${DOLT_IDENTITY_EMAIL:-signal-platform@localhost}"
+FUNDAMENTALS_PARITY_REPORT_DIR="${FUNDAMENTALS_PARITY_REPORT_DIR:-/var/lib/signal-platform/reports/fundamentals-parity}"
+
+fail() {
+  echo "ERROR: $*" >&2
+  exit 1
+}
+
+version_ok() {
+  local output
+  output="$("$DOLT_BINARY" version 2>/dev/null || true)"
+  grep -Eq "(^|[[:space:]])v?${DOLT_VERSION}([[:space:]]|$)" <<<"$output"
+}
+
+as_app_user() {
+  if [[ "$(id -un)" == "$APP_USER" ]]; then
+    "$@"
+  else
+    command -v runuser >/dev/null 2>&1 || fail "runuser is required"
+    runuser -u "$APP_USER" -- "$@"
+  fi
+}
+
+repo_command() {
+  (
+    cd "$EARNINGS_DIR"
+    as_app_user "$@"
+  )
+}
+
+repo_config_value() {
+  repo_command "$DOLT_BINARY" config --get "$1"
+}
+
+configure_identity() {
+  local name email
+  name="$(repo_config_value user.name 2>/dev/null || true)"
+  email="$(repo_config_value user.email 2>/dev/null || true)"
+  if [[ -z "$name" ]]; then
+    repo_command "$DOLT_BINARY" config --local --add user.name "$DOLT_IDENTITY_NAME"
+  fi
+  if [[ -z "$email" ]]; then
+    repo_command "$DOLT_BINARY" config --local --add user.email "$DOLT_IDENTITY_EMAIL"
+  fi
+}
+
+check_free_space() {
+  local available_kb
+  available_kb="$(df -Pk "$DOLT_DATA_DIR" | awk 'NR == 2 {print $4}')"
+  [[ "$available_kb" =~ ^[0-9]+$ ]] || fail "could not read free space for $DOLT_DATA_DIR"
+  if ! awk -v available="$available_kb" -v minimum_gb="$MIN_FREE_GB" \
+    'BEGIN { exit !(available >= minimum_gb * 1024 * 1024) }'; then
+    fail "$DOLT_DATA_DIR has less than ${MIN_FREE_GB} GB free"
+  fi
+}
+
+check_env() {
+  [[ -f "$ENV_FILE" ]] || fail "missing environment file: $ENV_FILE"
+  grep -Fqx "DOLT_BINARY=$DOLT_BINARY" "$ENV_FILE" \
+    || fail "set DOLT_BINARY=$DOLT_BINARY in $ENV_FILE"
+  grep -Fqx "DOLT_DATA_DIR=$DOLT_DATA_DIR" "$ENV_FILE" \
+    || fail "set DOLT_DATA_DIR=$DOLT_DATA_DIR in $ENV_FILE"
+  grep -Fqx "DOLT_EARNINGS_SUBDIR=$DOLT_EARNINGS_SUBDIR" "$ENV_FILE" \
+    || fail "set DOLT_EARNINGS_SUBDIR=$DOLT_EARNINGS_SUBDIR in $ENV_FILE"
+  grep -Eq '^SEC_USER_AGENT=.*@.*' "$ENV_FILE" \
+    || fail "SEC_USER_AGENT in $ENV_FILE must contain a real contact email"
+  grep -Fqx "FUNDAMENTALS_PARITY_REPORT_DIR=$FUNDAMENTALS_PARITY_REPORT_DIR" "$ENV_FILE" \
+    || fail "set FUNDAMENTALS_PARITY_REPORT_DIR=$FUNDAMENTALS_PARITY_REPORT_DIR in $ENV_FILE"
+}
+
+check_all() {
+  local identity_name identity_email
+  id "$APP_USER" >/dev/null 2>&1 || fail "missing service user: $APP_USER"
+  [[ -x "$DOLT_BINARY" ]] || fail "missing Dolt binary: $DOLT_BINARY"
+  version_ok || fail "expected Dolt $DOLT_VERSION at $DOLT_BINARY"
+  [[ -d "$EARNINGS_DIR/.dolt" ]] \
+    || fail "missing earnings clone: $EARNINGS_DIR"
+  if [[ "$(id -un)" == "$APP_USER" ]]; then
+    [[ -r "$EARNINGS_DIR/.dolt" ]] \
+      || fail "earnings clone is not readable by $APP_USER"
+  elif command -v runuser >/dev/null 2>&1; then
+    runuser -u "$APP_USER" -- test -r "$EARNINGS_DIR/.dolt" \
+      || fail "earnings clone is not readable by $APP_USER"
+  else
+    fail "run --check as $APP_USER (or install runuser)"
+  fi
+  identity_name="$(repo_config_value user.name 2>/dev/null || true)"
+  identity_email="$(repo_config_value user.email 2>/dev/null || true)"
+  [[ -n "$identity_name" ]] || fail "missing Dolt user.name for $EARNINGS_DIR"
+  [[ -n "$identity_email" ]] || fail "missing Dolt user.email for $EARNINGS_DIR"
+  [[ -d "$FUNDAMENTALS_PARITY_REPORT_DIR" ]] \
+    || fail "missing parity report directory: $FUNDAMENTALS_PARITY_REPORT_DIR"
+  if [[ "$(id -un)" == "$APP_USER" ]]; then
+    [[ -w "$FUNDAMENTALS_PARITY_REPORT_DIR" ]] \
+      || fail "parity report directory is not writable by $APP_USER"
+  else
+    runuser -u "$APP_USER" -- test -w "$FUNDAMENTALS_PARITY_REPORT_DIR" \
+      || fail "parity report directory is not writable by $APP_USER"
+  fi
+  check_free_space
+  check_env
+  echo "OK: Dolt $DOLT_VERSION and earnings clone are provisioned"
+}
+
+if [[ "${1:-}" == "--check" ]]; then
+  check_all
+  exit 0
+fi
+
+[[ "$EUID" -eq 0 ]] || fail "run provisioning as root (or use --check)"
+command -v curl >/dev/null 2>&1 || fail "curl is required"
+command -v runuser >/dev/null 2>&1 || fail "runuser is required"
+id "$APP_USER" >/dev/null 2>&1 || fail "missing service user: $APP_USER"
+
+if ! version_ok; then
+  installer="$(mktemp)"
+  trap 'rm -f "$installer"' EXIT
+  curl -fsSL \
+    "https://github.com/dolthub/dolt/releases/download/v${DOLT_VERSION}/install.sh" \
+    -o "$installer"
+  bash "$installer"
+fi
+version_ok || fail "Dolt $DOLT_VERSION installation failed"
+
+install -d -o "$APP_USER" -g "$APP_GROUP" -m 0750 "$DOLT_DATA_DIR"
+install -d -o "$APP_USER" -g "$APP_GROUP" -m 0750 "$FUNDAMENTALS_PARITY_REPORT_DIR"
+check_free_space
+
+if [[ ! -d "$EARNINGS_DIR/.dolt" ]]; then
+  [[ ! -e "$EARNINGS_DIR" ]] \
+    || fail "$EARNINGS_DIR exists but is not a Dolt clone"
+  runuser -u "$APP_USER" -- \
+    "$DOLT_BINARY" clone post-no-preference/earnings "$EARNINGS_DIR"
+fi
+
+configure_identity
+check_all

docs/dolt-integration-plan.md

@@ -0,0 +1,540 @@
+# Dolt bulk-data integration — implementation plan
+
+Status: approved 2026-07-21, revised through four review rounds; direction: KISS
+backend, UI value first. Hand-off document for the implementing agent;
+self-contained.
+
+## Objective
+
+Replace the free-tier fundamentals APIs (FMP, Finnhub, Alpha Vantage) with bulk
+data: SEC Company Facts for fundamentals, the DoltHub earnings repo for the
+earnings calendar/history, and — later, independently — the DoltHub stocks repo for
+historical OHLCV. PostgreSQL stays the production system of record.
+
+**Delivery order: two independent workstreams.**
+
+- **Workstream A (build first):** SEC fundamentals + Dolt earnings + API v1 +
+  FundamentalsPanel + decommission FMP/Finnhub/Alpha Vantage. Valuation uses the
+  existing Alpaca closes already in `ohlcv_records`. This alone achieves the goal
+  (killing the quota-limited APIs) and delivers all the UI value.
+- **Workstream B (later, optional until needed):** replace historical OHLCV with
+  the Dolt stocks repo. The most complex machinery (4.7 GB clone, split
+  adjustment, source-bar table, reconciliation) lives here and blocks nothing in A.
+
+**Guiding principle: KISS.** Plain daily importers with staging and atomic
+promotion — no forensic replay, no permanent archive store, no conflict tables, no
+aggregate tables. Engineering budget goes into the UI (quarter trends, peer
+comparison). Deferred until a concrete need: exact source replay, point-in-time
+backtest enforcement, fundamental metrics in scoring.
+
+**Non-negotiables**
+
+- The application never queries Dolt/DoltHub or SEC at request time. All access is
+  batch import → PostgreSQL. If a sync fails or the source is unchanged, production
+  continues on the last successfully imported data.
+- Do not replace PostgreSQL with Dolt/Doltgres. Never commit to the upstream clones.
+- No owned SEC Dolt repo: SEC JSON is normalized straight into PostgreSQL.
+- Scoring **code** is unchanged, but swapping the data source changes production
+  behavior: `app/services/scoring_service.py` (~line 450) scores pe_ratio /
+  revenue_growth / earnings_surprise from `fundamental_data`, so new definitions
+  change rankings even with identical code. Cutover of `fundamental_data`
+  population requires the **score-parity gate** (phase A5) — never silently. All
+  *new* metrics are display-only.
+- Intraday (10:00–15:00), near-close (15:30) and after-close (16:45) pipelines stay
+  on Alpaca unchanged.
+
+## Data sources
+
+1. **SEC Company Facts + submissions bulk files** (free, no key; costs bandwidth,
+   CPU and disk — optimize accordingly) — XBRL facts per **issuer (CIK), not per
+   ticker**. Tickers resolve to a CIK via SEC `company_tickers.json` (multi-class
+   issuers like GOOGL/GOOG share one CIK and one set of fundamentals). Submissions
+   also supply the SIC code (peer grouping) and `acceptanceDateTime`. Handle unit
+   variants and fiscal-period alignment (derive Q4 = FY − Q1..Q3 where needed).
+   **Amendments:** retain every accession immutably; readers select the newest
+   valid `accepted_at` snapshot per reporting period at read time. No flags, no
+   mutation.
+2. **`post-no-preference/earnings`** (DoltHub) — announcement date, BMO/AMC session
+   (partial), period end, EPS estimate/actual, surprise history. Small clone.
+   `scripts/import_dolthub_earnings.py` is a research/SQLite importer — reuse its
+   normalization/alignment logic (calendar↔EPS-history monotonic alignment, SUE
+   scaling) but write a production PostgreSQL importer; do not extend the script.
+3. **`post-no-preference/stocks`** (DoltHub, **workstream B**) — daily raw OHLCV
+   (unadjusted), symbol metadata, splits, dividends. Publishes ~01:30 ET the
+   following calendar day. Clone is ~4.7 GB.
+
+**Licensing (phase A0) — DECIDED 2026-07-22:** `post-no-preference/earnings` is
+**approved for private/internal ingestion under CC BY-SA 4.0**. Conditions the A2
+importer must honor: preserve the upstream license, attribution, and transformation
+notes (retain a CC BY-SA 4.0 reference + attribution to `post-no-preference/earnings`
+and a note of the transformations applied — e.g. in a repo `NOTICE`/attribution file
+and the importer module); **no public API, bulk export, or redistribution** of the
+data; re-review licensing before any public or commercial access. The
+`post-no-preference/stocks` repo (workstream B) is **not** covered here and will be
+reviewed separately if B begins.
+
+## Schema
+
+**Migration 026 (workstream A)** — current head: `025_trade_setup_scan_run_id`:
+
+- `data_import_runs` — lean: id, source (`sec_facts` | `dolt_earnings` |
+  `dolt_stocks`), revision (Dolt commit hash, or SEC archive SHA-256), status
+  (`running`/`validated`/`promoted`/`no_op`/`failed`), source_max_date, row_counts
+  JSON, validation JSON (includes reconciliation/discrepancy summaries — no
+  separate conflicts table; details go to structured logs), started_at,
+  completed_at, error_details. One run per source at a time (Postgres advisory
+  lock keyed by source).
+- `earnings_events` — ticker_id, announce_date, session (`bmo`/`amc`/`unknown`),
+  period_end, eps_estimate, eps_actual, source, import_run_id. Unique
+  (ticker_id, announce_date). **Rescheduling:** within each promotion transaction,
+  delete this source's future-dated rows (announce_date > today) and re-insert
+  from the new snapshot, so moved or cancelled dates never linger. Past rows
+  (results) are never deleted.
+- `tickers` — add nullable `cik`, `sic`, `sic_description` (from SEC submissions /
+  `company_tickers.json`; refreshed by the SEC import; multi-class tickers share
+  values). The only ticker↔issuer join point.
+- `fundamental_snapshots` — **CIK-keyed, one immutable row per accession**: cik,
+  accession (unique), form, filed_date, **accepted_at** (kept although PIT
+  enforcement is deferred — one timestamp now vs painful retrofit later),
+  **period_start, period_end, fiscal_year, fiscal_period** (the filing's own
+  `dei`/`us-gaap` period identity — required to align non-calendar fiscal years and
+  to derive discrete quarters from cumulative facts), and the **price-independent
+  raw facts** so metrics are recomputable. **Store facts as the filing reports
+  them, not as derived quarters:** duration facts (revenue, net income, diluted EPS,
+  CFO, capex, EBITDA inputs) retain the filing's normalized **cumulative YTD/FY**
+  value for the (period_start → period_end) span; balance-sheet facts (cash+ST
+  investments, total debt, shares outstanding) are **period-end** values.
+  ``shares_outstanding`` is a point-in-time count
+  (``dei:EntityCommonStockSharesOutstanding``), not the weighted-average diluted
+  share count — both consumers (est. market cap, YoY dilution) want a
+  point-in-time value. **Nothing
+  derived is frozen into a row:** discrete quarters (10-Q YTD deltas, Q4 = FY −
+  Q1..Q3), TTM, YoY and the four-period metric histories are all computed **at read time** by
+  picking the newest valid accepted_at snapshot for *each* required period — so
+  non-calendar fiscal years resolve correctly and a later amendment to a prior
+  quarter is reflected automatically without ever storing a stale derived quarter.
+  Readers pick the newest valid accepted_at per period; history powers the UI
+  reference comparisons and deterministic reads.
+- Keep `fundamental_data` (`app/models/fundamental.py`) as the latest-value compat
+  cache, repopulated by the daily SEC job — but only after the phase-A5 parity
+  gate.
+
+**Migration 027 (workstream B, written when B starts):**
+
+- `ohlcv_source_bars` — source-truth bar table, required because `ohlcv_records`
+  allows one row per (ticker_id, date) (`app/models/ohlcv.py:12`) and Alpaca
+  ingestion upserts it in place (`app/services/price_service.py:82`) — Dolt and
+  Alpaca bars cannot coexist there. Holds **Dolt raw (unadjusted) bars only** —
+  Alpaca bars are already split-adjusted at the provider (`app/providers/alpaca.py:77`
+  requests `Adjustment.SPLIT`) and live exclusively in `ohlcv_records`. Columns:
+  source (`dolt`), adjustment (`raw` — explicit), ticker_id, date, OHLCV,
+  import_run_id; unique (source, ticker_id, date). Changed bars are counted in the
+  run's validation JSON and logged before overwrite.
+- `corporate_actions` — ticker_id, type (`split`/`dividend`), ex_date,
+  ratio/amount, source, import_run_id. Unique (ticker_id, type, ex_date).
+- `ohlcv_records` — add nullable `import_run_id` FK and `source` text (default
+  `'alpaca'`).
+
+## Import framework
+
+Every importer: idempotent per revision (same Dolt commit / archive checksum →
+`no_op`, zero row changes); stage into a representation outside the live tables
+first (in-memory for the small workstream-A sources; a file/table handle is fine
+if workstream B ever needs it); promotion in one transaction;
+safe to retry; a failed or unchanged run leaves the current dataset untouched.
+Record every attempt in `data_import_runs`.
+
+**SEC access requirements (operational safeguards, per SEC fair-access policy):**
+send an identifying `User-Agent` with a contact email on every request; stay far
+below the 10 req/s limit (the bulk endpoints need only a handful of requests per
+run); exponential backoff on 429; a 403 means the User-Agent or request pattern is
+wrong — alert and stop, never retry-loop. See SEC developer resources
+(https://www.sec.gov/about/developer-resources).
+
+**Reproducibility scope (deliberately limited):** the normalized snapshots in
+PostgreSQL *are* the durable record. Keep only the last ~2 SEC archives on disk for
+debugging. Byte-level replay of old runs is out of scope until a concrete need.
+
+Dolt access: `dolt pull` on the persistent clone, record the resulting commit hash,
+read via `dolt sql -r csv` (no long-running sql-server). **The scheduler shares one
+event loop with the API** (`app/scheduler.py:73`) — run dolt/unzip/download
+subprocesses via `asyncio.create_subprocess_exec` (or an executor), never blocking
+calls. Check free disk space before pulling; alert and skip if below threshold.
+
+**Deployment constraints:** deploy is `rsync --delete` of the repo tree
+(`.gitea/workflows/deploy.yml:127`), so clones and archives must live **outside the
+deployment path** — an env-configured persistent directory (e.g.
+`DOLT_DATA_DIR=/var/lib/signal-platform/dolt`). The dolt binary is a new prod
+runtime dependency: install it once with the version-pinned provisioner in
+`deploy/provision_fundamentals.sh`; operational steps are in
+`docs/fundamentals-deployment.md`. The clone is reproducible from DoltHub; the
+normalized PostgreSQL rows remain part of the normal database backup.
+
+**Validation gates (block promotion, raise an alert via the existing system-events
+path):** source freshness as expected; tracked-universe coverage; no duplicate
+business keys; fundamental units/periods consistent; row-count deltas within
+reason; upstream schema change stops promotion. Workstream B adds: OHLC sanity
+(high ≥ open/close/low, low ≤ open/close/high, volume ≥ 0); no unexplained split
+discontinuities.
+
+**Split adjustment (workstream B):** `ohlcv_source_bars` + `corporate_actions` are
+the source of truth; canonical `ohlcv_records` is *generated* from them to match
+Alpaca `Adjustment.SPLIT`, selecting only adjustment = `raw` rows as input so
+adjustment is applied exactly once. A newly published split rewrites the symbol's
+entire adjusted history — treat whole-symbol rewrites as a normal import event
+(exempt that symbol from the row-count-delta gate for that run) and stamp rows with
+the import_run_id (a backtest↔prod parity guard exists; changed history changes
+backtests).
+
+## Scheduling (`app/scheduler.py` — `SCHEDULE_DEFAULTS` / `_CRON_JOBS`, ~line 1451)
+
+Follow the existing pattern: cron strings in SystemSettings via
+`app/services/settings_store.py`, day-of-week as names never numbers, logging via
+`_log_event`.
+
+Workstream A:
+
+- Dolt earnings import: daily ~02:30 ET (with the future-row replacement above).
+- **SEC fundamentals job: daily ~04:00 ET.** One job, three steps:
+  (a) detect a composite revision from the latest EDGAR daily-index date, the
+  exact tracked index rows, and the tracked-universe fingerprint — an unchanged
+  revision is a `no_op` before Company Facts are fetched;
+  (b) when changed, fetch and parse Company Facts only for tracked-universe CIKs
+  that filed, plus full available history for the first run or a newly added
+  issuer, through validation→atomic promotion. Universe resolution and the exact
+  index inputs are cached during revision detection and reused during staging;
+  CIKs are resolved from `company_tickers.json` without writes until promotion;
+  (c) **always, locally, and only after production activation** (the phase-A5
+  parity approval): refresh the legacy `fundamental_data` fields and mark affected
+  cached fundamental scores stale. **Sources differ per field** — do not assume all
+  five come from SEC: `pe_ratio` and `market_cap` from the newest valid snapshots ×
+  latest PostgreSQL close, each with its own formula — `pe_ratio` = latest close /
+  TTM diluted EPS; `market_cap` = issuer-wide shares outstanding × latest close;
+  `revenue_growth` from the snapshots alone; `earnings_surprise` and
+  `next_earnings_date` from `earnings_events` (the Dolt earnings feed — these two do
+  not exist in SEC facts). Before activation the job imports snapshots only
+  (shadow). Step (c) must run identically when SEC is unreachable — prices move
+  daily even when filings don't, and the earnings-derived fields already live in
+  PostgreSQL.
+  **The new API valuation object is not stored anywhere** — it is computed at
+  request time (below). No valuation cache or table exists.
+
+Workstream B:
+
+- Dolt OHLCV+splits pull/import: `0 2 * * tue-sat` ET. If source_max_date is not
+  fresh, retry hourly until ~06:00, then give up quietly. After a successful
+  import, reconcile the previous session's Dolt-derived bars against the Alpaca
+  bars; summary into the run's validation JSON, details to logs.
+- Move `schedule_daily_pipeline_cron` (morning refresh) from `0 2 * * *` to
+  `0 3 * * *` (only needed once the 02:00 slot is taken by the OHLCV pull).
+  **Late Dolt publication is a non-event:** the canonical scan runs at 15:30 on
+  Alpaca, so the morning pipeline runs normally even when the import hasn't
+  landed — no gating, no defensive coupling.
+
+## Metrics catalog (curated — TTM basis)
+
+Snapshots store **price-independent per-period facts** (the "snapshot" column below
+means *derived from stored snapshots, assembled across periods at read time* — see
+Schema — not frozen at import); price-dependent ratios are never frozen into
+snapshots and have **no storage location at all**: the API computes
+them at request time from the stored snapshots + the latest `ohlcv_records` close
+(both already in PostgreSQL, so this works identically when SEC is unreachable).
+The only stored price-dependent values are the legacy `fundamental_data` fields
+that scoring already reads, refreshed daily by step (c) after activation.
+
+| Metric | Definition | Where computed |
+|---|---|---|
+| Revenue growth YoY | TTM revenue vs prior TTM | snapshot |
+| EPS growth YoY | TTM diluted EPS vs prior TTM | snapshot |
+| Operating margin + 4q trend | TTM operating income / revenue | snapshot |
+| FCF margin | (TTM CFO − capex) / revenue | snapshot |
+| Net debt | total debt − (cash + ST investments); positive = net debt | snapshot |
+| Net debt / EBITDA | net debt / TTM EBITDA | snapshot |
+| Share count Δ YoY | shares outstanding vs year ago | snapshot |
+| Trailing P/E | price / TTM diluted EPS | request time |
+| FCF yield | TTM FCF / est. market cap | request time |
+| Est. market cap | issuer-wide shares outstanding × ticker price | request time |
+| Earnings surprise history | last 4+ from `earnings_events` | query |
+
+**Market cap is an estimate** (issuer-wide shares outstanding × one ticker's price —
+approximate for multi-class issuers). Share count comes from a single consolidated
+value, not a class sum: prefer the one `dei:EntityCommonStockSharesOutstanding`
+cover-page fact; if absent (e.g. Alphabet) fall back to
+`us-gaap:CommonStockSharesOutstanding` at period end. companyfacts is
+non-dimensional, so class-specific facts can't be summed reliably — never do that,
+and never substitute weighted-average/diluted shares; if conflicting values remain,
+store null. Label it "est." in the UI and round aggressively rather than withholding
+it; false precision is the failure mode, not the approximation.
+
+**Units follow existing app conventions:** percentages are percentage points
+(21.0 = 21%), P/E and net-debt/EBITDA are multiples, market cap and net debt are
+dollars.
+
+Deliberately **excluded**: ROIC (invested-capital/NOPAT normalization too noisy),
+gross margin (COGS tagging too inconsistent), any new composite score.
+
+## Peer comparison (read-time only)
+
+- Peer group = tracked-universe issuers sharing the **first two SIC digits**,
+  **deduplicated by CIK** — GOOG and GOOGL are one issuer, one observation, in
+  medians, percentiles and peer_count.
+- Computed at read time from current snapshots — no aggregate tables until
+  performance demonstrates a need.
+- Medians exclude null/invalid values. **Fewer than 5 valid peer issuers → omit
+  the peer result entirely** rather than showing a misleading universe comparison.
+- Percentile direction respects metric polarity (higher-is-better for FCF yield,
+  lower-is-better for P/E and leverage).
+
+## API contract (additive v1)
+
+Every existing top-level field is preserved unchanged (name, type, position) —
+backend and frontend ship independently, no breaking interval. New objects, exact
+names and types:
+
+```jsonc
+{
+  // ...all existing legacy fields, unchanged...
+  "earnings": {
+    "next": {"date": "YYYY-MM-DD", "session": "bmo|amc|unknown", "days_until": 12} | null,
+    "recent": [   // newest first, max 4, may be empty
+      {"announce_date": "YYYY-MM-DD", "period_end": "YYYY-MM-DD|null",
+       "eps_estimate": 1.02|null, "eps_actual": 1.10|null, "surprise_pct": 7.8|null}
+    ]
+  },
+  "metrics": [    // fixed row set — every key always present, value null when unavailable
+    {
+      "key": "revenue_growth_yoy",   // revenue_growth_yoy | eps_growth_yoy | operating_margin |
+                                     // fcf_margin | net_debt | net_debt_to_ebitda | share_count_change_yoy
+      "value": 18.0,                 // number | null — pp / multiples / dollars per units above
+      "history": [                   // oldest→newest, max 4 points, [] when unavailable
+        {"period_end": "YYYY-MM-DD", "value": 8.0}
+      ],
+      "industry": {                  // object | null — null when < 5 valid peer issuers (CIK-deduped)
+        "label": "SIC 73 peers",     // truthful 2-digit group label — grouping IS 2-digit,
+                                     // so no 4-digit description like "Prepackaged Software"
+        "median": 11.0,
+        "favorable_percentile": 82,  // 0-100, polarity-aware (higher = more favorable)
+        "peer_count": 12             // issuers, not tickers
+      },
+      "period_end": "YYYY-MM-DD|null",
+      "filed_date": "YYYY-MM-DD|null",
+      "source": "sec|dolt|legacy_api"
+    }
+  ],
+  "valuation": {  // object | null (null until SEC snapshots exist, phase A3); same industry sub-object rules
+    // computed at REQUEST TIME from stored snapshots + latest PostgreSQL close —
+    // no valuation cache or table; unaffected by SEC availability
+    "pe": 29.2|null, "fcf_yield": 3.8|null,
+    "market_cap_est": 1.2e9|null,    // estimated — UI labels "est."
+    "pe_industry": {...}|null, "fcf_yield_industry": {...}|null,
+    "price_date": "YYYY-MM-DD"       // close used for the ratios
+  }
+}
+```
+
+Null/freshness semantics: absent data is `null` with the row still present (the UI
+shows "n/a", never hides rows); every metric carries its own source, period and
+filing date — no panel-wide source label. The objects may serve partial data during
+rollout (e.g. `earnings` live, `metrics` still `legacy_api`); the shape never
+changes.
+
+## UI — `frontend/src/components/ticker/FundamentalsPanel.tsx`
+
+One distinctive visual device — the **Reference Rails** — in an otherwise restrained
+panel. Preserve the app's dark glass styling and numeric typography.
+
+```
+Fundamentals
+Growth accelerating · margins improving · valuation priced above peers
+Next earnings  Aug 3 · AMC       EPS surprises  ▂ ▅ ▃ ▆
+
+Operating trend              less favorable ← ref → more favorable
+Revenue growth                                             18%
+───────────────│━━━━●                   +3pp vs prior · accelerating
+Share count YoY                                           −1.7%
+───────────────│━━━━●                                    buying back
+
+Valuation & balance           less favorable ← median → more favorable
+P/E                                                       29.2×
+────●━━━━━━━━━━│────                  priced above peers · median 23.5× · 12 peers
+```
+
+- Growth and margins: horizontal rails compare the latest value with the prior
+  quarter or prior-period average; share-count YoY compares with zero. The rail
+  is normalized so right is always more favorable, including buybacks.
+- P/E, FCF yield and leverage: horizontal favorable-percentile rails with a
+  peer-median marker. No decorative rail when `industry` is null (< 5 peers).
+- Every row keeps the exact value and one deterministic comparison caption;
+  missing values render `n/a`, and insufficient peers render `peers n/a`.
+- Earnings: four bars around a shared zero baseline — cyan beats, coral misses, gray
+  unavailable — plus next date and BMO/AMC session countdown.
+- Accessibility: color is always paired with text; neutral/ambiguous stays gray;
+  rails and earnings bars expose complete ARIA descriptions.
+- Remove the hard-coded "FMP" source label; surface filing and price-date
+  provenance in the footer.
+
+**Deterministic reads — one shared rule set.** Implement as a single function with
+named constants; the metric reads and the header sentence use identical outputs. No
+LLM, no new composite score. Defaults (tunable constants, not scattered literals):
+
+- A series read requires ≥ 3 periods; otherwise show "—" and no read.
+- Growth metrics (pp): latest − prior ≥ +2.0 → "accelerating";
+  ≤ −2.0 → "decelerating"; else "steady".
+- Margins (latest vs mean of prior periods, pp): ≥ +1.0 → "improving";
+  ≤ −1.0 → "deteriorating"; else "stable" (phrased "above/below own average"
+  where the layout calls for it).
+- Share count YoY: > +1.0% → "N% dilution"; < −1.0% → "buying back"; else "flat".
+- Peer-relative: favorable_percentile ≥ 60 → favorable ("above peers");
+  ≤ 40 → adverse ("priced above peers" for P/E, "elevated leverage" for
+  net-debt/EBITDA); else "in line".
+- Header sentence: join the growth read, margin read and peer-relative valuation
+  read with " · ", omitting segments that have no read (e.g. "Growth accelerating
+  · margins stable · valuation above industry median"). **Segment sources are
+  fixed:** growth = revenue growth read; margins = operating margin read;
+  valuation = P/E peer-relative read, falling back to FCF yield when P/E is null.
+  This keeps the header unambiguous when sibling metrics (EPS vs revenue growth,
+  P/E vs FCF yield) point in different directions.
+
+## Decommissioning (end of workstream A)
+
+Remove completely: FMP (`app/providers/fmp.py`), Finnhub + Alpha Vantage
+(`app/providers/fundamentals_chain.py`), their config keys (`app/config.py`), and
+their wiring in `app/scheduler.py`, `app/routers/ingestion.py`,
+`app/services/ticker_universe_service.py`. Retain: Alpaca (prices), FRED,
+sentiment provider, Telegram. Note: decommissioning does **not** depend on
+workstream B — Alpaca remains the price source throughout.
+
+## Rollout
+
+**Workstream A:**
+
+- A0. License review **DONE** (earnings approved for private/internal use under
+  CC BY-SA 4.0, no redistribution — see Licensing above). The Dolt version,
+  persistent `DOLT_DATA_DIR`, clone, and production checks are captured in
+  `deploy/provision_fundamentals.sh` and `docs/fundamentals-deployment.md`.
+- A1. Migration 026, import-run framework.
+- A2. Earnings ingestion in shadow (writes `earnings_events`, prod untouched);
+  verify forward-calendar coverage and rescheduling behavior.
+- A3. SEC daily job in shadow (writes `fundamental_snapshots`). **Primary technical
+  risk here: Q4 derivation and fiscal-period alignment** — non-calendar fiscal years,
+  restatements/amendments, and XBRL unit/dimension variants; budget accordingly.
+- A4. API v1 + FundamentalsPanel + peer comparison — served from snapshots and
+  earnings_events, independent of the scoring cutover (the additive API supports
+  partial data). UI value ships before anything touches scoring inputs.
+- A5. **Score-parity gate** → `fundamental_data` cutover: compute candidate
+  pe_ratio/revenue_growth/earnings_surprise from SEC/Dolt side by side with the
+  API values across the tracked universe, report per-field deltas and resulting
+  fundamental-score/ranking changes, require explicit approval. Definition
+  changes (e.g. TTM vs provider convention) called out, not averaged away.
+  **Status 2026-07-24: the gate has been exercised and the evidence supports
+  approval** — see the handoff section below. Step (c) is implemented behind the
+  default-off `fundamental_data_sec_dolt_cutover_enabled` SystemSetting; the
+  remaining production action is flipping that switch on and observing it.
+- A6. Remove FMP/Finnhub/Alpha Vantage; keep monitoring + manual fallback.
+
+**Workstream B (independent, start when wanted):**
+
+- B0. Stocks clone (~4.7 GB) provisioned; migration 027.
+- B1. OHLCV + split adjustment in shadow (writes `ohlcv_source_bars` only; Alpaca
+  keeps owning `ohlcv_records`); historical backfill.
+- B2. Reconciliation window (≥ 2 weeks) vs Alpaca; review validation summaries.
+- B3. Promote Dolt as historical OHLCV source (canonical rebuilt from raw source
+  bars + splits); morning pipeline → 03:00.
+
+## Test plan
+
+- Daily SEC job: changed revision imports; unchanged conditional-HTTP check is a
+  `no_op` with zero downloads and zero row changes; validation failure leaves
+  production untouched; source unavailable still runs the local
+  `fundamental_data` refresh (step c, post-activation); before activation the job
+  never writes `fundamental_data`.
+- Valuation endpoint returns identical values with SEC reachable and unreachable
+  (pure PostgreSQL computation); no valuation rows exist in any table.
+- Multi-class tickers resolve to the same CIK snapshots; peer medians and
+  peer_count are CIK-deduplicated (GOOG+GOOGL = one observation).
+- Earnings rescheduling: a moved future date replaces the old row atomically; a
+  cancelled date disappears; historical results are never touched.
+- Amendment selection: for a period with multiple accessions, the newest valid
+  accepted_at wins at read time; older rows remain unchanged.
+- History arrays are chronological, ≤ 4 points.
+- Percentage-point units stay compatible with existing formatters and scoring
+  inputs.
+- Deterministic reads: threshold boundary cases (exactly +2.0pp, exactly 60th
+  percentile) resolve per the stated rules; header uses identical outputs and
+  falls back from P/E to FCF yield for the valuation segment when P/E is null.
+- Peer comparison disappears below 5 peer issuers; favorable-percentile direction
+  correct for both polarities.
+- Workstream B: split-adjusted OHLCV matches Alpaca on representative normal /
+  split / reverse-split symbols.
+- UI states: positive, adverse, neutral, insufficient history, insufficient
+  peers; mobile layout; non-color accessibility.
+- Unit, integration, scheduler and frontend suites pass.
+
+## Acceptance criteria
+
+- App works normally with Dolt/DoltHub/SEC unreachable.
+- Re-running the same revision: zero duplicate or changed rows.
+- **Upcoming earnings dates present and timely for the tracked universe** — the
+  forward calendar is the hardest thing to replace and gates decommissioning.
+- Coverage meets the tracked-universe target; scheduler runs cleanly with
+  FMP/Finnhub/AV keys removed from the environment.
+- Score-parity diff reviewed and approved before `fundamental_data` cutover.
+- Scheduled imports never block the API event loop.
+
+## Handoff — remaining work after the A5 parity investigation (2026-07-24)
+
+The 2026-07-23 parity report surfaced coverage gaps and wrong values; a nine-pass
+investigation traced every one to parser/identity bugs (not source data), fixed them,
+and reparsed production twice. Full evidence trail:
+`reports/fundamentals-parity-20260723-findings.md` (root causes, decisions, validation)
+plus the before/after reports (`fundamentals-parity-20260723T…` / `…20260724T….json`).
+Post-fix: candidate scores 504 of 511 vs legacy's 507 (gap = PSKY/Q new registrants +
+FITB, all explained); revenue-growth agreement 0.0038 median abs delta where both exist.
+Dennis reviewed the evidence 2026-07-24 and directed proceeding to cutover.
+
+**Task 1 — A5 activation (IMPLEMENTED 2026-07-24; production switch remains).** The
+post-activation local refresh of `fundamental_data` derives `pe_ratio` and
+`market_cap` from newest valid snapshots × latest PostgreSQL close, `revenue_growth`
+from snapshots, `earnings_surprise`/`next_earnings_date` from `earnings_events`; mark
+affected cached fundamental scores stale; must run identically when SEC is unreachable.
+It consumes `fundamentals_derivation.derive()` outputs, NOT raw snapshot fields —
+that path carries the split guard (`ttm_diluted_eps`
+nulls when contaminated, with `ttm_diluted_eps_caveat`) and the multi-class share
+fallback (`shares_outstanding` + `shares_outstanding_estimated`). Parity and activation
+share the same candidate builder. Activation is the explicit
+`fundamental_data_sec_dolt_cutover_enabled` SystemSetting and defaults off. It is
+managed by the **Fundamentals data source** card in Admin → Settings; while active,
+the weekly legacy collector skips itself so it cannot overwrite the SEC/Dolt cache.
+See `docs/fundamentals-deployment.md` for the production flip and rollback procedure.
+
+**Task 2 — A6 decommissioning.** After a short observation window: remove
+FMP/Finnhub/Alpha Vantage providers, config and env keys; keep monitoring + manual
+fallback. Gated by the acceptance criteria above — especially forward-calendar
+timeliness from `dolt_earnings` (its `source_max_date` ran ~5 weeks ahead as of
+2026-07-23, which passes).
+
+**Known caveats to carry (documented in the findings report, not bugs to fix):**
+- KLAC-class post-filing splits: P/E wrong until the next 10-Q; undetectable from
+  snapshots. Workstream B's `corporate_actions` table is the natural future fix.
+- BRK-B: no share count exists anywhere in companyfacts → no market cap, correctly.
+- FITB: unscored (split guard + no taggable revenue) — the one name that lost its
+  score relative to legacy; composite renormalises.
+- Share-change guard at 25% nulls P/E for stock-funded M&A too (COF, WAT…);
+  revisit only if the ~3% universe hit-rate proves painful.
+- `sec_cik_overrides` SystemSetting pins XOM → 34088 (applied in prod); the
+  `no_xbrl_filings` SystemEvent says when a new pin is needed.
+- After any future parser change, stored rows need `scripts/reparse_fundamentals.py`
+  (dry-run default; `--apply` rewrites) — snapshots are otherwise immutable.
+
+## Deferred (explicitly, until a concrete need appears)
+
+- Workstream B itself is deferred relative to A and blocks nothing in A.
+- Exact byte-level source replay of historical imports; permanent archive store.
+- Point-in-time backtest enforcement (`accepted_at` is stored now; derivation and
+  backtest visibility rules are built only when fundamentals enter
+  scoring/backtesting).
+- Fundamental metrics in the score; sector-relative scoring.
+- Aggregate/rollup tables for peer statistics.
+- Any valuation cache or table (request-time computation from snapshots + latest
+  close suffices).
+- A dedicated conflicts table (validation JSON + logs suffice).

docs/dolt-sec-a3-design.md

@@ -0,0 +1,252 @@
+# A3 design — SEC fundamentals importer
+
+Status: **design APPROVED 2026-07-22 — three decisions signed off (daily-index
+fetch, primary-period-only snapshots, full-history backfill) + four review
+correctness fixes folded in (composite revision incl. universe fingerprint;
+submissions pagination shards for full history; index↔Company-Facts consistency
+gate; insert-only immutability with discrepancy reporting; deterministic cash/debt
+composition). Ready to implement.**
+Companion to `docs/dolt-integration-plan.md` (workstream A, phase A3). Grounded in
+live SEC data probes (Apple CIK 0000320193, company_tickers, submissions, daily-index).
+
+## Objective (unchanged from the plan)
+
+Populate `fundamental_snapshots` (CIK-keyed, one immutable row per accession)
+and `tickers.cik/sic/sic_description` from SEC data, as a `SourceImporter`
+plugging into the A1 framework. Shadow only (A3): nothing reads snapshots until
+A4; `fundamental_data` is untouched until the A5 parity gate. All new metrics
+are display-only.
+
+## What the SEC data actually looks like (probed, not assumed)
+
+`data.sec.gov/api/xbrl/companyfacts/CIK##########.json` — one JSON per **issuer
+(CIK)** aggregating every period across every filing. Shape:
+`facts.us-gaap.<Concept>.units.<unit>[] = {start, end, val, fy, fp, form, filed, accn, frame}`.
+
+Ground-truth findings that drive the design:
+
+1. **`fp` is only `Q1|Q2|Q3|FY` — there is no `Q4`.** Q4 must be derived.
+2. **`fy`/`fp` are the *filing's* fiscal context, not each fact's period.** Proven:
+   Apple's FY2019 10-K carries a discrete Q3-FY2018 revenue fact
+   (`start 2018-07-01, end 2018-09-29, val 62.9B`) tagged `fp=FY` — it's a
+   comparative. **Period identity lives in `(start, end)` + the filing's
+   `reportDate`, never in `fp/fy`.** Selecting values by `fp` would silently mix
+   comparatives into the wrong period.
+3. SEC provides **both** discrete 3-month facts **and** YTD-cumulative facts
+   (Apple Q2 FY26: YTD `254,940` over 6mo *and* discrete `111,184` over 3mo;
+   `143,756 + 111,184 = 254,940`). This confirms the stored-YTD schema: store
+   cumulative YTD per filing, derive discretes/Q4/TTM at read time.
+4. Instant facts (`dei:EntityCommonStockSharesOutstanding`) end on the **cover
+   date** (2026-04-17), which differs from `period_end` (2026-03-28) → the
+   `shares_outstanding_date` column added in migration 026.
+5. **No conditional-GET support:** the companyfacts endpoint returns no `ETag`
+   and no `Last-Modified`. AAPL's file is 3.75 MB. So ~505 unconditional fetches
+   ≈ 0.5–1.5 GB *per run* — the plan's "conditional HTTP no-op" is impossible on
+   this endpoint. This is the fact that decides the fetch strategy (below).
+6. `submissions/CIK##########.json` supplies `sic`, `sicDescription`,
+   `fiscalYearEnd` (e.g. `0926`), and per-accession `reportDate` +
+   `acceptanceDateTime` — the keys for period selection and `accepted_at`.
+7. `company_tickers.json` uses **dash** tickers (`BRK-B`, `BRK-A`) and maps
+   `GOOGL`/`GOOG` to the **same** `cik_str` (1652044). The ticker→CIK join reuses
+   the earnings importer's `normalise_symbol` (dot→dash), so both sides match.
+
+## Decision 1 (APPROVED) — fetch strategy: EDGAR daily-index driven
+
+**Plan said** bulk `companyfacts.zip` + ETag no-op. **Reality:** the data.sec.gov
+endpoints expose no validators, and the bulk zip is multi-GB and changes ~daily
+(all of EDGAR), so ETag would rarely match → near-daily multi-GB download to get
+505 issuers. Per-CIK *conditional* fetch is impossible (finding 5). Per-CIK
+*unconditional* is 0.5–1.5 GB every night.
+
+**Recommended:** drive off the **EDGAR daily-index** (`daily-index/YYYY/QTRn/
+form.YYYYMMDD.idx` — fixed-width Form/Company/CIK/Date/accession, ~3300 rows/day,
+confirmed). Each run:
+
+- `detect_revision` → a **composite revision**, not just the date:
+  `latest-index-date` + a hash of the index content processed this run + a
+  **fingerprint of the tracked symbol→CIK set**. The CIK fingerprint is essential:
+  a newly added ticker changes the revision and forces a run, so a new ticker is
+  never `no_op`'d away or starved waiting for its issuer to file. Equal composite
+  revision → `no_op`.
+- **No backfill sentinel.** The *absence of a prior promoted run* is what triggers
+  the initial full-history backfill; `source_max_date` records the processed index
+  date each run.
+- `stage` → for each index date since the last processed one, parse the form
+  index, keep rows where `form ∈ {10-K, 10-Q, 10-K/A, 10-Q/A}` **and** CIK ∈
+  tracked set, then fetch `companyfacts/CIK.json` for **only those few issuers**
+  and extract their newly-reported period(s). Most nights this is a handful of
+  issuers → near-zero transfer, respectful of SEC fair-access.
+- **First run (backfill)**: no prior promoted run → fetch companyfacts for all
+  tracked CIKs once (~1 GB one-time) and seed **full** history. Full history needs
+  the paginated submissions shards — see "CIK resolution" below.
+
+Why this over the alternatives: transfer scales with *filings*, not with all of
+EDGAR or with the universe size every night; it restores the revision/no_op
+model; and it's the lightest load on SEC. Cost: daily-index parsing + date
+bookkeeping (store last-processed index date in `data_import_runs` /
+settings). **This deviates from the plan's "bulk zip" — requesting sign-off.**
+
+## Decision 2 (APPROVED) — snapshot mapping: primary-period, YTD, immutable
+
+One `fundamental_snapshots` row per accession, representing the filing's
+**primary current period only** (not its comparatives):
+
+- **Select the primary period by `end == submissions.reportDate[accn]`** (finding
+  2), *not* by `fp/fy`. `fiscal_period` label comes from the filing's own `fp`
+  (a 10-Q's own `fp` matches its current quarter; a 10-K → `FY`);
+  `fiscal_year`/`period_start`/`period_end` from the selected facts + submissions.
+- **Duration facts → cumulative YTD.** For each concept, pick the duration fact
+  with `accn == thisFiling`, `end == reportDate`, and `start ≈ fiscal-year start`
+  (derived from `fiscalYearEnd`), sanity-checked by span length (Q1≈3mo, Q2≈6mo,
+  Q3≈9mo, FY≈12mo). **If the YTD fact is absent, store null — never a discrete
+  masquerading as cumulative** (that would poison read-time differencing).
+- **Balance-sheet instants → at `end == reportDate`.** `shares_outstanding` is
+  the exception: prefer the `dei:EntityCommonStockSharesOutstanding` cover-page
+  fact and store *its own* `end` in `shares_outstanding_date` (cover date ≠
+  period_end); when there is no dei fact (e.g. Alphabet) fall back to
+  `us-gaap:CommonStockSharesOutstanding` at `reportDate`. A single consolidated
+  value — never a class sum (companyfacts is non-dimensional) nor
+  weighted-average/diluted; conflicting values → null.
+- **Amendments:** a real `10-K/A` / `10-Q/A` is a new accession → a new immutable
+  row for the same `(cik, fy, fp)`; readers pick the newest valid `accepted_at`.
+- **Out of scope (stated, not silent):** restatements that appear *only* as
+  comparatives inside a later normal filing are **not** captured — only a real
+  amendment updates a prior period. This narrows the plan's "newest accepted_at
+  per period" to amendment-driven updates; a deliberate KISS boundary.
+- **Immutable means insert-only, not upsert.** `promote` **inserts** new accession
+  rows with `ON CONFLICT (accession) DO NOTHING`. An accession never mutates: if a
+  re-fetch reconstructs *different* values for an accession already stored, that is
+  a **discrepancy to report** (into `validation_json` + a system event), never a
+  silent overwrite, and the original `import_run_id` is never replaced. (Ordinary
+  updates arrive as a *new* amendment accession, which is a new row.)
+
+## Read-time derivation (constrains the importer; built in A4)
+
+From the per-accession YTD rows, all at read time (newest `accepted_at` per
+period), following the schema decision already in the plan:
+
+- discrete quarter = YTD(Qn) − YTD(Qn−1); **Q4 = FY − YTD(Q3)**.
+- TTM = sum of the trailing four discrete quarters (e.g. TTM@Q2 = FY(prev) +
+  YTD(Q2) − YTD(Q2 prev year)).
+- YoY = period vs same period a year earlier.
+- **Hard rule the importer must enable: any missing period in a run → the derived
+  value is `null`, never a partial number.** So the importer must aim for
+  complete consecutive quarter runs per issuer and report gaps.
+
+## Metric tag catalog (prioritized us-gaap tags + fallbacks)
+
+Tagging is inconsistent across issuers (the plan's known risk). Each metric
+resolves through an ordered tag list; first present wins; unit-checked.
+
+| Snapshot field | Primary tag | Fallbacks | Unit |
+|---|---|---|---|
+| revenue | `RevenueFromContractWithCustomerExcludingAssessedTax` | `Revenues`, `SalesRevenueNet` | USD |
+| net_income | `NetIncomeLoss` | — | USD |
+| operating_income | `OperatingIncomeLoss` | — | USD |
+| diluted_eps | `EarningsPerShareDiluted` | — | USD/shares |
+| cfo | `NetCashProvidedByUsedInOperatingActivities` | `...ContinuingOperations` | USD |
+| capex | `PaymentsToAcquirePropertyPlantAndEquipment` | `PaymentsToAcquireProductiveAssets` | USD |
+| depreciation_amortization | `DepreciationDepletionAndAmortization` | `DepreciationAmortizationAndAccretionNet`, `DepreciationAndAmortization` | USD |
+| cash_and_st_investments | see composition rule | — | USD |
+| total_debt | see composition rule | — | USD |
+| shares_outstanding | `dei:EntityCommonStockSharesOutstanding` | — | shares |
+
+**Composite fields — deterministic, aggregate-first, no double counting.** Each
+source tag contributes at most once:
+
+- `cash_and_st_investments` = `CashAndCashEquivalentsAtCarryingValue`
+  **+ short-term investments**, where ST investments = the **first present** of
+  [`ShortTermInvestments`, `MarketableSecuritiesCurrent`] — never both summed.
+- `total_debt` = **long-term component + short-term component**, where
+  - long-term = first present of [`LongTermDebt` (the aggregate, already includes
+    current + noncurrent portions), **else** (`LongTermDebtNoncurrent` +
+    `LongTermDebtCurrent`)];
+  - short-term borrowings = first present of [`ShortTermBorrowings`,
+    `CommercialPaper`] (0 if neither).
+  So the long-term aggregate and its components are mutually exclusive, and CP vs
+  short-term-borrowings is a single pick — nothing is counted twice.
+
+EBITDA (for net-debt/EBITDA) is derived at read time = operating_income + D&A.
+Concepts absent for an issuer → that field is null (display-only; no synthesis).
+The exact tag lists live as named constants, tunable without touching logic.
+
+## Fiscal-period identity
+
+`fiscalYearEnd` (MMDD from submissions) anchors the fiscal-year start for YTD
+span checks and Q4 derivation. Non-calendar fiscal years (Apple's Sept) are
+handled because we key on `(start, end)` + `reportDate`, not calendar quarters.
+`fiscal_year`/`fiscal_period` are stored from the filing's own `fy`/`fp` for its
+primary period (safe — a filing's own context is correct for its current period).
+
+## CIK resolution & tickers backfill
+
+- From `company_tickers.json`: `normalise_symbol(ticker) → cik_str`. Set
+  `tickers.cik` for each tracked ticker (multi-class share one CIK).
+- From `submissions/CIK.json`: `sic`, `sicDescription`, `fiscalYearEnd` →
+  `tickers.sic/sic_description` (+ fiscal anchor for YTD/Q4).
+- **Submissions is paginated — full history needs the shards.** `filings.recent`
+  holds only the **latest 1000** filings (verified: Apple `recent` = 1000). Older
+  accessions live in `filings.files[]` = `[{name, filingFrom, filingTo,
+  filingCount}]` (e.g. `CIK0000320193-submissions-001.json`, 1236 filings
+  1994–2015), each a bare object with the **same parallel arrays** including
+  `reportDate`, `acceptanceDateTime`, and `isXBRL`. The full-history backfill must
+  **follow every `filings.files[].name`** to obtain period identity + `accepted_at`
+  + `isXBRL` for pre-1000 accessions. Incremental runs only need `recent`.
+- Refreshed by the SEC job; a newly added ticker self-resolves on its next run
+  (the CIK fingerprint in the revision forces that run) — until then its snapshots
+  are absent → metrics null, per the plan.
+
+## SourceImporter mapping (source = `sec_facts`)
+
+- `detect_revision` → latest daily-index date (or `backfill` sentinel on first run).
+- `stage` → resolve tracked CIKs; (incremental) parse indices since last date →
+  tracked filers → fetch their companyfacts → build per-accession snapshot rows;
+  (backfill) fetch all tracked companyfacts. In-memory staged set (KISS, per A1).
+- `validate` (fail-closed) → tracked-universe **coverage floor** (issuers with ≥1
+  snapshot); **unit/period sanity** (YTD spans within tolerance; EPS in USD/shares);
+  **no duplicate accession**; **filings skipped for missing period identity are
+  counted in `validation_json`** (carry-forward from A1 review); an unexpected
+  companyfacts shape (missing `facts`/`units`) stops promotion.
+- **Index↔Company-Facts consistency gate (the daily index and Company Facts are
+  separate SEC products that can lag each other):** for every tracked index
+  accession marked `isXBRL`, confirm that accession actually appears in the fetched
+  companyfacts before promotion. If any is missing → **fail the run and retry
+  later** — do **not** advance the revision and do **not** record an
+  incomplete/null snapshot for it. Non-XBRL amendments are skipped with a recorded
+  reason in `validation_json`. (The framework only stores the revision on a
+  promoted run, so a failed consistency check naturally leaves the revision behind
+  for retry.)
+- `promote` → **insert** snapshot rows (`ON CONFLICT (accession) DO NOTHING`;
+  immutable — see Decision 2), stamped `import_run_id`; refresh
+  `tickers.cik/sic/sic_description`. A re-fetch that reconstructs different values
+  for an existing accession is reported as a discrepancy, never a silent mutation.
+  Non-destructive (append-only accessions) — no future-row deletion like earnings.
+
+## SEC fair-access (operational, per the plan's non-negotiable)
+
+Identifying `User-Agent` with contact email on every request; well under 10 req/s
+with spacing; exponential backoff on 429; **403 → alert and stop, never
+retry-loop** — with one carved-out exception: `www.sec.gov/Archives` is served
+from an S3 bucket without a `ListBucket` grant, so an **absent** file 403s with
+S3's `AccessDenied` XML rather than 404 (every weekend/holiday daily index does
+this). That one shape is read as "missing"; a real rejection is the WAF's
+`text/html` "Undeclared Automated Tool" page and still stops the run.
+New config: `sec_user_agent`, `sec_request_spacing_seconds`,
+`sec_max_retries`. Keep only the last ~2 fetched artifacts on disk for debugging
+(reproducibility is the normalized Postgres rows, per the plan).
+
+## Explicitly out of scope for A3
+
+- `fundamental_data` cutover (A5 parity gate) — snapshots only in A3.
+- The read-time derivation, API object, and panel (A4).
+- Comparative-only restatements (Decision 2).
+- Point-in-time backtest enforcement (`accepted_at` stored, not yet enforced).
+
+## Decisions (signed off 2026-07-22)
+
+1. **Fetch** — EDGAR daily-index driven (Decision 1). Approved deviation from the
+   plan's bulk zip.
+2. **Snapshot mapping** — primary-period-only per accession; comparative-only
+   restatements out of scope (Decision 2). Approved.
+3. **Backfill depth** — seed **full** available history per issuer on first run
+   (cheap to store; powers the quarter tape / multi-year YoY). Approved.

docs/fundamentals-deployment.md

@@ -0,0 +1,257 @@
+# Fundamentals production deployment
+
+This is the one-time production setup for the Dolt earnings and SEC fundamentals
+imports. The A5 scoring cutover was approved on 2026-07-24; the compat-cache write
+path is still default-off until the explicit production switch below is set. Do
+not add OS cron entries: the application scheduler owns both jobs.
+
+## What the deployment adds
+
+- `Dolt Earnings Import (shadow)` runs daily at 02:30 America/New_York.
+- `SEC Fundamentals Import` runs daily at 04:00 America/New_York. Its local
+  `fundamental_data` refresh runs only when the A5 switch is enabled.
+- `Fundamentals Parity Report (read-only)` runs daily at 05:30 America/New_York.
+- Both jobs are visible, toggleable, and manually triggerable in Admin → Jobs.
+- Cron expressions are editable in Admin → Schedule.
+- Every attempt is recorded in `data_import_runs`; failures also create a system
+  event. A failed validation does not promote partial data.
+- An SEC filing still missing after the short publication-lag window enters
+  `sec_filing_gaps`. The daily importer retries it automatically; affected
+  tickers are excluded from actionable setups until a snapshot is recovered or
+  a later valid 10-K/10-Q supersedes the gap. Migration `028` materializes older
+  promoted gaps into this queue once, so setup reads never scan import history.
+
+The systemd service uses one application worker. The import framework also holds
+a PostgreSQL advisory lock per source, so an overlapping manual/scheduled run is
+skipped safely.
+
+## Prerequisites
+
+The production `.env` at `/opt/signalplatform/.env` must contain:
+
+```dotenv
+DOLT_BINARY=/usr/local/bin/dolt
+DOLT_DATA_DIR=/var/lib/signal-platform/dolt
+DOLT_EARNINGS_SUBDIR=earnings
+DOLT_MIN_FREE_DISK_GB=5.0
+SEC_USER_AGENT=signal-platform/1.0 (contact: real-address@example.com)
+SEC_REQUEST_SPACING_SECONDS=0.2
+FUNDAMENTALS_PARITY_REPORT_DIR=/var/lib/signal-platform/reports/fundamentals-parity
+```
+
+Use a real monitored contact address. Keep at least 5 GB free at the Dolt data
+path; 8–10 GB gives comfortable growth headroom. The data directory must stay
+outside `/opt/signalplatform`, because deployments use `rsync --delete` there.
+The parity-report directory is also persistent and owned by the service user;
+its small timestamped JSON/CSV bundles form the temporary A5 review trail.
+
+## One-time provisioning
+
+First deploy the commit containing this bundle to production. Then SSH to the
+server and run:
+
+```bash
+cd /opt/signalplatform
+sudo bash ./deploy/provision_fundamentals.sh
+sudo -u deploy bash ./deploy/provision_fundamentals.sh --check
+sudo systemctl restart signalplatform.service
+curl -fsS http://127.0.0.1:8998/api/v1/health
+```
+
+The provisioner is idempotent. It installs the pinned Dolt version, creates the
+persistent directory as `deploy:deploy`, clones
+`post-no-preference/earnings`, configures a repository-local author identity for
+`dolt pull`, verifies free space and `.env`, and refuses an unexpected
+Dolt version. It does not modify PostgreSQL or start an import. The public clone
+does not require `dolt login`.
+
+For a server provisioned before the author-identity check was added, repair the
+existing clone once with:
+
+```bash
+sudo -u deploy -H /usr/local/bin/dolt config --global --add user.name "Signal Platform"
+sudo -u deploy -H /usr/local/bin/dolt config --global --add user.email "signal-platform@localhost"
+```
+
+Do not replace the pinned version with `latest`. A future Dolt upgrade should be
+a reviewed change to `DOLT_VERSION`, followed by the same provision/check flow.
+
+## First-run verification
+
+In Admin → Jobs, wait until no other job is running, then:
+
+1. Trigger **Dolt Earnings Import (shadow)**. Expect `completed` with import
+   status `promoted`; a repeat without an upstream change should report `no_op`.
+2. Trigger **SEC Fundamentals Import**. The first run performs the
+   tracked-universe history backfill and can take materially longer than a daily
+   incremental run. Expect `completed` with import status `promoted`.
+3. Check Admin → System Events. There should be no new import error.
+4. Confirm the next-run times correspond to 02:30 and 04:00 New York time.
+5. Open several ticker pages and confirm the fundamentals panel has populated
+   data and still handles partial/missing issuers cleanly. A ticker held by the
+   quality gate should show **New setups paused** with the specific SEC reason.
+
+## A5 parity observation window
+
+After both shadow imports are healthy, trigger **Fundamentals Parity Report
+(read-only)** once in Admin → Jobs. The **A5 Fundamentals Parity** card above
+the jobs shows the latest coverage/delta summary and provides authenticated JSON
+and CSV downloads. The canonical server-side bundles are archived at:
+
+```text
+/var/lib/signal-platform/reports/fundamentals-parity/
+```
+
+The scheduler then generates one report daily at 05:30 New York time, after the
+02:30 Dolt and 04:00 SEC jobs. Review 5–7 consecutive reports before making the
+cutover decision. A report never writes `fundamental_data`, dimension/composite
+scores, rankings, qualification state, or an approval flag. Materiality bands
+only highlight rows for review; A5 still requires explicit approval.
+
+Each bundle contains legacy and candidate P/E, revenue growth, and earnings
+surprise; definition notes; source revisions and price dates; recomputed legacy
+and candidate fundamental scores; and per-universe fundamental-rank changes.
+Definition changes remain explicit even when numeric deltas are small.
+
+Optional database verification:
+
+```sql
+SELECT source, status, revision, source_max_date, started_at, completed_at,
+       validation_json
+FROM data_import_runs
+WHERE source IN ('dolt_earnings', 'sec_facts')
+ORDER BY id DESC
+LIMIT 10;
+
+SELECT count(*) FROM earnings_events WHERE source = 'dolt_earnings';
+SELECT count(*), count(DISTINCT cik) FROM fundamental_snapshots;
+```
+
+During the longer first SEC run, execute the following in a second SSH session.
+It opens an independent database connection and attempts the same source lock:
+
+```bash
+cd /opt/signalplatform
+sudo -u deploy .venv/bin/python - <<'PY'
+import asyncio
+
+from sqlalchemy import text
+
+from app.database import engine
+from app.services.data_import import _advisory_key
+
+
+async def main():
+    key = _advisory_key("sec_facts")
+    async with engine.connect() as connection:
+        acquired = await connection.scalar(
+            text("SELECT pg_try_advisory_lock(:key)"), {"key": key}
+        )
+        print("UNEXPECTED: lock acquired" if acquired else "OK: source lock is busy")
+        if acquired:
+            await connection.execute(
+                text("SELECT pg_advisory_unlock(:key)"), {"key": key}
+            )
+
+
+asyncio.run(main())
+PY
+```
+
+Expect `OK: source lock is busy`. This is the remaining live-PostgreSQL
+mutual-exclusion check; SQLite unit tests cannot exercise PostgreSQL advisory
+locks. A second Admin trigger should independently report the job as busy.
+
+## A5 production activation (approved 2026-07-24)
+
+The write path is controlled by the SystemSetting
+`fundamental_data_sec_dolt_cutover_enabled`. An absent value, `false`, or any
+value other than `true` leaves `fundamental_data` untouched. Before enabling it,
+confirm the normal PostgreSQL backup containing `fundamental_data` is current.
+
+In **Admin → Settings → Fundamentals data source**:
+
+1. Turn on **Use SEC + Dolt for scoring inputs** and accept the confirmation.
+2. Click **Run refresh now**. The SEC import may be `promoted` or `no_op`; either
+   result runs the local cache refresh.
+
+The weekly legacy collector is automatically skipped while the switch is on, so
+it cannot overwrite the activated cache. The switch remains visible even before
+its SystemSetting row exists because the safe default is off.
+
+If the Admin UI is unavailable, enable the cutover directly in PostgreSQL:
+
+```sql
+INSERT INTO system_settings (key, value, updated_at)
+VALUES ('fundamental_data_sec_dolt_cutover_enabled', 'true', now())
+ON CONFLICT (key) DO UPDATE
+SET value = EXCLUDED.value, updated_at = now();
+```
+
+Then trigger **SEC Fundamentals Import** once in Admin → Jobs. Once enabled, the
+same refresh also runs after an SEC network/validation failure or a source-lock
+skip, because it reads only PostgreSQL snapshots, earnings events, and closes.
+The job message appends the cache row count and changed score-input count when
+the import itself completed successfully.
+
+Verify the switch and refreshed rows:
+
+```sql
+SELECT key, value, updated_at
+FROM system_settings
+WHERE key = 'fundamental_data_sec_dolt_cutover_enabled';
+
+SELECT count(*) AS rows,
+       max(fetched_at) AS refreshed_at,
+       count(pe_ratio) AS pe_available,
+       count(revenue_growth) AS growth_available,
+       count(earnings_surprise) AS surprise_available,
+       count(next_earnings_date) AS next_date_available
+FROM fundamental_data;
+
+SELECT dimension, is_stale, count(*)
+FROM dimension_scores
+WHERE dimension = 'fundamental'
+GROUP BY dimension, is_stale;
+
+SELECT is_stale, count(*)
+FROM composite_scores
+GROUP BY is_stale;
+```
+
+The first refresh intentionally marks affected fundamental and composite score
+caches stale. The normal 15:30 near-close scanner recomputes them before using
+the rankings; until then, reads truthfully expose the stale state. Observe at
+least several scheduled cycles before A6 removes the legacy providers.
+
+## Failure and rollback
+
+- To stop the A5 cache writes without stopping SEC snapshot ingestion, turn off
+  **Use SEC + Dolt for scoring inputs** in Admin → Settings. If the UI is
+  unavailable, set `fundamental_data_sec_dolt_cutover_enabled` back to `false`
+  with the SQL above (changing only the value). This prevents the next local
+  refresh but does not restore rows already replaced. Restore `fundamental_data`
+  from the pre-cutover database backup, or—before A6—manually run the legacy
+  Fundamental Collector if its provider keys and quota are still available.
+- Disable a failing source-import job in Admin → Jobs only when ingestion itself
+  must stop. Existing promoted snapshots/events remain available.
+- Inspect the job runtime, latest `data_import_runs.validation_json`, service
+  logs, and Admin → System Events before retrying.
+- `unresolved_filing` is emitted once when a filing enters automatic retry. It
+  does not require a server command. If the gap is still current after 14 days,
+  `filing_gap_aged` is emitted once with the CIK, accession, and parser/mapping
+  reason. A later valid 10-K/10-Q retires the gap even when the original SEC
+  accession never becomes usable.
+- Successful co-registrant recovery is logged without a warning. New registrants
+  with no XBRL history are also logged quietly, but their ticker page explains
+  that setups remain paused and that successor shells may need `sec_cik_overrides`.
+- Re-run `sudo -u deploy bash ./deploy/provision_fundamentals.sh --check` for
+  binary, clone, permission, disk, or environment failures.
+- The Dolt clone is a reproducible cache and does not need a bespoke backup.
+  PostgreSQL (including `earnings_events`, `fundamental_snapshots`, and import
+  audit rows) must remain covered by the normal production database backup.
+- Do not proceed to A6 until the activated cache has completed the observation
+  window and the forward earnings calendar remains timely.
+- If report generation fails, inspect Admin → System Events and verify
+  `FUNDAMENTALS_PARITY_REPORT_DIR` exists and is writable by `deploy`. Existing
+  reports and all live data remain untouched.

docs/research/README.md

@@ -8,7 +8,9 @@ was run and the data said no.** Detail lives in the linked docs and in
 **The one-line summary of the whole platform:** it is a **long-only
 cross-sectional momentum book** — buy the top quintile by beta-adjusted 12-1
 momentum, tilt toward higher volatility, hold ≤ 10 names, cut at 1.5× ATR, then
-trail at 3× ATR for up to 30 trading days. Everything else in the app (composite
+trail at 3× ATR for up to 30 trading days. After an initial stop, require the
+daily production gate to fail and subsequently qualify again before re-entry.
+Everything else in the app (composite
 score, Structural S/R, the Gate Target Ladder, sentiment, fundamentals) is
 **display or screening**, not edge.
 
@@ -22,7 +24,8 @@ score, Structural S/R, the Gate Target Ladder, sentiment, fundamentals) is
 | 80/20 residual-momentum / 6m-volatility rank | Ranking tilt | Buys ~2pp CAGR over momentum-only; costs ~6pp drawdown |
 | 1.5× ATR initial stop | Real exit | Cuts losers fast |
 | 3× ATR trailing stop, 30-day max hold | Real exit | Best Sharpe of every exit tested |
-| Max 10 concurrent positions, 1% risk per trade | Sizing | Cap never binds in practice |
+| Post-stop normal gate reset | Re-entry policy | Stop always closes; a later gate failure and subsequent fresh qualification define the next signal episode. The selected study arm reached Sharpe 1.77 / CAGR 48.3% at capacity 10; live scan-before-outcome timing is stricter (Sharpe 1.68 / CAGR 44.8% analogue). [Full study](post-stop-reentry.md) |
+| Max 10 concurrent positions, 1% risk per trade | Sizing | The cap binds by signal count, but the focused bracket found negligible opportunity cost: cap 15 admitted every blocked setup and added only 0.0018 R/trade in affected paths. [Findings](portfolio-capacity-bracket-findings.md) |
 | Structural S/R | Human-facing product context | Clean, capped zones for charts and alerts; not read by the scanner |
 | Gate Target Ladder | Screening machinery | Volume-free transient proposals preserve the production candidate set exactly; never an exit |
 
@@ -44,6 +47,7 @@ score, Structural S/R, the Gate Target Ladder, sentiment, fundamentals) is
 | 10 | **Inverse-vol position sizing** | The apparent "win" was **mis-attributed**: the 20% notional cap bound on 95% of entries, so it measured concentration, not vol-sizing. Genuine inverse-vol cuts DD to −18.2% but costs ~58pp return at flat Sharpe | **Rejected** as edge; it's a risk-preference trade | `backtest-20260709-position-sizing*.json` |
 | 11 | **FIP path-smoothness** as tie-breaker/filter | Non-monotonic within the qualified set; thinning the entry stream costs more compounding than the tilt returns | **Rejected as a filter** — but see §4, it's the strongest raw signal we've measured | — |
 | 12 | **Fixed take-profit sweep** (R-multiples) | No interior optimum ever found — the best TP is "no TP" | **Rejected.** Momentum's edge lives in the right tail | `backtest_service.py:450` |
+| 13 | **Sector-residual 12-1** (`mom_12_1_sector_resid` / sector demean) as replacement for market residual | Short-window IC/A/B looked knife-edge green; deep repaired + **liquid-1500** retest: weeks 83, mild +IC **0.027** / t 1.69, **below iron bar 0.03** (FAIL). Demean already weaker | **Rejected / closed.** Keep production market residual. Do not resurrect without a new pre-registered protocol | [sector-residual-momentum.md](sector-residual-momentum.md) · `sector-resid-deep-20260719-113319.json` · history-depth supersession note |
 
 ---
 
@@ -57,12 +61,13 @@ invites overfitting.
 |---|---|
 | ATR trail multiple {1.5–4.0} | **Keep 3.0** — ≤2.0 whipsaws out the right tail; ≥2.5 is a plateau |
 | Momentum lookback (6-1, 3-1, 12-7 Novy-Marx, composites) | **Keep residual 12-1** — the others have IC ≈ 0 or weaker t-stats |
-| Selection cutoff {70…90} × book size {10, 15, 20} | **Keep 80 × 10** — monotonically worse in both directions |
+| Selection cutoff {70…90} × book size {10, 15, 20} | **Keep 80 × 10** — the focused daily bracket found no meaningful gain from cap 15, while weekly rank replacement hurt. [Findings](portfolio-capacity-bracket-findings.md) |
 | Position sizing (equal-weight, inverse-vol, risk-% sweep) | **Keep 1% fixed-fractional** |
 | Primary-target probability floor | **Keep 20%** — pruned lottery targets, 1,428 → 1,089 qualified, lifted Sharpe |
 | Primary-target R:R selector | **Keep 1.5** — target choice is intentionally independent of the later 2.0 activation floor |
 | Exit policy (hold / SMA50 / 20-day low / technical-40 / ATR trail) | **Keep 3× ATR trail** — best Sharpe (2.04) |
 | **Activation R:R floor `min_rr`** (swept 2026-07-12) | **Keep 2.0** — best in-sample *and* out-of-sample. But it is a **spike, not a plateau** — see below |
+| Post-stop re-entry (nine daily policy arms) | **Keep normal gate reset at production capacity 10** — Sharpe 1.77 vs 1.67 immediate and 1.47 fixed cooldown 5. The result changes with book capacity; see [post-stop-reentry.md](post-stop-reentry.md) |
 
 ### The `min_rr` sweep (2026-07-12)
 
@@ -76,7 +81,7 @@ Reports: `backtest-20260712-min-rr-sweep.json` (in-sample), `-oos.json` (test wi
 | min_rr | qualified | In-sample Sharpe / CAGR | **OOS** Sharpe / CAGR (entries ≥ 2024-07) |
 |---|---|---|---|
 | 0.0 (floor off) | 6636 | 1.98 / 58.5% | 2.02 / 66.2% |
-| 1.2 (code default) | 3897 | 1.34 / 33.9% | 1.12 / 28.8% |
+| 1.2 (old code default) | 3897 | 1.34 / 33.9% | 1.12 / 28.8% |
 | 1.5 | 3127 | 1.20 / 29.6% | 1.12 / 28.8% |
 | 1.75 | 1974 | 1.64 / 44.5% | 1.15 / 27.4% |
 | **2.0 (live)** | 1089 | **2.04 / 50.4%** | **2.78 / 73.3%** |
@@ -101,22 +106,56 @@ and it would also sever the last dependency the *gate* has on the weak S/R detec
 
 ---
 
-## 4. Open leads
+## 4. Phase A matrix (2026-07-18) — closed
 
-| Lead | Why it's interesting | Blocker |
-|---|---|---|
-| **`fip_id`** (information discreteness over the 12-1 window) | **Strongest cross-sectional signal measured on this universe** — IC −0.045, t = −2.91, correct sign | Doesn't improve *this* book (the momentum gate already captures it in-sample). Revisit when the universe broadens |
-| **Broader universe** (`nasdaq_all`) | Strengthens every week's cross-section and the IC t-stat | Also where `fip_id` could become tradeable |
-| **Forward paper-trade record** | The only true out-of-sample evidence the snapshot cannot give | Time |
-| **Better target model for clear-air names** | The return is demonstrably there (#2 wins on raw CAGR in *both* train and test); it's the *flat* 3× ATR target that makes it too expensive in risk | Needs a per-name model, not a constant k×ATR |
+Full write-up: **[phase-a-matrix.md](phase-a-matrix.md)** ·
+`reports/research-matrix-phase-a.json`.
+
+| Arm | Decision |
+|---|---|
+| Max-hold {45,60,90} | **Note and move on** — validation glitter, train collapse (regime interaction) |
+| Equity-curve vol targeting | **Reject as edge** on this sample; park vt25 as optional DD insurance only |
+| Correlation caps | **Reject**; sector caps stay Phase B with reduced expectations |
+| Next-open fill | **Discovery, not reject** — honest deployable ~Sharpe 1.2 / CAGR 30% under overnight scanner. Decision baseline until near-close ships = `next_open` |
+| `fip_id` re-derive | **Validated** (IC −0.045, t = −2.92) |
+
+### Execution recovery (same day) — closed as evidence
+
+Full write-up: **[execution-recovery.md](execution-recovery.md)** ·
+`reports/execution_recovery_matrix.json`.
+
+| Finding | Decision |
+|---|---|
+| Monotone fill timing (next_open → stale → close) + DD recovery | **When you fill**, not decaying alpha |
+| Live bracket **[1.57, 1.77]** full Sharpe | Near-close expected near top of bracket; no more fill-timing sim |
+| Gap-cap | **Dead** — third tail-trim instance |
+| Auto-`recover: false` | Not a null — bar hit the lower-bound arm by 0.03 train SE |
+
+**Highest-leverage open work:** **ops** — move the single daily R:R scan to
+`America/New_York` near-close (checklist in execution-recovery.md). Not more research
+knobs.
 
 ---
 
-## 5. Method rules learned the hard way
+## 5. Open leads
+
+| Lead | Why it's interesting | Blocker |
+|---|---|---|
+| **Near-close / MOC execution (ops)** | Recovers overnight momentum drift left on the table by a morning EU scan; evidence closed | Schedule + fill_mode shipped; live paper validation ongoing |
+| **`fip_id` / liquid breadth** | Fingerprint −0.045 / t −2.91; liquid unconditional **−0.017 / t −1.85** (not green); mom-conditional **−0.088 / t −4.58** | **Parked.** Orphan +0.0575 died (snapshot race). Breadth did not strengthen resid-mom t-stat. Optional reopen = pre-registered two-arm liquid-1500 book first. See [fip-breadth-ic.md](fip-breadth-ic.md) |
+| **Broader universe** | Composition changes factor signs (fip tug-of-war); vol-tilt on breadth is only a **directional hypothesis** (auth. −0.048 / t −1.36) | Any prod broaden must re-validate 80/20 tilt; offline research only; research.sqlite requires completion manifest |
+| **Forward paper-trade record** | The only true out-of-sample evidence the snapshot cannot give | Time; mark entries at actual near-close fill once ops ships |
+| **Better target model for clear-air names** | The return is demonstrably there (#2 wins on raw CAGR in *both* train and test); it's the *flat* 3× ATR target that makes it too expensive in risk | Needs a per-name model, not a constant k×ATR |
+| **Minimum effective-risk floor** | In cap-never-bound paths, the confounded 0.5% floor arm removed about 8% of fills while EV rose from 0.328 to 0.399 R and PF from 1.60 to 1.75, with exposure nearly unchanged | Run the frozen single-variable cap-10 A/B. [Specification](effective-risk-floor-ab.md) / [capacity findings](portfolio-capacity-bracket-findings.md) |
+
+---
+
+## 6. Method rules learned the hard way
 
 1. **Nested lookback windows are NOT out-of-sample.** The clear-air result (#2) was
    clean, large, and consistent across five nested windows — and still died on a
-   proper train/test split by entry date. Use `BACKTEST_HOLDOUT_SPLIT`.
+   proper train/test split by entry date. Use `BACKTEST_HOLDOUT_SPLIT` / a named
+   validation window — and do not pretend a repeatedly opened window is pristine.
 2. **Check what population an ablation actually admits.** The blanket fallback (#3)
    looked like it tested the "resistance famine" hypothesis. It didn't — 65% of the
    setups it let in were a different population entirely, and they drove the result.
@@ -126,10 +165,21 @@ and it would also sever the last dependency the *gate* has on the weak S/R detec
 4. **The iron rule:** a signal earns its way into selection *only* through the
    factor harness — |mean IC| ≳ 0.03, consistent sign, `reliable: true` (≥ 12
    non-overlapping windows). Never let an unvalidated score gate setups.
+5. **Momentum filters are guilty of tail-trimming until proven otherwise.**
+   Independent failures: take-profit exits, FIP as an in-book filter, gap-up entry
+   caps. Cosmetic quality up, P&L down — the right tail *is* the edge.
+6. **Fill timing is part of the strategy.** Close-fill reports are not deployable
+   numbers for an overnight scanner. Grade promotion under the fill mode you will
+   actually trade.
+7. **Incomplete research artifacts are not results.** The Phase B +0.0575 / t +5.12
+   liquid-fip row was orphaned within hours: it raced a partially built
+   `research.sqlite`. Extender now writes a completion manifest; breadth mode
+   refuses without a match. Same class of protection as calendar-truncation
+   asserts — do not re-mythologize numbers computed on half a universe.
 
 ---
 
-## 6. Why we stay with the current strategy
+## 7. Why we stay with the current strategy
 
 Everything we've tried to add has either failed the backtest, failed
 out-of-sample, or turned out to be measuring something other than what it claimed.
@@ -142,6 +192,21 @@ it is internal screening machinery whose broad historical-price-traffic behavior
 was preserved explicitly and volume-free, with exact full-period parity. It is
 still neither market structure nor an exit. The one component that *does* have
 measured predictive edge is the momentum gate, and every knob on it has been
-swept and confirmed.
+swept and confirmed. After an initial-stop exit, that same gate now also defines
+when a new episode may begin: one later failed observation followed by a fresh
+qualification. The [daily re-entry matrix](post-stop-reentry.md) supports this
+for the current 10-position book, but not as a universal rule for other
+portfolio capacities.
+
+Capacity is now closed as a negative result. The current daily Phase A control
+does reject 519 qualified entries because the ten-slot book is full versus 472
+admitted trades, so the older weekly “cap never binds” claim was stale. But the
+clean cap-15 arm admitted every opportunity the strategy requested and added
+only 0.0018 R/trade in paths where cap 10 bound. Weekly current-rank replacement
+reduced mean EV and created substantial churn. Keep cap 10 and do not build the
+replacement policy. See the [frozen specification](portfolio-capacity-bracket.md)
+and the separate [capacity findings](portfolio-capacity-bracket-findings.md).
+The only open follow-up from that run is the
+[frozen confound-free 0.5% minimum effective-risk-floor A/B](effective-risk-floor-ab.md).
 
 The next real evidence is **forward**, not backward: the live paper-trade record.

docs/research/earnings-gap-and-sue.md

@@ -0,0 +1,167 @@
+# Earnings gap diagnostic + SUE / PEAD (Tier-1 alpha research)
+
+**Status:** **CLOSED — SUE DEAD**.
+**Branch:** `research/earnings-gap-and-sue`
+**Production impact:** none. Local research only; no earnings filter or SUE integration is shipped.
+
+---
+
+## Pre-registration (locked before the final research run)
+
+### Data
+
+- Historical earnings announcements for the production universe, stored in the
+  real `earnings_events` table and deduplicated on symbol + announcement date.
+- The originally requested 2016 start is amended, with user approval, to the
+  public source's announcement coverage start of 2020-01-22. Earlier EPS-period
+  history may scale later surprises but may never activate a live signal.
+- Report coverage, pairing, duplicates/restatements, annual-rate sanity, and
+  announcement-session quality before either experiment.
+- Point-in-time: an earnings surprise is usable only from announcement date +1
+  trading day. Same-day use is forbidden.
+
+### Experiment 2a — earnings-gap risk (defense, report-only)
+
+Run the production-config book on the approximately 505-name production
+universe with close fills and 0.001 transaction cost per side. Join simulated
+trades to earnings by symbol and date.
+
+1. Among closed trades with realized net R ≤ -1.0, report the fraction with an
+   announcement strictly after entry and before exit, alongside the base rate
+   for all trades.
+2. Compare entries within three trading sessions before an announcement with
+   all other entries: count, mean/median R, win rate, p05, and p95.
+3. Compare stops within one trading session after an announcement with all
+   other stops and exits.
+
+Verdict is always `INFORMATIONAL`. Report only: no filter arm, recommendation,
+or implementation. The right tail must be shown alongside the left tail.
+
+### Experiment 2b — SUE / PEAD (offense)
+
+Signal `sue_latest`:
+
+\[
+\text{SUE} = \frac{\text{actual} - \text{estimate}}
+{\sigma(\text{trailing 8 surprises})}
+\]
+
+Use at least four trailing surprises; if estimate history fails the registered
+quality gate, use `(actual - estimate) / price` and name that fallback. Activate
+at announcement date +1 trading day, carry for 63 trading days, then drop the
+symbol from the cross-section.
+
+Evaluate mean weekly Spearman IC on the existing non-overlapping-window harness.
+Always report `sue_latest`, `mom_12_1`, and `mom_12_1_resid` on identical
+week-symbol-forward-return cells, plus SUE inside the top momentum quintile.
+
+### Mechanical verdict rule
+
+- **PASS** only if unconditional `sue_latest` has mean IC ≥ +0.03,
+  `reliable: true` (at least 12 windows), and positive signs in both the pre-2021
+  and post-2021 eras.
+- **FAIL** otherwise, with terminal verdict `SUE DEAD for this stack`.
+- PASS stops at `SUE PASS→PENDING_HUMAN`; integration design remains a separate
+  human decision. FAIL is terminal and no variants are proposed.
+
+---
+
+## Results
+
+### Data quality gate
+
+Approved earnings window: 2020-01-22 to 2026-07-17. Source mode: dolthub_public_bulk_clone.
+
+| check | result |
+|---|---:|
+| Prod symbols requested / tradable | 506 / 505 |
+| Manifest complete + live counts match | True |
+| Prod symbols with pre-2021 bars | 491 (97.2%) |
+| SPY benchmark depth | 2649 rows, 2016-01-04 to 2026-07-17 |
+| Snapshot depth gate | True |
+| Bulk source windows / requests logged | 1/1 / 1 |
+| Source repository / pinned commit | https://www.dolthub.com/repositories/post-no-preference/earnings @ 9n0et3hpj9j7vue8f3qsldon3qa5sdjj |
+| Source license / upstream provider documented | CC-BY-SA-4.0 / False |
+| Existing-source conflicts preserved | 940 rows / 1526 fields |
+| Symbols with >=8 announcements | 498 (98.6%) |
+| Symbols with >=8 paired announcements | 495 (98.0%) |
+| Events with estimate + actual | 12311/12414 (99.2%) |
+| Duplicate rows in keyed table | 0 |
+| Duplicate / restated payload rows fetched | 0 / 940 |
+| Mean announcements per active symbol-year | 4.08 (expected about 4) |
+| Symbols far off (<2 or >6/year, incl. zero) | 1 |
+| Recognised BMO/AMC/during | 92.8% (reliable=True) |
+| Point-in-time policy | announce_date_plus_1_trading_day_for_all_events |
+| SUE price fallback | not_used |
+
+Deduplication: UNIQUE(symbol, announce_date); normalise dot/dash symbols; retain one calendar row per key; preserve existing non-null session/EPS values from the prior FMP/Alpha Vantage partial backfill, then fill nulls and all remaining symbols from DoltHub; attach DoltHub period-end alignment
+
+Far-off announcement-rate symbols: SPCX
+
+### Experiment 2a - earnings-gap risk diagnostic
+
+Verdict: **INFORMATIONAL**. Report-only; no filter arm or implementation.
+
+Trade cohort is restricted to the approved earnings-coverage window 2020-01-22 to 2026-07-17; 0 simulated trades outside that window were excluded.
+
+| cohort | count | fraction |
+|---|---:|---:|
+| Realized net R <= -1.0 | 266 | - |
+| Losses with announcement strictly inside hold | 23 | 0.0865 |
+| All trades with announcement strictly inside hold | 115 | 0.2003 |
+
+| Entry cohort | count | mean R | median R | win rate | p05 R | p95 R |
+|---|---:|---:|---:|---:|---:|---:|
+| Within 3 sessions before earnings | 27 | 0.4837 | -1.0265 | 0.3333 | -1.1463 | 5.981 |
+| All other entries | 547 | 0.2734 | -0.8316 | 0.3565 | -1.1228 | 4.5888 |
+
+Tail deltas (pre minus other): p05=-0.0235, p95=1.3922.
+
+Registered directional tail condition is not present.
+
+| Exit cohort | count | mean R | median R | win rate | p05 R | p95 R |
+|---|---:|---:|---:|---:|---:|---:|
+| Stops within 1 session after earnings | 26 | -0.6434 | -0.9753 | 0.2308 | -2.4614 | 1.1142 |
+| All other stops | 433 | -0.5035 | -1.0278 | 0.1963 | -1.1373 | 1.3591 |
+| All other exits | 548 | 0.3273 | -0.8361 | 0.3613 | -1.0644 | 4.7224 |
+
+### Experiment 2b - SUE / post-earnings drift
+
+Mechanical verdict: **FAIL** - SUE DEAD for this stack
+
+Identical cross-sections:
+
+| signal | mean IC | t | windows | avg N | IC positive % | reliable |
+|---|---:|---:|---:|---:|---:|---|
+| sue_latest | 0.0148 | 1.27 | 56 | 450.9 | 51.8 | true |
+| mom_12_1 | 0.0195 | 0.74 | 56 | 450.9 | 58.9 | true |
+| mom_12_1_resid | 0.0262 | 1.07 | 56 | 450.9 | 55.4 | true |
+
+Unconditional SUE grade row:
+
+| signal | mean IC | t | windows | avg N | IC positive % | reliable |
+|---|---:|---:|---:|---:|---:|---|
+| sue_latest | 0.0151 | 1.29 | 56 | 451.4 | 51.8 | true |
+
+Era stability:
+
+| era | mean IC | t | windows | avg N | IC positive % | reliable |
+|---|---:|---:|---:|---:|---:|---|
+| pre-2021 | 0.0286 | 0.7 | 9 | 398.3 | 55.6 | false |
+| post-2021 | 0.0172 | 1.34 | 48 | 461.9 | 64.6 | true |
+
+Coverage: 501 symbols with live SUE; avg weekly N=453.1; scored non-overlap avg N=451.4.
+
+Cross-section is not flagged thin at the registered <100-name read.
+
+Momentum-conditional top-quintile SUE: mean IC=0.0213, t=1.3, windows=56, avg N=89.8.
+
+## Artifacts
+
+- `reports/earnings-2a-gap-20260720-dolthub-final.json` and companion Markdown
+- `reports/earnings-2b-sue-20260720-dolthub-final.json` and companion Markdown
+- `reports/earnings-backfill-status.json`
+
+Production changes: **none**. No earnings filter or SUE integration was implemented.
+
+## Final status: **Task 2 CLOSED (SUE DEAD)**

docs/research/effective-risk-floor-ab.md

@@ -0,0 +1,124 @@
+# Effective initial-risk floor A/B - frozen specification
+
+Date frozen: 2026-08-05
+Branch: research/portfolio-capacity-rebalancing
+Runner: scripts/run_portfolio_construction_matrix.py
+Study ID: risk-floor-ab
+
+## Question
+
+Does rejecting an otherwise qualified cap-10 entry when its actual initial
+stop-risk after cash and notional sizing is below 0.5% of marked equity improve
+trade selection?
+
+The completed capacity bracket cannot answer this. Its cash_unbounded arm
+removed the count cap and applied the 0.5% floor simultaneously. In the 70 paths
+where the control cap never bound, that arm still raised mean EV from 0.328 to
+0.399 R and profit factor from 1.60 to 1.75 while trades fell about 8% and
+exposure stayed nearly flat. Capacity was a no-op in those paths, so the floor
+is the plausible cause, but the prior arm remains confounded.
+
+This A/B changes only the floor. It has no formal promotion gate and does not
+automatically change production.
+
+## Frozen arms
+
+1. cap10_incumbent: current production-style cap-10 control, with no minimum
+   effective-risk floor.
+2. cap10_min_risk_005: the same cap-10 strategy, rejecting an entry only when
+   actual initial stop-risk after cash/notional sizing is below 0.5% of marked
+   equity.
+
+Both arms have max_positions=10, weekly replacement disabled, 1% target risk
+per trade, and identical admission ordering. The only differing simulator
+argument is min_initial_risk_fraction: None versus 0.005.
+
+All other settings remain the frozen daily Phase A control: current production
+construction universe, full-universe residual-momentum/low-volatility 80/20
+rank, threshold 80, normal gate-reset re-entry, close fills, 3x ATR trail,
+30-session maximum hold, 20% per-position notional ceiling, no leverage, and
+costs of 0.10% and 0.20% per fill.
+
+Every priced symbol contributes to the daily cross-sectional rank. Rank-only
+symbols cannot submit trades. Validation retains the 450-600-symbol production
+construction guardrail and the legacy-snapshot column-scoped loader.
+
+## Frozen cohorts
+
+Reuse the completed bracket's point-in-time daily candidate/rank cache and
+cohort manifest:
+
+- Empty book: first eligible session of each month in 2019-2025, with 504 prior
+  scoring sessions and 252 measurement sessions. This is the primary start-date
+  evidence.
+- Warm book: weekly seeds 63-126 sessions before each 2019-2025 annual anchor,
+  with state carried into the same 252-session measurement window. This is a
+  state-carrying replication, not independent evidence.
+
+The expected realization is 78 empty-book paths, 97 warm paths, seven annual
+clusters in each protocol, two costs, two arms, and 700 cells.
+
+Do not use warm-seed IQR as evidence. Six of seven completed-bracket anchors
+were structurally degenerate because fractional sizing is scale invariant and
+the 30-session maximum hold washed out books before anchors. The 2023 exception
+shows that state carrying itself works.
+
+## Reporting and interpretation
+
+For every protocol and cost, pair identical paths. Report:
+
+- mean, median, P25, and P75 paired net-EV changes in R;
+- positive-path and bit-identical-path fractions;
+- the median paired delta within each year and the median across seven years;
+- simple 90% cluster-bootstrap context for EV and Calmar, with no CI gate;
+- mean paired PF, Gain-to-Pain, Sortino, Calmar/MAR, CAGR, maximum drawdown,
+  total return, and Sharpe changes;
+- trades, floor rejections, holding time, cash, gross exposure, average/peak
+  positions, turnover, and costs.
+
+Means and identical-path fractions must appear beside medians so inert cohorts
+cannot turn a left- or right-skewed treatment into a misleading zero headline.
+For these 252-session windows, the implementation's full-window Calmar is CAGR
+divided by maximum drawdown, the same numeric definition commonly called MAR;
+do not present the duplicate label as a second independent metric.
+
+Today's production membership is projected backward. Use paired differences
+for the treatment conclusion; absolute profitability remains descriptive and
+survivorship-biased. Empty and warm protocols cover the same seven market years
+and must not be interpreted as independent replications.
+
+Interpretation is deliberately simple:
+
+- a positive result means the isolated floor improves the paired EV
+  distribution without an economically important loss of total-return or
+  drawdown quality;
+- a negative result closes the floor;
+- mixed EV/portfolio-quality results are reported as a trade-off, not forced
+  through a composite score.
+
+## Reproducibility and macOS execution
+
+The authoritative run refuses a dirty worktree. Its fingerprint includes the
+implementation commit, this specification hash, snapshot hash, candidate-cache
+key, construction view, cohort manifest, arm definitions, costs, and study
+version. Cells checkpoint atomically and --resume verifies the fingerprint.
+
+From the repository root on macOS:
+
+    python3 -m venv .venv
+    ./.venv/bin/python -m pip install -e '.[dev]'
+
+Preflight, reusing the completed bracket's candidate/rank cache:
+
+    ./.venv/bin/python scripts/run_portfolio_construction_matrix.py +      backtest_snapshots/research.sqlite +      --study risk-floor-ab +      --run-id prod505-effective-risk-floor-ab-daily-v1 +      --candidate-cache reports/.cache/prod505-capacity-bracket-daily-v1-candidates.pkl +      --workers 8 +      --resume +      --validate-only
+
+Authoritative run:
+
+    ./.venv/bin/python scripts/run_portfolio_construction_matrix.py +      backtest_snapshots/research.sqlite +      --study risk-floor-ab +      --run-id prod505-effective-risk-floor-ab-daily-v1 +      --candidate-cache reports/.cache/prod505-capacity-bracket-daily-v1-candidates.pkl +      --workers 8 +      --resume
+
+On an M2 Pro, eight workers is the explicit high-utilization setting. Use six
+instead on a memory-constrained machine; auto intentionally caps itself at six.
+Changing worker count does not change the fingerprint or results.
+
+Commit only the compact final JSON and Markdown reports. Candidate caches,
+checkpoints, raw curves, and trade ledgers remain ignored.

docs/research/execution-recovery.md

@@ -0,0 +1,197 @@
+# Execution recovery (2026-07-18) — when you fill is the edge you leave on the table
+
+Report: `reports/execution_recovery_matrix.json` / `.md`  
+Follows Phase A A4 ([phase-a-matrix.md](phase-a-matrix.md)).  
+Arms: `close_control` · `next_open` · `stale_close` · `next_open_gap2`.
+
+Mechanics verified before sign-off: `stale_close` re-anchors stop to
+fill − 1.5×ATR(signal day); no same-day stop after a close fill; each t−1
+candidate keeps its own gate/rank cross-section (no lookahead). Numbers trusted.
+
+---
+
+## Results
+
+| Arm | Full Sharpe | Full CAGR | Full DD | Train Sharpe | Val Sharpe |
+|---|---:|---:|---:|---:|---:|
+| **close_control** | **1.77** | **48.3%** | **21.6%** | 1.75 | 1.68 |
+| **stale_close** | **1.57** | **40.6%** | **21.8%** | 1.36 | **1.74** |
+| next_open | 1.20 | 30.0% | 28.2% | 0.94 | 1.44 |
+| next_open_gap2 | 1.06 | 25.2% | 27.8% | 0.72 | 1.41 |
+
+### Monotonicity (the strongest evidence)
+
+In every window, Sharpe recovers as the fill moves **toward** the signal:
+
+| Window | next_open → stale_close → close |
+|---|---|
+| Train | 0.94 → 1.36 → 1.75 |
+| Validation | 1.44 → 1.74 → 1.68* |
+| Full | 1.20 → 1.57 → 1.77 |
+
+\*Val close 1.68 is within SE of stale 1.74 — not a break of the story.
+
+A dead edge does **not** produce a monotone gradient in fill timing. A live edge
+that is progressively surrendered to execution delay does. Combined with full-period
+**DD recovery** (28.2% → 21.8% ≈ close’s 21.6%), this confirms the diagnosis:
+**when you fill**, not decaying alpha.
+
+### Auto-flag `recover: False` is not a null
+
+Pre-registered recovery applied the “≥ close − 0.5 SE” bar to **`stale_close`**,
+the **lower-bound** arm (one full session of lag). Flags:
+
+| Flag | Result |
+|---|---|
+| near_close_control (val) | **True** (Δ +0.06) |
+| beats_next_open | **True** (val Δ +0.30) |
+| train_ok | **False** (1.36 vs need ≥ ~1.39 — miss by ~0.03) |
+
+The floor missed “≥ close − 0.5 SE” by 0.03 while the live design is expected to
+sit **above** the floor. The flag worked correctly on the wrong object.
+
+**Log sentence:** *Partial recovery proven; full recovery needs same-day fill.*
+
+### Live outcome bracket
+
+| Bound | Arm | Meaning |
+|---|---|---|
+| Floor | stale_close ~1.57 full | One-session-stale close fill (conservative) |
+| Ceiling | close_control ~1.77 full | Same-day close; optimistic only by final ~15 min of signal info |
+
+Real near-close execution (scan ~15:30–15:40 ET on a ~99% complete bar, MOC by
+15:50/15:55) is signal-at-partial-bar filled at the same close the control uses.
+**Live truth is bracketed [1.57, 1.77]** with residual uncertainty of ~15 minutes
+of staleness, not 24 hours — expect near the **top** of the bracket.
+
+`stale_close` alone already justifies the schedule change. **No further fill-timing
+simulation on this snapshot** — the bracket is the result. You cannot simulate
+15:45 partial bars from daily data, and you do not need to.
+
+### Gap-cap — dead (third tail-trim instance)
+
+`next_open_gap2` worse than plain next_open on every window (full Sharpe 1.06 vs
+1.20). **262** full-period gap-ups skipped — they were continuations.
+
+This is the **third independent instance** of the same lesson:
+
+1. Take-profit exits (gate target as TP)  
+2. FIP as an in-book filter  
+3. **Gap-up entry caps**
+
+Any rule that trims the right tail improves cosmetic quality metrics and destroys
+P&L. **Standing method rule:** momentum filters must be presumed guilty of
+tail-trimming until shown otherwise.
+
+---
+
+## Decisions locked
+
+1. **Biggest lever is execution scheduling**, not a strategy rewrite.  
+2. ~~**Until near-close execution is live:** grade strategy promotion under
+   `fill_mode=next_open`; keep close-fill as historical control.~~
+   **Superseded 2026-07-18** — near-close shipped to prod (commits 736451e,
+   a71dd4a; migration 023). Decision baseline is now **`fill_mode=close`**
+   (≈ the live near-close path). Live fill quality is monitored via
+   `fill_mode=near_close` paper trades: entry price vs the day's final close
+   (available after the after-close fetch overwrites the partial bar).  
+3. **After near-close ships:** grade under a close-like / near-close fill mode
+   (actual live path). **← Active.**  
+4. **No more sim arms** on fill timing for this snapshot.  
+5. **Gap-cap:** do not ship.  
+6. **Strategy work** (nasdaq_all, fip_id, sector) waits until the execution path
+   is decided — those experiments must be graded under the fill mode you will trade.
+
+---
+
+## Ops design — implementation plan (code-checked)
+
+Assumptions verified against current code before ship:
+
+- Intraday pipeline already fetches/upserts the **in-progress day-t bar** all
+  session (`fetch_ohlcv` end_date defaults to today). Near-close job =
+  **OHLCV fetch → R:R scan** (no new snapshot synthesizer).
+- One global `schedule_timezone` (default `Europe/Berlin`); stored
+  `SystemSetting` values shadow code defaults — **defaults alone do not
+  migrate prod**.
+- `observe_reentry_gate_transitions` stamps timestamps with **no same-day
+  guard** today — dual scans would accelerate fail→requalify unless fixed in
+  `trade_policy`.
+
+### Semantic guard (ship step 1 — precondition)
+
+In `trade_policy` (not the scheduler):
+
+> `reentry_gate_requalified_at` may only be set when `reentry_gate_failed_at`
+> falls on an **earlier America/New_York trading date** than the current
+> observation.
+
+Manual mid-day scans stay allowed; same-day fail+qualify cannot unlock.
+Unit test: fail 10:00 / qualify 15:35 same day → still locked; qualify next day → unlocked.
+
+### Schedule split
+
+| Slot (America/New_York) | Jobs |
+|---|---|
+| Morning (~02:00) | OHLCV backfill, benchmark, sentiment, fundamentals — **no** qualifying R:R scan |
+| Near-close (~15:30 Mon–Fri) | OHLCV fetch (refresh day-t bar) → **R:R scan** (only daily qualifying observation) |
+| After close (~16:30–17:00) | **Outcome eval** on its own slot (not chained to the partial-bar scan) |
+| Intraday hourly | Unchanged in NY terms; last ~16:00 still mid-session under 15m feed |
+
+- Near-close scan **1–5 only**; US-holiday no-ops are fine (stale identical data
+  can’t flip gates) — comment only, no exchange calendar.
+- **Do not** run morning + near-close qualifying scans; move the scan, don’t add a second.
+
+### Behavior change to document (not an accident)
+
+With scan at ~15:35 ET, stops closed by **earlier same-day** intraday outcome
+evals can get a **same-day fail observation** — closer to the **promoted**
+`gate_reset` arm (stop-day close may establish failure) than today’s
+`strict_gate_reset` analogue (scan always before any eval). Stops after the
+bell still wait a day. Rewrite README “Live timing matters” / post-stop sections
+and a line here when shipping.
+
+### Feed / paper honesty
+
+- Document 15-minute delayed SIP: 15:35 scan may see ~15:20 prices; OK for 12-1.
+- Paper entry price ≈ scan entry (near close) is nearly automatic; add
+  **`fill_mode=near_close` era tag** so Track Record can separate morning-scan /
+  near-close / future broker-routed eras.
+- Morning sentiment staleness is display-only; gate is price-only (GTL parity /
+  neutral-sentiment backtest). One doc line closes that.
+
+### Stored settings migration (ship step 4)
+
+Flip global default TZ to `America/New_York` and re-express crons in NY time.
+**Also** migration (or documented Admin rewrite) of stored `schedule_*` keys so
+prod does not keep 07:00 Berlin silently.
+
+### Ship order
+
+1. `trade_policy` distinct-day requalify guard + unit test  
+2. Near-close job = existing fetch → scan; outcome eval own after-close slot  
+3. Paper `fill_mode=near_close` era tag; verify entry marking  
+4. Defaults + **stored settings migration** + README/research timing rewrite  
+
+### Out of scope
+
+Broker MOC routing, more fill-timing sim, nasdaq_all / fip / sector (grade later
+under the fill mode you trade).
+
+---
+
+## Implementation status
+
+| Item | Status |
+|---|---|
+| Research evidence | **Closed** — this doc + matrix report |
+| Distinct-day gate-reset guard | **Shipped** — `trade_policy` + unit test |
+| Schedule split + near-close scan→alert | **Shipped** — morning / near-close / after-close (fetch→outcome) |
+| Paper era tag | **Shipped** — `fill_mode=near_close` on new paper trades |
+| Settings migration + docs | **Shipped** — alembic 023 rewrites schedule_*; README updated |
+
+### Shipped behavior change (not accidental)
+
+Near-close scan at ~15:30 ET lets same-day fail observations after earlier
+intraday stop closes — closer to promoted `gate_reset` than the old
+morning-scan `strict_gate_reset` analogue. Documented in README.

docs/research/fip-breadth-ic.md

@@ -0,0 +1,216 @@
+# Broad-universe fip_id IC research (Phase B)
+
+**Status:** **Parked / closed for now.** Unconditional fip not green; mom-conditional lead logged; breadth-momentum thesis challenged. No book sim until reopen.  
+**Production impact:** none. Display card remains context-only. No deploy from this work.  
+**Artifacts:** research log + compact reports + env-gated harness hooks; tooling stays for a future reopen.
+
+## Scope
+
+- Research only — production universe, gate, scanner, schedule unchanged.
+- Snapshot: `research.sqlite` (~4,650 tickers = prod + nasdaq_all extend).
+- Liquid mask: top **1,500** by point-in-time 63d median $vol, price ≥ **$5**/week.
+- **Completion manifest required:** extender writes `<snapshot>.manifest.json`; breadth runners refuse without a matching complete manifest (see §Race guard).
+
+## Caveats
+
+- Survivorship bias (today’s constituents, history backfilled).
+- IEX volume undercount → relative $vol rank only.
+- Pool skew: Nasdaq-heavy; missing pure NYSE mid-caps.
+- Do not mix multi-signal tables across universe baselines.
+- **Do not cite orphaned 21:14 numbers** (see below).
+
+---
+
+## Fingerprint (505-name prod)
+
+| | Expected | Observed |
+|---|---:|---:|
+| mean IC | −0.045 | **−0.045** |
+| t-stat | −2.9 | **−2.91** |
+| weeks / N / reliable | ≥12 / ~500 / true | 35 / 497.7 / true |
+
+**Pass.** Formula + pipeline trustworthy.
+
+Residual momentum on the same fingerprint (what the production book ranks on): **IC +0.055 / t +1.98**.
+
+---
+
+## The orphan (21:14) — root cause
+
+| Source | fip IC (liquid ~1500) | t |
+|---|---:|---:|
+| Orphan run 21:14 (removed from tree; was `fip-breadth-20260718-211440-breadth.json`) | **+0.0575** | **+5.12** |
+| Single-sourced recompute on complete snapshot (2026-07-19) | **−0.0168** | **−1.85** |
+
+That is a **sign disagreement** on the same intended quantity. Method rule: the number you cannot reconcile is the number you cannot use.
+
+### Verdict: orphaned — raced the snapshot build
+
+**Not** “orphaned, unexplained.” The mechanism is derivable from the table itself:
+
+1. **Code was not the difference.** Reconcile shows the old harness path and the new shared filter produce **identical** results on current data (−0.0168 / −1.85). The implementation fork is closed.
+2. **Data was the difference.** On today’s complete snapshot the liquid mask **binds in 97.1% of weeks** at top-N = 1,500. Dense signals (e.g. `vol_6m`) post-mask at **exactly 1,500**. The orphaned report’s `vol_6m` averaged **~1,475** cross-section — a masked run on complete data cannot do that. At 21:14 the eligible pool was smaller than 1,500 and the mask never bound.
+3. **Timeline fits.** Extender fixes landed ~20:32 / 20:34; full fetch takes ~30 minutes; breadth run fired **21:14** against a partially built `research.sqlite`. Every number in that report was computed on an incomplete universe.
+
+**Do not cite +0.0575 / t +5.12.** It survived less than six hours of contact with project discipline — that is the system working, not time wasted. The orphan JSON was **deleted from the tree** (still in Git history) so it cannot be re-imported as evidence.
+
+**Kept artifacts**
+
+| File | Role |
+|---|---|
+| `reports/fip-reconcile-20260719-000520.json` | Authoritative single-sourced ICs (compact; membership dumps stripped) |
+| `reports/fip-breadth-20260718-211440-fingerprint.json` | Prod fingerprint pass |
+
+### Race guard (same class as calendar truncation)
+
+| Piece | Behavior |
+|---|---|
+| `extend_snapshot_universe.py` | Clears any prior manifest on start; on full completion writes `<output>.manifest.json` with `complete=true`, ticker / OHLCV / rank_only counts, `finished_at`. `--limit` smoke runs write `complete=false`. |
+| `run_fip_breadth_research.py` / `run_fip_breadth_diagnostics.py` | **Refuse** breadth mode unless a matching complete manifest exists and live counts equal the recorded totals. |
+
+Helper: `scripts/research_snapshot_manifest.py`.
+
+---
+
+## Authoritative unconditional liquid fip (post-reconciliation)
+
+| metric | value |
+|---|---:|
+| mean_ic | **−0.0168** |
+| ic_t_stat | **−1.85** |
+| weeks | 35 |
+| avg_cross_section (**post-mask IC sample**) | 1471.2 |
+| avg_raw_pool | 3214.4 |
+| avg_eligible_pre_mask | **2338.4** |
+| mask_binds_pct | **97.1%** |
+| reliable | true |
+
+**Mask binds hard** on complete data (eligible ≫ 1500). Post-mask IC N for fip is ~1471 because not every liquid name has a valid 12-1 fip path — that is signal availability, not a non-binding mask. Contrast orphan `vol_6m` avg N ~1475 vs complete-data `vol_6m` avg N **1500**.
+
+Harness `_signal_evaluation` vs manual IC through the same filter: **exact match** (−0.0168 / −1.85).
+
+**Iron rule unconditional:** **not green** (|IC| 0.017 < 0.03), correct mild-negative sign.
+
+---
+
+## Context table (orphaned 21:14 vs authoritative) — kill the myth numbers
+
+The context table died with the orphan. **−0.16 must not survive in the log.**
+
+| signal (liquid ~1500) | orphaned (21:14) | authoritative (shared filter) | consequence |
+|---|---:|---:|---|
+| **vol_6m** | −0.16 / t **−6.1** | **−0.048 / t −1.36** | “High-vol tilt harmful on breadth” **downgrades from finding to directional hypothesis** — not significant |
+| **raw mom** (`mom_12_1`) | +0.10 / t +4.6 | **+0.046 / t +1.91** | Below iron-rule bar on this pool |
+| **resid mom** (`mom_12_1_resid`) | +0.04 / t +2.3 | **+0.029 / t +1.33** | Ditto, and weaker than raw |
+
+### Breadth-momentum thesis — challenged
+
+That last pair is the sobering one. Momentum on liquid breadth is **marginal**. The “more breadth strengthens the momentum t-stat” thesis that motivated Phase B is **empirically wrong on this pool**: same 35 weeks, triple the names, residual-mom t-stat **fell** versus the 505-name fingerprint (**0.055 / 1.98** → **0.029 / 1.33**). The clean momentum edge lives in the large-cap universe already traded.
+
+Meanwhile the strongest reliable signal on liquid breadth is now **mom-conditional fip** (−0.088 / −4.58) — but a fip tilt presupposes a breadth momentum book worth tilting, and that is no longer free.
+
+---
+
+## Compositional story (supported)
+
+`fip_id = sign(PRET)×(%neg−%pos)` pools:
+
+- **Continuous winners** → want **negative** IC  
+- **Continuous bleeders** → want **positive** IC  
+
+| check | IC | t | read |
+|---|---:|---:|---|
+| Prod-universe subset inside liquid | **−0.044** | **−2.88** | Matches fingerprint → compositional, not regime change |
+| Tier 1–800 (senior) | **−0.035** | **−2.99** | Winner leg |
+| Tier 801–1500 (junior) | **+0.014** | +1.25 | More bleeder / junk weight |
+| Lagged membership (prior-week $vol) | −0.010 | −0.93 | Same sign as same-week; not a +5σ leak artifact |
+
+**Do not log “on Nasdaq, jumpy paths outperform.”** That would mythologize an orphaned +0.06.
+
+---
+
+## Platform-relevant test: momentum-conditional fip
+
+Among liquid top-1500, keep **mom_12_1 ≥ P80** (~294 names/week):
+
+| metric | value |
+|---|---:|
+| mean_ic | **−0.0879** |
+| ic_t_stat | **−4.58** |
+| ic_positive_pct | 22.9% |
+| weeks | 35 |
+| reliable | **true** |
+
+Computed on the **same single-sourced path** as the authoritative −0.017. This is the paper’s claim and the only version a gate could consume.
+
+| Decision | |
+|---|---|
+| Unconditional fip | **Closed** for production |
+| Mom-conditional fip | **Alive as book-tilt candidate only** — and only after a baseline breadth book proves itself |
+| Display card | Stays |
+| Production change | **None** |
+
+---
+
+## Vol-tilt warning (softened)
+
+| signal (liquid, single-sourced) | IC | t |
+|---|---:|---:|
+| vol_6m | −0.048 | **−1.36** |
+| mom_12_1 | +0.046 | +1.91 |
+| mom_12_1_resid | +0.029 | +1.33 |
+
+High-vol names **tend** to underperform on this pool relative to a clean S&P-like book — that is a **directional hypothesis**, not a finding. Production **80/20 high-vol tilt** was validated on S&P-like names. If the universe ever broadens in production, re-validate that tilt; do not treat the orphaned −0.16 / t −6.1 as evidence.
+
+---
+
+## What this means for the book experiment
+
+A fip tilt presupposes a breadth momentum book worth tilting — **that is no longer free.**
+
+**Caution against over-reacting the other way:** modest cross-sectional IC does not preclude a good book. The 505-name book turns resid-mom IC ~0.055 into Sharpe ~2 because the gate trades the **extreme tail**, not the linear sort. The breadth book might still work; it just has to **prove it** before the fip arm means anything. If the baseline cannot clearly beat the existing production book’s territory, fip’s future is a footnote regardless of −4.58.
+
+### Parked next step (if reopened): pre-registered two-arm design
+
+Not started — **design only**, pre-register before any sim:
+
+| Arm | Definition |
+|---|---|
+| **A — baseline** | Top-quintile residual (or raw — pick one and lock) momentum book on liquid-1500; **no fip**; honest costs; next-open or near-close fills; production-like capacity / risk / stops |
+| **B — +fip tilt** | Same book + mom-conditional fip tilt (among mom winners, prefer smoother paths / negative fip_id) |
+
+| Grade on | Spec |
+|---|---|
+| Split | Entry-date train / validation (`BACKTEST_HOLDOUT_SPLIT` naming — not pristine holdout) |
+| Metrics | Sharpe + Mertens/Lo SE, PSR, **DSR**; max DD; turnover; cost drag |
+| Promote bar | Arm A must be in production-book territory first; Arm B must beat A on validation with DSR-aware multiple-testing honesty |
+| Fail-closed | If A fails, fip is a footnote; do not shop tilts on a dead baseline |
+
+---
+
+## How to re-run (research branch only)
+
+```powershell
+# 1) Full extend writes completion manifest (required)
+.\.venv\Scripts\python.exe scripts\extend_snapshot_universe.py `
+  --source backtest_snapshots\prod.sqlite `
+  --output backtest_snapshots\research.sqlite
+
+# 2) Breadth / diagnostics refuse without matching manifest
+.\.venv\Scripts\python.exe scripts\run_fip_breadth_diagnostics.py `
+  --research-snapshot backtest_snapshots\research.sqlite `
+  --prod-snapshot backtest_snapshots\prod.sqlite `
+  --workers 6 --allow-spawn
+```
+
+---
+
+## Bottom line
+
+1. Formal iron-rule screen: **not green** either before or after reconciliation.  
+2. **+0.0575 / +5.12 is orphaned: raced the snapshot build** — authoritative unconditional liquid fip is **−0.017 / −1.9**; mask binds (~97%) on complete data.  
+3. Context-table myths die with the orphan: **vol −0.16 is not real**; authoritative vol is **−0.048 / t −1.36** (directional only).  
+4. Compositional tug-of-war is the right story; jumpiness premium is not.  
+5. **Breadth does not strengthen residual-mom t-stat** on this pool (0.055/1.98 → 0.029/1.33).  
+6. **Mom-conditional −0.088 / −4.6 stands** on the single-sourced path → optional next step is a **pre-registered two-arm breadth book** (baseline first), not a gate wire-in.  
+7. Manifest guard is in place so the race cannot recur silently.

docs/research/fundamentals-weight-backtest.md

@@ -0,0 +1,74 @@
+# Fundamentals ranking-overlay research
+
+Status: completed 2026-07-23. Decision: keep production scoring and qualification unchanged.
+
+## Question
+
+Does using point-in-time SEC fundamentals to reorder already-qualified long setups improve the production portfolio's risk-adjusted return? The experiments changed ranking only; qualification, execution, sizing, capacity, costs, ATR exits, and post-stop re-entry remained unchanged.
+
+## Method
+
+- The control was the production 80/20 residual-momentum / volatility rank.
+- SEC facts became visible only after `accepted_at`, using the newest visible accession per fiscal period.
+- Portfolio simulations used daily entry opportunities, close fills, the production gate-reset re-entry policy, and a 30-session horizon.
+- Train contained entries before 2024-01-01, validation covered 2024, and test began 2025-01-01.
+- Missing composite scores were neutral at 50.
+- Deflated Sharpe used the complete registered arm count for each experiment.
+
+The snapshot contained 511 tracked tickers, 507 unique CIKs, 30,494 SEC snapshot rows, and prices from 2021-06-24 through 2026-07-22.
+
+## Initial experiment
+
+The first registered matrix tested quality, growth, and balanced composites at 10%, 20%, 30%, and 40% weights: 13 trials including control.
+
+No overlay passed the train and validation requirements. The most attractive full-period result, balanced at 10%, failed validation and improved test Sharpe by only 0.06.
+
+Review also found that filing-time diluted EPS and shares are not reliably comparable across stock splits. A snapshot audit found share-count changes above 25% for 90 of 461 issuers with comparable 2021+ periods, including recognizable split ratios for AMZN, GOOG, NVDA, CMG, and GE plus some obvious unit anomalies. Consequently, EPS growth and share-count change cannot be trusted for historical ranking without point-in-time split factors.
+
+The complete initial result is recoverable from Git commit `7f944d7`.
+
+## Split-safe follow-up
+
+The follow-up excluded diluted-EPS growth and share-count change completely. It tested:
+
+- Quality: operating margin, FCF margin, and low net-debt/EBITDA, requiring at least two inputs.
+- Growth: revenue growth only.
+- Balanced: equal quality and growth weights.
+- Overlay weights: 5%, 10%, and 15%.
+
+This produced 10 registered trials including control. Growth coverage among qualified candidates was 88.96%; lack of data was not the limiting factor.
+
+| Window | Control Sharpe | Revenue-growth 5% | Delta |
+|---|---:|---:|---:|
+| Train | 1.26 | 1.31 | +0.05 |
+| Validation | 2.52 | 2.64 | +0.12 |
+| Test | 1.99 | 1.99 | 0.00 |
+| Full | 1.86 | 1.87 | +0.01 |
+
+Revenue growth at 5% mechanically passed the deliberately permissive "not worse" gate, but did not demonstrate an economically meaningful edge:
+
+- Test CAGR rose from 54.4% to 56.1%, while full-period CAGR fell from 52.1% to 51.5%.
+- Full-period trade overlap was 68.53%, so roughly one-third of selections changed for essentially unchanged Sharpe.
+- Revenue-growth IC was 0.0006 in train, 0.0053 in test, and 0.0116 full-period with a full-period t-stat of 0.55.
+- Growth weights of 10% and 15% deteriorated; quality and balanced composites failed.
+- The test window had already been inspected, so this follow-up was sensitivity evidence rather than a fresh out-of-sample result.
+
+The complete split-safe result is recoverable from Git commit `dba7ea7`.
+
+## Decision
+
+- Do not add fundamental weight to production ranking or the automated qualification gate.
+- Do not run another historical weight sweep on the same sample; it would add data-mining rather than new evidence.
+- Keep fundamentals informational and user-facing in the UI.
+- A5 source-parity and cutover work can proceed independently without changing scoring behavior.
+- Treat historical EPS growth and share-count change as non-comparable across corporate actions until a split-aware solution or a conservative UI guard exists.
+
+Revisit automated weighting only with materially better data, such as point-in-time split factors and historical constituent/delisting coverage, followed by genuinely new forward paper evidence.
+
+## Limitations
+
+The snapshot uses today's tracked universe rather than historical membership and delisted securities, creating survivorship bias. Absolute CAGR and Sharpe must not be interpreted as unbiased live expectations. The relative comparison is useful, but the observed test window and short number of independent factor windows limit statistical power.
+
+## Repository cleanup
+
+The experiment-only scorer, runner, Mac launcher, caches, tests, and expanded report bundles were removed after this decision. They remain recoverable from commits `eae4d34`, `34d6dda`, `7f944d7`, and `dba7ea7`. Production fundamentals derivation and ingestion remain unchanged.

docs/research/history-depth-extension.md

@@ -0,0 +1,206 @@
+# History-depth extension (Tier-1 alpha research)
+
+**Status:** **CLOSED.** Sector-residual deep test **FAIL** — Task 1 archived as rejected (see supersession).  
+**Branch:** `research/earnings-gap-and-sue`  
+**Superseded artifact (do not cite):** `reports/history-depth-20260719-103315.json` — **UNMASKED, TWO-TIER SNAPSHOT**  
+**Authoritative sector grade:** `reports/sector-resid-deep-20260719-113319.json`  
+**Production impact:** none. **Do not retune any production knob on deep history.**
+
+---
+
+## Pre-registration (locked before rebuild)
+
+### Motivation
+
+All current conclusions rest on ~35 non-overlapping weekly windows in essentially
+one post-2021 regime. Extending history toward max Alpaca daily-bar depth adds
+the 2018 vol shock and full 2020 crash (where the feed allows).
+
+### Protocol
+
+1. **Empirical coverage first** — bars per calendar year per symbol; document
+   where the feed thins out. Do **not** assume a uniform start date.
+2. **Rebuild the research snapshot completely** from prod source + max history
+   per symbol (`Adjustment.SPLIT`, ~200 req/min pacing via existing extender).
+3. **Race guard (rule 6)** — refuse analysis until completion manifest is
+   `complete=true` and live counts match.
+4. **Re-run full signal harness** (all existing signals incl. sector residual /
+   SUE if present) on the extended window.
+5. **Report per signal:** mean IC, t, window count, and **era split**
+   (pre-/post-2021) — diagnostic only, **not a tuning input**.
+6. **Log prominently:** survivorship bias grows with depth (today’s constituents
+   backfilled). Absolute Sharpe/CAGR on deep history is optimistic; payload is
+   **relative** signal comparisons and IC stability, not levels.
+7. **Do not retune** production knobs. If a knob’s confirmation looks
+   overturned on deep history → report only; human decides.
+
+### Success / interpretation (not promotion of a new signal)
+
+| outcome | meaning |
+|---|---|
+| Sector residual still ≥ market residual on deep IC + stable sign | strengthens Task 1 PROMOTE case |
+| Sector residual collapses pre-2021 | **PARK** Task 1 wire-in |
+| SUE remains weak after full earnings + depth | **DEAD** SUE for this stack |
+| Any production knob looks worse deep | report; no auto-retune |
+
+---
+
+## Data provenance
+
+| check | result |
+|---|---|
+| Snapshot | MacBook `research.sqlite` |
+| Manifest `complete` | **true** (finished 2026-07-19T08:19Z) |
+| Live counts match | yes — 4655 tickers / 6,609,926 OHLCV / 4149 rank_only |
+| `history_days` | 5000 |
+| fetch_ok / fail | 4152 / 0 |
+| Race guard | **pass** |
+
+> **SURVIVORSHIP BIAS:** today’s constituents backfilled historically. Absolute
+> Sharpe/CAGR levels on deep history are optimistic. Use **relative** signal IC
+> comparisons and era stability only — not levels.
+
+**Coverage JSON was empty in the auto-written doc** (harness-only phase after
+rebuild). Manifest is the race-guard source of truth for this run.
+
+Earlier MacBook files `history-depth-20260719-093853` … `095156` are intermediate
+/ incomplete passes — **do not cite**. Only **103315** is authoritative.
+
+---
+
+## Results (authoritative: 103315)
+
+### Full-window signal IC (broad research universe, deep bars)
+
+| signal | mean_ic | t | weeks | avg_N | notes |
+|---|---:|---:|---:|---:|---|
+| high_52w | **0.111** | **6.41** | 84 | 2280 | strong on deep breadth |
+| mom_12_1 | **0.066** | **5.11** | 83 | 2278 | raw momentum strong |
+| trend_200 | 0.055 | 4.30 | 85 | 2308 | |
+| mom_6_1 | 0.049 | 4.91 | 88 | 2376 | |
+| mom_3_1 | 0.036 | 3.58 | 90 | 2426 | |
+| fip_id | **+0.027** | **3.25** | 83 | 2278 | **sign flip vs prod fingerprint** |
+| mom_12_1_resid | 0.026 | 2.21 | 83 | 2278 | market residual still + but weaker than raw |
+| reversal_1m | ~0 | 0.31 | 91 | 2443 | dead |
+| vol_6m | **−0.123** | **−6.34** | 88 | 2376 | low-vol anomaly strong |
+| mom_12_1_sector_resid | 0.058 | 2.34 | **35** | **498** | **not deep-sample — see caveats** |
+| mom_12_1_sector_demeaned | 0.034 | 1.32 | **35** | **497** | same short fingerprint |
+
+### Era split (diagnostic only — not a tuning input)
+
+| signal | pre-2021 IC / t / w / N | post-2021 IC / t / w / N |
+|---|---|---|
+| mom_12_1 | +0.041 / 2.94 / 36 / 1425 | +0.079 / 3.64 / 48 / 2913 |
+| mom_12_1_resid | +0.023 / 1.68 / 36 / 1425 | +0.027 / 1.37 / 48 / 2913 |
+| fip_id | +0.012 / 1.35 / 36 / 1425 | +0.037 / 2.91 / 48 / 2913 |
+| vol_6m | −0.056 / −2.16 / 40 / 1461 | −0.162 / −4.94 / 48 / 3123 |
+| high_52w | +0.049 / 2.0 / 36 / 1422 | +0.138 / 4.34 / 48 / 2915 |
+| **sector_resid** | **absent** | 0.058 / 2.34 / 35 / 498 (short only) |
+| **sector_demeaned** | **absent** | 0.034 / 1.32 / 35 / 497 (short only) |
+
+---
+
+## Critical caveats (must read)
+
+### 1. Sector residual did **not** get a deep-history stress test
+
+`mom_12_1_sector_resid` / `_demeaned` still show **exactly** the Task‑1 short-window
+fingerprint: **35 weeks, N≈498, IC 0.0578, t 2.34**.
+
+On the same run, raw `mom_12_1` has **83 weeks, N≈2278**. So depth worked for
+price-only signals, but sector residual is still limited to the **~505 labeled
+prod names × short factor calendar** (sector map only covers prod; and/or sector
+ETF / two-factor path did not extend usable residual weeks).
+
+**Pre-registered rule:** “Sector residual collapses pre-2021 → PARK Task 1
+wire-in.” Pre-2021 sector residual is **absent** from the era table. That is a
+**PARK**, not a confirmation of the short-window PROMOTE.
+
+Do **not** claim “sector residual beats market residual on deep history” from
+this table — the two rows are **not the same cross-section or window count**.
+
+### 2. `fip_id` sign flips vs production fingerprint
+
+| sample | fip mean IC | t |
+|---|---:|---:|
+| Prod 505, ~5y (fingerprint) | **−0.045** | −2.91 |
+| Research breadth, deep (this run) | **+0.027** | +3.25 |
+
+This does **not** authorize resurrecting unconditional FIP as a book filter. It
+confirms earlier Phase‑B caution: FIP edge is **universe- and sample-dependent**.
+Production display card can stay context-only. Nested lookbacks still not OOS.
+
+### 3. Market residual vs raw momentum on deep breadth
+
+On deep broad IC, **raw 12‑1 (0.066 / t 5.1) ≫ market residual (0.026 / t 2.2)**.
+That does **not** by itself overturn production residual ranking (book A/B was
+on 505 + GTL gate, not pure factor IC), but it is a yellow flag for “residual is
+always the better rank key” stories on broad history. **No auto-retune.**
+
+### 4. Low-vol anomaly is the cleanest deep-history result
+
+`vol_6m` IC −0.12 / t −6.3 full; stronger post-2021. Consistent sign across eras.
+Production already blends **high**-vol (not low-vol) into the 80/20 rank — this
+report does not change that without a separate A/B. Flag for human awareness only.
+
+---
+
+## Verdicts (vs pre-registration)
+
+| question | verdict |
+|---|---|
+| Task 1 sector residual wire-in | **PARK** — no pre-2021 sector residual; deep-sample IC not established; short-window PROMOTE stays “human design only,” **not strengthened** by this run |
+| Sector demean | still **DEAD** for promotion (t 1.32, short only) |
+| SUE | **not re-scored here** (no `sue_latest` in harness table) — leave Task 2 **PARK** until full earnings backfill |
+| fip unconditional book filter | remains **rejected / parked** despite sign flip on broad deep sample |
+| Production residual / 80/20 / trail knobs | **no retune** from this report |
+| Overall Task 3 | **COMPLETE as diagnostic** — payload is relative IC + caveats above |
+
+---
+
+## What a human must decide next
+
+1. **Sector residual — decided:** CLOSED / REJECTED (archive complete). No wire-in.
+2. **Do not** retune residual vs raw, FIP, or vol blend from deep IC tables
+   without a pre-registered book A/B on the intended universe.
+3. Optional (separate threads only): finish earnings backfill and re-run SUE;
+   snapshot per-symbol depth guard as tooling.
+
+---
+
+## Artifacts
+
+| file | role |
+|---|---|
+| `reports/history-depth-20260719-103315.json` | Superseded unmasked/two-tier IC dump (do not cite for sector residual) |
+| `reports/sector-resid-deep-20260719-113319.json` | Authoritative sector-resid deep grade |
+| `reports/prod-book-universe-horizon-20260719-140737.json` | 505 vs liquid × horizon book matrix |
+
+Intermediate history-depth partials (093853–095156) and SANITY-FAIL noise were
+removed in branch cleanup.
+
+---
+
+## Supersession notice (2026-07-19 sector-resid deep test)
+
+The table and interpretation from **`history-depth-20260719-103315`** are **UNMASKED, TWO-TIER SNAPSHOT — superseded, directional only, do not cite**. Prod-universe names (and sector residual coverage) were left shallow while breadth names were deepened; sector residual weeks=35 was a data gap.
+
+### Sector-residual deep test outcome: **FAIL** (archived)
+
+**Task 1 CLOSED / REJECTED** — sector residual dead on deep evidence. Archived in
+the research log rejected table (#13). Do not resurrect without a new
+pre-registered protocol.
+
+| check | result |
+|---|---|
+| weeks | **83** (data fix worked) |
+| mean IC | **0.0268** (below 0.03 bar) → FAIL |
+| t vs resid same CS | 1.69 ≥ 1.30 pass |
+| era signs | both + pass |
+
+- Artifact: `reports/sector-resid-deep-20260719-113319.json`
+- Summary write-up: [sector-residual-momentum.md](sector-residual-momentum.md)
+
+**Future snapshot rebuilds must verify per-symbol depth** (earliest-bar
+uniformity across the intended universe) — guard is a to-do, not part of this
+order.

docs/research/phase-a-matrix.md

@@ -0,0 +1,127 @@
+# Phase A research matrix (2026-07-18) — results and decisions
+
+Report: `reports/research-matrix-phase-a.json` / `.md`  
+Branch: `research/portfolio-vol-and-followups`  
+Validation split: entries ≥ **2024-07-01** (called *validation*, not holdout — this window has been opened before).  
+Cadence: daily, production gate/rank/trail + gate-reset re-entry.  
+Pre-registered N for DSR: **20**.
+
+## Pre-registered promotion rule (unchanged after run start)
+
+Promote only if **all** of:
+
+1. Validation Sharpe ≥ control  
+2. Validation max DD not worse by more than **2pp**  
+3. Train Sharpe not worse (both-windows consistency)  
+
+Always report whether validation ΔSharpe exceeds **1 × SE** (expect most will not).
+
+Mechanics guards confirmed before reading results: calendar truncation asserted on every arm; next-open re-anchors stop to fill − 1.5×ATR(signal); vol scalars apply at entry only.
+
+---
+
+## Control baseline
+
+| Window | Sharpe | SE | CAGR | MaxDD | Calmar | Trades |
+|---|---:|---:|---:|---:|---:|---:|
+| Train | 1.75 | 0.68 | 49.8% | 17.9% | 2.78 | 240 |
+| **Validation** | **1.68** | **0.72** | **41.6%** | **20.9%** | **1.99** | **239** |
+| Full (close-fill) | 1.77 | 0.50 | 48.3% | 21.6% | 2.23 | 472 |
+
+**Capacity correction (2026-08-05):** the full close-fill control also records
+skipped_book_full = 519 versus 472 admitted trades, so the ten-slot book
+refuses 52.4% of admitted+blocked qualified opportunities. The older weekly
+claim that the cap never bound is stale and does not apply to this daily
+gate-reset configuration. Capacity is now isolated in the
+[focused bracket study](portfolio-capacity-bracket.md).
+
+Validation SE ≈ 0.72 — almost no arm clears a 1-SE delta.
+
+---
+
+## Per-arm decisions
+
+### A2 — Max hold {30, 45, 60, 90} — **note and move on**
+
+| Hold | Val Sharpe | Val DD | Train Sharpe | Trades train |
+|---:|---:|---:|---:|---:|
+| 30 | 1.68 | 20.9 | 1.75 | 240 |
+| 45 | **2.07** | 19.3 | **1.43** | 218 |
+| 60 | **2.12** | 19.3 | **1.11** | 186 |
+| 90 | 2.04 | 23.1 | 1.30 | 179 |
+
+Validation-only would have “found” +0.4 Sharpe. Train collapses: longer holds leave stale names blocking slots (240 → 186 trades at hold-60). This is a **regime interaction** (trend validation vs chop train), not a free knob. A regime-conditional hold is a large research program; prior regime-overlay work already argues against that path.
+
+**Decision: keep max hold 30. Do not ship longer static holds.**
+
+### A3 — Equity-curve vol targeting — **reject as edge; park as optional insurance**
+
+Scalars averaged 0.77–1.07 as designed (grid straddled historical book vol ~22–25%). Lower targets de-levered; **vt25** was nearly neutral (val Sharpe 1.63 vs 1.68). Wide clamp ≈ headline clamp. Lookback sensitivity did not unlock a win.
+
+This sample has **no major vol-regime shift**, so the run **rejects vol targeting as an edge on this data** — it does **not** reject crash-insurance value in a future high-vol regime. The ~0.02 Sharpe cost at vt25 is a nearly free insurance policy if drawdown tolerance ever tightens.
+
+**Decision: do not ship. Settles “Phase 2 = vol-scaled momentum” as an edge plan on this snapshot. Park vt25 as optional risk preference only.**
+
+### A5 — Correlation caps — **reject; sector caps stay Phase B with reduced expectations**
+
+Best near-miss: **0.6 skip** — val Sharpe 1.70, val DD **17.0%** (tempting), but train Sharpe 1.61 < 1.75 and full-period Sharpe **1.59 vs 1.77** (the cap deletes real momentum concentration profit). Half-size variants were worse.
+
+**Decision: no pure corr cap. Sector caps remain Phase B with reduced expectations.**
+
+### A4 — Next-open fill — **not a reject; the discovery**
+
+| | Close control | Next-open |
+|---|---:|---:|
+| Full Sharpe | 1.77 | **1.20** |
+| Full CAGR | 48.3% | **30.0%** |
+| Val Sharpe | 1.68 | 1.44 |
+| Val DD | 20.9% | **28.2%** |
+
+Overnight gap on validation entries: mean **−0.52%**, median −0.18%, p05 −4.5%, p95 +2.1% (n=243).
+
+This is not “slippage noise.” It is largely the **overnight momentum drift** that close-fill earns and a 07:00-Berlin scanner (signal yesterday’s close → fill tomorrow’s open) **structurally cannot**. Honest deployable number under that schedule is ~Sharpe 1.2 / CAGR 30%, not 1.77 / 48%.
+
+**Decision baseline going forward (until near-close ships):** grade **promotion** under
+`fill_mode=next_open`; keep close-fill as the historical control for comparability
+with prior reports.
+
+**Follow-up (done):** execution recovery matrix — see
+**[execution-recovery.md](execution-recovery.md)**. Short version: monotone fill-timing
+gradient + DD recovery prove this is *when you fill*; live bracket **[1.57, 1.77]**;
+gap-cap dead; no more fill-timing sim on this snapshot; ops move R:R scan to NY
+near-close (one scan/day).
+
+### `fip_id` re-derivation — **validated**
+
+Weekly IC fingerprint on this snapshot: **mean IC −0.045, t = −2.92**, reliable (35 weeks). Matches the July record. Safe to reuse when the universe broadens.
+
+---
+
+## Promotion table (rule as written)
+
+| Outcome | Arms |
+|---|---|
+| Promote | only `a2_hold_30` (identity with control) |
+| Reject | every other arm |
+
+No arm cleared ΔSharpe > 1 SE.
+
+---
+
+## What not to do next
+
+- Re-litigate rejected-table items, min_rr, GTL  
+- Regime-conditional max-hold as a “small” experiment  
+- Treat validation-only max-hold glitter as a free CAGR lift  
+- Ship vol targeting as edge without a vol-regime sample  
+- More fill-timing simulation on this snapshot (settled — see execution-recovery.md)  
+- Dual daily qualifying scans (would break gate-reset validation)  
+- Gap-up entry filters (third tail-trim failure)  
+
+## What to do next
+
+1. **Ship near-close execution** — ops checklist in [execution-recovery.md](execution-recovery.md)  
+   (one R:R scan/day in `America/New_York`, MOC window, partial-bar honesty).  
+2. Until that ships: **decision baseline = next_open**.  
+3. Strategy work (nasdaq_all, fip_id, sector) only **after** execution path is decided,
+   graded under the fill mode you will trade.

docs/research/portfolio-capacity-bracket-findings.md

@@ -0,0 +1,124 @@
+# Portfolio-capacity bracket — findings
+
+Date interpreted: 2026-08-05
+
+Status: **capacity and weekly replacement closed as negative results; the
+minimum effective-risk floor remains an open single-variable follow-up.**
+
+This document interprets the frozen v2 run without modifying its generated
+outputs:
+
+- result commit: `24482c6`;
+- simulation source commit: `6fc82ae8574de9104c83273e018391e75a5f8ac6`;
+- frozen specification SHA-256:
+  `f1e37783cf6d157ecc827d48211fa45da16f0a0ac19cd23686b3902d347a1898`;
+- JSON SHA-256:
+  `2435875667097db7416a0d96f412db81d2f2d09ba053748c9f2cfb8a0cba4417`;
+- Markdown SHA-256:
+  `dc3f5de25eb0a156ce51d0025c90e04ac0977e9502dec47bcf1b25bdcf609c81`.
+
+The run completed 78 empty-book paths, 97 warm-seed paths, seven annual
+clusters under both protocols, two cost levels, four arms, and 1,400 cells with
+no validation errors. The construction universe was 505 priced tradable
+symbols plus 4,149 priced rank-only symbols.
+
+## Capacity is economically free
+
+The clean capacity treatment is `cap15_incumbent`: it changes no sizing or
+admission rule. Its cap never bound in any cell (maximum observed position count
+12; zero full-book skips), so it absorbed every opportunity blocked by cap 10.
+
+At 0.10% per fill, split the 175 paths by whether the paired control recorded
+any `skipped_book_full`. Values below are mean paired changes in net EV per
+trade, in R:
+
+| Arm | Cap never bound (n=70) | Cap did bind (n=105) |
+|---|---:|---:|
+| `cap15_incumbent` | +0.0000 | +0.0018 |
+| `cash_unbounded` | +0.0714 | +0.0077 |
+| `cap10_weekly_top10` | -0.0246 | -0.0426 |
+
+The exact zero for cap15 in the never-bound stratum is also a harness validity
+check: when the treatment cannot act, results are identical. Where it does act,
+giving the strategy every slot it requested adds only 0.0018 R/trade. The old
+519-blocked-versus-472-admitted count was true, but it did not imply that the
+blocked opportunities were economically valuable.
+
+Decision: **keep the production cap at 10.** Do not remove it or raise it in the
+expectation of additional edge.
+
+## The positive arm measured the risk floor
+
+`cash_unbounded` combined two treatments: no count cap and a 0.5% minimum
+effective initial-risk fraction. Its EV effect is roughly nine times larger in
+the 70 paths where the control cap never bound, so capacity cannot explain the
+improvement.
+
+Within that never-bound stratum:
+
+| Measure | Control | `cash_unbounded` |
+|---|---:|---:|
+| Mean trades | 75.7 | 69.9 |
+| Mean cash | 27.8% | 28.2% |
+| Mean gross exposure | 72.2% | 71.8% |
+| Mean hold | 15.4 sessions | 15.6 sessions |
+| Mean EV | +0.328 R | +0.399 R |
+| Mean profit factor | 1.60 | 1.75 |
+
+The floor removes about 8% of fills while leaving exposure and holding time
+nearly unchanged. This is selection, not general de-risking: candidates that
+available sizing compresses below half the intended risk are worse on average.
+The report records repeated reject attempts, not the rejected candidates'
+ranks, so whether the effect is rank-mediated remains unknown.
+
+Next research: one single-variable A/B, `cap10_incumbent` versus cap 10 with
+`min_initial_risk_fraction=0.005`, with every other rule unchanged. Do not call
+the current `cash_unbounded` result causal evidence for that floor until this
+confound-free comparison is run.
+
+## Weekly replacement hurts
+
+Median paired deltas read zero because enough cohorts are inert. The distribution
+is not neutral:
+
+| Protocol | Mean ΔEV | P25 ΔEV | Identical paths |
+|---|---:|---:|---:|
+| Empty book | -0.0360 R | -0.0817 R | 27/78 (34.6%) |
+| Warm book | -0.0348 R | -0.1582 R | 14/97 (14.4%) |
+
+The arm made 2,170 replacements and 529 same-symbol re-entries within ten
+sessions, so 24% of replacements were associated with short-horizon churn.
+
+Decision: **reject weekly top-10 replacement.** Future reports should show mean
+paired effects and identical-path fractions beside medians whenever treatments
+are inert in a material share of cohorts.
+
+## Warm dispersion was mostly structurally degenerate
+
+For six of seven anchors, control EV IQR is numerical zero (approximately
+`1e-16`) and Calmar IQR is exactly zero. The displayed ratio `1.000` is therefore
+mostly the implementation's zero-over-zero convention, not evidence of equal
+nonzero dispersion.
+
+Two mechanics cause convergence: sizing and notional limits are fractions of
+equity, making R and ratio metrics scale-invariant; and the 30-session maximum
+hold is shorter than the 63-session minimum seed offset, allowing initial books
+to wash out before the anchor.
+
+The exception is 2023. Control measurement-start positions vary from 6 to 9,
+EV IQR is 0.0274 R, and Calmar IQR is 0.2675. The protocol therefore carries
+state correctly, but its chosen offsets usually erase the initialization effect
+it was intended to measure.
+
+Future initialization studies should use seed offsets shorter than maximum hold,
+approximately 5–25 sessions. The current empty-book cohorts remain the primary
+start-date evidence, but they necessarily mix initialization with market regime.
+
+## Final decisions
+
+1. Keep cap 10; its measured opportunity cost is negligible.
+2. Reject weekly rank replacement.
+3. Do not interpret the `cash_unbounded` improvement as a capacity effect.
+4. Run only the focused cap-10 effective-risk-floor A/B next.
+5. Report means, inert fractions, and absolute dispersion beside medians and
+   ratios in future sparse-treatment studies.

docs/research/portfolio-capacity-bracket.md

@@ -0,0 +1,169 @@
+# Portfolio-capacity bracket — frozen specification
+
+Date frozen: 2026-08-05
+Branch: research/portfolio-capacity-rebalancing
+Runner: scripts/run_portfolio_construction_matrix.py
+
+## Question and motivation
+
+The daily Phase A production control (a0_control: close fill, 30-session
+maximum hold, 1% fixed-fractional risk, no correlation or volatility overlay)
+recorded 472 trades and 519 otherwise qualified entries rejected because the
+ten-position book was full. The blocked share is 519 / (519 + 472) = 52.4%.
+The book is therefore materially arrival-order constrained.
+
+This supersedes the older statement that the ten-slot cap never bound. That
+statement came from a shorter, weekly, pre-gate-reset replay and is not evidence
+about the current daily strategy.
+
+The study brackets the value of capacity before tuning replacement details. It
+does not contain a formal promotion rule or automatically change production.
+Because the current ~505-name production membership is projected backward,
+paired arm-versus-control differences are the primary evidence. Absolute
+profitability is descriptive and survivorship-biased.
+
+Implementation correction: the first completed v1 artifact at commit `23fe39f`
+incorrectly allowed the snapshot's broad rank-only universe to submit trades.
+That artifact is invalid, is removed from the branch, and must not be used for
+strategy conclusions. Runner v2 fixes the construction/ranking partition below.
+
+## Frozen arms
+
+1. **cap10_incumbent:** exact production-style cap-10 control, no displacement.
+2. **cash_unbounded:** no position-count cap; cash/no leverage and the existing
+   20% per-position notional ceiling remain. Reject an entry if actual initial
+   stop-risk after cash/notional sizing is below 0.5% of marked equity.
+3. **cap10_weekly_top10:** on the final trading session of each ISO week, rank
+   holdings plus fresh same-day qualified entrants and retain the top ten.
+4. **cap15_incumbent:** cap 15, no displacement.
+
+All arms use the frozen Phase A control configuration: daily candidate replay,
+live-like full-universe residual-momentum/low-volatility 80/20 rank, activation
+threshold 80, normal gate-reset re-entry, close fill, 3×ATR trail, 30-session
+maximum hold, 1% risk, and costs of 0.10% and 0.20% per fill.
+
+Every priced symbol contributes to the daily cross-sectional rank. Only symbols
+not listed in the snapshot's `research_rank_only` side table may submit trade
+setups to any arm. The resulting construction universe must contain 450-600
+symbols (expected approximately 505); validation fails outside that frozen
+guardrail or when the side table references unknown ticker symbols.
+
+The daily replay uses zero outcome horizon: setup and rank observations continue
+through the snapshot's last session because portfolio simulation, unlike outcome
+grading, does not require 30 future bars.
+
+Control-parity note: a direct main-versus-branch comparison found identical
+total return, CAGR, maximum drawdown, and Sharpe. The branch intentionally
+changes only the first calendar year's `yearly_returns` convention: it starts
+from initial capital rather than equity after the first session, so day-one
+entry costs are now charged to year one. Older reports can therefore show a
+different first-year contextual return without a strategy-performance
+regression. New trade-detail and measurement-start fields are additive.
+
+### Weekly-selection mechanics
+
+- Ordinary exits run before entries/rebalancing.
+- Open slots may still fill from daily qualified entries during the week.
+- On the final ISO-week session, current holdings and that day's fresh qualified
+  entrants use the full-universe strategy_rank for that same date.
+- Stored entry-day rank is never used.
+- Holdings with missing current rank/data are protected and consume a slot;
+  entrants missing rank are ineligible.
+- Incumbents win exact rank ties; symbol is the deterministic final tie-breaker.
+- Rebalance exits pay costs and bypass cooldown/post-stop state.
+- Report entrant-pool sizes, replacements, turnover, and same-symbol re-entry
+  within 5/10/20 sessions.
+
+## Frozen cohorts
+
+research.sqlite is expected to cover 2016-01-04 through 2026-07-17. Residual
+momentum requires 252 benchmark sessions. Empty-book starts additionally require
+504 prior scoring sessions and 252 forward measurement sessions.
+
+- **Empty book:** first eligible session of each month, approximately January
+  2019 through July 2025; start with no positions and measure 252 sessions.
+- **Warm book:** first session of each year 2019–2025 is the measurement anchor.
+  Seed the portfolio on the first session of every ISO week falling 63–126
+  trading sessions before the anchor, carry all positions and gate-reset state
+  forward, and measure the same 252-session anchor window.
+
+Warm portfolio returns reset to marked equity immediately before the anchor
+session. P&L after the anchor from carried positions belongs to portfolio
+returns, while trade EV includes only entries on or after the anchor. Remaining
+positions liquidate at the last measurement close with costs.
+
+The validate-only mode must print realized cohort counts and fail unless both
+protocols contain the seven annual clusters 2019–2025 and every warm anchor has
+at least 12 seeds. It must also print ranking, rank-only, and tradable symbol
+counts plus the raw, removed, and retained qualified-long counts.
+
+## Reporting
+
+Primary reported measures:
+
+- net EV per trade in R, with costs and actual initial stop-risk dollars;
+- Calmar (CAGR / max drawdown);
+- profit factor on net trade R;
+- Gain-to-Pain (sum of all monthly returns / absolute sum of negative months);
+- Sortino using daily returns and zero target.
+
+Also report total return/CAGR, maximum drawdown, Sharpe, win rate, time
+underwater, exposure, cash, average/peak positions, sessions at capacity,
+turnover, costs, qualified/admitted/blocked opportunities, and minimum-risk
+rejections.
+
+For each arm/protocol/cost/metric, pair identical paths with cap10_incumbent,
+take the median paired delta within each start year or annual anchor, show all
+seven cluster values, and headline their median.
+
+Initialization dispersion is reported separately for EV and Calmar: calculate
+the seed-path IQR within each warm anchor, divide by the paired control IQR, show
+all seven ratios, and headline their median. Do not combine them into a composite.
+
+For context only, run a deterministic 10,000-replicate cluster bootstrap over
+the seven paired annual summaries and report the central 90% percentile interval
+for median EV and Calmar deltas and warm IQR ratios. These intervals are not
+promotion gates, independent-population confidence claims, or formal inference.
+
+## Reproducibility and execution
+
+Candidate replay/ranks cache under reports/.cache; each matrix cell checkpoints
+atomically and resume verifies a fingerprint over the implementation commit,
+this specification hash, snapshot SHA-256, cache key, arm definitions, costs,
+and cohort manifest. An authoritative run refuses a dirty worktree.
+
+The existing v1 candidate/rank cache is intentionally reusable: its
+full-universe current-day ranks are correct. Runner v2 derives a fingerprinted
+construction view by removing qualified rows whose symbols are rank-only. V2
+uses a versioned checkpoint directory, so invalid v1 portfolio cells are never
+resumed and the expensive daily rank replay does not need to run again.
+
+The loader reads only ticker ID/symbol and the OHLCV columns used by replay, so
+snapshots created before SEC metadata added `tickers.cik`, `tickers.sic`, and
+`tickers.sic_description` remain valid. Do not migrate or alter the research
+snapshot: its original SHA-256 is part of the run fingerprint.
+
+macOS environment setup from the repository root (zsh):
+
+    python3 -m venv .venv
+    ./.venv/bin/python -m pip install -e '.[dev]'
+
+Preflight:
+
+    ./.venv/bin/python scripts/run_portfolio_construction_matrix.py \
+      backtest_snapshots/research.sqlite \
+      --run-id prod505-capacity-bracket-daily-v1 \
+      --workers auto \
+      --resume \
+      --validate-only
+
+Authoritative run:
+
+    ./.venv/bin/python scripts/run_portfolio_construction_matrix.py \
+      backtest_snapshots/research.sqlite \
+      --run-id prod505-capacity-bracket-daily-v1 \
+      --workers auto \
+      --resume
+
+Commit only the compact final JSON and Markdown reports. Raw curves, trades,
+candidate caches, and checkpoints remain ignored.

docs/research/post-stop-reentry.md

@@ -0,0 +1,196 @@
+# Post-stop re-entry: daily policy study and production decision
+
+## Decision
+
+Use a **normal gate reset** after an initial-stop exit:
+
+1. The initial stop always closes the trade. It is never cancelled because the
+   ticker still passes the gate.
+2. Re-entry remains locked until a later full-universe daily scan observes the
+   ticker **failing** the production activation gate.
+3. The lock remains in place until a subsequent daily scan observes a **fresh
+   qualification**.
+4. Only then may the ticker return to the actionable setup list or be opened
+   through `create_trade`.
+
+Trailing-stop, time, target, and manual exits do not start this state machine.
+Scanner errors do not count as a gate failure. The two transitions are persisted
+on the latest initial-stop `PaperTrade`, so neither a service call nor a restart
+can bypass the rule.
+
+Migration 022 applies the policy prospectively. Existing initial-stop rows are
+grandfathered by marking both reset timestamps complete at their historical
+`closed_at`; otherwise their new NULL columns would be mistaken for active
+locks despite no scanner observations having existed. At runtime, only the
+actual latest closed trade per ticker can start a lock, and only when that exit
+was an initial stop. A newer trailing, time, target, or manual exit therefore
+cannot revive an older stop episode.
+
+This replaces the previously proposed fixed five-session lockdown. The normal
+reset counts an unqualified stop-day close when that close is observed after the
+stop. The stricter experiment, which required a failed close on a later session,
+was not promoted as the research policy.
+
+### Live scheduling boundary
+
+The live daily pipeline runs the R:R scan **before** Outcome Eval. A trade that
+is closed at its initial stop by that Outcome Eval—or by an intraday evaluation
+after the full scan—was therefore still open when the day's gate observation
+ran. Its stop-day state cannot establish the failure. The earliest possible
+failure is the next successful full scan, and a fresh qualification requires a
+subsequent full scan.
+
+The study simulator closes positions before checking same-session re-entry
+state, so its normal `gate_reset` arm can count the stop-day close. At this first
+transition boundary, current live ordering is instead analogous to
+`strict_gate_reset`. The distinction is material: the strict full-period row
+recorded Sharpe 1.68, CAGR 44.8%, and 23.4% drawdown; its disjoint 2025+ row
+recorded Sharpe 1.38, CAGR 32.9%, and 21.0% drawdown. The selected normal-reset
+result (Sharpe 1.77) is therefore policy-study evidence, not exact live
+scheduler-order parity. Changing that ordering would be a separate production
+decision.
+
+## Experiment design
+
+Source: [`reports/daily_reentry_matrix.json`](../../reports/daily_reentry_matrix.json),
+generated 2026-07-17.
+
+| Input | Value |
+|---|---|
+| Snapshot | Production SQLite snapshot through 2026-07-02 |
+| Period used by the all/5y rows | 2022-06-24 to 2026-07-02 |
+| Tickers | 505 |
+| Point-in-time candidate observations | 1,011,248 (492,850 long; 518,398 short) |
+| Live-universe rank observations | 584,393 |
+| Qualified candidates under `live_universe` ranking | 5,189 |
+| Entry cadence | Daily |
+| Selection and ordering | Production GTL gate; residual/high-vol 80/20 rank; long-only after ranking |
+| Exit | 1.5× ATR initial stop; 3× ATR trailing stop; 30-session maximum hold |
+| Portfolio | 10 positions; 1% risk per trade; $10,000 initial capital |
+| Trading cost | 0.1% per side in the primary matrix; 0.1–0.3% robustness sweep |
+| Holdout split | 2025-01-01 |
+
+The expensive daily candidate replay was performed once. Every policy arm then
+used the same candidates, prices, costs, position sizing, capacity, and exit
+logic. `live_universe` ranks all eligible tickers once per session like the live
+scanner. `backtest_legacy` retains the older candidate-only rank approximation as
+a sensitivity check.
+
+### Policies tested
+
+| Arm | Rule after an initial stop |
+|---|---|
+| `immediate` | No memory; a same-day close re-entry is possible |
+| `next_session` | Block only the stop session |
+| `cooldown_2/3/5` | Re-entry allowed at wait-session N |
+| `gate_reset` | Require a failed gate observation, then a later qualification; the stop-day close may establish the failure |
+| `strict_gate_reset` | Ignore the stop-day failure; require a later failed close and then requalification |
+| `gate_reset_improved` | Gate reset plus a higher new stop and non-weaker production rank |
+| `two_session_confirmation` | Require two consecutive qualified post-stop closes |
+
+## Primary result: production-like `live_universe` ranking
+
+The available history is shorter than five years, so the report's `5y` and
+`all` rows cover the same period.
+
+| Policy | Total return | CAGR | Max DD | Sharpe | Trades | Win rate | Post-stop re-entries |
+|---|---:|---:|---:|---:|---:|---:|---:|
+| Immediate | 348.4% | 45.2% | 24.3% | 1.67 | 489 | 35.6% | 155 |
+| Next session | 388.1% | 48.3% | 21.6% | 1.77 | 472 | 36.2% | 142 |
+| Cooldown 2 | 343.5% | 44.8% | 23.4% | 1.68 | 472 | 35.8% | 146 |
+| Cooldown 3 | 293.4% | 40.6% | 22.7% | 1.56 | 474 | 35.9% | 146 |
+| Cooldown 5 | 250.8% | 36.6% | 22.2% | 1.47 | 473 | 35.9% | 145 |
+| **Gate reset** | **388.1%** | **48.3%** | **21.6%** | **1.77** | **472** | **36.2%** | **142** |
+| Strict gate reset | 342.7% | 44.8% | 23.4% | 1.68 | 471 | 35.9% | 144 |
+| Gate reset + improved setup | 267.4% | 38.2% | 24.9% | 1.60 | 422 | 36.3% | 69 |
+| Two-session confirmation | 296.1% | 40.8% | **17.6%** | 1.65 | 441 | **37.9%** | 96 |
+
+At the production capacity, normal gate reset improved all four portfolio
+objectives relative to immediate re-entry: higher total return, CAGR, and
+Sharpe, with lower drawdown. The fixed five-session rule reduced churn but gave
+up too many profitable re-entry opportunities.
+
+`gate_reset` and `next_session` produced exactly the same executed portfolio in
+the `live_universe` runs. Their rules are not equivalent. In this sample, the
+portfolio-level candidate path happened to converge to the same trades. This is
+evidence that blocking same-day re-entry helped; it does **not** isolate an
+independent return premium for the reset condition itself.
+
+## Disjoint 2025+ test window
+
+These are separate books with entries on or after 2025-01-01. They are a useful
+temporal sensitivity check, but not forward evidence: the policy was still
+selected after the historical data existed.
+
+| Policy | Total return | CAGR | Max DD | Sharpe | Trades |
+|---|---:|---:|---:|---:|---:|
+| Immediate | 64.1% | 39.3% | 19.6% | 1.55 | 181 |
+| Next session | 68.5% | 41.8% | 19.2% | 1.66 | 183 |
+| **Gate reset** | **68.5%** | **41.8%** | **19.2%** | **1.66** | **183** |
+| Cooldown 5 | 52.7% | 32.7% | 19.8% | 1.43 | 175 |
+| Strict gate reset | 52.9% | 32.9% | 21.0% | 1.38 | 181 |
+| Two-session confirmation | 35.4% | 22.5% | **18.1%** | 1.02 | 183 |
+
+The gate-reset result did not depend solely on the earlier training period: it
+also beat immediate and the fixed five-session rule in the disjoint test book.
+
+## Cost and capacity sensitivity
+
+At the production capacity of 10, gate reset remained ahead of both immediate
+and cooldown 5 as costs increased.
+
+| Cost per side | Policy | Total return | CAGR | Max DD | Sharpe |
+|---:|---|---:|---:|---:|---:|
+| 0.1% | Immediate | 348.4% | 45.2% | 24.3% | 1.67 |
+| 0.1% | **Gate reset** | **388.1%** | **48.3%** | **21.6%** | **1.77** |
+| 0.1% | Cooldown 5 | 250.8% | 36.6% | 22.2% | 1.47 |
+| 0.2% | Immediate | 296.3% | 40.8% | 25.2% | 1.54 |
+| 0.2% | **Gate reset** | **333.0%** | **44.0%** | **22.9%** | **1.64** |
+| 0.2% | Cooldown 5 | 209.9% | 32.5% | 23.4% | 1.33 |
+| 0.3% | Immediate | 249.8% | 36.5% | 26.0% | 1.41 |
+| 0.3% | **Gate reset** | **284.0%** | **39.7%** | **24.2%** | **1.51** |
+| 0.3% | Cooldown 5 | 164.0% | 27.3% | 24.7% | 1.16 |
+
+The capacity sweep is a real limitation, not a footnote:
+
+| Capacity at 0.1% cost | Immediate Sharpe / CAGR / DD | Gate-reset Sharpe / CAGR / DD | Cooldown-5 Sharpe / CAGR / DD |
+|---:|---|---|---|
+| 5 | 1.33 / 31.9% / 16.8% | 1.37 / 32.8% / 17.5% | **1.46 / 35.8% / 18.3%** |
+| **10 (production)** | 1.67 / 45.2% / 24.3% | **1.77 / 48.3% / 21.6%** | 1.47 / 36.6% / 22.2% |
+| 15 | **1.66 / 44.8% / 24.3%** | 1.63 / 43.0% / **21.6%** | 1.33 / 32.8% / 22.2% |
+
+The promotion is therefore specific to the actual 10-position production book.
+At capacity 5, cooldown 5 ranked best; at capacity 15, immediate had slightly
+higher return and Sharpe while gate reset retained the shallower drawdown. Do
+not generalize the chosen rule to a differently sized portfolio without rerunning
+the matrix.
+
+## Legacy-rank sensitivity
+
+The older candidate-only ranking approximation also favored normal gate reset
+over immediate and cooldown 5, although `next_session` was slightly stronger.
+
+| Policy | Total return | CAGR | Max DD | Sharpe | Trades |
+|---|---:|---:|---:|---:|---:|
+| Immediate | 357.4% | 45.9% | 17.9% | 1.73 | 480 |
+| Next session | **421.5%** | **50.8%** | 18.5% | **1.86** | 466 |
+| **Gate reset** | 408.3% | 49.8% | 18.3% | 1.84 | 464 |
+| Cooldown 5 | 332.3% | 43.9% | 19.6% | 1.71 | 459 |
+| Strict gate reset | 351.3% | 45.5% | 20.4% | 1.72 | 457 |
+
+## Why gate reset was promoted
+
+- It is tied to a new signal episode instead of an arbitrary elapsed time.
+- At the production capacity, it beat immediate and five-session cooldown on
+  return, CAGR, drawdown, and Sharpe.
+- The advantage survived costs of 0.2% and 0.3% per side and the disjoint 2025+
+  test book.
+- It avoids cancelling a valid stop: the loss and transaction costs are always
+  realized before any later trade.
+- It avoids the extra filters that weakened strict reset, improved-setup reset,
+  and two-close confirmation.
+
+The correct interpretation is deliberately modest: **normal gate reset is the
+best production rule among the tested policies for the current 10-position
+book.** It is not proof that gate reset is a universal source of alpha. Forward
+paper-trade monitoring is still the only genuinely new evidence.

docs/research/prod-book-universe-horizon.md

@@ -0,0 +1,127 @@
+# Production book × universe × horizon matrix
+
+**Status:** PRE-REGISTERED — prepare / MacBook run; no production changes.  
+**Branch:** `research/earnings-gap-and-sue`  
+**Runner:** `scripts/run_prod_book_universe_matrix.py`
+
+---
+
+## Question
+
+How does the **live production book** (unchanged knobs) behave when we only vary:
+
+1. **History length** used for entries (≈4y vs since 2016-07)  
+2. **Tradable universe** (prod ~505 vs 505 + PIT liquid Nasdaq/breadth)
+
+No strategy modifications: same residual gate, 80/20 high-vol rank, GTL entry
+machinery, 3× ATR trail, 30d max hold, gate-reset re-entry, `fill_mode=close`,
+cost 10 bps/side, max 10, 1% risk.
+
+---
+
+## Pre-registered arms (locked)
+
+| id | label | Entry start | Tradable universe |
+|---|---|---|---|
+| **A** | prod_4y_505 | **2022-07-01** | Prod ~505 only |
+| **B** | prod_4y_505_liquid | **2022-07-01** | Prod ∪ liquid top-1500 |
+| **C** | prod_2016_505 | **2016-07-01** | Prod ~505 only |
+| **D** | prod_2016_505_liquid | **2016-07-01** | Prod ∪ liquid top-1500 |
+
+- **End:** last available bar in snapshot (no artificial end).  
+- **4y start** chosen to align with recent Phase‑A / book baselines (~mid‑2022 → mid‑2026).  
+- **2016-07-01** = first full month after typical Alpaca floor (~2016-01); residual 12‑1 needs ~1y bars so first residual ranks appear mid‑2017 where feed allows.
+
+### Universe definitions
+
+| set | definition |
+|---|---|
+| **Prod ~505** | Symbols **not** in `research_rank_only` on the research snapshot (the original prod-universe copy). |
+| **Liquid top-1500** | Point-in-time: among names with as-of close ≥ **$5** and valid 63d median $vol, keep top **1500** by that $vol. Same definition as breadth IC research. |
+| **Prod ∪ liquid** | A name may enter the book on date *t* if it is prod **or** in the liquid top-1500 at *t*. |
+
+Cross-sectional residual / vol / 80/20 ranks are **recomputed inside each arm’s
+eligible candidate set** that period (so breadth arms are not ranked against
+non-eligible thin names).
+
+### Explicit non-goals
+
+- No sector residual, SUE, FIP filter, gap-cap, take-profit, vol-target, corr-cap  
+- No retune of trail / cutoff / min_rr  
+- Survivorship: report levels with the standard caveat; **compare arms relatively**
+
+### Reporting (required table)
+
+Per arm: Sharpe, Sharpe SE (Mertens), CAGR %, max DD %, total return %, trades,
+win rate if available, start/end, n qualified longs. One markdown table + JSON.
+
+**No promotion rule** — descriptive matrix only. Human decides whether breadth
+or depth changes the risk story.
+
+---
+
+## Snapshot requirements
+
+- Prefer MacBook **deep** `research.sqlite` after sector-resid deepen (prod names
+  from ~2016, breadth deep, completion manifest `complete=true`).  
+- Race-guard before run.  
+- Sector map / sector ETFs optional (not used for ranking).
+
+---
+
+## Results
+
+Generated: `2026-07-19T14:07:37` · artifact
+`reports/prod-book-universe-horizon-20260719-140737.json`  
+Snapshot: MacBook deep `research.sqlite` (506 prod + breadth prices; 2.39M raw
+GTL candidates). Strategy knobs = live production (residual 80, 80/20 high-vol
+rank, ATR trail 3×, hold 30, gate-reset, `fill_mode=close`).
+
+> Survivorship: today's constituents backfilled. **Compare arms relatively.**
+> Absolute deep CAGR/Sharpe are not deployable forecasts.
+
+| arm | universe | entries from | Sharpe | SE | CAGR % | max DD % | total ret % | trades | win % | vs SPY |
+|---|---|---|---:|---:|---:|---:|---:|---:|---:|---:|
+| **A** | 505 only | 2022-07-01 | **1.32** | 0.49 | **31.8** | **18.9** | +205 | 374 | 35.6 | +96.5 |
+| **B** | 505 + liquid 1500 | 2022-07-01 | 0.14 | 0.50 | −1.8 | 55.3 | −7 | 706 | 29.3 | +95.0 |
+| **C** | 505 only | 2016-07-01 | **0.88** | 0.31 | **16.7** | **24.4** | +369 | 763 | 36.7 | +257 |
+| **D** | 505 + liquid 1500 | 2016-07-01 | −0.06 | 0.32 | −7.0 | 73.9 | −52 | 1567 | 28.0 | +254 |
+
+Qualified longs: A 1 448 · B 6 587 · C 2 450 · D 11 551.
+
+### Read (relative only)
+
+1. **Same strategy, broader liquid universe kills the book** (A→B and C→D).
+   Sharpe collapses; DD roughly triples; win rate drops ~6–8pp; trade count
+   ~doubles. This matches earlier breadth IC work: the production residual +
+   high-vol package is a **large-cap / prod-universe** edge, not a
+   “more names = better” edge.
+
+2. **Longer history on 505 stays positive but softer** (A→C). Sharpe 1.32 → 0.88,
+   CAGR 32% → 17%, DD 19% → 24%. Still well above the liquid-breadth arms.
+   Levels are optimistic (survivorship); the useful message is “edge does not
+   vanish when 2018/2020 are included,” not “expect 17% CAGR forever.”
+
+3. **Arm A vs older Phase‑A / short-window controls** (~Sharpe 1.7–2.1): this
+   matrix re-ranked on deep research.sqlite with a fixed entry start; numbers
+   need not match prior reports row-for-row. Use **this table for A–D
+   comparisons**, not for rewriting the production baseline number.
+
+4. **No production change implied.** Keep the live ~505 universe. Do not broaden
+   the tradable set to liquid Nasdaq under current knobs without a new
+   pre-registered design (and almost certainly a different rank/tilt package).
+
+## Verdict
+
+**Descriptive matrix complete.**
+
+| question | answer from this matrix |
+|---|---|
+| Prod book @ ~4y / 505 | Positive (arm A) |
+| Same + liquid Nasdaq | **No** — large degradation (arm B) |
+| Prod book since 2016 / 505 | Still positive, milder (arm C) |
+| Same + liquid Nasdaq deep | **No** — worst arm (arm D) |
+
+**PENDING_HUMAN** only for whether to log “universe broaden under current knobs”
+as rejected in the main research index. Strategy knobs unchanged either way.
+

docs/research/regime-monitor-v2.md

@@ -1,75 +0,0 @@
-# Regime Monitor v2 methodology
-
-The Regime Monitor is an observational AI/Tech risk thermometer. It does not
-gate entries, exits, position size, ranking, or alerts about individual setups.
-
-## Outputs
-
-**State** measures current structural stress:
-
-- Price structure, 40%: `max(P1, P2, P3)`, so the correlated 200-DMA, death-cross,
-  and drawdown readings receive one capped vote.
-- Fixed-basket breadth level, 25%.
-- HY option-adjusted credit spread, 20%.
-- VIX level, 15%.
-
-**Warning** measures deterioration and divergence:
-
-- Fixed-basket breadth divergence while SMH holds/rises, 50%.
-- 60-session SMH/SPY relative-strength deterioration, 30%.
-- Hyperscaler capex cuts, 12%.
-- Good-news-stock-down earnings reactions, 8%.
-
-Combined, RSP/SPY (former F4), and the NVDA canary (former P6) do not enter v2.
-
-## Scale and missing data
-
-Zero means ordinary/healthy, and only stress contributes positively. Automated
-capex `raising`/`holding` and no good-news-stock-down pattern map to zero;
-`mixed`, unknown, and stale observations are unavailable rather than neutral 50.
-Manual observations use the same categories: each hyperscaler is marked
-`raising`, `holding`, `cutting`, or `unknown`, while the earnings reaction is
-`yes`, `no`, or `mixed`. F1 is derived from the share of at least three known
-hyperscalers marked `cutting`; arbitrary numeric overrides are not accepted.
-
-Scores renormalize over available fixed weights, but a band is published only at
-75% or greater coverage. Trend deltas are suppressed when the participating
-pillar set changes. Bands are stable `<30`, watch `<60`, elevated `<80`, and
-breaking `>=80`.
-
-Credit uses named HY OAS anchors (3.5 mild, 5.0 elevated, 7.0 stressed) for 70%
-of its score and a ten-year upper-tail percentile for 30%.
-
-## Point-in-time record
-
-The first v2 run rebuilds the latest 400 trading sessions with sufficient sensor
-warm-up. Routine runs thereafter insert/update only the latest trading date.
-Fundamental observations have an effective date (normally the next session after
-collection) and are never replayed backward. The history API and main chart show
-only snapshots marked `methodology: v2`.
-
-Each snapshot stores the fixed basket symbols, hash, and freeze date. Reconstructed
-history before that freeze date is retrospective/exploratory; readings after it
-form the forward record.
-
-The automatic 400-session rebuild is intentionally one-shot: it runs only when
-no v2 snapshot exists. If an initial seed used partial data or the wrong basket,
-the operational reseed procedure is to remove the v2 snapshot rows and run the
-Regime Monitor job again. There is no routine force-rebuild flag.
-
-## Warning study
-
-The study calls the outcome a **10% correction**, not a regime break. The first
-70% of sessions freezes the 80th-percentile warning threshold; alarm episodes are
-measured on the final 30%. An alarm requires an upward crossing and another alarm
-requires a reset below the threshold. The report exposes warned/missed events,
-false alarms per year, median lead, sample dates, event count, report date, and
-whether the result is exploratory or a true forward holdout. UI claims are
-generated from that report; no performance sentence is hard-coded.
-
-## Operator rule
-
-Quadrant alerts default off for new/reset configurations. When enabled they
-require fresh inputs, at least 75% coverage on both axes, two consecutive daily
-confirmations, hysteresis, and cooldown. Every alert states: **Risk thermometer —
-not a trade signal.**

docs/research/regime-monitor-v3.md

@@ -0,0 +1,202 @@
+# Regime Monitor v3 methodology
+
+The Regime Monitor is an observational AI/Tech risk thermometer. It does not
+gate entries, exits, position size, ranking, or alerts about individual setups.
+
+v3 supersedes v2. Every parameter below was calibrated against the 408 v2
+sessions ending 2026-07-24, reproduced offline from the same Alpaca and FRED
+inputs the live job uses; the reproduction matched the stored prod distribution
+exactly (State avg 22.6/22.7, p80 35.1, max 91.2, P3 pegged 39, W1 live 108).
+
+## What changed and why
+
+**Fundamentals left the score.** F1 (capex) and F3 (good-news-stock-down)
+carried 12 + 8 of 100 Warning points. Pegged at maximum stress they produced a
+Warning of exactly 20.0 — below the event study's 25.3 alarm threshold, and
+still inside the "stable" band. The sourced observation could not change any
+published conclusion, so refreshing it looked like it did nothing. They are now
+a qualitative overlay reported beside the scores. Capex also stopped scoring
+`raising` and `holding` identically at 0: `holding` is the deceleration case and
+now scores 50, so a boom no longer reads the same as a stall.
+
+**The drawdown sensor stopped saturating.** v2 used `dd_pct * 5`, reaching 100 at
+a 20% drawdown — the 90th percentile of the observed distribution. 39 of 408
+sessions sat at exactly 100 with no resolution left, and the price pillar showed
+the top band on 13.5% of sessions. v3 uses named anchors with headroom past the
+observed 36% maximum, and blends leader/confirm 2:1 as P1 and P2 already did
+instead of taking `max()`. P3's realized share of State falls from 65% to 40%,
+matching its nominal weight.
+
+**Warning gained a sensor with range.** The HY OAS *level* is pinned at zero
+below the 3.5 mild anchor (2.77 at the cutover), so credit contributed nothing
+in a calm tape. Its 20-session rate of change still does, and spread widening is
+a classic lead.
+
+**The credit percentile leg was removed.** Its reference window silently shrank
+from 10 years to 3 when ICE restricted the upstream series in April 2026, after
+which it scored 20 points of stress at a spread the same sensor's anchors call
+"mild". See Calibration below.
+
+**Breadth loss counts during declines.** v2's divergence gate was
+`price_ret >= 0`, so the sensor zeroed during every selloff. On 2026-07-24 the
+basket shed 10 points of participation in 20 sessions while SMH fell 11.9% and
+Warning printed exactly 0. v3 tapers to a floor instead: deterioration counts
+fully when price masks it (true divergence, the dangerous pre-top case) and at
+35% when price confirms it. Breadth *level* lives in State, but breadth
+*velocity* appears nowhere else, so this is not double counting.
+
+**Bands are per axis.** v2 Warning never exceeded 64.9 in 408 sessions while
+State reached 91.2, yet both used 30/60/80 with quadrant dividers at 60. The
+upper half of the Warning axis was unreachable.
+
+## Outputs
+
+**State** — current structural stress:
+
+- Price structure, 40%: `max(P1, P2, P3)`, one capped vote for correlated reads.
+- Fixed-basket breadth level, 25%.
+- HY option-adjusted credit spread level, 20%.
+- VIX level, 15%.
+
+**Warning** — deterioration and divergence:
+
+- Fixed-basket breadth divergence, 45%.
+- 60-session SMH/SPY relative-strength deterioration, 30%.
+- HY OAS 20-session widening, 25%.
+
+Combined, RSP/SPY (former F4), and the NVDA canary (former P6) do not enter v3.
+
+## Calibration
+
+P3 drawdown anchors, as (drawdown %, score): 0→0, 4→10, 8→25, 16→50, 28→78,
+40→100, flat outside. Credit impulse is relative (+35% over 20 sessions = 100)
+rather than absolute, because +0.5pp means something very different at an OAS of
+2.7 than at 8.0.
+
+Bands are round, meaning-anchored numbers, not percentile fits — percentile
+thresholds would drift on every rebuild and silently rewrite what past snapshots
+meant. Realized shares over the calibration window:
+
+| Axis | stable | watch | elevated | breaking | thresholds |
+|------|--------|-------|----------|----------|------------|
+| State | 73.3% | 15.0% | 8.3% | 3.4% | 20 / 50 / 80 |
+| Warning | 69.4% | 19.6% | 7.6% | 3.4% | 20 / 40 / 60 |
+
+Quadrant dividers sit at each axis's watch/elevated boundary: State 50,
+Warning 40.
+
+Scores renormalize over available fixed weights, but a band is published only at
+75% or greater coverage. Trend deltas are suppressed when the participating
+pillar set changes. Zero means ordinary/healthy; only stress contributes.
+
+Credit level is the named HY OAS anchors alone: 3.5 mild, 5.0 elevated, 7.0
+stressed, linear between, and nothing else. v2 blended those anchors at 70% with
+a 30% upper-tail percentile over a nominally 10-year window.
+
+That leg was removed rather than repaired. ICE restricted FRED to a rolling
+3-year window for `BAMLH0A0HYM2` in April 2026 — the series metadata states it
+outright ("Starting in April 2026, this series will only include 3 years of
+observations"), and an unbounded request returns the same 795 observations as a
+30-year one. The v2 percentile therefore ranked the current spread against three
+uniformly tight years (range 2.59–4.61 over the calibration window), which made
+it fire early and saturate absurdly: at an OAS of 3.50 — the level the anchors
+call *mild*, scoring zero stress — the blended sensor read 20.1, and the
+percentile leg pegged at 100 by an OAS of 4.5. Across the 408 sessions it
+roughly tripled the credit sensor's average (2.70 vs 1.00) and more than doubled
+its nonzero days (60 vs 27).
+
+The anchors already encode the long-run distribution as constants, so the
+percentile was a second, noisier estimate of the same thing. What it was
+genuinely reaching for — "unusual versus recent history" — is now W3 on the
+Warning axis, computed as a rate of change, which is where deterioration
+belongs. Removing it moved State's average by −0.4 and its maximum by −3.8, left
+Warning bit-identical, and did not shift any band threshold.
+
+A long-history alternative (`BAA10Y`, Fed-published, 7,712 observations back to
+1997) was considered and rejected: ranking an HY spread against investment-grade
+history is not a coherent statistic, and it would rescue a leg that is redundant
+anyway.
+
+Every snapshot now records `data_quality.credit_history_days` and
+`vix_history_days`. This defect was invisible for roughly three months because
+nothing asserted the window the code claimed; the spans make a future upstream
+truncation show up in the record instead of quietly reshaping a sensor.
+
+**Survivorship caveat.** The basket was frozen 2026-07-15 but the calibration
+window reaches back to 2024, so names were partly selected for having done well.
+Every distribution above inherits that bias. It is the same bias v2 carried, so
+the v2/v3 comparison is like-for-like, but the absolute band shares are
+optimistic.
+
+## Point-in-time record
+
+The first run under a new `METHODOLOGY` rebuilds the latest 400 trading sessions
+with sufficient sensor warm-up; routine runs thereafter insert/update only the
+latest trading date. The history API and main chart show only snapshots matching
+the current methodology, so a bump reseeds the series rather than splicing two
+formulas into one line.
+
+The fundamental overlay keeps its effective date (normally the next session after
+collection) and is never replayed backward, so a rebuild cannot stamp today's
+observation onto historical snapshots. Because the observation is stored in a
+single slot, a refresh replaces the previously effective record: the snapshot
+therefore reports the overlay as `pending` until the new effective date, and the
+live reading additionally carries `fundamental_context` so a just-collected
+observation is visible immediately rather than appearing to have done nothing.
+
+Each snapshot stores the fixed basket symbols, hash, and freeze date.
+Reconstructed history before that freeze date is retrospective/exploratory.
+
+## Warning study
+
+The study calls the outcome a **10% correction**, not a regime break. The first
+70% of sessions freezes the 80th-percentile warning threshold; alarm episodes are
+measured on the final 30%. Because v3 dropped fundamentals from the score, the
+study now measures exactly the live Warning score rather than a technical-only
+approximation of it, and both are computed from one shared sensor definition
+(`warning_sensor_scores`) so they cannot drift apart.
+
+A cached report is discarded when its methodology no longer matches, so the panel
+reverts to "not run yet" after a bump rather than showing stale numbers. **Re-run
+the Event Study job after cutting over to v3.**
+
+### Reading the result
+
+The report carries a `reliability` block and the UI renders its warnings, because
+the headline numbers invite over-reading in two specific ways.
+
+**The holdout is thin.** The study detects 11 corrections across 5 years but the
+70/30 split leaves only 4 in the test period. Recall is therefore one event away
+from a materially different headline, and in practice the event that flips is
+decided by where the frozen threshold happens to land rather than by whether the
+score saw anything. The v3 cutover run illustrates it: v3 scored 2/4 against v2's
+3/4, but "v3 without the credit sensor" scores 3/4 at a *higher* threshold
+(35.5) than shipped v3 misses it at (32.3) — because the alarm rule needs a
+rising edge, and a lower threshold can mean the alarm already fired outside the
+20-session horizon and never reset below. Below `MIN_EVENTS_FOR_CONFIDENCE`
+holdout events the report says so explicitly.
+
+Some events carry no information at all for comparison: in that run every
+variant caught 2026-03-06, every variant missed 2026-06-05, and every variant
+"caught" 2025-11-20 with a 1-session lead, which is coincident rather than a
+warning.
+
+**Sensor coverage can straddle the split.** The score renormalises over available
+sensors, so a training window predating a sensor's history freezes the threshold
+on a different construct than the holdout is measured against. At the v3 cutover
+only 39% of training sessions had all three Warning sensors versus 100% of the
+test period, because credit history begins 2023-07-25.
+
+Restricting the threshold to sensor-matched training sessions was tried and is
+*not* the fix: those sessions are a calm recent stretch, so the threshold drops
+from 32.3 to 22.5 and false alarms rise from 3.3 to 8.6 per year. It trades a
+coverage bias for a regime-selection bias. The honest position is that the
+threshold is hypersensitive to window choice at this sample size; the report
+states its limits rather than pretending to a precision it does not have.
+
+## Operator rule
+
+Quadrant alerts default off for new/reset configurations. When enabled they
+require fresh inputs, at least 75% coverage on both axes, two consecutive daily
+confirmations, hysteresis, and cooldown. Every alert states: **Risk thermometer —
+not a trade signal.**

docs/research/sector-residual-momentum.md

@@ -0,0 +1,235 @@
+# Sector-residual momentum (Tier-1 alpha research)
+
+**Status:** **CLOSED / REJECTED** — do not resurrect without a new pre-registered protocol.  
+**Branch:** `research/earnings-gap-and-sue` (final grade) · earlier short-window work on `research/sector-residual-momentum`  
+**Production impact:** none. Market residual 12-1 remains the production momentum leg.  
+**Authoritative deep grade:** `reports/sector-resid-deep-20260719-113319.json` (**FAIL**)  
+**Short-window A/B (superseded for promotion):** `reports/sector-residual-20260719-083356.json` — knife-edge only; not decisive after deep masked retest.
+
+### Closure (2026-07-19)
+
+Pre-registered deep test on repaired snapshot + liquid-1500 mask:
+
+| check | result |
+|---|---|
+| weeks extended (≫ 35) | pass (83) |
+| sign +, reliable, eras both + | pass |
+| t ≥ `mom_12_1_resid` same CS | pass (1.69 ≥ 1.30) |
+| \|mean IC\| ≥ 0.03 | **fail (0.0268)** |
+
+**Verdict:** Task 1 CLOSED — sector residual dead on deep evidence.  
+`mom_12_1_sector_demeaned` remains DEAD for promotion. No further sector-residual variants from this thread.
+
+---
+
+## Pre-registration (locked before first research run)
+
+### Hypothesis
+
+Residualizing 12–1 momentum against the sector, not only the market, reduces
+factor volatility at similar return (Blitz / Huij / Martens-style) → higher
+Sharpe on the production book when the residual replaces market-only residual
+as the momentum leg.
+
+### Signals (candidates)
+
+| signal | construction |
+|---|---|
+| `mom_12_1_sector_resid` | Two-factor residual vs SPY + ticker’s sector ETF. Same window as `mom_12_1_resid`: ≥100 daily obs, 252-bar lookback, 21-bar skip; two-factor OLS betas **without intercept**; cumulate residual returns over the formation window. |
+| `mom_12_1_sector_demeaned` | Plain `mom_12_1` minus the **cross-sectional** mean of `mom_12_1` within the same GICS sector that week (≥2 names in sector). No regression. |
+
+### Baselines (same run, same cross-sections — iron rule)
+
+Always report side-by-side with:
+
+- `mom_12_1`
+- `mom_12_1_resid`
+
+Computed on the **identical** weekly non-overlapping cross-sections in this run.
+Never compare against IC numbers from another report.
+
+### Iron rule (IC harness)
+
+Source of truth: `_signal_evaluation` in `app/services/backtest_service.py`.
+
+- Mean weekly Spearman IC on **non-overlapping** weekly windows  
+- Bar: \|mean IC\| ≥ ~0.03, **consistent positive sign**, `reliable: true` (≥ 12 windows)
+
+### Promotion to portfolio A/B (candidate → book)
+
+A candidate promotes to A/B **only if**:
+
+1. It clears the iron-rule bar **and**
+2. Its IC **t-stat ≥** that of `mom_12_1_resid` on the same cross-sections.
+
+### Portfolio A/B grading (if and only if IC promotion fires)
+
+- Swap candidate in as the **momentum leg** of the production 80/20 momentum/vol
+  rank **and** as the gate-percentile signal.
+- `fill_mode=close`, `COST_PER_SIDE = 0.001`, full config otherwise unchanged.
+- Validation window = entries ≥ **2024-07-01** (call it **validation**, not
+  holdout — contaminated by prior experiments).
+- Pre-registered promotion bar:
+  - validation Sharpe ≥ control − 0.5·SE  
+  - full-period Sharpe and max-DD **not worse** than control  
+- Report Lo / Mertens-adjusted SEs.
+
+### Optional sector-cap sub-experiment
+
+Only if labels are in **and** A/B ran: max **3** positions per sector in the
+10-slot book. Same A/B grading. **Tail-trim presumption of guilt** (rule 4):
+report entry counts and both tails of the R distribution. Rising win rate with
+falling Sharpe/CAGR = red flag → do not promote.
+
+**This run:** sector-cap arm **not executed** (optional; A/B unconstrained book
+only). Can be a human-approved follow-up.
+
+### Verdict labels
+
+| label | meaning |
+|---|---|
+| **PROMOTE** | Clears pre-registered bar; human decides next (wire design separate) |
+| **PARK** | Inconclusive / weak; keep machinery, no book change |
+| **DEAD** | Failed iron rule or worse than residual baseline with clear sign |
+
+### Explicit non-goals
+
+- No production deploy from this doc  
+- Do not resurrect: take-profit exits, EV gate, regime entry-blocking,
+  inverse-vol sizing, gap-caps, unconditional FIP filter
+
+---
+
+## Data provenance
+
+### Snapshot race guard
+
+| check | result |
+|---|---|
+| Snapshot path | `backtest_snapshots/prod.sqlite` |
+| Manifest | none (expected for prod snapshot); bar-count sanity applied |
+| Tickers / OHLCV | **506** / **629,263** |
+| Bars min / avg / max | 14 / 1246.1 / 1261 |
+| OHLCV range | 2021-06-24 → 2026-07-02 |
+| Partial-build red flags | none (avg bars healthy) |
+
+Integrity fingerprint on same run: `fip_id` mean IC **−0.045** / t **−2.91**
+(35 weeks, N≈498) — matches the established prod fingerprint.
+
+### Sector labels
+
+| source | count |
+|---|---:|
+| Public S&P 500 GICS CSV | 496 newly filled |
+| FMP profile requests | 10 (all missing after CSV) |
+| Mapped / universe | **505 / 506 (99.8%)** |
+| With mappable ETF | 505 |
+| Still missing | **RHM** only |
+
+Persist path: `data/research/ticker_sector_map.json`.
+
+FMP aliases (`Technology`, `Consumer Defensive`, `Financial Services`) map to
+SPDRs via the alias table in `app/services/sector_map.py`.
+
+### Sector ETFs in `benchmark_prices` (auxiliary only — not tradable)
+
+| symbol | bars | min date | max date |
+|---|---:|---|---|
+| SPY | 1516 | 2020-07-06 | 2026-07-17 |
+| XLB…XLY (11) | 1512 each | 2020-07-10 | 2026-07-17 |
+
+Fetched via Alpaca `Adjustment.SPLIT` into **`benchmark_prices`** (same table as
+SPY) so they never enter the ticker universe or candidate replay.
+
+---
+
+## Results
+
+Generated: `2026-07-19T08:33:56`
+
+### IC harness (identical cross-sections, production 506-name universe)
+
+| signal | mean_ic | ic_t_stat | weeks | avg_N | reliable | ic+_pct | quintile spread |
+|---|---:|---:|---:|---:|---|---:|---:|
+| **mom_12_1_sector_resid** | **0.0578** | **2.34** | 35 | 497.7 | true | 65.7 | 0.0245 |
+| mom_12_1_resid | 0.0552 | 1.98 | 35 | 497.7 | true | 60.0 | 0.0207 |
+| mom_12_1 | 0.0531 | 1.61 | 35 | 497.7 | true | 65.7 | 0.0206 |
+| mom_12_1_sector_demeaned | 0.0340 | 1.32 | 35 | 496.7 | true | 62.9 | 0.0154 |
+
+### IC promotion grades
+
+| candidate | iron rule | t ≥ resid | promote_to_ab |
+|---|---|---|---|
+| `mom_12_1_sector_resid` | pass (IC 0.058, +sign, reliable) | **yes** (2.34 ≥ 1.98) | **yes** |
+| `mom_12_1_sector_demeaned` | pass (IC 0.034, +sign, reliable) | **no** (1.32 < 1.98) | **no** |
+
+### Portfolio A/B — `mom_12_1_sector_resid` as residual leg
+
+Config: production 80/20 residual/high-vol rank + gate percentile, `fill_mode=close`,
+cost 10 bps/side, ATR trail / gate-reset re-entry as live. Validation split
+2024-07-01.
+
+| window | arm | Sharpe | Sharpe SE (Mertens) | CAGR % | max DD % | trades | n_days |
+|---|---|---:|---:|---:|---:|---:|---:|
+| train | control (resid) | 1.30 | 0.685 | 29.2 | 21.4 | 176 | 525 |
+| train | treatment (sector resid) | **1.57** | 0.677 | **35.5** | **19.8** | 176 | 530 |
+| validation | control | **2.92** | 0.709 | **76.3** | **11.7** | 150 | 501 |
+| validation | treatment | 2.57 | 0.701 | 66.3 | 14.8 | 163 | 501 |
+| full | control | 2.09 | 0.497 | 51.6 | 21.4 | 322 | 1000 |
+| full | treatment | 2.09 | 0.491 | 51.0 | **19.8** | 337 | 1005 |
+
+**Pre-registered A/B checks**
+
+| check | result |
+|---|---|
+| val Sharpe ≥ control − 0.5·SE | **pass** (2.57 ≥ 2.92 − 0.5×0.701 = 2.5695) — **knife-edge** |
+| full Sharpe not worse | **pass** (2.09 = 2.09) |
+| full max DD not worse | **pass** (19.8 < 21.4) |
+
+Qualified long candidates: control 1086 vs treatment 1210 (sector residual
+gates a slightly larger set).
+
+---
+
+## Verdict (final — archived)
+
+| signal | verdict | note |
+|---|---|---|
+| **`mom_12_1_sector_resid`** | **CLOSED / REJECTED** | Deep masked IC 0.0268 &lt; 0.03 bar (`sector-resid-deep-20260719-113319`). Short-window PROMOTE superseded. |
+| **`mom_12_1_sector_demeaned`** | **DEAD** | Never cleared t vs market residual; stays dead. |
+
+Short-window evidence below is **historical only** (pre-deep retest). Do not use it
+to reopen promotion.
+
+### Read carefully (archived context)
+
+1. Short-window IC (0.058 / t 2.34 vs resid 0.055 / t 1.98 on 35 weeks) and knife-edge
+   A/B looked openable — that was the data gap era (shallow prod bars).
+2. Deep repaired + liquid-1500 retest closed the case: weeks 83, mild +IC, **below bar**.
+3. Production keeps **market** residual 12-1. Research harness may still *emit*
+   sector residual for diagnostics; it is not a promotion candidate.
+4. **Do not resurrect** without a new pre-registered protocol and new data.
+
+---
+
+## What a human must decide next
+
+**Nothing on Task 1** — archived. Optional: leave research machinery in tree
+(harmless) or delete later as cleanup; not a strategy decision.
+
+---
+
+## Implementation notes
+
+Research runners and sector-residual harness hooks were **removed after close**
+(2026-07-19 cleanup). Evidence remains in the report artifacts below. Do not
+re-add without a new pre-registered protocol.
+
+---
+
+## Artifacts
+
+| file | role |
+|---|---|
+| `reports/sector-resid-deep-20260719-113319.json` | **Authoritative deep FAIL** |
+| `reports/sector-residual-20260719-083356.json` | Short-window IC/A/B (superseded for promotion) |

frontend/harness.html

@@ -0,0 +1,18 @@
+<!doctype html>
+<html lang="en" class="dark">
+  <head>
+    <meta charset="UTF-8" />
+    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+    <title>FundamentalsPanel harness</title>
+    <link rel="preconnect" href="https://fonts.googleapis.com" />
+    <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
+    <link
+      href="https://fonts.googleapis.com/css2?family=Space+Grotesk:wght@400;500;600;700&family=Instrument+Sans:wght@400;500;600;700&family=IBM+Plex+Mono:wght@400;500;600&display=swap"
+      rel="stylesheet"
+    />
+  </head>
+  <body class="bg-[#0a0b11] text-gray-100 font-sans">
+    <div id="root"></div>
+    <script type="module" src="/src/dev/harness.tsx"></script>
+  </body>
+</html>

frontend/index.html

@@ -3,7 +3,9 @@
   <head>
     <meta charset="UTF-8" />
     <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+    <meta name="theme-color" content="#0a0b11" />
     <title>Signal — Trading Intelligence</title>
+    <link rel="icon" href="/favicon.svg" type="image/svg+xml" />
     <link rel="preconnect" href="https://fonts.googleapis.com" />
     <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
     <link

frontend/public/favicon.svg

@@ -0,0 +1,56 @@
+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 32 32" role="img" aria-label="Signal">
+  <defs>
+    <radialGradient id="void" cx="50%" cy="42%" r="70%">
+      <stop offset="0%" stop-color="#141622"/>
+      <stop offset="100%" stop-color="#0a0b11"/>
+    </radialGradient>
+    <radialGradient id="emberCore" cx="50%" cy="50%" r="50%">
+      <stop offset="0%" stop-color="#ffb199"/>
+      <stop offset="45%" stop-color="#ff6a45"/>
+      <stop offset="100%" stop-color="#e24a2a"/>
+    </radialGradient>
+    <radialGradient id="emberGlow" cx="50%" cy="50%" r="50%">
+      <stop offset="0%" stop-color="#ff6a45" stop-opacity="0.5"/>
+      <stop offset="55%" stop-color="#ff6a45" stop-opacity="0.1"/>
+      <stop offset="100%" stop-color="#ff6a45" stop-opacity="0"/>
+    </radialGradient>
+    <linearGradient id="rim" x1="0%" y1="0%" x2="100%" y2="100%">
+      <stop offset="0%" stop-color="#9fd8e8"/>
+      <stop offset="45%" stop-color="#2f9db2"/>
+      <stop offset="100%" stop-color="#1a6f82"/>
+    </linearGradient>
+    <filter id="soft" x="-40%" y="-40%" width="180%" height="180%">
+      <feGaussianBlur stdDeviation="0.55" result="b"/>
+      <feMerge>
+        <feMergeNode in="b"/>
+        <feMergeNode in="SourceGraphic"/>
+      </feMerge>
+    </filter>
+  </defs>
+
+  <rect width="32" height="32" rx="7" fill="url(#void)"/>
+
+  <circle cx="16" cy="16" r="11.2" fill="none" stroke="#2f9db2" stroke-opacity="0.18" stroke-width="0.75"/>
+  <circle cx="16" cy="16" r="8.4" fill="none" stroke="#2f9db2" stroke-opacity="0.28" stroke-width="0.85"/>
+
+  <path
+    d="M 7.4 18.8 A 9.2 9.2 0 0 1 24.6 13.2"
+    fill="none"
+    stroke="url(#rim)"
+    stroke-width="1.55"
+    stroke-linecap="round"
+    opacity="0.95"
+  />
+  <path
+    d="M 8.2 20.1 A 9.2 9.2 0 0 1 23.5 14.4"
+    fill="none"
+    stroke="#6ec9db"
+    stroke-width="0.55"
+    stroke-linecap="round"
+    opacity="0.35"
+  />
+
+  <circle cx="16" cy="16" r="9.5" fill="url(#emberGlow)"/>
+  <circle cx="16" cy="16" r="4" fill="url(#emberCore)" filter="url(#soft)"/>
+  <circle cx="15.2" cy="15" r="1.25" fill="#ffe0d4" opacity="0.55"/>
+</svg>

frontend/src/api/admin.ts

@@ -4,6 +4,7 @@ import type {
   AdminUser,
   AlertConfig,
   AlertTestResult,
+  FundamentalsCutoverConfig,
   PipelineReadiness,
   RecommendationConfig,
   ScheduleConfig,
@@ -56,9 +57,15 @@ export function updateSetting(key: string, value: string) {
     .then((r) => r.data);
 }
 
-export function updateRegistration(enabled: boolean) {
+export function getFundamentalsCutoverSettings() {
   return apiClient
-    .put<{ message: string }>('admin/settings/registration', { enabled })
+    .get<FundamentalsCutoverConfig>('admin/settings/fundamentals-cutover')
+    .then((r) => r.data);
+}
+
+export function updateFundamentalsCutoverSettings(enabled: boolean) {
+  return apiClient
+    .put<FundamentalsCutoverConfig>('admin/settings/fundamentals-cutover', { enabled })
     .then((r) => r.data);
 }
 
@@ -98,6 +105,41 @@ export function updateScheduleSettings(payload: Partial<ScheduleConfig>) {
     .then((r) => r.data);
 }
 
+export interface PerformanceConfig {
+  start_date: string;
+}
+
+export function getPerformanceSettings() {
+  return apiClient
+    .get<PerformanceConfig>('admin/settings/performance')
+    .then((r) => r.data);
+}
+
+export function updatePerformanceSettings(payload: Partial<PerformanceConfig>) {
+  return apiClient
+    .put<PerformanceConfig>('admin/settings/performance', payload)
+    .then((r) => r.data);
+}
+
+export interface ShadowBookConfig {
+  enabled: boolean;
+  capacity: number;
+  risk_pct: number;
+  start_equity: number;
+}
+
+export function getShadowBookSettings() {
+  return apiClient
+    .get<ShadowBookConfig>('admin/settings/shadow-book')
+    .then((r) => r.data);
+}
+
+export function updateShadowBookSettings(payload: Partial<ShadowBookConfig>) {
+  return apiClient
+    .put<ShadowBookConfig>('admin/settings/shadow-book', payload)
+    .then((r) => r.data);
+}
+
 export function getSentimentSettings() {
   return apiClient
     .get<SentimentProviderConfig>('admin/settings/sentiment')
@@ -204,6 +246,40 @@ export interface TriggerJobResponse {
   cadence?: BacktestCadence;
 }
 
+export interface ParityFieldStats {
+  legacy_available: number;
+  candidate_available: number;
+  both_available: number;
+  material_differences: number;
+  median_absolute_delta: number | null;
+  p95_absolute_delta: number | null;
+  max_absolute_delta: number | null;
+}
+
+export interface FundamentalsParityReport {
+  report_version: number;
+  generated_at: string;
+  as_of_date: string;
+  approval_status: string;
+  read_only: boolean;
+  summary: {
+    universe_count: number;
+    legacy_fundamental_score_available: number;
+    candidate_fundamental_score_available: number;
+    fundamental_scores_compared: number;
+    fundamental_score_material_changes: number;
+    fundamental_rank_changes: number;
+    field_stats: Record<string, ParityFieldStats>;
+  };
+  source_runs: Record<string, {
+    run_id: number;
+    status: string;
+    revision: string | null;
+    source_max_date: string | null;
+    completed_at: string | null;
+  } | null>;
+}
+
 export type BacktestTargetModel = 'production_gtl' | 'structural_sr';
 export type BacktestCadence = 'weekly' | 'daily';
 
@@ -230,6 +306,24 @@ export function triggerJob(
     .then((r) => r.data);
 }
 
+export function getFundamentalsParityReport() {
+  return apiClient
+    .get<FundamentalsParityReport | null>('admin/fundamentals-parity')
+    .then((r) => r.data);
+}
+
+export function getFundamentalsParityCsv() {
+  return apiClient
+    .get<{ filename: string; content: string } | null>('admin/fundamentals-parity/csv')
+    .then((r) => r.data);
+}
+
+export function getFundamentalsParityJson() {
+  return apiClient
+    .get<{ filename: string; content: string } | null>('admin/fundamentals-parity/json')
+    .then((r) => r.data);
+}
+
 // System events (operational warnings / errors)
 export interface SystemEvent {
   id: number;
@@ -275,9 +369,18 @@ export function acknowledgeSystemEvents(days = 7) {
 }
 
 // Data cleanup
+export interface CleanupResult {
+  ohlcv: number;
+  sentiment: number;
+  fundamentals: number;
+  sr_refresh_ok: number;
+  sr_refresh_failed: number;
+  sr_refresh_failures: { symbol: string; error: string }[];
+}
+
 export function cleanupData(olderThanDays: number) {
   return apiClient
-    .post<{ message: string }>('admin/data/cleanup', {
+    .post<CleanupResult>('admin/data/cleanup', {
       older_than_days: olderThanDays,
     })
     .then((r) => r.data);

frontend/src/api/paperTrades.ts

@@ -38,6 +38,39 @@ export function getEquityCurve() {
   return apiClient.get<EquityPoint[]>('paper-trades/equity-curve').then((r) => r.data);
 }
 
+export interface PerfPoint {
+  date: string;
+  manual_pnl: number;
+  shadow_pnl: number;
+  spy_pct: number;
+}
+
+export interface BookStats {
+  trades: number;
+  closed: number;
+  open: number;
+  win_rate: number | null;
+  total_r: number;
+  avg_r: number | null;
+  pnl: number;
+}
+
+export interface PerformanceSummary {
+  start_date: string | null;
+  series: PerfPoint[];
+  stats: {
+    manual?: BookStats;
+    shadow?: BookStats;
+    spy?: { pct: number };
+  };
+}
+
+export function getPerformance() {
+  return apiClient
+    .get<PerformanceSummary>('paper-trades/performance')
+    .then((r) => r.data);
+}
+
 export function closePaperTrade(id: number, closePrice?: number) {
   return apiClient
     .post<{ id: number; status: string }>(`paper-trades/${id}/close`, {

frontend/src/api/trades.ts

@@ -14,7 +14,3 @@ export function list(params?: TradeListParams) {
 export function bySymbol(symbol: string) {
   return apiClient.get<TradeSetup[]>(`trades/${symbol.toUpperCase()}`).then((r) => r.data);
 }
-
-export function history(symbol: string) {
-  return apiClient.get<TradeSetup[]>(`trades/${symbol.toUpperCase()}/history`).then((r) => r.data);
-}

frontend/src/components/admin/ActivationSettings.tsx

@@ -3,10 +3,11 @@ import type { ActivationConfig } from '../../lib/types';
 import { useActivationSettings, useUpdateActivationSettings } from '../../hooks/useAdmin';
 import { SkeletonTable } from '../ui/Skeleton';
 
+/** Mirrors app.services.admin_service.ACTIVATION_DEFAULTS — keep in sync. */
 const DEFAULTS: ActivationConfig = {
   min_momentum_percentile: 80,
-  min_rr: 1.2,
-  min_confidence: 55,
+  min_rr: 2.0,
+  min_confidence: 0,
   require_high_conviction: false,
   exclude_conflicts: false,
   exclude_neutral: true,

frontend/src/components/admin/FundamentalsCutoverSettings.tsx

@@ -0,0 +1,176 @@
+import {
+  useFundamentalsCutoverSettings,
+  useJobs,
+  useTriggerJob,
+  useUpdateFundamentalsCutoverSettings,
+} from '../../hooks/useAdmin';
+import { SkeletonCard } from '../ui/Skeleton';
+
+const SEC_JOB = 'sec_fundamentals_import';
+
+function formatRun(iso: string | null | undefined): string {
+  if (!iso) return 'not run in this process';
+  const minutes = Math.floor((Date.now() - new Date(iso).getTime()) / 60_000);
+  if (minutes < 1) return 'just now';
+  if (minutes < 60) return `${minutes}m ago`;
+  const hours = Math.floor(minutes / 60);
+  return hours < 24 ? `${hours}h ago` : `${Math.floor(hours / 24)}d ago`;
+}
+
+export function FundamentalsCutoverSettings() {
+  const cutover = useFundamentalsCutoverSettings();
+  const update = useUpdateFundamentalsCutoverSettings();
+  const trigger = useTriggerJob();
+  const { data: jobs } = useJobs();
+
+  if (cutover.isLoading) return <SkeletonCard />;
+  if (cutover.isError || !cutover.data) {
+    return (
+      <p className="text-sm text-red-400">
+        {(cutover.error as Error)?.message || 'Failed to load fundamentals data source'}
+      </p>
+    );
+  }
+
+  const enabled = cutover.data.enabled;
+  const secJob = jobs?.find((job) => job.name === SEC_JOB);
+  const runningJob = jobs?.find((job) => job.running);
+  const refreshBlocked = Boolean(runningJob && runningJob.name !== SEC_JOB);
+
+  const changeSource = () => {
+    const next = !enabled;
+    const confirmed = window.confirm(
+      next
+        ? 'Activate SEC + Dolt fundamentals? The next SEC import will replace the legacy cache and mark affected scores stale.'
+        : 'Pause SEC + Dolt cache refreshes? Existing cache values will stay in place; legacy values are not restored automatically.',
+    );
+    if (confirmed) update.mutate(next);
+  };
+
+  return (
+    <section className="glass overflow-hidden" aria-labelledby="fundamentals-source-title">
+      <div className={`h-0.5 ${enabled ? 'bg-gradient-to-r from-sky-500 via-cyan-300 to-emerald-400' : 'bg-white/[0.06]'}`} />
+      <div className="space-y-5 p-5">
+        <div className="flex flex-wrap items-start justify-between gap-3">
+          <div>
+            <div className="flex items-center gap-2">
+              <h3 id="fundamentals-source-title" className="text-sm font-semibold text-gray-200">
+                Fundamentals data source
+              </h3>
+              <span
+                className={`rounded-full border px-2 py-0.5 text-[10px] font-semibold uppercase tracking-[0.14em] ${
+                  enabled
+                    ? 'border-cyan-400/25 bg-cyan-400/10 text-cyan-300'
+                    : 'border-white/10 bg-white/[0.04] text-gray-500'
+                }`}
+              >
+                {enabled ? 'SEC + Dolt active' : 'Legacy cache'}
+              </span>
+            </div>
+            <p className="mt-1 max-w-3xl text-xs leading-relaxed text-gray-500">
+              Controls what repopulates <span className="num text-gray-400">fundamental_data</span>, the
+              compatibility cache used by scoring. SEC filings supply P/E, growth and estimated market
+              cap; Dolt supplies earnings dates and surprises. Everything is derived locally from PostgreSQL.
+            </p>
+          </div>
+        </div>
+
+        <div className="grid grid-cols-[minmax(0,1fr)_5rem_minmax(0,1fr)] items-center gap-3 rounded-xl border border-white/[0.06] bg-black/10 px-4 py-3">
+          <div className={enabled ? 'text-gray-600' : 'text-amber-200/90'}>
+            <div className="num text-[10px] uppercase tracking-[0.16em]">Legacy APIs</div>
+            <div className="mt-0.5 text-[11px]">FMP / Finnhub / Alpha Vantage</div>
+          </div>
+          <div className="relative h-px bg-white/10" aria-hidden="true">
+            <span
+              className={`absolute top-1/2 h-2.5 w-2.5 -translate-y-1/2 rounded-full border-2 border-[#0e120f] transition-all duration-300 ${
+                enabled
+                  ? 'right-0 bg-cyan-300 shadow-[0_0_12px_rgba(103,232,249,0.55)]'
+                  : 'left-0 bg-amber-300'
+              }`}
+            />
+          </div>
+          <div className={`text-right ${enabled ? 'text-cyan-200' : 'text-gray-600'}`}>
+            <div className="num text-[10px] uppercase tracking-[0.16em]">SEC + Dolt</div>
+            <div className="mt-0.5 text-[11px]">Bulk imports → PostgreSQL cache</div>
+          </div>
+        </div>
+
+        <div className="grid gap-4 border-t border-white/[0.06] pt-4 md:grid-cols-2">
+          <div className="flex items-start justify-between gap-4 rounded-xl bg-white/[0.025] p-3.5">
+            <div>
+              <div className="num text-[10px] uppercase tracking-[0.14em] text-gray-600">1 · Source</div>
+              <div className="mt-1 text-sm text-gray-200">Use SEC + Dolt for scoring inputs</div>
+              <p className="mt-1 text-[11px] leading-relaxed text-gray-500">
+                While active, the weekly legacy collector is skipped so it cannot overwrite the new cache.
+              </p>
+            </div>
+            <button
+              type="button"
+              role="switch"
+              aria-checked={enabled}
+              aria-label="Use SEC and Dolt fundamentals"
+              onClick={changeSource}
+              disabled={update.isPending}
+              className={`relative mt-1 inline-flex h-6 w-11 shrink-0 rounded-full border-2 border-transparent transition-colors focus:outline-none focus:ring-2 focus:ring-cyan-400/70 focus:ring-offset-2 focus:ring-offset-[#0e120f] disabled:cursor-wait disabled:opacity-50 ${
+                enabled ? 'bg-gradient-to-r from-sky-500 to-cyan-400' : 'bg-white/10'
+              }`}
+            >
+              <span
+                className={`pointer-events-none inline-block h-5 w-5 rounded-full bg-white shadow transition-transform ${
+                  enabled ? 'translate-x-5' : 'translate-x-0'
+                }`}
+              />
+            </button>
+          </div>
+
+          <div className="rounded-xl bg-white/[0.025] p-3.5">
+            <div className="num text-[10px] uppercase tracking-[0.14em] text-gray-600">2 · Refresh</div>
+            <div className="mt-1 flex flex-wrap items-center justify-between gap-3">
+              <div>
+                <div className="text-sm text-gray-200">Apply the source now</div>
+                <p className="mt-1 text-[11px] text-gray-500">
+                  {secJob?.running
+                    ? 'SEC import and cache refresh are running.'
+                    : secJob?.runtime_message || `Last SEC run: ${formatRun(secJob?.runtime_finished_at)}`}
+                </p>
+              </div>
+              <button
+                type="button"
+                onClick={() => trigger.mutate(SEC_JOB)}
+                disabled={
+                  !enabled ||
+                  trigger.isPending ||
+                  Boolean(secJob?.running) ||
+                  refreshBlocked ||
+                  secJob?.enabled === false
+                }
+                className="btn-primary px-3 py-2 text-xs disabled:cursor-not-allowed disabled:opacity-40"
+              >
+                <span>
+                  {secJob?.running
+                    ? 'Refreshing…'
+                    : trigger.isPending
+                      ? 'Starting…'
+                      : refreshBlocked
+                        ? 'Another job is running'
+                        : 'Run refresh now'}
+                </span>
+              </button>
+            </div>
+            {!enabled && (
+              <p className="mt-2 text-[11px] text-amber-300/70">Activate the source before running the refresh.</p>
+            )}
+            {enabled && secJob?.enabled === false && (
+              <p className="mt-2 text-[11px] text-amber-300/70">Enable the SEC Fundamentals job on the Jobs tab first.</p>
+            )}
+          </div>
+        </div>
+
+        <p className="text-[11px] leading-relaxed text-gray-600">
+          Rollback pauses future writes only. To restore pre-cutover values, use the database backup or
+          pause this source and manually run the legacy collector while its provider keys remain installed.
+        </p>
+      </div>
+    </section>
+  );
+}

frontend/src/components/admin/FundamentalsParityPanel.tsx

@@ -0,0 +1,156 @@
+import { useState } from 'react';
+import {
+  getFundamentalsParityCsv,
+  getFundamentalsParityJson,
+} from '../../api/admin';
+import { useFundamentalsParityReport } from '../../hooks/useAdmin';
+import { SkeletonTable } from '../ui/Skeleton';
+
+const FIELD_LABELS: Record<string, string> = {
+  pe_ratio: 'P/E',
+  revenue_growth: 'Revenue growth',
+  earnings_surprise: 'Earnings surprise',
+};
+
+function downloadText(filename: string, content: string, type: string) {
+  const blob = new Blob([content], { type });
+  const url = URL.createObjectURL(blob);
+  const anchor = document.createElement('a');
+  anchor.href = url;
+  anchor.download = filename;
+  anchor.click();
+  URL.revokeObjectURL(url);
+}
+
+export function FundamentalsParityPanel() {
+  const { data: report, isLoading, isError, error } = useFundamentalsParityReport();
+  const [downloading, setDownloading] = useState(false);
+
+  if (isLoading) return <SkeletonTable rows={2} cols={4} />;
+  if (isError) {
+    return <p className="text-sm text-red-400">{(error as Error).message}</p>;
+  }
+
+  if (!report) {
+    return (
+      <div className="glass p-5">
+        <h3 className="text-sm font-semibold text-gray-200">A5 Fundamentals Parity</h3>
+        <p className="mt-1 text-xs text-gray-500">
+          No report yet. Trigger “Fundamentals Parity Report (read-only)” below.
+        </p>
+      </div>
+    );
+  }
+
+  const summary = report.summary;
+  const generated = new Date(report.generated_at).toLocaleString();
+
+  async function downloadCsv() {
+    setDownloading(true);
+    try {
+      const artifact = await getFundamentalsParityCsv();
+      if (artifact) downloadText(artifact.filename, artifact.content, 'text/csv;charset=utf-8');
+    } finally {
+      setDownloading(false);
+    }
+  }
+
+  async function downloadJson() {
+    setDownloading(true);
+    try {
+      const artifact = await getFundamentalsParityJson();
+      if (artifact) downloadText(artifact.filename, artifact.content, 'application/json;charset=utf-8');
+    } finally {
+      setDownloading(false);
+    }
+  }
+
+  return (
+    <div className="glass p-5 space-y-4">
+      <div className="flex flex-wrap items-start justify-between gap-3">
+        <div>
+          <div className="flex flex-wrap items-center gap-2">
+            <h3 className="text-sm font-semibold text-gray-200">A5 Fundamentals Parity</h3>
+            <span className="rounded-full border border-amber-400/20 bg-amber-400/10 px-2 py-0.5 text-[10px] uppercase tracking-wide text-amber-300">
+              approval pending
+            </span>
+            <span className="rounded-full border border-cyan-400/20 bg-cyan-400/10 px-2 py-0.5 text-[10px] uppercase tracking-wide text-cyan-300">
+              read-only
+            </span>
+          </div>
+          <p className="mt-1 text-xs text-gray-500">
+            Generated {generated} · as of {report.as_of_date} · {summary.universe_count} tracked tickers
+          </p>
+        </div>
+        <div className="flex gap-2">
+          <button
+            type="button"
+            className="rounded border border-white/10 px-3 py-1.5 text-xs text-gray-300 hover:text-white"
+            onClick={downloadJson}
+            disabled={downloading}
+          >
+            Download JSON
+          </button>
+          <button
+            type="button"
+            className="rounded border border-white/10 px-3 py-1.5 text-xs text-gray-300 hover:text-white disabled:opacity-50"
+            onClick={downloadCsv}
+            disabled={downloading}
+          >
+            {downloading ? 'Preparing…' : 'Download CSV'}
+          </button>
+        </div>
+      </div>
+
+      <div className="grid gap-3 sm:grid-cols-2 lg:grid-cols-4">
+        <Summary label="Candidate score coverage" value={`${summary.candidate_fundamental_score_available}/${summary.universe_count}`} />
+        <Summary label="Scores compared" value={summary.fundamental_scores_compared} />
+        <Summary label="Material score moves" value={summary.fundamental_score_material_changes} />
+        <Summary label="Fundamental rank moves" value={summary.fundamental_rank_changes} />
+      </div>
+
+      <div className="overflow-x-auto">
+        <table className="w-full text-left text-xs">
+          <thead className="text-[10px] uppercase tracking-wider text-gray-500">
+            <tr>
+              <th className="pb-2 pr-4 font-medium">Field</th>
+              <th className="pb-2 px-3 font-medium">Legacy</th>
+              <th className="pb-2 px-3 font-medium">Candidate</th>
+              <th className="pb-2 px-3 font-medium">Compared</th>
+              <th className="pb-2 px-3 font-medium">Material</th>
+              <th className="pb-2 pl-3 font-medium">Median |Δ|</th>
+            </tr>
+          </thead>
+          <tbody className="divide-y divide-white/[0.06] text-gray-300">
+            {Object.entries(summary.field_stats).map(([key, stats]) => (
+              <tr key={key}>
+                <td className="py-2.5 pr-4">{FIELD_LABELS[key] ?? key}</td>
+                <td className="py-2.5 px-3 num">{stats.legacy_available}</td>
+                <td className="py-2.5 px-3 num">{stats.candidate_available}</td>
+                <td className="py-2.5 px-3 num">{stats.both_available}</td>
+                <td className="py-2.5 px-3 num">{stats.material_differences}</td>
+                <td className="py-2.5 pl-3 num">
+                  {stats.median_absolute_delta == null ? 'n/a' : stats.median_absolute_delta.toFixed(2)}
+                </td>
+              </tr>
+            ))}
+          </tbody>
+        </table>
+      </div>
+
+      <p className="text-[11px] leading-relaxed text-gray-500">
+        Materiality bands highlight review candidates only. They do not approve a cutover or write fundamentals,
+        scores, rankings, or qualification state.
+      </p>
+    </div>
+  );
+}
+
+function Summary({ label, value }: { label: string; value: string | number }) {
+  return (
+    <div className="rounded-lg border border-white/[0.07] bg-white/[0.025] px-3 py-2.5">
+      <div className="text-[10px] uppercase tracking-wider text-gray-500">{label}</div>
+      <div className="mt-1 num text-lg text-gray-200">{value}</div>
+    </div>
+  );
+}

frontend/src/components/admin/JobControls.tsx

@@ -25,7 +25,7 @@ function formatAgo(iso: string | null | undefined): string {
 
 function lastRunColor(status: string | null | undefined): string {
   if (status === 'error') return 'text-red-300';
-  if (status === 'rate_limited') return 'text-amber-300';
+  if (status === 'rate_limited' || status === 'deferred') return 'text-amber-300';
   return 'text-gray-500';
 }
 
@@ -127,7 +127,7 @@ export function JobControls() {
                     className={`text-[11px] font-medium ${
                       job.running
                         ? 'text-blue-300'
-                        : job.runtime_status === 'rate_limited'
+                        : job.runtime_status === 'rate_limited' || job.runtime_status === 'deferred'
                           ? 'text-amber-300'
                           : job.runtime_status === 'error'
                             ? 'text-red-300'
@@ -140,6 +140,8 @@ export function JobControls() {
                       ? 'Running'
                       : job.runtime_status === 'rate_limited'
                         ? 'Paused (rate-limited)'
+                        : job.runtime_status === 'deferred'
+                          ? 'Deferred (retrying)'
                         : job.runtime_status === 'error'
                           ? 'Last run error'
                           : job.enabled

frontend/src/components/admin/PerformanceSettings.tsx

@@ -0,0 +1,193 @@
+import { useEffect, useState } from 'react';
+import { useMutation, useQuery, useQueryClient } from '@tanstack/react-query';
+import {
+  getPerformanceSettings,
+  getShadowBookSettings,
+  updatePerformanceSettings,
+  updateShadowBookSettings,
+  type ShadowBookConfig,
+} from '../../api/admin';
+import { SkeletonCard } from '../ui/Skeleton';
+
+/** Performance window + the auto-traded shadow book.
+ *
+ *  These belong together: the shadow book is what the comparison measures, and
+ *  the start date is what keeps the comparison inside a single strategy
+ *  configuration.
+ */
+export function PerformanceSettings() {
+  const qc = useQueryClient();
+  const window = useQuery({ queryKey: ['admin', 'performance'], queryFn: getPerformanceSettings });
+  const shadow = useQuery({ queryKey: ['admin', 'shadow-book'], queryFn: getShadowBookSettings });
+
+  const [startDate, setStartDate] = useState('');
+  const [book, setBook] = useState<ShadowBookConfig | null>(null);
+
+  useEffect(() => {
+    if (window.data) setStartDate(window.data.start_date ?? '');
+  }, [window.data]);
+  useEffect(() => {
+    if (shadow.data) setBook(shadow.data);
+  }, [shadow.data]);
+
+  const saveWindow = useMutation({
+    mutationFn: () => updatePerformanceSettings({ start_date: startDate }),
+    onSuccess: () => {
+      qc.invalidateQueries({ queryKey: ['admin', 'performance'] });
+      qc.invalidateQueries({ queryKey: ['paper-trades', 'performance'] });
+    },
+  });
+
+  const saveBook = useMutation({
+    mutationFn: (payload: Partial<ShadowBookConfig>) => updateShadowBookSettings(payload),
+    onSuccess: (data) => {
+      qc.setQueryData(['admin', 'shadow-book'], data);
+      setBook(data);
+      qc.invalidateQueries({ queryKey: ['admin', 'shadow-book'] });
+      qc.invalidateQueries({ queryKey: ['paper-trades', 'performance'] });
+    },
+  });
+
+  if (window.isLoading || shadow.isLoading || !book) return <SkeletonCard />;
+
+  // Staged edits: nothing about the shadow book persists until Save, matching
+  // the other admin panels — and making the live-trade toggle deliberate.
+  const bookDirty =
+    !!shadow.data &&
+    (book.enabled !== shadow.data.enabled ||
+      book.capacity !== shadow.data.capacity ||
+      book.risk_pct !== shadow.data.risk_pct ||
+      book.start_equity !== shadow.data.start_equity);
+
+  return (
+    <div className="glass space-y-5 p-5">
+      <div>
+        <h3 className="text-sm font-semibold text-gray-200">Performance &amp; Shadow Book</h3>
+        <p className="mt-1 text-xs leading-relaxed text-gray-500">
+          The <span className="text-gray-300">shadow book</span> trades the validated strategy with no
+          human input: top-ranked qualified setups up to capacity, sized to a fixed risk, entered right
+          after the near-close scan. It shares the paper exit policy with your own trades, so the only
+          difference between the two books is <span className="text-gray-300">which setups get taken</span>.
+        </p>
+      </div>
+
+      <label className="block space-y-1">
+        <span className="text-xs text-gray-400">Performance since</span>
+        <div className="flex gap-2">
+          <input
+            type="date"
+            value={startDate}
+            onChange={(e) => setStartDate(e.target.value)}
+            className="input-glass w-48 px-3 py-2 text-sm"
+          />
+          <button
+            type="button"
+            onClick={() => saveWindow.mutate()}
+            disabled={saveWindow.isPending}
+            className="btn-glass px-3 py-2 text-sm"
+          >
+            {saveWindow.isPending ? 'Saving…' : 'Save'}
+          </button>
+          {startDate && (
+            <button
+              type="button"
+              onClick={() => {
+                setStartDate('');
+                updatePerformanceSettings({ start_date: '' }).then(() => {
+                  qc.invalidateQueries({ queryKey: ['admin', 'performance'] });
+                  qc.invalidateQueries({ queryKey: ['paper-trades', 'performance'] });
+                });
+              }}
+              className="btn-glass px-3 py-2 text-sm text-gray-400"
+            >
+              Clear
+            </button>
+          )}
+        </div>
+        <span className="block text-[11px] leading-relaxed text-gray-500">
+          Trades opened before this date are excluded from the Performance card. The strategy has been
+          revised repeatedly — pinning a start keeps the comparison inside one configuration instead of
+          averaging across rules that no longer exist. Empty shows all history.
+        </span>
+      </label>
+
+      <div className="border-t border-white/5 pt-4">
+        <label className="flex items-start gap-3">
+          <input
+            type="checkbox"
+            checked={book.enabled}
+            disabled={saveBook.isPending}
+            onChange={(e) => setBook({ ...book, enabled: e.target.checked })}
+            className="mt-0.5 disabled:cursor-wait disabled:opacity-50"
+          />
+          <span>
+            <span className="text-sm text-gray-200">Shadow book enabled</span>
+            <span className="block text-[11px] leading-relaxed text-gray-500">
+              Starts opening real paper positions automatically on the next near-close scan — takes
+              effect when you Save. Verify its first selections match the top-ranked qualified setups
+              before trusting the curve.
+            </span>
+          </span>
+        </label>
+
+        <div className="mt-4 grid gap-4 md:grid-cols-3">
+          <label className="block space-y-1">
+            <span className="text-xs text-gray-400">Capacity (positions)</span>
+            <input
+              type="number"
+              min={1}
+              max={100}
+              value={book.capacity}
+              disabled={saveBook.isPending}
+              onChange={(e) => setBook({ ...book, capacity: Number(e.target.value) })}
+              className="input-glass w-full px-3 py-2 text-sm disabled:cursor-wait disabled:opacity-50"
+            />
+          </label>
+          <label className="block space-y-1">
+            <span className="text-xs text-gray-400">Risk per trade (%)</span>
+            <input
+              type="number"
+              step="0.05"
+              min={0.05}
+              max={10}
+              value={book.risk_pct}
+              disabled={saveBook.isPending}
+              onChange={(e) => setBook({ ...book, risk_pct: Number(e.target.value) })}
+              className="input-glass w-full px-3 py-2 text-sm disabled:cursor-wait disabled:opacity-50"
+            />
+          </label>
+          <label className="block space-y-1">
+            <span className="text-xs text-gray-400">Start equity ($)</span>
+            <input
+              type="number"
+              min={1000}
+              step={1000}
+              value={book.start_equity}
+              disabled={saveBook.isPending}
+              onChange={(e) => setBook({ ...book, start_equity: Number(e.target.value) })}
+              className="input-glass w-full px-3 py-2 text-sm disabled:cursor-wait disabled:opacity-50"
+            />
+          </label>
+        </div>
+        <p className="mt-2 text-[11px] leading-relaxed text-gray-500">
+          Defaults match the validated configuration: 10 positions, 1% fixed-fractional risk. Start
+          equity is only a sizing base — the books are compared in R-multiples, not currency.
+        </p>
+
+        <div className="mt-4 flex items-center gap-3">
+          <button
+            type="button"
+            className="btn-primary px-4 py-2 text-sm disabled:opacity-50"
+            disabled={!bookDirty || saveBook.isPending}
+            onClick={() => saveBook.mutate(book)}
+          >
+            {saveBook.isPending ? 'Saving…' : 'Save shadow book'}
+          </button>
+          {bookDirty && !saveBook.isPending && (
+            <span className="text-[11px] text-amber-400/80">Unsaved changes</span>
+          )}
+        </div>
+      </div>
+    </div>
+  );
+}

frontend/src/components/admin/ScheduleSettings.tsx

@@ -4,34 +4,69 @@ import { useScheduleSettings, useUpdateScheduleSettings } from '../../hooks/useA
 import { SkeletonTable } from '../ui/Skeleton';
 
 const DEFAULTS: ScheduleConfig = {
-  schedule_timezone: 'Europe/Berlin',
-  schedule_daily_pipeline_cron: '0 7 * * *',
-  schedule_intraday_pipeline_cron: '0 14-22 * * 1-5',
-  schedule_fundamentals_cron: '0 4 * * 1',
+  schedule_timezone: 'America/New_York',
+  schedule_daily_pipeline_cron: '0 2 * * *',
+  schedule_dolt_earnings_cron: '30 2 * * *',
+  schedule_sec_fundamentals_cron: '0 4 * * *',
+  schedule_fundamentals_parity_cron: '30 5 * * *',
+  schedule_near_close_pipeline_cron: '30 15 * * mon-fri',
+  schedule_after_close_pipeline_cron: '45 16 * * mon-fri',
+  schedule_intraday_pipeline_cron: '0 10-15 * * mon-fri',
+  schedule_fundamentals_cron: '0 1 * * mon',
 };
 
 const FIELDS: { key: keyof ScheduleConfig; label: string; hint: string; mono?: boolean }[] = [
   {
     key: 'schedule_timezone',
     label: 'Timezone',
-    hint: 'IANA name, e.g. Europe/Berlin. All times below are in this zone.',
+    hint: 'IANA name. Prefer America/New_York so the near-close scan tracks the US cash close through DST.',
   },
   {
     key: 'schedule_daily_pipeline_cron',
-    label: 'Daily pipeline (full)',
-    hint: 'OHLCV → sentiment → R:R scan → outcomes → regime. Default 07:00 so data is ready by 8.',
+    label: 'Morning pipeline',
+    hint: 'OHLCV → benchmark → sentiment → regime → alerts (no R:R scan). Default 02:00 ET so regime-quadrant changes hit Telegram in the morning.',
+    mono: true,
+  },
+  {
+    key: 'schedule_dolt_earnings_cron',
+    label: 'Dolt earnings',
+    hint: 'Pull and import earnings dates/results daily at 02:30 ET. The activated cache refresh uses these local events.',
+    mono: true,
+  },
+  {
+    key: 'schedule_sec_fundamentals_cron',
+    label: 'SEC fundamentals',
+    hint: 'Import tracked-universe SEC facts daily at 04:00 ET and refresh the scoring cache when the cutover is active.',
+    mono: true,
+  },
+  {
+    key: 'schedule_fundamentals_parity_cron',
+    label: 'Fundamentals parity report',
+    hint: 'Read-only legacy vs SEC/Dolt comparison daily at 05:30 ET, after the bulk imports.',
+    mono: true,
+  },
+  {
+    key: 'schedule_near_close_pipeline_cron',
+    label: 'Near-close pipeline (scan + alert)',
+    hint: 'OHLCV fetch → R:R scan → Telegram. Default 15:30 ET Mon–Fri so manual MOC fills can still hit ~15:50/15:55.',
+    mono: true,
+  },
+  {
+    key: 'schedule_after_close_pipeline_cron',
+    label: 'After-close pipeline (outcome)',
+    hint: 'OHLCV fetch (final bar) → outcome eval. Default 16:45 ET Mon–Fri — not chained to the partial near-close bar.',
     mono: true,
   },
   {
     key: 'schedule_intraday_pipeline_cron',
     label: 'Intraday pipeline (light)',
-    hint: 'Refresh prices + resolve outcomes. Default hourly across the US session, weekdays.',
+    hint: 'Refresh prices + resolve outcomes mid-session. Default hourly 10:00–15:00 ET weekdays.',
     mono: true,
   },
   {
     key: 'schedule_fundamentals_cron',
-    label: 'Fundamentals (weekly)',
-    hint: 'Slow, rate-limited. Default early Monday so it finishes well before the day starts.',
+    label: 'Legacy fundamentals (weekly)',
+    hint: 'Fallback provider chain. Automatically skipped while the SEC + Dolt cutover is active.',
     mono: true,
   },
 ];
@@ -43,7 +78,7 @@ export function ScheduleSettings() {
   const [form, setForm] = useState<ScheduleConfig>(DEFAULTS);
 
   useEffect(() => {
-    if (data) setForm(data);
+    if (data) setForm({ ...DEFAULTS, ...data });
   }, [data]);
 
   if (isLoading) return <SkeletonTable rows={2} cols={2} />;
@@ -55,8 +90,8 @@ export function ScheduleSettings() {
         <h3 className="text-sm font-semibold text-gray-200">Pipeline Schedule</h3>
         <p className="mt-1 text-xs text-gray-500">
           When the jobs run, as 5-field cron (<span className="num">min hour day month weekday</span>).
-          Saved changes apply to the running scheduler immediately — no redeploy. The big nightly run
-          does the full refresh; the light intraday run just keeps prices current.
+          Saved changes apply to the running scheduler immediately — no redeploy.
+          One qualifying R:R scan per day is the near-close job; alerts fire immediately after that scan.
         </p>
       </div>
 
@@ -66,7 +101,7 @@ export function ScheduleSettings() {
             <span className="text-xs text-gray-400">{f.label}</span>
             <input
               type="text"
-              value={form[f.key]}
+              value={form[f.key] ?? ''}
               spellCheck={false}
               onChange={(e) => setForm((prev) => ({ ...prev, [f.key]: e.target.value }))}
               className={`w-full input-glass px-3 py-2 text-sm ${f.mono ? 'num' : ''}`}