- The breakeven study grades at the shadow book's capacity AND risk per
trade, and labels each with where it came from. A missing setting row
now reads "shadow_book default" instead of crediting the setting; the
committed report said shadow_book_capacity though the snapshot has no
such row.
- _production_book_config reads strategies through the same
_portfolio_monitor_strategies() hook as the monitor.
- breakeven_gate_reset is rejected when there is no post-stop episode or
cooldown for it to start, and the study skips its diagnostic arm for a
book without one, so a production config change cannot abort the run.
- identical_to_control is renamed ties_control_on_graded_metrics and needs
real values: be_2r ties on every graded metric but still changed 2 trades.
- _BookConfig drops an unused field and hashes by identity; the min-R:R
sweep swaps its gate in with dataclasses.replace instead of passing it.
- New tests cover the parity guard's success path, the no-config fallback
and the gate-reset check.
The breakeven report is regenerated: every row, diagnostic row and the
parity check are identical; only the labels, the risk fields and the
renamed flag changed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Pre-registered test of raising the stop to entry once a trade is +1R or
+2R, on top of the 3x ATR trail. be_1r fails the frozen rule (train worse
on all three graded metrics, test drawdown worse); be_2r changes 2 of 480
trades because the trail already reaches entry near +2R. The production
exit stays.
Breakeven exits get their own breakeven_stop label. They follow live
re-entry semantics (only initial-stop exits lock re-entry), and an
ungraded be_1r_gated diagnostic arm sends them through the post-stop gate
reset instead. The trail and breakeven ratchet share one helper, so
breakeven also applies on a next_open fill bar. breakeven_r is validated,
verdicts flag passes that are only ties, and the graded capacity comes
from the shadow book config. Every graded number and the parity row are
unchanged from the first run.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CI failed on test_run_backtest_rolls_back_a_failed_ticker_fetch: it asserted
"degraded" and got "failed". The code was right and the test was wrong.
run_backtest calls refresh_benchmark_prices unless it is in offline snapshot
mode, so whether SPY closes exist depends on network access and credentials.
That decides the status, because a missing benchmark means residual 12-1
momentum silently ran as raw 12-1 — a run that changed its own signal
definition, which the ledger reports as failed. A dev box reaches Alpaca and
sees "degraded"; CI does not and correctly sees "failed".
Any test asserting on status now pins the benchmark instead of inheriting
whatever the machine can reach. The every-ticker-fails test is pinned too, so
its "failed" verdict is attributable to the fetch failures rather than passing
for the wrong reason on a benchmark-less box, and the healthy-completeness test
can now assert "ok" outright.
Reproduced with BACKTEST_SNAPSHOT_OFFLINE=1, which skips the refresh and leaves
the benchmark empty exactly as CI has it: the old test fails there, the new one
passes. Full suite verified green both with and without a reachable benchmark.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HAH7vVtn7qbcn1EVheecHQ
A snapshot whose schema predated a migration made every _fetch_columns call
raise, both dispatch paths caught the per-ticker exception and continued, and
the run wrote a report with zero candidates that was indistinguishable from an
honest run that found no setups. Nothing said the run had lost its universe.
run_backtest now builds a completeness ledger and emits it as
report["data_quality"]: expected / loaded / evaluated counts, per-symbol
failures with stage and reason, bar-less symbols named, and benchmark
availability. Every universe symbol lands in exactly one terminal bucket and
the arithmetic is checked, so a counter that stops being maintained reports
itself as a failure rather than under-counting quietly.
Benchmark availability is a separate axis: without SPY, residual 12-1 momentum
silently degrades to raw 12-1, and the report now says so instead of leaving it
in a log line.
Three statuses, because severity differs. "failed" (nothing evaluated, empty
universe, unbalanced ledger, missing benchmark) replaces the performance tiles
with the callout rather than warning above them — a banner over ordinary-looking
numbers is exactly how the zero-candidate run would still have read as a quiet
week. "degraded" keeps the tiles with a note. A report with no data_quality
block reads as unknown, not clean; defaulting old reports to complete would
reproduce the bug being fixed.
The same gate applies to the two other consumers of backtest numbers:
productionBaseRates returns null for a failed run, and EvaluationPanel will not
report drift against a broken baseline.
"complete" means the processing and coverage checks passed. It does not mean
the prices are correct — a complete run over silently wrong bars is still
complete, and the card says so.
Also threads an explicit terminal-events option through the replay, validated
before dispatch so a stale events file aborts the run instead of quietly
degrading it. Verified against a healthy snapshot (degraded, 505/506) and one
with tickers.cik dropped (failed, 0/506, 506 fetch failures).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HAH7vVtn7qbcn1EVheecHQ
Every fix so far only applied to reports generated after deploy. The cached
report is served verbatim, so it keeps the recommendation the OLD build stored —
quoting the legacy policy book, naming a rejected exit as "recommended", and
carrying no basis_lookback, which let the lookback selector default to 3y and
put 3-year tiles beside an all-history recommendation with no divergence notice.
Exactly the contradiction the last three commits set out to remove, silently
present on the first page load after deploy and until the next scheduled run
overwrote it.
The recommendation is a pure function of the numbers already in the report — its
own note says it is derived from them on every run — so it is now re-derived on
read. A corrected recommendation appears immediately instead of after the next
backtest. On failure it is dropped rather than falling back to the stored one,
which is the stale derivation this replaces.
The test drives the real shape: an old-build report with a legacy recommendation
written straight to the settings row, read back through get_backtest_report.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two ways the recommendation still disagreed with the page it sits on.
It preferred the "all" monitor row while the UI defaulted its selector to "3y",
so a default page load showed one set of returns in the tiles and a different
set in the recommendation. The row it used is now published as basis_lookback
and the page defaults to it, so the two cannot open on different windows. The
test fixture gains a second monitor row with different numbers — with only an
"all" row present, a lookback mix-up could not fail.
Robustness picked its basis between "the recommended Nd hold" and "the S/R
target exit", naming an exit the production book replaced as recommended. There
is no ATR-trail ex-top-5% figure in the report, so it now always reports the
gate-level grading and says that is what it is, rather than dressing a legacy
number as a verdict on the production book. time_exit_sweep is no longer read
here at all.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The recommendation's "Book vs SPY" line read from portfolio_sim — the hold/target
policy book — while the tiles directly above read from portfolio_monitor, the
production ATR-trail book. Same SPY figure, different portfolio return, on one
screen. It now reads the same production row the tiles do.
Also removed, for the same reason: the "legacy exit diagnostic" comparing hold
against the S/R target. Both are exits the production book replaced, so a
recommendation between them could not lead to an action. And the fallback
headline, which advised the fixed-hold exit whenever a report had no production
row — a report that cannot describe the production baseline now states none.
portfolio_sim stays in the report payload: scripts/run_backtest_snapshot.py and
reports/compare_reports.py read it, and it is no longer surfaced in the UI. The
test fixture now carries a production monitor whose numbers differ from its
policy sim, so re-sourcing that line from the old place fails rather than passes
unnoticed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three portfolio metrics computed where their inputs already live in
_simulate_portfolio: Sortino off the existing daily return series, Gain-to-Pain
off a monthly aggregation of the equity curve, profit factor off closed-trade
dollar P&L.
Gain-to-Pain follows Schwager — sum of ALL monthly returns over the absolute
sum of the negative ones. The profit-factor-shaped variant,
sum(positive)/|sum(negative)|, sits exactly 1.0 higher for every input since
sum(all) = sum(pos) - |sum(neg)|; the test asserts against both so the wrong one
cannot pass. Sortino divides by len(rets), the full-sample lower partial moment,
not by the count of down days, which would shrink the denominator and inflate
the ratio.
No MAR field: calmar is already CAGR / max drawdown, the same number under the
other name (docs/research/effective-risk-floor-ab.md).
All three keys are emitted unconditionally even when None — the UI reads an
absent key as "report predates these metrics", so presence is a contract.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The first pass guarded the replay loop but not the portfolio-simulation block,
which re-fetches price columns and loads the benchmark and the live exit policy
from the same session much later. A failure in any of those swallows the
exception without clearing the transaction — the identical failure mode, with
the identical symptom: the report write is the first unguarded statement and
takes the blame.
The outer handler is the backstop for the price_columns loop, which has no
handler of its own; rolling back a session an inner handler already cleared is
a no-op.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every DB call in run_backtest is best-effort so one unreadable ticker cannot
abort the whole replay, but the handlers swallowed the exception without
clearing the transaction. asyncpg then reports "current transaction is
aborted" for every later statement, and the first unguarded one — the report
write — surfaced it as the job error, long after the real cause.
Add _rollback_quietly at the three swallowing sites (benchmark load, parallel
fetch, sequential replay), matching the guard price_service already uses.
Load plain symbols instead of Ticker instances: a rollback expires ORM objects
held across it, and touching an expired attribute afterwards triggers sync
lazy-loading, which raises on an AsyncSession. rr_scanner_service hit this
same trap. Only .symbol was ever used.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Add research-only snapshot extender, PIT dollar-volume mask for signal IC,
rank-only harness path, fingerprint+breadth runner, and docs. Fingerprint
reproduced IC -0.045 / t -2.91 on prod.sqlite. No production gate/schedule changes.
Document Phase A (max-hold/vol/corr closed; next-open as decision baseline).
Add stale_close and next_open gap-cap fill modes plus a small matrix to test
whether near-close scheduling recovers overnight momentum drift.
Ship shared Sharpe SE/PSR diagnostics, next-open fill and equity-curve vol targeting in the portfolio simulator, re-derived fip_id, and a checkpointed offline matrix runner for Mac-side validation sweeps.
Backtest report now includes research-only hold-to-horizon portfolio variants comparing raw vs residual 12-1 momentum, cutoff 80 vs 90, max 10 vs 15 positions, and SPY-200 risk scaling. A dynamic research recommendation panel flags residual momentum, cutoff 90, or regime scaling only when transparent promotion rules pass.
Adds signal_context_snapshots with migration 016 and captures one point-in-time context row per newly generated TradeSetup: setup fields, composite/dimensions, latest sentiment, latest fundamentals, and strategy_version=momentum_12_1_rr_time_v1. This is forward-only; no historical sentiment/fundamental backfill is attempted.
No live gate, paper-trade exit, or production ranking behavior changes.
Verification: 458 backend tests pass, ruff check app/ clean, frontend npm run build clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds a research-only 12-1 residual momentum signal to the cross-sectional signal-evaluation harness. The signal estimates benchmark beta over the 12-1 formation window and ranks cumulative stock return minus beta-adjusted benchmark return; it only appears when benchmark closes are available.
No production qualification behavior changes. The Backtest signal table labels the new row as 12-1 residual momentum. Tests cover benchmark-gated emission and beta removal while keeping stock-specific drift.
Verification: 453 backend tests pass, ruff check app/ clean, frontend npm run build clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The robustness warning was computed on the target-model distribution
while the same panel recommends the hold exit — internally inconsistent.
_robustness_stats (median, profit factor, ex-top-5% expectancy) is now
shared by _bucket_stats and _time_exit_bucket, the time-exit table shows
Median Net R and Ex-Top-5% per hold length, and _build_recommendation
reads the trimmed expectancy from the recommended exit's bucket (falling
back to the target model when no hold is recommended).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Robustness (answers 'is the edge just outliers?'):
- _bucket_stats gains median_net_r, profit_factor, and net_avg_r_ex_top5
(expectancy with the top 5% of winners removed); shown as stat tiles.
- Portfolio sim gains per-calendar-year returns, shown in the sim table.
Dynamic recommendation ('What this backtest recommends' panel):
- _build_recommendation derives advice from the report's own numbers on
every run — exit policy (target vs best hold, with sim CAGRs), which
gate floors earn their keep (ablation Hold column), best momentum
cutoff, book-vs-SPY verdict, and an outlier-dependence warning when
the trimmed expectancy goes non-positive.
Retired (conclusions reached, tables removed from report + UI):
- Take-profit sweep (no interior optimum — fixed TP is the wrong tool
for momentum), trailing sweep (converged to the hold-to-horizon exit),
probability calibration (model is display-only by decision).
- _tp_primitives slimmed to _risk_and_stop_day; trailing machinery gone.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Per-trade additions to the report:
- Gap-through-stop fills: stops now fill at the worse of the stop or the
bar's open across every exit model (target, TP, trailing, time), so a
loss can exceed -1R; targets never fill better than their level.
- best_r / worst_r, avg holding days, and net R per day of capital
deployed on the summary buckets and the time-exit sweep.
Portfolio simulation (the stats a per-setup replay cannot give):
- One capital-constrained book over the qualified setups: 10k start, max
10 concurrent positions (one per ticker, best momentum first), 1%
fixed-fractional risk with a 20% no-leverage notional cap, entries at
the detection close, 0.1%/side costs, daily mark-to-market.
- Two exit policies compared: S/R target race vs hold-to-horizon.
- Equity-curve stats: final equity, total return, CAGR, max drawdown,
annualized daily Sharpe, win rate, avg P&L, best/worst trade, avg
hold, entries skipped on a full book, and SPY price return over the
same window (benchmark history refreshed to cover the replay span).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The ablation judged floors under the target/stop model, but the exit
sweeps point at replacing that exit with a fixed hold — under which the
R:R floor's rationale (bigger payoff at the target) may not apply. Each
ablation row now also carries hold_avg_r / hold_net_avg_r / hold_total_r
(30d hold, initial stop only), so the Phase 3 gate decision can be read
under the exit policy that would actually be used.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
_window_setups computed them but _replay_ticker dropped them, so the
ablation's NEUTRAL/tightener checks saw None for every candidate and the
'without confidence floor' / 'without R:R floor' rows collapsed to 0
setups (impossible — removing a floor can only add setups). Regression
test now goes through the real _replay_ticker path.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Phase 1 of the strategy-measurement plan — report-only, no production
trading behavior changes:
- Cost haircut: every bucket/sweep now reports net_avg_r/net_total_r
alongside gross (COST_PER_SIDE=0.1% of notional, converted to R via
each setup's stop distance); params carry cost_per_side_pct.
- Gate ablation table: re-qualifies candidates at the current momentum
cutoff with one floor removed per row (confidence / R:R / NEUTRAL /
momentum-only) to show which floors earn their keep.
- Time-based exit sweep: hold 5/10/21/30 days with the initial ATR stop,
exit at the day-N close — the classic momentum implementation, to
disambiguate the wide-trailing result.
- TP sweep extended to +40/+50%, trailing to 25/30% so the optima are
interior instead of starred at the sweep edge.
- BacktestPanel: Net Avg R columns everywhere, gate-ablation and
time-exit tables, stars now mark best net avg R; stale cached reports
still render (all new fields optional/guarded).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Third exit model alongside target-vs-stop and the fixed take-profit. The TP sweep
showed the edge lives in the fat tail (avg R keeps rising as you let winners run),
but a fixed wide target is win-rate-brutal and gives everything back on a reversal.
A trailing stop harvests the tail while protecting gains.
Per setup the replay computes the realized R for several trail widths (3/5/7/10/
15/20%) in a single conservative pass — stop ratchets up via max(initial_stop,
peak*(1-trail)), exit on the pullback or at the horizon close, R vs the initial
risk. Aggregated into a trailing sweep (win rate = share closed in profit, avg R,
total R) over the qualified set and shown as a new table in the Backtest panel.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The target-vs-stop model counts a near-miss of a far S/R target as a full loss
and ignores the partial gains you actually bank — so it measures a different
strategy than "scalp the early pop, take +8%". Add a realistic take-profit exit
model next to it (original untouched).
Per setup the replay now also records risk%, whether the stop was hit, the
favourable excursion reachable before the stop (MFE), and the horizon-close move.
From those a fixed-take-profit sweep (4/6/8/10/12/15%) is scored in R: bank +X%
if reached before the stop, else -1R, else the horizon close. Hit rate = how
often +X% was banked (the MFE CDF), so you can pick the EV-optimal TP without
top-ticking fantasy. Shown as a new table in the Backtest panel; the IC,
calibration and momentum sweep are unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
dennisthiessenandClaude Opus 5.5