feat(backtest): make an incomplete run visibly incomplete
A snapshot whose schema predated a migration made every _fetch_columns call raise, both dispatch paths caught the per-ticker exception and continued, and the run wrote a report with zero candidates that was indistinguishable from an honest run that found no setups. Nothing said the run had lost its universe. run_backtest now builds a completeness ledger and emits it as report["data_quality"]: expected / loaded / evaluated counts, per-symbol failures with stage and reason, bar-less symbols named, and benchmark availability. Every universe symbol lands in exactly one terminal bucket and the arithmetic is checked, so a counter that stops being maintained reports itself as a failure rather than under-counting quietly. Benchmark availability is a separate axis: without SPY, residual 12-1 momentum silently degrades to raw 12-1, and the report now says so instead of leaving it in a log line. Three statuses, because severity differs. "failed" (nothing evaluated, empty universe, unbalanced ledger, missing benchmark) replaces the performance tiles with the callout rather than warning above them — a banner over ordinary-looking numbers is exactly how the zero-candidate run would still have read as a quiet week. "degraded" keeps the tiles with a note. A report with no data_quality block reads as unknown, not clean; defaulting old reports to complete would reproduce the bug being fixed. The same gate applies to the two other consumers of backtest numbers: productionBaseRates returns null for a failed run, and EvaluationPanel will not report drift against a broken baseline. "complete" means the processing and coverage checks passed. It does not mean the prices are correct — a complete run over silently wrong bars is still complete, and the card says so. Also threads an explicit terminal-events option through the replay, validated before dispatch so a stale events file aborts the run instead of quietly degrading it. Verified against a healthy snapshot (degraded, 505/506) and one with tickers.cik dropped (failed, 0/506, 506 fetch failures). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HAH7vVtn7qbcn1EVheecHQ
This commit is contained in:
1 parent
ebe9047403
commit
f2ef8e5438
11 files changed
+1013
-49
No files matched your search
@@ -1689,6 +1689,74 @@ async def test_run_backtest_rolls_back_a_failed_ticker_fetch(session, monkeypatc
|
||||
assert report["tickers"] == 2
|
||||
assert report["candidates"] >= 1
|
||||
|
||||
# ...and the report says one ticker was lost rather than presenting the
|
||||
# remaining one as the whole universe.
|
||||
quality = report["data_quality"]
|
||||
assert quality["expected"] == 2
|
||||
assert quality["evaluated"] == 1
|
||||
assert quality["failed"] == 1
|
||||
assert quality["failures"][0]["symbol"] == "AAA"
|
||||
assert quality["failures"][0]["stage"] == "fetch"
|
||||
assert "simulated OHLCV read failure" in quality["failures"][0]["error"]
|
||||
assert quality["complete"] is False
|
||||
assert quality["status"] == "degraded"
|
||||
|
||||
|
||||
async def test_run_backtest_reports_its_own_completeness(session):
|
||||
"""The healthy case: every ticker accounted for, so the report says so."""
|
||||
await _seed_oscillating_ticker(session, "OSC")
|
||||
report = await bt.run_backtest(session)
|
||||
|
||||
quality = report["data_quality"]
|
||||
assert quality["expected"] == report["tickers"] == 1
|
||||
assert quality["evaluated"] == 1
|
||||
assert quality["failed"] == 0
|
||||
assert quality["no_data"] == 0
|
||||
# Ledger identity — the three terminal buckets must cover the universe.
|
||||
assert quality["no_data"] + quality["failed"] + quality["evaluated"] == quality["expected"]
|
||||
|
||||
|
||||
async def test_run_backtest_marks_a_run_failed_when_every_ticker_fails(session, monkeypatch):
|
||||
"""The demonstrated failure: an unsynced snapshot raises on every read, the
|
||||
replay loop swallows each one, and the run writes a report with no
|
||||
candidates. It must come back "failed", not look like a quiet run.
|
||||
"""
|
||||
await _seed_oscillating_ticker(session, "AAA")
|
||||
await _seed_oscillating_ticker(session, "OSC")
|
||||
|
||||
async def always_failing_fetch(db, symbol):
|
||||
raise RuntimeError("no such column: tickers.cik")
|
||||
|
||||
monkeypatch.setattr(bt, "_fetch_columns", always_failing_fetch)
|
||||
|
||||
report = await bt.run_backtest(session)
|
||||
|
||||
assert report["candidates"] == 0
|
||||
quality = report["data_quality"]
|
||||
assert quality["status"] == "failed"
|
||||
assert quality["evaluated"] == 0
|
||||
assert quality["loaded"] == 0
|
||||
assert quality["failed"] == 2
|
||||
assert "no symbol was evaluated successfully" in quality["reasons"]
|
||||
|
||||
|
||||
async def test_run_backtest_flags_a_missing_benchmark(session, monkeypatch):
|
||||
"""Residual momentum degrades to raw without SPY. The numbers then describe
|
||||
a different gate than the one the report claims, so the run is not clean."""
|
||||
await _seed_oscillating_ticker(session, "OSC")
|
||||
|
||||
async def no_benchmark(db, **kwargs):
|
||||
return {}
|
||||
|
||||
monkeypatch.setattr(bt, "_load_benchmark_closes_for_backtest", no_benchmark)
|
||||
|
||||
report = await bt.run_backtest(session)
|
||||
benchmark = report["data_quality"]["benchmark"]
|
||||
|
||||
assert benchmark["sessions"] == 0
|
||||
assert benchmark["momentum_basis"] == "raw_fallback"
|
||||
assert report["data_quality"]["status"] == "failed"
|
||||
|
||||
|
||||
async def test_run_backtest_rolls_back_a_failed_portfolio_sim_load(session, monkeypatch):
|
||||
"""The portfolio-sim block loads the benchmark and the live exit policy from
|
||||
|
||||
Reference in new issue
Block a user