feat(backtest): make an incomplete run visibly incomplete

A snapshot whose schema predated a migration made every _fetch_columns call
raise, both dispatch paths caught the per-ticker exception and continued, and
the run wrote a report with zero candidates that was indistinguishable from an
honest run that found no setups. Nothing said the run had lost its universe.

run_backtest now builds a completeness ledger and emits it as
report["data_quality"]: expected / loaded / evaluated counts, per-symbol
failures with stage and reason, bar-less symbols named, and benchmark
availability. Every universe symbol lands in exactly one terminal bucket and
the arithmetic is checked, so a counter that stops being maintained reports
itself as a failure rather than under-counting quietly.

Benchmark availability is a separate axis: without SPY, residual 12-1 momentum
silently degrades to raw 12-1, and the report now says so instead of leaving it
in a log line.

Three statuses, because severity differs. "failed" (nothing evaluated, empty
universe, unbalanced ledger, missing benchmark) replaces the performance tiles
with the callout rather than warning above them — a banner over ordinary-looking
numbers is exactly how the zero-candidate run would still have read as a quiet
week. "degraded" keeps the tiles with a note. A report with no data_quality
block reads as unknown, not clean; defaulting old reports to complete would
reproduce the bug being fixed.

The same gate applies to the two other consumers of backtest numbers:
productionBaseRates returns null for a failed run, and EvaluationPanel will not
report drift against a broken baseline.

"complete" means the processing and coverage checks passed. It does not mean
the prices are correct — a complete run over silently wrong bars is still
complete, and the card says so.

Also threads an explicit terminal-events option through the replay, validated
before dispatch so a stale events file aborts the run instead of quietly
degrading it. Verified against a healthy snapshot (degraded, 505/506) and one
with tickers.cik dropped (failed, 0/506, 506 fetch failures).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HAH7vVtn7qbcn1EVheecHQ
This commit is contained in:
dennisthiessenandClaude Opus 5 committed 2026-09-07 19:29:48 +02:00
1 parent ebe9047403
commit f2ef8e5438
11 files changed
+1013 -49

No files matched your search

+68
View File
@@ -1689,6 +1689,74 @@ async def test_run_backtest_rolls_back_a_failed_ticker_fetch(session, monkeypatc
assert report["tickers"] == 2
assert report["candidates"] >= 1
# ...and the report says one ticker was lost rather than presenting the
# remaining one as the whole universe.
quality = report["data_quality"]
assert quality["expected"] == 2
assert quality["evaluated"] == 1
assert quality["failed"] == 1
assert quality["failures"][0]["symbol"] == "AAA"
assert quality["failures"][0]["stage"] == "fetch"
assert "simulated OHLCV read failure" in quality["failures"][0]["error"]
assert quality["complete"] is False
assert quality["status"] == "degraded"
async def test_run_backtest_reports_its_own_completeness(session):
"""The healthy case: every ticker accounted for, so the report says so."""
await _seed_oscillating_ticker(session, "OSC")
report = await bt.run_backtest(session)
quality = report["data_quality"]
assert quality["expected"] == report["tickers"] == 1
assert quality["evaluated"] == 1
assert quality["failed"] == 0
assert quality["no_data"] == 0
# Ledger identity — the three terminal buckets must cover the universe.
assert quality["no_data"] + quality["failed"] + quality["evaluated"] == quality["expected"]
async def test_run_backtest_marks_a_run_failed_when_every_ticker_fails(session, monkeypatch):
"""The demonstrated failure: an unsynced snapshot raises on every read, the
replay loop swallows each one, and the run writes a report with no
candidates. It must come back "failed", not look like a quiet run.
"""
await _seed_oscillating_ticker(session, "AAA")
await _seed_oscillating_ticker(session, "OSC")
async def always_failing_fetch(db, symbol):
raise RuntimeError("no such column: tickers.cik")
monkeypatch.setattr(bt, "_fetch_columns", always_failing_fetch)
report = await bt.run_backtest(session)
assert report["candidates"] == 0
quality = report["data_quality"]
assert quality["status"] == "failed"
assert quality["evaluated"] == 0
assert quality["loaded"] == 0
assert quality["failed"] == 2
assert "no symbol was evaluated successfully" in quality["reasons"]
async def test_run_backtest_flags_a_missing_benchmark(session, monkeypatch):
"""Residual momentum degrades to raw without SPY. The numbers then describe
a different gate than the one the report claims, so the run is not clean."""
await _seed_oscillating_ticker(session, "OSC")
async def no_benchmark(db, **kwargs):
return {}
monkeypatch.setattr(bt, "_load_benchmark_closes_for_backtest", no_benchmark)
report = await bt.run_backtest(session)
benchmark = report["data_quality"]["benchmark"]
assert benchmark["sessions"] == 0
assert benchmark["momentum_basis"] == "raw_fallback"
assert report["data_quality"]["status"] == "failed"
async def test_run_backtest_rolls_back_a_failed_portfolio_sim_load(session, monkeypatch):
"""The portfolio-sim block loads the benchmark and the live exit policy from