feat(backtest): make an incomplete run visibly incomplete

A snapshot whose schema predated a migration made every _fetch_columns call
raise, both dispatch paths caught the per-ticker exception and continued, and
the run wrote a report with zero candidates that was indistinguishable from an
honest run that found no setups. Nothing said the run had lost its universe.

run_backtest now builds a completeness ledger and emits it as
report["data_quality"]: expected / loaded / evaluated counts, per-symbol
failures with stage and reason, bar-less symbols named, and benchmark
availability. Every universe symbol lands in exactly one terminal bucket and
the arithmetic is checked, so a counter that stops being maintained reports
itself as a failure rather than under-counting quietly.

Benchmark availability is a separate axis: without SPY, residual 12-1 momentum
silently degrades to raw 12-1, and the report now says so instead of leaving it
in a log line.

Three statuses, because severity differs. "failed" (nothing evaluated, empty
universe, unbalanced ledger, missing benchmark) replaces the performance tiles
with the callout rather than warning above them — a banner over ordinary-looking
numbers is exactly how the zero-candidate run would still have read as a quiet
week. "degraded" keeps the tiles with a note. A report with no data_quality
block reads as unknown, not clean; defaulting old reports to complete would
reproduce the bug being fixed.

The same gate applies to the two other consumers of backtest numbers:
productionBaseRates returns null for a failed run, and EvaluationPanel will not
report drift against a broken baseline.

"complete" means the processing and coverage checks passed. It does not mean
the prices are correct — a complete run over silently wrong bars is still
complete, and the card says so.

Also threads an explicit terminal-events option through the replay, validated
before dispatch so a stale events file aborts the run instead of quietly
degrading it. Verified against a healthy snapshot (degraded, 505/506) and one
with tickers.cik dropped (failed, 0/506, 506 fetch failures).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HAH7vVtn7qbcn1EVheecHQ
This commit is contained in:
dennisthiessenandClaude Opus 5 committed 2026-09-07 19:29:48 +02:00
1 parent ebe9047403
commit f2ef8e5438
11 files changed
+1013 -49

No files matched your search

+25
View File
@@ -72,9 +72,29 @@ def _parse_args() -> argparse.Namespace:
default=None,
help="Add a disjoint train/test portfolio report split at YYYY-MM-DD.",
)
parser.add_argument(
"--terminal-events",
default=None,
help=(
"JSON map of {symbol: YYYY-MM-DD} marking symbols whose final bar is a "
"research terminal mark, written by prep_survivorship_snapshot.py. Those "
"symbols are replayed through their last real bar instead of stopping 30 "
"bars early; every other symbol is unaffected."
),
)
return parser.parse_args()
def _load_terminal_events(path: str | None) -> dict[str, date]:
"""Parse the terminal-event map. Explicit input only — never inferred."""
if not path:
return {}
raw = json.loads(Path(path).read_text(encoding="utf-8"))
if not isinstance(raw, dict):
raise SystemExit(f"Terminal events file must be a JSON object: {path}")
return {str(sym).upper(): date.fromisoformat(str(value)) for sym, value in raw.items()}
def _default_output_path(cadence: str) -> Path:
stamp = datetime.now().strftime("%Y%m%d-%H%M%S")
return Path("reports") / f"backtest-{cadence}-{stamp}.json"
@@ -190,6 +210,10 @@ async def _main() -> None:
raise SystemExit("--holdout-split must use YYYY-MM-DD") from exc
os.environ["BACKTEST_HOLDOUT_SPLIT"] = args.holdout_split
terminal_events = _load_terminal_events(args.terminal_events)
if terminal_events:
print(f"terminal events: {len(terminal_events)} symbols ({', '.join(sorted(terminal_events))})")
from app.config import settings
from app.services.backtest_service import run_backtest
@@ -222,6 +246,7 @@ async def _main() -> None:
progress_cb=progress,
target_model=args.target_model,
cadence=args.cadence,
terminal_events=terminal_events,
)
finally:
await engine.dispose()