parallelize the backtest across worker processes (true multi-core)
The replay was CPU-bound and single-core: the earlier asyncio.to_thread offload kept the API responsive but, because of the GIL, ran on one core. Per-ticker replay is independent, so fan it out across worker processes (which sidestep the GIL) for real multi-core speedup. - New `settings.backtest_workers` (default 4), capped to cpu_count-1 so a core stays free for the web server. - Uses a `forkserver` context (workers forked from a clean single-threaded server — avoids the fork-with-threads deadlock); falls back to `fork`. On spawn-only platforms (Windows) and for 1-ticker runs it uses the thread path, so dev/tests are unaffected. - Worker takes primitive column arrays (cheap to pickle), rebuilds bars, and returns (candidates, plain-dict signal series) — both picklable across the process boundary. Bars are still fetched in the event loop (ORM-safe). - Pool creation is guarded: if the pool can't start, the job falls back to the sequential thread path instead of failing. 334 backend tests pass (parallel path is POSIX/server-only, so it's covered by construction + the picklability/worker-count tests; the thread fallback is exercised by the run_backtest smoke test). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -73,6 +73,12 @@ class Settings(BaseSettings):
|
||||
# backtests become computable. ~252 trading days/year.
|
||||
ohlcv_history_days: int = 1825
|
||||
|
||||
# Backtest parallelism: replay tickers across this many worker processes on
|
||||
# POSIX (forkserver), capped to cpu_count-1 so a core stays free for the web
|
||||
# server. 1 disables it (sequential). No effect on Windows / spawn-only
|
||||
# platforms — those fall back to a single worker thread.
|
||||
backtest_workers: int = 4
|
||||
|
||||
# Database Pool
|
||||
db_pool_size: int = 5
|
||||
db_pool_timeout: int = 30
|
||||
|
||||
Reference in New Issue
Block a user