dennisthiessen/signal-platform
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: record fundamentals research decision and clean up

Browse Source
This commit is contained in:
dennisthiessen
2026-07-23 17:50:33 +02:00
parent dba7ea739b
commit 361cfd7883
21 changed files with 102 additions and 209972 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/research/fundamentals-weight-backtest.md
+46 -163
View File
@@ -1,191 +1,74 @@
# Point-in-time fundamentals weight backtest
# Fundamentals ranking-overlay research
Status: initial experiment completed; split-safe follow-up registered locally.
Running either protocol does not change production.
Status: completed 2026-07-23. Decision: keep production scoring and qualification unchanged.
## Question
Does reordering already-qualified long setups with SEC fundamentals improve the
production book's risk-adjusted return? The qualification gate, execution model,
position sizing, capacity, costs, ATR trail, and post-stop re-entry policy remain
unchanged. This isolates the incremental value of fundamentals as a ranking
overlay.
Does using point-in-time SEC fundamentals to reorder already-qualified long setups improve the production portfolio's risk-adjusted return? The experiments changed ranking only; qualification, execution, sizing, capacity, costs, ATR exits, and post-stop re-entry remained unchanged.
## Completed initial experiment
## Method
The control is the production 80/20 residual-momentum / volatility rank. The
runner tests three fundamental composites at weights 10%, 20%, 30%, and 40%:
- The control was the production 80/20 residual-momentum / volatility rank.
- SEC facts became visible only after `accepted_at`, using the newest visible accession per fiscal period.
- Portfolio simulations used daily entry opportunities, close fills, the production gate-reset re-entry policy, and a 30-session horizon.
- Train contained entries before 2024-01-01, validation covered 2024, and test began 2025-01-01.
- Missing composite scores were neutral at 50.
- Deflated Sharpe used the complete registered arm count for each experiment.
- Quality: operating margin, FCF margin, low net-debt/EBITDA, and low dilution.
At least two inputs must exist.
- Growth: revenue growth and diluted-EPS growth. At least one must exist.
- Balanced: equal weight to the quality and growth sub-scores. Both must exist.
The snapshot contained 511 tracked tickers, 507 unique CIKs, 30,494 SEC snapshot rows, and prices from 2021-06-24 through 2026-07-22.
Each raw metric is ranked favorably from 0 to 100 across the CIK-deduplicated
tracked universe. Missing composite scores are neutral at 50. The formula is:
## Initial experiment
`final rank = (1 - weight) * production rank + weight * fundamental rank`
The first registered matrix tested quality, growth, and balanced composites at 10%, 20%, 30%, and 40% weights: 13 trials including control.
There are 13 registered portfolio trials including the control. That complete
count is used by the Deflated Sharpe calculation.
No overlay passed the train and validation requirements. The most attractive full-period result, balanced at 10%, failed validation and improved test Sharpe by only 0.06.
No overlay passed the registered train and validation requirements. Review also
found that filing-time diluted EPS and shares are not guaranteed to use the same
split basis across periods. That makes EPS growth and share-count change unsafe
for historical ranking without point-in-time split factors. The initial result
remains an auditable rejection of its registered arms, but it is not evidence
that split-safe fundamentals have no value.
Review also found that filing-time diluted EPS and shares are not reliably comparable across stock splits. A snapshot audit found share-count changes above 25% for 90 of 461 issuers with comparable 2021+ periods, including recognizable split ratios for AMZN, GOOG, NVDA, CMG, and GE plus some obvious unit anomalies. Consequently, EPS growth and share-count change cannot be trusted for historical ranking without point-in-time split factors.
## Registered split-safe follow-up
The complete initial result is recoverable from Git commit `7f944d7`.
Run with `--protocol split-safe`. This is a smaller sensitivity experiment:
## Split-safe follow-up
- Quality: operating margin, FCF margin, and low net-debt/EBITDA. At least two
inputs must exist.
The follow-up excluded diluted-EPS growth and share-count change completely. It tested:
- Quality: operating margin, FCF margin, and low net-debt/EBITDA, requiring at least two inputs.
- Growth: revenue growth only.
- Balanced: equal weight to the quality and growth sub-scores. Both must exist.
- Balanced: equal quality and growth weights.
- Overlay weights: 5%, 10%, and 15%.
Diluted-EPS growth and share-count change are excluded completely: they are not
ranked, do not enter composites, and do not appear in factor-IC output. Historical
P/E and FCF yield remain excluded for the same split-basis reason. Earnings
surprise remains excluded because the completed Dolt SUE study failed its
promotion bar for this strategy.
This produced 10 registered trials including control. Growth coverage among qualified candidates was 88.96%; lack of data was not the limiting factor.
There are 10 registered portfolio trials including the control. The split-safe
report uses 10 in its Deflated Sharpe calculation. It has a separate score cache
and `fundamentals-splitsafe-*` output prefix, so it cannot be confused with or
silently reuse the initial experiment's scores.
| Window | Control Sharpe | Revenue-growth 5% | Delta |
|---|---:|---:|---:|
| Train | 1.26 | 1.31 | +0.05 |
| Validation | 2.52 | 2.64 | +0.12 |
| Test | 1.99 | 1.99 | 0.00 |
| Full | 1.86 | 1.87 | +0.01 |
The test window has already been inspected during the initial experiment. Keep
the original date boundaries and development selection discipline, but treat the
follow-up as sensitivity evidence. Any promotion still requires forward paper
evidence.
Revenue growth at 5% mechanically passed the deliberately permissive "not worse" gate, but did not demonstrate an economically meaningful edge:
## Point-in-time rule
- Test CAGR rose from 54.4% to 56.1%, while full-period CAGR fell from 52.1% to 51.5%.
- Full-period trade overlap was 68.53%, so roughly one-third of selections changed for essentially unchanged Sharpe.
- Revenue-growth IC was 0.0006 in train, 0.0053 in test, and 0.0116 full-period with a full-period t-stat of 0.55.
- Growth weights of 10% and 15% deteriorated; quality and balanced composites failed.
- The test window had already been inspected, so this follow-up was sensitivity evidence rather than a fresh out-of-sample result.
Only SEC rows accepted before midnight America/New_York at the start of a signal
date are visible. This is conservative relative to the daily pre-market SEC
import and prevents same-day filings or later amendments from leaking backward.
The derivation then selects the newest visible accession per fiscal period.
The complete split-safe result is recoverable from Git commit `dba7ea7`.
Default windows are fixed before the first run:
## Decision
- Train: entry date before 2024-01-01
- Validation: 2024-01-01 through 2024-12-31
- Test: entry date on or after 2025-01-01
- Do not add fundamental weight to production ranking or the automated qualification gate.
- Do not run another historical weight sweep on the same sample; it would add data-mining rather than new evidence.
- Keep fundamentals informational and user-facing in the UI.
- A5 source-parity and cutover work can proceed independently without changing scoring behavior.
- Treat historical EPS growth and share-count change as non-comparable across corporate actions until a split-aware solution or a conservative UI guard exists.
Do not move these boundaries after seeing results. The test window is used only
to check the single arm chosen from train and validation. Reports expose all
registered rows for auditability and correct multiple-testing accounting.
Revisit automated weighting only with materially better data, such as point-in-time split factors and historical constituent/delisting coverage, followed by genuinely new forward paper evidence.
## 1. Create the portable snapshot
## Limitations
Run this wherever the production PostgreSQL connection is already configured.
The exporter copies prices, the safe strategy settings, SEC snapshots, and Dolt
earnings rows. It does not copy credentials or unrelated system settings.
The snapshot uses today's tracked universe rather than historical membership and delisted securities, creating survivorship bias. Absolute CAGR and Sharpe must not be interpreted as unbiased live expectations. The relative comparison is useful, but the observed test window and short number of independent factor windows limit statistical power.
Windows PowerShell:
## Repository cleanup
```powershell
.venv\Scripts\python.exe scripts\create_backtest_snapshot.py `
--output backtest_snapshots\fundamentals-backtest.sqlite `
--force
```
Linux production host:
```bash
.venv/bin/python scripts/create_backtest_snapshot.py \
--output backtest_snapshots/fundamentals-backtest.sqlite \
--force
```
Copy only the SQLite file to the MacBook. `scp`, a local network share, or an
encrypted USB drive are all fine. Do not copy `.env`.
## 2. Prepare the MacBook
Use the same Git commit as the machine that created the report. From the repo:
```bash
python3.11 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e '.[dev]'
chmod +x scripts/run_fundamentals_macbook.sh
```
Put the snapshot at `backtest_snapshots/fundamentals-backtest.sqlite`, or pass a
different path to the launcher.
## 3. Run it
The launcher now defaults to the registered `split-safe` follow-up:
```bash
./scripts/run_fundamentals_macbook.sh \
backtest_snapshots/fundamentals-backtest.sqlite
```
The launcher defaults to logical CPU count minus one. Override it if the laptop
gets too warm or memory pressure rises:
```bash
WORKERS=8 ./scripts/run_fundamentals_macbook.sh \
backtest_snapshots/fundamentals-backtest.sqlite
```
To reproduce the completed initial matrix instead, opt in explicitly:
```bash
PROTOCOL=original ./scripts/run_fundamentals_macbook.sh \
backtest_snapshots/fundamentals-backtest.sqlite
```
The first run builds two caches under `reports/.cache`: production candidate
replay and protocol-specific point-in-time fundamental scores. If interrupted,
rerun the same command; valid caches are reused. Cache keys include the protocol,
snapshot size, and mtime, so neither a protocol switch nor a new snapshot can
reuse incompatible scores.
## 4. Bring the result back
The final line names one ZIP such as:
`reports/fundamentals-splitsafe-20260723-180000.zip`
That ZIP contains:
- the complete JSON report and reproducibility metadata;
- a readable Markdown summary;
- portfolio-arm CSV;
- factor-IC CSV;
- full-period trade CSV for every arm, allowing winner-concentration checks;
- this registered protocol.
Copy the ZIP into this workspace or attach it in the conversation. The snapshot
itself is not needed for the first evaluation unless a result looks inconsistent.
## Evaluation order
1. Data coverage and the accepted-at range.
2. Individual factor IC: sign, magnitude, consistency, and cross-section size.
3. Composite IC in train, validation, and test.
4. Development selection made without the test window.
5. Test Sharpe, CAGR, drawdown, yearly returns, trial-corrected DSR, and trade
overlap versus control.
6. Sensitivity to a few dominant winners and whether the effect is economically
large enough to justify production complexity.
The mechanical development bar requires train and validation Sharpe not below
control and validation drawdown no more than two percentage points worse. A test
pass is still research evidence, not automatic deployment.
## Known limitation
The snapshot contains today's tracked tickers, not historical constituent
membership or delisted names. This creates survivorship bias. The same biased
universe is used for control and overlays, so the local comparison is useful,
but its absolute Sharpe or CAGR must not be presented as an unbiased live
expectation. Forward paper performance remains the true out-of-sample check.
The experiment-only scorer, runner, Mac launcher, caches, tests, and expanded report bundles were removed after this decision. They remain recoverable from commits `eae4d34`, `34d6dda`, `7f944d7`, and `dba7ea7`. Production fundamentals derivation and ingestion remain unchanged.