research: volatility-standardized residual momentum screen, failed
Strategy review (2026-09-08) and a frozen, paired signal screen of residual 12-1 momentum divided by its formation-window residual volatility. Across 73 non-overlapping windows the normalized signal's IC is 0.0128 against 0.0145 for the matched control, below the frozen 0.03 floor, so the conditional portfolio replay was not run and nothing in production changes. Code-review fixes to the runner, before first commit: a missing OHLC price stops the run instead of reading as 0; the snapshot connection is closed explicitly (on Windows it otherwise kept the file locked); the paired-coverage check raises instead of asserting, so python -O cannot strip it; and source files are recorded as git blob ids with line endings normalized, so the report can be checked against history. The report was regenerated with the fixed runner; its screen section is byte-identical to the original run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
dennisthiessenandClaude Opus 5.5 committed
1 parent
70d17c1491
commit
7a9b3ad38b
9 files changed
+2659
No files matched your search
@@ -5,6 +5,13 @@ stop re-litigating settled questions: **if a row says "rejected", the experiment
|
|||||||
was run and the data said no.** Detail lives in the linked docs and in
|
was run and the data said no.** Detail lives in the linked docs and in
|
||||||
`reports/*.json` (all committed).
|
`reports/*.json` (all committed).
|
||||||
|
|
||||||
|
[Strategy research review (2026-09-08)](strategy-review-2026-09-08.md) compares
|
||||||
|
the existing evidence with published research and specifies a proposed
|
||||||
|
volatility-standardized residual-momentum experiment. The subsequent
|
||||||
|
[normalization screen failed](residual-normalization-findings.md): IC 0.0128
|
||||||
|
versus 0.0145 for matched control, below the frozen 0.03 floor. No portfolio
|
||||||
|
replay or production change followed.
|
||||||
|
|
||||||
**The one-line summary of the whole platform:** it is a **long-only
|
**The one-line summary of the whole platform:** it is a **long-only
|
||||||
cross-sectional momentum book** — buy the top quintile by beta-adjusted 12-1
|
cross-sectional momentum book** — buy the top quintile by beta-adjusted 12-1
|
||||||
momentum, tilt toward higher volatility, hold ≤ 10 names, cut at 1.5× ATR, then
|
momentum, tilt toward higher volatility, hold ≤ 10 names, cut at 1.5× ATR, then
|
||||||
@@ -48,6 +55,7 @@ score, Structural S/R, the Gate Target Ladder, sentiment, fundamentals) is
|
|||||||
| 11 | **FIP path-smoothness** as tie-breaker/filter | Non-monotonic within the qualified set; thinning the entry stream costs more compounding than the tilt returns | **Rejected as a filter** — but see §4, it's the strongest raw signal we've measured | — |
|
| 11 | **FIP path-smoothness** as tie-breaker/filter | Non-monotonic within the qualified set; thinning the entry stream costs more compounding than the tilt returns | **Rejected as a filter** — but see §4, it's the strongest raw signal we've measured | — |
|
||||||
| 12 | **Fixed take-profit sweep** (R-multiples) | No interior optimum ever found — the best TP is "no TP" | **Rejected.** Momentum's edge lives in the right tail | `backtest_service.py:450` |
|
| 12 | **Fixed take-profit sweep** (R-multiples) | No interior optimum ever found — the best TP is "no TP" | **Rejected.** Momentum's edge lives in the right tail | `backtest_service.py:450` |
|
||||||
| 13 | **Sector-residual 12-1** (`mom_12_1_sector_resid` / sector demean) as replacement for market residual | Short-window IC/A/B looked knife-edge green; deep repaired + **liquid-1500** retest: weeks 83, mild +IC **0.027** / t 1.69, **below iron bar 0.03** (FAIL). Demean already weaker | **Rejected / closed.** Keep production market residual. Do not resurrect without a new pre-registered protocol | [sector-residual-momentum.md](sector-residual-momentum.md) · `sector-resid-deep-20260719-113319.json` · history-depth supersession note |
|
| 13 | **Sector-residual 12-1** (`mom_12_1_sector_resid` / sector demean) as replacement for market residual | Short-window IC/A/B looked knife-edge green; deep repaired + **liquid-1500** retest: weeks 83, mild +IC **0.027** / t 1.69, **below iron bar 0.03** (FAIL). Demean already weaker | **Rejected / closed.** Keep production market residual. Do not resurrect without a new pre-registered protocol | [sector-residual-momentum.md](sector-residual-momentum.md) · `sector-resid-deep-20260719-113319.json` · history-depth supersession note |
|
||||||
|
| 14 | **Volatility-standardized market-residual 12-1** | 73 non-overlapping windows: IC **0.01284** vs matched control **0.01446**, below frozen 0.03 floor. Paired delta -0.00162, 95% interval [-0.00496, +0.00156] | **Closed negative at signal screen.** No demonstrated improvement; difference is inconclusive. Conditional portfolio stage not run | [Findings](residual-normalization-findings.md) · [Protocol](residual-normalization.md) · `residual-normalization-20260908.json` |
|
||||||
| 15 | **Breakeven stop** (stop to entry once the trade is +1R / +2R, on top of the 3× ATR trail) | `be_1r` fails the frozen rule: train worse on Sharpe, CAGR and drawdown, test better on return but worse on drawdown. `be_2r` changed 2 of 480 trades, because the trail already reaches entry near +2R. A post-hoc arm that gates re-entry after breakeven exits does better but still fails on drawdown | **Rejected / closed for this snapshot.** Keep the production exit. Reopen only with a new pre-registered test on unseen data | [breakeven-stop.md](breakeven-stop.md) · `backtest-20261009-breakeven-stop.json` |
|
| 15 | **Breakeven stop** (stop to entry once the trade is +1R / +2R, on top of the 3× ATR trail) | `be_1r` fails the frozen rule: train worse on Sharpe, CAGR and drawdown, test better on return but worse on drawdown. `be_2r` changed 2 of 480 trades, because the trail already reaches entry near +2R. A post-hoc arm that gates re-entry after breakeven exits does better but still fails on drawdown | **Rejected / closed for this snapshot.** Keep the production exit. Reopen only with a new pre-registered test on unseen data | [breakeven-stop.md](breakeven-stop.md) · `backtest-20261009-breakeven-stop.json` |
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|||||||
@@ -0,0 +1,76 @@
|
|||||||
|
# Residual normalization — findings
|
||||||
|
|
||||||
|
**2026-09-08: closed negative at the signal screen. Keep production unchanged.**
|
||||||
|
The proposed normalization did not meet the rule frozen in
|
||||||
|
[the experiment protocol](residual-normalization.md), and its point estimates
|
||||||
|
did not improve on the current momentum signal. No portfolio replay was run:
|
||||||
|
the protocol requires a passing signal screen before that stage.
|
||||||
|
|
||||||
|
The candidate used the production SPY-adjusted 12-1 return sum divided by sample
|
||||||
|
standard deviation of the same 231 daily adjusted returns. All comparisons used
|
||||||
|
identical as-of dates and 30-SPY-session forward-return endpoints. No new factor,
|
||||||
|
lookback, volatility weight, sizing rule, or exit was tested.
|
||||||
|
|
||||||
|
| Era | Non-overlapping windows | Current signal, matched coverage | Normalized signal | Difference |
|
||||||
|
|---|---:|---:|---:|---:|
|
||||||
|
| Full period | 73 | 0.01446 | 0.01284 | -0.00162 |
|
||||||
|
| Before 2021 | 30 | 0.00776 | 0.00496 | -0.00280 |
|
||||||
|
| From 2021 | 42 | 0.02377 | 0.02259 | -0.00118 |
|
||||||
|
|
||||||
|
Values are mean **Spearman information coefficients (IC)**: how well signal
|
||||||
|
ordering agrees with subsequent stock returns. They are neither Sharpe ratios
|
||||||
|
nor trading returns. Higher is better for this positive-momentum hypothesis.
|
||||||
|
The candidate needed full-period IC >= 0.03, at least 12 windows, and positive
|
||||||
|
mean IC in both eras. It passed the count and era-sign conditions but failed the
|
||||||
|
0.03 threshold. The rule was evaluated before rounding.
|
||||||
|
|
||||||
|
The paired normalized-minus-current IC difference has a 95% moving-block interval
|
||||||
|
of **[-0.00496, +0.00156]**. This spans zero: the study does not establish that
|
||||||
|
normalization is worse, and it supplies no convincing improvement evidence.
|
||||||
|
The mean top-minus-bottom-quintile 30-session return spread also fell, from
|
||||||
|
**0.967% to 0.751%**. That is an unconditional diagnostic, not the P&L of the
|
||||||
|
long-only, gated, capital-constrained production book.
|
||||||
|
|
||||||
|
**Coverage and checks**
|
||||||
|
|
||||||
|
- The snapshot completion manifest matches 4,650 registry tickers, 5,081,073
|
||||||
|
OHLCV rows, and 4,144 research-only names. The experiment used the remaining
|
||||||
|
506 production-universe registry names, averaging 485.4 paired stocks per
|
||||||
|
retained window.
|
||||||
|
- Actual retained as-of dates run from **2017-01-06 to 2026-05-01**; the final
|
||||||
|
forward endpoint is **2026-06-15**. Later dates cannot necessarily add another
|
||||||
|
complete non-overlapping 30-session window. SPY history starts in January
|
||||||
|
2016, so a full one-year signal cannot start in July 2016.
|
||||||
|
- One retained window crosses the 2021 boundary and belongs to the full-period
|
||||||
|
result only, explaining why the era counts sum to 72 rather than 73.
|
||||||
|
- The unrestricted production control had **35,441** retained stock/date cells;
|
||||||
|
paired control and treatment each had **35,433**. Its mean IC was 0.01447,
|
||||||
|
essentially the same as paired control. Coverage restriction did not account
|
||||||
|
for the observed result.
|
||||||
|
- Numerator parity was checked against the production implementation on each
|
||||||
|
eligible evaluated cell. The snapshot and frozen source hashes were unchanged
|
||||||
|
across the run. SQLite was opened read-only.
|
||||||
|
- Twelve targeted tests passed, including numerical invariance, production
|
||||||
|
parity, lookahead exclusion, zero-variance handling, missing-data pairing,
|
||||||
|
calendar overlap, and an end-to-end screen check. Ruff passed for the runner
|
||||||
|
and its tests. The existing Python runtime was usable outside the filesystem
|
||||||
|
sandbox; no environment replacement was needed.
|
||||||
|
|
||||||
|
**Interpretation and decision**
|
||||||
|
|
||||||
|
My proposed mechanism was plausible but unsupported by this experiment. Do not
|
||||||
|
add normalization to production or search denominator variants on this history.
|
||||||
|
This ends this specific experiment, not all possible residual-momentum research.
|
||||||
|
|
||||||
|
The current signal also falls below 0.03 on this particular long-history screen.
|
||||||
|
That cautions against treating its predictive evidence as universally strong;
|
||||||
|
it does not by itself measure or invalidate the complete production book. The
|
||||||
|
unconditional universe, true session spacing, dates, and return objective differ
|
||||||
|
from earlier setup/portfolio studies. Historical constituents are still projected
|
||||||
|
backward, this history has already been inspected, and the bootstrap intervals
|
||||||
|
remain exploratory. No new CAGR, drawdown, cost-stress, or live result is claimed.
|
||||||
|
|
||||||
|
**Artifacts:** [machine-readable results](../../reports/residual-normalization-20260908.json),
|
||||||
|
[generated tables](../../reports/residual-normalization-20260908.md),
|
||||||
|
[runner](../../scripts/run_residual_normalization.py), and
|
||||||
|
[tests](../../tests/unit/test_residual_normalization.py).
|
||||||
@@ -0,0 +1,87 @@
|
|||||||
|
# Residual normalization experiment
|
||||||
|
|
||||||
|
Protocol frozen 2026-09-08 before computing either signal's results. Implements
|
||||||
|
the first candidate in [the strategy review](strategy-review-2026-09-08.md).
|
||||||
|
This is offline research; it does not change production selection or settings.
|
||||||
|
|
||||||
|
**Question:** does dividing the current SPY-adjusted 12-1 score by its own daily
|
||||||
|
residual standard deviation improve predictive ordering and portfolio economics?
|
||||||
|
|
||||||
|
**Signal:** use the existing 252-bar lookback, skip 21 bars, estimate SPY beta
|
||||||
|
with centered covariance/variance, and retain the fitted intercept's contribution
|
||||||
|
exactly as production does. The treatment divides the production sum by sample
|
||||||
|
standard deviation (`ddof=1`) over the same 231 adjusted daily returns. No
|
||||||
|
denominator clipping, alternate window, factor addition, or weight sweep.
|
||||||
|
An exactly market-tracking series can leave machine-rounding residuals; sample
|
||||||
|
standard deviation <= 64 machine epsilons times `max(1, max(abs(stock_returns)))`
|
||||||
|
is treated as numerical zero, not divided into the signal.
|
||||||
|
|
||||||
|
**Data:** the completed `backtest_snapshots/research.sqlite`, restricted to
|
||||||
|
tickers absent from `research_rank_only` (506 production-universe registry names).
|
||||||
|
Nominal as-of start 2016-07-01; end bounded by actual SPY history and availability
|
||||||
|
of a 30-SPY-session forward endpoint. This snapshot starts SPY in January 2016,
|
||||||
|
so a fully aligned one-year signal cannot start in July 2016. Report actual
|
||||||
|
signal and forward-return dates. All history is previously inspected and
|
||||||
|
survivorship-biased; none is labeled fresh out-of-sample.
|
||||||
|
|
||||||
|
Require exactly 231 return pairs over consecutive benchmark sessions in the
|
||||||
|
formation window, positive finite prices, and nonzero residual variance for
|
||||||
|
both paired arms. A separate unrestricted production control retains production's
|
||||||
|
minimum-100-pair behavior. Report the removed population. Forward returns use
|
||||||
|
the same exact stock endpoint dates 30 SPY sessions apart, without filling absent
|
||||||
|
prices, without subtracting SPY, and without changing the stored return basis.
|
||||||
|
|
||||||
|
**Stage 1 — signal screen:** consider the last SPY session of each ISO week.
|
||||||
|
Keep a date with at least 20 common eligible stocks only if its as-of session is
|
||||||
|
at or after the preceding retained forward endpoint. Thus successive intervals
|
||||||
|
share at most an endpoint, not a return, including around holidays and year ends.
|
||||||
|
Use those dates for all three arms. Report per-window Spearman IC, top-minus-bottom
|
||||||
|
quintile forward-return spread, cross-sectional coverage, and paired IC deltas.
|
||||||
|
Era summaries use windows ending before 2021-01-01 and windows starting on/after
|
||||||
|
2021-01-01; boundary-crossing windows remain in full-period results only.
|
||||||
|
|
||||||
|
The screen passes only if the treatment has full-period mean IC >= 0.03, at
|
||||||
|
least 12 non-overlapping full-period windows, and strictly positive mean IC in
|
||||||
|
both eras. Mean IC and signs are checked before rounding. No additional paired
|
||||||
|
delta threshold is silently added. Report uncertainty whether the screen passes
|
||||||
|
or fails. For IC intervals use paired moving blocks of four retained windows,
|
||||||
|
5,000 replicates, NumPy PCG64 seed 20260908. These are exploratory intervals;
|
||||||
|
non-overlap alone does not establish statistical independence.
|
||||||
|
|
||||||
|
**Stage 2 — conditional portfolio comparison:** run only if stage 1 passes.
|
||||||
|
Replay daily, with unrestricted control, common-coverage control, and normalized
|
||||||
|
treatment; rank the full available production universe once per ticker/date.
|
||||||
|
Use unchanged 80/20 momentum/6-month-volatility ordering, top-20% momentum gate,
|
||||||
|
GTL qualification, capacity 15, 1% fixed-fractional risk, 20% notional cap,
|
||||||
|
cash constraints, 1.5 ATR initial stop, 3 ATR trail, 30-session hold, no
|
||||||
|
take-profit, and independent gate-reset histories. Keep the volatility rank
|
||||||
|
universe and its values identical between the paired arms. Use the snapshot's
|
||||||
|
non-momentum configuration and assert it agrees with the documented production
|
||||||
|
floors before running. Keep all non-qualified scans for gate-reset observations.
|
||||||
|
|
||||||
|
Use close fills as the historical execution scenario and costs 10 and 20 bps per
|
||||||
|
side. On a common calendar report paired returns, trade overlap, exposure,
|
||||||
|
turnover, risk per fill, and winning/losing contributions. Advancement requires
|
||||||
|
positive paired Sharpe improvement, no lower point-estimate CAGR, no deeper
|
||||||
|
point-estimate max drawdown, and a positive lower 95% bootstrap bound for Sharpe
|
||||||
|
improvement at both costs and both 21-/63-session block lengths. Use 5,000
|
||||||
|
paired replicates and seed 20260908. Bootstrap the realized return streams;
|
||||||
|
this does not model counterfactual changes in decisions along resampled prices.
|
||||||
|
|
||||||
|
A failed screen ends this experiment. A portfolio pass would justify forward
|
||||||
|
research, not automatic promotion. No historical variation or rule relaxation
|
||||||
|
will be chosen after seeing this result. Point-in-time membership and actual
|
||||||
|
execution evidence remain separate requirements for deployment.
|
||||||
|
|
||||||
|
**Reproduce stage 1:**
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
.venv\Scripts\python.exe scripts/run_residual_normalization.py
|
||||||
|
.venv\Scripts\python.exe -m pytest tests/unit/test_residual_normalization.py -q
|
||||||
|
```
|
||||||
|
|
||||||
|
The runner reads SQLite in read-only mode, validates the manifest, records
|
||||||
|
snapshot/protocol/code hashes and package versions, checks production-numerator
|
||||||
|
parity on the evaluated cells, and verifies the snapshot hash again before
|
||||||
|
writing completed results. Errors must not become skipped tickers or partial
|
||||||
|
successful reports.
|
||||||
@@ -0,0 +1,80 @@
|
|||||||
|
# Strategy research review — 2026-09-08
|
||||||
|
|
||||||
|
**Subsequent experiment:** the first candidate below was implemented and
|
||||||
|
[failed the frozen signal screen](residual-normalization-findings.md) on
|
||||||
|
2026-09-08. Its IC was 0.0128 versus 0.0145 for matched control, below the 0.03
|
||||||
|
floor. No portfolio replay or production change followed. The original research
|
||||||
|
rationale below is preserved as the proposal made before seeing that result.
|
||||||
|
|
||||||
|
**Recommendation:** make volatility-standardized market-residual momentum the next small research experiment. Keep the current strategy as the control. Before treating any winner as deployable, establish its result on historical membership and a forward record with observable execution. A separate cross-asset trend strategy is the more substantial diversification candidate.
|
||||||
|
|
||||||
|
**Status:** research synthesis and proposed experiment, not a new backtest or a promotion. Objective assumed: better net returns relative to drawdown. Historical figures below were checked against the saved JSON reports; production code and settings were not changed. No replacement has been demonstrated superior by this review.
|
||||||
|
|
||||||
|
**What the existing evidence actually supports**
|
||||||
|
|
||||||
|
The current code defaults are a long-only, top-quintile market-residual 12-1 momentum book, ordered by 80% momentum percentile and 20% six-month volatility percentile. Capacity is 15, with 1% account risk per entry subject to cash and notional limits, a 1.5 ATR initial stop, a 3 ATR trail, a 30-session maximum hold, and gate-reset re-entry. Capacity and near-close scheduling have already shipped; some introductory research text still describes capacity 10 or scheduling as open work. These are code defaults and documented implementation, not an inspection of live Admin settings. See [momentum implementation](../../app/services/momentum_service.py), [capacity findings](portfolio-capacity-bracket-findings.md), and [execution decision](execution-recovery.md).
|
||||||
|
|
||||||
|
| Existing experiment | Recorded result | Implication |
|
||||||
|
|---|---|---|
|
||||||
|
| Production universe, entries from July 2022 | Sharpe 1.32; CAGR 31.8%; max drawdown 18.9% | Positive on that particular replay. |
|
||||||
|
| Same package, production plus liquid breadth | Sharpe 0.14; CAGR -1.8%; max drawdown 55.3% | Expanding the tradable universe under the existing rules already failed. |
|
||||||
|
| Production universe, entries from July 2016 | Sharpe 0.88; CAGR 16.7%; max drawdown 24.4% | Longer-history evidence is appreciably weaker. |
|
||||||
|
| Deep history plus liquid breadth | Sharpe -0.06; CAGR -7.0%; max drawdown 73.9% | More history did not rescue breadth. |
|
||||||
|
| Restore 85 removed constituents, September sensitivity study | Sharpe 2.09 → 1.57; CAGR 51.6% → 38.1% | Cross-sectional universe composition materially changes results. |
|
||||||
|
| Close versus next-open execution, July study | Sharpe 1.77 → 1.20; CAGR 48.3% → 30.0% | Execution assumptions can dominate a signal improvement. |
|
||||||
|
|
||||||
|
The first four rows come from one [universe/horizon matrix](../../reports/prod-book-universe-horizon-20260719-140737.json), using the then-current capacity of 10. They are not new measurements of today's capacity-15 book. The last two rows come from separate [survivorship](survivorship-audit.md) and [execution](../../reports/execution_recovery_matrix.json) studies. Do not pool these as one baseline: cadence, history, universe, configuration, and end dates differ. Today's constituents projected backward remain a limitation of the deep-history study. Restoring removals is also not point-in-time membership, and the 13.5 percentage-point CAGR difference is not an estimate of survivorship bias. The sensitivity study attributes the portfolio change to universe recomposition, not positions in the failed banks.
|
||||||
|
|
||||||
|
**First candidate: standardize the existing residual signal**
|
||||||
|
|
||||||
|
The code estimates a stock's beta to SPY over the 12-1 formation window and sums daily stock returns minus beta times SPY returns. It deliberately does not subtract a fitted intercept. It also does not divide this score by residual volatility. That is a legitimate practical approximation; the name alone does not make it a replication of the academic strategy.
|
||||||
|
|
||||||
|
Blitz, Huij and Martens estimate market, size, and value exposures using 36 monthly observations, then standardize the 12-1 residual return by its volatility. Their main portfolios are monthly long-short decile portfolios. They explicitly omit the estimated alpha from the momentum calculation. Their reported improvement therefore does not establish an improvement for this concentrated long-only ATR strategy. [Residual Momentum, accepted manuscript, pp. 10–12](https://repub.eur.nl/pub/22252/ResidualMomentum-2011.pdf)
|
||||||
|
|
||||||
|
My proposed first experiment isolates standardization using the existing SPY model:
|
||||||
|
|
||||||
|
```text
|
||||||
|
u[d] = stock_return[d] - beta * SPY_return[d]
|
||||||
|
control_score = sum(u[d])
|
||||||
|
candidate_score = control_score / sample_standard_deviation(u[d])
|
||||||
|
```
|
||||||
|
|
||||||
|
Use the same 231 daily return pairs from the existing 252-session lookback with the final 21 sessions skipped. Require a complete, aligned window and positive residual standard deviation in both arms for the paired signal comparison. Retain a separate unrestricted control to quantify any coverage effect. Do not add a fitted-intercept subtraction, new lookback, or new volatility weight.
|
||||||
|
|
||||||
|
**Hypothesis, not finding:** measuring a stock-specific rise relative to its own noise may improve ordering and reduce sensitivity to unusually volatile names. This changes the signal itself; it is distinct from the previously rejected inverse-volatility position sizing or standalone low-volatility ranking. Nevertheless, those negative results and the strategy's dependence on a few large winners lower confidence: standardization could also remove valuable winners. The research search found no recorded direct test of this exact normalization in the current scripts and research documents.
|
||||||
|
|
||||||
|
Keep the existing 80/20 blend in the first portfolio comparison, replacing only its momentum input and the momentum gate input. Removing the high-volatility tilt simultaneously would obscure the mechanism. If this narrow test fails, record the failure without searching over denominator windows, powers, or clipping rules on the same history.
|
||||||
|
|
||||||
|
**Second candidate: a fuller factor-residual model**
|
||||||
|
|
||||||
|
A later, separately specified candidate could remove market, size, and value exposures with a longer estimation window. This addresses factor contamination that single-market beta adjustment leaves behind. It is distinct from the rejected sector-residual experiment, but that rejection still argues for modest expectations. Model estimation, monthly versus daily updates, coverage, and the existing daily gate-reset behavior would all need explicit treatment.
|
||||||
|
|
||||||
|
The factor data are available from the [Kenneth French Data Library](https://mba.tuck.dartmouth.edu/pages/faculty/ken.french/data_library.html), which states that historical returns are reconstructed during updates and can change. Freeze the source vintage and record when factors were actually available. A factor observation dated month-end is not automatically usable for a trade at that same close. Current downloadable history alone does not establish historical publication availability. A monthly paper replication and a daily deployable adaptation must be labeled separately.
|
||||||
|
|
||||||
|
Prioritize the simpler normalization experiment because it isolates one mechanism without adding a factor-data dependency. Neither candidate fixes historical universe membership or corporate-action errors. The existing [total-return A/B](tr-total-return-ab.md) remains unresolved; silently changing dividend treatment in the same comparison would introduce another cause for any difference.
|
||||||
|
|
||||||
|
**Larger alternative: an independent cross-asset trend allocation**
|
||||||
|
|
||||||
|
Research a separate portfolio of liquid equity-index, bond, commodity, and currency exposures, taking positions according to each market's own past return. The purpose is to diversify the source of portfolio returns beyond stock selection. This would be a new strategy and data pipeline, with its own capital allocation and execution model.
|
||||||
|
|
||||||
|
Hurst, Ooi and Pedersen study time-series momentum across futures and forwards and extend the historical examination back to 1880. Their results motivate investigating the strategy family; they do not supply an expected return or optimal allocation for this platform. [A Century of Evidence on Trend-Following Investing](https://www.aqr.com/insights/research/journal-article/a-century-of-evidence-on-trend-following-investing)
|
||||||
|
|
||||||
|
A long-only ETF/cash version may be easier to operate, but cannot reproduce the short positions, leverage, or historical instrument coverage of the published strategy. Start with one fixed specification and evaluate both its standalone economics and its contribution to a fixed combined portfolio after costs. Diversification must be measured, especially during equity losses; shared use of the word momentum does not determine correlation. This is a larger effort than the first candidate and remains untested here.
|
||||||
|
|
||||||
|
**What to leave closed**
|
||||||
|
|
||||||
|
The local record does not support another round of take-profit exits, SPY moving-average entry bans, gap-up caps, high-volatility breadth expansion, earnings-surprise gates, fundamental score weights, or ATR/hold-period sweeps on the same snapshots. See the [experiment index](README.md), [earnings findings](earnings-gap-and-sue.md), and [fundamental-weight findings](fundamentals-weight-backtest.md). Repeating them would add selection opportunities rather than independent evidence.
|
||||||
|
|
||||||
|
Crash-management papers also need careful translation. Daniel, Jagannathan and Kim identify a short-side mechanism in long-short momentum crashes. That is not direct evidence for adding an equity-regime veto to this long-only strategy. [Tail Risk in Momentum Strategy Returns](https://www.nber.org/papers/w18169)
|
||||||
|
|
||||||
|
**Proposed evaluation contract for the first candidate**
|
||||||
|
|
||||||
|
1. **Freeze the inputs and trial first.** Record snapshot and code hashes, symbol/date eligibility, cash treatment, costs, signal formula, and report boundaries. Verify the research snapshot completion manifest against actual counts. Treat the previously inspected 2016–2026 history as exploratory evidence, including periods historically labeled validation or test.
|
||||||
|
2. **Screen the signal on identical cells.** Compare the candidate and current score against the same 30-session forward returns. Select actual non-overlapping session intervals; report window count, mean Spearman IC, uncertainty, quintile spreads, and pre-/post-2021 signs. The existing IC >= 0.03, consistent-sign, >= 12-window rule is a screening requirement. The harness's `reliable` flag counts windows; it is not a statistical significance result. Measure paired IC differences as well as each signal's absolute IC.
|
||||||
|
3. **Replay the whole policy if the screen passes.** Use daily scans, capacity 15, unchanged cash/notional limits, stops, trail, hold, costs, and 80/20 weights. Recompute qualifications and gate-reset histories separately for each arm. Reusing the incumbent's qualified candidates or lock histories would miss decisions caused by the treatment.
|
||||||
|
4. **Grade portfolio economics.** For the default objective, propose a conservative advancement rule: positive paired Sharpe improvement, no lower point-estimate CAGR, and no deeper point-estimate maximum drawdown, with a paired 95% interval for Sharpe improvement above zero. Report 21- and 63-session block-bootstrap intervals, with seed and replicate count frozen before execution. Require the conclusion to survive both block lengths and a doubled-cost stress from 10 to 20 bps per side. These are proposed research criteria, not previously passed tests. Failure or inadequate precision means no promotion; do not relax the rule after seeing results.
|
||||||
|
5. **Check why the book changed.** Report turnover, exposure, entry overlap, risk per fill, best/worst contributors, and foregone winners. Pair daily portfolio returns on a common calendar. Overlapping start-date paths are sensitivity checks, not independent observations. Resampling realized returns does not simulate how different price histories would change future decisions.
|
||||||
|
6. **Establish deployability separately.** Use dated constituent membership, symbol identity and delisting treatment, and a documented price/return basis. For execution, retain the actual scan inputs, order timestamps, decision prices, and fills. The current shadow book records near-close setup prices; those are simulated entries, not broker execution evidence. Daily bars cannot reconstruct the exact partial bar that a near-close scan saw. Historical close/stale-close results are scenarios, not a guaranteed bracket for future live Sharpe.
|
||||||
|
7. **Reserve prospective evidence.** Choose a fixed forward evaluation schedule before collecting the comparison, with separate paper books. Require enough calendar history to assess uncertainty; a small trade count or an early favorable month is not a promotion criterion. Maintain the full experiment ledger. Deflated Sharpe can address multiple selection and non-normality when the relevant trial information is available, but cannot repair lookahead or invent untouched data. [Bailey and López de Prado, The Deflated Sharpe Ratio](https://www.davidhbailey.com/dhbpapers/deflated-sharpe.pdf)
|
||||||
|
|
||||||
|
**Deliverable and limits:** this review supplies a prioritized candidate and a concrete evaluation design. It verifies existing evidence rather than reporting a newly discovered edge. No new replay was executed; the checked-in virtual environment points to an unavailable Python interpreter in this session. The research recommendation does not depend on treating that missing run as a result.
|
||||||
@@ -1,5 +1,12 @@
|
|||||||
# Backtest report index
|
# Backtest report index
|
||||||
|
|
||||||
|
The 2026-09-08 residual-normalization signal experiment is preserved in
|
||||||
|
`residual-normalization-20260908.json` and its generated Markdown table.
|
||||||
|
The JSON includes every retained window, coverage, paired uncertainty, the
|
||||||
|
mechanical verdict, the snapshot hash, and the git blob ids of the source it
|
||||||
|
ran. The screen failed its frozen IC floor, so no portfolio replay was run. Interpretation:
|
||||||
|
[`residual-normalization-findings.md`](../docs/research/residual-normalization-findings.md).
|
||||||
|
|
||||||
The 2026-10-09 breakeven-stop exit test is preserved in
|
The 2026-10-09 breakeven-stop exit test is preserved in
|
||||||
`backtest-20261009-breakeven-stop.json` (regenerated 2026-10-10 after a code
|
`backtest-20261009-breakeven-stop.json` (regenerated 2026-10-10 after a code
|
||||||
review, with no graded number changed). Its `breakeven_stop_study` section holds
|
review, with no graded number changed). Its `breakeven_stop_study` section holds
|
||||||
|
|||||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,35 @@
|
|||||||
|
# Residual normalization — signal screen
|
||||||
|
|
||||||
|
Generated: 2026-10-10T07:53:06.165467+00:00
|
||||||
|
|
||||||
|
**Verdict: FAIL**
|
||||||
|
|
||||||
|
Previously inspected historical data; no production change or new out-of-sample claim.
|
||||||
|
|
||||||
|
| Era | Arm | Windows | Avg stocks | Mean IC | IC 95% interval | Quintile spread |
|
||||||
|
|---|---|---:|---:|---:|---|---:|
|
||||||
|
| full | unrestricted_control | 73 | 485.5 | 0.0145 | [-0.0257, 0.0450] | 0.97% |
|
||||||
|
| full | paired_control | 73 | 485.4 | 0.0145 | [-0.0257, 0.0450] | 0.97% |
|
||||||
|
| full | normalized | 73 | 485.4 | 0.0128 | [-0.0271, 0.0442] | 0.75% |
|
||||||
|
| pre_2021 | unrestricted_control | 30 | 472.0 | 0.0078 | [-0.0338, 0.0682] | 0.76% |
|
||||||
|
| pre_2021 | paired_control | 30 | 471.7 | 0.0078 | [-0.0338, 0.0683] | 0.75% |
|
||||||
|
| pre_2021 | normalized | 30 | 471.7 | 0.0050 | [-0.0395, 0.0661] | 0.46% |
|
||||||
|
| post_2021 | unrestricted_control | 42 | 495.2 | 0.0238 | [-0.0125, 0.0630] | 1.26% |
|
||||||
|
| post_2021 | paired_control | 42 | 495.2 | 0.0238 | [-0.0125, 0.0630] | 1.26% |
|
||||||
|
| post_2021 | normalized | 42 | 495.2 | 0.0226 | [-0.0147, 0.0631] | 1.06% |
|
||||||
|
|
||||||
|
Paired normalized-minus-control IC differences:
|
||||||
|
|
||||||
|
- full: -0.0016; 95% interval [-0.004960466538844113, 0.0015584142071432597].
|
||||||
|
- pre_2021: -0.0028; 95% interval [-0.008151700038659202, 0.001515881215178447].
|
||||||
|
- post_2021: -0.0012; 95% interval [-0.005965195533675296, 0.003076462018338155].
|
||||||
|
|
||||||
|
Screen checks: `{"mean_ic_ge_0_03": false, "at_least_12_windows": true, "pre_2021_positive": true, "post_2021_positive": true}`
|
||||||
|
|
||||||
|
Portfolio stage: **not_run_failed_signal_screen**.
|
||||||
|
|
||||||
|
Signal IC and quintile spreads are predictive diagnostics, not simulated strategy returns.
|
||||||
|
Intervals resample paired blocks of four non-overlapping forward windows; they remain exploratory.
|
||||||
|
The unrestricted control shows the effect of requiring complete formation history.
|
||||||
|
|
||||||
|
[Frozen protocol](../docs/research/residual-normalization.md)
|
||||||
@@ -0,0 +1,360 @@
|
|||||||
|
"""Frozen, paired residual-volatility normalization screen. Offline only.
|
||||||
|
|
||||||
|
See docs/research/residual-normalization.md. A failed screen terminates the
|
||||||
|
experiment; a passing screen allows the separately specified daily book replay.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import contextlib
|
||||||
|
import hashlib
|
||||||
|
import importlib.metadata
|
||||||
|
import json
|
||||||
|
import math
|
||||||
|
import sqlite3
|
||||||
|
import statistics
|
||||||
|
import sys
|
||||||
|
from collections import Counter
|
||||||
|
from datetime import date, datetime, timezone
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
|
||||||
|
ROOT = Path(__file__).resolve().parents[1]
|
||||||
|
if str(ROOT) not in sys.path:
|
||||||
|
sys.path.insert(0, str(ROOT))
|
||||||
|
|
||||||
|
from app.services.backtest_service import _quintile_spread, _spearman # noqa: E402
|
||||||
|
from app.services.momentum_service import compute_residual_12_1_momentum # noqa: E402
|
||||||
|
|
||||||
|
START = date(2016, 7, 1)
|
||||||
|
ERA_SPLIT = date(2021, 1, 1)
|
||||||
|
LOOKBACK = 252
|
||||||
|
SKIP = 21
|
||||||
|
PAIRS = LOOKBACK - SKIP
|
||||||
|
HORIZON = 30
|
||||||
|
MIN_CROSS = 20
|
||||||
|
MIN_WINDOWS = 12
|
||||||
|
IC_FLOOR = 0.03
|
||||||
|
SEED = 20260908
|
||||||
|
REPLICATES = 5000
|
||||||
|
IC_BLOCK = 4
|
||||||
|
ARMS = ("unrestricted_control", "paired_control", "normalized")
|
||||||
|
PROTOCOL = ROOT / "docs/research/residual-normalization.md"
|
||||||
|
|
||||||
|
|
||||||
|
def sha256(path: Path) -> str:
|
||||||
|
with path.open("rb") as source:
|
||||||
|
return hashlib.file_digest(source, "sha256").hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
def git_blob_id(path: Path) -> str:
|
||||||
|
"""The id ``git hash-object`` gives this file. Line endings are normalized
|
||||||
|
first, as git does on commit, so a CRLF checkout and an LF checkout of the
|
||||||
|
same source record the same id — one that can be looked up in history."""
|
||||||
|
data = path.read_bytes().replace(b"\r\n", b"\n")
|
||||||
|
return hashlib.sha1(b"blob %d\0" % len(data) + data).hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
def read_snapshot(path: Path) -> tuple[dict, dict, dict]:
|
||||||
|
"""Fail closed on incomplete data; never open a live application database."""
|
||||||
|
path = path.resolve(strict=True)
|
||||||
|
manifest = json.loads(Path(str(path) + ".manifest.json").read_text(encoding="utf-8"))
|
||||||
|
if manifest.get("complete") is not True:
|
||||||
|
raise ValueError("Snapshot completion manifest is not complete")
|
||||||
|
if any(Path(str(path) + suffix).exists() for suffix in ("-wal", "-journal")):
|
||||||
|
raise ValueError("Snapshot has a journal; use a completed, quiescent copy")
|
||||||
|
# sqlite3's own context manager only ends the transaction; closing releases
|
||||||
|
# the file handle, which on Windows otherwise pins the snapshot for the run.
|
||||||
|
with contextlib.closing(sqlite3.connect(path.as_uri() + "?mode=ro", uri=True)) as con:
|
||||||
|
con.execute("PRAGMA query_only=ON")
|
||||||
|
counts = {
|
||||||
|
key: con.execute(f"SELECT COUNT(*) FROM {table}").fetchone()[0]
|
||||||
|
for key, table in (
|
||||||
|
("ticker_count", "tickers"),
|
||||||
|
("ohlcv_row_count", "ohlcv_records"),
|
||||||
|
("rank_only_count", "research_rank_only"),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
if any(manifest.get(k) != v for k, v in counts.items()):
|
||||||
|
raise ValueError(f"Snapshot completion counts do not match: {counts}")
|
||||||
|
tickers = con.execute(
|
||||||
|
"SELECT id, symbol FROM tickers WHERE symbol NOT IN "
|
||||||
|
"(SELECT symbol FROM research_rank_only) ORDER BY symbol"
|
||||||
|
).fetchall()
|
||||||
|
benchmark = {
|
||||||
|
date.fromisoformat(d[:10]): float(c)
|
||||||
|
for d, c in con.execute(
|
||||||
|
"SELECT date, close FROM benchmark_prices WHERE symbol='SPY' ORDER BY date"
|
||||||
|
)
|
||||||
|
}
|
||||||
|
if not benchmark or any(not math.isfinite(c) or c <= 0 for c in benchmark.values()):
|
||||||
|
raise ValueError("Missing or invalid SPY history")
|
||||||
|
prices = {}
|
||||||
|
for ticker_id, symbol in tickers:
|
||||||
|
rows = con.execute(
|
||||||
|
"SELECT date, open, high, low, close, volume FROM ohlcv_records "
|
||||||
|
"WHERE ticker_id=? ORDER BY date", (ticker_id,),
|
||||||
|
).fetchall()
|
||||||
|
dates = [date.fromisoformat(r[0][:10]) for r in rows]
|
||||||
|
if len(dates) != len(set(dates)):
|
||||||
|
raise ValueError(f"Duplicate stock dates: {symbol}")
|
||||||
|
# A missing price must stop the run: read as 0 it would put a -100%
|
||||||
|
# return into the unrestricted control while the paired arms drop it.
|
||||||
|
if any(r[k] is None for r in rows for k in range(1, 5)):
|
||||||
|
raise ValueError(f"Missing OHLC price: {symbol}")
|
||||||
|
columns = (
|
||||||
|
([d.toordinal() for d in dates],)
|
||||||
|
+ tuple([float(r[k]) for r in rows] for k in range(1, 5))
|
||||||
|
+ ([float(r[5] or 0) for r in rows],)
|
||||||
|
)
|
||||||
|
prices[symbol] = columns
|
||||||
|
return prices, benchmark, {**manifest, "verified_counts": counts}
|
||||||
|
|
||||||
|
|
||||||
|
def signal_at(
|
||||||
|
dates: list[date], closes: list[float], i: int,
|
||||||
|
benchmark: dict[date, float], benchmark_index: dict[date, int],
|
||||||
|
) -> dict:
|
||||||
|
"""Production numerator, paired with a strictly aligned sample denominator."""
|
||||||
|
empty = {"unrestricted_control": None, "paired_control": None,
|
||||||
|
"normalized": None, "residual_sd": None, "reason": "short_history"}
|
||||||
|
if i < LOOKBACK:
|
||||||
|
return empty
|
||||||
|
window_dates = dates[i - LOOKBACK:i + 1]
|
||||||
|
window_closes = closes[i - LOOKBACK:i + 1]
|
||||||
|
raw = compute_residual_12_1_momentum(window_dates, window_closes, benchmark)
|
||||||
|
if raw is not None and not math.isfinite(raw):
|
||||||
|
raise ValueError("Non-finite production signal")
|
||||||
|
empty["unrestricted_control"] = raw
|
||||||
|
formation_dates = window_dates[:PAIRS + 1]
|
||||||
|
formation_closes = window_closes[:PAIRS + 1]
|
||||||
|
indices = [benchmark_index.get(d) for d in formation_dates]
|
||||||
|
if any(x is None for x in indices) or any(
|
||||||
|
b != a + 1 for a, b in zip(indices, indices[1:])
|
||||||
|
):
|
||||||
|
return {**empty, "reason": "unaligned_formation"}
|
||||||
|
if any(not math.isfinite(c) or c <= 0 for c in formation_closes):
|
||||||
|
return {**empty, "reason": "invalid_formation_price"}
|
||||||
|
market = [benchmark[d] for d in formation_dates]
|
||||||
|
stock_returns = [b / a - 1 for a, b in zip(formation_closes, formation_closes[1:])]
|
||||||
|
market_returns = [b / a - 1 for a, b in zip(market, market[1:])]
|
||||||
|
mean_s, mean_m = statistics.fmean(stock_returns), statistics.fmean(market_returns)
|
||||||
|
var_m = sum((r - mean_m) ** 2 for r in market_returns)
|
||||||
|
if var_m <= 0 or raw is None:
|
||||||
|
return {**empty, "reason": "unavailable_beta"}
|
||||||
|
beta = sum((s - mean_s) * (m - mean_m)
|
||||||
|
for s, m in zip(stock_returns, market_returns)) / var_m
|
||||||
|
residuals = [s - beta * m for s, m in zip(stock_returns, market_returns)]
|
||||||
|
# Independent arithmetic verifies that the denominator uses the numerator's
|
||||||
|
# exact return window. It must never silently redefine production momentum.
|
||||||
|
if not math.isclose(sum(residuals), raw, rel_tol=1e-10, abs_tol=1e-12):
|
||||||
|
raise AssertionError("Production numerator parity failed")
|
||||||
|
sd = statistics.stdev(residuals)
|
||||||
|
# Treat machine-precision residuals of a perfect market tracker as zero,
|
||||||
|
# not as infinite signal strength. This is a numerical degeneracy check,
|
||||||
|
# not a tunable volatility floor.
|
||||||
|
numerical_zero = 64 * np.finfo(float).eps * max(1.0, max(map(abs, stock_returns)))
|
||||||
|
if sd <= numerical_zero:
|
||||||
|
return {**empty, "reason": "zero_residual_variance"}
|
||||||
|
normalized = raw / sd
|
||||||
|
if not math.isfinite(normalized):
|
||||||
|
raise ValueError("Non-finite normalized signal")
|
||||||
|
return {"unrestricted_control": raw, "paired_control": raw,
|
||||||
|
"normalized": normalized, "residual_sd": sd, "reason": "eligible"}
|
||||||
|
|
||||||
|
|
||||||
|
def weekly_indices(calendar: list[date]) -> list[int]:
|
||||||
|
last = {}
|
||||||
|
for i, day in enumerate(calendar):
|
||||||
|
last[day.isocalendar()[:2]] = i
|
||||||
|
return sorted(i for i in last.values() if calendar[i] >= START and i + HORIZON < len(calendar))
|
||||||
|
|
||||||
|
|
||||||
|
def nonoverlapping_indices(indices: list[int], horizon: int = HORIZON) -> list[int]:
|
||||||
|
kept = []
|
||||||
|
for i in sorted(indices):
|
||||||
|
if not kept or i >= kept[-1] + horizon:
|
||||||
|
kept.append(i)
|
||||||
|
return kept
|
||||||
|
|
||||||
|
|
||||||
|
def interval(values) -> list[float] | None:
|
||||||
|
if len(values) == 0:
|
||||||
|
return None
|
||||||
|
return [float(v) for v in np.quantile(values, [0.025, 0.975])]
|
||||||
|
|
||||||
|
|
||||||
|
def bootstrap_means(matrix: np.ndarray) -> dict:
|
||||||
|
"""Shared moving-block indices preserve both the dates and arm pairing."""
|
||||||
|
n = len(matrix)
|
||||||
|
if n < IC_BLOCK:
|
||||||
|
return {"reason": "fewer_windows_than_block", "arms": {}, "paired_delta_ci95": None}
|
||||||
|
rng = np.random.default_rng(SEED)
|
||||||
|
starts = rng.integers(0, n - IC_BLOCK + 1, (REPLICATES, math.ceil(n / IC_BLOCK)))
|
||||||
|
indices = (starts[:, :, None] + np.arange(IC_BLOCK)).reshape(REPLICATES, -1)[:, :n]
|
||||||
|
means = matrix[indices].mean(axis=1)
|
||||||
|
return {
|
||||||
|
"arms": {arm: interval(means[:, i]) for i, arm in enumerate(ARMS)},
|
||||||
|
"paired_delta_ci95": interval(means[:, 2] - means[:, 1]),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def summarize(windows: list[dict]) -> dict:
|
||||||
|
if not windows:
|
||||||
|
return {"windows": 0, "arms": {}, "paired_mean_ic_delta": None}
|
||||||
|
matrix = np.array([[w["arms"][arm]["ic"] for arm in ARMS] for w in windows])
|
||||||
|
uncertainty = bootstrap_means(matrix)
|
||||||
|
arms = {}
|
||||||
|
for i, arm in enumerate(ARMS):
|
||||||
|
values = matrix[:, i]
|
||||||
|
se = float(values.std(ddof=1) / math.sqrt(len(values))) if len(values) > 1 else None
|
||||||
|
arms[arm] = {
|
||||||
|
"mean_ic": float(values.mean()),
|
||||||
|
"ic_t_stat_descriptive": float(values.mean() / se) if se and se > 0 else None,
|
||||||
|
"ic_ci95": uncertainty["arms"].get(arm),
|
||||||
|
"positive_window_fraction": float((values > 0).mean()),
|
||||||
|
"avg_cross_section": statistics.fmean(w["arms"][arm]["n"] for w in windows),
|
||||||
|
"mean_quintile_spread": statistics.fmean(
|
||||||
|
w["arms"][arm]["quintile_spread"] for w in windows
|
||||||
|
),
|
||||||
|
"reliable_window_count": len(windows) >= MIN_WINDOWS,
|
||||||
|
}
|
||||||
|
return {"windows": len(windows), "first_asof": windows[0]["asof"],
|
||||||
|
"last_asof": windows[-1]["asof"], "last_forward_end": windows[-1]["forward_end"],
|
||||||
|
"arms": arms, "paired_mean_ic_delta": float((matrix[:, 2] - matrix[:, 1]).mean()),
|
||||||
|
"paired_ic_delta_ci95": uncertainty["paired_delta_ci95"]}
|
||||||
|
|
||||||
|
|
||||||
|
def screen_pass(summaries: dict) -> dict:
|
||||||
|
full = summaries["full"]
|
||||||
|
means = {era: summary.get("arms", {}).get("normalized", {}).get("mean_ic")
|
||||||
|
for era, summary in summaries.items()}
|
||||||
|
checks = {
|
||||||
|
"mean_ic_ge_0_03": means["full"] is not None and means["full"] >= IC_FLOOR,
|
||||||
|
"at_least_12_windows": full["windows"] >= MIN_WINDOWS,
|
||||||
|
"pre_2021_positive": means["pre_2021"] is not None and means["pre_2021"] > 0,
|
||||||
|
"post_2021_positive": means["post_2021"] is not None and means["post_2021"] > 0,
|
||||||
|
}
|
||||||
|
return {"passed": all(checks.values()), "checks": checks,
|
||||||
|
"failed_checks": [k for k, v in checks.items() if not v]}
|
||||||
|
|
||||||
|
|
||||||
|
def run_screen(prices: dict, benchmark: dict) -> dict:
|
||||||
|
calendar = sorted(benchmark)
|
||||||
|
bench_index = {d: i for i, d in enumerate(calendar)}
|
||||||
|
weekly = weekly_indices(calendar)
|
||||||
|
observations = {i: [] for i in weekly}
|
||||||
|
reasons = Counter()
|
||||||
|
for count, (symbol, columns) in enumerate(sorted(prices.items()), 1):
|
||||||
|
dates = [date.fromordinal(int(o)) for o in columns[0]]
|
||||||
|
closes = columns[4]
|
||||||
|
index = {d: i for i, d in enumerate(dates)}
|
||||||
|
for asof in weekly:
|
||||||
|
i, forward = index.get(calendar[asof]), index.get(calendar[asof + HORIZON])
|
||||||
|
if i is None or forward is None:
|
||||||
|
reasons["missing_forward_or_asof"] += 1
|
||||||
|
continue
|
||||||
|
if any(not math.isfinite(closes[j]) or closes[j] <= 0 for j in (i, forward)):
|
||||||
|
reasons["invalid_forward_or_asof"] += 1
|
||||||
|
continue
|
||||||
|
signal = signal_at(dates, closes, i, benchmark, bench_index)
|
||||||
|
reasons[signal["reason"]] += 1
|
||||||
|
observations[asof].append({"symbol": symbol, **signal,
|
||||||
|
"forward": closes[forward] / closes[i] - 1})
|
||||||
|
if count % 50 == 0 or count == len(prices):
|
||||||
|
print(f"signal screen: {count}/{len(prices)} symbols", flush=True)
|
||||||
|
usable = [i for i in weekly if sum(o["normalized"] is not None for o in observations[i]) >= MIN_CROSS]
|
||||||
|
kept = nonoverlapping_indices(usable)
|
||||||
|
windows = []
|
||||||
|
for i in kept:
|
||||||
|
rows = observations[i]
|
||||||
|
arms = {}
|
||||||
|
for arm in ARMS:
|
||||||
|
pairs = [(o[arm], o["forward"]) for o in rows if o[arm] is not None]
|
||||||
|
ic = _spearman([p[0] for p in pairs], [p[1] for p in pairs])
|
||||||
|
if ic is None:
|
||||||
|
raise ValueError(f"Degenerate {arm} cross-section on {calendar[i]}")
|
||||||
|
arms[arm] = {"n": len(pairs), "ic": ic, "quintile_spread": _quintile_spread(pairs)}
|
||||||
|
# Raised, not asserted: python -O must not strip the pairing guarantee.
|
||||||
|
if arms["paired_control"]["n"] != arms["normalized"]["n"]:
|
||||||
|
raise AssertionError(f"Paired arms scored different stocks on {calendar[i]}")
|
||||||
|
windows.append({"asof": calendar[i].isoformat(),
|
||||||
|
"forward_end": calendar[i + HORIZON].isoformat(), "arms": arms})
|
||||||
|
summaries = {
|
||||||
|
"full": summarize(windows),
|
||||||
|
"pre_2021": summarize([w for w in windows if w["forward_end"] < ERA_SPLIT.isoformat()]),
|
||||||
|
"post_2021": summarize([w for w in windows if w["asof"] >= ERA_SPLIT.isoformat()]),
|
||||||
|
}
|
||||||
|
return {"summaries": summaries, "verdict": screen_pass(summaries), "windows": windows,
|
||||||
|
"coverage": {"registry_symbols": len(prices), "weekly_asofs_considered": len(weekly),
|
||||||
|
"usable_weekly_asofs": len(usable), "all_weekly_cell_reasons": dict(reasons),
|
||||||
|
"retained_unrestricted_cells": sum(w["arms"][ARMS[0]]["n"] for w in windows),
|
||||||
|
"retained_paired_cells": sum(w["arms"][ARMS[1]]["n"] for w in windows)},
|
||||||
|
"bootstrap": {"seed": SEED, "replicates": REPLICATES, "window_block": IC_BLOCK,
|
||||||
|
"generator": "NumPy PCG64", "interval": "percentile 95%"}}
|
||||||
|
|
||||||
|
|
||||||
|
def render_report(report: dict) -> str:
|
||||||
|
screen = report["screen"]
|
||||||
|
lines = ["# Residual normalization — signal screen", "",
|
||||||
|
f"Generated: {report['generated_at']}", "",
|
||||||
|
f"**Verdict: {'PASS' if screen['verdict']['passed'] else 'FAIL'}**", "",
|
||||||
|
"Previously inspected historical data; no production change or new out-of-sample claim.", "",
|
||||||
|
"| Era | Arm | Windows | Avg stocks | Mean IC | IC 95% interval | Quintile spread |",
|
||||||
|
"|---|---|---:|---:|---:|---|---:|"]
|
||||||
|
for era, summary in screen["summaries"].items():
|
||||||
|
for arm, row in summary["arms"].items():
|
||||||
|
ci = row["ic_ci95"]
|
||||||
|
ci_text = f"[{ci[0]:.4f}, {ci[1]:.4f}]" if ci else "unavailable"
|
||||||
|
lines.append(f"| {era} | {arm} | {summary['windows']} | {row['avg_cross_section']:.1f} "
|
||||||
|
f"| {row['mean_ic']:.4f} | {ci_text} | {100 * row['mean_quintile_spread']:.2f}% |")
|
||||||
|
lines.extend(["", "Paired normalized-minus-control IC differences:", ""])
|
||||||
|
for era, summary in screen["summaries"].items():
|
||||||
|
if summary["windows"]:
|
||||||
|
lines.append(f"- {era}: {summary['paired_mean_ic_delta']:.4f}; "
|
||||||
|
f"95% interval {summary['paired_ic_delta_ci95']}.")
|
||||||
|
lines.extend(["", f"Screen checks: `{json.dumps(screen['verdict']['checks'])}`", "",
|
||||||
|
f"Portfolio stage: **{report['portfolio_stage']}**.", "",
|
||||||
|
"Signal IC and quintile spreads are predictive diagnostics, not simulated strategy returns.",
|
||||||
|
"Intervals resample paired blocks of four non-overlapping forward windows; they remain exploratory.",
|
||||||
|
"The unrestricted control shows the effect of requiring complete formation history.", "",
|
||||||
|
"[Frozen protocol](../docs/research/residual-normalization.md)", ""])
|
||||||
|
return "\n".join(lines)
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
parser = argparse.ArgumentParser(description=__doc__)
|
||||||
|
parser.add_argument("--snapshot", type=Path, default=ROOT / "backtest_snapshots/research.sqlite")
|
||||||
|
parser.add_argument("--out", type=Path, default=ROOT / "reports/residual-normalization-20260908.json")
|
||||||
|
args = parser.parse_args()
|
||||||
|
snapshot = args.snapshot.resolve(strict=True)
|
||||||
|
before = sha256(snapshot)
|
||||||
|
code_paths = [Path(__file__).resolve(), PROTOCOL, ROOT / "app/services/momentum_service.py",
|
||||||
|
ROOT / "app/services/backtest_service.py"]
|
||||||
|
hashes = {p.relative_to(ROOT).as_posix(): git_blob_id(p) for p in code_paths}
|
||||||
|
prices, benchmark, manifest = read_snapshot(snapshot)
|
||||||
|
print(f"manifest verified: {len(prices)} production-universe names; snapshot {before[:16]}", flush=True)
|
||||||
|
screen = run_screen(prices, benchmark)
|
||||||
|
if sha256(snapshot) != before or any(git_blob_id(ROOT / p) != h for p, h in hashes.items()):
|
||||||
|
raise RuntimeError("Snapshot, protocol, or implementation changed during the run")
|
||||||
|
report = {
|
||||||
|
"schema_version": 1, "generated_at": datetime.now(timezone.utc).isoformat(),
|
||||||
|
"status": "complete_signal_screen", "snapshot": str(snapshot), "snapshot_sha256": before,
|
||||||
|
"source_git_blobs": hashes, "manifest": manifest, "python": sys.version,
|
||||||
|
"packages": {p: importlib.metadata.version(p) for p in ("numpy", "sqlalchemy")},
|
||||||
|
"screen": screen,
|
||||||
|
"portfolio_stage": "eligible_pending_replay" if screen["verdict"]["passed"] else "not_run_failed_signal_screen",
|
||||||
|
"production_changed": False,
|
||||||
|
}
|
||||||
|
args.out.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
args.out.write_text(json.dumps(report, indent=2, allow_nan=False) + "\n", encoding="utf-8")
|
||||||
|
args.out.with_suffix(".md").write_text(render_report(report), encoding="utf-8")
|
||||||
|
print(json.dumps({"verdict": screen["verdict"], "summaries": screen["summaries"],
|
||||||
|
"report": str(args.out)}, indent=2), flush=True)
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
@@ -0,0 +1,238 @@
|
|||||||
|
"""Economic invariants and calendar/coverage guards for normalization research."""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import sqlite3
|
||||||
|
from datetime import date, timedelta
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from app.services.momentum_service import compute_residual_12_1_momentum
|
||||||
|
from scripts import run_residual_normalization as study
|
||||||
|
|
||||||
|
|
||||||
|
def sessions(n=400):
|
||||||
|
days, day = [], date(2019, 1, 2)
|
||||||
|
while len(days) < n:
|
||||||
|
if day.weekday() < 5:
|
||||||
|
days.append(day)
|
||||||
|
day += timedelta(days=1)
|
||||||
|
return days
|
||||||
|
|
||||||
|
|
||||||
|
def series(beta=1.3, scale=1.0):
|
||||||
|
days = sessions()
|
||||||
|
rng = np.random.default_rng(59)
|
||||||
|
market = rng.normal(0.0003, 0.01, len(days) - 1)
|
||||||
|
specific = rng.normal(0.0008, 0.005, len(days) - 1)
|
||||||
|
spy = [100.0, *list(100 * np.cumprod(1 + market))]
|
||||||
|
stock = [100.0, *list(100 * np.cumprod(1 + beta * market + scale * specific))]
|
||||||
|
return days, stock, dict(zip(days, spy)), {d: i for i, d in enumerate(days)}
|
||||||
|
|
||||||
|
|
||||||
|
def test_numerator_is_production_and_denominator_uses_sample_variance():
|
||||||
|
days, stock, spy, index = series()
|
||||||
|
i = 300
|
||||||
|
got = study.signal_at(days, stock, i, spy, index)
|
||||||
|
assert got["paired_control"] == compute_residual_12_1_momentum(days[:i + 1], stock[:i + 1], spy)
|
||||||
|
s = np.diff(np.array(stock[i - 252:i - 21 + 1])) / np.array(stock[i - 252:i - 21])
|
||||||
|
mclose = np.array([spy[d] for d in days[i - 252:i - 21 + 1]])
|
||||||
|
m = np.diff(mclose) / mclose[:-1]
|
||||||
|
# Independent least-squares slope calculation, keeping alpha in the score.
|
||||||
|
intercept, beta = np.linalg.lstsq(np.column_stack([np.ones(len(m)), m]), s, rcond=None)[0]
|
||||||
|
assert intercept != 0
|
||||||
|
expected_sd = (s - beta * m).std(ddof=1)
|
||||||
|
assert got["residual_sd"] == pytest.approx(expected_sd)
|
||||||
|
assert got["normalized"] == pytest.approx(got["paired_control"] / expected_sd)
|
||||||
|
|
||||||
|
|
||||||
|
def test_normalization_removes_specific_return_scale_but_keeps_direction():
|
||||||
|
days, stock, spy, index = series(scale=1)
|
||||||
|
_, larger, _, _ = series(scale=3)
|
||||||
|
a = study.signal_at(days, stock, 350, spy, index)
|
||||||
|
b = study.signal_at(days, larger, 350, spy, index)
|
||||||
|
assert abs(a["paired_control"]) > 0.01
|
||||||
|
assert b["paired_control"] == pytest.approx(3 * a["paired_control"])
|
||||||
|
assert b["normalized"] == pytest.approx(a["normalized"])
|
||||||
|
assert np.sign(a["normalized"]) == np.sign(a["paired_control"])
|
||||||
|
|
||||||
|
|
||||||
|
def test_market_beta_change_does_not_become_specific_strength():
|
||||||
|
days, stock, spy, index = series(beta=0.5)
|
||||||
|
_, high_beta, _, _ = series(beta=2.5)
|
||||||
|
a = study.signal_at(days, stock, 350, spy, index)
|
||||||
|
b = study.signal_at(days, high_beta, 350, spy, index)
|
||||||
|
assert b["normalized"] == pytest.approx(a["normalized"])
|
||||||
|
|
||||||
|
|
||||||
|
def test_skipped_month_and_future_are_not_signal_inputs():
|
||||||
|
days, stock, spy, index = series()
|
||||||
|
i = 300
|
||||||
|
baseline = study.signal_at(days, stock, i, spy, index)
|
||||||
|
for k in range(i - 20, len(stock)):
|
||||||
|
stock[k] *= 1.8
|
||||||
|
spy[days[k]] *= 0.7
|
||||||
|
assert study.signal_at(days, stock, i, spy, index) == baseline
|
||||||
|
|
||||||
|
|
||||||
|
def test_missing_benchmark_pair_stays_only_in_unrestricted_control():
|
||||||
|
days, stock, spy, index = series()
|
||||||
|
del spy[days[200]]
|
||||||
|
index = {d: i for i, d in enumerate(sorted(spy))}
|
||||||
|
got = study.signal_at(days, stock, 350, spy, index)
|
||||||
|
assert got["unrestricted_control"] is not None
|
||||||
|
assert got["paired_control"] is None
|
||||||
|
assert got["normalized"] is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_missing_stock_session_cannot_be_called_daily_aligned():
|
||||||
|
days, stock, spy, index = series()
|
||||||
|
del days[200]
|
||||||
|
del stock[200]
|
||||||
|
got = study.signal_at(days, stock, 349, spy, index)
|
||||||
|
assert got["unrestricted_control"] is not None
|
||||||
|
assert got["reason"] == "unaligned_formation"
|
||||||
|
|
||||||
|
|
||||||
|
def test_perfect_market_tracker_is_not_an_infinite_score():
|
||||||
|
days, _, spy, index = series()
|
||||||
|
got = study.signal_at(days, [spy[d] for d in days], 350, spy, index)
|
||||||
|
assert got["normalized"] is None
|
||||||
|
assert got["reason"] == "zero_residual_variance"
|
||||||
|
|
||||||
|
|
||||||
|
def test_nonoverlap_uses_sessions_and_permits_only_a_shared_endpoint():
|
||||||
|
# Arbitrary indices stand in for a calendar with holidays/year boundaries.
|
||||||
|
candidates = [100, 104, 109, 114, 119, 123, 128, 133, 138, 143, 148, 153, 158, 163]
|
||||||
|
assert study.nonoverlapping_indices(candidates) == [100, 133, 163]
|
||||||
|
|
||||||
|
|
||||||
|
def test_weekly_endpoint_not_moved_earlier_to_obtain_forward_data():
|
||||||
|
days = sessions(100)
|
||||||
|
got = study.weekly_indices(days)
|
||||||
|
last_per_week = {}
|
||||||
|
for i, day in enumerate(days):
|
||||||
|
last_per_week[day.isocalendar()[:2]] = i
|
||||||
|
assert all(last_per_week[days[i].isocalendar()[:2]] == i for i in got)
|
||||||
|
assert all(i + study.HORIZON < len(days) for i in got)
|
||||||
|
|
||||||
|
|
||||||
|
def test_identical_columns_have_zero_paired_bootstrap_difference():
|
||||||
|
x = np.linspace(-0.1, 0.2, 25)
|
||||||
|
matrix = np.column_stack([x, x, x])
|
||||||
|
got = study.bootstrap_means(matrix)
|
||||||
|
assert got["paired_delta_ci95"] == [0.0, 0.0]
|
||||||
|
assert got == study.bootstrap_means(matrix)
|
||||||
|
|
||||||
|
|
||||||
|
def test_screen_uses_unrounded_floor_and_requires_both_eras():
|
||||||
|
def summaries(value, pre=0.01, post=0.04):
|
||||||
|
return {era: {"windows": 20, "arms": {"normalized": {"mean_ic": mean}}}
|
||||||
|
for era, mean in (("full", value), ("pre_2021", pre), ("post_2021", post))}
|
||||||
|
assert not study.screen_pass(summaries(0.0299999))["passed"]
|
||||||
|
assert study.screen_pass(summaries(0.03))["passed"]
|
||||||
|
assert not study.screen_pass(summaries(0.04, pre=-0.001))["passed"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_screen_keeps_paired_stocks_and_forward_endpoints_identical(monkeypatch):
|
||||||
|
days = sessions()
|
||||||
|
benchmark = {d: 100 + i for i, d in enumerate(days)}
|
||||||
|
prices = {}
|
||||||
|
for k in range(25):
|
||||||
|
closes = [100 + k + i * (k + 1) / 100 for i in range(len(days))]
|
||||||
|
prices[f"S{k:02}"] = ([d.toordinal() for d in days], [], [], [], closes, [])
|
||||||
|
|
||||||
|
def fake_signal(dates, closes, i, spy, index):
|
||||||
|
value = closes[0]
|
||||||
|
# One name remains in the unrestricted control but not the paired set.
|
||||||
|
excluded = value == 100
|
||||||
|
return {"unrestricted_control": value, "paired_control": None if excluded else value,
|
||||||
|
"normalized": None if excluded else value / 2,
|
||||||
|
"reason": "unaligned_formation" if excluded else "eligible"}
|
||||||
|
|
||||||
|
monkeypatch.setattr(study, "signal_at", fake_signal)
|
||||||
|
report = study.run_screen(prices, benchmark)
|
||||||
|
assert report["windows"]
|
||||||
|
index = {d.isoformat(): i for i, d in enumerate(days)}
|
||||||
|
previous_end = None
|
||||||
|
for window in report["windows"]:
|
||||||
|
assert index[window["forward_end"]] - index[window["asof"]] == 30
|
||||||
|
if previous_end:
|
||||||
|
assert window["asof"] >= previous_end
|
||||||
|
previous_end = window["forward_end"]
|
||||||
|
assert window["arms"]["unrestricted_control"]["n"] == 25
|
||||||
|
assert window["arms"]["paired_control"]["n"] == 24
|
||||||
|
assert window["arms"]["normalized"]["n"] == 24
|
||||||
|
assert window["arms"]["normalized"]["ic"] == window["arms"]["paired_control"]["ic"]
|
||||||
|
|
||||||
|
|
||||||
|
def write_snapshot(path, closes):
|
||||||
|
"""Minimal snapshot: one ticker, SPY, and a matching completion manifest."""
|
||||||
|
days = sessions(len(closes))
|
||||||
|
con = sqlite3.connect(path)
|
||||||
|
con.executescript(
|
||||||
|
"CREATE TABLE tickers (id INTEGER, symbol TEXT);"
|
||||||
|
"CREATE TABLE research_rank_only (symbol TEXT);"
|
||||||
|
"CREATE TABLE ohlcv_records (ticker_id INTEGER, date TEXT, open REAL, high REAL,"
|
||||||
|
" low REAL, close REAL, volume REAL);"
|
||||||
|
"CREATE TABLE benchmark_prices (symbol TEXT, date TEXT, close REAL);"
|
||||||
|
"INSERT INTO tickers VALUES (1, 'AAA');"
|
||||||
|
)
|
||||||
|
con.executemany(
|
||||||
|
"INSERT INTO ohlcv_records VALUES (1, ?, ?, ?, ?, ?, NULL)",
|
||||||
|
[(d.isoformat(), c, c, c, c) for d, c in zip(days, closes)],
|
||||||
|
)
|
||||||
|
con.executemany(
|
||||||
|
"INSERT INTO benchmark_prices VALUES ('SPY', ?, 100.0)", [(d.isoformat(),) for d in days]
|
||||||
|
)
|
||||||
|
con.commit()
|
||||||
|
con.close()
|
||||||
|
Path(str(path) + ".manifest.json").write_text(json.dumps({
|
||||||
|
"complete": True, "ticker_count": 1, "ohlcv_row_count": len(closes), "rank_only_count": 0,
|
||||||
|
}))
|
||||||
|
|
||||||
|
|
||||||
|
def test_missing_price_stops_the_run_instead_of_reading_as_zero(tmp_path):
|
||||||
|
path = tmp_path / "snap.sqlite"
|
||||||
|
write_snapshot(path, [100.0, None, 102.0])
|
||||||
|
with pytest.raises(ValueError, match="Missing OHLC price: AAA"):
|
||||||
|
study.read_snapshot(path)
|
||||||
|
|
||||||
|
|
||||||
|
def test_snapshot_read_keeps_missing_volume_and_releases_the_file(tmp_path):
|
||||||
|
path = tmp_path / "snap.sqlite"
|
||||||
|
write_snapshot(path, [100.0, 101.0, 102.0])
|
||||||
|
prices, benchmark, _ = study.read_snapshot(path)
|
||||||
|
assert prices["AAA"][4] == [100.0, 101.0, 102.0]
|
||||||
|
assert prices["AAA"][5] == [0.0, 0.0, 0.0]
|
||||||
|
assert len(benchmark) == 3
|
||||||
|
path.unlink() # Windows refuses while a connection still holds the file.
|
||||||
|
|
||||||
|
|
||||||
|
def test_paired_coverage_mismatch_raises_rather_than_asserts(monkeypatch):
|
||||||
|
days = sessions()
|
||||||
|
benchmark = {d: 100 + i for i, d in enumerate(days)}
|
||||||
|
prices = {
|
||||||
|
f"S{k:02}": ([d.toordinal() for d in days], [], [], [],
|
||||||
|
[100 + k + i * (k + 1) / 100 for i in range(len(days))], [])
|
||||||
|
for k in range(25)
|
||||||
|
}
|
||||||
|
|
||||||
|
def broken_pairing(dates, closes, i, spy, index):
|
||||||
|
value = closes[0]
|
||||||
|
return {"unrestricted_control": value, "paired_control": value,
|
||||||
|
"normalized": None if value == 100 else value / 2, "reason": "eligible"}
|
||||||
|
|
||||||
|
monkeypatch.setattr(study, "signal_at", broken_pairing)
|
||||||
|
with pytest.raises(AssertionError, match="Paired arms scored different stocks"):
|
||||||
|
study.run_screen(prices, benchmark)
|
||||||
|
|
||||||
|
|
||||||
|
def test_source_ids_are_git_blob_ids_whatever_the_line_endings(tmp_path):
|
||||||
|
lf, crlf = tmp_path / "lf.txt", tmp_path / "crlf.txt"
|
||||||
|
lf.write_bytes(b"hello\n")
|
||||||
|
crlf.write_bytes(b"hello\r\n")
|
||||||
|
# `printf 'hello\n' | git hash-object --stdin`
|
||||||
|
assert study.git_blob_id(lf) == "ce013625030ba8dba906f756967f9e9ca394464a"
|
||||||
|
assert study.git_blob_id(crlf) == study.git_blob_id(lf)
|
||||||
Reference in new issue
Block a user