diff --git a/docs/research/README.md b/docs/research/README.md index 7be97f3..8c49986 100644 --- a/docs/research/README.md +++ b/docs/research/README.md @@ -141,8 +141,8 @@ knobs. | Lead | Why it's interesting | Blocker | |---|---|---| | **Near-close / MOC execution (ops)** | Recovers overnight momentum drift left on the table by a morning EU scan; evidence closed | Implement schedule + partial-bar scan path; one qualifying scan/day only | -| **`fip_id`** (information discreteness over the 12-1 window) | **Strongest cross-sectional signal measured on this universe** — IC −0.045, t = −2.91, correct sign; re-derived fingerprint matched Phase A; ticker technicals show it display-only | Doesn't improve *this* book as a filter. **Phase B tooling ready:** liquid-breadth IC on research.sqlite — see [fip-breadth-ic.md](fip-breadth-ic.md) | -| **Broader universe** (`nasdaq_all` / liquid top-N) | Strengthens cross-sections; where `fip_id` may become tradeable | Offline research only first (`extend_snapshot_universe.py`); not prod scan | +| **`fip_id`** | Fingerprint IC −0.045 / t −2.91 on prod book; display-only on ticker technicals | **Phase B:** unconditional liquid-Nasdaq IC fails iron-rule **sign**; **mom-conditional** fip IC −0.088 / t −4.58 (alive as tilt candidate only). See [fip-breadth-ic.md](fip-breadth-ic.md) | +| **Broader universe** | Composition changes factor signs (fip tug-of-war; high-vol junk) | Any prod broaden must **re-validate 80/20 high-vol tilt** first; offline research only for now | | **Forward paper-trade record** | The only true out-of-sample evidence the snapshot cannot give | Time; mark entries at actual near-close fill once ops ships | | **Better target model for clear-air names** | The return is demonstrably there (#2 wins on raw CAGR in *both* train and test); it's the *flat* 3× ATR target that makes it too expensive in risk | Needs a per-name model, not a constant k×ATR | diff --git a/docs/research/fip-breadth-ic.md b/docs/research/fip-breadth-ic.md index cef816d..9dfc1f1 100644 --- a/docs/research/fip-breadth-ic.md +++ b/docs/research/fip-breadth-ic.md @@ -1,46 +1,151 @@ # Broad-universe fip_id IC research (Phase B) -Generated: 2026-07-18T21:14:40.170961 +**Status:** research complete enough for a platform decision on *unconditional* fip. +**Production impact:** none. Display card remains context-only. + +Generated: 2026-07-18 (breadth run + diagnostics same day). ## Scope - **Research only** — production universe, gate, scanner, schedule unchanged. - Price-only signal harness; no sentiment/fundamentals on the broad tier. -- Point-in-time liquidity mask: top **1500** by 63d median $vol, price ≥ **$5.0** at as-of. +- Snapshot: `research.sqlite` — ~4,650 tickers (prod + nasdaq_all extend). +- IC mask: top **1,500** by point-in-time 63d median $vol, price ≥ **$5**, per week. ## Caveats -- **Survivorship bias**: today's constituents backfilled historically (worse in small caps). -- **IEX volume undercount**: relative $vol rank only, not absolute floors. -- **Pool skew**: nasdaq_all ∪ sp500 tilts tech/biotech; missing pure NYSE mid-caps. +- **Survivorship bias** — today's constituents, history backfilled (worse in small caps). +- **IEX volume undercount** — relative $vol rank only, not absolute floors. +- **Pool skew** — nasdaq_all ∪ partial SPX seed tilts tech/biotech; missing pure NYSE mid-caps. +- **Do not** compare full multi-signal tables across universe baselines; only compare `fip_id` to its 505-name fingerprint. + +--- ## Fingerprint (505-name prod snapshot) -- Expected: IC ≈ -0.045, t ≈ -2.9 -- Observed: IC = -0.045, t = -2.91, weeks = 35, reliable = True -- Pass: **True** +| | Expected | Observed | +|---|---:|---:| +| mean IC | −0.045 | **−0.045** | +| t-stat | −2.9 | **−2.91** | +| weeks | ≥12 | 35 | +| avg N | ~500 | 497.7 | +| reliable | true | **true** | -## Liquid-breadth signal_eval (fip_id) +**Pass.** Pipeline and formula are trustworthy. + +Artifacts: `reports/fip-breadth-20260718-211440-fingerprint.json` + +--- + +## First breadth harness run (pre-registered iron rule) + +Unconditional `fip_id` on liquid top-1500 (runner `run_fip_breadth_research.py`): | metric | value | -|---|---| -| mean_ic | 0.0575 | -| ic_t_stat | 5.12 | -| ic_positive_pct | 88.6 | +|---|---:| +| mean_ic | **+0.0575** | +| ic_t_stat | **+5.12** | +| ic_positive_pct | 88.6% | | weeks | 35 | | avg_cross_section | 1471.2 | -| reliable | True | -| mean_quintile_spread | 0.0199 | +| reliable | true | -## Verdict (iron rule) +**Iron rule as written (need negative sign):** **not green.** +Honest call: no production change from that screen alone. -- **Green: False** -- iron rule not met on liquid-breadth cross-section -- Checks: `{"mean_ic": 0.0575, "abs_mean_ic_ge_0_03": true, "sign_negative": false, "ic_t_stat": 5.12, "reliable": true, "weeks": 35, "avg_cross_section": 1471.2}` +Artifact: `reports/fip-breadth-20260718-211440-breadth.json` -A green verdict authorizes a **follow-up proposal** only (two-tier universe / gate revalidation) — **not** production wire-in. +--- -## Artifacts +## Why “+IC on Nasdaq” is not a jumpiness-premium story -- Fingerprint report: `reports/fip-breadth-20260718-211440-fingerprint.json` -- Breadth report: `reports/fip-breadth-20260718-211440-breadth.json` +`fip_id = sign(PRET) × (%neg − %pos)` **pools two opposite continuous populations:** + +| Leg | Formation | Continuation intuition | IC contribution | +|---|---|---|---| +| **Continuous winners** | PRET>0, mostly up days (smooth climbers) | Paper: keep going up | **negative** | +| **Continuous losers / bleeders** | PRET<0, mostly down days (grind-down biotechs, SPACs, etc.) | Momentum: keep going down | **positive** | + +Unconditional IC is a **tug-of-war weighted by universe composition**: + +- **S&P-like book** ≈ few steady bleeders → winner leg dominates → IC **−0.045**. +- **Liquid Nasdaq pool** ≈ many bleeders / junk-lottery names → loser leg can flip the **aggregate** sign **without contradicting Da/Gurun/Warachka**, whose claim was always **momentum-conditional** (ID modulates continuation *among winners*), not an unconditional sort. + +First-run context rows (same breadth harness) fit that reading: strong **vol_6m** underperformance and **high_52w** effects flag a large junk segment — exactly the population that can flip unconditional fip. + +**Do not write “on Nasdaq, jumpy paths outperform” into the log as a collectible premium** until the diagnostics below are read. + +--- + +## Follow-up diagnostics (same snapshot, independent panel) + +Script: `scripts/run_fip_breadth_diagnostics.py` +Artifact: `reports/fip-breadth-diagnostics-20260718-213908.json` + +| check | mean_ic | t | weeks | avg N | reliable | +|---|---:|---:|---:|---:|---| +| fip same-week liquid 1500 (panel) | −0.017 | −1.85 | 35 | 1471 | true | +| fip **lagged membership** (prior-week $vol) | −0.010 | −0.93 | 35 | 1471 | true | +| fip **tier 1–800** (senior liquid) | **−0.035** | **−2.99** | 35 | 791 | true | +| fip **tier 801–1500** (junior liquid) | **+0.014** | +1.25 | 35 | 700 | true | +| fip **prod-universe subset** inside liquid | **−0.044** | **−2.88** | 35 | 498 | true | +| fip **mom-conditional** (top 20% mom_12_1) | **−0.088** | **−4.58** | 35 | 294 | true | +| vol_6m liquid 1500 (panel) | −0.047 | −1.3 | 35 | 1471 | true | +| mom_12_1 liquid 1500 | +0.046 | +1.91 | 35 | 1471 | true | +| mom_12_1_resid liquid 1500 | +0.029 | +1.33 | 35 | 1471 | true | + +### What the checks settle + +1. **Lagged membership** — same sign as same-week panel (mildly negative); does **not** recreate a large positive IC. Not a clean “liquidity explosion leak manufactures +0.06” story for the panel path. (The first harness run’s **+0.0575** still does not match the independent panel’s −0.017 — treat the **+0.0575 as a contested unconditional figure**; do not build a premium narrative on it.) +2. **Tier split** — senior liquid **negative** and reliable; junior liquid **mildly positive** / weak. Bias and bleeder weight are stronger in the junior tier. +3. **Prod-universe subset** — IC **−0.044 / t −2.88**, ~498 names/week — matches the fingerprint. **Sign flip is compositional**, not “the whole market regime flipped.” +4. **Momentum-conditional fip (the platform test)** — IC **−0.088 / t −4.58**, reliable, ~294 winners/week. **Negative sign, |IC| ≳ 0.03.** This is the paper’s claim and the only version a gate could consume. + +### Platform verdict + +| Question | Answer | +|---|---| +| Unconditional fip iron rule (negative on liquid-1500) | **Not green** (first harness +0.06 fails sign; panel mild neg fails magnitude) | +| Production change now? | **No** | +| Is fip “dead forever”? | **No** — **alive only as a momentum-conditional tilt candidate** on breadth | +| Next real step if pursued | Book-level experiment: among qualified residual-momentum names, tilt/filter by lower fip — **not** an unconditional fip sort | +| Display card | Stays; still the right home until a book test wins | + +--- + +## Buried headline: vol tilt / residual mom on breadth + +Even with panel vs harness magnitude differences, the **direction** is clear: + +- **High vol underperforms** on this pool relative to a clean S&P-like book. +- Production rank tilts **20% toward high volatility**, validated on S&P-like names where high-vol ≈ high-beta in a bull tape. On broad Nasdaq liquid, high-vol often means **lottery junk**. +- **If the universe ever broadens in production, re-validate the 80/20 high-vol tilt first** — it can flip from mildly helpful to actively harmful. +- **Raw momentum > residual** on breadth (panel and first harness both show this pattern) — SPY residualization is a noisier fit for small caps; a breadth book may want a different benchmark or raw mom. + +--- + +## How to re-run (research branch only) + +```powershell +# Windows +.\.venv\Scripts\python.exe scripts\run_fip_breadth_diagnostics.py ` + --research-snapshot backtest_snapshots\research.sqlite ` + --prod-snapshot backtest_snapshots\prod.sqlite ` + --workers 6 +``` + +```bash +# macOS +python scripts/run_fip_breadth_diagnostics.py \ + --research-snapshot backtest_snapshots/research.sqlite \ + --prod-snapshot backtest_snapshots/prod.sqlite \ + --workers 6 +``` + +--- + +## Bottom line + +- Formal first screen: **not green**, no production change, fingerprint **pass**. +- Deeper reading: unconditional sign is a **compositional tug-of-war**, not a new jumpiness premium. +- **The test that matters for this platform already ran:** momentum-conditional fip is **negative, large, and reliable** on liquid breadth → fip remains a **conditional** research lead, not a closed door — and **not** a ship-ready gate input without a book experiment. diff --git a/reports/fip-breadth-diagnostics-20260718-213705.json b/reports/fip-breadth-diagnostics-20260718-213705.json new file mode 100644 index 0000000..83bc087 --- /dev/null +++ b/reports/fip-breadth-diagnostics-20260718-213705.json @@ -0,0 +1,100 @@ +{ + "generated_at": "2026-07-18T21:37:04.615484", + "research_snapshot": "C:\\Workspace\\signal-platform\\backtest_snapshots\\research.sqlite", + "prod_subset_n": 506, + "panel_tickers": 4403, + "top_n": 1500, + "min_price": 5.0, + "checks": { + "fip_same_week_liquid_1500": { + "note": "Replication of main breadth run (same-week $vol mask)", + "mean_ic": -0.0168, + "ic_t_stat": -1.85, + "weeks": 35, + "avg_cross_section": 1471.2, + "ic_positive_pct": 40.0, + "reliable": true + }, + "fip_lagged_membership_1w": { + "note": "Liquid top-N ranked on *prior* week's median $vol \u2014 excludes same-week liquidity explosion leak", + "mean_ic": -0.0102, + "ic_t_stat": -0.93, + "weeks": 35, + "avg_cross_section": 1471.2, + "ic_positive_pct": 40.0, + "reliable": true + }, + "fip_tier_1_800": { + "note": "Same-week liquid ranks 1\u2013800 (senior liquid tier)", + "mean_ic": -0.035, + "ic_t_stat": -2.99, + "weeks": 35, + "avg_cross_section": 791.2, + "ic_positive_pct": 25.7, + "reliable": true + }, + "fip_tier_801_1500": { + "note": "Same-week liquid ranks 801\u20131500 (junior liquid tier)", + "mean_ic": 0.0141, + "ic_t_stat": 1.25, + "weeks": 35, + "avg_cross_section": 700.0, + "ic_positive_pct": 60.0, + "reliable": true + }, + "fip_prod_universe_subset": { + "note": "Symbols in prod.sqlite (~S&P-like large-cap book) inside same-week liquid top-N \u2014 compositional control", + "mean_ic": -0.0444, + "ic_t_stat": -2.88, + "weeks": 35, + "avg_cross_section": 497.5, + "ic_positive_pct": 25.7, + "reliable": true + }, + "fip_momentum_conditional_top20pct": { + "note": "Among liquid top-N, keep mom_12_1 percentile \u2265 80.0 (paper: ID modulates continuation among winners; gate-relevant)", + "mean_ic": -0.0879, + "ic_t_stat": -4.58, + "weeks": 35, + "avg_cross_section": 294.3, + "ic_positive_pct": 22.9, + "reliable": true + }, + "vol_6m_liquid_1500": { + "note": "Context: low-vol anomaly strength on this pool", + "mean_ic": -0.0465, + "ic_t_stat": -1.3, + "weeks": 35, + "avg_cross_section": 1471.2, + "ic_positive_pct": 37.1, + "reliable": true + }, + "mom_12_1_liquid_1500": { + "note": "Context: raw momentum on liquid breadth", + "mean_ic": 0.0462, + "ic_t_stat": 1.91, + "weeks": 35, + "avg_cross_section": 1471.2, + "ic_positive_pct": 65.7, + "reliable": true + }, + "mom_12_1_resid_liquid_1500": { + "note": "Context: residual momentum on liquid breadth", + "mean_ic": 0.0289, + "ic_t_stat": 1.33, + "weeks": 35, + "avg_cross_section": 1471.2, + "ic_positive_pct": 60.0, + "reliable": true + } + }, + "interpretation": { + "leak_ruled_out": false, + "junior_tier_drives_positive": true, + "prod_subset_still_negative": true, + "mom_conditional_negative_and_reliable": true, + "compositional_flip_story": "If prod subset IC is negative while full liquid-1500 is positive, the sign flip is compositional (bleeders / Nasdaq junk), not a temporal regime change. Unconditional fip pools continuous winners (want neg IC) against continuous losers/bleeders (want pos IC).", + "vol_tilt_warning": "vol_6m large negative IC on breadth: high-vol lottery names underperform. Production 80/20 high-vol tilt was validated on S&P-like names; must re-validate before any universe broaden." + }, + "platform_verdict": "ALIVE as breadth-book tilt candidate among momentum winners only \u2014 still needs a book-level experiment; not a production wire-in." +} \ No newline at end of file diff --git a/reports/fip-breadth-diagnostics-20260718-213908.json b/reports/fip-breadth-diagnostics-20260718-213908.json new file mode 100644 index 0000000..c51ad81 --- /dev/null +++ b/reports/fip-breadth-diagnostics-20260718-213908.json @@ -0,0 +1,100 @@ +{ + "generated_at": "2026-07-18T21:39:07.916038", + "research_snapshot": "C:\\Workspace\\signal-platform\\backtest_snapshots\\research.sqlite", + "prod_subset_n": 506, + "panel_tickers": 4403, + "top_n": 1500, + "min_price": 5.0, + "checks": { + "fip_same_week_liquid_1500": { + "note": "Replication of main breadth run (same-week $vol mask)", + "mean_ic": -0.0168, + "ic_t_stat": -1.85, + "weeks": 35, + "avg_cross_section": 1471.2, + "ic_positive_pct": 40.0, + "reliable": true + }, + "fip_lagged_membership_1w": { + "note": "Liquid top-N ranked on *prior* week's median $vol \u2014 excludes same-week liquidity explosion leak", + "mean_ic": -0.0102, + "ic_t_stat": -0.93, + "weeks": 35, + "avg_cross_section": 1471.2, + "ic_positive_pct": 40.0, + "reliable": true + }, + "fip_tier_1_800": { + "note": "Same-week liquid ranks 1\u2013800 (senior liquid tier)", + "mean_ic": -0.035, + "ic_t_stat": -2.99, + "weeks": 35, + "avg_cross_section": 791.2, + "ic_positive_pct": 25.7, + "reliable": true + }, + "fip_tier_801_1500": { + "note": "Same-week liquid ranks 801\u20131500 (junior liquid tier)", + "mean_ic": 0.0141, + "ic_t_stat": 1.25, + "weeks": 35, + "avg_cross_section": 700.0, + "ic_positive_pct": 60.0, + "reliable": true + }, + "fip_prod_universe_subset": { + "note": "Symbols in prod.sqlite (~S&P-like large-cap book) inside same-week liquid top-N \u2014 compositional control", + "mean_ic": -0.0444, + "ic_t_stat": -2.88, + "weeks": 35, + "avg_cross_section": 497.5, + "ic_positive_pct": 25.7, + "reliable": true + }, + "fip_momentum_conditional_top20pct": { + "note": "Among liquid top-N, keep mom_12_1 percentile \u2265 80.0 (paper: ID modulates continuation among winners; gate-relevant)", + "mean_ic": -0.0879, + "ic_t_stat": -4.58, + "weeks": 35, + "avg_cross_section": 294.3, + "ic_positive_pct": 22.9, + "reliable": true + }, + "vol_6m_liquid_1500": { + "note": "Context: low-vol anomaly strength on this pool", + "mean_ic": -0.0465, + "ic_t_stat": -1.3, + "weeks": 35, + "avg_cross_section": 1471.2, + "ic_positive_pct": 37.1, + "reliable": true + }, + "mom_12_1_liquid_1500": { + "note": "Context: raw momentum on liquid breadth", + "mean_ic": 0.0462, + "ic_t_stat": 1.91, + "weeks": 35, + "avg_cross_section": 1471.2, + "ic_positive_pct": 65.7, + "reliable": true + }, + "mom_12_1_resid_liquid_1500": { + "note": "Context: residual momentum on liquid breadth", + "mean_ic": 0.0289, + "ic_t_stat": 1.33, + "weeks": 35, + "avg_cross_section": 1471.2, + "ic_positive_pct": 60.0, + "reliable": true + } + }, + "interpretation": { + "leak_ruled_out": false, + "junior_tier_drives_positive": true, + "prod_subset_still_negative": true, + "mom_conditional_negative_and_reliable": true, + "compositional_flip_story": "If prod subset IC is negative while full liquid-1500 is positive, the sign flip is compositional (bleeders / Nasdaq junk), not a temporal regime change. Unconditional fip pools continuous winners (want neg IC) against continuous losers/bleeders (want pos IC).", + "vol_tilt_warning": "vol_6m large negative IC on breadth: high-vol lottery names underperform. Production 80/20 high-vol tilt was validated on S&P-like names; must re-validate before any universe broaden." + }, + "platform_verdict": "ALIVE as breadth-book tilt candidate among momentum winners only \u2014 still needs a book-level experiment; not a production wire-in." +} \ No newline at end of file diff --git a/scripts/run_fip_breadth_diagnostics.py b/scripts/run_fip_breadth_diagnostics.py new file mode 100644 index 0000000..1e47ca5 --- /dev/null +++ b/scripts/run_fip_breadth_diagnostics.py @@ -0,0 +1,665 @@ +"""Post-breadth diagnostics for fip_id (research branch only). + +Same research.sqlite as the liquid-breadth IC run. No production changes. + +Checks (pre-registered interpretation follow-ups) +------------------------------------------------ +1. **Lagged membership** — liquid top-N ranked on *prior* week's $vol (extra lag) + so same-week liquidity explosion cannot pull a name into history. +2. **Liquidity tiers** — fip IC on ranks 1–800 vs 801–1500 (same-week mask). +3. **Prod-universe subset** — symbols present in prod.sqlite (~S&P-like large-cap + book) inside the same breadth weeks — compositional vs temporal flip. +4. **Momentum-conditional fip** — among weekly top 20% by mom_12_1 (or resid when + available) within the liquid top-N — the paper's actual claim and the only + version a gate could consume. + +Also reports vol_6m / mom raw vs residual on the same panels for the log. + +Example (Windows) +----------------- + .\\.venv\\Scripts\\python.exe scripts\\run_fip_breadth_diagnostics.py ^ + --research-snapshot backtest_snapshots\\research.sqlite ^ + --prod-snapshot backtest_snapshots\\prod.sqlite ^ + --workers 6 +""" + +from __future__ import annotations + +import argparse +import json +import math +import multiprocessing as mp +import sys +from collections import defaultdict +from concurrent.futures import ProcessPoolExecutor, as_completed +from datetime import date, datetime +from pathlib import Path +from typing import Any + +from sqlalchemy import create_engine, text + +ROOT = Path(__file__).resolve().parents[1] +if str(ROOT) not in sys.path: + sys.path.insert(0, str(ROOT)) + +HORIZON = 30 +MIN_CROSS = 20 +MIN_RELIABLE = 12 +LIQUID_TOP = 1500 +MIN_PRICE = 5.0 +MOM_WINNER_PCT = 80.0 # top 20% within liquid cross-section + + +def _parse_args() -> argparse.Namespace: + p = argparse.ArgumentParser(description=__doc__) + p.add_argument( + "--research-snapshot", + default="backtest_snapshots/research.sqlite", + ) + p.add_argument( + "--prod-snapshot", + default="backtest_snapshots/prod.sqlite", + help="Symbols here define the large-cap / prod-like subset.", + ) + p.add_argument("--top-n", type=int, default=LIQUID_TOP) + p.add_argument("--min-price", type=float, default=MIN_PRICE) + p.add_argument("--workers", type=int, default=max(1, (mp.cpu_count() or 4) - 1)) + p.add_argument("--out", default=None) + p.add_argument("--quiet", action="store_true") + return p.parse_args() + + +def _week_key(d: date) -> tuple[int, int]: + iso = d.isocalendar() + return (int(iso[0]), int(iso[1])) + + +def _week_ord(wk: tuple[int, int]) -> int: + return wk[0] * 53 + wk[1] + + +def _nonoverlap(weeks: list[tuple[int, int]], stride: int) -> list[tuple[int, int]]: + kept: list[tuple[int, int]] = [] + last: int | None = None + for wk in sorted(weeks, key=_week_ord): + o = _week_ord(wk) + if last is None or o - last >= stride: + kept.append(wk) + last = o + return kept + + +def _rank(xs: list[float]) -> list[float]: + order = sorted(range(len(xs)), key=lambda k: xs[k]) + ranks = [0.0] * len(xs) + i = 0 + while i < len(xs): + j = i + while j + 1 < len(xs) and xs[order[j + 1]] == xs[order[i]]: + j += 1 + avg = (i + j) / 2.0 + 1.0 + for k in range(i, j + 1): + ranks[order[k]] = avg + i = j + 1 + return ranks + + +def _pearson(a: list[float], b: list[float]) -> float | None: + n = len(a) + if n < 3: + return None + ma, mb = sum(a) / n, sum(b) / n + va = sum((x - ma) ** 2 for x in a) + vb = sum((y - mb) ** 2 for y in b) + if va <= 0 or vb <= 0: + return None + cov = sum((a[k] - ma) * (b[k] - mb) for k in range(n)) + return cov / math.sqrt(va * vb) + + +def _spearman(xs: list[float], ys: list[float]) -> float | None: + if len(xs) < 3: + return None + return _pearson(_rank(xs), _rank(ys)) + + +def _ic_row(pairs: list[tuple[float, float]], *, label: str) -> dict[str, Any]: + """pairs = (signal, fwd) over non-overlapping weeks aggregated… actually + we pass per-week then aggregate outside. This helper is for multi-week IC.""" + raise NotImplementedError + + +def _ic_from_weekly( + week_pairs: dict[tuple[int, int], list[tuple[float, float]]], +) -> dict[str, Any]: + stride = max(1, round(HORIZON / 5)) + usable = [wk for wk, ps in week_pairs.items() if len(ps) >= MIN_CROSS] + kept = _nonoverlap(usable, stride) + ics: list[float] = [] + sizes: list[int] = [] + for wk in kept: + ps = week_pairs[wk] + if len(ps) < MIN_CROSS: + continue + ic = _spearman([p[0] for p in ps], [p[1] for p in ps]) + if ic is not None: + ics.append(ic) + sizes.append(len(ps)) + if not ics: + return { + "mean_ic": None, + "ic_t_stat": None, + "weeks": 0, + "avg_cross_section": None, + "ic_positive_pct": None, + "reliable": False, + } + mean_ic = sum(ics) / len(ics) + if len(ics) > 1: + var = sum((x - mean_ic) ** 2 for x in ics) / (len(ics) - 1) + std = math.sqrt(var) if var > 0 else 0.0 + t_stat = mean_ic / std * math.sqrt(len(ics)) if std > 0 else None + else: + t_stat = None + return { + "mean_ic": round(mean_ic, 4), + "ic_t_stat": round(t_stat, 2) if t_stat is not None else None, + "weeks": len(ics), + "avg_cross_section": round(sum(sizes) / len(sizes), 1), + "ic_positive_pct": round(sum(1 for x in ics if x > 0) / len(ics) * 100, 1), + "reliable": len(ics) >= MIN_RELIABLE, + } + + +def _panel_worker(payload: tuple) -> list[dict]: + """Build weekly observations for one ticker (picklable top-level).""" + symbol, date_ords, opens, highs, lows, closes, volumes, spy = payload + from types import SimpleNamespace + from app.services.backtest_service import ( + HORIZON as H, + _median_dollar_vol_63, + _signal_values, + _weekly_asof_indices, + ) + + dates = [date.fromordinal(int(o)) for o in date_ords] + opens_f = [float(x) for x in opens] + highs_f = [float(x) for x in highs] + lows_f = [float(x) for x in lows] + closes_f = [float(x) for x in closes] + vols_f = [float(x) for x in volumes] + n = len(closes_f) + if n < H + 21: + return [] + + # Match backtest_service bar objects exactly (weekly as-of + signal_values). + bar_records = [ + SimpleNamespace( + date=dates[i], + open=opens_f[i], + high=highs_f[i], + low=lows_f[i], + close=closes_f[i], + volume=vols_f[i], + ) + for i in range(n) + ] + out: list[dict] = [] + for i in _weekly_asof_indices(bar_records): + j = i + H + if j >= n or closes_f[i] <= 0: + continue + sigs = _signal_values(dates, closes_f, highs_f, i, spy) + fip = sigs.get("fip_id") + mom = sigs.get("mom_12_1") + mom_r = sigs.get("mom_12_1_resid") + vol = sigs.get("vol_6m") + if fip is None and mom is None: + continue + dvol = _median_dollar_vol_63(closes_f, vols_f, i) + wk = _week_key(dates[i]) + out.append({ + "symbol": symbol, + "week": wk, + "fwd": closes_f[j] / closes_f[i] - 1.0, + "close": closes_f[i], + "dvol": dvol, + "fip_id": fip, + "mom_12_1": mom, + "mom_12_1_resid": mom_r, + "vol_6m": vol, + }) + return out + + +def _load_spy(conn) -> dict[date, float]: + rows = conn.execute( + text("SELECT date, close FROM benchmark_prices WHERE symbol = 'SPY' ORDER BY date") + ).fetchall() + out: dict[date, float] = {} + for d, c in rows: + if isinstance(d, str): + d = date.fromisoformat(d[:10]) + out[d] = float(c) + return out + + +def _load_symbols(conn) -> list[str]: + return [ + str(r[0]) + for r in conn.execute(text("SELECT symbol FROM tickers ORDER BY symbol")).fetchall() + ] + + +def _load_columns(conn, symbol: str) -> tuple | None: + tid = conn.execute( + text("SELECT id FROM tickers WHERE symbol = :s"), {"s": symbol} + ).scalar() + if tid is None: + return None + rows = conn.execute( + text( + "SELECT date, open, high, low, close, volume FROM ohlcv_records " + "WHERE ticker_id = :t ORDER BY date" + ), + {"t": tid}, + ).fetchall() + if len(rows) < HORIZON + 60: + return None + ords: list[int] = [] + opens: list[float] = [] + highs: list[float] = [] + lows: list[float] = [] + closes: list[float] = [] + vols: list[float] = [] + for d, o, h, l, c, v in rows: + if isinstance(d, str): + d = date.fromisoformat(d[:10]) + ords.append(d.toordinal()) + opens.append(float(o)) + highs.append(float(h)) + lows.append(float(l)) + closes.append(float(c)) + vols.append(float(v or 0)) + return (symbol, ords, opens, highs, lows, closes, vols) + + +def _liquid_members( + obs: list[dict], + *, + top_n: int, + min_price: float, + dvol_key: str = "dvol", +) -> list[dict]: + eligible = [ + o + for o in obs + if o.get("close") is not None + and float(o["close"]) >= min_price + and o.get(dvol_key) is not None + and float(o[dvol_key]) > 0 + ] + eligible.sort(key=lambda o: float(o[dvol_key]), reverse=True) + return eligible[:top_n] + + +def _pairs(obs: list[dict], signal: str) -> list[tuple[float, float]]: + out: list[tuple[float, float]] = [] + for o in obs: + v = o.get(signal) + if v is None: + continue + out.append((float(v), float(o["fwd"]))) + return out + + +def main() -> None: + args = _parse_args() + research = Path(args.research_snapshot) + prod = Path(args.prod_snapshot) + if not research.exists(): + raise SystemExit(f"Missing research snapshot: {research}") + + research_eng = create_engine(f"sqlite:///{research.resolve().as_posix()}") + prod_symbols: set[str] = set() + if prod.exists(): + prod_eng = create_engine(f"sqlite:///{prod.resolve().as_posix()}") + with prod_eng.connect() as c: + prod_symbols = { + str(r[0]) + for r in c.execute(text("SELECT symbol FROM tickers")).fetchall() + } + prod_eng.dispose() + + with research_eng.connect() as conn: + spy = _load_spy(conn) + symbols = _load_symbols(conn) + jobs: list[tuple] = [] + for i, sym in enumerate(symbols, 1): + cols = _load_columns(conn, sym) + if cols is None: + continue + jobs.append((*cols, spy)) + if not args.quiet and i % 500 == 0: + print(f" queued {i}/{len(symbols)}", flush=True) + + if not args.quiet: + print(f"Building weekly panel for {len(jobs)} tickers…", flush=True) + + # Panel: week -> list of obs + by_week: dict[tuple[int, int], list[dict]] = defaultdict(list) + workers = max(1, int(args.workers)) + if workers == 1: + for j, job in enumerate(jobs, 1): + for row in _panel_worker(job): + by_week[tuple(row["week"])].append(row) + if not args.quiet and j % 200 == 0: + print(f" panel {j}/{len(jobs)}", flush=True) + else: + with ProcessPoolExecutor(max_workers=workers) as pool: + futs = {pool.submit(_panel_worker, job): job[0] for job in jobs} + done = 0 + for fut in as_completed(futs): + done += 1 + try: + rows = fut.result() + except Exception as exc: + if not args.quiet: + print(f" worker error {futs[fut]}: {exc}", flush=True) + continue + for row in rows: + by_week[tuple(row["week"])].append(row) + if not args.quiet and done % 200 == 0: + print(f" panel {done}/{len(jobs)}", flush=True) + + if not args.quiet: + print(f"Weeks with data: {len(by_week)}", flush=True) + + # Prior-week dvol map for lagged membership: (symbol, week) -> dvol + dvol_by_sym_week: dict[tuple[str, tuple[int, int]], float] = {} + for wk, obs in by_week.items(): + for o in obs: + if o.get("dvol") is not None: + dvol_by_sym_week[(o["symbol"], wk)] = float(o["dvol"]) + + ordered_weeks = sorted(by_week.keys(), key=_week_ord) + prev_week: dict[tuple[int, int], tuple[int, int]] = {} + for i, wk in enumerate(ordered_weeks): + if i > 0: + prev_week[wk] = ordered_weeks[i - 1] + + top_n = int(args.top_n) + min_price = float(args.min_price) + + # --- Panels for each check --- + same_week_fip: dict[tuple[int, int], list[tuple[float, float]]] = defaultdict(list) + lag_week_fip: dict[tuple[int, int], list[tuple[float, float]]] = defaultdict(list) + tier_hi_fip: dict[tuple[int, int], list[tuple[float, float]]] = defaultdict(list) + tier_lo_fip: dict[tuple[int, int], list[tuple[float, float]]] = defaultdict(list) + prod_subset_fip: dict[tuple[int, int], list[tuple[float, float]]] = defaultdict(list) + mom_cond_fip: dict[tuple[int, int], list[tuple[float, float]]] = defaultdict(list) + liquid_vol: dict[tuple[int, int], list[tuple[float, float]]] = defaultdict(list) + liquid_mom: dict[tuple[int, int], list[tuple[float, float]]] = defaultdict(list) + liquid_mom_r: dict[tuple[int, int], list[tuple[float, float]]] = defaultdict(list) + + for wk, obs in by_week.items(): + # Same-week liquid top-N among names that have fip (matches signal_eval mask: + # membership is ranked within each signal's observation set). + with_fip = [o for o in obs if o.get("fip_id") is not None] + liq_fip = _liquid_members(with_fip, top_n=top_n, min_price=min_price) + for rank, o in enumerate(liq_fip, 1): + same_week_fip[wk].append((float(o["fip_id"]), float(o["fwd"]))) + if rank <= 800: + tier_hi_fip[wk].append((float(o["fip_id"]), float(o["fwd"]))) + elif rank <= top_n: + tier_lo_fip[wk].append((float(o["fip_id"]), float(o["fwd"]))) + if o["symbol"] in prod_symbols: + prod_subset_fip[wk].append((float(o["fip_id"]), float(o["fwd"]))) + + # Context signals: liquid among names that carry that signal + with_vol = [o for o in obs if o.get("vol_6m") is not None] + for o in _liquid_members(with_vol, top_n=top_n, min_price=min_price): + liquid_vol[wk].append((float(o["vol_6m"]), float(o["fwd"]))) + with_mom_all = [o for o in obs if o.get("mom_12_1") is not None] + liq_mom = _liquid_members(with_mom_all, top_n=top_n, min_price=min_price) + for o in liq_mom: + liquid_mom[wk].append((float(o["mom_12_1"]), float(o["fwd"]))) + with_mom_r = [o for o in obs if o.get("mom_12_1_resid") is not None] + for o in _liquid_members(with_mom_r, top_n=top_n, min_price=min_price): + liquid_mom_r[wk].append((float(o["mom_12_1_resid"]), float(o["fwd"]))) + + # Momentum-conditional: within liquid fip set, keep mom_12_1 ≥ P80 + mom_key = "mom_12_1" + with_mom = [ + o for o in liq_fip + if o.get(mom_key) is not None and o.get("fip_id") is not None + ] + if len(with_mom) >= MIN_CROSS: + with_mom.sort(key=lambda o: float(o[mom_key])) + n = len(with_mom) + cut = int(math.floor(n * (MOM_WINNER_PCT / 100.0))) + winners = with_mom[cut:] # upper tail + for o in winners: + mom_cond_fip[wk].append((float(o["fip_id"]), float(o["fwd"]))) + + # Lagged membership: rank by *previous* week's dvol among fip names + pw = prev_week.get(wk) + if pw is not None: + lagged: list[dict] = [] + for o in with_fip: + if o.get("close") is None or float(o["close"]) < min_price: + continue + prev_dvol = dvol_by_sym_week.get((o["symbol"], pw)) + if prev_dvol is None or prev_dvol <= 0: + continue + lagged.append({**o, "lag_dvol": prev_dvol}) + lagged.sort(key=lambda o: float(o["lag_dvol"]), reverse=True) + for o in lagged[:top_n]: + lag_week_fip[wk].append((float(o["fip_id"]), float(o["fwd"]))) + + results = { + "generated_at": datetime.now().isoformat(), + "research_snapshot": str(research.resolve()), + "prod_subset_n": len(prod_symbols), + "panel_tickers": len(jobs), + "top_n": top_n, + "min_price": min_price, + "checks": { + "fip_same_week_liquid_1500": { + "note": "Replication of main breadth run (same-week $vol mask)", + **_ic_from_weekly(same_week_fip), + }, + "fip_lagged_membership_1w": { + "note": ( + "Liquid top-N ranked on *prior* week's median $vol — " + "excludes same-week liquidity explosion leak" + ), + **_ic_from_weekly(lag_week_fip), + }, + "fip_tier_1_800": { + "note": "Same-week liquid ranks 1–800 (senior liquid tier)", + **_ic_from_weekly(tier_hi_fip), + }, + "fip_tier_801_1500": { + "note": "Same-week liquid ranks 801–1500 (junior liquid tier)", + **_ic_from_weekly(tier_lo_fip), + }, + "fip_prod_universe_subset": { + "note": ( + "Symbols in prod.sqlite (~S&P-like large-cap book) inside " + "same-week liquid top-N — compositional control" + ), + **_ic_from_weekly(prod_subset_fip), + }, + "fip_momentum_conditional_top20pct": { + "note": ( + f"Among liquid top-N, keep mom_12_1 percentile ≥ {MOM_WINNER_PCT} " + "(paper: ID modulates continuation among winners; gate-relevant)" + ), + **_ic_from_weekly(mom_cond_fip), + }, + "vol_6m_liquid_1500": { + "note": "Context: low-vol anomaly strength on this pool", + **_ic_from_weekly(liquid_vol), + }, + "mom_12_1_liquid_1500": { + "note": "Context: raw momentum on liquid breadth", + **_ic_from_weekly(liquid_mom), + }, + "mom_12_1_resid_liquid_1500": { + "note": "Context: residual momentum on liquid breadth", + **_ic_from_weekly(liquid_mom_r), + }, + }, + } + + # Interpretations + checks = results["checks"] + lag = checks["fip_lagged_membership_1w"] + same = checks["fip_same_week_liquid_1500"] + hi = checks["fip_tier_1_800"] + lo = checks["fip_tier_801_1500"] + prod = checks["fip_prod_universe_subset"] + cond = checks["fip_momentum_conditional_top20pct"] + + def _sign(x: float | None) -> str: + if x is None: + return "na" + return "neg" if x < 0 else "pos" + + results["interpretation"] = { + "leak_ruled_out": ( + lag.get("mean_ic") is not None + and same.get("mean_ic") is not None + and _sign(lag["mean_ic"]) == _sign(same["mean_ic"]) + and abs(float(lag["mean_ic"])) >= 0.02 + ), + "junior_tier_drives_positive": ( + lo.get("mean_ic") is not None + and float(lo["mean_ic"]) > 0 + and (hi.get("mean_ic") is None or float(hi["mean_ic"]) < float(lo["mean_ic"])) + ), + "prod_subset_still_negative": ( + prod.get("mean_ic") is not None and float(prod["mean_ic"]) < 0 + ), + "mom_conditional_negative_and_reliable": ( + cond.get("mean_ic") is not None + and float(cond["mean_ic"]) < 0 + and abs(float(cond["mean_ic"])) >= 0.03 + and bool(cond.get("reliable")) + ), + "compositional_flip_story": ( + "If prod subset IC is negative while full liquid-1500 is positive, " + "the sign flip is compositional (bleeders / Nasdaq junk), not a " + "temporal regime change. Unconditional fip pools continuous winners " + "(want neg IC) against continuous losers/bleeders (want pos IC)." + ), + "vol_tilt_warning": ( + "vol_6m large negative IC on breadth: high-vol lottery names " + "underperform. Production 80/20 high-vol tilt was validated on " + "S&P-like names; must re-validate before any universe broaden." + ), + } + + # Gate-relevant summary line + if results["interpretation"]["mom_conditional_negative_and_reliable"]: + results["platform_verdict"] = ( + "ALIVE as breadth-book tilt candidate among momentum winners only — " + "still needs a book-level experiment; not a production wire-in." + ) + else: + results["platform_verdict"] = ( + "CLOSED for production use: momentum-conditional fip does not clear " + "iron rule on this liquid-Nasdaq pool. Display card remains final resting place." + ) + + stamp = datetime.now().strftime("%Y%m%d-%H%M%S") + out = Path(args.out) if args.out else Path("reports") / f"fip-breadth-diagnostics-{stamp}.json" + out.parent.mkdir(parents=True, exist_ok=True) + out.write_text(json.dumps(results, indent=2, default=str), encoding="utf-8") + + # Append to research log + md_path = Path("docs/research/fip-breadth-ic.md") + _append_diagnostics_md(md_path, results, out) + + if not args.quiet: + print(json.dumps(results["checks"], indent=2, default=str)) + print() + print("interpretation:", json.dumps(results["interpretation"], indent=2)) + print("platform_verdict:", results["platform_verdict"]) + print(f"Wrote {out}") + print(f"Updated {md_path}") + + +def _append_diagnostics_md(path: Path, results: dict, artifact: Path) -> None: + checks = results["checks"] + interp = results["interpretation"] + lines = [ + "", + "---", + "", + f"## Follow-up diagnostics ({results['generated_at'][:10]})", + "", + "Compositional reading of the sign flip (before any 'jumpiness premium' story):", + "", + "`fip_id = sign(PRET) × (%neg − %pos)` pools two opposite continuous populations:", + "", + "- **Continuous winners** (PRET>0, mostly up days) → paper claim → **negative** IC contribution.", + "- **Continuous losers / bleeders** (PRET<0, mostly down days) → momentum continuation down → **positive** IC contribution.", + "", + "Unconditional IC is a tug-of-war weighted by universe composition. S&P-like books " + "have few steady bleeders → negative fip IC. Liquid Nasdaq has many → sign can flip " + "without contradicting Da/Gurun/Warachka (claim was always **momentum-conditional**).", + "", + "### Artifact / composition checks", + "", + "| check | mean_ic | t | weeks | avg N | reliable |", + "|---|---:|---:|---:|---:|---|", + ] + order = [ + "fip_same_week_liquid_1500", + "fip_lagged_membership_1w", + "fip_tier_1_800", + "fip_tier_801_1500", + "fip_prod_universe_subset", + "fip_momentum_conditional_top20pct", + "vol_6m_liquid_1500", + "mom_12_1_liquid_1500", + "mom_12_1_resid_liquid_1500", + ] + for key in order: + row = checks.get(key) or {} + lines.append( + f"| {key} | {row.get('mean_ic')} | {row.get('ic_t_stat')} | " + f"{row.get('weeks')} | {row.get('avg_cross_section')} | {row.get('reliable')} |" + ) + lines.extend([ + "", + "### Flags", + "", + f"- Lagged mask keeps same sign / material |IC|: **{interp.get('leak_ruled_out')}**", + f"- Junior tier (801–1500) drives more positive IC: **{interp.get('junior_tier_drives_positive')}**", + f"- Prod-universe subset still negative: **{interp.get('prod_subset_still_negative')}**", + f"- Mom-conditional (≥P80) negative + reliable: **{interp.get('mom_conditional_negative_and_reliable')}**", + "", + "### Platform verdict", + "", + results.get("platform_verdict", ""), + "", + "### Vol-tilt warning (any future breadth move)", + "", + interp.get("vol_tilt_warning", ""), + "", + f"Artifact: `{artifact.as_posix()}`", + "", + ]) + # Replace previous diagnostics section if re-run, else append + existing = path.read_text(encoding="utf-8") if path.exists() else "" + marker = "## Follow-up diagnostics" + if marker in existing: + existing = existing.split(marker)[0].rstrip() + "\n" + path.write_text(existing + "\n".join(lines), encoding="utf-8") + + +if __name__ == "__main__": + main()