dennisthiessen/signal-platform
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

research: park Phase B fip breadth; race guard and compact evidence

Browse Source 
Log the 21:14 orphan as a snapshot-build race, rewrite the context table to
authoritative ICs only, and soften the vol-tilt warning. Add extender completion
manifest + breadth refuse guard; strip intermediate/orphaned reports; park the
thread (no book sim, no deploy).
This commit is contained in:
dennisthiessen
2026-07-19 00:32:20 +02:00
parent 7d60e54f5a
commit 2311999e57
16 changed files with 562 additions and 8103 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/research/README.md
+8 -3
View File
@@ -140,9 +140,9 @@ knobs.
| Lead | Why it's interesting | Blocker |
|---|---|---|
| **Near-close / MOC execution (ops)** | Recovers overnight momentum drift left on the table by a morning EU scan; evidence closed | Implement schedule + partial-bar scan path; one qualifying scan/day only |
| **`fip_id`** | Fingerprint IC −0.045 / t −2.91 on prod book; display-only on ticker technicals | **Phase B:** unconditional liquid-Nasdaq IC fails iron-rule **sign**; **mom-conditional** fip IC −0.088 / t −4.58 (alive as tilt candidate only). See [fip-breadth-ic.md](fip-breadth-ic.md) |
| **Broader universe** | Composition changes factor signs (fip tug-of-war; high-vol junk) | Any prod broaden must **re-validate 80/20 high-vol tilt** first; offline research only for now |
| **Near-close / MOC execution (ops)** | Recovers overnight momentum drift left on the table by a morning EU scan; evidence closed | Schedule + fill_mode shipped; live paper validation ongoing |
| **`fip_id` / liquid breadth** | Fingerprint −0.045 / t −2.91; liquid unconditional **−0.017 / t −1.85** (not green); mom-conditional **−0.088 / t −4.58** | **Parked.** Orphan +0.0575 died (snapshot race). Breadth did not strengthen resid-mom t-stat. Optional reopen = pre-registered two-arm liquid-1500 book first. See [fip-breadth-ic.md](fip-breadth-ic.md) |
| **Broader universe** | Composition changes factor signs (fip tug-of-war); vol-tilt on breadth is only a **directional hypothesis** (auth. −0.048 / t −1.36) | Any prod broaden must re-validate 80/20 tilt; offline research only; research.sqlite requires completion manifest |
| **Forward paper-trade record** | The only true out-of-sample evidence the snapshot cannot give | Time; mark entries at actual near-close fill once ops ships |
| **Better target model for clear-air names** | The return is demonstrably there (#2 wins on raw CAGR in *both* train and test); it's the *flat* 3× ATR target that makes it too expensive in risk | Needs a per-name model, not a constant k×ATR |
@@ -169,6 +169,11 @@ knobs.
6. **Fill timing is part of the strategy.** Close-fill reports are not deployable
numbers for an overnight scanner. Grade promotion under the fill mode you will
actually trade.
7. **Incomplete research artifacts are not results.** The Phase B +0.0575 / t +5.12
liquid-fip row was orphaned within hours: it raced a partially built
`research.sqlite`. Extender now writes a completion manifest; breadth mode
refuses without a match. Same class of protection as calendar-truncation
asserts — do not re-mythologize numbers computed on half a universe.
---
docs/research/fip-breadth-ic.md
+97 -26
View File
@@ -1,13 +1,15 @@
# Broad-universe fip_id IC research (Phase B)
**Status:** unconditional fip closed; mom-conditional lead confirmed on single-sourced path.
**Production impact:** none. Display card remains context-only.
**Status:** **Parked / closed for now.** Unconditional fip not green; mom-conditional lead logged; breadth-momentum thesis challenged. No book sim until reopen.
**Production impact:** none. Display card remains context-only. No deploy from this work.
**Artifacts:** research log + compact reports + env-gated harness hooks; tooling stays for a future reopen.
## Scope
- Research only — production universe, gate, scanner, schedule unchanged.
- Snapshot: `research.sqlite` (~4,650 tickers = prod + nasdaq_all extend).
- Liquid mask: top **1,500** by point-in-time 63d median $vol, price ≥ **$5**/week.
- **Completion manifest required:** extender writes `<snapshot>.manifest.json`; breadth runners refuse without a matching complete manifest (see §Race guard).
## Caveats
@@ -15,6 +17,7 @@
- IEX volume undercount → relative $vol rank only.
- Pool skew: Nasdaq-heavy; missing pure NYSE mid-caps.
- Do not mix multi-signal tables across universe baselines.
- **Do not cite orphaned 21:14 numbers** (see below).
---
@@ -28,47 +31,83 @@
**Pass.** Formula + pipeline trustworthy.
Residual momentum on the same fingerprint (what the production book ranks on): **IC +0.055 / t +1.98**.
---
## Discrepancy (must not be papered over)
## The orphan (21:14) — root cause
| Source | fip IC (liquid ~1500) | t |
|---|---:|---:|
| Report `fip-breadth-20260718-211440-breadth.json` | **+0.0575** | **+5.12** |
| Single-sourced recompute (2026-07-19) | **−0.0168** | **−1.85** |
| Orphan run 21:14 (removed from tree; was `fip-breadth-20260718-211440-breadth.json`) | **+0.0575** | **+5.12** |
| Single-sourced recompute on complete snapshot (2026-07-19) | **−0.0168** | **−1.85** |
That is a **sign disagreement** on the same intended quantity. Method rule: the number you cannot reconcile is the number you cannot use.
### What we did
### Verdict: orphaned — raced the snapshot build
1. **Single-sourced the mask** — diagnostics call harness `_signal_series` + `_filter_liquid_breadth_week_rich` only (no parallel mask).
2. **Documented avg_cross_section semantics** — always **post-mask** IC sample size.
3. **Logged pre-mask stats** so “did top-N bind?” is answerable.
**Not** “orphaned, unexplained.” The mechanism is derivable from the table itself:
### Authoritative unconditional liquid fip (post-reconciliation)
1. **Code was not the difference.** Reconcile shows the old harness path and the new shared filter produce **identical** results on current data (−0.0168 / −1.85). The implementation fork is closed.
2. **Data was the difference.** On today’s complete snapshot the liquid mask **binds in 97.1% of weeks** at top-N = 1,500. Dense signals (e.g. `vol_6m`) post-mask at **exactly 1,500**. The orphaned report’s `vol_6m` averaged **~1,475** cross-section — a masked run on complete data cannot do that. At 21:14 the eligible pool was smaller than 1,500 and the mask never bound.
3. **Timeline fits.** Extender fixes landed ~20:32 / 20:34; full fetch takes ~30 minutes; breadth run fired **21:14** against a partially built `research.sqlite`. Every number in that report was computed on an incomplete universe.
**Do not cite +0.0575 / t +5.12.** It survived less than six hours of contact with project discipline — that is the system working, not time wasted. The orphan JSON was **deleted from the tree** (still in Git history) so it cannot be re-imported as evidence.
**Kept artifacts**
| File | Role |
|---|---|
| `reports/fip-reconcile-20260719-000520.json` | Authoritative single-sourced ICs (compact; membership dumps stripped) |
| `reports/fip-breadth-20260718-211440-fingerprint.json` | Prod fingerprint pass |
### Race guard (same class as calendar truncation)
| Piece | Behavior |
|---|---|
| `extend_snapshot_universe.py` | Clears any prior manifest on start; on full completion writes `<output>.manifest.json` with `complete=true`, ticker / OHLCV / rank_only counts, `finished_at`. `--limit` smoke runs write `complete=false`. |
| `run_fip_breadth_research.py` / `run_fip_breadth_diagnostics.py` | **Refuse** breadth mode unless a matching complete manifest exists and live counts equal the recorded totals. |
Helper: `scripts/research_snapshot_manifest.py`.
---
## Authoritative unconditional liquid fip (post-reconciliation)
| metric | value |
|---|---:|
| mean_ic | **−0.0168** |
| ic_t_stat | **−1.85** |
| weeks | 35 |
| avg_cross_section (**post-mask**) | 1471.2 |
| avg_cross_section (**post-mask IC sample**) | 1471.2 |
| avg_raw_pool | 3214.4 |
| avg_eligible_pre_mask | **2338.4** |
| mask_binds_pct | **97.1%** |
| reliable | true |
**Mask binds hard** (eligible ≫ 1500). The hypothesis that “1471 meant the mask never bound / unmasked +5σ” is **false**.
**Mask binds hard** on complete data (eligible ≫ 1500). Post-mask IC N for fip is ~1471 because not every liquid name has a valid 12-1 fip path — that is signal availability, not a non-binding mask. Contrast orphan `vol_6m` avg N ~1475 vs complete-data `vol_6m` avg N **1500**.
Harness `_signal_evaluation` vs manual IC through the same filter: **exact match** (−0.0168 / −1.85).
### Verdict on the orphan
**Iron rule unconditional:** **not green** (|IC| 0.017 < 0.03), correct mild-negative sign.
The **+0.0575 / t +5.12** row is **orphaned**. Do not cite it. Root cause of that single run is not fully forensic-reconstructed (no dual dump from the original process remains), but every single-sourced recompute on this snapshot lands near **−0.017**, and the tier blend (≈800×−0.035 + ≈670×+0.014)/1471 ≈ **−0.013** is internally consistent with that number—not with +0.058.
---
**Iron rule unconditional:** still **not green** (|IC| 0.017 < 0.03), and now with the correct mild-negative sign.
## Context table (orphaned 21:14 vs authoritative) — kill the myth numbers
Artifact: `reports/fip-reconcile-20260719-000520.json`
The context table died with the orphan. **−0.16 must not survive in the log.**
| signal (liquid ~1500) | orphaned (21:14) | authoritative (shared filter) | consequence |
|---|---:|---:|---|
| **vol_6m** | −0.16 / t **−6.1** | **−0.048 / t −1.36** | “High-vol tilt harmful on breadth” **downgrades from finding to directional hypothesis** — not significant |
| **raw mom** (`mom_12_1`) | +0.10 / t +4.6 | **+0.046 / t +1.91** | Below iron-rule bar on this pool |
| **resid mom** (`mom_12_1_resid`) | +0.04 / t +2.3 | **+0.029 / t +1.33** | Ditto, and weaker than raw |
### Breadth-momentum thesis — challenged
That last pair is the sobering one. Momentum on liquid breadth is **marginal**. The “more breadth strengthens the momentum t-stat” thesis that motivated Phase B is **empirically wrong on this pool**: same 35 weeks, triple the names, residual-mom t-stat **fell** versus the 505-name fingerprint (**0.055 / 1.98** → **0.029 / 1.33**). The clean momentum edge lives in the large-cap universe already traded.
Meanwhile the strongest reliable signal on liquid breadth is now **mom-conditional fip** (−0.088 / −4.58) — but a fip tilt presupposes a breadth momentum book worth tilting, and that is no longer free.
---
@@ -107,27 +146,57 @@ Computed on the **same single-sourced path** as the authoritative −0.017. This
| Decision | |
|---|---|
| Unconditional fip | **Closed** for production |
| Mom-conditional fip | **Alive as book-tilt candidate only** — book sim before any gate talk |
| Mom-conditional fip | **Alive as book-tilt candidate only** — and only after a baseline breadth book proves itself |
| Display card | Stays |
| Production change | **None** |
---
## Vol-tilt / residual-mom warning (any future breadth move)
## Vol-tilt warning (softened)
| signal (liquid, single-sourced) | IC | t |
|---|---:|---:|
| vol_6m | −0.048 | −1.4 |
| mom_12_1 | +0.046 | +1.9 |
| mom_12_1_resid | +0.029 | +1.3 |
| vol_6m | −0.048 | **−1.36** |
| mom_12_1 | +0.046 | +1.91 |
| mom_12_1_resid | +0.029 | +1.33 |
High-vol names tend to underperform on this pool relative to a clean S&P-like book. Production **80/20 high-vol tilt** was validated on S&P-like names. **If the universe ever broadens in production, re-validate that tilt first** — it can flip from mildly helpful to harmful. Raw momentum also looks stronger than SPY residualization here (noisier fit for small caps).
High-vol names **tend** to underperform on this pool relative to a clean S&P-like book — that is a **directional hypothesis**, not a finding. Production **80/20 high-vol tilt** was validated on S&P-like names. If the universe ever broadens in production, re-validate that tilt; do not treat the orphaned −0.16 / t −6.1 as evidence.
---
## What this means for the book experiment
A fip tilt presupposes a breadth momentum book worth tilting — **that is no longer free.**
**Caution against over-reacting the other way:** modest cross-sectional IC does not preclude a good book. The 505-name book turns resid-mom IC ~0.055 into Sharpe ~2 because the gate trades the **extreme tail**, not the linear sort. The breadth book might still work; it just has to **prove it** before the fip arm means anything. If the baseline cannot clearly beat the existing production book’s territory, fip’s future is a footnote regardless of −4.58.
### Parked next step (if reopened): pre-registered two-arm design
Not started — **design only**, pre-register before any sim:
| Arm | Definition |
|---|---|
| **A — baseline** | Top-quintile residual (or raw — pick one and lock) momentum book on liquid-1500; **no fip**; honest costs; next-open or near-close fills; production-like capacity / risk / stops |
| **B — +fip tilt** | Same book + mom-conditional fip tilt (among mom winners, prefer smoother paths / negative fip_id) |
| Grade on | Spec |
|---|---|
| Split | Entry-date train / validation (`BACKTEST_HOLDOUT_SPLIT` naming — not pristine holdout) |
| Metrics | Sharpe + Mertens/Lo SE, PSR, **DSR**; max DD; turnover; cost drag |
| Promote bar | Arm A must be in production-book territory first; Arm B must beat A on validation with DSR-aware multiple-testing honesty |
| Fail-closed | If A fails, fip is a footnote; do not shop tilts on a dead baseline |
---
## How to re-run (research branch only)
```powershell
# 1) Full extend writes completion manifest (required)
.\.venv\Scripts\python.exe scripts\extend_snapshot_universe.py `
--source backtest_snapshots\prod.sqlite `
--output backtest_snapshots\research.sqlite
# 2) Breadth / diagnostics refuse without matching manifest
.\.venv\Scripts\python.exe scripts\run_fip_breadth_diagnostics.py `
--research-snapshot backtest_snapshots\research.sqlite `
--prod-snapshot backtest_snapshots\prod.sqlite `
@@ -139,7 +208,9 @@ High-vol names tend to underperform on this pool relative to a clean S&P-like bo
## Bottom line
1. Formal iron-rule screen: **not green** either before or after reconciliation.
2. **+0.0575 / +5.12 is orphaned** — authoritative unconditional liquid fip is **−0.017 / −1.9**; mask binds (~97%).
3. Compositional tug-of-war is the right story; jumpiness premium is not.
4. **Mom-conditional −0.088 / −4.6 stands on the single-sourced path** → optional next research step is a **book** A/B, not a gate wire-in.
5. Log any future reader who sees both numbers: trust the reconcile artifact, not the orphaned breadth headline.
2. **+0.0575 / +5.12 is orphaned: raced the snapshot build** — authoritative unconditional liquid fip is **−0.017 / −1.9**; mask binds (~97%) on complete data.
3. Context-table myths die with the orphan: **vol −0.16 is not real**; authoritative vol is **−0.048 / t −1.36** (directional only).
4. Compositional tug-of-war is the right story; jumpiness premium is not.
5. **Breadth does not strengthen residual-mom t-stat** on this pool (0.055/1.98 → 0.029/1.33).
6. **Mom-conditional −0.088 / −4.6 stands** on the single-sourced path → optional next step is a **pre-registered two-arm breadth book** (baseline first), not a gate wire-in.
7. Manifest guard is in place so the race cannot recur silently.