dennisthiessen/signal-platform
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

research: Task 2 closed — SUE dead, earnings gap informational

Browse Source 
Earnings backfill sourced from the public DoltHub earnings repo at a
pinned commit rather than the FMP API: reproducible for anyone re-running
the study, and it burns no request quota. 12,414 events, 98.6% of symbols
with >=8 announcements, 99.2% paired actual/estimate, no keyed duplicates.

2a earnings-gap diagnostic: INFORMATIONAL, no filter shipped. The
pre-earnings cohort's right tail was better, so the registered
avoid-earnings condition failed. Note the raw 23/266 vs 115/574 incidence
gap is largely a duration confound -- severe losses stop out fast and have
less time to span an announcement -- so it is not evidence that holding
through earnings is safe.

2b SUE: FAIL against the pre-registered +0.03 bar (unconditional IC
+0.0151 over 56 reliable windows, momentum-conditional +0.0213). Signs
stable across eras, so this is a clean null rather than an ambiguous one,
consistent with post-earnings drift having decayed in large caps.

Closes the Tier-1 arc: Task 1 dead on deep evidence, Task 2 dead here,
Task 3 complete as diagnostic. No in-sample research thread remains open.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
dennisthiessen
2026-07-20 21:10:46 +02:00
co-authored by Claude Fable 5
parent 1fa3d70dec
commit c7c60a64f2
14 changed files with 3705 additions and 1828 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/research/earnings-gap-and-sue.md
+111 -146
View File
@@ -1,202 +1,167 @@
# Earnings gap diagnostic + SUE / PEAD (Tier-1 alpha research)
**Status:** **PARK** (incomplete earnings coverage; SUE fails iron rule on available sample).
**Branch:** `research/earnings-gap-and-sue`
**Production impact:** none. Local research only. **No filters shipped from 2a.**
**Artifacts:** `reports/earnings-gap-sue-20260719-093129.json` (+ companion `.md`)
**Status:** **CLOSED — SUE DEAD**.
**Branch:** `research/earnings-gap-and-sue`
**Production impact:** none. Local research only; no earnings filter or SUE integration is shipped.
---
## Pre-registration (locked before first research run)
## Pre-registration (locked before the final research run)
### Data
- Historical earnings calendar for the production universe over the full snapshot
window (and deeper if the feed provides it).
- Preferred source: FMP **date-range earnings-calendar** (bulk). If unavailable on
free tier, fall back to per-symbol `/stable/earnings` with request accounting.
- Store in a real local table `earnings_events` (symbol + announce_date key).
- Point-in-time: a surprise is usable only from **announce date + 1 trading day**
onward.
- Historical earnings announcements for the production universe, stored in the
real `earnings_events` table and deduplicated on symbol + announcement date.
- The originally requested 2016 start is amended, with user approval, to the
public source's announcement coverage start of 2020-01-22. Earlier EPS-period
history may scale later surprises but may never activate a live signal.
- Report coverage, pairing, duplicates/restatements, annual-rate sanity, and
announcement-session quality before either experiment.
- Point-in-time: an earnings surprise is usable only from announcement date +1
trading day. Same-day use is forbidden.
### Experiment 2a — earnings-gap risk (defense, report-only)
Join simulated production-config trades (`fill_mode=close`) with earnings dates.
Run the production-config book on the approximately 505-name production
universe with close fills and 0.001 transaction cost per side. Join simulated
trades to earnings by symbol and date.
**Pre-registered questions:**
1. Among closed trades with realized net R ≤ -1.0, report the fraction with an
announcement strictly after entry and before exit, alongside the base rate
for all trades.
2. Compare entries within three trading sessions before an announcement with
all other entries: count, mean/median R, win rate, p05, and p95.
3. Compare stops within one trading session after an announcement with all
other stops and exits.
1. What fraction of losses worse than **−1R** occur with an earnings announcement
**between entry and exit** (inclusive of the holding window)?
2. What is the mean R of entries taken within **3 trading days BEFORE** an
announcement vs all other entries — report **both tails** of the R
distribution (rule 4: any earnings-avoid entry filter is presumed guilty of
right-tail trimming until the win distribution shows otherwise)?
**Output:** distributions and counts only.
**No filter is shipped.** If numbers argue for a filter → report and stop.
Verdict is always `INFORMATIONAL`. Report only: no filter arm, recommendation,
or implementation. The right tail must be shown alongside the left tail.
### Experiment 2b — SUE / PEAD (offense)
Signal `sue_latest`:
\[
\text{SUE} = \frac{\text{actual} - \text{estimate}}{\sigma(\text{trailing 8 surprises})}
\text{SUE} = \frac{\text{actual} - \text{estimate}}
{\sigma(\text{trailing 8 surprises})}
\]
Fallback if estimate history is thin: scale surprise by price.
Carry forward from announce+1 for **63 trading days**, else NaN (name drops out
of that cross-section).
Use at least four trailing surprises; if estimate history fails the registered
quality gate, use `(actual - estimate) / price` and name that fallback. Activate
at announcement date +1 trading day, carry for 63 trading days, then drop the
symbol from the cross-section.
**Iron rule (IC harness):** mean weekly Spearman IC on non-overlapping weeks;
\|mean IC\| ≥ ~0.03, **positive** sign (drift), `reliable: true` (≥12 windows).
Evaluate mean weekly Spearman IC on the existing non-overlapping-window harness.
Always report `sue_latest`, `mom_12_1`, and `mom_12_1_resid` on identical
week-symbol-forward-return cells, plus SUE inside the top momentum quintile.
Always side-by-side with `mom_12_1` and `mom_12_1_resid` on **identical**
cross-sections.
### Mechanical verdict rule
Also report **momentum-conditional** IC (within top momentum quintile).
**If it passes iron rule:** STOP and report. Book-integration design is a
separate human-approved step — do not wire.
### Verdict labels
| label | meaning |
|---|---|
| **PROMOTE** | (2b only) iron rule cleared → human designs tilt/gate |
| **PARK** | Interesting but incomplete / weak |
| **DEAD** | No edge / diagnostic argues against action |
| **REPORT-ONLY** | (2a) always — never auto-filter |
---
## Data provenance
| item | result |
|---|---|
| Snapshot | `backtest_snapshots/prod.sqlite` (506 names) |
| FMP bulk `earnings-calendar` | **402 Premium** — not available on free tier |
| FMP per-symbol `/stable/earnings` | used; hit daily rate limit ~225 reqs |
| Alpha Vantage `EARNINGS` | used for +24 symbols (announce = `reportedDate`) |
| Symbols with events | **48 / 506 (9.5%)** |
| Total events | 5,612 (5,018 with actual+estimate) |
| Announce range | 1985-08-31 → 2026-07-16 |
| FMP requests (first day) | 260 FMP + 25 AV (see `reports/earnings-backfill-status.json`) |
**Incomplete backfill is first-class.** 2a under-detects earnings overlaps; 2b SUE
cross-section averages **~47 names**, not ~500. Resume:
```bash
# Day N (FMP free ~250/day; AV free ~25/day — prefer FMP after reset)
python scripts/backfill_earnings_events.py \
--snapshot backtest_snapshots/prod.sqlite \
--provider fmp --force-symbol --limit 250 --sleep 0.4
# When done==506:
python scripts/run_earnings_research.py \
--snapshot backtest_snapshots/prod.sqlite \
--workers 6 --allow-spawn
```
- **PASS** only if unconditional `sue_latest` has mean IC ≥ +0.03,
`reliable: true` (at least 12 windows), and positive signs in both the pre-2021
and post-2021 eras.
- **FAIL** otherwise, with terminal verdict `SUE DEAD for this stack`.
- PASS stops at `SUE PASS→PENDING_HUMAN`; integration design remains a separate
human decision. FAIL is terminal and no variants are proposed.
---
## Results
Generated: `2026-07-19T09:31:29`
### Data quality gate
### 2a — Earnings-gap risk (report-only)
Approved earnings window: 2020-01-22 to 2026-07-17. Source mode: dolthub_public_bulk_clone.
Production book sim: Sharpe 2.09 (SE 0.497), CAGR 51.6%, max DD 21.4%, **322 trades**,
`fill_mode=close`.
#### Q1 — Losses worse than −1R with earnings in hold
| metric | value |
| check | result |
|---|---:|
| n losses &lt; −1R | 28 |
| of which earnings in hold | **1** |
| fraction | **3.6%** |
| all trades with earnings in hold | 14 / 322 (4.4%) |
| Prod symbols requested / tradable | 506 / 505 |
| Manifest complete + live counts match | True |
| Prod symbols with pre-2021 bars | 491 (97.2%) |
| SPY benchmark depth | 2649 rows, 2016-01-04 to 2026-07-17 |
| Snapshot depth gate | True |
| Bulk source windows / requests logged | 1/1 / 1 |
| Source repository / pinned commit | https://www.dolthub.com/repositories/post-no-preference/earnings @ 9n0et3hpj9j7vue8f3qsldon3qa5sdjj |
| Source license / upstream provider documented | CC-BY-SA-4.0 / False |
| Existing-source conflicts preserved | 940 rows / 1526 fields |
| Symbols with >=8 announcements | 498 (98.6%) |
| Symbols with >=8 paired announcements | 495 (98.0%) |
| Events with estimate + actual | 12311/12414 (99.2%) |
| Duplicate rows in keyed table | 0 |
| Duplicate / restated payload rows fetched | 0 / 940 |
| Mean announcements per active symbol-year | 4.08 (expected about 4) |
| Symbols far off (<2 or >6/year, incl. zero) | 1 |
| Recognised BMO/AMC/during | 92.8% (reliable=True) |
| Point-in-time policy | announce_date_plus_1_trading_day_for_all_events |
| SUE price fallback | not_used |
**Read:** On incomplete earnings labels this is a **lower bound** on earnings
overlap, not a clean “earnings rarely hurt.” Do **not** conclude earnings risk is
immaterial until coverage ≥ ~95% of the book’s names.
Deduplication: UNIQUE(symbol, announce_date); normalise dot/dash symbols; retain one calendar row per key; preserve existing non-null session/EPS values from the prior FMP/Alpha Vantage partial backfill, then fill nulls and all remaining symbols from DoltHub; attach DoltHub period-end alignment
#### Q2 — Entry within 3 trading days before announce (both tails)
Far-off announcement-rate symbols: SPCX
| cohort | n | mean R | win rate | p05 | p50 | p95 | max |
|---|---:|---:|---:|---:|---:|---:|---:|
| pre-earn (≤3d before) | **4** | 1.94 | 50% | −1.24 | 1.12 | 6.26 | 6.84 |
| other | 318 | 0.70 | 37% | −1.11 | −0.83 | 6.08 | **12.87** |
| all | 322 | 0.71 | 37% | −1.12 | −0.83 | 6.22 | 12.87 |
### Experiment 2a - earnings-gap risk diagnostic
**Tail-trim presumption:** n=4 is not a sample. Point estimate does **not** show
right-tail destruction of pre-earn entries (p95 similar; max actually higher in
“other”). **No earnings-avoid filter is supported.** Re-run after full backfill.
Verdict: **INFORMATIONAL**. Report-only; no filter arm or implementation.
---
Trade cohort is restricted to the approved earnings-coverage window 2020-01-22 to 2026-07-17; 0 simulated trades outside that window were excluded.
### 2b — SUE / PEAD IC
| cohort | count | fraction |
|---|---:|---:|
| Realized net R <= -1.0 | 266 | - |
| Losses with announcement strictly inside hold | 23 | 0.0865 |
| All trades with announcement strictly inside hold | 115 | 0.2003 |
#### Full-universe harness (mom on ~500; SUE only where labeled)
| Entry cohort | count | mean R | median R | win rate | p05 R | p95 R |
|---|---:|---:|---:|---:|---:|---:|
| Within 3 sessions before earnings | 27 | 0.4837 | -1.0265 | 0.3333 | -1.1463 | 5.981 |
| All other entries | 547 | 0.2734 | -0.8316 | 0.3565 | -1.1228 | 4.5888 |
| signal | mean_ic | ic_t_stat | weeks | avg_N | reliable |
|---|---:|---:|---:|---:|---|
| mom_12_1_sector_resid | 0.0578 | 2.34 | 35 | 497.7 | true |
| mom_12_1_resid | 0.0552 | 1.98 | 35 | 497.7 | true |
| mom_12_1 | 0.0531 | 1.61 | 35 | 497.7 | true |
| **sue_latest** | **0.0172** | **0.6** | 44 | **47.4** | true |
| fip_id | −0.045 | −2.91 | 35 | 497.7 | true |
Tail deltas (pre minus other): p05=-0.0235, p95=1.3922.
#### Identical SUE subset (fair side-by-side — use this while coverage is thin)
Registered directional tail condition is not present.
| signal | mean_ic | ic_t_stat | weeks | avg_N |
|---|---:|---:|---:|---:|
| sue_latest | 0.0172 | 0.6 | 44 | 47.4 |
| mom_12_1 | −0.0174 | −0.42 | 35 | 47.3 |
| mom_12_1_resid | −0.0104 | −0.27 | 35 | 47.3 |
| Exit cohort | count | mean R | median R | win rate | p05 R | p95 R |
|---|---:|---:|---:|---:|---:|---:|
| Stops within 1 session after earnings | 26 | -0.6434 | -0.9753 | 0.2308 | -2.4614 | 1.1142 |
| All other stops | 433 | -0.5035 | -1.0278 | 0.1963 | -1.1373 | 1.3591 |
| All other exits | 548 | 0.3273 | -0.8361 | 0.3613 | -1.0644 | 4.7224 |
On the thin labeled subset, momentum itself is noise — so the subset is not yet
a meaningful PEAD test.
### Experiment 2b - SUE / post-earnings drift
#### Momentum-conditional SUE (top mom quintile)
Mechanical verdict: **FAIL** - SUE DEAD for this stack
| metric | value |
|---|---:|
| mean IC | **−0.0065** |
| t | −0.1 |
| weeks | 35 |
Identical cross-sections:
Wrong sign vs “ride positive surprises inside the momentum gate.”
| signal | mean IC | t | windows | avg N | IC positive % | reliable |
|---|---:|---:|---:|---:|---:|---|
| sue_latest | 0.0148 | 1.27 | 56 | 450.9 | 51.8 | true |
| mom_12_1 | 0.0195 | 0.74 | 56 | 450.9 | 58.9 | true |
| mom_12_1_resid | 0.0262 | 1.07 | 56 | 450.9 | 55.4 | true |
**Iron rule:** fail (\|IC\| 0.017 &lt; 0.03; t 0.6). **No promote.**
Unconditional SUE grade row:
---
| signal | mean IC | t | windows | avg N | IC positive % | reliable |
|---|---:|---:|---:|---:|---:|---|
| sue_latest | 0.0151 | 1.29 | 56 | 451.4 | 51.8 | true |
## Verdict
Era stability:
| piece | verdict |
|---|---|
| **2a earnings-gap** | **REPORT-ONLY** — no filter. Coverage too thin for risk claims; tails do not argue for an avoid-filter on n=4. |
| **2b SUE** | **PARK** (effectively not green). Mild positive IC on ~48 names; fails iron bar; mom-conditional flat/negative. Re-score after full backfill before DEAD. |
| **Production** | **no change** |
| era | mean IC | t | windows | avg N | IC positive % | reliable |
|---|---:|---:|---:|---:|---:|---|
| pre-2021 | 0.0286 | 0.7 | 9 | 398.3 | 55.6 | false |
| post-2021 | 0.0172 | 1.34 | 48 | 461.9 | 64.6 | true |
---
Coverage: 501 symbols with live SUE; avg weekly N=453.1; scored non-overlap avg N=451.4.
## What a human must decide next
Cross-section is not flagged thin at the registered <100-name read.
1. Resume multi-day earnings backfill to **506/506**, then re-run
`run_earnings_research.py` (heavy — MacBook OK).
2. Do **not** ship an earnings-avoid entry filter from 2a.
3. Do **not** wire SUE until a full-coverage IC clears the iron rule (and
preferably mom-conditional &gt; 0).
4. Do not merge into main strategy docs without review.
Momentum-conditional top-quintile SUE: mean IC=0.0213, t=1.3, windows=56, avg N=89.8.
---
## Artifacts
## Implementation notes
- `reports/earnings-2a-gap-20260720-dolthub-final.json` and companion Markdown
- `reports/earnings-2b-sue-20260720-dolthub-final.json` and companion Markdown
- `reports/earnings-backfill-status.json`
| piece | role |
|---|---|
| `scripts/backfill_earnings_events.py` | bulk attempt → FMP/AV per-symbol; `earnings_events` + meta on snapshot |
| `scripts/run_earnings_research.py` | 2a trade join + 2b SUE IC / mom-conditional |
| Snapshot table `earnings_events` | real table (not SystemSetting JSON) |
Production changes: **none**. No earnings filter or SUE integration was implemented.
## Final status: **Task 2 CLOSED (SUE DEAD)**