research: Task 2 closed — SUE dead, earnings gap informational
Earnings backfill sourced from the public DoltHub earnings repo at a pinned commit rather than the FMP API: reproducible for anyone re-running the study, and it burns no request quota. 12,414 events, 98.6% of symbols with >=8 announcements, 99.2% paired actual/estimate, no keyed duplicates. 2a earnings-gap diagnostic: INFORMATIONAL, no filter shipped. The pre-earnings cohort's right tail was better, so the registered avoid-earnings condition failed. Note the raw 23/266 vs 115/574 incidence gap is largely a duration confound -- severe losses stop out fast and have less time to span an announcement -- so it is not evidence that holding through earnings is safe. 2b SUE: FAIL against the pre-registered +0.03 bar (unconditional IC +0.0151 over 56 reliable windows, momentum-conditional +0.0213). Signs stable across eras, so this is a clean null rather than an ambiguous one, consistent with post-earnings drift having decayed in large caps. Closes the Tier-1 arc: Task 1 dead on deep evidence, Task 2 dead here, Task 3 complete as diagnostic. No in-sample research thread remains open. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Claude Fable 5
@@ -1,202 +1,167 @@
|
|||||||
# Earnings gap diagnostic + SUE / PEAD (Tier-1 alpha research)
|
# Earnings gap diagnostic + SUE / PEAD (Tier-1 alpha research)
|
||||||
|
|
||||||
**Status:** **PARK** (incomplete earnings coverage; SUE fails iron rule on available sample).
|
**Status:** **CLOSED — SUE DEAD**.
|
||||||
**Branch:** `research/earnings-gap-and-sue`
|
**Branch:** `research/earnings-gap-and-sue`
|
||||||
**Production impact:** none. Local research only. **No filters shipped from 2a.**
|
**Production impact:** none. Local research only; no earnings filter or SUE integration is shipped.
|
||||||
**Artifacts:** `reports/earnings-gap-sue-20260719-093129.json` (+ companion `.md`)
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Pre-registration (locked before first research run)
|
## Pre-registration (locked before the final research run)
|
||||||
|
|
||||||
### Data
|
### Data
|
||||||
|
|
||||||
- Historical earnings calendar for the production universe over the full snapshot
|
- Historical earnings announcements for the production universe, stored in the
|
||||||
window (and deeper if the feed provides it).
|
real `earnings_events` table and deduplicated on symbol + announcement date.
|
||||||
- Preferred source: FMP **date-range earnings-calendar** (bulk). If unavailable on
|
- The originally requested 2016 start is amended, with user approval, to the
|
||||||
free tier, fall back to per-symbol `/stable/earnings` with request accounting.
|
public source's announcement coverage start of 2020-01-22. Earlier EPS-period
|
||||||
- Store in a real local table `earnings_events` (symbol + announce_date key).
|
history may scale later surprises but may never activate a live signal.
|
||||||
- Point-in-time: a surprise is usable only from **announce date + 1 trading day**
|
- Report coverage, pairing, duplicates/restatements, annual-rate sanity, and
|
||||||
onward.
|
announcement-session quality before either experiment.
|
||||||
|
- Point-in-time: an earnings surprise is usable only from announcement date +1
|
||||||
|
trading day. Same-day use is forbidden.
|
||||||
|
|
||||||
### Experiment 2a — earnings-gap risk (defense, report-only)
|
### Experiment 2a — earnings-gap risk (defense, report-only)
|
||||||
|
|
||||||
Join simulated production-config trades (`fill_mode=close`) with earnings dates.
|
Run the production-config book on the approximately 505-name production
|
||||||
|
universe with close fills and 0.001 transaction cost per side. Join simulated
|
||||||
|
trades to earnings by symbol and date.
|
||||||
|
|
||||||
**Pre-registered questions:**
|
1. Among closed trades with realized net R ≤ -1.0, report the fraction with an
|
||||||
|
announcement strictly after entry and before exit, alongside the base rate
|
||||||
|
for all trades.
|
||||||
|
2. Compare entries within three trading sessions before an announcement with
|
||||||
|
all other entries: count, mean/median R, win rate, p05, and p95.
|
||||||
|
3. Compare stops within one trading session after an announcement with all
|
||||||
|
other stops and exits.
|
||||||
|
|
||||||
1. What fraction of losses worse than **−1R** occur with an earnings announcement
|
Verdict is always `INFORMATIONAL`. Report only: no filter arm, recommendation,
|
||||||
**between entry and exit** (inclusive of the holding window)?
|
or implementation. The right tail must be shown alongside the left tail.
|
||||||
2. What is the mean R of entries taken within **3 trading days BEFORE** an
|
|
||||||
announcement vs all other entries — report **both tails** of the R
|
|
||||||
distribution (rule 4: any earnings-avoid entry filter is presumed guilty of
|
|
||||||
right-tail trimming until the win distribution shows otherwise)?
|
|
||||||
|
|
||||||
**Output:** distributions and counts only.
|
|
||||||
**No filter is shipped.** If numbers argue for a filter → report and stop.
|
|
||||||
|
|
||||||
### Experiment 2b — SUE / PEAD (offense)
|
### Experiment 2b — SUE / PEAD (offense)
|
||||||
|
|
||||||
Signal `sue_latest`:
|
Signal `sue_latest`:
|
||||||
|
|
||||||
\[
|
\[
|
||||||
\text{SUE} = \frac{\text{actual} - \text{estimate}}{\sigma(\text{trailing 8 surprises})}
|
\text{SUE} = \frac{\text{actual} - \text{estimate}}
|
||||||
|
{\sigma(\text{trailing 8 surprises})}
|
||||||
\]
|
\]
|
||||||
|
|
||||||
Fallback if estimate history is thin: scale surprise by price.
|
Use at least four trailing surprises; if estimate history fails the registered
|
||||||
Carry forward from announce+1 for **63 trading days**, else NaN (name drops out
|
quality gate, use `(actual - estimate) / price` and name that fallback. Activate
|
||||||
of that cross-section).
|
at announcement date +1 trading day, carry for 63 trading days, then drop the
|
||||||
|
symbol from the cross-section.
|
||||||
|
|
||||||
**Iron rule (IC harness):** mean weekly Spearman IC on non-overlapping weeks;
|
Evaluate mean weekly Spearman IC on the existing non-overlapping-window harness.
|
||||||
\|mean IC\| ≥ ~0.03, **positive** sign (drift), `reliable: true` (≥12 windows).
|
Always report `sue_latest`, `mom_12_1`, and `mom_12_1_resid` on identical
|
||||||
|
week-symbol-forward-return cells, plus SUE inside the top momentum quintile.
|
||||||
|
|
||||||
Always side-by-side with `mom_12_1` and `mom_12_1_resid` on **identical**
|
### Mechanical verdict rule
|
||||||
cross-sections.
|
|
||||||
|
|
||||||
Also report **momentum-conditional** IC (within top momentum quintile).
|
- **PASS** only if unconditional `sue_latest` has mean IC ≥ +0.03,
|
||||||
|
`reliable: true` (at least 12 windows), and positive signs in both the pre-2021
|
||||||
**If it passes iron rule:** STOP and report. Book-integration design is a
|
and post-2021 eras.
|
||||||
separate human-approved step — do not wire.
|
- **FAIL** otherwise, with terminal verdict `SUE DEAD for this stack`.
|
||||||
|
- PASS stops at `SUE PASS→PENDING_HUMAN`; integration design remains a separate
|
||||||
### Verdict labels
|
human decision. FAIL is terminal and no variants are proposed.
|
||||||
|
|
||||||
| label | meaning |
|
|
||||||
|---|---|
|
|
||||||
| **PROMOTE** | (2b only) iron rule cleared → human designs tilt/gate |
|
|
||||||
| **PARK** | Interesting but incomplete / weak |
|
|
||||||
| **DEAD** | No edge / diagnostic argues against action |
|
|
||||||
| **REPORT-ONLY** | (2a) always — never auto-filter |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Data provenance
|
|
||||||
|
|
||||||
| item | result |
|
|
||||||
|---|---|
|
|
||||||
| Snapshot | `backtest_snapshots/prod.sqlite` (506 names) |
|
|
||||||
| FMP bulk `earnings-calendar` | **402 Premium** — not available on free tier |
|
|
||||||
| FMP per-symbol `/stable/earnings` | used; hit daily rate limit ~225 reqs |
|
|
||||||
| Alpha Vantage `EARNINGS` | used for +24 symbols (announce = `reportedDate`) |
|
|
||||||
| Symbols with events | **48 / 506 (9.5%)** |
|
|
||||||
| Total events | 5,612 (5,018 with actual+estimate) |
|
|
||||||
| Announce range | 1985-08-31 → 2026-07-16 |
|
|
||||||
| FMP requests (first day) | 260 FMP + 25 AV (see `reports/earnings-backfill-status.json`) |
|
|
||||||
|
|
||||||
**Incomplete backfill is first-class.** 2a under-detects earnings overlaps; 2b SUE
|
|
||||||
cross-section averages **~47 names**, not ~500. Resume:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Day N (FMP free ~250/day; AV free ~25/day — prefer FMP after reset)
|
|
||||||
python scripts/backfill_earnings_events.py \
|
|
||||||
--snapshot backtest_snapshots/prod.sqlite \
|
|
||||||
--provider fmp --force-symbol --limit 250 --sleep 0.4
|
|
||||||
|
|
||||||
# When done==506:
|
|
||||||
python scripts/run_earnings_research.py \
|
|
||||||
--snapshot backtest_snapshots/prod.sqlite \
|
|
||||||
--workers 6 --allow-spawn
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Results
|
## Results
|
||||||
|
|
||||||
Generated: `2026-07-19T09:31:29`
|
### Data quality gate
|
||||||
|
|
||||||
### 2a — Earnings-gap risk (report-only)
|
Approved earnings window: 2020-01-22 to 2026-07-17. Source mode: dolthub_public_bulk_clone.
|
||||||
|
|
||||||
Production book sim: Sharpe 2.09 (SE 0.497), CAGR 51.6%, max DD 21.4%, **322 trades**,
|
| check | result |
|
||||||
`fill_mode=close`.
|
|
||||||
|
|
||||||
#### Q1 — Losses worse than −1R with earnings in hold
|
|
||||||
|
|
||||||
| metric | value |
|
|
||||||
|---|---:|
|
|---|---:|
|
||||||
| n losses < −1R | 28 |
|
| Prod symbols requested / tradable | 506 / 505 |
|
||||||
| of which earnings in hold | **1** |
|
| Manifest complete + live counts match | True |
|
||||||
| fraction | **3.6%** |
|
| Prod symbols with pre-2021 bars | 491 (97.2%) |
|
||||||
| all trades with earnings in hold | 14 / 322 (4.4%) |
|
| SPY benchmark depth | 2649 rows, 2016-01-04 to 2026-07-17 |
|
||||||
|
| Snapshot depth gate | True |
|
||||||
|
| Bulk source windows / requests logged | 1/1 / 1 |
|
||||||
|
| Source repository / pinned commit | https://www.dolthub.com/repositories/post-no-preference/earnings @ 9n0et3hpj9j7vue8f3qsldon3qa5sdjj |
|
||||||
|
| Source license / upstream provider documented | CC-BY-SA-4.0 / False |
|
||||||
|
| Existing-source conflicts preserved | 940 rows / 1526 fields |
|
||||||
|
| Symbols with >=8 announcements | 498 (98.6%) |
|
||||||
|
| Symbols with >=8 paired announcements | 495 (98.0%) |
|
||||||
|
| Events with estimate + actual | 12311/12414 (99.2%) |
|
||||||
|
| Duplicate rows in keyed table | 0 |
|
||||||
|
| Duplicate / restated payload rows fetched | 0 / 940 |
|
||||||
|
| Mean announcements per active symbol-year | 4.08 (expected about 4) |
|
||||||
|
| Symbols far off (<2 or >6/year, incl. zero) | 1 |
|
||||||
|
| Recognised BMO/AMC/during | 92.8% (reliable=True) |
|
||||||
|
| Point-in-time policy | announce_date_plus_1_trading_day_for_all_events |
|
||||||
|
| SUE price fallback | not_used |
|
||||||
|
|
||||||
**Read:** On incomplete earnings labels this is a **lower bound** on earnings
|
Deduplication: UNIQUE(symbol, announce_date); normalise dot/dash symbols; retain one calendar row per key; preserve existing non-null session/EPS values from the prior FMP/Alpha Vantage partial backfill, then fill nulls and all remaining symbols from DoltHub; attach DoltHub period-end alignment
|
||||||
overlap, not a clean “earnings rarely hurt.” Do **not** conclude earnings risk is
|
|
||||||
immaterial until coverage ≥ ~95% of the book’s names.
|
|
||||||
|
|
||||||
#### Q2 — Entry within 3 trading days before announce (both tails)
|
Far-off announcement-rate symbols: SPCX
|
||||||
|
|
||||||
| cohort | n | mean R | win rate | p05 | p50 | p95 | max |
|
### Experiment 2a - earnings-gap risk diagnostic
|
||||||
|---|---:|---:|---:|---:|---:|---:|---:|
|
|
||||||
| pre-earn (≤3d before) | **4** | 1.94 | 50% | −1.24 | 1.12 | 6.26 | 6.84 |
|
|
||||||
| other | 318 | 0.70 | 37% | −1.11 | −0.83 | 6.08 | **12.87** |
|
|
||||||
| all | 322 | 0.71 | 37% | −1.12 | −0.83 | 6.22 | 12.87 |
|
|
||||||
|
|
||||||
**Tail-trim presumption:** n=4 is not a sample. Point estimate does **not** show
|
Verdict: **INFORMATIONAL**. Report-only; no filter arm or implementation.
|
||||||
right-tail destruction of pre-earn entries (p95 similar; max actually higher in
|
|
||||||
“other”). **No earnings-avoid filter is supported.** Re-run after full backfill.
|
|
||||||
|
|
||||||
---
|
Trade cohort is restricted to the approved earnings-coverage window 2020-01-22 to 2026-07-17; 0 simulated trades outside that window were excluded.
|
||||||
|
|
||||||
### 2b — SUE / PEAD IC
|
| cohort | count | fraction |
|
||||||
|
|---|---:|---:|
|
||||||
|
| Realized net R <= -1.0 | 266 | - |
|
||||||
|
| Losses with announcement strictly inside hold | 23 | 0.0865 |
|
||||||
|
| All trades with announcement strictly inside hold | 115 | 0.2003 |
|
||||||
|
|
||||||
#### Full-universe harness (mom on ~500; SUE only where labeled)
|
| Entry cohort | count | mean R | median R | win rate | p05 R | p95 R |
|
||||||
|
|---|---:|---:|---:|---:|---:|---:|
|
||||||
|
| Within 3 sessions before earnings | 27 | 0.4837 | -1.0265 | 0.3333 | -1.1463 | 5.981 |
|
||||||
|
| All other entries | 547 | 0.2734 | -0.8316 | 0.3565 | -1.1228 | 4.5888 |
|
||||||
|
|
||||||
| signal | mean_ic | ic_t_stat | weeks | avg_N | reliable |
|
Tail deltas (pre minus other): p05=-0.0235, p95=1.3922.
|
||||||
|---|---:|---:|---:|---:|---|
|
|
||||||
| mom_12_1_sector_resid | 0.0578 | 2.34 | 35 | 497.7 | true |
|
|
||||||
| mom_12_1_resid | 0.0552 | 1.98 | 35 | 497.7 | true |
|
|
||||||
| mom_12_1 | 0.0531 | 1.61 | 35 | 497.7 | true |
|
|
||||||
| **sue_latest** | **0.0172** | **0.6** | 44 | **47.4** | true |
|
|
||||||
| fip_id | −0.045 | −2.91 | 35 | 497.7 | true |
|
|
||||||
|
|
||||||
#### Identical SUE subset (fair side-by-side — use this while coverage is thin)
|
Registered directional tail condition is not present.
|
||||||
|
|
||||||
| signal | mean_ic | ic_t_stat | weeks | avg_N |
|
| Exit cohort | count | mean R | median R | win rate | p05 R | p95 R |
|
||||||
|---|---:|---:|---:|---:|
|
|---|---:|---:|---:|---:|---:|---:|
|
||||||
| sue_latest | 0.0172 | 0.6 | 44 | 47.4 |
|
| Stops within 1 session after earnings | 26 | -0.6434 | -0.9753 | 0.2308 | -2.4614 | 1.1142 |
|
||||||
| mom_12_1 | −0.0174 | −0.42 | 35 | 47.3 |
|
| All other stops | 433 | -0.5035 | -1.0278 | 0.1963 | -1.1373 | 1.3591 |
|
||||||
| mom_12_1_resid | −0.0104 | −0.27 | 35 | 47.3 |
|
| All other exits | 548 | 0.3273 | -0.8361 | 0.3613 | -1.0644 | 4.7224 |
|
||||||
|
|
||||||
On the thin labeled subset, momentum itself is noise — so the subset is not yet
|
### Experiment 2b - SUE / post-earnings drift
|
||||||
a meaningful PEAD test.
|
|
||||||
|
|
||||||
#### Momentum-conditional SUE (top mom quintile)
|
Mechanical verdict: **FAIL** - SUE DEAD for this stack
|
||||||
|
|
||||||
| metric | value |
|
Identical cross-sections:
|
||||||
|---|---:|
|
|
||||||
| mean IC | **−0.0065** |
|
|
||||||
| t | −0.1 |
|
|
||||||
| weeks | 35 |
|
|
||||||
|
|
||||||
Wrong sign vs “ride positive surprises inside the momentum gate.”
|
| signal | mean IC | t | windows | avg N | IC positive % | reliable |
|
||||||
|
|---|---:|---:|---:|---:|---:|---|
|
||||||
|
| sue_latest | 0.0148 | 1.27 | 56 | 450.9 | 51.8 | true |
|
||||||
|
| mom_12_1 | 0.0195 | 0.74 | 56 | 450.9 | 58.9 | true |
|
||||||
|
| mom_12_1_resid | 0.0262 | 1.07 | 56 | 450.9 | 55.4 | true |
|
||||||
|
|
||||||
**Iron rule:** fail (\|IC\| 0.017 < 0.03; t 0.6). **No promote.**
|
Unconditional SUE grade row:
|
||||||
|
|
||||||
---
|
| signal | mean IC | t | windows | avg N | IC positive % | reliable |
|
||||||
|
|---|---:|---:|---:|---:|---:|---|
|
||||||
|
| sue_latest | 0.0151 | 1.29 | 56 | 451.4 | 51.8 | true |
|
||||||
|
|
||||||
## Verdict
|
Era stability:
|
||||||
|
|
||||||
| piece | verdict |
|
| era | mean IC | t | windows | avg N | IC positive % | reliable |
|
||||||
|---|---|
|
|---|---:|---:|---:|---:|---:|---|
|
||||||
| **2a earnings-gap** | **REPORT-ONLY** — no filter. Coverage too thin for risk claims; tails do not argue for an avoid-filter on n=4. |
|
| pre-2021 | 0.0286 | 0.7 | 9 | 398.3 | 55.6 | false |
|
||||||
| **2b SUE** | **PARK** (effectively not green). Mild positive IC on ~48 names; fails iron bar; mom-conditional flat/negative. Re-score after full backfill before DEAD. |
|
| post-2021 | 0.0172 | 1.34 | 48 | 461.9 | 64.6 | true |
|
||||||
| **Production** | **no change** |
|
|
||||||
|
|
||||||
---
|
Coverage: 501 symbols with live SUE; avg weekly N=453.1; scored non-overlap avg N=451.4.
|
||||||
|
|
||||||
## What a human must decide next
|
Cross-section is not flagged thin at the registered <100-name read.
|
||||||
|
|
||||||
1. Resume multi-day earnings backfill to **506/506**, then re-run
|
Momentum-conditional top-quintile SUE: mean IC=0.0213, t=1.3, windows=56, avg N=89.8.
|
||||||
`run_earnings_research.py` (heavy — MacBook OK).
|
|
||||||
2. Do **not** ship an earnings-avoid entry filter from 2a.
|
|
||||||
3. Do **not** wire SUE until a full-coverage IC clears the iron rule (and
|
|
||||||
preferably mom-conditional > 0).
|
|
||||||
4. Do not merge into main strategy docs without review.
|
|
||||||
|
|
||||||
---
|
## Artifacts
|
||||||
|
|
||||||
## Implementation notes
|
- `reports/earnings-2a-gap-20260720-dolthub-final.json` and companion Markdown
|
||||||
|
- `reports/earnings-2b-sue-20260720-dolthub-final.json` and companion Markdown
|
||||||
|
- `reports/earnings-backfill-status.json`
|
||||||
|
|
||||||
| piece | role |
|
Production changes: **none**. No earnings filter or SUE integration was implemented.
|
||||||
|---|---|
|
|
||||||
| `scripts/backfill_earnings_events.py` | bulk attempt → FMP/AV per-symbol; `earnings_events` + meta on snapshot |
|
## Final status: **Task 2 CLOSED (SUE DEAD)**
|
||||||
| `scripts/run_earnings_research.py` | 2a trade join + 2b SUE IC / mom-conditional |
|
|
||||||
| Snapshot table `earnings_events` | real table (not SystemSetting JSON) |
|
|
||||||
|
|||||||
@@ -0,0 +1,348 @@
|
|||||||
|
{
|
||||||
|
"generated_at": "2026-07-20T07:02:10.934892+00:00",
|
||||||
|
"snapshot": "C:\\Workspace\\signal-platform\\backtest_snapshots\\research.sqlite",
|
||||||
|
"snapshot_depth": {
|
||||||
|
"manifest": {
|
||||||
|
"schema_version": 1,
|
||||||
|
"snapshot": "research.sqlite",
|
||||||
|
"snapshot_resolved": "C:\\Workspace\\signal-platform\\backtest_snapshots\\research.sqlite",
|
||||||
|
"complete": true,
|
||||||
|
"finished_at": "2026-07-19T14:22:15.706192+00:00",
|
||||||
|
"ticker_count": 4650,
|
||||||
|
"ohlcv_row_count": 5081073,
|
||||||
|
"rank_only_count": 4144,
|
||||||
|
"sources": {
|
||||||
|
"pool": "source_snapshot"
|
||||||
|
},
|
||||||
|
"history_days": 5000,
|
||||||
|
"min_bars": 1262,
|
||||||
|
"fetch_ok": 505,
|
||||||
|
"fetch_fail": 1,
|
||||||
|
"limit": null,
|
||||||
|
"extra": {
|
||||||
|
"prod_symbols_at_start": 506,
|
||||||
|
"pool_size": 506,
|
||||||
|
"to_fetch": 506,
|
||||||
|
"source_symbols_only": true,
|
||||||
|
"benchmark_spy_rows": 2649
|
||||||
|
},
|
||||||
|
"live_counts": {
|
||||||
|
"ticker_count": 4650,
|
||||||
|
"ohlcv_row_count": 5081073,
|
||||||
|
"rank_only_count": 4144
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"requested_symbols": 506,
|
||||||
|
"tradable_symbols": 505,
|
||||||
|
"missing_symbols": [],
|
||||||
|
"zero_bar_symbols": [
|
||||||
|
"RHM"
|
||||||
|
],
|
||||||
|
"bar_count": {
|
||||||
|
"min": 24,
|
||||||
|
"median": 2649,
|
||||||
|
"max": 2649
|
||||||
|
},
|
||||||
|
"symbols_with_pre2021_bars": 491,
|
||||||
|
"symbols_with_pre2021_bars_pct": 97.2,
|
||||||
|
"shallow_symbols_lt_1000_bars": [
|
||||||
|
{
|
||||||
|
"symbol": "SPCX",
|
||||||
|
"first_bar": "2026-06-12",
|
||||||
|
"last_bar": "2026-07-17",
|
||||||
|
"bars": 24
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"symbol": "Q",
|
||||||
|
"first_bar": "2025-11-03",
|
||||||
|
"last_bar": "2026-07-17",
|
||||||
|
"bars": 176
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"symbol": "PSKY",
|
||||||
|
"first_bar": "2025-08-07",
|
||||||
|
"last_bar": "2026-07-17",
|
||||||
|
"bars": 237
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"symbol": "SNDK",
|
||||||
|
"first_bar": "2025-02-13",
|
||||||
|
"last_bar": "2026-07-17",
|
||||||
|
"bars": 357
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"symbol": "GEV",
|
||||||
|
"first_bar": "2024-04-02",
|
||||||
|
"last_bar": "2026-07-17",
|
||||||
|
"bars": 575
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"symbol": "SOLV",
|
||||||
|
"first_bar": "2024-04-01",
|
||||||
|
"last_bar": "2026-07-17",
|
||||||
|
"bars": 576
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"symbol": "VLTO",
|
||||||
|
"first_bar": "2023-10-02",
|
||||||
|
"last_bar": "2026-07-17",
|
||||||
|
"bars": 700
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"symbol": "KVUE",
|
||||||
|
"first_bar": "2023-05-04",
|
||||||
|
"last_bar": "2026-07-17",
|
||||||
|
"bars": 803
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"symbol": "GEHC",
|
||||||
|
"first_bar": "2022-12-15",
|
||||||
|
"last_bar": "2026-07-17",
|
||||||
|
"bars": 898
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"price_window": {
|
||||||
|
"min": "2016-01-04",
|
||||||
|
"max": "2026-07-17"
|
||||||
|
},
|
||||||
|
"benchmark_spy": {
|
||||||
|
"rows": 2649,
|
||||||
|
"min": "2016-01-04",
|
||||||
|
"max": "2026-07-17"
|
||||||
|
},
|
||||||
|
"gate_threshold": {
|
||||||
|
"min_tradable_symbols": 505,
|
||||||
|
"max_missing_or_zero_bar": 1,
|
||||||
|
"min_symbols_with_pre2021_bars_pct": 80.0,
|
||||||
|
"benchmark_min_rows": 1000,
|
||||||
|
"benchmark_must_begin_pre2021": true
|
||||||
|
},
|
||||||
|
"gate_pass": true
|
||||||
|
},
|
||||||
|
"data_quality": {
|
||||||
|
"window": {
|
||||||
|
"from": "2020-01-22",
|
||||||
|
"to": "2026-07-17"
|
||||||
|
},
|
||||||
|
"prod_symbols": 505,
|
||||||
|
"events": 12414,
|
||||||
|
"symbols_with_any_event": 504,
|
||||||
|
"symbols_with_ge8_announcements": 498,
|
||||||
|
"symbols_with_ge8_announcements_pct": 98.6,
|
||||||
|
"symbols_with_ge8_paired_announcements": 495,
|
||||||
|
"symbols_with_ge8_paired_announcements_pct": 98.0,
|
||||||
|
"events_with_actual_and_estimate": 12311,
|
||||||
|
"events_with_actual_and_estimate_pct": 99.2,
|
||||||
|
"duplicate_rows_in_table": 0,
|
||||||
|
"duplicate_rows_fetched": 0,
|
||||||
|
"restated_rows_fetched": 940,
|
||||||
|
"dedupe_policy": "UNIQUE(symbol, announce_date); normalise dot/dash symbols; retain one calendar row per key; preserve existing non-null session/EPS values from the prior FMP/Alpha Vantage partial backfill, then fill nulls and all remaining symbols from DoltHub; attach DoltHub period-end alignment",
|
||||||
|
"events_per_symbol_year": {
|
||||||
|
"mean_active_span_rate": 4.08,
|
||||||
|
"expected": "approximately 4",
|
||||||
|
"far_off_rule": "active-span rate <2 or >6, plus zero-event symbols",
|
||||||
|
"far_off_count": 1,
|
||||||
|
"far_off_symbols": [
|
||||||
|
{
|
||||||
|
"symbol": "SPCX",
|
||||||
|
"events": 0,
|
||||||
|
"events_per_year": 0.0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"announcement_session": {
|
||||||
|
"recognised_bmo_amc_or_during": 11520,
|
||||||
|
"recognised_pct": 92.8,
|
||||||
|
"reliable": true,
|
||||||
|
"assessment": "usable"
|
||||||
|
},
|
||||||
|
"point_in_time_policy": "announce_date_plus_1_trading_day_for_all_events",
|
||||||
|
"sue_scaling": {
|
||||||
|
"primary": "eps_surprise_over_stdev_of_prior_8_surprises_min_4",
|
||||||
|
"fallback_trigger": "paired event coverage <50% or symbols with >=8 paired events <50%",
|
||||||
|
"fallback_needed": false,
|
||||||
|
"fallback_name": "not_used"
|
||||||
|
},
|
||||||
|
"backfill": {
|
||||||
|
"mode": "dolthub_public_bulk_clone",
|
||||||
|
"window": {
|
||||||
|
"from": "2020-01-22",
|
||||||
|
"to": "2026-07-17"
|
||||||
|
},
|
||||||
|
"coverage_amendment": {
|
||||||
|
"approved_by_user": true,
|
||||||
|
"reason": "FMP free tier blocks historical bulk earnings",
|
||||||
|
"original_start": "2016-01-04",
|
||||||
|
"amended_announcement_start": "2020-01-22"
|
||||||
|
},
|
||||||
|
"source": {
|
||||||
|
"repository": "https://www.dolthub.com/repositories/post-no-preference/earnings",
|
||||||
|
"commit": "9n0et3hpj9j7vue8f3qsldon3qa5sdjj",
|
||||||
|
"license": "CC-BY-SA-4.0",
|
||||||
|
"upstream_provider_documented": false
|
||||||
|
},
|
||||||
|
"bulk_windows_total": 1,
|
||||||
|
"bulk_windows_done": 1,
|
||||||
|
"bulk_requests_logged_total": 1,
|
||||||
|
"bulk_exports": 2,
|
||||||
|
"calendar": {
|
||||||
|
"raw_rows": 117482,
|
||||||
|
"universe_rows_in_window": 12342,
|
||||||
|
"deduped_rows_in_window": 12342,
|
||||||
|
"duplicate_rows": 0,
|
||||||
|
"restated_rows": 0
|
||||||
|
},
|
||||||
|
"eps_history": {
|
||||||
|
"raw_rows": 165050,
|
||||||
|
"universe_rows": 18515,
|
||||||
|
"deduped_rows": 18515,
|
||||||
|
"duplicate_rows": 0,
|
||||||
|
"restated_rows": 0,
|
||||||
|
"complete_actual_and_estimate": 18304
|
||||||
|
},
|
||||||
|
"pairing": {
|
||||||
|
"method": "minimum-cost monotonic alignment per symbol",
|
||||||
|
"allowed_announce_minus_period_end_days": [
|
||||||
|
-14,
|
||||||
|
90
|
||||||
|
],
|
||||||
|
"matched_calendar_events": 12271,
|
||||||
|
"unmatched_calendar_events": 71,
|
||||||
|
"unmatched_periods_in_pairing_window": 538,
|
||||||
|
"announce_minus_period_end_days": {
|
||||||
|
"min": -10,
|
||||||
|
"median": 30,
|
||||||
|
"max": 89
|
||||||
|
},
|
||||||
|
"pre_2020_eps_history_use": "trailing_surprise_stdev_only; never treated as an announcement or live signal event"
|
||||||
|
},
|
||||||
|
"duplicate_rows_logged_total": 0,
|
||||||
|
"restated_rows_logged_total": 940,
|
||||||
|
"conflicting_existing_rows": 940,
|
||||||
|
"conflicting_existing_fields": 1526,
|
||||||
|
"preserved_existing_fields": 2945,
|
||||||
|
"existing_enrichment_events_not_in_dolthub_calendar": 72,
|
||||||
|
"dedupe_policy": "UNIQUE(symbol, announce_date); normalise dot/dash symbols; retain one calendar row per key; preserve existing non-null session/EPS values from the prior FMP/Alpha Vantage partial backfill, then fill nulls and all remaining symbols from DoltHub; attach DoltHub period-end alignment",
|
||||||
|
"events_in_window": 12414,
|
||||||
|
"events_with_actual_and_estimate": 12311,
|
||||||
|
"symbols_done": 506,
|
||||||
|
"symbols_universe": 506,
|
||||||
|
"symbols_with_dolthub_calendar": 504,
|
||||||
|
"symbols_without_dolthub_calendar": [
|
||||||
|
"RHM",
|
||||||
|
"SPCX"
|
||||||
|
],
|
||||||
|
"announce_date_range": {
|
||||||
|
"min": "2020-01-22",
|
||||||
|
"max": "2026-07-17"
|
||||||
|
},
|
||||||
|
"complete": true
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"production_impact": "none",
|
||||||
|
"experiment": "2a",
|
||||||
|
"result": {
|
||||||
|
"verdict": "INFORMATIONAL",
|
||||||
|
"costs": {
|
||||||
|
"per_side": 0.001,
|
||||||
|
"r_is_net_of_round_trip_costs": true
|
||||||
|
},
|
||||||
|
"closed_trades": 574,
|
||||||
|
"q1_loss_concentration": {
|
||||||
|
"loss_definition": "realized_net_R <= -1.0",
|
||||||
|
"holding_period_definition": "announcement strictly after entry and before exit",
|
||||||
|
"losses_count": 266,
|
||||||
|
"losses_with_announcement_count": 23,
|
||||||
|
"losses_with_announcement_fraction": 0.0865,
|
||||||
|
"all_trades_with_announcement_count": 115,
|
||||||
|
"all_trades_with_announcement_fraction": 0.2003
|
||||||
|
},
|
||||||
|
"q2_entries_within_3_trading_days_before_announcement": {
|
||||||
|
"pre_earnings": {
|
||||||
|
"count": 27,
|
||||||
|
"mean_r": 0.4837,
|
||||||
|
"median_r": -1.0265,
|
||||||
|
"win_rate": 0.3333,
|
||||||
|
"p05_r": -1.1463,
|
||||||
|
"p95_r": 5.981,
|
||||||
|
"min_r": -1.2449,
|
||||||
|
"max_r": 8.8996
|
||||||
|
},
|
||||||
|
"all_other_entries": {
|
||||||
|
"count": 547,
|
||||||
|
"mean_r": 0.2734,
|
||||||
|
"median_r": -0.8316,
|
||||||
|
"win_rate": 0.3565,
|
||||||
|
"p05_r": -1.1228,
|
||||||
|
"p95_r": 4.5888,
|
||||||
|
"min_r": -6.0161,
|
||||||
|
"max_r": 19.98
|
||||||
|
},
|
||||||
|
"tail_deltas_pre_minus_other": {
|
||||||
|
"p05_r": -0.0235,
|
||||||
|
"p95_r": 1.3922
|
||||||
|
},
|
||||||
|
"directional_tail_condition_present": false,
|
||||||
|
"tail_read": "Registered directional tail condition is not present."
|
||||||
|
},
|
||||||
|
"q3_stop_exits_within_1_trading_day_after_announcement": {
|
||||||
|
"stops_after_earnings": {
|
||||||
|
"count": 26,
|
||||||
|
"mean_r": -0.6434,
|
||||||
|
"median_r": -0.9753,
|
||||||
|
"win_rate": 0.2308,
|
||||||
|
"p05_r": -2.4614,
|
||||||
|
"p95_r": 1.1142,
|
||||||
|
"min_r": -2.6413,
|
||||||
|
"max_r": 1.7012
|
||||||
|
},
|
||||||
|
"all_other_stops": {
|
||||||
|
"count": 433,
|
||||||
|
"mean_r": -0.5035,
|
||||||
|
"median_r": -1.0278,
|
||||||
|
"win_rate": 0.1963,
|
||||||
|
"p05_r": -1.1373,
|
||||||
|
"p95_r": 1.3591,
|
||||||
|
"min_r": -6.0161,
|
||||||
|
"max_r": 19.98
|
||||||
|
},
|
||||||
|
"all_other_exits": {
|
||||||
|
"count": 548,
|
||||||
|
"mean_r": 0.3273,
|
||||||
|
"median_r": -0.8361,
|
||||||
|
"win_rate": 0.3613,
|
||||||
|
"p05_r": -1.0644,
|
||||||
|
"p95_r": 4.7224,
|
||||||
|
"min_r": -6.0161,
|
||||||
|
"max_r": 19.98
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"implementation": "REPORT_ONLY_NO_FILTER_ARM_NO_FILTER_CHANGE",
|
||||||
|
"analysis_window": {
|
||||||
|
"from": "2020-01-22",
|
||||||
|
"to": "2026-07-17",
|
||||||
|
"rule": "entry_on_or_after_start_and_exit_on_or_before_end",
|
||||||
|
"simulation_trades_total": 574,
|
||||||
|
"trades_excluded_outside_earnings_coverage": 0
|
||||||
|
},
|
||||||
|
"run_config": {
|
||||||
|
"universe_symbols": 505,
|
||||||
|
"fill_mode": "close",
|
||||||
|
"cost_per_side": 0.001,
|
||||||
|
"momentum_cutoff": 80.0,
|
||||||
|
"exit_policy": "atr_trail3",
|
||||||
|
"hold_days": 30,
|
||||||
|
"max_positions": 10,
|
||||||
|
"risk_per_trade": 0.01
|
||||||
|
},
|
||||||
|
"sim_summary": {
|
||||||
|
"start_date": "2020-01-22",
|
||||||
|
"end_date": "2026-07-13",
|
||||||
|
"trades": 574,
|
||||||
|
"sharpe": 1.03,
|
||||||
|
"cagr_pct": 22.9,
|
||||||
|
"max_drawdown_pct": 26.6,
|
||||||
|
"total_return_pct": 280.6
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,58 @@
|
|||||||
|
# Earnings Task 2a - gap diagnostic
|
||||||
|
|
||||||
|
### Data quality gate
|
||||||
|
|
||||||
|
Approved earnings window: 2020-01-22 to 2026-07-17. Source mode: dolthub_public_bulk_clone.
|
||||||
|
|
||||||
|
| check | result |
|
||||||
|
|---|---:|
|
||||||
|
| Prod symbols requested / tradable | 506 / 505 |
|
||||||
|
| Manifest complete + live counts match | True |
|
||||||
|
| Prod symbols with pre-2021 bars | 491 (97.2%) |
|
||||||
|
| SPY benchmark depth | 2649 rows, 2016-01-04 to 2026-07-17 |
|
||||||
|
| Snapshot depth gate | True |
|
||||||
|
| Bulk source windows / requests logged | 1/1 / 1 |
|
||||||
|
| Source repository / pinned commit | https://www.dolthub.com/repositories/post-no-preference/earnings @ 9n0et3hpj9j7vue8f3qsldon3qa5sdjj |
|
||||||
|
| Source license / upstream provider documented | CC-BY-SA-4.0 / False |
|
||||||
|
| Existing-source conflicts preserved | 940 rows / 1526 fields |
|
||||||
|
| Symbols with >=8 announcements | 498 (98.6%) |
|
||||||
|
| Symbols with >=8 paired announcements | 495 (98.0%) |
|
||||||
|
| Events with estimate + actual | 12311/12414 (99.2%) |
|
||||||
|
| Duplicate rows in keyed table | 0 |
|
||||||
|
| Duplicate / restated payload rows fetched | 0 / 940 |
|
||||||
|
| Mean announcements per active symbol-year | 4.08 (expected about 4) |
|
||||||
|
| Symbols far off (<2 or >6/year, incl. zero) | 1 |
|
||||||
|
| Recognised BMO/AMC/during | 92.8% (reliable=True) |
|
||||||
|
| Point-in-time policy | announce_date_plus_1_trading_day_for_all_events |
|
||||||
|
| SUE price fallback | not_used |
|
||||||
|
|
||||||
|
Deduplication: UNIQUE(symbol, announce_date); normalise dot/dash symbols; retain one calendar row per key; preserve existing non-null session/EPS values from the prior FMP/Alpha Vantage partial backfill, then fill nulls and all remaining symbols from DoltHub; attach DoltHub period-end alignment
|
||||||
|
|
||||||
|
Far-off announcement-rate symbols: SPCX
|
||||||
|
|
||||||
|
### Experiment 2a - earnings-gap risk diagnostic
|
||||||
|
|
||||||
|
Verdict: **INFORMATIONAL**. Report-only; no filter arm or implementation.
|
||||||
|
|
||||||
|
Trade cohort is restricted to the approved earnings-coverage window 2020-01-22 to 2026-07-17; 0 simulated trades outside that window were excluded.
|
||||||
|
|
||||||
|
| cohort | count | fraction |
|
||||||
|
|---|---:|---:|
|
||||||
|
| Realized net R <= -1.0 | 266 | - |
|
||||||
|
| Losses with announcement strictly inside hold | 23 | 0.0865 |
|
||||||
|
| All trades with announcement strictly inside hold | 115 | 0.2003 |
|
||||||
|
|
||||||
|
| Entry cohort | count | mean R | median R | win rate | p05 R | p95 R |
|
||||||
|
|---|---:|---:|---:|---:|---:|---:|
|
||||||
|
| Within 3 sessions before earnings | 27 | 0.4837 | -1.0265 | 0.3333 | -1.1463 | 5.981 |
|
||||||
|
| All other entries | 547 | 0.2734 | -0.8316 | 0.3565 | -1.1228 | 4.5888 |
|
||||||
|
|
||||||
|
Tail deltas (pre minus other): p05=-0.0235, p95=1.3922.
|
||||||
|
|
||||||
|
Registered directional tail condition is not present.
|
||||||
|
|
||||||
|
| Exit cohort | count | mean R | median R | win rate | p05 R | p95 R |
|
||||||
|
|---|---:|---:|---:|---:|---:|---:|
|
||||||
|
| Stops within 1 session after earnings | 26 | -0.6434 | -0.9753 | 0.2308 | -2.4614 | 1.1142 |
|
||||||
|
| All other stops | 433 | -0.5035 | -1.0278 | 0.1963 | -1.1373 | 1.3591 |
|
||||||
|
| All other exits | 548 | 0.3273 | -0.8361 | 0.3613 | -1.0644 | 4.7224 |
|
||||||
@@ -0,0 +1,349 @@
|
|||||||
|
{
|
||||||
|
"generated_at": "2026-07-20T07:02:10.934892+00:00",
|
||||||
|
"snapshot": "C:\\Workspace\\signal-platform\\backtest_snapshots\\research.sqlite",
|
||||||
|
"snapshot_depth": {
|
||||||
|
"manifest": {
|
||||||
|
"schema_version": 1,
|
||||||
|
"snapshot": "research.sqlite",
|
||||||
|
"snapshot_resolved": "C:\\Workspace\\signal-platform\\backtest_snapshots\\research.sqlite",
|
||||||
|
"complete": true,
|
||||||
|
"finished_at": "2026-07-19T14:22:15.706192+00:00",
|
||||||
|
"ticker_count": 4650,
|
||||||
|
"ohlcv_row_count": 5081073,
|
||||||
|
"rank_only_count": 4144,
|
||||||
|
"sources": {
|
||||||
|
"pool": "source_snapshot"
|
||||||
|
},
|
||||||
|
"history_days": 5000,
|
||||||
|
"min_bars": 1262,
|
||||||
|
"fetch_ok": 505,
|
||||||
|
"fetch_fail": 1,
|
||||||
|
"limit": null,
|
||||||
|
"extra": {
|
||||||
|
"prod_symbols_at_start": 506,
|
||||||
|
"pool_size": 506,
|
||||||
|
"to_fetch": 506,
|
||||||
|
"source_symbols_only": true,
|
||||||
|
"benchmark_spy_rows": 2649
|
||||||
|
},
|
||||||
|
"live_counts": {
|
||||||
|
"ticker_count": 4650,
|
||||||
|
"ohlcv_row_count": 5081073,
|
||||||
|
"rank_only_count": 4144
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"requested_symbols": 506,
|
||||||
|
"tradable_symbols": 505,
|
||||||
|
"missing_symbols": [],
|
||||||
|
"zero_bar_symbols": [
|
||||||
|
"RHM"
|
||||||
|
],
|
||||||
|
"bar_count": {
|
||||||
|
"min": 24,
|
||||||
|
"median": 2649,
|
||||||
|
"max": 2649
|
||||||
|
},
|
||||||
|
"symbols_with_pre2021_bars": 491,
|
||||||
|
"symbols_with_pre2021_bars_pct": 97.2,
|
||||||
|
"shallow_symbols_lt_1000_bars": [
|
||||||
|
{
|
||||||
|
"symbol": "SPCX",
|
||||||
|
"first_bar": "2026-06-12",
|
||||||
|
"last_bar": "2026-07-17",
|
||||||
|
"bars": 24
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"symbol": "Q",
|
||||||
|
"first_bar": "2025-11-03",
|
||||||
|
"last_bar": "2026-07-17",
|
||||||
|
"bars": 176
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"symbol": "PSKY",
|
||||||
|
"first_bar": "2025-08-07",
|
||||||
|
"last_bar": "2026-07-17",
|
||||||
|
"bars": 237
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"symbol": "SNDK",
|
||||||
|
"first_bar": "2025-02-13",
|
||||||
|
"last_bar": "2026-07-17",
|
||||||
|
"bars": 357
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"symbol": "GEV",
|
||||||
|
"first_bar": "2024-04-02",
|
||||||
|
"last_bar": "2026-07-17",
|
||||||
|
"bars": 575
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"symbol": "SOLV",
|
||||||
|
"first_bar": "2024-04-01",
|
||||||
|
"last_bar": "2026-07-17",
|
||||||
|
"bars": 576
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"symbol": "VLTO",
|
||||||
|
"first_bar": "2023-10-02",
|
||||||
|
"last_bar": "2026-07-17",
|
||||||
|
"bars": 700
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"symbol": "KVUE",
|
||||||
|
"first_bar": "2023-05-04",
|
||||||
|
"last_bar": "2026-07-17",
|
||||||
|
"bars": 803
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"symbol": "GEHC",
|
||||||
|
"first_bar": "2022-12-15",
|
||||||
|
"last_bar": "2026-07-17",
|
||||||
|
"bars": 898
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"price_window": {
|
||||||
|
"min": "2016-01-04",
|
||||||
|
"max": "2026-07-17"
|
||||||
|
},
|
||||||
|
"benchmark_spy": {
|
||||||
|
"rows": 2649,
|
||||||
|
"min": "2016-01-04",
|
||||||
|
"max": "2026-07-17"
|
||||||
|
},
|
||||||
|
"gate_threshold": {
|
||||||
|
"min_tradable_symbols": 505,
|
||||||
|
"max_missing_or_zero_bar": 1,
|
||||||
|
"min_symbols_with_pre2021_bars_pct": 80.0,
|
||||||
|
"benchmark_min_rows": 1000,
|
||||||
|
"benchmark_must_begin_pre2021": true
|
||||||
|
},
|
||||||
|
"gate_pass": true
|
||||||
|
},
|
||||||
|
"data_quality": {
|
||||||
|
"window": {
|
||||||
|
"from": "2020-01-22",
|
||||||
|
"to": "2026-07-17"
|
||||||
|
},
|
||||||
|
"prod_symbols": 505,
|
||||||
|
"events": 12414,
|
||||||
|
"symbols_with_any_event": 504,
|
||||||
|
"symbols_with_ge8_announcements": 498,
|
||||||
|
"symbols_with_ge8_announcements_pct": 98.6,
|
||||||
|
"symbols_with_ge8_paired_announcements": 495,
|
||||||
|
"symbols_with_ge8_paired_announcements_pct": 98.0,
|
||||||
|
"events_with_actual_and_estimate": 12311,
|
||||||
|
"events_with_actual_and_estimate_pct": 99.2,
|
||||||
|
"duplicate_rows_in_table": 0,
|
||||||
|
"duplicate_rows_fetched": 0,
|
||||||
|
"restated_rows_fetched": 940,
|
||||||
|
"dedupe_policy": "UNIQUE(symbol, announce_date); normalise dot/dash symbols; retain one calendar row per key; preserve existing non-null session/EPS values from the prior FMP/Alpha Vantage partial backfill, then fill nulls and all remaining symbols from DoltHub; attach DoltHub period-end alignment",
|
||||||
|
"events_per_symbol_year": {
|
||||||
|
"mean_active_span_rate": 4.08,
|
||||||
|
"expected": "approximately 4",
|
||||||
|
"far_off_rule": "active-span rate <2 or >6, plus zero-event symbols",
|
||||||
|
"far_off_count": 1,
|
||||||
|
"far_off_symbols": [
|
||||||
|
{
|
||||||
|
"symbol": "SPCX",
|
||||||
|
"events": 0,
|
||||||
|
"events_per_year": 0.0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"announcement_session": {
|
||||||
|
"recognised_bmo_amc_or_during": 11520,
|
||||||
|
"recognised_pct": 92.8,
|
||||||
|
"reliable": true,
|
||||||
|
"assessment": "usable"
|
||||||
|
},
|
||||||
|
"point_in_time_policy": "announce_date_plus_1_trading_day_for_all_events",
|
||||||
|
"sue_scaling": {
|
||||||
|
"primary": "eps_surprise_over_stdev_of_prior_8_surprises_min_4",
|
||||||
|
"fallback_trigger": "paired event coverage <50% or symbols with >=8 paired events <50%",
|
||||||
|
"fallback_needed": false,
|
||||||
|
"fallback_name": "not_used"
|
||||||
|
},
|
||||||
|
"backfill": {
|
||||||
|
"mode": "dolthub_public_bulk_clone",
|
||||||
|
"window": {
|
||||||
|
"from": "2020-01-22",
|
||||||
|
"to": "2026-07-17"
|
||||||
|
},
|
||||||
|
"coverage_amendment": {
|
||||||
|
"approved_by_user": true,
|
||||||
|
"reason": "FMP free tier blocks historical bulk earnings",
|
||||||
|
"original_start": "2016-01-04",
|
||||||
|
"amended_announcement_start": "2020-01-22"
|
||||||
|
},
|
||||||
|
"source": {
|
||||||
|
"repository": "https://www.dolthub.com/repositories/post-no-preference/earnings",
|
||||||
|
"commit": "9n0et3hpj9j7vue8f3qsldon3qa5sdjj",
|
||||||
|
"license": "CC-BY-SA-4.0",
|
||||||
|
"upstream_provider_documented": false
|
||||||
|
},
|
||||||
|
"bulk_windows_total": 1,
|
||||||
|
"bulk_windows_done": 1,
|
||||||
|
"bulk_requests_logged_total": 1,
|
||||||
|
"bulk_exports": 2,
|
||||||
|
"calendar": {
|
||||||
|
"raw_rows": 117482,
|
||||||
|
"universe_rows_in_window": 12342,
|
||||||
|
"deduped_rows_in_window": 12342,
|
||||||
|
"duplicate_rows": 0,
|
||||||
|
"restated_rows": 0
|
||||||
|
},
|
||||||
|
"eps_history": {
|
||||||
|
"raw_rows": 165050,
|
||||||
|
"universe_rows": 18515,
|
||||||
|
"deduped_rows": 18515,
|
||||||
|
"duplicate_rows": 0,
|
||||||
|
"restated_rows": 0,
|
||||||
|
"complete_actual_and_estimate": 18304
|
||||||
|
},
|
||||||
|
"pairing": {
|
||||||
|
"method": "minimum-cost monotonic alignment per symbol",
|
||||||
|
"allowed_announce_minus_period_end_days": [
|
||||||
|
-14,
|
||||||
|
90
|
||||||
|
],
|
||||||
|
"matched_calendar_events": 12271,
|
||||||
|
"unmatched_calendar_events": 71,
|
||||||
|
"unmatched_periods_in_pairing_window": 538,
|
||||||
|
"announce_minus_period_end_days": {
|
||||||
|
"min": -10,
|
||||||
|
"median": 30,
|
||||||
|
"max": 89
|
||||||
|
},
|
||||||
|
"pre_2020_eps_history_use": "trailing_surprise_stdev_only; never treated as an announcement or live signal event"
|
||||||
|
},
|
||||||
|
"duplicate_rows_logged_total": 0,
|
||||||
|
"restated_rows_logged_total": 940,
|
||||||
|
"conflicting_existing_rows": 940,
|
||||||
|
"conflicting_existing_fields": 1526,
|
||||||
|
"preserved_existing_fields": 2945,
|
||||||
|
"existing_enrichment_events_not_in_dolthub_calendar": 72,
|
||||||
|
"dedupe_policy": "UNIQUE(symbol, announce_date); normalise dot/dash symbols; retain one calendar row per key; preserve existing non-null session/EPS values from the prior FMP/Alpha Vantage partial backfill, then fill nulls and all remaining symbols from DoltHub; attach DoltHub period-end alignment",
|
||||||
|
"events_in_window": 12414,
|
||||||
|
"events_with_actual_and_estimate": 12311,
|
||||||
|
"symbols_done": 506,
|
||||||
|
"symbols_universe": 506,
|
||||||
|
"symbols_with_dolthub_calendar": 504,
|
||||||
|
"symbols_without_dolthub_calendar": [
|
||||||
|
"RHM",
|
||||||
|
"SPCX"
|
||||||
|
],
|
||||||
|
"announce_date_range": {
|
||||||
|
"min": "2020-01-22",
|
||||||
|
"max": "2026-07-17"
|
||||||
|
},
|
||||||
|
"complete": true
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"production_impact": "none",
|
||||||
|
"experiment": "2b",
|
||||||
|
"result": {
|
||||||
|
"verdict": "FAIL",
|
||||||
|
"verdict_detail": "SUE DEAD for this stack",
|
||||||
|
"grade_rule": {
|
||||||
|
"mean_ic_ge_0_03_positive": false,
|
||||||
|
"reliable_ge_12_windows": true,
|
||||||
|
"positive_sign_pre_and_post_2021": true,
|
||||||
|
"pass": false
|
||||||
|
},
|
||||||
|
"sue_unconditional": {
|
||||||
|
"signal": "sue_latest",
|
||||||
|
"weeks": 56,
|
||||||
|
"avg_cross_section": 451.4,
|
||||||
|
"mean_ic": 0.0151,
|
||||||
|
"ic_t_stat": 1.29,
|
||||||
|
"ic_positive_pct": 51.8,
|
||||||
|
"mean_quintile_spread": 0.0041,
|
||||||
|
"reliable": true
|
||||||
|
},
|
||||||
|
"era_split": {
|
||||||
|
"pre_2021": {
|
||||||
|
"signal": "sue_latest",
|
||||||
|
"weeks": 9,
|
||||||
|
"avg_cross_section": 398.3,
|
||||||
|
"mean_ic": 0.0286,
|
||||||
|
"ic_t_stat": 0.7,
|
||||||
|
"ic_positive_pct": 55.6,
|
||||||
|
"mean_quintile_spread": 0.0084,
|
||||||
|
"reliable": false
|
||||||
|
},
|
||||||
|
"post_2021": {
|
||||||
|
"signal": "sue_latest",
|
||||||
|
"weeks": 48,
|
||||||
|
"avg_cross_section": 461.9,
|
||||||
|
"mean_ic": 0.0172,
|
||||||
|
"ic_t_stat": 1.34,
|
||||||
|
"ic_positive_pct": 64.6,
|
||||||
|
"mean_quintile_spread": 0.0038,
|
||||||
|
"reliable": true
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"signal_eval_identical_cross_sections": {
|
||||||
|
"sue_latest": {
|
||||||
|
"signal": "sue_latest",
|
||||||
|
"weeks": 56,
|
||||||
|
"avg_cross_section": 450.9,
|
||||||
|
"mean_ic": 0.0148,
|
||||||
|
"ic_t_stat": 1.27,
|
||||||
|
"ic_positive_pct": 51.8,
|
||||||
|
"mean_quintile_spread": 0.004,
|
||||||
|
"reliable": true
|
||||||
|
},
|
||||||
|
"mom_12_1": {
|
||||||
|
"signal": "mom_12_1",
|
||||||
|
"weeks": 56,
|
||||||
|
"avg_cross_section": 450.9,
|
||||||
|
"mean_ic": 0.0195,
|
||||||
|
"ic_t_stat": 0.74,
|
||||||
|
"ic_positive_pct": 58.9,
|
||||||
|
"mean_quintile_spread": 0.0104,
|
||||||
|
"reliable": true
|
||||||
|
},
|
||||||
|
"mom_12_1_resid": {
|
||||||
|
"signal": "mom_12_1_resid",
|
||||||
|
"weeks": 56,
|
||||||
|
"avg_cross_section": 450.9,
|
||||||
|
"mean_ic": 0.0262,
|
||||||
|
"ic_t_stat": 1.07,
|
||||||
|
"ic_positive_pct": 55.4,
|
||||||
|
"mean_quintile_spread": 0.0114,
|
||||||
|
"reliable": true
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"identical_cross_section_definition": "same week-symbol-forward-return cells where sue_latest, mom_12_1, and mom_12_1_resid are all non-null",
|
||||||
|
"momentum_conditional_top_quintile": {
|
||||||
|
"mean_ic": 0.0213,
|
||||||
|
"ic_t_stat": 1.3,
|
||||||
|
"weeks": 56,
|
||||||
|
"avg_cross_section": 89.8,
|
||||||
|
"population": "top_mom_12_1_quintile_only"
|
||||||
|
},
|
||||||
|
"coverage": {
|
||||||
|
"symbols_with_live_sue": 501,
|
||||||
|
"avg_weekly_live_n_all_weeks": 453.1,
|
||||||
|
"avg_cross_section_n_scored_nonoverlap": 451.4,
|
||||||
|
"thin_cross_section_lt_100": false,
|
||||||
|
"warning": null
|
||||||
|
},
|
||||||
|
"scaling": {
|
||||||
|
"method": "eps_surprise_over_stdev_of_prior_8_surprises_min_4",
|
||||||
|
"fallback": "not_used",
|
||||||
|
"counts": {
|
||||||
|
"standard_scaled_events": 12149,
|
||||||
|
"events_scaled_from_period_history": 12149,
|
||||||
|
"price_fallback_events": 0,
|
||||||
|
"dropped_insufficient_trailing_history": 95,
|
||||||
|
"dropped_missing_period_alignment": 67,
|
||||||
|
"dropped_zero_stdev": 0
|
||||||
|
},
|
||||||
|
"pre_coverage_history_policy": "period-end EPS surprises may scale later events but are never treated as live signals without an announcement date",
|
||||||
|
"availability": "announce_date_plus_1_trading_day",
|
||||||
|
"carry_trading_days": 63
|
||||||
|
},
|
||||||
|
"universe_symbols": 505
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,62 @@
|
|||||||
|
# Earnings Task 2b - SUE / PEAD
|
||||||
|
|
||||||
|
### Data quality gate
|
||||||
|
|
||||||
|
Approved earnings window: 2020-01-22 to 2026-07-17. Source mode: dolthub_public_bulk_clone.
|
||||||
|
|
||||||
|
| check | result |
|
||||||
|
|---|---:|
|
||||||
|
| Prod symbols requested / tradable | 506 / 505 |
|
||||||
|
| Manifest complete + live counts match | True |
|
||||||
|
| Prod symbols with pre-2021 bars | 491 (97.2%) |
|
||||||
|
| SPY benchmark depth | 2649 rows, 2016-01-04 to 2026-07-17 |
|
||||||
|
| Snapshot depth gate | True |
|
||||||
|
| Bulk source windows / requests logged | 1/1 / 1 |
|
||||||
|
| Source repository / pinned commit | https://www.dolthub.com/repositories/post-no-preference/earnings @ 9n0et3hpj9j7vue8f3qsldon3qa5sdjj |
|
||||||
|
| Source license / upstream provider documented | CC-BY-SA-4.0 / False |
|
||||||
|
| Existing-source conflicts preserved | 940 rows / 1526 fields |
|
||||||
|
| Symbols with >=8 announcements | 498 (98.6%) |
|
||||||
|
| Symbols with >=8 paired announcements | 495 (98.0%) |
|
||||||
|
| Events with estimate + actual | 12311/12414 (99.2%) |
|
||||||
|
| Duplicate rows in keyed table | 0 |
|
||||||
|
| Duplicate / restated payload rows fetched | 0 / 940 |
|
||||||
|
| Mean announcements per active symbol-year | 4.08 (expected about 4) |
|
||||||
|
| Symbols far off (<2 or >6/year, incl. zero) | 1 |
|
||||||
|
| Recognised BMO/AMC/during | 92.8% (reliable=True) |
|
||||||
|
| Point-in-time policy | announce_date_plus_1_trading_day_for_all_events |
|
||||||
|
| SUE price fallback | not_used |
|
||||||
|
|
||||||
|
Deduplication: UNIQUE(symbol, announce_date); normalise dot/dash symbols; retain one calendar row per key; preserve existing non-null session/EPS values from the prior FMP/Alpha Vantage partial backfill, then fill nulls and all remaining symbols from DoltHub; attach DoltHub period-end alignment
|
||||||
|
|
||||||
|
Far-off announcement-rate symbols: SPCX
|
||||||
|
|
||||||
|
### Experiment 2b - SUE / post-earnings drift
|
||||||
|
|
||||||
|
Mechanical verdict: **FAIL** - SUE DEAD for this stack
|
||||||
|
|
||||||
|
Identical cross-sections:
|
||||||
|
|
||||||
|
| signal | mean IC | t | windows | avg N | IC positive % | reliable |
|
||||||
|
|---|---:|---:|---:|---:|---:|---|
|
||||||
|
| sue_latest | 0.0148 | 1.27 | 56 | 450.9 | 51.8 | true |
|
||||||
|
| mom_12_1 | 0.0195 | 0.74 | 56 | 450.9 | 58.9 | true |
|
||||||
|
| mom_12_1_resid | 0.0262 | 1.07 | 56 | 450.9 | 55.4 | true |
|
||||||
|
|
||||||
|
Unconditional SUE grade row:
|
||||||
|
|
||||||
|
| signal | mean IC | t | windows | avg N | IC positive % | reliable |
|
||||||
|
|---|---:|---:|---:|---:|---:|---|
|
||||||
|
| sue_latest | 0.0151 | 1.29 | 56 | 451.4 | 51.8 | true |
|
||||||
|
|
||||||
|
Era stability:
|
||||||
|
|
||||||
|
| era | mean IC | t | windows | avg N | IC positive % | reliable |
|
||||||
|
|---|---:|---:|---:|---:|---:|---|
|
||||||
|
| pre-2021 | 0.0286 | 0.7 | 9 | 398.3 | 55.6 | false |
|
||||||
|
| post-2021 | 0.0172 | 1.34 | 48 | 461.9 | 64.6 | true |
|
||||||
|
|
||||||
|
Coverage: 501 symbols with live SUE; avg weekly N=453.1; scored non-overlap avg N=451.4.
|
||||||
|
|
||||||
|
Cross-section is not flagged thin at the registered <100-name read.
|
||||||
|
|
||||||
|
Momentum-conditional top-quintile SUE: mean IC=0.0213, t=1.3, windows=56, avg N=89.8.
|
||||||
@@ -1,15 +1,75 @@
|
|||||||
{
|
{
|
||||||
"mode": "per_symbol",
|
"mode": "dolthub_public_bulk_clone",
|
||||||
"fmp_requests": 25,
|
"window": {
|
||||||
"events_written_this_run": 2541,
|
"from": "2020-01-22",
|
||||||
"total_events": 5612,
|
"to": "2026-07-17"
|
||||||
"symbols_done": 48,
|
|
||||||
"symbols_universe": 506,
|
|
||||||
"announce_date_range": {
|
|
||||||
"min": "1985-08-31",
|
|
||||||
"max": "2026-07-16"
|
|
||||||
},
|
},
|
||||||
"events_with_actual_and_estimate": 5018,
|
"coverage_amendment": {
|
||||||
"budget": 25,
|
"approved_by_user": true,
|
||||||
"complete": false
|
"reason": "FMP free tier blocks historical bulk earnings",
|
||||||
|
"original_start": "2016-01-04",
|
||||||
|
"amended_announcement_start": "2020-01-22"
|
||||||
|
},
|
||||||
|
"source": {
|
||||||
|
"repository": "https://www.dolthub.com/repositories/post-no-preference/earnings",
|
||||||
|
"commit": "9n0et3hpj9j7vue8f3qsldon3qa5sdjj",
|
||||||
|
"license": "CC-BY-SA-4.0",
|
||||||
|
"upstream_provider_documented": false
|
||||||
|
},
|
||||||
|
"bulk_windows_total": 1,
|
||||||
|
"bulk_windows_done": 1,
|
||||||
|
"bulk_requests_logged_total": 1,
|
||||||
|
"bulk_exports": 2,
|
||||||
|
"calendar": {
|
||||||
|
"raw_rows": 117482,
|
||||||
|
"universe_rows_in_window": 12342,
|
||||||
|
"deduped_rows_in_window": 12342,
|
||||||
|
"duplicate_rows": 0,
|
||||||
|
"restated_rows": 0
|
||||||
|
},
|
||||||
|
"eps_history": {
|
||||||
|
"raw_rows": 165050,
|
||||||
|
"universe_rows": 18515,
|
||||||
|
"deduped_rows": 18515,
|
||||||
|
"duplicate_rows": 0,
|
||||||
|
"restated_rows": 0,
|
||||||
|
"complete_actual_and_estimate": 18304
|
||||||
|
},
|
||||||
|
"pairing": {
|
||||||
|
"method": "minimum-cost monotonic alignment per symbol",
|
||||||
|
"allowed_announce_minus_period_end_days": [
|
||||||
|
-14,
|
||||||
|
90
|
||||||
|
],
|
||||||
|
"matched_calendar_events": 12271,
|
||||||
|
"unmatched_calendar_events": 71,
|
||||||
|
"unmatched_periods_in_pairing_window": 538,
|
||||||
|
"announce_minus_period_end_days": {
|
||||||
|
"min": -10,
|
||||||
|
"median": 30,
|
||||||
|
"max": 89
|
||||||
|
},
|
||||||
|
"pre_2020_eps_history_use": "trailing_surprise_stdev_only; never treated as an announcement or live signal event"
|
||||||
|
},
|
||||||
|
"duplicate_rows_logged_total": 0,
|
||||||
|
"restated_rows_logged_total": 940,
|
||||||
|
"conflicting_existing_rows": 940,
|
||||||
|
"conflicting_existing_fields": 1526,
|
||||||
|
"preserved_existing_fields": 2945,
|
||||||
|
"existing_enrichment_events_not_in_dolthub_calendar": 72,
|
||||||
|
"dedupe_policy": "UNIQUE(symbol, announce_date); normalise dot/dash symbols; retain one calendar row per key; preserve existing non-null session/EPS values from the prior FMP/Alpha Vantage partial backfill, then fill nulls and all remaining symbols from DoltHub; attach DoltHub period-end alignment",
|
||||||
|
"events_in_window": 12414,
|
||||||
|
"events_with_actual_and_estimate": 12311,
|
||||||
|
"symbols_done": 506,
|
||||||
|
"symbols_universe": 506,
|
||||||
|
"symbols_with_dolthub_calendar": 504,
|
||||||
|
"symbols_without_dolthub_calendar": [
|
||||||
|
"RHM",
|
||||||
|
"SPCX"
|
||||||
|
],
|
||||||
|
"announce_date_range": {
|
||||||
|
"min": "2020-01-22",
|
||||||
|
"max": "2026-07-17"
|
||||||
|
},
|
||||||
|
"complete": true
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,331 +0,0 @@
|
|||||||
{
|
|
||||||
"generated_at": "2026-07-19T09:31:29.078611",
|
|
||||||
"data_provenance": {
|
|
||||||
"snapshot": "C:\\Workspace\\signal-platform\\backtest_snapshots\\prod.sqlite",
|
|
||||||
"n_earnings_events": 5612,
|
|
||||||
"backfill_meta": {
|
|
||||||
"done": 48,
|
|
||||||
"universe_tickers": 506
|
|
||||||
},
|
|
||||||
"announce_range": {
|
|
||||||
"min": "1985-08-31",
|
|
||||||
"max": "2026-07-16"
|
|
||||||
},
|
|
||||||
"with_actual_and_estimate": 5018
|
|
||||||
},
|
|
||||||
"experiment_2a": {
|
|
||||||
"sim_summary": {
|
|
||||||
"sharpe": 2.09,
|
|
||||||
"sharpe_se": 0.497,
|
|
||||||
"cagr_pct": 51.6,
|
|
||||||
"max_drawdown_pct": 21.4,
|
|
||||||
"trades": 322,
|
|
||||||
"total_return_pct": 424.6
|
|
||||||
},
|
|
||||||
"n_trades_parsed": 322,
|
|
||||||
"q1_losses_worse_than_minus_1r": {
|
|
||||||
"n_losses_lt_minus_1r": 28,
|
|
||||||
"n_with_earnings_in_hold": 1,
|
|
||||||
"fraction_with_earnings": 0.0357,
|
|
||||||
"all_trades_with_earnings_in_hold": 14,
|
|
||||||
"fraction_all_trades_with_earnings": 0.0435
|
|
||||||
},
|
|
||||||
"q2_entry_within_3d_before_announce": {
|
|
||||||
"pre_earn_entries": {
|
|
||||||
"n": 4,
|
|
||||||
"mean": 1.9379,
|
|
||||||
"win_rate": 0.5,
|
|
||||||
"p05": -1.2428,
|
|
||||||
"p25": -0.8833,
|
|
||||||
"p50": 1.1209,
|
|
||||||
"p75": 3.942,
|
|
||||||
"p95": 6.2623,
|
|
||||||
"min": -1.3327,
|
|
||||||
"max": 6.8424
|
|
||||||
},
|
|
||||||
"other_entries": {
|
|
||||||
"n": 318,
|
|
||||||
"mean": 0.6965,
|
|
||||||
"win_rate": 0.3711,
|
|
||||||
"p05": -1.1052,
|
|
||||||
"p25": -1.0,
|
|
||||||
"p50": -0.8259,
|
|
||||||
"p75": 2.1053,
|
|
||||||
"p95": 6.077,
|
|
||||||
"min": -3.2587,
|
|
||||||
"max": 12.8654
|
|
||||||
},
|
|
||||||
"all_entries": {
|
|
||||||
"n": 322,
|
|
||||||
"mean": 0.7119,
|
|
||||||
"win_rate": 0.3727,
|
|
||||||
"p05": -1.1209,
|
|
||||||
"p25": -1.0,
|
|
||||||
"p50": -0.8251,
|
|
||||||
"p75": 2.1595,
|
|
||||||
"p95": 6.2246,
|
|
||||||
"min": -3.2587,
|
|
||||||
"max": 12.8654
|
|
||||||
},
|
|
||||||
"tail_trim_note": "Compare p95/max and mean of pre_earn vs other. Rising win_rate with falling mean/p95 = right-tail trim red flag."
|
|
||||||
},
|
|
||||||
"note": "REPORT-ONLY \u2014 no filter shipped."
|
|
||||||
},
|
|
||||||
"experiment_2b": {
|
|
||||||
"signal_eval_side_by_side": {
|
|
||||||
"mom_12_1": {
|
|
||||||
"signal": "mom_12_1",
|
|
||||||
"weeks": 35,
|
|
||||||
"avg_cross_section": 497.7,
|
|
||||||
"mean_ic": 0.0531,
|
|
||||||
"ic_t_stat": 1.61,
|
|
||||||
"ic_positive_pct": 65.7,
|
|
||||||
"mean_quintile_spread": 0.0206,
|
|
||||||
"reliable": true
|
|
||||||
},
|
|
||||||
"mom_12_1_resid": {
|
|
||||||
"signal": "mom_12_1_resid",
|
|
||||||
"weeks": 35,
|
|
||||||
"avg_cross_section": 497.7,
|
|
||||||
"mean_ic": 0.0552,
|
|
||||||
"ic_t_stat": 1.98,
|
|
||||||
"ic_positive_pct": 60.0,
|
|
||||||
"mean_quintile_spread": 0.0207,
|
|
||||||
"reliable": true
|
|
||||||
},
|
|
||||||
"mom_12_1_sector_resid": {
|
|
||||||
"signal": "mom_12_1_sector_resid",
|
|
||||||
"weeks": 35,
|
|
||||||
"avg_cross_section": 497.7,
|
|
||||||
"mean_ic": 0.0578,
|
|
||||||
"ic_t_stat": 2.34,
|
|
||||||
"ic_positive_pct": 65.7,
|
|
||||||
"mean_quintile_spread": 0.0245,
|
|
||||||
"reliable": true
|
|
||||||
},
|
|
||||||
"mom_12_1_sector_demeaned": {
|
|
||||||
"signal": "mom_12_1_sector_demeaned",
|
|
||||||
"weeks": 35,
|
|
||||||
"avg_cross_section": 496.7,
|
|
||||||
"mean_ic": 0.034,
|
|
||||||
"ic_t_stat": 1.32,
|
|
||||||
"ic_positive_pct": 62.9,
|
|
||||||
"mean_quintile_spread": 0.0154,
|
|
||||||
"reliable": true
|
|
||||||
},
|
|
||||||
"sue_latest": {
|
|
||||||
"signal": "sue_latest",
|
|
||||||
"weeks": 44,
|
|
||||||
"avg_cross_section": 47.4,
|
|
||||||
"mean_ic": 0.0172,
|
|
||||||
"ic_t_stat": 0.6,
|
|
||||||
"ic_positive_pct": 47.7,
|
|
||||||
"mean_quintile_spread": 0.0064,
|
|
||||||
"reliable": true
|
|
||||||
},
|
|
||||||
"fip_id": {
|
|
||||||
"signal": "fip_id",
|
|
||||||
"weeks": 35,
|
|
||||||
"avg_cross_section": 497.7,
|
|
||||||
"mean_ic": -0.045,
|
|
||||||
"ic_t_stat": -2.91,
|
|
||||||
"ic_positive_pct": 25.7,
|
|
||||||
"mean_quintile_spread": -0.0168,
|
|
||||||
"reliable": true
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"signal_eval_identical_sue_subset": {
|
|
||||||
"mom_12_1": {
|
|
||||||
"signal": "mom_12_1",
|
|
||||||
"weeks": 35,
|
|
||||||
"avg_cross_section": 47.3,
|
|
||||||
"mean_ic": -0.0174,
|
|
||||||
"ic_t_stat": -0.42,
|
|
||||||
"ic_positive_pct": 45.7,
|
|
||||||
"mean_quintile_spread": 0.0077,
|
|
||||||
"reliable": true
|
|
||||||
},
|
|
||||||
"mom_12_1_resid": {
|
|
||||||
"signal": "mom_12_1_resid",
|
|
||||||
"weeks": 35,
|
|
||||||
"avg_cross_section": 47.3,
|
|
||||||
"mean_ic": -0.0104,
|
|
||||||
"ic_t_stat": -0.27,
|
|
||||||
"ic_positive_pct": 51.4,
|
|
||||||
"mean_quintile_spread": 0.0075,
|
|
||||||
"reliable": true
|
|
||||||
},
|
|
||||||
"sue_latest": {
|
|
||||||
"signal": "sue_latest",
|
|
||||||
"weeks": 44,
|
|
||||||
"avg_cross_section": 47.4,
|
|
||||||
"mean_ic": 0.0172,
|
|
||||||
"ic_t_stat": 0.6,
|
|
||||||
"ic_positive_pct": 47.7,
|
|
||||||
"mean_quintile_spread": 0.0064,
|
|
||||||
"reliable": true
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"identical_subset_note": "Mom baselines re-scored only on (week, symbol) cells where SUE exists. Use this table when backfill is incomplete \u2014 full-universe mom N is not comparable.",
|
|
||||||
"full_signal_eval": [
|
|
||||||
{
|
|
||||||
"signal": "vol_6m",
|
|
||||||
"weeks": 39,
|
|
||||||
"avg_cross_section": 498.2,
|
|
||||||
"mean_ic": 0.0609,
|
|
||||||
"ic_t_stat": 1.48,
|
|
||||||
"ic_positive_pct": 64.1,
|
|
||||||
"mean_quintile_spread": 0.0337,
|
|
||||||
"reliable": true
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"signal": "mom_12_1_sector_resid",
|
|
||||||
"weeks": 35,
|
|
||||||
"avg_cross_section": 497.7,
|
|
||||||
"mean_ic": 0.0578,
|
|
||||||
"ic_t_stat": 2.34,
|
|
||||||
"ic_positive_pct": 65.7,
|
|
||||||
"mean_quintile_spread": 0.0245,
|
|
||||||
"reliable": true
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"signal": "mom_12_1_resid",
|
|
||||||
"weeks": 35,
|
|
||||||
"avg_cross_section": 497.7,
|
|
||||||
"mean_ic": 0.0552,
|
|
||||||
"ic_t_stat": 1.98,
|
|
||||||
"ic_positive_pct": 60.0,
|
|
||||||
"mean_quintile_spread": 0.0207,
|
|
||||||
"reliable": true
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"signal": "mom_12_1",
|
|
||||||
"weeks": 35,
|
|
||||||
"avg_cross_section": 497.7,
|
|
||||||
"mean_ic": 0.0531,
|
|
||||||
"ic_t_stat": 1.61,
|
|
||||||
"ic_positive_pct": 65.7,
|
|
||||||
"mean_quintile_spread": 0.0206,
|
|
||||||
"reliable": true
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"signal": "mom_12_1_sector_demeaned",
|
|
||||||
"weeks": 35,
|
|
||||||
"avg_cross_section": 496.7,
|
|
||||||
"mean_ic": 0.034,
|
|
||||||
"ic_t_stat": 1.32,
|
|
||||||
"ic_positive_pct": 62.9,
|
|
||||||
"mean_quintile_spread": 0.0154,
|
|
||||||
"reliable": true
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"signal": "sue_latest",
|
|
||||||
"weeks": 44,
|
|
||||||
"avg_cross_section": 47.4,
|
|
||||||
"mean_ic": 0.0172,
|
|
||||||
"ic_t_stat": 0.6,
|
|
||||||
"ic_positive_pct": 47.7,
|
|
||||||
"mean_quintile_spread": 0.0064,
|
|
||||||
"reliable": true
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"signal": "trend_200",
|
|
||||||
"weeks": 37,
|
|
||||||
"avg_cross_section": 497.9,
|
|
||||||
"mean_ic": 0.0161,
|
|
||||||
"ic_t_stat": 0.44,
|
|
||||||
"ic_positive_pct": 59.5,
|
|
||||||
"mean_quintile_spread": 0.006,
|
|
||||||
"reliable": true
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"signal": "reversal_1m",
|
|
||||||
"weeks": 43,
|
|
||||||
"avg_cross_section": 498.7,
|
|
||||||
"mean_ic": 0.0059,
|
|
||||||
"ic_t_stat": 0.22,
|
|
||||||
"ic_positive_pct": 53.5,
|
|
||||||
"mean_quintile_spread": 0.0053,
|
|
||||||
"reliable": true
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"signal": "mom_6_1",
|
|
||||||
"weeks": 39,
|
|
||||||
"avg_cross_section": 498.2,
|
|
||||||
"mean_ic": 0.0051,
|
|
||||||
"ic_t_stat": 0.21,
|
|
||||||
"ic_positive_pct": 56.4,
|
|
||||||
"mean_quintile_spread": 0.0087,
|
|
||||||
"reliable": true
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"signal": "mom_3_1",
|
|
||||||
"weeks": 42,
|
|
||||||
"avg_cross_section": 498.5,
|
|
||||||
"mean_ic": -0.0064,
|
|
||||||
"ic_t_stat": -0.25,
|
|
||||||
"ic_positive_pct": 50.0,
|
|
||||||
"mean_quintile_spread": 0.0046,
|
|
||||||
"reliable": true
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"signal": "high_52w",
|
|
||||||
"weeks": 35,
|
|
||||||
"avg_cross_section": 497.7,
|
|
||||||
"mean_ic": -0.0086,
|
|
||||||
"ic_t_stat": -0.26,
|
|
||||||
"ic_positive_pct": 54.3,
|
|
||||||
"mean_quintile_spread": -0.0088,
|
|
||||||
"reliable": true
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"signal": "fip_id",
|
|
||||||
"weeks": 35,
|
|
||||||
"avg_cross_section": 497.7,
|
|
||||||
"mean_ic": -0.045,
|
|
||||||
"ic_t_stat": -2.91,
|
|
||||||
"ic_positive_pct": 25.7,
|
|
||||||
"mean_quintile_spread": -0.0168,
|
|
||||||
"reliable": true
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"sue_grade": {
|
|
||||||
"green": false,
|
|
||||||
"checks": {
|
|
||||||
"mean_ic": 0.0172,
|
|
||||||
"sign_positive": true,
|
|
||||||
"abs_ge_0_03": false,
|
|
||||||
"reliable": true,
|
|
||||||
"ic_t_stat": 0.6,
|
|
||||||
"weeks": 44
|
|
||||||
},
|
|
||||||
"reason": "iron rule not met",
|
|
||||||
"row": {
|
|
||||||
"signal": "sue_latest",
|
|
||||||
"weeks": 44,
|
|
||||||
"avg_cross_section": 47.4,
|
|
||||||
"mean_ic": 0.0172,
|
|
||||||
"ic_t_stat": 0.6,
|
|
||||||
"ic_positive_pct": 47.7,
|
|
||||||
"mean_quintile_spread": 0.0064,
|
|
||||||
"reliable": true
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"momentum_conditional_sue": {
|
|
||||||
"mean_ic": -0.0065,
|
|
||||||
"ic_t_stat": -0.1,
|
|
||||||
"weeks": 35,
|
|
||||||
"note": "IC of sue_latest within top mom_12_1 quintile (non-overlapping weeks)"
|
|
||||||
},
|
|
||||||
"sue_coverage": {
|
|
||||||
"symbols_with_sue": 48,
|
|
||||||
"avg_weeks_with_sue": 47.1,
|
|
||||||
"weeks_with_min_cross_section": 256
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"verdict": "PARK",
|
|
||||||
"verdict_detail": "SUE IC=0.0172 below iron bar or unreliable; keep data, no wire.",
|
|
||||||
"human_next": "- No SUE book change.\n- Read 2a tails before considering any earnings-avoid filter.",
|
|
||||||
"report_path": "reports/earnings-gap-sue-20260719-093129.json",
|
|
||||||
"fmp_note": "Bulk earnings-calendar is paid (402 on free tier). Backfill used per-symbol /stable/earnings; see earnings-backfill-status.json."
|
|
||||||
}
|
|
||||||
@@ -1,202 +0,0 @@
|
|||||||
# Earnings gap diagnostic + SUE / PEAD (Tier-1 alpha research)
|
|
||||||
|
|
||||||
**Status:** **PARK** (incomplete earnings coverage; SUE fails iron rule on available sample).
|
|
||||||
**Branch:** `research/earnings-gap-and-sue`
|
|
||||||
**Production impact:** none. Local research only. **No filters shipped from 2a.**
|
|
||||||
**Artifacts:** `reports/earnings-gap-sue-20260719-093129.json` (+ companion `.md`)
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Pre-registration (locked before first research run)
|
|
||||||
|
|
||||||
### Data
|
|
||||||
|
|
||||||
- Historical earnings calendar for the production universe over the full snapshot
|
|
||||||
window (and deeper if the feed provides it).
|
|
||||||
- Preferred source: FMP **date-range earnings-calendar** (bulk). If unavailable on
|
|
||||||
free tier, fall back to per-symbol `/stable/earnings` with request accounting.
|
|
||||||
- Store in a real local table `earnings_events` (symbol + announce_date key).
|
|
||||||
- Point-in-time: a surprise is usable only from **announce date + 1 trading day**
|
|
||||||
onward.
|
|
||||||
|
|
||||||
### Experiment 2a — earnings-gap risk (defense, report-only)
|
|
||||||
|
|
||||||
Join simulated production-config trades (`fill_mode=close`) with earnings dates.
|
|
||||||
|
|
||||||
**Pre-registered questions:**
|
|
||||||
|
|
||||||
1. What fraction of losses worse than **−1R** occur with an earnings announcement
|
|
||||||
**between entry and exit** (inclusive of the holding window)?
|
|
||||||
2. What is the mean R of entries taken within **3 trading days BEFORE** an
|
|
||||||
announcement vs all other entries — report **both tails** of the R
|
|
||||||
distribution (rule 4: any earnings-avoid entry filter is presumed guilty of
|
|
||||||
right-tail trimming until the win distribution shows otherwise)?
|
|
||||||
|
|
||||||
**Output:** distributions and counts only.
|
|
||||||
**No filter is shipped.** If numbers argue for a filter → report and stop.
|
|
||||||
|
|
||||||
### Experiment 2b — SUE / PEAD (offense)
|
|
||||||
|
|
||||||
Signal `sue_latest`:
|
|
||||||
|
|
||||||
\[
|
|
||||||
\text{SUE} = \frac{\text{actual} - \text{estimate}}{\sigma(\text{trailing 8 surprises})}
|
|
||||||
\]
|
|
||||||
|
|
||||||
Fallback if estimate history is thin: scale surprise by price.
|
|
||||||
Carry forward from announce+1 for **63 trading days**, else NaN (name drops out
|
|
||||||
of that cross-section).
|
|
||||||
|
|
||||||
**Iron rule (IC harness):** mean weekly Spearman IC on non-overlapping weeks;
|
|
||||||
\|mean IC\| ≥ ~0.03, **positive** sign (drift), `reliable: true` (≥12 windows).
|
|
||||||
|
|
||||||
Always side-by-side with `mom_12_1` and `mom_12_1_resid` on **identical**
|
|
||||||
cross-sections.
|
|
||||||
|
|
||||||
Also report **momentum-conditional** IC (within top momentum quintile).
|
|
||||||
|
|
||||||
**If it passes iron rule:** STOP and report. Book-integration design is a
|
|
||||||
separate human-approved step — do not wire.
|
|
||||||
|
|
||||||
### Verdict labels
|
|
||||||
|
|
||||||
| label | meaning |
|
|
||||||
|---|---|
|
|
||||||
| **PROMOTE** | (2b only) iron rule cleared → human designs tilt/gate |
|
|
||||||
| **PARK** | Interesting but incomplete / weak |
|
|
||||||
| **DEAD** | No edge / diagnostic argues against action |
|
|
||||||
| **REPORT-ONLY** | (2a) always — never auto-filter |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Data provenance
|
|
||||||
|
|
||||||
| item | result |
|
|
||||||
|---|---|
|
|
||||||
| Snapshot | `backtest_snapshots/prod.sqlite` (506 names) |
|
|
||||||
| FMP bulk `earnings-calendar` | **402 Premium** — not available on free tier |
|
|
||||||
| FMP per-symbol `/stable/earnings` | used; hit daily rate limit ~225 reqs |
|
|
||||||
| Alpha Vantage `EARNINGS` | used for +24 symbols (announce = `reportedDate`) |
|
|
||||||
| Symbols with events | **48 / 506 (9.5%)** |
|
|
||||||
| Total events | 5,612 (5,018 with actual+estimate) |
|
|
||||||
| Announce range | 1985-08-31 → 2026-07-16 |
|
|
||||||
| FMP requests (first day) | 260 FMP + 25 AV (see `reports/earnings-backfill-status.json`) |
|
|
||||||
|
|
||||||
**Incomplete backfill is first-class.** 2a under-detects earnings overlaps; 2b SUE
|
|
||||||
cross-section averages **~47 names**, not ~500. Resume:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Day N (FMP free ~250/day; AV free ~25/day — prefer FMP after reset)
|
|
||||||
python scripts/backfill_earnings_events.py \
|
|
||||||
--snapshot backtest_snapshots/prod.sqlite \
|
|
||||||
--provider fmp --force-symbol --limit 250 --sleep 0.4
|
|
||||||
|
|
||||||
# When done==506:
|
|
||||||
python scripts/run_earnings_research.py \
|
|
||||||
--snapshot backtest_snapshots/prod.sqlite \
|
|
||||||
--workers 6 --allow-spawn
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Results
|
|
||||||
|
|
||||||
Generated: `2026-07-19T09:31:29`
|
|
||||||
|
|
||||||
### 2a — Earnings-gap risk (report-only)
|
|
||||||
|
|
||||||
Production book sim: Sharpe 2.09 (SE 0.497), CAGR 51.6%, max DD 21.4%, **322 trades**,
|
|
||||||
`fill_mode=close`.
|
|
||||||
|
|
||||||
#### Q1 — Losses worse than −1R with earnings in hold
|
|
||||||
|
|
||||||
| metric | value |
|
|
||||||
|---|---:|
|
|
||||||
| n losses < −1R | 28 |
|
|
||||||
| of which earnings in hold | **1** |
|
|
||||||
| fraction | **3.6%** |
|
|
||||||
| all trades with earnings in hold | 14 / 322 (4.4%) |
|
|
||||||
|
|
||||||
**Read:** On incomplete earnings labels this is a **lower bound** on earnings
|
|
||||||
overlap, not a clean “earnings rarely hurt.” Do **not** conclude earnings risk is
|
|
||||||
immaterial until coverage ≥ ~95% of the book’s names.
|
|
||||||
|
|
||||||
#### Q2 — Entry within 3 trading days before announce (both tails)
|
|
||||||
|
|
||||||
| cohort | n | mean R | win rate | p05 | p50 | p95 | max |
|
|
||||||
|---|---:|---:|---:|---:|---:|---:|---:|
|
|
||||||
| pre-earn (≤3d before) | **4** | 1.94 | 50% | −1.24 | 1.12 | 6.26 | 6.84 |
|
|
||||||
| other | 318 | 0.70 | 37% | −1.11 | −0.83 | 6.08 | **12.87** |
|
|
||||||
| all | 322 | 0.71 | 37% | −1.12 | −0.83 | 6.22 | 12.87 |
|
|
||||||
|
|
||||||
**Tail-trim presumption:** n=4 is not a sample. Point estimate does **not** show
|
|
||||||
right-tail destruction of pre-earn entries (p95 similar; max actually higher in
|
|
||||||
“other”). **No earnings-avoid filter is supported.** Re-run after full backfill.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### 2b — SUE / PEAD IC
|
|
||||||
|
|
||||||
#### Full-universe harness (mom on ~500; SUE only where labeled)
|
|
||||||
|
|
||||||
| signal | mean_ic | ic_t_stat | weeks | avg_N | reliable |
|
|
||||||
|---|---:|---:|---:|---:|---|
|
|
||||||
| mom_12_1_sector_resid | 0.0578 | 2.34 | 35 | 497.7 | true |
|
|
||||||
| mom_12_1_resid | 0.0552 | 1.98 | 35 | 497.7 | true |
|
|
||||||
| mom_12_1 | 0.0531 | 1.61 | 35 | 497.7 | true |
|
|
||||||
| **sue_latest** | **0.0172** | **0.6** | 44 | **47.4** | true |
|
|
||||||
| fip_id | −0.045 | −2.91 | 35 | 497.7 | true |
|
|
||||||
|
|
||||||
#### Identical SUE subset (fair side-by-side — use this while coverage is thin)
|
|
||||||
|
|
||||||
| signal | mean_ic | ic_t_stat | weeks | avg_N |
|
|
||||||
|---|---:|---:|---:|---:|
|
|
||||||
| sue_latest | 0.0172 | 0.6 | 44 | 47.4 |
|
|
||||||
| mom_12_1 | −0.0174 | −0.42 | 35 | 47.3 |
|
|
||||||
| mom_12_1_resid | −0.0104 | −0.27 | 35 | 47.3 |
|
|
||||||
|
|
||||||
On the thin labeled subset, momentum itself is noise — so the subset is not yet
|
|
||||||
a meaningful PEAD test.
|
|
||||||
|
|
||||||
#### Momentum-conditional SUE (top mom quintile)
|
|
||||||
|
|
||||||
| metric | value |
|
|
||||||
|---|---:|
|
|
||||||
| mean IC | **−0.0065** |
|
|
||||||
| t | −0.1 |
|
|
||||||
| weeks | 35 |
|
|
||||||
|
|
||||||
Wrong sign vs “ride positive surprises inside the momentum gate.”
|
|
||||||
|
|
||||||
**Iron rule:** fail (\|IC\| 0.017 < 0.03; t 0.6). **No promote.**
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Verdict
|
|
||||||
|
|
||||||
| piece | verdict |
|
|
||||||
|---|---|
|
|
||||||
| **2a earnings-gap** | **REPORT-ONLY** — no filter. Coverage too thin for risk claims; tails do not argue for an avoid-filter on n=4. |
|
|
||||||
| **2b SUE** | **PARK** (effectively not green). Mild positive IC on ~48 names; fails iron bar; mom-conditional flat/negative. Re-score after full backfill before DEAD. |
|
|
||||||
| **Production** | **no change** |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## What a human must decide next
|
|
||||||
|
|
||||||
1. Resume multi-day earnings backfill to **506/506**, then re-run
|
|
||||||
`run_earnings_research.py` (heavy — MacBook OK).
|
|
||||||
2. Do **not** ship an earnings-avoid entry filter from 2a.
|
|
||||||
3. Do **not** wire SUE until a full-coverage IC clears the iron rule (and
|
|
||||||
preferably mom-conditional > 0).
|
|
||||||
4. Do not merge into main strategy docs without review.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Implementation notes
|
|
||||||
|
|
||||||
| piece | role |
|
|
||||||
|---|---|
|
|
||||||
| `scripts/backfill_earnings_events.py` | bulk attempt → FMP/AV per-symbol; `earnings_events` + meta on snapshot |
|
|
||||||
| `scripts/run_earnings_research.py` | 2a trade join + 2b SUE IC / mom-conditional |
|
|
||||||
| Snapshot table `earnings_events` | real table (not SystemSetting JSON) |
|
|
||||||
+380
-397
@@ -1,15 +1,13 @@
|
|||||||
"""Backfill historical earnings into a snapshot ``earnings_events`` table.
|
"""Bulk-only historical earnings backfill for a local SQLite snapshot.
|
||||||
|
|
||||||
Prefers FMP bulk date-range ``earnings-calendar`` (one request per window).
|
The job uses FMP's date-range earnings-calendar endpoint. One request covers all
|
||||||
On free-tier 402/403, falls back to per-symbol ``/stable/earnings`` with
|
symbols in a date window; per-symbol endpoints are intentionally not available
|
||||||
resume support and request counting (≈250 req/day free tier).
|
in this task runner. Successful windows are committed independently so a later
|
||||||
|
run resumes after a daily quota boundary without repeating completed windows.
|
||||||
|
|
||||||
Research only — writes to the local snapshot SQLite, never production Postgres.
|
Example:
|
||||||
|
python scripts/backfill_earnings_events.py --snapshot backtest_snapshots/prod.sqlite \
|
||||||
Example
|
--from-date 2012-01-01 --window-days 30 --limit 250
|
||||||
-------
|
|
||||||
python scripts/backfill_earnings_events.py \\
|
|
||||||
--snapshot backtest_snapshots/prod.sqlite --limit 250
|
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
@@ -17,10 +15,11 @@ from __future__ import annotations
|
|||||||
import argparse
|
import argparse
|
||||||
import asyncio
|
import asyncio
|
||||||
import json
|
import json
|
||||||
|
import math
|
||||||
import sys
|
import sys
|
||||||
import time
|
|
||||||
from datetime import date, datetime, timedelta, timezone
|
from datetime import date, datetime, timedelta, timezone
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
import httpx
|
import httpx
|
||||||
from sqlalchemy import create_engine, text
|
from sqlalchemy import create_engine, text
|
||||||
@@ -34,7 +33,7 @@ from app.ssl_bootstrap import bootstrap_ssl # noqa: E402
|
|||||||
bootstrap_ssl()
|
bootstrap_ssl()
|
||||||
|
|
||||||
FMP_STABLE = "https://financialmodelingprep.com/stable"
|
FMP_STABLE = "https://financialmodelingprep.com/stable"
|
||||||
DDL = """
|
EVENTS_DDL = """
|
||||||
CREATE TABLE IF NOT EXISTS earnings_events (
|
CREATE TABLE IF NOT EXISTS earnings_events (
|
||||||
id INTEGER PRIMARY KEY,
|
id INTEGER PRIMARY KEY,
|
||||||
symbol TEXT NOT NULL,
|
symbol TEXT NOT NULL,
|
||||||
@@ -49,7 +48,6 @@ CREATE TABLE IF NOT EXISTS earnings_events (
|
|||||||
UNIQUE(symbol, announce_date)
|
UNIQUE(symbol, announce_date)
|
||||||
)
|
)
|
||||||
"""
|
"""
|
||||||
# Side table tracks which symbols have been fully pulled (resume).
|
|
||||||
META_DDL = """
|
META_DDL = """
|
||||||
CREATE TABLE IF NOT EXISTS earnings_backfill_meta (
|
CREATE TABLE IF NOT EXISTS earnings_backfill_meta (
|
||||||
symbol TEXT PRIMARY KEY,
|
symbol TEXT PRIMARY KEY,
|
||||||
@@ -59,239 +57,238 @@ CREATE TABLE IF NOT EXISTS earnings_backfill_meta (
|
|||||||
note TEXT
|
note TEXT
|
||||||
)
|
)
|
||||||
"""
|
"""
|
||||||
|
WINDOW_DDL = """
|
||||||
|
CREATE TABLE IF NOT EXISTS earnings_backfill_windows (
|
||||||
|
from_date TEXT NOT NULL,
|
||||||
|
to_date TEXT NOT NULL,
|
||||||
|
status TEXT NOT NULL,
|
||||||
|
requests INTEGER NOT NULL DEFAULT 0,
|
||||||
|
rows_raw INTEGER NOT NULL DEFAULT 0,
|
||||||
|
rows_universe INTEGER NOT NULL DEFAULT 0,
|
||||||
|
duplicate_rows INTEGER NOT NULL DEFAULT 0,
|
||||||
|
restated_rows INTEGER NOT NULL DEFAULT 0,
|
||||||
|
updated_at TEXT NOT NULL,
|
||||||
|
note TEXT,
|
||||||
|
PRIMARY KEY(from_date, to_date)
|
||||||
|
)
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
def _parse_args() -> argparse.Namespace:
|
def _parse_args() -> argparse.Namespace:
|
||||||
p = argparse.ArgumentParser(description=__doc__)
|
parser = argparse.ArgumentParser(description=__doc__)
|
||||||
p.add_argument("--snapshot", default="backtest_snapshots/prod.sqlite")
|
parser.add_argument("--snapshot", default="backtest_snapshots/prod.sqlite")
|
||||||
p.add_argument(
|
parser.add_argument("--from-date", default="2012-01-01")
|
||||||
"--from-date",
|
parser.add_argument("--to-date", default=None)
|
||||||
default="2020-01-01",
|
parser.add_argument("--window-days", type=int, default=30)
|
||||||
help="Bulk calendar window start (also filters per-symbol rows).",
|
parser.add_argument("--limit", type=int, default=250)
|
||||||
)
|
parser.add_argument("--sleep", type=float, default=0.35)
|
||||||
p.add_argument(
|
parser.add_argument(
|
||||||
"--to-date",
|
"--refetch-windows",
|
||||||
default=None,
|
|
||||||
help="Bulk calendar window end (default: today).",
|
|
||||||
)
|
|
||||||
p.add_argument(
|
|
||||||
"--limit",
|
|
||||||
type=int,
|
|
||||||
default=250,
|
|
||||||
help="Max FMP requests this run (free-tier cushion).",
|
|
||||||
)
|
|
||||||
p.add_argument("--sleep", type=float, default=0.35)
|
|
||||||
p.add_argument(
|
|
||||||
"--force-symbol",
|
|
||||||
action="store_true",
|
action="store_true",
|
||||||
help="Skip bulk attempt; go straight to per-symbol.",
|
help="Re-fetch date windows already logged as done.",
|
||||||
)
|
)
|
||||||
p.add_argument(
|
return parser.parse_args()
|
||||||
"--refetch-done",
|
|
||||||
action="store_true",
|
|
||||||
help="Re-fetch symbols already marked done.",
|
|
||||||
)
|
|
||||||
p.add_argument(
|
|
||||||
"--provider",
|
|
||||||
choices=("fmp", "alpha_vantage", "auto"),
|
|
||||||
default="auto",
|
|
||||||
help="Earnings provider. auto tries FMP bulk then FMP/AV per-symbol.",
|
|
||||||
)
|
|
||||||
return p.parse_args()
|
|
||||||
|
|
||||||
|
|
||||||
def _ensure_tables(engine) -> None:
|
def _ensure_tables(engine) -> None:
|
||||||
with engine.begin() as conn:
|
with engine.begin() as conn:
|
||||||
conn.execute(text(DDL))
|
conn.execute(text(EVENTS_DDL))
|
||||||
conn.execute(text(META_DDL))
|
conn.execute(text(META_DDL))
|
||||||
|
conn.execute(text(WINDOW_DDL))
|
||||||
|
|
||||||
|
|
||||||
def _upsert_events(conn, rows: list[dict], source: str) -> int:
|
def _number(value: Any) -> float | None:
|
||||||
if not rows:
|
if value is None or value == "":
|
||||||
return 0
|
return None
|
||||||
now = datetime.now(timezone.utc).isoformat()
|
try:
|
||||||
written = 0
|
result = float(value)
|
||||||
for r in rows:
|
except (TypeError, ValueError):
|
||||||
conn.execute(
|
return None
|
||||||
text(
|
return result if math.isfinite(result) else None
|
||||||
"""
|
|
||||||
INSERT INTO earnings_events (
|
|
||||||
symbol, announce_date, announce_time,
|
def _normalise_session(value: Any) -> str | None:
|
||||||
eps_estimate, eps_actual, revenue_estimate, revenue_actual,
|
if value is None:
|
||||||
source, fetched_at
|
return None
|
||||||
) VALUES (
|
cleaned = str(value).strip().lower().replace("_", " ").replace("-", " ")
|
||||||
:symbol, :announce_date, :announce_time,
|
aliases = {
|
||||||
:eps_estimate, :eps_actual, :revenue_estimate, :revenue_actual,
|
"bmo": "bmo",
|
||||||
:source, :fetched_at
|
"before market open": "bmo",
|
||||||
)
|
"before open": "bmo",
|
||||||
ON CONFLICT(symbol, announce_date) DO UPDATE SET
|
"amc": "amc",
|
||||||
announce_time=excluded.announce_time,
|
"after market close": "amc",
|
||||||
eps_estimate=excluded.eps_estimate,
|
"after close": "amc",
|
||||||
eps_actual=excluded.eps_actual,
|
"during market hours": "during",
|
||||||
revenue_estimate=excluded.revenue_estimate,
|
"dmh": "during",
|
||||||
revenue_actual=excluded.revenue_actual,
|
}
|
||||||
source=excluded.source,
|
return aliases.get(cleaned, cleaned or None)
|
||||||
fetched_at=excluded.fetched_at
|
|
||||||
"""
|
|
||||||
),
|
|
||||||
{
|
|
||||||
"symbol": r["symbol"],
|
|
||||||
"announce_date": r["announce_date"],
|
|
||||||
"announce_time": r.get("announce_time"),
|
|
||||||
"eps_estimate": r.get("eps_estimate"),
|
|
||||||
"eps_actual": r.get("eps_actual"),
|
|
||||||
"revenue_estimate": r.get("revenue_estimate"),
|
|
||||||
"revenue_actual": r.get("revenue_actual"),
|
|
||||||
"source": source,
|
|
||||||
"fetched_at": now,
|
|
||||||
},
|
|
||||||
)
|
|
||||||
written += 1
|
|
||||||
return written
|
|
||||||
|
|
||||||
|
|
||||||
def _parse_bulk_item(item: dict) -> dict | None:
|
def _parse_bulk_item(item: dict) -> dict | None:
|
||||||
sym = (item.get("symbol") or "").strip().upper()
|
symbol = str(item.get("symbol") or "").strip().upper().replace(".", "-")
|
||||||
d = item.get("date") or item.get("earningsDate")
|
raw_date = item.get("date") or item.get("earningsDate")
|
||||||
if not sym or not d:
|
if not symbol or not raw_date:
|
||||||
return None
|
return None
|
||||||
return {
|
return {
|
||||||
"symbol": sym.replace(".", "-"),
|
"symbol": symbol,
|
||||||
"announce_date": str(d)[:10],
|
"announce_date": str(raw_date)[:10],
|
||||||
"announce_time": item.get("time") or item.get("announceTime"),
|
"announce_time": _normalise_session(
|
||||||
"eps_estimate": _f(item.get("epsEstimated") or item.get("estimatedEarning")),
|
item.get("time") or item.get("announceTime")
|
||||||
"eps_actual": _f(item.get("epsActual") or item.get("eps")),
|
),
|
||||||
"revenue_estimate": _f(item.get("revenueEstimated")),
|
"eps_estimate": _number(
|
||||||
"revenue_actual": _f(item.get("revenueActual")),
|
item.get("epsEstimated")
|
||||||
|
if item.get("epsEstimated") is not None
|
||||||
|
else item.get("estimatedEarning")
|
||||||
|
),
|
||||||
|
"eps_actual": _number(
|
||||||
|
item.get("epsActual")
|
||||||
|
if item.get("epsActual") is not None
|
||||||
|
else item.get("eps")
|
||||||
|
),
|
||||||
|
"revenue_estimate": _number(item.get("revenueEstimated")),
|
||||||
|
"revenue_actual": _number(item.get("revenueActual")),
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
def _parse_symbol_item(item: dict, symbol: str) -> dict | None:
|
def _windows(start: date, end: date, window_days: int) -> list[tuple[date, date]]:
|
||||||
d = item.get("date")
|
if window_days < 1:
|
||||||
if not d:
|
raise ValueError("window_days must be positive")
|
||||||
return None
|
result: list[tuple[date, date]] = []
|
||||||
return {
|
cursor = start
|
||||||
"symbol": symbol.replace(".", "-").upper(),
|
while cursor <= end:
|
||||||
"announce_date": str(d)[:10],
|
window_end = min(end, cursor + timedelta(days=window_days - 1))
|
||||||
"announce_time": item.get("time"),
|
result.append((cursor, window_end))
|
||||||
"eps_estimate": _f(item.get("epsEstimated")),
|
cursor = window_end + timedelta(days=1)
|
||||||
"eps_actual": _f(item.get("epsActual")),
|
return result
|
||||||
"revenue_estimate": _f(item.get("revenueEstimated")),
|
|
||||||
"revenue_actual": _f(item.get("revenueActual")),
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def _f(v) -> float | None:
|
def _dedupe_bulk_rows(rows: list[dict]) -> tuple[list[dict], int, int]:
|
||||||
if v is None or v == "":
|
"""Prefer the most complete duplicate; use the later row as the tie-break."""
|
||||||
return None
|
fields = (
|
||||||
try:
|
"announce_time",
|
||||||
return float(v)
|
"eps_estimate",
|
||||||
except (TypeError, ValueError):
|
"eps_actual",
|
||||||
return None
|
"revenue_estimate",
|
||||||
|
"revenue_actual",
|
||||||
|
|
||||||
async def _try_bulk(
|
|
||||||
client: httpx.AsyncClient,
|
|
||||||
api_key: str,
|
|
||||||
start: date,
|
|
||||||
end: date,
|
|
||||||
*,
|
|
||||||
window_days: int = 30,
|
|
||||||
) -> tuple[list[dict], int, str | None]:
|
|
||||||
"""Return (rows, requests_used, error_note)."""
|
|
||||||
rows: list[dict] = []
|
|
||||||
reqs = 0
|
|
||||||
cur = start
|
|
||||||
while cur <= end:
|
|
||||||
win_end = min(end, cur + timedelta(days=window_days - 1))
|
|
||||||
resp = await client.get(
|
|
||||||
f"{FMP_STABLE}/earnings-calendar",
|
|
||||||
params={
|
|
||||||
"from": cur.isoformat(),
|
|
||||||
"to": win_end.isoformat(),
|
|
||||||
"apikey": api_key,
|
|
||||||
},
|
|
||||||
)
|
)
|
||||||
reqs += 1
|
chosen: dict[tuple[str, str], dict] = {}
|
||||||
if resp.status_code in (402, 403):
|
duplicate_extras = 0
|
||||||
return [], reqs, f"bulk_unavailable status={resp.status_code}"
|
restated = 0
|
||||||
if resp.status_code == 429:
|
for row in rows:
|
||||||
return rows, reqs, "rate_limited"
|
key = (str(row["symbol"]), str(row["announce_date"]))
|
||||||
resp.raise_for_status()
|
previous = chosen.get(key)
|
||||||
data = resp.json()
|
if previous is None:
|
||||||
if not isinstance(data, list):
|
chosen[key] = row
|
||||||
return [], reqs, f"unexpected bulk payload type={type(data)}"
|
continue
|
||||||
for item in data:
|
duplicate_extras += 1
|
||||||
|
if any(
|
||||||
|
previous.get(field) is not None
|
||||||
|
and row.get(field) is not None
|
||||||
|
and previous.get(field) != row.get(field)
|
||||||
|
for field in fields
|
||||||
|
):
|
||||||
|
restated += 1
|
||||||
|
previous_score = sum(previous.get(field) is not None for field in fields)
|
||||||
|
new_score = sum(row.get(field) is not None for field in fields)
|
||||||
|
if new_score >= previous_score:
|
||||||
|
chosen[key] = row
|
||||||
|
return list(chosen.values()), duplicate_extras, restated
|
||||||
|
|
||||||
|
|
||||||
|
def _upsert_events(conn, rows: list[dict]) -> int:
|
||||||
|
if not rows:
|
||||||
|
return 0
|
||||||
|
fetched_at = datetime.now(timezone.utc).isoformat()
|
||||||
|
statement = text(
|
||||||
|
"""
|
||||||
|
INSERT INTO earnings_events (
|
||||||
|
symbol, announce_date, announce_time, eps_estimate, eps_actual,
|
||||||
|
revenue_estimate, revenue_actual, source, fetched_at
|
||||||
|
) VALUES (
|
||||||
|
:symbol, :announce_date, :announce_time, :eps_estimate, :eps_actual,
|
||||||
|
:revenue_estimate, :revenue_actual, 'fmp_earnings_calendar', :fetched_at
|
||||||
|
)
|
||||||
|
ON CONFLICT(symbol, announce_date) DO UPDATE SET
|
||||||
|
announce_time=COALESCE(excluded.announce_time, earnings_events.announce_time),
|
||||||
|
eps_estimate=COALESCE(excluded.eps_estimate, earnings_events.eps_estimate),
|
||||||
|
eps_actual=COALESCE(excluded.eps_actual, earnings_events.eps_actual),
|
||||||
|
revenue_estimate=COALESCE(excluded.revenue_estimate, earnings_events.revenue_estimate),
|
||||||
|
revenue_actual=COALESCE(excluded.revenue_actual, earnings_events.revenue_actual),
|
||||||
|
source=excluded.source,
|
||||||
|
fetched_at=excluded.fetched_at
|
||||||
|
"""
|
||||||
|
)
|
||||||
|
conn.execute(statement, [{**row, "fetched_at": fetched_at} for row in rows])
|
||||||
|
return len(rows)
|
||||||
|
|
||||||
|
|
||||||
|
async def _fetch_bulk_window(
|
||||||
|
client: httpx.AsyncClient, api_key: str, start: date, end: date
|
||||||
|
) -> tuple[list[dict], int, str | None]:
|
||||||
|
response = await client.get(
|
||||||
|
f"{FMP_STABLE}/earnings-calendar",
|
||||||
|
params={"from": start.isoformat(), "to": end.isoformat(), "apikey": api_key},
|
||||||
|
)
|
||||||
|
if response.status_code in (402, 403):
|
||||||
|
return [], response.status_code, "bulk_endpoint_unavailable"
|
||||||
|
if response.status_code == 429:
|
||||||
|
return [], response.status_code, "daily_limit_reached"
|
||||||
|
response.raise_for_status()
|
||||||
|
payload = response.json()
|
||||||
|
if not isinstance(payload, list):
|
||||||
|
return [], response.status_code, f"unexpected_payload:{type(payload).__name__}"
|
||||||
|
rows = []
|
||||||
|
for item in payload:
|
||||||
if isinstance(item, dict):
|
if isinstance(item, dict):
|
||||||
parsed = _parse_bulk_item(item)
|
parsed = _parse_bulk_item(item)
|
||||||
if parsed:
|
if parsed:
|
||||||
rows.append(parsed)
|
rows.append(parsed)
|
||||||
cur = win_end + timedelta(days=1)
|
return rows, response.status_code, None
|
||||||
return rows, reqs, None
|
|
||||||
|
|
||||||
|
|
||||||
async def _fetch_symbol(
|
def _write_window_status(
|
||||||
client: httpx.AsyncClient, api_key: str, symbol: str
|
engine,
|
||||||
) -> list[dict]:
|
*,
|
||||||
resp = await client.get(
|
start: date,
|
||||||
f"{FMP_STABLE}/earnings",
|
end: date,
|
||||||
params={"symbol": symbol, "apikey": api_key},
|
status: str,
|
||||||
|
raw_n: int = 0,
|
||||||
|
universe_n: int = 0,
|
||||||
|
duplicate_n: int = 0,
|
||||||
|
restated_n: int = 0,
|
||||||
|
note: str | None = None,
|
||||||
|
) -> None:
|
||||||
|
with engine.begin() as conn:
|
||||||
|
conn.execute(
|
||||||
|
text(
|
||||||
|
"""
|
||||||
|
INSERT INTO earnings_backfill_windows(
|
||||||
|
from_date, to_date, status, requests, rows_raw, rows_universe,
|
||||||
|
duplicate_rows, restated_rows, updated_at, note
|
||||||
|
) VALUES (:a, :b, :status, 1, :raw, :uni, :dup, :rest, :now, :note)
|
||||||
|
ON CONFLICT(from_date, to_date) DO UPDATE SET
|
||||||
|
status=excluded.status,
|
||||||
|
requests=earnings_backfill_windows.requests + 1,
|
||||||
|
rows_raw=excluded.rows_raw,
|
||||||
|
rows_universe=excluded.rows_universe,
|
||||||
|
duplicate_rows=excluded.duplicate_rows,
|
||||||
|
restated_rows=excluded.restated_rows,
|
||||||
|
updated_at=excluded.updated_at,
|
||||||
|
note=excluded.note
|
||||||
|
"""
|
||||||
|
),
|
||||||
|
{
|
||||||
|
"a": start.isoformat(),
|
||||||
|
"b": end.isoformat(),
|
||||||
|
"status": status,
|
||||||
|
"raw": raw_n,
|
||||||
|
"uni": universe_n,
|
||||||
|
"dup": duplicate_n,
|
||||||
|
"rest": restated_n,
|
||||||
|
"now": datetime.now(timezone.utc).isoformat(),
|
||||||
|
"note": note,
|
||||||
|
},
|
||||||
)
|
)
|
||||||
if resp.status_code == 429:
|
|
||||||
raise RuntimeError("rate_limited")
|
|
||||||
if resp.status_code == 402:
|
|
||||||
return []
|
|
||||||
resp.raise_for_status()
|
|
||||||
data = resp.json()
|
|
||||||
if not isinstance(data, list):
|
|
||||||
return []
|
|
||||||
out: list[dict] = []
|
|
||||||
for item in data:
|
|
||||||
if isinstance(item, dict):
|
|
||||||
parsed = _parse_symbol_item(item, symbol)
|
|
||||||
if parsed:
|
|
||||||
out.append(parsed)
|
|
||||||
return out
|
|
||||||
|
|
||||||
|
|
||||||
async def _fetch_symbol_alpha_vantage(
|
|
||||||
client: httpx.AsyncClient, api_key: str, symbol: str
|
|
||||||
) -> list[dict]:
|
|
||||||
"""Alpha Vantage EARNINGS — includes reportedDate (announce) + estimate/actual."""
|
|
||||||
resp = await client.get(
|
|
||||||
"https://www.alphavantage.co/query",
|
|
||||||
params={"function": "EARNINGS", "symbol": symbol, "apikey": api_key},
|
|
||||||
)
|
|
||||||
if resp.status_code == 429:
|
|
||||||
raise RuntimeError("rate_limited")
|
|
||||||
resp.raise_for_status()
|
|
||||||
data = resp.json()
|
|
||||||
if not isinstance(data, dict):
|
|
||||||
return []
|
|
||||||
note = str(data.get("Note") or data.get("Information") or "")
|
|
||||||
if "rate limit" in note.lower() or "Thank you for using Alpha Vantage" in note:
|
|
||||||
raise RuntimeError("rate_limited")
|
|
||||||
if data.get("Error Message"):
|
|
||||||
return []
|
|
||||||
quarterly = data.get("quarterlyEarnings") or []
|
|
||||||
out: list[dict] = []
|
|
||||||
for item in quarterly:
|
|
||||||
if not isinstance(item, dict):
|
|
||||||
continue
|
|
||||||
# Prefer announce (reportedDate); fall back to fiscal end (worse PIT).
|
|
||||||
ad = item.get("reportedDate") or item.get("fiscalDateEnding")
|
|
||||||
if not ad:
|
|
||||||
continue
|
|
||||||
out.append({
|
|
||||||
"symbol": symbol.replace(".", "-").upper(),
|
|
||||||
"announce_date": str(ad)[:10],
|
|
||||||
"announce_time": item.get("reportTime"),
|
|
||||||
"eps_estimate": _f(item.get("estimatedEPS")),
|
|
||||||
"eps_actual": _f(item.get("reportedEPS")),
|
|
||||||
"revenue_estimate": None,
|
|
||||||
"revenue_actual": None,
|
|
||||||
})
|
|
||||||
return out
|
|
||||||
|
|
||||||
|
|
||||||
async def _main() -> None:
|
async def _main() -> None:
|
||||||
@@ -304,228 +301,214 @@ async def _main() -> None:
|
|||||||
|
|
||||||
if not settings.fmp_api_key:
|
if not settings.fmp_api_key:
|
||||||
raise SystemExit("FMP_API_KEY required")
|
raise SystemExit("FMP_API_KEY required")
|
||||||
|
|
||||||
start = date.fromisoformat(args.from_date)
|
start = date.fromisoformat(args.from_date)
|
||||||
end = date.fromisoformat(args.to_date) if args.to_date else date.today()
|
end = date.fromisoformat(args.to_date) if args.to_date else date.today()
|
||||||
engine = create_engine(
|
if start > end:
|
||||||
f"sqlite:///{snapshot.resolve().as_posix()}",
|
raise SystemExit("--from-date must not be after --to-date")
|
||||||
future=True,
|
|
||||||
)
|
|
||||||
_ensure_tables(engine)
|
|
||||||
|
|
||||||
|
engine = create_engine(f"sqlite:///{snapshot.resolve().as_posix()}", future=True)
|
||||||
|
_ensure_tables(engine)
|
||||||
|
all_windows = _windows(start, end, int(args.window_days))
|
||||||
with engine.connect() as conn:
|
with engine.connect() as conn:
|
||||||
symbols = [
|
symbols = [
|
||||||
str(r[0]).upper().replace(".", "-")
|
str(row[0]).upper().replace(".", "-")
|
||||||
for r in conn.execute(text("SELECT symbol FROM tickers ORDER BY symbol"))
|
for row in conn.execute(text("SELECT symbol FROM tickers ORDER BY symbol"))
|
||||||
]
|
]
|
||||||
done = set()
|
completed = {
|
||||||
if not args.refetch_done:
|
(str(row[0]), str(row[1]))
|
||||||
done = {
|
for row in conn.execute(
|
||||||
str(r[0])
|
|
||||||
for r in conn.execute(
|
|
||||||
text(
|
text(
|
||||||
"SELECT symbol FROM earnings_backfill_meta "
|
"SELECT from_date, to_date FROM earnings_backfill_windows "
|
||||||
"WHERE status='done' AND n_events > 0"
|
"WHERE status='done'"
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
pending = [
|
||||||
pending = [s for s in symbols if s not in done]
|
window
|
||||||
|
for window in all_windows
|
||||||
|
if args.refetch_windows
|
||||||
|
or (window[0].isoformat(), window[1].isoformat()) not in completed
|
||||||
|
]
|
||||||
|
universe = set(symbols)
|
||||||
print(f"Snapshot: {snapshot}")
|
print(f"Snapshot: {snapshot}")
|
||||||
print(f"Universe: {len(symbols)}; pending: {len(pending)}; done: {len(done)}")
|
print(f"Universe: {len(symbols)} symbols")
|
||||||
print(f"Window filter: {start} → {end}")
|
print(f"Window: {start} -> {end}")
|
||||||
print(f"Provider: {args.provider}")
|
print(
|
||||||
|
f"Bulk windows: {len(all_windows)} total; "
|
||||||
req_budget = int(args.limit)
|
f"{len(all_windows) - len(pending)} done; {len(pending)} pending"
|
||||||
reqs_used = 0
|
|
||||||
events_written = 0
|
|
||||||
mode = "per_symbol"
|
|
||||||
use_av = args.provider in ("alpha_vantage", "auto") and bool(
|
|
||||||
getattr(settings, "alpha_vantage_api_key", "")
|
|
||||||
)
|
)
|
||||||
use_fmp = args.provider in ("fmp", "auto") and bool(settings.fmp_api_key)
|
print("Provider: FMP bulk earnings-calendar only")
|
||||||
|
|
||||||
|
requests_this_run = 0
|
||||||
|
rows_upserted = 0
|
||||||
|
duplicate_rows = 0
|
||||||
|
restated_rows = 0
|
||||||
|
stop_note: str | None = None
|
||||||
async with httpx.AsyncClient(timeout=60.0) as client:
|
async with httpx.AsyncClient(timeout=60.0) as client:
|
||||||
if (
|
for index, (window_start, window_end) in enumerate(pending, 1):
|
||||||
not args.force_symbol
|
if requests_this_run >= int(args.limit):
|
||||||
and req_budget > 0
|
stop_note = "request_budget_exhausted"
|
||||||
and use_fmp
|
break
|
||||||
and args.provider != "alpha_vantage"
|
try:
|
||||||
):
|
raw_rows, status_code, error = await _fetch_bulk_window(
|
||||||
print("Attempting bulk earnings-calendar…")
|
client, settings.fmp_api_key, window_start, window_end
|
||||||
bulk_rows, bulk_reqs, err = await _try_bulk(
|
|
||||||
client, settings.fmp_api_key, start, end
|
|
||||||
)
|
)
|
||||||
reqs_used += bulk_reqs
|
except Exception as exc:
|
||||||
if err:
|
raw_rows, status_code = [], 0
|
||||||
print(f" Bulk unavailable: {err} (requests={bulk_reqs})")
|
error = f"request_error:{type(exc).__name__}:{exc}"
|
||||||
else:
|
requests_this_run += 1
|
||||||
# Filter to universe.
|
if error:
|
||||||
uni = set(symbols)
|
_write_window_status(
|
||||||
bulk_rows = [r for r in bulk_rows if r["symbol"] in uni]
|
engine,
|
||||||
|
start=window_start,
|
||||||
|
end=window_end,
|
||||||
|
status="error",
|
||||||
|
note=f"http={status_code} {error}"[:300],
|
||||||
|
)
|
||||||
|
stop_note = error
|
||||||
|
print(
|
||||||
|
f"STOP {window_start}..{window_end}: {error} "
|
||||||
|
f"(http={status_code}, request={requests_this_run})"
|
||||||
|
)
|
||||||
|
break
|
||||||
|
|
||||||
|
in_universe = [row for row in raw_rows if row["symbol"] in universe]
|
||||||
|
deduped, duplicate_n, restated_n = _dedupe_bulk_rows(in_universe)
|
||||||
with engine.begin() as conn:
|
with engine.begin() as conn:
|
||||||
events_written += _upsert_events(conn, bulk_rows, "fmp_earnings_calendar")
|
rows_upserted += _upsert_events(conn, deduped)
|
||||||
for sym in symbols:
|
_write_window_status(
|
||||||
n = conn.execute(
|
engine,
|
||||||
|
start=window_start,
|
||||||
|
end=window_end,
|
||||||
|
status="done",
|
||||||
|
raw_n=len(raw_rows),
|
||||||
|
universe_n=len(deduped),
|
||||||
|
duplicate_n=duplicate_n,
|
||||||
|
restated_n=restated_n,
|
||||||
|
note="bulk",
|
||||||
|
)
|
||||||
|
duplicate_rows += duplicate_n
|
||||||
|
restated_rows += restated_n
|
||||||
|
if index == 1 or index % 10 == 0 or index == len(pending):
|
||||||
|
print(
|
||||||
|
f"progress windows={index}/{len(pending)} "
|
||||||
|
f"requests={requests_this_run}/{args.limit} "
|
||||||
|
f"last={window_start}..{window_end} rows={len(deduped)}"
|
||||||
|
)
|
||||||
|
if args.sleep > 0:
|
||||||
|
await asyncio.sleep(float(args.sleep))
|
||||||
|
|
||||||
|
with engine.begin() as conn:
|
||||||
|
windows_done = int(
|
||||||
|
conn.execute(
|
||||||
text(
|
text(
|
||||||
"SELECT COUNT(*) FROM earnings_events WHERE symbol=:s"
|
"SELECT COUNT(*) FROM earnings_backfill_windows "
|
||||||
|
"WHERE status='done' AND from_date >= :a AND to_date <= :b"
|
||||||
),
|
),
|
||||||
{"s": sym},
|
{"a": start.isoformat(), "b": end.isoformat()},
|
||||||
).scalar_one()
|
).scalar_one()
|
||||||
|
)
|
||||||
|
complete = windows_done >= len(all_windows)
|
||||||
|
if complete:
|
||||||
|
now = datetime.now(timezone.utc).isoformat()
|
||||||
|
for symbol in symbols:
|
||||||
|
count = int(
|
||||||
|
conn.execute(
|
||||||
|
text(
|
||||||
|
"SELECT COUNT(*) FROM earnings_events "
|
||||||
|
"WHERE symbol=:symbol AND announce_date BETWEEN :a AND :b"
|
||||||
|
),
|
||||||
|
{"symbol": symbol, "a": start.isoformat(), "b": end.isoformat()},
|
||||||
|
).scalar_one()
|
||||||
|
)
|
||||||
conn.execute(
|
conn.execute(
|
||||||
text(
|
text(
|
||||||
"""
|
"""
|
||||||
INSERT INTO earnings_backfill_meta(symbol, status, n_events, updated_at, note)
|
INSERT INTO earnings_backfill_meta(symbol, status, n_events, updated_at, note)
|
||||||
VALUES (:s, 'done', :n, :t, 'bulk')
|
VALUES (:symbol, 'done', :count, :now, 'bulk_complete')
|
||||||
ON CONFLICT(symbol) DO UPDATE SET
|
ON CONFLICT(symbol) DO UPDATE SET
|
||||||
status='done', n_events=excluded.n_events,
|
status='done', n_events=excluded.n_events,
|
||||||
updated_at=excluded.updated_at, note=excluded.note
|
updated_at=excluded.updated_at, note=excluded.note
|
||||||
"""
|
"""
|
||||||
),
|
),
|
||||||
{
|
{"symbol": symbol, "count": count, "now": now},
|
||||||
"s": sym,
|
|
||||||
"n": int(n),
|
|
||||||
"t": datetime.now(timezone.utc).isoformat(),
|
|
||||||
},
|
|
||||||
)
|
)
|
||||||
mode = "bulk"
|
params = {"a": start.isoformat(), "b": end.isoformat()}
|
||||||
print(f" Bulk wrote {events_written} events; requests={bulk_reqs}")
|
|
||||||
pending = []
|
|
||||||
|
|
||||||
# Per-symbol fallback / completion.
|
|
||||||
fmp_limited = False
|
|
||||||
for sym in pending:
|
|
||||||
if reqs_used >= req_budget:
|
|
||||||
print(f"Request budget exhausted ({req_budget}). Resume later.")
|
|
||||||
break
|
|
||||||
items: list[dict] = []
|
|
||||||
source = "fmp_earnings"
|
|
||||||
note = "per_symbol"
|
|
||||||
try:
|
|
||||||
if use_fmp and not fmp_limited and args.provider != "alpha_vantage":
|
|
||||||
items = await _fetch_symbol(client, settings.fmp_api_key, sym)
|
|
||||||
source = "fmp_earnings"
|
|
||||||
note = "fmp_per_symbol"
|
|
||||||
# Empty list may mean soft-limit or no data — try AV if available.
|
|
||||||
if not items and use_av:
|
|
||||||
items = await _fetch_symbol_alpha_vantage(
|
|
||||||
client, settings.alpha_vantage_api_key, sym
|
|
||||||
)
|
|
||||||
source = "alpha_vantage_earnings"
|
|
||||||
note = "av_after_fmp_empty"
|
|
||||||
reqs_used += 1 # count AV call separately below too
|
|
||||||
elif use_av:
|
|
||||||
items = await _fetch_symbol_alpha_vantage(
|
|
||||||
client, settings.alpha_vantage_api_key, sym
|
|
||||||
)
|
|
||||||
source = "alpha_vantage_earnings"
|
|
||||||
note = "av_per_symbol"
|
|
||||||
else:
|
|
||||||
raise RuntimeError("no provider available")
|
|
||||||
except Exception as exc:
|
|
||||||
msg = str(exc)
|
|
||||||
print(f" FAIL {sym}: {msg}")
|
|
||||||
reqs_used += 1
|
|
||||||
if "rate_limited" in msg and note.startswith("fmp"):
|
|
||||||
fmp_limited = True
|
|
||||||
with engine.begin() as conn:
|
|
||||||
conn.execute(
|
|
||||||
text(
|
|
||||||
"""
|
|
||||||
INSERT INTO earnings_backfill_meta(symbol, status, n_events, updated_at, note)
|
|
||||||
VALUES (:s, 'error', 0, :t, :n)
|
|
||||||
ON CONFLICT(symbol) DO UPDATE SET
|
|
||||||
status='error', updated_at=excluded.updated_at, note=excluded.note
|
|
||||||
"""
|
|
||||||
),
|
|
||||||
{
|
|
||||||
"s": sym,
|
|
||||||
"t": datetime.now(timezone.utc).isoformat(),
|
|
||||||
"n": msg[:200],
|
|
||||||
},
|
|
||||||
)
|
|
||||||
if args.sleep > 0:
|
|
||||||
await asyncio.sleep(args.sleep)
|
|
||||||
continue
|
|
||||||
|
|
||||||
reqs_used += 1
|
|
||||||
# Keep all rows with dates on/before end — SUE needs trailing history.
|
|
||||||
filtered = [
|
|
||||||
r for r in items if r["announce_date"] <= end.isoformat()
|
|
||||||
]
|
|
||||||
# Do NOT mark empty as done — leave pending for another provider/day.
|
|
||||||
status = "done" if filtered else "empty"
|
|
||||||
with engine.begin() as conn:
|
|
||||||
n_w = _upsert_events(conn, filtered, source) if filtered else 0
|
|
||||||
events_written += n_w
|
|
||||||
conn.execute(
|
|
||||||
text(
|
|
||||||
"""
|
|
||||||
INSERT INTO earnings_backfill_meta(symbol, status, n_events, updated_at, note)
|
|
||||||
VALUES (:s, :st, :n, :t, :note)
|
|
||||||
ON CONFLICT(symbol) DO UPDATE SET
|
|
||||||
status=excluded.status, n_events=excluded.n_events,
|
|
||||||
updated_at=excluded.updated_at, note=excluded.note
|
|
||||||
"""
|
|
||||||
),
|
|
||||||
{
|
|
||||||
"s": sym,
|
|
||||||
"st": status,
|
|
||||||
"n": len(filtered),
|
|
||||||
"t": datetime.now(timezone.utc).isoformat(),
|
|
||||||
"note": note,
|
|
||||||
},
|
|
||||||
)
|
|
||||||
if reqs_used % 10 == 0 or reqs_used == 1:
|
|
||||||
print(
|
|
||||||
f" progress reqs={reqs_used}/{req_budget} last={sym} "
|
|
||||||
f"events_batch={len(filtered)} src={source}"
|
|
||||||
)
|
|
||||||
# AV free tier is ~5/min or 25/day — be polite when using it.
|
|
||||||
sleep_s = float(args.sleep)
|
|
||||||
if source.startswith("alpha_vantage"):
|
|
||||||
sleep_s = max(sleep_s, 12.0)
|
|
||||||
if sleep_s > 0:
|
|
||||||
await asyncio.sleep(sleep_s)
|
|
||||||
|
|
||||||
with engine.connect() as conn:
|
|
||||||
total_events = int(
|
total_events = int(
|
||||||
conn.execute(text("SELECT COUNT(*) FROM earnings_events")).scalar_one()
|
conn.execute(
|
||||||
|
text(
|
||||||
|
"SELECT COUNT(*) FROM earnings_events "
|
||||||
|
"WHERE symbol IN (SELECT symbol FROM tickers) "
|
||||||
|
"AND announce_date BETWEEN :a AND :b"
|
||||||
|
),
|
||||||
|
params,
|
||||||
|
).scalar_one()
|
||||||
)
|
)
|
||||||
done_n = int(
|
paired_events = int(
|
||||||
|
conn.execute(
|
||||||
|
text(
|
||||||
|
"SELECT COUNT(*) FROM earnings_events "
|
||||||
|
"WHERE symbol IN (SELECT symbol FROM tickers) "
|
||||||
|
"AND announce_date BETWEEN :a AND :b "
|
||||||
|
"AND eps_actual IS NOT NULL AND eps_estimate IS NOT NULL"
|
||||||
|
),
|
||||||
|
params,
|
||||||
|
).scalar_one()
|
||||||
|
)
|
||||||
|
date_range = conn.execute(
|
||||||
|
text(
|
||||||
|
"SELECT MIN(announce_date), MAX(announce_date) FROM earnings_events "
|
||||||
|
"WHERE symbol IN (SELECT symbol FROM tickers) "
|
||||||
|
"AND announce_date BETWEEN :a AND :b"
|
||||||
|
),
|
||||||
|
params,
|
||||||
|
).fetchone()
|
||||||
|
done_symbols = int(
|
||||||
conn.execute(
|
conn.execute(
|
||||||
text("SELECT COUNT(*) FROM earnings_backfill_meta WHERE status='done'")
|
text("SELECT COUNT(*) FROM earnings_backfill_meta WHERE status='done'")
|
||||||
).scalar_one()
|
).scalar_one()
|
||||||
)
|
)
|
||||||
d_range = conn.execute(
|
totals = conn.execute(
|
||||||
text("SELECT MIN(announce_date), MAX(announce_date) FROM earnings_events")
|
|
||||||
).fetchone()
|
|
||||||
with_actual = int(
|
|
||||||
conn.execute(
|
|
||||||
text(
|
text(
|
||||||
"SELECT COUNT(*) FROM earnings_events "
|
"SELECT COALESCE(SUM(requests),0), COALESCE(SUM(duplicate_rows),0), "
|
||||||
"WHERE eps_actual IS NOT NULL AND eps_estimate IS NOT NULL"
|
"COALESCE(SUM(restated_rows),0) FROM earnings_backfill_windows "
|
||||||
)
|
"WHERE from_date >= :a AND to_date <= :b"
|
||||||
).scalar_one()
|
),
|
||||||
)
|
params,
|
||||||
|
).fetchone()
|
||||||
|
|
||||||
summary = {
|
summary = {
|
||||||
"mode": mode,
|
"mode": "fmp_bulk_date_range_only",
|
||||||
"fmp_requests": reqs_used,
|
"window": {"from": start.isoformat(), "to": end.isoformat()},
|
||||||
"events_written_this_run": events_written,
|
"window_days": int(args.window_days),
|
||||||
"total_events": total_events,
|
"bulk_windows_total": len(all_windows),
|
||||||
"symbols_done": done_n,
|
"bulk_windows_done": windows_done,
|
||||||
|
"bulk_requests_this_run": requests_this_run,
|
||||||
|
"bulk_requests_logged_total": int(totals[0]),
|
||||||
|
"rows_upserted_this_run": rows_upserted,
|
||||||
|
"duplicate_rows_this_run": duplicate_rows,
|
||||||
|
"restated_rows_this_run": restated_rows,
|
||||||
|
"duplicate_rows_logged_total": int(totals[1]),
|
||||||
|
"restated_rows_logged_total": int(totals[2]),
|
||||||
|
"dedupe_policy": (
|
||||||
|
"UNIQUE(symbol, announce_date); prefer more non-null fields, then "
|
||||||
|
"the provider's later occurrence; non-null bulk fields replace prior "
|
||||||
|
"values while null bulk fields retain existing values"
|
||||||
|
),
|
||||||
|
"events_in_window": total_events,
|
||||||
|
"events_with_actual_and_estimate": paired_events,
|
||||||
|
"symbols_done": done_symbols,
|
||||||
"symbols_universe": len(symbols),
|
"symbols_universe": len(symbols),
|
||||||
"announce_date_range": {"min": d_range[0], "max": d_range[1]},
|
"announce_date_range": {"min": date_range[0], "max": date_range[1]},
|
||||||
"events_with_actual_and_estimate": with_actual,
|
"request_budget": int(args.limit),
|
||||||
"budget": req_budget,
|
"stop_note": stop_note,
|
||||||
"complete": done_n >= len(symbols),
|
"complete": complete,
|
||||||
}
|
}
|
||||||
|
output = Path("reports/earnings-backfill-status.json")
|
||||||
|
output.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
output.write_text(json.dumps(summary, indent=2) + "\n", encoding="utf-8")
|
||||||
print(json.dumps(summary, indent=2))
|
print(json.dumps(summary, indent=2))
|
||||||
out = Path("reports") / "earnings-backfill-status.json"
|
print(f"Wrote {output}")
|
||||||
out.parent.mkdir(parents=True, exist_ok=True)
|
|
||||||
out.write_text(json.dumps(summary, indent=2) + "\n", encoding="utf-8")
|
|
||||||
print(f"Wrote {out}")
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
|
|||||||
@@ -97,6 +97,14 @@ def _parse_args() -> argparse.Namespace:
|
|||||||
default=5,
|
default=5,
|
||||||
help="Retries per symbol on RateLimitError.",
|
help="Retries per symbol on RateLimitError.",
|
||||||
)
|
)
|
||||||
|
p.add_argument(
|
||||||
|
"--source-symbols-only",
|
||||||
|
action="store_true",
|
||||||
|
help=(
|
||||||
|
"Refresh only symbols present in --source. Useful for repairing "
|
||||||
|
"per-symbol depth without re-fetching the broad rank-only pool."
|
||||||
|
),
|
||||||
|
)
|
||||||
p.add_argument("--quiet", action="store_true")
|
p.add_argument("--quiet", action="store_true")
|
||||||
return p.parse_args()
|
return p.parse_args()
|
||||||
|
|
||||||
@@ -218,6 +226,15 @@ async def _main() -> None:
|
|||||||
if not source.exists():
|
if not source.exists():
|
||||||
raise SystemExit(f"Source snapshot not found: {source}")
|
raise SystemExit(f"Source snapshot not found: {source}")
|
||||||
|
|
||||||
|
source_engine = create_engine(
|
||||||
|
f"sqlite:///{source.resolve().as_posix()}", future=True
|
||||||
|
)
|
||||||
|
with source_engine.connect() as conn:
|
||||||
|
source_symbols = {
|
||||||
|
str(row[0]) for row in conn.execute(text("SELECT symbol FROM tickers"))
|
||||||
|
}
|
||||||
|
source_engine.dispose()
|
||||||
|
|
||||||
# Any rebuild/update invalidates prior completion until we finish cleanly.
|
# Any rebuild/update invalidates prior completion until we finish cleanly.
|
||||||
clear_manifest(output)
|
clear_manifest(output)
|
||||||
|
|
||||||
@@ -241,6 +258,11 @@ async def _main() -> None:
|
|||||||
start = end - timedelta(days=int(args.history_days))
|
start = end - timedelta(days=int(args.history_days))
|
||||||
|
|
||||||
print("Resolving universe pool (nasdaq_all ∪ sp500)…")
|
print("Resolving universe pool (nasdaq_all ∪ sp500)…")
|
||||||
|
if args.source_symbols_only:
|
||||||
|
pool = sorted(source_symbols)
|
||||||
|
sources = {"pool": "source_snapshot"}
|
||||||
|
print(" source snapshot: symbol pool selected")
|
||||||
|
else:
|
||||||
pool, sources = await _resolve_pool()
|
pool, sources = await _resolve_pool()
|
||||||
print(f"Pool size: {len(pool)} (sources={sources})")
|
print(f"Pool size: {len(pool)} (sources={sources})")
|
||||||
|
|
||||||
@@ -257,7 +279,7 @@ async def _main() -> None:
|
|||||||
text("SELECT id, symbol FROM tickers")
|
text("SELECT id, symbol FROM tickers")
|
||||||
).fetchall()
|
).fetchall()
|
||||||
existing_ids = {str(sym): int(tid) for tid, sym in existing_rows}
|
existing_ids = {str(sym): int(tid) for tid, sym in existing_rows}
|
||||||
prod_symbols = set(existing_ids)
|
prod_symbols = set(source_symbols)
|
||||||
|
|
||||||
bar_counts: dict[str, int] = {}
|
bar_counts: dict[str, int] = {}
|
||||||
for sym, tid in existing_ids.items():
|
for sym, tid in existing_ids.items():
|
||||||
@@ -363,7 +385,7 @@ async def _main() -> None:
|
|||||||
for b in bars
|
for b in bars
|
||||||
],
|
],
|
||||||
)
|
)
|
||||||
if is_new:
|
if is_new and sym not in prod_symbols:
|
||||||
write.execute(
|
write.execute(
|
||||||
text(
|
text(
|
||||||
"INSERT OR REPLACE INTO research_rank_only "
|
"INSERT OR REPLACE INTO research_rank_only "
|
||||||
@@ -385,6 +407,39 @@ async def _main() -> None:
|
|||||||
f"elapsed={elapsed/60:.1f}m last={sym} bars={len(bars)}"
|
f"elapsed={elapsed/60:.1f}m last={sym} bars={len(bars)}"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
benchmark_rows = 0
|
||||||
|
try:
|
||||||
|
benchmark_bars = await _fetch_symbol_bars(
|
||||||
|
provider,
|
||||||
|
"SPY",
|
||||||
|
start,
|
||||||
|
end,
|
||||||
|
max_retries=args.max_retries,
|
||||||
|
sleep_s=args.sleep,
|
||||||
|
)
|
||||||
|
with engine.begin() as write:
|
||||||
|
write.execute(
|
||||||
|
text(
|
||||||
|
"DELETE FROM benchmark_prices WHERE symbol='SPY' "
|
||||||
|
"AND date >= :start AND date <= :end"
|
||||||
|
),
|
||||||
|
{"start": start.isoformat(), "end": end.isoformat()},
|
||||||
|
)
|
||||||
|
if benchmark_bars:
|
||||||
|
write.execute(
|
||||||
|
text(
|
||||||
|
"INSERT INTO benchmark_prices(symbol, date, close) "
|
||||||
|
"VALUES ('SPY', :date, :close)"
|
||||||
|
),
|
||||||
|
[
|
||||||
|
{"date": bar.date.isoformat(), "close": float(bar.close)}
|
||||||
|
for bar in benchmark_bars
|
||||||
|
],
|
||||||
|
)
|
||||||
|
benchmark_rows = len(benchmark_bars)
|
||||||
|
except Exception as exc:
|
||||||
|
print(f" benchmark SPY refresh FAIL {exc}")
|
||||||
|
|
||||||
rank_only_n = conn.execute(
|
rank_only_n = conn.execute(
|
||||||
text("SELECT COUNT(*) FROM research_rank_only")
|
text("SELECT COUNT(*) FROM research_rank_only")
|
||||||
).scalar_one()
|
).scalar_one()
|
||||||
@@ -409,6 +464,8 @@ async def _main() -> None:
|
|||||||
"prod_symbols_at_start": len(prod_symbols),
|
"prod_symbols_at_start": len(prod_symbols),
|
||||||
"pool_size": len(pool),
|
"pool_size": len(pool),
|
||||||
"to_fetch": len(to_fetch),
|
"to_fetch": len(to_fetch),
|
||||||
|
"source_symbols_only": bool(args.source_symbols_only),
|
||||||
|
"benchmark_spy_rows": benchmark_rows,
|
||||||
},
|
},
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,657 @@
|
|||||||
|
"""Import the public post-no-preference/earnings DoltHub database.
|
||||||
|
|
||||||
|
The earnings calendar and EPS history are separate tables in the source. This
|
||||||
|
importer aligns them monotonically per symbol, keeps every calendar event for
|
||||||
|
the defensive gap study, and stores the longer EPS history separately for SUE
|
||||||
|
scaling. EPS history without an announcement date is never exposed as a live
|
||||||
|
signal event.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import csv
|
||||||
|
import json
|
||||||
|
import math
|
||||||
|
import sqlite3
|
||||||
|
from collections import defaultdict
|
||||||
|
from datetime import date, datetime, timedelta, timezone
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
|
||||||
|
EVENTS_DDL = """
|
||||||
|
CREATE TABLE IF NOT EXISTS earnings_events (
|
||||||
|
id INTEGER PRIMARY KEY,
|
||||||
|
symbol TEXT NOT NULL,
|
||||||
|
announce_date TEXT NOT NULL,
|
||||||
|
announce_time TEXT,
|
||||||
|
eps_estimate REAL,
|
||||||
|
eps_actual REAL,
|
||||||
|
revenue_estimate REAL,
|
||||||
|
revenue_actual REAL,
|
||||||
|
source TEXT NOT NULL,
|
||||||
|
fetched_at TEXT NOT NULL,
|
||||||
|
period_end_date TEXT,
|
||||||
|
UNIQUE(symbol, announce_date)
|
||||||
|
)
|
||||||
|
"""
|
||||||
|
META_DDL = """
|
||||||
|
CREATE TABLE IF NOT EXISTS earnings_backfill_meta (
|
||||||
|
symbol TEXT PRIMARY KEY,
|
||||||
|
status TEXT NOT NULL,
|
||||||
|
n_events INTEGER NOT NULL DEFAULT 0,
|
||||||
|
updated_at TEXT NOT NULL,
|
||||||
|
note TEXT
|
||||||
|
)
|
||||||
|
"""
|
||||||
|
SURPRISE_HISTORY_DDL = """
|
||||||
|
CREATE TABLE IF NOT EXISTS earnings_surprise_history (
|
||||||
|
symbol TEXT NOT NULL,
|
||||||
|
period_end_date TEXT NOT NULL,
|
||||||
|
eps_estimate REAL,
|
||||||
|
eps_actual REAL,
|
||||||
|
source TEXT NOT NULL,
|
||||||
|
fetched_at TEXT NOT NULL,
|
||||||
|
PRIMARY KEY(symbol, period_end_date)
|
||||||
|
)
|
||||||
|
"""
|
||||||
|
|
||||||
|
SKIP_EVENT_COST = 45.0
|
||||||
|
SKIP_PERIOD_COST = 45.0
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_args() -> argparse.Namespace:
|
||||||
|
parser = argparse.ArgumentParser(description=__doc__)
|
||||||
|
parser.add_argument("--snapshot", default="backtest_snapshots/prod.sqlite")
|
||||||
|
parser.add_argument("--calendar-csv", required=True)
|
||||||
|
parser.add_argument("--history-csv", required=True)
|
||||||
|
parser.add_argument("--from-date", default="2020-01-22")
|
||||||
|
parser.add_argument("--to-date", required=True)
|
||||||
|
parser.add_argument("--source-commit", required=True)
|
||||||
|
parser.add_argument(
|
||||||
|
"--source-url",
|
||||||
|
default="https://www.dolthub.com/repositories/post-no-preference/earnings",
|
||||||
|
)
|
||||||
|
parser.add_argument("--max-period-lag-days", type=int, default=90)
|
||||||
|
parser.add_argument("--max-period-lead-days", type=int, default=14)
|
||||||
|
parser.add_argument(
|
||||||
|
"--status-output", default="reports/earnings-backfill-status.json"
|
||||||
|
)
|
||||||
|
return parser.parse_args()
|
||||||
|
|
||||||
|
|
||||||
|
def _normalise_symbol(value: Any) -> str:
|
||||||
|
return str(value or "").strip().upper().replace(".", "-")
|
||||||
|
|
||||||
|
|
||||||
|
def _normalise_session(value: Any) -> str | None:
|
||||||
|
cleaned = str(value or "").strip().lower().replace("_", " ").replace("-", " ")
|
||||||
|
aliases = {
|
||||||
|
"before market open": "bmo",
|
||||||
|
"before open": "bmo",
|
||||||
|
"bmo": "bmo",
|
||||||
|
"after market close": "amc",
|
||||||
|
"after close": "amc",
|
||||||
|
"amc": "amc",
|
||||||
|
"during market hours": "during",
|
||||||
|
"dmh": "during",
|
||||||
|
}
|
||||||
|
return aliases.get(cleaned, cleaned or None)
|
||||||
|
|
||||||
|
|
||||||
|
def _number(value: Any) -> float | None:
|
||||||
|
if value is None or str(value).strip() == "":
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
result = float(value)
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
return None
|
||||||
|
return result if math.isfinite(result) else None
|
||||||
|
|
||||||
|
|
||||||
|
def _read_calendar(
|
||||||
|
path: Path,
|
||||||
|
universe: set[str],
|
||||||
|
start: date,
|
||||||
|
end: date,
|
||||||
|
) -> tuple[dict[str, list[dict[str, Any]]], dict[str, int]]:
|
||||||
|
by_key: dict[tuple[str, date], dict[str, Any]] = {}
|
||||||
|
raw_rows = 0
|
||||||
|
universe_rows = 0
|
||||||
|
duplicate_rows = 0
|
||||||
|
restated_rows = 0
|
||||||
|
with path.open(newline="", encoding="utf-8-sig") as handle:
|
||||||
|
for raw in csv.DictReader(handle):
|
||||||
|
raw_rows += 1
|
||||||
|
symbol = _normalise_symbol(raw.get("act_symbol"))
|
||||||
|
raw_date = str(raw.get("date") or "")[:10]
|
||||||
|
if symbol not in universe or not raw_date:
|
||||||
|
continue
|
||||||
|
event_date = date.fromisoformat(raw_date)
|
||||||
|
if not start <= event_date <= end:
|
||||||
|
continue
|
||||||
|
universe_rows += 1
|
||||||
|
row = {
|
||||||
|
"symbol": symbol,
|
||||||
|
"announce_date": event_date,
|
||||||
|
"announce_time": _normalise_session(raw.get("when")),
|
||||||
|
}
|
||||||
|
key = (symbol, event_date)
|
||||||
|
previous = by_key.get(key)
|
||||||
|
if previous is not None:
|
||||||
|
duplicate_rows += 1
|
||||||
|
if (
|
||||||
|
previous.get("announce_time") is not None
|
||||||
|
and row.get("announce_time") is not None
|
||||||
|
and previous["announce_time"] != row["announce_time"]
|
||||||
|
):
|
||||||
|
restated_rows += 1
|
||||||
|
if row.get("announce_time") is not None:
|
||||||
|
by_key[key] = row
|
||||||
|
else:
|
||||||
|
by_key[key] = row
|
||||||
|
grouped: dict[str, list[dict[str, Any]]] = defaultdict(list)
|
||||||
|
for row in by_key.values():
|
||||||
|
grouped[row["symbol"]].append(row)
|
||||||
|
for rows in grouped.values():
|
||||||
|
rows.sort(key=lambda item: item["announce_date"])
|
||||||
|
return grouped, {
|
||||||
|
"raw_rows": raw_rows,
|
||||||
|
"universe_rows_in_window": universe_rows,
|
||||||
|
"deduped_rows_in_window": len(by_key),
|
||||||
|
"duplicate_rows": duplicate_rows,
|
||||||
|
"restated_rows": restated_rows,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _read_history(
|
||||||
|
path: Path, universe: set[str]
|
||||||
|
) -> tuple[dict[str, list[dict[str, Any]]], dict[str, int]]:
|
||||||
|
by_key: dict[tuple[str, date], dict[str, Any]] = {}
|
||||||
|
raw_rows = 0
|
||||||
|
universe_rows = 0
|
||||||
|
duplicate_rows = 0
|
||||||
|
restated_rows = 0
|
||||||
|
fields = ("eps_actual", "eps_estimate")
|
||||||
|
with path.open(newline="", encoding="utf-8-sig") as handle:
|
||||||
|
for raw in csv.DictReader(handle):
|
||||||
|
raw_rows += 1
|
||||||
|
symbol = _normalise_symbol(raw.get("act_symbol"))
|
||||||
|
raw_date = str(raw.get("period_end_date") or "")[:10]
|
||||||
|
if symbol not in universe or not raw_date:
|
||||||
|
continue
|
||||||
|
universe_rows += 1
|
||||||
|
period_end = date.fromisoformat(raw_date)
|
||||||
|
row = {
|
||||||
|
"symbol": symbol,
|
||||||
|
"period_end_date": period_end,
|
||||||
|
"eps_actual": _number(raw.get("reported")),
|
||||||
|
"eps_estimate": _number(raw.get("estimate")),
|
||||||
|
}
|
||||||
|
key = (symbol, period_end)
|
||||||
|
previous = by_key.get(key)
|
||||||
|
if previous is not None:
|
||||||
|
duplicate_rows += 1
|
||||||
|
if any(
|
||||||
|
previous.get(field) is not None
|
||||||
|
and row.get(field) is not None
|
||||||
|
and previous[field] != row[field]
|
||||||
|
for field in fields
|
||||||
|
):
|
||||||
|
restated_rows += 1
|
||||||
|
previous_score = sum(previous.get(field) is not None for field in fields)
|
||||||
|
row_score = sum(row.get(field) is not None for field in fields)
|
||||||
|
if row_score >= previous_score:
|
||||||
|
by_key[key] = row
|
||||||
|
else:
|
||||||
|
by_key[key] = row
|
||||||
|
grouped: dict[str, list[dict[str, Any]]] = defaultdict(list)
|
||||||
|
for row in by_key.values():
|
||||||
|
grouped[row["symbol"]].append(row)
|
||||||
|
for rows in grouped.values():
|
||||||
|
rows.sort(key=lambda item: item["period_end_date"])
|
||||||
|
return grouped, {
|
||||||
|
"raw_rows": raw_rows,
|
||||||
|
"universe_rows": universe_rows,
|
||||||
|
"deduped_rows": len(by_key),
|
||||||
|
"duplicate_rows": duplicate_rows,
|
||||||
|
"restated_rows": restated_rows,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _match_cost(event: dict[str, Any], period: dict[str, Any]) -> float:
|
||||||
|
delta = (event["announce_date"] - period["period_end_date"]).days
|
||||||
|
missing_session_penalty = 3.0 if event.get("announce_time") is None else 0.0
|
||||||
|
return float(abs(delta - 30)) + missing_session_penalty
|
||||||
|
|
||||||
|
|
||||||
|
def _align_symbol(
|
||||||
|
events: list[dict[str, Any]],
|
||||||
|
periods: list[dict[str, Any]],
|
||||||
|
*,
|
||||||
|
max_lag_days: int,
|
||||||
|
max_lead_days: int,
|
||||||
|
) -> tuple[list[tuple[int, int]], list[int], list[int]]:
|
||||||
|
"""Return a minimum-cost monotonic calendar-to-period alignment."""
|
||||||
|
n_events = len(events)
|
||||||
|
n_periods = len(periods)
|
||||||
|
scores = [[0.0] * (n_periods + 1) for _ in range(n_events + 1)]
|
||||||
|
choices = [[""] * (n_periods + 1) for _ in range(n_events + 1)]
|
||||||
|
for event_index in range(n_events - 1, -1, -1):
|
||||||
|
scores[event_index][n_periods] = (
|
||||||
|
scores[event_index + 1][n_periods] + SKIP_EVENT_COST
|
||||||
|
)
|
||||||
|
choices[event_index][n_periods] = "event"
|
||||||
|
for period_index in range(n_periods - 1, -1, -1):
|
||||||
|
scores[n_events][period_index] = (
|
||||||
|
scores[n_events][period_index + 1] + SKIP_PERIOD_COST
|
||||||
|
)
|
||||||
|
choices[n_events][period_index] = "period"
|
||||||
|
|
||||||
|
for event_index in range(n_events - 1, -1, -1):
|
||||||
|
for period_index in range(n_periods - 1, -1, -1):
|
||||||
|
options = [
|
||||||
|
(
|
||||||
|
scores[event_index + 1][period_index] + SKIP_EVENT_COST,
|
||||||
|
2,
|
||||||
|
"event",
|
||||||
|
),
|
||||||
|
(
|
||||||
|
scores[event_index][period_index + 1] + SKIP_PERIOD_COST,
|
||||||
|
1,
|
||||||
|
"period",
|
||||||
|
),
|
||||||
|
]
|
||||||
|
delta = (
|
||||||
|
events[event_index]["announce_date"]
|
||||||
|
- periods[period_index]["period_end_date"]
|
||||||
|
).days
|
||||||
|
if -max_lead_days <= delta <= max_lag_days:
|
||||||
|
options.append(
|
||||||
|
(
|
||||||
|
scores[event_index + 1][period_index + 1]
|
||||||
|
+ _match_cost(events[event_index], periods[period_index]),
|
||||||
|
0,
|
||||||
|
"match",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
score, _, choice = min(options)
|
||||||
|
scores[event_index][period_index] = score
|
||||||
|
choices[event_index][period_index] = choice
|
||||||
|
|
||||||
|
matches: list[tuple[int, int]] = []
|
||||||
|
unmatched_events: list[int] = []
|
||||||
|
unmatched_periods: list[int] = []
|
||||||
|
event_index = 0
|
||||||
|
period_index = 0
|
||||||
|
while event_index < n_events or period_index < n_periods:
|
||||||
|
if event_index >= n_events:
|
||||||
|
unmatched_periods.extend(range(period_index, n_periods))
|
||||||
|
break
|
||||||
|
if period_index >= n_periods:
|
||||||
|
unmatched_events.extend(range(event_index, n_events))
|
||||||
|
break
|
||||||
|
choice = choices[event_index][period_index]
|
||||||
|
if choice == "match":
|
||||||
|
matches.append((event_index, period_index))
|
||||||
|
event_index += 1
|
||||||
|
period_index += 1
|
||||||
|
elif choice == "period":
|
||||||
|
unmatched_periods.append(period_index)
|
||||||
|
period_index += 1
|
||||||
|
else:
|
||||||
|
unmatched_events.append(event_index)
|
||||||
|
event_index += 1
|
||||||
|
return matches, unmatched_events, unmatched_periods
|
||||||
|
|
||||||
|
|
||||||
|
def _ensure_schema(connection: sqlite3.Connection) -> None:
|
||||||
|
connection.execute(EVENTS_DDL)
|
||||||
|
columns = {
|
||||||
|
str(row[1])
|
||||||
|
for row in connection.execute("PRAGMA table_info(earnings_events)")
|
||||||
|
}
|
||||||
|
if "period_end_date" not in columns:
|
||||||
|
connection.execute("ALTER TABLE earnings_events ADD COLUMN period_end_date TEXT")
|
||||||
|
connection.execute(META_DDL)
|
||||||
|
connection.execute(SURPRISE_HISTORY_DDL)
|
||||||
|
|
||||||
|
|
||||||
|
def _main() -> None:
|
||||||
|
args = _parse_args()
|
||||||
|
snapshot = Path(args.snapshot)
|
||||||
|
calendar_csv = Path(args.calendar_csv)
|
||||||
|
history_csv = Path(args.history_csv)
|
||||||
|
for path in (snapshot, calendar_csv, history_csv):
|
||||||
|
if not path.exists():
|
||||||
|
raise SystemExit(f"Missing input: {path}")
|
||||||
|
start = date.fromisoformat(args.from_date)
|
||||||
|
end = date.fromisoformat(args.to_date)
|
||||||
|
if start > end:
|
||||||
|
raise SystemExit("--from-date must not be after --to-date")
|
||||||
|
|
||||||
|
connection = sqlite3.connect(snapshot)
|
||||||
|
try:
|
||||||
|
universe = {
|
||||||
|
_normalise_symbol(row[0])
|
||||||
|
for row in connection.execute("SELECT symbol FROM tickers")
|
||||||
|
}
|
||||||
|
finally:
|
||||||
|
connection.close()
|
||||||
|
calendar, calendar_stats = _read_calendar(calendar_csv, universe, start, end)
|
||||||
|
history, history_stats = _read_history(history_csv, universe)
|
||||||
|
|
||||||
|
aligned_events: list[dict[str, Any]] = []
|
||||||
|
pairing_deltas: list[int] = []
|
||||||
|
unmatched_calendar = 0
|
||||||
|
unmatched_periods_in_pairing_window = 0
|
||||||
|
matched = 0
|
||||||
|
for symbol in sorted(universe):
|
||||||
|
events = calendar.get(symbol, [])
|
||||||
|
lower = start - timedelta(days=int(args.max_period_lag_days))
|
||||||
|
upper = end + timedelta(days=int(args.max_period_lead_days))
|
||||||
|
periods = [
|
||||||
|
row
|
||||||
|
for row in history.get(symbol, [])
|
||||||
|
if lower <= row["period_end_date"] <= upper
|
||||||
|
]
|
||||||
|
matches, unmatched_events, unmatched_periods = _align_symbol(
|
||||||
|
events,
|
||||||
|
periods,
|
||||||
|
max_lag_days=int(args.max_period_lag_days),
|
||||||
|
max_lead_days=int(args.max_period_lead_days),
|
||||||
|
)
|
||||||
|
matched_by_event = {event_index: period_index for event_index, period_index in matches}
|
||||||
|
matched += len(matches)
|
||||||
|
unmatched_calendar += len(unmatched_events)
|
||||||
|
unmatched_periods_in_pairing_window += len(unmatched_periods)
|
||||||
|
for event_index, event in enumerate(events):
|
||||||
|
row = dict(event)
|
||||||
|
period_index = matched_by_event.get(event_index)
|
||||||
|
if period_index is None:
|
||||||
|
row.update(
|
||||||
|
{
|
||||||
|
"period_end_date": None,
|
||||||
|
"eps_actual": None,
|
||||||
|
"eps_estimate": None,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
period = periods[period_index]
|
||||||
|
row.update(
|
||||||
|
{
|
||||||
|
"period_end_date": period["period_end_date"],
|
||||||
|
"eps_actual": period["eps_actual"],
|
||||||
|
"eps_estimate": period["eps_estimate"],
|
||||||
|
}
|
||||||
|
)
|
||||||
|
pairing_deltas.append(
|
||||||
|
(event["announce_date"] - period["period_end_date"]).days
|
||||||
|
)
|
||||||
|
aligned_events.append(row)
|
||||||
|
|
||||||
|
now = datetime.now(timezone.utc).isoformat()
|
||||||
|
source = f"dolthub_post_no_preference@{args.source_commit}"
|
||||||
|
conflicting_existing_rows = 0
|
||||||
|
conflicting_existing_fields = 0
|
||||||
|
preserved_existing_fields = 0
|
||||||
|
incoming_keys = {
|
||||||
|
(row["symbol"], row["announce_date"].isoformat()) for row in aligned_events
|
||||||
|
}
|
||||||
|
connection = sqlite3.connect(snapshot)
|
||||||
|
try:
|
||||||
|
_ensure_schema(connection)
|
||||||
|
existing = {
|
||||||
|
(str(row[0]), str(row[1])): row
|
||||||
|
for row in connection.execute(
|
||||||
|
"""
|
||||||
|
SELECT symbol, announce_date, announce_time, eps_estimate,
|
||||||
|
eps_actual, period_end_date, source
|
||||||
|
FROM earnings_events
|
||||||
|
WHERE announce_date BETWEEN ? AND ?
|
||||||
|
""",
|
||||||
|
(start.isoformat(), end.isoformat()),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
upsert = """
|
||||||
|
INSERT INTO earnings_events(
|
||||||
|
symbol, announce_date, announce_time, eps_estimate, eps_actual,
|
||||||
|
revenue_estimate, revenue_actual, source, fetched_at, period_end_date
|
||||||
|
) VALUES (?, ?, ?, ?, ?, NULL, NULL, ?, ?, ?)
|
||||||
|
ON CONFLICT(symbol, announce_date) DO UPDATE SET
|
||||||
|
announce_time=COALESCE(earnings_events.announce_time, excluded.announce_time),
|
||||||
|
eps_estimate=COALESCE(earnings_events.eps_estimate, excluded.eps_estimate),
|
||||||
|
eps_actual=COALESCE(earnings_events.eps_actual, excluded.eps_actual),
|
||||||
|
period_end_date=COALESCE(excluded.period_end_date, earnings_events.period_end_date),
|
||||||
|
source=excluded.source,
|
||||||
|
fetched_at=excluded.fetched_at
|
||||||
|
"""
|
||||||
|
for row in aligned_events:
|
||||||
|
key = (row["symbol"], row["announce_date"].isoformat())
|
||||||
|
old = existing.get(key)
|
||||||
|
retained = 0
|
||||||
|
conflicts = 0
|
||||||
|
if old is not None:
|
||||||
|
old_values = {
|
||||||
|
"announce_time": old[2],
|
||||||
|
"eps_estimate": old[3],
|
||||||
|
"eps_actual": old[4],
|
||||||
|
"period_end_date": old[5],
|
||||||
|
}
|
||||||
|
new_values = {
|
||||||
|
"announce_time": row.get("announce_time"),
|
||||||
|
"eps_estimate": row.get("eps_estimate"),
|
||||||
|
"eps_actual": row.get("eps_actual"),
|
||||||
|
"period_end_date": (
|
||||||
|
row["period_end_date"].isoformat()
|
||||||
|
if row.get("period_end_date")
|
||||||
|
else None
|
||||||
|
),
|
||||||
|
}
|
||||||
|
for field, new_value in new_values.items():
|
||||||
|
old_value = old_values[field]
|
||||||
|
if field != "period_end_date" and old_value is not None:
|
||||||
|
retained += 1
|
||||||
|
if new_value is not None and old_value is not None:
|
||||||
|
if field in {"eps_estimate", "eps_actual"}:
|
||||||
|
differs = not math.isclose(
|
||||||
|
float(new_value), float(old_value), rel_tol=0.0, abs_tol=1e-9
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
differs = str(new_value) != str(old_value)
|
||||||
|
conflicts += int(differs)
|
||||||
|
conflicting_existing_rows += int(conflicts > 0)
|
||||||
|
conflicting_existing_fields += conflicts
|
||||||
|
preserved_existing_fields += retained
|
||||||
|
row_source = source
|
||||||
|
if retained and old is not None:
|
||||||
|
row_source = f"{old[6]}+calendar:{source}"
|
||||||
|
connection.execute(
|
||||||
|
upsert,
|
||||||
|
(
|
||||||
|
row["symbol"],
|
||||||
|
row["announce_date"].isoformat(),
|
||||||
|
row.get("announce_time"),
|
||||||
|
row.get("eps_estimate"),
|
||||||
|
row.get("eps_actual"),
|
||||||
|
row_source,
|
||||||
|
now,
|
||||||
|
(
|
||||||
|
row["period_end_date"].isoformat()
|
||||||
|
if row.get("period_end_date")
|
||||||
|
else None
|
||||||
|
),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
history_upsert = """
|
||||||
|
INSERT INTO earnings_surprise_history(
|
||||||
|
symbol, period_end_date, eps_estimate, eps_actual, source, fetched_at
|
||||||
|
) VALUES (?, ?, ?, ?, ?, ?)
|
||||||
|
ON CONFLICT(symbol, period_end_date) DO UPDATE SET
|
||||||
|
eps_estimate=COALESCE(excluded.eps_estimate, earnings_surprise_history.eps_estimate),
|
||||||
|
eps_actual=COALESCE(excluded.eps_actual, earnings_surprise_history.eps_actual),
|
||||||
|
source=excluded.source,
|
||||||
|
fetched_at=excluded.fetched_at
|
||||||
|
"""
|
||||||
|
for symbol, rows in history.items():
|
||||||
|
connection.executemany(
|
||||||
|
history_upsert,
|
||||||
|
[
|
||||||
|
(
|
||||||
|
symbol,
|
||||||
|
row["period_end_date"].isoformat(),
|
||||||
|
row.get("eps_estimate"),
|
||||||
|
row.get("eps_actual"),
|
||||||
|
source,
|
||||||
|
now,
|
||||||
|
)
|
||||||
|
for row in rows
|
||||||
|
],
|
||||||
|
)
|
||||||
|
|
||||||
|
for symbol in sorted(universe):
|
||||||
|
count = int(
|
||||||
|
connection.execute(
|
||||||
|
"""
|
||||||
|
SELECT COUNT(*) FROM earnings_events
|
||||||
|
WHERE symbol=? AND announce_date BETWEEN ? AND ?
|
||||||
|
""",
|
||||||
|
(symbol, start.isoformat(), end.isoformat()),
|
||||||
|
).fetchone()[0]
|
||||||
|
)
|
||||||
|
connection.execute(
|
||||||
|
"""
|
||||||
|
INSERT INTO earnings_backfill_meta(symbol, status, n_events, updated_at, note)
|
||||||
|
VALUES (?, 'done', ?, ?, 'dolthub_bulk_complete')
|
||||||
|
ON CONFLICT(symbol) DO UPDATE SET
|
||||||
|
status='done', n_events=excluded.n_events,
|
||||||
|
updated_at=excluded.updated_at, note=excluded.note
|
||||||
|
""",
|
||||||
|
(symbol, count, now),
|
||||||
|
)
|
||||||
|
connection.commit()
|
||||||
|
|
||||||
|
params = (start.isoformat(), end.isoformat())
|
||||||
|
total_events = int(
|
||||||
|
connection.execute(
|
||||||
|
"""
|
||||||
|
SELECT COUNT(*) FROM earnings_events
|
||||||
|
WHERE symbol IN (SELECT symbol FROM tickers)
|
||||||
|
AND announce_date BETWEEN ? AND ?
|
||||||
|
""",
|
||||||
|
params,
|
||||||
|
).fetchone()[0]
|
||||||
|
)
|
||||||
|
paired_events = int(
|
||||||
|
connection.execute(
|
||||||
|
"""
|
||||||
|
SELECT COUNT(*) FROM earnings_events
|
||||||
|
WHERE symbol IN (SELECT symbol FROM tickers)
|
||||||
|
AND announce_date BETWEEN ? AND ?
|
||||||
|
AND eps_actual IS NOT NULL AND eps_estimate IS NOT NULL
|
||||||
|
""",
|
||||||
|
params,
|
||||||
|
).fetchone()[0]
|
||||||
|
)
|
||||||
|
date_range = connection.execute(
|
||||||
|
"""
|
||||||
|
SELECT MIN(announce_date), MAX(announce_date) FROM earnings_events
|
||||||
|
WHERE symbol IN (SELECT symbol FROM tickers)
|
||||||
|
AND announce_date BETWEEN ? AND ?
|
||||||
|
""",
|
||||||
|
params,
|
||||||
|
).fetchone()
|
||||||
|
source_symbols = set(calendar)
|
||||||
|
history_complete = int(
|
||||||
|
connection.execute(
|
||||||
|
"""
|
||||||
|
SELECT COUNT(*) FROM earnings_surprise_history
|
||||||
|
WHERE symbol IN (SELECT symbol FROM tickers)
|
||||||
|
AND eps_actual IS NOT NULL AND eps_estimate IS NOT NULL
|
||||||
|
"""
|
||||||
|
).fetchone()[0]
|
||||||
|
)
|
||||||
|
finally:
|
||||||
|
connection.close()
|
||||||
|
|
||||||
|
deltas = sorted(pairing_deltas)
|
||||||
|
summary = {
|
||||||
|
"mode": "dolthub_public_bulk_clone",
|
||||||
|
"window": {"from": start.isoformat(), "to": end.isoformat()},
|
||||||
|
"coverage_amendment": {
|
||||||
|
"approved_by_user": True,
|
||||||
|
"reason": "FMP free tier blocks historical bulk earnings",
|
||||||
|
"original_start": "2016-01-04",
|
||||||
|
"amended_announcement_start": start.isoformat(),
|
||||||
|
},
|
||||||
|
"source": {
|
||||||
|
"repository": args.source_url,
|
||||||
|
"commit": args.source_commit,
|
||||||
|
"license": "CC-BY-SA-4.0",
|
||||||
|
"upstream_provider_documented": False,
|
||||||
|
},
|
||||||
|
"bulk_windows_total": 1,
|
||||||
|
"bulk_windows_done": 1,
|
||||||
|
"bulk_requests_logged_total": 1,
|
||||||
|
"bulk_exports": 2,
|
||||||
|
"calendar": calendar_stats,
|
||||||
|
"eps_history": {**history_stats, "complete_actual_and_estimate": history_complete},
|
||||||
|
"pairing": {
|
||||||
|
"method": "minimum-cost monotonic alignment per symbol",
|
||||||
|
"allowed_announce_minus_period_end_days": [
|
||||||
|
-int(args.max_period_lead_days),
|
||||||
|
int(args.max_period_lag_days),
|
||||||
|
],
|
||||||
|
"matched_calendar_events": matched,
|
||||||
|
"unmatched_calendar_events": unmatched_calendar,
|
||||||
|
"unmatched_periods_in_pairing_window": unmatched_periods_in_pairing_window,
|
||||||
|
"announce_minus_period_end_days": {
|
||||||
|
"min": min(deltas) if deltas else None,
|
||||||
|
"median": deltas[len(deltas) // 2] if deltas else None,
|
||||||
|
"max": max(deltas) if deltas else None,
|
||||||
|
},
|
||||||
|
"pre_2020_eps_history_use": (
|
||||||
|
"trailing_surprise_stdev_only; never treated as an announcement "
|
||||||
|
"or live signal event"
|
||||||
|
),
|
||||||
|
},
|
||||||
|
"duplicate_rows_logged_total": (
|
||||||
|
calendar_stats["duplicate_rows"] + history_stats["duplicate_rows"]
|
||||||
|
),
|
||||||
|
"restated_rows_logged_total": (
|
||||||
|
calendar_stats["restated_rows"]
|
||||||
|
+ history_stats["restated_rows"]
|
||||||
|
+ conflicting_existing_rows
|
||||||
|
),
|
||||||
|
"conflicting_existing_rows": conflicting_existing_rows,
|
||||||
|
"conflicting_existing_fields": conflicting_existing_fields,
|
||||||
|
"preserved_existing_fields": preserved_existing_fields,
|
||||||
|
"existing_enrichment_events_not_in_dolthub_calendar": max(
|
||||||
|
0, total_events - len(incoming_keys)
|
||||||
|
),
|
||||||
|
"dedupe_policy": (
|
||||||
|
"UNIQUE(symbol, announce_date); normalise dot/dash symbols; retain one "
|
||||||
|
"calendar row per key; preserve existing non-null session/EPS values from "
|
||||||
|
"the prior FMP/Alpha Vantage partial backfill, then fill nulls and all "
|
||||||
|
"remaining symbols from DoltHub; attach DoltHub period-end alignment"
|
||||||
|
),
|
||||||
|
"events_in_window": total_events,
|
||||||
|
"events_with_actual_and_estimate": paired_events,
|
||||||
|
"symbols_done": len(universe),
|
||||||
|
"symbols_universe": len(universe),
|
||||||
|
"symbols_with_dolthub_calendar": len(source_symbols),
|
||||||
|
"symbols_without_dolthub_calendar": sorted(universe - source_symbols),
|
||||||
|
"announce_date_range": {"min": date_range[0], "max": date_range[1]},
|
||||||
|
"complete": True,
|
||||||
|
}
|
||||||
|
output = Path(args.status_output)
|
||||||
|
output.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
output.write_text(json.dumps(summary, indent=2) + "\n", encoding="utf-8")
|
||||||
|
print(json.dumps(summary, indent=2))
|
||||||
|
print(f"Wrote {output}")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
_main()
|
||||||
+1365
-721
File diff suppressed because it is too large
Load Diff
@@ -120,12 +120,14 @@ case "$PHASE" in
|
|||||||
ssl) ssl_check ;;
|
ssl) ssl_check ;;
|
||||||
earnings)
|
earnings)
|
||||||
need_file "$PROD_SNAP"
|
need_file "$PROD_SNAP"
|
||||||
log "Earnings backfill + research (parked experiment)"
|
need_file "$RESEARCH_SNAP"
|
||||||
|
log "Earnings Task 2 bulk backfill + registered 2a/2b closeout"
|
||||||
"$PYTHON" scripts/backfill_earnings_events.py \
|
"$PYTHON" scripts/backfill_earnings_events.py \
|
||||||
--snapshot "$PROD_SNAP" --provider fmp --force-symbol \
|
--snapshot "$PROD_SNAP" --from-date 2016-01-04 --window-days 30 \
|
||||||
--limit "$FMP_LIMIT" --sleep "$FMP_SLEEP"
|
--limit "$FMP_LIMIT" --sleep "$FMP_SLEEP"
|
||||||
"$PYTHON" scripts/run_earnings_research.py \
|
"$PYTHON" scripts/run_earnings_research.py \
|
||||||
--snapshot "$PROD_SNAP" --workers "$WORKERS" --allow-spawn
|
--snapshot "$RESEARCH_SNAP" --universe-snapshot "$PROD_SNAP" \
|
||||||
|
--earnings-snapshot "$PROD_SNAP" --workers "$WORKERS" --allow-spawn
|
||||||
;;
|
;;
|
||||||
prod_book)
|
prod_book)
|
||||||
need_file "$RESEARCH_SNAP"
|
need_file "$RESEARCH_SNAP"
|
||||||
|
|||||||
@@ -0,0 +1,225 @@
|
|||||||
|
from datetime import date, timedelta
|
||||||
|
|
||||||
|
from scripts.backfill_earnings_events import _dedupe_bulk_rows, _windows
|
||||||
|
from scripts.import_dolthub_earnings import _align_symbol
|
||||||
|
from scripts.run_earnings_research import (
|
||||||
|
_analyse_2a_trades,
|
||||||
|
_build_sue_series,
|
||||||
|
_mechanical_sue_grade,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_dolthub_alignment_is_monotonic_across_close_calendar_events() -> None:
|
||||||
|
events = [
|
||||||
|
{"announce_date": date(2020, 3, 17), "announce_time": "bmo"},
|
||||||
|
{"announce_date": date(2020, 4, 30), "announce_time": "bmo"},
|
||||||
|
]
|
||||||
|
periods = [
|
||||||
|
{"period_end_date": date(2019, 12, 31)},
|
||||||
|
{"period_end_date": date(2020, 3, 31)},
|
||||||
|
]
|
||||||
|
matches, unmatched_events, unmatched_periods = _align_symbol(
|
||||||
|
events, periods, max_lag_days=90, max_lead_days=14
|
||||||
|
)
|
||||||
|
assert matches == [(0, 0), (1, 1)]
|
||||||
|
assert unmatched_events == []
|
||||||
|
assert unmatched_periods == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_dolthub_alignment_allows_fiscal_period_label_after_announcement() -> None:
|
||||||
|
events = [
|
||||||
|
{"announce_date": date(2023, 2, 28), "announce_time": "bmo"},
|
||||||
|
{"announce_date": date(2023, 5, 23), "announce_time": "bmo"},
|
||||||
|
]
|
||||||
|
periods = [
|
||||||
|
{"period_end_date": date(2023, 2, 28)},
|
||||||
|
{"period_end_date": date(2023, 5, 31)},
|
||||||
|
]
|
||||||
|
matches, _, _ = _align_symbol(
|
||||||
|
events, periods, max_lag_days=90, max_lead_days=14
|
||||||
|
)
|
||||||
|
assert matches == [(0, 0), (1, 1)]
|
||||||
|
|
||||||
|
|
||||||
|
def test_bulk_windows_cover_range_without_overlap() -> None:
|
||||||
|
result = _windows(date(2020, 1, 1), date(2020, 1, 10), 4)
|
||||||
|
assert result == [
|
||||||
|
(date(2020, 1, 1), date(2020, 1, 4)),
|
||||||
|
(date(2020, 1, 5), date(2020, 1, 8)),
|
||||||
|
(date(2020, 1, 9), date(2020, 1, 10)),
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def test_bulk_dedupe_prefers_more_complete_and_counts_restatement() -> None:
|
||||||
|
rows = [
|
||||||
|
{
|
||||||
|
"symbol": "AAPL",
|
||||||
|
"announce_date": "2024-01-01",
|
||||||
|
"announce_time": None,
|
||||||
|
"eps_estimate": 1.0,
|
||||||
|
"eps_actual": 1.1,
|
||||||
|
"revenue_estimate": None,
|
||||||
|
"revenue_actual": None,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"symbol": "AAPL",
|
||||||
|
"announce_date": "2024-01-01",
|
||||||
|
"announce_time": "amc",
|
||||||
|
"eps_estimate": 1.0,
|
||||||
|
"eps_actual": 1.2,
|
||||||
|
"revenue_estimate": 10.0,
|
||||||
|
"revenue_actual": 11.0,
|
||||||
|
},
|
||||||
|
]
|
||||||
|
deduped, duplicates, restated = _dedupe_bulk_rows(rows)
|
||||||
|
assert duplicates == 1
|
||||||
|
assert restated == 1
|
||||||
|
assert deduped == [rows[1]]
|
||||||
|
|
||||||
|
|
||||||
|
def test_2a_uses_net_r_strict_hold_and_next_session_stop() -> None:
|
||||||
|
calendar = [
|
||||||
|
date(2024, 1, 2),
|
||||||
|
date(2024, 1, 3),
|
||||||
|
date(2024, 1, 4),
|
||||||
|
date(2024, 1, 5),
|
||||||
|
date(2024, 1, 8),
|
||||||
|
date(2024, 1, 9),
|
||||||
|
]
|
||||||
|
events = [
|
||||||
|
{
|
||||||
|
"symbol": "AAPL",
|
||||||
|
"announce_date": date(2024, 1, 5),
|
||||||
|
}
|
||||||
|
]
|
||||||
|
trades = [
|
||||||
|
{
|
||||||
|
"symbol": "AAPL",
|
||||||
|
"entry_date": "2024-01-03",
|
||||||
|
"exit_date": "2024-01-08",
|
||||||
|
"entry": 100.0,
|
||||||
|
"initial_stop": 90.0,
|
||||||
|
"fill": 90.0,
|
||||||
|
"r": -1.0,
|
||||||
|
"reason": "stop",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"symbol": "MSFT",
|
||||||
|
"entry_date": "2024-01-02",
|
||||||
|
"exit_date": "2024-01-09",
|
||||||
|
"entry": 100.0,
|
||||||
|
"initial_stop": 90.0,
|
||||||
|
"fill": 110.0,
|
||||||
|
"r": 1.0,
|
||||||
|
"reason": "time",
|
||||||
|
},
|
||||||
|
]
|
||||||
|
result = _analyse_2a_trades(
|
||||||
|
trades, events, calendar, cost_per_side=0.001
|
||||||
|
)
|
||||||
|
assert result["q1_loss_concentration"]["losses_count"] == 1
|
||||||
|
assert result["q1_loss_concentration"]["losses_with_announcement_count"] == 1
|
||||||
|
assert (
|
||||||
|
result["q2_entries_within_3_trading_days_before_announcement"][
|
||||||
|
"pre_earnings"
|
||||||
|
]["count"]
|
||||||
|
== 1
|
||||||
|
)
|
||||||
|
assert (
|
||||||
|
result["q3_stop_exits_within_1_trading_day_after_announcement"][
|
||||||
|
"stops_after_earnings"
|
||||||
|
]["count"]
|
||||||
|
== 1
|
||||||
|
)
|
||||||
|
assert result["q1_loss_concentration"]["loss_definition"] == (
|
||||||
|
"realized_net_R <= -1.0"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_sue_needs_four_prior_surprises_and_starts_next_trading_day() -> None:
|
||||||
|
dates = [date(2024, 1, 1) + timedelta(days=index) for index in range(100)]
|
||||||
|
columns = (
|
||||||
|
[value.toordinal() for value in dates],
|
||||||
|
[100.0] * len(dates),
|
||||||
|
[101.0] * len(dates),
|
||||||
|
[99.0] * len(dates),
|
||||||
|
[100.0] * len(dates),
|
||||||
|
[1_000_000] * len(dates),
|
||||||
|
)
|
||||||
|
event_dates = [date(2024, 1, 2) + timedelta(days=10 * index) for index in range(5)]
|
||||||
|
surprises = [0.1, -0.2, 0.3, -0.1, 0.4]
|
||||||
|
events = {
|
||||||
|
"AAPL": [
|
||||||
|
{
|
||||||
|
"announce_date": event_date,
|
||||||
|
"eps_actual": 1.0 + surprise,
|
||||||
|
"eps_estimate": 1.0,
|
||||||
|
}
|
||||||
|
for event_date, surprise in zip(event_dates, surprises)
|
||||||
|
]
|
||||||
|
}
|
||||||
|
series, counts = _build_sue_series(
|
||||||
|
events, {"AAPL": columns}, use_price_fallback=False
|
||||||
|
)
|
||||||
|
first_live = event_dates[-1] + timedelta(days=1)
|
||||||
|
assert first_live in series["AAPL"]
|
||||||
|
assert event_dates[-1] not in series["AAPL"]
|
||||||
|
assert counts["standard_scaled_events"] == 1
|
||||||
|
assert counts["price_fallback_events"] == 0
|
||||||
|
|
||||||
|
|
||||||
|
def test_sue_uses_period_history_only_for_scaling() -> None:
|
||||||
|
dates = [date(2020, 1, 1) + timedelta(days=index) for index in range(100)]
|
||||||
|
columns = (
|
||||||
|
[value.toordinal() for value in dates],
|
||||||
|
[100.0] * len(dates),
|
||||||
|
[101.0] * len(dates),
|
||||||
|
[99.0] * len(dates),
|
||||||
|
[100.0] * len(dates),
|
||||||
|
[1_000_000] * len(dates),
|
||||||
|
)
|
||||||
|
event_date = date(2020, 2, 3)
|
||||||
|
events = {
|
||||||
|
"AAPL": [
|
||||||
|
{
|
||||||
|
"announce_date": event_date,
|
||||||
|
"period_end_date": date(2019, 12, 31),
|
||||||
|
"eps_actual": 1.4,
|
||||||
|
"eps_estimate": 1.0,
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
history = {
|
||||||
|
"AAPL": [
|
||||||
|
{
|
||||||
|
"period_end_date": date(2018, 12, 31)
|
||||||
|
+ timedelta(days=90 * index),
|
||||||
|
"eps_actual": 1.0 + surprise,
|
||||||
|
"eps_estimate": 1.0,
|
||||||
|
}
|
||||||
|
for index, surprise in enumerate([0.1, -0.2, 0.3, -0.1])
|
||||||
|
]
|
||||||
|
}
|
||||||
|
series, counts = _build_sue_series(
|
||||||
|
events,
|
||||||
|
{"AAPL": columns},
|
||||||
|
use_price_fallback=False,
|
||||||
|
surprise_history_by_symbol=history,
|
||||||
|
)
|
||||||
|
assert event_date + timedelta(days=1) in series["AAPL"]
|
||||||
|
assert event_date not in series["AAPL"]
|
||||||
|
assert counts["events_scaled_from_period_history"] == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_sue_grade_requires_positive_both_eras() -> None:
|
||||||
|
full = {"mean_ic": 0.03, "reliable": True}
|
||||||
|
passed, stable = _mechanical_sue_grade(
|
||||||
|
full, {"mean_ic": 0.01}, {"mean_ic": 0.02}
|
||||||
|
)
|
||||||
|
assert passed is True
|
||||||
|
assert stable is True
|
||||||
|
failed, stable = _mechanical_sue_grade(
|
||||||
|
full, {"mean_ic": -0.01}, {"mean_ic": 0.02}
|
||||||
|
)
|
||||||
|
assert failed is False
|
||||||
|
assert stable is False
|
||||||
Reference in New Issue
Block a user