From 23de8c95405c05d20be1a94b0d22d5b48a6f8cd3 Mon Sep 17 00:00:00 2001 From: Dennis Thiessen Date: Fri, 7 Aug 2026 13:03:32 +0200 Subject: [PATCH] docs(dolt-plan): drop workstream B and record why MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit B would have replaced Alpaca historical OHLCV with the DoltHub stocks repo. It was scoped inside a plan whose goal was killing the quota-limited free-tier APIs — which A6 achieved without it, since Alpaca was never one of them. Its only concrete benefit was `corporate_actions` for the KLAC-class post-filing split (TTM EPS pre-split against a post-split price). That needs split events, not a 4.7 GB clone, and the Alpaca SDK already in the venv exposes them. Against that: fundamentals are 20% of the composite and P/E one of three inputs, so the wart is small and self-correcting at the next filing; and B would have made symbol history mutable as a routine event, which the backtest/prod parity guard exists to catch. The design is kept as a record, struck through rather than deleted, with the reasoning next to it so this isn't re-derived. Also corrects the doc's status header, the never-written migration 027 (that number went to weighted_avg_diluted_shares), and the note that the stocks repo's license was never reviewed. Co-Authored-By: Claude Opus 5 --- docs/dolt-integration-plan.md | 90 +++++++++++++++++++++++++---------- 1 file changed, 66 insertions(+), 24 deletions(-) diff --git a/docs/dolt-integration-plan.md b/docs/dolt-integration-plan.md index 6d8e859..8e8a594 100644 --- a/docs/dolt-integration-plan.md +++ b/docs/dolt-integration-plan.md @@ -1,15 +1,18 @@ # Dolt bulk-data integration — implementation plan -Status: approved 2026-07-21, revised through four review rounds; direction: KISS -backend, UI value first. Hand-off document for the implementing agent; -self-contained. +Status: **workstream A complete and deployed** (A0–A6, last step 2026-08-07); +**workstream B dropped 2026-08-07** — see § Why B was dropped. Approved 2026-07-21, +revised through five review rounds; direction: KISS backend, UI value first. +Originally a hand-off document for the implementing agent; now the design record. +Current operations live in `docs/fundamentals-deployment.md`. ## Objective Replace the free-tier fundamentals APIs (FMP, Finnhub, Alpha Vantage) with bulk -data: SEC Company Facts for fundamentals, the DoltHub earnings repo for the -earnings calendar/history, and — later, independently — the DoltHub stocks repo for -historical OHLCV. PostgreSQL stays the production system of record. +data: SEC Company Facts for fundamentals and the DoltHub earnings repo for the +earnings calendar/history. PostgreSQL stays the production system of record. +(A third source — the DoltHub stocks repo for historical OHLCV — was planned as +workstream B and dropped; Alpaca remains the price source.) **Delivery order: two independent workstreams.** @@ -17,9 +20,9 @@ historical OHLCV. PostgreSQL stays the production system of record. FundamentalsPanel + decommission FMP/Finnhub/Alpha Vantage. Valuation uses the existing Alpaca closes already in `ohlcv_records`. This alone achieves the goal (killing the quota-limited APIs) and delivers all the UI value. -- **Workstream B (later, optional until needed):** replace historical OHLCV with - the Dolt stocks repo. The most complex machinery (4.7 GB clone, split - adjustment, source-bar table, reconciliation) lives here and blocks nothing in A. +- **Workstream B — DROPPED 2026-08-07, see below.** Would have replaced historical + OHLCV with the Dolt stocks repo. Its design is retained further down as a record, + not as a backlog item. **Guiding principle: KISS.** Plain daily importers with staging and atomic promotion — no forensic replay, no permanent archive store, no conflict tables, no @@ -70,8 +73,9 @@ notes (retain a CC BY-SA 4.0 reference + attribution to `post-no-preference/earn and a note of the transformations applied — e.g. in a repo `NOTICE`/attribution file and the importer module); **no public API, bulk export, or redistribution** of the data; re-review licensing before any public or commercial access. The -`post-no-preference/stocks` repo (workstream B) is **not** covered here and will be -reviewed separately if B begins. +`post-no-preference/stocks` repo (workstream B) is **not** covered here. B was +dropped before any licensing review, so that repo has never been assessed — any +future use of it starts that review from scratch. ## Schema @@ -119,7 +123,9 @@ reviewed separately if B begins. cache, repopulated by the daily SEC job — but only after the phase-A5 parity gate. -**Migration 027 (workstream B, written when B starts):** +**Migration 027 (workstream B — NEVER WRITTEN; B was dropped, and `027` was +subsequently used for `fundamental_snapshots.weighted_avg_diluted_shares`). The +design below is a record only:** - `ohlcv_source_bars` — source-truth bar table, required because `ohlcv_records` allows one row per (ticker_id, date) (`app/models/ohlcv.py:12`) and Alpaca @@ -219,7 +225,7 @@ Workstream A: **The new API valuation object is not stored anywhere** — it is computed at request time (below). No valuation cache or table exists. -Workstream B: +Workstream B (dropped — never built): - Dolt OHLCV+splits pull/import: `0 2 * * tue-sat` ET. If source_max_date is not fresh, retry hourly until ~06:00, then give up quietly. After a successful @@ -437,14 +443,48 @@ workstream B — Alpaca remains the price source throughout. once the deploy was confirmed healthy; the archived parity bundles stay as the A5 evidence trail. -**Workstream B (independent, start when wanted):** +**Workstream B — DROPPED 2026-08-07.** The phases below are recorded for anyone +who revisits the decision; none of them are scheduled work. -- B0. Stocks clone (~4.7 GB) provisioned; migration 027. -- B1. OHLCV + split adjustment in shadow (writes `ohlcv_source_bars` only; Alpaca - keeps owning `ohlcv_records`); historical backfill. -- B2. Reconciliation window (≥ 2 weeks) vs Alpaca; review validation summaries. -- B3. Promote Dolt as historical OHLCV source (canonical rebuilt from raw source - bars + splits); morning pipeline → 03:00. +- ~~B0. Stocks clone (~4.7 GB) provisioned; migration 027.~~ +- ~~B1. OHLCV + split adjustment in shadow (writes `ohlcv_source_bars` only; Alpaca + keeps owning `ohlcv_records`); historical backfill.~~ +- ~~B2. Reconciliation window (≥ 2 weeks) vs Alpaca; review validation summaries.~~ +- ~~B3. Promote Dolt as historical OHLCV source (canonical rebuilt from raw source + bars + splits); morning pipeline → 03:00.~~ + +### Why B was dropped + +Reviewed after A6 shipped. Four reasons, in order of weight: + +1. **Its motivation no longer exists.** B was scoped inside a plan whose goal was + killing the quota-limited free-tier APIs. Alpaca was never one of them, and the + plan always said so (§ Decommissioning: "Alpaca remains the price source + throughout"). A6 achieved the goal. What remained was swapping one working + price source for another. +2. **Its only concrete benefit is reachable far more cheaply.** The prize was + `corporate_actions`, the documented fix for the KLAC-class post-filing split + (TTM EPS pre-split vs a post-split price → P/E 6.19 instead of ~13, invisible to + snapshots). That needs *split events*, not 4.7 GB of bars — and the Alpaca SDK + already in the venv exposes them via + `alpaca.data.historical.corporate_actions.CorporateActionsClient.get_corporate_actions` + with `CorporateActionsRequest` / `CorporateActionsType`. See the follow-up below. +3. **The benefit is small.** Fundamentals carry 20% of the composite, P/E is one of + three fundamental inputs, and only names that split between their last 10-Q and + today are affected — a handful at a time, self-correcting at the next filing. +4. **B would add a risk the current setup does not carry.** By design a newly + published split rewrites a symbol's entire adjusted history. A backtest↔prod + parity guard exists precisely because changed history invalidates comparisons; + B makes history mutable as a routine event. It also needs its own license + review — the A0 CC BY-SA decision covers only `post-no-preference/earnings`. + +**Optional follow-up, not scheduled:** a small `corporate_actions` table populated +from Alpaca, used to null or correct P/E when a split post-dates the newest +snapshot. Roughly a day's work; captures essentially all of B's value with no +clone, no `ohlcv_source_bars`, no split-adjustment pipeline and no reconciliation +window. Worth doing only if the wart starts costing something — it has been visible +and harmless since July 2026. Note that migration numbering has moved on: head is +`030`, so any such table would be `031+`, not the `027` named below. ## Test plan @@ -469,8 +509,8 @@ workstream B — Alpaca remains the price source throughout. falls back from P/E to FCF yield for the valuation segment when P/E is null. - Peer comparison disappears below 5 peer issuers; favorable-percentile direction correct for both polarities. -- Workstream B: split-adjusted OHLCV matches Alpaca on representative normal / - split / reverse-split symbols. +- ~~Workstream B: split-adjusted OHLCV matches Alpaca on representative normal / + split / reverse-split symbols.~~ (dropped) - UI states: positive, adverse, neutral, insufficient history, insufficient peers; mobile layout; non-color accessibility. - Unit, integration, scheduler and frontend suites pass. @@ -520,7 +560,9 @@ remain: migration `030` dropped the tombstone rows after the deploy was verified **Known caveats to carry (documented in the findings report, not bugs to fix):** - KLAC-class post-filing splits: P/E wrong until the next 10-Q; undetectable from - snapshots. Workstream B's `corporate_actions` table is the natural future fix. + snapshots. Still open and still harmless. The fix, if ever wanted, is a small + `corporate_actions` table fed from Alpaca — **not** workstream B, which was + dropped; see § Why B was dropped. - BRK-B: no share count exists anywhere in companyfacts → no market cap, correctly. - FITB: unscored (split guard + no taggable revenue) — the one name that lost its score relative to legacy; composite renormalises. @@ -533,7 +575,7 @@ remain: migration `030` dropped the tombstone rows after the deploy was verified ## Deferred (explicitly, until a concrete need appears) -- Workstream B itself is deferred relative to A and blocks nothing in A. +- Workstream B: **dropped** 2026-08-07, not deferred — see § Why B was dropped. - Exact byte-level source replay of historical imports; permanent archive store. - Point-in-time backtest enforcement (`accepted_at` is stored now; derivation and backtest visibility rules are built only when fundamentals enter