docs(dolt-plan): drop workstream B and record why
B would have replaced Alpaca historical OHLCV with the DoltHub stocks repo. It was scoped inside a plan whose goal was killing the quota-limited free-tier APIs — which A6 achieved without it, since Alpaca was never one of them. Its only concrete benefit was `corporate_actions` for the KLAC-class post-filing split (TTM EPS pre-split against a post-split price). That needs split events, not a 4.7 GB clone, and the Alpaca SDK already in the venv exposes them. Against that: fundamentals are 20% of the composite and P/E one of three inputs, so the wart is small and self-correcting at the next filing; and B would have made symbol history mutable as a routine event, which the backtest/prod parity guard exists to catch. The design is kept as a record, struck through rather than deleted, with the reasoning next to it so this isn't re-derived. Also corrects the doc's status header, the never-written migration 027 (that number went to weighted_avg_diluted_shares), and the note that the stocks repo's license was never reviewed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -1,15 +1,18 @@
|
||||
# Dolt bulk-data integration — implementation plan
|
||||
|
||||
Status: approved 2026-07-21, revised through four review rounds; direction: KISS
|
||||
backend, UI value first. Hand-off document for the implementing agent;
|
||||
self-contained.
|
||||
Status: **workstream A complete and deployed** (A0–A6, last step 2026-08-07);
|
||||
**workstream B dropped 2026-08-07** — see § Why B was dropped. Approved 2026-07-21,
|
||||
revised through five review rounds; direction: KISS backend, UI value first.
|
||||
Originally a hand-off document for the implementing agent; now the design record.
|
||||
Current operations live in `docs/fundamentals-deployment.md`.
|
||||
|
||||
## Objective
|
||||
|
||||
Replace the free-tier fundamentals APIs (FMP, Finnhub, Alpha Vantage) with bulk
|
||||
data: SEC Company Facts for fundamentals, the DoltHub earnings repo for the
|
||||
earnings calendar/history, and — later, independently — the DoltHub stocks repo for
|
||||
historical OHLCV. PostgreSQL stays the production system of record.
|
||||
data: SEC Company Facts for fundamentals and the DoltHub earnings repo for the
|
||||
earnings calendar/history. PostgreSQL stays the production system of record.
|
||||
(A third source — the DoltHub stocks repo for historical OHLCV — was planned as
|
||||
workstream B and dropped; Alpaca remains the price source.)
|
||||
|
||||
**Delivery order: two independent workstreams.**
|
||||
|
||||
@@ -17,9 +20,9 @@ historical OHLCV. PostgreSQL stays the production system of record.
|
||||
FundamentalsPanel + decommission FMP/Finnhub/Alpha Vantage. Valuation uses the
|
||||
existing Alpaca closes already in `ohlcv_records`. This alone achieves the goal
|
||||
(killing the quota-limited APIs) and delivers all the UI value.
|
||||
- **Workstream B (later, optional until needed):** replace historical OHLCV with
|
||||
the Dolt stocks repo. The most complex machinery (4.7 GB clone, split
|
||||
adjustment, source-bar table, reconciliation) lives here and blocks nothing in A.
|
||||
- **Workstream B — DROPPED 2026-08-07, see below.** Would have replaced historical
|
||||
OHLCV with the Dolt stocks repo. Its design is retained further down as a record,
|
||||
not as a backlog item.
|
||||
|
||||
**Guiding principle: KISS.** Plain daily importers with staging and atomic
|
||||
promotion — no forensic replay, no permanent archive store, no conflict tables, no
|
||||
@@ -70,8 +73,9 @@ notes (retain a CC BY-SA 4.0 reference + attribution to `post-no-preference/earn
|
||||
and a note of the transformations applied — e.g. in a repo `NOTICE`/attribution file
|
||||
and the importer module); **no public API, bulk export, or redistribution** of the
|
||||
data; re-review licensing before any public or commercial access. The
|
||||
`post-no-preference/stocks` repo (workstream B) is **not** covered here and will be
|
||||
reviewed separately if B begins.
|
||||
`post-no-preference/stocks` repo (workstream B) is **not** covered here. B was
|
||||
dropped before any licensing review, so that repo has never been assessed — any
|
||||
future use of it starts that review from scratch.
|
||||
|
||||
## Schema
|
||||
|
||||
@@ -119,7 +123,9 @@ reviewed separately if B begins.
|
||||
cache, repopulated by the daily SEC job — but only after the phase-A5 parity
|
||||
gate.
|
||||
|
||||
**Migration 027 (workstream B, written when B starts):**
|
||||
**Migration 027 (workstream B — NEVER WRITTEN; B was dropped, and `027` was
|
||||
subsequently used for `fundamental_snapshots.weighted_avg_diluted_shares`). The
|
||||
design below is a record only:**
|
||||
|
||||
- `ohlcv_source_bars` — source-truth bar table, required because `ohlcv_records`
|
||||
allows one row per (ticker_id, date) (`app/models/ohlcv.py:12`) and Alpaca
|
||||
@@ -219,7 +225,7 @@ Workstream A:
|
||||
**The new API valuation object is not stored anywhere** — it is computed at
|
||||
request time (below). No valuation cache or table exists.
|
||||
|
||||
Workstream B:
|
||||
Workstream B (dropped — never built):
|
||||
|
||||
- Dolt OHLCV+splits pull/import: `0 2 * * tue-sat` ET. If source_max_date is not
|
||||
fresh, retry hourly until ~06:00, then give up quietly. After a successful
|
||||
@@ -437,14 +443,48 @@ workstream B — Alpaca remains the price source throughout.
|
||||
once the deploy was confirmed healthy; the archived parity bundles stay as the
|
||||
A5 evidence trail.
|
||||
|
||||
**Workstream B (independent, start when wanted):**
|
||||
**Workstream B — DROPPED 2026-08-07.** The phases below are recorded for anyone
|
||||
who revisits the decision; none of them are scheduled work.
|
||||
|
||||
- B0. Stocks clone (~4.7 GB) provisioned; migration 027.
|
||||
- B1. OHLCV + split adjustment in shadow (writes `ohlcv_source_bars` only; Alpaca
|
||||
keeps owning `ohlcv_records`); historical backfill.
|
||||
- B2. Reconciliation window (≥ 2 weeks) vs Alpaca; review validation summaries.
|
||||
- B3. Promote Dolt as historical OHLCV source (canonical rebuilt from raw source
|
||||
bars + splits); morning pipeline → 03:00.
|
||||
- ~~B0. Stocks clone (~4.7 GB) provisioned; migration 027.~~
|
||||
- ~~B1. OHLCV + split adjustment in shadow (writes `ohlcv_source_bars` only; Alpaca
|
||||
keeps owning `ohlcv_records`); historical backfill.~~
|
||||
- ~~B2. Reconciliation window (≥ 2 weeks) vs Alpaca; review validation summaries.~~
|
||||
- ~~B3. Promote Dolt as historical OHLCV source (canonical rebuilt from raw source
|
||||
bars + splits); morning pipeline → 03:00.~~
|
||||
|
||||
### Why B was dropped
|
||||
|
||||
Reviewed after A6 shipped. Four reasons, in order of weight:
|
||||
|
||||
1. **Its motivation no longer exists.** B was scoped inside a plan whose goal was
|
||||
killing the quota-limited free-tier APIs. Alpaca was never one of them, and the
|
||||
plan always said so (§ Decommissioning: "Alpaca remains the price source
|
||||
throughout"). A6 achieved the goal. What remained was swapping one working
|
||||
price source for another.
|
||||
2. **Its only concrete benefit is reachable far more cheaply.** The prize was
|
||||
`corporate_actions`, the documented fix for the KLAC-class post-filing split
|
||||
(TTM EPS pre-split vs a post-split price → P/E 6.19 instead of ~13, invisible to
|
||||
snapshots). That needs *split events*, not 4.7 GB of bars — and the Alpaca SDK
|
||||
already in the venv exposes them via
|
||||
`alpaca.data.historical.corporate_actions.CorporateActionsClient.get_corporate_actions`
|
||||
with `CorporateActionsRequest` / `CorporateActionsType`. See the follow-up below.
|
||||
3. **The benefit is small.** Fundamentals carry 20% of the composite, P/E is one of
|
||||
three fundamental inputs, and only names that split between their last 10-Q and
|
||||
today are affected — a handful at a time, self-correcting at the next filing.
|
||||
4. **B would add a risk the current setup does not carry.** By design a newly
|
||||
published split rewrites a symbol's entire adjusted history. A backtest↔prod
|
||||
parity guard exists precisely because changed history invalidates comparisons;
|
||||
B makes history mutable as a routine event. It also needs its own license
|
||||
review — the A0 CC BY-SA decision covers only `post-no-preference/earnings`.
|
||||
|
||||
**Optional follow-up, not scheduled:** a small `corporate_actions` table populated
|
||||
from Alpaca, used to null or correct P/E when a split post-dates the newest
|
||||
snapshot. Roughly a day's work; captures essentially all of B's value with no
|
||||
clone, no `ohlcv_source_bars`, no split-adjustment pipeline and no reconciliation
|
||||
window. Worth doing only if the wart starts costing something — it has been visible
|
||||
and harmless since July 2026. Note that migration numbering has moved on: head is
|
||||
`030`, so any such table would be `031+`, not the `027` named below.
|
||||
|
||||
## Test plan
|
||||
|
||||
@@ -469,8 +509,8 @@ workstream B — Alpaca remains the price source throughout.
|
||||
falls back from P/E to FCF yield for the valuation segment when P/E is null.
|
||||
- Peer comparison disappears below 5 peer issuers; favorable-percentile direction
|
||||
correct for both polarities.
|
||||
- Workstream B: split-adjusted OHLCV matches Alpaca on representative normal /
|
||||
split / reverse-split symbols.
|
||||
- ~~Workstream B: split-adjusted OHLCV matches Alpaca on representative normal /
|
||||
split / reverse-split symbols.~~ (dropped)
|
||||
- UI states: positive, adverse, neutral, insufficient history, insufficient
|
||||
peers; mobile layout; non-color accessibility.
|
||||
- Unit, integration, scheduler and frontend suites pass.
|
||||
@@ -520,7 +560,9 @@ remain: migration `030` dropped the tombstone rows after the deploy was verified
|
||||
|
||||
**Known caveats to carry (documented in the findings report, not bugs to fix):**
|
||||
- KLAC-class post-filing splits: P/E wrong until the next 10-Q; undetectable from
|
||||
snapshots. Workstream B's `corporate_actions` table is the natural future fix.
|
||||
snapshots. Still open and still harmless. The fix, if ever wanted, is a small
|
||||
`corporate_actions` table fed from Alpaca — **not** workstream B, which was
|
||||
dropped; see § Why B was dropped.
|
||||
- BRK-B: no share count exists anywhere in companyfacts → no market cap, correctly.
|
||||
- FITB: unscored (split guard + no taggable revenue) — the one name that lost its
|
||||
score relative to legacy; composite renormalises.
|
||||
@@ -533,7 +575,7 @@ remain: migration `030` dropped the tombstone rows after the deploy was verified
|
||||
|
||||
## Deferred (explicitly, until a concrete need appears)
|
||||
|
||||
- Workstream B itself is deferred relative to A and blocks nothing in A.
|
||||
- Workstream B: **dropped** 2026-08-07, not deferred — see § Why B was dropped.
|
||||
- Exact byte-level source replay of historical imports; permanent archive store.
|
||||
- Point-in-time backtest enforcement (`accepted_at` is stored now; derivation and
|
||||
backtest visibility rules are built only when fundamentals enter
|
||||
|
||||
Reference in New Issue
Block a user