dennisthiessen/signal-platform
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: A3 design signed off — daily-index, primary-period, full backfill

Browse Source 
All three decisions approved: fetch via EDGAR daily-index (not bulk zip),
one snapshot per accession for its primary period (comparative-only
restatements out of scope), full-history backfill on first run. Doc status
flipped to approved / ready to implement.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
dennisthiessen
2026-07-22 12:33:56 +02:00
co-authored by Claude Opus 4.8
parent 18fca28b7c
commit d923f41f85
1 changed files with 13 additions and 12 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/dolt-sec-a3-design.md
+13 -12
View File
@@ -1,9 +1,9 @@
# A3 design — SEC fundamentals importer
Status: **design pass, 2026-07-22 — awaiting sign-off on two plan deviations
(fetch strategy, snapshot mapping).** Not yet implemented. Companion to
`docs/dolt-integration-plan.md` (workstream A, phase A3). Grounded in live SEC
data probes (Apple CIK 0000320193, company_tickers, submissions, daily-index).
Status: **design APPROVED 2026-07-22 — all three decisions signed off (daily-index
fetch, primary-period-only snapshots, full-history backfill). Ready to implement.**
Companion to `docs/dolt-integration-plan.md` (workstream A, phase A3). Grounded in
live SEC data probes (Apple CIK 0000320193, company_tickers, submissions, daily-index).
## Objective (unchanged from the plan)
@@ -46,7 +46,7 @@ Ground-truth findings that drive the design:
`GOOGL`/`GOOG` to the **same** `cik_str` (1652044). The ticker→CIK join reuses
the earnings importer's `normalise_symbol` (dot→dash), so both sides match.
## Decision 1 (needs sign-off) — fetch strategy: EDGAR daily-index driven
## Decision 1 (APPROVED) — fetch strategy: EDGAR daily-index driven
**Plan said** bulk `companyfacts.zip` + ETag no-op. **Reality:** the data.sec.gov
endpoints expose no validators, and the bulk zip is multi-GB and changes ~daily
@@ -74,7 +74,7 @@ model; and it's the lightest load on SEC. Cost: daily-index parsing + date
bookkeeping (store last-processed index date in `data_import_runs` /
settings). **This deviates from the plan's "bulk zip" — requesting sign-off.**
## Decision 2 (needs sign-off) — snapshot mapping: primary-period, YTD, immutable
## Decision 2 (APPROVED) — snapshot mapping: primary-period, YTD, immutable
One `fundamental_snapshots` row per accession, representing the filing's
**primary current period only** (not its comparatives):
@@ -179,10 +179,11 @@ retry-loop**. New config: `sec_user_agent`, `sec_request_spacing_seconds`,
- Comparative-only restatements (Decision 2).
- Point-in-time backtest enforcement (`accepted_at` stored, not yet enforced).
## Open questions for sign-off
## Decisions (signed off 2026-07-22)
1. **Decision 1** (daily-index fetch vs the plan's bulk zip) — approve the deviation?
2. **Decision 2** (primary-period-only; comparative restatements out of scope) — approve?
3. Backfill depth: seed **full** available history per issuer on first run, or cap
at N years (e.g. 5, matching `ohlcv_history_days`)? Full history is cheap to
store and powers the quarter tape / YoY.
1. **Fetch** — EDGAR daily-index driven (Decision 1). Approved deviation from the
plan's bulk zip.
2. **Snapshot mapping** — primary-period-only per accession; comparative-only
restatements out of scope (Decision 2). Approved.
3. **Backfill depth** — seed **full** available history per issuer on first run
(cheap to store; powers the quarter tape / multi-year YoY). Approved.