From b1397fa82efc554f67d3538cd105a6cfe735e55e Mon Sep 17 00:00:00 2001 From: Dennis Thiessen Date: Wed, 22 Jul 2026 14:04:12 +0200 Subject: [PATCH] =?UTF-8?q?docs:=20A3=20design=20=E2=80=94=20four=20review?= =?UTF-8?q?=20correctness=20fixes?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fold in the A3 design review: 1. Composite revision = latest-index-date + index-content-hash + tracked symbol->CIK fingerprint, so a newly added ticker forces a run instead of being no_op'd/starved. No backfill sentinel — absence of a prior promoted run triggers backfill; source_max_date records the processed index date. 2. Full history needs the paginated submissions shards: filings.recent caps at 1000; older accessions (reportDate/acceptanceDateTime/isXBRL) live in filings.files[] shards (verified on Apple: recent=1000, one 1994-2015 shard). 3. Index<->Company-Facts consistency gate: they are separate SEC products that can lag; for every tracked isXBRL index accession, confirm it exists in Company Facts before promotion, else fail+retry (never record a null/partial snapshot). Non-XBRL amendments skipped with a recorded reason. 4. Immutable = insert-only (ON CONFLICT DO NOTHING); a differing re-fetch is a reported discrepancy, never a silent mutation / import_run_id replacement. Plus deterministic, mutually-exclusive cash/debt composition (aggregate-first; each source tag counted at most once). Co-Authored-By: Claude Opus 4.8 --- docs/dolt-sec-a3-design.md | 80 +++++++++++++++++++++++++++++++------- 1 file changed, 67 insertions(+), 13 deletions(-) diff --git a/docs/dolt-sec-a3-design.md b/docs/dolt-sec-a3-design.md index b6a4991..6a4ea5b 100644 --- a/docs/dolt-sec-a3-design.md +++ b/docs/dolt-sec-a3-design.md @@ -1,7 +1,11 @@ # A3 design — SEC fundamentals importer -Status: **design APPROVED 2026-07-22 — all three decisions signed off (daily-index -fetch, primary-period-only snapshots, full-history backfill). Ready to implement.** +Status: **design APPROVED 2026-07-22 — three decisions signed off (daily-index +fetch, primary-period-only snapshots, full-history backfill) + four review +correctness fixes folded in (composite revision incl. universe fingerprint; +submissions pagination shards for full history; index↔Company-Facts consistency +gate; insert-only immutability with discrepancy reporting; deterministic cash/debt +composition). Ready to implement.** Companion to `docs/dolt-integration-plan.md` (workstream A, phase A3). Grounded in live SEC data probes (Apple CIK 0000320193, company_tickers, submissions, daily-index). @@ -58,15 +62,23 @@ endpoints expose no validators, and the bulk zip is multi-GB and changes ~daily form.YYYYMMDD.idx` — fixed-width Form/Company/CIK/Date/accession, ~3300 rows/day, confirmed). Each run: -- `detect_revision` → the latest available daily-index date. If it equals the - last processed date, **`no_op`** (framework's model works cleanly again). +- `detect_revision` → a **composite revision**, not just the date: + `latest-index-date` + a hash of the index content processed this run + a + **fingerprint of the tracked symbol→CIK set**. The CIK fingerprint is essential: + a newly added ticker changes the revision and forces a run, so a new ticker is + never `no_op`'d away or starved waiting for its issuer to file. Equal composite + revision → `no_op`. +- **No backfill sentinel.** The *absence of a prior promoted run* is what triggers + the initial full-history backfill; `source_max_date` records the processed index + date each run. - `stage` → for each index date since the last processed one, parse the form index, keep rows where `form ∈ {10-K, 10-Q, 10-K/A, 10-Q/A}` **and** CIK ∈ tracked set, then fetch `companyfacts/CIK.json` for **only those few issuers** and extract their newly-reported period(s). Most nights this is a handful of issuers → near-zero transfer, respectful of SEC fair-access. -- **First run (backfill)** has no last-processed date: fetch companyfacts for all - tracked CIKs once (~1 GB one-time) to seed history, then go incremental. +- **First run (backfill)**: no prior promoted run → fetch companyfacts for all + tracked CIKs once (~1 GB one-time) and seed **full** history. Full history needs + the paginated submissions shards — see "CIK resolution" below. Why this over the alternatives: transfer scales with *filings*, not with all of EDGAR or with the universe size every night; it restores the revision/no_op @@ -97,6 +109,12 @@ One `fundamental_snapshots` row per accession, representing the filing's comparatives inside a later normal filing are **not** captured — only a real amendment updates a prior period. This narrows the plan's "newest accepted_at per period" to amendment-driven updates; a deliberate KISS boundary. +- **Immutable means insert-only, not upsert.** `promote` **inserts** new accession + rows with `ON CONFLICT (accession) DO NOTHING`. An accession never mutates: if a + re-fetch reconstructs *different* values for an accession already stored, that is + a **discrepancy to report** (into `validation_json` + a system event), never a + silent overwrite, and the original `import_run_id` is never replaced. (Ordinary + updates arrive as a *new* amendment accession, which is a new row.) ## Read-time derivation (constrains the importer; built in A4) @@ -125,10 +143,25 @@ resolves through an ordered tag list; first present wins; unit-checked. | cfo | `NetCashProvidedByUsedInOperatingActivities` | `...ContinuingOperations` | USD | | capex | `PaymentsToAcquirePropertyPlantAndEquipment` | `PaymentsToAcquireProductiveAssets` | USD | | depreciation_amortization | `DepreciationDepletionAndAmortization` | `DepreciationAmortizationAndAccretionNet`, `DepreciationAndAmortization` | USD | -| cash_and_st_investments | `CashAndCashEquivalentsAtCarryingValue` (+ ST inv) | ST inv: `ShortTermInvestments`, `MarketableSecuritiesCurrent` | USD | -| total_debt | `LongTermDebtNoncurrent` + `LongTermDebtCurrent` | `LongTermDebt`; +`CommercialPaper`/`ShortTermBorrowings` if present | USD | +| cash_and_st_investments | see composition rule | — | USD | +| total_debt | see composition rule | — | USD | | shares_outstanding | `dei:EntityCommonStockSharesOutstanding` | — | shares | +**Composite fields — deterministic, aggregate-first, no double counting.** Each +source tag contributes at most once: + +- `cash_and_st_investments` = `CashAndCashEquivalentsAtCarryingValue` + **+ short-term investments**, where ST investments = the **first present** of + [`ShortTermInvestments`, `MarketableSecuritiesCurrent`] — never both summed. +- `total_debt` = **long-term component + short-term component**, where + - long-term = first present of [`LongTermDebt` (the aggregate, already includes + current + noncurrent portions), **else** (`LongTermDebtNoncurrent` + + `LongTermDebtCurrent`)]; + - short-term borrowings = first present of [`ShortTermBorrowings`, + `CommercialPaper`] (0 if neither). + So the long-term aggregate and its components are mutually exclusive, and CP vs + short-term-borrowings is a single pick — nothing is counted twice. + EBITDA (for net-debt/EBITDA) is derived at read time = operating_income + D&A. Concepts absent for an issuer → that field is null (display-only; no synthesis). The exact tag lists live as named constants, tunable without touching logic. @@ -145,9 +178,19 @@ primary period (safe — a filing's own context is correct for its current perio - From `company_tickers.json`: `normalise_symbol(ticker) → cik_str`. Set `tickers.cik` for each tracked ticker (multi-class share one CIK). -- From `submissions/CIK.json`: `sic`, `sicDescription` → `tickers.sic/sic_description`. +- From `submissions/CIK.json`: `sic`, `sicDescription`, `fiscalYearEnd` → + `tickers.sic/sic_description` (+ fiscal anchor for YTD/Q4). +- **Submissions is paginated — full history needs the shards.** `filings.recent` + holds only the **latest 1000** filings (verified: Apple `recent` = 1000). Older + accessions live in `filings.files[]` = `[{name, filingFrom, filingTo, + filingCount}]` (e.g. `CIK0000320193-submissions-001.json`, 1236 filings + 1994–2015), each a bare object with the **same parallel arrays** including + `reportDate`, `acceptanceDateTime`, and `isXBRL`. The full-history backfill must + **follow every `filings.files[].name`** to obtain period identity + `accepted_at` + + `isXBRL` for pre-1000 accessions. Incremental runs only need `recent`. - Refreshed by the SEC job; a newly added ticker self-resolves on its next run - (until then its snapshots are absent → metrics null, per the plan). + (the CIK fingerprint in the revision forces that run) — until then its snapshots + are absent → metrics null, per the plan. ## SourceImporter mapping (source = `sec_facts`) @@ -160,9 +203,20 @@ primary period (safe — a filing's own context is correct for its current perio **no duplicate accession**; **filings skipped for missing period identity are counted in `validation_json`** (carry-forward from A1 review); an unexpected companyfacts shape (missing `facts`/`units`) stops promotion. -- `promote` → upsert snapshot rows keyed by unique `accession`, stamped - `import_run_id`; refresh `tickers.cik/sic/sic_description`. Non-destructive - (append-only accessions) — no future-row deletion like earnings. +- **Index↔Company-Facts consistency gate (the daily index and Company Facts are + separate SEC products that can lag each other):** for every tracked index + accession marked `isXBRL`, confirm that accession actually appears in the fetched + companyfacts before promotion. If any is missing → **fail the run and retry + later** — do **not** advance the revision and do **not** record an + incomplete/null snapshot for it. Non-XBRL amendments are skipped with a recorded + reason in `validation_json`. (The framework only stores the revision on a + promoted run, so a failed consistency check naturally leaves the revision behind + for retry.) +- `promote` → **insert** snapshot rows (`ON CONFLICT (accession) DO NOTHING`; + immutable — see Decision 2), stamped `import_run_id`; refresh + `tickers.cik/sic/sic_description`. A re-fetch that reconstructs different values + for an existing accession is reported as a discrepancy, never a silent mutation. + Non-destructive (append-only accessions) — no future-row deletion like earnings. ## SEC fair-access (operational, per the plan's non-negotiable)