signal-platform/docs/dolt-sec-a3-design.md

A3 design — SEC fundamentals importer

Status: design APPROVED 2026-07-22 — three decisions signed off (daily-index fetch, primary-period-only snapshots, full-history backfill) + four review correctness fixes folded in (composite revision incl. universe fingerprint; submissions pagination shards for full history; index↔Company-Facts consistency gate; insert-only immutability with discrepancy reporting; deterministic cash/debt composition). Ready to implement. Companion to docs/dolt-integration-plan.md (workstream A, phase A3). Grounded in live SEC data probes (Apple CIK 0000320193, company_tickers, submissions, daily-index).

Objective (unchanged from the plan)

Populate fundamental_snapshots (CIK-keyed, one immutable row per accession) and tickers.cik/sic/sic_description from SEC data, as a SourceImporter plugging into the A1 framework. Shadow only (A3): nothing reads snapshots until A4; fundamental_data is untouched until the A5 parity gate. All new metrics are display-only.

What the SEC data actually looks like (probed, not assumed)

data.sec.gov/api/xbrl/companyfacts/CIK##########.json — one JSON per issuer (CIK) aggregating every period across every filing. Shape: facts.us-gaap.<Concept>.units.<unit>[] = {start, end, val, fy, fp, form, filed, accn, frame}.

Ground-truth findings that drive the design:

1. fp is only Q1|Q2|Q3|FY — there is no Q4. Q4 must be derived.
2. fy/fp are the filing's fiscal context, not each fact's period. Proven: Apple's FY2019 10-K carries a discrete Q3-FY2018 revenue fact (start 2018-07-01, end 2018-09-29, val 62.9B) tagged fp=FY — it's a comparative. Period identity lives in (start, end) + the filing's reportDate, never in fp/fy. Selecting values by fp would silently mix comparatives into the wrong period.
3. SEC provides both discrete 3-month facts and YTD-cumulative facts (Apple Q2 FY26: YTD 254,940 over 6mo and discrete 111,184 over 3mo; 143,756 + 111,184 = 254,940). This confirms the stored-YTD schema: store cumulative YTD per filing, derive discretes/Q4/TTM at read time.
4. Instant facts (dei:EntityCommonStockSharesOutstanding) end on the cover date (2026-04-17), which differs from period_end (2026-03-28) → the shares_outstanding_date column added in migration 026.
5. No conditional-GET support: the companyfacts endpoint returns no ETag and no Last-Modified. AAPL's file is 3.75 MB. So ~505 unconditional fetches ≈ 0.5–1.5 GB per run — the plan's "conditional HTTP no-op" is impossible on this endpoint. This is the fact that decides the fetch strategy (below).
6. submissions/CIK##########.json supplies sic, sicDescription, fiscalYearEnd (e.g. 0926), and per-accession reportDate + acceptanceDateTime — the keys for period selection and accepted_at.
7. company_tickers.json uses dash tickers (BRK-B, BRK-A) and maps GOOGL/GOOG to the same cik_str (1652044). The ticker→CIK join reuses the earnings importer's normalise_symbol (dot→dash), so both sides match.

Decision 1 (APPROVED) — fetch strategy: EDGAR daily-index driven

Plan said bulk companyfacts.zip + ETag no-op. Reality: the data.sec.gov endpoints expose no validators, and the bulk zip is multi-GB and changes ~daily (all of EDGAR), so ETag would rarely match → near-daily multi-GB download to get 505 issuers. Per-CIK conditional fetch is impossible (finding 5). Per-CIK unconditional is 0.5–1.5 GB every night.

Recommended: drive off the EDGAR daily-index (daily-index/YYYY/QTRn/ form.YYYYMMDD.idx — fixed-width Form/Company/CIK/Date/accession, ~3300 rows/day, confirmed). Each run:

• detect_revision → a composite revision, not just the date: latest-index-date + a hash of the index content processed this run + a fingerprint of the tracked symbol→CIK set. The CIK fingerprint is essential: a newly added ticker changes the revision and forces a run, so a new ticker is never no_op'd away or starved waiting for its issuer to file. Equal composite revision → no_op.
• No backfill sentinel. The absence of a prior promoted run is what triggers the initial full-history backfill; source_max_date records the processed index date each run.
• stage → for each index date since the last processed one, parse the form index, keep rows where form ∈ {10-K, 10-Q, 10-K/A, 10-Q/A} and CIK ∈ tracked set, then fetch companyfacts/CIK.json for only those few issuers and extract their newly-reported period(s). Most nights this is a handful of issuers → near-zero transfer, respectful of SEC fair-access.
• First run (backfill): no prior promoted run → fetch companyfacts for all tracked CIKs once (~1 GB one-time) and seed full history. Full history needs the paginated submissions shards — see "CIK resolution" below.

Why this over the alternatives: transfer scales with filings, not with all of EDGAR or with the universe size every night; it restores the revision/no_op model; and it's the lightest load on SEC. Cost: daily-index parsing + date bookkeeping (store last-processed index date in data_import_runs / settings). This deviates from the plan's "bulk zip" — requesting sign-off.

Decision 2 (APPROVED) — snapshot mapping: primary-period, YTD, immutable

One fundamental_snapshots row per accession, representing the filing's primary current period only (not its comparatives):

• Select the primary period by end == submissions.reportDate[accn] (finding 2), not by fp/fy. fiscal_period label comes from the filing's own fp (a 10-Q's own fp matches its current quarter; a 10-K → FY); fiscal_year/period_start/period_end from the selected facts + submissions.
• Duration facts → cumulative YTD. For each concept, pick the duration fact with accn == thisFiling, end == reportDate, and start ≈ fiscal-year start (derived from fiscalYearEnd), sanity-checked by span length (Q1≈3mo, Q2≈6mo, Q3≈9mo, FY≈12mo). If the YTD fact is absent, store null — never a discrete masquerading as cumulative (that would poison read-time differencing).
• Balance-sheet instants → at end == reportDate. shares_outstanding is the exception: prefer the dei:EntityCommonStockSharesOutstanding cover-page fact and store its own end in shares_outstanding_date (cover date ≠ period_end); when there is no dei fact (e.g. Alphabet) fall back to us-gaap:CommonStockSharesOutstanding at reportDate. A single consolidated value — never a class sum (companyfacts is non-dimensional) nor weighted-average/diluted; conflicting values → null.
• Amendments: a real 10-K/A / 10-Q/A is a new accession → a new immutable row for the same (cik, fy, fp); readers pick the newest valid accepted_at.
• Out of scope (stated, not silent): restatements that appear only as comparatives inside a later normal filing are not captured — only a real amendment updates a prior period. This narrows the plan's "newest accepted_at per period" to amendment-driven updates; a deliberate KISS boundary.
• Immutable means insert-only, not upsert. promote inserts new accession rows with ON CONFLICT (accession) DO NOTHING. An accession never mutates: if a re-fetch reconstructs different values for an accession already stored, that is a discrepancy to report (into validation_json + a system event), never a silent overwrite, and the original import_run_id is never replaced. (Ordinary updates arrive as a new amendment accession, which is a new row.)

Read-time derivation (constrains the importer; built in A4)

From the per-accession YTD rows, all at read time (newest accepted_at per period), following the schema decision already in the plan:

• discrete quarter = YTD(Qn) − YTD(Qn−1); Q4 = FY − YTD(Q3).
• TTM = sum of the trailing four discrete quarters (e.g. TTM@Q2 = FY(prev) + YTD(Q2) − YTD(Q2 prev year)).
• YoY = period vs same period a year earlier.
• Hard rule the importer must enable: any missing period in a run → the derived value is null, never a partial number. So the importer must aim for complete consecutive quarter runs per issuer and report gaps.

Metric tag catalog (prioritized us-gaap tags + fallbacks)

Tagging is inconsistent across issuers (the plan's known risk). Each metric resolves through an ordered tag list; first present wins; unit-checked.

Snapshot field	Primary tag	Fallbacks	Unit
revenue	RevenueFromContractWithCustomerExcludingAssessedTax	Revenues, SalesRevenueNet	USD
net_income	NetIncomeLoss	—	USD
operating_income	OperatingIncomeLoss	—	USD
diluted_eps	EarningsPerShareDiluted	—	USD/shares
cfo	NetCashProvidedByUsedInOperatingActivities	...ContinuingOperations	USD
capex	PaymentsToAcquirePropertyPlantAndEquipment	PaymentsToAcquireProductiveAssets	USD
depreciation_amortization	DepreciationDepletionAndAmortization	DepreciationAmortizationAndAccretionNet, DepreciationAndAmortization	USD
cash_and_st_investments	see composition rule	—	USD
total_debt	see composition rule	—	USD
shares_outstanding	dei:EntityCommonStockSharesOutstanding	—	shares

Composite fields — deterministic, aggregate-first, no double counting. Each source tag contributes at most once:

• cash_and_st_investments = CashAndCashEquivalentsAtCarryingValue + short-term investments, where ST investments = the first present of [ShortTermInvestments, MarketableSecuritiesCurrent] — never both summed.
• total_debt = long-term component + short-term component, where
  • long-term = first present of [LongTermDebt (the aggregate, already includes current + noncurrent portions), else (LongTermDebtNoncurrent + LongTermDebtCurrent)];
  • short-term borrowings = first present of [ShortTermBorrowings, CommercialPaper] (0 if neither). So the long-term aggregate and its components are mutually exclusive, and CP vs short-term-borrowings is a single pick — nothing is counted twice.

EBITDA (for net-debt/EBITDA) is derived at read time = operating_income + D&A. Concepts absent for an issuer → that field is null (display-only; no synthesis). The exact tag lists live as named constants, tunable without touching logic.

Fiscal-period identity

fiscalYearEnd (MMDD from submissions) anchors the fiscal-year start for YTD span checks and Q4 derivation. Non-calendar fiscal years (Apple's Sept) are handled because we key on (start, end) + reportDate, not calendar quarters. fiscal_year/fiscal_period are stored from the filing's own fy/fp for its primary period (safe — a filing's own context is correct for its current period).

CIK resolution & tickers backfill

• From company_tickers.json: normalise_symbol(ticker) → cik_str. Set tickers.cik for each tracked ticker (multi-class share one CIK).
• From submissions/CIK.json: sic, sicDescription, fiscalYearEnd → tickers.sic/sic_description (+ fiscal anchor for YTD/Q4).
• Submissions is paginated — full history needs the shards. filings.recent holds only the latest 1000 filings (verified: Apple recent = 1000). Older accessions live in filings.files[] = [{name, filingFrom, filingTo, filingCount}] (e.g. CIK0000320193-submissions-001.json, 1236 filings 1994–2015), each a bare object with the same parallel arrays including reportDate, acceptanceDateTime, and isXBRL. The full-history backfill must follow every filings.files[].name to obtain period identity + accepted_at
  • isXBRL for pre-1000 accessions. Incremental runs only need recent.
• Refreshed by the SEC job; a newly added ticker self-resolves on its next run (the CIK fingerprint in the revision forces that run) — until then its snapshots are absent → metrics null, per the plan.

SourceImporter mapping (source = sec_facts)

• detect_revision → latest daily-index date (or backfill sentinel on first run).
• stage → resolve tracked CIKs; (incremental) parse indices since last date → tracked filers → fetch their companyfacts → build per-accession snapshot rows; (backfill) fetch all tracked companyfacts. In-memory staged set (KISS, per A1).
• validate (fail-closed) → tracked-universe coverage floor (issuers with ≥1 snapshot); unit/period sanity (YTD spans within tolerance; EPS in USD/shares); no duplicate accession; filings skipped for missing period identity are counted in validation_json (carry-forward from A1 review); an unexpected companyfacts shape (missing facts/units) stops promotion.
• Index↔Company-Facts consistency gate (the daily index and Company Facts are separate SEC products that can lag each other): for every tracked index accession marked isXBRL, confirm that accession actually appears in the fetched companyfacts before promotion. If any is missing → fail the run and retry later — do not advance the revision and do not record an incomplete/null snapshot for it. Non-XBRL amendments are skipped with a recorded reason in validation_json. (The framework only stores the revision on a promoted run, so a failed consistency check naturally leaves the revision behind for retry.)
• promote → insert snapshot rows (ON CONFLICT (accession) DO NOTHING; immutable — see Decision 2), stamped import_run_id; refresh tickers.cik/sic/sic_description. A re-fetch that reconstructs different values for an existing accession is reported as a discrepancy, never a silent mutation. Non-destructive (append-only accessions) — no future-row deletion like earnings.

SEC fair-access (operational, per the plan's non-negotiable)

Identifying User-Agent with contact email on every request; well under 10 req/s with spacing; exponential backoff on 429; 403 → alert and stop, never retry-loop — with one carved-out exception: www.sec.gov/Archives is served from an S3 bucket without a ListBucket grant, so an absent file 403s with S3's AccessDenied XML rather than 404 (every weekend/holiday daily index does this). That one shape is read as "missing"; a real rejection is the WAF's text/html "Undeclared Automated Tool" page and still stops the run. New config: sec_user_agent, sec_request_spacing_seconds, sec_max_retries. Keep only the last ~2 fetched artifacts on disk for debugging (reproducibility is the normalized Postgres rows, per the plan).

Explicitly out of scope for A3

• fundamental_data cutover (A5 parity gate) — snapshots only in A3.
• The read-time derivation, API object, and panel (A4).
• Comparative-only restatements (Decision 2).
• Point-in-time backtest enforcement (accepted_at stored, not yet enforced).

Decisions (signed off 2026-07-22)

1. Fetch — EDGAR daily-index driven (Decision 1). Approved deviation from the plan's bulk zip.
2. Snapshot mapping — primary-period-only per accession; comparative-only restatements out of scope (Decision 2). Approved.
3. Backfill depth — seed full available history per issuer on first run (cheap to store; powers the quarter tape / multi-year YoY). Approved.