signal-platform/docs/dolt-sec-a3-design.md

A3 design — SEC fundamentals importer

Status: design pass, 2026-07-22 — awaiting sign-off on two plan deviations (fetch strategy, snapshot mapping). Not yet implemented. Companion to docs/dolt-integration-plan.md (workstream A, phase A3). Grounded in live SEC data probes (Apple CIK 0000320193, company_tickers, submissions, daily-index).

Objective (unchanged from the plan)

Populate fundamental_snapshots (CIK-keyed, one immutable row per accession) and tickers.cik/sic/sic_description from SEC data, as a SourceImporter plugging into the A1 framework. Shadow only (A3): nothing reads snapshots until A4; fundamental_data is untouched until the A5 parity gate. All new metrics are display-only.

What the SEC data actually looks like (probed, not assumed)

data.sec.gov/api/xbrl/companyfacts/CIK##########.json — one JSON per issuer (CIK) aggregating every period across every filing. Shape: facts.us-gaap.<Concept>.units.<unit>[] = {start, end, val, fy, fp, form, filed, accn, frame}.

Ground-truth findings that drive the design:

1. fp is only Q1|Q2|Q3|FY — there is no Q4. Q4 must be derived.
2. fy/fp are the filing's fiscal context, not each fact's period. Proven: Apple's FY2019 10-K carries a discrete Q3-FY2018 revenue fact (start 2018-07-01, end 2018-09-29, val 62.9B) tagged fp=FY — it's a comparative. Period identity lives in (start, end) + the filing's reportDate, never in fp/fy. Selecting values by fp would silently mix comparatives into the wrong period.
3. SEC provides both discrete 3-month facts and YTD-cumulative facts (Apple Q2 FY26: YTD 254,940 over 6mo and discrete 111,184 over 3mo; 143,756 + 111,184 = 254,940). This confirms the stored-YTD schema: store cumulative YTD per filing, derive discretes/Q4/TTM at read time.
4. Instant facts (dei:EntityCommonStockSharesOutstanding) end on the cover date (2026-04-17), which differs from period_end (2026-03-28) → the shares_outstanding_date column added in migration 026.
5. No conditional-GET support: the companyfacts endpoint returns no ETag and no Last-Modified. AAPL's file is 3.75 MB. So ~505 unconditional fetches ≈ 0.5–1.5 GB per run — the plan's "conditional HTTP no-op" is impossible on this endpoint. This is the fact that decides the fetch strategy (below).
6. submissions/CIK##########.json supplies sic, sicDescription, fiscalYearEnd (e.g. 0926), and per-accession reportDate + acceptanceDateTime — the keys for period selection and accepted_at.
7. company_tickers.json uses dash tickers (BRK-B, BRK-A) and maps GOOGL/GOOG to the same cik_str (1652044). The ticker→CIK join reuses the earnings importer's normalise_symbol (dot→dash), so both sides match.

Decision 1 (needs sign-off) — fetch strategy: EDGAR daily-index driven

Plan said bulk companyfacts.zip + ETag no-op. Reality: the data.sec.gov endpoints expose no validators, and the bulk zip is multi-GB and changes ~daily (all of EDGAR), so ETag would rarely match → near-daily multi-GB download to get 505 issuers. Per-CIK conditional fetch is impossible (finding 5). Per-CIK unconditional is 0.5–1.5 GB every night.

Recommended: drive off the EDGAR daily-index (daily-index/YYYY/QTRn/ form.YYYYMMDD.idx — fixed-width Form/Company/CIK/Date/accession, ~3300 rows/day, confirmed). Each run:

• detect_revision → the latest available daily-index date. If it equals the last processed date, no_op (framework's model works cleanly again).
• stage → for each index date since the last processed one, parse the form index, keep rows where form ∈ {10-K, 10-Q, 10-K/A, 10-Q/A} and CIK ∈ tracked set, then fetch companyfacts/CIK.json for only those few issuers and extract their newly-reported period(s). Most nights this is a handful of issuers → near-zero transfer, respectful of SEC fair-access.
• First run (backfill) has no last-processed date: fetch companyfacts for all tracked CIKs once (~1 GB one-time) to seed history, then go incremental.

Why this over the alternatives: transfer scales with filings, not with all of EDGAR or with the universe size every night; it restores the revision/no_op model; and it's the lightest load on SEC. Cost: daily-index parsing + date bookkeeping (store last-processed index date in data_import_runs / settings). This deviates from the plan's "bulk zip" — requesting sign-off.

Decision 2 (needs sign-off) — snapshot mapping: primary-period, YTD, immutable

One fundamental_snapshots row per accession, representing the filing's primary current period only (not its comparatives):

• Select the primary period by end == submissions.reportDate[accn] (finding 2), not by fp/fy. fiscal_period label comes from the filing's own fp (a 10-Q's own fp matches its current quarter; a 10-K → FY); fiscal_year/period_start/period_end from the selected facts + submissions.
• Duration facts → cumulative YTD. For each concept, pick the duration fact with accn == thisFiling, end == reportDate, and start ≈ fiscal-year start (derived from fiscalYearEnd), sanity-checked by span length (Q1≈3mo, Q2≈6mo, Q3≈9mo, FY≈12mo). If the YTD fact is absent, store null — never a discrete masquerading as cumulative (that would poison read-time differencing).
• Balance-sheet instants → at end == reportDate. shares_outstanding is the exception: take dei:EntityCommonStockSharesOutstanding for that accession and store its own end in shares_outstanding_date (cover date ≠ period_end).
• Amendments: a real 10-K/A / 10-Q/A is a new accession → a new immutable row for the same (cik, fy, fp); readers pick the newest valid accepted_at.
• Out of scope (stated, not silent): restatements that appear only as comparatives inside a later normal filing are not captured — only a real amendment updates a prior period. This narrows the plan's "newest accepted_at per period" to amendment-driven updates; a deliberate KISS boundary.

Read-time derivation (constrains the importer; built in A4)

From the per-accession YTD rows, all at read time (newest accepted_at per period), following the schema decision already in the plan:

• discrete quarter = YTD(Qn) − YTD(Qn−1); Q4 = FY − YTD(Q3).
• TTM = sum of the trailing four discrete quarters (e.g. TTM@Q2 = FY(prev) + YTD(Q2) − YTD(Q2 prev year)).
• YoY = period vs same period a year earlier.
• Hard rule the importer must enable: any missing period in a run → the derived value is null, never a partial number. So the importer must aim for complete consecutive quarter runs per issuer and report gaps.

Metric tag catalog (prioritized us-gaap tags + fallbacks)

Tagging is inconsistent across issuers (the plan's known risk). Each metric resolves through an ordered tag list; first present wins; unit-checked.

Snapshot field	Primary tag	Fallbacks	Unit
revenue	RevenueFromContractWithCustomerExcludingAssessedTax	Revenues, SalesRevenueNet	USD
net_income	NetIncomeLoss	—	USD
operating_income	OperatingIncomeLoss	—	USD
diluted_eps	EarningsPerShareDiluted	—	USD/shares
cfo	NetCashProvidedByUsedInOperatingActivities	...ContinuingOperations	USD
capex	PaymentsToAcquirePropertyPlantAndEquipment	PaymentsToAcquireProductiveAssets	USD
depreciation_amortization	DepreciationDepletionAndAmortization	DepreciationAmortizationAndAccretionNet, DepreciationAndAmortization	USD
cash_and_st_investments	CashAndCashEquivalentsAtCarryingValue (+ ST inv)	ST inv: ShortTermInvestments, MarketableSecuritiesCurrent	USD
total_debt	LongTermDebtNoncurrent + LongTermDebtCurrent	LongTermDebt; +CommercialPaper/ShortTermBorrowings if present	USD
shares_outstanding	dei:EntityCommonStockSharesOutstanding	—	shares

EBITDA (for net-debt/EBITDA) is derived at read time = operating_income + D&A. Concepts absent for an issuer → that field is null (display-only; no synthesis). The exact tag lists live as named constants, tunable without touching logic.

Fiscal-period identity

fiscalYearEnd (MMDD from submissions) anchors the fiscal-year start for YTD span checks and Q4 derivation. Non-calendar fiscal years (Apple's Sept) are handled because we key on (start, end) + reportDate, not calendar quarters. fiscal_year/fiscal_period are stored from the filing's own fy/fp for its primary period (safe — a filing's own context is correct for its current period).

CIK resolution & tickers backfill

• From company_tickers.json: normalise_symbol(ticker) → cik_str. Set tickers.cik for each tracked ticker (multi-class share one CIK).
• From submissions/CIK.json: sic, sicDescription → tickers.sic/sic_description.
• Refreshed by the SEC job; a newly added ticker self-resolves on its next run (until then its snapshots are absent → metrics null, per the plan).

SourceImporter mapping (source = sec_facts)

• detect_revision → latest daily-index date (or backfill sentinel on first run).
• stage → resolve tracked CIKs; (incremental) parse indices since last date → tracked filers → fetch their companyfacts → build per-accession snapshot rows; (backfill) fetch all tracked companyfacts. In-memory staged set (KISS, per A1).
• validate (fail-closed) → tracked-universe coverage floor (issuers with ≥1 snapshot); unit/period sanity (YTD spans within tolerance; EPS in USD/shares); no duplicate accession; filings skipped for missing period identity are counted in validation_json (carry-forward from A1 review); an unexpected companyfacts shape (missing facts/units) stops promotion.
• promote → upsert snapshot rows keyed by unique accession, stamped import_run_id; refresh tickers.cik/sic/sic_description. Non-destructive (append-only accessions) — no future-row deletion like earnings.

SEC fair-access (operational, per the plan's non-negotiable)

Identifying User-Agent with contact email on every request; well under 10 req/s with spacing; exponential backoff on 429; 403 → alert and stop, never retry-loop. New config: sec_user_agent, sec_request_spacing_seconds, sec_max_retries. Keep only the last ~2 fetched artifacts on disk for debugging (reproducibility is the normalized Postgres rows, per the plan).

Explicitly out of scope for A3

• fundamental_data cutover (A5 parity gate) — snapshots only in A3.
• The read-time derivation, API object, and panel (A4).
• Comparative-only restatements (Decision 2).
• Point-in-time backtest enforcement (accepted_at stored, not yet enforced).

Open questions for sign-off

1. Decision 1 (daily-index fetch vs the plan's bulk zip) — approve the deviation?
2. Decision 2 (primary-period-only; comparative restatements out of scope) — approve?
3. Backfill depth: seed full available history per issuer on first run, or cap at N years (e.g. 5, matching ohlcv_history_days)? Full history is cheap to store and powers the quarter tape / YoY.