signal-platform/docs/dolt-sec-a3-design.md

# A3 design — SEC fundamentals importer

Status: **design APPROVED 2026-07-22 — three decisions signed off (daily-index
fetch, primary-period-only snapshots, full-history backfill) + four review
correctness fixes folded in (composite revision incl. universe fingerprint;
submissions pagination shards for full history; index↔Company-Facts consistency
gate; insert-only immutability with discrepancy reporting; deterministic cash/debt
composition). Ready to implement.**
Companion to `docs/dolt-integration-plan.md` (workstream A, phase A3). Grounded in
live SEC data probes (Apple CIK 0000320193, company_tickers, submissions, daily-index).

## Objective (unchanged from the plan)

Populate `fundamental_snapshots` (CIK-keyed, one immutable row per accession)
and `tickers.cik/sic/sic_description` from SEC data, as a `SourceImporter`
plugging into the A1 framework. Shadow only (A3): nothing reads snapshots until
A4; `fundamental_data` is untouched until the A5 parity gate. All new metrics
are display-only.

## What the SEC data actually looks like (probed, not assumed)

`data.sec.gov/api/xbrl/companyfacts/CIK##########.json` — one JSON per **issuer
(CIK)** aggregating every period across every filing. Shape:
`facts.us-gaap.<Concept>.units.<unit>[] = {start, end, val, fy, fp, form, filed, accn, frame}`.

Ground-truth findings that drive the design:

1. **`fp` is only `Q1|Q2|Q3|FY` — there is no `Q4`.** Q4 must be derived.
2. **`fy`/`fp` are the *filing's* fiscal context, not each fact's period.** Proven:
   Apple's FY2019 10-K carries a discrete Q3-FY2018 revenue fact
   (`start 2018-07-01, end 2018-09-29, val 62.9B`) tagged `fp=FY` — it's a
   comparative. **Period identity lives in `(start, end)` + the filing's
   `reportDate`, never in `fp/fy`.** Selecting values by `fp` would silently mix
   comparatives into the wrong period.
3. SEC provides **both** discrete 3-month facts **and** YTD-cumulative facts
   (Apple Q2 FY26: YTD `254,940` over 6mo *and* discrete `111,184` over 3mo;
   `143,756 + 111,184 = 254,940`). This confirms the stored-YTD schema: store
   cumulative YTD per filing, derive discretes/Q4/TTM at read time.
4. Instant facts (`dei:EntityCommonStockSharesOutstanding`) end on the **cover
   date** (2026-04-17), which differs from `period_end` (2026-03-28) → the
   `shares_outstanding_date` column added in migration 026.
5. **No conditional-GET support:** the companyfacts endpoint returns no `ETag`
   and no `Last-Modified`. AAPL's file is 3.75 MB. So ~505 unconditional fetches
   ≈ 0.5–1.5 GB *per run* — the plan's "conditional HTTP no-op" is impossible on
   this endpoint. This is the fact that decides the fetch strategy (below).
6. `submissions/CIK##########.json` supplies `sic`, `sicDescription`,
   `fiscalYearEnd` (e.g. `0926`), and per-accession `reportDate` +
   `acceptanceDateTime` — the keys for period selection and `accepted_at`.
7. `company_tickers.json` uses **dash** tickers (`BRK-B`, `BRK-A`) and maps
   `GOOGL`/`GOOG` to the **same** `cik_str` (1652044). The ticker→CIK join reuses
   the earnings importer's `normalise_symbol` (dot→dash), so both sides match.

## Decision 1 (APPROVED) — fetch strategy: EDGAR daily-index driven

**Plan said** bulk `companyfacts.zip` + ETag no-op. **Reality:** the data.sec.gov
endpoints expose no validators, and the bulk zip is multi-GB and changes ~daily
(all of EDGAR), so ETag would rarely match → near-daily multi-GB download to get
505 issuers. Per-CIK *conditional* fetch is impossible (finding 5). Per-CIK
*unconditional* is 0.5–1.5 GB every night.

**Recommended:** drive off the **EDGAR daily-index** (`daily-index/YYYY/QTRn/
form.YYYYMMDD.idx` — fixed-width Form/Company/CIK/Date/accession, ~3300 rows/day,
confirmed). Each run:

- `detect_revision` → a **composite revision**, not just the date:
  `latest-index-date` + a hash of the index content processed this run + a
  **fingerprint of the tracked symbol→CIK set**. The CIK fingerprint is essential:
  a newly added ticker changes the revision and forces a run, so a new ticker is
  never `no_op`'d away or starved waiting for its issuer to file. Equal composite
  revision → `no_op`.
- **No backfill sentinel.** The *absence of a prior promoted run* is what triggers
  the initial full-history backfill; `source_max_date` records the processed index
  date each run.
- `stage` → for each index date since the last processed one, parse the form
  index, keep rows where `form ∈ {10-K, 10-Q, 10-K/A, 10-Q/A}` **and** CIK ∈
  tracked set, then fetch `companyfacts/CIK.json` for **only those few issuers**
  and extract their newly-reported period(s). Most nights this is a handful of
  issuers → near-zero transfer, respectful of SEC fair-access.
- **First run (backfill)**: no prior promoted run → fetch companyfacts for all
  tracked CIKs once (~1 GB one-time) and seed **full** history. Full history needs
  the paginated submissions shards — see "CIK resolution" below.

Why this over the alternatives: transfer scales with *filings*, not with all of
EDGAR or with the universe size every night; it restores the revision/no_op
model; and it's the lightest load on SEC. Cost: daily-index parsing + date
bookkeeping (store last-processed index date in `data_import_runs` /
settings). **This deviates from the plan's "bulk zip" — requesting sign-off.**

## Decision 2 (APPROVED) — snapshot mapping: primary-period, YTD, immutable

One `fundamental_snapshots` row per accession, representing the filing's
**primary current period only** (not its comparatives):

- **Select the primary period by `end == submissions.reportDate[accn]`** (finding
  2), *not* by `fp/fy`. `fiscal_period` label comes from the filing's own `fp`
  (a 10-Q's own `fp` matches its current quarter; a 10-K → `FY`);
  `fiscal_year`/`period_start`/`period_end` from the selected facts + submissions.
- **Duration facts → cumulative YTD.** For each concept, pick the duration fact
  with `accn == thisFiling`, `end == reportDate`, and `start ≈ fiscal-year start`
  (derived from `fiscalYearEnd`), sanity-checked by span length (Q1≈3mo, Q2≈6mo,
  Q3≈9mo, FY≈12mo). **If the YTD fact is absent, store null — never a discrete
  masquerading as cumulative** (that would poison read-time differencing).
- **Balance-sheet instants → at `end == reportDate`.** `shares_outstanding` is
  the exception: prefer the `dei:EntityCommonStockSharesOutstanding` cover-page
  fact and store *its own* `end` in `shares_outstanding_date` (cover date ≠
  period_end); when there is no dei fact (e.g. Alphabet) fall back to
  `us-gaap:CommonStockSharesOutstanding` at `reportDate`. A single consolidated
  value — never a class sum (companyfacts is non-dimensional) nor
  weighted-average/diluted; conflicting values → null.
- **Amendments:** a real `10-K/A` / `10-Q/A` is a new accession → a new immutable
  row for the same `(cik, fy, fp)`; readers pick the newest valid `accepted_at`.
- **Out of scope (stated, not silent):** restatements that appear *only* as
  comparatives inside a later normal filing are **not** captured — only a real
  amendment updates a prior period. This narrows the plan's "newest accepted_at
  per period" to amendment-driven updates; a deliberate KISS boundary.
- **Immutable means insert-only, not upsert.** `promote` **inserts** new accession
  rows with `ON CONFLICT (accession) DO NOTHING`. An accession never mutates: if a
  re-fetch reconstructs *different* values for an accession already stored, that is
  a **discrepancy to report** (into `validation_json` + a system event), never a
  silent overwrite, and the original `import_run_id` is never replaced. (Ordinary
  updates arrive as a *new* amendment accession, which is a new row.)

## Read-time derivation (constrains the importer; built in A4)

From the per-accession YTD rows, all at read time (newest `accepted_at` per
period), following the schema decision already in the plan:

- discrete quarter = YTD(Qn) − YTD(Qn−1); **Q4 = FY − YTD(Q3)**.
- TTM = sum of the trailing four discrete quarters (e.g. TTM@Q2 = FY(prev) +
  YTD(Q2) − YTD(Q2 prev year)).
- YoY = period vs same period a year earlier.
- **Hard rule the importer must enable: any missing period in a run → the derived
  value is `null`, never a partial number.** So the importer must aim for
  complete consecutive quarter runs per issuer and report gaps.

## Metric tag catalog (prioritized us-gaap tags + fallbacks)

Tagging is inconsistent across issuers (the plan's known risk). Each metric
resolves through an ordered tag list; first present wins; unit-checked.

| Snapshot field | Primary tag | Fallbacks | Unit |
|---|---|---|---|
| revenue | `RevenueFromContractWithCustomerExcludingAssessedTax` | `Revenues`, `SalesRevenueNet` | USD |
| net_income | `NetIncomeLoss` | — | USD |
| operating_income | `OperatingIncomeLoss` | — | USD |
| diluted_eps | `EarningsPerShareDiluted` | — | USD/shares |
| cfo | `NetCashProvidedByUsedInOperatingActivities` | `...ContinuingOperations` | USD |
| capex | `PaymentsToAcquirePropertyPlantAndEquipment` | `PaymentsToAcquireProductiveAssets` | USD |
| depreciation_amortization | `DepreciationDepletionAndAmortization` | `DepreciationAmortizationAndAccretionNet`, `DepreciationAndAmortization` | USD |
| cash_and_st_investments | see composition rule | — | USD |
| total_debt | see composition rule | — | USD |
| shares_outstanding | `dei:EntityCommonStockSharesOutstanding` | — | shares |

**Composite fields — deterministic, aggregate-first, no double counting.** Each
source tag contributes at most once:

- `cash_and_st_investments` = `CashAndCashEquivalentsAtCarryingValue`
  **+ short-term investments**, where ST investments = the **first present** of
  [`ShortTermInvestments`, `MarketableSecuritiesCurrent`] — never both summed.
- `total_debt` = **long-term component + short-term component**, where
  - long-term = first present of [`LongTermDebt` (the aggregate, already includes
    current + noncurrent portions), **else** (`LongTermDebtNoncurrent` +
    `LongTermDebtCurrent`)];
  - short-term borrowings = first present of [`ShortTermBorrowings`,
    `CommercialPaper`] (0 if neither).
  So the long-term aggregate and its components are mutually exclusive, and CP vs
  short-term-borrowings is a single pick — nothing is counted twice.

EBITDA (for net-debt/EBITDA) is derived at read time = operating_income + D&A.
Concepts absent for an issuer → that field is null (display-only; no synthesis).
The exact tag lists live as named constants, tunable without touching logic.

## Fiscal-period identity

`fiscalYearEnd` (MMDD from submissions) anchors the fiscal-year start for YTD
span checks and Q4 derivation. Non-calendar fiscal years (Apple's Sept) are
handled because we key on `(start, end)` + `reportDate`, not calendar quarters.
`fiscal_year`/`fiscal_period` are stored from the filing's own `fy`/`fp` for its
primary period (safe — a filing's own context is correct for its current period).

## CIK resolution & tickers backfill

- From `company_tickers.json`: `normalise_symbol(ticker) → cik_str`. Set
  `tickers.cik` for each tracked ticker (multi-class share one CIK).
- From `submissions/CIK.json`: `sic`, `sicDescription`, `fiscalYearEnd` →
  `tickers.sic/sic_description` (+ fiscal anchor for YTD/Q4).
- **Submissions is paginated — full history needs the shards.** `filings.recent`
  holds only the **latest 1000** filings (verified: Apple `recent` = 1000). Older
  accessions live in `filings.files[]` = `[{name, filingFrom, filingTo,
  filingCount}]` (e.g. `CIK0000320193-submissions-001.json`, 1236 filings
  1994–2015), each a bare object with the **same parallel arrays** including
  `reportDate`, `acceptanceDateTime`, and `isXBRL`. The full-history backfill must
  **follow every `filings.files[].name`** to obtain period identity + `accepted_at`
  + `isXBRL` for pre-1000 accessions. Incremental runs only need `recent`.
- Refreshed by the SEC job; a newly added ticker self-resolves on its next run
  (the CIK fingerprint in the revision forces that run) — until then its snapshots
  are absent → metrics null, per the plan.

## SourceImporter mapping (source = `sec_facts`)

- `detect_revision` → latest daily-index date (or `backfill` sentinel on first run).
- `stage` → resolve tracked CIKs; (incremental) parse indices since last date →
  tracked filers → fetch their companyfacts → build per-accession snapshot rows;
  (backfill) fetch all tracked companyfacts. In-memory staged set (KISS, per A1).
- `validate` (fail-closed) → tracked-universe **coverage floor** (issuers with ≥1
  snapshot); **unit/period sanity** (YTD spans within tolerance; EPS in USD/shares);
  **no duplicate accession**; **filings skipped for missing period identity are
  counted in `validation_json`** (carry-forward from A1 review); an unexpected
  companyfacts shape (missing `facts`/`units`) stops promotion.
- **Index↔Company-Facts consistency gate (the daily index and Company Facts are
  separate SEC products that can lag each other):** for every tracked index
  accession marked `isXBRL`, confirm that accession actually appears in the fetched
  companyfacts before promotion. If any is missing → **fail the run and retry
  later** — do **not** advance the revision and do **not** record an
  incomplete/null snapshot for it. Non-XBRL amendments are skipped with a recorded
  reason in `validation_json`. (The framework only stores the revision on a
  promoted run, so a failed consistency check naturally leaves the revision behind
  for retry.)
- `promote` → **insert** snapshot rows (`ON CONFLICT (accession) DO NOTHING`;
  immutable — see Decision 2), stamped `import_run_id`; refresh
  `tickers.cik/sic/sic_description`. A re-fetch that reconstructs different values
  for an existing accession is reported as a discrepancy, never a silent mutation.
  Non-destructive (append-only accessions) — no future-row deletion like earnings.

## SEC fair-access (operational, per the plan's non-negotiable)

Identifying `User-Agent` with contact email on every request; well under 10 req/s
with spacing; exponential backoff on 429; **403 → alert and stop, never
retry-loop** — with one carved-out exception: `www.sec.gov/Archives` is served
from an S3 bucket without a `ListBucket` grant, so an **absent** file 403s with
S3's `AccessDenied` XML rather than 404 (every weekend/holiday daily index does
this). That one shape is read as "missing"; a real rejection is the WAF's
`text/html` "Undeclared Automated Tool" page and still stops the run.
New config: `sec_user_agent`, `sec_request_spacing_seconds`,
`sec_max_retries`. Keep only the last ~2 fetched artifacts on disk for debugging
(reproducibility is the normalized Postgres rows, per the plan).

## Explicitly out of scope for A3

- `fundamental_data` cutover (A5 parity gate) — snapshots only in A3.
- The read-time derivation, API object, and panel (A4).
- Comparative-only restatements (Decision 2).
- Point-in-time backtest enforcement (`accepted_at` stored, not yet enforced).

## Decisions (signed off 2026-07-22)

1. **Fetch** — EDGAR daily-index driven (Decision 1). Approved deviation from the
   plan's bulk zip.
2. **Snapshot mapping** — primary-period-only per accession; comparative-only
   restatements out of scope (Decision 2). Approved.
3. **Backfill depth** — seed **full** available history per issuer on first run
   (cheap to store; powers the quarter tape / multi-year YoY). Approved.