docs: A3 design pass — SEC fundamentals importer

Design (not implementation) for phase A3, grounded in live SEC data probes.
Key findings: fp has no Q4 (derive it); fy/fp are the filing's context not each
fact's period (select by end==reportDate); SEC provides both discrete and YTD
facts (confirms stored-YTD schema); companyfacts endpoint has no ETag/
Last-Modified (conditional GET impossible); tickers are dash-form and GOOG/GOOGL
share one CIK.

Two plan deviations flagged for sign-off:
1. Fetch via the EDGAR daily-index (fetch companyfacts only for tracked issuers
   that filed) rather than the multi-GB bulk zip — lighter and restores the
   revision/no_op model.
2. One snapshot row per accession for its primary period (YTD-cumulative);
   comparative-only restatements out of scope (only real 10-K/A updates a period).

Plus a metric tag catalog, read-time derivation rules (missing period -> null),
CIK resolution, validation gates, and SEC fair-access handling.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

docs/dolt-sec-a3-design.md

@@ -0,0 +1,188 @@
+# A3 design — SEC fundamentals importer
+
+Status: **design pass, 2026-07-22 — awaiting sign-off on two plan deviations
+(fetch strategy, snapshot mapping).** Not yet implemented. Companion to
+`docs/dolt-integration-plan.md` (workstream A, phase A3). Grounded in live SEC
+data probes (Apple CIK 0000320193, company_tickers, submissions, daily-index).
+
+## Objective (unchanged from the plan)
+
+Populate `fundamental_snapshots` (CIK-keyed, one immutable row per accession)
+and `tickers.cik/sic/sic_description` from SEC data, as a `SourceImporter`
+plugging into the A1 framework. Shadow only (A3): nothing reads snapshots until
+A4; `fundamental_data` is untouched until the A5 parity gate. All new metrics
+are display-only.
+
+## What the SEC data actually looks like (probed, not assumed)
+
+`data.sec.gov/api/xbrl/companyfacts/CIK##########.json` — one JSON per **issuer
+(CIK)** aggregating every period across every filing. Shape:
+`facts.us-gaap.<Concept>.units.<unit>[] = {start, end, val, fy, fp, form, filed, accn, frame}`.
+
+Ground-truth findings that drive the design:
+
+1. **`fp` is only `Q1|Q2|Q3|FY` — there is no `Q4`.** Q4 must be derived.
+2. **`fy`/`fp` are the *filing's* fiscal context, not each fact's period.** Proven:
+   Apple's FY2019 10-K carries a discrete Q3-FY2018 revenue fact
+   (`start 2018-07-01, end 2018-09-29, val 62.9B`) tagged `fp=FY` — it's a
+   comparative. **Period identity lives in `(start, end)` + the filing's
+   `reportDate`, never in `fp/fy`.** Selecting values by `fp` would silently mix
+   comparatives into the wrong period.
+3. SEC provides **both** discrete 3-month facts **and** YTD-cumulative facts
+   (Apple Q2 FY26: YTD `254,940` over 6mo *and* discrete `111,184` over 3mo;
+   `143,756 + 111,184 = 254,940`). This confirms the stored-YTD schema: store
+   cumulative YTD per filing, derive discretes/Q4/TTM at read time.
+4. Instant facts (`dei:EntityCommonStockSharesOutstanding`) end on the **cover
+   date** (2026-04-17), which differs from `period_end` (2026-03-28) → the
+   `shares_outstanding_date` column added in migration 026.
+5. **No conditional-GET support:** the companyfacts endpoint returns no `ETag`
+   and no `Last-Modified`. AAPL's file is 3.75 MB. So ~505 unconditional fetches
+   ≈ 0.5–1.5 GB *per run* — the plan's "conditional HTTP no-op" is impossible on
+   this endpoint. This is the fact that decides the fetch strategy (below).
+6. `submissions/CIK##########.json` supplies `sic`, `sicDescription`,
+   `fiscalYearEnd` (e.g. `0926`), and per-accession `reportDate` +
+   `acceptanceDateTime` — the keys for period selection and `accepted_at`.
+7. `company_tickers.json` uses **dash** tickers (`BRK-B`, `BRK-A`) and maps
+   `GOOGL`/`GOOG` to the **same** `cik_str` (1652044). The ticker→CIK join reuses
+   the earnings importer's `normalise_symbol` (dot→dash), so both sides match.
+
+## Decision 1 (needs sign-off) — fetch strategy: EDGAR daily-index driven
+
+**Plan said** bulk `companyfacts.zip` + ETag no-op. **Reality:** the data.sec.gov
+endpoints expose no validators, and the bulk zip is multi-GB and changes ~daily
+(all of EDGAR), so ETag would rarely match → near-daily multi-GB download to get
+505 issuers. Per-CIK *conditional* fetch is impossible (finding 5). Per-CIK
+*unconditional* is 0.5–1.5 GB every night.
+
+**Recommended:** drive off the **EDGAR daily-index** (`daily-index/YYYY/QTRn/
+form.YYYYMMDD.idx` — fixed-width Form/Company/CIK/Date/accession, ~3300 rows/day,
+confirmed). Each run:
+
+- `detect_revision` → the latest available daily-index date. If it equals the
+  last processed date, **`no_op`** (framework's model works cleanly again).
+- `stage` → for each index date since the last processed one, parse the form
+  index, keep rows where `form ∈ {10-K, 10-Q, 10-K/A, 10-Q/A}` **and** CIK ∈
+  tracked set, then fetch `companyfacts/CIK.json` for **only those few issuers**
+  and extract their newly-reported period(s). Most nights this is a handful of
+  issuers → near-zero transfer, respectful of SEC fair-access.
+- **First run (backfill)** has no last-processed date: fetch companyfacts for all
+  tracked CIKs once (~1 GB one-time) to seed history, then go incremental.
+
+Why this over the alternatives: transfer scales with *filings*, not with all of
+EDGAR or with the universe size every night; it restores the revision/no_op
+model; and it's the lightest load on SEC. Cost: daily-index parsing + date
+bookkeeping (store last-processed index date in `data_import_runs` /
+settings). **This deviates from the plan's "bulk zip" — requesting sign-off.**
+
+## Decision 2 (needs sign-off) — snapshot mapping: primary-period, YTD, immutable
+
+One `fundamental_snapshots` row per accession, representing the filing's
+**primary current period only** (not its comparatives):
+
+- **Select the primary period by `end == submissions.reportDate[accn]`** (finding
+  2), *not* by `fp/fy`. `fiscal_period` label comes from the filing's own `fp`
+  (a 10-Q's own `fp` matches its current quarter; a 10-K → `FY`);
+  `fiscal_year`/`period_start`/`period_end` from the selected facts + submissions.
+- **Duration facts → cumulative YTD.** For each concept, pick the duration fact
+  with `accn == thisFiling`, `end == reportDate`, and `start ≈ fiscal-year start`
+  (derived from `fiscalYearEnd`), sanity-checked by span length (Q1≈3mo, Q2≈6mo,
+  Q3≈9mo, FY≈12mo). **If the YTD fact is absent, store null — never a discrete
+  masquerading as cumulative** (that would poison read-time differencing).
+- **Balance-sheet instants → at `end == reportDate`.** `shares_outstanding` is
+  the exception: take `dei:EntityCommonStockSharesOutstanding` for that accession
+  and store *its own* `end` in `shares_outstanding_date` (cover date ≠ period_end).
+- **Amendments:** a real `10-K/A` / `10-Q/A` is a new accession → a new immutable
+  row for the same `(cik, fy, fp)`; readers pick the newest valid `accepted_at`.
+- **Out of scope (stated, not silent):** restatements that appear *only* as
+  comparatives inside a later normal filing are **not** captured — only a real
+  amendment updates a prior period. This narrows the plan's "newest accepted_at
+  per period" to amendment-driven updates; a deliberate KISS boundary.
+
+## Read-time derivation (constrains the importer; built in A4)
+
+From the per-accession YTD rows, all at read time (newest `accepted_at` per
+period), following the schema decision already in the plan:
+
+- discrete quarter = YTD(Qn) − YTD(Qn−1); **Q4 = FY − YTD(Q3)**.
+- TTM = sum of the trailing four discrete quarters (e.g. TTM@Q2 = FY(prev) +
+  YTD(Q2) − YTD(Q2 prev year)).
+- YoY = period vs same period a year earlier.
+- **Hard rule the importer must enable: any missing period in a run → the derived
+  value is `null`, never a partial number.** So the importer must aim for
+  complete consecutive quarter runs per issuer and report gaps.
+
+## Metric tag catalog (prioritized us-gaap tags + fallbacks)
+
+Tagging is inconsistent across issuers (the plan's known risk). Each metric
+resolves through an ordered tag list; first present wins; unit-checked.
+
+| Snapshot field | Primary tag | Fallbacks | Unit |
+|---|---|---|---|
+| revenue | `RevenueFromContractWithCustomerExcludingAssessedTax` | `Revenues`, `SalesRevenueNet` | USD |
+| net_income | `NetIncomeLoss` | — | USD |
+| operating_income | `OperatingIncomeLoss` | — | USD |
+| diluted_eps | `EarningsPerShareDiluted` | — | USD/shares |
+| cfo | `NetCashProvidedByUsedInOperatingActivities` | `...ContinuingOperations` | USD |
+| capex | `PaymentsToAcquirePropertyPlantAndEquipment` | `PaymentsToAcquireProductiveAssets` | USD |
+| depreciation_amortization | `DepreciationDepletionAndAmortization` | `DepreciationAmortizationAndAccretionNet`, `DepreciationAndAmortization` | USD |
+| cash_and_st_investments | `CashAndCashEquivalentsAtCarryingValue` (+ ST inv) | ST inv: `ShortTermInvestments`, `MarketableSecuritiesCurrent` | USD |
+| total_debt | `LongTermDebtNoncurrent` + `LongTermDebtCurrent` | `LongTermDebt`; +`CommercialPaper`/`ShortTermBorrowings` if present | USD |
+| shares_outstanding | `dei:EntityCommonStockSharesOutstanding` | — | shares |
+
+EBITDA (for net-debt/EBITDA) is derived at read time = operating_income + D&A.
+Concepts absent for an issuer → that field is null (display-only; no synthesis).
+The exact tag lists live as named constants, tunable without touching logic.
+
+## Fiscal-period identity
+
+`fiscalYearEnd` (MMDD from submissions) anchors the fiscal-year start for YTD
+span checks and Q4 derivation. Non-calendar fiscal years (Apple's Sept) are
+handled because we key on `(start, end)` + `reportDate`, not calendar quarters.
+`fiscal_year`/`fiscal_period` are stored from the filing's own `fy`/`fp` for its
+primary period (safe — a filing's own context is correct for its current period).
+
+## CIK resolution & tickers backfill
+
+- From `company_tickers.json`: `normalise_symbol(ticker) → cik_str`. Set
+  `tickers.cik` for each tracked ticker (multi-class share one CIK).
+- From `submissions/CIK.json`: `sic`, `sicDescription` → `tickers.sic/sic_description`.
+- Refreshed by the SEC job; a newly added ticker self-resolves on its next run
+  (until then its snapshots are absent → metrics null, per the plan).
+
+## SourceImporter mapping (source = `sec_facts`)
+
+- `detect_revision` → latest daily-index date (or `backfill` sentinel on first run).
+- `stage` → resolve tracked CIKs; (incremental) parse indices since last date →
+  tracked filers → fetch their companyfacts → build per-accession snapshot rows;
+  (backfill) fetch all tracked companyfacts. In-memory staged set (KISS, per A1).
+- `validate` (fail-closed) → tracked-universe **coverage floor** (issuers with ≥1
+  snapshot); **unit/period sanity** (YTD spans within tolerance; EPS in USD/shares);
+  **no duplicate accession**; **filings skipped for missing period identity are
+  counted in `validation_json`** (carry-forward from A1 review); an unexpected
+  companyfacts shape (missing `facts`/`units`) stops promotion.
+- `promote` → upsert snapshot rows keyed by unique `accession`, stamped
+  `import_run_id`; refresh `tickers.cik/sic/sic_description`. Non-destructive
+  (append-only accessions) — no future-row deletion like earnings.
+
+## SEC fair-access (operational, per the plan's non-negotiable)
+
+Identifying `User-Agent` with contact email on every request; well under 10 req/s
+with spacing; exponential backoff on 429; **403 → alert and stop, never
+retry-loop**. New config: `sec_user_agent`, `sec_request_spacing_seconds`,
+`sec_max_retries`. Keep only the last ~2 fetched artifacts on disk for debugging
+(reproducibility is the normalized Postgres rows, per the plan).
+
+## Explicitly out of scope for A3
+
+- `fundamental_data` cutover (A5 parity gate) — snapshots only in A3.
+- The read-time derivation, API object, and panel (A4).
+- Comparative-only restatements (Decision 2).
+- Point-in-time backtest enforcement (`accepted_at` stored, not yet enforced).
+
+## Open questions for sign-off
+
+1. **Decision 1** (daily-index fetch vs the plan's bulk zip) — approve the deviation?
+2. **Decision 2** (primary-period-only; comparative restatements out of scope) — approve?
+3. Backfill depth: seed **full** available history per issuer on first run, or cap
+   at N years (e.g. 5, matching `ohlcv_history_days`)? Full history is cheap to
+   store and powers the quarter tape / YoY.