Pure parser (no I/O/DB) turning one issuer's companyfacts + submissions filing
metadata into per-accession snapshot rows for the filing's primary period.
- Period identity from end == reportDate, never fy/fp (fy/fp is the filing's
context; comparatives repeat it).
- Duration facts stored as cumulative YTD: pick the fact whose span matches the
fiscal-period-to-date length (Q1~3mo..FY~12mo) within tolerance; no YTD-length
fact -> null (never a discrete masquerading as YTD).
- Balance-sheet instants at end == reportDate; shares_outstanding is the dei
cover-page fact whose own end (cover date) is stored in shares_outstanding_date.
- Cash and debt composites are aggregate-first and mutually exclusive (each
source tag counted at most once).
- Carries filing_date through submissions rows (snapshot.filed_date).
Verified on REAL Apple companyfacts: 44 snapshots, 0 skipped, YTD revenue
124.3B->219.7B->313.7B->416.2B across FY2025 (Q4 derives at read time), every
shares_date is the cover date != period_end. Tests: 6 fixture + 1 skip-guarded
live-invariants (monotonic YTD, cover-date shares).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Addresses the slice-1 review:
1. Resolution is now read-only (A1 transaction contract). resolve_ciks /
fetch_sic_updates compute proposals and mutate nothing; a new
apply_ticker_updates issues the writes, called only in promote — so a failed
validation can't leak ticker changes on the framework's failure commit.
2. Only 404 means "missing". Added SecNotFoundError; daily_index /
latest_index_date catch only that. 403, exhausted 429, 5xx, timeouts, and
transport/parse errors now propagate instead of looking like "no index".
3. Fair-access enforced when opening a REAL client (transport=None): reject
blank/placeholder/non-email User-Agent and sub-0.11s spacing. Mock transports
skip it (tests use 0 spacing).
4. submissions(include_history=False) by default — only the one-time full
backfill fetches the history shards; SIC/incremental work makes no extra
requests.
Plus: retry transient 5xx/network errors and honor Retry-After during the 1 GB
backfill; compose_revision rejects a missing index date (no "None:..." revision).
Re-verified live vs real SEC (fair-access validation passes, shard merge intact).
Tests: 18 (added error propagation, read-only resolution, fair-access, recent-only
submissions, reject-None revision).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
First A3 implementation checkpoint (design: docs/dolt-sec-a3-design.md).
- sec_client.py: async SEC EDGAR client honoring fair-access — identifying
User-Agent (config), request spacing < 10 req/s, exponential backoff on 429,
and 403 -> SecForbiddenError (alert and stop, never retry-loop). Fetchers:
company_tickers (normalised, multi-class share CIK), submissions (merges the
paginated filings.files shards so full history is visible), companyfacts,
daily_index (fixed-width form.idx parse), latest_index_date.
- sec_universe.py: resolve_ciks (tickers.cik backfill), refresh_sic
(sic/sic_description), and the composite-revision pieces — universe_fingerprint
(a new ticker changes the revision, so it's never no_op'd/starved),
index_content_hash, compose_revision.
- config + .env.example: SEC_USER_AGENT (must be a real contact email) + spacing
/ retries / timeout.
Verified live against real SEC: AAPL->320193, GOOG==GOOGL, BRK-B resolved;
submissions shard-merge proven (131 filings back to 1993); daily index parsed.
Tests: 11 (mocked-transport parsing + 403/429 handling + resolution/fingerprint).
Full suite 713 passed. Next slice: companyfacts -> snapshot parser + importer.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
dennisthiessenandClaude Opus 4.8