The fundamentals import has been dead since 2026-07-25, alerting
SecForbiddenError on form.20260725.idx with "set a real sec_user_agent
contact email". The User-Agent was never the problem.
www.sec.gov/Archives is served from an S3 bucket with no ListBucket
grant, so a MISSING key cannot answer 404 — it returns 403 with S3's
AccessDenied XML. SEC publishes a daily index for business days only, so
2026-07-25 (a Saturday) is simply absent. _get mapped every 403 to the
fatal SecForbiddenError, which made daily_index's `except
SecNotFoundError` unreachable for the exact case it was written for:
the first weekend an incremental walk crossed killed the run, and
last_processed never advanced past Friday.
Latent until activation, not a change at SEC: with no promoted run the
importer takes the backfill path and makes zero daily_index calls, so
the walk was first exercised by the first incremental run.
Verified live 2026-07-30: Sat/Sun 403 with AccessDenied XML while Fri
(51 rows) and Mon (26 rows) return 200 on the same UA; a genuine
rejection is instead the WAF's text/html "Undeclared Automated Tool"
page, served even for files that exist. So the downgrade to "missing" is
gated on all three: the /Archives/ prefix, an XML content type, and
S3's own error code. Every other 403 still alerts and stops.
Missing weekday indexes now log at WARNING — if a rejection page were
ever misread as absent, the importer must not advance past real filings
quietly.
No state to reset: _last_processed_index_date reads promoted runs only,
so the next run walks 2026-07-25..29, skipping the weekend.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1. Removed the 45-day index-walk cap: it discarded the older part of a long
outage while still advancing source_max_date, permanently losing filings.
The walk now covers every unprocessed date (a large gap is one-time cost).
2. Discrepancy detection meets the immutability contract: it compares ALL source
snapshot fields (not five), read-only during stage/validate, reports the
differing accessions + fields in validation_json, and promote emits a warning
system event (in-transaction) — never mutating the stored row.
3. Malformed companyfacts (missing facts/units structure) are recorded separately
and FAIL validation, instead of silently degrading to skipped rows that the
50% backfill coverage floor could still pass.
Also corrected the stale "sum share classes" / DEI-only wording in the snapshot
model docstring and the A3 design doc to describe the us-gaap fallback.
Tests: +4 regressions (>45-day gap loses nothing, newly-added issuer backfills
without filing, malformed payload fails, shares discrepancy detected + evented).
23 passed, 1 skipped.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fold in the A3 design review:
1. Composite revision = latest-index-date + index-content-hash + tracked
symbol->CIK fingerprint, so a newly added ticker forces a run instead of
being no_op'd/starved. No backfill sentinel — absence of a prior promoted
run triggers backfill; source_max_date records the processed index date.
2. Full history needs the paginated submissions shards: filings.recent caps at
1000; older accessions (reportDate/acceptanceDateTime/isXBRL) live in
filings.files[] shards (verified on Apple: recent=1000, one 1994-2015 shard).
3. Index<->Company-Facts consistency gate: they are separate SEC products that
can lag; for every tracked isXBRL index accession, confirm it exists in
Company Facts before promotion, else fail+retry (never record a null/partial
snapshot). Non-XBRL amendments skipped with a recorded reason.
4. Immutable = insert-only (ON CONFLICT DO NOTHING); a differing re-fetch is a
reported discrepancy, never a silent mutation / import_run_id replacement.
Plus deterministic, mutually-exclusive cash/debt composition (aggregate-first;
each source tag counted at most once).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
All three decisions approved: fetch via EDGAR daily-index (not bulk zip),
one snapshot per accession for its primary period (comparative-only
restatements out of scope), full-history backfill on first run. Doc status
flipped to approved / ready to implement.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Design (not implementation) for phase A3, grounded in live SEC data probes.
Key findings: fp has no Q4 (derive it); fy/fp are the filing's context not each
fact's period (select by end==reportDate); SEC provides both discrete and YTD
facts (confirms stored-YTD schema); companyfacts endpoint has no ETag/
Last-Modified (conditional GET impossible); tickers are dash-form and GOOG/GOOGL
share one CIK.
Two plan deviations flagged for sign-off:
1. Fetch via the EDGAR daily-index (fetch companyfacts only for tracked issuers
that filed) rather than the multi-GB bulk zip — lighter and restores the
revision/no_op model.
2. One snapshot row per accession for its primary period (YTD-cumulative);
comparative-only restatements out of scope (only real 10-K/A updates a period).
Plus a metric tag catalog, read-time derivation rules (missing period -> null),
CIK resolution, validation gates, and SEC fair-access handling.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
dennisthiessenandClaude Opus 5