Stop reading an absent SEC daily index as a fair-access block

The fundamentals import has been dead since 2026-07-25, alerting
SecForbiddenError on form.20260725.idx with "set a real sec_user_agent
contact email". The User-Agent was never the problem.

www.sec.gov/Archives is served from an S3 bucket with no ListBucket
grant, so a MISSING key cannot answer 404 — it returns 403 with S3's
AccessDenied XML. SEC publishes a daily index for business days only, so
2026-07-25 (a Saturday) is simply absent. _get mapped every 403 to the
fatal SecForbiddenError, which made daily_index's `except
SecNotFoundError` unreachable for the exact case it was written for:
the first weekend an incremental walk crossed killed the run, and
last_processed never advanced past Friday.

Latent until activation, not a change at SEC: with no promoted run the
importer takes the backfill path and makes zero daily_index calls, so
the walk was first exercised by the first incremental run.

Verified live 2026-07-30: Sat/Sun 403 with AccessDenied XML while Fri
(51 rows) and Mon (26 rows) return 200 on the same UA; a genuine
rejection is instead the WAF's text/html "Undeclared Automated Tool"
page, served even for files that exist. So the downgrade to "missing" is
gated on all three: the /Archives/ prefix, an XML content type, and
S3's own error code. Every other 403 still alerts and stops.

Missing weekday indexes now log at WARNING — if a rejection page were
ever misread as absent, the importer must not advance past real filings
quietly.

No state to reset: _last_processed_index_date reads promoted runs only,
so the next run walks 2026-07-25..29, skipping the weekend.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-07-30 10:37:26 +02:00
co-authored by Claude Opus 5
parent a7aefa6fe7
commit 5b4fdab85c
3 changed files with 120 additions and 6 deletions
+6 -1
View File
@@ -226,7 +226,12 @@ primary period (safe — a filing's own context is correct for its current perio
Identifying `User-Agent` with contact email on every request; well under 10 req/s
with spacing; exponential backoff on 429; **403 → alert and stop, never
retry-loop**. New config: `sec_user_agent`, `sec_request_spacing_seconds`,
retry-loop** — with one carved-out exception: `www.sec.gov/Archives` is served
from an S3 bucket without a `ListBucket` grant, so an **absent** file 403s with
S3's `AccessDenied` XML rather than 404 (every weekend/holiday daily index does
this). That one shape is read as "missing"; a real rejection is the WAF's
`text/html` "Undeclared Automated Tool" page and still stops the run.
New config: `sec_user_agent`, `sec_request_spacing_seconds`,
`sec_max_retries`. Keep only the last ~2 fetched artifacts on disk for debugging
(reproducibility is the normalized Postgres rows, per the plan).