Stop reading an absent SEC daily index as a fair-access block
The fundamentals import has been dead since 2026-07-25, alerting SecForbiddenError on form.20260725.idx with "set a real sec_user_agent contact email". The User-Agent was never the problem. www.sec.gov/Archives is served from an S3 bucket with no ListBucket grant, so a MISSING key cannot answer 404 — it returns 403 with S3's AccessDenied XML. SEC publishes a daily index for business days only, so 2026-07-25 (a Saturday) is simply absent. _get mapped every 403 to the fatal SecForbiddenError, which made daily_index's `except SecNotFoundError` unreachable for the exact case it was written for: the first weekend an incremental walk crossed killed the run, and last_processed never advanced past Friday. Latent until activation, not a change at SEC: with no promoted run the importer takes the backfill path and makes zero daily_index calls, so the walk was first exercised by the first incremental run. Verified live 2026-07-30: Sat/Sun 403 with AccessDenied XML while Fri (51 rows) and Mon (26 rows) return 200 on the same UA; a genuine rejection is instead the WAF's text/html "Undeclared Automated Tool" page, served even for files that exist. So the downgrade to "missing" is gated on all three: the /Archives/ prefix, an XML content type, and S3's own error code. Every other 403 still alerts and stops. Missing weekday indexes now log at WARNING — if a rejection page were ever misread as absent, the importer must not advance past real filings quietly. No state to reset: _last_processed_index_date reads promoted runs only, so the next run walks 2026-07-25..29, skipping the weekend. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -226,7 +226,12 @@ primary period (safe — a filing's own context is correct for its current perio
|
||||
|
||||
Identifying `User-Agent` with contact email on every request; well under 10 req/s
|
||||
with spacing; exponential backoff on 429; **403 → alert and stop, never
|
||||
retry-loop**. New config: `sec_user_agent`, `sec_request_spacing_seconds`,
|
||||
retry-loop** — with one carved-out exception: `www.sec.gov/Archives` is served
|
||||
from an S3 bucket without a `ListBucket` grant, so an **absent** file 403s with
|
||||
S3's `AccessDenied` XML rather than 404 (every weekend/holiday daily index does
|
||||
this). That one shape is read as "missing"; a real rejection is the WAF's
|
||||
`text/html` "Undeclared Automated Tool" page and still stops the run.
|
||||
New config: `sec_user_agent`, `sec_request_spacing_seconds`,
|
||||
`sec_max_retries`. Keep only the last ~2 fetched artifacts on disk for debugging
|
||||
(reproducibility is the normalized Postgres rows, per the plan).
|
||||
|
||||
|
||||
Reference in New Issue
Block a user