Stop reading an absent SEC daily index as a fair-access block

The fundamentals import has been dead since 2026-07-25, alerting SecForbiddenError on form.20260725.idx with "set a real sec_user_agent contact email". The User-Agent was never the problem. www.sec.gov/Archives is served from an S3 bucket with no ListBucket grant, so a MISSING key cannot answer 404 — it returns 403 with S3's AccessDenied XML. SEC publishes a daily index for business days only, so 2026-07-25 (a Saturday) is simply absent. _get mapped every 403 to the fatal SecForbiddenError, which made daily_index's `except SecNotFoundError` unreachable for the exact case it was written for: the first weekend an incremental walk crossed killed the run, and last_processed never advanced past Friday. Latent until activation, not a change at SEC: with no promoted run the importer takes the backfill path and makes zero daily_index calls, so the walk was first exercised by the first incremental run. Verified live 2026-07-30: Sat/Sun 403 with AccessDenied XML while Fri (51 rows) and Mon (26 rows) return 200 on the same UA; a genuine rejection is instead the WAF's text/html "Undeclared Automated Tool" page, served even for files that exist. So the downgrade to "missing" is gated on all three: the /Archives/ prefix, an XML content type, and S3's own error code. Every other 403 still alerts and stops. Missing weekday indexes now log at WARNING — if a rejection page were ever misread as absent, the importer must not advance past real filings quietly. No state to reset: _last_processed_index_date reads promoted runs only, so the next run walks 2026-07-25..29, skipping the weekend. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 10:37:26 +02:00
parent a7aefa6fe7
commit 5b4fdab85c
3 changed files with 120 additions and 6 deletions
@@ -207,6 +207,69 @@ async def test_only_404_is_treated_as_missing():
        assert await c.latest_index_date(today=date(2026, 7, 22)) is None


+# The two shapes a real SEC 403 takes (captured live 2026-07-30). /Archives is
+# S3-backed with no ListBucket grant, so an ABSENT file comes back as S3's
+# AccessDenied XML; a genuine fair-access rejection is the WAF interstitial.
+S3_ACCESS_DENIED = (
+    '<?xml version="1.0" encoding="UTF-8"?>'
+    "<Error><Code>AccessDenied</Code><Message>Access Denied</Message>"
+    "<RequestId>5AWQBRAEX3NPAPHB</RequestId><HostId>MHFbU0a3k0ER</HostId></Error>"
+)
+WAF_HTML = (
+    "<!DOCTYPE html><html><head><title>SEC.gov | Your Request Originates from "
+    "an Undeclared Automated Tool</title></head><body>...</body></html>"
+)
+
+
+def _forbidden(body: str, content_type: str) -> httpx.Response:
+    return httpx.Response(
+        403, content=body.encode(), headers={"Content-Type": content_type}
+    )
+
+
+async def test_archives_access_denied_is_absent_not_forbidden():
+    # SEC publishes no daily index on weekends, and the bucket reports the absent
+    # key as 403/AccessDenied. Treating that as fatal wedged the importer on the
+    # first Saturday of an incremental walk (2026-07-25); it must read as "missing".
+    def handler(request: httpx.Request) -> httpx.Response:
+        url = str(request.url)
+        if url.endswith("QTR2/index.json"):
+            return httpx.Response(200, json={"directory": {"item": [{"name": "form.20260630.idx"}]}})
+        return _forbidden(S3_ACCESS_DENIED, "application/xml")
+
+    def client() -> SecClient:
+        return SecClient(
+            transport=httpx.MockTransport(handler), spacing_seconds=0, max_retries=0
+        )
+
+    async with client() as c:
+        assert await c.daily_index(date(2026, 7, 25)) == []
+    async with client() as c:
+        # QTR3 absent → the previous-quarter fallback now actually fires.
+        assert await c.latest_index_date(today=date(2026, 7, 22)) == date(2026, 6, 30)
+
+
+async def test_archives_waf_rejection_stays_forbidden():
+    # A real UA/pattern rejection is served for files that DO exist — never
+    # downgrade it, or a blocked run would look like an empty index.
+    def handler(request: httpx.Request) -> httpx.Response:
+        return _forbidden(WAF_HTML, "text/html")
+
+    async with SecClient(transport=httpx.MockTransport(handler), spacing_seconds=0) as c:
+        with pytest.raises(SecForbiddenError):
+            await c.daily_index(date(2026, 7, 21))
+
+
+async def test_access_denied_outside_archives_stays_forbidden():
+    # The downgrade is gated on the Archives prefix; data.sec.gov is not S3-backed.
+    def handler(request: httpx.Request) -> httpx.Response:
+        return _forbidden(S3_ACCESS_DENIED, "application/xml")
+
+    async with SecClient(transport=httpx.MockTransport(handler), spacing_seconds=0) as c:
+        with pytest.raises(SecForbiddenError):
+            await c.companyfacts(320193)
+
+
 async def test_fair_access_validation_on_real_client():
    # Placeholder email rejected.
    with pytest.raises(SecError):