Stop reading an absent SEC daily index as a fair-access block

The fundamentals import has been dead since 2026-07-25, alerting
SecForbiddenError on form.20260725.idx with "set a real sec_user_agent
contact email". The User-Agent was never the problem.

www.sec.gov/Archives is served from an S3 bucket with no ListBucket
grant, so a MISSING key cannot answer 404 — it returns 403 with S3's
AccessDenied XML. SEC publishes a daily index for business days only, so
2026-07-25 (a Saturday) is simply absent. _get mapped every 403 to the
fatal SecForbiddenError, which made daily_index's `except
SecNotFoundError` unreachable for the exact case it was written for:
the first weekend an incremental walk crossed killed the run, and
last_processed never advanced past Friday.

Latent until activation, not a change at SEC: with no promoted run the
importer takes the backfill path and makes zero daily_index calls, so
the walk was first exercised by the first incremental run.

Verified live 2026-07-30: Sat/Sun 403 with AccessDenied XML while Fri
(51 rows) and Mon (26 rows) return 200 on the same UA; a genuine
rejection is instead the WAF's text/html "Undeclared Automated Tool"
page, served even for files that exist. So the downgrade to "missing" is
gated on all three: the /Archives/ prefix, an XML content type, and
S3's own error code. Every other 403 still alerts and stops.

Missing weekday indexes now log at WARNING — if a rejection page were
ever misread as absent, the importer must not advance past real filings
quietly.

No state to reset: _last_processed_index_date reads promoted runs only,
so the next run walks 2026-07-25..29, skipping the weekend.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

app/services/sec_client.py

@@ -8,7 +8,9 @@ the daily filing index — behind one client that honors SEC's fair-access polic
 - request spacing well under the 10 req/s limit;
 - exponential backoff + retry on 429;
 - **403 → alert and stop** (raise ``SecForbiddenError``), never a retry-loop — a
-  403 means the UA or request pattern is wrong and retrying won't fix it.
+  403 means the UA or request pattern is wrong and retrying won't fix it. The one
+  exception is S3's ``AccessDenied`` on an ``/Archives/`` path, which is how the
+  bucket reports an absent file (``_is_absent_archive_key``).
 
 Parsing lives here (index fixed-width, submissions pagination); DB writes and the
 snapshot mapping live in the importer. No conditional GETs — the companyfacts
@@ -53,11 +55,44 @@ class SecForbiddenError(SecError):
 
 
 class SecNotFoundError(SecError):
-    """SEC returned 404 — the resource does not exist (e.g. no index for a day).
+    """The resource does not exist (e.g. no daily index published for a day).
 
     The *only* error a caller may treat as 'missing' — every other SecError
-    (403, exhausted retries, 5xx, timeout) must propagate so a fetch failure is
-    never mistaken for an empty result."""
+    (fair-access rejection, exhausted retries, 5xx, timeout) must propagate so a
+    fetch failure is never mistaken for an empty result.
+
+    Raised for a 404, and for the one 403 that also means "absent": see
+    ``_is_absent_archive_key``."""
+
+
+def _is_absent_archive_key(url: str, resp: httpx.Response) -> bool:
+    """True when a 403 means "this file does not exist", not "you are blocked".
+
+    ``www.sec.gov/Archives`` is served straight out of an S3 bucket that grants
+    no ``s3:ListBucket``, so a missing key cannot be answered with 404 — S3
+    returns **403 with its ``AccessDenied`` XML** instead. SEC publishes a daily
+    index only for business days, so every weekend and market holiday inside an
+    incremental walk lands on exactly this response (verified 2026-07-30:
+    ``form.20260725.idx``, a Saturday, 403s while the Friday and Monday files
+    return 200 on the same User-Agent).
+
+    A genuine fair-access rejection is distinguishable and must stay fatal: it is
+    SEC's WAF interstitial — ``text/html``, "Your Request Originates from an
+    Undeclared Automated Tool" — and it is returned for files that *do* exist,
+    on any path. Hence the narrow gate: the Archives prefix plus S3's own error
+    document. Nothing else may be downgraded to "missing"."""
+    try:
+        parsed = httpx.URL(url)
+    except (TypeError, ValueError):  # pragma: no cover — url comes from us
+        return False
+    if parsed.host != "www.sec.gov" or not parsed.path.startswith("/Archives/"):
+        return False
+    if "xml" not in resp.headers.get("Content-Type", "").lower():
+        return False
+    try:
+        return "<Code>AccessDenied</Code>" in resp.text
+    except (UnicodeDecodeError, httpx.HTTPError):  # pragma: no cover
+        return False
 
 
 def _looks_like_contact_email(ua: str) -> bool:
@@ -155,6 +190,8 @@ class SecClient:
 
             code = resp.status_code
             if code == 403:
+                if _is_absent_archive_key(url, resp):
+                    raise SecNotFoundError(f"SEC 403/AccessDenied (absent) for {url}")
                 raise SecForbiddenError(
                     f"SEC 403 for {url} — User-Agent/pattern rejected; set a real "
                     "sec_user_agent contact email"
@@ -252,7 +289,16 @@ class SecClient:
         try:
             text = await self.get_text(url)
         except SecNotFoundError:
-            logger.info("no daily index for %s (404)", day)
+            # Absent is normal on a weekend (SEC publishes business days only). On a
+            # weekday it is not: a SEC hiccup — or a rejection page misread as absent
+            # — would otherwise let the importer advance past real filings silently,
+            # so surface it at WARNING instead of hiding it in the info stream.
+            logger.log(
+                logging.INFO if day.weekday() >= 5 else logging.WARNING,
+                "no daily index published for %s (%s)",
+                day,
+                f"{day:%a}",
+            )
             return []  # weekend/holiday/not-yet-published; other errors propagate
         return _parse_form_index(text)