CI only lints app/, so 11 findings had accumulated in tests/ and scripts/.
Mechanical and behaviour-neutral, but two were not auto-fixable and needed a
judgement call rather than `ruff --fix`:
- E741 in run_fip_breadth_diagnostics: `l` is the OHLCV low and is genuinely
used, so this was a naming fix (`l` -> `lo`), not a deletion.
- F841 in the same file: `vol_ix`/`momr_ix` are assigned from a pure local
`_index()` and never read, so removing them cannot change any output. Their
upstream `vol_weeks`/`momr_weeks` maps *are* used further down and stay; the
comment above `_index` was corrected to say so.
The rest are unused imports and f-strings without placeholders (literal
markdown table headers, so identical output).
Verified beyond the linter, since py_compile does not catch a removed-but-used
import: every removed symbol has zero remaining references, all scripts compile,
and the full unit suite passes (852 passed, 1 skipped).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The fundamentals import has been dead since 2026-07-25, alerting
SecForbiddenError on form.20260725.idx with "set a real sec_user_agent
contact email". The User-Agent was never the problem.
www.sec.gov/Archives is served from an S3 bucket with no ListBucket
grant, so a MISSING key cannot answer 404 — it returns 403 with S3's
AccessDenied XML. SEC publishes a daily index for business days only, so
2026-07-25 (a Saturday) is simply absent. _get mapped every 403 to the
fatal SecForbiddenError, which made daily_index's `except
SecNotFoundError` unreachable for the exact case it was written for:
the first weekend an incremental walk crossed killed the run, and
last_processed never advanced past Friday.
Latent until activation, not a change at SEC: with no promoted run the
importer takes the backfill path and makes zero daily_index calls, so
the walk was first exercised by the first incremental run.
Verified live 2026-07-30: Sat/Sun 403 with AccessDenied XML while Fri
(51 rows) and Mon (26 rows) return 200 on the same UA; a genuine
rejection is instead the WAF's text/html "Undeclared Automated Tool"
page, served even for files that exist. So the downgrade to "missing" is
gated on all three: the /Archives/ prefix, an XML content type, and
S3's own error code. Every other 403 still alerts and stops.
Missing weekday indexes now log at WARNING — if a rejection page were
ever misread as absent, the importer must not advance past real filings
quietly.
No state to reset: _last_processed_index_date reads promoted runs only,
so the next run walks 2026-07-25..29, skipping the weekend.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Pure parser (no I/O/DB) turning one issuer's companyfacts + submissions filing
metadata into per-accession snapshot rows for the filing's primary period.
- Period identity from end == reportDate, never fy/fp (fy/fp is the filing's
context; comparatives repeat it).
- Duration facts stored as cumulative YTD: pick the fact whose span matches the
fiscal-period-to-date length (Q1~3mo..FY~12mo) within tolerance; no YTD-length
fact -> null (never a discrete masquerading as YTD).
- Balance-sheet instants at end == reportDate; shares_outstanding is the dei
cover-page fact whose own end (cover date) is stored in shares_outstanding_date.
- Cash and debt composites are aggregate-first and mutually exclusive (each
source tag counted at most once).
- Carries filing_date through submissions rows (snapshot.filed_date).
Verified on REAL Apple companyfacts: 44 snapshots, 0 skipped, YTD revenue
124.3B->219.7B->313.7B->416.2B across FY2025 (Q4 derives at read time), every
shares_date is the cover date != period_end. Tests: 6 fixture + 1 skip-guarded
live-invariants (monotonic YTD, cover-date shares).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Addresses the slice-1 review:
1. Resolution is now read-only (A1 transaction contract). resolve_ciks /
fetch_sic_updates compute proposals and mutate nothing; a new
apply_ticker_updates issues the writes, called only in promote — so a failed
validation can't leak ticker changes on the framework's failure commit.
2. Only 404 means "missing". Added SecNotFoundError; daily_index /
latest_index_date catch only that. 403, exhausted 429, 5xx, timeouts, and
transport/parse errors now propagate instead of looking like "no index".
3. Fair-access enforced when opening a REAL client (transport=None): reject
blank/placeholder/non-email User-Agent and sub-0.11s spacing. Mock transports
skip it (tests use 0 spacing).
4. submissions(include_history=False) by default — only the one-time full
backfill fetches the history shards; SIC/incremental work makes no extra
requests.
Plus: retry transient 5xx/network errors and honor Retry-After during the 1 GB
backfill; compose_revision rejects a missing index date (no "None:..." revision).
Re-verified live vs real SEC (fair-access validation passes, shard merge intact).
Tests: 18 (added error propagation, read-only resolution, fair-access, recent-only
submissions, reject-None revision).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
First A3 implementation checkpoint (design: docs/dolt-sec-a3-design.md).
- sec_client.py: async SEC EDGAR client honoring fair-access — identifying
User-Agent (config), request spacing < 10 req/s, exponential backoff on 429,
and 403 -> SecForbiddenError (alert and stop, never retry-loop). Fetchers:
company_tickers (normalised, multi-class share CIK), submissions (merges the
paginated filings.files shards so full history is visible), companyfacts,
daily_index (fixed-width form.idx parse), latest_index_date.
- sec_universe.py: resolve_ciks (tickers.cik backfill), refresh_sic
(sic/sic_description), and the composite-revision pieces — universe_fingerprint
(a new ticker changes the revision, so it's never no_op'd/starved),
index_content_hash, compose_revision.
- config + .env.example: SEC_USER_AGENT (must be a real contact email) + spacing
/ retries / timeout.
Verified live against real SEC: AAPL->320193, GOOG==GOOGL, BRK-B resolved;
submissions shard-merge proven (131 filings back to 1993); daily index parsed.
Tests: 11 (mocked-transport parsing + 403/429 handling + resolution/fingerprint).
Full suite 713 passed. Next slice: companyfacts -> snapshot parser + importer.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
dennisthiessenandClaude Opus 5