A weekday with no published index is usually just a market holiday
(~10/yr), not a fault. The WARNING still earns its place as the guard
against inferring missing from an error code, but the comment should
not claim more than it can.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The fundamentals import has been dead since 2026-07-25, alerting
SecForbiddenError on form.20260725.idx with "set a real sec_user_agent
contact email". The User-Agent was never the problem.
www.sec.gov/Archives is served from an S3 bucket with no ListBucket
grant, so a MISSING key cannot answer 404 — it returns 403 with S3's
AccessDenied XML. SEC publishes a daily index for business days only, so
2026-07-25 (a Saturday) is simply absent. _get mapped every 403 to the
fatal SecForbiddenError, which made daily_index's `except
SecNotFoundError` unreachable for the exact case it was written for:
the first weekend an incremental walk crossed killed the run, and
last_processed never advanced past Friday.
Latent until activation, not a change at SEC: with no promoted run the
importer takes the backfill path and makes zero daily_index calls, so
the walk was first exercised by the first incremental run.
Verified live 2026-07-30: Sat/Sun 403 with AccessDenied XML while Fri
(51 rows) and Mon (26 rows) return 200 on the same UA; a genuine
rejection is instead the WAF's text/html "Undeclared Automated Tool"
page, served even for files that exist. So the downgrade to "missing" is
gated on all three: the /Archives/ prefix, an XML content type, and
S3's own error code. Every other 403 still alerts and stops.
Missing weekday indexes now log at WARNING — if a rejection page were
ever misread as absent, the importer must not advance past real filings
quietly.
No state to reset: _last_processed_index_date reads promoted runs only,
so the next run walks 2026-07-25..29, skipping the weekend.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The fundamentals import had been failing for three days on two tracked
filings the index listed but Company Facts appeared not to have. They were
not lagging: SEC filed the XBRL of NEE's and DOW's 2026-07-24 combined
parent/subsidiary 10-Qs under the co-registrant's CIK (Florida Power &
Light, Dow Chemical), so the ticker-carrying filer's own facts file never
receives that accession. This does not self-correct - an NEE filing
misattributed the same way in 2014 is still misfiled.
Because source_max_date advances only on a promoted run, the failure was
self-perpetuating: every later run re-walked the same index day and re-hit
the same two filings.
- Recover from the co-registrant file. The daily index lists every
co-registrant of an accession, which is the only pointer to where the
facts actually landed. Rows are re-stamped to the real filer, since
parse_snapshots stamps the CIK of the payload it read.
- Guard the recovery with a share-count continuity check against the
issuer's own history, so a subsidiary's standalone facts can never be
stored as the parent's. No history, no recovery.
- Bound the blocking: a filing still unresolvable after
MISSING_XBRL_RETRY_DAYS promotes with a named unresolved_filing warning
instead of wedging every later import.
- Name the offending filings in the alert and record them in
validation_json, separating not_in_companyfacts from not_in_submissions.
The gate previously reported a count and discarded the accessions.
Verified against live SEC data: both filings recover with the correct CIK
and the guard rejects a mismatched reference.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A conditional clause in the summary sentence carried an f prefix with no
placeholders, failing `ruff check app/` and blocking the deploy. Literal only;
the rendered text is unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The v3 cutover run scored 2/4 corrections warned against v2's 3/4, which reads
like a regression and is not one. Only 4 of the 11 detected corrections fall in
the holdout, so recall is one event from a different headline -- and the event
that flips is decided by threshold placement, not by what the score saw. "v3
without the credit sensor" catches 2025-02-21 at a *higher* threshold (35.5)
than shipped v3 misses it at (32.3), because the alarm rule needs a rising edge
and a lower threshold can fire outside the horizon then never reset below.
Two caveats are now computed and surfaced rather than left for the reader to
infer:
- Holdout event count against MIN_EVENTS_FOR_CONFIDENCE. The summary sentence
states how many of the detected corrections actually fall in the test period.
- Warning-sensor coverage across the split. The score renormalises over what is
available, so a training window predating a sensor's history freezes the
threshold on a different construct than the holdout is measured against. At
the cutover that is 39% of training sessions with all three sensors versus
100% of the test period, credit history beginning 2023-07-25.
Restricting the threshold to sensor-matched training sessions was tested and
rejected: those sessions are a calm recent stretch, so the threshold falls from
32.3 to 22.5 and false alarms rise from 3.3 to 8.6/yr. It swaps a coverage bias
for a regime-selection bias. The report states its limits instead.
_warning_series now returns per-session sensor counts alongside the scores.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The LLM-sourced capex/earnings observations carried 12+8 of 100 Warning points,
so both pegged at 100 produced a Warning of 20.0 -- below the event study's 25.3
alarm threshold and still inside the "stable" band. The reading was
arithmetically incapable of changing anything on screen, which is why refreshing
it appeared to do nothing. They are now a qualitative overlay reported beside
the scores rather than diluted into them.
Calibrated against the 408 v2 sessions to 2026-07-24, reproduced offline from
Alpaca + FRED; the harness matched the stored prod distribution exactly before
any parameter was changed.
State:
- P3 used dd_pct * 5, reaching 100 at a 20% drawdown -- the 90th percentile of
the observed distribution -- so 39/408 sessions sat at exactly 100 with no
resolution left during the part of a selloff that matters most. Replaced with
anchored breakpoints keeping headroom past the observed 36% maximum, blended
2:1 like P1/P2 instead of max(). P3's realized share of State falls from 65%
to 40%, matching its nominal weight.
- Credit level is now anchors-only. ICE capped FRED's BAMLH0A0HYM2 at a rolling
3-year window in April 2026, silently turning the 10-year percentile leg into
a 3-year one that scored 20 points of stress at an OAS of 3.5 -- the level its
own anchors call "mild". The anchors already encode the long-run distribution.
Warning:
- Added HY OAS 20-session widening (25%). The level is pinned at zero below the
3.5 anchor; its rate of change is not.
- Divergence tapers to a 0.35 floor instead of a hard price_ret >= 0 gate, which
zeroed the sensor through every decline: on 2026-07-24 the basket shed 10
points of participation in 20 sessions and Warning printed exactly 0.
- The event study and the live monitor now share one sensor definition, so they
cannot silently drift apart.
Bands are per axis (State 20/50/80, Warning 20/40/60) with quadrant dividers at
50/40; v2 Warning never exceeded 64.9 against a shared 60, leaving that half of
the quadrant unreachable. Realized shares: State 73/15/8/3%, Warning 69/20/8/3%.
Snapshots now record credit_history_days and vix_history_days -- the percentile
defect went unnoticed for months because nothing asserted the window the code
claimed.
Cutover: the first run rebuilds 400 sessions automatically; the Event Study job
must be re-run, as its cached report self-invalidates on the methodology check.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The parity gate has been exercised: nine-pass investigation, fixes, two prod
reparses, and an after-report at 504/511 candidate coverage with agreement
unchanged. What remains of workstream A is the activation itself (step c, the
post-approval fundamental_data refresh — not yet implemented) and A6
decommissioning, both now specified in a handoff section with the caveats the
next implementer must carry (KLAC splits, BRK-B, FITB, the 25% guard, CIK
overrides, reparse-after-parser-changes).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Closes the parity investigation: prod reparse runs 6+7 reconciled against the
dry run, the collision check that caught the BEN regression (and its residue --
36 historical 53-week-drift rows, deliberately left), and the 2026-07-24 parity
report diffed against the 2026-07-23 baseline. Candidate coverage 482 -> 504 of
511 with the gap fully explained (PSKY/Q new registrants, FITB guard-tripped
with no revenue), agreement unchanged where both sides exist, and the remaining
deltas are documented definition differences. Includes the after-report and a
correction to the seventh pass (FITB loses its score, not just one input).
Recommends approving the A5 cutover with KLAC as the one carried caveat.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Regression found by the post-reparse collision check. _period_identity trusted
submissions.fiscalYearEnd, which is not reliable: Franklin Resources (BEN)
declares 1231 while every one of its 10-Ks ends 09-30.
The effect was data loss, not just a bad label. BEN's real fiscal Q1 (Dec 31)
sat 0 days from the claimed year end, matching no quarter band, so it fell back
to SEC's fy/fp; its fiscal Q2 (Mar 31) computed 275 days out and was labelled
Q1. Both landed on the same key, the collision discarded one, and BEN lost TTM
EPS and revenue growth entirely — values it had before this branch.
A 10-K's reportDate IS the fiscal year end by definition, so resolve_fiscal_
year_end() now prefers the issuer's most recent annual filing and treats the
declared value as a fallback for issuers with no 10-K in the set.
Scanned the full tracked universe: 2 of 506 issuers declare a year end more
than 21 days from their own 10-K — BEN (91d, broken) and DELL (29d, mislabelled
but functionally correct). Both now derive correctly and match the legacy
provider: BEN revenue growth 3.8243 vs 3.82, DELL 38.5735 vs 38.57. Controls
(AAPL, COST, PEP, DPZ, IRM, JPM, CRM, STX, AVY) byte-identical.
826 unit tests pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Found in review. _merge_amendments rebuilds a period from _MERGED_FIELDS +
_CARRIED_FIELDS alone, so a column in neither list is absent from the merged
row, not just stale — and callers read it with getattr(..., None), which
silently yields None. weighted_avg_diluted_shares was never added when the
market-cap fallback landed (_SNAPSHOT_COLS in the importer was updated, its
counterpart in the derivation was not).
The failure needed both of this branch's fixes at once: a multi-class issuer
with a partial amendment on its latest period (META with a Part-III-only
10-K/A) would silently lose market cap and FCF yield again.
Adds the field, a regression test for that case, and a guard test asserting
the merge/carry lists cover every SnapshotRow field, so the next column added
fails loudly rather than losing data quietly. Confirmed the guard catches the
original bug.
Also from review:
- Expose pe_caveat in the valuation payload, so a P/E suppressed by split
contamination says why instead of looking like missing data (the caveat was
set but never read).
- no_xbrl_filings now names both causes; the old text advised pinning a CIK
override, which is wrong for a genuine new registrant that simply has not
filed yet and clears itself.
- Document that fiscalYearEnd is the issuer's current calendar, so a fiscal-
year-end change degrades old periods (fallback/newest-wins), not current ones.
- Parser-level tests for _select_weighted_avg_shares (shortest-span-wins and
concept priority), which only had derivation-level coverage.
823 unit tests pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Nine-pass investigation of the 2026-07-23 A5 parity report: for each coverage
gap and wrong value, the root cause traced against live SEC company facts, the
fix, and its live-data validation. Also records the decisions taken (weighted-
average share fallback, keep ASC-606 revenue basis, basic-EPS fallback, keep the
25% split-guard threshold) and what remains genuinely unfixable from this data
(KLAC post-filing split, BRK-B dimensional share count). Includes the source
parity report the findings analyse.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Operational plumbing to land the parser fixes and to make silent resolution
failures visible.
- Reparse: SecFundamentalsImporter(reparse=True) restages every accession with
the current parser and rewrites the ones that now reconstruct differently,
writing the full column set so a row is never half old-parse. Snapshots stay
immutable with respect to SEC; the stored row is our reconstruction, and after
a parser fix keeping it is a stale cache, not history. run_import(force=True)
bypasses the unchanged-revision no-op, since the staleness is on our side, not
the source's. Exposed as scripts/reparse_fundamentals.py, dry-run by default.
- CIK overrides: sec_universe reads a {symbol: cik} pin from
SystemSetting['sec_cik_overrides'], applied ahead of company_tickers.json, for
when SEC maps a ticker to a successor shell with no filings (XOM -> a zero-
filing "ExxonMobil Holdings Corp" while every 10-K/Q is under CIK 34088).
- Resolution validation: a tracked issuer resolving to a registrant with no XBRL
filings now records no_xbrl_filings and raises a warning naming the CIKs and
the override setting, instead of silently yielding nothing on every run.
- _diff_fields compares datetime instants, not representations: accepted_at
round-trips naive from SQLite but tz-aware from Postgres, which otherwise made
a reparse of identical data report every row as changed (and false-positived
the pre-existing discrepancy warning).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The A5 parity report surfaced coverage gaps and wrong values that all traced
to the SEC facts parser and read-time derivation rather than to bad source
data. Fixes, each validated by replaying the production parser + derivation
against live company facts:
- Period identity is derived from period_end against the issuer's fiscal
calendar, not SEC's fy/fp fields, which collide (two period ends on one key,
one silently discarded) and invert (a period sorting before one that precedes
it) often enough to break the quarter chain. Recovers BXP, CRM, CRWD, FRT,
MTD, NTAP, PPL, STX, WDAY. Fixed labels are internal ordering keys only (not
in any API schema), so a filer whose year ends in early January shifting by
one is harmless.
- Revenue concept list gains RevenuesNetOfInterestExpense (banks) and the
IncludingAssessedTax variant (REITs/consumer); EPS gains the continuing-ops
variant (REG/FCX) and, last, basic EPS for a period tagging no diluted
variant at all (PPL). All appended, so any issuer that already resolved keeps
its concept.
- YTD span tolerance 20 -> 25 days, covering 4-4-5 retail calendars whose
36-week YTD-Q3 (251-252d) previously missed by ~2 (COST, PEP, DPZ).
- Amendment resolution is per field: a partial 10-K/A (Part III only, no
financial facts) no longer blanks the period (DVN).
- TTM diluted EPS is suppressed when a split contaminates the trailing window
(BKNG's mixed-unit sum produced a P/E of 1.10 that clamped to a perfect
fundamental sub-score). A post-filing split with no share-count evidence
(KLAC) remains undetectable from this data.
- Multi-class share fallback: weighted_avg_diluted_shares is captured and used
for market cap when the cover-page count is absent (dimensional, so missing
from company facts for META/CMCSA/CHTR/FOXA/NWSA/LEN). Within ~0.6% of the
true count on controls; flagged shares_estimated in the API. BRK-B has no
weighted-average fact either and stays unavailable.
820 unit tests pass; new tests confirmed to fail against the pre-fix code.
Effect is inert until existing rows are reparsed (see reparse path).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
max-w-md (448px) under-sized the panel vs its real full-width tab placement and
never triggered the desktop two-column layout. Widen to max-w-3xl; note to
resize to ~390px for the mobile (single-column) check.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the four-cell quarter tape (too many equally-weighted numbers) with one
comparison rail per metric, so the panel answers "improving? sound? fairly
valued?" instead of asking the reader to decode it.
- Operating trend (revenue/EPS growth, operating/FCF margin, share count): a rail
centered on a truthful reference — prior quarter (growth), prior-period average
(margins), or zero (share count) — with a dot at the current delta and a bar
back to the reference, plus a shaded neutral band (backend's +-2pp / +-1pp /
+-1% rules). Value, read, reference label, and signed delta stay visible;
per-quarter history drops out of the default view.
- Valuation & balance (net debt/EBITDA, P/E, FCF yield): a 0-100 favorable-
percentile rail with the peer median fixed at 50; right is always more
favorable (percentile is polarity-aware). median + peer_count shown.
- Not a progress bar: reference line, not a 100% target.
- Horizon tokens: cyan #6EC9DB favorable / coral #EF9182 adverse / #5D6373 track,
replacing emerald/rose. Two columns on desktop, single column (rows stack) on
mobile. Null -> n/a with no rail; insufficient peers -> "peers n/a", no track.
- Kept: compact earnings line, provenance footer, local-date parsing, aria-labels
on every rail. tsc -b passes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Addresses the static review + adds a dev-only visual harness:
1. Tape no longer overflows narrow mobile: each row stacks (label + read on one
line, cells below) under sm, keeping the single-line grid on desktop.
2. Date-only strings (earnings, price_date) are parsed as LOCAL calendar dates,
so a viewer west of UTC no longer sees the previous day.
3. Peer context is visible ("med X · Np") on every width and the percentile
strip carries a full aria-label — no longer hover-only / desktop-only.
4. Per-metric provenance + freshness surfaced (SEC filings · latest quarter,
filed date) replacing the removed panel-wide FMP label.
5. Same-day earnings render "today", not "in 0d".
Harness: frontend/harness.html + src/dev/harness.tsx (dev-only, served at
/harness.html by vite, not in the production build) render full /
partial-insufficient-peer / empty fixtures for desktop + ~390px review.
tsc -b passes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reshapes FundamentalsPanel to consume the additive API v1, within the app's
existing dark-glass language.
- types.ts updated to the exact v1 shape (metrics/earnings/valuation/reads +
legacy fields preserved).
- The quarter tape is the single distinctive device: per-metric 4-cell tape
(revenue/EPS growth, operating + FCF margin, share count) with the latest cell
toned by the deterministic read; color is always paired with the read text.
- Restrained peer strips for Net debt/EBITDA, P/E, FCF yield: value + a
polarity-aware percentile bar with a median marker + the read; hidden ("peers
n/a") when industry is null (< 5 peers).
- Earnings: next date/session/countdown + last-N beat/miss arrows (▲/▼/·) with
text aria-labels; explicit "no date" state.
- Explicit n/a, insufficient-peer, and no-earnings states; header shows the
deterministic sentence. Removed the hard-coded "FMP" source label.
Frontend tsc -b passes; backend suite 778 passed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- The eps_growth_yoy read was never computed, leaving that fixed by_key entry
null even with sufficient EPS history; now growth_read() is applied to EPS
history just like revenue.
- Tests: same-day earnings returns as next with days_until 0 (and not in
recent); zero close guards valuation to null; eps read populated. Fixture
seeds three fiscal years so YoY growth reads have a >=3 run. 9 API tests pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1. Multi-class subject is priced by the REQUESTED ticker: the peer group's
representative for the subject CIK is overridden to the requested ticker_id
(other issuers pick a deterministic-by-symbol rep), so GOOGL's P/E uses
GOOGL's price, not GOOG's. Differing-price GOOG/GOOGL test added.
2. reads matches the selected contract: header is null when there is no read;
by_key is a fixed map over every metric key plus pe and fcf_yield, null when
unavailable (was a sparse dict).
3. Earnings use the New York calendar date; same-day is UPCOMING (days_until 0),
recent is strictly earlier.
4. Valuation is null when there is no usable price (> 0 required for P/E and
market cap); when present, price_date is non-null.
Added a real router/API-envelope test with a seeded legacy record (the endpoint,
not just the schema merge). 6 tests pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
GET /fundamentals/{symbol} now returns the additive v1 objects alongside the
unchanged legacy fields (no legacy growth mapped onto the SEC TTM metric).
- earnings: next (date/session/days_until) + recent (<=4, with surprise_pct)
from earnings_events.
- metrics: fixed key set (value + dated history + per-metric SIC-peer industry
object + source=sec); net_debt has no industry (size-dependent).
- valuation: P/E, FCF yield, market_cap_est computed at REQUEST TIME from the
derived TTM inputs x the latest ohlcv close (no stored valuation); guarded to
null on missing/invalid inputs; pe_industry / fcf_yield_industry peer stats.
- reads: deterministic outputs in a SEPARATE object (header + per-metric reads).
Peer queries are batched and CIK-deduplicated by 2-digit SIC; industry omitted
below 5 valid peers. Schema extended with optional typed sub-models; the router
merges legacy + v1 so every existing field is preserved.
Tests: 4 (full assembly incl. peer industry + valuation + additive-merge, no-cik
null metrics, <5-peers omitted, price-guarded valuation).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1. Peer percentile is now a tie-aware rank against the OTHER issuers
((worse + 0.5*tied)/(peers-1)): an all-equal group maps to 50 (not 100), the
median maps to 50, a unique best to 100, a unique worst to 0.
2. Deterministic reads use the consecutive non-null suffix ending at the latest
point (>=3 values): a null latest or an internal gap yields no read, so a read
never reflects a period displayed as n/a.
3. Peer filtering excludes non-finite (NaN/±inf) as well as null, including an
invalid subject.
Tests updated + added (all-equal, median rank, non-finite, latest-null history,
internal gap). 15 passed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Completes the pure read-time core.
fundamentals_peers.py: median + polarity-aware favorable percentile + peer_count
for a subject within its SIC group (CIK-deduped by the caller); returns None
below MIN_PEERS=5 so the caller omits the industry object. Absolute net_debt is
intentionally NOT peer-eligible (size-dependent) — leverage compares via
net_debt_to_ebitda. HIGHER_IS_BETTER polarity map + two_digit_sic() grouping key.
fundamentals_reads.py: one shared deterministic rule set (no LLM): growth_read
(+-2pp), margin_read (latest vs mean-of-prior, +-1pp), share_count_read (+-1%),
peer_read (60/40 bands, polarity-aware phrasing per metric), header_sentence
(growth · margins · valuation, omitting empty). Tunable named constants; >=3
periods required for a series read.
Tests: 8 peer + 5 reads, anchored on the boundary cases (exactly +2.0pp, exactly
60th percentile, exactly +1.0pp margin). 13 passed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1. net_debt requires BOTH cash and total_debt; a missing side is null, not
treated as zero (which would be a partial, misleading value).
2. net_debt_to_ebitda is null when TTM EBITDA <= 0 — a negative denominator
would otherwise rank a distressed issuer as favorably low-leverage.
3. The quarter tape is the CONSECUTIVE run ending at the latest period (stops at
a gap), so trend text never compares non-adjacent quarters as if consecutive.
4. YoY growth is null when the prior-year TTM is <= 0 (e.g. loss->profit), which
is not a meaningful percentage.
Also corrected the plan's net-debt formula to total debt − (cash + ST) matching
the positive-means-net-debt implementation. +4 tests. 10 passed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Derives the display metrics from the stored YTD snapshots at read time (no I/O,
no DB), per the A3 schema decision. Given an issuer's snapshot rows it produces:
- amendment selection (newest accepted_at per fiscal period);
- discrete quarters = YTD(Qn) - YTD(Qn-1), Q4 = YTD(FY) - YTD(Q3);
- TTM = trailing four discrete quarters; missing period -> null, never partial;
- metric series (value + 4-quarter tape, each point dated): revenue_growth_yoy,
eps_growth_yoy, operating_margin, fcf_margin, net_debt, net_debt_to_ebitda,
share_count_change_yoy;
- request-time valuation inputs (ttm_diluted_eps, ttm_fcf, shares_outstanding)
for the API to combine with price.
Units per app convention (percentages = pp, leverage = multiple, dollars).
Tests: 6 (growth+Q4, margins, net-debt/EBITDA+dilution, valuation inputs,
missing-period-null, amendment selection). Verified on real Apple snapshots:
op margin 32.6%, net-debt/EBITDA 0.10, buyback -1.7%/yr, TTM EPS $8.26.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Extend the companyfacts structural check to reject a concept with a
missing/non-dict `units` mapping (not just the top-level `facts`), so a
partially-malformed payload fails promotion instead of silently dropping that
concept's facts. New fixture proves it fails.
- Strengthen the newly-added-issuer test: keep latest_index equal to the prior
run so ONLY the universe fingerprint changes the revision — proving the
fingerprint alone prevents a new ticker from being starved/no_op'd.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1. Removed the 45-day index-walk cap: it discarded the older part of a long
outage while still advancing source_max_date, permanently losing filings.
The walk now covers every unprocessed date (a large gap is one-time cost).
2. Discrepancy detection meets the immutability contract: it compares ALL source
snapshot fields (not five), read-only during stage/validate, reports the
differing accessions + fields in validation_json, and promote emits a warning
system event (in-transaction) — never mutating the stored row.
3. Malformed companyfacts (missing facts/units structure) are recorded separately
and FAIL validation, instead of silently degrading to skipped rows that the
50% backfill coverage floor could still pass.
Also corrected the stale "sum share classes" / DEI-only wording in the snapshot
model docstring and the A3 design doc to describe the us-gaap fallback.
Tests: +4 regressions (>45-day gap loses nothing, newly-added issuer backfills
without filing, malformed payload fails, shares discrepancy detected + evented).
23 passed, 1 skipped.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
SecFundamentalsImporter (SourceImporter, source=sec_facts): populates immutable
fundamental_snapshots from Company Facts and back-fills tickers.cik/sic, driven
by the EDGAR daily index. Shadow only. Guardrails per review:
- detect_revision caches the resolved universe + exact tracked index rows and
composes the revision from them; stage consumes those same cached inputs
(no index/universe refetch) so promoted data matches the computed revision.
- Resolution is read-only in stage (proposals only); ticker writes happen in
promote via apply_ticker_updates.
- validate runs the index<->Company-Facts consistency gate before any write:
a tracked XBRL index accession missing from Company Facts fails the run
(they lag independently) so we retry, not record null. Non-XBRL amendments
are skipped with a recorded reason. Backfill has a coverage floor.
- promote inserts ON CONFLICT (accession) DO NOTHING (immutable), reports
differing existing accessions without mutating, and applies ticker updates in
the same transaction.
- Full-history backfill on first run / for newly-added issuers (include_history);
incremental fetch only for issuers that filed.
Parser: split parse result into skipped_filings vs field_issues (coverage must
not count field warnings); header notes the us-gaap shares fallback; added
companyfacts_accessions() for the gate.
Verified live end-to-end (AAPL + GOOGL backfill): 112 snapshots, cik/sic set,
GOOGL shares via us-gaap fallback, AAPL via dei. Tests: 6 importer (backfill,
incremental, consistency-gate fail, non-XBRL skip, read-only-on-failure,
conflict-discrepancy) + parser ParseResult updates.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1. Multi-class shares: prefer the single dei:EntityCommonStockSharesOutstanding
cover-page fact; else fall back to us-gaap:CommonStockSharesOutstanding at
period end (Alphabet has no dei fact). Never sum class facts (companyfacts is
non-dimensional) and never use weighted-average/diluted; conflicting values ->
null, counted as an "ambiguous shares outstanding" note in validation. Plan's
"sum class-specific" wording corrected. Verified live: Alphabet shares now
populate (12.1B), Apple still uses its dei cover date.
2. Fiscal context is the majority (fy, fp) among facts ending at reportDate, with
ties rejected — no longer the arbitrary first fact.
3. Hardening: catalog selectors require taxonomy == "us-gaap"; indexing drops
malformed facts (missing accession/end, non-finite value) so a custom concept
or bad date can't be selected.
Tests: +8 (dei precedence, us-gaap fallback, conflict->null, no weighted-average,
tie-context skip, foreign-taxonomy/malformed ignored, ambiguous-shares note).
14 passed, 1 skipped.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Pure parser (no I/O/DB) turning one issuer's companyfacts + submissions filing
metadata into per-accession snapshot rows for the filing's primary period.
- Period identity from end == reportDate, never fy/fp (fy/fp is the filing's
context; comparatives repeat it).
- Duration facts stored as cumulative YTD: pick the fact whose span matches the
fiscal-period-to-date length (Q1~3mo..FY~12mo) within tolerance; no YTD-length
fact -> null (never a discrete masquerading as YTD).
- Balance-sheet instants at end == reportDate; shares_outstanding is the dei
cover-page fact whose own end (cover date) is stored in shares_outstanding_date.
- Cash and debt composites are aggregate-first and mutually exclusive (each
source tag counted at most once).
- Carries filing_date through submissions rows (snapshot.filed_date).
Verified on REAL Apple companyfacts: 44 snapshots, 0 skipped, YTD revenue
124.3B->219.7B->313.7B->416.2B across FY2025 (Q4 derives at read time), every
shares_date is the cover date != period_end. Tests: 6 fixture + 1 skip-guarded
live-invariants (monotonic YTD, cover-date shares).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Addresses the slice-1 review:
1. Resolution is now read-only (A1 transaction contract). resolve_ciks /
fetch_sic_updates compute proposals and mutate nothing; a new
apply_ticker_updates issues the writes, called only in promote — so a failed
validation can't leak ticker changes on the framework's failure commit.
2. Only 404 means "missing". Added SecNotFoundError; daily_index /
latest_index_date catch only that. 403, exhausted 429, 5xx, timeouts, and
transport/parse errors now propagate instead of looking like "no index".
3. Fair-access enforced when opening a REAL client (transport=None): reject
blank/placeholder/non-email User-Agent and sub-0.11s spacing. Mock transports
skip it (tests use 0 spacing).
4. submissions(include_history=False) by default — only the one-time full
backfill fetches the history shards; SIC/incremental work makes no extra
requests.
Plus: retry transient 5xx/network errors and honor Retry-After during the 1 GB
backfill; compose_revision rejects a missing index date (no "None:..." revision).
Re-verified live vs real SEC (fair-access validation passes, shard merge intact).
Tests: 18 (added error propagation, read-only resolution, fair-access, recent-only
submissions, reject-None revision).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
First A3 implementation checkpoint (design: docs/dolt-sec-a3-design.md).
- sec_client.py: async SEC EDGAR client honoring fair-access — identifying
User-Agent (config), request spacing < 10 req/s, exponential backoff on 429,
and 403 -> SecForbiddenError (alert and stop, never retry-loop). Fetchers:
company_tickers (normalised, multi-class share CIK), submissions (merges the
paginated filings.files shards so full history is visible), companyfacts,
daily_index (fixed-width form.idx parse), latest_index_date.
- sec_universe.py: resolve_ciks (tickers.cik backfill), refresh_sic
(sic/sic_description), and the composite-revision pieces — universe_fingerprint
(a new ticker changes the revision, so it's never no_op'd/starved),
index_content_hash, compose_revision.
- config + .env.example: SEC_USER_AGENT (must be a real contact email) + spacing
/ retries / timeout.
Verified live against real SEC: AAPL->320193, GOOG==GOOGL, BRK-B resolved;
submissions shard-merge proven (131 filings back to 1993); daily index parsed.
Tests: 11 (mocked-transport parsing + 403/429 handling + resolution/fingerprint).
Full suite 713 passed. Next slice: companyfacts -> snapshot parser + importer.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fold in the A3 design review:
1. Composite revision = latest-index-date + index-content-hash + tracked
symbol->CIK fingerprint, so a newly added ticker forces a run instead of
being no_op'd/starved. No backfill sentinel — absence of a prior promoted
run triggers backfill; source_max_date records the processed index date.
2. Full history needs the paginated submissions shards: filings.recent caps at
1000; older accessions (reportDate/acceptanceDateTime/isXBRL) live in
filings.files[] shards (verified on Apple: recent=1000, one 1994-2015 shard).
3. Index<->Company-Facts consistency gate: they are separate SEC products that
can lag; for every tracked isXBRL index accession, confirm it exists in
Company Facts before promotion, else fail+retry (never record a null/partial
snapshot). Non-XBRL amendments skipped with a recorded reason.
4. Immutable = insert-only (ON CONFLICT DO NOTHING); a differing re-fetch is a
reported discrepancy, never a silent mutation / import_run_id replacement.
Plus deterministic, mutually-exclusive cash/debt composition (aggregate-first;
each source tag counted at most once).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
