Regression found by the post-reparse collision check. _period_identity trusted
submissions.fiscalYearEnd, which is not reliable: Franklin Resources (BEN)
declares 1231 while every one of its 10-Ks ends 09-30.
The effect was data loss, not just a bad label. BEN's real fiscal Q1 (Dec 31)
sat 0 days from the claimed year end, matching no quarter band, so it fell back
to SEC's fy/fp; its fiscal Q2 (Mar 31) computed 275 days out and was labelled
Q1. Both landed on the same key, the collision discarded one, and BEN lost TTM
EPS and revenue growth entirely — values it had before this branch.
A 10-K's reportDate IS the fiscal year end by definition, so resolve_fiscal_
year_end() now prefers the issuer's most recent annual filing and treats the
declared value as a fallback for issuers with no 10-K in the set.
Scanned the full tracked universe: 2 of 506 issuers declare a year end more
than 21 days from their own 10-K — BEN (91d, broken) and DELL (29d, mislabelled
but functionally correct). Both now derive correctly and match the legacy
provider: BEN revenue growth 3.8243 vs 3.82, DELL 38.5735 vs 38.57. Controls
(AAPL, COST, PEP, DPZ, IRM, JPM, CRM, STX, AVY) byte-identical.
826 unit tests pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Found in review. _merge_amendments rebuilds a period from _MERGED_FIELDS +
_CARRIED_FIELDS alone, so a column in neither list is absent from the merged
row, not just stale — and callers read it with getattr(..., None), which
silently yields None. weighted_avg_diluted_shares was never added when the
market-cap fallback landed (_SNAPSHOT_COLS in the importer was updated, its
counterpart in the derivation was not).
The failure needed both of this branch's fixes at once: a multi-class issuer
with a partial amendment on its latest period (META with a Part-III-only
10-K/A) would silently lose market cap and FCF yield again.
Adds the field, a regression test for that case, and a guard test asserting
the merge/carry lists cover every SnapshotRow field, so the next column added
fails loudly rather than losing data quietly. Confirmed the guard catches the
original bug.
Also from review:
- Expose pe_caveat in the valuation payload, so a P/E suppressed by split
contamination says why instead of looking like missing data (the caveat was
set but never read).
- no_xbrl_filings now names both causes; the old text advised pinning a CIK
override, which is wrong for a genuine new registrant that simply has not
filed yet and clears itself.
- Document that fiscalYearEnd is the issuer's current calendar, so a fiscal-
year-end change degrades old periods (fallback/newest-wins), not current ones.
- Parser-level tests for _select_weighted_avg_shares (shortest-span-wins and
concept priority), which only had derivation-level coverage.
823 unit tests pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The A5 parity report surfaced coverage gaps and wrong values that all traced
to the SEC facts parser and read-time derivation rather than to bad source
data. Fixes, each validated by replaying the production parser + derivation
against live company facts:
- Period identity is derived from period_end against the issuer's fiscal
calendar, not SEC's fy/fp fields, which collide (two period ends on one key,
one silently discarded) and invert (a period sorting before one that precedes
it) often enough to break the quarter chain. Recovers BXP, CRM, CRWD, FRT,
MTD, NTAP, PPL, STX, WDAY. Fixed labels are internal ordering keys only (not
in any API schema), so a filer whose year ends in early January shifting by
one is harmless.
- Revenue concept list gains RevenuesNetOfInterestExpense (banks) and the
IncludingAssessedTax variant (REITs/consumer); EPS gains the continuing-ops
variant (REG/FCX) and, last, basic EPS for a period tagging no diluted
variant at all (PPL). All appended, so any issuer that already resolved keeps
its concept.
- YTD span tolerance 20 -> 25 days, covering 4-4-5 retail calendars whose
36-week YTD-Q3 (251-252d) previously missed by ~2 (COST, PEP, DPZ).
- Amendment resolution is per field: a partial 10-K/A (Part III only, no
financial facts) no longer blanks the period (DVN).
- TTM diluted EPS is suppressed when a split contaminates the trailing window
(BKNG's mixed-unit sum produced a P/E of 1.10 that clamped to a perfect
fundamental sub-score). A post-filing split with no share-count evidence
(KLAC) remains undetectable from this data.
- Multi-class share fallback: weighted_avg_diluted_shares is captured and used
for market cap when the cover-page count is absent (dimensional, so missing
from company facts for META/CMCSA/CHTR/FOXA/NWSA/LEN). Within ~0.6% of the
true count on controls; flagged shares_estimated in the API. BRK-B has no
weighted-average fact either and stays unavailable.
820 unit tests pass; new tests confirmed to fail against the pre-fix code.
Effect is inert until existing rows are reparsed (see reparse path).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
SecFundamentalsImporter (SourceImporter, source=sec_facts): populates immutable
fundamental_snapshots from Company Facts and back-fills tickers.cik/sic, driven
by the EDGAR daily index. Shadow only. Guardrails per review:
- detect_revision caches the resolved universe + exact tracked index rows and
composes the revision from them; stage consumes those same cached inputs
(no index/universe refetch) so promoted data matches the computed revision.
- Resolution is read-only in stage (proposals only); ticker writes happen in
promote via apply_ticker_updates.
- validate runs the index<->Company-Facts consistency gate before any write:
a tracked XBRL index accession missing from Company Facts fails the run
(they lag independently) so we retry, not record null. Non-XBRL amendments
are skipped with a recorded reason. Backfill has a coverage floor.
- promote inserts ON CONFLICT (accession) DO NOTHING (immutable), reports
differing existing accessions without mutating, and applies ticker updates in
the same transaction.
- Full-history backfill on first run / for newly-added issuers (include_history);
incremental fetch only for issuers that filed.
Parser: split parse result into skipped_filings vs field_issues (coverage must
not count field warnings); header notes the us-gaap shares fallback; added
companyfacts_accessions() for the gate.
Verified live end-to-end (AAPL + GOOGL backfill): 112 snapshots, cik/sic set,
GOOGL shares via us-gaap fallback, AAPL via dei. Tests: 6 importer (backfill,
incremental, consistency-gate fail, non-XBRL skip, read-only-on-failure,
conflict-discrepancy) + parser ParseResult updates.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1. Multi-class shares: prefer the single dei:EntityCommonStockSharesOutstanding
cover-page fact; else fall back to us-gaap:CommonStockSharesOutstanding at
period end (Alphabet has no dei fact). Never sum class facts (companyfacts is
non-dimensional) and never use weighted-average/diluted; conflicting values ->
null, counted as an "ambiguous shares outstanding" note in validation. Plan's
"sum class-specific" wording corrected. Verified live: Alphabet shares now
populate (12.1B), Apple still uses its dei cover date.
2. Fiscal context is the majority (fy, fp) among facts ending at reportDate, with
ties rejected — no longer the arbitrary first fact.
3. Hardening: catalog selectors require taxonomy == "us-gaap"; indexing drops
malformed facts (missing accession/end, non-finite value) so a custom concept
or bad date can't be selected.
Tests: +8 (dei precedence, us-gaap fallback, conflict->null, no weighted-average,
tie-context skip, foreign-taxonomy/malformed ignored, ambiguous-shares note).
14 passed, 1 skipped.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Pure parser (no I/O/DB) turning one issuer's companyfacts + submissions filing
metadata into per-accession snapshot rows for the filing's primary period.
- Period identity from end == reportDate, never fy/fp (fy/fp is the filing's
context; comparatives repeat it).
- Duration facts stored as cumulative YTD: pick the fact whose span matches the
fiscal-period-to-date length (Q1~3mo..FY~12mo) within tolerance; no YTD-length
fact -> null (never a discrete masquerading as YTD).
- Balance-sheet instants at end == reportDate; shares_outstanding is the dei
cover-page fact whose own end (cover date) is stored in shares_outstanding_date.
- Cash and debt composites are aggregate-first and mutually exclusive (each
source tag counted at most once).
- Carries filing_date through submissions rows (snapshot.filed_date).
Verified on REAL Apple companyfacts: 44 snapshots, 0 skipped, YTD revenue
124.3B->219.7B->313.7B->416.2B across FY2025 (Q4 derives at read time), every
shares_date is the cover date != period_end. Tests: 6 fixture + 1 skip-guarded
live-invariants (monotonic YTD, cover-date shares).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
dennisthiessenandClaude Opus 4.8