fix(sec): correct fundamentals derivation from SEC company facts
The A5 parity report surfaced coverage gaps and wrong values that all traced to the SEC facts parser and read-time derivation rather than to bad source data. Fixes, each validated by replaying the production parser + derivation against live company facts: - Period identity is derived from period_end against the issuer's fiscal calendar, not SEC's fy/fp fields, which collide (two period ends on one key, one silently discarded) and invert (a period sorting before one that precedes it) often enough to break the quarter chain. Recovers BXP, CRM, CRWD, FRT, MTD, NTAP, PPL, STX, WDAY. Fixed labels are internal ordering keys only (not in any API schema), so a filer whose year ends in early January shifting by one is harmless. - Revenue concept list gains RevenuesNetOfInterestExpense (banks) and the IncludingAssessedTax variant (REITs/consumer); EPS gains the continuing-ops variant (REG/FCX) and, last, basic EPS for a period tagging no diluted variant at all (PPL). All appended, so any issuer that already resolved keeps its concept. - YTD span tolerance 20 -> 25 days, covering 4-4-5 retail calendars whose 36-week YTD-Q3 (251-252d) previously missed by ~2 (COST, PEP, DPZ). - Amendment resolution is per field: a partial 10-K/A (Part III only, no financial facts) no longer blanks the period (DVN). - TTM diluted EPS is suppressed when a split contaminates the trailing window (BKNG's mixed-unit sum produced a P/E of 1.10 that clamped to a perfect fundamental sub-score). A post-filing split with no share-count evidence (KLAC) remains undetectable from this data. - Multi-class share fallback: weighted_avg_diluted_shares is captured and used for market cap when the cover-page count is absent (dimensional, so missing from company facts for META/CMCSA/CHTR/FOXA/NWSA/LEN). Within ~0.6% of the true count on controls; flagged shares_estimated in the API. BRK-B has no weighted-average fact either and stays unavailable. 820 unit tests pass; new tests confirmed to fail against the pre-fix code. Effect is inert until existing rows are reparsed (see reparse path). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 4.8
@@ -8,6 +8,9 @@ fixture and verifiable against a real companyfacts pull.
|
||||
The load-bearing rules (design Decision 2 + review):
|
||||
- Period identity comes from `end == submissions.reportDate`, never `fy/fp`
|
||||
(fy/fp is the *filing's* context; comparatives inside a filing repeat it).
|
||||
This applies to the stored `fiscal_year`/`fiscal_period` too: they are derived
|
||||
from `reportDate` against the issuer's `fiscalYearEnd` (see `_period_identity`),
|
||||
because SEC's fy/fp collide and invert often enough to break the quarter chain.
|
||||
- Duration facts are stored as **cumulative YTD**: pick the fact whose span
|
||||
matches the fiscal-period-to-date length (Q1≈3mo … FY≈12mo) within tolerance.
|
||||
If no YTD-length fact exists, store null — never a discrete masquerading as YTD.
|
||||
@@ -15,7 +18,10 @@ The load-bearing rules (design Decision 2 + review):
|
||||
is a single consolidated value: the cover-page `dei` fact (its own cover-date
|
||||
`end` stored separately) if present, else `us-gaap:CommonStockSharesOutstanding`
|
||||
at period end (e.g. Alphabet has no `dei` fact) — never a class sum or the
|
||||
weighted-average/diluted count.
|
||||
weighted-average/diluted count. Multi-class issuers report it per class, which
|
||||
is dimensional and therefore absent from companyfacts entirely, so
|
||||
`weighted_avg_diluted_shares` is stored alongside as an explicit fallback for
|
||||
market cap — a separate column, never backfilled into `shares_outstanding`.
|
||||
- Cash and debt composites are aggregate-first and mutually exclusive (each
|
||||
source tag counted at most once).
|
||||
|
||||
@@ -29,7 +35,7 @@ from __future__ import annotations
|
||||
import logging
|
||||
import math
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import date, datetime
|
||||
from datetime import date, datetime, timedelta
|
||||
from typing import Any, NamedTuple
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -37,14 +43,35 @@ logger = logging.getLogger(__name__)
|
||||
# Expected YTD span (days) per fiscal period; a duration fact must land within
|
||||
# tolerance of this to count as the period's cumulative value.
|
||||
_EXPECTED_YTD_DAYS = {"Q1": 91, "Q2": 182, "Q3": 273, "FY": 365}
|
||||
_YTD_TOLERANCE_DAYS = 20 # covers 52/53-week fiscal calendars
|
||||
# Period identity (see _period_identity): how far a quarter end sits before its
|
||||
# fiscal-year end, and how far a fiscal-year end may drift from the nominal MMDD.
|
||||
# The quarter bands are 91 days apart, so ±35 stays unambiguous even for a 4-4-5
|
||||
# filer whose 16-week Q4 puts Q3 112 days out.
|
||||
_QUARTER_DAYS_TO_FY_END = {"Q1": 273, "Q2": 182, "Q3": 91}
|
||||
_QUARTER_TOLERANCE_DAYS = 35
|
||||
_FYE_DRIFT_TOLERANCE_DAYS = 21
|
||||
# Covers 52/53-week calendars *and* 4-4-5 retail ones (12/12/12/16 weeks), whose
|
||||
# YTD-Q3 is 36 weeks = 251-252 days and missed a 20-day tolerance by ~2 -- so
|
||||
# COST/PEP lost Q3 every year, breaking the quarter chain and nulling TTM + YoY.
|
||||
# Q1 84d, Q2 168d and FY 364d were always inside. Adjacent periods stay
|
||||
# unambiguous at 25 (66-116, 157-207, 248-298, 340-390).
|
||||
_YTD_TOLERANCE_DAYS = 25
|
||||
|
||||
# us-gaap duration concepts (money), priority order; first present wins.
|
||||
_DURATION_USD = {
|
||||
# Order is load-bearing (first present wins) and the tail entries are
|
||||
# deliberately *appended*: every issuer that already resolved keeps the same
|
||||
# concept, and only issuers that resolved to nothing gain a value.
|
||||
# - IncludingAssessedTax: REITs/consumer filers that tag only this variant
|
||||
# (e.g. ARE, KHC) reported no revenue at all.
|
||||
# - RevenuesNetOfInterestExpense: the banks' total-revenue tag. JPM/GS/WFC
|
||||
# tag it in every 10-Q and `Revenues` only (if at all) in the 10-K.
|
||||
"revenue": [
|
||||
"RevenueFromContractWithCustomerExcludingAssessedTax",
|
||||
"Revenues",
|
||||
"SalesRevenueNet",
|
||||
"RevenueFromContractWithCustomerIncludingAssessedTax",
|
||||
"RevenuesNetOfInterestExpense",
|
||||
],
|
||||
"net_income": ["NetIncomeLoss"],
|
||||
"operating_income": ["OperatingIncomeLoss"],
|
||||
@@ -62,7 +89,27 @@ _DURATION_USD = {
|
||||
"DepreciationAndAmortization",
|
||||
],
|
||||
}
|
||||
_EPS_CONCEPTS = ["EarningsPerShareDiluted"] # unit USD/shares
|
||||
# unit USD/shares. Appended (not reordered) so any issuer that already resolved
|
||||
# keeps the same concept. REG tags only the continuing-operations variant on every
|
||||
# filing; FCX switches by form type -- EarningsPerShareDiluted in its 10-Qs, the
|
||||
# continuing-ops tag in its 10-K -- which nulled the FY row and killed Q4 + TTM.
|
||||
# The basic variants are a last resort for a period that tags no diluted EPS at
|
||||
# all (PPL's 2026 Q1). Basic ignores option/convert dilution so it slightly
|
||||
# overstates EPS (~1.2% for PPL), but only fires when diluted is entirely absent,
|
||||
# and high-dilution names always tag diluted -- so it never displaces a real one.
|
||||
_EPS_CONCEPTS = [
|
||||
"EarningsPerShareDiluted",
|
||||
"IncomeLossFromContinuingOperationsPerDilutedShare",
|
||||
"EarningsPerShareBasic",
|
||||
"IncomeLossFromContinuingOperationsPerBasicShare",
|
||||
]
|
||||
# Weighted-average diluted share count (unit "shares"), the market-cap fallback
|
||||
# for multi-class issuers whose cover-page count is dimensional and therefore
|
||||
# absent from companyfacts. Always present, since EPS is computed from it.
|
||||
_WEIGHTED_AVG_SHARE_CONCEPTS = [
|
||||
"WeightedAverageNumberOfDilutedSharesOutstanding",
|
||||
"WeightedAverageNumberOfSharesOutstandingBasicAndDiluted",
|
||||
]
|
||||
# us-gaap instant (balance-sheet) concepts, at end == reportDate.
|
||||
_CASH = ["CashAndCashEquivalentsAtCarryingValue"]
|
||||
_ST_INVESTMENTS = ["ShortTermInvestments", "MarketableSecuritiesCurrent"] # pick one
|
||||
@@ -104,6 +151,7 @@ class SnapshotRow:
|
||||
total_debt: float | None = None
|
||||
shares_outstanding: float | None = None
|
||||
shares_outstanding_date: date | None = None
|
||||
weighted_avg_diluted_shares: float | None = None
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -127,9 +175,14 @@ def parse_snapshots(
|
||||
companyfacts: dict[str, Any],
|
||||
filings: dict[str, FilingMeta],
|
||||
accessions: set[str],
|
||||
fiscal_year_end: str | None = None,
|
||||
) -> ParseResult:
|
||||
"""Build snapshot rows for ``accessions`` (those with facts + filing meta).
|
||||
|
||||
``fiscal_year_end`` is the issuer's ``submissions.fiscalYearEnd`` (MMDD) and
|
||||
is what makes period identity independent of SEC's unreliable fy/fp fields
|
||||
(see ``_period_identity``). Omitting it falls back to the old fy/fp behaviour.
|
||||
|
||||
``skipped_filings`` = no row produced (missing facts/meta or no usable period
|
||||
identity); ``field_issues`` = a row was produced but a field is null/ambiguous.
|
||||
Callers must not use field issues as failed-row coverage.
|
||||
@@ -143,7 +196,7 @@ def parse_snapshots(
|
||||
if meta is None or not facts:
|
||||
result.skipped_filings.append({"accession": accn, "reason": "no facts or filing metadata"})
|
||||
continue
|
||||
row, note = _parse_one(cik, accn, facts, meta)
|
||||
row, note = _parse_one(cik, accn, facts, meta, fiscal_year_end)
|
||||
if row is None:
|
||||
result.skipped_filings.append({"accession": accn, "reason": note or "unparseable"})
|
||||
continue
|
||||
@@ -190,12 +243,18 @@ def _index_by_accession(companyfacts: dict[str, Any]) -> dict[str, list[Fact]]:
|
||||
|
||||
|
||||
def _parse_one(
|
||||
cik: str, accn: str, facts: list[Fact], meta: FilingMeta
|
||||
cik: str, accn: str, facts: list[Fact], meta: FilingMeta,
|
||||
fiscal_year_end: str | None = None,
|
||||
) -> tuple[SnapshotRow | None, str | None]:
|
||||
"""Returns (row, note). row is None when there's no usable period identity;
|
||||
note is a validation reason (row-skip reason when row is None, else a
|
||||
field-level issue such as ambiguous shares)."""
|
||||
fy, fp = _fiscal_context(facts, meta.report_date)
|
||||
fy, fp = _period_identity(meta, fiscal_year_end)
|
||||
if fy is None or fp is None:
|
||||
# No fiscal calendar, or a period the calendar cannot place (a transition
|
||||
# period). Fall back to the filing's own context: an imperfect label still
|
||||
# beats dropping the filing entirely.
|
||||
fy, fp = _fiscal_context(facts, meta.report_date)
|
||||
if fy is None or fp not in _EXPECTED_YTD_DAYS:
|
||||
return None, "no usable period identity"
|
||||
|
||||
@@ -227,9 +286,80 @@ def _parse_one(
|
||||
shares, shares_date, ambiguous = _select_shares(facts, meta.report_date)
|
||||
row.shares_outstanding = shares
|
||||
row.shares_outstanding_date = shares_date
|
||||
row.weighted_avg_diluted_shares = _select_weighted_avg_shares(facts, meta.report_date)
|
||||
|
||||
return row, ("ambiguous shares outstanding" if ambiguous else None)
|
||||
|
||||
|
||||
def _period_identity(
|
||||
meta: FilingMeta, fiscal_year_end: str | None
|
||||
) -> tuple[int | None, str | None]:
|
||||
"""(fiscal_year, fiscal_period) from the period end and the issuer's fiscal
|
||||
calendar — never from the fy/fp fields.
|
||||
|
||||
SEC's fy/fp describe the *filing*, and they are unreliable as period identity:
|
||||
observed in production, a 10-Q labelled ``FY`` (BXP), a year ending 2025-12-31
|
||||
labelled 2024 (FRT, a December filer), a year ending 2025-06-27 labelled 2027
|
||||
(STX), and four different period ends all labelled 2022 Q3 (PPL). Because
|
||||
readers key on (fiscal_year, fiscal_period), colliding labels silently discard
|
||||
a period and inverted ones scramble the quarter chain — nulling TTM and YoY.
|
||||
|
||||
``period_end`` is authoritative, so identity is derived from it: the form
|
||||
decides FY vs quarter, and distance to the fiscal-year end decides which
|
||||
quarter. Labels need not match the issuer's own naming — a filer whose year
|
||||
ends in early January (DPZ) shifts by one — they need to be unique, monotonic
|
||||
and YoY-aligned, which is all the derivation asks of them. Nothing outside the
|
||||
derivation reads these columns.
|
||||
"""
|
||||
fy = _fiscal_year_of(meta.report_date, fiscal_year_end)
|
||||
if fy is None:
|
||||
return None, None
|
||||
if meta.form.startswith("10-K"):
|
||||
return fy, "FY"
|
||||
nominal_end = _nominal_fy_end(fy, fiscal_year_end)
|
||||
if nominal_end is None:
|
||||
return None, None
|
||||
remaining = (nominal_end - meta.report_date).days
|
||||
best = min(
|
||||
_QUARTER_DAYS_TO_FY_END,
|
||||
key=lambda k: abs(_QUARTER_DAYS_TO_FY_END[k] - remaining),
|
||||
)
|
||||
if abs(_QUARTER_DAYS_TO_FY_END[best] - remaining) > _QUARTER_TOLERANCE_DAYS:
|
||||
return None, None # transition period or odd filing — let the caller fall back
|
||||
return fy, best
|
||||
|
||||
|
||||
def _nominal_fy_end(year: int, fiscal_year_end: str | None) -> date | None:
|
||||
"""The issuer's nominal fiscal-year end in ``year`` from a MMDD string."""
|
||||
if not fiscal_year_end or len(fiscal_year_end) != 4 or not fiscal_year_end.isdigit():
|
||||
return None
|
||||
month, day = int(fiscal_year_end[:2]), int(fiscal_year_end[2:])
|
||||
if not 1 <= month <= 12 or not 1 <= day <= 31:
|
||||
return None
|
||||
while day > 28: # 52/53-week ends land on 0229/0230/0231 in some filings
|
||||
try:
|
||||
return date(year, month, day)
|
||||
except ValueError:
|
||||
day -= 1
|
||||
return date(year, month, day)
|
||||
|
||||
|
||||
def _fiscal_year_of(period_end: date, fiscal_year_end: str | None) -> int | None:
|
||||
"""Which fiscal year ``period_end`` belongs to.
|
||||
|
||||
A 52/53-week calendar's real year end drifts around the nominal MMDD (and can
|
||||
cross the calendar year), so allow drift before rolling into the next year.
|
||||
"""
|
||||
nominal = _nominal_fy_end(period_end.year, fiscal_year_end)
|
||||
if nominal is None:
|
||||
return None
|
||||
return (
|
||||
period_end.year
|
||||
if period_end <= nominal + timedelta(days=_FYE_DRIFT_TOLERANCE_DAYS)
|
||||
else period_end.year + 1
|
||||
)
|
||||
|
||||
|
||||
def _fiscal_context(facts: list[Fact], report_date: date) -> tuple[int | None, str | None]:
|
||||
"""The filing's (fy, fp) taken as the majority context among the facts that
|
||||
end at reportDate (the current-period facts, which share the filing's
|
||||
@@ -352,6 +482,34 @@ def _select_shares(
|
||||
return None, None, False # simply absent — not a conflict
|
||||
|
||||
|
||||
def _select_weighted_avg_shares(facts: list[Fact], report_date: date) -> float | None:
|
||||
"""The most recent quarter's weighted-average diluted share count.
|
||||
|
||||
Deliberately the **shortest** duration ending at reportDate, not the YTD one:
|
||||
the shorter the window the closer the average sits to the current count, which
|
||||
is what a market cap wants. Measured against issuers where the true
|
||||
point-in-time count is available, the quarter average is within ~0.6%.
|
||||
"""
|
||||
best: tuple[int, float] | None = None
|
||||
for concept in _WEIGHTED_AVG_SHARE_CONCEPTS:
|
||||
for f in facts:
|
||||
if (
|
||||
f.taxonomy != "us-gaap"
|
||||
or f.concept != concept
|
||||
or f.unit != "shares"
|
||||
or f.start is None
|
||||
or f.end != report_date
|
||||
or f.val <= 0
|
||||
):
|
||||
continue
|
||||
span = (f.end - f.start).days
|
||||
if best is None or span < best[0]:
|
||||
best = (span, float(f.val))
|
||||
if best is not None:
|
||||
return best[1] # first present concept wins, as elsewhere
|
||||
return None
|
||||
|
||||
|
||||
def _d(value: Any) -> date | None:
|
||||
if not value:
|
||||
return None
|
||||
|
||||
Reference in New Issue
Block a user