fix(sec): correct fundamentals derivation from SEC company facts
The A5 parity report surfaced coverage gaps and wrong values that all traced to the SEC facts parser and read-time derivation rather than to bad source data. Fixes, each validated by replaying the production parser + derivation against live company facts: - Period identity is derived from period_end against the issuer's fiscal calendar, not SEC's fy/fp fields, which collide (two period ends on one key, one silently discarded) and invert (a period sorting before one that precedes it) often enough to break the quarter chain. Recovers BXP, CRM, CRWD, FRT, MTD, NTAP, PPL, STX, WDAY. Fixed labels are internal ordering keys only (not in any API schema), so a filer whose year ends in early January shifting by one is harmless. - Revenue concept list gains RevenuesNetOfInterestExpense (banks) and the IncludingAssessedTax variant (REITs/consumer); EPS gains the continuing-ops variant (REG/FCX) and, last, basic EPS for a period tagging no diluted variant at all (PPL). All appended, so any issuer that already resolved keeps its concept. - YTD span tolerance 20 -> 25 days, covering 4-4-5 retail calendars whose 36-week YTD-Q3 (251-252d) previously missed by ~2 (COST, PEP, DPZ). - Amendment resolution is per field: a partial 10-K/A (Part III only, no financial facts) no longer blanks the period (DVN). - TTM diluted EPS is suppressed when a split contaminates the trailing window (BKNG's mixed-unit sum produced a P/E of 1.10 that clamped to a perfect fundamental sub-score). A post-filing split with no share-count evidence (KLAC) remains undetectable from this data. - Multi-class share fallback: weighted_avg_diluted_shares is captured and used for market cap when the cover-page count is absent (dimensional, so missing from company facts for META/CMCSA/CHTR/FOXA/NWSA/LEN). Within ~0.6% of the true count on controls; flagged shares_estimated in the API. BRK-B has no weighted-average fact either and stays unavailable. 820 unit tests pass; new tests confirmed to fail against the pre-fix code. Effect is inert until existing rows are reparsed (see reparse path). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 4.8
@@ -8,8 +8,10 @@ schema decision. No I/O, no DB: it takes an issuer's snapshot rows (ORM rows or
|
||||
any objects with the same attributes) and returns structured metrics.
|
||||
|
||||
Rules:
|
||||
- **Amendment selection:** for each (fiscal_year, fiscal_period), the row with
|
||||
the newest `accepted_at` wins.
|
||||
- **Amendment selection:** for each (fiscal_year, fiscal_period), the newest
|
||||
`accepted_at` wins **per field**, falling back to the newest row that actually
|
||||
reports one. A partial amendment (a 10-K/A adding Part III carries no financial
|
||||
facts) must not blank the period.
|
||||
- **Discrete quarter** = YTD(Qn) − YTD(Qn−1); Q1 = YTD(Q1); **Q4 = YTD(FY) −
|
||||
YTD(Q3)**. Any missing period → the derived value is null, never partial.
|
||||
- **TTM** = sum of the trailing four discrete quarters ending at a period.
|
||||
@@ -21,6 +23,7 @@ from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import date
|
||||
from types import SimpleNamespace
|
||||
from typing import Any, Iterable
|
||||
|
||||
_FP_TO_Q = {"Q1": 1, "Q2": 2, "Q3": 3, "FY": 4}
|
||||
@@ -38,6 +41,20 @@ _FLOW_FIELDS = (
|
||||
"revenue", "net_income", "operating_income", "diluted_eps", "cfo", "capex",
|
||||
"depreciation_amortization",
|
||||
)
|
||||
# Reported facts resolved independently across a period's accessions (see
|
||||
# _merge_amendments); period identity/provenance is taken from the newest one.
|
||||
_MERGED_FIELDS = (
|
||||
*_FLOW_FIELDS,
|
||||
"cash_and_st_investments", "total_debt", "shares_outstanding",
|
||||
"shares_outstanding_date",
|
||||
# period_start is set alongside revenue by the parser, so it follows the same
|
||||
# fallback: a bare amendment reports neither and must not blank it.
|
||||
"period_start",
|
||||
)
|
||||
_CARRIED_FIELDS = (
|
||||
"fiscal_year", "fiscal_period", "period_end", "filed_date",
|
||||
"accepted_at", "form", "accession", "cik",
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -60,8 +77,15 @@ class DerivedFundamentals:
|
||||
metrics: dict[str, MetricSeries] = field(default_factory=dict)
|
||||
# request-time valuation inputs (ratios are computed in the API with price)
|
||||
ttm_diluted_eps: float | None = None
|
||||
# Set when ttm_diluted_eps was suppressed rather than simply unavailable.
|
||||
ttm_diluted_eps_caveat: str | None = None
|
||||
ttm_fcf: float | None = None
|
||||
shares_outstanding: float | None = None
|
||||
# True when shares_outstanding came from the weighted-average diluted count
|
||||
# because the point-in-time cover-page count was absent (always so for
|
||||
# multi-class issuers). Consumers must label anything derived from it as
|
||||
# estimated — it is a period average, not a point-in-time count.
|
||||
shares_outstanding_estimated: bool = False
|
||||
latest_period_end: date | None = None
|
||||
latest_filed_date: date | None = None
|
||||
|
||||
@@ -85,6 +109,15 @@ def derive(snapshots: Iterable[Any]) -> DerivedFundamentals:
|
||||
result.latest_period_end = latest_row.period_end
|
||||
result.latest_filed_date = latest_row.filed_date
|
||||
result.shares_outstanding = getattr(latest_row, "shares_outstanding", None)
|
||||
if result.shares_outstanding is None:
|
||||
# Multi-class issuers (META, CMCSA, BRK-B, CHTR, FOXA, NWSA, LEN) report
|
||||
# the cover-page count per class, which is dimensional and so absent from
|
||||
# companyfacts — leaving market cap and FCF yield silently unavailable for
|
||||
# some of the largest names. The weighted-average diluted count is always
|
||||
# present and within ~0.6% of the true count where both exist, so fall
|
||||
# back to it and mark the result estimated rather than show nothing.
|
||||
result.shares_outstanding = getattr(latest_row, "weighted_avg_diluted_shares", None)
|
||||
result.shares_outstanding_estimated = result.shares_outstanding is not None
|
||||
result.ttm_diluted_eps = _ttm(discrete["diluted_eps"], *latest)
|
||||
ttm_cfo = _ttm(discrete["cfo"], *latest)
|
||||
ttm_capex = _ttm(discrete["capex"], *latest)
|
||||
@@ -102,7 +135,14 @@ def derive(snapshots: Iterable[Any]) -> DerivedFundamentals:
|
||||
"net_debt_to_ebitda": _leverage_series(selected, discrete, tape),
|
||||
"share_count_change_yoy": _share_change_series(selected, tape),
|
||||
}
|
||||
_guard_split_sensitive_metrics(result.metrics)
|
||||
# TTM EPS sums four quarters of *per-share* values, so a split inside that
|
||||
# window mixes pre- and post-split units — the same distortion the guard
|
||||
# already catches for the series, and the one that produced BKNG's P/E of
|
||||
# 1.10. Left unguarded it does not merely mislead: a nonsense-low P/E clamps
|
||||
# to a perfect 100 fundamental sub-score, so it must null out like the rest.
|
||||
if _guard_split_sensitive_metrics(result.metrics):
|
||||
result.ttm_diluted_eps = None
|
||||
result.ttm_diluted_eps_caveat = SPLIT_SENSITIVE_CAVEAT
|
||||
for series in result.metrics.values():
|
||||
series.period_end = latest_row.period_end
|
||||
series.filed_date = latest_row.filed_date
|
||||
@@ -112,17 +152,57 @@ def derive(snapshots: Iterable[Any]) -> DerivedFundamentals:
|
||||
# -- period selection --------------------------------------------------------
|
||||
|
||||
def _select_latest_per_period(snapshots: Iterable[Any]) -> dict[tuple[int, str], Any]:
|
||||
best: dict[tuple[int, str], Any] = {}
|
||||
grouped: dict[tuple[int, str], list[Any]] = {}
|
||||
for row in snapshots:
|
||||
fp = getattr(row, "fiscal_period", None)
|
||||
fy = getattr(row, "fiscal_year", None)
|
||||
if fp not in _FP_TO_Q or fy is None:
|
||||
continue
|
||||
key = (fy, fp)
|
||||
cur = best.get(key)
|
||||
if cur is None or _accepted(row) > _accepted(cur):
|
||||
best[key] = row
|
||||
return best
|
||||
grouped.setdefault((fy, fp), []).append(row)
|
||||
return {key: _merge_amendments(rows) for key, rows in grouped.items()}
|
||||
|
||||
|
||||
def _merge_amendments(rows: list[Any]) -> Any:
|
||||
"""Resolve one period from its accessions: newest wins, per field.
|
||||
|
||||
Amendments are frequently partial — a 10-K/A filed only to add Part III
|
||||
reports no financial facts at all. Taking the newest accession wholesale
|
||||
would blank every field it omits and null the period downstream (and with
|
||||
it TTM and YoY, which need an unbroken quarter chain), so each field falls
|
||||
back to the newest accession that actually reports it.
|
||||
|
||||
Only rows sharing the newest row's ``period_end`` are merged. A same-key row
|
||||
covering a *different* period is a mislabelled filing, not an amendment, and
|
||||
blending the two would silently mix fiscal years.
|
||||
"""
|
||||
if len(rows) == 1:
|
||||
return rows[0]
|
||||
ordered = sorted(rows, key=_amendment_order, reverse=True) # newest first
|
||||
newest = ordered[0]
|
||||
same_period = [
|
||||
row
|
||||
for row in ordered
|
||||
if getattr(row, "period_end", None) == getattr(newest, "period_end", None)
|
||||
]
|
||||
if len(same_period) == 1:
|
||||
return newest
|
||||
merged = SimpleNamespace(**{name: getattr(newest, name, None) for name in _CARRIED_FIELDS})
|
||||
for name in _MERGED_FIELDS:
|
||||
merged_value = None
|
||||
for row in same_period: # newest first
|
||||
value = getattr(row, name, None)
|
||||
if value is not None:
|
||||
merged_value = value
|
||||
break
|
||||
setattr(merged, name, merged_value)
|
||||
return merged
|
||||
|
||||
|
||||
def _amendment_order(row: Any) -> tuple[bool, Any]:
|
||||
# (has-timestamp, timestamp) so a row without one sorts oldest instead of
|
||||
# raising when compared against a row that has one.
|
||||
accepted = _accepted(row)
|
||||
return (accepted is not None, accepted)
|
||||
|
||||
|
||||
def _accepted(row: Any):
|
||||
@@ -255,18 +335,21 @@ def _share_change_series(selected, tape) -> MetricSeries:
|
||||
return _series(pts)
|
||||
|
||||
|
||||
def _guard_split_sensitive_metrics(metrics: dict[str, MetricSeries]) -> None:
|
||||
def _guard_split_sensitive_metrics(metrics: dict[str, MetricSeries]) -> bool:
|
||||
"""Suppress historical comparisons likely distorted by a corporate action.
|
||||
|
||||
Company Facts has no point-in-time split factors. A large YoY share-count
|
||||
move can therefore make both the point-in-time share comparison and
|
||||
per-share EPS growth non-comparable. Keep the raw facts in snapshots, but
|
||||
expose nulls plus an explicit caveat in the user-facing derived series.
|
||||
|
||||
Returns True when the *latest* period is suspect, so callers can apply the
|
||||
same suppression to per-share scalars derived from that window.
|
||||
"""
|
||||
shares = metrics.get("share_count_change_yoy")
|
||||
eps = metrics.get("eps_growth_yoy")
|
||||
if shares is None or eps is None:
|
||||
return
|
||||
return False
|
||||
|
||||
suspect_periods = {
|
||||
point.period_end
|
||||
@@ -275,18 +358,21 @@ def _guard_split_sensitive_metrics(metrics: dict[str, MetricSeries]) -> None:
|
||||
and abs(point.value) >= SPLIT_SUSPECT_SHARE_CHANGE_PCT
|
||||
}
|
||||
if not suspect_periods:
|
||||
return
|
||||
return False
|
||||
|
||||
latest_suspect = False
|
||||
for series in (shares, eps):
|
||||
latest_guarded = bool(
|
||||
series.history and series.history[-1].period_end in suspect_periods
|
||||
)
|
||||
latest_suspect = latest_suspect or latest_guarded
|
||||
for point in series.history:
|
||||
if point.period_end in suspect_periods:
|
||||
point.value = None
|
||||
series.value = series.history[-1].value if series.history else None
|
||||
if latest_guarded:
|
||||
series.caveat = SPLIT_SENSITIVE_CAVEAT
|
||||
return latest_suspect
|
||||
|
||||
|
||||
def _net_debt(row: Any) -> float | None:
|
||||
|
||||
Reference in New Issue
Block a user