Closes the parity investigation: prod reparse runs 6+7 reconciled against the dry run, the collision check that caught the BEN regression (and its residue -- 36 historical 53-week-drift rows, deliberately left), and the 2026-07-24 parity report diffed against the 2026-07-23 baseline. Candidate coverage 482 -> 504 of 511 with the gap fully explained (PSKY/Q new registrants, FITB guard-tripped with no revenue), agreement unchanged where both sides exist, and the remaining deltas are documented definition differences. Includes the after-report and a correction to the seventh pass (FITB loses its score, not just one input). Recommends approving the A5 cutover with KLAC as the one carried caveat. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
926 lines
50 KiB
Markdown
926 lines
50 KiB
Markdown
# A5 parity report — root-cause findings
|
||
|
||
Investigation of `fundamentals-parity-20260723T210658161480Z.json` (511 tickers,
|
||
generated 2026-07-23). Method: replayed the production parser
|
||
(`sec_facts_parser.parse_snapshots`) and derivation (`fundamentals_derivation.derive`)
|
||
against **live SEC companyfacts**, using the importer's own `_filing_meta` and
|
||
backfill accession set, then cross-checked prices against IBKR. No database was
|
||
available locally, so every conclusion below is reproduced from source data rather
|
||
than read out of prod.
|
||
|
||
Repro script: `scratchpad/diag.py` (`--history` replays the full backfill path).
|
||
Every claim below was verified on the named issuer. Names that were *not*
|
||
individually inspected are listed as unclassified — an earlier draft of this
|
||
document guessed their cause from fiscal-year-end dates and was wrong for most of
|
||
them, so the guessing is not repeated here.
|
||
|
||
## Verdict
|
||
|
||
Where both sides have a value the candidate data is good: P/E spearman 0.968,
|
||
revenue growth agreeing to 4 decimals for most names, score spearman 0.825. Every
|
||
defect found is a **parser/derivation bug or an identity problem** — not a data
|
||
quality problem with SEC or Dolt. The largest cluster is period identity, which is
|
||
exactly the risk A3 flagged as primary.
|
||
|
||
## 1. P/E outliers — splits corrupt TTM EPS, and the split guard doesn't cover it
|
||
|
||
`derive()` sets `result.ttm_diluted_eps` at `fundamentals_derivation.py:88` and only
|
||
calls `_guard_split_sensitive_metrics()` at line 105, which annotates `result.metrics`
|
||
(the `MetricSeries` objects). `ttm_diluted_eps` is a bare scalar and is never guarded.
|
||
`fundamentals_parity_service._pe()` consumes it directly.
|
||
|
||
The cleanest evidence that the *candidate* side is the broken one: reconcile each
|
||
P/E against the report's own price. Legacy comes out sane in both cases, candidate
|
||
does not.
|
||
|
||
**BKNG — guard fired, nobody listened.** Share count jumps 31.7M → 774.9M between the
|
||
FY2025 10-K and the 2026 Q1 10-Q (≈25:1 split). TTM EPS therefore sums three pre-split
|
||
quarters (27.31 + 84.01 + 44.18 = 155.50) plus one post-split quarter (1.36) =
|
||
**156.86** — mixed units. Live price $172.83 matches the price the report implies
|
||
exactly (1.1018 × 156.86 = 172.83), so the price is correct and current. Against that
|
||
price, legacy's P/E of 22.44 implies EPS ≈ 7.70 — a coherent post-split number, versus
|
||
the candidate's 156.86. The derivation *did* raise `"Not comparable: share count
|
||
changed at least 25%; possible split or corporate action."` on `eps_growth_yoy` and
|
||
`share_count_change_yoy` — P/E never sees it.
|
||
|
||
**KLAC — the guard cannot fire.** The split post-dates the most recent 10-Q (period end
|
||
2026-03-31), so no snapshot shows any share-count change (`share_count_change_yoy` =
|
||
−1.2%). TTM EPS **35.31** is internally consistent and entirely pre-split; the price
|
||
($223.30 live, ≈218.7 in the report) is post-split. Reconciling: legacy P/E 60.21
|
||
against the report price implies EPS ≈ 3.63 ≈ 35.31/9.7 — i.e. legacy is consistent
|
||
with a ~10:1 split and correct, and the candidate is off by exactly the split factor.
|
||
(IBKR's split-adjusted `open_52w` of 89.36 corroborates 10:1.)
|
||
|
||
This is the important case: **a split after the latest filing is undetectable from
|
||
snapshots alone.** No share-count test can catch it. Reconciliation needs a corporate
|
||
actions source or a price-vs-EPS plausibility check.
|
||
|
||
**COF — not a bug, a definition difference.** Shares 383M → 639M in 2025 Q2 is the
|
||
Discover acquisition. TTM GAAP EPS is genuinely $3.92 because the merger-charge quarter
|
||
(−10.19) sits in the window. Candidate P/E 51.01 is arithmetically correct on a GAAP TTM
|
||
basis; legacy's 11.61 is an adjusted/forward convention. Disclose, don't fix. Note this
|
||
single row drives the report's largest change (rank 1 → 456).
|
||
|
||
## 2. Bank revenue growth — concept-mapping gap (confirmed)
|
||
|
||
`sec_facts_parser._DURATION_USD["revenue"]` is:
|
||
|
||
```
|
||
RevenueFromContractWithCustomerExcludingAssessedTax, Revenues, SalesRevenueNet
|
||
```
|
||
|
||
Banks tag **`RevenuesNetOfInterestExpense`** in their 10-Qs:
|
||
|
||
| filer | 2026 Q1 10-Q tags present | parsed `revenue` |
|
||
|---|---|---|
|
||
| JPM | `RevenuesNetOfInterestExpense` 49,836M, `NoninterestIncome`, `InterestIncomeExpenseNet` | **null** |
|
||
| GS | `RevenuesNetOfInterestExpense` 17,227M, `InterestAndDividendIncomeOperating`, … | **null** |
|
||
| WFC | `RevenuesNetOfInterestExpense` 21,436M, … | **null** |
|
||
|
||
JPM's FY2025 10-K *also* tags `Revenues` (182,447M — identical value), so only the annual
|
||
row populates; GS never tags `Revenues` at all. Revenue growth needs five consecutive
|
||
quarterly values, so it is null for the whole cluster (JPM, GS, MS, WFC, TFC, MTB, FITB,
|
||
RF, SYF, BNY, BX, BLK, SPGI, ACGL, CBOE).
|
||
|
||
A second variant of the same gap: **ARE** and **KHC** tag
|
||
`RevenueFromContractWithCustomer**Including**AssessedTax` — also absent from the list —
|
||
so revenue is null on every row while EPS parses fine.
|
||
|
||
**Fix:** add `RevenuesNetOfInterestExpense` and the `IncludingAssessedTax` variant.
|
||
|
||
**Latent risk while you're in there:** `RevenueFromContractWithCustomerExcludingAssessedTax`
|
||
is *first* and "first present wins". For a bank that tags it, it captures only ASC-606 fee
|
||
revenue, not total revenue — a silently **understated** number rather than a null, which is
|
||
worse. DVN shows the same hazard from the other side: its 2026 Q1 tags both
|
||
`RevenueFromContractWithCustomerExcludingAssessedTax` (4,508M) and `Revenues` (3,807M),
|
||
an 18% difference decided purely by list order.
|
||
|
||
## 3. Period identity — the largest cluster, three confirmed mechanisms
|
||
|
||
### 3a. Fiscal-year label collisions (CRM, FRT, STX)
|
||
|
||
`_fiscal_context` majority-votes SEC's `fy`/`fp` fields, and `_select_latest_per_period`
|
||
keys on `(fiscal_year, fiscal_period)`. When SEC's labels disagree with the calendar, two
|
||
distinct periods collide on one key and **one is silently discarded**:
|
||
|
||
- **CRM** — two rows keyed `2025 FY`, ending 2025-01-31 and 2026-01-31.
|
||
- **FRT** — two rows keyed `2024 FY`, ending 2024-12-31 and 2025-12-31.
|
||
- **STX** — the year ending 2025-06-27 is labelled **`2027 FY`**, so it sorts *after*
|
||
`2026 Q3` (period end 2026-04-03) and is taken as the latest quarter.
|
||
|
||
The survivor's `period_end` then contradicts the fiscal ordering, Q4 derivation and the
|
||
consecutive-quarter chain break, and TTM EPS + YoY both go null.
|
||
|
||
**FRT is a calendar-year (Dec) filer**, so this is *not* limited to non-calendar fiscal
|
||
years — the earlier assumption that it was is wrong. Any filer SEC labels inconsistently
|
||
is exposed.
|
||
|
||
### 3b. Amendment selection blanks a period (DVN)
|
||
|
||
DVN has two rows for `2025 FY` (both ending 2025-12-31): the 10-K with complete financials,
|
||
and a **10-K/A carrying no financial facts at the report date** (`rev=None eps=None`).
|
||
`_select_latest_per_period` takes the newest `accepted_at`, so **the empty amendment wins**
|
||
and the FY2025 row becomes all-null, breaking the chain.
|
||
|
||
This is the most dangerous of the three: it is not exotic. Any issuer filing a 10-K/A —
|
||
including routine Part III amendments that restate nothing — silently loses that period.
|
||
The rule needs to prefer the newest accession *that actually carries the fact*, per field,
|
||
rather than the newest accession outright.
|
||
|
||
### 3c. 4-4-5 retail calendar — Q3 only, misses by ~2 days (COST, PEP)
|
||
|
||
`_EXPECTED_YTD_DAYS["Q3"] = 273` with `_YTD_TOLERANCE_DAYS = 20` accepts 253–293 days. A
|
||
12/12/12/16-week filer's YTD-Q3 is 36 weeks ≈ **251–252 days** — just under the floor.
|
||
|
||
Confirmed, facts present and rejected:
|
||
|
||
- COST 2026 Q3: `RevenueFromContractWithCustomerExcludingAssessedTax` span=**251d**
|
||
val=207,431M, `EarningsPerShareDiluted` span=251d val=14.01 → row stored with
|
||
`rev=None eps=None start=None`. Same for 2025 Q3 and 2024 Q3.
|
||
- PEP: every Q3 row is `rev=None eps=None`; Q1/Q2/FY all populate.
|
||
|
||
Q1 (83d vs 91±20), Q2 (167d vs 182±20) and FY (363–364d vs 365±20) all pass — only Q3
|
||
fails, every year. The code comment claims the tolerance "covers 52/53-week fiscal
|
||
calendars"; it does not cover 4-4-5 ones.
|
||
|
||
Note this does **not** apply to ordinary 13-week 52/53-week filers (STX's Q3 YTD is 279d and
|
||
passes) — their failures are 3a, not this.
|
||
|
||
**Fix:** widen the Q3 tolerance to ~25 days, or derive the expected span from the filer's own
|
||
fiscal calendar rather than a fixed 91/182/273.
|
||
|
||
## 4. CIK identity (XOM)
|
||
|
||
SEC's `company_tickers.json` now maps **XOM → CIK 2115436 "ExxonMobil Holdings Corp", which
|
||
has 0 filings**. All 26 XBRL 10-K/10-Qs sit under the old CIK **34088 "EXXON MOBIL CORP"**.
|
||
XOM therefore has no snapshots at all, and nothing in the pipeline notices that a tracked
|
||
issuer resolved to a CIK with zero filings.
|
||
|
||
PSKY (5 filings) and Q (3 filings) are genuinely new registrants — expected, not a bug.
|
||
|
||
## Status of the 25 names that lose their fundamental score
|
||
|
||
Production requires ≥2 metrics (`scoring_service.py:502`), the same rule the parity harness
|
||
uses, so these genuinely drop the fundamental dimension and the composite renormalises over
|
||
the remaining four.
|
||
|
||
| cause (confirmed on the named issuer) | names |
|
||
|---|---|
|
||
| FY label collision (3a) | CRM, FRT, STX |
|
||
| 4-4-5 Q3 span (3c) | COST, PEP |
|
||
| revenue concept gap (§2) | ARE, KHC |
|
||
| amendment blanks period (3b) | DVN |
|
||
| CIK identity (§4) | XOM |
|
||
| new registrant — expected | PSKY, Q |
|
||
| **not yet classified** | AZO, BXP, CRWD, FCX, HAL, MOS, MTD, NTAP, PPL, REG, SJM, SWKS, WDAY |
|
||
|
||
13 of 25 confirmed. The unclassified 13 have not been inspected and should not be assumed to
|
||
share a cause — the confirmed set already spans five distinct mechanisms.
|
||
|
||
## Recommended order of work
|
||
|
||
1. **Amendment selection (3b)** — highest blast radius, affects any 10-K/A filer, and the
|
||
current rule is wrong in principle rather than at the margin.
|
||
2. **Revenue concept list (§2)** — add `RevenuesNetOfInterestExpense` and
|
||
`IncludingAssessedTax`; audit the ASC-606-first priority, which can understate rather
|
||
than null.
|
||
3. **Q3 YTD span tolerance (3c)** — effectively one line.
|
||
4. **XOM CIK remap (§4)** — plus a validation that flags any tracked ticker resolving to a
|
||
CIK with zero XBRL filings.
|
||
5. **Split safety for `ttm_diluted_eps` (§1)** — propagate the existing guard to the scalar,
|
||
and add a price-vs-EPS plausibility check for splits that post-date the last filing.
|
||
6. **Fiscal-period identity (3a)** — the deepest fix; consider keying period identity on
|
||
`period_end` rather than SEC's `fy`/`fp`.
|
||
|
||
Re-run the parity report after these and re-classify the remaining 13 before making a
|
||
cutover decision. The current report should not be approved as-is: its coverage gaps are
|
||
artifacts of the above, not real absences in the source data.
|
||
|
||
---
|
||
|
||
# Fixes applied (items 1–3)
|
||
|
||
| # | change | file | effective |
|
||
|---|---|---|---|
|
||
| 1 | amendment resolution is now **per field** — newest accession that actually reports a fact wins; only rows sharing the newest `period_end` are merged, so a mislabelled filing is never blended in | `fundamentals_derivation.py` | **read time — immediately** |
|
||
| 2 | appended `RevenueFromContractWithCustomerIncludingAssessedTax` and `RevenuesNetOfInterestExpense` to the revenue concept list | `sec_facts_parser.py` | parse time — **needs reparse** |
|
||
| 3 | YTD span tolerance 20 → 25 days, covering 4-4-5 retail calendars | `sec_facts_parser.py` | parse time — **needs reparse** |
|
||
|
||
Fix 2 is deliberately **additive**: the new tags go at the end of the priority list, so
|
||
every issuer that already resolved keeps the same concept and only issuers that resolved
|
||
to nothing gain a value. A regression test pins that ordering.
|
||
|
||
Tests: 7 added across `test_sec_facts_parser.py` and `test_fundamentals_derivation.py`.
|
||
The 5 behaviour-changing ones were confirmed to fail against the pre-fix code; the other 2
|
||
are invariance guards that pass both ways. Full unit suite: 795 passed.
|
||
|
||
## Validation against live SEC data
|
||
|
||
Re-ran the parser + derivation on live companyfacts. Every targeted name recovers, and
|
||
the recovered values independently agree with the legacy provider:
|
||
|
||
| name | cause | revenue growth before → after | legacy | TTM EPS after |
|
||
|---|---|---|---|---|
|
||
| COST | 4-4-5 Q3 | null → **9.2311** | 9.23 | 19.88 |
|
||
| PEP | 4-4-5 Q3 | null → **5.6197** | 5.62 | 7.63 |
|
||
| KHC | concept (Including) | null → **−1.7457** | −1.75 | −4.85 |
|
||
| DVN | partial 10-K/A | null → **0.0956** | −1.51 | 3.59 |
|
||
| ARE | concept (Including) | null → **−5.3462** | −9.53 | −6.27 |
|
||
| JPM | concept (bank) | null → **3.3388** | 108.98 | 20.89 |
|
||
| GS | concept (bank) | null → **11.1974** | 6.67 | 54.75 |
|
||
| WFC | concept (bank) | null → **4.1847** | 72.75 | 6.47 |
|
||
|
||
COST/PEP/KHC matching legacy to two decimals is strong evidence the parse is now correct.
|
||
The banks are the opposite case and worth noting for the cutover argument: legacy's JPM
|
||
109% and WFC 73% "revenue growth" are not plausible for a bank, while the SEC-derived
|
||
3.3% and 4.2% are — here the candidate is **better** than what it would replace. DVN and
|
||
ARE still differ from legacy; DVN is the `Revenues` vs ASC-606 ambiguity noted in §2 and
|
||
is the one open definition question.
|
||
|
||
Regression check on names that were already correct — IRM, KLAC, BKNG — reproduces their
|
||
previous values exactly (IRM 15.6375, KLAC 13.3895, BKNG 14.9506; TTM EPS unchanged).
|
||
Nothing that worked before moved.
|
||
|
||
### Concept consistency across the bank chains (checked, clean)
|
||
|
||
Because `Revenues` still outranks `RevenuesNetOfInterestExpense`, a filer could resolve the
|
||
FY row to one concept and its quarters to the other — which would make
|
||
`Q4 = YTD(FY) − YTD(Q3)` a subtraction across two definitions, and poison every TTM window
|
||
containing it. Checked all 15 recovered banks (`scratchpad/concept_check.py`):
|
||
|
||
- **14 resolve a single concept across the whole chain** (GS, WFC, MS, TFC, MTB, FITB, RF,
|
||
SYF, BNY, BX, BLK, SPGI, ACGL, CBOE).
|
||
- **JPM is mixed but benign**: its FY2025 row tags both, at an *identical* 182,447M, so Q4
|
||
subtracts like for like. No filer showed the two tags disagreeing where both appear.
|
||
|
||
So the "candidate beats legacy for banks" claim above is safe as stated. **Residual risk:**
|
||
a future filer whose two tags differ would fail silently. Cheapest hardening is to treat
|
||
the two as one logical revenue concept rather than separate priority entries; the detector
|
||
script above turns this into a one-command check.
|
||
|
||
## Operational note — the parser fixes need a deliberate reparse
|
||
|
||
`sec_fundamentals_importer.promote()` treats snapshots as **immutable per accession**: a
|
||
re-run skips any accession already stored and records a `snapshot_discrepancy` SystemEvent
|
||
instead. So fixes 2 and 3 change nothing for rows already in the database — recovering
|
||
COST/PEP/JPM/etc. requires deleting the affected snapshot rows and re-importing, or adding
|
||
an explicit reparse path. Usefully, the discrepancy warning names exactly which stored
|
||
accessions now reconstruct differently, so a dry run over existing data will enumerate the
|
||
blast radius before anything is rewritten.
|
||
|
||
---
|
||
|
||
# Second pass — all 25 lost names now classified
|
||
|
||
Re-ran `diag.py --history` over every previously unclassified name, with fixes 1–3 in
|
||
place. (One name, DPZ, had been dropped from the unclassified list when this document was
|
||
rewritten; it is included here.)
|
||
|
||
## 11 of 25 recover
|
||
|
||
COST, PEP, KHC, DVN, ARE, **AZO, MOS, SJM, SWKS, HAL, DPZ** — and again the recovered
|
||
revenue growth matches the legacy provider to two decimals on every one:
|
||
|
||
| name | candidate | legacy | | name | candidate | legacy |
|
||
|---|---|---|---|---|---|---|
|
||
| AZO | 5.7405 | 5.74 | | SWKS | 2.3303 | 2.33 |
|
||
| MOS | 12.3388 | 12.34 | | HAL | −1.7201 | −1.72 |
|
||
| SJM | 3.7222 | 3.72 | | DPZ | 5.1573 | 5.16 |
|
||
|
||
Precisely: all 11 clear the ≥2-metric floor and regain a fundamental score. P/E returns for
|
||
AZO, MOS, SWKS, DPZ, COST, PEP and DVN. ARE, KHC and SJM have genuinely negative TTM EPS,
|
||
so their P/E stays null correctly. **HAL's TTM EPS is still null and the cause is not yet
|
||
established** — it scores on revenue growth + surprise. Loose end.
|
||
|
||
## 14 remain, in four causes
|
||
|
||
| cause | names | count |
|
||
|---|---|---|
|
||
| **fiscal-year label collisions (§3a)** | CRM, FRT, STX, BXP, CRWD, MTD, NTAP, WDAY, PPL | **9** |
|
||
| **EPS concept gap (new — §5 below)** | FCX, REG | 2 |
|
||
| CIK identity (§4) | XOM | 1 |
|
||
| new registrant — expected, not a bug | PSKY, Q | 2 |
|
||
|
||
The label bug is now the dominant cause by a wide margin, and it is more varied than first
|
||
described — it is not only colliding `fiscal_year` values:
|
||
|
||
- **BXP** — a *10-Q* for period end 2026-03-31 is labelled `2026 **FY**`. The **fiscal
|
||
period** is wrong, not just the year, so `_select_ytd` then measures the 90-day fact
|
||
against the 365-day FY expectation and rejects it too.
|
||
- **NTAP, WDAY, MTD, CRWD** — two different period-ends colliding on one key (the pattern
|
||
first seen on CRM/FRT).
|
||
- **PPL** — the worst observed: **four** rows keyed `2022 Q3`, with period ends 2022-09-30,
|
||
2023-03-31, 2023-06-30 and 2023-09-30.
|
||
|
||
## 5. New cause — EPS concept coverage
|
||
|
||
`_EPS_CONCEPTS = ["EarningsPerShareDiluted"]` is the only tag read. Confirmed by listing
|
||
every `USD/shares` duration concept in the relevant filings:
|
||
|
||
- **REG** tags only `IncomeLossFromContinuingOperationsPerDilutedShare`, on every filing —
|
||
EPS is null everywhere, so no TTM EPS and no P/E, ever.
|
||
- **FCX** is the nastier shape: its **10-Qs** tag `EarningsPerShareDiluted`, but its
|
||
**10-K** tags only `IncomeLossFromContinuingOperationsPerDilutedShare`. The FY row loses
|
||
EPS, so `Q4 = YTD(FY) − YTD(Q3)` is undefined and TTM dies — an issuer that switches
|
||
concept *by form type* looks like partial data rather than a mapping gap.
|
||
|
||
**Fix:** append `IncomeLossFromContinuingOperationsPerDilutedShare` to `_EPS_CONCEPTS`.
|
||
Same additive shape as the revenue fix; recovers REG outright and FCX's FY row.
|
||
|
||
**Related decision, not a fix:** PPL's 2026 Q1 tags *no diluted variant at all* — only
|
||
`EarningsPerShareBasic` and `IncomeLossFromContinuingOperationsPerBasicShare`. Adding the
|
||
diluted continuing-ops tag does not help it. Falling back to basic EPS is a definition
|
||
change (basic ≠ diluted) and should be an explicit call, not a silent one.
|
||
|
||
---
|
||
|
||
# Third pass — fixes #2 and #3 applied
|
||
|
||
| # | change | file | effective |
|
||
|---|---|---|---|
|
||
| 2a | `_guard_split_sensitive_metrics()` now returns whether the *latest* period is split-suspect, and `derive()` nulls `ttm_diluted_eps` (setting `ttm_diluted_eps_caveat`) when it is | `fundamentals_derivation.py` | read time — immediately |
|
||
| 3 | appended `IncomeLossFromContinuingOperationsPerDilutedShare` to `_EPS_CONCEPTS` | `sec_facts_parser.py` | parse time — needs reparse |
|
||
|
||
5 tests added; the 3 behaviour-changing ones confirmed to fail against pre-fix code, 2 are
|
||
invariance guards. Full unit suite: **800 passed**.
|
||
|
||
## Validated on live data
|
||
|
||
| name | before | after | |
|
||
|---|---|---|---|
|
||
| FCX | TTM EPS null | **1.89** | recovered |
|
||
| REG | TTM EPS null | **2.92** | recovered |
|
||
| BKNG | TTM EPS 156.86 → P/E **1.10** | **null** + caveat | false perfect score removed |
|
||
| COF | TTM EPS 3.92 → P/E 51.01 | **null** + caveat | see side effect below |
|
||
| KLAC | TTM EPS 35.31 → P/E **6.19** | unchanged | **still wrong — 2b not fixed** |
|
||
| IRM, COST | — | unchanged | no regression |
|
||
|
||
FCX and REG regain a fundamental score (EPS + surprise clears the ≥2 floor). Their
|
||
**revenue growth is still null** — both are also blocked by the label bug (REG has a
|
||
mislabelled duplicate `2024 Q2`; FCX is missing its 2024 FY row entirely).
|
||
|
||
## Threshold decision — RESOLVED: keep 25%
|
||
|
||
Measured against the database (`scratchpad/share_change_check.sql`): **15 of 467 comparable
|
||
issuers (3.2%)** trip the ≥25% guard on their latest period.
|
||
|
||
| band | names | cause |
|
||
|---|---|---|
|
||
| ≥200% | BKNG 23.8×, ORLY 14.5×, NFLX 9.8×, NOW 5.0×, TPL 3.0× | forward splits |
|
||
| 50–142% | CHTR (query artifact), **AMCR −68% (1-for-5 reverse split)**, WAT, COF | split + stock-funded M&A |
|
||
| 25–47% | OMC, BG, HBAN, FITB, COHR, RKLB | stock-funded M&A, ordinary dilution |
|
||
|
||
**Keep the threshold at 25%**, for three reasons — the first of which is empirical and came
|
||
out of checking AMCR:
|
||
|
||
1. **A real split trips at only 68%.** AMCR's 1-for-5 reverse consolidation
|
||
(2,308,359,941 → 462,045,690 shares, ratio 4.996, between the Nov 2025 and Feb 2026
|
||
10-Qs) shows up as −68%. Raising the bar to 100% to spare the M&A cases would have let a
|
||
genuine split straight through. Split magnitude and M&A magnitude overlap in practice,
|
||
not just in theory.
|
||
2. **The cost is milder than first described.** Losing P/E leaves revenue growth + earnings
|
||
surprise = 2 metrics, which still clears the ≥2 floor. Affected issuers keep a
|
||
fundamental score; they lose one of three inputs.
|
||
3. **The severities are asymmetric.** A missed split yields a P/E off by 10–25×, clamping to
|
||
a *perfect 100* sub-score. Over-nulling yields a missing input the scorer already handles
|
||
by renormalising.
|
||
|
||
Honest caveat: the guard is blunt — it detects that a share base moved, not how much damage
|
||
resulted. AMCR's pre-fix P/E was 28.61 against legacy's 29.47, i.e. only ~10-15% off, because
|
||
most of its YTD figures had already been restated on the post-split basis. So the guard
|
||
sometimes removes a roughly-usable number. That is the accepted price of a rule that cannot
|
||
measure the split factor.
|
||
|
||
Two data notes from the same check:
|
||
|
||
- **CHTR is a query artifact, not a guard trip.** The SQL picks the newest period *with* a
|
||
share count, while `derive()` picks the newest period and then reads shares off it. CHTR's
|
||
recent snapshots have a null `shares_outstanding`, so the query fell back to the 2016 Time
|
||
Warner merger. In the real path its change is None and the guard never fires — so the true
|
||
count is ~14. But it also means **CHTR has no recent share count, which breaks its market
|
||
cap in the API** — a separate small bug.
|
||
- **AMCR was suspected of being a `shares_outstanding` parsing bug and is not.** It is a real
|
||
corporate action, correctly detected. `abs()` in the guard already handles reverse splits.
|
||
|
||
## Side effect — COF
|
||
|
||
The guard fires on *any* ≥25% YoY share-count move, not only splits. COF's 383M → 639M jump
|
||
is the Discover acquisition, so it now nulls too and **loses the P/E of 51.01** that this
|
||
document previously called "arithmetically correct on a GAAP TTM basis".
|
||
|
||
I think nulling is right: TTM EPS sums four quarters whose per-share figures use different
|
||
weighted-average denominators, and across a 67% share change that sum is not a meaningful
|
||
per-share number regardless of whether the cause was a split or an acquisition. It follows
|
||
the formula without being a valid result.
|
||
|
||
But the cost is real and worth stating plainly: **any issuer doing a large stock-funded
|
||
acquisition loses its P/E for four quarters.** That frequency has not been measured — it
|
||
needs a count of `|share_count_change_yoy| ≥ 25%` across the universe, which needs the
|
||
database. If it turns out to be common, the alternative is a higher or split-shaped
|
||
threshold, at the cost of letting more BKNG-class errors through.
|
||
|
||
## 2b is genuinely unfixed
|
||
|
||
KLAC's split post-dates its most recent 10-Q, so no snapshot carries any share-count
|
||
evidence and no guard built on share counts can fire. Its P/E is still 6.19 — the true P/E
|
||
divided by the split factor. I did not ship a heuristic for this: the obvious one, flagging
|
||
implausibly low P/Es, would misfire on genuinely cheap names — CHTR (3.42) and CMCSA (4.30)
|
||
sit below KLAC's corrupted 6.19 in this very report. Detecting it needs an actual
|
||
corporate-actions source, or a price-vs-share-count reconciliation against an external
|
||
market-cap reference.
|
||
|
||
---
|
||
|
||
# Fourth pass — the reparse path
|
||
|
||
Snapshots are immutable per accession, so the parser fixes never reached stored rows.
|
||
`promote()` skipped them and logged a discrepancy. Reparse is the deliberate exception:
|
||
immutability protects *SEC's* record, but the stored row is **our reconstruction** — after a
|
||
parser fix, keeping it is preserving a stale cache, not preserving history.
|
||
|
||
| change | file |
|
||
|---|---|
|
||
| `run_import(..., force=True)` bypasses the unchanged-revision no-op. The revision tracks the *source*; a fix on our side leaves it unchanged, so the gate would skip the run | `data_import.py` |
|
||
| `SecFundamentalsImporter(reparse=True)` — forces full-history staging, and `promote()` rewrites the accessions whose reconstruction changed, stamping `import_run_id` | `sec_fundamentals_importer.py` |
|
||
| `scripts/reparse_fundamentals.py` — **dry run by default**, `--apply` to write | new |
|
||
|
||
Unchanged rows are never touched; only accessions appearing in `staged.discrepancies` are
|
||
rewritten. The update writes the full `_SNAPSHOT_COLS` set via the same `_row_values()` the
|
||
insert uses, so a rewritten row can never be half old-parse and half new-parse. `created_at`
|
||
keeps its original value. Nothing is wired into the scheduler.
|
||
|
||
## A real bug the tests caught: false-positive discrepancies
|
||
|
||
`test_reparse_leaves_unchanged_rows_untouched` failed on first run — reparsing *identical*
|
||
data reported a change. Cause: `accepted_at` is written tz-aware UTC but
|
||
`DateTime(timezone=True)` only preserves tzinfo on Postgres; SQLite returns it naive, so
|
||
`_diff_fields` compared representations and saw a difference.
|
||
|
||
Left alone this would have made the dry-run report claim **every row needs rewriting** —
|
||
exactly the misleading signal that makes a blast-radius report worthless. `_diff_fields` now
|
||
compares datetime *instants* via `_same_value()`. This also fixes a latent false positive in
|
||
the pre-existing `snapshot_discrepancy` warning, which shares the same code path.
|
||
|
||
## Verification
|
||
|
||
4 reparse tests added, driven through the real import framework with the fake SEC client.
|
||
The key one seeds the database through the **pre-fix parser** (monkeypatching
|
||
`_YTD_TOLERANCE_DAYS` back to 20 so a 4-4-5 Q3 is rejected and stored as null), then reparses
|
||
with the fixed parser and asserts the row is rewritten in place with new provenance — the
|
||
production scenario end to end. Also covered: unchanged rows keep their original
|
||
`import_run_id`; `reparse=False` still reports and refuses to mutate; `force` bypasses the
|
||
no-op. Full suite: **804 passed**.
|
||
|
||
Not verifiable here: this reads and writes production Postgres, which is unreachable from
|
||
this machine, so the SQLite harness is the limit of what could be self-tested. The UPDATE is
|
||
plain SQLAlchemy Core with no dialect-specific constructs.
|
||
|
||
## Running it
|
||
|
||
```
|
||
python scripts/reparse_fundamentals.py # dry run, writes nothing
|
||
python scripts/reparse_fundamentals.py --apply # rewrite changed rows
|
||
```
|
||
|
||
Two cautions for whoever runs it:
|
||
|
||
- **Read the dry run for *kinds* of change, not just the count.** The tolerance 20→25 change
|
||
newly accepts facts for arbitrary filers, not only the names investigated here. Sample
|
||
changed rows for issuers that were never on the list and confirm they are recovered nulls
|
||
and corrected values — not something unexpected.
|
||
- **It refetches Company Facts for every tracked issuer** under the SEC throttle, because the
|
||
facts a fixed parser now accepts were never stored. Expect a long run; the dry run pays
|
||
that cost too, so budget for two passes.
|
||
|
||
Scope: this rewrites `fundamental_snapshots` only. Those rows currently feed the fundamentals
|
||
API/UI and the parity report — scoring still reads the legacy `fundamental_data` table, and
|
||
nothing in the backtest path touches `FundamentalSnapshot`. So a reparse **cannot** move
|
||
composite scores or backtests until the A5 cutover happens. The plan's "changed history
|
||
changes backtests" caution applies to workstream B's OHLCV rewrites, not to this.
|
||
|
||
---
|
||
|
||
# Fifth pass — period identity
|
||
|
||
The parser's own stated rule was *"period identity comes from `end == reportDate`, never
|
||
`fy/fp`"* — but `_fiscal_context()` derived the stored `fiscal_year`/`fiscal_period` by
|
||
majority-voting exactly those fy/fp fields. The labelling contradicted the module's own
|
||
principle, and SEC's labels are unreliable enough to break the quarter chain.
|
||
|
||
`_period_identity()` now derives both from `period_end` against the issuer's
|
||
`submissions.fiscalYearEnd`: **the form decides FY vs quarter** (a 10-Q can no longer be
|
||
labelled FY), and **distance to the fiscal-year end decides which quarter**. The MMDD is
|
||
threaded through `parse_snapshots(..., fiscal_year_end=...)`; without it the old fy/fp path
|
||
is used unchanged, so nothing regresses for issuers lacking a calendar.
|
||
|
||
**Rejected approach:** classifying the period by fact spans. Every 10-Q carries both a YTD
|
||
*and* a discrete fact ending at reportDate, so "best span match" reads COST's Q2 (167d) as a
|
||
Q1; and taking the *longest* span mislabelled IRM's Q3 2020 10-Q as FY because that filing
|
||
carries a 12-month fact. The prototype caught this as a regression on a working name before
|
||
any code was written. Distance-to-year-end needs no facts at all and is unambiguous — the
|
||
quarter bands sit 91 days apart, so ±35 absorbs even a 4-4-5 filer's 16-week Q4.
|
||
|
||
**Labels no longer match issuer naming in one case, deliberately.** A filer whose year ends
|
||
in early January (DPZ, `fiscalYearEnd` 0102) shifts by one. That is harmless: `fiscal_year`
|
||
and `fiscal_period` appear nowhere in the API schemas or routers — they are internal keys the
|
||
derivation uses for ordering, YTD differencing and YoY pairing, and the API surfaces
|
||
`period_end`. The requirement is uniqueness, monotonicity and YoY alignment, not nomenclature.
|
||
DPZ's derived values are byte-identical before and after the shift, which is the proof.
|
||
|
||
## Prototype evidence (before implementing)
|
||
|
||
Collisions = two period ends on one key, one silently discarded. Inversions = a period
|
||
sorting before one that precedes it.
|
||
|
||
| | CRM | FRT | STX | BXP | PPL | MTD | NTAP | WDAY | CRWD | COST | PEP | IRM | DPZ | AMCR | AAPL |
|
||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
||
| collisions before | 1 | 1 | 0 | 0 | 4 | 5 | 2 | 3 | 2 | 0 | 0 | 0 | 1 | 0 | 0 |
|
||
| collisions after | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
|
||
| inversions before | 1 | 1 | 1 | 0 | 2 | 10 | 2 | 3 | 5 | 1 | 0 | 0 | 2 | 0 | 0 |
|
||
| inversions after | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
|
||
|
||
## Validated on live data
|
||
|
||
8 of the 9 recover fully, every one matching the legacy provider to two decimals:
|
||
|
||
| name | TTM EPS | revenue growth | legacy |
|
||
|---|---|---|---|
|
||
| CRM | 8.63 | 10.9818 | 10.98 |
|
||
| FRT | 5.77 | 7.4263 | 7.43 |
|
||
| STX | 10.54 | 28.9227 | 28.92 |
|
||
| BXP | 1.99 | 1.6227 | 1.62 |
|
||
| MTD | 42.57 | 6.7785 | 6.78 |
|
||
| NTAP | 6.35 | 5.3713 | 5.37 |
|
||
| WDAY | 3.21 | 13.3165 | 13.32 |
|
||
| CRWD | −0.10 | 23.1667 | 23.17 |
|
||
|
||
**PPL is partial**: revenue growth recovers (8.3353) but TTM EPS is still null — its 2026 Q1
|
||
tags no diluted EPS variant at all, which is the open basic-vs-diluted decision, not this bug.
|
||
Note legacy claims −58.81% revenue growth for a utility; 8.34% is far more plausible.
|
||
|
||
**Two bonus recoveries**: FCX and REG had recovered EPS in the fourth pass but their revenue
|
||
growth was still blocked by label collisions. REG now reads 7.7569 against legacy's 7.76.
|
||
FCX reads 5.4378 against legacy's −24.23 — a genuine disagreement, likely the same
|
||
`Revenues` vs ASC-606 ambiguity flagged for DVN in §2, and worth resolving with that decision.
|
||
|
||
**Regression check — all byte-identical:** IRM 0.92/15.637543, COST 19.88/9.231107,
|
||
PEP 7.63/5.619741, DPZ 17.64/5.157289, AMCR null/64.834349, JPM 20.89/3.338823,
|
||
DVN 3.59/0.095648, AZO 145.39/5.740494. Nothing that worked before moved.
|
||
|
||
7 tests added at `_period_identity` covering each production shape (10-Q-labelled-FY,
|
||
December collision, January and mid-year ends, 4-4-5 quarters, the January-crossing shift,
|
||
and the no-calendar fallback). Full suite: **811 passed**.
|
||
|
||
## Reparse note
|
||
|
||
This changes `fiscal_year`/`fiscal_period` for a large share of rows — every non-December
|
||
filer, not only the broken ones. The dry-run count will be **much** larger than for the
|
||
earlier fixes, and that is expected. Read it by field: `fiscal_year`/`fiscal_period` churn is
|
||
the intended relabelling; changes to *value* columns are the recoveries.
|
||
|
||
## Where the 25 stand now
|
||
|
||
22 of 25 have a fundamental score again. Remaining: **XOM** (CIK identity, still unfixed) and
|
||
**PSKY / Q**, which are new registrants without enough filing history — correct behaviour,
|
||
not a bug.
|
||
|
||
---
|
||
|
||
# Sixth pass — CIK identity, and a much larger finding about share counts
|
||
|
||
## XOM: pinned, plus the validation that should have caught it
|
||
|
||
`company_tickers.json` maps XOM to CIK 2115436 "ExxonMobil Holdings Corp", which has **zero
|
||
XBRL filings**, while every 10-K/10-Q — including one filed 2026-05-04 — is still under CIK
|
||
34088. Which registrant is the real filer is a judgement about a corporate event, so it is
|
||
**pinned explicitly** rather than guessed:
|
||
|
||
- `sec_universe.cik_overrides()` reads a `{symbol: cik}` JSON map from
|
||
`SystemSetting['sec_cik_overrides']` and applies it ahead of `company_tickers.json`.
|
||
A malformed setting is logged and ignored, never fatal.
|
||
- **To fix XOM, set:** `sec_cik_overrides = {"XOM": 34088}`.
|
||
|
||
The more valuable half is that nothing noticed. A tracked issuer resolving to a registrant
|
||
with no XBRL filings can never produce a snapshot, and is restaged on *every* run forever.
|
||
The importer now records those in `staged.no_xbrl_filings`, reports them in the validation
|
||
summary (`no_xbrl_filings_count`), and raises a `no_xbrl_filings` SystemEvent naming the CIKs
|
||
and pointing at the override setting. It warns rather than fails — one misresolved ticker
|
||
must not block the whole import.
|
||
|
||
3 tests added. Full suite: **814 passed**.
|
||
|
||
## CHTR was not a bug, and the real problem is much bigger
|
||
|
||
I previously called this "a separate small bug". Both halves were wrong.
|
||
|
||
CHTR's `dei:EntityCommonStockSharesOutstanding` facts stop at **2016-06-30** — exactly when
|
||
the Time Warner Cable / Bright House deal closed and Charter became a multi-class issuer.
|
||
Since then the cover page reports the count **per share class**, which is dimensional, and
|
||
companyfacts is non-dimensional — so the facts are simply not in the API. Its recent filings
|
||
tag no consolidated common-share concept at all, only preferred and treasury.
|
||
|
||
This is not specific to CHTR. Of 12 issuers checked, **7 have no share count at all**:
|
||
|
||
| issuer | latest `shares_outstanding` | dei fact history |
|
||
|---|---|---|
|
||
| META | null (4/4 recent) | **never tagged** (n=0) |
|
||
| CMCSA | null (4/4 recent) | stops 2009-12-31 |
|
||
| BRK-B | null (4/4 recent) | stops 2011-04-29 |
|
||
| CHTR | null (4/4 recent) | stops 2016-06-30 |
|
||
| FOXA, NWSA, LEN | null (4/4 recent) | — |
|
||
| GOOGL / GOOG | 12,230,000,000 | works via the `us-gaap` fallback |
|
||
|
||
So **market cap is silently unavailable for a meaningful slice of the large-cap universe**,
|
||
and it is a source limitation rather than a parser defect: the two obvious workarounds are
|
||
both already-rejected design decisions — class sums are impossible (the per-class facts are
|
||
not in companyfacts at all), and the weighted-average diluted count is explicitly excluded
|
||
because market cap needs a point-in-time value.
|
||
|
||
**No code change made.** Substituting weighted-average diluted shares would silently
|
||
overturn a deliberate design decision and produce a subtly wrong market cap for exactly the
|
||
biggest, most-watched names. That is a call to make explicitly, so it is listed as a decision
|
||
below rather than quietly implemented.
|
||
|
||
---
|
||
|
||
# Seventh pass — multi-class share counts (decision taken: weighted-average fallback)
|
||
|
||
## Why this fallback, and why not the alternatives
|
||
|
||
Two candidates existed. The one **not** taken: derive the count as
|
||
`net_income ÷ diluted_eps` from columns already stored — no migration at all, and measured
|
||
accurate (GOOGL +0.48%, MRNA −0.45%, AAPL +0.19%, MSFT +0.18%). Rejected because it depends
|
||
on the derived quarter chain — the very thing these fixes have been repairing, and FOXA
|
||
already fails it — and because the two-class EPS method makes `net_income` differ from the
|
||
EPS numerator for exactly the multi-class issuers this targets.
|
||
|
||
Taken instead: store the **reported** `WeightedAverageNumberOfDilutedSharesOutstanding`.
|
||
It is the number the filer computed, needs no chain, and covers one issuer more.
|
||
|
||
| control | point-in-time | wavg diluted (latest qtr) | ratio |
|
||
|---|---|---|---|
|
||
| GOOGL | 12,230,000,000 | 12,309,000,000 | 0.9936 |
|
||
| MRNA | 396,786,259 | 395,000,000 | 1.0045 |
|
||
| AAPL | 14,687,356,000 | 14,725,873,000 | 0.9974 |
|
||
| MSFT | 7,428,434,704 | 7,445,000,000 | 0.9978 |
|
||
|
||
## Shape of the change
|
||
|
||
- **Migration 027** adds `fundamental_snapshots.weighted_avg_diluted_shares`. A separate
|
||
column, never backfilled into `shares_outstanding`, so the point-in-time column keeps its
|
||
strict meaning and the fallback stays a read-time decision.
|
||
- **Parser** stores the **shortest**-span fact ending at `period_end` (the most recent
|
||
quarter's average, closest to the current count) — deliberately not the YTD one, since an
|
||
average is not cumulative and the YTD convention does not apply.
|
||
- **Derivation** falls back only when the cover-page count is absent, and sets
|
||
`shares_outstanding_estimated`.
|
||
- **API** exposes `shares_estimated`, so `market_cap_est` and `fcf_yield` are never presented
|
||
as exact when they rest on a period average.
|
||
|
||
## Validated on live data
|
||
|
||
| issuer | shares_outstanding | estimated |
|
||
|---|---|---|
|
||
| GOOGL, AAPL, MSFT, MRNA | unchanged point-in-time values | **False** |
|
||
| META | 2,564,000,000 | True |
|
||
| CMCSA | 3,570,000,000 | True |
|
||
| CHTR | 126,849,271 | True |
|
||
| FOXA | 432,000,000 | True |
|
||
| NWSA | 555,700,000 | True |
|
||
| LEN | 240,776,000 | True |
|
||
| **BRK-B** | **still null** | False |
|
||
|
||
6 of 7 recovered, no regression on the controls. **BRK-B remains unavailable** and honestly
|
||
so: Berkshire reports per *equivalent Class A share*, dimensionally, so it has no consolidated
|
||
weighted-average fact either. Nothing in companyfacts can give it a share count.
|
||
|
||
Known caveat, accepted: for issuers using the two-class method the count is the EPS
|
||
denominator. For CHTR that is Class A only — which is also the basis on which Charter's equity
|
||
market cap is normally quoted, so it is the right number for this purpose, but it is not
|
||
"all shares of all classes".
|
||
|
||
3 tests added. Full suite: **817 passed**. Alembic single head at 027.
|
||
|
||
**Needs the reparse to land:** existing rows have `weighted_avg_diluted_shares = NULL` until
|
||
`scripts/reparse_fundamentals.py --apply` runs, so market cap stays missing for these issuers
|
||
until then.
|
||
|
||
---
|
||
|
||
# Eighth pass — revenue basis (decision: keep ASC-606, no change)
|
||
|
||
The two concepts measure different things: `RevenueFromContractWithCustomerExcludingAssessedTax`
|
||
is customer-contract revenue (an E&P's oil/gas/NGL sales), while `Revenues` is the total
|
||
income-statement line, which for commodity producers folds in mark-to-market derivative
|
||
gains/losses. That is why DVN's ASC-606 figure is *larger*: 4,508M of sales minus ~701M of
|
||
hedging losses gives the 3,807M `Revenues` line.
|
||
|
||
Measured across 21 issuers (deliberately energy-weighted, where the gap concentrates):
|
||
|
||
- Both tags present and differing >1%: **5 of 21** — DVN +18.4%, COP −14.3%, OXY +6.5%,
|
||
FCX −2.8%, PPL +1.6%. Everyone else tags one, or they are identical (COST +0.0%).
|
||
- Concept choice **flips within an issuer's chain: 0 of 21**. Whichever tag wins, the series
|
||
is internally consistent, so YoY never compares two definitions.
|
||
|
||
**Decision: keep ASC-606 first, change nothing.** Derivative gains/losses are mean-reverting
|
||
and sign-flipping; folding them into "revenue growth" turns the sub-score into a partial
|
||
hedging-P&L read for exactly the affected names. The consistency argument for switching is
|
||
empirically absent (zero flips), and changing would churn every dual-tagging issuer's stored
|
||
value — widening the reparse diff — to make ~5 names noisier.
|
||
|
||
**Correction to the fourth/fifth-pass note:** FCX's disagreement with legacy (+5.44% vs
|
||
−24.23%) is **not** this ambiguity. Its two tags differ by only 2.8%, and FCX's own revenue
|
||
rose 22,703M → 25,186M YoY, so −24% is not credible — legacy is simply wrong there, and this
|
||
decision does not touch it. So the basis choice moves only DVN, COP, OXY.
|
||
|
||
The mirror hazard — an issuer where ASC-606 is only a *fragment* of revenue (a bank's fee
|
||
income) — was checked (all 15 recovered banks resolve total revenue, not a fragment). A
|
||
fragment-detection warning was prototyped and then **removed**: with no UI surface it would
|
||
only have lived in the run summary, and the case it guards against is not currently present.
|
||
Documented and closed rather than shipped as dead plumbing. If a fragment case ever appears,
|
||
it shows up as an implausibly low revenue in the next parity report.
|
||
|
||
---
|
||
|
||
# Ninth pass — basic-EPS fallback (PPL) and HAL resolved
|
||
|
||
## PPL: basic-EPS fallback (decision taken)
|
||
|
||
PPL's 2026 Q1 tags no diluted EPS variant at all, only basic — a single-filing omission
|
||
(its other quarters tag diluted), but that one missing period broke the quarter chain and
|
||
nulled TTM. `EarningsPerShareBasic` / `IncomeLossFromContinuingOperationsPerBasicShare` are
|
||
now appended to `_EPS_CONCEPTS`, last, so they only fire when no diluted variant exists.
|
||
|
||
Evidence (19-name scan): a basic fallback helps exactly **1 name (PPL)**. Basic-vs-diluted is
|
||
~0.5–1.2% for most, +1.2% for PPL. The one name where it genuinely diverges (TSLA +13.3%)
|
||
already tags diluted, so it never reaches the fallback. Basic is always ≥ diluted, so the
|
||
result slightly overstates EPS / understates P/E — accepted, since it fires only on an
|
||
otherwise-null period.
|
||
|
||
Validated: PPL TTM EPS null → **1.63** (≈$36 / 1.63 = 22.1 vs legacy P/E 22.43). AAPL, MSFT,
|
||
DUK, HAL unchanged — diluted still wins wherever present. 2 tests added. Full suite: **819
|
||
passed**.
|
||
|
||
## HAL: resolved, and it was never our bug
|
||
|
||
HAL's TTM EPS is now **1.81** (≈$33 / 1.81 = 18.2 vs legacy P/E 18.01) — the period-identity
|
||
and EPS-concept work already fixed it. The "unexplained null" is closed.
|
||
|
||
Its 2024 EPS values are garbage (680000, 1480000, …) because **Halliburton's own 2024 XBRL
|
||
tags `EarningsPerShareDiluted = 680000` in unit USD/shares** — a filer scale error in the
|
||
source, faithfully stored. It only poisons TTM windows that include 2024, which the current
|
||
point-in-time report does not use, so no code change: clamping EPS to "plausible" values would
|
||
risk masking real ones. Documented as a known source-data quirk.
|
||
|
||
This does surface a latent robustness point (not acted on): a single fat-fingered per-share
|
||
value poisons any TTM window it lands in. It is invisible in the current report and out of
|
||
scope here, but worth a note if historical TTM series are ever surfaced.
|
||
|
||
## All 25 lost names accounted for
|
||
|
||
| status | names |
|
||
|---|---|
|
||
| **recovered** (22) | ARE, AZO, BXP, COST, CRM, CRWD, DPZ, DVN, FCX, FRT, HAL, KHC, MOS, MTD, NTAP, PEP, PPL, REG, SJM, STX, SWKS, WDAY |
|
||
| **XOM** | fixed by the `sec_cik_overrides` pin (needs the setting applied) |
|
||
| **PSKY, Q** | new registrants without enough filing history — correct behaviour, not a bug |
|
||
|
||
## Still outstanding
|
||
|
||
Revised after the second pass, in the order I would take them:
|
||
|
||
Everything actionable without a live database is now done. What remains is one hard
|
||
data limitation and two operational steps that only run against production.
|
||
|
||
1. ~~**§3a fiscal-period identity**~~ — **done**, fifth pass.
|
||
2. ~~**§1 split contamination, part (a)**~~ — **done**, third pass.
|
||
3. ~~**§5 EPS concept gap**~~ — **done**, third pass.
|
||
4. ~~**§4 XOM CIK remap** + zero-filings validation~~ — **done**, sixth pass.
|
||
5. ~~**Reparse path**~~ — **done**, fourth pass.
|
||
6. ~~**Multi-class share counts**~~ — **done**, seventh pass (weighted-average fallback).
|
||
7. ~~**DVN/FCX revenue basis**~~ — **decided**, eighth pass (keep ASC-606, no change).
|
||
8. ~~**PPL basic-EPS fallback**~~ — **done**, ninth pass.
|
||
9. ~~**HAL null TTM EPS**~~ — **resolved**, ninth pass (already fixed; 2024 is a filer error).
|
||
|
||
## Review finding — two fixes on this branch silently interacted
|
||
|
||
Caught in review, not by me. `_merge_amendments` (the per-field amendment fix, first pass)
|
||
builds the merged period from `_MERGED_FIELDS` + `_CARRIED_FIELDS` alone, so a column in
|
||
neither list is **absent** from the merged row, not merely stale — and every caller reads it
|
||
with `getattr(row, name, None)`, which quietly returns `None`.
|
||
|
||
`weighted_avg_diluted_shares` (the market-cap fallback, seventh pass) was never added to
|
||
`_MERGED_FIELDS`. The failure needed both fixes to be present at once: a multi-class issuer
|
||
*and* a partial amendment on its latest period — META with a Part-III-only 10-K/A — would
|
||
silently lose market cap and FCF yield again, i.e. the seventh pass's fix undone by the
|
||
first pass's mechanism. I updated `_SNAPSHOT_COLS` in the importer when adding the column
|
||
but not `_MERGED_FIELDS` in the derivation.
|
||
|
||
Fixed, with a regression test for the specific case. The more useful addition is a guard —
|
||
`test_merge_lists_cover_every_parser_field` asserts the two lists cover every `SnapshotRow`
|
||
field, so the *next* column added fails loudly instead of losing data quietly. Verified it
|
||
would have caught this one.
|
||
|
||
Lesson worth keeping: a hand-maintained field list that reconstructs an object is a silent
|
||
data-loss footgun. `_SNAPSHOT_COLS` (importer) and `_MERGED_FIELDS` (derivation) must both
|
||
track the parser's `SnapshotRow`, and only one of them is now enforced by a test.
|
||
|
||
## Genuinely unfixable from this data
|
||
|
||
- **§1 part (b)** — a split post-dating the last filing (KLAC). No snapshot carries
|
||
share-count evidence, so no guard built on share counts can fire. Needs a corporate-actions
|
||
source or an external market-cap reconciliation.
|
||
- **BRK-B market cap** — Berkshire reports per equivalent Class A share, dimensionally, so it
|
||
has neither a cover-page count nor a weighted-average one. Nothing in companyfacts can give
|
||
it a share count.
|
||
|
||
## Operational steps (production only — cannot run from here)
|
||
|
||
- Apply the setting `sec_cik_overrides = {"XOM": 34088}`.
|
||
- Run `scripts/reparse_fundamentals.py` — dry run first, then `--apply`. This is what lands
|
||
every parser-side fix (revenue/EPS concepts, Q3 span, period identity, weighted-average
|
||
shares via migration 027) onto existing rows. Until it runs, those fixes are inert in prod.
|
||
|
||
## Standing decision, revisit only if it bites
|
||
|
||
- **COF-class share-change threshold** — kept at 25%. Revisit only if the 3.2% universe
|
||
hit-rate proves painful.
|
||
|
||
## Known source-data quirk, not acted on
|
||
|
||
- A single fat-fingered per-share value in a filer's XBRL (HAL 2024) poisons any TTM window
|
||
it lands in. Invisible in the current point-in-time report; relevant only if historical TTM
|
||
series are ever surfaced.
|
||
|
||
PSKY and Q need nothing — they are new registrants without enough filing history, which is
|
||
correct behaviour.
|
||
|
||
---
|
||
|
||
# Closing — post-reparse verification (2026-07-24)
|
||
|
||
## Production reparse
|
||
|
||
Two apply runs against prod (`scripts/reparse_fundamentals.py --apply`):
|
||
|
||
- **Run 6** (all fixes through the seventh pass): 262 inserted, 28,664 rewritten —
|
||
99.4% of which was backfilling the new `weighted_avg_diluted_shares` column; the
|
||
behavioural churn matched the dry run exactly. The five duration facts clustering at
|
||
190–234 changed rows each is the 4-4-5 Q3 recovery signature. `accepted_at` changed on
|
||
only 76 rows (0.25%), confirming the tz-comparison fix works against real Postgres.
|
||
- **Run 7** (after the fiscal-year-end fix below): 3 inserted, 322 rewritten — BEN, DELL,
|
||
and boundary-year relabels for 53-week filers whose derived MMDD shifted a few days.
|
||
|
||
`cik_updates: 1` on run 6 was the XOM pin taking effect; XOM now has 68 snapshot rows,
|
||
latest period end 2026-03-31.
|
||
|
||
## Regression caught by the collision check — and its fix
|
||
|
||
The before/after key-collision query (~130 rows max 6 → 44 rows all 2) surfaced one real
|
||
regression: **BEN**. `submissions.fiscalYearEnd` declares `1231` while every Franklin
|
||
Resources 10-K ends 09-30, so `_period_identity` — which trusted the declared value — put
|
||
BEN's real fiscal Q1 zero days from the claimed year end (no band matched) and labelled its
|
||
fiscal Q2 as Q1. The collision discarded a period and BEN lost TTM EPS and revenue growth it
|
||
had before the branch. Fixed in `3d42ca7`: `resolve_fiscal_year_end()` prefers the issuer's
|
||
own most recent 10-K reportDate (which *is* the fiscal year end by definition) and treats
|
||
the declared field as fallback. Full-universe scan: 2 of 506 issuers mis-declare (BEN 91d,
|
||
DELL 29d); both now derive correctly (BEN rg 3.8243 vs legacy 3.82; DELL 38.5735 vs 38.57).
|
||
|
||
Residual collisions after run 7: 36 rows, all count-2, **latest year 2023** — the 53-week
|
||
drift class (AVY/CDNS/RVTY/JNJ/TDY/DPZ at 5–6-year intervals). Newest-wins degrades one
|
||
historical FY row; no current period is affected. Left alone deliberately: eliminating them
|
||
means modelling each filer's actual 52/53-week calendar per year, for rows feeding no
|
||
current metric.
|
||
|
||
## The verdict: 2026-07-24 parity report vs the 2026-07-23 baseline
|
||
|
||
| metric | baseline | after | |
|
||
|---|---|---|---|
|
||
| candidate scored | 482 | **504** | legacy scores 507; gap = PSKY, Q (new registrants) + FITB |
|
||
| revenue_growth candidate available | 442 | **489** | banks, REITs, 4-4-5 recovered |
|
||
| pe_ratio candidate available | 432 | **452** | net of the split-guard nulls |
|
||
| revenue_growth median abs delta | 0.0038 | **0.0038** | 47 names added at unchanged agreement |
|
||
| pe_ratio median / p95 abs delta | 0.5883 / 7.73 | **0.5576 / 6.03** | corrupted outliers gone |
|
||
|
||
Revenue `material_differences` rose 84 → 96: the newly compared names include the cases
|
||
where **legacy is the wrong side** (JPM 108.98% vs 3.34%, PPL −58.81% vs 8.34%, FCX −24.23%
|
||
vs +5.44%). Material is symmetric; these flag the provider being corrected.
|
||
|
||
The split guard is visible in the report: BKNG (1.10), COF, TPL, AMCR, WAT all null P/E now.
|
||
**KLAC (6.19) is the one known-wrong value left** — the post-filing split documented as
|
||
unfixable without a corporate-actions source.
|
||
|
||
**Correction to the seventh pass:** the claim that guard-tripped issuers "keep their
|
||
fundamental score, losing one of three inputs" fails for **FITB**, the one name that also
|
||
lacks revenue growth (its recent filings tag only ASC-606 fee-income fragments, then nothing)
|
||
— nulling its contaminated P/E (Comerica merger, 661M → 902M shares) drops it to one metric
|
||
and no score. Accepted: the composite renormalises, and legacy's 58% "revenue growth" for a
|
||
bank was itself junk.
|
||
|
||
## Recommendation
|
||
|
||
The A5 gate evidence now supports approving the cutover: coverage within 3 of legacy with
|
||
every gap explained, agreement essentially exact where both sides exist, every corrupted
|
||
value either fixed or deliberately nulled with a caveat, and the remaining score deltas are
|
||
documented definition differences — called out, not averaged away, as the plan requires.
|
||
Carry KLAC as the one known caveat in the approval note.
|