fix(sec): tell an attribution collision apart from a changed reconstruction
snapshot_discrepancy named the accessions but not the columns, so it could not distinguish "our numbers moved" from "the same filing is attributed twice". The fields were already computed for validation_json and simply dropped from the message; they are now in it. A difference in cik ALONE is no longer reported as a reconstruction change at all. Every fact matched, so two tracked CIKs are claiming one filing and the fix is the universe, not the parser: it raises accession_cik_collision naming both CIKs and sec_cik_overrides. It also never self-heals — the losing CIK stores no row, so _ciks_with_snapshots never sees it and it is full-history backfilled and re-reported every run until its ticker is re-pointed or retired. Observed 2026-08-19 for EQR: after Equity Residential renamed to Vivmark Residential (VMRK, CIK 906107), SEC's own company_tickers.json left the old symbol on ERP Operating LP (CIK 931182), the non-traded co-registrant of their combined 10-Qs. Both were tracked, both reconstructed the same two filings. The reparse path now excludes cik-only differences from its rewrite set: rewriting one would re-stamp the filing onto the co-registrant, taking it from the issuer that actually filed it, which no parser fix asks for. No stored value was wrong in that incident — reports/ carries the full reproduction for this and for the companyfacts staleness behind the gate fix. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Lo99z3jWqu9X9ueBU3Z3D
This commit is contained in:
Claude Opus 5
@@ -0,0 +1,150 @@
|
||||
# SEC fundamentals alerts, 2026-08-21
|
||||
|
||||
Two `sec_facts` warnings, investigated against live SEC data. Both originate in SEC's
|
||||
own published data — a stale per-company Company-Facts file (1) and a stale
|
||||
ticker→CIK mapping (2) — and neither is a parser defect: no stored fundamental value
|
||||
is wrong. Every SEC-side probe below reproduces offline from public endpoints; the
|
||||
four database facts used are quoted where they appear.
|
||||
|
||||
## 1. `filing_gap_aged` — 43 gaps, all `not_in_companyfacts`
|
||||
|
||||
**Root cause: SEC's per-company Company-Facts files are stale for these issuers,
|
||||
while the same filings are present in SEC's own `frames` aggregation.**
|
||||
|
||||
All ten named filings are real 10-Qs filed 2026-07-28/29, present in the issuer's
|
||||
`submissions` with `isXBRL=1`, with complete R-files and XBRL in the EDGAR archive
|
||||
— and absent from `companyfacts/CIK*.json`:
|
||||
|
||||
| CIK | issuer | accession | filed | in `companyfacts` | newest fact in file |
|
||||
|---|---|---|---|---|---|
|
||||
| 0000001800 | Abbott | 0001628280-26-050134 | 2026-07-28 | no | 2026-04-29 |
|
||||
| 0000021344 | Coca-Cola | 0001628280-26-050503 | 2026-07-29 | no | 2026-04-30 |
|
||||
| 0000024741 | Corning | 0000024741-26-000255 | 2026-07-29 | no | 2026-05-01 |
|
||||
| 0000029989 | Omnicom | 0000029989-26-000019 | 2026-07-29 | no | 2026-04-29 |
|
||||
| 0000037996 | Ford | 0000037996-26-000156 | 2026-07-29 | no | 2026-04-30 |
|
||||
| 0000040533 | General Dynamics | 0000040533-26-000032 | 2026-07-29 | no | 2026-07-01 |
|
||||
| 0000048898 | Hubbell | 0001628280-26-050405 | 2026-07-29 | no | 2026-06-04 |
|
||||
| 0000049071 | Humana | 0000049071-26-000050 | 2026-07-29 | no | 2026-04-29 |
|
||||
| 0000049196 | Huntington Bancshares | 0000049196-26-000066 | 2026-07-28 | no | 2026-04-30 |
|
||||
| 0000062996 | Masco | 0000062996-26-000027 | 2026-07-29 | no | 2026-04-22 |
|
||||
|
||||
Ruled out, with evidence:
|
||||
|
||||
- **Not a global SEC outage.** Company Facts is current for other issuers filing the
|
||||
same days — MSFT `0001193125-26-323660` @2026-07-29, AAPL @2026-07-31, P&G
|
||||
@2026-08-04, Chevron @2026-08-06, JPMorgan @2026-08-20.
|
||||
- **Not a CDN/cache artifact.** A cache-busted request with `Cache-Control: no-cache`
|
||||
returns the identical stale 3.39 MB payload; the response carries no cache headers.
|
||||
- **Not our filter.** The scan covers every taxonomy/concept/unit in the payload.
|
||||
- **Not a metadata discriminator.** Gap and non-gap filings are identical on
|
||||
`isXBRL`, `isInlineXBRL`, `reportDate`, `primaryDocDescription`.
|
||||
- **SEC does have the facts.** `frames/us-gaap/Assets/USD/CY2026Q2I.json` lists
|
||||
Abbott at exactly the missing accession `0001628280-26-050134`, and Coca-Cola and
|
||||
Ford at theirs. The per-company endpoints are the degraded ones:
|
||||
`companyconcept/CIK0000001800/us-gaap/Assets.json` returns `"units":{"USD":{}}`.
|
||||
|
||||
**Consequence, and why the gate changed.** Retrying `companyfacts` cannot recover
|
||||
these — Abbott's file has been stale since April. And because `active_gaps`
|
||||
supersedes a gap only on a *successfully ingested later* filing, a stale file also
|
||||
swallows Q3: the pause was open-ended, not seasonal, on 43 large caps.
|
||||
|
||||
**Fix** (`app/services/fundamentals_quality_service.py`): once `filing_gap_aged` has
|
||||
escalated a gap (`escalated_at`), it stops pausing setups **if** the issuer's own
|
||||
newest stored 10-K/10-Q is under `GAP_GATE_RECENT_FILING_DAYS` (180) old. Pause hands
|
||||
off to the alert; an issuer with nothing that recent stays paused. `active_gaps` is
|
||||
deliberately untouched, so `_retry_backlog` keeps retrying and a recovered filing
|
||||
still resolves normally. The bound is applied to the queue path *and* the
|
||||
`validation_json` summary path, which mirrors the same filings — bounding only one
|
||||
leaves the behaviour unchanged in production.
|
||||
|
||||
**This is a bounded reprieve, not a removal — know the two ways it ends.** Abbott's
|
||||
newest ingested filing is `0001628280-26-028357`, filed 2026-04-29, so its recency
|
||||
window closes around **2026-10-26**; most of the 43 sit on late-April filings and
|
||||
turn back to paused within days of each other. That crossing is **silent**: the
|
||||
importer escalates only gaps with `escalated_at IS NULL`, so `filing_gap_aged` does
|
||||
not re-fire for a gap it has already reported. Separately, a Q3 10-Q that also fails
|
||||
to ingest creates a *new* un-escalated gap on the same CIK, which re-pauses it at
|
||||
once (that one does raise its own `filing_gap_aged` 14 days later). Whether the
|
||||
silent re-block deserves a re-escalation signal is an open call, deliberately not
|
||||
made here — "one actionable escalation rather than a daily warning" is the existing
|
||||
design intent.
|
||||
|
||||
**Not done, with reasons.** A `frames`-backed recovery source was considered and
|
||||
rejected: frames are calendar-aligned with a tolerance (off-fiscal filers drop out)
|
||||
and carry one fact per issuer per period, so amendment/restatement semantics differ
|
||||
from Company Facts — lossy as a snapshot source, not merely expensive. Parsing the
|
||||
filing's own inline-XBRL instance is the authoritative alternative but is a new
|
||||
subsystem (contexts, dimensions, unit refs) duplicating the parser's fact model.
|
||||
|
||||
## 2. `snapshot_discrepancy` — 0000906107-15-000012 / -000016
|
||||
|
||||
**Root cause: two tracked tickers claim the same filing, because SEC's
|
||||
`company_tickers.json` still points the old symbol at a non-traded co-registrant.
|
||||
No stored value is wrong and no reparse is warranted.**
|
||||
|
||||
CIK 0000906107 is **Vivmark Residential** (VMRK, formerly Equity Residential). Both
|
||||
alerted accessions are **combined EQR + ERP Operating LP 10-Qs** — one accession, two
|
||||
registrants (0000906107 and 0000931182) — the pattern behind the existing
|
||||
co-registrant recovery path.
|
||||
|
||||
The stored rows are **byte-identical** to what the current parser reconstructs from
|
||||
EQR's own Company Facts — every column, verified: `cik` (`0000906107`), `form`,
|
||||
`filed_date`, `accepted_at`, both period dates, `fiscal_year`/`fiscal_period`,
|
||||
`revenue`, `net_income`, `operating_income`, `diluted_eps`, `cfo`, the two nulls,
|
||||
`cash_and_st_investments`, `total_debt` (340,900,000 / null),
|
||||
`shares_outstanding`, `shares_outstanding_date`, `weighted_avg_diluted_shares`. Both
|
||||
carry `import_run_id = 6`, and CIK 0000906107 holds all 69 of its filings across runs
|
||||
6–30, so the issuer's own history is complete.
|
||||
|
||||
Run 63 (2026-08-19) recorded
|
||||
`fields: ["cik"]` for both accessions, and the universe explains it:
|
||||
|
||||
```
|
||||
tickers: VMRK -> 0000906107 (Vivmark Residential, ex-Equity Residential)
|
||||
EQR -> 0000931182 (ERP Operating Ltd Partnership)
|
||||
```
|
||||
|
||||
SEC's own `company_tickers.json` carries `{"cik_str": 931182, "ticker": "EQR",
|
||||
"title": "ERP OPERATING LTD PARTNERSHIP"}` — after the rename, the old symbol stayed
|
||||
attached to the **non-traded operating partnership**, the co-registrant on those
|
||||
combined 10-Qs. `resolve_ciks` reads `active_only` tickers and follows SEC, so
|
||||
0000931182 is tracked. Its Company Facts holds 7 accessions, exactly 2 of them
|
||||
EQR-prefixed, so its backfill reconstructs exactly those two rows, stamps them
|
||||
`cik=0000931182`, and collides with the rows already stored under 0000906107 —
|
||||
identical in every fact, differing only in attribution.
|
||||
|
||||
It cannot self-heal. The collision loser never stores a row (the insert is skipped as
|
||||
immutable), so `_ciks_with_snapshots` never sees 0000931182, and it is full-history
|
||||
backfilled — refetching every submissions shard and its companyfacts — **on every
|
||||
run**, re-raising the warning each time. `fundamental_snapshots` for 0000906107 holds
|
||||
all 69 filings across runs 6–30, so the issuer's own history is complete and correct.
|
||||
|
||||
### Fixes
|
||||
|
||||
**Code** (`sec_fundamentals_importer.py`): a `cik`-only difference is no longer
|
||||
reported as a reconstruction discrepancy. It raises `accession_cik_collision`, naming
|
||||
both CIKs and pointing at `sec_cik_overrides`, because the fix is the universe, not
|
||||
the parser. The reparse path also excludes these from its rewrite set — rewriting a
|
||||
cik-only difference would re-stamp the filing onto the co-registrant and take it from
|
||||
the issuer that filed it. (A reparse run while both CIKs are tracked fails validation
|
||||
on `duplicate accession in staged snapshots` instead, which is a safe stop.)
|
||||
|
||||
**Data — needs an operator, and the alert repeats daily until then.** `EQR` is a stale
|
||||
symbol: the security now trades as `VMRK`, which is already tracked at the correct
|
||||
CIK. Retiring the `EQR` ticker ends the loop. A `sec_cik_overrides` pin of
|
||||
`EQR -> 906107` would silence the collision but leave two tickers on one security,
|
||||
double-counting the issuer in scans — retirement is the right action.
|
||||
|
||||
**Not fixed, deliberately:** the permanent-backfill loop itself. A tracked CIK whose
|
||||
only parseable filings belong to another CIK is re-backfilled every run; ending that
|
||||
in code means teaching `_ciks_with_snapshots` about foreign-owned accessions, which is
|
||||
more state for a condition that is now loudly and specifically reported.
|
||||
|
||||
### Separate observation: `total_debt` on this issuer looks wrong
|
||||
|
||||
Independent of the alert, and unchanged by any fix here: the parser reconstructs
|
||||
`total_debt = 340,900,000` for EQR's 2015 Q1 and `null` for Q2, while the REIT carried
|
||||
roughly $10bn of debt. `_compose_debt` returns the short-term component alone when
|
||||
every `_LONG_TERM_DEBT_AGG` concept **and** the `LongTermDebtNoncurrent`/`Current`
|
||||
pair miss — which is what happened here, and Q2 matched neither. Worth checking
|
||||
against a current REIT filer before trusting `total_debt` for that sector.
|
||||
Reference in New Issue
Block a user