fix(sec): tell an attribution collision apart from a changed reconstruction
snapshot_discrepancy named the accessions but not the columns, so it could not distinguish "our numbers moved" from "the same filing is attributed twice". The fields were already computed for validation_json and simply dropped from the message; they are now in it. A difference in cik ALONE is no longer reported as a reconstruction change at all. Every fact matched, so two tracked CIKs are claiming one filing and the fix is the universe, not the parser: it raises accession_cik_collision naming both CIKs and sec_cik_overrides. It also never self-heals — the losing CIK stores no row, so _ciks_with_snapshots never sees it and it is full-history backfilled and re-reported every run until its ticker is re-pointed or retired. Observed 2026-08-19 for EQR: after Equity Residential renamed to Vivmark Residential (VMRK, CIK 906107), SEC's own company_tickers.json left the old symbol on ERP Operating LP (CIK 931182), the non-traded co-registrant of their combined 10-Qs. Both were tracked, both reconstructed the same two filings. The reparse path now excludes cik-only differences from its rewrite set: rewriting one would re-stamp the filing onto the co-registrant, taking it from the issuer that actually filed it, which no parser fix asks for. No stored value was wrong in that incident — reports/ carries the full reproduction for this and for the companyfacts staleness behind the gate fix. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Lo99z3jWqu9X9ueBU3Z3D
This commit is contained in:
@@ -36,7 +36,11 @@ Guardrails (design + reviews):
|
||||
excluded from actionable setups until its filing is recovered.
|
||||
- ``promote`` inserts snapshots ``ON CONFLICT (accession) DO NOTHING`` (immutable),
|
||||
reports differing existing accessions, and applies ticker updates in the same
|
||||
transaction.
|
||||
transaction. A difference in ``cik`` **alone** is reported separately as an
|
||||
``accession_cik_collision``: every fact matched, so two tracked CIKs are
|
||||
claiming one filing and the fix is the universe, not the parser. It never
|
||||
self-heals on its own — the losing CIK stores no row, so it is backfilled and
|
||||
re-reported every run until its ticker is re-pointed or retired.
|
||||
- ``reparse=True`` is the one exception to immutability, and it is deliberate:
|
||||
it restages every accession with the current parser and **rewrites** the rows
|
||||
that now reconstruct differently. Immutability protects SEC's record (one row
|
||||
@@ -67,7 +71,7 @@ from app.services import sec_universe
|
||||
from app.services.data_import import STATUS_PROMOTED, ValidationResult
|
||||
from app.services.sec_client import SecClient, SecError, cik10
|
||||
from app.services.sec_facts_parser import FilingMeta, SnapshotRow
|
||||
from app.services.sec_universe import ResolvedUniverse
|
||||
from app.services.sec_universe import CIK_OVERRIDES_KEY, ResolvedUniverse
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -281,7 +285,18 @@ class SecFundamentalsImporter:
|
||||
if old is not None:
|
||||
fields = _diff_fields(row, old)
|
||||
if fields:
|
||||
staged.discrepancies.append({"accession": row.accession, "fields": fields})
|
||||
# Carry both CIKs. promote() reads a bare ["cik"] as an
|
||||
# attribution collision rather than a changed
|
||||
# reconstruction, which holds only because _COMPARE_COLS
|
||||
# spans every stored fact: a fact column added to the
|
||||
# model but not to _SNAPSHOT_COLS would go uncompared and
|
||||
# let a real difference through as a collision.
|
||||
staged.discrepancies.append({
|
||||
"accession": row.accession,
|
||||
"fields": fields,
|
||||
"cik": row.cik,
|
||||
"stored_cik": old.cik,
|
||||
})
|
||||
return staged
|
||||
|
||||
async def _stage_issuer(
|
||||
@@ -555,8 +570,15 @@ class SecFundamentalsImporter:
|
||||
inserted = 0
|
||||
updated = 0
|
||||
# Only accessions whose reconstruction actually changed are rewritten;
|
||||
# an unchanged stored row is left completely alone.
|
||||
changed = {d["accession"] for d in staged.discrepancies} if self.reparse else set()
|
||||
# an unchanged stored row is left completely alone. A cik-only difference
|
||||
# is excluded on purpose: the facts are identical there, so rewriting
|
||||
# would re-stamp the filing onto the colliding co-registrant — taking it
|
||||
# from the issuer that actually filed it, which no parser fix asks for.
|
||||
changed = (
|
||||
{d["accession"] for d in staged.discrepancies if d["fields"] != ["cik"]}
|
||||
if self.reparse
|
||||
else set()
|
||||
)
|
||||
for row in staged.rows:
|
||||
if row.accession in staged.existing_accessions:
|
||||
if row.accession in changed:
|
||||
@@ -641,10 +663,51 @@ class SecFundamentalsImporter:
|
||||
if gap["accession"] not in existing_gap_accessions
|
||||
]
|
||||
|
||||
# Two tracked issuers claiming one filing is not a reconstruction change:
|
||||
# every fact matched and only the CIK stamp differs, so re-parsing or
|
||||
# reparsing fixes nothing — the universe resolution does. It is reported
|
||||
# separately because it also does not self-heal: the loser of the
|
||||
# collision never stores a row, so `_ciks_with_snapshots` never sees it,
|
||||
# and it is full-history backfilled (and re-reported) on every run until
|
||||
# a human re-points or retires the ticker. Observed 2026-08 for EQR,
|
||||
# which SEC's own company_tickers.json maps to ERP Operating LP, the
|
||||
# non-traded co-registrant of the issuer now trading as VMRK.
|
||||
collisions = [d for d in staged.discrepancies if d["fields"] == ["cik"]]
|
||||
if collisions:
|
||||
named = ", ".join(
|
||||
f"{d['accession']} (stored {d['stored_cik']}, parsed {d['cik']})"
|
||||
for d in collisions[:10]
|
||||
)
|
||||
db.add(SystemEvent(
|
||||
severity="warning",
|
||||
source="sec_facts",
|
||||
code="accession_cik_collision",
|
||||
message=(
|
||||
f"{len(collisions)} filing(s) are claimed by two tracked CIKs — "
|
||||
"the reconstruction is identical, only the attribution differs, "
|
||||
"so one of the two is a co-registrant the universe should not "
|
||||
f"track. Re-point or retire the ticker (see {CIK_OVERRIDES_KEY}); "
|
||||
f"this repeats every run until then: {named}"
|
||||
)[:4000],
|
||||
dedup_key=f"sec_facts:accession_cik_collision:{run_id}",
|
||||
created_at=_now(),
|
||||
))
|
||||
|
||||
# Warn (in-transaction, so it commits atomically with the promotion) when
|
||||
# any existing accession reconstructed differently — kept immutable.
|
||||
if staged.discrepancies:
|
||||
accns = ", ".join(d["accession"] for d in staged.discrepancies[:10])
|
||||
reconstruction_diffs = [
|
||||
d for d in staged.discrepancies if d["fields"] != ["cik"]
|
||||
]
|
||||
if reconstruction_diffs:
|
||||
# Name the columns, not just the accession: "differs in revenue"
|
||||
# (our numbers moved) and "differs in period_start" (the filing was
|
||||
# re-placed in the calendar) need different responses, and the alert
|
||||
# is where that call gets made. The fields are already computed for
|
||||
# validation_json — they were simply dropped from the message.
|
||||
accns = ", ".join(
|
||||
f"{d['accession']} ({', '.join(d['fields'])})"
|
||||
for d in reconstruction_diffs[:10]
|
||||
)
|
||||
disposition = (
|
||||
f"REWRITTEN by reparse run {run_id}" if self.reparse else "kept immutable"
|
||||
)
|
||||
@@ -653,7 +716,7 @@ class SecFundamentalsImporter:
|
||||
source="sec_facts",
|
||||
code="snapshot_reparse" if self.reparse else "snapshot_discrepancy",
|
||||
message=(
|
||||
f"{len(staged.discrepancies)} stored accession(s) reconstructed "
|
||||
f"{len(reconstruction_diffs)} stored accession(s) reconstructed "
|
||||
f"differently; {disposition}: {accns}"
|
||||
)[:4000],
|
||||
dedup_key=f"sec_facts:discrepancy:{run_id}",
|
||||
|
||||
Reference in New Issue
Block a user