fix(sec): tell an attribution collision apart from a changed reconstruction
Deploy / lint (push) Successful in 12s
Deploy / test (push) Successful in 1m43s
Deploy / deploy (push) Successful in 39s

snapshot_discrepancy named the accessions but not the columns, so it could not
distinguish "our numbers moved" from "the same filing is attributed twice".
The fields were already computed for validation_json and simply dropped from
the message; they are now in it.

A difference in cik ALONE is no longer reported as a reconstruction change at
all. Every fact matched, so two tracked CIKs are claiming one filing and the
fix is the universe, not the parser: it raises accession_cik_collision naming
both CIKs and sec_cik_overrides. It also never self-heals — the losing CIK
stores no row, so _ciks_with_snapshots never sees it and it is full-history
backfilled and re-reported every run until its ticker is re-pointed or retired.

Observed 2026-08-19 for EQR: after Equity Residential renamed to Vivmark
Residential (VMRK, CIK 906107), SEC's own company_tickers.json left the old
symbol on ERP Operating LP (CIK 931182), the non-traded co-registrant of their
combined 10-Qs. Both were tracked, both reconstructed the same two filings.

The reparse path now excludes cik-only differences from its rewrite set:
rewriting one would re-stamp the filing onto the co-registrant, taking it from
the issuer that actually filed it, which no parser fix asks for.

No stored value was wrong in that incident — reports/ carries the full
reproduction for this and for the companyfacts staleness behind the gate fix.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Lo99z3jWqu9X9ueBU3Z3D
This commit is contained in:
2026-08-21 16:46:40 +02:00
co-authored by Claude Opus 5
parent a13dbc9710
commit c15b51439e
3 changed files with 326 additions and 8 deletions
+71 -8
View File
@@ -36,7 +36,11 @@ Guardrails (design + reviews):
excluded from actionable setups until its filing is recovered.
- ``promote`` inserts snapshots ``ON CONFLICT (accession) DO NOTHING`` (immutable),
reports differing existing accessions, and applies ticker updates in the same
transaction.
transaction. A difference in ``cik`` **alone** is reported separately as an
``accession_cik_collision``: every fact matched, so two tracked CIKs are
claiming one filing and the fix is the universe, not the parser. It never
self-heals on its own — the losing CIK stores no row, so it is backfilled and
re-reported every run until its ticker is re-pointed or retired.
- ``reparse=True`` is the one exception to immutability, and it is deliberate:
it restages every accession with the current parser and **rewrites** the rows
that now reconstruct differently. Immutability protects SEC's record (one row
@@ -67,7 +71,7 @@ from app.services import sec_universe
from app.services.data_import import STATUS_PROMOTED, ValidationResult
from app.services.sec_client import SecClient, SecError, cik10
from app.services.sec_facts_parser import FilingMeta, SnapshotRow
from app.services.sec_universe import ResolvedUniverse
from app.services.sec_universe import CIK_OVERRIDES_KEY, ResolvedUniverse
logger = logging.getLogger(__name__)
@@ -281,7 +285,18 @@ class SecFundamentalsImporter:
if old is not None:
fields = _diff_fields(row, old)
if fields:
staged.discrepancies.append({"accession": row.accession, "fields": fields})
# Carry both CIKs. promote() reads a bare ["cik"] as an
# attribution collision rather than a changed
# reconstruction, which holds only because _COMPARE_COLS
# spans every stored fact: a fact column added to the
# model but not to _SNAPSHOT_COLS would go uncompared and
# let a real difference through as a collision.
staged.discrepancies.append({
"accession": row.accession,
"fields": fields,
"cik": row.cik,
"stored_cik": old.cik,
})
return staged
async def _stage_issuer(
@@ -555,8 +570,15 @@ class SecFundamentalsImporter:
inserted = 0
updated = 0
# Only accessions whose reconstruction actually changed are rewritten;
# an unchanged stored row is left completely alone.
changed = {d["accession"] for d in staged.discrepancies} if self.reparse else set()
# an unchanged stored row is left completely alone. A cik-only difference
# is excluded on purpose: the facts are identical there, so rewriting
# would re-stamp the filing onto the colliding co-registrant — taking it
# from the issuer that actually filed it, which no parser fix asks for.
changed = (
{d["accession"] for d in staged.discrepancies if d["fields"] != ["cik"]}
if self.reparse
else set()
)
for row in staged.rows:
if row.accession in staged.existing_accessions:
if row.accession in changed:
@@ -641,10 +663,51 @@ class SecFundamentalsImporter:
if gap["accession"] not in existing_gap_accessions
]
# Two tracked issuers claiming one filing is not a reconstruction change:
# every fact matched and only the CIK stamp differs, so re-parsing or
# reparsing fixes nothing — the universe resolution does. It is reported
# separately because it also does not self-heal: the loser of the
# collision never stores a row, so `_ciks_with_snapshots` never sees it,
# and it is full-history backfilled (and re-reported) on every run until
# a human re-points or retires the ticker. Observed 2026-08 for EQR,
# which SEC's own company_tickers.json maps to ERP Operating LP, the
# non-traded co-registrant of the issuer now trading as VMRK.
collisions = [d for d in staged.discrepancies if d["fields"] == ["cik"]]
if collisions:
named = ", ".join(
f"{d['accession']} (stored {d['stored_cik']}, parsed {d['cik']})"
for d in collisions[:10]
)
db.add(SystemEvent(
severity="warning",
source="sec_facts",
code="accession_cik_collision",
message=(
f"{len(collisions)} filing(s) are claimed by two tracked CIKs — "
"the reconstruction is identical, only the attribution differs, "
"so one of the two is a co-registrant the universe should not "
f"track. Re-point or retire the ticker (see {CIK_OVERRIDES_KEY}); "
f"this repeats every run until then: {named}"
)[:4000],
dedup_key=f"sec_facts:accession_cik_collision:{run_id}",
created_at=_now(),
))
# Warn (in-transaction, so it commits atomically with the promotion) when
# any existing accession reconstructed differently — kept immutable.
if staged.discrepancies:
accns = ", ".join(d["accession"] for d in staged.discrepancies[:10])
reconstruction_diffs = [
d for d in staged.discrepancies if d["fields"] != ["cik"]
]
if reconstruction_diffs:
# Name the columns, not just the accession: "differs in revenue"
# (our numbers moved) and "differs in period_start" (the filing was
# re-placed in the calendar) need different responses, and the alert
# is where that call gets made. The fields are already computed for
# validation_json — they were simply dropped from the message.
accns = ", ".join(
f"{d['accession']} ({', '.join(d['fields'])})"
for d in reconstruction_diffs[:10]
)
disposition = (
f"REWRITTEN by reparse run {run_id}" if self.reparse else "kept immutable"
)
@@ -653,7 +716,7 @@ class SecFundamentalsImporter:
source="sec_facts",
code="snapshot_reparse" if self.reparse else "snapshot_discrepancy",
message=(
f"{len(staged.discrepancies)} stored accession(s) reconstructed "
f"{len(reconstruction_diffs)} stored accession(s) reconstructed "
f"differently; {disposition}: {accns}"
)[:4000],
dedup_key=f"sec_facts:discrepancy:{run_id}",