feat(sec): reparse path, CIK overrides, resolution validation
Operational plumbing to land the parser fixes and to make silent resolution
failures visible.
- Reparse: SecFundamentalsImporter(reparse=True) restages every accession with
the current parser and rewrites the ones that now reconstruct differently,
writing the full column set so a row is never half old-parse. Snapshots stay
immutable with respect to SEC; the stored row is our reconstruction, and after
a parser fix keeping it is a stale cache, not history. run_import(force=True)
bypasses the unchanged-revision no-op, since the staleness is on our side, not
the source's. Exposed as scripts/reparse_fundamentals.py, dry-run by default.
- CIK overrides: sec_universe reads a {symbol: cik} pin from
SystemSetting['sec_cik_overrides'], applied ahead of company_tickers.json, for
when SEC maps a ticker to a successor shell with no filings (XOM -> a zero-
filing "ExxonMobil Holdings Corp" while every 10-K/Q is under CIK 34088).
- Resolution validation: a tracked issuer resolving to a registrant with no XBRL
filings now records no_xbrl_filings and raises a warning naming the CIKs and
the override setting, instead of silently yielding nothing on every run.
- _diff_fields compares datetime instants, not representations: accepted_at
round-trips naive from SQLite but tz-aware from Postgres, which otherwise made
a reparse of identical data report every row as changed (and false-positived
the pre-existing discrepancy warning).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -145,11 +145,17 @@ async def run_import(
|
||||
importer: SourceImporter,
|
||||
*,
|
||||
engine: AsyncEngine | None = None,
|
||||
force: bool = False,
|
||||
) -> DataImportRun | None:
|
||||
"""Run one import for ``importer``.
|
||||
|
||||
Returns the recorded ``DataImportRun`` (promoted / no_op / failed), or None
|
||||
when the per-source advisory lock is already held (another run is active).
|
||||
|
||||
``force`` runs even when the revision is unchanged. The revision tracks the
|
||||
*source*, so a re-import driven by a change on our side — a parser fix that
|
||||
makes stored rows stale — is a no_op under the normal gate. Manually invoked
|
||||
only; scheduled jobs must leave it False so an unchanged source stays a no_op.
|
||||
"""
|
||||
engine = engine or app_engine
|
||||
source = importer.source
|
||||
@@ -189,7 +195,7 @@ async def run_import(
|
||||
revision = await importer.detect_revision(session)
|
||||
run.revision = revision
|
||||
last_rev = await _last_promoted_revision(session, source)
|
||||
if revision is not None and revision == last_rev:
|
||||
if not force and revision is not None and revision == last_rev:
|
||||
run.status = STATUS_NO_OP
|
||||
run.completed_at = _now()
|
||||
await session.commit()
|
||||
|
||||
Reference in New Issue
Block a user