Review fixes to the import-run framework (A1):
1. detect_revision ran outside the failure handler, so a failed revision probe
(the most likely external failure) escaped unrecorded — violating "every
attempt is recorded". Now the running row is created FIRST, then
detect_revision + last-revision lookup + stage + validate + promote all run
inside the same handler; the row converts to no_op when the revision is
unchanged. New test covers a detection exception → recorded failed + alert.
2. asyncio.CancelledError (BaseException, not caught by except Exception) left a
permanent running row on deploy/scheduler shutdown. Now caught explicitly:
best-effort mark failed, then re-raise the cancellation (never swallowed).
New test asserts the run is failed and the error re-propagates.
Full suite 682 passed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
run_import + SourceImporter Protocol (detect_revision/stage/validate/promote)
giving every bulk importer the plan's non-negotiables, KISS:
- one run per source at a time — Postgres session-level advisory lock held on a
single pinned engine.connect() so it survives the running-row and promotion
commits; no-op on SQLite.
- idempotent per revision — cheap detect_revision compared to the last promoted
run; unchanged revision records a no_op with zero writes (no fetch).
- staging (in-memory, no physical staging tables) → validate (read-only) →
atomic promote + run-row flip in one transaction.
- failed validation or mid-run exception marks the run failed, alerts via
system_event_service, and leaves live tables untouched.
Every attempt recorded in data_import_runs; conflicts summary in validation_json
(no conflicts table). Concrete SEC/earnings importers land in later phases.
Tests: 6 orchestration tests (no_op / promote / new-revision / failed-untouched
/ promote-exception-rollback) + deterministic advisory-key derivation. Full
suite 680 passed. Advisory-lock mutual exclusion is PG-verify-pending (SQLite
no-ops it — flagged, not covered).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
dennisthiessenandClaude Opus 4.8