feat(deploy): schedule fundamentals shadow imports

This commit is contained in:
2026-07-23 12:41:27 +02:00
parent 71d21a45b6
commit 88fc90cc6f
10 changed files with 493 additions and 19 deletions
+16 -12
View File
@@ -164,8 +164,11 @@ calls. Check free disk space before pulling; alert and skip if below threshold.
**Deployment constraints:** deploy is `rsync --delete` of the repo tree
(`.gitea/workflows/deploy.yml:127`), so clones and archives must live **outside the
deployment path** — an env-configured persistent directory (e.g.
`DOLT_DATA_DIR=/var/lib/signal-platform/dolt`), backed up. The dolt binary is a new
prod runtime dependency: install with a pinned version in the deploy workflow.
`DOLT_DATA_DIR=/var/lib/signal-platform/dolt`). The dolt binary is a new prod
runtime dependency: install it once with the version-pinned provisioner in
`deploy/provision_fundamentals.sh`; operational steps are in
`docs/fundamentals-deployment.md`. The clone is reproducible from DoltHub; the
normalized PostgreSQL rows remain part of the normal database backup.
**Validation gates (block promotion, raise an alert via the existing system-events
path):** source freshness as expected; tracked-universe coverage; no duplicate
@@ -193,13 +196,14 @@ Workstream A:
- Dolt earnings import: daily ~02:30 ET (with the future-row replacement above).
- **SEC fundamentals job: daily ~04:00 ET.** One job, three steps:
(a) freshness check **using conditional HTTP metadata (ETag/Last-Modified /
If-None-Match) before downloading** — an unchanged archive is a `no_op` without
pulling the multi-GB file; verify by SHA-256 after any actual download;
(b) when changed, **parse and write only tracked-universe CIKs** through
staging→promotion (the tracked universe is `ticker_universe_service`'s set;
CIKs are resolved from `company_tickers.json` each run, so a newly added ticker
self-resolves on its next SEC run — until then its metrics are simply null);
(a) detect a composite revision from the latest EDGAR daily-index date, the
exact tracked index rows, and the tracked-universe fingerprint — an unchanged
revision is a `no_op` before Company Facts are fetched;
(b) when changed, fetch and parse Company Facts only for tracked-universe CIKs
that filed, plus full available history for the first run or a newly added
issuer, through validation→atomic promotion. Universe resolution and the exact
index inputs are cached during revision detection and reused during staging;
CIKs are resolved from `company_tickers.json` without writes until promotion;
(c) **always, locally, and only after production activation** (the phase-A5
parity approval): refresh the legacy `fundamental_data` fields and mark affected
cached fundamental scores stale. **Sources differ per field** — do not assume all
@@ -405,9 +409,9 @@ workstream B — Alpaca remains the price source throughout.
**Workstream A:**
- A0. License review **DONE** (earnings approved for private/internal use under
CC BY-SA 4.0, no redistribution — see Licensing above); still pending at deploy
time: dolt binary pinned in deploy; `DOLT_DATA_DIR` outside the rsync tree
(earnings clone only — small), provisioned and backed up.
CC BY-SA 4.0, no redistribution — see Licensing above). The Dolt version,
persistent `DOLT_DATA_DIR`, clone, and production checks are captured in
`deploy/provision_fundamentals.sh` and `docs/fundamentals-deployment.md`.
- A1. Migration 026, import-run framework.
- A2. Earnings ingestion in shadow (writes `earnings_events`, prod untouched);
verify forward-calendar coverage and rescheduling behavior.