feat(deploy): schedule fundamentals shadow imports
This commit is contained in:
@@ -164,8 +164,11 @@ calls. Check free disk space before pulling; alert and skip if below threshold.
|
||||
**Deployment constraints:** deploy is `rsync --delete` of the repo tree
|
||||
(`.gitea/workflows/deploy.yml:127`), so clones and archives must live **outside the
|
||||
deployment path** — an env-configured persistent directory (e.g.
|
||||
`DOLT_DATA_DIR=/var/lib/signal-platform/dolt`), backed up. The dolt binary is a new
|
||||
prod runtime dependency: install with a pinned version in the deploy workflow.
|
||||
`DOLT_DATA_DIR=/var/lib/signal-platform/dolt`). The dolt binary is a new prod
|
||||
runtime dependency: install it once with the version-pinned provisioner in
|
||||
`deploy/provision_fundamentals.sh`; operational steps are in
|
||||
`docs/fundamentals-deployment.md`. The clone is reproducible from DoltHub; the
|
||||
normalized PostgreSQL rows remain part of the normal database backup.
|
||||
|
||||
**Validation gates (block promotion, raise an alert via the existing system-events
|
||||
path):** source freshness as expected; tracked-universe coverage; no duplicate
|
||||
@@ -193,13 +196,14 @@ Workstream A:
|
||||
|
||||
- Dolt earnings import: daily ~02:30 ET (with the future-row replacement above).
|
||||
- **SEC fundamentals job: daily ~04:00 ET.** One job, three steps:
|
||||
(a) freshness check **using conditional HTTP metadata (ETag/Last-Modified /
|
||||
If-None-Match) before downloading** — an unchanged archive is a `no_op` without
|
||||
pulling the multi-GB file; verify by SHA-256 after any actual download;
|
||||
(b) when changed, **parse and write only tracked-universe CIKs** through
|
||||
staging→promotion (the tracked universe is `ticker_universe_service`'s set;
|
||||
CIKs are resolved from `company_tickers.json` each run, so a newly added ticker
|
||||
self-resolves on its next SEC run — until then its metrics are simply null);
|
||||
(a) detect a composite revision from the latest EDGAR daily-index date, the
|
||||
exact tracked index rows, and the tracked-universe fingerprint — an unchanged
|
||||
revision is a `no_op` before Company Facts are fetched;
|
||||
(b) when changed, fetch and parse Company Facts only for tracked-universe CIKs
|
||||
that filed, plus full available history for the first run or a newly added
|
||||
issuer, through validation→atomic promotion. Universe resolution and the exact
|
||||
index inputs are cached during revision detection and reused during staging;
|
||||
CIKs are resolved from `company_tickers.json` without writes until promotion;
|
||||
(c) **always, locally, and only after production activation** (the phase-A5
|
||||
parity approval): refresh the legacy `fundamental_data` fields and mark affected
|
||||
cached fundamental scores stale. **Sources differ per field** — do not assume all
|
||||
@@ -405,9 +409,9 @@ workstream B — Alpaca remains the price source throughout.
|
||||
**Workstream A:**
|
||||
|
||||
- A0. License review **DONE** (earnings approved for private/internal use under
|
||||
CC BY-SA 4.0, no redistribution — see Licensing above); still pending at deploy
|
||||
time: dolt binary pinned in deploy; `DOLT_DATA_DIR` outside the rsync tree
|
||||
(earnings clone only — small), provisioned and backed up.
|
||||
CC BY-SA 4.0, no redistribution — see Licensing above). The Dolt version,
|
||||
persistent `DOLT_DATA_DIR`, clone, and production checks are captured in
|
||||
`deploy/provision_fundamentals.sh` and `docs/fundamentals-deployment.md`.
|
||||
- A1. Migration 026, import-run framework.
|
||||
- A2. Earnings ingestion in shadow (writes `earnings_events`, prod untouched);
|
||||
verify forward-calendar coverage and rescheduling behavior.
|
||||
|
||||
Reference in New Issue
Block a user