feat(deploy): schedule fundamentals shadow imports
This commit is contained in:
@@ -164,8 +164,11 @@ calls. Check free disk space before pulling; alert and skip if below threshold.
|
||||
**Deployment constraints:** deploy is `rsync --delete` of the repo tree
|
||||
(`.gitea/workflows/deploy.yml:127`), so clones and archives must live **outside the
|
||||
deployment path** — an env-configured persistent directory (e.g.
|
||||
`DOLT_DATA_DIR=/var/lib/signal-platform/dolt`), backed up. The dolt binary is a new
|
||||
prod runtime dependency: install with a pinned version in the deploy workflow.
|
||||
`DOLT_DATA_DIR=/var/lib/signal-platform/dolt`). The dolt binary is a new prod
|
||||
runtime dependency: install it once with the version-pinned provisioner in
|
||||
`deploy/provision_fundamentals.sh`; operational steps are in
|
||||
`docs/fundamentals-deployment.md`. The clone is reproducible from DoltHub; the
|
||||
normalized PostgreSQL rows remain part of the normal database backup.
|
||||
|
||||
**Validation gates (block promotion, raise an alert via the existing system-events
|
||||
path):** source freshness as expected; tracked-universe coverage; no duplicate
|
||||
@@ -193,13 +196,14 @@ Workstream A:
|
||||
|
||||
- Dolt earnings import: daily ~02:30 ET (with the future-row replacement above).
|
||||
- **SEC fundamentals job: daily ~04:00 ET.** One job, three steps:
|
||||
(a) freshness check **using conditional HTTP metadata (ETag/Last-Modified /
|
||||
If-None-Match) before downloading** — an unchanged archive is a `no_op` without
|
||||
pulling the multi-GB file; verify by SHA-256 after any actual download;
|
||||
(b) when changed, **parse and write only tracked-universe CIKs** through
|
||||
staging→promotion (the tracked universe is `ticker_universe_service`'s set;
|
||||
CIKs are resolved from `company_tickers.json` each run, so a newly added ticker
|
||||
self-resolves on its next SEC run — until then its metrics are simply null);
|
||||
(a) detect a composite revision from the latest EDGAR daily-index date, the
|
||||
exact tracked index rows, and the tracked-universe fingerprint — an unchanged
|
||||
revision is a `no_op` before Company Facts are fetched;
|
||||
(b) when changed, fetch and parse Company Facts only for tracked-universe CIKs
|
||||
that filed, plus full available history for the first run or a newly added
|
||||
issuer, through validation→atomic promotion. Universe resolution and the exact
|
||||
index inputs are cached during revision detection and reused during staging;
|
||||
CIKs are resolved from `company_tickers.json` without writes until promotion;
|
||||
(c) **always, locally, and only after production activation** (the phase-A5
|
||||
parity approval): refresh the legacy `fundamental_data` fields and mark affected
|
||||
cached fundamental scores stale. **Sources differ per field** — do not assume all
|
||||
@@ -405,9 +409,9 @@ workstream B — Alpaca remains the price source throughout.
|
||||
**Workstream A:**
|
||||
|
||||
- A0. License review **DONE** (earnings approved for private/internal use under
|
||||
CC BY-SA 4.0, no redistribution — see Licensing above); still pending at deploy
|
||||
time: dolt binary pinned in deploy; `DOLT_DATA_DIR` outside the rsync tree
|
||||
(earnings clone only — small), provisioned and backed up.
|
||||
CC BY-SA 4.0, no redistribution — see Licensing above). The Dolt version,
|
||||
persistent `DOLT_DATA_DIR`, clone, and production checks are captured in
|
||||
`deploy/provision_fundamentals.sh` and `docs/fundamentals-deployment.md`.
|
||||
- A1. Migration 026, import-run framework.
|
||||
- A2. Earnings ingestion in shadow (writes `earnings_events`, prod untouched);
|
||||
verify forward-calendar coverage and rescheduling behavior.
|
||||
|
||||
@@ -0,0 +1,133 @@
|
||||
# Fundamentals production deployment
|
||||
|
||||
This is the one-time production setup for the Dolt earnings and SEC fundamentals
|
||||
imports. Both imports remain shadow inputs until the separate A5 scoring-cutover
|
||||
approval. Do not add OS cron entries: the application scheduler owns both jobs.
|
||||
|
||||
## What the deployment adds
|
||||
|
||||
- `Dolt Earnings Import (shadow)` runs daily at 02:30 America/New_York.
|
||||
- `SEC Fundamentals Import (shadow)` runs daily at 04:00 America/New_York.
|
||||
- Both jobs are visible, toggleable, and manually triggerable in Admin → Jobs.
|
||||
- Cron expressions are editable in Admin → Schedule.
|
||||
- Every attempt is recorded in `data_import_runs`; failures also create a system
|
||||
event. A failed validation does not promote partial data.
|
||||
|
||||
The systemd service uses one application worker. The import framework also holds
|
||||
a PostgreSQL advisory lock per source, so an overlapping manual/scheduled run is
|
||||
skipped safely.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
The production `.env` at `/opt/signalplatform/.env` must contain:
|
||||
|
||||
```dotenv
|
||||
DOLT_BINARY=/usr/local/bin/dolt
|
||||
DOLT_DATA_DIR=/var/lib/signal-platform/dolt
|
||||
DOLT_EARNINGS_SUBDIR=earnings
|
||||
DOLT_MIN_FREE_DISK_GB=5.0
|
||||
SEC_USER_AGENT=signal-platform/1.0 (contact: real-address@example.com)
|
||||
SEC_REQUEST_SPACING_SECONDS=0.2
|
||||
```
|
||||
|
||||
Use a real monitored contact address. Keep at least 5 GB free at the Dolt data
|
||||
path; 8–10 GB gives comfortable growth headroom. The data directory must stay
|
||||
outside `/opt/signalplatform`, because deployments use `rsync --delete` there.
|
||||
|
||||
## One-time provisioning
|
||||
|
||||
First deploy the commit containing this bundle to production. Then SSH to the
|
||||
server and run:
|
||||
|
||||
```bash
|
||||
cd /opt/signalplatform
|
||||
sudo bash ./deploy/provision_fundamentals.sh
|
||||
sudo -u deploy bash ./deploy/provision_fundamentals.sh --check
|
||||
sudo systemctl restart signalplatform.service
|
||||
curl -fsS http://127.0.0.1:8998/api/v1/health
|
||||
```
|
||||
|
||||
The provisioner is idempotent. It installs the pinned Dolt version, creates the
|
||||
persistent directory as `deploy:deploy`, clones
|
||||
`post-no-preference/earnings`, verifies free space and `.env`, and refuses an
|
||||
unexpected Dolt version. It does not modify PostgreSQL or start an import.
|
||||
|
||||
Do not replace the pinned version with `latest`. A future Dolt upgrade should be
|
||||
a reviewed change to `DOLT_VERSION`, followed by the same provision/check flow.
|
||||
|
||||
## First-run verification
|
||||
|
||||
In Admin → Jobs, wait until no other job is running, then:
|
||||
|
||||
1. Trigger **Dolt Earnings Import (shadow)**. Expect `completed` with import
|
||||
status `promoted`; a repeat without an upstream change should report `no_op`.
|
||||
2. Trigger **SEC Fundamentals Import (shadow)**. The first run performs the
|
||||
tracked-universe history backfill and can take materially longer than a daily
|
||||
incremental run. Expect `completed` with import status `promoted`.
|
||||
3. Check Admin → System Events. There should be no new import error.
|
||||
4. Confirm the next-run times correspond to 02:30 and 04:00 New York time.
|
||||
5. Open several ticker pages and confirm the fundamentals panel has populated
|
||||
data and still handles partial/missing issuers cleanly.
|
||||
|
||||
Optional database verification:
|
||||
|
||||
```sql
|
||||
SELECT source, status, revision, source_max_date, started_at, completed_at,
|
||||
validation_json
|
||||
FROM data_import_runs
|
||||
WHERE source IN ('dolt_earnings', 'sec_facts')
|
||||
ORDER BY id DESC
|
||||
LIMIT 10;
|
||||
|
||||
SELECT count(*) FROM earnings_events WHERE source = 'dolt_earnings';
|
||||
SELECT count(*), count(DISTINCT cik) FROM fundamental_snapshots;
|
||||
```
|
||||
|
||||
During the longer first SEC run, execute the following in a second SSH session.
|
||||
It opens an independent database connection and attempts the same source lock:
|
||||
|
||||
```bash
|
||||
cd /opt/signalplatform
|
||||
sudo -u deploy .venv/bin/python - <<'PY'
|
||||
import asyncio
|
||||
|
||||
from sqlalchemy import text
|
||||
|
||||
from app.database import engine
|
||||
from app.services.data_import import _advisory_key
|
||||
|
||||
|
||||
async def main():
|
||||
key = _advisory_key("sec_facts")
|
||||
async with engine.connect() as connection:
|
||||
acquired = await connection.scalar(
|
||||
text("SELECT pg_try_advisory_lock(:key)"), {"key": key}
|
||||
)
|
||||
print("UNEXPECTED: lock acquired" if acquired else "OK: source lock is busy")
|
||||
if acquired:
|
||||
await connection.execute(
|
||||
text("SELECT pg_advisory_unlock(:key)"), {"key": key}
|
||||
)
|
||||
|
||||
|
||||
asyncio.run(main())
|
||||
PY
|
||||
```
|
||||
|
||||
Expect `OK: source lock is busy`. This is the remaining live-PostgreSQL
|
||||
mutual-exclusion check; SQLite unit tests cannot exercise PostgreSQL advisory
|
||||
locks. A second Admin trigger should independently report the job as busy.
|
||||
|
||||
## Failure and rollback
|
||||
|
||||
- Disable the failing shadow job in Admin → Jobs. This stops scheduled imports
|
||||
without changing existing data or the legacy scoring path.
|
||||
- Inspect the job runtime, latest `data_import_runs.validation_json`, service
|
||||
logs, and Admin → System Events before retrying.
|
||||
- Re-run `sudo -u deploy bash ./deploy/provision_fundamentals.sh --check` for
|
||||
binary, clone, permission, disk, or environment failures.
|
||||
- The Dolt clone is a reproducible cache and does not need a bespoke backup.
|
||||
PostgreSQL (including `earnings_events`, `fundamental_snapshots`, and import
|
||||
audit rows) must remain covered by the normal production database backup.
|
||||
- Do not proceed to A5 while either shadow feed is unhealthy or the parity gate
|
||||
has not received explicit approval.
|
||||
Reference in New Issue
Block a user