feat(deploy): schedule fundamentals shadow imports

2026-07-23 12:41:27 +02:00
parent 71d21a45b6
commit 88fc90cc6f
10 changed files with 493 additions and 19 deletions
@@ -0,0 +1,133 @@
+# Fundamentals production deployment
+
+This is the one-time production setup for the Dolt earnings and SEC fundamentals
+imports. Both imports remain shadow inputs until the separate A5 scoring-cutover
+approval. Do not add OS cron entries: the application scheduler owns both jobs.
+
+## What the deployment adds
+
+- `Dolt Earnings Import (shadow)` runs daily at 02:30 America/New_York.
+- `SEC Fundamentals Import (shadow)` runs daily at 04:00 America/New_York.
+- Both jobs are visible, toggleable, and manually triggerable in Admin → Jobs.
+- Cron expressions are editable in Admin → Schedule.
+- Every attempt is recorded in `data_import_runs`; failures also create a system
+  event. A failed validation does not promote partial data.
+
+The systemd service uses one application worker. The import framework also holds
+a PostgreSQL advisory lock per source, so an overlapping manual/scheduled run is
+skipped safely.
+
+## Prerequisites
+
+The production `.env` at `/opt/signalplatform/.env` must contain:
+
+```dotenv
+DOLT_BINARY=/usr/local/bin/dolt
+DOLT_DATA_DIR=/var/lib/signal-platform/dolt
+DOLT_EARNINGS_SUBDIR=earnings
+DOLT_MIN_FREE_DISK_GB=5.0
+SEC_USER_AGENT=signal-platform/1.0 (contact: real-address@example.com)
+SEC_REQUEST_SPACING_SECONDS=0.2
+```
+
+Use a real monitored contact address. Keep at least 5 GB free at the Dolt data
+path; 8–10 GB gives comfortable growth headroom. The data directory must stay
+outside `/opt/signalplatform`, because deployments use `rsync --delete` there.
+
+## One-time provisioning
+
+First deploy the commit containing this bundle to production. Then SSH to the
+server and run:
+
+```bash
+cd /opt/signalplatform
+sudo bash ./deploy/provision_fundamentals.sh
+sudo -u deploy bash ./deploy/provision_fundamentals.sh --check
+sudo systemctl restart signalplatform.service
+curl -fsS http://127.0.0.1:8998/api/v1/health
+```
+
+The provisioner is idempotent. It installs the pinned Dolt version, creates the
+persistent directory as `deploy:deploy`, clones
+`post-no-preference/earnings`, verifies free space and `.env`, and refuses an
+unexpected Dolt version. It does not modify PostgreSQL or start an import.
+
+Do not replace the pinned version with `latest`. A future Dolt upgrade should be
+a reviewed change to `DOLT_VERSION`, followed by the same provision/check flow.
+
+## First-run verification
+
+In Admin → Jobs, wait until no other job is running, then:
+
+1. Trigger **Dolt Earnings Import (shadow)**. Expect `completed` with import
+   status `promoted`; a repeat without an upstream change should report `no_op`.
+2. Trigger **SEC Fundamentals Import (shadow)**. The first run performs the
+   tracked-universe history backfill and can take materially longer than a daily
+   incremental run. Expect `completed` with import status `promoted`.
+3. Check Admin → System Events. There should be no new import error.
+4. Confirm the next-run times correspond to 02:30 and 04:00 New York time.
+5. Open several ticker pages and confirm the fundamentals panel has populated
+   data and still handles partial/missing issuers cleanly.
+
+Optional database verification:
+
+```sql
+SELECT source, status, revision, source_max_date, started_at, completed_at,
+       validation_json
+FROM data_import_runs
+WHERE source IN ('dolt_earnings', 'sec_facts')
+ORDER BY id DESC
+LIMIT 10;
+
+SELECT count(*) FROM earnings_events WHERE source = 'dolt_earnings';
+SELECT count(*), count(DISTINCT cik) FROM fundamental_snapshots;
+```
+
+During the longer first SEC run, execute the following in a second SSH session.
+It opens an independent database connection and attempts the same source lock:
+
+```bash
+cd /opt/signalplatform
+sudo -u deploy .venv/bin/python - <<'PY'
+import asyncio
+
+from sqlalchemy import text
+
+from app.database import engine
+from app.services.data_import import _advisory_key
+
+
+async def main():
+    key = _advisory_key("sec_facts")
+    async with engine.connect() as connection:
+        acquired = await connection.scalar(
+            text("SELECT pg_try_advisory_lock(:key)"), {"key": key}
+        )
+        print("UNEXPECTED: lock acquired" if acquired else "OK: source lock is busy")
+        if acquired:
+            await connection.execute(
+                text("SELECT pg_advisory_unlock(:key)"), {"key": key}
+            )
+
+
+asyncio.run(main())
+PY
+```
+
+Expect `OK: source lock is busy`. This is the remaining live-PostgreSQL
+mutual-exclusion check; SQLite unit tests cannot exercise PostgreSQL advisory
+locks. A second Admin trigger should independently report the job as busy.
+
+## Failure and rollback
+
+- Disable the failing shadow job in Admin → Jobs. This stops scheduled imports
+  without changing existing data or the legacy scoring path.
+- Inspect the job runtime, latest `data_import_runs.validation_json`, service
+  logs, and Admin → System Events before retrying.
+- Re-run `sudo -u deploy bash ./deploy/provision_fundamentals.sh --check` for
+  binary, clone, permission, disk, or environment failures.
+- The Dolt clone is a reproducible cache and does not need a bespoke backup.
+  PostgreSQL (including `earnings_events`, `fundamental_snapshots`, and import
+  audit rows) must remain covered by the normal production database backup.
+- Do not proceed to A5 while either shadow feed is unhealthy or the parity gate
+  has not received explicit approval.