244 lines
10 KiB
Markdown
244 lines
10 KiB
Markdown
# Fundamentals production deployment
|
||
|
||
This is the one-time production setup for the Dolt earnings and SEC fundamentals
|
||
imports. The A5 scoring cutover was approved on 2026-07-24; the compat-cache write
|
||
path is still default-off until the explicit production switch below is set. Do
|
||
not add OS cron entries: the application scheduler owns both jobs.
|
||
|
||
## What the deployment adds
|
||
|
||
- `Dolt Earnings Import (shadow)` runs daily at 02:30 America/New_York.
|
||
- `SEC Fundamentals Import` runs daily at 04:00 America/New_York. Its local
|
||
`fundamental_data` refresh runs only when the A5 switch is enabled.
|
||
- `Fundamentals Parity Report (read-only)` runs daily at 05:30 America/New_York.
|
||
- Both jobs are visible, toggleable, and manually triggerable in Admin → Jobs.
|
||
- Cron expressions are editable in Admin → Schedule.
|
||
- Every attempt is recorded in `data_import_runs`; failures also create a system
|
||
event. A failed validation does not promote partial data.
|
||
|
||
The systemd service uses one application worker. The import framework also holds
|
||
a PostgreSQL advisory lock per source, so an overlapping manual/scheduled run is
|
||
skipped safely.
|
||
|
||
## Prerequisites
|
||
|
||
The production `.env` at `/opt/signalplatform/.env` must contain:
|
||
|
||
```dotenv
|
||
DOLT_BINARY=/usr/local/bin/dolt
|
||
DOLT_DATA_DIR=/var/lib/signal-platform/dolt
|
||
DOLT_EARNINGS_SUBDIR=earnings
|
||
DOLT_MIN_FREE_DISK_GB=5.0
|
||
SEC_USER_AGENT=signal-platform/1.0 (contact: real-address@example.com)
|
||
SEC_REQUEST_SPACING_SECONDS=0.2
|
||
FUNDAMENTALS_PARITY_REPORT_DIR=/var/lib/signal-platform/reports/fundamentals-parity
|
||
```
|
||
|
||
Use a real monitored contact address. Keep at least 5 GB free at the Dolt data
|
||
path; 8–10 GB gives comfortable growth headroom. The data directory must stay
|
||
outside `/opt/signalplatform`, because deployments use `rsync --delete` there.
|
||
The parity-report directory is also persistent and owned by the service user;
|
||
its small timestamped JSON/CSV bundles form the temporary A5 review trail.
|
||
|
||
## One-time provisioning
|
||
|
||
First deploy the commit containing this bundle to production. Then SSH to the
|
||
server and run:
|
||
|
||
```bash
|
||
cd /opt/signalplatform
|
||
sudo bash ./deploy/provision_fundamentals.sh
|
||
sudo -u deploy bash ./deploy/provision_fundamentals.sh --check
|
||
sudo systemctl restart signalplatform.service
|
||
curl -fsS http://127.0.0.1:8998/api/v1/health
|
||
```
|
||
|
||
The provisioner is idempotent. It installs the pinned Dolt version, creates the
|
||
persistent directory as `deploy:deploy`, clones
|
||
`post-no-preference/earnings`, configures a repository-local author identity for
|
||
`dolt pull`, verifies free space and `.env`, and refuses an unexpected
|
||
Dolt version. It does not modify PostgreSQL or start an import. The public clone
|
||
does not require `dolt login`.
|
||
|
||
For a server provisioned before the author-identity check was added, repair the
|
||
existing clone once with:
|
||
|
||
```bash
|
||
sudo -u deploy -H /usr/local/bin/dolt config --global --add user.name "Signal Platform"
|
||
sudo -u deploy -H /usr/local/bin/dolt config --global --add user.email "signal-platform@localhost"
|
||
```
|
||
|
||
Do not replace the pinned version with `latest`. A future Dolt upgrade should be
|
||
a reviewed change to `DOLT_VERSION`, followed by the same provision/check flow.
|
||
|
||
## First-run verification
|
||
|
||
In Admin → Jobs, wait until no other job is running, then:
|
||
|
||
1. Trigger **Dolt Earnings Import (shadow)**. Expect `completed` with import
|
||
status `promoted`; a repeat without an upstream change should report `no_op`.
|
||
2. Trigger **SEC Fundamentals Import**. The first run performs the
|
||
tracked-universe history backfill and can take materially longer than a daily
|
||
incremental run. Expect `completed` with import status `promoted`.
|
||
3. Check Admin → System Events. There should be no new import error.
|
||
4. Confirm the next-run times correspond to 02:30 and 04:00 New York time.
|
||
5. Open several ticker pages and confirm the fundamentals panel has populated
|
||
data and still handles partial/missing issuers cleanly.
|
||
|
||
## A5 parity observation window
|
||
|
||
After both shadow imports are healthy, trigger **Fundamentals Parity Report
|
||
(read-only)** once in Admin → Jobs. The **A5 Fundamentals Parity** card above
|
||
the jobs shows the latest coverage/delta summary and provides authenticated JSON
|
||
and CSV downloads. The canonical server-side bundles are archived at:
|
||
|
||
```text
|
||
/var/lib/signal-platform/reports/fundamentals-parity/
|
||
```
|
||
|
||
The scheduler then generates one report daily at 05:30 New York time, after the
|
||
02:30 Dolt and 04:00 SEC jobs. Review 5–7 consecutive reports before making the
|
||
cutover decision. A report never writes `fundamental_data`, dimension/composite
|
||
scores, rankings, qualification state, or an approval flag. Materiality bands
|
||
only highlight rows for review; A5 still requires explicit approval.
|
||
|
||
Each bundle contains legacy and candidate P/E, revenue growth, and earnings
|
||
surprise; definition notes; source revisions and price dates; recomputed legacy
|
||
and candidate fundamental scores; and per-universe fundamental-rank changes.
|
||
Definition changes remain explicit even when numeric deltas are small.
|
||
|
||
Optional database verification:
|
||
|
||
```sql
|
||
SELECT source, status, revision, source_max_date, started_at, completed_at,
|
||
validation_json
|
||
FROM data_import_runs
|
||
WHERE source IN ('dolt_earnings', 'sec_facts')
|
||
ORDER BY id DESC
|
||
LIMIT 10;
|
||
|
||
SELECT count(*) FROM earnings_events WHERE source = 'dolt_earnings';
|
||
SELECT count(*), count(DISTINCT cik) FROM fundamental_snapshots;
|
||
```
|
||
|
||
During the longer first SEC run, execute the following in a second SSH session.
|
||
It opens an independent database connection and attempts the same source lock:
|
||
|
||
```bash
|
||
cd /opt/signalplatform
|
||
sudo -u deploy .venv/bin/python - <<'PY'
|
||
import asyncio
|
||
|
||
from sqlalchemy import text
|
||
|
||
from app.database import engine
|
||
from app.services.data_import import _advisory_key
|
||
|
||
|
||
async def main():
|
||
key = _advisory_key("sec_facts")
|
||
async with engine.connect() as connection:
|
||
acquired = await connection.scalar(
|
||
text("SELECT pg_try_advisory_lock(:key)"), {"key": key}
|
||
)
|
||
print("UNEXPECTED: lock acquired" if acquired else "OK: source lock is busy")
|
||
if acquired:
|
||
await connection.execute(
|
||
text("SELECT pg_advisory_unlock(:key)"), {"key": key}
|
||
)
|
||
|
||
|
||
asyncio.run(main())
|
||
PY
|
||
```
|
||
|
||
Expect `OK: source lock is busy`. This is the remaining live-PostgreSQL
|
||
mutual-exclusion check; SQLite unit tests cannot exercise PostgreSQL advisory
|
||
locks. A second Admin trigger should independently report the job as busy.
|
||
|
||
## A5 production activation (approved 2026-07-24)
|
||
|
||
The write path is controlled by the SystemSetting
|
||
`fundamental_data_sec_dolt_cutover_enabled`. An absent value, `false`, or any
|
||
value other than `true` leaves `fundamental_data` untouched. Before enabling it,
|
||
confirm the normal PostgreSQL backup containing `fundamental_data` is current.
|
||
|
||
In **Admin → Settings → Fundamentals data source**:
|
||
|
||
1. Turn on **Use SEC + Dolt for scoring inputs** and accept the confirmation.
|
||
2. Click **Run refresh now**. The SEC import may be `promoted` or `no_op`; either
|
||
result runs the local cache refresh.
|
||
|
||
The weekly legacy collector is automatically skipped while the switch is on, so
|
||
it cannot overwrite the activated cache. The switch remains visible even before
|
||
its SystemSetting row exists because the safe default is off.
|
||
|
||
If the Admin UI is unavailable, enable the cutover directly in PostgreSQL:
|
||
|
||
```sql
|
||
INSERT INTO system_settings (key, value, updated_at)
|
||
VALUES ('fundamental_data_sec_dolt_cutover_enabled', 'true', now())
|
||
ON CONFLICT (key) DO UPDATE
|
||
SET value = EXCLUDED.value, updated_at = now();
|
||
```
|
||
|
||
Then trigger **SEC Fundamentals Import** once in Admin → Jobs. Once enabled, the
|
||
same refresh also runs after an SEC network/validation failure or a source-lock
|
||
skip, because it reads only PostgreSQL snapshots, earnings events, and closes.
|
||
The job message appends the cache row count and changed score-input count when
|
||
the import itself completed successfully.
|
||
|
||
Verify the switch and refreshed rows:
|
||
|
||
```sql
|
||
SELECT key, value, updated_at
|
||
FROM system_settings
|
||
WHERE key = 'fundamental_data_sec_dolt_cutover_enabled';
|
||
|
||
SELECT count(*) AS rows,
|
||
max(fetched_at) AS refreshed_at,
|
||
count(pe_ratio) AS pe_available,
|
||
count(revenue_growth) AS growth_available,
|
||
count(earnings_surprise) AS surprise_available,
|
||
count(next_earnings_date) AS next_date_available
|
||
FROM fundamental_data;
|
||
|
||
SELECT dimension, is_stale, count(*)
|
||
FROM dimension_scores
|
||
WHERE dimension = 'fundamental'
|
||
GROUP BY dimension, is_stale;
|
||
|
||
SELECT is_stale, count(*)
|
||
FROM composite_scores
|
||
GROUP BY is_stale;
|
||
```
|
||
|
||
The first refresh intentionally marks affected fundamental and composite score
|
||
caches stale. The normal 15:30 near-close scanner recomputes them before using
|
||
the rankings; until then, reads truthfully expose the stale state. Observe at
|
||
least several scheduled cycles before A6 removes the legacy providers.
|
||
|
||
## Failure and rollback
|
||
|
||
- To stop the A5 cache writes without stopping SEC snapshot ingestion, turn off
|
||
**Use SEC + Dolt for scoring inputs** in Admin → Settings. If the UI is
|
||
unavailable, set `fundamental_data_sec_dolt_cutover_enabled` back to `false`
|
||
with the SQL above (changing only the value). This prevents the next local
|
||
refresh but does not restore rows already replaced. Restore `fundamental_data`
|
||
from the pre-cutover database backup, or—before A6—manually run the legacy
|
||
Fundamental Collector if its provider keys and quota are still available.
|
||
- Disable a failing source-import job in Admin → Jobs only when ingestion itself
|
||
must stop. Existing promoted snapshots/events remain available.
|
||
- Inspect the job runtime, latest `data_import_runs.validation_json`, service
|
||
logs, and Admin → System Events before retrying.
|
||
- Re-run `sudo -u deploy bash ./deploy/provision_fundamentals.sh --check` for
|
||
binary, clone, permission, disk, or environment failures.
|
||
- The Dolt clone is a reproducible cache and does not need a bespoke backup.
|
||
PostgreSQL (including `earnings_events`, `fundamental_snapshots`, and import
|
||
audit rows) must remain covered by the normal production database backup.
|
||
- Do not proceed to A6 until the activated cache has completed the observation
|
||
window and the forward earnings calendar remains timely.
|
||
- If report generation fails, inspect Admin → System Events and verify
|
||
`FUNDAMENTALS_PARITY_REPORT_DIR` exists and is writable by `deploy`. Existing
|
||
reports and all live data remain untouched.
|