signal-platform/docs/fundamentals-deployment.md

Fundamentals production deployment

This is the one-time production setup for the Dolt earnings and SEC fundamentals imports. Since A6 (2026-08) these are the only fundamentals sources — the FMP/Finnhub/Alpha Vantage providers, the weekly legacy collector and the A5 parity report are gone, and the cache write path is unconditional. Do not add OS cron entries: the application scheduler owns both jobs.

What the deployment adds

• Dolt Earnings Import runs daily at 02:30 America/New_York.
• SEC Fundamentals Import runs daily at 04:00 America/New_York, then refreshes fundamental_data — the compat cache scoring reads — from stored snapshots, earnings events and closes.
• Both jobs are visible, toggleable, and manually triggerable in Admin → Jobs. Disabling the SEC job stops its SEC network fetch only; the local cache refresh still runs, because prices and earnings move daily even when no filing does.
• Cron expressions are editable in Admin → Schedule.
• Every attempt is recorded in data_import_runs; failures also create a system event. A failed validation does not promote partial data.
• An SEC filing still missing after the short publication-lag window enters sec_filing_gaps. The daily importer retries it automatically; affected tickers are excluded from actionable setups until a snapshot is recovered or a later valid 10-K/10-Q supersedes the gap. Migration 028 materializes older promoted gaps into this queue once, so setup reads never scan import history.

The systemd service uses one application worker. The import framework also holds a PostgreSQL advisory lock per source, so an overlapping manual/scheduled run is skipped safely.

Prerequisites

The production .env at /opt/signalplatform/.env must contain:

DOLT_BINARY=/usr/local/bin/dolt
DOLT_DATA_DIR=/var/lib/signal-platform/dolt
DOLT_EARNINGS_SUBDIR=earnings
DOLT_MIN_FREE_DISK_GB=5.0
SEC_USER_AGENT=signal-platform/1.0 (contact: real-address@example.com)
SEC_REQUEST_SPACING_SECONDS=0.2

Use a real monitored contact address. Keep at least 5 GB free at the Dolt data path; 8–10 GB gives comfortable growth headroom. The data directory must stay outside /opt/signalplatform, because deployments use rsync --delete there.

FMP_API_KEY, FINNHUB_API_KEY and ALPHA_VANTAGE_API_KEY must be removed from this file. Nothing reads them any more, and leaving them installed is the one thing that would let a rolled-back pre-A6 process resume the legacy collector and overwrite the SEC/Dolt cache.

One-time provisioning

First deploy the commit containing this bundle to production. Then SSH to the server and run:

cd /opt/signalplatform
sudo bash ./deploy/provision_fundamentals.sh
sudo -u deploy bash ./deploy/provision_fundamentals.sh --check
sudo systemctl restart signalplatform.service
curl -fsS http://127.0.0.1:8998/api/v1/health

The provisioner is idempotent. It installs the pinned Dolt version, creates the persistent directory as deploy:deploy, clones post-no-preference/earnings, configures a repository-local author identity for dolt pull, verifies free space and .env, and refuses an unexpected Dolt version. It does not modify PostgreSQL or start an import. The public clone does not require dolt login.

For a server provisioned before the author-identity check was added, repair the existing clone once with:

sudo -u deploy -H /usr/local/bin/dolt config --global --add user.name "Signal Platform"
sudo -u deploy -H /usr/local/bin/dolt config --global --add user.email "signal-platform@localhost"

Do not replace the pinned version with latest. A future Dolt upgrade should be a reviewed change to DOLT_VERSION, followed by the same provision/check flow.

First-run verification

In Admin → Jobs, wait until no other job is running, then:

1. Trigger Dolt Earnings Import (shadow). Expect completed with import status promoted; a repeat without an upstream change should report no_op.
2. Trigger SEC Fundamentals Import. The first run performs the tracked-universe history backfill and can take materially longer than a daily incremental run. Expect completed with import status promoted.
3. Check Admin → System Events. There should be no new import error.
4. Confirm the next-run times correspond to 02:30 and 04:00 New York time.
5. Open several ticker pages and confirm the fundamentals panel has populated data and still handles partial/missing issuers cleanly. A ticker held by the quality gate should show New setups paused with the specific SEC reason.

Verification

Optional database verification:

SELECT source, status, revision, source_max_date, started_at, completed_at,
       validation_json
FROM data_import_runs
WHERE source IN ('dolt_earnings', 'sec_facts')
ORDER BY id DESC
LIMIT 10;

SELECT count(*) FROM earnings_events WHERE source = 'dolt_earnings';
SELECT count(*), count(DISTINCT cik) FROM fundamental_snapshots;

During the longer first SEC run, execute the following in a second SSH session. It opens an independent database connection and attempts the same source lock:

cd /opt/signalplatform
sudo -u deploy .venv/bin/python - <<'PY'
import asyncio

from sqlalchemy import text

from app.database import engine
from app.services.data_import import _advisory_key


async def main():
    key = _advisory_key("sec_facts")
    async with engine.connect() as connection:
        acquired = await connection.scalar(
            text("SELECT pg_try_advisory_lock(:key)"), {"key": key}
        )
        print("UNEXPECTED: lock acquired" if acquired else "OK: source lock is busy")
        if acquired:
            await connection.execute(
                text("SELECT pg_advisory_unlock(:key)"), {"key": key}
            )


asyncio.run(main())
PY

Expect OK: source lock is busy. This is the remaining live-PostgreSQL mutual-exclusion check; SQLite unit tests cannot exercise PostgreSQL advisory locks. A second Admin trigger should independently report the job as busy.

The fundamentals cache

fundamental_data is the compat cache scoring reads. The SEC Fundamentals Import rebuilds it every run from data already in PostgreSQL: newest valid snapshots x latest close for pe_ratio and market_cap, snapshots alone for revenue_growth, and earnings_events for earnings_surprise and next_earnings_date. It therefore also runs after an SEC network/validation failure, a no_op, a source-lock skip, or with the job disabled — no network access is involved. The job message appends the cache row count and the changed score-input count.

A refresh marks affected fundamental and composite score caches stale. The normal 15:30 near-close scanner recomputes them before using the rankings; until then, reads truthfully expose the stale state.

Verify the refreshed rows:

SELECT count(*) AS rows,
       max(fetched_at) AS refreshed_at,
       count(pe_ratio) AS pe_available,
       count(revenue_growth) AS growth_available,
       count(earnings_surprise) AS surprise_available,
       count(next_earnings_date) AS next_date_available
FROM fundamental_data;

SELECT dimension, is_stale, count(*)
FROM dimension_scores
WHERE dimension = 'fundamental'
GROUP BY dimension, is_stale;

Retired settings rows (delete later)

Migration 029 pinned two SystemSetting rows as rollback tombstones rather than deleting them, because migrations run before the service restarts and a rolled-back pre-A6 process reads an absent job_..._enabled row as enabled:

key	pinned value
fundamental_data_sec_dolt_cutover_enabled	true
job_fundamental_collector_enabled	false

Neither controls anything now; Admin -> Settings hides both. Once the A6 rollback window has closed, delete the rows and the MANAGED_SETTINGS filter in frontend/src/components/admin/SettingsForm.tsx.

Failure and rollback

• There is no provider fallback any more. To recover bad fundamental_data values, restore the table from the PostgreSQL backup; the next scheduled run will rebuild it from the current snapshots. Confirm a recent backup exists before any change that could corrupt the snapshots.
• Disable a failing source-import job in Admin → Jobs only when SEC network access itself must stop — the local cache refresh keeps running, and existing promoted snapshots/events remain available. To freeze the cache as well, disable the job and accept that P/E, market cap and earnings dates go stale.
• Inspect the job runtime, latest data_import_runs.validation_json, service logs, and Admin → System Events before retrying.
• unresolved_filing is emitted once when a filing enters automatic retry. It does not require a server command. If the gap is still current after 14 days, filing_gap_aged is emitted once with the CIK, accession, and parser/mapping reason. A later valid 10-K/10-Q retires the gap even when the original SEC accession never becomes usable.
• Successful co-registrant recovery is logged without a warning. New registrants with no XBRL history are also logged quietly, but their ticker page explains that setups remain paused and that successor shells may need sec_cik_overrides.
• Re-run sudo -u deploy bash ./deploy/provision_fundamentals.sh --check for binary, clone, permission, disk, or environment failures.
• The Dolt clone is a reproducible cache and does not need a bespoke backup. PostgreSQL (including earnings_events, fundamental_snapshots, and import audit rows) must remain covered by the normal production database backup.