chore: decommission FMP, Finnhub and Alpha Vantage (A6)
The A5 cutover has been on and observed in production, so SEC Company Facts + DoltHub earnings are already the live source for `fundamental_data`. This removes everything the legacy path still occupied. Gone: the three providers and their config/env keys; the weekly `fundamental_collector` job; the cutover toggle (SEC + Dolt is now the unconditional path, so `off` can no longer silently freeze scoring inputs); the A5 parity report, whose deltas became structurally zero once the candidate builder started writing the table it compared against; and the FMP tier of universe bootstrap. Two behavioral notes: - Disabling **SEC Fundamentals Import** now stops the SEC network fetch only. The local cache refresh moved outside the job-enable check, because candidates also derive from daily closes and earnings events — freezing those on an ingestion pause would stale scoring with no fallback left to recover from. - `/ingestion/fetch?sources=fundamentals` still accepts the key and reports `skipped`; there is no per-ticker fetch any more. Migration 029 does not blanket-delete the leftover settings rows. Migrations run before the service restart, and pre-A6 code reads an absent `job_*_enabled` row as *enabled* — so the two behavior-bearing keys become tombstones pinned to safe values (hidden in Admin) and only the inert three are deleted. Removing the provider keys from the production `.env` is the matching rollout step. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -1,17 +1,21 @@
|
||||
# Fundamentals production deployment
|
||||
|
||||
This is the one-time production setup for the Dolt earnings and SEC fundamentals
|
||||
imports. The A5 scoring cutover was approved on 2026-07-24; the compat-cache write
|
||||
path is still default-off until the explicit production switch below is set. Do
|
||||
not add OS cron entries: the application scheduler owns both jobs.
|
||||
imports. Since A6 (2026-08) these are the *only* fundamentals sources — the
|
||||
FMP/Finnhub/Alpha Vantage providers, the weekly legacy collector and the A5 parity
|
||||
report are gone, and the cache write path is unconditional. Do not add OS cron
|
||||
entries: the application scheduler owns both jobs.
|
||||
|
||||
## What the deployment adds
|
||||
|
||||
- `Dolt Earnings Import (shadow)` runs daily at 02:30 America/New_York.
|
||||
- `SEC Fundamentals Import` runs daily at 04:00 America/New_York. Its local
|
||||
`fundamental_data` refresh runs only when the A5 switch is enabled.
|
||||
- `Fundamentals Parity Report (read-only)` runs daily at 05:30 America/New_York.
|
||||
- `Dolt Earnings Import` runs daily at 02:30 America/New_York.
|
||||
- `SEC Fundamentals Import` runs daily at 04:00 America/New_York, then refreshes
|
||||
`fundamental_data` — the compat cache scoring reads — from stored snapshots,
|
||||
earnings events and closes.
|
||||
- Both jobs are visible, toggleable, and manually triggerable in Admin → Jobs.
|
||||
**Disabling the SEC job stops its SEC network fetch only**; the local cache
|
||||
refresh still runs, because prices and earnings move daily even when no filing
|
||||
does.
|
||||
- Cron expressions are editable in Admin → Schedule.
|
||||
- Every attempt is recorded in `data_import_runs`; failures also create a system
|
||||
event. A failed validation does not promote partial data.
|
||||
@@ -36,14 +40,16 @@ DOLT_EARNINGS_SUBDIR=earnings
|
||||
DOLT_MIN_FREE_DISK_GB=5.0
|
||||
SEC_USER_AGENT=signal-platform/1.0 (contact: real-address@example.com)
|
||||
SEC_REQUEST_SPACING_SECONDS=0.2
|
||||
FUNDAMENTALS_PARITY_REPORT_DIR=/var/lib/signal-platform/reports/fundamentals-parity
|
||||
```
|
||||
|
||||
Use a real monitored contact address. Keep at least 5 GB free at the Dolt data
|
||||
path; 8–10 GB gives comfortable growth headroom. The data directory must stay
|
||||
outside `/opt/signalplatform`, because deployments use `rsync --delete` there.
|
||||
The parity-report directory is also persistent and owned by the service user;
|
||||
its small timestamped JSON/CSV bundles form the temporary A5 review trail.
|
||||
|
||||
`FMP_API_KEY`, `FINNHUB_API_KEY` and `ALPHA_VANTAGE_API_KEY` must be **removed**
|
||||
from this file. Nothing reads them any more, and leaving them installed is the
|
||||
one thing that would let a rolled-back pre-A6 process resume the legacy
|
||||
collector and overwrite the SEC/Dolt cache.
|
||||
|
||||
## One-time provisioning
|
||||
|
||||
@@ -91,27 +97,7 @@ In Admin → Jobs, wait until no other job is running, then:
|
||||
data and still handles partial/missing issuers cleanly. A ticker held by the
|
||||
quality gate should show **New setups paused** with the specific SEC reason.
|
||||
|
||||
## A5 parity observation window
|
||||
|
||||
After both shadow imports are healthy, trigger **Fundamentals Parity Report
|
||||
(read-only)** once in Admin → Jobs. The **A5 Fundamentals Parity** card above
|
||||
the jobs shows the latest coverage/delta summary and provides authenticated JSON
|
||||
and CSV downloads. The canonical server-side bundles are archived at:
|
||||
|
||||
```text
|
||||
/var/lib/signal-platform/reports/fundamentals-parity/
|
||||
```
|
||||
|
||||
The scheduler then generates one report daily at 05:30 New York time, after the
|
||||
02:30 Dolt and 04:00 SEC jobs. Review 5–7 consecutive reports before making the
|
||||
cutover decision. A report never writes `fundamental_data`, dimension/composite
|
||||
scores, rankings, qualification state, or an approval flag. Materiality bands
|
||||
only highlight rows for review; A5 still requires explicit approval.
|
||||
|
||||
Each bundle contains legacy and candidate P/E, revenue growth, and earnings
|
||||
surprise; definition notes; source revisions and price dates; recomputed legacy
|
||||
and candidate fundamental scores; and per-universe fundamental-rank changes.
|
||||
Definition changes remain explicit even when numeric deltas are small.
|
||||
## Verification
|
||||
|
||||
Optional database verification:
|
||||
|
||||
@@ -162,45 +148,24 @@ Expect `OK: source lock is busy`. This is the remaining live-PostgreSQL
|
||||
mutual-exclusion check; SQLite unit tests cannot exercise PostgreSQL advisory
|
||||
locks. A second Admin trigger should independently report the job as busy.
|
||||
|
||||
## A5 production activation (approved 2026-07-24)
|
||||
## The fundamentals cache
|
||||
|
||||
The write path is controlled by the SystemSetting
|
||||
`fundamental_data_sec_dolt_cutover_enabled`. An absent value, `false`, or any
|
||||
value other than `true` leaves `fundamental_data` untouched. Before enabling it,
|
||||
confirm the normal PostgreSQL backup containing `fundamental_data` is current.
|
||||
`fundamental_data` is the compat cache scoring reads. The SEC Fundamentals
|
||||
Import rebuilds it every run from data already in PostgreSQL: newest valid
|
||||
snapshots x latest close for `pe_ratio` and `market_cap`, snapshots alone for
|
||||
`revenue_growth`, and `earnings_events` for `earnings_surprise` and
|
||||
`next_earnings_date`. It therefore also runs after an SEC network/validation
|
||||
failure, a `no_op`, a source-lock skip, or with the job disabled — no network
|
||||
access is involved. The job message appends the cache row count and the changed
|
||||
score-input count.
|
||||
|
||||
In **Admin → Settings → Fundamentals data source**:
|
||||
A refresh marks affected fundamental and composite score caches stale. The
|
||||
normal 15:30 near-close scanner recomputes them before using the rankings; until
|
||||
then, reads truthfully expose the stale state.
|
||||
|
||||
1. Turn on **Use SEC + Dolt for scoring inputs** and accept the confirmation.
|
||||
2. Click **Run refresh now**. The SEC import may be `promoted` or `no_op`; either
|
||||
result runs the local cache refresh.
|
||||
|
||||
The weekly legacy collector is automatically skipped while the switch is on, so
|
||||
it cannot overwrite the activated cache. The switch remains visible even before
|
||||
its SystemSetting row exists because the safe default is off.
|
||||
|
||||
If the Admin UI is unavailable, enable the cutover directly in PostgreSQL:
|
||||
Verify the refreshed rows:
|
||||
|
||||
```sql
|
||||
INSERT INTO system_settings (key, value, updated_at)
|
||||
VALUES ('fundamental_data_sec_dolt_cutover_enabled', 'true', now())
|
||||
ON CONFLICT (key) DO UPDATE
|
||||
SET value = EXCLUDED.value, updated_at = now();
|
||||
```
|
||||
|
||||
Then trigger **SEC Fundamentals Import** once in Admin → Jobs. Once enabled, the
|
||||
same refresh also runs after an SEC network/validation failure or a source-lock
|
||||
skip, because it reads only PostgreSQL snapshots, earnings events, and closes.
|
||||
The job message appends the cache row count and changed score-input count when
|
||||
the import itself completed successfully.
|
||||
|
||||
Verify the switch and refreshed rows:
|
||||
|
||||
```sql
|
||||
SELECT key, value, updated_at
|
||||
FROM system_settings
|
||||
WHERE key = 'fundamental_data_sec_dolt_cutover_enabled';
|
||||
|
||||
SELECT count(*) AS rows,
|
||||
max(fetched_at) AS refreshed_at,
|
||||
count(pe_ratio) AS pe_available,
|
||||
@@ -213,28 +178,33 @@ SELECT dimension, is_stale, count(*)
|
||||
FROM dimension_scores
|
||||
WHERE dimension = 'fundamental'
|
||||
GROUP BY dimension, is_stale;
|
||||
|
||||
SELECT is_stale, count(*)
|
||||
FROM composite_scores
|
||||
GROUP BY is_stale;
|
||||
```
|
||||
|
||||
The first refresh intentionally marks affected fundamental and composite score
|
||||
caches stale. The normal 15:30 near-close scanner recomputes them before using
|
||||
the rankings; until then, reads truthfully expose the stale state. Observe at
|
||||
least several scheduled cycles before A6 removes the legacy providers.
|
||||
### Retired settings rows (delete later)
|
||||
|
||||
Migration `029` pinned two SystemSetting rows as rollback tombstones rather than
|
||||
deleting them, because migrations run before the service restarts and a
|
||||
rolled-back pre-A6 process reads an absent `job_..._enabled` row as *enabled*:
|
||||
|
||||
| key | pinned value |
|
||||
|---|---|
|
||||
| `fundamental_data_sec_dolt_cutover_enabled` | `true` |
|
||||
| `job_fundamental_collector_enabled` | `false` |
|
||||
|
||||
Neither controls anything now; Admin -> Settings hides both. Once the A6
|
||||
rollback window has closed, delete the rows and the `MANAGED_SETTINGS` filter in
|
||||
`frontend/src/components/admin/SettingsForm.tsx`.
|
||||
|
||||
## Failure and rollback
|
||||
|
||||
- To stop the A5 cache writes without stopping SEC snapshot ingestion, turn off
|
||||
**Use SEC + Dolt for scoring inputs** in Admin → Settings. If the UI is
|
||||
unavailable, set `fundamental_data_sec_dolt_cutover_enabled` back to `false`
|
||||
with the SQL above (changing only the value). This prevents the next local
|
||||
refresh but does not restore rows already replaced. Restore `fundamental_data`
|
||||
from the pre-cutover database backup, or—before A6—manually run the legacy
|
||||
Fundamental Collector if its provider keys and quota are still available.
|
||||
- Disable a failing source-import job in Admin → Jobs only when ingestion itself
|
||||
must stop. Existing promoted snapshots/events remain available.
|
||||
- **There is no provider fallback any more.** To recover bad `fundamental_data`
|
||||
values, restore the table from the PostgreSQL backup; the next scheduled run
|
||||
will rebuild it from the current snapshots. Confirm a recent backup exists
|
||||
before any change that could corrupt the snapshots.
|
||||
- Disable a failing source-import job in Admin → Jobs only when SEC network
|
||||
access itself must stop — the local cache refresh keeps running, and existing
|
||||
promoted snapshots/events remain available. To freeze the cache as well,
|
||||
disable the job *and* accept that P/E, market cap and earnings dates go stale.
|
||||
- Inspect the job runtime, latest `data_import_runs.validation_json`, service
|
||||
logs, and Admin → System Events before retrying.
|
||||
- `unresolved_filing` is emitted once when a filing enters automatic retry. It
|
||||
@@ -250,8 +220,3 @@ least several scheduled cycles before A6 removes the legacy providers.
|
||||
- The Dolt clone is a reproducible cache and does not need a bespoke backup.
|
||||
PostgreSQL (including `earnings_events`, `fundamental_snapshots`, and import
|
||||
audit rows) must remain covered by the normal production database backup.
|
||||
- Do not proceed to A6 until the activated cache has completed the observation
|
||||
window and the forward earnings calendar remains timely.
|
||||
- If report generation fails, inspect Admin → System Events and verify
|
||||
`FUNDAMENTALS_PARITY_REPORT_DIR` exists and is writable by `deploy`. Existing
|
||||
reports and all live data remain untouched.
|
||||
|
||||
Reference in New Issue
Block a user