dennisthiessen/signal-platform
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

chore: decommission FMP, Finnhub and Alpha Vantage (A6)

Browse Source 
The A5 cutover has been on and observed in production, so SEC Company Facts +
DoltHub earnings are already the live source for `fundamental_data`. This
removes everything the legacy path still occupied.

Gone: the three providers and their config/env keys; the weekly
`fundamental_collector` job; the cutover toggle (SEC + Dolt is now the
unconditional path, so `off` can no longer silently freeze scoring inputs); the
A5 parity report, whose deltas became structurally zero once the candidate
builder started writing the table it compared against; and the FMP tier of
universe bootstrap.

Two behavioral notes:

- Disabling **SEC Fundamentals Import** now stops the SEC network fetch only.
  The local cache refresh moved outside the job-enable check, because candidates
  also derive from daily closes and earnings events — freezing those on an
  ingestion pause would stale scoring with no fallback left to recover from.
- `/ingestion/fetch?sources=fundamentals` still accepts the key and reports
  `skipped`; there is no per-ticker fetch any more.

Migration 029 does not blanket-delete the leftover settings rows. Migrations run
before the service restart, and pre-A6 code reads an absent `job_*_enabled` row
as *enabled* — so the two behavior-bearing keys become tombstones pinned to safe
values (hidden in Admin) and only the inert three are deleted. Removing the
provider keys from the production `.env` is the matching rollout step.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
dennisthiessen
2026-08-07 11:19:28 +02:00
co-authored by Claude Opus 5
parent f5d4b516ab
commit 3e83d63b05
51 changed files with 368 additions and 3787 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/dolt-integration-plan.md
+12 -6
View File
@@ -430,7 +430,11 @@ workstream B — Alpaca remains the price source throughout.
approval** — see the handoff section below. Step (c) is implemented behind the
default-off `fundamental_data_sec_dolt_cutover_enabled` SystemSetting; the
remaining production action is flipping that switch on and observing it.
- A6. Remove FMP/Finnhub/Alpha Vantage; keep monitoring + manual fallback.
- A6. **DONE 2026-08-07.** FMP/Finnhub/Alpha Vantage removed, along with the
weekly `fundamental_collector` job, the A5 cutover toggle (SEC+Dolt is now the
unconditional path) and the parity report. Migration `029` tombstones the two
behavior-bearing settings rows for the rollback window; the archived parity
bundles stay as the A5 evidence trail.
**Workstream B (independent, start when wanted):**
@@ -507,11 +511,13 @@ managed by the **Fundamentals data source** card in Admin → Settings; while ac
the weekly legacy collector skips itself so it cannot overwrite the SEC/Dolt cache.
See `docs/fundamentals-deployment.md` for the production flip and rollback procedure.
**Task 2 — A6 decommissioning.** After a short observation window: remove
FMP/Finnhub/Alpha Vantage providers, config and env keys; keep monitoring + manual
fallback. Gated by the acceptance criteria above — especially forward-calendar
timeliness from `dolt_earnings` (its `source_max_date` ran ~5 weeks ahead as of
2026-07-23, which passes).
**Task 2 — A6 decommissioning: DONE 2026-08-07.** The cutover ran on and was
observed in production, so the legacy providers, their config/env keys, the weekly
collector job and the parity report were all removed. Two consequences to carry:
(1) `fundamental_data` now has no provider fallback — recovery is restore-from-backup;
(2) disabling **SEC Fundamentals Import** stops the SEC fetch only, because the local
cache refresh was deliberately moved outside the job-enable check. Remaining
follow-up: delete the migration-029 tombstone rows once the rollback window closes.
**Known caveats to carry (documented in the findings report, not bugs to fix):**
- KLAC-class post-filing splits: P/E wrong until the next 10-Q; undetectable from
docs/fundamentals-deployment.md
+52 -87
View File
@@ -1,17 +1,21 @@
# Fundamentals production deployment
This is the one-time production setup for the Dolt earnings and SEC fundamentals
imports. The A5 scoring cutover was approved on 2026-07-24; the compat-cache write
path is still default-off until the explicit production switch below is set. Do
not add OS cron entries: the application scheduler owns both jobs.
imports. Since A6 (2026-08) these are the *only* fundamentals sources — the
FMP/Finnhub/Alpha Vantage providers, the weekly legacy collector and the A5 parity
report are gone, and the cache write path is unconditional. Do not add OS cron
entries: the application scheduler owns both jobs.
## What the deployment adds
- `Dolt Earnings Import (shadow)` runs daily at 02:30 America/New_York.
- `SEC Fundamentals Import` runs daily at 04:00 America/New_York. Its local
`fundamental_data` refresh runs only when the A5 switch is enabled.
- `Fundamentals Parity Report (read-only)` runs daily at 05:30 America/New_York.
- `Dolt Earnings Import` runs daily at 02:30 America/New_York.
- `SEC Fundamentals Import` runs daily at 04:00 America/New_York, then refreshes
`fundamental_data` — the compat cache scoring reads — from stored snapshots,
earnings events and closes.
- Both jobs are visible, toggleable, and manually triggerable in Admin → Jobs.
**Disabling the SEC job stops its SEC network fetch only**; the local cache
refresh still runs, because prices and earnings move daily even when no filing
does.
- Cron expressions are editable in Admin → Schedule.
- Every attempt is recorded in `data_import_runs`; failures also create a system
event. A failed validation does not promote partial data.
@@ -36,14 +40,16 @@ DOLT_EARNINGS_SUBDIR=earnings
DOLT_MIN_FREE_DISK_GB=5.0
SEC_USER_AGENT=signal-platform/1.0 (contact: real-address@example.com)
SEC_REQUEST_SPACING_SECONDS=0.2
FUNDAMENTALS_PARITY_REPORT_DIR=/var/lib/signal-platform/reports/fundamentals-parity
```
Use a real monitored contact address. Keep at least 5 GB free at the Dolt data
path; 8–10 GB gives comfortable growth headroom. The data directory must stay
outside `/opt/signalplatform`, because deployments use `rsync --delete` there.
The parity-report directory is also persistent and owned by the service user;
its small timestamped JSON/CSV bundles form the temporary A5 review trail.
`FMP_API_KEY`, `FINNHUB_API_KEY` and `ALPHA_VANTAGE_API_KEY` must be **removed**
from this file. Nothing reads them any more, and leaving them installed is the
one thing that would let a rolled-back pre-A6 process resume the legacy
collector and overwrite the SEC/Dolt cache.
## One-time provisioning
@@ -91,27 +97,7 @@ In Admin → Jobs, wait until no other job is running, then:
data and still handles partial/missing issuers cleanly. A ticker held by the
quality gate should show **New setups paused** with the specific SEC reason.
## A5 parity observation window
After both shadow imports are healthy, trigger **Fundamentals Parity Report
(read-only)** once in Admin → Jobs. The **A5 Fundamentals Parity** card above
the jobs shows the latest coverage/delta summary and provides authenticated JSON
and CSV downloads. The canonical server-side bundles are archived at:
```text
/var/lib/signal-platform/reports/fundamentals-parity/
```
The scheduler then generates one report daily at 05:30 New York time, after the
02:30 Dolt and 04:00 SEC jobs. Review 5–7 consecutive reports before making the
cutover decision. A report never writes `fundamental_data`, dimension/composite
scores, rankings, qualification state, or an approval flag. Materiality bands
only highlight rows for review; A5 still requires explicit approval.
Each bundle contains legacy and candidate P/E, revenue growth, and earnings
surprise; definition notes; source revisions and price dates; recomputed legacy
and candidate fundamental scores; and per-universe fundamental-rank changes.
Definition changes remain explicit even when numeric deltas are small.
## Verification
Optional database verification:
@@ -162,45 +148,24 @@ Expect `OK: source lock is busy`. This is the remaining live-PostgreSQL
mutual-exclusion check; SQLite unit tests cannot exercise PostgreSQL advisory
locks. A second Admin trigger should independently report the job as busy.
## A5 production activation (approved 2026-07-24)
## The fundamentals cache
The write path is controlled by the SystemSetting
`fundamental_data_sec_dolt_cutover_enabled`. An absent value, `false`, or any
value other than `true` leaves `fundamental_data` untouched. Before enabling it,
confirm the normal PostgreSQL backup containing `fundamental_data` is current.
`fundamental_data` is the compat cache scoring reads. The SEC Fundamentals
Import rebuilds it every run from data already in PostgreSQL: newest valid
snapshots x latest close for `pe_ratio` and `market_cap`, snapshots alone for
`revenue_growth`, and `earnings_events` for `earnings_surprise` and
`next_earnings_date`. It therefore also runs after an SEC network/validation
failure, a `no_op`, a source-lock skip, or with the job disabled — no network
access is involved. The job message appends the cache row count and the changed
score-input count.
In **Admin → Settings → Fundamentals data source**:
A refresh marks affected fundamental and composite score caches stale. The
normal 15:30 near-close scanner recomputes them before using the rankings; until
then, reads truthfully expose the stale state.
1. Turn on **Use SEC + Dolt for scoring inputs** and accept the confirmation.
2. Click **Run refresh now**. The SEC import may be `promoted` or `no_op`; either
result runs the local cache refresh.
The weekly legacy collector is automatically skipped while the switch is on, so
it cannot overwrite the activated cache. The switch remains visible even before
its SystemSetting row exists because the safe default is off.
If the Admin UI is unavailable, enable the cutover directly in PostgreSQL:
Verify the refreshed rows:
```sql
INSERT INTO system_settings (key, value, updated_at)
VALUES ('fundamental_data_sec_dolt_cutover_enabled', 'true', now())
ON CONFLICT (key) DO UPDATE
SET value = EXCLUDED.value, updated_at = now();
```
Then trigger **SEC Fundamentals Import** once in Admin → Jobs. Once enabled, the
same refresh also runs after an SEC network/validation failure or a source-lock
skip, because it reads only PostgreSQL snapshots, earnings events, and closes.
The job message appends the cache row count and changed score-input count when
the import itself completed successfully.
Verify the switch and refreshed rows:
```sql
SELECT key, value, updated_at
FROM system_settings
WHERE key = 'fundamental_data_sec_dolt_cutover_enabled';
SELECT count(*) AS rows,
max(fetched_at) AS refreshed_at,
count(pe_ratio) AS pe_available,
@@ -213,28 +178,33 @@ SELECT dimension, is_stale, count(*)
FROM dimension_scores
WHERE dimension = 'fundamental'
GROUP BY dimension, is_stale;
SELECT is_stale, count(*)
FROM composite_scores
GROUP BY is_stale;
```
The first refresh intentionally marks affected fundamental and composite score
caches stale. The normal 15:30 near-close scanner recomputes them before using
the rankings; until then, reads truthfully expose the stale state. Observe at
least several scheduled cycles before A6 removes the legacy providers.
### Retired settings rows (delete later)
Migration `029` pinned two SystemSetting rows as rollback tombstones rather than
deleting them, because migrations run before the service restarts and a
rolled-back pre-A6 process reads an absent `job_..._enabled` row as *enabled*:
| key | pinned value |
|---|---|
| `fundamental_data_sec_dolt_cutover_enabled` | `true` |
| `job_fundamental_collector_enabled` | `false` |
Neither controls anything now; Admin -> Settings hides both. Once the A6
rollback window has closed, delete the rows and the `MANAGED_SETTINGS` filter in
`frontend/src/components/admin/SettingsForm.tsx`.
## Failure and rollback
- To stop the A5 cache writes without stopping SEC snapshot ingestion, turn off
**Use SEC + Dolt for scoring inputs** in Admin → Settings. If the UI is
unavailable, set `fundamental_data_sec_dolt_cutover_enabled` back to `false`
with the SQL above (changing only the value). This prevents the next local
refresh but does not restore rows already replaced. Restore `fundamental_data`
from the pre-cutover database backup, or—before A6—manually run the legacy
Fundamental Collector if its provider keys and quota are still available.
- Disable a failing source-import job in Admin → Jobs only when ingestion itself
must stop. Existing promoted snapshots/events remain available.
- **There is no provider fallback any more.** To recover bad `fundamental_data`
values, restore the table from the PostgreSQL backup; the next scheduled run
will rebuild it from the current snapshots. Confirm a recent backup exists
before any change that could corrupt the snapshots.
- Disable a failing source-import job in Admin → Jobs only when SEC network
access itself must stop — the local cache refresh keeps running, and existing
promoted snapshots/events remain available. To freeze the cache as well,
disable the job *and* accept that P/E, market cap and earnings dates go stale.
- Inspect the job runtime, latest `data_import_runs.validation_json`, service
logs, and Admin → System Events before retrying.
- `unresolved_filing` is emitted once when a filing enters automatic retry. It
@@ -250,8 +220,3 @@ least several scheduled cycles before A6 removes the legacy providers.
- The Dolt clone is a reproducible cache and does not need a bespoke backup.
PostgreSQL (including `earnings_events`, `fundamental_snapshots`, and import
audit rows) must remain covered by the normal production database backup.
- Do not proceed to A6 until the activated cache has completed the observation
window and the forward earnings calendar remains timely.
- If report generation fails, inspect Admin → System Events and verify
`FUNDAMENTALS_PARITY_REPORT_DIR` exists and is writable by `deploy`. Existing
reports and all live data remain untouched.