The A5 cutover has been on and observed in production, so SEC Company Facts +
DoltHub earnings are already the live source for `fundamental_data`. This
removes everything the legacy path still occupied.
Gone: the three providers and their config/env keys; the weekly
`fundamental_collector` job; the cutover toggle (SEC + Dolt is now the
unconditional path, so `off` can no longer silently freeze scoring inputs); the
A5 parity report, whose deltas became structurally zero once the candidate
builder started writing the table it compared against; and the FMP tier of
universe bootstrap.
Two behavioral notes:
- Disabling **SEC Fundamentals Import** now stops the SEC network fetch only.
The local cache refresh moved outside the job-enable check, because candidates
also derive from daily closes and earnings events — freezing those on an
ingestion pause would stale scoring with no fallback left to recover from.
- `/ingestion/fetch?sources=fundamentals` still accepts the key and reports
`skipped`; there is no per-ticker fetch any more.
Migration 029 does not blanket-delete the leftover settings rows. Migrations run
before the service restart, and pre-A6 code reads an absent `job_*_enabled` row
as *enabled* — so the two behavior-bearing keys become tombstones pinned to safe
values (hidden in Admin) and only the inert three are deleted. Removing the
provider keys from the production `.env` is the matching rollout step.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two independent bugs left the near-close scan running on the previous
session's close, silently degrading live execution to the stale_close
floor (~1.57 Sharpe) instead of the intended ~1.77 close-fill case.
1. OHLCV window never covered the current day. Daily bars are stamped at
session start (04:00Z under EDT), so an end of midnight-on-end_date
landed before that day's bar and dropped it. Widening the window alone
fails the whole request with 'subscription does not permit querying
recent SIP data', so end is also clamped to now-20min. Today's bar is
now returned, roughly 20 minutes behind live -- within the staleness
the near-close design already assumed.
Intraday runs therefore store a partial bar and ingestion progress
reaches today, which made incremental resume skip the after-close
refresh entirely. collect_ohlcv_final() re-pulls the last sessions so
the consolidated bar overwrites the partial one before outcome eval.
2. APScheduler's from_crontab() passes day-of-week to its own field where
0=Monday, so '1-5' meant Tue-Sat: every Monday was skipped and the
scanner ran Saturdays on stale data. Weekday schedules now use names.
Stored settings already corrected via Admin; this fixes the defaults.
Tests cover both: today's bar inside the window, the delayed-data clamp,
historical windows untruncated, and a week of fire times asserting Monday
is present and weekends are not.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Finnhub profile2 reports marketCapitalization in millions; storing it
as dollars made mega-caps like SPCX show as micro (e.g. 1.8M). Normalize
on ingest, add unit tests, and include a one-shot SQL backfill script.
Richer LLM output (same grounded call, ~no extra cost):
- All providers now also return a recommendation (buy/hold/avoid) and a thorough
reasoning paragraph; Gemini now actually captures reasoning + grounding
citations (it was dropping them). Stored on sentiment_scores (migration 008),
exposed in the API; display-only — NOT fed into the composite/EV.
- Ticker Sentiment panel shows an "LLM view" badge and a "Full analysis & sources"
expander with the complete reasoning + citations.
Search-budget scoping (Gemini grounding free tier = 5000/mo):
- collect_sentiment now targets only watchlist + open paper trades + top-N by
composite, skips tickers refreshed within sentiment_fresh_hours (72h), and caps
per run (sentiment_max_per_run). Once the relevant set is fresh, runs spend 0
searches until it ages out — bounding monthly usage well under the free tier.
- Widened sentiment lookback to 7d (scoring + display) so sparser collection
still feeds the dimension score.
Deploy: alembic upgrade (sentiment_scores.recommendation). Switch provider to
Gemini Flash in Admin for the cost win (grounded, cheapest).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Finnhub's earnings calendar now supplies next_earnings_date through the
fundamentals chain; persisted on fundamental_data (migration 006) and exposed in
the fundamentals API. The recommendation panel warns when earnings fall within
the ~30-day target horizon (a report can gap price through stop/target) and
otherwise shows the next date. Informational only.
Deploy: run alembic upgrade (new fundamental_data.next_earnings_date column).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Root cause of "price plan needed in bulk but fine on manual reload": on free
tiers FMP returns only market cap (others 402) and the chain merged that as a
partial success — so when the Finnhub/Alpha Vantage fallbacks were rate-limited
during a bulk run, the chain silently returned market-cap-only and the
collector's backoff never engaged. Manual single fetches worked because the
fallbacks weren't throttled at that moment.
Fixes:
- Chain distinguishes RateLimitError from other failures: if a fallback is
rate-limited and fields are still missing, raise RateLimitError (unless
allow_partial=True) so the collector backs off and retries.
- Bulk job paces requests (fundamental_request_spacing_seconds, default 3s) to
stay under Finnhub's ~60/min, and on retry-exhaustion stores partial data and
continues instead of aborting the whole run.
- Manual fetch passes allow_partial=True so a lone 429 doesn't fail the refresh.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Triggered by CNC showing "LONG (High Confidence)" with SHORT reasoning
and no long setup.
- A: recommendation action + reasoning are ticker-level and identical
on both setups; reasoning always matches the shown action
- B: recommended_action only picks a direction with a tradeable setup;
strong bias with no setup (e.g. price at ATH) → NEUTRAL with an
explanatory reason instead of a fake LONG_HIGH
- C: confidence is a directional-agreement model — opposing signals push
it below 50 (SHORT on a 92-technical/99-momentum stock ~0%, not 55%)
- D: fundamental score requires >=2 real metrics (market-cap-only no
longer yields a high score)
- E: RSI score peaks at healthy momentum (~60) and penalizes
overbought/oversold extremes instead of treating RSI 90 as maximal
- F: fundamentals chain merges fields across providers (FMP market cap
+ Finnhub P/E) instead of stopping at the first with any field
- NEUTRAL label: "No Clear Setup" (covers untradeable-bias case)
Scores recompute on next scan/scoring run; C and E shift score
distributions intentionally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
xAI returned 410 — search_parameters/Live Search is retired. Route xAI
through the Responses API web_search tool instead (same path as OpenAI):
- OpenAISentimentProvider parametrized with base_url / tool_type / source
- xAI builds it against https://api.x.ai/v1 with the web_search tool
- Drop the dead Live Search code from the generic compatible provider
- Frontend label: "xAI Grok — web search"
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Providers (admin-switchable, no redeploy):
- DeepSeek and any OpenAI-compatible endpoint (OpenRouter, Together,
Groq, local Ollama) via a generic Chat Completions adapter + base_url
- xAI Grok with Live Search (search_parameters web+X, citations) —
grounded tier alongside OpenAI and Gemini
- DeepSeek / generic compatible endpoints are ungrounded (no web
search); UI shows an amber warning and labels each provider's grounding
- Optional env fallbacks DEEPSEEK_API_KEY / XAI_API_KEY
UI: replace native <select> (unstyleable white popup on Windows) with a
custom dark Dropdown component everywhere — sentiment provider, scanner
filters, market sort, indicators, admin universe, user role.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
dennisthiessenandClaude Opus 5