Nineteen jobs rendered as one alphabetical list in which a pipeline, one of its
steps, a standalone cron job and a manual-only job were indistinguishable.
Four sections, ordered by the API (category rank, then trading-day order within
it) so the client does not re-derive ordering: Pipelines, Pipeline steps,
Standalone scheduled, Manual only. An unrecognised category still renders, under
"Other" -- a stray section beats a job silently vanishing from the admin page.
Sections rather than nesting steps under their parent, which is what the flat
"runs via pipeline" label invited. Membership is many-to-many -- data_collector
runs in all four pipelines, alerts and outcome_evaluator in two each -- so
nesting means duplicating those rows, and the duplicates would each carry a
Trigger button despite not being distinct actions: only plain collect_ohlcv is
registered, while the near-close and after-close variants are different
coroutines that are not individually triggerable. Instead each pipeline card
lists its step sequence and each step says which pipelines run it, which is the
same information without a button that lies.
Every job now answers "when does this next run" the same way: its own timer, its
soonest enabled parent's ("Next via Intraday Pipeline in 42m"), or "manual
only". Jobs with no recorded run say so explicitly rather than showing nothing.
The status chip and the rate-limit banner still read runtime_* only, so a
persisted failure cannot pin either to a stale state.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Job outcomes lived only in scheduler._job_runtime, an in-memory dict. Every
deploy wiped it, so Admin -> Jobs could report "Active" with no indication a job
had ever run or how it ended -- which is the main thing that page is for.
New job_run_state table (migration 031): one row per job, upserted on job_name.
Deliberately not history -- system_events already grows unbounded with no
retention job, and a second append-only operational table would repeat that
debt. Adding history later is purely additive.
Written from two hooks, NOT from _runtime_finish. That looked cheapest (one
function, ~40 call sites) but unit tests invoke job coroutines directly, so it
would fire detached DB writes at the real session factory throughout the suite,
and there is no testing flag to guard on.
- An APScheduler EVENT_JOB_EXECUTED/ERROR listener covers everything the
scheduler fires, including manual triggers. Its detached task is held in a
module-level set (a bare create_task result can be collected mid-flight) and
drained in the app lifespan before engine.dispose().
- _run_pipeline persists directly, and must: pipeline steps are plain
coroutine calls that emit no scheduler events, so the listener cannot see
them. The step persist sits AFTER the except that swallows step errors --
inside it, exactly the failed runs worth seeing would be skipped. The
orchestrator persists in the finally, and the disabled early-return persists
too, or "skipped" is silently dropped.
_persist_job_run never raises: a persistence failure must not break an otherwise
successful pipeline.
The API reports this as last_run_* and leaves runtime_* meaning strictly live
in-memory state. Reusing runtime_status would have been a regression, not a
no-op: JobControls drives the status chip from it (a job that errored eight days
ago would read "Last run error" forever instead of "Active") and picks the
rate-limit banner from it (a week-old rate limit would pin the banner
permanently). Tests pin the split.
The table starts empty; each job fills its row the next time it finishes. No
backfill from system_events, which records only warning/error outcomes under a
different status vocabulary and would invent successes that never happened.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Groundwork for the Admin -> Jobs cleanup. Three sources of truth collapse into
app/job_catalog.py, which imports nothing from app so both the scheduler and
admin_service can import it at module level (admin_service otherwise has to
import the scheduler inside functions to dodge a cycle).
PIPELINE_MEMBERS is now DERIVED from the four pipeline step lists instead of
being a literal set in admin_service duplicating four lists in scheduler.py with
nothing asserting they agreed. A test pins that the derivation reproduces the
previous hand-maintained 9 names exactly, so this is behaviour-preserving.
Deletes the private _JOB_NAMES list, which held 16 of the 19 jobs:
benchmark_collector, outcome_evaluator and shadow_book had no runtime row, and
so no "last run" line in the panel, until their first run in a given process.
_job_runtime is now seeded from the catalog, and a test pins the invariant.
Next-run is decided by category rather than by reading a timestamp. A pipeline
step has no schedule of its own, so it reports its parent's ("next via Morning
Pipeline in 3h") instead of nothing; a manual job says manual_only rather than
rendering a date. This also fixes a real bug: triggering a paused job set
next_run_time=now, APScheduler re-armed the 520-week backstop behind it, and the
panel displayed "next run in ~87600h". Two independent guards -- the category
rule, plus _visible_next_run dropping anything past a year -- and an APScheduler
listener that re-pauses steps and manual jobs once their run finishes. The
listener is registered at module level because configure_scheduler is called
more than once and add_listener does not deduplicate.
Migrates backtest and ticker_universe_sync from interval to cron (Sun 03:00 ET
and 01:00 ET). configure_scheduler calls remove_all_jobs() on every startup, so
an interval countdown restarts each deploy -- a 168h backtest needed a week of
uninterrupted uptime to fire even once. The codebase already documented this
pitfall as the reason cron was adopted; these two were never migrated. Both are
now editable in Admin -> Schedule.
Also: list_jobs went from one settings query per job (19) to one for all of
them, and data_backfill is hidden from the listing while staying registered and
API-triggerable.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
"Market Regime" and "Regime Monitor" sat next to each other in Admin -> Jobs
(pipeline steps 4 and 5) reading as the same job. They are unrelated, and the
names had it backwards: "Market Regime" is the SPY 50/200 guard that drives the
TopBar trend dot and the counter-trend warning on setups, so it changes what a
setup shows; "Regime Monitor" is the observational AI/Tech thermometer that
explicitly feeds no trades. The more consequential job had the vaguer name.
market_regime "Market Regime" -> "Market Trend (SPY)"
regime_monitor "Regime Monitor" -> "AI/Tech Risk Monitor"
Display strings only. The job *ids* are persisted -- they key the pipeline step
list, cron config, runtime tracking and run history -- so they are untouched,
as is the /regime route, which keeps existing links working.
The label the admin UI renders comes from JOB_LABELS in admin_service (via
routers/jobs.py), not from the scheduler's APScheduler `name=`. Both are updated;
only the former is user-visible.
Carries the vocabulary through the rest of the surface so it does not half-land:
page title, nav ("Regime" -> "Risk"), the empty-state instruction that names the
job to run, the quadrant alert toggle, the morning-pipeline hint, and the
Telegram alert headline ("Regime quadrant change" -> "AI/Tech risk quadrant
change"). No test asserts any of these strings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A6 deployed cleanly and the provider keys are gone from the production `.env`,
which makes the legacy collector inert regardless of any settings row. The two
tombstones migration 029 pinned have no remaining job, and nothing in the
codebase reads either key.
Migration 030 deletes them and drops the Admin filter that hid them. Unlike
029's, its downgrade is meaningful — it restores both rows at their safe values,
since going back past this revision means going back toward code that reads them.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The A5 cutover has been on and observed in production, so SEC Company Facts +
DoltHub earnings are already the live source for `fundamental_data`. This
removes everything the legacy path still occupied.
Gone: the three providers and their config/env keys; the weekly
`fundamental_collector` job; the cutover toggle (SEC + Dolt is now the
unconditional path, so `off` can no longer silently freeze scoring inputs); the
A5 parity report, whose deltas became structurally zero once the candidate
builder started writing the table it compared against; and the FMP tier of
universe bootstrap.
Two behavioral notes:
- Disabling **SEC Fundamentals Import** now stops the SEC network fetch only.
The local cache refresh moved outside the job-enable check, because candidates
also derive from daily closes and earnings events — freezing those on an
ingestion pause would stale scoring with no fallback left to recover from.
- `/ingestion/fetch?sources=fundamentals` still accepts the key and reports
`skipped`; there is no per-ticker fetch any more.
Migration 029 does not blanket-delete the leftover settings rows. Migrations run
before the service restart, and pre-A6 code reads an absent `job_*_enabled` row
as *enabled* — so the two behavior-bearing keys become tombstones pinned to safe
values (hidden in Admin) and only the inert three are deleted. Removing the
provider keys from the production `.env` is the matching rollout step.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The shadow book toggle saved on change and the numeric fields on blur,
inconsistent with every other admin panel (which stages edits behind a
Save button). Stage all four fields in local state and commit them
together on Save, with an unsaved-changes hint. This also makes enabling
the live-trade toggle a deliberate two-step action rather than an
unguarded single click.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The manual paper book only contains trades taken by hand, inside a 20
minute window, on days someone was available. The backtest that validated
this strategy auto-takes the top-ranked qualified setups up to capacity
every session. The forward record was therefore measuring strategy plus
discretion plus availability -- and degrading silently on busy days.
The shadow book closes that gap: it mirrors the backtest's selection rule
(top strategy_rank qualified, up to capacity, 1% fixed-fractional risk)
and shares the manual book's exit policy, so the only difference between
the two books is which setups get taken. Selection ordering reuses the
strategy_rank the scanner already stores rather than recomputing it, so
the two cannot drift apart. It runs as a near-close pipeline step right
after the scan, marking entries at the same prices a human would see.
Gate-reset re-entry state is now scoped per book -- the books diverge as
soon as their entries differ, and each must see only its own stops.
Performance view rewritten around the comparison:
- three series (shadow, manual, SPY) from a new endpoint
- SPY changes from a per-trade cost-basis counterfactual to plain
buy-and-hold %, since one line has to serve two books
- headline stats are R-multiples, not currency: the books size
differently, so only R compares across them
- configurable start date, because the strategy has been revised
repeatedly and pre-cutover trades ran under rules that no longer
exist
Migration 024 also repairs the numeric weekday crons written by 023,
rewriting only rows still holding the broken form so hand-corrected
settings survive. Its literals are inlined because bound parameters
render as NULL under 'alembic upgrade --sql'.
The shadow book is opt-in and writes nothing until enabled. Verify its
first selections match a backtest of that day's cross-section before
trusting any point on the curve.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Regime is computed at 02:00 ET; without a morning alert pass, quadrant
changes waited until 15:30. Dispatcher is change-driven so quiet days stay
quiet. Drop unused alerts_frequency config.
Move the only qualifying R:R scan to 15:30 ET with chained Telegram alerts,
put outcome eval after a final-bar OHLCV fetch, enforce NY trading-day
requalify semantics, stamp paper trades fill_mode=near_close, and migrate
stored schedule_* keys to America/New_York.
Ship greenfield min_rr=2.0 and conf=0, read-only Structural S/R, indicator
cache invalidation, and UI/gate language that treats GTL as screening not exit.
Align strategy_rank missing-vol fallback live vs backtest, single-source
PRIMARY_TARGET_MIN_RR, expand prod parity tests, and drop dead FE clients.
Persist job and ingestion warnings/errors for 7 days, surface a dismissible top-nav badge, treat stale OHLCV as a warning (e.g. ticker renames), and show market bar age on the ticker freshness chip.
Production strategy change based on the July 2026 backtest: paper trades now default to a 30-trading-day hold with the initial stop (classic momentum hold-and-rerank), while target and trailing exits remain available in Admin. The exit policy API/UI now carries hold_days and close_reason can be 'time'.
The activation confidence floor default is now 0/off because the gate ablation showed it added no per-trade edge while filtering out usable setups. Migration 015 clears stored activation_min_confidence and paper_exit_mode so the new defaults take effect; this intentionally resets Track Record comparability from this deploy.
Verification: 451 backend tests pass, ruff check app/ clean, frontend npm run build clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Store an optional company name on Ticker (migration 014) and backfill it from
Alpaca's asset list in a single Trading-API call for the whole universe — no
per-ticker fetch. Runs automatically at the end of universe bootstrap and via a
manual "Backfill Names" button (admin) / POST /admin/tickers/backfill-names.
The name ships on /tickers; a shared symbol→name map (useTickerNames) lets any view
show it without its own request. Displayed subtly next to the symbol — in the global
search, the ticker header, and as a small muted line under the symbol in Top Setups
and Open Trades (no extra column, truncated so it never widens the table).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Applies the backtest-validated trailing stop to live paper trading, and surfaces
it transparently.
Exit (A):
- New paper-trade exit policy (paper_exit_mode=trailing, paper_trailing_pct=12),
tunable in Admin → Paper-Trade Exit. resolve_open_trades runs a trailing stop
(initial stop as floor, ratchets up from the peak; target ignored — the
validated rule) and records close_reason (trailing|stop|target|manual; +migration
013).
- list_trades enriches open trades with the live trailing-stop level + distance %.
Open Trades panel shows the active tactic and a Trail Stop column.
Alerts (B):
- Daily digest now lists open trades with unrealized gain, trailing stop, and how
far away it is.
- New "trade closed" alert: one summary per auto-close (trailing/target/stop, not
manual) — direction, reason, days held, P&L abs+%/R — covering wins AND
stop-loss losses. Deduped by trade id; toggle in Admin alerts.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A NEUTRAL ("No Clear Setup") recommendation means the engine found no clear
directional trade, yet such setups could still qualify and even be crowned the
top pick purely on momentum rank (e.g. an extended momentum leader with a far,
5%-probability target). A NEUTRAL signal isn't actionable, so it shouldn't
qualify.
New `exclude_neutral` activation flag (default on): setup_qualifies drops setups
whose recommended_action is NEUTRAL. It lives in the shared gate, so it flows
through the dashboard's qualified/top-pick selection, the track record's
qualified stats, and the backtest (which computes recommended_action and gates on
meets_core). Toggleable in Admin → Settings → Activation; the frontend mirror and
activationSummary ("directional") match.
Re-run the backtest after enabling to confirm it holds/improves expectancy.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fires once when the regime monitor shifts quadrant (regime index x early
warning), so you don't have to watch the tab. Two guards against spam:
- Hysteresis: each axis only flips once the value crosses its divider by a
margin, so a point parked on a boundary keeps its quadrant instead of
flip-flopping day to day.
- Cooldown: a genuine change stays quiet for a few days after the last alert.
Seeds the baseline silently on first run; reuses the existing Telegram dispatch
+ AlertLog. New per-trigger toggle in Admin → Alerts (on by default).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
min_target_probability is gone: it filtered on the probability model the
calibration has repeatedly shown to be weak and overconfident, it was redundant
with the momentum gate, and as an off-by-default knob it just invited bad tuning.
Removed from the backend gate, activation config/schema, the frontend mirror
(qualifiesSetup / activationSummary), and ActivationSettings. The probability
model stays where it does real work (primary-target selection + display).
Charts: with multi-year history the all-bars default was unreadable. Added
time-range presets (1M / 3M / 6M / YTD / 1Y / 3Y / 5Y / All), defaulting to 1Y;
clicking a preset always re-applies (snaps back after a manual zoom). Y-axis
autoscale and wheel-zoom / drag-pan were already there.
339 backend tests pass; frontend build clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The 5-year backtest confirmed the EV gate adds negative value (high threshold =
worst expectancy) and that 12-1 month momentum is the one price signal with a
plausible, right-signed cross-sectional IC (~0.05). So "qualified" now means:
clears the R:R + confidence floors AND the ticker ranks in the top
`min_momentum_percentile` of the universe by 12-1 momentum that week.
- qualification.py: drop expected_value_r / the EV gate; add a momentum-percentile
gate (duck-typed `momentum_percentile`, only enforced when attached + threshold
set, else defers to floors). Mirrored in frontend qualification.ts.
- activation config/schema: min_expected_value -> min_momentum_percentile
(default 80 = top quintile). ActivationSettings, DashboardPage (ranks/【shows】
momentum instead of EV), and the BacktestPanel sweep follow.
- backtest: rank each ISO week's universe by 12-1 momentum, assign a percentile,
and qualify the top slice; the sweep now sweeps the percentile cutoff.
Also offload the backtest's per-ticker compute to a worker thread so the heavy
~5y run no longer blocks the API event loop (the "backend offline" flicker).
Production setups don't carry momentum_percentile yet — wiring the scanner to
attach it (a universe momentum-rank step) is the next step; until then the live
gate defers to floors while the backtest measures the momentum selection. 330
backend tests pass; frontend build clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Diagnosing "no qualified signals for 5 days": setups were generated but none
qualified. The gate required BOTH a high min_rr (2.0) AND a high
min_target_probability (60), which became contradictory after the Jun-15
probability recalibration — probability already embeds R:R via the 1/(rr+1) ruin
term, so high-R:R targets are inherently low-probability and nothing cleared both.
Gate is now expected value (R): p*rr - (1-p) from the primary target's
probability. R:R and confidence stay as floors; high-conviction / exclude-conflicts
/ min-target-probability become optional tighteners (default off). Defaults:
min_expected_value=0.15, min_rr=1.2, min_confidence=55. EV is only enforced when
computable. Migration 009 clears stored activation_* rows so the new defaults
apply. Backtest sweeps min_expected_value instead of target probability.
Scheduling: pipelines are now cron-configurable in Admin -> Jobs. daily_pipeline
(full, default 0 7 * * *) plus a new light intraday_pipeline (OHLCV + outcome eval,
default hourly US session) that keeps prices/live-R:R current without setup churn.
Fundamentals on its own early weekly cron. Timezone configurable (default
Europe/Berlin). Moving interval->CronTrigger also fixes the restart-deferral bug
where an interval job's countdown resets on every process restart.
319 backend unit tests pass; frontend tsc clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Jobs were independent 24h timers with no ordering, so the scanner could run on
stale OHLCV, and manual runs desynced the offsets. New daily_pipeline job runs
the data→signal flow in dependency order: OHLCV → fundamentals → sentiment →
R:R scan → outcome eval (+paper close) → market regime. Each step keeps its own
enable flag and runtime status; a failing step is logged and the pipeline
continues.
The member jobs are registered PAUSED (no auto-fire) so they only run via the
pipeline — but stay manually triggerable from Admin → Jobs (shown as "runs in
daily pipeline"). Alerts (hourly), ticker universe sync, and backtest keep their
own independent cadence.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Jobs panel only surfaced live progress; once a job finished you couldn't see
when it ran, whether it succeeded, or its message (e.g. a regime/collector error).
Add a "Last run <ago> · <status> — <message>" line per job, colored by status,
from the runtime_* fields the backend already returns.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Closes the action loop — instead of polling the dashboard, the platform pushes
actionable signals to Telegram. New hourly 'alerts' job dispatches four
toggleable triggers, deduped via a new alert_log table (cooldown-based for
qualified/S-R/digest, watermark-based for score deterioration). Admin → Settings
gains a Telegram panel (write-only bot token, chat ID, per-trigger toggles, Send
Test). Credentials follow DB > env precedence (TELEGRAM_BOT_TOKEN / _CHAT_ID).
Backend: alert_service + AlertLog model + migration 005, scheduler job, admin
endpoints/schema. Frontend: AlertSettings panel, hooks, api, types.
Deploy: run alembic upgrade (new alert_log table).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Track Record: new "Reset" action (POST /admin/track-record/reset) deletes all
trade setups so stats start fresh after material scoring/setup changes — live
setups regenerate on the next scan. Guarded by a confirm dialog.
Recommendation config: remove distance_penalty_factor, which was exposed in the
admin UI but consumed nowhere (the touch-probability model superseded it). A
knob that silently does nothing is worse than no knob. Remaining defaults are
left as-is — they're reasonable, and the honest way to tune them is backtesting
against accumulated outcomes, not invented "researched" numbers.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
xAI returned 410 — search_parameters/Live Search is retired. Route xAI
through the Responses API web_search tool instead (same path as OpenAI):
- OpenAISentimentProvider parametrized with base_url / tool_type / source
- xAI builds it against https://api.x.ai/v1 with the web_search tool
- Drop the dead Live Search code from the generic compatible provider
- Frontend label: "xAI Grok — web search"
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Providers (admin-switchable, no redeploy):
- DeepSeek and any OpenAI-compatible endpoint (OpenRouter, Together,
Groq, local Ollama) via a generic Chat Completions adapter + base_url
- xAI Grok with Live Search (search_parameters web+X, citations) —
grounded tier alongside OpenAI and Gemini
- DeepSeek / generic compatible endpoints are ungrounded (no web
search); UI shows an amber warning and labels each provider's grounding
- Optional env fallbacks DEEPSEEK_API_KEY / XAI_API_KEY
UI: replace native <select> (unstyleable white popup on Windows) with a
custom dark Dropdown component everywhere — sentiment provider, scanner
filters, market sort, indicators, admin universe, user role.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Make "qualified" mean an edge candidate, not just R:R + confidence.
The gate now also requires (all admin-configurable, defaults on):
- high conviction: recommended_action LONG_HIGH / SHORT_HIGH only
- clean read: risk_level Low (no contradicting signals)
- probable primary target: best target probability >= min (default 60)
- Shared predicate: app/services/qualification.py +
frontend/src/lib/qualification.ts (mirrored)
- Activation config extended (min_target_probability,
require_high_conviction, exclude_conflicts) with bool-aware
get/update + validation
- /trades/performance switched to ?qualified_only=true, applying
the full gate server-side; confidence breakdown stays unfiltered
- Dashboard "Qualified", Signals "Qualified only" toggle, and
Track Record all use the one gate; Admin gains the new controls
Sentiment provider runtime config (prior change) included.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Admin-configurable thresholds (min R:R, default 2.0; min confidence,
default 70%) defining what counts as an actionable signal:
- Admin Settings: new Activation Thresholds panel
(GET/PUT /admin/settings/activation)
- GET /trades/activation exposes values to all users with access
- Signals/Setups: filters initialize from activation values
- Track Record: "Qualified signals only" toggle (default on) via
min_rr/min_confidence params on /trades/performance; the
confidence breakdown always covers the full population so the
thresholds can be validated against outcomes
- Dashboard: "Qualified" metric and qualified-first Top Setups
- Outcome evaluator unchanged: every setup is still evaluated
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Add shared UI primitives: Button, Field/Input/Select, PageHeader,
Section, Callout, Tabs, Disclosure
- Replace gradient buttons with single blue-accent btn-primary
- Reserve gradient text for the brand wordmark only
- Rework Scanner page onto the glass system; collapse explainer and
glossary into a disclosure, move filters into a glass toolbar
- Restructure Ticker Detail into tabs (Analysis / Indicators / S/R)
with chart and recommendation always visible
- Align Watchlist, Rankings, Admin, Login/Register to shared primitives
- Unify stray indigo/violet/gray accents into the blue family
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
