feat(jobs): persist each job's last run so it survives a restart

Job outcomes lived only in scheduler._job_runtime, an in-memory dict. Every
deploy wiped it, so Admin -> Jobs could report "Active" with no indication a job
had ever run or how it ended -- which is the main thing that page is for.

New job_run_state table (migration 031): one row per job, upserted on job_name.
Deliberately not history -- system_events already grows unbounded with no
retention job, and a second append-only operational table would repeat that
debt. Adding history later is purely additive.

Written from two hooks, NOT from _runtime_finish. That looked cheapest (one
function, ~40 call sites) but unit tests invoke job coroutines directly, so it
would fire detached DB writes at the real session factory throughout the suite,
and there is no testing flag to guard on.

  - An APScheduler EVENT_JOB_EXECUTED/ERROR listener covers everything the
    scheduler fires, including manual triggers. Its detached task is held in a
    module-level set (a bare create_task result can be collected mid-flight) and
    drained in the app lifespan before engine.dispose().
  - _run_pipeline persists directly, and must: pipeline steps are plain
    coroutine calls that emit no scheduler events, so the listener cannot see
    them. The step persist sits AFTER the except that swallows step errors --
    inside it, exactly the failed runs worth seeing would be skipped. The
    orchestrator persists in the finally, and the disabled early-return persists
    too, or "skipped" is silently dropped.

_persist_job_run never raises: a persistence failure must not break an otherwise
successful pipeline.

The API reports this as last_run_* and leaves runtime_* meaning strictly live
in-memory state. Reusing runtime_status would have been a regression, not a
no-op: JobControls drives the status chip from it (a job that errored eight days
ago would read "Last run error" forever instead of "Active") and picks the
rate-limit banner from it (a week-old rate limit would pin the banner
permanently). Tests pin the split.

The table starts empty; each job fills its row the next time it finishes. No
backfill from system_events, which records only warning/error outcomes under a
different status vocabulary and would invent successes that never happened.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

app/services/admin_service.py

@@ -18,7 +18,7 @@ from app.models.settings import SystemSetting
 from app.models.ticker import Ticker
 from app.models.trade_setup import TradeSetup
 from app.models.user import User
-from app.services import settings_store
+from app.services import job_run_store, settings_store
 
 logger = logging.getLogger(__name__)
 
@@ -689,11 +689,13 @@ async def list_jobs(db: AsyncSession) -> list[dict]:
         name: flags.get(f"job_{name}_enabled", "true") == "true"
         for name in VALID_JOB_NAMES
     }
+    last_runs = await job_run_store.get_map(db, visible)
 
     jobs_out = []
     for name in visible:
         job = scheduler.get_job(name)
         runtime = get_job_runtime_snapshot(name)
+        last = last_runs.get(name)
 
         jobs_out.append({
             "name": name,
@@ -717,6 +719,14 @@ async def list_jobs(db: AsyncSession) -> list[dict]:
             "runtime_started_at": runtime.get("started_at"),
             "runtime_finished_at": runtime.get("finished_at"),
             "runtime_message": runtime.get("message"),
+            # Survives restarts, unlike runtime_*. Reported separately so the
+            # status chip keeps meaning "state now" rather than "last outcome,
+            # forever" -- an error a week ago must not read as Inactive today.
+            "last_run_at": last.finished_at.isoformat() if last else None,
+            "last_run_status": last.status if last else None,
+            "last_run_message": last.message if last else None,
+            "last_run_processed": last.processed if last else None,
+            "last_run_total": last.total if last else None,
             **_next_run_fields(scheduler, name, enabled_map),
         })
 

app/services/job_run_store.py

@@ -0,0 +1,70 @@
+"""Single source for JobRunState reads/writes.
+
+Mirrors ``settings_store``: ``record_finish`` never commits — the caller owns
+the transaction — and reads are batched so the admin listing stays one query.
+"""
+
+from __future__ import annotations
+
+import logging
+from collections.abc import Iterable
+from datetime import datetime, timezone
+
+from sqlalchemy import select
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.models.job_run_state import JobRunState
+
+logger = logging.getLogger(__name__)
+
+
+def _as_datetime(value: object) -> datetime | None:
+    """Runtime snapshots carry ISO strings; the column wants a datetime."""
+    if isinstance(value, datetime):
+        return value
+    if isinstance(value, str) and value:
+        try:
+            return datetime.fromisoformat(value)
+        except ValueError:
+            return None
+    return None
+
+
+async def get_map(db: AsyncSession, job_names: Iterable[str]) -> dict[str, JobRunState]:
+    """Return {job_name: row} for the given jobs that have ever finished."""
+    result = await db.execute(
+        select(JobRunState).where(JobRunState.job_name.in_(list(job_names)))
+    )
+    return {row.job_name: row for row in result.scalars().all()}
+
+
+async def record_finish(db: AsyncSession, job_name: str, runtime: dict) -> JobRunState:
+    """Upsert the last-run row from a scheduler runtime snapshot.
+
+    Select-then-update-or-insert rather than a dialect-specific upsert, matching
+    ``settings_store.upsert_setting`` — tests run on SQLite, prod on Postgres.
+    """
+    existing = await db.execute(
+        select(JobRunState).where(JobRunState.job_name == job_name)
+    )
+    row = existing.scalar_one_or_none()
+
+    finished_at = _as_datetime(runtime.get("finished_at")) or datetime.now(timezone.utc)
+    status = str(runtime.get("status") or "completed")
+    message = runtime.get("message")
+    values = {
+        "status": status,
+        "started_at": _as_datetime(runtime.get("started_at")),
+        "finished_at": finished_at,
+        "processed": runtime.get("processed"),
+        "total": runtime.get("total"),
+        "message": str(message)[:4000] if message else None,
+    }
+
+    if row is None:
+        row = JobRunState(job_name=job_name, **values)
+        db.add(row)
+    else:
+        for field, value in values.items():
+            setattr(row, field, value)
+    return row