fix(jobs): make last-run writes atomic and drain them properly on shutdown

Two review findings on 22bee28, both reproduced before fixing.

[1] Concurrent writes could lose or rewind a row. record_finish did
select-then-insert-or-update, and pipelines are separate scheduler jobs that can
overlap while sharing step ids -- data_collector belongs to all four. Reproduced
both halves: two sessions that SELECT before either INSERTs make the second
commit raise IntegrityError, which _persist_job_run swallows, so the run
silently vanishes; and a later write carrying an OLDER finished_at rewound the
row from 12:00 back to 09:00, dragging the status with it, so the panel would
report a stale outcome as the latest.

Now a single atomic INSERT ... ON CONFLICT (job_name) DO UPDATE, guarded by
WHERE job_run_state.finished_at < excluded.finished_at so an older completion
can never overwrite a newer one. Dialect-specific because prod is Postgres and
tests are SQLite; both support it (SQLite >= 3.24, ours is 3.45). updated_at is
set explicitly, as the model's onupdate hook does not fire for a core upsert,
and get_map now uses populate_existing since core writes leave any
previously-loaded ORM instance stale in the identity map.

[2] Shutdown could dispose the engine underneath a pending write.
scheduler.shutdown(wait=False) returns before APScheduler dispatches its
completion events, and those events are what create persist tasks -- so a single
snapshot of the task set missed writes still to be queued. flush_job_run_persists
now settles briefly for pending callbacks, then drains in a loop until the set
stays empty, with the deadline still bounding total shutdown time. Left
shutdown(wait=False) alone deliberately: waiting would block a deploy restart
behind a long-running scan.

Six regression tests: interleaved first writes, older-never-rewinds,
newer-still-wins, a task queued mid-drain, prompt return when idle, and giving
up rather than hanging shutdown.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

app/scheduler.py

@@ -353,11 +353,34 @@ def _schedule_persist(job_name: str) -> None:
     task.add_done_callback(_persist_tasks.discard)
 
 
-async def flush_job_run_persists(timeout: float = 5.0) -> None:
-    """Await in-flight last-run writes. Called from the app's shutdown path."""
-    if not _persist_tasks:
-        return
-    await asyncio.wait(set(_persist_tasks), timeout=timeout)
+async def flush_job_run_persists(timeout: float = 5.0, settle: float = 0.05) -> None:
+    """Drain last-run writes, including ones queued while we are draining.
+
+    ``scheduler.shutdown(wait=False)`` returns before APScheduler has dispatched
+    its job-completion events, and those events are what create persist tasks. A
+    single snapshot of the set therefore misses writes still to be queued, and
+    ``engine.dispose()`` could then close the pool underneath them. So: give the
+    loop a moment for pending callbacks to land, then keep draining until the
+    set stays empty or the deadline passes.
+    """
+    loop = asyncio.get_running_loop()
+    deadline = loop.time() + timeout
+    # Bounded settle so callbacks dispatched by shutdown get to queue their work
+    # before the first emptiness check decides there is nothing to wait for.
+    await asyncio.sleep(min(settle, timeout))
+    while True:
+        pending = {task for task in _persist_tasks if not task.done()}
+        if not pending:
+            return
+        remaining = deadline - loop.time()
+        if remaining <= 0:
+            logger.warning(
+                "Timed out draining %d last-run write(s); some may be lost", len(pending)
+            )
+            return
+        await asyncio.wait(pending, timeout=remaining)
+        # Loop rather than return: a completion callback may have queued another.
+        await asyncio.sleep(0)
 
 
 def get_job_runtime_snapshot(job_name: str | None = None) -> dict[str, dict[str, object]] | dict[str, object]: