Commit Graph
207 Commits
Author SHA1 Message Date
dennisthiessenandClaude Opus 4.8 2038b84b72 feat(fundamentals): A4 — pure peer comparison + deterministic reads
Completes the pure read-time core.

fundamentals_peers.py: median + polarity-aware favorable percentile + peer_count
for a subject within its SIC group (CIK-deduped by the caller); returns None
below MIN_PEERS=5 so the caller omits the industry object. Absolute net_debt is
intentionally NOT peer-eligible (size-dependent) — leverage compares via
net_debt_to_ebitda. HIGHER_IS_BETTER polarity map + two_digit_sic() grouping key.

fundamentals_reads.py: one shared deterministic rule set (no LLM): growth_read
(+-2pp), margin_read (latest vs mean-of-prior, +-1pp), share_count_read (+-1%),
peer_read (60/40 bands, polarity-aware phrasing per metric), header_sentence
(growth · margins · valuation, omitting empty). Tunable named constants; >=3
periods required for a series read.

Tests: 8 peer + 5 reads, anchored on the boundary cases (exactly +2.0pp, exactly
60th percentile, exactly +1.0pp margin). 13 passed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-22 20:25:48 +02:00
dennisthiessenandClaude Opus 4.8 256880899b fix(fundamentals): A4a review — stricter null semantics in derivation
1. net_debt requires BOTH cash and total_debt; a missing side is null, not
   treated as zero (which would be a partial, misleading value).
2. net_debt_to_ebitda is null when TTM EBITDA <= 0 — a negative denominator
   would otherwise rank a distressed issuer as favorably low-leverage.
3. The quarter tape is the CONSECUTIVE run ending at the latest period (stops at
   a gap), so trend text never compares non-adjacent quarters as if consecutive.
4. YoY growth is null when the prior-year TTM is <= 0 (e.g. loss->profit), which
   is not a meaningful percentage.

Also corrected the plan's net-debt formula to total debt − (cash + ST) matching
the positive-means-net-debt implementation. +4 tests. 10 passed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-22 20:20:29 +02:00
dennisthiessenandClaude Opus 4.8 a549942afe feat(fundamentals): A4a — pure read-time metric derivation
Derives the display metrics from the stored YTD snapshots at read time (no I/O,
no DB), per the A3 schema decision. Given an issuer's snapshot rows it produces:

- amendment selection (newest accepted_at per fiscal period);
- discrete quarters = YTD(Qn) - YTD(Qn-1), Q4 = YTD(FY) - YTD(Q3);
- TTM = trailing four discrete quarters; missing period -> null, never partial;
- metric series (value + 4-quarter tape, each point dated): revenue_growth_yoy,
  eps_growth_yoy, operating_margin, fcf_margin, net_debt, net_debt_to_ebitda,
  share_count_change_yoy;
- request-time valuation inputs (ttm_diluted_eps, ttm_fcf, shares_outstanding)
  for the API to combine with price.

Units per app convention (percentages = pp, leverage = multiple, dollars).
Tests: 6 (growth+Q4, margins, net-debt/EBITDA+dilution, valuation inputs,
missing-period-null, amendment selection). Verified on real Apple snapshots:
op margin 32.6%, net-debt/EBITDA 0.10, buyback -1.7%/yr, TTM EPS $8.26.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-22 20:10:52 +02:00
dennisthiessenandClaude Opus 4.8 96d3b40560 fix(sec): A3 sign-off hardening — validate per-concept units structure
- Extend the companyfacts structural check to reject a concept with a
  missing/non-dict `units` mapping (not just the top-level `facts`), so a
  partially-malformed payload fails promotion instead of silently dropping that
  concept's facts. New fixture proves it fails.
- Strengthen the newly-added-issuer test: keep latest_index equal to the prior
  run so ONLY the universe fingerprint changes the revision — proving the
  fingerprint alone prevents a new ticker from being starved/no_op'd.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-22 20:04:11 +02:00
dennisthiessenandClaude Opus 4.8 8dcdcac2a6 fix(sec): A3 slice-2b review — no index cap, full discrepancy + malformed gate
1. Removed the 45-day index-walk cap: it discarded the older part of a long
   outage while still advancing source_max_date, permanently losing filings.
   The walk now covers every unprocessed date (a large gap is one-time cost).
2. Discrepancy detection meets the immutability contract: it compares ALL source
   snapshot fields (not five), read-only during stage/validate, reports the
   differing accessions + fields in validation_json, and promote emits a warning
   system event (in-transaction) — never mutating the stored row.
3. Malformed companyfacts (missing facts/units structure) are recorded separately
   and FAIL validation, instead of silently degrading to skipped rows that the
   50% backfill coverage floor could still pass.

Also corrected the stale "sum share classes" / DEI-only wording in the snapshot
model docstring and the A3 design doc to describe the us-gaap fallback.

Tests: +4 regressions (>45-day gap loses nothing, newly-added issuer backfills
without filing, malformed payload fails, shares discrepancy detected + evented).
23 passed, 1 skipped.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-22 19:55:58 +02:00
dennisthiessenandClaude Opus 4.8 f7ce85a33e feat(sec): A3 slice 2b — SEC fundamentals importer (shadow ingestion)
SecFundamentalsImporter (SourceImporter, source=sec_facts): populates immutable
fundamental_snapshots from Company Facts and back-fills tickers.cik/sic, driven
by the EDGAR daily index. Shadow only. Guardrails per review:

- detect_revision caches the resolved universe + exact tracked index rows and
  composes the revision from them; stage consumes those same cached inputs
  (no index/universe refetch) so promoted data matches the computed revision.
- Resolution is read-only in stage (proposals only); ticker writes happen in
  promote via apply_ticker_updates.
- validate runs the index<->Company-Facts consistency gate before any write:
  a tracked XBRL index accession missing from Company Facts fails the run
  (they lag independently) so we retry, not record null. Non-XBRL amendments
  are skipped with a recorded reason. Backfill has a coverage floor.
- promote inserts ON CONFLICT (accession) DO NOTHING (immutable), reports
  differing existing accessions without mutating, and applies ticker updates in
  the same transaction.
- Full-history backfill on first run / for newly-added issuers (include_history);
  incremental fetch only for issuers that filed.

Parser: split parse result into skipped_filings vs field_issues (coverage must
not count field warnings); header notes the us-gaap shares fallback; added
companyfacts_accessions() for the gate.

Verified live end-to-end (AAPL + GOOGL backfill): 112 snapshots, cik/sic set,
GOOGL shares via us-gaap fallback, AAPL via dei. Tests: 6 importer (backfill,
incremental, consistency-gate fail, non-XBRL skip, read-only-on-failure,
conflict-discrepancy) + parser ParseResult updates.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-22 19:39:46 +02:00
dennisthiessenandClaude Opus 4.8 4754dbc17b fix(sec): A3 slice-2a review — shares fallback, robust context, hardening
1. Multi-class shares: prefer the single dei:EntityCommonStockSharesOutstanding
   cover-page fact; else fall back to us-gaap:CommonStockSharesOutstanding at
   period end (Alphabet has no dei fact). Never sum class facts (companyfacts is
   non-dimensional) and never use weighted-average/diluted; conflicting values ->
   null, counted as an "ambiguous shares outstanding" note in validation. Plan's
   "sum class-specific" wording corrected. Verified live: Alphabet shares now
   populate (12.1B), Apple still uses its dei cover date.
2. Fiscal context is the majority (fy, fp) among facts ending at reportDate, with
   ties rejected — no longer the arbitrary first fact.
3. Hardening: catalog selectors require taxonomy == "us-gaap"; indexing drops
   malformed facts (missing accession/end, non-finite value) so a custom concept
   or bad date can't be selected.

Tests: +8 (dei precedence, us-gaap fallback, conflict->null, no weighted-average,
tie-context skip, foreign-taxonomy/malformed ignored, ambiguous-shares note).
14 passed, 1 skipped.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-22 16:40:05 +02:00
dennisthiessenandClaude Opus 4.8 7413de9301 feat(sec): A3 slice 2a — companyfacts -> snapshot parser
Pure parser (no I/O/DB) turning one issuer's companyfacts + submissions filing
metadata into per-accession snapshot rows for the filing's primary period.

- Period identity from end == reportDate, never fy/fp (fy/fp is the filing's
  context; comparatives repeat it).
- Duration facts stored as cumulative YTD: pick the fact whose span matches the
  fiscal-period-to-date length (Q1~3mo..FY~12mo) within tolerance; no YTD-length
  fact -> null (never a discrete masquerading as YTD).
- Balance-sheet instants at end == reportDate; shares_outstanding is the dei
  cover-page fact whose own end (cover date) is stored in shares_outstanding_date.
- Cash and debt composites are aggregate-first and mutually exclusive (each
  source tag counted at most once).
- Carries filing_date through submissions rows (snapshot.filed_date).

Verified on REAL Apple companyfacts: 44 snapshots, 0 skipped, YTD revenue
124.3B->219.7B->313.7B->416.2B across FY2025 (Q4 derives at read time), every
shares_date is the cover date != period_end. Tests: 6 fixture + 1 skip-guarded
live-invariants (monotonic YTD, cover-date shares).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-22 16:11:22 +02:00
dennisthiessenandClaude Opus 4.8 5939be7b7f fix(sec): A3 slice-1 review — read-only resolution, error propagation, fair-access
Addresses the slice-1 review:

1. Resolution is now read-only (A1 transaction contract). resolve_ciks /
   fetch_sic_updates compute proposals and mutate nothing; a new
   apply_ticker_updates issues the writes, called only in promote — so a failed
   validation can't leak ticker changes on the framework's failure commit.
2. Only 404 means "missing". Added SecNotFoundError; daily_index /
   latest_index_date catch only that. 403, exhausted 429, 5xx, timeouts, and
   transport/parse errors now propagate instead of looking like "no index".
3. Fair-access enforced when opening a REAL client (transport=None): reject
   blank/placeholder/non-email User-Agent and sub-0.11s spacing. Mock transports
   skip it (tests use 0 spacing).
4. submissions(include_history=False) by default — only the one-time full
   backfill fetches the history shards; SIC/incremental work makes no extra
   requests.

Plus: retry transient 5xx/network errors and honor Retry-After during the 1 GB
backfill; compose_revision rejects a missing index date (no "None:..." revision).

Re-verified live vs real SEC (fair-access validation passes, shard merge intact).
Tests: 18 (added error propagation, read-only resolution, fair-access, recent-only
submissions, reject-None revision).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-22 14:49:33 +02:00
dennisthiessenandClaude Opus 4.8 cc67aebe61 feat(sec): A3 slice 1 — SEC client + CIK/SIC resolution + composite revision
First A3 implementation checkpoint (design: docs/dolt-sec-a3-design.md).

- sec_client.py: async SEC EDGAR client honoring fair-access — identifying
  User-Agent (config), request spacing < 10 req/s, exponential backoff on 429,
  and 403 -> SecForbiddenError (alert and stop, never retry-loop). Fetchers:
  company_tickers (normalised, multi-class share CIK), submissions (merges the
  paginated filings.files shards so full history is visible), companyfacts,
  daily_index (fixed-width form.idx parse), latest_index_date.
- sec_universe.py: resolve_ciks (tickers.cik backfill), refresh_sic
  (sic/sic_description), and the composite-revision pieces — universe_fingerprint
  (a new ticker changes the revision, so it's never no_op'd/starved),
  index_content_hash, compose_revision.
- config + .env.example: SEC_USER_AGENT (must be a real contact email) + spacing
  / retries / timeout.

Verified live against real SEC: AAPL->320193, GOOG==GOOGL, BRK-B resolved;
submissions shard-merge proven (131 filings back to 1993); daily index parsed.
Tests: 11 (mocked-transport parsing + 403/429 handling + resolution/fingerprint).
Full suite 713 passed. Next slice: companyfacts -> snapshot parser + importer.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-22 14:15:56 +02:00
dennisthiessenandClaude Opus 4.8 5d275c8df7 fix(dolt): A2 review — subprocess timeouts, stronger initial gate, HASHOF
Addresses the A2 review:

1. Every dolt subprocess is now bounded by a hard timeout
   (dolt_command_timeout_seconds, default 600s); on expiry the process is killed
   and DoltError raised — a hung pull/sql can no longer pin the import
   connection and advisory lock indefinitely. Tested (timeout + non-zero exit).
2. Initial-load validate is stronger: besides zero-future, an initial load now
   requires a real forward horizon (>= 21d, under the ~35d observed on the
   clone) AND universe coverage >= 50% (a broken symbol join can't seed a hollow
   calendar). Subsequent runs keep the 50% collapse gate.
3. Revision uses DOLT_HASHOF('HEAD') — formally HEAD, not dolt_log-by-timestamp.
4. Free-disk floor raised 2 GB -> 5 GB (safe headroom over the ~1.7 GB clone).

Full suite 702 passed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-22 11:50:50 +02:00
dennisthiessenandClaude Opus 4.8 54ae8ba153 feat(dolt): A2 — DoltHub earnings importer (shadow ingestion)
A SourceImporter that ingests post-no-preference/earnings into earnings_events
for the tracked universe. Shadow by construction (nothing reads earnings_events
until A4).

- earnings_alignment.py: pure calendar<->EPS-history min-cost monotonic DP,
  reused from scripts/import_dolthub_earnings.py with identical constants (not
  extending that one-off script); symbol/session normalization; unit-tested
  against the pinned constants.
- dolt_client.py: async dolt CLI wrapper (pull / current_commit / query_csv via
  asyncio.create_subprocess_exec — never blocks the shared event loop) + disk
  guard before pull.
- dolt_earnings_importer.py: detect_revision = pull + HEAD hash; stage = query
  earnings_calendar + eps_history, dedup, align, map act_symbol->ticker_id
  (normalize both sides so dotted BRK.B joins); promote is destructive
  (delete future dolt_earnings rows + upsert; past never deleted) so validate is
  FAIL-CLOSED — blocks when the staged forward calendar is empty or has collapsed
  below 50% of what's loaded (the forward calendar is the acceptance gate).
- NOTICE: CC BY-SA 4.0 attribution; config: DOLT_BINARY / DOLT_DATA_DIR / etc.

Verified end-to-end against the real 1.68 GB clone (5 tickers: 133 events, 128
paired, forward calendar to 2026-08-26, BRK.B joined). Tests: 9 alignment + 7
importer + 1 skip-guarded real-clone smoke. Full suite 699 passed.

Remaining for A2: wire the daily ~02:30 ET shadow cron — deferred to pair with
the deploy-time dolt install + DOLT_DATA_DIR provisioning.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-22 10:55:27 +02:00
dennisthiessenandClaude Opus 4.8 e5a62ca648 refactor(dolt): pass run_id to SourceImporter.promote
Both real target tables (fundamental_snapshots, earnings_events) carry an
import_run_id; stamping requires the current run's id. A2 (the earnings
importer) is the first real consumer, so promote gains a run_id argument rather
than having importers hack the running row out of the framework. Protocol +
call site updated; the A1 fake importer now stamps and asserts import_run_id.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-22 10:55:11 +02:00
dennisthiessenandClaude Opus 4.8 8b45bdb361 fix(dolt): record/alert revision-detection failures + handle cancellation
Review fixes to the import-run framework (A1):

1. detect_revision ran outside the failure handler, so a failed revision probe
   (the most likely external failure) escaped unrecorded — violating "every
   attempt is recorded". Now the running row is created FIRST, then
   detect_revision + last-revision lookup + stage + validate + promote all run
   inside the same handler; the row converts to no_op when the revision is
   unchanged. New test covers a detection exception → recorded failed + alert.

2. asyncio.CancelledError (BaseException, not caught by except Exception) left a
   permanent running row on deploy/scheduler shutdown. Now caught explicitly:
   best-effort mark failed, then re-raise the cancellation (never swallowed).
   New test asserts the run is failed and the error re-propagates.

Full suite 682 passed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-22 09:25:14 +02:00
dennisthiessenandClaude Opus 4.8 febd741671 feat(dolt): source-agnostic import-run framework (A1)
run_import + SourceImporter Protocol (detect_revision/stage/validate/promote)
giving every bulk importer the plan's non-negotiables, KISS:

- one run per source at a time — Postgres session-level advisory lock held on a
  single pinned engine.connect() so it survives the running-row and promotion
  commits; no-op on SQLite.
- idempotent per revision — cheap detect_revision compared to the last promoted
  run; unchanged revision records a no_op with zero writes (no fetch).
- staging (in-memory, no physical staging tables) → validate (read-only) →
  atomic promote + run-row flip in one transaction.
- failed validation or mid-run exception marks the run failed, alerts via
  system_event_service, and leaves live tables untouched.

Every attempt recorded in data_import_runs; conflicts summary in validation_json
(no conflicts table). Concrete SEC/earnings importers land in later phases.

Tests: 6 orchestration tests (no_op / promote / new-revision / failed-untouched
/ promote-exception-rollback) + deterministic advisory-key derivation. Full
suite 680 passed. Advisory-lock mutual exclusion is PG-verify-pending (SQLite
no-ops it — flagged, not covered).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-22 00:18:27 +02:00
dennisthiessenandClaude Fable 5 565484de87 fix: select shadow book setups by scan run id, not a time window
Deploy / lint (push) Successful in 9s
Deploy / test (push) Successful in 1m21s
Deploy / deploy (push) Successful in 39s
The run-id marker proved which scan wrote last, but the shadow book still
selected setups by detected_at >= scan_start. An overlapping manual scan
could insert rows in that same window; if the pipeline's scan wrote the
marker last its id matched and the shadow book proceeded, then swept in --
or ranked highest -- a manual-scan row. The identity check gated entry but
selection did not.

Carry the run id onto the rows. Migration 025 adds an indexed
trade_setups.scan_run_id. scan_all_tickers computes one id per run
(pipeline's when a step, else fresh), passes it to scan_ticker which stamps
every row after enhancement, and writes the same id to the completion
marker. The shadow book selects WHERE scan_run_id == the matched id, so a
concurrent scan's rows are excluded by identity regardless of their
detected_at. The now-unused STARTED marker is dropped; COMPLETED
(freshness) and RUN_ID (identity) remain.

Decisive test: the pipeline's id matches, but a same-window manual row with
a higher rank is present and is excluded -- only the pipeline's own row is
traded. A time-window select would have swept it in and ranked it first.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-21 11:25:03 +02:00
dennisthiessenandClaude Fable 5 05ba138d35 fix: match shadow book to its pipeline's scan by run id, not timestamp
A manually triggered rr_scanner and the scheduled near-close pipeline are
separate APScheduler jobs; max_instances=1 serialises a job only against
itself, so they can overlap. A manual scan starting just before the
pipeline can finish just after it began and overwrite the scan markers.
Its completion timestamp is then later than the pipeline start, so the
previous 'completed >= pipeline_start' check accepted its batch as though
it were the pipeline's own -- exactly when the pipeline's scan may have
failed.

Replace the timestamp comparison with an exact run-id match. A new
pipeline_run module holds a per-task run-id contextvar (separate module so
the scanner and scheduler import it without a cycle). _run_pipeline binds a
fresh id per invocation; scan_all_tickers stamps that id -- or a fresh one
when run standalone -- into the scan markers, written with started/completed
in a single commit. The shadow step requires the stored run id to equal its
pipeline's id exactly, so a concurrent manual scan (its own id) or a failed
pipeline scan (a prior run's id) can never be mistaken for it. Direct Admin
triggers have no pipeline context and keep the freshness fallback.

Known residual: the id match governs whether shadow proceeds; setup
selection remains detected_at >= scan start, so a fully per-run setup
isolation would need a run_id column on trade_setups (not required here).

Tests cover the reported race (manual scan finishing last is refused), a
failed pipeline scan, the id-match accept path, and contextvar propagation
and non-leakage across tasks.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-21 10:47:32 +02:00
dennisthiessenandClaude Fable 5 807cc4bdfa fix: bind shadow book to its own pipeline's scan, not wall-clock freshness
A 6-hour freshness window proves only that some scan ran recently, which a
manual mid-day scan satisfies. Scenario: a manual scan succeeds at 13:00;
the 15:30 near-close pipeline's scan step is disabled or fails; at 15:30
the 13:00 completion is still 'fresh', so the shadow step trades that
earlier batch despite no successful scan in the current pipeline.

_run_pipeline now records its start in a per-task contextvar, visible to
the steps it awaits. run_shadow_book reads it and requires the scan
completion marker to be at/after the pipeline start, so a scan that failed
or was disabled in this pass (marker left at a prior run, before the
pipeline began) cannot be substituted by an earlier manual scan. A direct
Admin trigger has no pipeline context and falls back to the freshness
window -- an explicit operator action, not an automated one.

Tests pin the reported case: a fresh manual scan predating the pipeline
start is refused; the pipeline's own post-start scan is accepted.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-21 10:03:44 +02:00
dennisthiessenandClaude Fable 5 6a10c8ff09 fix: guarantee shadow scan freshness, long-only, user-scoped setup list
Second review round on the shadow book; all three findings were real.

- Scan freshness is now proven, not assumed. Pipeline steps run and fail
  independently, so a disabled or failed scan step still let the shadow
  step run on the newest *stored* setups -- a prior session's picks at
  stale prices. scan_all_tickers now records a run boundary
  (last_scan_run_started_at / _completed_at) only on successful
  completion; the shadow book refuses to trade unless COMPLETED is fresh
  and selects only setups with detected_at >= the run start. Deduplication
  to the latest row per ticker now happens BEFORE qualification, so a newer
  unqualified row suppresses an older qualified one rather than the reverse.

- Shadow selection is hard long-only. setup_qualifies only enforces
  long-only when min_momentum_percentile > 0, but 0 is a legal admin
  setting, and the cash accounting assumes long positions -- so the
  constraint is enforced in shadow selection regardless of gate config.

- The personal setup list excludes only the caller's own open positions.
  get_trade_setups gained exclude_open_trade_user_id; the trades route
  passes the authenticated user, while the Telegram broadcast stays global
  since it has no single owner.

New tests cover stale/absent scan markers, prior-run exclusion, newer
unqualified suppressing older qualified, long-only under a disabled gate,
and both sides of the user-scoped exclusion.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-21 09:31:24 +02:00
dennisthiessenandClaude Fable 5 247a92a89f fix: harden shadow book against book leakage (review of ba2df8b)
Review of the shadow book found seven ways the two books could leak into
each other; all are fixed here. The most serious silently invalidated the
comparison the shadow book exists to make.

- Shadow holdings no longer suppress the manual candidate list. The
  open-trade exclusion filtered on any book, so shadow taking the
  top-ranked names removed exactly those from the user's list and alerts,
  confining the discretionary book to leftovers. Scoped to the manual
  book. Closed-trade alerts and paper-book equity were leaking the same
  way and are likewise scoped.

- Shadow sizing now matches _simulate_portfolio: min(1% risk, 20% notional
  cap, available cash) from marked equity, plus the sub- dust guard.
  Previously risk-only from realized equity, so a tight stop produced a
  multiples-of-equity leveraged position the strategy would never take.

- Shadow only trades setups from the scan that just ran (<6h old) with one
  setup per ticker. A failed or disabled scan step could otherwise open
  positions from a prior session at stale prices.

- Gate-reset transitions are observed for both books, so a shadow stop-out
  completes fail -> requalify instead of staying locked forever.

- Manual list/close endpoints default to the manual book and reject
  hand-closing shadow trades; the performance endpoint is scoped to the
  caller so 'your picks' is not every user's book.

- run_shadow_book is registered as a paused job so Admin can trigger it.

Also anchors three pre-existing paper-trade tests (and the new alpaca
window test) on the UTC date. They build fixtures from the local date but
the service stamps opened_at in UTC, so they failed only between 00:00 and
02:00 in a UTC+hh timezone -- latent on ba2df8b, exposed by the clock.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-21 09:11:14 +02:00
dennisthiessenandClaude Fable 5 ba2df8b9fd feat: shadow book + shadow-vs-manual performance comparison
The manual paper book only contains trades taken by hand, inside a 20
minute window, on days someone was available. The backtest that validated
this strategy auto-takes the top-ranked qualified setups up to capacity
every session. The forward record was therefore measuring strategy plus
discretion plus availability -- and degrading silently on busy days.

The shadow book closes that gap: it mirrors the backtest's selection rule
(top strategy_rank qualified, up to capacity, 1% fixed-fractional risk)
and shares the manual book's exit policy, so the only difference between
the two books is which setups get taken. Selection ordering reuses the
strategy_rank the scanner already stores rather than recomputing it, so
the two cannot drift apart. It runs as a near-close pipeline step right
after the scan, marking entries at the same prices a human would see.

Gate-reset re-entry state is now scoped per book -- the books diverge as
soon as their entries differ, and each must see only its own stops.

Performance view rewritten around the comparison:
  - three series (shadow, manual, SPY) from a new endpoint
  - SPY changes from a per-trade cost-basis counterfactual to plain
    buy-and-hold %, since one line has to serve two books
  - headline stats are R-multiples, not currency: the books size
    differently, so only R compares across them
  - configurable start date, because the strategy has been revised
    repeatedly and pre-cutover trades ran under rules that no longer
    exist

Migration 024 also repairs the numeric weekday crons written by 023,
rewriting only rows still holding the broken form so hand-corrected
settings survive. Its literals are inlined because bound parameters
render as NULL under 'alembic upgrade --sql'.

The shadow book is opt-in and writes nothing until enabled. Verify its
first selections match a backtest of that day's cross-section before
trusting any point on the curve.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 23:44:41 +02:00
dennisthiessen bb8aa655a1 research: clean up closed Tier-1 scaffolding from branch
Drop intermediate history-depth reports, sector-residual runners/map/code hooks
(evidence stays in final reports + docs), and slim MacBook helper to ssl/earnings/
prod-book-matrix only. SSL bootstrap and archived research conclusions retained.
2026-07-19 14:41:52 +02:00
dennisthiessen fa25b6ee68 research: sector residual, earnings gap/SUE, history-depth scaffolding
Tier-1 alpha research (local only, no production deploy):

Sector residual momentum: two-factor SPY+sector residual and sector demean signals, IC harness + A/B. Sector resid clears pre-registered bars narrowly (PROMOTE for human wire design only). Sector demean fails t vs market resid.

Earnings: earnings_events backfill (FMP bulk paid; FMP/AV per-symbol), 2a gap diagnostic report-only, 2b SUE IC (PARK; incomplete 48/506 coverage).

History-depth: pre-registered doc + runner for MacBook deep rebuild/harness.

Do not ship production residual or filters from this branch.
2026-07-19 09:33:34 +02:00
dennisthiessen 7d60e54f5a research: single-source liquid mask; orphan +0.06 fip IC
Harness and diagnostics share _filter_liquid_breadth_week_rich. Recompute
shows unconditional liquid fip IC -0.017 (mask binds 97%); mom-conditional
-0.088/t-4.58 stands. Document +0.0575 as orphaned.
2026-07-19 00:06:04 +02:00
dennisthiessen 9704e0d85a feat: Phase B fip_id liquid-breadth research tooling
Deploy / lint (push) Successful in 9s
Deploy / deploy (push) Canceled after 0s
Deploy / test (push) Canceled after 1m5s
Add research-only snapshot extender, PIT dollar-volume mask for signal IC,
rank-only harness path, fingerprint+breadth runner, and docs. Fingerprint
reproduced IC -0.045 / t -2.91 on prod.sqlite. No production gate/schedule changes.
2026-07-18 20:22:11 +02:00
dennisthiessen dc08a805a8 fix: recalibrate FIP path labels to live equity scale
Deploy / lint (push) Successful in 9s
Deploy / test (push) Canceled after 0s
Deploy / deploy (push) Canceled after 0s
Replace inert ±0.25 bands with ~p25/p75 cutoffs from the prod snapshot
(−0.08 / 0.00). Document zero-return dilution and left-skewed distribution.
2026-07-18 19:27:45 +02:00
dennisthiessen 19d674ed62 feat: show FIP path-smoothness in ticker technicals
Deploy / lint (push) Successful in 9s
Deploy / test (push) Successful in 1m20s
Deploy / deploy (push) Successful in 41s
Display-only Da/Gurun/Warachka information discreteness on the ticker
indicator panel. Shared compute with the backtest harness; not wired into
gate or rank.
2026-07-18 19:22:02 +02:00
dennisthiessen 736451e26f feat: near-close scan schedule and distinct-day gate reset
Move the only qualifying R:R scan to 15:30 ET with chained Telegram alerts,
put outcome eval after a final-bar OHLCV fetch, enforce NY trading-day
requalify semantics, stamp paper trades fill_mode=near_close, and migrate
stored schedule_* keys to America/New_York.
2026-07-18 17:55:39 +02:00
dennisthiessen 3eb6192a1e feat: log Phase A decisions and add execution-recovery matrix
Document Phase A (max-hold/vol/corr closed; next-open as decision baseline).
Add stale_close and next_open gap-cap fill modes plus a small matrix to test
whether near-close scheduling recovers overnight momentum drift.
2026-07-18 16:27:10 +02:00
dennisthiessen 529343ce82 feat: add Phase A research matrix (vol target, fill, corr, SE/DSR)
Ship shared Sharpe SE/PSR diagnostics, next-open fill and equity-curve vol targeting in the portfolio simulator, re-derived fip_id, and a checkpointed offline matrix runner for Mac-side validation sweeps.
2026-07-18 15:04:44 +02:00
dennisthiessen cad4b49e7c fix: harden Structural S/R after OHLCV writes and surface cleanup failures
Deploy / lint (push) Successful in 8s
Deploy / test (push) Successful in 1m18s
Deploy / deploy (push) Successful in 40s
Honor custom S/R tolerance as a transient detect, refresh levels after OHLCV
mutations without failing committed price writes, report per-ticker S/R
rebuild failures from admin cleanup, and warn in the admin UI when refresh is partial.
2026-07-18 13:44:34 +02:00
dennisthiessen b0e33e1606 fix: align production defaults and close review parity gaps
Ship greenfield min_rr=2.0 and conf=0, read-only Structural S/R, indicator
cache invalidation, and UI/gate language that treats GTL as screening not exit.
Align strategy_rank missing-vol fallback live vs backtest, single-source
PRIMARY_TARGET_MIN_RR, expand prod parity tests, and drop dead FE clients.
2026-07-18 13:03:22 +02:00
dennisthiessen d13c54e3c7 fix: grandfather pre-cutover stop episodes 2026-07-17 20:28:42 +02:00
dennisthiessen 5155d00d9e feat: require gate reset before post-stop reentry 2026-07-17 19:30:40 +02:00
dennisthiessen bbc7383d3a feat: compare legacy and live ranking universes 2026-07-17 17:07:35 +02:00
dennisthiessen 9800114fc4 fix: align daily matrix ranking universe 2026-07-17 16:39:43 +02:00
dennisthiessen 0cd9ee7689 feat: add daily reentry policy matrix 2026-07-17 16:11:18 +02:00
dennisthiessen 65a462271c feat: add selectable daily backtest cadence 2026-07-17 14:41:24 +02:00
dennisthiessen bc50ba9136 fix: harden post-stop reentry lockdown 2026-07-17 14:17:57 +02:00
dennisthiessen 1e9f2dc4fb feat: add five-session post-stop reentry lockdown 2026-07-17 13:21:06 +02:00
dennisthiessen f714782fa4 fix: use categorical regime fundamentals
Deploy / lint (push) Successful in 8s
Deploy / test (push) Successful in 1m13s
Deploy / deploy (push) Successful in 37s
2026-07-15 10:13:22 +02:00
dennisthiessen 81c6f5fe2f fix: refresh latest regime trading session
Deploy / lint (push) Successful in 10s
Deploy / test (push) Successful in 1m14s
Deploy / deploy (push) Successful in 39s
2026-07-15 09:13:42 +02:00
dennisthiessen 1d5b1489be feat: replace regime monitor with v2 methodology 2026-07-15 09:02:56 +02:00
dennisthiessen fd21067a40 Fix S&P 500 universe parse so renames like BNY are discovered.
Deploy / lint (push) Successful in 8s
Deploy / test (push) Successful in 1m6s
Deploy / deploy (push) Successful in 37s
Wikipedia no longer uses plain symbol table cells; parse exchange links and NyseSymbol templates, surface the list source in bootstrap results, and keep legacy cell parsing as a fallback.
2026-07-14 14:14:58 +02:00
dennisthiessen f59c0c3484 Add system alerts log with nav badge and Admin Alerts tab.
Deploy / lint (push) Successful in 8s
Deploy / test (push) Failing after 1m4s
Deploy / deploy (push) Has been skipped
Persist job and ingestion warnings/errors for 7 days, surface a dismissible top-nav badge, treat stale OHLCV as a warning (e.g. ticker renames), and show market bar age on the ticker freshness chip.
2026-07-14 13:38:04 +02:00
dennisthiessen 8db535b889 Fix manual refresh dropping qualified ranks and clarify trade UI.
Deploy / lint (push) Successful in 9s
Deploy / test (push) Successful in 1m20s
Deploy / deploy (push) Successful in 39s
Single-ticker fetch now attaches residual-momentum ranks so setups do not silently fail the activation gate. Exit plan is a timeline, chart labels move left of the price scale, and missing ranks surface explicitly.
2026-07-14 10:10:30 +02:00
dennisthiessen bddaeb9110 Cleanup retired S/R research scaffolding
- Remove unused _gate_eligible_levels filtering logic and its tests (research-only)
- Add prominent RESEARCH/DIAGNOSTIC markers and docs to clear-air/ATR fallback helpers
- Document production vs research BACKTEST_* environment variables in backtest_service
- Minor cleanups: update legacy report text, improve outdated function docstring
2026-07-13 18:52:24 +02:00
dennisthiessen bee5a5ce89 Finalize GTL and retire S/R research harness 2026-07-13 17:58:04 +02:00
dennisthiessen 9d362bd568 Default online backtest to production GTL 2026-07-13 16:51:55 +02:00
dennisthiessen 8e09f239c8 Retire completed GTL tuning harnesses 2026-07-13 16:40:29 +02:00