Completes the pure read-time core.
fundamentals_peers.py: median + polarity-aware favorable percentile + peer_count
for a subject within its SIC group (CIK-deduped by the caller); returns None
below MIN_PEERS=5 so the caller omits the industry object. Absolute net_debt is
intentionally NOT peer-eligible (size-dependent) — leverage compares via
net_debt_to_ebitda. HIGHER_IS_BETTER polarity map + two_digit_sic() grouping key.
fundamentals_reads.py: one shared deterministic rule set (no LLM): growth_read
(+-2pp), margin_read (latest vs mean-of-prior, +-1pp), share_count_read (+-1%),
peer_read (60/40 bands, polarity-aware phrasing per metric), header_sentence
(growth · margins · valuation, omitting empty). Tunable named constants; >=3
periods required for a series read.
Tests: 8 peer + 5 reads, anchored on the boundary cases (exactly +2.0pp, exactly
60th percentile, exactly +1.0pp margin). 13 passed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1. net_debt requires BOTH cash and total_debt; a missing side is null, not
treated as zero (which would be a partial, misleading value).
2. net_debt_to_ebitda is null when TTM EBITDA <= 0 — a negative denominator
would otherwise rank a distressed issuer as favorably low-leverage.
3. The quarter tape is the CONSECUTIVE run ending at the latest period (stops at
a gap), so trend text never compares non-adjacent quarters as if consecutive.
4. YoY growth is null when the prior-year TTM is <= 0 (e.g. loss->profit), which
is not a meaningful percentage.
Also corrected the plan's net-debt formula to total debt − (cash + ST) matching
the positive-means-net-debt implementation. +4 tests. 10 passed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Derives the display metrics from the stored YTD snapshots at read time (no I/O,
no DB), per the A3 schema decision. Given an issuer's snapshot rows it produces:
- amendment selection (newest accepted_at per fiscal period);
- discrete quarters = YTD(Qn) - YTD(Qn-1), Q4 = YTD(FY) - YTD(Q3);
- TTM = trailing four discrete quarters; missing period -> null, never partial;
- metric series (value + 4-quarter tape, each point dated): revenue_growth_yoy,
eps_growth_yoy, operating_margin, fcf_margin, net_debt, net_debt_to_ebitda,
share_count_change_yoy;
- request-time valuation inputs (ttm_diluted_eps, ttm_fcf, shares_outstanding)
for the API to combine with price.
Units per app convention (percentages = pp, leverage = multiple, dollars).
Tests: 6 (growth+Q4, margins, net-debt/EBITDA+dilution, valuation inputs,
missing-period-null, amendment selection). Verified on real Apple snapshots:
op margin 32.6%, net-debt/EBITDA 0.10, buyback -1.7%/yr, TTM EPS $8.26.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Extend the companyfacts structural check to reject a concept with a
missing/non-dict `units` mapping (not just the top-level `facts`), so a
partially-malformed payload fails promotion instead of silently dropping that
concept's facts. New fixture proves it fails.
- Strengthen the newly-added-issuer test: keep latest_index equal to the prior
run so ONLY the universe fingerprint changes the revision — proving the
fingerprint alone prevents a new ticker from being starved/no_op'd.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1. Removed the 45-day index-walk cap: it discarded the older part of a long
outage while still advancing source_max_date, permanently losing filings.
The walk now covers every unprocessed date (a large gap is one-time cost).
2. Discrepancy detection meets the immutability contract: it compares ALL source
snapshot fields (not five), read-only during stage/validate, reports the
differing accessions + fields in validation_json, and promote emits a warning
system event (in-transaction) — never mutating the stored row.
3. Malformed companyfacts (missing facts/units structure) are recorded separately
and FAIL validation, instead of silently degrading to skipped rows that the
50% backfill coverage floor could still pass.
Also corrected the stale "sum share classes" / DEI-only wording in the snapshot
model docstring and the A3 design doc to describe the us-gaap fallback.
Tests: +4 regressions (>45-day gap loses nothing, newly-added issuer backfills
without filing, malformed payload fails, shares discrepancy detected + evented).
23 passed, 1 skipped.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
SecFundamentalsImporter (SourceImporter, source=sec_facts): populates immutable
fundamental_snapshots from Company Facts and back-fills tickers.cik/sic, driven
by the EDGAR daily index. Shadow only. Guardrails per review:
- detect_revision caches the resolved universe + exact tracked index rows and
composes the revision from them; stage consumes those same cached inputs
(no index/universe refetch) so promoted data matches the computed revision.
- Resolution is read-only in stage (proposals only); ticker writes happen in
promote via apply_ticker_updates.
- validate runs the index<->Company-Facts consistency gate before any write:
a tracked XBRL index accession missing from Company Facts fails the run
(they lag independently) so we retry, not record null. Non-XBRL amendments
are skipped with a recorded reason. Backfill has a coverage floor.
- promote inserts ON CONFLICT (accession) DO NOTHING (immutable), reports
differing existing accessions without mutating, and applies ticker updates in
the same transaction.
- Full-history backfill on first run / for newly-added issuers (include_history);
incremental fetch only for issuers that filed.
Parser: split parse result into skipped_filings vs field_issues (coverage must
not count field warnings); header notes the us-gaap shares fallback; added
companyfacts_accessions() for the gate.
Verified live end-to-end (AAPL + GOOGL backfill): 112 snapshots, cik/sic set,
GOOGL shares via us-gaap fallback, AAPL via dei. Tests: 6 importer (backfill,
incremental, consistency-gate fail, non-XBRL skip, read-only-on-failure,
conflict-discrepancy) + parser ParseResult updates.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1. Multi-class shares: prefer the single dei:EntityCommonStockSharesOutstanding
cover-page fact; else fall back to us-gaap:CommonStockSharesOutstanding at
period end (Alphabet has no dei fact). Never sum class facts (companyfacts is
non-dimensional) and never use weighted-average/diluted; conflicting values ->
null, counted as an "ambiguous shares outstanding" note in validation. Plan's
"sum class-specific" wording corrected. Verified live: Alphabet shares now
populate (12.1B), Apple still uses its dei cover date.
2. Fiscal context is the majority (fy, fp) among facts ending at reportDate, with
ties rejected — no longer the arbitrary first fact.
3. Hardening: catalog selectors require taxonomy == "us-gaap"; indexing drops
malformed facts (missing accession/end, non-finite value) so a custom concept
or bad date can't be selected.
Tests: +8 (dei precedence, us-gaap fallback, conflict->null, no weighted-average,
tie-context skip, foreign-taxonomy/malformed ignored, ambiguous-shares note).
14 passed, 1 skipped.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Pure parser (no I/O/DB) turning one issuer's companyfacts + submissions filing
metadata into per-accession snapshot rows for the filing's primary period.
- Period identity from end == reportDate, never fy/fp (fy/fp is the filing's
context; comparatives repeat it).
- Duration facts stored as cumulative YTD: pick the fact whose span matches the
fiscal-period-to-date length (Q1~3mo..FY~12mo) within tolerance; no YTD-length
fact -> null (never a discrete masquerading as YTD).
- Balance-sheet instants at end == reportDate; shares_outstanding is the dei
cover-page fact whose own end (cover date) is stored in shares_outstanding_date.
- Cash and debt composites are aggregate-first and mutually exclusive (each
source tag counted at most once).
- Carries filing_date through submissions rows (snapshot.filed_date).
Verified on REAL Apple companyfacts: 44 snapshots, 0 skipped, YTD revenue
124.3B->219.7B->313.7B->416.2B across FY2025 (Q4 derives at read time), every
shares_date is the cover date != period_end. Tests: 6 fixture + 1 skip-guarded
live-invariants (monotonic YTD, cover-date shares).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Addresses the slice-1 review:
1. Resolution is now read-only (A1 transaction contract). resolve_ciks /
fetch_sic_updates compute proposals and mutate nothing; a new
apply_ticker_updates issues the writes, called only in promote — so a failed
validation can't leak ticker changes on the framework's failure commit.
2. Only 404 means "missing". Added SecNotFoundError; daily_index /
latest_index_date catch only that. 403, exhausted 429, 5xx, timeouts, and
transport/parse errors now propagate instead of looking like "no index".
3. Fair-access enforced when opening a REAL client (transport=None): reject
blank/placeholder/non-email User-Agent and sub-0.11s spacing. Mock transports
skip it (tests use 0 spacing).
4. submissions(include_history=False) by default — only the one-time full
backfill fetches the history shards; SIC/incremental work makes no extra
requests.
Plus: retry transient 5xx/network errors and honor Retry-After during the 1 GB
backfill; compose_revision rejects a missing index date (no "None:..." revision).
Re-verified live vs real SEC (fair-access validation passes, shard merge intact).
Tests: 18 (added error propagation, read-only resolution, fair-access, recent-only
submissions, reject-None revision).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
First A3 implementation checkpoint (design: docs/dolt-sec-a3-design.md).
- sec_client.py: async SEC EDGAR client honoring fair-access — identifying
User-Agent (config), request spacing < 10 req/s, exponential backoff on 429,
and 403 -> SecForbiddenError (alert and stop, never retry-loop). Fetchers:
company_tickers (normalised, multi-class share CIK), submissions (merges the
paginated filings.files shards so full history is visible), companyfacts,
daily_index (fixed-width form.idx parse), latest_index_date.
- sec_universe.py: resolve_ciks (tickers.cik backfill), refresh_sic
(sic/sic_description), and the composite-revision pieces — universe_fingerprint
(a new ticker changes the revision, so it's never no_op'd/starved),
index_content_hash, compose_revision.
- config + .env.example: SEC_USER_AGENT (must be a real contact email) + spacing
/ retries / timeout.
Verified live against real SEC: AAPL->320193, GOOG==GOOGL, BRK-B resolved;
submissions shard-merge proven (131 filings back to 1993); daily index parsed.
Tests: 11 (mocked-transport parsing + 403/429 handling + resolution/fingerprint).
Full suite 713 passed. Next slice: companyfacts -> snapshot parser + importer.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fold in the A3 design review:
1. Composite revision = latest-index-date + index-content-hash + tracked
symbol->CIK fingerprint, so a newly added ticker forces a run instead of
being no_op'd/starved. No backfill sentinel — absence of a prior promoted
run triggers backfill; source_max_date records the processed index date.
2. Full history needs the paginated submissions shards: filings.recent caps at
1000; older accessions (reportDate/acceptanceDateTime/isXBRL) live in
filings.files[] shards (verified on Apple: recent=1000, one 1994-2015 shard).
3. Index<->Company-Facts consistency gate: they are separate SEC products that
can lag; for every tracked isXBRL index accession, confirm it exists in
Company Facts before promotion, else fail+retry (never record a null/partial
snapshot). Non-XBRL amendments skipped with a recorded reason.
4. Immutable = insert-only (ON CONFLICT DO NOTHING); a differing re-fetch is a
reported discrepancy, never a silent mutation / import_run_id replacement.
Plus deterministic, mutually-exclusive cash/debt composition (aggregate-first;
each source tag counted at most once).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
All three decisions approved: fetch via EDGAR daily-index (not bulk zip),
one snapshot per accession for its primary period (comparative-only
restatements out of scope), full-history backfill on first run. Doc status
flipped to approved / ready to implement.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Design (not implementation) for phase A3, grounded in live SEC data probes.
Key findings: fp has no Q4 (derive it); fy/fp are the filing's context not each
fact's period (select by end==reportDate); SEC provides both discrete and YTD
facts (confirms stored-YTD schema); companyfacts endpoint has no ETag/
Last-Modified (conditional GET impossible); tickers are dash-form and GOOG/GOOGL
share one CIK.
Two plan deviations flagged for sign-off:
1. Fetch via the EDGAR daily-index (fetch companyfacts only for tracked issuers
that filed) rather than the multi-GB bulk zip — lighter and restores the
revision/no_op model.
2. One snapshot row per accession for its primary period (YTD-cumulative);
comparative-only restatements out of scope (only real 10-K/A updates a period).
Plus a metric tag catalog, read-time derivation rules (missing period -> null),
CIK resolution, validation gates, and SEC fair-access handling.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Addresses the A2 review:
1. Every dolt subprocess is now bounded by a hard timeout
(dolt_command_timeout_seconds, default 600s); on expiry the process is killed
and DoltError raised — a hung pull/sql can no longer pin the import
connection and advisory lock indefinitely. Tested (timeout + non-zero exit).
2. Initial-load validate is stronger: besides zero-future, an initial load now
requires a real forward horizon (>= 21d, under the ~35d observed on the
clone) AND universe coverage >= 50% (a broken symbol join can't seed a hollow
calendar). Subsequent runs keep the 50% collapse gate.
3. Revision uses DOLT_HASHOF('HEAD') — formally HEAD, not dolt_log-by-timestamp.
4. Free-disk floor raised 2 GB -> 5 GB (safe headroom over the ~1.7 GB clone).
Full suite 702 passed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A1 carry-forward. The SEC cover-page share count
(dei:EntityCommonStockSharesOutstanding) is reported "as of" its own date, which
can differ from the fiscal period_end — store that date so market cap uses the
right point-in-time count. Migration 026 edited in place (never run with data).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A SourceImporter that ingests post-no-preference/earnings into earnings_events
for the tracked universe. Shadow by construction (nothing reads earnings_events
until A4).
- earnings_alignment.py: pure calendar<->EPS-history min-cost monotonic DP,
reused from scripts/import_dolthub_earnings.py with identical constants (not
extending that one-off script); symbol/session normalization; unit-tested
against the pinned constants.
- dolt_client.py: async dolt CLI wrapper (pull / current_commit / query_csv via
asyncio.create_subprocess_exec — never blocks the shared event loop) + disk
guard before pull.
- dolt_earnings_importer.py: detect_revision = pull + HEAD hash; stage = query
earnings_calendar + eps_history, dedup, align, map act_symbol->ticker_id
(normalize both sides so dotted BRK.B joins); promote is destructive
(delete future dolt_earnings rows + upsert; past never deleted) so validate is
FAIL-CLOSED — blocks when the staged forward calendar is empty or has collapsed
below 50% of what's loaded (the forward calendar is the acceptance gate).
- NOTICE: CC BY-SA 4.0 attribution; config: DOLT_BINARY / DOLT_DATA_DIR / etc.
Verified end-to-end against the real 1.68 GB clone (5 tickers: 133 events, 128
paired, forward calendar to 2026-08-26, BRK.B joined). Tests: 9 alignment + 7
importer + 1 skip-guarded real-clone smoke. Full suite 699 passed.
Remaining for A2: wire the daily ~02:30 ET shadow cron — deferred to pair with
the deploy-time dolt install + DOLT_DATA_DIR provisioning.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Both real target tables (fundamental_snapshots, earnings_events) carry an
import_run_id; stamping requires the current run's id. A2 (the earnings
importer) is the first real consumer, so promote gains a run_id argument rather
than having importers hack the running row out of the framework. Protocol +
call site updated; the A1 fake importer now stamps and asserts import_run_id.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Local dev uses a Dolt clone of post-no-preference/earnings under dolt-data/
(git-ignored). Production keeps clones in DOLT_DATA_DIR outside the repo tree —
the deploy is rsync --delete of the tree, so a clone inside it would be unsafe.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
post-no-preference/earnings approved for private/internal ingestion under
CC BY-SA 4.0. Conditions the A2 importer must honor: preserve upstream license /
attribution / transformation notes; no public API, bulk export, or
redistribution; re-review before any public or commercial access. The stocks
repo (workstream B) is not covered and will be reviewed separately if B begins.
A0 rollout item marked done (dolt binary pin + DOLT_DATA_DIR still pending at
deploy time).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Review items 3-4 on the plan doc:
- Replace remaining "diluted shares" (the share *count*) with point-in-time
shares_outstanding (dei:EntityCommonStockSharesOutstanding) across schema,
metrics catalog and market-cap note. "diluted EPS" is left as-is (correctly a
duration fact). Adds the multi-class rule: derive the issuer-wide count from
the consolidated cover-page figure OR by summing class-specific facts (GOOG +
GOOGL) — never both, to avoid double counting.
- The framework stages into a representation *outside the live tables* (in-memory
for workstream A; a file/table handle is fine if B needs it), not physical
"staging tables" — wording now matches the implementation.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Review fixes to the import-run framework (A1):
1. detect_revision ran outside the failure handler, so a failed revision probe
(the most likely external failure) escaped unrecorded — violating "every
attempt is recorded". Now the running row is created FIRST, then
detect_revision + last-revision lookup + stage + validate + promote all run
inside the same handler; the row converts to no_op when the revision is
unchanged. New test covers a detection exception → recorded failed + alert.
2. asyncio.CancelledError (BaseException, not caught by except Exception) left a
permanent running row on deploy/scheduler shutdown. Now caught explicitly:
best-effort mark failed, then re-raise the cancellation (never swallowed).
New test asserts the run is failed and the error re-propagates.
Full suite 682 passed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
run_import + SourceImporter Protocol (detect_revision/stage/validate/promote)
giving every bulk importer the plan's non-negotiables, KISS:
- one run per source at a time — Postgres session-level advisory lock held on a
single pinned engine.connect() so it survives the running-row and promotion
commits; no-op on SQLite.
- idempotent per revision — cheap detect_revision compared to the last promoted
run; unchanged revision records a no_op with zero writes (no fetch).
- staging (in-memory, no physical staging tables) → validate (read-only) →
atomic promote + run-row flip in one transaction.
- failed validation or mid-run exception marks the run failed, alerts via
system_event_service, and leaves live tables untouched.
Every attempt recorded in data_import_runs; conflicts summary in validation_json
(no conflicts table). Concrete SEC/earnings importers land in later phases.
Tests: 6 orchestration tests (no_op / promote / new-revision / failed-untouched
/ promote-exception-rollback) + deterministic advisory-key derivation. Full
suite 680 passed. Advisory-lock mutual exclusion is PG-verify-pending (SQLite
no-ops it — flagged, not covered).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
First reviewable slice of workstream A: schema only, no importers, no data.
- data_import_runs: lean batch-import audit (source/revision/status,
row_counts_json + validation_json as Text-holding-JSON per repo convention).
- fundamental_snapshots: CIK-keyed, one immutable row per accession; stores
per-period raw facts (duration = cumulative YTD/FY, balance-sheet =
period-end) plus period_start/period_end/fiscal_year/fiscal_period so
discrete quarters, Q4, TTM and YoY are derived at read time.
- earnings_events: Dolt-sourced calendar + surprise history, unique
(ticker_id, announce_date).
- tickers: nullable cik/sic/sic_description — the ticker<->issuer join point.
fundamental_data is left untouched (cutover gated separately at A5). Models
registered in app/models/__init__.py; Ticker gains an earnings_events
relationship. Verified: create_all builds the tables, mappers configure, and
migration 026 renders valid Postgres DDL up and down.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds the Dolt bulk-data integration hand-off plan (authored in prior
session) and applies clarifications found in a pre-handoff review:
- step (c): fields refresh from three distinct sources, not "snapshots +
close" — earnings_surprise/next_earnings_date come from earnings_events,
not SEC facts. pe_ratio = close / TTM diluted EPS and market_cap =
issuer-wide diluted shares × close stated as separate formulas.
- fundamental_snapshots made implementable: add period_start, fiscal_year,
fiscal_period; duration facts store the filing's cumulative YTD/FY values,
balance-sheet facts store period-end values. Discrete quarters, Q4, TTM
and YoY are derived at read time (correct for non-calendar fiscal years;
amendments never freeze a stale derived quarter).
- flag Q4 derivation / fiscal-period alignment as the primary A3 risk.
- anchor "tracked universe" to ticker_universe_service + per-run CIK
resolution.
All code anchors in the doc verified accurate against the current tree.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The shadow book toggle saved on change and the numeric fields on blur,
inconsistent with every other admin panel (which stages edits behind a
Save button). Stage all four fields in local state and commit them
together on Save, with an unsaved-changes hint. This also makes enabling
the live-trade toggle a deliberate two-step action rather than an
unguarded single click.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The run-id marker proved which scan wrote last, but the shadow book still
selected setups by detected_at >= scan_start. An overlapping manual scan
could insert rows in that same window; if the pipeline's scan wrote the
marker last its id matched and the shadow book proceeded, then swept in --
or ranked highest -- a manual-scan row. The identity check gated entry but
selection did not.
Carry the run id onto the rows. Migration 025 adds an indexed
trade_setups.scan_run_id. scan_all_tickers computes one id per run
(pipeline's when a step, else fresh), passes it to scan_ticker which stamps
every row after enhancement, and writes the same id to the completion
marker. The shadow book selects WHERE scan_run_id == the matched id, so a
concurrent scan's rows are excluded by identity regardless of their
detected_at. The now-unused STARTED marker is dropped; COMPLETED
(freshness) and RUN_ID (identity) remain.
Decisive test: the pipeline's id matches, but a same-window manual row with
a higher rank is present and is excluded -- only the pipeline's own row is
traded. A time-window select would have swept it in and ranked it first.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A manually triggered rr_scanner and the scheduled near-close pipeline are
separate APScheduler jobs; max_instances=1 serialises a job only against
itself, so they can overlap. A manual scan starting just before the
pipeline can finish just after it began and overwrite the scan markers.
Its completion timestamp is then later than the pipeline start, so the
previous 'completed >= pipeline_start' check accepted its batch as though
it were the pipeline's own -- exactly when the pipeline's scan may have
failed.
Replace the timestamp comparison with an exact run-id match. A new
pipeline_run module holds a per-task run-id contextvar (separate module so
the scanner and scheduler import it without a cycle). _run_pipeline binds a
fresh id per invocation; scan_all_tickers stamps that id -- or a fresh one
when run standalone -- into the scan markers, written with started/completed
in a single commit. The shadow step requires the stored run id to equal its
pipeline's id exactly, so a concurrent manual scan (its own id) or a failed
pipeline scan (a prior run's id) can never be mistaken for it. Direct Admin
triggers have no pipeline context and keep the freshness fallback.
Known residual: the id match governs whether shadow proceeds; setup
selection remains detected_at >= scan start, so a fully per-run setup
isolation would need a run_id column on trade_setups (not required here).
Tests cover the reported race (manual scan finishing last is refused), a
failed pipeline scan, the id-match accept path, and contextvar propagation
and non-leakage across tasks.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A 6-hour freshness window proves only that some scan ran recently, which a
manual mid-day scan satisfies. Scenario: a manual scan succeeds at 13:00;
the 15:30 near-close pipeline's scan step is disabled or fails; at 15:30
the 13:00 completion is still 'fresh', so the shadow step trades that
earlier batch despite no successful scan in the current pipeline.
_run_pipeline now records its start in a per-task contextvar, visible to
the steps it awaits. run_shadow_book reads it and requires the scan
completion marker to be at/after the pipeline start, so a scan that failed
or was disabled in this pass (marker left at a prior run, before the
pipeline began) cannot be substituted by an earlier manual scan. A direct
Admin trigger has no pipeline context and falls back to the freshness
window -- an explicit operator action, not an automated one.
Tests pin the reported case: a fresh manual scan predating the pipeline
start is refused; the pipeline's own post-start scan is accepted.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Second review round on the shadow book; all three findings were real.
- Scan freshness is now proven, not assumed. Pipeline steps run and fail
independently, so a disabled or failed scan step still let the shadow
step run on the newest *stored* setups -- a prior session's picks at
stale prices. scan_all_tickers now records a run boundary
(last_scan_run_started_at / _completed_at) only on successful
completion; the shadow book refuses to trade unless COMPLETED is fresh
and selects only setups with detected_at >= the run start. Deduplication
to the latest row per ticker now happens BEFORE qualification, so a newer
unqualified row suppresses an older qualified one rather than the reverse.
- Shadow selection is hard long-only. setup_qualifies only enforces
long-only when min_momentum_percentile > 0, but 0 is a legal admin
setting, and the cash accounting assumes long positions -- so the
constraint is enforced in shadow selection regardless of gate config.
- The personal setup list excludes only the caller's own open positions.
get_trade_setups gained exclude_open_trade_user_id; the trades route
passes the authenticated user, while the Telegram broadcast stays global
since it has no single owner.
New tests cover stale/absent scan markers, prior-run exclusion, newer
unqualified suppressing older qualified, long-only under a disabled gate,
and both sides of the user-scoped exclusion.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Review of the shadow book found seven ways the two books could leak into
each other; all are fixed here. The most serious silently invalidated the
comparison the shadow book exists to make.
- Shadow holdings no longer suppress the manual candidate list. The
open-trade exclusion filtered on any book, so shadow taking the
top-ranked names removed exactly those from the user's list and alerts,
confining the discretionary book to leftovers. Scoped to the manual
book. Closed-trade alerts and paper-book equity were leaking the same
way and are likewise scoped.
- Shadow sizing now matches _simulate_portfolio: min(1% risk, 20% notional
cap, available cash) from marked equity, plus the sub- dust guard.
Previously risk-only from realized equity, so a tight stop produced a
multiples-of-equity leveraged position the strategy would never take.
- Shadow only trades setups from the scan that just ran (<6h old) with one
setup per ticker. A failed or disabled scan step could otherwise open
positions from a prior session at stale prices.
- Gate-reset transitions are observed for both books, so a shadow stop-out
completes fail -> requalify instead of staying locked forever.
- Manual list/close endpoints default to the manual book and reject
hand-closing shadow trades; the performance endpoint is scoped to the
caller so 'your picks' is not every user's book.
- run_shadow_book is registered as a paused job so Admin can trigger it.
Also anchors three pre-existing paper-trade tests (and the new alpaca
window test) on the UTC date. They build fixtures from the local date but
the service stamps opened_at in UTC, so they failed only between 00:00 and
02:00 in a UTC+hh timezone -- latent on ba2df8b, exposed by the clock.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The manual paper book only contains trades taken by hand, inside a 20
minute window, on days someone was available. The backtest that validated
this strategy auto-takes the top-ranked qualified setups up to capacity
every session. The forward record was therefore measuring strategy plus
discretion plus availability -- and degrading silently on busy days.
The shadow book closes that gap: it mirrors the backtest's selection rule
(top strategy_rank qualified, up to capacity, 1% fixed-fractional risk)
and shares the manual book's exit policy, so the only difference between
the two books is which setups get taken. Selection ordering reuses the
strategy_rank the scanner already stores rather than recomputing it, so
the two cannot drift apart. It runs as a near-close pipeline step right
after the scan, marking entries at the same prices a human would see.
Gate-reset re-entry state is now scoped per book -- the books diverge as
soon as their entries differ, and each must see only its own stops.
Performance view rewritten around the comparison:
- three series (shadow, manual, SPY) from a new endpoint
- SPY changes from a per-trade cost-basis counterfactual to plain
buy-and-hold %, since one line has to serve two books
- headline stats are R-multiples, not currency: the books size
differently, so only R compares across them
- configurable start date, because the strategy has been revised
repeatedly and pre-cutover trades ran under rules that no longer
exist
Migration 024 also repairs the numeric weekday crons written by 023,
rewriting only rows still holding the broken form so hand-corrected
settings survive. Its literals are inlined because bound parameters
render as NULL under 'alembic upgrade --sql'.
The shadow book is opt-in and writes nothing until enabled. Verify its
first selections match a backtest of that day's cross-section before
trusting any point on the curve.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Earnings backfill sourced from the public DoltHub earnings repo at a
pinned commit rather than the FMP API: reproducible for anyone re-running
the study, and it burns no request quota. 12,414 events, 98.6% of symbols
with >=8 announcements, 99.2% paired actual/estimate, no keyed duplicates.
2a earnings-gap diagnostic: INFORMATIONAL, no filter shipped. The
pre-earnings cohort's right tail was better, so the registered
avoid-earnings condition failed. Note the raw 23/266 vs 115/574 incidence
gap is largely a duration confound -- severe losses stop out fast and have
less time to span an announcement -- so it is not evidence that holding
through earnings is safe.
2b SUE: FAIL against the pre-registered +0.03 bar (unconditional IC
+0.0151 over 56 reliable windows, momentum-conditional +0.0213). Signs
stable across eras, so this is a clean null rather than an ambiguous one,
consistent with post-earnings drift having decayed in large caps.
Closes the Tier-1 arc: Task 1 dead on deep evidence, Task 2 dead here,
Task 3 complete as diagnostic. No in-sample research thread remains open.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two independent bugs left the near-close scan running on the previous
session's close, silently degrading live execution to the stale_close
floor (~1.57 Sharpe) instead of the intended ~1.77 close-fill case.
1. OHLCV window never covered the current day. Daily bars are stamped at
session start (04:00Z under EDT), so an end of midnight-on-end_date
landed before that day's bar and dropped it. Widening the window alone
fails the whole request with 'subscription does not permit querying
recent SIP data', so end is also clamped to now-20min. Today's bar is
now returned, roughly 20 minutes behind live -- within the staleness
the near-close design already assumed.
Intraday runs therefore store a partial bar and ingestion progress
reaches today, which made incremental resume skip the after-close
refresh entirely. collect_ohlcv_final() re-pulls the last sessions so
the consolidated bar overwrites the partial one before outcome eval.
2. APScheduler's from_crontab() passes day-of-week to its own field where
0=Monday, so '1-5' meant Tue-Sat: every Monday was skipped and the
scanner ran Saturdays on stale data. Weekday schedules now use names.
Stored settings already corrected via Admin; this fixes the defaults.
Tests cover both: today's bar inside the window, the delayed-data clamp,
historical windows untruncated, and a week of fire times asserting Monday
is present and weekends are not.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Drop intermediate history-depth reports, sector-residual runners/map/code hooks
(evidence stays in final reports + docs), and slim MacBook helper to ssl/earnings/
prod-book-matrix only. SSL bootstrap and archived research conclusions retained.
Four-arm results: 505 stays positive (softer on deep history); liquid breadth
destroys book under current knobs. Stop tracking 1.3GB pkl cache under reports/.cache.
Pre-register A-D (4y/2016 x 505/505+liquid) with unchanged production knobs.
Runner caches full GTL candidates then re-ranks per arm; MacBook entry via
run_tier1_macbook.sh --prod-book-matrix.
Deep masked retest failed the iron IC bar (0.027 < 0.03). Log as rejected #13 in
research README; close sector-residual and history-depth docs. Production market
residual unchanged.
SANITY-FAIL report showed megacaps/ETFs already at empirical 2016-01-04 floor
(2649 bars) after deepen; check wrongly required ~2013. Pass when megacaps leave
the old 2021 two-tier floor and match SPY; XLC listing exception retained.
Terminal follow-up for Task 1: detect/refetch shallow two-tier symbols and sector
ETFs at 5000d, regenerate manifest, run ONE liquid-1500 harness with era split, grade
mom_12_1_sector_resid mechanically. Bundled as run_tier1_macbook.sh --sector-resid-deep.
Authoritative report history-depth-20260719-103315: race guard pass on deep
research.sqlite. Sector residual still short-window only (no pre-2021); fip sign
flips on broad deep sample; no production retune.
Extract app/ssl_bootstrap.py (shared with FastAPI main), wire it into research
scripts, and teach run_tier1_macbook.sh to locate combined-ca-bundle.pem, certifi,
optional USE_CORP_PROXY, plus --ssl-check diagnostics.
Tier-1 alpha research (local only, no production deploy):
Sector residual momentum: two-factor SPY+sector residual and sector demean signals, IC harness + A/B. Sector resid clears pre-registered bars narrowly (PROMOTE for human wire design only). Sector demean fails t vs market resid.
Earnings: earnings_events backfill (FMP bulk paid; FMP/AV per-symbol), 2a gap diagnostic report-only, 2b SUE IC (PARK; incomplete 48/506 coverage).
History-depth: pre-registered doc + runner for MacBook deep rebuild/harness.
Do not ship production residual or filters from this branch.
Brings env-gated liquid-breadth harness hooks, research tooling, compact
evidence, and the completion-manifest race guard. No production behavior
change when liquid env vars are unset. Nothing to deploy.
Log the 21:14 orphan as a snapshot-build race, rewrite the context table to
authoritative ICs only, and soften the vol-tilt warning. Add extender completion
manifest + breadth refuse guard; strip intermediate/orphaned reports; park the
thread (no book sim, no deploy).
Add lagged/tier/prod-subset/mom-conditional checks on research.sqlite.
Log: unconditional sign is a winner/bleeder tug-of-war; mom-conditional
fip stays negative and reliable; warn on high-vol tilt if universe broadens.
Add research-only snapshot extender, PIT dollar-volume mask for signal IC,
rank-only harness path, fingerprint+breadth runner, and docs. Fingerprint
reproduced IC -0.045 / t -2.91 on prod.sqlite. No production gate/schedule changes.
Display-only Da/Gurun/Warachka information discreteness on the ticker
indicator panel. Shared compute with the backtest harness; not wired into
gate or rank.
Regime is computed at 02:00 ET; without a morning alert pass, quadrant
changes waited until 15:30. Dispatcher is change-driven so quiet days stay
quiet. Drop unused alerts_frequency config.
Move the only qualifying R:R scan to 15:30 ET with chained Telegram alerts,
put outcome eval after a final-bar OHLCV fetch, enforce NY trading-day
requalify semantics, stamp paper trades fill_mode=near_close, and migrate
stored schedule_* keys to America/New_York.
Record monotone fill-timing gradient, live [1.57,1.77] bracket, recover-flag
interpretation, gap-cap as third tail-trim, and pre-scheduler ops checklist.
Document Phase A (max-hold/vol/corr closed; next-open as decision baseline).
Add stale_close and next_open gap-cap fill modes plus a small matrix to test
whether near-close scheduling recovers overnight momentum drift.
Ship shared Sharpe SE/PSR diagnostics, next-open fill and equity-curve vol targeting in the portfolio simulator, re-derived fip_id, and a checkpointed offline matrix runner for Mac-side validation sweeps.
Honor custom S/R tolerance as a transient detect, refresh levels after OHLCV
mutations without failing committed price writes, report per-ticker S/R
rebuild failures from admin cleanup, and warn in the admin UI when refresh is partial.
Ship greenfield min_rr=2.0 and conf=0, read-only Structural S/R, indicator
cache invalidation, and UI/gate language that treats GTL as screening not exit.
Align strategy_rank missing-vol fallback live vs backtest, single-source
PRIMARY_TARGET_MIN_RR, expand prod parity tests, and drop dead FE clients.
Finnhub profile2 reports marketCapitalization in millions; storing it
as dollars made mega-caps like SPCX show as micro (e.g. 1.8M). Normalize
on ingest, add unit tests, and include a one-shot SQL backfill script.
Wikipedia no longer uses plain symbol table cells; parse exchange links and NyseSymbol templates, surface the list source in bootstrap results, and keep legacy cell parsing as a fallback.
Persist job and ingestion warnings/errors for 7 days, surface a dismissible top-nav badge, treat stale OHLCV as a warning (e.g. ticker renames), and show market bar age on the ticker freshness chip.
Reconstruct the ATR/% trail bar-by-bar so the mini chart shows a ratchet path instead of a flat current level, and de-emphasize screening targets under trailing exit modes.
Keep support/resistance visible even near trade lines, tighten the how-this-exits spine so steps connect, unify step styling, and add space above the panel.
De-clutter setup labels (left roles, right prices; hide Entry near Now; S/R in-plot), and draw green/red arrows for paper trade entry and exit on the matching sessions.
Single-ticker fetch now attaches residual-momentum ranks so setups do not silently fail the activation gate. Exit plan is a timeline, chart labels move left of the price scale, and missing ranks surface explicitly.
- Remove unused _gate_eligible_levels filtering logic and its tests (research-only)
- Add prominent RESEARCH/DIAGNOSTIC markers and docs to clear-air/ATR fallback helpers
- Document production vs research BACKTEST_* environment variables in backtest_service
- Minor cleanups: update legacy report text, improve outdated function docstring
dennisthiessenandClaude Opus 4.8