max-w-md (448px) under-sized the panel vs its real full-width tab placement and
never triggered the desktop two-column layout. Widen to max-w-3xl; note to
resize to ~390px for the mobile (single-column) check.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the four-cell quarter tape (too many equally-weighted numbers) with one
comparison rail per metric, so the panel answers "improving? sound? fairly
valued?" instead of asking the reader to decode it.
- Operating trend (revenue/EPS growth, operating/FCF margin, share count): a rail
centered on a truthful reference — prior quarter (growth), prior-period average
(margins), or zero (share count) — with a dot at the current delta and a bar
back to the reference, plus a shaded neutral band (backend's +-2pp / +-1pp /
+-1% rules). Value, read, reference label, and signed delta stay visible;
per-quarter history drops out of the default view.
- Valuation & balance (net debt/EBITDA, P/E, FCF yield): a 0-100 favorable-
percentile rail with the peer median fixed at 50; right is always more
favorable (percentile is polarity-aware). median + peer_count shown.
- Not a progress bar: reference line, not a 100% target.
- Horizon tokens: cyan #6EC9DB favorable / coral #EF9182 adverse / #5D6373 track,
replacing emerald/rose. Two columns on desktop, single column (rows stack) on
mobile. Null -> n/a with no rail; insufficient peers -> "peers n/a", no track.
- Kept: compact earnings line, provenance footer, local-date parsing, aria-labels
on every rail. tsc -b passes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Addresses the static review + adds a dev-only visual harness:
1. Tape no longer overflows narrow mobile: each row stacks (label + read on one
line, cells below) under sm, keeping the single-line grid on desktop.
2. Date-only strings (earnings, price_date) are parsed as LOCAL calendar dates,
so a viewer west of UTC no longer sees the previous day.
3. Peer context is visible ("med X · Np") on every width and the percentile
strip carries a full aria-label — no longer hover-only / desktop-only.
4. Per-metric provenance + freshness surfaced (SEC filings · latest quarter,
filed date) replacing the removed panel-wide FMP label.
5. Same-day earnings render "today", not "in 0d".
Harness: frontend/harness.html + src/dev/harness.tsx (dev-only, served at
/harness.html by vite, not in the production build) render full /
partial-insufficient-peer / empty fixtures for desktop + ~390px review.
tsc -b passes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reshapes FundamentalsPanel to consume the additive API v1, within the app's
existing dark-glass language.
- types.ts updated to the exact v1 shape (metrics/earnings/valuation/reads +
legacy fields preserved).
- The quarter tape is the single distinctive device: per-metric 4-cell tape
(revenue/EPS growth, operating + FCF margin, share count) with the latest cell
toned by the deterministic read; color is always paired with the read text.
- Restrained peer strips for Net debt/EBITDA, P/E, FCF yield: value + a
polarity-aware percentile bar with a median marker + the read; hidden ("peers
n/a") when industry is null (< 5 peers).
- Earnings: next date/session/countdown + last-N beat/miss arrows (▲/▼/·) with
text aria-labels; explicit "no date" state.
- Explicit n/a, insufficient-peer, and no-earnings states; header shows the
deterministic sentence. Removed the hard-coded "FMP" source label.
Frontend tsc -b passes; backend suite 778 passed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- The eps_growth_yoy read was never computed, leaving that fixed by_key entry
null even with sufficient EPS history; now growth_read() is applied to EPS
history just like revenue.
- Tests: same-day earnings returns as next with days_until 0 (and not in
recent); zero close guards valuation to null; eps read populated. Fixture
seeds three fiscal years so YoY growth reads have a >=3 run. 9 API tests pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1. Multi-class subject is priced by the REQUESTED ticker: the peer group's
representative for the subject CIK is overridden to the requested ticker_id
(other issuers pick a deterministic-by-symbol rep), so GOOGL's P/E uses
GOOGL's price, not GOOG's. Differing-price GOOG/GOOGL test added.
2. reads matches the selected contract: header is null when there is no read;
by_key is a fixed map over every metric key plus pe and fcf_yield, null when
unavailable (was a sparse dict).
3. Earnings use the New York calendar date; same-day is UPCOMING (days_until 0),
recent is strictly earlier.
4. Valuation is null when there is no usable price (> 0 required for P/E and
market cap); when present, price_date is non-null.
Added a real router/API-envelope test with a seeded legacy record (the endpoint,
not just the schema merge). 6 tests pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
GET /fundamentals/{symbol} now returns the additive v1 objects alongside the
unchanged legacy fields (no legacy growth mapped onto the SEC TTM metric).
- earnings: next (date/session/days_until) + recent (<=4, with surprise_pct)
from earnings_events.
- metrics: fixed key set (value + dated history + per-metric SIC-peer industry
object + source=sec); net_debt has no industry (size-dependent).
- valuation: P/E, FCF yield, market_cap_est computed at REQUEST TIME from the
derived TTM inputs x the latest ohlcv close (no stored valuation); guarded to
null on missing/invalid inputs; pe_industry / fcf_yield_industry peer stats.
- reads: deterministic outputs in a SEPARATE object (header + per-metric reads).
Peer queries are batched and CIK-deduplicated by 2-digit SIC; industry omitted
below 5 valid peers. Schema extended with optional typed sub-models; the router
merges legacy + v1 so every existing field is preserved.
Tests: 4 (full assembly incl. peer industry + valuation + additive-merge, no-cik
null metrics, <5-peers omitted, price-guarded valuation).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1. Peer percentile is now a tie-aware rank against the OTHER issuers
((worse + 0.5*tied)/(peers-1)): an all-equal group maps to 50 (not 100), the
median maps to 50, a unique best to 100, a unique worst to 0.
2. Deterministic reads use the consecutive non-null suffix ending at the latest
point (>=3 values): a null latest or an internal gap yields no read, so a read
never reflects a period displayed as n/a.
3. Peer filtering excludes non-finite (NaN/±inf) as well as null, including an
invalid subject.
Tests updated + added (all-equal, median rank, non-finite, latest-null history,
internal gap). 15 passed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Completes the pure read-time core.
fundamentals_peers.py: median + polarity-aware favorable percentile + peer_count
for a subject within its SIC group (CIK-deduped by the caller); returns None
below MIN_PEERS=5 so the caller omits the industry object. Absolute net_debt is
intentionally NOT peer-eligible (size-dependent) — leverage compares via
net_debt_to_ebitda. HIGHER_IS_BETTER polarity map + two_digit_sic() grouping key.
fundamentals_reads.py: one shared deterministic rule set (no LLM): growth_read
(+-2pp), margin_read (latest vs mean-of-prior, +-1pp), share_count_read (+-1%),
peer_read (60/40 bands, polarity-aware phrasing per metric), header_sentence
(growth · margins · valuation, omitting empty). Tunable named constants; >=3
periods required for a series read.
Tests: 8 peer + 5 reads, anchored on the boundary cases (exactly +2.0pp, exactly
60th percentile, exactly +1.0pp margin). 13 passed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1. net_debt requires BOTH cash and total_debt; a missing side is null, not
treated as zero (which would be a partial, misleading value).
2. net_debt_to_ebitda is null when TTM EBITDA <= 0 — a negative denominator
would otherwise rank a distressed issuer as favorably low-leverage.
3. The quarter tape is the CONSECUTIVE run ending at the latest period (stops at
a gap), so trend text never compares non-adjacent quarters as if consecutive.
4. YoY growth is null when the prior-year TTM is <= 0 (e.g. loss->profit), which
is not a meaningful percentage.
Also corrected the plan's net-debt formula to total debt − (cash + ST) matching
the positive-means-net-debt implementation. +4 tests. 10 passed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Derives the display metrics from the stored YTD snapshots at read time (no I/O,
no DB), per the A3 schema decision. Given an issuer's snapshot rows it produces:
- amendment selection (newest accepted_at per fiscal period);
- discrete quarters = YTD(Qn) - YTD(Qn-1), Q4 = YTD(FY) - YTD(Q3);
- TTM = trailing four discrete quarters; missing period -> null, never partial;
- metric series (value + 4-quarter tape, each point dated): revenue_growth_yoy,
eps_growth_yoy, operating_margin, fcf_margin, net_debt, net_debt_to_ebitda,
share_count_change_yoy;
- request-time valuation inputs (ttm_diluted_eps, ttm_fcf, shares_outstanding)
for the API to combine with price.
Units per app convention (percentages = pp, leverage = multiple, dollars).
Tests: 6 (growth+Q4, margins, net-debt/EBITDA+dilution, valuation inputs,
missing-period-null, amendment selection). Verified on real Apple snapshots:
op margin 32.6%, net-debt/EBITDA 0.10, buyback -1.7%/yr, TTM EPS $8.26.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Extend the companyfacts structural check to reject a concept with a
missing/non-dict `units` mapping (not just the top-level `facts`), so a
partially-malformed payload fails promotion instead of silently dropping that
concept's facts. New fixture proves it fails.
- Strengthen the newly-added-issuer test: keep latest_index equal to the prior
run so ONLY the universe fingerprint changes the revision — proving the
fingerprint alone prevents a new ticker from being starved/no_op'd.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1. Removed the 45-day index-walk cap: it discarded the older part of a long
outage while still advancing source_max_date, permanently losing filings.
The walk now covers every unprocessed date (a large gap is one-time cost).
2. Discrepancy detection meets the immutability contract: it compares ALL source
snapshot fields (not five), read-only during stage/validate, reports the
differing accessions + fields in validation_json, and promote emits a warning
system event (in-transaction) — never mutating the stored row.
3. Malformed companyfacts (missing facts/units structure) are recorded separately
and FAIL validation, instead of silently degrading to skipped rows that the
50% backfill coverage floor could still pass.
Also corrected the stale "sum share classes" / DEI-only wording in the snapshot
model docstring and the A3 design doc to describe the us-gaap fallback.
Tests: +4 regressions (>45-day gap loses nothing, newly-added issuer backfills
without filing, malformed payload fails, shares discrepancy detected + evented).
23 passed, 1 skipped.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
SecFundamentalsImporter (SourceImporter, source=sec_facts): populates immutable
fundamental_snapshots from Company Facts and back-fills tickers.cik/sic, driven
by the EDGAR daily index. Shadow only. Guardrails per review:
- detect_revision caches the resolved universe + exact tracked index rows and
composes the revision from them; stage consumes those same cached inputs
(no index/universe refetch) so promoted data matches the computed revision.
- Resolution is read-only in stage (proposals only); ticker writes happen in
promote via apply_ticker_updates.
- validate runs the index<->Company-Facts consistency gate before any write:
a tracked XBRL index accession missing from Company Facts fails the run
(they lag independently) so we retry, not record null. Non-XBRL amendments
are skipped with a recorded reason. Backfill has a coverage floor.
- promote inserts ON CONFLICT (accession) DO NOTHING (immutable), reports
differing existing accessions without mutating, and applies ticker updates in
the same transaction.
- Full-history backfill on first run / for newly-added issuers (include_history);
incremental fetch only for issuers that filed.
Parser: split parse result into skipped_filings vs field_issues (coverage must
not count field warnings); header notes the us-gaap shares fallback; added
companyfacts_accessions() for the gate.
Verified live end-to-end (AAPL + GOOGL backfill): 112 snapshots, cik/sic set,
GOOGL shares via us-gaap fallback, AAPL via dei. Tests: 6 importer (backfill,
incremental, consistency-gate fail, non-XBRL skip, read-only-on-failure,
conflict-discrepancy) + parser ParseResult updates.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1. Multi-class shares: prefer the single dei:EntityCommonStockSharesOutstanding
cover-page fact; else fall back to us-gaap:CommonStockSharesOutstanding at
period end (Alphabet has no dei fact). Never sum class facts (companyfacts is
non-dimensional) and never use weighted-average/diluted; conflicting values ->
null, counted as an "ambiguous shares outstanding" note in validation. Plan's
"sum class-specific" wording corrected. Verified live: Alphabet shares now
populate (12.1B), Apple still uses its dei cover date.
2. Fiscal context is the majority (fy, fp) among facts ending at reportDate, with
ties rejected — no longer the arbitrary first fact.
3. Hardening: catalog selectors require taxonomy == "us-gaap"; indexing drops
malformed facts (missing accession/end, non-finite value) so a custom concept
or bad date can't be selected.
Tests: +8 (dei precedence, us-gaap fallback, conflict->null, no weighted-average,
tie-context skip, foreign-taxonomy/malformed ignored, ambiguous-shares note).
14 passed, 1 skipped.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Pure parser (no I/O/DB) turning one issuer's companyfacts + submissions filing
metadata into per-accession snapshot rows for the filing's primary period.
- Period identity from end == reportDate, never fy/fp (fy/fp is the filing's
context; comparatives repeat it).
- Duration facts stored as cumulative YTD: pick the fact whose span matches the
fiscal-period-to-date length (Q1~3mo..FY~12mo) within tolerance; no YTD-length
fact -> null (never a discrete masquerading as YTD).
- Balance-sheet instants at end == reportDate; shares_outstanding is the dei
cover-page fact whose own end (cover date) is stored in shares_outstanding_date.
- Cash and debt composites are aggregate-first and mutually exclusive (each
source tag counted at most once).
- Carries filing_date through submissions rows (snapshot.filed_date).
Verified on REAL Apple companyfacts: 44 snapshots, 0 skipped, YTD revenue
124.3B->219.7B->313.7B->416.2B across FY2025 (Q4 derives at read time), every
shares_date is the cover date != period_end. Tests: 6 fixture + 1 skip-guarded
live-invariants (monotonic YTD, cover-date shares).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Addresses the slice-1 review:
1. Resolution is now read-only (A1 transaction contract). resolve_ciks /
fetch_sic_updates compute proposals and mutate nothing; a new
apply_ticker_updates issues the writes, called only in promote — so a failed
validation can't leak ticker changes on the framework's failure commit.
2. Only 404 means "missing". Added SecNotFoundError; daily_index /
latest_index_date catch only that. 403, exhausted 429, 5xx, timeouts, and
transport/parse errors now propagate instead of looking like "no index".
3. Fair-access enforced when opening a REAL client (transport=None): reject
blank/placeholder/non-email User-Agent and sub-0.11s spacing. Mock transports
skip it (tests use 0 spacing).
4. submissions(include_history=False) by default — only the one-time full
backfill fetches the history shards; SIC/incremental work makes no extra
requests.
Plus: retry transient 5xx/network errors and honor Retry-After during the 1 GB
backfill; compose_revision rejects a missing index date (no "None:..." revision).
Re-verified live vs real SEC (fair-access validation passes, shard merge intact).
Tests: 18 (added error propagation, read-only resolution, fair-access, recent-only
submissions, reject-None revision).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
First A3 implementation checkpoint (design: docs/dolt-sec-a3-design.md).
- sec_client.py: async SEC EDGAR client honoring fair-access — identifying
User-Agent (config), request spacing < 10 req/s, exponential backoff on 429,
and 403 -> SecForbiddenError (alert and stop, never retry-loop). Fetchers:
company_tickers (normalised, multi-class share CIK), submissions (merges the
paginated filings.files shards so full history is visible), companyfacts,
daily_index (fixed-width form.idx parse), latest_index_date.
- sec_universe.py: resolve_ciks (tickers.cik backfill), refresh_sic
(sic/sic_description), and the composite-revision pieces — universe_fingerprint
(a new ticker changes the revision, so it's never no_op'd/starved),
index_content_hash, compose_revision.
- config + .env.example: SEC_USER_AGENT (must be a real contact email) + spacing
/ retries / timeout.
Verified live against real SEC: AAPL->320193, GOOG==GOOGL, BRK-B resolved;
submissions shard-merge proven (131 filings back to 1993); daily index parsed.
Tests: 11 (mocked-transport parsing + 403/429 handling + resolution/fingerprint).
Full suite 713 passed. Next slice: companyfacts -> snapshot parser + importer.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fold in the A3 design review:
1. Composite revision = latest-index-date + index-content-hash + tracked
symbol->CIK fingerprint, so a newly added ticker forces a run instead of
being no_op'd/starved. No backfill sentinel — absence of a prior promoted
run triggers backfill; source_max_date records the processed index date.
2. Full history needs the paginated submissions shards: filings.recent caps at
1000; older accessions (reportDate/acceptanceDateTime/isXBRL) live in
filings.files[] shards (verified on Apple: recent=1000, one 1994-2015 shard).
3. Index<->Company-Facts consistency gate: they are separate SEC products that
can lag; for every tracked isXBRL index accession, confirm it exists in
Company Facts before promotion, else fail+retry (never record a null/partial
snapshot). Non-XBRL amendments skipped with a recorded reason.
4. Immutable = insert-only (ON CONFLICT DO NOTHING); a differing re-fetch is a
reported discrepancy, never a silent mutation / import_run_id replacement.
Plus deterministic, mutually-exclusive cash/debt composition (aggregate-first;
each source tag counted at most once).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
All three decisions approved: fetch via EDGAR daily-index (not bulk zip),
one snapshot per accession for its primary period (comparative-only
restatements out of scope), full-history backfill on first run. Doc status
flipped to approved / ready to implement.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Design (not implementation) for phase A3, grounded in live SEC data probes.
Key findings: fp has no Q4 (derive it); fy/fp are the filing's context not each
fact's period (select by end==reportDate); SEC provides both discrete and YTD
facts (confirms stored-YTD schema); companyfacts endpoint has no ETag/
Last-Modified (conditional GET impossible); tickers are dash-form and GOOG/GOOGL
share one CIK.
Two plan deviations flagged for sign-off:
1. Fetch via the EDGAR daily-index (fetch companyfacts only for tracked issuers
that filed) rather than the multi-GB bulk zip — lighter and restores the
revision/no_op model.
2. One snapshot row per accession for its primary period (YTD-cumulative);
comparative-only restatements out of scope (only real 10-K/A updates a period).
Plus a metric tag catalog, read-time derivation rules (missing period -> null),
CIK resolution, validation gates, and SEC fair-access handling.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Addresses the A2 review:
1. Every dolt subprocess is now bounded by a hard timeout
(dolt_command_timeout_seconds, default 600s); on expiry the process is killed
and DoltError raised — a hung pull/sql can no longer pin the import
connection and advisory lock indefinitely. Tested (timeout + non-zero exit).
2. Initial-load validate is stronger: besides zero-future, an initial load now
requires a real forward horizon (>= 21d, under the ~35d observed on the
clone) AND universe coverage >= 50% (a broken symbol join can't seed a hollow
calendar). Subsequent runs keep the 50% collapse gate.
3. Revision uses DOLT_HASHOF('HEAD') — formally HEAD, not dolt_log-by-timestamp.
4. Free-disk floor raised 2 GB -> 5 GB (safe headroom over the ~1.7 GB clone).
Full suite 702 passed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A1 carry-forward. The SEC cover-page share count
(dei:EntityCommonStockSharesOutstanding) is reported "as of" its own date, which
can differ from the fiscal period_end — store that date so market cap uses the
right point-in-time count. Migration 026 edited in place (never run with data).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A SourceImporter that ingests post-no-preference/earnings into earnings_events
for the tracked universe. Shadow by construction (nothing reads earnings_events
until A4).
- earnings_alignment.py: pure calendar<->EPS-history min-cost monotonic DP,
reused from scripts/import_dolthub_earnings.py with identical constants (not
extending that one-off script); symbol/session normalization; unit-tested
against the pinned constants.
- dolt_client.py: async dolt CLI wrapper (pull / current_commit / query_csv via
asyncio.create_subprocess_exec — never blocks the shared event loop) + disk
guard before pull.
- dolt_earnings_importer.py: detect_revision = pull + HEAD hash; stage = query
earnings_calendar + eps_history, dedup, align, map act_symbol->ticker_id
(normalize both sides so dotted BRK.B joins); promote is destructive
(delete future dolt_earnings rows + upsert; past never deleted) so validate is
FAIL-CLOSED — blocks when the staged forward calendar is empty or has collapsed
below 50% of what's loaded (the forward calendar is the acceptance gate).
- NOTICE: CC BY-SA 4.0 attribution; config: DOLT_BINARY / DOLT_DATA_DIR / etc.
Verified end-to-end against the real 1.68 GB clone (5 tickers: 133 events, 128
paired, forward calendar to 2026-08-26, BRK.B joined). Tests: 9 alignment + 7
importer + 1 skip-guarded real-clone smoke. Full suite 699 passed.
Remaining for A2: wire the daily ~02:30 ET shadow cron — deferred to pair with
the deploy-time dolt install + DOLT_DATA_DIR provisioning.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Both real target tables (fundamental_snapshots, earnings_events) carry an
import_run_id; stamping requires the current run's id. A2 (the earnings
importer) is the first real consumer, so promote gains a run_id argument rather
than having importers hack the running row out of the framework. Protocol +
call site updated; the A1 fake importer now stamps and asserts import_run_id.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Local dev uses a Dolt clone of post-no-preference/earnings under dolt-data/
(git-ignored). Production keeps clones in DOLT_DATA_DIR outside the repo tree —
the deploy is rsync --delete of the tree, so a clone inside it would be unsafe.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
post-no-preference/earnings approved for private/internal ingestion under
CC BY-SA 4.0. Conditions the A2 importer must honor: preserve upstream license /
attribution / transformation notes; no public API, bulk export, or
redistribution; re-review before any public or commercial access. The stocks
repo (workstream B) is not covered and will be reviewed separately if B begins.
A0 rollout item marked done (dolt binary pin + DOLT_DATA_DIR still pending at
deploy time).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Review items 3-4 on the plan doc:
- Replace remaining "diluted shares" (the share *count*) with point-in-time
shares_outstanding (dei:EntityCommonStockSharesOutstanding) across schema,
metrics catalog and market-cap note. "diluted EPS" is left as-is (correctly a
duration fact). Adds the multi-class rule: derive the issuer-wide count from
the consolidated cover-page figure OR by summing class-specific facts (GOOG +
GOOGL) — never both, to avoid double counting.
- The framework stages into a representation *outside the live tables* (in-memory
for workstream A; a file/table handle is fine if B needs it), not physical
"staging tables" — wording now matches the implementation.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Review fixes to the import-run framework (A1):
1. detect_revision ran outside the failure handler, so a failed revision probe
(the most likely external failure) escaped unrecorded — violating "every
attempt is recorded". Now the running row is created FIRST, then
detect_revision + last-revision lookup + stage + validate + promote all run
inside the same handler; the row converts to no_op when the revision is
unchanged. New test covers a detection exception → recorded failed + alert.
2. asyncio.CancelledError (BaseException, not caught by except Exception) left a
permanent running row on deploy/scheduler shutdown. Now caught explicitly:
best-effort mark failed, then re-raise the cancellation (never swallowed).
New test asserts the run is failed and the error re-propagates.
Full suite 682 passed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
run_import + SourceImporter Protocol (detect_revision/stage/validate/promote)
giving every bulk importer the plan's non-negotiables, KISS:
- one run per source at a time — Postgres session-level advisory lock held on a
single pinned engine.connect() so it survives the running-row and promotion
commits; no-op on SQLite.
- idempotent per revision — cheap detect_revision compared to the last promoted
run; unchanged revision records a no_op with zero writes (no fetch).
- staging (in-memory, no physical staging tables) → validate (read-only) →
atomic promote + run-row flip in one transaction.
- failed validation or mid-run exception marks the run failed, alerts via
system_event_service, and leaves live tables untouched.
Every attempt recorded in data_import_runs; conflicts summary in validation_json
(no conflicts table). Concrete SEC/earnings importers land in later phases.
Tests: 6 orchestration tests (no_op / promote / new-revision / failed-untouched
/ promote-exception-rollback) + deterministic advisory-key derivation. Full
suite 680 passed. Advisory-lock mutual exclusion is PG-verify-pending (SQLite
no-ops it — flagged, not covered).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
First reviewable slice of workstream A: schema only, no importers, no data.
- data_import_runs: lean batch-import audit (source/revision/status,
row_counts_json + validation_json as Text-holding-JSON per repo convention).
- fundamental_snapshots: CIK-keyed, one immutable row per accession; stores
per-period raw facts (duration = cumulative YTD/FY, balance-sheet =
period-end) plus period_start/period_end/fiscal_year/fiscal_period so
discrete quarters, Q4, TTM and YoY are derived at read time.
- earnings_events: Dolt-sourced calendar + surprise history, unique
(ticker_id, announce_date).
- tickers: nullable cik/sic/sic_description — the ticker<->issuer join point.
fundamental_data is left untouched (cutover gated separately at A5). Models
registered in app/models/__init__.py; Ticker gains an earnings_events
relationship. Verified: create_all builds the tables, mappers configure, and
migration 026 renders valid Postgres DDL up and down.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds the Dolt bulk-data integration hand-off plan (authored in prior
session) and applies clarifications found in a pre-handoff review:
- step (c): fields refresh from three distinct sources, not "snapshots +
close" — earnings_surprise/next_earnings_date come from earnings_events,
not SEC facts. pe_ratio = close / TTM diluted EPS and market_cap =
issuer-wide diluted shares × close stated as separate formulas.
- fundamental_snapshots made implementable: add period_start, fiscal_year,
fiscal_period; duration facts store the filing's cumulative YTD/FY values,
balance-sheet facts store period-end values. Discrete quarters, Q4, TTM
and YoY are derived at read time (correct for non-calendar fiscal years;
amendments never freeze a stale derived quarter).
- flag Q4 derivation / fiscal-period alignment as the primary A3 risk.
- anchor "tracked universe" to ticker_universe_service + per-run CIK
resolution.
All code anchors in the doc verified accurate against the current tree.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
