Operational plumbing to land the parser fixes and to make silent resolution
failures visible.
- Reparse: SecFundamentalsImporter(reparse=True) restages every accession with
the current parser and rewrites the ones that now reconstruct differently,
writing the full column set so a row is never half old-parse. Snapshots stay
immutable with respect to SEC; the stored row is our reconstruction, and after
a parser fix keeping it is a stale cache, not history. run_import(force=True)
bypasses the unchanged-revision no-op, since the staleness is on our side, not
the source's. Exposed as scripts/reparse_fundamentals.py, dry-run by default.
- CIK overrides: sec_universe reads a {symbol: cik} pin from
SystemSetting['sec_cik_overrides'], applied ahead of company_tickers.json, for
when SEC maps a ticker to a successor shell with no filings (XOM -> a zero-
filing "ExxonMobil Holdings Corp" while every 10-K/Q is under CIK 34088).
- Resolution validation: a tracked issuer resolving to a registrant with no XBRL
filings now records no_xbrl_filings and raises a warning naming the CIKs and
the override setting, instead of silently yielding nothing on every run.
- _diff_fields compares datetime instants, not representations: accepted_at
round-trips naive from SQLite but tz-aware from Postgres, which otherwise made
a reparse of identical data report every row as changed (and false-positived
the pre-existing discrepancy warning).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The A5 parity report surfaced coverage gaps and wrong values that all traced
to the SEC facts parser and read-time derivation rather than to bad source
data. Fixes, each validated by replaying the production parser + derivation
against live company facts:
- Period identity is derived from period_end against the issuer's fiscal
calendar, not SEC's fy/fp fields, which collide (two period ends on one key,
one silently discarded) and invert (a period sorting before one that precedes
it) often enough to break the quarter chain. Recovers BXP, CRM, CRWD, FRT,
MTD, NTAP, PPL, STX, WDAY. Fixed labels are internal ordering keys only (not
in any API schema), so a filer whose year ends in early January shifting by
one is harmless.
- Revenue concept list gains RevenuesNetOfInterestExpense (banks) and the
IncludingAssessedTax variant (REITs/consumer); EPS gains the continuing-ops
variant (REG/FCX) and, last, basic EPS for a period tagging no diluted
variant at all (PPL). All appended, so any issuer that already resolved keeps
its concept.
- YTD span tolerance 20 -> 25 days, covering 4-4-5 retail calendars whose
36-week YTD-Q3 (251-252d) previously missed by ~2 (COST, PEP, DPZ).
- Amendment resolution is per field: a partial 10-K/A (Part III only, no
financial facts) no longer blanks the period (DVN).
- TTM diluted EPS is suppressed when a split contaminates the trailing window
(BKNG's mixed-unit sum produced a P/E of 1.10 that clamped to a perfect
fundamental sub-score). A post-filing split with no share-count evidence
(KLAC) remains undetectable from this data.
- Multi-class share fallback: weighted_avg_diluted_shares is captured and used
for market cap when the cover-page count is absent (dimensional, so missing
from company facts for META/CMCSA/CHTR/FOXA/NWSA/LEN). Within ~0.6% of the
true count on controls; flagged shares_estimated in the API. BRK-B has no
weighted-average fact either and stays unavailable.
820 unit tests pass; new tests confirmed to fail against the pre-fix code.
Effect is inert until existing rows are reparsed (see reparse path).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
max-w-md (448px) under-sized the panel vs its real full-width tab placement and
never triggered the desktop two-column layout. Widen to max-w-3xl; note to
resize to ~390px for the mobile (single-column) check.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the four-cell quarter tape (too many equally-weighted numbers) with one
comparison rail per metric, so the panel answers "improving? sound? fairly
valued?" instead of asking the reader to decode it.
- Operating trend (revenue/EPS growth, operating/FCF margin, share count): a rail
centered on a truthful reference — prior quarter (growth), prior-period average
(margins), or zero (share count) — with a dot at the current delta and a bar
back to the reference, plus a shaded neutral band (backend's +-2pp / +-1pp /
+-1% rules). Value, read, reference label, and signed delta stay visible;
per-quarter history drops out of the default view.
- Valuation & balance (net debt/EBITDA, P/E, FCF yield): a 0-100 favorable-
percentile rail with the peer median fixed at 50; right is always more
favorable (percentile is polarity-aware). median + peer_count shown.
- Not a progress bar: reference line, not a 100% target.
- Horizon tokens: cyan #6EC9DB favorable / coral #EF9182 adverse / #5D6373 track,
replacing emerald/rose. Two columns on desktop, single column (rows stack) on
mobile. Null -> n/a with no rail; insufficient peers -> "peers n/a", no track.
- Kept: compact earnings line, provenance footer, local-date parsing, aria-labels
on every rail. tsc -b passes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Addresses the static review + adds a dev-only visual harness:
1. Tape no longer overflows narrow mobile: each row stacks (label + read on one
line, cells below) under sm, keeping the single-line grid on desktop.
2. Date-only strings (earnings, price_date) are parsed as LOCAL calendar dates,
so a viewer west of UTC no longer sees the previous day.
3. Peer context is visible ("med X · Np") on every width and the percentile
strip carries a full aria-label — no longer hover-only / desktop-only.
4. Per-metric provenance + freshness surfaced (SEC filings · latest quarter,
filed date) replacing the removed panel-wide FMP label.
5. Same-day earnings render "today", not "in 0d".
Harness: frontend/harness.html + src/dev/harness.tsx (dev-only, served at
/harness.html by vite, not in the production build) render full /
partial-insufficient-peer / empty fixtures for desktop + ~390px review.
tsc -b passes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reshapes FundamentalsPanel to consume the additive API v1, within the app's
existing dark-glass language.
- types.ts updated to the exact v1 shape (metrics/earnings/valuation/reads +
legacy fields preserved).
- The quarter tape is the single distinctive device: per-metric 4-cell tape
(revenue/EPS growth, operating + FCF margin, share count) with the latest cell
toned by the deterministic read; color is always paired with the read text.
- Restrained peer strips for Net debt/EBITDA, P/E, FCF yield: value + a
polarity-aware percentile bar with a median marker + the read; hidden ("peers
n/a") when industry is null (< 5 peers).
- Earnings: next date/session/countdown + last-N beat/miss arrows (▲/▼/·) with
text aria-labels; explicit "no date" state.
- Explicit n/a, insufficient-peer, and no-earnings states; header shows the
deterministic sentence. Removed the hard-coded "FMP" source label.
Frontend tsc -b passes; backend suite 778 passed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- The eps_growth_yoy read was never computed, leaving that fixed by_key entry
null even with sufficient EPS history; now growth_read() is applied to EPS
history just like revenue.
- Tests: same-day earnings returns as next with days_until 0 (and not in
recent); zero close guards valuation to null; eps read populated. Fixture
seeds three fiscal years so YoY growth reads have a >=3 run. 9 API tests pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1. Multi-class subject is priced by the REQUESTED ticker: the peer group's
representative for the subject CIK is overridden to the requested ticker_id
(other issuers pick a deterministic-by-symbol rep), so GOOGL's P/E uses
GOOGL's price, not GOOG's. Differing-price GOOG/GOOGL test added.
2. reads matches the selected contract: header is null when there is no read;
by_key is a fixed map over every metric key plus pe and fcf_yield, null when
unavailable (was a sparse dict).
3. Earnings use the New York calendar date; same-day is UPCOMING (days_until 0),
recent is strictly earlier.
4. Valuation is null when there is no usable price (> 0 required for P/E and
market cap); when present, price_date is non-null.
Added a real router/API-envelope test with a seeded legacy record (the endpoint,
not just the schema merge). 6 tests pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
GET /fundamentals/{symbol} now returns the additive v1 objects alongside the
unchanged legacy fields (no legacy growth mapped onto the SEC TTM metric).
- earnings: next (date/session/days_until) + recent (<=4, with surprise_pct)
from earnings_events.
- metrics: fixed key set (value + dated history + per-metric SIC-peer industry
object + source=sec); net_debt has no industry (size-dependent).
- valuation: P/E, FCF yield, market_cap_est computed at REQUEST TIME from the
derived TTM inputs x the latest ohlcv close (no stored valuation); guarded to
null on missing/invalid inputs; pe_industry / fcf_yield_industry peer stats.
- reads: deterministic outputs in a SEPARATE object (header + per-metric reads).
Peer queries are batched and CIK-deduplicated by 2-digit SIC; industry omitted
below 5 valid peers. Schema extended with optional typed sub-models; the router
merges legacy + v1 so every existing field is preserved.
Tests: 4 (full assembly incl. peer industry + valuation + additive-merge, no-cik
null metrics, <5-peers omitted, price-guarded valuation).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1. Peer percentile is now a tie-aware rank against the OTHER issuers
((worse + 0.5*tied)/(peers-1)): an all-equal group maps to 50 (not 100), the
median maps to 50, a unique best to 100, a unique worst to 0.
2. Deterministic reads use the consecutive non-null suffix ending at the latest
point (>=3 values): a null latest or an internal gap yields no read, so a read
never reflects a period displayed as n/a.
3. Peer filtering excludes non-finite (NaN/±inf) as well as null, including an
invalid subject.
Tests updated + added (all-equal, median rank, non-finite, latest-null history,
internal gap). 15 passed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Completes the pure read-time core.
fundamentals_peers.py: median + polarity-aware favorable percentile + peer_count
for a subject within its SIC group (CIK-deduped by the caller); returns None
below MIN_PEERS=5 so the caller omits the industry object. Absolute net_debt is
intentionally NOT peer-eligible (size-dependent) — leverage compares via
net_debt_to_ebitda. HIGHER_IS_BETTER polarity map + two_digit_sic() grouping key.
fundamentals_reads.py: one shared deterministic rule set (no LLM): growth_read
(+-2pp), margin_read (latest vs mean-of-prior, +-1pp), share_count_read (+-1%),
peer_read (60/40 bands, polarity-aware phrasing per metric), header_sentence
(growth · margins · valuation, omitting empty). Tunable named constants; >=3
periods required for a series read.
Tests: 8 peer + 5 reads, anchored on the boundary cases (exactly +2.0pp, exactly
60th percentile, exactly +1.0pp margin). 13 passed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1. net_debt requires BOTH cash and total_debt; a missing side is null, not
treated as zero (which would be a partial, misleading value).
2. net_debt_to_ebitda is null when TTM EBITDA <= 0 — a negative denominator
would otherwise rank a distressed issuer as favorably low-leverage.
3. The quarter tape is the CONSECUTIVE run ending at the latest period (stops at
a gap), so trend text never compares non-adjacent quarters as if consecutive.
4. YoY growth is null when the prior-year TTM is <= 0 (e.g. loss->profit), which
is not a meaningful percentage.
Also corrected the plan's net-debt formula to total debt − (cash + ST) matching
the positive-means-net-debt implementation. +4 tests. 10 passed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Derives the display metrics from the stored YTD snapshots at read time (no I/O,
no DB), per the A3 schema decision. Given an issuer's snapshot rows it produces:
- amendment selection (newest accepted_at per fiscal period);
- discrete quarters = YTD(Qn) - YTD(Qn-1), Q4 = YTD(FY) - YTD(Q3);
- TTM = trailing four discrete quarters; missing period -> null, never partial;
- metric series (value + 4-quarter tape, each point dated): revenue_growth_yoy,
eps_growth_yoy, operating_margin, fcf_margin, net_debt, net_debt_to_ebitda,
share_count_change_yoy;
- request-time valuation inputs (ttm_diluted_eps, ttm_fcf, shares_outstanding)
for the API to combine with price.
Units per app convention (percentages = pp, leverage = multiple, dollars).
Tests: 6 (growth+Q4, margins, net-debt/EBITDA+dilution, valuation inputs,
missing-period-null, amendment selection). Verified on real Apple snapshots:
op margin 32.6%, net-debt/EBITDA 0.10, buyback -1.7%/yr, TTM EPS $8.26.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Extend the companyfacts structural check to reject a concept with a
missing/non-dict `units` mapping (not just the top-level `facts`), so a
partially-malformed payload fails promotion instead of silently dropping that
concept's facts. New fixture proves it fails.
- Strengthen the newly-added-issuer test: keep latest_index equal to the prior
run so ONLY the universe fingerprint changes the revision — proving the
fingerprint alone prevents a new ticker from being starved/no_op'd.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1. Removed the 45-day index-walk cap: it discarded the older part of a long
outage while still advancing source_max_date, permanently losing filings.
The walk now covers every unprocessed date (a large gap is one-time cost).
2. Discrepancy detection meets the immutability contract: it compares ALL source
snapshot fields (not five), read-only during stage/validate, reports the
differing accessions + fields in validation_json, and promote emits a warning
system event (in-transaction) — never mutating the stored row.
3. Malformed companyfacts (missing facts/units structure) are recorded separately
and FAIL validation, instead of silently degrading to skipped rows that the
50% backfill coverage floor could still pass.
Also corrected the stale "sum share classes" / DEI-only wording in the snapshot
model docstring and the A3 design doc to describe the us-gaap fallback.
Tests: +4 regressions (>45-day gap loses nothing, newly-added issuer backfills
without filing, malformed payload fails, shares discrepancy detected + evented).
23 passed, 1 skipped.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
SecFundamentalsImporter (SourceImporter, source=sec_facts): populates immutable
fundamental_snapshots from Company Facts and back-fills tickers.cik/sic, driven
by the EDGAR daily index. Shadow only. Guardrails per review:
- detect_revision caches the resolved universe + exact tracked index rows and
composes the revision from them; stage consumes those same cached inputs
(no index/universe refetch) so promoted data matches the computed revision.
- Resolution is read-only in stage (proposals only); ticker writes happen in
promote via apply_ticker_updates.
- validate runs the index<->Company-Facts consistency gate before any write:
a tracked XBRL index accession missing from Company Facts fails the run
(they lag independently) so we retry, not record null. Non-XBRL amendments
are skipped with a recorded reason. Backfill has a coverage floor.
- promote inserts ON CONFLICT (accession) DO NOTHING (immutable), reports
differing existing accessions without mutating, and applies ticker updates in
the same transaction.
- Full-history backfill on first run / for newly-added issuers (include_history);
incremental fetch only for issuers that filed.
Parser: split parse result into skipped_filings vs field_issues (coverage must
not count field warnings); header notes the us-gaap shares fallback; added
companyfacts_accessions() for the gate.
Verified live end-to-end (AAPL + GOOGL backfill): 112 snapshots, cik/sic set,
GOOGL shares via us-gaap fallback, AAPL via dei. Tests: 6 importer (backfill,
incremental, consistency-gate fail, non-XBRL skip, read-only-on-failure,
conflict-discrepancy) + parser ParseResult updates.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1. Multi-class shares: prefer the single dei:EntityCommonStockSharesOutstanding
cover-page fact; else fall back to us-gaap:CommonStockSharesOutstanding at
period end (Alphabet has no dei fact). Never sum class facts (companyfacts is
non-dimensional) and never use weighted-average/diluted; conflicting values ->
null, counted as an "ambiguous shares outstanding" note in validation. Plan's
"sum class-specific" wording corrected. Verified live: Alphabet shares now
populate (12.1B), Apple still uses its dei cover date.
2. Fiscal context is the majority (fy, fp) among facts ending at reportDate, with
ties rejected — no longer the arbitrary first fact.
3. Hardening: catalog selectors require taxonomy == "us-gaap"; indexing drops
malformed facts (missing accession/end, non-finite value) so a custom concept
or bad date can't be selected.
Tests: +8 (dei precedence, us-gaap fallback, conflict->null, no weighted-average,
tie-context skip, foreign-taxonomy/malformed ignored, ambiguous-shares note).
14 passed, 1 skipped.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Pure parser (no I/O/DB) turning one issuer's companyfacts + submissions filing
metadata into per-accession snapshot rows for the filing's primary period.
- Period identity from end == reportDate, never fy/fp (fy/fp is the filing's
context; comparatives repeat it).
- Duration facts stored as cumulative YTD: pick the fact whose span matches the
fiscal-period-to-date length (Q1~3mo..FY~12mo) within tolerance; no YTD-length
fact -> null (never a discrete masquerading as YTD).
- Balance-sheet instants at end == reportDate; shares_outstanding is the dei
cover-page fact whose own end (cover date) is stored in shares_outstanding_date.
- Cash and debt composites are aggregate-first and mutually exclusive (each
source tag counted at most once).
- Carries filing_date through submissions rows (snapshot.filed_date).
Verified on REAL Apple companyfacts: 44 snapshots, 0 skipped, YTD revenue
124.3B->219.7B->313.7B->416.2B across FY2025 (Q4 derives at read time), every
shares_date is the cover date != period_end. Tests: 6 fixture + 1 skip-guarded
live-invariants (monotonic YTD, cover-date shares).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Addresses the slice-1 review:
1. Resolution is now read-only (A1 transaction contract). resolve_ciks /
fetch_sic_updates compute proposals and mutate nothing; a new
apply_ticker_updates issues the writes, called only in promote — so a failed
validation can't leak ticker changes on the framework's failure commit.
2. Only 404 means "missing". Added SecNotFoundError; daily_index /
latest_index_date catch only that. 403, exhausted 429, 5xx, timeouts, and
transport/parse errors now propagate instead of looking like "no index".
3. Fair-access enforced when opening a REAL client (transport=None): reject
blank/placeholder/non-email User-Agent and sub-0.11s spacing. Mock transports
skip it (tests use 0 spacing).
4. submissions(include_history=False) by default — only the one-time full
backfill fetches the history shards; SIC/incremental work makes no extra
requests.
Plus: retry transient 5xx/network errors and honor Retry-After during the 1 GB
backfill; compose_revision rejects a missing index date (no "None:..." revision).
Re-verified live vs real SEC (fair-access validation passes, shard merge intact).
Tests: 18 (added error propagation, read-only resolution, fair-access, recent-only
submissions, reject-None revision).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
First A3 implementation checkpoint (design: docs/dolt-sec-a3-design.md).
- sec_client.py: async SEC EDGAR client honoring fair-access — identifying
User-Agent (config), request spacing < 10 req/s, exponential backoff on 429,
and 403 -> SecForbiddenError (alert and stop, never retry-loop). Fetchers:
company_tickers (normalised, multi-class share CIK), submissions (merges the
paginated filings.files shards so full history is visible), companyfacts,
daily_index (fixed-width form.idx parse), latest_index_date.
- sec_universe.py: resolve_ciks (tickers.cik backfill), refresh_sic
(sic/sic_description), and the composite-revision pieces — universe_fingerprint
(a new ticker changes the revision, so it's never no_op'd/starved),
index_content_hash, compose_revision.
- config + .env.example: SEC_USER_AGENT (must be a real contact email) + spacing
/ retries / timeout.
Verified live against real SEC: AAPL->320193, GOOG==GOOGL, BRK-B resolved;
submissions shard-merge proven (131 filings back to 1993); daily index parsed.
Tests: 11 (mocked-transport parsing + 403/429 handling + resolution/fingerprint).
Full suite 713 passed. Next slice: companyfacts -> snapshot parser + importer.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fold in the A3 design review:
1. Composite revision = latest-index-date + index-content-hash + tracked
symbol->CIK fingerprint, so a newly added ticker forces a run instead of
being no_op'd/starved. No backfill sentinel — absence of a prior promoted
run triggers backfill; source_max_date records the processed index date.
2. Full history needs the paginated submissions shards: filings.recent caps at
1000; older accessions (reportDate/acceptanceDateTime/isXBRL) live in
filings.files[] shards (verified on Apple: recent=1000, one 1994-2015 shard).
3. Index<->Company-Facts consistency gate: they are separate SEC products that
can lag; for every tracked isXBRL index accession, confirm it exists in
Company Facts before promotion, else fail+retry (never record a null/partial
snapshot). Non-XBRL amendments skipped with a recorded reason.
4. Immutable = insert-only (ON CONFLICT DO NOTHING); a differing re-fetch is a
reported discrepancy, never a silent mutation / import_run_id replacement.
Plus deterministic, mutually-exclusive cash/debt composition (aggregate-first;
each source tag counted at most once).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
All three decisions approved: fetch via EDGAR daily-index (not bulk zip),
one snapshot per accession for its primary period (comparative-only
restatements out of scope), full-history backfill on first run. Doc status
flipped to approved / ready to implement.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Design (not implementation) for phase A3, grounded in live SEC data probes.
Key findings: fp has no Q4 (derive it); fy/fp are the filing's context not each
fact's period (select by end==reportDate); SEC provides both discrete and YTD
facts (confirms stored-YTD schema); companyfacts endpoint has no ETag/
Last-Modified (conditional GET impossible); tickers are dash-form and GOOG/GOOGL
share one CIK.
Two plan deviations flagged for sign-off:
1. Fetch via the EDGAR daily-index (fetch companyfacts only for tracked issuers
that filed) rather than the multi-GB bulk zip — lighter and restores the
revision/no_op model.
2. One snapshot row per accession for its primary period (YTD-cumulative);
comparative-only restatements out of scope (only real 10-K/A updates a period).
Plus a metric tag catalog, read-time derivation rules (missing period -> null),
CIK resolution, validation gates, and SEC fair-access handling.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Addresses the A2 review:
1. Every dolt subprocess is now bounded by a hard timeout
(dolt_command_timeout_seconds, default 600s); on expiry the process is killed
and DoltError raised — a hung pull/sql can no longer pin the import
connection and advisory lock indefinitely. Tested (timeout + non-zero exit).
2. Initial-load validate is stronger: besides zero-future, an initial load now
requires a real forward horizon (>= 21d, under the ~35d observed on the
clone) AND universe coverage >= 50% (a broken symbol join can't seed a hollow
calendar). Subsequent runs keep the 50% collapse gate.
3. Revision uses DOLT_HASHOF('HEAD') — formally HEAD, not dolt_log-by-timestamp.
4. Free-disk floor raised 2 GB -> 5 GB (safe headroom over the ~1.7 GB clone).
Full suite 702 passed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A1 carry-forward. The SEC cover-page share count
(dei:EntityCommonStockSharesOutstanding) is reported "as of" its own date, which
can differ from the fiscal period_end — store that date so market cap uses the
right point-in-time count. Migration 026 edited in place (never run with data).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A SourceImporter that ingests post-no-preference/earnings into earnings_events
for the tracked universe. Shadow by construction (nothing reads earnings_events
until A4).
- earnings_alignment.py: pure calendar<->EPS-history min-cost monotonic DP,
reused from scripts/import_dolthub_earnings.py with identical constants (not
extending that one-off script); symbol/session normalization; unit-tested
against the pinned constants.
- dolt_client.py: async dolt CLI wrapper (pull / current_commit / query_csv via
asyncio.create_subprocess_exec — never blocks the shared event loop) + disk
guard before pull.
- dolt_earnings_importer.py: detect_revision = pull + HEAD hash; stage = query
earnings_calendar + eps_history, dedup, align, map act_symbol->ticker_id
(normalize both sides so dotted BRK.B joins); promote is destructive
(delete future dolt_earnings rows + upsert; past never deleted) so validate is
FAIL-CLOSED — blocks when the staged forward calendar is empty or has collapsed
below 50% of what's loaded (the forward calendar is the acceptance gate).
- NOTICE: CC BY-SA 4.0 attribution; config: DOLT_BINARY / DOLT_DATA_DIR / etc.
Verified end-to-end against the real 1.68 GB clone (5 tickers: 133 events, 128
paired, forward calendar to 2026-08-26, BRK.B joined). Tests: 9 alignment + 7
importer + 1 skip-guarded real-clone smoke. Full suite 699 passed.
Remaining for A2: wire the daily ~02:30 ET shadow cron — deferred to pair with
the deploy-time dolt install + DOLT_DATA_DIR provisioning.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Both real target tables (fundamental_snapshots, earnings_events) carry an
import_run_id; stamping requires the current run's id. A2 (the earnings
importer) is the first real consumer, so promote gains a run_id argument rather
than having importers hack the running row out of the framework. Protocol +
call site updated; the A1 fake importer now stamps and asserts import_run_id.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Local dev uses a Dolt clone of post-no-preference/earnings under dolt-data/
(git-ignored). Production keeps clones in DOLT_DATA_DIR outside the repo tree —
the deploy is rsync --delete of the tree, so a clone inside it would be unsafe.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
post-no-preference/earnings approved for private/internal ingestion under
CC BY-SA 4.0. Conditions the A2 importer must honor: preserve upstream license /
attribution / transformation notes; no public API, bulk export, or
redistribution; re-review before any public or commercial access. The stocks
repo (workstream B) is not covered and will be reviewed separately if B begins.
A0 rollout item marked done (dolt binary pin + DOLT_DATA_DIR still pending at
deploy time).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Review items 3-4 on the plan doc:
- Replace remaining "diluted shares" (the share *count*) with point-in-time
shares_outstanding (dei:EntityCommonStockSharesOutstanding) across schema,
metrics catalog and market-cap note. "diluted EPS" is left as-is (correctly a
duration fact). Adds the multi-class rule: derive the issuer-wide count from
the consolidated cover-page figure OR by summing class-specific facts (GOOG +
GOOGL) — never both, to avoid double counting.
- The framework stages into a representation *outside the live tables* (in-memory
for workstream A; a file/table handle is fine if B needs it), not physical
"staging tables" — wording now matches the implementation.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Review fixes to the import-run framework (A1):
1. detect_revision ran outside the failure handler, so a failed revision probe
(the most likely external failure) escaped unrecorded — violating "every
attempt is recorded". Now the running row is created FIRST, then
detect_revision + last-revision lookup + stage + validate + promote all run
inside the same handler; the row converts to no_op when the revision is
unchanged. New test covers a detection exception → recorded failed + alert.
2. asyncio.CancelledError (BaseException, not caught by except Exception) left a
permanent running row on deploy/scheduler shutdown. Now caught explicitly:
best-effort mark failed, then re-raise the cancellation (never swallowed).
New test asserts the run is failed and the error re-propagates.
Full suite 682 passed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
run_import + SourceImporter Protocol (detect_revision/stage/validate/promote)
giving every bulk importer the plan's non-negotiables, KISS:
- one run per source at a time — Postgres session-level advisory lock held on a
single pinned engine.connect() so it survives the running-row and promotion
commits; no-op on SQLite.
- idempotent per revision — cheap detect_revision compared to the last promoted
run; unchanged revision records a no_op with zero writes (no fetch).
- staging (in-memory, no physical staging tables) → validate (read-only) →
atomic promote + run-row flip in one transaction.
- failed validation or mid-run exception marks the run failed, alerts via
system_event_service, and leaves live tables untouched.
Every attempt recorded in data_import_runs; conflicts summary in validation_json
(no conflicts table). Concrete SEC/earnings importers land in later phases.
Tests: 6 orchestration tests (no_op / promote / new-revision / failed-untouched
/ promote-exception-rollback) + deterministic advisory-key derivation. Full
suite 680 passed. Advisory-lock mutual exclusion is PG-verify-pending (SQLite
no-ops it — flagged, not covered).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
First reviewable slice of workstream A: schema only, no importers, no data.
- data_import_runs: lean batch-import audit (source/revision/status,
row_counts_json + validation_json as Text-holding-JSON per repo convention).
- fundamental_snapshots: CIK-keyed, one immutable row per accession; stores
per-period raw facts (duration = cumulative YTD/FY, balance-sheet =
period-end) plus period_start/period_end/fiscal_year/fiscal_period so
discrete quarters, Q4, TTM and YoY are derived at read time.
- earnings_events: Dolt-sourced calendar + surprise history, unique
(ticker_id, announce_date).
- tickers: nullable cik/sic/sic_description — the ticker<->issuer join point.
fundamental_data is left untouched (cutover gated separately at A5). Models
registered in app/models/__init__.py; Ticker gains an earnings_events
relationship. Verified: create_all builds the tables, mappers configure, and
migration 026 renders valid Postgres DDL up and down.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds the Dolt bulk-data integration hand-off plan (authored in prior
session) and applies clarifications found in a pre-handoff review:
- step (c): fields refresh from three distinct sources, not "snapshots +
close" — earnings_surprise/next_earnings_date come from earnings_events,
not SEC facts. pe_ratio = close / TTM diluted EPS and market_cap =
issuer-wide diluted shares × close stated as separate formulas.
- fundamental_snapshots made implementable: add period_start, fiscal_year,
fiscal_period; duration facts store the filing's cumulative YTD/FY values,
balance-sheet facts store period-end values. Discrete quarters, Q4, TTM
and YoY are derived at read time (correct for non-calendar fiscal years;
amendments never freeze a stale derived quarter).
- flag Q4 derivation / fiscal-period alignment as the primary A3 risk.
- anchor "tracked universe" to ticker_universe_service + per-run CIK
resolution.
All code anchors in the doc verified accurate against the current tree.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The shadow book toggle saved on change and the numeric fields on blur,
inconsistent with every other admin panel (which stages edits behind a
Save button). Stage all four fields in local state and commit them
together on Save, with an unsaved-changes hint. This also makes enabling
the live-trade toggle a deliberate two-step action rather than an
unguarded single click.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The run-id marker proved which scan wrote last, but the shadow book still
selected setups by detected_at >= scan_start. An overlapping manual scan
could insert rows in that same window; if the pipeline's scan wrote the
marker last its id matched and the shadow book proceeded, then swept in --
or ranked highest -- a manual-scan row. The identity check gated entry but
selection did not.
Carry the run id onto the rows. Migration 025 adds an indexed
trade_setups.scan_run_id. scan_all_tickers computes one id per run
(pipeline's when a step, else fresh), passes it to scan_ticker which stamps
every row after enhancement, and writes the same id to the completion
marker. The shadow book selects WHERE scan_run_id == the matched id, so a
concurrent scan's rows are excluded by identity regardless of their
detected_at. The now-unused STARTED marker is dropped; COMPLETED
(freshness) and RUN_ID (identity) remain.
Decisive test: the pipeline's id matches, but a same-window manual row with
a higher rank is present and is excluded -- only the pipeline's own row is
traded. A time-window select would have swept it in and ranked it first.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
