CI only lints app/, so 11 findings had accumulated in tests/ and scripts/.
Mechanical and behaviour-neutral, but two were not auto-fixable and needed a
judgement call rather than `ruff --fix`:
- E741 in run_fip_breadth_diagnostics: `l` is the OHLCV low and is genuinely
used, so this was a naming fix (`l` -> `lo`), not a deletion.
- F841 in the same file: `vol_ix`/`momr_ix` are assigned from a pure local
`_index()` and never read, so removing them cannot change any output. Their
upstream `vol_weeks`/`momr_weeks` maps *are* used further down and stay; the
comment above `_index` was corrected to say so.
The rest are unused imports and f-strings without placeholders (literal
markdown table headers, so identical output).
Verified beyond the linter, since py_compile does not catch a removed-but-used
import: every removed symbol has zero remaining references, all scripts compile,
and the full unit suite passes (852 passed, 1 skipped).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two review findings on 46ace50.
[P1] Raising HY_OAS_WINDOW_DAYS to 700 only reached newly computed rows. A
routine run recomputes the latest trading date alone, and `rebuilding` was keyed
on "no v3 snapshot exists at all", which is false once the cutover has run --
so every row already written kept the credit gap the wider window exists to
close, indefinitely.
Adds SENSOR_REVISION: stamped into each snapshot, absent on pre-marker rows
(read as 1), and a stored revision below the current one triggers exactly one
reseed. Deliberately not METHODOLOGY, which would partition the history API and
discard the cached event study -- neither warranted, since the study recomputes
its Warning series from source rather than reading snapshots and so cannot be
staled by a reseed.
The reseed is bounded by REBUILD_LOOKBACK_DAYS in calendar days rather than a
session count, because the binding constraint is the OAS fetch: each replayed
row needs W3's 20-business-day lookback inside HY_OAS_WINDOW_DAYS. Replaying by
session count would have left the oldest stored rows unrepaired -- the exact
rows the fix targets. At 672 days the replay covers ~464 sessions, W3's oldest
requirement lands on the first fetched OAS day, and the ~400-session series the
cutover wrote is fully covered. A test asserts that relationship so the two
constants cannot drift back into recreating the gap.
[P2] With nothing ever collected, current_observation returned available=true
and the default placeholders -- "unknown" for every hyperscaler, "mixed" for the
reaction -- so the card announced a reading that never happened. Those are the
absence of an observation, not an observation of absence. Gated on `observed`
(non-null fetched_at, the one field every path writing real content stamps),
which blanks the content and drives a proper empty state naming where an admin
collects one. This was a regression from 46ace50; fundamental_overlay never had
it, since no observation means no effective date means pending.
Also renames the leftover v2 identifiers in the touched paths
(rewrite_existing_v2, latest_v2).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The test built the importer with today=date.today() while running against a
fixed local clone with do_pull=False, so its forward horizon shrank by a day
per real day. It has now decayed past the initial-load gate -- 19d against the
21d MIN_FORWARD_HORIZON_DAYS floor -- and would have kept failing, worse each
day.
Anchors today to the clone's own calendar (max reporting date across the seeded
dot-free symbols, minus 35 days, mirroring the ~35d horizon the importer's own
comment cites) and uses that date in the forward-calendar assertion. Also
surfaces run.error_details on failure, which is how the cause was found.
Test-only. MIN_FORWARD_HORIZON_DAYS and the importer are untouched: production
pulls fresh data on every run and was never affected by this.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The page had twelve stacked blocks, several of them different views of the
same numbers. The quadrant plot and the score-history chart drew the same two
series from the same query key, which read as two datasets; they are now one
card with a Time | Path toggle. The two pillar disclosures become one grouped
table, and three prose blocks (data quality, basket, coverage) become one
provenance chip strip. Page text is now limited to what changes how the reader
interprets today's number; the rest moved to the methodology doc.
Removes three stale-threshold bugs of one class. The quadrant fell back to v2's
60/60 dividers when quadrant_config was absent -- the real values are 50/40 and
they feed alert_service, so the chart could disagree with what actually fires.
The gauge fell back to v2's 30/60/80 band ticks, and drew a divider line that
always landed on its own "elevated" tick. The time series' reference lines were
at 30/60/80, which correspond to nothing in v3; they are now per-axis dashed
lines read from the same quadrant_config. Rendering also surfaced a live
clipping bug inherited from the old chart: margin.left -18 against YAxis
width 28 left ~10px for a 3-digit label, so every Y tick was cut off.
HY_OAS_WINDOW_DAYS was 400 *calendar* days while a rebuild replays
REBUILD_SESSIONS = 400 *trading* sessions (~579 calendar days), so the oldest
~180 days of any rebuild got no OAS at all and both credit sensors returned
None. State then lands at 80% coverage and Warning at exactly MIN_COVERAGE, so
both still publish bands -- a series that looks homogeneous while its oldest
rows were scored without credit. Widened to 700. This needs no methodology
bump: C1 reads [-1] and W3 reads [-21], both from the end, so widening only
prepends and every live score is bit-identical. Sequenced deliberately, since
acting on the open findings below bumps METHODOLOGY and fires the rebuild.
A just-collected fundamental observation was hidden until its effective date --
one day, three over a weekend -- because the live reading called the
point-in-time function, so refreshing appeared to do nothing. That was the
opposite of what the doc claimed. fundamental_overlay stays the gated record
(it runs for every replayed date during a rebuild); current_observation is the
live reading and reports the effective date instead of blanking the content.
Nothing in the overlay is scored, so showing it early cannot reach a published
number.
Documents four calculation findings. Three are not implemented, since each
changes a published score and so requires a v4 cut: State's top band is a
credit-event band (credit returns 0.0 rather than None below the 3.5 anchor, so
it is pinned at zero at weight 20 -- with everything else pegged State computes
to exactly 80.0, the breaking threshold); V1 saturates at VIX 30; and the
deliberate max(P1,P2,P3) defeats P3's anchoring because P1 is binary.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
B would have replaced Alpaca historical OHLCV with the DoltHub stocks repo. It
was scoped inside a plan whose goal was killing the quota-limited free-tier APIs
— which A6 achieved without it, since Alpaca was never one of them.
Its only concrete benefit was `corporate_actions` for the KLAC-class post-filing
split (TTM EPS pre-split against a post-split price). That needs split events,
not a 4.7 GB clone, and the Alpaca SDK already in the venv exposes them. Against
that: fundamentals are 20% of the composite and P/E one of three inputs, so the
wart is small and self-correcting at the next filing; and B would have made
symbol history mutable as a routine event, which the backtest/prod parity guard
exists to catch.
The design is kept as a record, struck through rather than deleted, with the
reasoning next to it so this isn't re-derived. Also corrects the doc's status
header, the never-written migration 027 (that number went to
weighted_avg_diluted_shares), and the note that the stocks repo's license was
never reviewed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A6 deployed cleanly and the provider keys are gone from the production `.env`,
which makes the legacy collector inert regardless of any settings row. The two
tombstones migration 029 pinned have no remaining job, and nothing in the
codebase reads either key.
Migration 030 deletes them and drops the Admin filter that hid them. Unlike
029's, its downgrade is meaningful — it restores both rows at their safe values,
since going back past this revision means going back toward code that reads them.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three follow-ups from review of the A6 commits.
The cache-summary re-finalize raised a second durable system event on a failed
run: _runtime_finish emits for `error`/`rate_limited`, and the dedup key
includes the message, so "SEC unavailable" and "SEC unavailable · cache 511 · 2
score inputs changed" landed as two unacknowledged Admin events. Adds
`emit_event` so a re-finalize that only rewords an outcome stays silent, with a
regression test asserting exactly one event.
The rollback section still claimed disabling the SEC job freezes the cache — the
opposite of what the same page says two lines earlier, and of what the code now
does. Rewritten: there is no Admin cache-off switch, restoring `fundamental_data`
alone is temporary because the next run rebuilds it from the same snapshots and
code, and a real freeze means stopping the service.
Remaining "shadow" wording: the two import jobs have never been shadow since
activation, so `_run_shadow_import` -> `_run_source_import`, its section heading,
the deployment doc's job label, and the plan doc's "production switch remains"
handoff paragraph are all brought up to date.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The runtime message only appended the cache summary when the import itself
completed. On a deferred, failed or source-locked run Admin → Jobs showed just
the import outcome, so an operator had no signal that `fundamental_data` had
advanced — contradicting the claim now made in the docstring, the schedule hint
and the deployment doc.
The import status still varies and stays the headline; the cache summary is
appended to all of them. Adds a source-locked regression test.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The A5 cutover has been on and observed in production, so SEC Company Facts +
DoltHub earnings are already the live source for `fundamental_data`. This
removes everything the legacy path still occupied.
Gone: the three providers and their config/env keys; the weekly
`fundamental_collector` job; the cutover toggle (SEC + Dolt is now the
unconditional path, so `off` can no longer silently freeze scoring inputs); the
A5 parity report, whose deltas became structurally zero once the candidate
builder started writing the table it compared against; and the FMP tier of
universe bootstrap.
Two behavioral notes:
- Disabling **SEC Fundamentals Import** now stops the SEC network fetch only.
The local cache refresh moved outside the job-enable check, because candidates
also derive from daily closes and earnings events — freezing those on an
ingestion pause would stale scoring with no fallback left to recover from.
- `/ingestion/fetch?sources=fundamentals` still accepts the key and reports
`skipped`; there is no per-ticker fetch any more.
Migration 029 does not blanket-delete the leftover settings rows. Migrations run
before the service restart, and pre-A6 code reads an absent `job_*_enabled` row
as *enabled* — so the two behavior-bearing keys become tombstones pinned to safe
values (hidden in Admin) and only the inert three are deleted. Removing the
provider keys from the production `.env` is the matching rollout step.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Brings the durable artifacts of research/portfolio-capacity-rebalancing onto
main so the rationale for raising the count cap lives with the code that cites
it. The matrix runner, the research simulator hooks and the study's unit tests
are deliberately left behind; they remain at tag research/portfolio-capacity-final.
Corrects conclusions that were reached on EV per trade and are now superseded:
the findings doc's decisions 1 (keep cap 10) and 4 (run the risk-floor A/B) are
struck through and answered in a new correction section, and the research README
and phase-A matrix entries are updated to match. The frozen specification itself
is untouched -- its recorded SHA-256 f1e37783 still verifies.
effective-risk-floor-ab.md is retained but marked CLOSED/NEGATIVE: the study it
proposes is already answered by cap15 vs cash_unbounded (-0.753pp CAGR while
EV/trade rises), and its EV-based pass rule would have shipped it.
scripts/research_rankings.py replaces a fourth copy of the historical rank-map
helper; run_research_matrix, run_execution_recovery_matrix and
run_daily_reentry_matrix now share it. The shared version adds a duplicate
observation guard and a deterministic symbol tie-break the copies lacked.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The capacity bracket study (reports/portfolio-construction-prod505-capacity-
bracket-daily-v1) showed a book whose count cap never binds earns +1.1pp CAGR
over the old 10 -- 51 of 175 paired cohorts better, 2 worse -- at unchanged
drawdown (+0.007pp) and better Calmar in 51 of the 52 cohorts that moved.
The headline EV-per-trade delta is ~0 (+0.001), which is the trap: capacity
does not change trade quality, it changes trade COUNT. Flat EV/trade means the
blocked entries were just as good as the taken ones, so refusing them cost their
whole contribution to return. Judge capacity on CAGR, never on EV per trade.
15 is headroom, not a target. cap15 peaked at 12 positions with zero full-book
skips, so cash plus SIM_NOTIONAL_CAP is the real ceiling and 15/20/None are the
same experiment.
SIM_MAX_POSITIONS and the shadow book's DEFAULT_CAPACITY move together to keep
backtest and production in parity. Historical research arms pass max_positions
explicitly, so their labels and past results are unaffected.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
TradeChart shows a fixed 21-bar window. Once a trade is older than that,
its entry bar precedes the window and entryIdx goes negative, so the price
and trail paths index past the start of series/stopPath and emit NaN
coordinates -- the browser then drops both paths entirely, leaving only the
horizontal level lines. Clamp the index to the left edge and drop the entry
marker when the entry bar is outside the window; the full-width entry line
already carries it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
test_rr_scanner_bug_exploration.py and test_rr_scanner_fix_check.py both
assert one invariant: the headline target is the probability-based near
level, not the far max-R:R lottery. That is already covered directly by
test_recommendation_service.py's _select_primary_target tests, which also
reach cases these never did (empty list, probability floor, activation vs
scanner floor), and end to end by test_rr_scanner_integration.py's
full-flow test -- a strict superset of their deterministic cases: three
resistance and three support levels, both directions, plus persistence
and rr_ratio consistency.
The two files also duplicated each other, and their docstrings had gone
stale: test_deterministic_long_three_levels documented a hand-computed
_compute_quality_score winner even though the assertion is about the
probability primary that supersedes it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The autouse _setup_db fixture ran create_all + drop_all for every test in
the suite, including the many that never open a session. That cycle costs
~49ms against these 22 tables; truncating them instead costs ~6ms for the
same guarantee of an empty database per test.
Build the schema once, then delete every row before each subsequent test.
No model sets sqlite_autoincrement, so SQLite reuses rowids after a full
delete and generated ids still restart at 1.
Measured over 874 tests, deterministic order: 138.6s -> 74.5s (~46%).
Verified green under pytest-randomly's default random ordering as well.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A weekday with no published index is usually just a market holiday
(~10/yr), not a fault. The WARNING still earns its place as the guard
against inferring missing from an error code, but the comment should
not claim more than it can.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The fundamentals import has been dead since 2026-07-25, alerting
SecForbiddenError on form.20260725.idx with "set a real sec_user_agent
contact email". The User-Agent was never the problem.
www.sec.gov/Archives is served from an S3 bucket with no ListBucket
grant, so a MISSING key cannot answer 404 — it returns 403 with S3's
AccessDenied XML. SEC publishes a daily index for business days only, so
2026-07-25 (a Saturday) is simply absent. _get mapped every 403 to the
fatal SecForbiddenError, which made daily_index's `except
SecNotFoundError` unreachable for the exact case it was written for:
the first weekend an incremental walk crossed killed the run, and
last_processed never advanced past Friday.
Latent until activation, not a change at SEC: with no promoted run the
importer takes the backfill path and makes zero daily_index calls, so
the walk was first exercised by the first incremental run.
Verified live 2026-07-30: Sat/Sun 403 with AccessDenied XML while Fri
(51 rows) and Mon (26 rows) return 200 on the same UA; a genuine
rejection is instead the WAF's text/html "Undeclared Automated Tool"
page, served even for files that exist. So the downgrade to "missing" is
gated on all three: the /Archives/ prefix, an XML content type, and
S3's own error code. Every other 403 still alerts and stops.
Missing weekday indexes now log at WARNING — if a rejection page were
ever misread as absent, the importer must not advance past real filings
quietly.
No state to reset: _last_processed_index_date reads promoted runs only,
so the next run walks 2026-07-25..29, skipping the weekend.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The fundamentals import had been failing for three days on two tracked
filings the index listed but Company Facts appeared not to have. They were
not lagging: SEC filed the XBRL of NEE's and DOW's 2026-07-24 combined
parent/subsidiary 10-Qs under the co-registrant's CIK (Florida Power &
Light, Dow Chemical), so the ticker-carrying filer's own facts file never
receives that accession. This does not self-correct - an NEE filing
misattributed the same way in 2014 is still misfiled.
Because source_max_date advances only on a promoted run, the failure was
self-perpetuating: every later run re-walked the same index day and re-hit
the same two filings.
- Recover from the co-registrant file. The daily index lists every
co-registrant of an accession, which is the only pointer to where the
facts actually landed. Rows are re-stamped to the real filer, since
parse_snapshots stamps the CIK of the payload it read.
- Guard the recovery with a share-count continuity check against the
issuer's own history, so a subsidiary's standalone facts can never be
stored as the parent's. No history, no recovery.
- Bound the blocking: a filing still unresolvable after
MISSING_XBRL_RETRY_DAYS promotes with a named unresolved_filing warning
instead of wedging every later import.
- Name the offending filings in the alert and record them in
validation_json, separating not_in_companyfacts from not_in_submissions.
The gate previously reported a count and discarded the accessions.
Verified against live SEC data: both filings recover with the correct CIK
and the guard rejects a mismatched reference.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A conditional clause in the summary sentence carried an f prefix with no
placeholders, failing `ruff check app/` and blocking the deploy. Literal only;
the rendered text is unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The v3 cutover run scored 2/4 corrections warned against v2's 3/4, which reads
like a regression and is not one. Only 4 of the 11 detected corrections fall in
the holdout, so recall is one event from a different headline -- and the event
that flips is decided by threshold placement, not by what the score saw. "v3
without the credit sensor" catches 2025-02-21 at a *higher* threshold (35.5)
than shipped v3 misses it at (32.3), because the alarm rule needs a rising edge
and a lower threshold can fire outside the horizon then never reset below.
Two caveats are now computed and surfaced rather than left for the reader to
infer:
- Holdout event count against MIN_EVENTS_FOR_CONFIDENCE. The summary sentence
states how many of the detected corrections actually fall in the test period.
- Warning-sensor coverage across the split. The score renormalises over what is
available, so a training window predating a sensor's history freezes the
threshold on a different construct than the holdout is measured against. At
the cutover that is 39% of training sessions with all three sensors versus
100% of the test period, credit history beginning 2023-07-25.
Restricting the threshold to sensor-matched training sessions was tested and
rejected: those sessions are a calm recent stretch, so the threshold falls from
32.3 to 22.5 and false alarms rise from 3.3 to 8.6/yr. It swaps a coverage bias
for a regime-selection bias. The report states its limits instead.
_warning_series now returns per-session sensor counts alongside the scores.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The LLM-sourced capex/earnings observations carried 12+8 of 100 Warning points,
so both pegged at 100 produced a Warning of 20.0 -- below the event study's 25.3
alarm threshold and still inside the "stable" band. The reading was
arithmetically incapable of changing anything on screen, which is why refreshing
it appeared to do nothing. They are now a qualitative overlay reported beside
the scores rather than diluted into them.
Calibrated against the 408 v2 sessions to 2026-07-24, reproduced offline from
Alpaca + FRED; the harness matched the stored prod distribution exactly before
any parameter was changed.
State:
- P3 used dd_pct * 5, reaching 100 at a 20% drawdown -- the 90th percentile of
the observed distribution -- so 39/408 sessions sat at exactly 100 with no
resolution left during the part of a selloff that matters most. Replaced with
anchored breakpoints keeping headroom past the observed 36% maximum, blended
2:1 like P1/P2 instead of max(). P3's realized share of State falls from 65%
to 40%, matching its nominal weight.
- Credit level is now anchors-only. ICE capped FRED's BAMLH0A0HYM2 at a rolling
3-year window in April 2026, silently turning the 10-year percentile leg into
a 3-year one that scored 20 points of stress at an OAS of 3.5 -- the level its
own anchors call "mild". The anchors already encode the long-run distribution.
Warning:
- Added HY OAS 20-session widening (25%). The level is pinned at zero below the
3.5 anchor; its rate of change is not.
- Divergence tapers to a 0.35 floor instead of a hard price_ret >= 0 gate, which
zeroed the sensor through every decline: on 2026-07-24 the basket shed 10
points of participation in 20 sessions and Warning printed exactly 0.
- The event study and the live monitor now share one sensor definition, so they
cannot silently drift apart.
Bands are per axis (State 20/50/80, Warning 20/40/60) with quadrant dividers at
50/40; v2 Warning never exceeded 64.9 against a shared 60, leaving that half of
the quadrant unreachable. Realized shares: State 73/15/8/3%, Warning 69/20/8/3%.
Snapshots now record credit_history_days and vix_history_days -- the percentile
defect went unnoticed for months because nothing asserted the window the code
claimed.
Cutover: the first run rebuilds 400 sessions automatically; the Event Study job
must be re-run, as its cached report self-invalidates on the methodology check.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The parity gate has been exercised: nine-pass investigation, fixes, two prod
reparses, and an after-report at 504/511 candidate coverage with agreement
unchanged. What remains of workstream A is the activation itself (step c, the
post-approval fundamental_data refresh — not yet implemented) and A6
decommissioning, both now specified in a handoff section with the caveats the
next implementer must carry (KLAC splits, BRK-B, FITB, the 25% guard, CIK
overrides, reparse-after-parser-changes).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Closes the parity investigation: prod reparse runs 6+7 reconciled against the
dry run, the collision check that caught the BEN regression (and its residue --
36 historical 53-week-drift rows, deliberately left), and the 2026-07-24 parity
report diffed against the 2026-07-23 baseline. Candidate coverage 482 -> 504 of
511 with the gap fully explained (PSKY/Q new registrants, FITB guard-tripped
with no revenue), agreement unchanged where both sides exist, and the remaining
deltas are documented definition differences. Includes the after-report and a
correction to the seventh pass (FITB loses its score, not just one input).
Recommends approving the A5 cutover with KLAC as the one carried caveat.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Regression found by the post-reparse collision check. _period_identity trusted
submissions.fiscalYearEnd, which is not reliable: Franklin Resources (BEN)
declares 1231 while every one of its 10-Ks ends 09-30.
The effect was data loss, not just a bad label. BEN's real fiscal Q1 (Dec 31)
sat 0 days from the claimed year end, matching no quarter band, so it fell back
to SEC's fy/fp; its fiscal Q2 (Mar 31) computed 275 days out and was labelled
Q1. Both landed on the same key, the collision discarded one, and BEN lost TTM
EPS and revenue growth entirely — values it had before this branch.
A 10-K's reportDate IS the fiscal year end by definition, so resolve_fiscal_
year_end() now prefers the issuer's most recent annual filing and treats the
declared value as a fallback for issuers with no 10-K in the set.
Scanned the full tracked universe: 2 of 506 issuers declare a year end more
than 21 days from their own 10-K — BEN (91d, broken) and DELL (29d, mislabelled
but functionally correct). Both now derive correctly and match the legacy
provider: BEN revenue growth 3.8243 vs 3.82, DELL 38.5735 vs 38.57. Controls
(AAPL, COST, PEP, DPZ, IRM, JPM, CRM, STX, AVY) byte-identical.
826 unit tests pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Found in review. _merge_amendments rebuilds a period from _MERGED_FIELDS +
_CARRIED_FIELDS alone, so a column in neither list is absent from the merged
row, not just stale — and callers read it with getattr(..., None), which
silently yields None. weighted_avg_diluted_shares was never added when the
market-cap fallback landed (_SNAPSHOT_COLS in the importer was updated, its
counterpart in the derivation was not).
The failure needed both of this branch's fixes at once: a multi-class issuer
with a partial amendment on its latest period (META with a Part-III-only
10-K/A) would silently lose market cap and FCF yield again.
Adds the field, a regression test for that case, and a guard test asserting
the merge/carry lists cover every SnapshotRow field, so the next column added
fails loudly rather than losing data quietly. Confirmed the guard catches the
original bug.
Also from review:
- Expose pe_caveat in the valuation payload, so a P/E suppressed by split
contamination says why instead of looking like missing data (the caveat was
set but never read).
- no_xbrl_filings now names both causes; the old text advised pinning a CIK
override, which is wrong for a genuine new registrant that simply has not
filed yet and clears itself.
- Document that fiscalYearEnd is the issuer's current calendar, so a fiscal-
year-end change degrades old periods (fallback/newest-wins), not current ones.
- Parser-level tests for _select_weighted_avg_shares (shortest-span-wins and
concept priority), which only had derivation-level coverage.
823 unit tests pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Nine-pass investigation of the 2026-07-23 A5 parity report: for each coverage
gap and wrong value, the root cause traced against live SEC company facts, the
fix, and its live-data validation. Also records the decisions taken (weighted-
average share fallback, keep ASC-606 revenue basis, basic-EPS fallback, keep the
25% split-guard threshold) and what remains genuinely unfixable from this data
(KLAC post-filing split, BRK-B dimensional share count). Includes the source
parity report the findings analyse.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Operational plumbing to land the parser fixes and to make silent resolution
failures visible.
- Reparse: SecFundamentalsImporter(reparse=True) restages every accession with
the current parser and rewrites the ones that now reconstruct differently,
writing the full column set so a row is never half old-parse. Snapshots stay
immutable with respect to SEC; the stored row is our reconstruction, and after
a parser fix keeping it is a stale cache, not history. run_import(force=True)
bypasses the unchanged-revision no-op, since the staleness is on our side, not
the source's. Exposed as scripts/reparse_fundamentals.py, dry-run by default.
- CIK overrides: sec_universe reads a {symbol: cik} pin from
SystemSetting['sec_cik_overrides'], applied ahead of company_tickers.json, for
when SEC maps a ticker to a successor shell with no filings (XOM -> a zero-
filing "ExxonMobil Holdings Corp" while every 10-K/Q is under CIK 34088).
- Resolution validation: a tracked issuer resolving to a registrant with no XBRL
filings now records no_xbrl_filings and raises a warning naming the CIKs and
the override setting, instead of silently yielding nothing on every run.
- _diff_fields compares datetime instants, not representations: accepted_at
round-trips naive from SQLite but tz-aware from Postgres, which otherwise made
a reparse of identical data report every row as changed (and false-positived
the pre-existing discrepancy warning).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The A5 parity report surfaced coverage gaps and wrong values that all traced
to the SEC facts parser and read-time derivation rather than to bad source
data. Fixes, each validated by replaying the production parser + derivation
against live company facts:
- Period identity is derived from period_end against the issuer's fiscal
calendar, not SEC's fy/fp fields, which collide (two period ends on one key,
one silently discarded) and invert (a period sorting before one that precedes
it) often enough to break the quarter chain. Recovers BXP, CRM, CRWD, FRT,
MTD, NTAP, PPL, STX, WDAY. Fixed labels are internal ordering keys only (not
in any API schema), so a filer whose year ends in early January shifting by
one is harmless.
- Revenue concept list gains RevenuesNetOfInterestExpense (banks) and the
IncludingAssessedTax variant (REITs/consumer); EPS gains the continuing-ops
variant (REG/FCX) and, last, basic EPS for a period tagging no diluted
variant at all (PPL). All appended, so any issuer that already resolved keeps
its concept.
- YTD span tolerance 20 -> 25 days, covering 4-4-5 retail calendars whose
36-week YTD-Q3 (251-252d) previously missed by ~2 (COST, PEP, DPZ).
- Amendment resolution is per field: a partial 10-K/A (Part III only, no
financial facts) no longer blanks the period (DVN).
- TTM diluted EPS is suppressed when a split contaminates the trailing window
(BKNG's mixed-unit sum produced a P/E of 1.10 that clamped to a perfect
fundamental sub-score). A post-filing split with no share-count evidence
(KLAC) remains undetectable from this data.
- Multi-class share fallback: weighted_avg_diluted_shares is captured and used
for market cap when the cover-page count is absent (dimensional, so missing
from company facts for META/CMCSA/CHTR/FOXA/NWSA/LEN). Within ~0.6% of the
true count on controls; flagged shares_estimated in the API. BRK-B has no
weighted-average fact either and stays unavailable.
820 unit tests pass; new tests confirmed to fail against the pre-fix code.
Effect is inert until existing rows are reparsed (see reparse path).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
