dennisthiessenandClaude Opus 4.8
8dcdcac2a6
1. Removed the 45-day index-walk cap: it discarded the older part of a long outage while still advancing source_max_date, permanently losing filings. The walk now covers every unprocessed date (a large gap is one-time cost). 2. Discrepancy detection meets the immutability contract: it compares ALL source snapshot fields (not five), read-only during stage/validate, reports the differing accessions + fields in validation_json, and promote emits a warning system event (in-transaction) — never mutating the stored row. 3. Malformed companyfacts (missing facts/units structure) are recorded separately and FAIL validation, instead of silently degrading to skipped rows that the 50% backfill coverage floor could still pass. Also corrected the stale "sum share classes" / DEI-only wording in the snapshot model docstring and the A3 design doc to describe the us-gaap fallback. Tests: +4 regressions (>45-day gap loses nothing, newly-added issuer backfills without filing, malformed payload fails, shares discrepancy detected + evented). 23 passed, 1 skipped. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
80 lines
4.5 KiB
Python
80 lines
4.5 KiB
Python
from datetime import date, datetime
|
|
|
|
from sqlalchemy import Date, DateTime, Float, ForeignKey, Index, String, UniqueConstraint
|
|
from sqlalchemy.orm import Mapped, mapped_column
|
|
|
|
from app.database import Base
|
|
|
|
|
|
class FundamentalSnapshot(Base):
|
|
"""CIK-keyed, one immutable row per SEC accession.
|
|
|
|
Keyed by issuer (CIK), not ticker — multi-class issuers (GOOG/GOOGL) share
|
|
one CIK and one set of fundamentals; the ``tickers.cik`` column is the only
|
|
join point. Amendments are retained: every accession is a distinct immutable
|
|
row, and readers pick the newest valid ``accepted_at`` per
|
|
(cik, fiscal_year, fiscal_period) at read time — no flags, no mutation.
|
|
|
|
**Facts are stored as the filing reports them, never as derived quarters.**
|
|
Duration facts (revenue, net_income, operating_income, diluted_eps, cfo,
|
|
capex, depreciation_amortization) hold the filing's normalized **cumulative
|
|
YTD/FY** value over (period_start -> period_end). Balance-sheet facts
|
|
(cash_and_st_investments, total_debt, shares_outstanding) are **period-end**
|
|
values. ``shares_outstanding`` is a single consolidated point-in-time count —
|
|
the ``dei:EntityCommonStockSharesOutstanding`` cover-page fact, or
|
|
``us-gaap:CommonStockSharesOutstanding`` at period end when no dei fact exists
|
|
(e.g. Alphabet). It is never a class sum (companyfacts is non-dimensional) nor
|
|
the weighted-average diluted count, since both consumers (estimated market cap,
|
|
YoY dilution read) want a point-in-time value. Discrete quarters (10-Q YTD
|
|
deltas, Q4 = FY - Q1..Q3), TTM, YoY and
|
|
the quarter tape are all derived at read time — so non-calendar fiscal years
|
|
resolve correctly and a later amendment never leaves a stale frozen quarter.
|
|
"""
|
|
|
|
__tablename__ = "fundamental_snapshots"
|
|
__table_args__ = (
|
|
UniqueConstraint("accession", name="uq_fundamental_snapshots_accession"),
|
|
Index("ix_fundamental_snapshots_cik_period", "cik", "fiscal_year", "fiscal_period"),
|
|
Index("ix_fundamental_snapshots_cik_period_end", "cik", "period_end"),
|
|
)
|
|
|
|
id: Mapped[int] = mapped_column(primary_key=True)
|
|
cik: Mapped[str] = mapped_column(String(10), nullable=False)
|
|
accession: Mapped[str] = mapped_column(String(25), nullable=False)
|
|
form: Mapped[str] = mapped_column(String(12), nullable=False) # 10-Q, 10-K, 10-K/A ...
|
|
filed_date: Mapped[date] = mapped_column(Date, nullable=False)
|
|
# Kept although PIT enforcement is deferred (one timestamp now vs painful retrofit).
|
|
accepted_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
|
|
|
|
# Period identity — required to align non-calendar fiscal years and to derive
|
|
# discrete quarters from cumulative facts.
|
|
period_start: Mapped[date | None] = mapped_column(Date, nullable=True)
|
|
period_end: Mapped[date] = mapped_column(Date, nullable=False)
|
|
fiscal_year: Mapped[int] = mapped_column(nullable=False)
|
|
fiscal_period: Mapped[str] = mapped_column(String(4), nullable=False) # Q1|Q2|Q3|Q4|FY
|
|
|
|
# Duration facts — cumulative YTD/FY over (period_start -> period_end).
|
|
revenue: Mapped[float | None] = mapped_column(Float, nullable=True)
|
|
net_income: Mapped[float | None] = mapped_column(Float, nullable=True)
|
|
operating_income: Mapped[float | None] = mapped_column(Float, nullable=True)
|
|
diluted_eps: Mapped[float | None] = mapped_column(Float, nullable=True)
|
|
cfo: Mapped[float | None] = mapped_column(Float, nullable=True) # cash flow from operations
|
|
capex: Mapped[float | None] = mapped_column(Float, nullable=True)
|
|
depreciation_amortization: Mapped[float | None] = mapped_column(Float, nullable=True)
|
|
|
|
# Balance-sheet facts — period-end values.
|
|
cash_and_st_investments: Mapped[float | None] = mapped_column(Float, nullable=True)
|
|
total_debt: Mapped[float | None] = mapped_column(Float, nullable=True)
|
|
shares_outstanding: Mapped[float | None] = mapped_column(Float, nullable=True)
|
|
# The cover-page share count (dei:EntityCommonStockSharesOutstanding) is
|
|
# reported "as of" its own date, which can differ from period_end — store it
|
|
# so market cap uses the right point-in-time count.
|
|
shares_outstanding_date: Mapped[date | None] = mapped_column(Date, nullable=True)
|
|
|
|
import_run_id: Mapped[int | None] = mapped_column(
|
|
ForeignKey("data_import_runs.id", ondelete="SET NULL"), nullable=True
|
|
)
|
|
created_at: Mapped[datetime] = mapped_column(
|
|
DateTime(timezone=True), default=datetime.utcnow, nullable=False
|
|
)
|