fix bulk fundamentals: rate limits masked by partial FMP success
Root cause of "price plan needed in bulk but fine on manual reload": on free tiers FMP returns only market cap (others 402) and the chain merged that as a partial success — so when the Finnhub/Alpha Vantage fallbacks were rate-limited during a bulk run, the chain silently returned market-cap-only and the collector's backoff never engaged. Manual single fetches worked because the fallbacks weren't throttled at that moment. Fixes: - Chain distinguishes RateLimitError from other failures: if a fallback is rate-limited and fields are still missing, raise RateLimitError (unless allow_partial=True) so the collector backs off and retries. - Bulk job paces requests (fundamental_request_spacing_seconds, default 3s) to stay under Finnhub's ~60/min, and on retry-exhaustion stores partial data and continues instead of aborting the whole run. - Manual fetch passes allow_partial=True so a lone 429 doesn't fail the refresh. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -220,16 +220,31 @@ class ChainedFundamentalProvider:
|
||||
raise ProviderError("No fundamental providers configured")
|
||||
self._providers = providers
|
||||
|
||||
async def fetch_fundamentals(self, ticker: str) -> FundamentalData:
|
||||
async def fetch_fundamentals(self, ticker: str, allow_partial: bool = False) -> FundamentalData:
|
||||
"""Merge fundamentals across providers.
|
||||
|
||||
``allow_partial`` controls behaviour when a fallback provider is *rate
|
||||
limited* and we end up with missing fields. By default we raise
|
||||
RateLimitError so the caller (the bulk collector) can back off and retry
|
||||
the ticker once the window frees — otherwise a transient 429 on Finnhub
|
||||
would be silently stored as market-cap-only. Pass ``allow_partial=True``
|
||||
(manual single fetches, or the collector's final give-up attempt) to
|
||||
accept whatever was gathered instead of raising.
|
||||
"""
|
||||
merged: dict[str, float | None] = {f: None for f in _FUNDAMENTAL_FIELDS}
|
||||
field_source: dict[str, str] = {}
|
||||
errors: list[str] = []
|
||||
rate_limited = False
|
||||
|
||||
for provider_name, provider in self._providers:
|
||||
if all(merged[f] is not None for f in _FUNDAMENTAL_FIELDS):
|
||||
break
|
||||
try:
|
||||
data = await provider.fetch_fundamentals(ticker)
|
||||
except RateLimitError as exc:
|
||||
rate_limited = True
|
||||
errors.append(f"{provider_name}: RateLimitError: {exc}")
|
||||
continue
|
||||
except Exception as exc:
|
||||
errors.append(f"{provider_name}: {type(exc).__name__}: {exc}")
|
||||
continue
|
||||
@@ -241,6 +256,17 @@ class ChainedFundamentalProvider:
|
||||
merged[field] = value
|
||||
field_source[field] = provider_name
|
||||
|
||||
missing = [f for f in _FUNDAMENTAL_FIELDS if merged[f] is None]
|
||||
|
||||
# A rate limit left data incomplete: signal it (unless partial is OK) so
|
||||
# the collector backs off rather than persisting a degraded record.
|
||||
if rate_limited and missing and not allow_partial:
|
||||
attempts = "; ".join(errors[:6])
|
||||
raise RateLimitError(
|
||||
f"Fundamentals incomplete for {ticker} due to provider rate limits "
|
||||
f"(missing {', '.join(missing)}). Attempts: {attempts}"
|
||||
)
|
||||
|
||||
if all(merged[f] is None for f in _FUNDAMENTAL_FIELDS):
|
||||
attempts = "; ".join(errors[:6]) if errors else "no usable metrics from any provider"
|
||||
raise ProviderError(f"All fundamentals providers failed for {ticker}. Attempts: {attempts}")
|
||||
|
||||
Reference in New Issue
Block a user