256 lines
19 KiB
Markdown
256 lines
19 KiB
Markdown
# Implementation Plan: Stock Data Backend
|
||
|
||
## Overview
|
||
|
||
Incremental build of the investing-signal platform: foundation first (config, DB, models, auth), then domain services (tickers, OHLCV, ingestion, indicators, S/R, sentiment, fundamentals), then scoring/ranking (scoring engine, R:R scanner, watchlist), then scheduled jobs, deployment templates, and final wiring. Each step builds on the previous and ends integrated.
|
||
|
||
## Tasks
|
||
|
||
- [x] 1. Project scaffolding, configuration, and database foundation
|
||
- [x] 1.1 Create project structure with `pyproject.toml`, `.env.example`, `alembic.ini`, and `app/` package
|
||
- Create `pyproject.toml` with dependencies: fastapi, uvicorn, sqlalchemy[asyncio], asyncpg, alembic, pydantic-settings, python-jose, passlib[bcrypt], apscheduler, httpx, alpaca-py, google-genai, hypothesis
|
||
- Create `.env.example` with all environment variables from design
|
||
- Create `app/__init__.py`, `app/config.py` (pydantic-settings `Settings` class)
|
||
- Create `app/database.py` (async SQLAlchemy engine, session factory, connection pooling)
|
||
- _Requirements: Design Constraints (connection pooling, config)_
|
||
|
||
- [x] 1.2 Create all SQLAlchemy ORM models and Alembic initial migration
|
||
- Create `app/models/__init__.py` and model files: `ticker.py`, `ohlcv.py`, `user.py`, `sentiment.py`, `fundamental.py`, `score.py`, `sr_level.py`, `trade_setup.py`, `watchlist.py`, `settings.py`
|
||
- Implement all 12 entities from the ERD: User, Ticker, OHLCVRecord, SentimentScore, FundamentalData, SRLevel, DimensionScore, CompositeScore, TradeSetup, WatchlistEntry, SystemSetting, IngestionProgress
|
||
- Include composite unique constraints, indexes, and cascade deletes per design
|
||
- Initialize Alembic (`alembic/env.py`) and generate initial migration
|
||
- _Requirements: 2.1, 2.2, Design Constraints (composite index on ticker+date)_
|
||
|
||
- [x] 1.3 Create shared schemas, exception hierarchy, and API envelope
|
||
- Create `app/schemas/common.py` with `APIEnvelope` model (status, data, error)
|
||
- Create `app/middleware.py` with global exception handler mapping `AppError` subclasses to JSON envelope responses
|
||
- Create exception classes: `AppError`, `ValidationError`, `NotFoundError`, `DuplicateError`, `AuthenticationError`, `AuthorizationError`, `ProviderError`, `RateLimitError`
|
||
- _Requirements: Design Constraints (JSON envelope, HTTP status codes)_
|
||
|
||
- [x] 1.4 Create FastAPI app entry point with lifespan, health check, and dependency injection
|
||
- Create `app/main.py` with FastAPI app, lifespan handler (DB pool startup/shutdown, default admin creation)
|
||
- Create `app/dependencies.py` with `Depends()` factories for DB session, current user, admin guard
|
||
- Create `app/routers/health.py` with unauthenticated `/api/v1/health` endpoint
|
||
- Wire health router into app
|
||
- _Requirements: 13.1, Design Constraints (health check, graceful shutdown, versioned URLs)_
|
||
|
||
- [x] 2. Authentication and admin services
|
||
- [x] 2.1 Implement Auth Service and auth router
|
||
- Create `app/services/auth_service.py`: registration (configurable on/off, creates no-access user), login (bcrypt verify, JWT generation with 60-min expiry), token validation
|
||
- Create `app/schemas/auth.py`: RegisterRequest, LoginRequest, TokenResponse
|
||
- Create `app/routers/auth.py`: `POST /api/v1/auth/register`, `POST /api/v1/auth/login`
|
||
- Implement JWT middleware in `app/dependencies.py` for `get_current_user` and `require_admin`
|
||
- _Requirements: 12.1, 12.2, 12.3, 12.4, 12.5, 12.6_
|
||
|
||
- [ ]* 2.2 Write property tests for auth (Properties 34-38)
|
||
- **Property 34: Registration creates no-access user** — _Validates: Requirements 12.1_
|
||
- **Property 35: Registration disabled rejects all attempts** — _Validates: Requirements 12.2_
|
||
- **Property 36: Login returns valid JWT** — _Validates: Requirements 12.3_
|
||
- **Property 37: Invalid credentials return generic error** — _Validates: Requirements 12.4_
|
||
- **Property 38: Access control enforcement** — _Validates: Requirements 12.5_
|
||
|
||
- [x] 2.3 Implement Admin Service and admin router
|
||
- Create `app/services/admin_service.py`: grant/revoke access, toggle registration, list users, reset passwords, create accounts, system settings CRUD, data cleanup (delete old OHLCV/sentiment/fundamentals preserving tickers/users/scores), job control
|
||
- Create `app/schemas/admin.py`: UserManagement, SystemSettingUpdate, DataCleanupRequest
|
||
- Create `app/routers/admin.py`: admin-only endpoints under `/api/v1/admin/`
|
||
- _Requirements: 13.1, 13.2, 13.3, 13.4, 13.5_
|
||
|
||
- [ ]* 2.4 Write property tests for admin (Properties 39-40)
|
||
- **Property 39: Admin user management operations** — _Validates: Requirements 13.2_
|
||
- **Property 40: Data cleanup preserves structure** — _Validates: Requirements 13.4_
|
||
|
||
- [x] 3. Checkpoint - Ensure all tests pass
|
||
- Ensure all tests pass, ask the user if questions arise.
|
||
|
||
- [x] 4. Ticker management and OHLCV price storage
|
||
- [x] 4.1 Implement Ticker Registry service and router
|
||
- Create `app/services/ticker_service.py`: add (validate non-empty, uppercase, alphanumeric, check uniqueness), delete (cascade all associated data), list (sorted alphabetically)
|
||
- Create `app/schemas/ticker.py`: TickerCreate, TickerResponse
|
||
- Create `app/routers/tickers.py`: `POST /api/v1/tickers`, `GET /api/v1/tickers`, `DELETE /api/v1/tickers/{symbol}`
|
||
- _Requirements: 1.1, 1.2, 1.3, 1.4, 1.5, 1.6_
|
||
|
||
- [ ]* 4.2 Write property tests for ticker management (Properties 1-4)
|
||
- **Property 1: Ticker creation round-trip** — _Validates: Requirements 1.1_
|
||
- **Property 2: Duplicate ticker rejection** — _Validates: Requirements 1.2_
|
||
- **Property 3: Whitespace ticker rejection** — _Validates: Requirements 1.3_
|
||
- **Property 4: Ticker deletion cascades** — _Validates: Requirements 1.5_
|
||
|
||
- [x] 4.3 Implement Price Store service and OHLCV router
|
||
- Create `app/services/price_service.py`: upsert OHLCV (validate high >= low, prices >= 0, volume >= 0, date <= today, ticker exists), query by ticker + date range
|
||
- Create `app/schemas/ohlcv.py`: OHLCVCreate, OHLCVResponse
|
||
- Create `app/routers/ohlcv.py`: `POST /api/v1/ohlcv`, `GET /api/v1/ohlcv/{symbol}`
|
||
- On upsert: invalidate LRU cache for ticker, mark composite score as stale
|
||
- _Requirements: 2.1, 2.2, 2.3, 2.4_
|
||
|
||
- [ ]* 4.4 Write property tests for OHLCV (Properties 5-7)
|
||
- **Property 5: OHLCV storage round-trip** — _Validates: Requirements 2.1, 2.2_
|
||
- **Property 6: OHLCV validation rejects invalid records** — _Validates: Requirements 2.3_
|
||
- **Property 7: OHLCV rejects unregistered tickers** — _Validates: Requirements 2.4_
|
||
|
||
- [x] 5. Market data provider and ingestion pipeline
|
||
- [x] 5.1 Implement provider protocols and concrete implementations
|
||
- Create `app/providers/protocol.py`: `MarketDataProvider` Protocol (fetch_ohlcv), `SentimentProvider` Protocol (fetch_sentiment), `FundamentalProvider` Protocol (fetch_fundamentals)
|
||
- Create `app/providers/alpaca.py`: Alpaca OHLCV provider using `alpaca-py` SDK — fetches daily bars by ticker and date range
|
||
- Create `app/providers/gemini_sentiment.py`: Gemini sentiment provider using `google-genai` with search grounding — sends structured prompt per ticker, parses JSON response (classification + confidence)
|
||
- Create `app/providers/fmp.py`: Financial Modeling Prep fundamentals provider using `httpx` — fetches P/E, revenue growth, earnings surprise, market cap
|
||
- _Requirements: Design Constraints (provider behind interface)_
|
||
|
||
- [x] 5.2 Implement Ingestion Pipeline service and router
|
||
- Create `app/services/ingestion_service.py`: fetch + upsert with rate-limit handling (track `last_ingested_date`, return partial progress on rate limit, resume from last date + 1 day), provider error handling (descriptive error, no data modification)
|
||
- Create `app/routers/ingestion.py`: `POST /api/v1/ingestion/fetch/{symbol}`
|
||
- _Requirements: 3.1, 3.2, 3.3, 3.4_
|
||
|
||
- [ ]* 5.3 Write property tests for ingestion (Properties 8-9)
|
||
- **Property 8: Provider error preserves existing data** — _Validates: Requirements 3.2, 7.3, 8.3_
|
||
- **Property 9: Rate-limit resume continuity** — _Validates: Requirements 3.3, 3.4, 4.5_
|
||
|
||
- [x] 6. Checkpoint - Ensure all tests pass
|
||
- Ensure all tests pass, ask the user if questions arise.
|
||
|
||
- [x] 7. Technical analysis and S/R detection
|
||
- [x] 7.1 Implement LRU cache wrapper with invalidation
|
||
- Create `app/cache.py`: LRU cache wrapper (max 1000 entries) keyed on ticker + date range + indicator type, with per-ticker invalidation method
|
||
- _Requirements: Design Constraints (LRU cache)_
|
||
|
||
- [x] 7.2 Implement Technical Analysis service and indicators router
|
||
- Create `app/services/indicator_service.py`: compute ADX (28+ bars), EMA (period+1 bars, default 20/50), RSI (15+ bars, 14-period), ATR (15+ bars, 14-period), Volume Profile (20+ bars, POC/Value Area/HVN/LVN), Pivot Points (5+ bars, swing highs/lows)
|
||
- Each indicator returns raw values + normalized 0-100 score
|
||
- Implement EMA cross signal (bullish/bearish/neutral based on short vs long EMA comparison)
|
||
- Enforce minimum data requirements, return error if insufficient
|
||
- Create `app/schemas/indicator.py`: IndicatorRequest, IndicatorResponse, EMACrossResponse
|
||
- Create `app/routers/indicators.py`: `GET /api/v1/indicators/{symbol}/{indicator_type}`, `GET /api/v1/indicators/{symbol}/ema-cross`
|
||
- _Requirements: 5.1, 5.2, 5.3, 5.4_
|
||
|
||
- [ ]* 7.3 Write property tests for indicators (Properties 11-14)
|
||
- **Property 11: Score bounds invariant** — _Validates: Requirements 5.2, 6.2, 9.1_
|
||
- **Property 12: Indicator minimum data enforcement** — _Validates: Requirements 5.4_
|
||
- **Property 13: EMA cross directional bias** — _Validates: Requirements 5.3_
|
||
- **Property 14: Indicator computation determinism** — _Validates: Requirements 5.1_
|
||
|
||
- [x] 7.4 Implement S/R Detector service and router
|
||
- Create `app/services/sr_service.py`: detect SR levels from Volume Profile (HVN/LVN) and Pivot Points (swing highs/lows), assign strength scores (0-100 based on price respect count), merge levels within tolerance (default 0.5%), tag as support/resistance relative to current price, recalculate on new OHLCV data
|
||
- Create `app/schemas/sr_level.py`: SRLevelResponse
|
||
- Create `app/routers/sr_levels.py`: `GET /api/v1/sr-levels/{symbol}` (sorted by strength descending)
|
||
- _Requirements: 6.1, 6.2, 6.3, 6.4, 6.5, 6.6_
|
||
|
||
- [ ]* 7.5 Write property tests for S/R detection (Properties 15-17)
|
||
- **Property 15: SR level support/resistance tagging** — _Validates: Requirements 6.3_
|
||
- **Property 16: SR level merging within tolerance** — _Validates: Requirements 6.5_
|
||
- **Property 17: SR level detection from data** — _Validates: Requirements 6.1_
|
||
|
||
- [x] 8. Sentiment and fundamental data services
|
||
- [x] 8.1 Implement Sentiment service and router
|
||
- Create `app/services/sentiment_service.py`: store sentiment records (classification, confidence, source, timestamp), compute dimension score with time-decay weighted average over configurable lookback window (default 24h)
|
||
- Create `app/schemas/sentiment.py`: SentimentResponse
|
||
- Create `app/routers/sentiment.py`: `GET /api/v1/sentiment/{symbol}`
|
||
- _Requirements: 7.1, 7.2, 7.3, 7.4_
|
||
|
||
- [ ]* 8.2 Write property tests for sentiment (Properties 18-19)
|
||
- **Property 18: Sentiment score data shape** — _Validates: Requirements 7.2_
|
||
- **Property 19: Sentiment dimension score uses time decay** — _Validates: Requirements 7.4_
|
||
|
||
- [x] 8.3 Implement Fundamental Data service and router
|
||
- Create `app/services/fundamental_service.py`: store fundamental data (P/E, revenue growth, earnings surprise, market cap), mark fundamental dimension score as stale on new data
|
||
- Create `app/schemas/fundamental.py`: FundamentalResponse
|
||
- Create `app/routers/fundamentals.py`: `GET /api/v1/fundamentals/{symbol}`
|
||
- _Requirements: 8.1, 8.2, 8.3, 8.4_
|
||
|
||
- [ ]* 8.4 Write property test for fundamentals (Property 20)
|
||
- **Property 20: Fundamental data storage round-trip** — _Validates: Requirements 8.1_
|
||
|
||
- [x] 9. Checkpoint - Ensure all tests pass
|
||
- Ensure all tests pass, ask the user if questions arise.
|
||
|
||
- [x] 10. Scoring engine, R:R scanner, and watchlist
|
||
- [x] 10.1 Implement Scoring Engine service and router
|
||
- Create `app/services/scoring_service.py`: compute dimension scores (technical, sr_quality, sentiment, fundamental, momentum) each 0-100, compute composite score as weighted average of available dimensions with re-normalized weights, staleness marking/recomputation on demand, weight update triggers full recomputation
|
||
- Create `app/schemas/score.py`: ScoreResponse, WeightUpdateRequest, RankingResponse
|
||
- Create `app/routers/scores.py`: `GET /api/v1/scores/{symbol}`, `GET /api/v1/rankings`, `PUT /api/v1/scores/weights`
|
||
- _Requirements: 9.1, 9.2, 9.3, 9.4, 9.5, 9.6, 9.7_
|
||
|
||
- [ ]* 10.2 Write property tests for scoring (Properties 21-25)
|
||
- **Property 21: Composite score is weighted average** — _Validates: Requirements 9.2_
|
||
- **Property 22: Missing dimensions re-normalize weights** — _Validates: Requirements 9.3_
|
||
- **Property 23: Staleness marking on data change** — _Validates: Requirements 9.4_
|
||
- **Property 24: Stale score recomputation on demand** — _Validates: Requirements 9.5_
|
||
- **Property 25: Weight update triggers full recomputation** — _Validates: Requirements 9.7_
|
||
|
||
- [x] 10.3 Implement R:R Scanner service and router
|
||
- Create `app/services/rr_scanner_service.py`: scan tickers for trade setups (long: target = nearest SR above, stop = entry - ATR×multiplier; short: target = nearest SR below, stop = entry + ATR×multiplier), filter by R:R threshold (default 3:1), recalculate/prune on data change, skip tickers without sufficient SR/ATR data
|
||
- Create `app/schemas/trade_setup.py`: TradeSetupResponse
|
||
- Create `app/routers/trades.py`: `GET /api/v1/trades` (sorted by R:R desc, secondary composite desc, optional direction filter)
|
||
- _Requirements: 10.1, 10.2, 10.3, 10.4, 10.5, 10.6, 10.7, 10.8_
|
||
|
||
- [ ]* 10.4 Write property tests for R:R scanner (Properties 26-29)
|
||
- **Property 26: Trade setup R:R threshold filtering** — _Validates: Requirements 10.1_
|
||
- **Property 27: Trade setup computation correctness** — _Validates: Requirements 10.2, 10.3_
|
||
- **Property 28: Trade setup data completeness** — _Validates: Requirements 10.4_
|
||
- **Property 29: Trade setup pruning on data change** — _Validates: Requirements 10.5_
|
||
|
||
- [x] 10.5 Implement Watchlist service and router
|
||
- Create `app/services/watchlist_service.py`: auto-populate top-X by composite score (default 10), manual add/remove (tagged, not subject to auto-population), enforce cap (auto + 10 manual, default max 20), update auto entries on score recomputation
|
||
- Create `app/schemas/watchlist.py`: WatchlistEntryResponse (includes composite score, dimension scores, R:R ratio, SR levels)
|
||
- Create `app/routers/watchlist.py`: `GET /api/v1/watchlist`, `POST /api/v1/watchlist/{symbol}`, `DELETE /api/v1/watchlist/{symbol}` (sortable by composite, dimension, or R:R)
|
||
- _Requirements: 11.1, 11.2, 11.3, 11.4, 11.5, 11.6_
|
||
|
||
- [ ]* 10.6 Write property tests for watchlist (Properties 30-33)
|
||
- **Property 30: Watchlist auto-population** — _Validates: Requirements 11.1_
|
||
- **Property 31: Watchlist entry data completeness** — _Validates: Requirements 11.2_
|
||
- **Property 32: Manual watchlist entries persist through auto-population** — _Validates: Requirements 11.3_
|
||
- **Property 33: Watchlist size cap enforcement** — _Validates: Requirements 11.4_
|
||
|
||
- [x] 11. Checkpoint - Ensure all tests pass
|
||
- Ensure all tests pass, ask the user if questions arise.
|
||
|
||
- [x] 12. Scheduled jobs and sorting correctness
|
||
- [x] 12.1 Implement APScheduler job definitions and scheduler integration
|
||
- Create `app/scheduler.py`: define scheduled jobs for Data Collector (OHLCV fetch for all tickers, configurable frequency), Sentiment Collector (default 30 min), Fundamental Collector (default daily), R:R Scanner (configurable frequency)
|
||
- Each job: process all tracked tickers independently (one failure doesn't stop others), log errors with structured JSON, handle rate limits (record last successful ticker, resume next run)
|
||
- Wire scheduler into FastAPI lifespan (start on startup, shutdown gracefully)
|
||
- _Requirements: 4.1, 4.2, 4.3, 4.4, 4.5, 7.1, 8.2, 10.6_
|
||
|
||
- [ ]* 12.2 Write property test for scheduled collection (Property 10)
|
||
- **Property 10: Scheduled collection processes all tickers** — _Validates: Requirements 4.1, 4.3, 7.1, 8.2_
|
||
|
||
- [ ]* 12.3 Write property test for sorting correctness (Property 41)
|
||
- **Property 41: Sorting correctness** — _Validates: Requirements 1.4, 6.6, 9.6, 10.8, 11.6_
|
||
|
||
- [x] 13. Test infrastructure and shared fixtures
|
||
- [x] 13.1 Create test configuration and shared fixtures
|
||
- Create `tests/conftest.py`: test DB session fixture (transaction rollback per test), FastAPI test client fixture, mock `MarketDataProvider`, hypothesis custom strategies (`valid_ticker_symbols`, `whitespace_strings`, `valid_ohlcv_records`, `invalid_ohlcv_records`, `dimension_scores`, `weight_configs`, `sr_levels`, `sentiment_scores`, `trade_setups`)
|
||
- Create `tests/__init__.py`, `tests/unit/__init__.py`, `tests/property/__init__.py`
|
||
- _Requirements: Design (Testing Strategy)_
|
||
|
||
- [x] 14. Deployment templates and CI/CD
|
||
- [x] 14.1 Create deployment configuration files
|
||
- Create `deploy/nginx.conf` (reverse proxy for signal.thiessen.io)
|
||
- Create `deploy/stock-data-backend.service` (systemd unit file)
|
||
- Create `deploy/setup_db.sh` (idempotent DB creation + migration script)
|
||
- Create `.gitea/workflows/deploy.yml` (lint → test → deploy pipeline)
|
||
- _Requirements: Design (Deployment and Infrastructure)_
|
||
|
||
- [x] 15. Final wiring and integration
|
||
- [x] 15.1 Wire all routers into FastAPI app and verify OpenAPI docs
|
||
- Register all routers in `app/main.py` under `/api/v1/` prefix
|
||
- Verify Swagger/OpenAPI docs endpoint works at `/docs`
|
||
- Ensure all middleware (logging, error handling, auth) is applied
|
||
- _Requirements: Design Constraints (OpenAPI/Swagger, versioned URLs)_
|
||
|
||
- [ ]* 15.2 Write integration tests for key API flows
|
||
- Test end-to-end: register → login → add ticker → fetch data → get indicators → get scores → get watchlist
|
||
- Test auth enforcement: unauthenticated → 401, no-access user → 403, admin endpoints → 403 for non-admin
|
||
- Test error flows: duplicate ticker → 409, invalid OHLCV → 400, missing ticker → 404
|
||
- _Requirements: 1.1-1.6, 2.1-2.4, 12.1-12.6_
|
||
|
||
- [x] 16. Final checkpoint - Ensure all tests pass
|
||
- Ensure all tests pass, ask the user if questions arise.
|
||
|
||
## Notes
|
||
|
||
- Tasks marked with `*` are optional and can be skipped for faster MVP
|
||
- Each task references specific requirements for traceability
|
||
- Checkpoints ensure incremental validation
|
||
- Property tests validate the 41 correctness properties from the design document using `hypothesis`
|
||
- Unit tests validate specific examples and edge cases
|
||
- All code is Python 3.12+ with FastAPI, SQLAlchemy async, and PostgreSQL
|