dennisthiessen/signal-platform
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
181cfe65888ad4a0e6955445fe52808fc2bdbedd
signal-platform/.kiro/specs/stock-data-backend/tasks.md
Dennis Thiessen 61ab24490d
Some checks failed
Deploy / lint (push) Failing after 7s
Details
Deploy / test (push) Has been skipped
Details
Deploy / deploy (push) Has been skipped
Details
first commit
2026-02-20 17:31:01 +01:00

19 KiB
Raw Blame History

Implementation Plan: Stock Data Backend

Overview

Incremental build of the investing-signal platform: foundation first (config, DB, models, auth), then domain services (tickers, OHLCV, ingestion, indicators, S/R, sentiment, fundamentals), then scoring/ranking (scoring engine, R:R scanner, watchlist), then scheduled jobs, deployment templates, and final wiring. Each step builds on the previous and ends integrated.

Tasks

  • 1. Project scaffolding, configuration, and database foundation

    • 1.1 Create project structure with pyproject.toml, .env.example, alembic.ini, and app/ package

      • Create pyproject.toml with dependencies: fastapi, uvicorn, sqlalchemy[asyncio], asyncpg, alembic, pydantic-settings, python-jose, passlib[bcrypt], apscheduler, httpx, alpaca-py, google-genai, hypothesis
      • Create .env.example with all environment variables from design
      • Create app/__init__.py, app/config.py (pydantic-settings Settings class)
      • Create app/database.py (async SQLAlchemy engine, session factory, connection pooling)
      • Requirements: Design Constraints (connection pooling, config)

    • 1.2 Create all SQLAlchemy ORM models and Alembic initial migration

      • Create app/models/__init__.py and model files: ticker.py, ohlcv.py, user.py, sentiment.py, fundamental.py, score.py, sr_level.py, trade_setup.py, watchlist.py, settings.py
      • Implement all 12 entities from the ERD: User, Ticker, OHLCVRecord, SentimentScore, FundamentalData, SRLevel, DimensionScore, CompositeScore, TradeSetup, WatchlistEntry, SystemSetting, IngestionProgress
      • Include composite unique constraints, indexes, and cascade deletes per design
      • Initialize Alembic (alembic/env.py) and generate initial migration
      • Requirements: 2.1, 2.2, Design Constraints (composite index on ticker+date)

    • 1.3 Create shared schemas, exception hierarchy, and API envelope

      • Create app/schemas/common.py with APIEnvelope model (status, data, error)
      • Create app/middleware.py with global exception handler mapping AppError subclasses to JSON envelope responses
      • Create exception classes: AppError, ValidationError, NotFoundError, DuplicateError, AuthenticationError, AuthorizationError, ProviderError, RateLimitError
      • Requirements: Design Constraints (JSON envelope, HTTP status codes)

    • 1.4 Create FastAPI app entry point with lifespan, health check, and dependency injection

      • Create app/main.py with FastAPI app, lifespan handler (DB pool startup/shutdown, default admin creation)
      • Create app/dependencies.py with Depends() factories for DB session, current user, admin guard
      • Create app/routers/health.py with unauthenticated /api/v1/health endpoint
      • Wire health router into app
      • Requirements: 13.1, Design Constraints (health check, graceful shutdown, versioned URLs)

  • 2. Authentication and admin services

    • 2.1 Implement Auth Service and auth router

      • Create app/services/auth_service.py: registration (configurable on/off, creates no-access user), login (bcrypt verify, JWT generation with 60-min expiry), token validation
      • Create app/schemas/auth.py: RegisterRequest, LoginRequest, TokenResponse
      • Create app/routers/auth.py: POST /api/v1/auth/register, POST /api/v1/auth/login
      • Implement JWT middleware in app/dependencies.py for get_current_user and require_admin
      • Requirements: 12.1, 12.2, 12.3, 12.4, 12.5, 12.6

    • * 2.2 Write property tests for auth (Properties 34-38)

      • Property 34: Registration creates no-access user — Validates: Requirements 12.1
      • Property 35: Registration disabled rejects all attempts — Validates: Requirements 12.2
      • Property 36: Login returns valid JWT — Validates: Requirements 12.3
      • Property 37: Invalid credentials return generic error — Validates: Requirements 12.4
      • Property 38: Access control enforcement — Validates: Requirements 12.5

    • 2.3 Implement Admin Service and admin router

      • Create app/services/admin_service.py: grant/revoke access, toggle registration, list users, reset passwords, create accounts, system settings CRUD, data cleanup (delete old OHLCV/sentiment/fundamentals preserving tickers/users/scores), job control
      • Create app/schemas/admin.py: UserManagement, SystemSettingUpdate, DataCleanupRequest
      • Create app/routers/admin.py: admin-only endpoints under /api/v1/admin/
      • Requirements: 13.1, 13.2, 13.3, 13.4, 13.5

    • * 2.4 Write property tests for admin (Properties 39-40)

      • Property 39: Admin user management operations — Validates: Requirements 13.2
      • Property 40: Data cleanup preserves structure — Validates: Requirements 13.4

  • 3. Checkpoint - Ensure all tests pass

    • Ensure all tests pass, ask the user if questions arise.

  • 4. Ticker management and OHLCV price storage

    • 4.1 Implement Ticker Registry service and router

      • Create app/services/ticker_service.py: add (validate non-empty, uppercase, alphanumeric, check uniqueness), delete (cascade all associated data), list (sorted alphabetically)
      • Create app/schemas/ticker.py: TickerCreate, TickerResponse
      • Create app/routers/tickers.py: POST /api/v1/tickers, GET /api/v1/tickers, DELETE /api/v1/tickers/{symbol}
      • Requirements: 1.1, 1.2, 1.3, 1.4, 1.5, 1.6

    • * 4.2 Write property tests for ticker management (Properties 1-4)

      • Property 1: Ticker creation round-trip — Validates: Requirements 1.1
      • Property 2: Duplicate ticker rejection — Validates: Requirements 1.2
      • Property 3: Whitespace ticker rejection — Validates: Requirements 1.3
      • Property 4: Ticker deletion cascades — Validates: Requirements 1.5

    • 4.3 Implement Price Store service and OHLCV router

      • Create app/services/price_service.py: upsert OHLCV (validate high >= low, prices >= 0, volume >= 0, date <= today, ticker exists), query by ticker + date range
      • Create app/schemas/ohlcv.py: OHLCVCreate, OHLCVResponse
      • Create app/routers/ohlcv.py: POST /api/v1/ohlcv, GET /api/v1/ohlcv/{symbol}
      • On upsert: invalidate LRU cache for ticker, mark composite score as stale
      • Requirements: 2.1, 2.2, 2.3, 2.4

    • * 4.4 Write property tests for OHLCV (Properties 5-7)

      • Property 5: OHLCV storage round-trip — Validates: Requirements 2.1, 2.2
      • Property 6: OHLCV validation rejects invalid records — Validates: Requirements 2.3
      • Property 7: OHLCV rejects unregistered tickers — Validates: Requirements 2.4

  • 5. Market data provider and ingestion pipeline

    • 5.1 Implement provider protocols and concrete implementations

      • Create app/providers/protocol.py: MarketDataProvider Protocol (fetch_ohlcv), SentimentProvider Protocol (fetch_sentiment), FundamentalProvider Protocol (fetch_fundamentals)
      • Create app/providers/alpaca.py: Alpaca OHLCV provider using alpaca-py SDK — fetches daily bars by ticker and date range
      • Create app/providers/gemini_sentiment.py: Gemini sentiment provider using google-genai with search grounding — sends structured prompt per ticker, parses JSON response (classification + confidence)
      • Create app/providers/fmp.py: Financial Modeling Prep fundamentals provider using httpx — fetches P/E, revenue growth, earnings surprise, market cap
      • Requirements: Design Constraints (provider behind interface)

    • 5.2 Implement Ingestion Pipeline service and router

      • Create app/services/ingestion_service.py: fetch + upsert with rate-limit handling (track last_ingested_date, return partial progress on rate limit, resume from last date + 1 day), provider error handling (descriptive error, no data modification)
      • Create app/routers/ingestion.py: POST /api/v1/ingestion/fetch/{symbol}
      • Requirements: 3.1, 3.2, 3.3, 3.4

    • * 5.3 Write property tests for ingestion (Properties 8-9)

      • Property 8: Provider error preserves existing data — Validates: Requirements 3.2, 7.3, 8.3
      • Property 9: Rate-limit resume continuity — Validates: Requirements 3.3, 3.4, 4.5

  • 6. Checkpoint - Ensure all tests pass

    • Ensure all tests pass, ask the user if questions arise.

  • 7. Technical analysis and S/R detection

    • 7.1 Implement LRU cache wrapper with invalidation

      • Create app/cache.py: LRU cache wrapper (max 1000 entries) keyed on ticker + date range + indicator type, with per-ticker invalidation method
      • Requirements: Design Constraints (LRU cache)

    • 7.2 Implement Technical Analysis service and indicators router

      • Create app/services/indicator_service.py: compute ADX (28+ bars), EMA (period+1 bars, default 20/50), RSI (15+ bars, 14-period), ATR (15+ bars, 14-period), Volume Profile (20+ bars, POC/Value Area/HVN/LVN), Pivot Points (5+ bars, swing highs/lows)
      • Each indicator returns raw values + normalized 0-100 score
      • Implement EMA cross signal (bullish/bearish/neutral based on short vs long EMA comparison)
      • Enforce minimum data requirements, return error if insufficient
      • Create app/schemas/indicator.py: IndicatorRequest, IndicatorResponse, EMACrossResponse
      • Create app/routers/indicators.py: GET /api/v1/indicators/{symbol}/{indicator_type}, GET /api/v1/indicators/{symbol}/ema-cross
      • Requirements: 5.1, 5.2, 5.3, 5.4

    • * 7.3 Write property tests for indicators (Properties 11-14)

      • Property 11: Score bounds invariant — Validates: Requirements 5.2, 6.2, 9.1
      • Property 12: Indicator minimum data enforcement — Validates: Requirements 5.4
      • Property 13: EMA cross directional bias — Validates: Requirements 5.3
      • Property 14: Indicator computation determinism — Validates: Requirements 5.1

    • 7.4 Implement S/R Detector service and router

      • Create app/services/sr_service.py: detect SR levels from Volume Profile (HVN/LVN) and Pivot Points (swing highs/lows), assign strength scores (0-100 based on price respect count), merge levels within tolerance (default 0.5%), tag as support/resistance relative to current price, recalculate on new OHLCV data
      • Create app/schemas/sr_level.py: SRLevelResponse
      • Create app/routers/sr_levels.py: GET /api/v1/sr-levels/{symbol} (sorted by strength descending)
      • Requirements: 6.1, 6.2, 6.3, 6.4, 6.5, 6.6

    • * 7.5 Write property tests for S/R detection (Properties 15-17)

      • Property 15: SR level support/resistance tagging — Validates: Requirements 6.3
      • Property 16: SR level merging within tolerance — Validates: Requirements 6.5
      • Property 17: SR level detection from data — Validates: Requirements 6.1

  • 8. Sentiment and fundamental data services

    • 8.1 Implement Sentiment service and router

      • Create app/services/sentiment_service.py: store sentiment records (classification, confidence, source, timestamp), compute dimension score with time-decay weighted average over configurable lookback window (default 24h)
      • Create app/schemas/sentiment.py: SentimentResponse
      • Create app/routers/sentiment.py: GET /api/v1/sentiment/{symbol}
      • Requirements: 7.1, 7.2, 7.3, 7.4

    • * 8.2 Write property tests for sentiment (Properties 18-19)

      • Property 18: Sentiment score data shape — Validates: Requirements 7.2
      • Property 19: Sentiment dimension score uses time decay — Validates: Requirements 7.4

    • 8.3 Implement Fundamental Data service and router

      • Create app/services/fundamental_service.py: store fundamental data (P/E, revenue growth, earnings surprise, market cap), mark fundamental dimension score as stale on new data
      • Create app/schemas/fundamental.py: FundamentalResponse
      • Create app/routers/fundamentals.py: GET /api/v1/fundamentals/{symbol}
      • Requirements: 8.1, 8.2, 8.3, 8.4

    • * 8.4 Write property test for fundamentals (Property 20)

      • Property 20: Fundamental data storage round-trip — Validates: Requirements 8.1

  • 9. Checkpoint - Ensure all tests pass

    • Ensure all tests pass, ask the user if questions arise.

  • 10. Scoring engine, R:R scanner, and watchlist

    • 10.1 Implement Scoring Engine service and router

      • Create app/services/scoring_service.py: compute dimension scores (technical, sr_quality, sentiment, fundamental, momentum) each 0-100, compute composite score as weighted average of available dimensions with re-normalized weights, staleness marking/recomputation on demand, weight update triggers full recomputation
      • Create app/schemas/score.py: ScoreResponse, WeightUpdateRequest, RankingResponse
      • Create app/routers/scores.py: GET /api/v1/scores/{symbol}, GET /api/v1/rankings, PUT /api/v1/scores/weights
      • Requirements: 9.1, 9.2, 9.3, 9.4, 9.5, 9.6, 9.7

    • * 10.2 Write property tests for scoring (Properties 21-25)

      • Property 21: Composite score is weighted average — Validates: Requirements 9.2
      • Property 22: Missing dimensions re-normalize weights — Validates: Requirements 9.3
      • Property 23: Staleness marking on data change — Validates: Requirements 9.4
      • Property 24: Stale score recomputation on demand — Validates: Requirements 9.5
      • Property 25: Weight update triggers full recomputation — Validates: Requirements 9.7

    • 10.3 Implement R:R Scanner service and router

      • Create app/services/rr_scanner_service.py: scan tickers for trade setups (long: target = nearest SR above, stop = entry - ATR×multiplier; short: target = nearest SR below, stop = entry + ATR×multiplier), filter by R:R threshold (default 3:1), recalculate/prune on data change, skip tickers without sufficient SR/ATR data
      • Create app/schemas/trade_setup.py: TradeSetupResponse
      • Create app/routers/trades.py: GET /api/v1/trades (sorted by R:R desc, secondary composite desc, optional direction filter)
      • Requirements: 10.1, 10.2, 10.3, 10.4, 10.5, 10.6, 10.7, 10.8

    • * 10.4 Write property tests for R:R scanner (Properties 26-29)

      • Property 26: Trade setup R:R threshold filtering — Validates: Requirements 10.1
      • Property 27: Trade setup computation correctness — Validates: Requirements 10.2, 10.3
      • Property 28: Trade setup data completeness — Validates: Requirements 10.4
      • Property 29: Trade setup pruning on data change — Validates: Requirements 10.5

    • 10.5 Implement Watchlist service and router

      • Create app/services/watchlist_service.py: auto-populate top-X by composite score (default 10), manual add/remove (tagged, not subject to auto-population), enforce cap (auto + 10 manual, default max 20), update auto entries on score recomputation
      • Create app/schemas/watchlist.py: WatchlistEntryResponse (includes composite score, dimension scores, R:R ratio, SR levels)
      • Create app/routers/watchlist.py: GET /api/v1/watchlist, POST /api/v1/watchlist/{symbol}, DELETE /api/v1/watchlist/{symbol} (sortable by composite, dimension, or R:R)
      • Requirements: 11.1, 11.2, 11.3, 11.4, 11.5, 11.6

    • * 10.6 Write property tests for watchlist (Properties 30-33)

      • Property 30: Watchlist auto-population — Validates: Requirements 11.1
      • Property 31: Watchlist entry data completeness — Validates: Requirements 11.2
      • Property 32: Manual watchlist entries persist through auto-population — Validates: Requirements 11.3
      • Property 33: Watchlist size cap enforcement — Validates: Requirements 11.4

  • 11. Checkpoint - Ensure all tests pass

    • Ensure all tests pass, ask the user if questions arise.

  • 12. Scheduled jobs and sorting correctness

    • 12.1 Implement APScheduler job definitions and scheduler integration

      • Create app/scheduler.py: define scheduled jobs for Data Collector (OHLCV fetch for all tickers, configurable frequency), Sentiment Collector (default 30 min), Fundamental Collector (default daily), R:R Scanner (configurable frequency)
      • Each job: process all tracked tickers independently (one failure doesn't stop others), log errors with structured JSON, handle rate limits (record last successful ticker, resume next run)
      • Wire scheduler into FastAPI lifespan (start on startup, shutdown gracefully)
      • Requirements: 4.1, 4.2, 4.3, 4.4, 4.5, 7.1, 8.2, 10.6

    • * 12.2 Write property test for scheduled collection (Property 10)

      • Property 10: Scheduled collection processes all tickers — Validates: Requirements 4.1, 4.3, 7.1, 8.2

    • * 12.3 Write property test for sorting correctness (Property 41)

      • Property 41: Sorting correctness — Validates: Requirements 1.4, 6.6, 9.6, 10.8, 11.6

  • 13. Test infrastructure and shared fixtures

    • 13.1 Create test configuration and shared fixtures
      • Create tests/conftest.py: test DB session fixture (transaction rollback per test), FastAPI test client fixture, mock MarketDataProvider, hypothesis custom strategies (valid_ticker_symbols, whitespace_strings, valid_ohlcv_records, invalid_ohlcv_records, dimension_scores, weight_configs, sr_levels, sentiment_scores, trade_setups)
      • Create tests/__init__.py, tests/unit/__init__.py, tests/property/__init__.py
      • Requirements: Design (Testing Strategy)

  • 14. Deployment templates and CI/CD

    • 14.1 Create deployment configuration files
      • Create deploy/nginx.conf (reverse proxy for signal.thiessen.io)
      • Create deploy/stock-data-backend.service (systemd unit file)
      • Create deploy/setup_db.sh (idempotent DB creation + migration script)
      • Create .gitea/workflows/deploy.yml (lint → test → deploy pipeline)
      • Requirements: Design (Deployment and Infrastructure)

  • 15. Final wiring and integration

    • 15.1 Wire all routers into FastAPI app and verify OpenAPI docs

      • Register all routers in app/main.py under /api/v1/ prefix
      • Verify Swagger/OpenAPI docs endpoint works at /docs
      • Ensure all middleware (logging, error handling, auth) is applied
      • Requirements: Design Constraints (OpenAPI/Swagger, versioned URLs)

    • * 15.2 Write integration tests for key API flows

      • Test end-to-end: register → login → add ticker → fetch data → get indicators → get scores → get watchlist
      • Test auth enforcement: unauthenticated → 401, no-access user → 403, admin endpoints → 403 for non-admin
      • Test error flows: duplicate ticker → 409, invalid OHLCV → 400, missing ticker → 404
      • Requirements: 1.1-1.6, 2.1-2.4, 12.1-12.6

  • 16. Final checkpoint - Ensure all tests pass

    • Ensure all tests pass, ask the user if questions arise.

Notes

  • Tasks marked with * are optional and can be skipped for faster MVP
  • Each task references specific requirements for traceability
  • Checkpoints ensure incremental validation
  • Property tests validate the 41 correctness properties from the design document using hypothesis
  • Unit tests validate specific examples and edge cases
  • All code is Python 3.12+ with FastAPI, SQLAlchemy async, and PostgreSQL
Reference in New Issue View Git Blame Copy Permalink