signal-platform/.kiro/specs/dashboard-enhancements/design.md

Design Document — Dashboard Enhancements

Overview

This design covers four enhancements to the TickerDetailPage dashboard:

1. Sentiment drill-down — Store OpenAI reasoning text and web search citations in the DB; expose via API; render in an expandable detail section within SentimentPanel.
2. Fundamentals drill-down — Track which FMP endpoints returned 402 (paid-plan-required) and surface those reasons in the API and an expandable detail section within FundamentalsPanel.
3. TradingView-style chart — Add mouse-wheel zoom, click-drag pan, and a crosshair overlay with price/date labels to the existing canvas-based CandlestickChart.
4. S/R clustering — Cluster nearby S/R levels into zones with aggregated strength, filter to top N, and render as shaded rectangles instead of dashed lines.

All changes are additive to existing components and preserve the glassmorphism UI style.

Architecture

graph TD
    subgraph Backend
        OAI[OpenAI Responses API] -->|reasoning + annotations| SP[OpenAISentimentProvider]
        SP -->|SentimentData + reasoning + citations| SS[sentiment_service]
        SS -->|persist| DB[(PostgreSQL)]

        FMP[FMP Stable API] -->|402 metadata| FP[FMPFundamentalProvider]
        FP -->|FundamentalData + unavailable_fields| FS[fundamental_service]
        FS -->|persist| DB

        DB -->|query| SRS[sr_service]
        SRS -->|cluster_sr_zones| SRS
        SRS -->|SRZone list| SRAPI[/sr-levels endpoint/]
    end

    subgraph Frontend
        SRAPI -->|zones JSON| Chart[CandlestickChart]
        Chart -->|canvas render| ZR[Zone rectangles + crosshair + zoom/pan]

        SS -->|API| SentAPI[/sentiment endpoint/]
        SentAPI -->|reasoning + citations| SPan[SentimentPanel]
        SPan -->|expand/collapse| DD1[Detail Section]

        FS -->|API| FundAPI[/fundamentals endpoint/]
        FundAPI -->|unavailable_fields| FPan[FundamentalsPanel]
        FPan -->|expand/collapse| DD2[Detail Section]
    end

The changes touch four layers:

• Provider layer — Extract additional data from external API responses (OpenAI annotations, FMP 402 reasons).
• Service layer — Store new fields, add zone clustering logic.
• API/Schema layer — Extend response schemas with new fields.
• Frontend components — Add interactive chart features and expandable detail sections.

Components and Interfaces

1. Sentiment Provider Changes (app/providers/openai_sentiment.py)

The SentimentData DTO gains two optional fields:

@dataclass(frozen=True, slots=True)
class SentimentData:
    ticker: str
    classification: str
    confidence: int
    source: str
    timestamp: datetime
    reasoning: str = ""                          # NEW
    citations: list[dict[str, str]] = field(default_factory=list)  # NEW: [{"url": ..., "title": ...}]

The provider already parses reasoning from the JSON response but discards it. The change:

• Return reasoning from the parsed JSON in the SentimentData.
• Iterate response.output items looking for type == "web_search_call" output items, then extract URL annotations from the subsequent message content blocks that have annotations with type == "url_citation". Each annotation yields {"url": annotation.url, "title": annotation.title}.
• If no annotations exist, return an empty list (no error).

2. Sentiment DB Model Changes (app/models/sentiment.py)

Add two columns to SentimentScore:

reasoning: Mapped[str] = mapped_column(Text, nullable=False, default="")
citations_json: Mapped[str] = mapped_column(Text, nullable=False, default="[]")

Citations are stored as a JSON-encoded string (list of {url, title} dicts). This avoids a separate table for a simple list of links.

Alembic migration adds these two columns with defaults so existing rows are unaffected.

3. Sentiment Service Changes (app/services/sentiment_service.py)

store_sentiment() gains reasoning: str and citations: list[dict] parameters. It serializes citations to JSON and stores both fields.

4. Sentiment Schema Changes (app/schemas/sentiment.py)

class CitationItem(BaseModel):
    url: str
    title: str

class SentimentScoreResult(BaseModel):
    id: int
    classification: Literal["bullish", "bearish", "neutral"]
    confidence: int = Field(ge=0, le=100)
    source: str
    timestamp: datetime
    reasoning: str = ""                    # NEW
    citations: list[CitationItem] = []     # NEW

5. FMP Provider Changes (app/providers/fmp.py)

FundamentalData DTO gains an unavailable_fields dict:

@dataclass(frozen=True, slots=True)
class FundamentalData:
    ticker: str
    pe_ratio: float | None
    revenue_growth: float | None
    earnings_surprise: float | None
    market_cap: float | None
    fetched_at: datetime
    unavailable_fields: dict[str, str] = field(default_factory=dict)  # NEW: {"pe_ratio": "requires paid plan", ...}

In _fetch_json_optional, when a 402 is received, the provider records which fields map to that endpoint. The mapping:

• ratios-ttm → pe_ratio
• financial-growth → revenue_growth
• earnings → earnings_surprise

After all fetches, any field that is None AND whose endpoint returned 402 gets an entry in unavailable_fields.

6. Fundamentals DB Model Changes (app/models/fundamental.py)

Add one column:

unavailable_fields_json: Mapped[str] = mapped_column(Text, nullable=False, default="{}")

Stored as JSON-encoded {"field_name": "reason"} dict.

7. Fundamentals Schema Changes (app/schemas/fundamental.py)

class FundamentalResponse(BaseModel):
    symbol: str
    pe_ratio: float | None = None
    revenue_growth: float | None = None
    earnings_surprise: float | None = None
    market_cap: float | None = None
    fetched_at: datetime | None = None
    unavailable_fields: dict[str, str] = {}  # NEW

8. S/R Zone Clustering (app/services/sr_service.py)

New function cluster_sr_zones():

def cluster_sr_zones(
    levels: list[dict],
    current_price: float,
    tolerance: float = 0.02,  # 2% default clustering tolerance
    max_zones: int | None = None,
) -> list[dict]:
    """Cluster nearby S/R levels into zones.
    
    Returns list of zone dicts:
    {
        "low": float,
        "high": float,
        "midpoint": float,
        "strength": int,  # sum of constituent strengths, capped at 100
        "type": "support" | "resistance",
        "level_count": int,
    }
    """

Algorithm:

1. Sort levels by price_level ascending.
2. Greedy merge: walk sorted levels; if the next level is within tolerance (percentage of the current cluster midpoint) of the current cluster, merge it in. Otherwise, start a new cluster.
3. For each cluster: low = min price, high = max price, midpoint = (low + high) / 2, strength = sum of constituent strengths capped at 100.
4. Tag each zone as "support" if midpoint < current_price, else "resistance".
5. Sort by strength descending.
6. If max_zones is set, return only the top N.

9. S/R Schema Changes (app/schemas/sr_level.py)

class SRZoneResult(BaseModel):
    low: float
    high: float
    midpoint: float
    strength: int = Field(ge=0, le=100)
    type: Literal["support", "resistance"]
    level_count: int

class SRLevelResponse(BaseModel):
    symbol: str
    levels: list[SRLevelResult]
    zones: list[SRZoneResult] = []  # NEW
    count: int

10. S/R Router Changes (app/routers/sr_levels.py)

Add max_zones query parameter (default 6). After fetching levels, call cluster_sr_zones() and include zones in the response.

11. CandlestickChart Enhancements (frontend/src/components/charts/CandlestickChart.tsx)

State additions:

• visibleRange: { start: number, end: number } — indices into the data array for the currently visible window.
• isPanning: boolean, panStartX: number — for drag-to-pan.
• crosshair: { x: number, y: number } | null — cursor position for crosshair rendering.

New event handlers:

• onWheel — Adjust visibleRange (zoom in = narrow range, zoom out = widen range). Clamp to min 10 bars, max full dataset.
• onMouseDown / onMouseMove / onMouseUp — When zoomed in, click-drag pans the visible range left/right.
• onMouseMove (extended) — Track cursor position for crosshair. Draw vertical + horizontal lines and axis labels.
• onMouseLeave — Clear crosshair state.

The draw() function changes:

• Use data.slice(visibleRange.start, visibleRange.end) instead of full data.
• After drawing candles, if crosshair is set, draw crosshair lines and labels.
• Replace S/R dashed lines with shaded zone rectangles when zones prop is provided.

New prop: zones?: SRZone[] (from the API response).

12. SentimentPanel Drill-Down (frontend/src/components/ticker/SentimentPanel.tsx)

Add useState<boolean>(false) for expand/collapse. When expanded, render:

• Reasoning text in a <p> block.
• Citations as a list of <a> links with title and URL.

Toggle button uses a chevron icon below the summary metrics.

13. FundamentalsPanel Drill-Down (frontend/src/components/ticker/FundamentalsPanel.tsx)

Add useState<boolean>(false) for expand/collapse. Changes:

• When a metric is null and unavailable_fields[field_name] exists, show the reason text (e.g., "Requires paid plan") in amber instead of "—".
• When expanded, show data source name ("FMP"), fetch timestamp, and a list of unavailable fields with reasons.

14. Frontend Type Updates (frontend/src/lib/types.ts)

// Sentiment additions
export interface CitationItem {
  url: string;
  title: string;
}

export interface SentimentScore {
  id: number;
  classification: 'bullish' | 'bearish' | 'neutral';
  confidence: number;
  source: string;
  timestamp: string;
  reasoning: string;       // NEW
  citations: CitationItem[]; // NEW
}

// Fundamentals additions
export interface FundamentalResponse {
  symbol: string;
  pe_ratio: number | null;
  revenue_growth: number | null;
  earnings_surprise: number | null;
  market_cap: number | null;
  fetched_at: string | null;
  unavailable_fields: Record<string, string>; // NEW
}

// S/R Zone
export interface SRZone {
  low: number;
  high: number;
  midpoint: number;
  strength: number;
  type: 'support' | 'resistance';
  level_count: number;
}

export interface SRLevelResponse {
  symbol: string;
  levels: SRLevel[];
  zones: SRZone[];  // NEW
  count: number;
}

Data Models

Database Schema Changes

sentiment_scores table — new columns

Column	Type	Default	Description
reasoning	TEXT	""	AI reasoning text from OpenAI response
citations_json	TEXT	"[]"	JSON array of {url, title} citation objects

fundamental_data table — new column

Column	Type	Default	Description
unavailable_fields_json	TEXT	"{}"	JSON dict of {field_name: reason} for missing data

No new tables are needed. The S/R zones are computed on-the-fly from existing sr_levels rows — they are not persisted.

Alembic Migration

A single migration file adds the three new columns with server defaults so existing rows are populated automatically:

op.add_column('sentiment_scores', sa.Column('reasoning', sa.Text(), server_default='', nullable=False))
op.add_column('sentiment_scores', sa.Column('citations_json', sa.Text(), server_default='[]', nullable=False))
op.add_column('fundamental_data', sa.Column('unavailable_fields_json', sa.Text(), server_default='{}', nullable=False))

Correctness Properties

A property is a characteristic or behavior that should hold true across all valid executions of a system — essentially, a formal statement about what the system should do. Properties serve as the bridge between human-readable specifications and machine-verifiable correctness guarantees.

Property 1: Sentiment reasoning extraction

For any valid OpenAI Responses API response containing a JSON body with a reasoning field, the OpenAISentimentProvider.fetch_sentiment() method should return a SentimentData whose reasoning field equals the reasoning string from the parsed JSON.

Validates: Requirements 1.1

Property 2: Sentiment citations extraction

For any valid OpenAI Responses API response containing zero or more url_citation annotations across its output items, the OpenAISentimentProvider.fetch_sentiment() method should return a SentimentData whose citations list contains exactly the URLs and titles from those annotations, in order. When no annotations exist, the citations list should be empty (no error raised).

Validates: Requirements 1.2, 1.4

Property 3: Sentiment data round-trip

For any sentiment record with arbitrary reasoning text and citations list, storing it via store_sentiment() and then retrieving it via the /sentiment/{symbol} API endpoint should return a response where the latest score's reasoning and citations fields match the originally stored values.

Validates: Requirements 1.3

Property 4: Expanded sentiment detail displays all data

For any SentimentScore with non-empty reasoning and a list of citations, when the SentimentPanel detail section is expanded, the rendered output should contain the reasoning text and every citation's title and URL as clickable links.

Validates: Requirements 2.2, 2.3

Property 5: Sentiment detail collapse hides content

For any SentimentPanel state where the detail section is expanded, collapsing it should result in the reasoning text and citations being hidden from the DOM while the summary metrics (classification, confidence, dimension score, source count) remain visible.

Validates: Requirements 2.4

Property 6: FMP 402 reason recording

For any subset of supplementary FMP endpoints (ratios-ttm, financial-growth, earnings) that return HTTP 402, the FMPFundamentalProvider.fetch_fundamentals() method should return a FundamentalData whose unavailable_fields dict contains an entry for each corresponding metric field name with the reason "requires paid plan".

Validates: Requirements 3.1

Property 7: Fundamentals unavailable_fields round-trip

For any fundamental data record with an arbitrary unavailable_fields dict, storing it via store_fundamental() and retrieving it via the /fundamentals/{symbol} API endpoint should return a response whose unavailable_fields matches the originally stored dict.

Validates: Requirements 3.2

Property 8: Null field display depends on reason existence

For any FundamentalResponse where a metric field is null, the FundamentalsPanel should display the reason text from unavailable_fields (if present for that field) or a dash character "—" (if no reason exists for that field).

Validates: Requirements 3.3, 3.4

Property 9: Fundamentals expanded detail content

For any FundamentalResponse with a fetch timestamp and unavailable fields, when the FundamentalsPanel detail section is expanded, the rendered output should contain the data source name, the formatted fetch timestamp, and each unavailable field's name and reason.

Validates: Requirements 4.2

Property 10: Zoom adjusts visible range proportionally

For any dataset of length N (N ≥ 10) and any current visible range [start, end], applying a positive wheel delta (zoom in) should produce a new range that is strictly narrower (fewer bars), and applying a negative wheel delta (zoom out) should produce a new range that is strictly wider (more bars), unless already at the limit.

Validates: Requirements 5.1, 5.2, 5.3

Property 11: Pan shifts visible range

For any dataset and any visible range that does not cover the full dataset, a horizontal drag of Δx pixels should shift the visible range start and end indices by a proportional amount in the corresponding direction, without changing the range width.

Validates: Requirements 5.4

Property 12: Zoom range invariant

For any sequence of zoom and pan operations on a dataset of length N, the visible range should always satisfy: end - start >= 10 AND end - start <= N AND start >= 0 AND end <= N.

Validates: Requirements 5.5

Property 13: Coordinate-to-value mapping

For any chart configuration with a visible price range [lo, hi] and visible data slice, the yToPrice function should map any y-coordinate within the chart area to a price within [lo, hi], and the xToBarIndex function should map any x-coordinate within the chart area to a valid index within the visible data slice.

Validates: Requirements 6.3, 6.4

Property 14: Clustering merges nearby levels

For any set of S/R levels and a clustering tolerance T, after calling cluster_sr_zones(), no two distinct zones should have midpoints within T percent of each other. Equivalently, all input levels that are within T percent of each other must end up in the same zone.

Validates: Requirements 7.2

Property 15: Zone strength is capped sum

For any SR zone produced by cluster_sr_zones(), the zone's strength should equal min(100, sum(constituent_level_strengths)).

Validates: Requirements 7.3

Property 16: Zone type tagging

For any SR zone and current price, the zone's type should be "support" if the zone midpoint is less than the current price, and "resistance" otherwise.

Validates: Requirements 7.4

Property 17: Zone filtering returns top N by strength

For any set of SR zones and a limit N, cluster_sr_zones(..., max_zones=N) should return at most N zones, and those zones should be the N zones with the highest strength scores from the full unfiltered set.

Validates: Requirements 8.2

Error Handling

Backend

Scenario	Handling
OpenAI response has no reasoning field in JSON	Default to empty string "" — no error
OpenAI response has no url_citation annotations	Return empty citations list — no error
OpenAI response JSON parse failure	Existing ProviderError handling unchanged
FMP endpoint returns 402	Record in unavailable_fields, return None for that metric — no error
FMP profile endpoint fails	Existing ProviderError propagation unchanged
citations_json column contains invalid JSON	Catch json.JSONDecodeError in schema serialization, default to []
unavailable_fields_json column contains invalid JSON	Catch json.JSONDecodeError, default to {}
cluster_sr_zones() receives empty levels list	Return empty zones list
max_zones is 0 or negative	Return empty zones list

Frontend

Scenario	Handling
reasoning is empty string	Detail section shows "No reasoning available" placeholder
citations is empty array	Detail section omits citations subsection
unavailable_fields is empty object	All null metrics show "—" as before
Chart data has fewer than 10 bars	Disable zoom (show all bars, no zoom controls)
Wheel event fires rapidly	Debounce zoom recalculation to 1 frame via requestAnimationFrame
Zone low equals high (single-level zone)	Render as a thin line (minimum 2px height rectangle)

Testing Strategy

Property-Based Testing

Library: Hypothesis (Python backend), fast-check (TypeScript frontend)

Each property test runs a minimum of 100 iterations. Each test is tagged with a comment referencing the design property:

# Feature: dashboard-enhancements, Property 14: Clustering merges nearby levels

Backend property tests (Hypothesis):

• Property 1: Generate random JSON strings with reasoning fields → verify extraction.
• Property 2: Generate mock OpenAI response objects with 0–10 annotations → verify citations list.
• Property 3: Generate random reasoning + citations → store → retrieve via test client → compare.
• Property 6: Generate random 402/200 combinations for 3 endpoints → verify unavailable_fields mapping.
• Property 7: Generate random unavailable_fields dicts → store → retrieve → compare.
• Property 10–12: Generate random datasets (10–500 bars) and zoom/pan sequences → verify range invariants.
• Property 13: Generate random chart dimensions and price ranges → verify coordinate mapping round-trips.
• Property 14: Generate random level sets (1–50 levels, prices 1–1000, strengths 1–100) and tolerances (0.5%–5%) → verify no two zones should have been merged.
• Property 15: Generate random level sets → cluster → verify each zone's strength = min(100, sum).
• Property 16: Generate random zones and current prices → verify type tagging.
• Property 17: Generate random zone sets and limits → verify top-N selection.

Frontend property tests (fast-check):

• Property 4: Generate random reasoning strings and citation lists → render SentimentPanel expanded → verify DOM content.
• Property 5: Generate random sentiment data → expand then collapse → verify summary visible, detail hidden.
• Property 8: Generate random FundamentalResponse with various null/non-null + reason combinations → verify displayed text.
• Property 9: Generate random FundamentalResponse → expand → verify source, timestamp, reasons in DOM.

Unit Tests

Unit tests cover specific examples and edge cases:

• Sentiment provider with a real-shaped OpenAI response fixture (example for 1.1, 1.2).
• Sentiment provider with no annotations (edge case for 1.4).
• FMP provider with all-402 responses (edge case for 3.1).
• FMP provider with mixed 200/402 responses (example for 3.1).
• SentimentPanel default collapsed state (example for 2.5).
• FundamentalsPanel default collapsed state (example for 4.4).
• Chart with exactly 10 bars — zoom in should be blocked (edge case for 5.5).
• Chart with 1 bar — zoom disabled entirely (edge case).
• Crosshair removed on mouse leave (example for 6.5).
• cluster_sr_zones() with empty input (edge case).
• cluster_sr_zones() with all levels at the same price (edge case).
• cluster_sr_zones() with levels exactly at tolerance boundary (edge case).
• Default max_zones = 6 in the dashboard (example for 8.3).
• Zone with single constituent level (edge case — low == high).