signal-platform/.kiro/specs/dashboard-enhancements/design.md

# Design Document — Dashboard Enhancements

## Overview

This design covers four enhancements to the TickerDetailPage dashboard:

1. **Sentiment drill-down** — Store OpenAI reasoning text and web search citations in the DB; expose via API; render in an expandable detail section within SentimentPanel.
2. **Fundamentals drill-down** — Track which FMP endpoints returned 402 (paid-plan-required) and surface those reasons in the API and an expandable detail section within FundamentalsPanel.
3. **TradingView-style chart** — Add mouse-wheel zoom, click-drag pan, and a crosshair overlay with price/date labels to the existing canvas-based CandlestickChart.
4. **S/R clustering** — Cluster nearby S/R levels into zones with aggregated strength, filter to top N, and render as shaded rectangles instead of dashed lines.

All changes are additive to existing components and preserve the glassmorphism UI style.

## Architecture

```mermaid
graph TD
    subgraph Backend
        OAI[OpenAI Responses API] -->|reasoning + annotations| SP[OpenAISentimentProvider]
        SP -->|SentimentData + reasoning + citations| SS[sentiment_service]
        SS -->|persist| DB[(PostgreSQL)]

        FMP[FMP Stable API] -->|402 metadata| FP[FMPFundamentalProvider]
        FP -->|FundamentalData + unavailable_fields| FS[fundamental_service]
        FS -->|persist| DB

        DB -->|query| SRS[sr_service]
        SRS -->|cluster_sr_zones| SRS
        SRS -->|SRZone list| SRAPI[/sr-levels endpoint/]
    end

    subgraph Frontend
        SRAPI -->|zones JSON| Chart[CandlestickChart]
        Chart -->|canvas render| ZR[Zone rectangles + crosshair + zoom/pan]

        SS -->|API| SentAPI[/sentiment endpoint/]
        SentAPI -->|reasoning + citations| SPan[SentimentPanel]
        SPan -->|expand/collapse| DD1[Detail Section]

        FS -->|API| FundAPI[/fundamentals endpoint/]
        FundAPI -->|unavailable_fields| FPan[FundamentalsPanel]
        FPan -->|expand/collapse| DD2[Detail Section]
    end
```

The changes touch four layers:
- **Provider layer** — Extract additional data from external API responses (OpenAI annotations, FMP 402 reasons).
- **Service layer** — Store new fields, add zone clustering logic.
- **API/Schema layer** — Extend response schemas with new fields.
- **Frontend components** — Add interactive chart features and expandable detail sections.

## Components and Interfaces

### 1. Sentiment Provider Changes (`app/providers/openai_sentiment.py`)

The `SentimentData` DTO gains two optional fields:

```python
@dataclass(frozen=True, slots=True)
class SentimentData:
    ticker: str
    classification: str
    confidence: int
    source: str
    timestamp: datetime
    reasoning: str = ""                          # NEW
    citations: list[dict[str, str]] = field(default_factory=list)  # NEW: [{"url": ..., "title": ...}]
```

The provider already parses `reasoning` from the JSON response but discards it. The change:
- Return `reasoning` from the parsed JSON in the `SentimentData`.
- Iterate `response.output` items looking for `type == "web_search_call"` output items, then extract URL annotations from the subsequent message content blocks that have `annotations` with `type == "url_citation"`. Each annotation yields `{"url": annotation.url, "title": annotation.title}`.
- If no annotations exist, return an empty list (no error).

### 2. Sentiment DB Model Changes (`app/models/sentiment.py`)

Add two columns to `SentimentScore`:

```python
reasoning: Mapped[str] = mapped_column(Text, nullable=False, default="")
citations_json: Mapped[str] = mapped_column(Text, nullable=False, default="[]")
```

Citations are stored as a JSON-encoded string (list of `{url, title}` dicts). This avoids a separate table for a simple list of links.

Alembic migration adds these two columns with defaults so existing rows are unaffected.

### 3. Sentiment Service Changes (`app/services/sentiment_service.py`)

`store_sentiment()` gains `reasoning: str` and `citations: list[dict]` parameters. It serializes citations to JSON and stores both fields.

### 4. Sentiment Schema Changes (`app/schemas/sentiment.py`)

```python
class CitationItem(BaseModel):
    url: str
    title: str

class SentimentScoreResult(BaseModel):
    id: int
    classification: Literal["bullish", "bearish", "neutral"]
    confidence: int = Field(ge=0, le=100)
    source: str
    timestamp: datetime
    reasoning: str = ""                    # NEW
    citations: list[CitationItem] = []     # NEW
```

### 5. FMP Provider Changes (`app/providers/fmp.py`)

`FundamentalData` DTO gains an `unavailable_fields` dict:

```python
@dataclass(frozen=True, slots=True)
class FundamentalData:
    ticker: str
    pe_ratio: float | None
    revenue_growth: float | None
    earnings_surprise: float | None
    market_cap: float | None
    fetched_at: datetime
    unavailable_fields: dict[str, str] = field(default_factory=dict)  # NEW: {"pe_ratio": "requires paid plan", ...}
```

In `_fetch_json_optional`, when a 402 is received, the provider records which fields map to that endpoint. The mapping:
- `ratios-ttm` → `pe_ratio`
- `financial-growth` → `revenue_growth`
- `earnings` → `earnings_surprise`

After all fetches, any field that is `None` AND whose endpoint returned 402 gets an entry in `unavailable_fields`.

### 6. Fundamentals DB Model Changes (`app/models/fundamental.py`)

Add one column:

```python
unavailable_fields_json: Mapped[str] = mapped_column(Text, nullable=False, default="{}")
```

Stored as JSON-encoded `{"field_name": "reason"}` dict.

### 7. Fundamentals Schema Changes (`app/schemas/fundamental.py`)

```python
class FundamentalResponse(BaseModel):
    symbol: str
    pe_ratio: float | None = None
    revenue_growth: float | None = None
    earnings_surprise: float | None = None
    market_cap: float | None = None
    fetched_at: datetime | None = None
    unavailable_fields: dict[str, str] = {}  # NEW
```

### 8. S/R Zone Clustering (`app/services/sr_service.py`)

New function `cluster_sr_zones()`:

```python
def cluster_sr_zones(
    levels: list[dict],
    current_price: float,
    tolerance: float = 0.02,  # 2% default clustering tolerance
    max_zones: int | None = None,
) -> list[dict]:
    """Cluster nearby S/R levels into zones.
    
    Returns list of zone dicts:
    {
        "low": float,
        "high": float,
        "midpoint": float,
        "strength": int,  # sum of constituent strengths, capped at 100
        "type": "support" | "resistance",
        "level_count": int,
    }
    """
```

Algorithm:
1. Sort levels by `price_level` ascending.
2. Greedy merge: walk sorted levels; if the next level is within `tolerance` (percentage of the current cluster midpoint) of the current cluster, merge it in. Otherwise, start a new cluster.
3. For each cluster: `low` = min price, `high` = max price, `midpoint` = (low + high) / 2, `strength` = sum of constituent strengths capped at 100.
4. Tag each zone as `"support"` if midpoint < current_price, else `"resistance"`.
5. Sort by strength descending.
6. If `max_zones` is set, return only the top N.

### 9. S/R Schema Changes (`app/schemas/sr_level.py`)

```python
class SRZoneResult(BaseModel):
    low: float
    high: float
    midpoint: float
    strength: int = Field(ge=0, le=100)
    type: Literal["support", "resistance"]
    level_count: int

class SRLevelResponse(BaseModel):
    symbol: str
    levels: list[SRLevelResult]
    zones: list[SRZoneResult] = []  # NEW
    count: int
```

### 10. S/R Router Changes (`app/routers/sr_levels.py`)

Add `max_zones` query parameter (default 6). After fetching levels, call `cluster_sr_zones()` and include zones in the response.

### 11. CandlestickChart Enhancements (`frontend/src/components/charts/CandlestickChart.tsx`)

State additions:
- `visibleRange: { start: number, end: number }` — indices into the data array for the currently visible window.
- `isPanning: boolean`, `panStartX: number` — for drag-to-pan.
- `crosshair: { x: number, y: number } | null` — cursor position for crosshair rendering.

New event handlers:
- `onWheel` — Adjust `visibleRange` (zoom in = narrow range, zoom out = widen range). Clamp to min 10 bars, max full dataset.
- `onMouseDown` / `onMouseMove` / `onMouseUp` — When zoomed in, click-drag pans the visible range left/right.
- `onMouseMove` (extended) — Track cursor position for crosshair. Draw vertical + horizontal lines and axis labels.
- `onMouseLeave` — Clear crosshair state.

The `draw()` function changes:
- Use `data.slice(visibleRange.start, visibleRange.end)` instead of full `data`.
- After drawing candles, if `crosshair` is set, draw crosshair lines and labels.
- Replace S/R dashed lines with shaded zone rectangles when `zones` prop is provided.

New prop: `zones?: SRZone[]` (from the API response).

### 12. SentimentPanel Drill-Down (`frontend/src/components/ticker/SentimentPanel.tsx`)

Add `useState<boolean>(false)` for expand/collapse. When expanded, render:
- Reasoning text in a `<p>` block.
- Citations as a list of `<a>` links with title and URL.

Toggle button uses a chevron icon below the summary metrics.

### 13. FundamentalsPanel Drill-Down (`frontend/src/components/ticker/FundamentalsPanel.tsx`)

Add `useState<boolean>(false)` for expand/collapse. Changes:
- When a metric is `null` and `unavailable_fields[field_name]` exists, show the reason text (e.g., "Requires paid plan") in amber instead of "—".
- When expanded, show data source name ("FMP"), fetch timestamp, and a list of unavailable fields with reasons.

### 14. Frontend Type Updates (`frontend/src/lib/types.ts`)

```typescript
// Sentiment additions
export interface CitationItem {
  url: string;
  title: string;
}

export interface SentimentScore {
  id: number;
  classification: 'bullish' | 'bearish' | 'neutral';
  confidence: number;
  source: string;
  timestamp: string;
  reasoning: string;       // NEW
  citations: CitationItem[]; // NEW
}

// Fundamentals additions
export interface FundamentalResponse {
  symbol: string;
  pe_ratio: number | null;
  revenue_growth: number | null;
  earnings_surprise: number | null;
  market_cap: number | null;
  fetched_at: string | null;
  unavailable_fields: Record<string, string>; // NEW
}

// S/R Zone
export interface SRZone {
  low: number;
  high: number;
  midpoint: number;
  strength: number;
  type: 'support' | 'resistance';
  level_count: number;
}

export interface SRLevelResponse {
  symbol: string;
  levels: SRLevel[];
  zones: SRZone[];  // NEW
  count: number;
}
```

## Data Models

### Database Schema Changes

#### `sentiment_scores` table — new columns

| Column | Type | Default | Description |
|--------|------|---------|-------------|
| `reasoning` | TEXT | `""` | AI reasoning text from OpenAI response |
| `citations_json` | TEXT | `"[]"` | JSON array of `{url, title}` citation objects |

#### `fundamental_data` table — new column

| Column | Type | Default | Description |
|--------|------|---------|-------------|
| `unavailable_fields_json` | TEXT | `"{}"` | JSON dict of `{field_name: reason}` for missing data |

No new tables are needed. The S/R zones are computed on-the-fly from existing `sr_levels` rows — they are not persisted.

### Alembic Migration

A single migration file adds the three new columns with server defaults so existing rows are populated automatically:

```python
op.add_column('sentiment_scores', sa.Column('reasoning', sa.Text(), server_default='', nullable=False))
op.add_column('sentiment_scores', sa.Column('citations_json', sa.Text(), server_default='[]', nullable=False))
op.add_column('fundamental_data', sa.Column('unavailable_fields_json', sa.Text(), server_default='{}', nullable=False))
```


## Correctness Properties

*A property is a characteristic or behavior that should hold true across all valid executions of a system — essentially, a formal statement about what the system should do. Properties serve as the bridge between human-readable specifications and machine-verifiable correctness guarantees.*

### Property 1: Sentiment reasoning extraction

*For any* valid OpenAI Responses API response containing a JSON body with a `reasoning` field, the `OpenAISentimentProvider.fetch_sentiment()` method should return a `SentimentData` whose `reasoning` field equals the reasoning string from the parsed JSON.

**Validates: Requirements 1.1**

### Property 2: Sentiment citations extraction

*For any* valid OpenAI Responses API response containing zero or more `url_citation` annotations across its output items, the `OpenAISentimentProvider.fetch_sentiment()` method should return a `SentimentData` whose `citations` list contains exactly the URLs and titles from those annotations, in order. When no annotations exist, the citations list should be empty (no error raised).

**Validates: Requirements 1.2, 1.4**

### Property 3: Sentiment data round-trip

*For any* sentiment record with arbitrary reasoning text and citations list, storing it via `store_sentiment()` and then retrieving it via the `/sentiment/{symbol}` API endpoint should return a response where the latest score's `reasoning` and `citations` fields match the originally stored values.

**Validates: Requirements 1.3**

### Property 4: Expanded sentiment detail displays all data

*For any* `SentimentScore` with non-empty reasoning and a list of citations, when the SentimentPanel detail section is expanded, the rendered output should contain the reasoning text and every citation's title and URL as clickable links.

**Validates: Requirements 2.2, 2.3**

### Property 5: Sentiment detail collapse hides content

*For any* SentimentPanel state where the detail section is expanded, collapsing it should result in the reasoning text and citations being hidden from the DOM while the summary metrics (classification, confidence, dimension score, source count) remain visible.

**Validates: Requirements 2.4**

### Property 6: FMP 402 reason recording

*For any* subset of supplementary FMP endpoints (ratios-ttm, financial-growth, earnings) that return HTTP 402, the `FMPFundamentalProvider.fetch_fundamentals()` method should return a `FundamentalData` whose `unavailable_fields` dict contains an entry for each corresponding metric field name with the reason "requires paid plan".

**Validates: Requirements 3.1**

### Property 7: Fundamentals unavailable_fields round-trip

*For any* fundamental data record with an arbitrary `unavailable_fields` dict, storing it via `store_fundamental()` and retrieving it via the `/fundamentals/{symbol}` API endpoint should return a response whose `unavailable_fields` matches the originally stored dict.

**Validates: Requirements 3.2**

### Property 8: Null field display depends on reason existence

*For any* `FundamentalResponse` where a metric field is null, the FundamentalsPanel should display the reason text from `unavailable_fields` (if present for that field) or a dash character "—" (if no reason exists for that field).

**Validates: Requirements 3.3, 3.4**

### Property 9: Fundamentals expanded detail content

*For any* `FundamentalResponse` with a fetch timestamp and unavailable fields, when the FundamentalsPanel detail section is expanded, the rendered output should contain the data source name, the formatted fetch timestamp, and each unavailable field's name and reason.

**Validates: Requirements 4.2**

### Property 10: Zoom adjusts visible range proportionally

*For any* dataset of length N (N ≥ 10) and any current visible range [start, end], applying a positive wheel delta (zoom in) should produce a new range that is strictly narrower (fewer bars), and applying a negative wheel delta (zoom out) should produce a new range that is strictly wider (more bars), unless already at the limit.

**Validates: Requirements 5.1, 5.2, 5.3**

### Property 11: Pan shifts visible range

*For any* dataset and any visible range that does not cover the full dataset, a horizontal drag of Δx pixels should shift the visible range start and end indices by a proportional amount in the corresponding direction, without changing the range width.

**Validates: Requirements 5.4**

### Property 12: Zoom range invariant

*For any* sequence of zoom and pan operations on a dataset of length N, the visible range should always satisfy: `end - start >= 10` AND `end - start <= N` AND `start >= 0` AND `end <= N`.

**Validates: Requirements 5.5**

### Property 13: Coordinate-to-value mapping

*For any* chart configuration with a visible price range [lo, hi] and visible data slice, the `yToPrice` function should map any y-coordinate within the chart area to a price within [lo, hi], and the `xToBarIndex` function should map any x-coordinate within the chart area to a valid index within the visible data slice.

**Validates: Requirements 6.3, 6.4**

### Property 14: Clustering merges nearby levels

*For any* set of S/R levels and a clustering tolerance T, after calling `cluster_sr_zones()`, no two distinct zones should have midpoints within T percent of each other. Equivalently, all input levels that are within T percent of each other must end up in the same zone.

**Validates: Requirements 7.2**

### Property 15: Zone strength is capped sum

*For any* SR zone produced by `cluster_sr_zones()`, the zone's strength should equal `min(100, sum(constituent_level_strengths))`.

**Validates: Requirements 7.3**

### Property 16: Zone type tagging

*For any* SR zone and current price, the zone's type should be `"support"` if the zone midpoint is less than the current price, and `"resistance"` otherwise.

**Validates: Requirements 7.4**

### Property 17: Zone filtering returns top N by strength

*For any* set of SR zones and a limit N, `cluster_sr_zones(..., max_zones=N)` should return at most N zones, and those zones should be the N zones with the highest strength scores from the full unfiltered set.

**Validates: Requirements 8.2**

## Error Handling

### Backend

| Scenario | Handling |
|----------|----------|
| OpenAI response has no `reasoning` field in JSON | Default to empty string `""` — no error |
| OpenAI response has no `url_citation` annotations | Return empty citations list — no error |
| OpenAI response JSON parse failure | Existing `ProviderError` handling unchanged |
| FMP endpoint returns 402 | Record in `unavailable_fields`, return `None` for that metric — no error |
| FMP profile endpoint fails | Existing `ProviderError` propagation unchanged |
| `citations_json` column contains invalid JSON | Catch `json.JSONDecodeError` in schema serialization, default to `[]` |
| `unavailable_fields_json` column contains invalid JSON | Catch `json.JSONDecodeError`, default to `{}` |
| `cluster_sr_zones()` receives empty levels list | Return empty zones list |
| `max_zones` is 0 or negative | Return empty zones list |

### Frontend

| Scenario | Handling |
|----------|----------|
| `reasoning` is empty string | Detail section shows "No reasoning available" placeholder |
| `citations` is empty array | Detail section omits citations subsection |
| `unavailable_fields` is empty object | All null metrics show "—" as before |
| Chart data has fewer than 10 bars | Disable zoom (show all bars, no zoom controls) |
| Wheel event fires rapidly | Debounce zoom recalculation to 1 frame via `requestAnimationFrame` |
| Zone `low` equals `high` (single-level zone) | Render as a thin line (minimum 2px height rectangle) |

## Testing Strategy

### Property-Based Testing

Library: **Hypothesis** (Python backend), **fast-check** (TypeScript frontend)

Each property test runs a minimum of 100 iterations. Each test is tagged with a comment referencing the design property:

```
# Feature: dashboard-enhancements, Property 14: Clustering merges nearby levels
```

Backend property tests (Hypothesis):
- **Property 1**: Generate random JSON strings with reasoning fields → verify extraction.
- **Property 2**: Generate mock OpenAI response objects with 0–10 annotations → verify citations list.
- **Property 3**: Generate random reasoning + citations → store → retrieve via test client → compare.
- **Property 6**: Generate random 402/200 combinations for 3 endpoints → verify unavailable_fields mapping.
- **Property 7**: Generate random unavailable_fields dicts → store → retrieve → compare.
- **Property 10–12**: Generate random datasets (10–500 bars) and zoom/pan sequences → verify range invariants.
- **Property 13**: Generate random chart dimensions and price ranges → verify coordinate mapping round-trips.
- **Property 14**: Generate random level sets (1–50 levels, prices 1–1000, strengths 1–100) and tolerances (0.5%–5%) → verify no two zones should have been merged.
- **Property 15**: Generate random level sets → cluster → verify each zone's strength = min(100, sum).
- **Property 16**: Generate random zones and current prices → verify type tagging.
- **Property 17**: Generate random zone sets and limits → verify top-N selection.

Frontend property tests (fast-check):
- **Property 4**: Generate random reasoning strings and citation lists → render SentimentPanel expanded → verify DOM content.
- **Property 5**: Generate random sentiment data → expand then collapse → verify summary visible, detail hidden.
- **Property 8**: Generate random FundamentalResponse with various null/non-null + reason combinations → verify displayed text.
- **Property 9**: Generate random FundamentalResponse → expand → verify source, timestamp, reasons in DOM.

### Unit Tests

Unit tests cover specific examples and edge cases:
- Sentiment provider with a real-shaped OpenAI response fixture (example for 1.1, 1.2).
- Sentiment provider with no annotations (edge case for 1.4).
- FMP provider with all-402 responses (edge case for 3.1).
- FMP provider with mixed 200/402 responses (example for 3.1).
- SentimentPanel default collapsed state (example for 2.5).
- FundamentalsPanel default collapsed state (example for 4.4).
- Chart with exactly 10 bars — zoom in should be blocked (edge case for 5.5).
- Chart with 1 bar — zoom disabled entirely (edge case).
- Crosshair removed on mouse leave (example for 6.5).
- `cluster_sr_zones()` with empty input (edge case).
- `cluster_sr_zones()` with all levels at the same price (edge case).
- `cluster_sr_zones()` with levels exactly at tolerance boundary (edge case).
- Default max_zones = 6 in the dashboard (example for 8.3).
- Zone with single constituent level (edge case — low == high).