Event study: report its own statistical limits
The v3 cutover run scored 2/4 corrections warned against v2's 3/4, which reads like a regression and is not one. Only 4 of the 11 detected corrections fall in the holdout, so recall is one event from a different headline -- and the event that flips is decided by threshold placement, not by what the score saw. "v3 without the credit sensor" catches 2025-02-21 at a *higher* threshold (35.5) than shipped v3 misses it at (32.3), because the alarm rule needs a rising edge and a lower threshold can fire outside the horizon then never reset below. Two caveats are now computed and surfaced rather than left for the reader to infer: - Holdout event count against MIN_EVENTS_FOR_CONFIDENCE. The summary sentence states how many of the detected corrections actually fall in the test period. - Warning-sensor coverage across the split. The score renormalises over what is available, so a training window predating a sensor's history freezes the threshold on a different construct than the holdout is measured against. At the cutover that is 39% of training sessions with all three sensors versus 100% of the test period, credit history beginning 2023-07-25. Restricting the threshold to sensor-matched training sessions was tested and rejected: those sessions are a calm recent stretch, so the threshold falls from 32.3 to 22.5 and false alarms rise from 3.3 to 8.6/yr. It swaps a coverage bias for a regime-selection bias. The report states its limits instead. _warning_series now returns per-session sensor counts alongside the scores. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -604,6 +604,18 @@ export interface EventStudyReport {
|
||||
warn_threshold: number;
|
||||
basket_hash: string;
|
||||
basket_asof: string;
|
||||
credit_sensor_from?: string | null;
|
||||
};
|
||||
/** How far the headline metrics can be trusted. See _reliability(). */
|
||||
reliability?: {
|
||||
events_detected: number;
|
||||
events_in_holdout: number;
|
||||
minimum_events: number;
|
||||
underpowered: boolean;
|
||||
sensors_expected: number;
|
||||
train_full_sensor_share: number;
|
||||
holdout_full_sensor_share: number;
|
||||
sensor_coverage_mismatch: boolean;
|
||||
};
|
||||
sample?: {
|
||||
start: string;
|
||||
|
||||
Reference in New Issue
Block a user