Files

T

dennisthiessen bee5a5ce89 Finalize GTL and retire S/R research harness

2026-07-13 17:58:04 +02:00

47 KiB

Raw Blame History

S/R levels: detection quality, the target exit, and the entry gate

Date: 2026-07-12 Question that started it: are our support/resistance levels built the way best practice says they should be, and do we actually use them that way?

Final answer: one level model should not serve two different jobs. The clean Structural S/R detector now supplies persisted chart and alert structure. The transient Gate Target Ladder preserves the broad historical-price traffic proposals that the setup screen depends on. Its headline target affects entry qualification only; honoring it as a take-profit is decisively worse. The final volume-free implementation reproduced the production candidate set and portfolio exactly. The sections below retain the investigation that led to that split.

1. How the levels are built today

Update (2026-07-12 detector rewrite): several gaps below were addressed in sr_service / indicator_service — close-bin VP, local-peak HVN, POC/VAH/VAL as candidates, LVN dropped from S/R, pivot prominence + lookbacks, rejection- weighted recency strength, ATR-adaptive merge, hard cap, round numbers. The table documents the pre-rewrite failure modes measured on the snapshot; keep it for historical context. Re-measure density on prod after deploy if gate rates shift.

app/services/sr_service.py::detect_sr_levels (post-rewrite):

Candidates = VP POC / VAH / VAL / local HVN peaks (lookback 252) + prominent swing pivots (lookback 504) + nearby round numbers.
Strength = rejection-weighted touches on last 252 bars with recency decay (pass-throughs down-weighted); method base + confluence on merge.
Nearby levels merged with ATR-adaptive tolerance (clamped ~0.4–1.5%); capped (~16, interleaved S/R); tagged support if below spot, else resistance.

Where the pre-rewrite detector departed from best practice

Measured on backtest_snapshots/prod.sqlite (AAPL, 1261 bars, spot $308.63):

Gap	Evidence
POC / VAH / VAL are computed then discarded. `sr_service` reads only `hvn`/`lvn`. The canonical volume-profile levels never become S/R.	POC $148.32, VAH $230.45 — unused
HVN = "any bin above the mean", so nearly every bin is a candidate. A real HVN is a local peak in the histogram.	8 of 20 bins HVN, other 12 LVN → 73 levels, median spacing $2.44 (0.79% of spot) — a price grid, not detected structure
HVN and LVN are scored and used identically, though they encode opposite dynamics (acceptance vs. rejection).	both appended as plain candidates
Volume is double-counted: a bar's full volume is added to every bin it spans rather than distributed.	binned total = 1.48× true volume
"Touch" = level fell inside the bar's range — a pass-through counts the same as a rejection. Strength therefore measures how central a price is in the 5-year range, not how often price reversed there.	21 of 73 levels pin at exactly 100 → the `sr_strength` magnet in the probability model is near-constant
No recency decay, unbounded lookback.	35 AAPL "support" levels sit >35% below spot
Round-number levels absent — the mechanism with the best empirical support (Osler 2000).	not implemented

Not a defect: the pivot window=2 is the standard 5-bar Williams fractal. What it lacked pre-rewrite is a prominence filter — AAPL yielded 338 pivots over 1261 bars, one every ~3.7 bars.

The structural problem: resistance famine

Levels are tagged relative to spot, so a stock near its highs has almost nothing above it. Across all 504 tickers in the snapshot, grouped by proximity to the 52-week high:

Population	Median resistance levels	% with <3
≥98% of 52w high — what the momentum gate buys	1	64%
90–98%	7	12%
<90%	22	0%

AAPL at $308: 71 support levels, 2 resistance levels. 26 tickers have zero.

This matters because _build_setups_at does if not targets: continue — no resistance above ⇒ no long setup at all, and <3 targets adds a target-availability conflict. The detector is structurally most blind exactly where the momentum edge is strongest. recommendation_service.py:521 already says so out loud: "price is extended near highs (no resistance target above), so no high-conviction long setup is available."

2. Does the target even act as an exit? No.

Production runs paper_exit_mode = "atr_trailing" (DEFAULT_EXIT_MODE), and _atr_trailing_close(direction, entry, init_stop, atr_multiplier, hold_days, ...) does not take target as a parameter. Only the non-default mode == "target" branch consults it.

So the S/R target's real causal role is:

the entry gate — rr ≥ 1.5, primary-target prob ≥ 20%, ≥3 targets, and no-resistance-above ⇒ no setup;
the displayed target table.

It decides whether you enter and plays no part in how you exit.

3. Experiment: should the target be honored as a take-profit? — REJECTED

Dennis's proposal: keep the 3× ATR trail, but also take profit when the S/R target is hit. This was not representable in the simulator (exit_policy is one string; the existing "target" policy replaces the trail). Added atr_trail3_target, which runs both — trade ends at whichever comes first.

Identical 80/20 momentum entry and qualification for every row — the exit is the only policy that changes. (Trade counts still vary 303–427, because exits free portfolio slots at different times; the conclusion is robust to that.) Report: reports/backtest-20260712-sr-target-exit.json

exit policy	Sharpe	CAGR	MaxDD	trades	win%
`low20`	2.04	53.2%	21.9%	305	37.7
`atr_trail3` (production)	2.04	50.4%	21.4%	320	37.5
`hold`	2.00	51.9%	22.2%	303	38.6
`technical40`	2.00	51.9%	22.2%	303	38.6
`sma50`	1.76	40.8%	24.4%	385	33.8
`target` (take-profit, no trail)	1.59	32.4%	25.0%	415	41.9
`atr_trail3_target` (trail + take-profit)	1.47	28.9%	23.5%	427	40.0

Decision: do not ship. The target must not become an exit.

Adding the take-profit to the trail: Sharpe 2.04 → 1.47, CAGR halved (50.4% → 28.9%), and drawdown got worse (21.4% → 23.5%). No risk compensation.
Win rate rose (37.5% → 40.0%) — the tell. You win more often and earn far less: the take-profit converts the few 5R/8R/15R runners into 1.5R wins while every loser still costs a full −1R. Momentum's edge is that right tail.
The combination (1.47) is worse than the take-profit alone (1.59): once upside is capped at the target, the trail's benefit (riding a winner far past any target) is gone but its cost (shakeouts on pullbacks) remains. Worst of both.

This confirms and extends the note at backtest_service.py:450 — swept fixed take-profits never found an interior optimum; the S/R target is no better.

Caveat: single in-sample run over full history. The effect is large (CAGR halved), not marginal.

4. Open: is the S/R entry gate net-positive?

The target is now proven useless as an exit, so the gate is its only justification — and the gate is what starves the momentum names.

Evidence for keeping it (gate_ablation, prod baseline): dropping the R:R floor halves per-setup expectancy, 0.583 → 0.301 net avg R (momentum_only = 0.345). Inside the tradeable pool the S/R-derived R:R floor is doing real selection — it favors names whose nearest resistance is far away, i.e. clear air above.

But that ablation cannot see the famine: it re-qualifies candidates that already exist, and starved names never enter the candidate set (if not targets: continue). So "remove the floors" is the wrong test — it just reproduces the rows above.

The right test changes target generation: when a direction has no S/R target, synthesize one at k×ATR so the name becomes a candidate. One variable moves; the previously-vetoed names now trade. Implemented behind BACKTEST_ATR_TARGET_FALLBACK=<k> (k=3 matches the trail; rr = 3/1.5 = 2.0, which clears the 1.5 floor, and an aligned momentum name lands ~34% probability, clearing the 20% floor).

Read the result against the production baseline (atr_trail3, Sharpe 2.04):

> 2.04 → the veto costs money; let the breakouts in.
≈ 2.04 → famine is a wash; a detector rewrite is cosmetic.
< 2.04 → the veto earns its keep by keeping us out of over-extended names, and S/R gating is vindicated.

Result: the veto EARNS ITS KEEP. Keep S/R in the gate.

Treatment reports/backtest-20260712-sr-gate-ablation-treatment.json vs control reports/backtest-20260711-prod-baseline.json. Admitting the vetoed names is a big change: qualified setups go 1089 → 4230 (~4×).

Production exit (atr_trail3), full history:

	Sharpe	CAGR	MaxDD	Calmar	trades
control (veto ON)	2.04	50.4%	21.4%	2.36	320
treatment (veto OFF)	1.82	58.6%	21.0%	2.79	404

Full history alone looks like a genuine trade-off — more return, more volatility, Sharpe down but Calmar up. The lookback split is what settles it:

window	control (veto ON)	treatment (veto OFF)
6m	Sharpe 2.87, CAGR 76.6%, DD 8.1%	Sharpe 1.50, CAGR 53.6%, DD 15.8%
1y	Sharpe 2.47, CAGR 66.8%, DD 8.8%	Sharpe 1.69, CAGR 66.0%, DD 15.8%
3y	Sharpe 2.12, CAGR 52.3%, DD 17.7%	Sharpe 2.11, CAGR 75.9%, DD 21.0%
5y	Sharpe 1.83, CAGR 38.8%, DD 21.4%	Sharpe 1.62, CAGR 44.9%, DD 21.0%
all	Sharpe 2.04, CAGR 50.4%, DD 21.4%	Sharpe 1.82, CAGR 58.6%, DD 21.0%

Per-setup expectancy: all_floors net avg R 0.583 → 0.280.

Decision: do not ship the fallback. Keep the gate as it is. The flat 3×ATR fallback is worse on Sharpe and on per-setup expectancy, and production needs no further defense than that.

But be careful what this run does and does not prove

It does not isolate the famine hypothesis. The fallback fires on any empty generate_targets result — and that includes the ATR/R:R distance filters in TargetGenerator (target closer than 1 ATR, or beyond max_atr_multiple), not just "no resistance above." Measured at the last bar across 502 tickers, of the long setups the fallback admits:

26 (35%) have genuinely no resistance above — the clear-air famine case
49 (65%) do have resistance above; the ATR/R:R filters rejected it — a different population entirely

That matches the report's own tell: qualified setups exploded 1089 → 4230 (~4×) while candidates rose only ~16%. So the degradation may be driven mostly by that 65%, and the clear-air breakouts this investigation was about are a minority of what was admitted.

What the run actually supports: "a flat 3×ATR fallback for all S/R-starved setups degrades performance." It does not support the stronger claim that a stock in clear air is a worse risk-adjusted buy, or that the veto is functioning as an over-extension filter. That mechanism is unproven.

The window split is also less clean than it first looks. The verdict rests on the two smallest samples — 6m (n=30) and 1y (n=72) — where Sharpe 2.87 is noise-dominated. The statistically sturdier 3y window (n=230 → 303) shows equal Sharpe (2.12 vs 2.11) with substantially higher treatment CAGR (52.3% → 75.9%). Full-history Calmar also favors the treatment (2.79 vs 2.36). So the result is metric- and window-dependent; only the flat-fallback rejection is solid.

Second contamination (by design): the fallback gives every admitted name the same rr = 3/1.5 = 2.0, so there is no R:R discrimination within the admitted set.

To actually test the famine, the fallback must fire only when there is no resistance above (not on ATR/R:R filter misses). That is the clear-air run below.

4b. The clean test: fire the fallback ONLY in clear air — the veto DOES cost money

BACKTEST_FALLBACK_CLEAR_AIR_ONLY=1 restricts the fallback to setups with no S/R level ahead at all, excluding the 65% that were merely ATR/R:R distance-filter misses. Same whole-portfolio simulation, same 10-slot book, same momentum ranking.

Report: reports/backtest-20260712-sr-gate-ablation-clearair.json

Production exit (atr_trail3), full history:

arm	Sharpe	CAGR	MaxDD	Calmar	trades
control — veto ON (production)	2.04	50.4%	21.4%	2.36	320
blanket fallback (contaminated)	1.82	58.6%	21.0%	2.79	404
clear-air-only fallback	2.07	62.3%	20.1%	3.10	363

Strictly better than production on all three headline metrics at once — higher Sharpe, ~12 points more CAGR, and lower drawdown. Qualified setups 1089 → 2072 (vs. 4230 for the blanket version).

By lookback (production strategy):

window	control (veto ON)	clear-air fallback
6m (n=30→38)	Sharpe 2.87, CAGR 76.6%, DD 8.1%	Sharpe 2.21, CAGR 78.8%, DD 13.0%
1y (n=72→86)	Sharpe 2.47, CAGR 66.8%, DD 8.8%	Sharpe 2.28, CAGR 87.6%, DD 12.9%
3y (n=230→270)	Sharpe 2.12, CAGR 52.3%, DD 17.7%	Sharpe 2.18, CAGR 68.6%, DD 14.3%
5y (n=320→363)	Sharpe 1.83, CAGR 38.8%, DD 21.4%	Sharpe 1.85, CAGR 47.6%, DD 20.1%
all	Sharpe 2.04, CAGR 50.4%, DD 21.4%	Sharpe 2.07, CAGR 62.3%, DD 20.1%

In every statistically sturdy window (3y, 5y, all) the clear-air fallback wins on Sharpe, CAGR and drawdown. The short windows (6m n=30, 1y n=72 — noise-dominated) favor control on Sharpe/DD while the treatment still earns more (1y CAGR 66.8% → 87.6%).

This reverses §4 and confirms the hypothesis that opened the investigation. The S/R veto on clear-air names was costing money; the blanket run masked it because the 65% loophole population (ATR/R:R filter misses) is genuinely bad and dominated the result. Isolate the two, and they pull in opposite directions:

clear-air names (no resistance above): portfolio-accretive
ATR/R:R distance-filter misses: portfolio-destructive

Per-setup expectancy is consistent with this: net avg R all_floors — control 0.583, clear-air 0.402, blanket 0.280. The admitted clear-air setups are individually a bit weaker, but they carry the highest momentum ranks, so they win book slots and deliver outsized portfolio returns.

Caveats before shipping: in-sample, single snapshot; the fallback still assigns a constant rr = 2.0 to every admitted name (no discrimination within the set). Needs an out-of-sample run before production — see §4c, which is where it comes undone.

4c. Out-of-sample holdout — the §4b result does NOT survive

Everything in §4b is in-sample: the rule was chosen by looking at the same 5 years it was then graded on. The portfolio_monitor lookbacks (6m/1y/3y/5y) are not a holdout — they are nested windows all ending today, so each one overlaps the data the idea came from.

Real split (BACKTEST_HOLDOUT_SPLIT=2024-07-01, production strategy, disjoint books):

train = entries before 2024-07-01 (~3y)
test = entries on/after 2024-07-01 (~2y, never informed the rule)

Reports: reports/backtest-20260712-holdout-control.json, reports/backtest-20260712-holdout-clearair.json

window	arm	Sharpe	CAGR	MaxDD	trades
train (2022-06 → 2024-08)	control	1.31	29.6%	21.4%	174
train	clear-air	1.63	42.6%	20.1%	191
test (2024-07 → 2026-07)	control	2.78	73.3%	11.7%	150
test	clear-air	2.45	83.0%	14.3%	176

Harness bug, found and fixed 2026-07-12. The train row first reported Sharpe 0.95 / CAGR 14.6% — wrong. Its equity curve ran to the end of the data while its entries stopped at the split, so the book sat in flat cash for two years and deflated its own metrics. _simulate_portfolio now truncates the calendar to hold_days after the last entry whenever end_date is set. The verdict is unaffected — it rests on the test row, whose entries and curve both start at the split and were always clean. But the broken numbers looked like a result, and nearly produced a false conclusion ("the first half of the sample was mediocre"). Corrected numbers above.

In train the clear-air rule wins on every metric. Out of sample it does not. On the held-out two years it delivers more raw return (+9.7pp CAGR) but at lower Sharpe (2.78 → 2.45) and higher drawdown (11.7% → 14.3%).

So the §4b headline — "strictly better on all three metrics" — was an in-sample artifact. Out of sample the rule is not a free win; it is a risk/return trade: it buys extra return by taking more risk, and on a risk-adjusted basis it is slightly worse than production.

Decision: do NOT ship the clear-air fallback. This project's decision metric is Sharpe throughout (every ranking in the report, and the 2026-07-10 primary-target A/B was accepted on Sharpe 1.51 → 2.00). By that standard the honest read of the only uncontaminated evidence is no improvement.

Notes for anyone revisiting:

Both arms show a large regime shift (train Sharpe ~1.3–1.6, test Sharpe ~2.5–2.8) — the test window was simply a much better market. That is why relative comparison within a window is the only valid read.
n = 150/176 in test is decent but not large; the Sharpe gap (0.33) is not overwhelming. This is "not confirmed," not "definitively refuted."
The famine hypothesis is therefore real but not exploitable as tried: the clear-air names do add return (consistently, in both train and test), but the flat 3×ATR target admits them at a risk cost that eats the risk-adjusted benefit. A better target model for those names (§ next runs) is the remaining avenue.

A wholesale "ATR target for everyone" variant was deliberately not run as the headline: with a fixed k×ATR target and a 1.5×ATR stop, rr = k/1.5 is constant across every name, which erases the very selection the 0.301 credits. A loss there would be uninterpretable.

5. Reproducing

Both research paths are off by default — the default report is byte-identical to the shipped baseline (5 exit rows, no fallback), and the full unit suite including the backtest↔prod parity guard passes.

# Exit book incl. the rejected take-profit rows
BACKTEST_RESEARCH_EXITS=1 python scripts/run_backtest_snapshot.py \
    backtest_snapshots/prod.sqlite --workers 7 --allow-spawn

# S/R gate ablation: synthesize a 3xATR target where S/R offers none
BACKTEST_ATR_TARGET_FALLBACK=3 python scripts/run_backtest_snapshot.py \
    backtest_snapshots/prod.sqlite --workers 7 --allow-spawn

Note --allow-spawn is required on Windows: _mp_context() has no fork/ forkserver there and silently falls back to a single thread without it.

6. Standing decisions

Measured:

The target must not be an exit. Tested, rejected, decisively — Sharpe 2.04 → 1.47, CAGR halved. Momentum's edge is the right tail; a take-profit truncates it. (§3)
Do NOT ship the clear-air fallback — it failed out-of-sample. In-sample it looked strictly better (Sharpe 2.04 → 2.07, CAGR 50.4% → 62.3%, DD 21.4% → 20.1%), but on a genuine holdout (entries after 2024-07-01, never seen by the rule) it is worse on Sharpe (2.78 → 2.45) and Calmar, better only on raw CAGR (+9.7pp). The in-sample "free win" was an artifact. Production gate stays as-is. (§4b, §4c)
The famine is real, but not exploitable as tried. Clear-air names do add return consistently (train and test) — the veto genuinely leaves money on the table. But a flat 3×ATR target admits them at a risk cost that cancels the risk-adjusted benefit. (§4c)
Do NOT relax the veto indiscriminately. The ATR/R:R distance-filter misses (65% of a blanket fallback) are portfolio-destructive and swamp everything — Sharpe 1.82, net avg R 0.280. The two populations pull in opposite directions and must be separated. (§4)

Reasoned, not measured — treat as hypotheses:

The detector's flaws probably don't reach P&L directly. No run ever varied detection quality — "good S/R vs bad S/R → P&L" has never been measured. Fix the §1 gaps for the displayed levels and the UX; do not promise a return improvement.

Method note (the expensive lesson): the in-sample result in §4b was clean, large, consistent across five nested windows — and still didn't survive a holdout. Nested lookbacks are not out-of-sample. Split by entry date before believing anything.

7. Archived S/R v2 investigation (2026-07-12/13)

The detector rewrite was decomposed into the causal arms below. They are names in the historical experiment record, not supported runtime configuration:

arm	behavior
`production_control`	deployed detector plus legacy 1.5 primary selection
`rr_aligned_control`	deployed detector; primary selection uses activation `min_rr`
`rewrite`	rewritten detector with activation-aligned primary selection
`soft_zones`	rewrite plus max-strength/confluence zone aggregation
`confirmed_rounds`	soft zones; standalone rounds need two rejection clusters
`gate_v2`	confirmed rounds plus uncapped gate evidence

The matrix runner, comparator, environment switch, and candidate-level audit were removed when the investigation closed. The compact comparison JSONs, cohort CSVs, and this narrative retain the decisions; Git history retains the raw implementation and reports for forensic reconstruction.

Validation result: confirmed_rounds is rejected and is no longer a lockable arm. It remains in the corrected training matrix only to preserve the causal experiment record. The first training reports used an entry end bound without forwarding it to the portfolio calendar, leaving each book in flat cash through the test period. The simulator now treats BACKTEST_ENTRY_END as an inclusive entry bound and truncates the calendar after the final position can resolve.

The isolated rr_aligned_control validation also failed (Sharpe 2.78 to 1.32, CAGR 73.3% to 33.1%, drawdown 11.7% to 18.4%). The live 1.5 primary-selection behavior is therefore frozen: although it predates the 2.0 activation gate, it acts as a useful selectivity mechanism. The final detector matrix holds that behavior constant and varies only detection/zone policy:

rewrite_legacy_primary
soft_zones_legacy_primary
confirmed_rounds_legacy_primary
gate_v2_legacy_primary

The post-2024 interval has informed earlier research, so this is validation rather than a pristine holdout; do not sweep variants on it. No deployment follows automatically. A lower validation Sharpe or higher drawdown remains a no-ship result even when CAGR rises.

8. Final detector-only result: no rewritten gate arm advances

The last matrix froze production's effective gate (primary min_rr=1.5, activation min_rr=2.0) and varied only level detection/zone policy on entries through 2024-06-30. Corrected portfolio calendars end after the last position can resolve; there is no flat-cash tail.

arm	Sharpe	CAGR	MaxDD	qualified	net avg R	ex-top-5%
production control	1.28	28.8%	21.4%	676	0.230	0.066
rewrite + legacy primary	0.96	21.0%	22.2%	1,200	0.037	-0.102
soft zones + legacy primary	1.08	25.1%	17.8%	1,161	0.045	-0.096
confirmed rounds + legacy primary	1.14	22.1%	20.2%	570	0.189	0.045
gate v2 + legacy primary	0.87	16.7%	20.7%	604	0.187	0.041

No arm advances to validation. The raw rewrite retains only 249 of 676 production setups, removes 427 good setups, and adds 951 setups with negative expectancy. Round confirmation repairs the added cohort but still removes 430 production setups whose 30-day average (+0.680R) exceeds the additions (+0.511R). Uncapping recovers only 16 of those missing setups. The old detector averages 43.3 gate levels versus 15.0 rewritten and 24.0 rewritten-uncapped levels.

Standing no-ship decision: keep both the deployed detector and the legacy 1.5 primary-selection behavior in the trading path. The rewritten structure may only proceed as a separately computed display model. Do not merge this research branch into production as-is.

Hidden-feature isolation

The deployed detector accidentally measures long-memory historical price traffic rather than genuine S/R. A dedicated matrix holds the complete gate fixed and changes one legacy component at a time:

legacy_geometry_neutral: old locations, every merged strength fixed at 50;
legacy_pivots_only: unfiltered full-history pivots, no VP grid;
legacy_traffic_grid_only: deployed HVN+LVN grid, no pivots.

The traffic matrix first established whether geometry, pivots, or the range-occupancy grid reproduced production on pre-2024 training data.

Corrected training result:

arm	qualified	Sharpe	CAGR	MaxDD	net avg R	ex-top-5%
production control	676	1.28	28.8%	21.4%	0.230	0.066
corrected neutral geometry	505	1.48	33.5%	18.2%	0.230	0.087
pivots only	717	1.38	32.9%	25.2%	0.236	0.076
traffic grid only	504	1.72	41.8%	16.8%	0.271	0.136

The traffic grid is the first research arm to beat control simultaneously on Sharpe, CAGR, drawdown, and robust expectancy. It retains 264 production setups at +0.214R ex-top-5%, adds 240 at +0.051R, and removes 412 at only +0.010R. Against corrected neutral geometry, only 239 qualified setups overlap; the 265 traffic-only setups return +0.140R ex-top-5% versus +0.073R for the 266 neutral-only setups. This is different selection, not merely a lower trade count.

The old volume_profile name is misleading. Its helper returns both HVN and LVN bins, so their union retains almost every one of the 20 evenly spaced centers over the expanding historical high-low range (19.987 levels on average in the audit). The later strength calculation does not use volume; it counts bars whose ranges cross each center. The candidate feature is therefore:

a normalized, expanding 20-bin price-range grid;
range-touch occupancy strength;
no full-history pivot ladder or pivot/grid strength saturation.

Two final training arms isolate the first two items explicitly:

legacy_range_grid_touch: all 20 range centers, no volume calculation, legacy touch strength;
legacy_range_grid_neutral: identical centers, strength fixed at 50 after clustering.

The touch arm reproduced legacy_traffic_grid_only exactly: all 121,464 candidates, 504 qualified setups, cohort membership, expectancy, and portfolio metrics match. Volume contributes nothing. Neutral strength won the training portfolio comparison (Sharpe 1.98 versus 1.72), but failed the locked validation:

validation arm	Sharpe	CAGR	MaxDD	net avg R	ex-top-5%
production control	2.78	73.3%	11.7%	0.174	0.022
neutral range grid	1.85	43.2%	15.2%	0.178	0.039

The neutral grid is a no-ship. The validation failure prompted a causal audit of the control rather than another detector sweep. One relationship survives both periods: dense legacy ladders are a proxy for a wide multiplicative price range.

control cohort	training ex-top-5%	validation ex-top-5%
at least 70 legacy levels	+0.165R	+0.185R
fewer than 70 levels	+0.004R	-0.379R

Level count is not independently useful after controlling for the last 504 trading days' range. For log(max(high) / min(low)) >= 1.0315 (about a 2.8x high/low ratio), the overlap cohort returns +0.292R training and +0.322R validation ex-top-5%. High density without high range returns -0.096R and +0.029R. Correlation between the explicit range and legacy level count is 0.864 training and 0.822 validation.

This isolates the hidden feature as a two-year realized price-excursion factor, accidentally encoded by how many full-history pivots survive a 0.5% merge. It is not evidence that the arbitrary lines are structural. A rounded threshold of log range >= 1.0 remains positive across a 0.9/1.0/1.1 sensitivity plateau.

Two diagnostic-only arms now test whether the explicit scalar replaces the side effect:

production_range504: deployed targets plus the explicit range gate;
rewrite_range504_legacy_primary: clean targets, frozen primary selection, plus the identical range gate.

Result:

training arm	qualified	Sharpe	CAGR	MaxDD	net avg R	ex-top-5%
production control	676	1.28	28.8%	21.4%	0.230	0.066
production + range504	180	1.65	31.4%	15.9%	0.476	0.250
clean rewrite	1,200	0.96	21.0%	22.2%	0.037	-0.102
clean rewrite + range504	291	1.46	31.5%	18.4%	0.292	0.158

The scalar recovers most of the detector rewrite's regression. The clean arm now beats the original production baseline on Sharpe, CAGR, drawdown, and robust expectancy. Old target geometry retains a smaller edge over the equally filtered clean arm.

The residual cohort is target selection, not another range feature. The clean arm retains 63 exceptionally strong common setups (+0.665R ex-top-5%), adds 228 weak setups (+0.018R), and misses 117 strong old-geometry setups (+0.175R). Of the clean-only additions, 156 have a standalone round-number primary; all 162 round-only qualified setups have fewer than two observed rejections and return -0.024R ex-top-5%. Structural-only and round-confluent primaries return +0.394R and +0.369R.

For 113 of the 117 missed setups, the clean detector does produce a target, but its selected primary averages 1.56R; 77 land in the 1.5-2R veto band. A final two-arm residual matrix therefore holds the clean detector and range factor fixed:

rewrite_range504_structural_legacy_primary: exclude standalone round targets, retain the deployed 1.5 primary floor;
rewrite_range504_structural_primary2: identical, but select the primary from targets clearing 2.0R.

Full-period production comparison

The frozen rewrite_range504_structural_legacy_primary candidate was run beside a fresh production_control over the complete snapshot with identical portfolio and exit settings. The retained decision files are reports/sr-full-production-vs-candidate-comparison.json and its cohort CSV; the redundant full candidate-row reports were removed after consolidation.

This is an apples-to-apples full-history diagnostic against the current live production path. It is not a new untouched holdout because the post-2024 data was already inspected while isolating the range factor.

Full-period results reject the clean range-gated candidate as a replacement: it improves robust setup expectancy and drawdown, but cuts the qualified set from 1,086 to 290 and the live-path book from 321 to 170 trades. Its 73 unique qualified symbols are also concentrated (36.2% of setups in the top ten names), so overlapping setup expectancy does not translate into independent portfolio opportunity.

Structural confirmation as a ranking overlay

The next experiment preserves the production detector, setup geometry, qualified universe, activation gate, and live exit. For each production setup, the clean detector is evaluated point-in-time only to attach a binary feature: whether rewrite_range504_structural_legacy_primary also clears its core gate. That confirmation receives a single pre-registered 5% weight:

overlay_rank = 95% * production_80_20_rank + 5% * structural_confirmation

There is deliberately no weight sweep and no union with clean-only setups. The report must first reproduce the production qualified count and production book; otherwise the comparison is invalid. The candidate advances only if the overlay improves full-period Sharpe, does not worsen drawdown, and retains at least 90% of production CAGR. The 1-year and 6-month rows must not both deteriorate. This remains contaminated full-history research, not promotion validation.

Result: reject the overlay. Production parity is exact, but the overlay reduces full-period Sharpe from 2.03 to 2.00 and CAGR from 50.0% to 48.4%; max drawdown improves from 21.4% to 20.0%. Confirmation has real standalone quality (+0.263R ex-top-5% versus +0.024R), but little marginal ranking value: confirmed setups already average production rank 92.3 and 46.1% come from ten symbols. The 5% boost changes ordering on only 45 active dates and overweights the same concentrated names.

Explicit gate target ladder

The legacy detector's volume-profile label is misleading. It returns both HVN and LVN bins, whose union is the complete 20-bin price-range grid, then adds unfiltered pivots and touch strength. For the gate this behaves as a broad target-proposal ladder, not human-facing support/resistance.

Component name: Gate Target Ladder (GTL). "Structural S/R" names the separate, persisted human-facing model. "Gate Target Ladder" names this transient screening component and avoids implying that its dense proposals are real support/resistance.

Runtime decision flow

flowchart TD
    O["Ticker OHLCV history"] --> STRUCT["Structural S/R<br/>clean detector"]
    STRUCT --> STORE[("Persist SRLevel")]
    STORE --> HUMAN["Charts and alerts"]

    O --> LADDER["Gate Target Ladder<br/>20 range centers + 5-bar pivots"]
    LADDER --> SCORE["Count historical price traffic<br/>strength + 0.5% merge + side tag"]
    SCORE --> SCAN{"Directional proposal<br/>with R:R ≥ 1.5?"}
    SCAN -->|no| NOSETUP["No setup for that direction"]
    SCAN -->|yes| ZONES["Cluster 2% target zones<br/>use reachable near edge"]
    ZONES --> FILTER["ATR-distance filter<br/>retain up to 5 near-to-far candidates"]
    FILTER --> PROB["Estimate target-before-stop<br/>reach probability"]
    PROB --> PRIMARY["Headline = most likely candidate<br/>clearing R:R ≥ 1.5 and probability ≥ 20%"]
    PRIMARY --> ACTIVATE{"Activation gate<br/>headline R:R ≥ 2.0<br/>probability ≥ 20%<br/>momentum/direction pass?"}
    ACTIVATE -->|no| OBS["Store as unqualified observation"]
    ACTIVATE -->|yes| QUAL["Eligible for production book"]
    QUAL --> EXIT["ATR stop/trail or max hold<br/>target is never an exit"]

Step by step:

scan_ticker loads the ticker's OHLCV history and computes the 1.5× ATR initial stop. It does not query persisted SRLevel rows for targets.
detect_gate_target_ladder creates 20 evenly spaced centers over the observed low/high range and adds unfiltered five-bar swing highs/lows. The implementation performs no volume calculation.
Each proposal is scored by the share of historical bars whose range crosses it. Proposals within 0.5% are merged, their traffic strengths combine, and they are tagged support/resistance relative to the latest close. The scanner materializes them with negative transient IDs; they are never persisted.
A direction exists only when at least one proposal clears the scanner's 1.5 R:R floor. This is setup construction, not the later live activation gate.
The recommendation layer clusters proposals into 2% target zones, uses each zone's reachable near edge, removes unsuitable ATR distances, and keeps up to five candidates spanning near, moderate and far distances.
Each retained candidate gets a target-before-stop reach probability based on distance, R:R, traffic strength and signal alignment.
The headline target is the most likely candidate clearing both R:R ≥ 1.5 and probability ≥ 20%. If none does, the most likely target overall remains headline so a distant high-R:R lottery target cannot game qualification.
The separate live gate then requires headline R:R ≥ 2.0 and probability ≥ 20%, plus the residual-momentum and direction rules. A traded setup still exits only through the ATR stop/trail or maximum hold.

Ticker-chart diagnostic

The optional GTL traffic overlay uses a volume-profile-like layout because price-axis bars make the ladder's density easy to read. The comparison stops at the layout:

Profile	Bar width measures	Valid interpretation
Volume profile	Traded volume assigned to a price bin	Where trading activity was accepted
GTL price traffic	Relative count of historical OHLCV bars crossing a GTL proposal	How strongly the legacy gate geometry revisited that price

The chart renders GTL traffic as violet bars extending left from the current price axis. It includes only proposals inside the displayed price range, so old far-away ladder levels do not compress the candles. Hover reveals price, crossings, capped strength, side and source. The overlay is off by default, loaded only on demand from GET /gate-target-ladder/{symbol}, and must remain visually distinct from persisted Structural S/R.

The production GTL replaces only the irrelevant volume pass with the complete range grid. It retains pivots, touch strength, merge geometry, primary selection, qualification, ranking, and exit behavior. Grid levels are labelled range_grid, making the internal purpose explicit.

The arm advances only on exact parity with the full-period production control: no added or removed qualified setups and identical production-book Sharpe, CAGR, drawdown, and trade count. Passing parity supports a dual-purpose design: clean structure for charts and alerts, explicit target ladder for the gate. Failing parity means the supposedly irrelevant volume pass still affects an edge case and must be located before any architectural change.

Result: exact parity. Both arms produce 1,086 qualified setups from 202,765 candidates, with 1,086 retained, zero added, and zero removed. Both production books have Sharpe 2.03, CAGR 50.0%, max drawdown 21.4%, and 321 trades. The retained cohort is +0.2086R net average and +0.0492R after removing the top 5%. This proves that neither volume nor the volume_profile interpretation is part of the deployed edge.

Implementation decision: keep the clean detect_sr_levels output persisted as human-facing S/R for charts and alerts. The scanner instead builds detect_gate_target_ladder directly from its OHLCV window, materializes it only for the current scan, and never writes those proposal levels to SRLevel. The primary-target selector retains its independently researched 1.5 floor; the later live activation gate remains 2.0, and the ATR-trailing exit is unchanged. The production_gtl backtest model calls the same pure helper as the live scanner, so the final full-period rerun is an implementation-parity check rather than another detector experiment.

Final implementation-parity result after commit 8161c35: pass. The regenerated full-period report differs from the pre-integration parity report only in generated_at; all 202,765 candidates, 1,086 qualified setups, cohort statistics, and portfolio results are unchanged. This closes the local backtest gate for the dual-purpose implementation. It does not itself authorize or perform a production deployment.

GTL tuning matrix

Exact parity established a safe, explicit control but did not prove that the inherited GTL constants were optimal. A temporary offline harness exposed those constants without changing the live scanner. That harness has now been retired; the compact consolidated reports remain as the reproducible decision record.

The single-command matrix contains 20 full-period arms. Each non-control arm changes exactly one input:

Knob	Frozen control	Alternatives	Question isolated
History	All available bars	252 / 504 / 756 bars	Is recent or long-cycle range geometry useful?
Candidate cap	5	8 / unlimited	Does early pruning discard the useful headline?
Max target distance	Existing volatility-dependent rule	5.5 / 8 ATR universally	Is the medium-volatility unlimited branch the hidden edge?
Traffic touch padding	0.5%	0 / 0.25%	Does padded price traffic carry information?
Proposal merge	0.5%	0.25% / 1%	Is proposal density or consolidation important?
Target zones	2%	1% / 3%	Does the reachable near edge manufacture the gate geometry?
Range centers	20	12 / 32	Is coarse ladder density the useful feature?
Pivots	Five-bar swings	None / eleven-bar swings	Do pivots add anything beyond the range ladder?
Traffic strength scale	500	250 / 1000	Does strength saturation affect probability/selection?

Arms execute sequentially so multiprocessing pools never compete. Each arm produces full-period production metrics, train/test books split at 2024-07-01, robust expectancy after removing the top 5% of setups, and retained/added/ removed cohorts against control. The consolidated JSON and Markdown table are checkpointed after every arm; successful runs delete temporary per-arm reports unless --keep-arm-reports is set.

The pre-registered screen requires all of the following versus control: full/train/test Sharpe not worse, full-period drawdown not worse, at least 80% of production trades retained, and positive qualified expectancy after removing the top 5%. Passing identifies a candidate for forward paper validation, not an automatic deployment. The post-2024 interval has already influenced this research, so the split is a robustness check rather than a pristine holdout.

Result on the 2026-07-13 snapshot: 20/20 arms completed; no replacement arm passed all six checks. The frozen control remained best on full-period Sharpe (2.03), CAGR (50.0%), and post-2024 Sharpe (2.78), with 321 production trades and 21.4% drawdown. The closest replacement, 0.25% touch padding, still fell to Sharpe 1.94 / CAGR 47.9%. This rejects direct constant replacement; the inherited behavior is not explained by one obvious GTL knob.

The paired cohorts do expose a narrower mechanism worth testing:

Variant	Retained control setups	Added by variant	Removed from control
0.25% touch	1,049 at +0.214R (+0.058 ex-top-5%)	18 at -0.235R	37 at -0.056R
Strength 1000	1,037 at +0.225R (+0.069)	122 at +0.301R (+0.152)	49 at +0.142R (-0.044)
0.25% merge	791 at +0.238R (+0.078)	365 at +0.161R (+0.023)	295 at +0.129R (-0.008)
Grid without pivots	428 at +0.232R (+0.100)	392 at +0.176R (+0.046)	658 at +0.208R (+0.041)

Replacement mixes the retained and added cohorts and also discards the removed cohort, so its portfolio result could not say which part helped. The archived confirmation matrix therefore preserved frozen control geometry and decomposed each selected variant into:

intersection — only control setups also core-qualified by the variant;
union — all core-qualified control setups plus genuinely added variant setups, using tuned geometry only for those additions.

It also tests pre-registered intersections among the three high-breadth confirmers. Its control path exactly reproduced the completed tuning matrix.

Result: 13/13 arms completed with exact control parity; no arm passed all six guardrails. The decomposition does identify one near-hit:

Arm	Full Sharpe	Train	Post-2024	CAGR	Max DD	Trades
Control	2.03	1.28	2.78	50.0%	21.4%	321
Strength-1000 intersection	2.06	1.30	2.82	50.7%	21.7%	316

Strength confirmation removes only 49 of 1,086 qualified control setups. Those removed setups average +0.142R, but turn negative after removing their largest 5% of outcomes (-0.044R); the retained 1,037 average +0.212R and +0.054R ex-top-5%. This is consistent with a weak tail-dependence filter. It is not yet a winner: the original drawdown guardrail remains fixed, and 21.7% is worse than 21.4% even though the difference is small.

The final parameter test was deliberately one-dimensional. It swept coarse strength scales around 1000 (625, 750, 875, 1000, 1125, 1250, 1500, 2000), using intersection only. It reproduced both the frozen control and the completed strength-1000 result exactly. Promotion required at least two adjacent non-control scales to pass all six original checks; an isolated winner was rejected as sensitivity.

Final result: 9/9 arms completed, control parity passed, and the strength-1000 replication passed. Scale 1500 was the only arm to clear all six original checks, but neither adjacent scale (1250 or 2000) cleared them, so the pre-registered stable-plateau requirement failed. It also traded away return and setup quality despite its screen pass: CAGR fell from 50.0% to 48.8% and qualified expectancy from +0.209R to +0.188R.

The sensitivity curve shows a real but non-dominating trade-off. Scales 750–1000 produce small Sharpe improvements in parts of the sample but each misses a different unchanged guardrail; higher scales eventually reduce drawdown by filtering more setups, while CAGR, expectancy, and then Sharpe decline. There is no robust parameter neighborhood that improves the whole book.

Final decision: keep the frozen Gate Target Ladder and do not deploy a strength-confirmation gate. The GTL remains the explicit, volume-free compatibility component that exactly reproduces the validated production screen. Clean Structural S/R remains the separate human-facing chart/alert model. This snapshot is now exhausted for GTL fitting; any future challenger must be pre-registered and evaluated on genuinely new forward data rather than another iteration over the same history.

The scheduled backtest, Admin UI, and local snapshot runner therefore default to production_gtl. A manual UI/local run may select structural_sr as an explicit comparison; every report records the model and whether it is the production path. After deployment, rerun the Admin backtest once to replace any cached report produced by the former default.

The temporary GTL matrix scripts, configurable detector branches, and confirmation hooks were removed after this decision. The normal snapshot backtester now exposes only the production GTL and clean Structural S/R comparison models.

The post-2024 window has been opened and is now analysis data, not a valid final promotion holdout. These arms can isolate mechanism, but neither may ship without new future data or a separately pre-registered walk-forward protocol.

Next runs, if picked back up:

A per-name target model for clear-air setups instead of a constant k×ATR. This is the one avenue left: the return is demonstrably there (§4c), it's the flat target that makes it too expensive in risk. Grade on the §4c holdout, not full history.
Sweep k (fallback distance); only k=3 was tried. Grade on the holdout.
A volatility-aware admission rule for clear-air names — the OOS failure is a drawdown/vol story (11.7% → 14.3%), so sizing them down may recover the Sharpe.

47 KiB Raw Blame History Unescape Escape