momentum-system/plans/015-backtest-replay-dashboard.md

Plan 015: Backtest replay dashboard — how the bot WOULD have traded past weeks under CURRENT settings

Executor instructions: Follow this plan step by step. Run every verification command and confirm the expected result before moving to the next step. If anything in the "STOP conditions" section occurs, stop and report — do not improvise. When done, update the status row for this plan in plans/README.md — unless a reviewer dispatched you and told you they maintain the index.

Drift check (run first): git diff --stat 43a3ace..HEAD -- api/main.py api/static/ research/experiments/replay.py strategy/exits.py strategy/population.py tests/unit/api/ On mismatch with "Current state" excerpts, STOP.

Status

• Priority: P2
• Effort: L
• Risk: MED (read-only, but the replay must be HONEST — a leaky or fidelity-free replay shown as truth is worse than no page; every simulation primitive here is reused from the adversarially-reviewed Experiment OS, not reimplemented)
• Depends on: plans/013-symbol-chart-trade-markers.md (chart component + vendored lib reused for per-trade drill-in)
• Category: direction / research-tooling
• Planned at: commit 43a3ace, 2026-07-16 (plan written 2026-07-17)

Why this matters

Owner's ask: "a separate dashboard to show when theoretical entries and exits would have happened with backtested trades over days or weeks to get a feel of how the bot would have acted from past data as if it was current with the current settings." The Experiment OS already contains every piece of an honest replay: theoretical ORB fires from setup_events, the SHARED live exit engine simulated on real bar paths with adverse-fill fidelity, the SHARED population gate, and walk-forward regime scores. What's missing is only the assembly — a /backtest page that runs those pieces over a chosen window with the CURRENTLY-configured settings and shows the resulting trades, per-day P&L, and equity curve.

Current state

All primitives exist — REUSE them; write no new simulation logic:

• research/experiments/replay.py:
  • reconstruct_fires_from_setups(store, since=None, limit=None) (line ~164) — theoretical fires from setup_events (entry = entry_reference_price, stop = invalidation_price, sim_baseline=True). The 18k-strong backtestable ORB universe.
  • _fire_in_population(con, cfg, fire) (line ~201) — screens a fire through the SHARED strategy/population.py gate on open-knowable inputs. population_config(pop_json) (line ~196) builds the config from a dict.
  • _sim_baseline_r(fire, bars_after, fidelity) (line ~306) — the LIVE exit config (ExitConfig.from_env(), cached in _live_exit_cfg) simulated on the bar path with adverse-fill accounting (_adverse(fidelity), entry fill = entry + adverse ticks). THIS is "current settings".
  • Look at run() for the batched per-session minute-bar loading pattern (one query per session, not per fire) — copy that idiom.
• strategy/exits.py: simulate_exit(entry_price, init_stop, bars_after, cfg) -> ExitResult (line 229) with r_multiple, reason, exit_index (index into bars_after where the position fully closed — this gives the EXIT TIMESTAMP for plotting), fills: list[ExitFill(frac, price, reason)]. For per-trade exit markers call simulate_exit directly (not _sim_baseline_r, which returns only the scalar R) — but keep the SAME adverse-fill entry (entry + _adverse(fidelity)) and the same _live_exit_cfg() so numbers match the harness.
• Regime gate: regime_scores table has walk-forward favorable per (model_version='meta_v1', session_date) — the current live gate's decision layer. Keep-or-zero per session (see iv_regime_gate, replay.py:429: no score row → KEEP, fail-open).
• Live current settings sources: ExitConfig.from_env() (exits), population from env via trading_execution.ExecutionSettings.from_env() fields price_band_min/price_band_max/min_entry_gap_pct/min_day_volume (see _population_reject, trading_execution.py:~895, for the exact field→PopulationConfig mapping to replicate), regime gate on/off via TRADING_REGIME_GATE_ENABLED.
• Server: api/main.py, stdlib ThreadingHTTPServer (each request gets a thread — a slow backtest request does NOT block the live SSE stream). Static HTML served via _send_html(STATIC_DIR / "dashboard.html"); admin.html shows the two-page pattern.
• Cost estimate (measured on this repo): batched replay ≈ sub-second per session; a 30-session window lands in the ~10–30 s range. Synchronous request + client-side loading state is acceptable at v1 WITH the session cap below; no job queue.
• Fidelity defaults: the pre-registered experiments use {"atr_frac": 0.25, "spread_mult": 3, "adverse_ticks": 1} — use the same (see experiments/regime_meta_v1.json).

Commands you will need

Purpose	Command	Expected on success
API tests	python3 -m pytest tests/unit/api/ -q	pass (pre-existing local urllib failures possible — compare to clean-HEAD stash run)
Replay tests	python3 -m pytest tests/unit/experiments/ -q	all pass (must stay green — you're consuming, not changing, replay)
Full suite	python3 -m pytest tests/ -q --ignore=tests/unit/api	all pass
Lint (CI gate)	ruff check --select F api/ storage/ research/	exit 0
JS syntax gate	plan 012 Step 5 snippet + node --check	exit 0

Scope

In scope:

• api/static/backtest.html (create — the separate dashboard page)
• api/main.py (route /backtest → the page; route /api/backtest → the replay; a small in-memory result cache on DashboardState)
• research/experiments/replay.py — ONLY if a tiny public helper is needed to avoid importing privates from api/ (e.g. def replay_window(store, since, until, *, population_cfg, regime_gate, fidelity) -> list[dict]); simulation internals unchanged.
• tests/unit/api/ (new test file)

Out of scope:

• ANY change to simulation semantics (simulate_exit, _sim_baseline_r, population logic, fidelity math) — consumers only.
• trading_execution.py, run_live_paper.py, live loop config.
• Writing anything to the DB (results are computed + cached in memory only).
• Parameter sweeps / what-if config editing in the UI — v1 replays the CURRENT env settings only (see Maintenance notes).

Git workflow

• Match repo style; suggested: research-tooling: /backtest replay dashboard — theoretical trades under current settings.
• Do NOT push unless instructed.

Steps

Step 1: replay_window helper in replay.py

Public function assembling existing pieces (no new math):

def replay_window(store, since, until, *, population_cfg=None,
                  regime_gate=False, fidelity=None) -> dict:
    """Theoretical trades for [since, until] under the CURRENT live exit
    config. Reuses reconstruct_fires_from_setups + _fire_in_population +
    simulate_exit with the harness's adverse-fill accounting."""

• fires = reconstruct_fires_from_setups(store, since=...) filtered to session_date <= until.
• population screen when population_cfg given (_fire_in_population).
• regime screen when regime_gate: drop fires whose session has a regime_scores row (model meta_v1) with favorable = false; sessions without a row are KEPT (fail-open, mirrors iv_regime_gate).
• per session, batch-load minute bars once (copy run()'s idiom); per fire: bars_after = bars strictly after fill_ts; entry_fill = entry_price + _adverse(fidelity); res = simulate_exit(entry_fill, stop_price, bars_after, _live_exit_cfg()); skip fires with no bars or risk <= 0 (count them as skipped).
• returns {"trades": [...], "daily": [...], "summary": {...}, "skipped": N} where each trade carries {symbol, session_date, entry_ts, entry_price, entry_fill, stop_price, exit_ts (bars_after.iloc[exit_index].timestamp), exit_reason, r_multiple, fills: [{frac, price, reason}]} — timestamps as ISO strings (naive ET, converted by the API layer like plan 013 does), daily = per-session {date, n, sum_r, cum_r}, summary = {n, sum_r, avg_r, hit_rate, sessions}.

Verify: python3 -m pytest tests/unit/experiments/ -q still green (no behavior change to existing paths); new unit test (Test plan case 1) passes.

Step 2: /api/backtest route + cache

• GET /api/backtest?days=10 (alternatively since/until ISO dates). Cap: 45 sessions — clamp and report {"clamped": true} beyond it.
• Build inputs from CURRENT env exactly as live does: population cfg from ExecutionSettings.from_env() fields (the _population_reject mapping), regime_gate from TRADING_REGIME_GATE_ENABLED, fidelity = {"atr_frac": 0.25, "spread_mult": 3, "adverse_ticks": 1}.
• Echo the effective settings in the response ("settings": {...}) — the page must SHOW what config produced the numbers, or the owner can't trust a stale tab.
• Cache on DashboardState: key = (since, until, repr(settings)); value + monotonic timestamp; TTL 10 min; lock around compute so two concurrent requests don't both run the replay (threading.Lock, compute-once).
• Convert naive-ET timestamps to UTC epoch seconds for chart use (same conversion as plan 013 Step 3).

Verify: curl -s "http://127.0.0.1:8765/api/backtest?days=5" | python3 -m json.tool | head -30 → trades + daily + summary + settings; second call returns in <1 s (cache).

Step 3: the /backtest page

api/static/backtest.html, served at /backtest (add the route next to the dashboard's). Reuse the dashboard's CSS variables/panel styles (copy the :root block; a shared stylesheet refactor is out of scope). Layout:

• Header: window picker (5/10/20/45 sessions buttons + custom date range), the echoed settings rendered as chips (target_r=2.0, gap≥10%, regime gate ON, fidelity), and a bold banner: "THEORETICAL — simulated fills (adverse-tick fidelity), setup-sourced entries; not live P&L."
• Summary strip: n trades, sum R, avg R, hit rate, sessions, skipped count.
• Equity curve: cumulative R per session (reuse the dashboard's existing canvas equity-spark drawing approach — see renderEquity / renderDaily in dashboard.html — or a Lightweight Charts line series; either is fine, pick ONE).
• Per-day bars: like the dashboard's Daily Performance panel; click a day → filters the trade table.
• Trade table: date, symbol, entry ts/price, exit ts/price, exit reason, R (green/red). Click a row → chart modal (plan 013's component pattern: vendored Lightweight Charts, candles for that session via /api/chart scoped hours, plus THEORETICAL entry (blue arrowUp) and exit (purple arrowDown, text = exit reason) markers from the trade row — purple distinguishes simulated from the live-fill green/red of plan 013).
• Loading state while /api/backtest runs (10–30 s for big windows): spinner
  • "replaying N sessions…" — no silent hang.

Verify: node --check on extracted scripts → exit 0; manual: open http://127.0.0.1:8765/backtest, pick 5 sessions → summary + curve + table render; row click opens chart with theoretical markers.

Step 4: honesty checks in the page

Two footer lines, always visible:

• entries are setup_events-sourced ORB triggers (the labeler's population), entry fill = trigger + adverse ticks — optimistic vs real slippage on thin names;
• sessions missing from setup_events (nightly build gaps) are silently absent — show sessions replayed: X of Y calendar trading days computed from the daily array vs the requested window.

Verify: visual — both lines present; the X-of-Y count is right for a window containing a known gap (e.g. a weekend).

Test plan

New tests/unit/api/test_backtest_endpoint.py (fixtures modeled on tests/unit/experiments/ seeding + tests/unit/api/ server fixtures):

1. replay_window: seed 2 sessions × 2 setups with synthetic minute bars (reuse the _seed_session idiom from tests/unit/research/test_regime.py: ~130) → returns 4 trades with exit_ts inside the session, daily has 2 rows, cum_r monotone-consistent with sum_r.
2. Population screen: one setup priced outside the band → excluded, counted in the response, not in trades.
3. Regime gate: seed regime_scores meta_v1 unfavorable for session 1 → its trades drop when regime_gate=True; session without a score row KEPT.
4. /api/backtest?days=9999 → clamped, clamped: true.
5. Cache: two identical calls → second returns the identical object (mock or count compute invocations via monkeypatch on replay_window).

Verify: python3 -m pytest tests/unit/api/test_backtest_endpoint.py -q → all pass.

Done criteria

• /backtest renders: settings chips, THEORETICAL banner, summary strip, equity curve, per-day bars, trade table, chart drill-in with entry/exit markers
• Numbers come from simulate_exit + _live_exit_cfg + harness fidelity — grep -n "def simulate\|risk\s*=" api/ shows NO simulation math in api/ (assembly only)
• 45-session clamp, 10-min cache, compute-once lock all in place
• All 5 new tests pass; tests/unit/experiments/ untouched and green; full suite passes; CI ruff gate clean; node --check clean
• git status clean outside scope; plans/README.md updated

STOP conditions

• Any step tempts you to modify simulate_exit, _sim_baseline_r, population logic, or fidelity math — STOP; this plan consumes them.
• reconstruct_fires_from_setups coverage for the chosen window is near-zero (nightly labeler gap — see plans/README 009 note about the 07-15 backfill): report the data gap, don't pad with a different fire source.
• A 5-session replay exceeds ~60 s on the dev machine — the batching idiom wasn't copied correctly; stop after one fix attempt.
• Plan 013's vendored chart lib is absent (013 not landed) — report; don't CDN it.

Maintenance notes

• v1 replays CURRENT env only. The obvious next step ("what if I change env X and replay") is a parameter-sweep UI — it belongs on top of the Experiment OS's iv_exit_config (which already takes an arbitrary exit config), and should be a new plan; do not bolt query-param config overrides onto /api/backtest ad hoc, that becomes an unregistered experiment lab with no pre-registration discipline.
• The equity curve is R-based, not dollar-based — sizing is deliberately out (fixed-fractional sizing depends on live equity; mixing it in would fake precision).
• Reviewer scrutiny: the fires' bars_after slicing (strictly AFTER fill_ts — off-by-one here silently peeks); ET→UTC conversion; the compute lock (a stampede of tabs must not run N replays).
• The nightly setup_events build is this page's data supply — if the scheduler misses a night (see plan 009 note), yesterday will be missing here too; the X-of-Y sessions line surfaces that.