From 33708ffe6adc4020a31899ba0e66c7ee082146f8 Mon Sep 17 00:00:00 2001 From: momentum dev Date: Fri, 17 Jul 2026 06:00:45 -0500 Subject: [PATCH] =?UTF-8?q?plans(012-015):=20dashboard=20visibility=20mile?= =?UTF-8?q?stone=20=E2=80=94=20order=20limits,=20symbol=20charts,=20decisi?= =?UTF-8?q?on=20trace,=20/backtest=20replay?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Owner-requested UI work, advisor-planned for executor handoff: - 012: show working limit/stop/target prices on positions + order lifecycle, cut dead columns (order-id, approval Mode) - 013: symbol drill-in — 24h candlestick + live buy/sell fill markers (TradingView Lightweight Charts v5.2, vendored, Apache-2.0) - 014: live decision trace — surface entry_gate_evaluated receipts as a feed (value→threshold per gate) + chart reject markers - 015: /backtest replay page — theoretical entries/exits over past sessions under CURRENT settings, assembled strictly from Experiment-OS primitives All four are read-path only; zero live-loop surface. Co-Authored-By: Claude Fable 5 --- plans/012-dashboard-order-limits-declutter.md | 237 ++++++++++++++ plans/013-symbol-chart-trade-markers.md | 306 ++++++++++++++++++ plans/014-live-decision-trace.md | 211 ++++++++++++ plans/015-backtest-replay-dashboard.md | 283 ++++++++++++++++ plans/README.md | 10 + 5 files changed, 1047 insertions(+) create mode 100644 plans/012-dashboard-order-limits-declutter.md create mode 100644 plans/013-symbol-chart-trade-markers.md create mode 100644 plans/014-live-decision-trace.md create mode 100644 plans/015-backtest-replay-dashboard.md diff --git a/plans/012-dashboard-order-limits-declutter.md b/plans/012-dashboard-order-limits-declutter.md new file mode 100644 index 0000000..3851ab6 --- /dev/null +++ b/plans/012-dashboard-order-limits-declutter.md @@ -0,0 +1,237 @@ +# Plan 012: Show the actual buy/sell limit prices everywhere they act, and cut the dashboard fields that carry no decision weight + +> **Executor instructions**: Follow this plan step by step. Run every +> verification command and confirm the expected result before moving to the +> next step. If anything in the "STOP conditions" section occurs, stop and +> report — do not improvise. When done, update the status row for this plan +> in `plans/README.md` — unless a reviewer dispatched you and told you they +> maintain the index. +> +> **Drift check (run first)**: `git diff --stat 43a3ace..HEAD -- api/static/dashboard.html storage/projections.py api/main.py tests/unit/api/` +> If any in-scope file changed since this plan was written, compare the +> "Current state" excerpts against the live code before proceeding; on a +> mismatch, treat it as a STOP condition. + +## Status + +- **Priority**: P2 +- **Effort**: S +- **Risk**: LOW (read-only UI + one projection field; no trading-path code) +- **Depends on**: none +- **Category**: dx / operator-visibility +- **Planned at**: commit `43a3ace`, 2026-07-16 + +## Why this matters + +The owner watches this dashboard to understand what the bot is DOING with +money, and today the most decision-relevant numbers — the actual limit price a +buy was submitted at, the resting protective stop, and the take-profit limit — +are either hidden (bracket legs appear only as anonymous rows in Order +Lifecycle) or collapsed into a single generic "Price" column. Meanwhile some +columns carry no decision weight (truncated broker order-id, an approval +"Mode" column that always reads `auto`). This plan makes the working orders +legible (limit/stop/fill split out, bracket legs attached to their position) +and removes the dead columns. + +## Current state + +- `api/static/dashboard.html` (1492 lines) — the ONE dashboard UI (owner + preference: extend this page at :8765; do NOT touch the retired NiceGUI + control room). Single inline `", html, re.S) +pathlib.Path("/tmp/dash_check.js").write_text("\n;\n".join(scripts)) +EOF +node --check /tmp/dash_check.js +``` + +→ exit 0. If a Playwright headless chromium is available in the environment, +also render `http://127.0.0.1:8765` after `python3 dashboard_api.py` and assert +zero `pageerror` events (this repo has caught blank-page JS bugs this way; see +Suggested executor toolkit). + +## Suggested executor toolkit + +- Headless render check: launch `chrome-headless-shell` via Playwright with + `args=["--no-sandbox"]`, navigate with `wait_until="load"` (NOT + "networkidle" — the SSE `/api/stream` never idles), capture + `page.on("pageerror")`. A passing API check is NOT a passing page. + +## Test plan + +- If `query_order_lifecycle_snapshot` changes: extend its existing unit test + (find it with `grep -rn "order_lifecycle" tests/unit/`) with one case + asserting `limit_price`/`stop_price`/`filled_avg_price`/`type` pass through + from a synthetic `account_orders_updated`-style payload. Model after the + neighboring projection tests in the same file. +- UI has no JS test harness — the `node --check` gate plus (optional) + headless render IS the UI verification. Do not introduce a JS test framework + for this plan. + +## Done criteria + +- [ ] Positions table shows `Stop` and `Target` columns populated from live + bracket legs (em-dash when absent) +- [ ] Order Lifecycle shows `Type, Limit, Stop, Fill` and no order-id column; + order-id recoverable via Symbol-cell `title` +- [ ] Approval queue has no `Mode` column; risk chip shows absolute entry and + stop prices +- [ ] `node --check` on the extracted script exits 0 +- [ ] `python3 -m pytest tests/ -q --ignore=tests/unit/api` all pass; + `ruff check --select F api/ storage/` exits 0 +- [ ] `git status` shows no modifications outside the in-scope list +- [ ] `plans/README.md` status row updated + +## STOP conditions + +Stop and report back (do not improvise) if: + +- The order snapshot dicts do NOT contain `limit_price`/`stop_price`/`type` + (the excerpt above was verified against production on 2026-07-16 — if absent, + the projection chain changed and this plan is stale). +- Adding pass-through fields to `query_order_lifecycle_snapshot` requires + touching how events are WRITTEN (it must stay a read-side change). +- `node --check` fails on the UNMODIFIED extracted script (pre-existing + breakage — report, don't fix someone else's bug in this plan). + +## Maintenance notes + +- Plan 013 (symbol drill-in chart) reuses `stopBy`/`targetBy`/`entryLimitBy` + as chart level-lines — keep them top-level in `render(d)`, not inlined. +- Reviewer should scrutinize: string→number coercion on broker prices (the + `"2.05"` strings), and that the leg maps filter by working `status` (a + cancelled stop leg must not render as protection). +- Deferred deliberately: cutting whole panels (Watch States, Account). The + owner said "fields that don't really matter" — start with dead columns; + panel-level cuts need the owner pointing at specific panels. diff --git a/plans/013-symbol-chart-trade-markers.md b/plans/013-symbol-chart-trade-markers.md new file mode 100644 index 0000000..b7aaab6 --- /dev/null +++ b/plans/013-symbol-chart-trade-markers.md @@ -0,0 +1,306 @@ +# Plan 013: Symbol drill-in — last-24h candlestick chart with the bot's live buy/sell markers + +> **Executor instructions**: Follow this plan step by step. Run every +> verification command and confirm the expected result before moving to the +> next step. If anything in the "STOP conditions" section occurs, stop and +> report — do not improvise. When done, update the status row for this plan +> in `plans/README.md` — unless a reviewer dispatched you and told you they +> maintain the index. +> +> **Drift check (run first)**: `git diff --stat 43a3ace..HEAD -- api/main.py api/static/ storage/projections.py tests/unit/api/` +> If any in-scope file changed since this plan was written, compare the +> "Current state" excerpts against the live code before proceeding; on a +> mismatch, treat it as a STOP condition. + +## Status + +- **Priority**: P2 +- **Effort**: M +- **Risk**: LOW-MED (new read-only endpoint + vendored JS lib; no trading-path code) +- **Depends on**: plans/012-dashboard-order-limits-declutter.md (reuses its + `stopBy`/`targetBy`/`entryLimitBy` maps for level lines; 013 works without + 012 but then must build those maps itself — prefer landing 012 first) +- **Category**: direction / operator-visibility +- **Planned at**: commit `43a3ace`, 2026-07-16 + +## Why this matters + +The owner's ask, verbatim: "look at a symbol and see the last 24 hours and the +marker of buy and sell that the model has on it in real time." Today the +dashboard has only 60-minute canvas sparklines — no premarket context, no +trade markers, no way to see WHERE the bot bought and sold on the tape. The +building blocks all exist (1-min bars incl. premarket in Postgres, buy/sell +fill events, an SSE stream that pushes on every change); this plan wires them +into a click-to-open chart modal. + +**Do not hand-roll the chart.** The purpose-built existing solution is +**TradingView Lightweight Charts v5.2** (Apache-2.0, ~45 KB standalone bundle, +`window.LightweightCharts` global — researched 2026-07-16): candlestick series, +`createSeriesMarkers(series, markers)` for buy/sell arrows, `series.update(bar)` +for real-time, `createPriceLine` for entry/stop/target levels. Its license +requires a TradingView attribution notice — Step 1 handles it. + +## Current state + +- `api/main.py` — stdlib HTTP server, no framework. + - Routes live in `do_GET` (line ~522): flat `if route == "/api/..."` chain. + `/api/bars` (line ~537) is the existing single-session sparkline endpoint. + - `DashboardState.bars()` (line ~326) queries via + `research.query.query_minute_bars(con, symbol, session_date)` under + `self._lock`, returns `{"symbol", "bars": [{t,c,h,l,v,vwap}...]}`. Note it + LACKS the open — a candlestick chart needs `o` too. + - SSE: `_stream()` (line ~481) pushes a full `snapshot` event whenever + `state.change_stamp()` changes; heartbeat ~15 s. + - `STATIC_DIR / "dashboard.html"` served at `/`; static dir is + `api/static/`. NOTE: only `dashboard.html`/`admin.html` are served today — + there is NO generic static-file route; Step 2 adds one for the vendored + lib. +- `storage/projections.py:403` — `query_fills_feed(store, limit, for_date)`: + already extracts exactly the marker fields from `order_filled` / + `order_submitted` events: `{kind, order_id, symbol, side, quantity, price, + timestamp}`. It has no symbol filter (Step 3 adds one). +- `api/static/dashboard.html` — single-page inline-JS UI. + - Symbol cells render as `SYMBOL` across all tables. + - Sparkline pattern to imitate for fetching: `loadBars(symbol, entry, stop, + canvas)` (line ~759) with an 8-second per-symbol cache. + - SSE hookup: the page listens for the `snapshot` event and calls + `render(d)` — the chart modal must refresh off that same event, not its + own timer (a hidden extra poller is how dashboards rot). + - Plan 012 adds `stopBy` / `targetBy` / `entryLimitBy` maps at the top of + `render(d)` — reuse them for price lines. +- Data (verified against production 2026-07-16): + - `minute_bars(symbol, timestamp, session_date, is_premarket, + is_regular_hours, is_afterhours, open, high, low, close, volume, vwap)` — + 1-min bars incl. premarket/after-hours. **Timestamps are NAIVE + US/Eastern** (`2026-07-16T10:33:21`). ~960 bars per symbol per 24 h. The + research connection uses `?` placeholders (facade converts to `%s`). + - `events(event_type, timestamp, payload_json)` — `order_filled` / + `order_submitted` rows carry `symbol/side/price/fill_price/quantity`; + timestamps also naive ET. +- Timezone convention: Lightweight Charts wants UTC epoch **seconds**. Convert + in Python: `datetime(...naive ET...).replace(tzinfo=ZoneInfo("America/New_York")).timestamp()`. + Do NOT ship naive strings to the chart. +- Conventions: no build step, no npm — vendored file + inline JS. Escape all + interpolated strings with the page's `esc()`. Read-only endpoints take no + token (the token gate is POST-only). + +## Commands you will need + +| Purpose | Command | Expected on success | +|---------|---------|---------------------| +| API tests | `python3 -m pytest tests/unit/api/ -q` | pass (pre-existing local-sandbox urllib failures possible — compare against a stash run of clean HEAD) | +| Full suite | `python3 -m pytest tests/ -q --ignore=tests/unit/api` | all pass | +| Lint (CI gate) | `ruff check --select F api/ storage/ research/` | exit 0 | +| JS syntax gate | extract inline `` + BEFORE the main inline script. +- Delegate clicks: any `` (positions, ready signals, watch + states, order lifecycle, feed) opens the modal for that symbol. Use ONE + document-level click listener checking `closest("td.sym")` — do not attach + per-row handlers (rows re-render every SSE tick). +- On open: fetch `/api/chart?symbol=S&hours=24`; build + `LightweightCharts.createChart`, add a CandlestickSeries, `setData(bars)`; + `createSeriesMarkers` mapping `side==="buy"` → `{position:"belowBar", + shape:"arrowUp", color:"var green", text:"B "+qty}` and sell → aboveBar / + arrowDown / red; submitted-but-unfilled (`kind==="order_submitted"`) render + in a muted gray so a resting order is visually distinct from a fill. Add + `createPriceLine` for each non-null level (entry blue, stop red, target + green — same colors the sparkline refLines use, line ~742). +- Real-time: inside the existing SSE `snapshot` handler, if the modal is open + and ≥5 s since the last chart fetch, re-fetch `/api/chart` for the open + symbol and `setData`/`setMarkers` again (full replace — 960 rows is cheap; + do NOT build incremental `update()` plumbing in v1). Close on Escape and on + backdrop click; destroy the chart instance (`chart.remove()`) on close. + +**Verify**: `node --check` on extracted inline scripts → exit 0. Manual: open +:8765, click a symbol in Positions/Ready Signals → modal with candles; with +the loop running, a new fill appears as an arrow within one SSE tick + ≤5 s. + +### Step 5: Render gate + +Run the headless render check (toolkit note): zero `pageerror`, and +`document.getElementById("chart-modal")` exists. If Playwright is unavailable +in the environment, state that explicitly in the completion report instead of +skipping silently. + +## Test plan + +New file `tests/unit/api/test_chart_endpoint.py`, modeled structurally on +`tests/unit/api/test_dashboard_sparklines.py` (same server/state fixtures): + +1. `/api/chart?symbol=X` with seeded minute_bars across a session boundary → + bars sorted ascending, `time` is int epoch seconds, bars older than the + window excluded. +2. Seeded `order_filled` + `order_submitted` events for X and for another + symbol → markers contain only X's, with `kind` and `side` intact. +3. No research con attached → `{"bars": [], "reason": ...}`, HTTP 200. +4. `query_fills_feed(symbol=None)` unchanged vs before (regression guard on + the default path — assert same rows as an unfiltered call). + +**Verify**: `python3 -m pytest tests/unit/api/test_chart_endpoint.py -q` → all +pass. + +## Done criteria + +- [ ] Clicking any symbol cell opens a 24-hour candlestick chart (premarket + bars included) with buy/sell arrows at the bot's fills and muted arrows + for submitted-unfilled orders +- [ ] Entry / stop / target price lines drawn when the symbol has working legs +- [ ] Chart refreshes from the existing SSE signal (no new standalone poller; + ≥5 s throttle) and cleans up on close +- [ ] Vendored lib pinned at v5.2, served locally; NOTICE file + visible + TradingView attribution present; path-traversal request returns 404 +- [ ] `python3 -m pytest tests/unit/api/test_chart_endpoint.py -q` all pass; + full suite (minus known-broken-local api group if pre-existing) passes; + `ruff check --select F api/ storage/ research/` exits 0 +- [ ] `node --check` clean on vendored file and extracted inline scripts +- [ ] `git status` clean outside the in-scope list; `plans/README.md` updated + +## STOP conditions + +Stop and report back (do not improvise) if: + +- The vendored v5.2 standalone bundle cannot be obtained (no network in the + executor environment) — report; do NOT substitute a CDN tag or an older + cached version silently. +- `minute_bars` timestamps turn out NOT to be naive US/Eastern (spot-check + `SELECT timestamp FROM minute_bars ORDER BY timestamp DESC LIMIT 1` against + the wall clock) — the epoch conversion in Step 3 would be wrong; report. +- The SSE `snapshot` handler in `dashboard.html` is not a single obvious + hookup point (drift since `43a3ace`). +- Plan 012 has not landed AND building the level maps yourself would duplicate + >20 lines — report the dependency instead of forking the logic. + +## Maintenance notes + +- The 5-second full-replace refresh is a deliberate ceiling + (`ponytail`-style): if the owner later wants tick-level updates, switch to + `series.update()` fed by a dedicated lightweight endpoint — do not shorten + the throttle on the full re-fetch. +- Marker volume: `query_fills_feed(limit=200)` bounds payload size; a + 200-trade day on one symbol would truncate oldest-first — acceptable, noted. +- The library is pinned and vendored; upgrading it is a deliberate act (check + the v5→v6 marker API for breaking changes when that day comes). +- Reviewer scrutiny points: path-traversal guard on the vendor route; ET→UTC + epoch conversion around DST transitions; chart teardown on close (leaked + chart instances accumulate canvases). diff --git a/plans/014-live-decision-trace.md b/plans/014-live-decision-trace.md new file mode 100644 index 0000000..e08d963 --- /dev/null +++ b/plans/014-live-decision-trace.md @@ -0,0 +1,211 @@ +# Plan 014: Live decision trace — show WHY the bot did or didn't enter, per gate, with the config values in play + +> **Executor instructions**: Follow this plan step by step. Run every +> verification command and confirm the expected result before moving to the +> next step. If anything in the "STOP conditions" section occurs, stop and +> report — do not improvise. When done, update the status row for this plan +> in `plans/README.md` — unless a reviewer dispatched you and told you they +> maintain the index. +> +> **Drift check (run first)**: `git diff --stat 43a3ace..HEAD -- api/main.py api/static/dashboard.html storage/projections.py tests/unit/api/` +> On mismatch with "Current state" excerpts, STOP. + +## Status + +- **Priority**: P2 +- **Effort**: M +- **Risk**: LOW (read-only; new endpoint over existing events + UI panel/overlay) +- **Depends on**: plans/013-symbol-chart-trade-markers.md (the chart modal this + overlays; the Decision Feed panel half works standalone) +- **Category**: direction / operator-visibility +- **Planned at**: commit `43a3ace`, 2026-07-16 (plan written 2026-07-17) + +## Why this matters + +Owner's ask: "see the logic running live... visually clear... make additional +determinations as to some of the smaller env variables and see their impact on +the trade as to when and where the bot thought it would be." The bot ALREADY +emits exactly this as `entry_gate_evaluated` decision receipts (plan 005) — +one event per entry evaluation with a per-gate breakdown including the value +seen and the threshold applied. Today those receipts are write-only: nothing +on the dashboard reads them. This plan surfaces them two ways: a live +Decision Feed panel (latest gate evaluations, reject reasons, value-vs- +threshold), and reject/entry markers on the plan-013 symbol chart so the +owner sees WHERE on the tape the bot considered and declined. + +## Current state + +- Receipts: `trading_execution.py:1138` `_emit_gate_receipt` emits + `entry_gate_evaluated` (env `TRADING_GATE_RECEIPTS`, default on). Payload + verified in production 2026-07-16: + + ```json + {"timestamp": "2026-07-16T10:41:40.255354", "symbol": "ATAI", + "source": "fast", "passed": false, "reject_reason": "already_active", + "message": "ATAI entry gates (fast) — rejected: already_active", + "gates": {"session": {"outcome": "pass"}, "regime": {"outcome": "pass"}, + "late_confirmation": {"outcome": "pass"}, + "population": {"outcome": "pass"}, + "rank": {"value": 5, "threshold": 0, "outcome": "pass"}}} + ``` + + The `gates` map is a soft contract (additive only). Gate keys observed: + `session, regime, late_confirmation, population, rank` (+ others the slow + path adds); entries may carry `value` and `threshold`. +- Volume: ~1,300 receipts/session. **Retention: 14 days** — the nightly + housekeeping in `research/premarket_eval.py` purges `entry_gate_evaluated` + older than 14 days (it is in the spam list). The feature is a LIVE lens, not + a historical archive; say so in the UI hint. +- `api/main.py` — stdlib ThreadingHTTPServer; GET routes in `do_GET` + (line ~522) as a flat if-chain; `_send_json` helper. `DashboardState` + wraps the event store (`self.open_store()`), `store.query_events( + event_type=..., since=..., until=..., limit=...)` is the query interface + (see `storage/projections.py:403` `query_fills_feed` for the exact idiom, + including `json.loads(event.get("payload_json"))`). +- `api/static/dashboard.html` — panel pattern: `
` + blocks (see `p-feed` Activity Feed, line ~528, and its `render` at + line ~1298 for the row style to imitate). Plan 013 adds a chart modal with + `createSeriesMarkers`; its refresh path re-fetches on the SSE `snapshot` + event with a ≥5 s throttle. +- Effective config: the admin page already exposes env; the receipts carry the + numeric threshold per gate — the feed shows `value → threshold` per reject, + which is the "which env var did this" answer without new plumbing. + +## Commands you will need + +| Purpose | Command | Expected on success | +|---------|---------|---------------------| +| API tests | `python3 -m pytest tests/unit/api/ -q` | pass (pre-existing local urllib failures possible — compare to a clean-HEAD stash run) | +| Full suite | `python3 -m pytest tests/ -q --ignore=tests/unit/api` | all pass | +| Lint (CI gate) | `ruff check --select F api/ storage/` | exit 0 | +| JS syntax gate | extract inline `