momentum-system/plans/007-operator-control-plane.md

Plan 007: Operator control plane — the dashboard tells the truth and accepts guarded commands

Executor instructions: Follow this plan step by step. Run every verification command and confirm the expected result before moving to the next step. If anything in the "STOP conditions" section occurs, stop and report — do not improvise. When done, update the status row for this plan in plans/README.md.

Drift check (run first): git diff --stat d54a2a9..HEAD -- api/ dashboard_api.py admin_api.py telegram_bot.py run_live_paper.py trading_execution.py storage/event_schema.py deploy/ docker-compose.yml Diffs introduced by plans 001–006 are EXPECTED — see plans/README.md "Expected drift". On any in-scope drift, compare "Current state" excerpts before proceeding; mismatch = STOP.

Status

• Priority: P2 (except step 1, which is a P1 production bug fix)
• Effort: M
• Risk: MED (introduces a live kill switch; mitigated by HMAC guard, idempotency, and the loop consuming commands through its existing protective-close helper)
• Depends on: 001 (CI test gate), 003 (clean shutdown/flatten paths). The headless smoke test is CREATED in this plan (step 6) — it does not exist in 001. Ledger panel additionally wants 006's tables (render "no experiments yet" until then).
• Category: bug / dx / capability
• Planned at: commit d54a2a9, 2026-07-13

Why this matters

The production :8765 dashboard currently lies and cannot act: every POST endpoint crashes (api/main.py:415-416 uses os.environ and hmac.compare_digest but the module imports at lines 24-30 include neither — NameError on any approve/reject/exit/watch action); the LIVE badge keys on execution_mode === "schwab_live" (dashboard.html:1061) while the admin go-live flow (admin_api.py:276-295) only swaps ALPACA_BASE_URL/keys and never touches TRADING_EXECUTION_MODE — the badge would read PAPER while trading real money; sparklines are permanently dead in production; the action UI renders in the read-only deployment even though has_execution is shipped (api/main.py:148) and never read; and no manual flatten path exists anywhere in the production surface (dashboard container runs execution_service=None — dashboard_api.py:36; the qabot hard-refuses action words — telegram_bot.py:29-30). The operator's two moments of need — "get me flat NOW" and "what is this thing actually running?" — both currently require shelling into a container. Additionally the SSE change detector opens a fresh Postgres connection (re-applying the 473-line schema unless PG_SKIP_SCHEMA=1) plus COUNT(*) every second per client.

Current state

• api/main.py:22-33 — imports: json, threading, time, datetime/timezone, http.server, pathlib, urllib.parse, storage.*. No os, no hmac. Lines 410-417:

  def do_POST(self):  # noqa: N802
      ...
      _tok = os.environ.get("DASHBOARD_TOKEN")
      if _tok and not hmac.compare_digest(self.headers.get("X-Dashboard-Token", ""), _tok):
  

• SSE loop api/main.py:~350-365: count = self.state.event_count() every ~1s tick; event_count() constructs work per call; storage/db_pg.py:275-277 re-applies pg_schema.sql on connection unless PG_SKIP_SCHEMA=1; storage/event_store.py:200 count = COUNT(*).
• has_execution shipped in snapshots (api/main.py:148: "has_execution": self.execution_service is not None), zero uses in api/static/dashboard.html (grep).
• LIVE badge: api/static/dashboard.html:419 <span id="mode-badge" class="badge paper">PAPER</span>; badge CSS at 89-94 (.badge.paper blue / .badge.live red); JS keys on the schwab_live literal (~line 1061).
• Admin go-live: admin_api.py:276-295 — /api/mode writes ALPACA_BASE_URL + live keys to overrides; note "restart the loop to apply".
• Standalone dashboard: dashboard_api.py:36 — DashboardState(db_path, execution_service=None, execution_mode=mode); docstring: "For approve/reject/exit buttons to act, run the full orchestrator".
• qabot: telegram_bot.py:29-30 _ACTION_WORDS = ("buy ", "sell ", "flatten", "close all", …) — refusal list; is_authorized (220-221) = chat-id string equality.
• Loop guard cadence: 5s (run_live_paper.py guard task) — command pickup latency bound. Flatten primitive: runtime/flatten.py:97 cancel_protective_and_close(client, symbol, orders=None, …).
• Event conventions: typed events in storage/event_schema.py; the loop already polls the store on several cadences.
• Design system (from api/static/dashboard.html, verified): tokens at :root lines 8-27 (--bg #080a10, --panel #111622, --border #232c41, --text #e3e8f2, --muted #7884a2, --accent #5ba3ff, --green #34d17e, --red #ff5a60, --amber #f7b13b, --mono stack); panel convention at 137-169 — a new panel is <details class="panel" id="p-x"><summary><h2>Title <span class="count">…</span></h2></summary><div id="x"></div></details>, span-6 grid cell (.full for span-12), collapth/persistence free via the details.panel classes; render via the shared helpers ($(id), fill(), table(), esc() — keep the stricter global esc at ~line 590; .pill/.count/.hint/.pos/.neg); ALL numerics in the mono stack with tabular-nums; dark-only — use tokens, never hardcoded colors.
• Headless verification recipe exists (project memory): Playwright chrome-headless-shell renders the page to catch JS errors curl can't see.

Design contract (from the design-fit review — follow, don't re-design)

• Experiment ledger panel: standard details.panel (span 6), columns experiment · n · bar · observed · verdict; verdict pill colors: PASS --green, FAIL --red, INCONCLUSIVE --muted, RUNNING/KILLED --accent/--amber; row-expand shows the pre-registered bars vs observed (mirror the Watch States expand pattern). Empty state: No experiments registered — plan 006 fills this.
• Kill switch: red ghost button FLATTEN ALL in the header right of the mode badge, visible only when snapshot.has_execution || COMMAND_BUS_ENABLED. Two-step: first click arms (button turns solid red, label CONFIRM FLATTEN (3 positions) — 5s, countdown auto-disarms); second click within 5s sends. Per-position EXIT ghost-red buttons in Open Positions rows, same two-step inline. PAUSE ENTRIES amber ghost toggle → solid amber + a thin amber banner while paused. No keyboard shortcut for any destructive action. Toast copy names the exact action: Flatten command sent — loop executes within ~5s.
• LIVE badge: derive from broker truth in the snapshot (step 4), tooltip = account + base-URL host; when live, the existing top shimmer line switches to red (one CSS rule off a body.live-mode class).
• Effective-config panel lives on ADMIN :8767 (its design system, its domain): per-key row key · effective value · source where source is a uniform-fill chip (baked | compose | overrides.env | learned); drift row highlighted when a required .env.example key is missing. Secrets masked (*** + last 4).

Commands you will need

Purpose	Command	Expected
Suite	DATABASE_URL=<pg url> python -m pytest -q	all pass
Dashboard API tests	DATABASE_URL=<pg url> python -m pytest tests/ -k "dashboard or api" -q	pass (loopback bind failures on WSL hosts are environmental, not regressions)
Headless smoke	(new) python tests/smoke_dashboard_headless.py	exits 0, "0 JS errors"

Scope

In scope:

• api/main.py (imports fix; SSE cheapening; command + config endpoints; snapshot fields)
• api/static/dashboard.html (badge truth, has_execution gating, kill-switch UI, ledger panel, sparkline wiring)
• dashboard_api.py (env plumbed for research DB sparklines if that's the gap — see step 5)
• storage/event_schema.py (OperatorCommandEvent), run_live_paper.py (command consumer on the 5s guard), trading_execution.py ONLY if pause-entries needs a flag hook (keep minimal)
• admin_api.py (+ its static page) — effective-config endpoint + table
• telegram_bot.py (guarded /flatten command emitting the same event)
• tests/ additions incl. tests/smoke_dashboard_headless.py
• .env.example (DASHBOARD_TOKEN documented as REQUIRED-when-commands-on, COMMAND_BUS_ENABLED)
• deploy/docker/truenas-compose.yml + docker-compose.yml: add PG_SKIP_SCHEMA=1 + DASHBOARD_TOKEN to the dashboard service env

Out of scope:

• Reviving NiceGUI controlroom (owner-rejected), any new UI framework/dependency, light theme.
• Approve/reject queue redesign; existing POST actions beyond making them not crash.
• Experiment tables themselves (plan 006).
• LISTEN/NOTIFY — explicitly deferred: the shared cached signal below is the lazy sufficient fix; NOTIFY is the upgrade path if it measurably falls short.

Git workflow

• Branch: advisor/007-operator-control-plane
• Step 1 ships as its own first commit (it is a production bug fix candidates for cherry-pick).
• Do NOT push or open a PR.

Steps

Step 1: Fix the POST crash + token plumbing

Add import os and import hmac to api/main.py's imports. Add DASHBOARD_TOKEN to the deploy templates' dashboard-service env (empty default + comment). Write the missing regression test: a POST with no token configured → routes (asserting no NameError); with token configured → 401 without header, 200-path with correct header.

Verify: DATABASE_URL=… python -m pytest tests/ -k "dashboard" -q → new POST tests pass.

Step 2: Cheapen the SSE change detector

In api/main.py: one module-level cached signal shared by all SSE clients — a small _ChangeSignal holding (stamp, checked_at) refreshed at most once per second GLOBALLY (lock + monotonic stamp), using ONE long-lived EventStore connection created with schema application skipped (constructor arg or PG_SKIP_SCHEMA — mirror storage/db_pg.py:275-277). Freshness query: SELECT max(timestamp) FROM events — events.id is VARCHAR PRIMARY KEY (storage/pg_schema.sql:18), NOT serial, so max(id) would be lexicographic garbage; idx_events_timestamp exists (pg_schema.sql:27) so max(timestamp) is an index-tail seek.

Verify: unit test — two fake clients, 10 ticks → underlying query runs ≤ ~10 times total (inject a counting stub); suite green.

Step 3: Operator command bus

1. storage/event_schema.py: OperatorCommandEvent — command (flatten_all|flatten_symbol|pause_entries|resume_entries), symbol (nullable), requested_by, nonce (uuid), plus standard fields.
2. api/main.py: POST /api/command — REQUIRES DASHBOARD_TOKEN configured (503 with clear message if unset: commands never run unauthenticated), validates command against the enum, emits the event. Emitting is allowed even when execution_service is None — the standalone dashboard becomes a thin emitter; expose command_bus: true in snapshots when the token is set and COMMAND_BUS_ENABLED=1.
3. run_live_paper.py: in the 5s guard task, poll operator_command events newer than the last-consumed stamp (keep a consumed-nonce set for the session; also persist consumption as an emitted operator_command_done event so a restart doesn't re-execute — REPLAY SAFETY: on boot, ignore commands older than 10 minutes). Execute: flatten_all → rt["execution"].close_session("operator_flatten") — decide deliberately: close_session also blocks re-entry for the day (that is the kill-switch semantic the safety posture wants; document it in the button tooltip). Interaction with plan 003's daily self-restart: an operator flatten must NOT trigger the exit-after-EOD-flatten path — _request_exit_after_eod is called only from step_eod_flatten's success branch, so route the operator command through close_session directly (never through step_eod_flatten) and add a test asserting no exit request fires on an operator flatten. flatten_symbol → cancel_protective_and_close for that symbol; pause_entries/resume_entries → a _operator_paused flag checked in _evaluate_entry_gates (plan 005) or, pre-005, at the top of BOTH request_approvals_for_ready_signals AND submit_breakout_now in trading_execution.py (miss one and pause silently fails on that path — the drift class plan 005 documents).
4. telegram_bot.py: /flatten and /flatten SYM commands — REMOVE "flatten" from _ACTION_WORDS refusals only for the slash-command path, keep prose refusals; authorized chat only (is_authorized); emits the same event; replies Command sent — loop executes within ~5s.
5. Dashboard UI per the design contract (two-step arm/confirm, gating, toasts).

Verify: unit tests — command consumed once (nonce dedup), stale-on-boot ignored, pause blocks a would-pass entry, unauthorized qabot chat refused; suite green.

Step 4: Truthful LIVE badge (+ has_execution gating)

Snapshot gains broker block (from env the services already have): {"base_url_host": <host of ALPACA_BASE_URL or default paper host>, "is_paper": host startswith "paper-api"}. Dashboard JS: badge = is_paper ? PAPER : LIVE (drop the schwab_live keying; keep the badge classes), tooltip shows host; body.live-mode class flips the shimmer red (one CSS rule). Gate every action control (approve/reject/exit/inject + new commands) on has_execution || command_bus.

Verify: unit test on the snapshot block (paper URL → is_paper true; live URL → false). Headless smoke (step 6) renders both states via a query stub if cheap, else at least default state with zero JS errors.

Step 5: Production sparklines

dashboard_api.py/api/main.py: the standalone dashboard's bars/sparkline endpoints need the research DB connection the embedded mode has. Identify the exact gap by grepping research_con / bars queries in api/main.py — wire the standalone entrypoint to construct the same reader (read-only) the orchestrator passes. If the gap is env-only (DATABASE_URL already reaches it), fix the query path; report in the commit message what the actual missing wire was.

Verify: run python dashboard_api.py against a dev DB → curl -s "127.0.0.1:8765/api/bars?symbol=X" returns HTTP 200 with a JSON array (rows when the dev DB has bars for X; an empty array on a bare DB is a PASS — the fix is the non-None reader, not the data). With bars present, headless smoke shows no fetch error.

Step 6: Headless smoke test + effective-config panel

1. tests/smoke_dashboard_headless.py — NEW file; the recipe is not in the repo, so here it is in full:
   • Setup (once per environment): pip install playwright && python -m playwright install chrome-headless-shell. On minimal Linux/WSL where the browser fails to launch with a libasound error, stub it: mkdir -p ~/.lib-stubs && touch ~/.lib-stubs/libasound.so.2 && export LD_LIBRARY_PATH=~/.lib-stubs:$LD_LIBRARY_PATH (the dashboard uses WebAudio only after user gesture; a dead stub is fine for rendering).
   • Script shape (sync API): start the dashboard server on an ephemeral port against the test DB (reuse how tests/ boot create_server), then:

     from playwright.sync_api import sync_playwright
     errors = []
     with sync_playwright() as p:
         b = p.chromium.launch(channel="chrome-headless-shell", args=["--no-sandbox"])
         page = b.new_page()
         page.on("console", lambda m: errors.append(m.text) if m.type == "error" else None)
         page.on("pageerror", lambda e: errors.append(str(e)))
         page.goto(f"http://127.0.0.1:{port}", wait_until="networkidle")
         assert page.locator(".panel").count() >= 8, "panels missing"
         assert page.locator("#mode-badge").count() == 1, "badge missing"
         b.close()
     assert not errors, f"JS errors: {errors}"
     

   • If channel="chrome-headless-shell" is unsupported by the installed Playwright version, use p.chromium.launch(headless=True) after python -m playwright install chromium. Wire into CI as a non-blocking step first (loopback binds can be flaky on WSL hosts), promote to blocking after one green week.
2. Admin effective-config: admin_api.py endpoint GET /api/config/effective — merge order baked-env < compose-env < data/overrides.env < learned_params.json-applied keys; return {key: {value_masked, source}}; static table per the design contract. Mask any key matching (KEY|SECRET|TOKEN|PASSWORD).

Verify: smoke test exits 0 locally; curl -s :8767/api/config/effective | python -m json.tool shows masked values + sources; admin tests (new, minimal: auth + masking) pass.

Test plan

• POST regression (no-NameError), token 401/200 (step 1)
• SSE shared-signal query budget (step 2)
• Command bus: nonce dedup, boot-stale ignore, pause gate (blocks BOTH path functions), qabot auth, and operator flatten does NOT request the plan-003 daily-restart exit (step 3)
• Badge truth from base URL (step 4)
• Admin: effective-config masking + auth (step 6)
• tests/smoke_dashboard_headless.py — the JS-error net that would have caught the NameError's sibling class
• Verification: DATABASE_URL=… python -m pytest -q green (~12 new).

Done criteria

• grep -n "^import os\|^import hmac" api/main.py → both present; POST works in tests
• SSE detector: one shared signal, PG_SKIP_SCHEMA honored, dashboard service env updated in both compose files
• OperatorCommandEvent end-to-end: dashboard button → event → loop executes → operator_command_done; qabot /flatten works for the authorized chat
• Badge reflects broker truth; action UI gated on has_execution || command_bus
• Standalone dashboard serves sparkline data
• Effective-config table live on :8767 with masked values + per-key source
• Headless smoke exits 0; full suite green
• No new colors outside :root tokens in dashboard.html diff
• plans/README.md updated

STOP conditions

• api/main.py at HEAD already imports os/hmac (someone hotfixed) — skip step 1, continue.
• The events table lacks BOTH a serial id and an indexed timestamp usable for the cheap change signal — report before inventing a third scheme.
• close_session proves unusable as flatten_all (e.g. it exits the process after plan 003) — coordinate: the command consumer must flatten WITHOUT the daily self-restart exit; if the two plans' semantics collide, stop and report the interaction.
• Any step requires adding a JS dependency/framework — hard stop (zero-dependency page is a constraint, not a preference).

Maintenance notes

• The command bus is deliberately minimal: enum commands, token-gated, nonce-deduped, 10-min staleness window. Adding parameterized commands (e.g. "resize position") should raise design review — the audit trail and replay-safety analysis must be redone per command.
• The two-step confirm's 5s disarm and the loop's 5s pickup are independent timers; total worst-case latency ~10s — acceptable because broker-side stops remain the fast protection layer. Do not "improve" by having the dashboard call the broker directly; the loop is the single execution authority.
• When plan 006's tables exist, add the ledger panel data key (experiment_verdicts latest per experiment) to DashboardState.snapshots() following the established pattern (filter on native timestamp column first — payload_json casts on the full events table time out; see the panel-adding convention).
• Reviewer scrutiny: XSS — every interpolated value in new panels goes through the global esc() (line ~590); the config panel doubly so (env values are attacker-influenced in principle).