momentum-system/docs/LIVE-CUTOVER.md

Live-Money Cutover Runbook

Staged 2026-07-22. Everything below is pre-built; flip day is configuration, not construction. Target: minimal downtime, zero blending of paper and live records.

Architecture decision

Real money runs as a second, fully isolated compose stack ("momentum-live"): own Postgres, own volumes, own dashboard. Rationale: the ~30 dashboard/P&L projections do not filter by event mode — one missed filter would silently blend money records. A separate database makes blending impossible by construction. The paper stack stays untouched as the research substrate (nightly labeler, experiments, backtests, regime scoring).

What is already staged

Piece	Where	State
Live compose (paper keys, current settings)	deploy/docker/truenas-compose.live.local.yaml (LOCAL, secrets, never commit)	ready; flip checklist in its header
DB seed script	deploy/docker/bootstrap_live_db.sh	ready
Equity-floor kill switch	TRADING_EQUITY_FLOOR (commit 5287c8e)	in image, 0=off
Boot order reconciliation	TRADING_BOOT_RECONCILE (default on, 5287c8e)	in image
Stale-session dashboard alarm	commit 2e39b75/5f33799	in image

Known dependencies (discovered during staging)

1. One paper account cannot serve two stacks. Both composes currently carry the same paper keys. Running both would double-trade one account AND each stack's boot-reconcile would cancel the other's resting orders as orphans. Before first boot: create a second Alpaca paper key pair for the live stack's shakedown (or pause the paper stack's loop during it).
2. Fresh DB must be seeded, not born empty. Empty DB = rvol baselines missing (~1 day to self-heal) and, worse, the regime gate fail-opens for weeks (no premarket history) — the one validated protection would be inert exactly at go-live. bootstrap_live_db.sh seeds from the paper stack's nightly dump and truncates the money-record tables (events, telemetry_events, simulated/paper tables), so research history carries over while live P&L starts at zero. The dump also carries every index.
3. data/learned_params.json — copy from the paper stack's data dir into momentum-live/data/ (nightly tune runs only on the paper stack).
4. Telegram — both stacks share one chat; create a second bot token for the live stack (TELEGRAM_BOT_TOKEN in its compose) or alerts interleave indistinguishably.
5. Host prerequisites: dirs /mnt/Volume/Storage/Apps/momentum-live/data/backups created; host ports 8865 (dashboard) and 5434 (postgres) free.

Config model (added 2026-07-22)

Strategy knobs live in the config_overrides database table (commit 495a3fa), edited via the admin panel (paper :8767, live :8867) with a whitelist (TRADING_*, TRIGGER_*, WATCHER_*, NEWS_*, OLLAMA_*, DISCOVER_*; secrets refused). Boot precedence: compose env < data/overrides.env file < database — every service applies the table last; DB down falls through silently so boot never blocks on config. Compose and the overrides file retain only infra + secrets (keys, URLs, DB address — the DB address cannot live inside the DB). One-shot migration inside any stack container:

python -m runtime.env_overrides seed-from-env

copies that container's effective knobs into the table, after which the compose knob lines are dead weight (strip at any redeploy). For the live stack: run the seed after bootstrap_live_db.sh (the seeded table travels with the DB restore otherwise — which is fine too, since the paper dump's table carries the same knobs; just re-check flip-day values). Flip-day knob changes (risk, regime gate, equity floor) happen in the admin UI; only keys + ALPACA_BASE_URL remain compose edits (secret-marked, deliberately not UI-editable).

Cutover procedure (~15 min, paper stack keeps running throughout)

1. mkdir -p /mnt/Volume/Storage/Apps/momentum-live/data/backups
2. Add the TrueNAS app from truenas-compose.live.local.yaml; start ONLY the postgres service (or start all and let the loop crash-loop until step 3 — harmless, restart: unless-stopped).
3. Seed: ./bootstrap_live_db.sh <latest paper backup> <live-pg-container> (paper backups land nightly 20:10 ET in /mnt/Volume/Storage/Apps/momentum/data/backups/).
4. cp .../momentum/data/learned_params.json .../momentum-live/data/ and cp deploy/docker/overrides.live.seed.env .../momentum-live/data/overrides.env
5. Shakedown on the SECOND paper key pair for 1-2 sessions: watch the 8865 dashboard — armed triggers populate, entries fire, no stale-session banner, boot-reconcile quiet.
6. The flip (edit live compose env, both loop + dashboard, then redeploy):
   • ALPACA_BASE_URL: https://api.alpaca.markets
   • ALPACA_API_KEY / ALPACA_SECRET_KEY: live keys
   • TRADING_RISK_PER_TRADE_PCT: '0.0025' (start tiny)
   • TRADING_MAX_RISK_DOLLARS: '250' (hard ceiling back on)
   • TRADING_REGIME_GATE_ENABLED: '1' (the one validated lever)
   • TRADING_EQUITY_FLOOR: 0.90 x starting equity (absolute dollars)
7. Verify within the first hour: dashboard LIVE badge, one order's audit trail end-to-end, python -m research.slippage --since <today> after the first close, Telegram alert received.

Rollback

Point the live compose back at paper keys/URL and redeploy — the stack itself is the same code either way. The paper stack was never touched. If the live DB is ever suspect, it is disposable pre-flip: destroy pgdata, re-run the seed.

Notes

• FINRA PDT rule retired June 4 2026 (Alpaca Intraday Margin Framework) — no $25k floor; standard margin minimums apply.
• Measured fill reality on gate-passing names: 0.3-0.5% median spread, ~12% profit drag vs paper; p90 tail 2.4-3.6% and book-walks are the real risks (research/slippage.py is the standing report).
• Gate for the flip: 2-3 consecutive green regime-gated paper weeks on the fixed stack (week of 07-20: +$1,732, 8-for-8, regime OFF/watch-mode).