diff --git a/deploy/docker/bootstrap_live_db.sh b/deploy/docker/bootstrap_live_db.sh new file mode 100644 index 0000000..98d7fc1 --- /dev/null +++ b/deploy/docker/bootstrap_live_db.sh @@ -0,0 +1,41 @@ +#!/bin/bash +# Seed the LIVE stack's fresh Postgres from the paper stack's nightly dump, +# then wipe the money-record tables so live P&L starts at zero while every +# research baseline (minute/daily bars, rvol + regime history, learned +# setups) carries over. Run ON CUBE after the live stack's postgres is up. +# +# ./bootstrap_live_db.sh +# e.g. ./bootstrap_live_db.sh \ +# /mnt/Volume/Storage/Apps/momentum/data/backups/momentum-2026-07-28.sql.gz \ +# ix-momentum-live-postgres-1 +# +# Idempotent-ish: restoring over a non-empty DB will error on existing objects; +# for a re-run, recreate the live stack's pgdata volume first (it is DISPOSABLE +# until the flip — that's the point of staging). +set -euo pipefail + +BACKUP="${1:?usage: bootstrap_live_db.sh }" +CONTAINER="${2:?usage: bootstrap_live_db.sh }" +PSQL=(docker exec -i "$CONTAINER" psql -U momentum -d momentum -v ON_ERROR_STOP=0 -q) + +echo "[1/3] restoring $BACKUP into $CONTAINER ..." +gunzip -c "$BACKUP" | "${PSQL[@]}" >/dev/null + +echo "[2/3] zeroing money-record tables (all trades/P&L derive from events) ..." +docker exec -i "$CONTAINER" psql -U momentum -d momentum -v ON_ERROR_STOP=1 <<'SQL' +TRUNCATE events, telemetry_events; +-- paper-only artifacts that must not leak into the live stack's views +TRUNCATE simulated_orders, simulated_fills, paper_missed_trades, paper_sessions; +SQL + +echo "[3/3] sanity ..." +docker exec -i "$CONTAINER" psql -U momentum -d momentum -t <<'SQL' +SELECT 'minute_bars ' || count(*) || ' (want millions)' FROM minute_bars +UNION ALL SELECT 'daily_bars ' || count(*) FROM daily_bars +UNION ALL SELECT 'setup_events ' || count(*) FROM setup_events +UNION ALL SELECT 'regime_scores ' || count(*) FROM regime_scores +UNION ALL SELECT 'events (MUST be 0) ' || count(*) FROM events; +SQL + +echo "done. Copy the paper stack's data/learned_params.json into the live" +echo "stack's data volume, then start the loop." diff --git a/docs/LIVE-CUTOVER.md b/docs/LIVE-CUTOVER.md new file mode 100644 index 0000000..1da1a49 --- /dev/null +++ b/docs/LIVE-CUTOVER.md @@ -0,0 +1,86 @@ +# Live-Money Cutover Runbook + +Staged 2026-07-22. Everything below is pre-built; flip day is configuration, +not construction. Target: minimal downtime, zero blending of paper and live +records. + +## Architecture decision + +Real money runs as a **second, fully isolated compose stack** ("momentum-live"): +own Postgres, own volumes, own dashboard. Rationale: the ~30 dashboard/P&L +projections do not filter by event mode — one missed filter would silently +blend money records. A separate database makes blending impossible by +construction. The paper stack stays untouched as the research substrate +(nightly labeler, experiments, backtests, regime scoring). + +## What is already staged + +| Piece | Where | State | +|---|---|---| +| Live compose (paper keys, current settings) | `deploy/docker/truenas-compose.live.local.yaml` (LOCAL, secrets, never commit) | ready; flip checklist in its header | +| DB seed script | `deploy/docker/bootstrap_live_db.sh` | ready | +| Equity-floor kill switch | `TRADING_EQUITY_FLOOR` (commit 5287c8e) | in image, 0=off | +| Boot order reconciliation | `TRADING_BOOT_RECONCILE` (default on, 5287c8e) | in image | +| Stale-session dashboard alarm | commit 2e39b75/5f33799 | in image | + +## Known dependencies (discovered during staging) + +1. **One paper account cannot serve two stacks.** Both composes currently + carry the same paper keys. Running both would double-trade one account AND + each stack's boot-reconcile would cancel the other's resting orders as + orphans. Before first boot: create a second Alpaca paper key pair for the + live stack's shakedown (or pause the paper stack's loop during it). +2. **Fresh DB must be seeded, not born empty.** Empty DB = rvol baselines + missing (~1 day to self-heal) and, worse, the regime gate fail-opens for + weeks (no premarket history) — the one validated protection would be inert + exactly at go-live. `bootstrap_live_db.sh` seeds from the paper stack's + nightly dump and truncates the money-record tables (`events`, + `telemetry_events`, simulated/paper tables), so research history carries + over while live P&L starts at zero. The dump also carries every index. +3. **`data/learned_params.json`** — copy from the paper stack's data dir into + `momentum-live/data/` (nightly tune runs only on the paper stack). +4. **Telegram** — both stacks share one chat; create a second bot token for + the live stack (`TELEGRAM_BOT_TOKEN` in its compose) or alerts interleave + indistinguishably. +5. Host prerequisites: dirs `/mnt/Volume/Storage/Apps/momentum-live/data/backups` + created; host ports **8865** (dashboard) and **5434** (postgres) free. + +## Cutover procedure (~15 min, paper stack keeps running throughout) + +1. `mkdir -p /mnt/Volume/Storage/Apps/momentum-live/data/backups` +2. Add the TrueNAS app from `truenas-compose.live.local.yaml`; start ONLY the + postgres service (or start all and let the loop crash-loop until step 3 — + harmless, `restart: unless-stopped`). +3. Seed: `./bootstrap_live_db.sh ` + (paper backups land nightly 20:10 ET in + `/mnt/Volume/Storage/Apps/momentum/data/backups/`). +4. `cp .../momentum/data/learned_params.json .../momentum-live/data/` +5. Shakedown on the SECOND paper key pair for 1-2 sessions: watch the 8865 + dashboard — armed triggers populate, entries fire, no stale-session banner, + boot-reconcile quiet. +6. **The flip** (edit live compose env, both loop + dashboard, then redeploy): + - `ALPACA_BASE_URL: https://api.alpaca.markets` + - `ALPACA_API_KEY` / `ALPACA_SECRET_KEY`: live keys + - `TRADING_RISK_PER_TRADE_PCT: '0.0025'` (start tiny) + - `TRADING_MAX_RISK_DOLLARS: '250'` (hard ceiling back on) + - `TRADING_REGIME_GATE_ENABLED: '1'` (the one validated lever) + - `TRADING_EQUITY_FLOOR:` 0.90 x starting equity (absolute dollars) +7. Verify within the first hour: dashboard LIVE badge, one order's audit trail + end-to-end, `python -m research.slippage --since ` after the first + close, Telegram alert received. + +## Rollback + +Point the live compose back at paper keys/URL and redeploy — the stack itself +is the same code either way. The paper stack was never touched. If the live DB +is ever suspect, it is disposable pre-flip: destroy pgdata, re-run the seed. + +## Notes + +- FINRA PDT rule retired June 4 2026 (Alpaca Intraday Margin Framework) — no + $25k floor; standard margin minimums apply. +- Measured fill reality on gate-passing names: 0.3-0.5% median spread, ~12% + profit drag vs paper; p90 tail 2.4-3.6% and book-walks are the real risks + (`research/slippage.py` is the standing report). +- Gate for the flip: 2-3 consecutive green regime-gated paper weeks on the + fixed stack (week of 07-20: +$1,732, 8-for-8, regime OFF/watch-mode).