docs: live-money cutover runbook + one-command DB seed script

docs/LIVE-CUTOVER.md: full flip-day procedure — staged pieces, discovered
dependencies (second paper key pair, DB seeding, learned_params copy,
telegram token, ports/dirs), 15-min cutover with the paper stack running
throughout, verification list, rollback. bootstrap_live_db.sh seeds the
live stack's Postgres from the paper nightly dump and truncates the
money-record tables so research history carries over while live P&L
starts at zero.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
momentum devandClaude Fable 5 committed 2026-07-22 17:29:06 -05:00
1 parent 5287c8ebfe
commit 148e300af4
2 files changed
+127

No files matched your search

+41
View File
@@ -0,0 +1,41 @@
#!/bin/bash
# Seed the LIVE stack's fresh Postgres from the paper stack's nightly dump,
# then wipe the money-record tables so live P&L starts at zero while every
# research baseline (minute/daily bars, rvol + regime history, learned
# setups) carries over. Run ON CUBE after the live stack's postgres is up.
#
# ./bootstrap_live_db.sh <backup.sql.gz> <live-postgres-container>
# e.g. ./bootstrap_live_db.sh \
# /mnt/Volume/Storage/Apps/momentum/data/backups/momentum-2026-07-28.sql.gz \
# ix-momentum-live-postgres-1
#
# Idempotent-ish: restoring over a non-empty DB will error on existing objects;
# for a re-run, recreate the live stack's pgdata volume first (it is DISPOSABLE
# until the flip — that's the point of staging).
set -euo pipefail
BACKUP="${1:?usage: bootstrap_live_db.sh <backup.sql.gz> <live-pg-container>}"
CONTAINER="${2:?usage: bootstrap_live_db.sh <backup.sql.gz> <live-pg-container>}"
PSQL=(docker exec -i "$CONTAINER" psql -U momentum -d momentum -v ON_ERROR_STOP=0 -q)
echo "[1/3] restoring $BACKUP into $CONTAINER ..."
gunzip -c "$BACKUP" | "${PSQL[@]}" >/dev/null
echo "[2/3] zeroing money-record tables (all trades/P&L derive from events) ..."
docker exec -i "$CONTAINER" psql -U momentum -d momentum -v ON_ERROR_STOP=1 <<'SQL'
TRUNCATE events, telemetry_events;
-- paper-only artifacts that must not leak into the live stack's views
TRUNCATE simulated_orders, simulated_fills, paper_missed_trades, paper_sessions;
SQL
echo "[3/3] sanity ..."
docker exec -i "$CONTAINER" psql -U momentum -d momentum -t <<'SQL'
SELECT 'minute_bars ' || count(*) || ' (want millions)' FROM minute_bars
UNION ALL SELECT 'daily_bars ' || count(*) FROM daily_bars
UNION ALL SELECT 'setup_events ' || count(*) FROM setup_events
UNION ALL SELECT 'regime_scores ' || count(*) FROM regime_scores
UNION ALL SELECT 'events (MUST be 0) ' || count(*) FROM events;
SQL
echo "done. Copy the paper stack's data/learned_params.json into the live"
echo "stack's data volume, then start the loop."
+86
View File
@@ -0,0 +1,86 @@
# Live-Money Cutover Runbook
Staged 2026-07-22. Everything below is pre-built; flip day is configuration,
not construction. Target: minimal downtime, zero blending of paper and live
records.
## Architecture decision
Real money runs as a **second, fully isolated compose stack** ("momentum-live"):
own Postgres, own volumes, own dashboard. Rationale: the ~30 dashboard/P&L
projections do not filter by event mode — one missed filter would silently
blend money records. A separate database makes blending impossible by
construction. The paper stack stays untouched as the research substrate
(nightly labeler, experiments, backtests, regime scoring).
## What is already staged
| Piece | Where | State |
|---|---|---|
| Live compose (paper keys, current settings) | `deploy/docker/truenas-compose.live.local.yaml` (LOCAL, secrets, never commit) | ready; flip checklist in its header |
| DB seed script | `deploy/docker/bootstrap_live_db.sh` | ready |
| Equity-floor kill switch | `TRADING_EQUITY_FLOOR` (commit 5287c8e) | in image, 0=off |
| Boot order reconciliation | `TRADING_BOOT_RECONCILE` (default on, 5287c8e) | in image |
| Stale-session dashboard alarm | commit 2e39b75/5f33799 | in image |
## Known dependencies (discovered during staging)
1. **One paper account cannot serve two stacks.** Both composes currently
carry the same paper keys. Running both would double-trade one account AND
each stack's boot-reconcile would cancel the other's resting orders as
orphans. Before first boot: create a second Alpaca paper key pair for the
live stack's shakedown (or pause the paper stack's loop during it).
2. **Fresh DB must be seeded, not born empty.** Empty DB = rvol baselines
missing (~1 day to self-heal) and, worse, the regime gate fail-opens for
weeks (no premarket history) — the one validated protection would be inert
exactly at go-live. `bootstrap_live_db.sh` seeds from the paper stack's
nightly dump and truncates the money-record tables (`events`,
`telemetry_events`, simulated/paper tables), so research history carries
over while live P&L starts at zero. The dump also carries every index.
3. **`data/learned_params.json`** — copy from the paper stack's data dir into
`momentum-live/data/` (nightly tune runs only on the paper stack).
4. **Telegram** — both stacks share one chat; create a second bot token for
the live stack (`TELEGRAM_BOT_TOKEN` in its compose) or alerts interleave
indistinguishably.
5. Host prerequisites: dirs `/mnt/Volume/Storage/Apps/momentum-live/data/backups`
created; host ports **8865** (dashboard) and **5434** (postgres) free.
## Cutover procedure (~15 min, paper stack keeps running throughout)
1. `mkdir -p /mnt/Volume/Storage/Apps/momentum-live/data/backups`
2. Add the TrueNAS app from `truenas-compose.live.local.yaml`; start ONLY the
postgres service (or start all and let the loop crash-loop until step 3 —
harmless, `restart: unless-stopped`).
3. Seed: `./bootstrap_live_db.sh <latest paper backup> <live-pg-container>`
(paper backups land nightly 20:10 ET in
`/mnt/Volume/Storage/Apps/momentum/data/backups/`).
4. `cp .../momentum/data/learned_params.json .../momentum-live/data/`
5. Shakedown on the SECOND paper key pair for 1-2 sessions: watch the 8865
dashboard — armed triggers populate, entries fire, no stale-session banner,
boot-reconcile quiet.
6. **The flip** (edit live compose env, both loop + dashboard, then redeploy):
- `ALPACA_BASE_URL: https://api.alpaca.markets`
- `ALPACA_API_KEY` / `ALPACA_SECRET_KEY`: live keys
- `TRADING_RISK_PER_TRADE_PCT: '0.0025'` (start tiny)
- `TRADING_MAX_RISK_DOLLARS: '250'` (hard ceiling back on)
- `TRADING_REGIME_GATE_ENABLED: '1'` (the one validated lever)
- `TRADING_EQUITY_FLOOR:` 0.90 x starting equity (absolute dollars)
7. Verify within the first hour: dashboard LIVE badge, one order's audit trail
end-to-end, `python -m research.slippage --since <today>` after the first
close, Telegram alert received.
## Rollback
Point the live compose back at paper keys/URL and redeploy — the stack itself
is the same code either way. The paper stack was never touched. If the live DB
is ever suspect, it is disposable pre-flip: destroy pgdata, re-run the seed.
## Notes
- FINRA PDT rule retired June 4 2026 (Alpaca Intraday Margin Framework) — no
$25k floor; standard margin minimums apply.
- Measured fill reality on gate-passing names: 0.3-0.5% median spread, ~12%
profit drag vs paper; p90 tail 2.4-3.6% and book-walks are the real risks
(`research/slippage.py` is the standing report).
- Gate for the flip: 2-3 consecutive green regime-gated paper weeks on the
fixed stack (week of 07-20: +$1,732, 8-for-8, regime OFF/watch-mode).