docs: live-money cutover runbook + one-command DB seed script
docs/LIVE-CUTOVER.md: full flip-day procedure — staged pieces, discovered dependencies (second paper key pair, DB seeding, learned_params copy, telegram token, ports/dirs), 15-min cutover with the paper stack running throughout, verification list, rollback. bootstrap_live_db.sh seeds the live stack's Postgres from the paper nightly dump and truncates the money-record tables so research history carries over while live P&L starts at zero. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
1 parent
5287c8ebfe
commit
148e300af4
2 files changed
+127
No files matched your search
@@ -0,0 +1,41 @@
|
||||
#!/bin/bash
|
||||
# Seed the LIVE stack's fresh Postgres from the paper stack's nightly dump,
|
||||
# then wipe the money-record tables so live P&L starts at zero while every
|
||||
# research baseline (minute/daily bars, rvol + regime history, learned
|
||||
# setups) carries over. Run ON CUBE after the live stack's postgres is up.
|
||||
#
|
||||
# ./bootstrap_live_db.sh <backup.sql.gz> <live-postgres-container>
|
||||
# e.g. ./bootstrap_live_db.sh \
|
||||
# /mnt/Volume/Storage/Apps/momentum/data/backups/momentum-2026-07-28.sql.gz \
|
||||
# ix-momentum-live-postgres-1
|
||||
#
|
||||
# Idempotent-ish: restoring over a non-empty DB will error on existing objects;
|
||||
# for a re-run, recreate the live stack's pgdata volume first (it is DISPOSABLE
|
||||
# until the flip — that's the point of staging).
|
||||
set -euo pipefail
|
||||
|
||||
BACKUP="${1:?usage: bootstrap_live_db.sh <backup.sql.gz> <live-pg-container>}"
|
||||
CONTAINER="${2:?usage: bootstrap_live_db.sh <backup.sql.gz> <live-pg-container>}"
|
||||
PSQL=(docker exec -i "$CONTAINER" psql -U momentum -d momentum -v ON_ERROR_STOP=0 -q)
|
||||
|
||||
echo "[1/3] restoring $BACKUP into $CONTAINER ..."
|
||||
gunzip -c "$BACKUP" | "${PSQL[@]}" >/dev/null
|
||||
|
||||
echo "[2/3] zeroing money-record tables (all trades/P&L derive from events) ..."
|
||||
docker exec -i "$CONTAINER" psql -U momentum -d momentum -v ON_ERROR_STOP=1 <<'SQL'
|
||||
TRUNCATE events, telemetry_events;
|
||||
-- paper-only artifacts that must not leak into the live stack's views
|
||||
TRUNCATE simulated_orders, simulated_fills, paper_missed_trades, paper_sessions;
|
||||
SQL
|
||||
|
||||
echo "[3/3] sanity ..."
|
||||
docker exec -i "$CONTAINER" psql -U momentum -d momentum -t <<'SQL'
|
||||
SELECT 'minute_bars ' || count(*) || ' (want millions)' FROM minute_bars
|
||||
UNION ALL SELECT 'daily_bars ' || count(*) FROM daily_bars
|
||||
UNION ALL SELECT 'setup_events ' || count(*) FROM setup_events
|
||||
UNION ALL SELECT 'regime_scores ' || count(*) FROM regime_scores
|
||||
UNION ALL SELECT 'events (MUST be 0) ' || count(*) FROM events;
|
||||
SQL
|
||||
|
||||
echo "done. Copy the paper stack's data/learned_params.json into the live"
|
||||
echo "stack's data volume, then start the loop."
|
||||
@@ -0,0 +1,86 @@
|
||||
# Live-Money Cutover Runbook
|
||||
|
||||
Staged 2026-07-22. Everything below is pre-built; flip day is configuration,
|
||||
not construction. Target: minimal downtime, zero blending of paper and live
|
||||
records.
|
||||
|
||||
## Architecture decision
|
||||
|
||||
Real money runs as a **second, fully isolated compose stack** ("momentum-live"):
|
||||
own Postgres, own volumes, own dashboard. Rationale: the ~30 dashboard/P&L
|
||||
projections do not filter by event mode — one missed filter would silently
|
||||
blend money records. A separate database makes blending impossible by
|
||||
construction. The paper stack stays untouched as the research substrate
|
||||
(nightly labeler, experiments, backtests, regime scoring).
|
||||
|
||||
## What is already staged
|
||||
|
||||
| Piece | Where | State |
|
||||
|---|---|---|
|
||||
| Live compose (paper keys, current settings) | `deploy/docker/truenas-compose.live.local.yaml` (LOCAL, secrets, never commit) | ready; flip checklist in its header |
|
||||
| DB seed script | `deploy/docker/bootstrap_live_db.sh` | ready |
|
||||
| Equity-floor kill switch | `TRADING_EQUITY_FLOOR` (commit 5287c8e) | in image, 0=off |
|
||||
| Boot order reconciliation | `TRADING_BOOT_RECONCILE` (default on, 5287c8e) | in image |
|
||||
| Stale-session dashboard alarm | commit 2e39b75/5f33799 | in image |
|
||||
|
||||
## Known dependencies (discovered during staging)
|
||||
|
||||
1. **One paper account cannot serve two stacks.** Both composes currently
|
||||
carry the same paper keys. Running both would double-trade one account AND
|
||||
each stack's boot-reconcile would cancel the other's resting orders as
|
||||
orphans. Before first boot: create a second Alpaca paper key pair for the
|
||||
live stack's shakedown (or pause the paper stack's loop during it).
|
||||
2. **Fresh DB must be seeded, not born empty.** Empty DB = rvol baselines
|
||||
missing (~1 day to self-heal) and, worse, the regime gate fail-opens for
|
||||
weeks (no premarket history) — the one validated protection would be inert
|
||||
exactly at go-live. `bootstrap_live_db.sh` seeds from the paper stack's
|
||||
nightly dump and truncates the money-record tables (`events`,
|
||||
`telemetry_events`, simulated/paper tables), so research history carries
|
||||
over while live P&L starts at zero. The dump also carries every index.
|
||||
3. **`data/learned_params.json`** — copy from the paper stack's data dir into
|
||||
`momentum-live/data/` (nightly tune runs only on the paper stack).
|
||||
4. **Telegram** — both stacks share one chat; create a second bot token for
|
||||
the live stack (`TELEGRAM_BOT_TOKEN` in its compose) or alerts interleave
|
||||
indistinguishably.
|
||||
5. Host prerequisites: dirs `/mnt/Volume/Storage/Apps/momentum-live/data/backups`
|
||||
created; host ports **8865** (dashboard) and **5434** (postgres) free.
|
||||
|
||||
## Cutover procedure (~15 min, paper stack keeps running throughout)
|
||||
|
||||
1. `mkdir -p /mnt/Volume/Storage/Apps/momentum-live/data/backups`
|
||||
2. Add the TrueNAS app from `truenas-compose.live.local.yaml`; start ONLY the
|
||||
postgres service (or start all and let the loop crash-loop until step 3 —
|
||||
harmless, `restart: unless-stopped`).
|
||||
3. Seed: `./bootstrap_live_db.sh <latest paper backup> <live-pg-container>`
|
||||
(paper backups land nightly 20:10 ET in
|
||||
`/mnt/Volume/Storage/Apps/momentum/data/backups/`).
|
||||
4. `cp .../momentum/data/learned_params.json .../momentum-live/data/`
|
||||
5. Shakedown on the SECOND paper key pair for 1-2 sessions: watch the 8865
|
||||
dashboard — armed triggers populate, entries fire, no stale-session banner,
|
||||
boot-reconcile quiet.
|
||||
6. **The flip** (edit live compose env, both loop + dashboard, then redeploy):
|
||||
- `ALPACA_BASE_URL: https://api.alpaca.markets`
|
||||
- `ALPACA_API_KEY` / `ALPACA_SECRET_KEY`: live keys
|
||||
- `TRADING_RISK_PER_TRADE_PCT: '0.0025'` (start tiny)
|
||||
- `TRADING_MAX_RISK_DOLLARS: '250'` (hard ceiling back on)
|
||||
- `TRADING_REGIME_GATE_ENABLED: '1'` (the one validated lever)
|
||||
- `TRADING_EQUITY_FLOOR:` 0.90 x starting equity (absolute dollars)
|
||||
7. Verify within the first hour: dashboard LIVE badge, one order's audit trail
|
||||
end-to-end, `python -m research.slippage --since <today>` after the first
|
||||
close, Telegram alert received.
|
||||
|
||||
## Rollback
|
||||
|
||||
Point the live compose back at paper keys/URL and redeploy — the stack itself
|
||||
is the same code either way. The paper stack was never touched. If the live DB
|
||||
is ever suspect, it is disposable pre-flip: destroy pgdata, re-run the seed.
|
||||
|
||||
## Notes
|
||||
|
||||
- FINRA PDT rule retired June 4 2026 (Alpaca Intraday Margin Framework) — no
|
||||
$25k floor; standard margin minimums apply.
|
||||
- Measured fill reality on gate-passing names: 0.3-0.5% median spread, ~12%
|
||||
profit drag vs paper; p90 tail 2.4-3.6% and book-walks are the real risks
|
||||
(`research/slippage.py` is the standing report).
|
||||
- Gate for the flip: 2-3 consecutive green regime-gated paper weeks on the
|
||||
fixed stack (week of 07-20: +$1,732, 8-for-8, regime OFF/watch-mode).
|
||||
Reference in new issue
Block a user