momentum-system/plans/001-production-safety-rails.md

Plan 001: Gate every production image on the test suite, and make loop-death and data-loss impossible to miss

Executor instructions: Follow this plan step by step. Run every verification command and confirm the expected result before moving to the next step. If anything in the "STOP conditions" section occurs, stop and report — do not improvise. When done, update the status row for this plan in plans/README.md.

Drift check (run first): git diff --stat d54a2a9..HEAD -- .github/workflows/ telegram_bot.py docker-compose.yml deploy/docker/truenas-compose.yml If any in-scope file changed since this plan was written, compare the "Current state" excerpts against the live code before proceeding; on a mismatch, treat it as a STOP condition.

Status

• Priority: P1
• Effort: S
• Risk: LOW
• Depends on: none
• Category: dx / tests / ops
• Planned at: commit d54a2a9, 2026-07-13

Why this matters

Production is a Docker Compose stack on TrueNAS that pulls :latest from a registry. Today .github/workflows/docker.yml builds and pushes that image on every master push with zero test run — and the repo has 376 passing tests sitting unused in CI. This already bit: api/main.py at HEAD references os and hmac without importing them, so every dashboard POST endpoint crashes with NameError in production (fixed separately in plan 007). Additionally: if the trading loop dies, nothing notices (the Telegram alerter lives inside the loop process — loop-down means alerts-down), and the production Postgres has no automated backup (pg_dump appears only in the one-time migration script). This plan is the lever that makes every other plan safe to ship.

Current state

• .github/workflows/docker.yml — builds + pushes ghcr.io/fez5587/momentum-app:latest and fez5587/momentum-app:latest on every master push. The jobs: block contains exactly one job (build) with checkout → two docker logins → docker/build-push-action@v6. No test step exists.
• tests/ — 376 tests collected. They need a reachable Postgres and isolate via throwaway mem_* schemas (tests/conftest.py sweeps them at session end). Invocation per README.md:112:

  DATABASE_URL=postgresql://user:pass@host:5432/momentum python -m pytest
  

• telegram_bot.py — a separate always-up container (docker-compose.yml:91-96, service qabot; restart: unless-stopped inherited from the x-app anchor at docker-compose.yml:27) with its own EventStore access. It answers questions but never checks whether the loop is alive. telegram_bot.py:219-220:

  def is_authorized(chat_id, allowed) -> bool:
      return allowed is not None and str(chat_id) == str(allowed)
  

• The loop's own notifier is in-process: run_live_paper.py:1033-1034 (notifier = TelegramNotifier(rt["store"])) — it cannot report the loop's own death.
• Heartbeat truth lives in the event store: RUNBOOK.md:31 — SELECT max(timestamp) FROM events; should be < ~2 min old during any market hour. Timestamps are ET-naive (RUNBOOK.md:34-36) — compare against ET wall clock, not UTC.
• Market-hours awareness already exists: runtime/market_calendar.py (holiday-aware, hardcoded 2026/27 NYSE calendar).
• Backups: grep -rn "pg_dump" --include="*.sh" --include="*.yml" . → the one-time deploy/docker/migrate_db.sh plus one comment at docker-compose.yml:9. The app image has NO pg_dump binary (Dockerfile is FROM python:3.13-slim installing only runtime libs), and the server is postgres:18 — a Debian-stock postgresql-client is an older major and pg_dump refuses to dump a newer server. So the backup must run with a version-matched client (the postgres:18 image itself — step 3). The scheduler service (docker-compose.yml:119-140 and deploy/docker/truenas-compose.yml:~190) is a while true shell loop that fires python -m research.premarket_eval at 20:00 ET Mon–Fri:

  if [ "$$(date +%H:%M)" = "20:00" ] && [ "$$(date +%u)" -le 5 ]; then
    echo "[scheduler] premarket_eval $$(date)"
    python -m research.premarket_eval || echo "[scheduler] eval FAILED"
  

• Dead duplicate workflow: .github/workflows/docker-publish.yml triggers on branches main and claude/kubernetes-image-package-kkdkom — the repo has no main branch (git branch -a shows only master), and it pushes a different image name (momentum-system) referenced only by the parked helm chart. Deleting it is plan 002's job; do not touch it here.

Commands you will need

Purpose	Command	Expected on success
Run tests locally	DATABASE_URL=<your pg url> python -m pytest -q	376 passed (±environmental skips)
Lint (new)	ruff check --select F .	exit 0 after step 2
Validate workflow YAML	python3 -c "import yaml,sys;yaml.safe_load(open('.github/workflows/docker.yml'))"	exit 0

Scope

In scope (the only files you should modify):

• .github/workflows/docker.yml
• telegram_bot.py
• docker-compose.yml (new db-backup service + qabot env only)
• deploy/docker/truenas-compose.yml (same)
• deploy/docker/backup_db.sh (create)
• tests/unit/test_qabot_deadman.py (create)
• .env.example (document new keys)
• plans/README.md (status row only)

Out of scope (do NOT touch, even though they look related):

• .github/workflows/docker-publish.yml — dead; deleted in plan 002.
• api/main.py POST NameError — plan 007.
• run_live_paper.py / the loop's in-process notifier — plan 003 territory.
• Any change to how tests connect to Postgres (tests/conftest.py).

Git workflow

• Branch: advisor/001-production-safety-rails
• Commit style: short imperative lowercase summary like the repo's log (admin: degrade gracefully when …). One commit per step.
• Do NOT push or open a PR unless the operator instructed it.

Steps

Step 1: Add a test job that gates the image push

In .github/workflows/docker.yml, add a test job before build and make build depend on it (needs: test). The test job needs a Postgres service container:

jobs:
  test:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:18
        env:
          POSTGRES_USER: momentum
          POSTGRES_PASSWORD: momentum
          POSTGRES_DB: momentum
        ports: ["5432:5432"]
        options: >-
          --health-cmd "pg_isready -U momentum" --health-interval 5s
          --health-timeout 5s --health-retries 10
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.13" }
      - run: pip install -r requirements.txt && pip install pytest ruff
      - run: ruff check --select F .
      - run: python -m pytest -q
        env:
          DATABASE_URL: postgresql://momentum:momentum@localhost:5432/momentum
  build:
    needs: test
    ...existing job unchanged...

The Dockerfile base is FROM python:3.13-slim (Dockerfile:8) — keep CI matched to it. If requirements.txt does not install pytest, keep the explicit pip install pytest.

Verify: python3 -c "import yaml;d=yaml.safe_load(open('.github/workflows/docker.yml'));assert d['jobs']['build']['needs']=='test';assert 'services' in d['jobs']['test']" → exit 0.

Verify (after push to a branch, if operator allows): the Actions run shows test before build.

Note: ruff check --select F . may fail on pre-existing dead code (plan 002 deletes comprehensive_backtest.py, which crashes on import but is import-dead, not an F-level error necessarily). Run it locally first; if pre-existing F errors exist that this plan must not fix, scope ruff to the money path instead: ruff check --select F run_live_paper.py trading_execution.py api/ alpaca_paper/ runtime/ storage/ strategy/ research/ and record the exclusions in the workflow comment.

Step 2: Dead-man's switch in the qabot

In telegram_bot.py, add a background check to the existing poll loop (the bot already polls Telegram every few seconds and has an EventStore): every 120s, read SELECT max(timestamp) FROM events (use the existing EventStore/projection helper if one surfaces max timestamp; otherwise a raw query through its existing connection). Convert to ET-naive comparison (the column is ET-naive per RUNBOOK). Alert conditions:

• During regular market hours (use runtime/market_calendar.py — reuse its is-open helpers; do NOT re-implement holidays): if now_et - max_ts > DEADMAN_STALE_MIN (env, default 5 minutes), send ONE Telegram message: ⚠️ DEAD-MAN: no events for N min during market hours — loop may be down. Re-alert at most every 30 min (keep a last_alerted timestamp in memory).
• Nightly OK-ping (env DEADMAN_DAILY_PING=1 default on): once per day at first check after 20:30 ET, send ✅ heartbeat: loop wrote events today; last event <ts>. (Do NOT report the backup marker file here — the qabot container does not mount the backups path; backup failures surface in the db-backup service logs.)

Config keys: DEADMAN_STALE_MIN, DEADMAN_DAILY_PING. Document both in .env.example with one-line whys.

Verify: DATABASE_URL=<pg url> python -m pytest tests/unit/test_qabot_deadman.py -q → new tests pass (see Test plan).

Step 3: Scheduled pg_dump — runs in a postgres:18 container, NOT the app image

The app image has no pg_dump (see Current state), so the backup gets its own tiny service using the SAME image as the database server (version-matched client for free).

1. Create deploy/docker/backup_db.sh (mirror the style of deploy/docker/migrate_db.sh): pg_dump "$DATABASE_URL" | gzip > "$BACKUP_DIR/momentum-$(date +%F).sql.gz", prune backups older than BACKUP_KEEP_DAYS (default 14), write $BACKUP_DIR/last_backup_ok marker with timestamp on success.
2. Add a db-backup service to BOTH compose files (root docker-compose.yml and deploy/docker/truenas-compose.yml), modeled on the scheduler's shell-loop pattern but on the postgres image:

  db-backup:
    image: postgres:18
    restart: unless-stopped
    environment:
      DATABASE_URL: ${DATABASE_URL}
      BACKUP_DIR: /backups
      BACKUP_KEEP_DAYS: "14"
    volumes:
      - ./deploy/docker/backup_db.sh:/backup_db.sh:ro
      - <the app-data volume/dataset>/backups:/backups
    command:
      - /bin/sh
      - -c
      - |
        echo "[backup] up; will fire 20:10 ET Mon-Fri"
        while true; do
          if [ "$$(date +%H:%M)" = "20:10" ] && [ "$$(date +%u)" -le 5 ]; then
            sh /backup_db.sh || echo "[backup] FAILED"
            sleep 120
          fi
          sleep 20
        done

Match each file's existing conventions for the data volume/dataset path and TZ env (the scheduler block shows both). Do NOT add pg_dump to the app image or the scheduler block. Document BACKUP_DIR/BACKUP_KEEP_DAYS in .env.example.

Verify (version-matched client): docker compose run --rm db-backup pg_dump --version → pg_dump (PostgreSQL) 18.x.

Verify (end-to-end, dev DB): docker compose run --rm -e BACKUP_DIR=/backups db-backup sh /backup_db.sh && ls <hostpath>/backups/ → one momentum-*.sql.gz + last_backup_ok; gunzip -t <hostpath>/backups/momentum-*.sql.gz → exit 0.

Test plan

• tests/unit/test_qabot_deadman.py (create; model structure on any existing tests/unit/test_*.py that stubs the EventStore — tests/unit/test_notifier*.py if present, else the simplest unit test in tests/unit/):
  1. stale heartbeat during market hours → alert function called once;
  2. stale heartbeat repeated within 30 min → no second alert;
  3. fresh heartbeat → no alert;
  4. stale heartbeat outside market hours (weekend date) → no alert. Inject the clock and the send function; do not hit Telegram or Postgres (stub the max-timestamp read).
• Verification: DATABASE_URL=<pg url> python -m pytest -q → full suite green, 4 new tests included.

Done criteria

• .github/workflows/docker.yml has a test job with a postgres service; build has needs: test
• ruff check --select F <scoped paths> exits 0 locally
• DATABASE_URL=… python -m pytest -q exits 0 with the 4 new dead-man tests
• deploy/docker/backup_db.sh exists; docker compose run --rm db-backup pg_dump --version → 18.x; end-to-end run produces a valid gzip and the marker file
• Both compose files define the db-backup service
• .env.example documents DEADMAN_STALE_MIN, DEADMAN_DAILY_PING, BACKUP_DIR, BACKUP_KEEP_DAYS
• git status shows no modified files outside the in-scope list (which includes plans/README.md)
• plans/README.md status row updated

STOP conditions

Stop and report back (do not improvise) if:

• docker.yml at HEAD no longer matches the "one build job, no test step" shape.
• The test suite does not pass locally BEFORE your changes (pre-existing red ≠ yours to fix here; report it).
• ruff check --select F reports more than ~10 pre-existing errors even after scoping — the lint step then needs a human decision, not silent exclusions.
• telegram_bot.py has no clean place to hook a periodic check (e.g. its poll loop is blocking with no timeout) — report the structure you found instead of restructuring the bot.

Maintenance notes

• The CI Postgres pin (postgres:18) should track the production compose image; if production upgrades, update both.
• The dead-man threshold assumes the loop writes at least one event every ~2 min during market hours (RUNBOOK's own invariant). If a future change makes quiet periods legitimate, raise DEADMAN_STALE_MIN rather than silencing the alert.
• Backup restore path is documented nowhere — a follow-up worth 10 lines in RUNBOOK.md: gunzip -c … | psql $DATABASE_URL.
• Reviewer should scrutinize: the ET-naive vs aware comparison in the dead-man check (the classic 4-hour-stale artifact, RUNBOOK.md:34).