# RUNBOOK

Operating the Momentum stack day to day. Deployment shapes and setup live in
[deploy/README.md](deploy/README.md); this is "it's running — now what."

## Daily rhythm (all times ET)

| When | What happens | Verify |
|---|---|---|
| ~04:00+ | premarket ingest + discovery begin | trigger board fills with gappers |
| 09:30 | session gate opens; entries allowed 09:30–15:55 | first `TRIGGER FIRED` / auto-approved orders |
| 09:35 | opening ranges complete → triggers arm | board states flip waiting→armed |
| 15:55 | fresh entries barred | — |
| 16:00± | EOD flatten of open positions | positions = 0 |
| 20:00 Mon–Fri | nightly job: SIP re-fetch, event retention, log rotation, shadow tracks, float/dilution caches | `appended:` line in scheduler log |

## Health checks

**Compose (production):**
```bash
docker ps                      # all momentum containers Up, loop (healthy)
docker logs -f <loop>          # discover/ingest/watch/execute cycle lines
curl -s localhost:8765/api/snapshots | head -c 200    # dashboard serves
# admin panel: http://<host>:8767 — services, DB, Alpaca, Ollama cards
```

**Local (systemd):** `systemctl --user status 'momentum-*'` + the same curls.

**Database truth (any shape):**
```sql
SELECT max(timestamp) FROM events;           -- heartbeat: < ~2 min during any hour
SELECT count(*) FROM minute_bars;            -- history intact: ~19M+ (2026-07)
```
Timestamps are **ET-naive**: compare against ET now, not UTC (a constant
"4-hour stale" reading is a timezone artifact, not an outage).

## Failure modes seen in production (and the fix)

| Symptom | Cause | Fix |
|---|---|---|
| Entries queue as "manual" in the dashboard | `TRADING_AUTO_APPROVE` missing from the loop's env | add `=1` to the loop service env; stale queue rows are zombies that clear with the session |
| Orders submitted after hours | (fixed 2026-07-10) missing session gate | `TRADING_SESSION_ONLY=1` — default on; verify if reintroduced |
| Duplicate resting orders after a loop crash/restart | armed/submitted state is in-memory | diff Alpaca open orders after any loop restart; cancel dupes (`get_orders` unscoped — `symbols=` can silently return empty) |
| `PermissionError` on `/app/data/...` | host dir not owned by container uid | `chown -R 10001:10001 <data dir>` |
| Postgres serving a tiny DB when history should exist | wrong pgdata mount (path typo/case, nested cluster, stale container) | `docker inspect <postgres> --format '{{range .Mounts}}...'`; verify by **size at destination**; recreate containers after volume edits (restart is not enough) |
| Autovacuum cancelled every 60s on a big table | a frequent connector re-applying schema DDL | healthchecks/one-shot scripts must set `PG_SKIP_SCHEMA=1` |
| Env edits in the YAML do nothing | TrueNAS expanded the anchors; `x-app` is decorative | put env in each **service's** block; verify with `docker exec <svc> sh -c 'echo $VAR'` |
| Loop trading names it shouldn't | check gates in order: static open gap 10–35% (or ≥150% + rvol≥5), $2–20, day-vol, quality ≥0.60, VWAP, session window | `./momentum inspect criteria SYM` or the events `risk_rule_triggered` payloads |

## Safety posture

- Paper account; bracket orders carry the protective stop; the exit manager
  enforces stops on anything naked, ratchets breakeven at +5%, catastrophe-
  stops at −10%/1.5×R.
- **When in doubt, stop the loop container** — positions stay protected by
  broker-side stop legs; a missed day costs nothing (the no-trade day is a
  legitimate outcome). Never run two loops against one account.
- Config experiments: use the admin panel's profiles; treat each switch as a
  hypothesis and compare the day against `validated-core` behavior.

## Where things live

- **Everything queryable**: PostgreSQL (`DATABASE_URL`) — events, bars,
  research, shadow tracks, caches.
- **Local disk / app-data volume**: logs (rotated nightly), transcripts,
  `learned_params.json`, `overrides.env` (admin panel), `profiles/`.
- **Historical plans/era docs**: `docs/history/` — don't follow them.