momentum-system/RUNBOOK.md

RUNBOOK

Operating the Momentum stack day to day. Deployment shapes and setup live in deploy/README.md; this is "it's running — now what."

Daily rhythm (all times ET)

When	What happens	Verify
~04:00+	premarket ingest + discovery begin	trigger board fills with gappers
09:30	session gate opens; entries allowed 09:30–15:55	first TRIGGER FIRED / auto-approved orders
09:35	opening ranges complete → triggers arm	board states flip waiting→armed
15:55	fresh entries barred	—
16:00±	EOD flatten of open positions	positions = 0
20:00 Mon–Fri	nightly job: SIP re-fetch, event retention, log rotation, shadow tracks, float/dilution caches	appended: line in scheduler log

Health checks

Compose (production):

docker ps                      # all momentum containers Up, loop (healthy)
docker logs -f <loop>          # discover/ingest/watch/execute cycle lines
curl -s localhost:8765/api/snapshots | head -c 200    # dashboard serves
# admin panel: http://<host>:8767 — services, DB, Alpaca, Ollama cards

Local (systemd): systemctl --user status 'momentum-*' + the same curls.

Database truth (any shape):

SELECT max(timestamp) FROM events;           -- heartbeat: < ~2 min during any hour
SELECT count(*) FROM minute_bars;            -- history intact: ~19M+ (2026-07)

Timestamps are ET-naive: compare against ET now, not UTC (a constant "4-hour stale" reading is a timezone artifact, not an outage).

Failure modes seen in production (and the fix)

Symptom	Cause	Fix
Entries queue as "manual" in the dashboard	TRADING_AUTO_APPROVE missing from the loop's env	add =1 to the loop service env; stale queue rows are zombies that clear with the session
Orders submitted after hours	(fixed 2026-07-10) missing session gate	TRADING_SESSION_ONLY=1 — default on; verify if reintroduced
Duplicate resting orders after a loop crash/restart	armed/submitted state is in-memory	diff Alpaca open orders after any loop restart; cancel dupes (get_orders unscoped — symbols= can silently return empty)
PermissionError on /app/data/...	host dir not owned by container uid	chown -R 10001:10001 <data dir>
Postgres serving a tiny DB when history should exist	wrong pgdata mount (path typo/case, nested cluster, stale container)	docker inspect <postgres> --format '{{range .Mounts}}...'; verify by size at destination; recreate containers after volume edits (restart is not enough)
Autovacuum cancelled every 60s on a big table	a frequent connector re-applying schema DDL	healthchecks/one-shot scripts must set PG_SKIP_SCHEMA=1
Env edits in the YAML do nothing	TrueNAS expanded the anchors; x-app is decorative	put env in each service's block; verify with docker exec <svc> sh -c 'echo $VAR'
Loop trading names it shouldn't	check gates in order: static open gap 10–35% (or ≥150% + rvol≥5), $2–20, day-vol, quality ≥0.60, VWAP, session window	./momentum inspect criteria SYM or the events risk_rule_triggered payloads

Safety posture

• Paper account; bracket orders carry the protective stop; the exit manager enforces stops on anything naked, ratchets breakeven at +5%, catastrophe- stops at −10%/1.5×R.
• When in doubt, stop the loop container — positions stay protected by broker-side stop legs; a missed day costs nothing (the no-trade day is a legitimate outcome). Never run two loops against one account.
• Config experiments: use the admin panel's profiles; treat each switch as a hypothesis and compare the day against validated-core behavior.

Where things live

• Everything queryable: PostgreSQL (DATABASE_URL) — events, bars, research, shadow tracks, caches.
• Local disk / app-data volume: logs (rotated nightly), transcripts, learned_params.json, overrides.env (admin panel), profiles/.
• Historical plans/era docs: docs/history/ — don't follow them.