momentum devandClaude Opus 4.8
5ac2cc7b89
- 10 era documents (phase plans, build summaries, redesign writeups, old runbooks, completed consolidation plan) moved to docs/history/ with a do-not-follow README. - Root README: current validated population ($2-20, 10-35% static gap + squeeze exception, SIP), Deployment section, automated-maintenance reality, corrected repo map (admin panel, youtube_claims, deploy/). - RUNBOOK rewritten for the running system: daily rhythm, health checks per deployment, and the production failure-mode table earned this week (env drift, pgdata mount traps, uid 10001, zombie approvals, PG_SKIP_ SCHEMA, timezone artifacts). - deploy/README.md is now the three-mode index: compose (canonical), systemd (local dev — unit templates now committed), helm (refresh staged as issue #8). Old k8s doc moved to deploy/helm/README.md. - docker/README.md: six TrueNAS gotchas, each from a real outage. - Admin panel improvements staged as GitHub issues #4-#7. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
68 lines
4.0 KiB
Markdown
68 lines
4.0 KiB
Markdown
# RUNBOOK
|
||
|
||
Operating the Momentum stack day to day. Deployment shapes and setup live in
|
||
[deploy/README.md](deploy/README.md); this is "it's running — now what."
|
||
|
||
## Daily rhythm (all times ET)
|
||
|
||
| When | What happens | Verify |
|
||
|---|---|---|
|
||
| ~04:00+ | premarket ingest + discovery begin | trigger board fills with gappers |
|
||
| 09:30 | session gate opens; entries allowed 09:30–15:55 | first `TRIGGER FIRED` / auto-approved orders |
|
||
| 09:35 | opening ranges complete → triggers arm | board states flip waiting→armed |
|
||
| 15:55 | fresh entries barred | — |
|
||
| 16:00± | EOD flatten of open positions | positions = 0 |
|
||
| 20:00 Mon–Fri | nightly job: SIP re-fetch, event retention, log rotation, shadow tracks, float/dilution caches | `appended:` line in scheduler log |
|
||
|
||
## Health checks
|
||
|
||
**Compose (production):**
|
||
```bash
|
||
docker ps # all momentum containers Up, loop (healthy)
|
||
docker logs -f <loop> # discover/ingest/watch/execute cycle lines
|
||
curl -s localhost:8765/api/snapshots | head -c 200 # dashboard serves
|
||
# admin panel: http://<host>:8767 — services, DB, Alpaca, Ollama cards
|
||
```
|
||
|
||
**Local (systemd):** `systemctl --user status 'momentum-*'` + the same curls.
|
||
|
||
**Database truth (any shape):**
|
||
```sql
|
||
SELECT max(timestamp) FROM events; -- heartbeat: < ~2 min during any hour
|
||
SELECT count(*) FROM minute_bars; -- history intact: ~19M+ (2026-07)
|
||
```
|
||
Timestamps are **ET-naive**: compare against ET now, not UTC (a constant
|
||
"4-hour stale" reading is a timezone artifact, not an outage).
|
||
|
||
## Failure modes seen in production (and the fix)
|
||
|
||
| Symptom | Cause | Fix |
|
||
|---|---|---|
|
||
| Entries queue as "manual" in the dashboard | `TRADING_AUTO_APPROVE` missing from the loop's env | add `=1` to the loop service env; stale queue rows are zombies that clear with the session |
|
||
| Orders submitted after hours | (fixed 2026-07-10) missing session gate | `TRADING_SESSION_ONLY=1` — default on; verify if reintroduced |
|
||
| Duplicate resting orders after a loop crash/restart | armed/submitted state is in-memory | diff Alpaca open orders after any loop restart; cancel dupes (`get_orders` unscoped — `symbols=` can silently return empty) |
|
||
| `PermissionError` on `/app/data/...` | host dir not owned by container uid | `chown -R 10001:10001 <data dir>` |
|
||
| Postgres serving a tiny DB when history should exist | wrong pgdata mount (path typo/case, nested cluster, stale container) | `docker inspect <postgres> --format '{{range .Mounts}}...'`; verify by **size at destination**; recreate containers after volume edits (restart is not enough) |
|
||
| Autovacuum cancelled every 60s on a big table | a frequent connector re-applying schema DDL | healthchecks/one-shot scripts must set `PG_SKIP_SCHEMA=1` |
|
||
| Env edits in the YAML do nothing | TrueNAS expanded the anchors; `x-app` is decorative | put env in each **service's** block; verify with `docker exec <svc> sh -c 'echo $VAR'` |
|
||
| Loop trading names it shouldn't | check gates in order: static open gap 10–35% (or ≥150% + rvol≥5), $2–20, day-vol, quality ≥0.60, VWAP, session window | `./momentum inspect criteria SYM` or the events `risk_rule_triggered` payloads |
|
||
|
||
## Safety posture
|
||
|
||
- Paper account; bracket orders carry the protective stop; the exit manager
|
||
enforces stops on anything naked, ratchets breakeven at +5%, catastrophe-
|
||
stops at −10%/1.5×R.
|
||
- **When in doubt, stop the loop container** — positions stay protected by
|
||
broker-side stop legs; a missed day costs nothing (the no-trade day is a
|
||
legitimate outcome). Never run two loops against one account.
|
||
- Config experiments: use the admin panel's profiles; treat each switch as a
|
||
hypothesis and compare the day against `validated-core` behavior.
|
||
|
||
## Where things live
|
||
|
||
- **Everything queryable**: PostgreSQL (`DATABASE_URL`) — events, bars,
|
||
research, shadow tracks, caches.
|
||
- **Local disk / app-data volume**: logs (rotated nightly), transcripts,
|
||
`learned_params.json`, `overrides.env` (admin panel), `profiles/`.
|
||
- **Historical plans/era docs**: `docs/history/` — don't follow them.
|