Docs cleanup: current-state rewrite + three deployment modes + history/

- 10 era documents (phase plans, build summaries, redesign writeups, old
  runbooks, completed consolidation plan) moved to docs/history/ with a
  do-not-follow README.
- Root README: current validated population ($2-20, 10-35% static gap +
  squeeze exception, SIP), Deployment section, automated-maintenance
  reality, corrected repo map (admin panel, youtube_claims, deploy/).
- RUNBOOK rewritten for the running system: daily rhythm, health checks
  per deployment, and the production failure-mode table earned this week
  (env drift, pgdata mount traps, uid 10001, zombie approvals, PG_SKIP_
  SCHEMA, timezone artifacts).
- deploy/README.md is now the three-mode index: compose (canonical),
  systemd (local dev — unit templates now committed), helm (refresh
  staged as issue #8). Old k8s doc moved to deploy/helm/README.md.
- docker/README.md: six TrueNAS gotchas, each from a real outage.
- Admin panel improvements staged as GitHub issues #4-#7.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

RUNBOOK.md

@@ -1,227 +1,67 @@
 # RUNBOOK
 
-Operational guide for running the momentum system. This file replaces the
-original MILESTONE*.md documents, which were unrecoverable during the
-rebuild.
+Operating the Momentum stack day to day. Deployment shapes and setup live in
+[deploy/README.md](deploy/README.md); this is "it's running — now what."
 
----
+## Daily rhythm (all times ET)
 
-## 1. Why live paper trading never worked (root causes, now fixed)
-
-1. **Minute bars were never ingested.** The watcher reads 1-minute bars from
-   `data/research/market.duckdb`, but the old ingestion only wrote *daily*
-   bars for a handful of mega-caps. Fix: `research/ingestion/market_data.py`
-   ingests 1-minute Alpaca IEX bars continuously, and
-   `run_live_paper.py` schedules it every `LIVE_BARS_INTERVAL_SECONDS`.
-
-2. **Session flags were hardcoded.** Every old row had
-   `is_regular_hours = TRUE`. Fix: flags are derived from the bar timestamp
-   in US/Eastern (premarket 04:00–09:30, regular 09:30–16:00, afterhours
-   16:00–20:00) — see `classify_session()` and its tests.
-
-3. **The candidate filter excluded everything ingested.** The query had a
-   hardcoded `close BETWEEN 1 AND 20` while only mega-caps (price ≫ 20) were
-   in the database, so the watchlist was always empty. Fix: the band is
-   configurable (`WATCHER_PRICE_MIN` / `WATCHER_PRICE_MAX`) and the default
-   symbol list now actually gets ingested.
-
-4. **No orchestrator.** Ingestion, watcher, broker sync, and execution were
-   never wired into one loop. Fix: `run_live_paper.py`.
-
-5. **Broken source files.** `runtime/providers/registry.py`,
-   `schwab/market/models.py`, `option_chain_service.py`, and
-   `schwab/streaming/client.py` had syntax errors that crashed imports.
-   All repaired/rewritten.
-
-6. **Schwab tokens were committed to git.** `data/schwab_tokens.json` is now
-   git-ignored and written `chmod 0600`. **Action required:** if that file
-   exists anywhere in your repo history, revoke the refresh token in the
-   Schwab developer portal and re-run the OAuth flow. Scrub history with
-   `git filter-repo --path data/schwab_tokens.json --invert-paths` if the
-   repo was ever shared.
-
----
-
-## 2. Environment setup
-
-```bash
-pip install -r requirements.txt
-cp .env.example .env
-```
-
-Minimum for paper trading: `ALPACA_API_KEY`, `ALPACA_SECRET_KEY`
-(free keys are fine — the system uses the IEX feed). Everything else has a
-sensible default; see `.env.example` for the full catalog.
-
-Key knobs:
-
-| Variable | Default | Meaning |
+| When | What happens | Verify |
 |---|---|---|
-| `TRADING_EXECUTION_MODE` | `alpaca_paper` | `schwab_live` routes to real money — leave on paper |
-| `TRADING_AUTO_APPROVE` | `0` | `1` = orders submit without dashboard approval (or pass `--auto-approve`) |
-| `TRADING_MAX_ORDERS_PER_TICK` | `1` | throttle new entries |
-| `TRADING_MAX_CONCURRENT_POSITIONS` | `3` | cap on simultaneous open positions |
-| `TRADING_RISK_PER_TRADE_PCT` | `0.01` | 1% of equity risked per trade (sizing = risk / (entry − stop)) |
-| `TRADING_REWARD_MULTIPLE` | `2.0` | profit target = entry + this × risk (2R by default) |
-| `TRADING_ENTRY_ORDER_TYPE` | `limit` | `limit` rests at the trigger so an unfilled entry can be cancelled; `market` fills instantly |
-| `TRADING_ENTRY_TIMEOUT_BARS` | `2` | cancel an unfilled entry after this many minutes resting (wall-clock; `0` disables the time box) |
-| `TRADING_ENTRY_INVALIDATE_PCT` | `0.0` | cancel an unfilled entry if the **live last-trade price** trades this fraction below the trigger (`0.0` = any break below entry; `0.005` = 0.5% below; negative disables) |
-| `TRADING_ENTRY_GUARD_INTERVAL_SECONDS` | `5` | how often the fast guard re-checks armed entries against the live price — lower = faster break detection, more quote calls |
-| `TRADING_MAX_DAILY_LOSS_PCT` | `0.03` | daily-loss circuit breaker |
-| `WATCHER_SYMBOLS` | `AAPL,TSLA,…` | static watchlist; `--discover` adds screener actives |
-| `WATCHER_PRICE_MIN/MAX` | `1` / `20` | candidate price band |
-| `WATCHER_EVENT_DB_PATH` | `./data/momentum.duckdb` | the event store |
-| `DASHBOARD_HOST/PORT` | `127.0.0.1` / `8010` | UI address |
+| ~04:00+ | premarket ingest + discovery begin | trigger board fills with gappers |
+| 09:30 | session gate opens; entries allowed 09:30–15:55 | first `TRIGGER FIRED` / auto-approved orders |
+| 09:35 | opening ranges complete → triggers arm | board states flip waiting→armed |
+| 15:55 | fresh entries barred | — |
+| 16:00± | EOD flatten of open positions | positions = 0 |
+| 20:00 Mon–Fri | nightly job: SIP re-fetch, event retention, log rotation, shadow tracks, float/dilution caches | `appended:` line in scheduler log |
 
----
-
-## 3. Start commands
-
-**Full live paper loop (recommended):**
+## Health checks
 
+**Compose (production):**
 ```bash
-python run_live_paper.py
-# dashboard:  http://127.0.0.1:8010
-# flags:      --once  --symbols SNDL,COSM  --auto-approve  --discover  --no-dashboard
+docker ps                      # all momentum containers Up, loop (healthy)
+docker logs -f <loop>          # discover/ingest/watch/execute cycle lines
+curl -s localhost:8765/api/snapshots | head -c 200    # dashboard serves
+# admin panel: http://<host>:8767 — services, DB, Alpaca, Ollama cards
 ```
 
-Each tick prints one status line:
+**Local (systemd):** `systemctl --user status 'momentum-*'` + the same curls.
 
+**Database truth (any shape):**
+```sql
+SELECT max(timestamp) FROM events;           -- heartbeat: < ~2 min during any hour
+SELECT count(*) FROM minute_bars;            -- history intact: ~19M+ (2026-07)
 ```
-[09:31:02] ingest: 240 minute rows | watch: evaluated=5 ready=['SNDL'] blocked=3 | sync: ok | execute: approvals_requested=['…'] auto_executed=0
-```
+Timestamps are **ET-naive**: compare against ET now, not UTC (a constant
+"4-hour stale" reading is a timezone artifact, not an outage).
 
-**Standalone read-only dashboard** (e.g. to review a past session):
+## Failure modes seen in production (and the fix)
 
-```bash
-python dashboard_api.py
-```
+| Symptom | Cause | Fix |
+|---|---|---|
+| Entries queue as "manual" in the dashboard | `TRADING_AUTO_APPROVE` missing from the loop's env | add `=1` to the loop service env; stale queue rows are zombies that clear with the session |
+| Orders submitted after hours | (fixed 2026-07-10) missing session gate | `TRADING_SESSION_ONLY=1` — default on; verify if reintroduced |
+| Duplicate resting orders after a loop crash/restart | armed/submitted state is in-memory | diff Alpaca open orders after any loop restart; cancel dupes (`get_orders` unscoped — `symbols=` can silently return empty) |
+| `PermissionError` on `/app/data/...` | host dir not owned by container uid | `chown -R 10001:10001 <data dir>` |
+| Postgres serving a tiny DB when history should exist | wrong pgdata mount (path typo/case, nested cluster, stale container) | `docker inspect <postgres> --format '{{range .Mounts}}...'`; verify by **size at destination**; recreate containers after volume edits (restart is not enough) |
+| Autovacuum cancelled every 60s on a big table | a frequent connector re-applying schema DDL | healthchecks/one-shot scripts must set `PG_SKIP_SCHEMA=1` |
+| Env edits in the YAML do nothing | TrueNAS expanded the anchors; `x-app` is decorative | put env in each **service's** block; verify with `docker exec <svc> sh -c 'echo $VAR'` |
+| Loop trading names it shouldn't | check gates in order: static open gap 10–35% (or ≥150% + rvol≥5), $2–20, day-vol, quality ≥0.60, VWAP, session window | `./momentum inspect criteria SYM` or the events `risk_rule_triggered` payloads |
 
-Approve/reject/exit buttons require the orchestrator (it attaches the
-execution service to the same server); the standalone runner returns 503 on
-actions by design.
+## Safety posture
 
-**Other tools:**
+- Paper account; bracket orders carry the protective stop; the exit manager
+  enforces stops on anything naked, ratchets breakeven at +5%, catastrophe-
+  stops at −10%/1.5×R.
+- **When in doubt, stop the loop container** — positions stay protected by
+  broker-side stop legs; a missed day costs nothing (the no-trade day is a
+  legitimate outcome). Never run two loops against one account.
+- Config experiments: use the admin panel's profiles; treat each switch as a
+  hypothesis and compare the day against `validated-core` behavior.
 
-```bash
-python fetch_minute_bars.py SNDL COSM --lookback 390   # ad-hoc ingestion
-python research_cli.py symbols --date 2026-06-11       # what's in the DB
-python research_cli.py gappers --date 2026-06-11       # gap scan → scanner_snapshots
-python backtest_cli.py --date 2026-06-10               # backtest a stored session
-python milestone3_verify.py                            # Schwab health walk
-python -m pytest                                       # full test suite
-```
+## Where things live
 
----
-
-## 4. The trading day, end to end
-
-1. **Ingest** — every 60 s, 1-minute IEX bars for the watchlist land in
-   `data/research/market.duckdb` with correct session flags. On boot, a
-   30-day daily backfill gives gap % and relative volume a baseline.
-2. **Watch** — every 30 s the watcher pulls in-band candidates, evaluates
-   nine weighted criteria (gap, RVOL, impulse, pullback, pullback volume,
-   VWAP, candle quality, breakout, data sufficiency) plus structure
-   detection, and drives `discovered → watching → ready | blocked | late`.
-   A `signal_ready` is emitted at most once per symbol per session.
-3. **Sync** — Alpaca account, positions, and orders snapshot into the event
-   store so projections and risk guards see broker truth.
-4. **Execute** — ready signals become orders sized at `risk_per_trade_pct`
-   of equity, with the stop from the setup and the target at
-   entry + `reward_multiple` × risk (2R by default). Manual mode parks them
-   in the dashboard queue; **auto mode (`--auto-approve`) submits
-   immediately, no approval.** Guards: max concurrent positions, no re-entry
-   into held symbols, one request per symbol per session.
-5. **Manage** — approve/reject from the dashboard; **Exit** sends a market
-   sell for the synced quantity.
-
-### The entry lifecycle (auto-arm → fill or back out)
-
-The earlier system would "see" a setup as something that had *already
-happened* rather than a trade about to happen, and never pull the trigger.
-The execution model now separates recognition from conviction from
-discipline:
-
-1. **Recognize** — when all the criteria pass, a `signal_ready` fires. The
-   setup is evaluated against the **last bar's timestamp**, not the wall
-   clock, so a live 9:40 breakout is judged against the 9:40 cutoff instead
-   of being dismissed as hours stale.
-2. **Arm with conviction** — in auto mode the order is submitted right away
-   as a **bracket** (entry + stop + target placed atomically). With
-   `TRADING_ENTRY_ORDER_TYPE=limit` (default) the entry rests at the trigger,
-   so "unfilled" is a real, observable state.
-3. **Back out at a defined point** — a fast **guard loop**
-   (`TRADING_ENTRY_GUARD_INTERVAL_SECONDS`, default 5 s) re-checks every
-   armed-but-unfilled entry and cancels it if either:
-   - the **live last-trade price** (pulled tick-by-tick from Alpaca's data
-     API, not a bar close) trades back below the trigger by
-     `TRADING_ENTRY_INVALIDATE_PCT`, or
-   - it has rested longer than `TRADING_ENTRY_TIMEOUT_BARS` minutes
-     (wall-clock — running the guard often does **not** shorten this window).
-   A cancel frees the risk budget and the concurrent-position slot, and emits
-   a `risk_rule_triggered` (`entry_backout`) you can see in the activity feed.
-   Once the order **fills**, it stops being tracked and the bracket's stop and
-   target manage the open position — no second-guessing.
-
-Tuning cheat-sheet: react to a fakeout faster → lower
-`TRADING_ENTRY_GUARD_INTERVAL_SECONDS` (e.g. `2`); more patience on the fill →
-raise `TRADING_ENTRY_TIMEOUT_BARS`; give the breakout room to wobble → set
-`TRADING_ENTRY_INVALIDATE_PCT` to something like `0.005`; chase fills instead
-of resting → `TRADING_ENTRY_ORDER_TYPE=market` (note: market fills can't time
-out); bigger winners → raise `TRADING_REWARD_MULTIPLE`.
-
-> **Quote rate limits.** The guard makes one latest-trade call per armed
-> symbol per pass (a ~2 s cache collapses duplicates). At the 5 s default with
-> a few open entries you're well inside Alpaca's free-tier 200 req/min. If you
-> drop the interval to 1 s with several armed entries, watch the budget —
-> bars ingestion and account sync also draw from it.
-
-### Reading the dashboard
-
-- **Badge** — blue `PAPER · ALPACA` vs red `LIVE · SCHWAB`. If it's red and
-  you didn't mean it, stop and check `TRADING_EXECUTION_MODE`.
-- **P&L strip** (top) — session realized / unrealized / total, win rate,
-  average R, open/closed counts. Green/red signed.
-- **Approval Queue** — pending entries with a **risk preview** ($ risk,
-  shares, notional, stop %) and Approve/Reject. Keyboard: `A` / `R` act on the
-  top row.
-- **Ready Signals / Open Positions** — each row has a live **sparkline** (last
-  ~40 one-minute closes) with dashed entry (blue) and stop (red) lines.
-- **Watch States** — click any row to expand the **nine-criteria breakdown**
-  (✓ pass / ✕ fail / · not-evaluated). The add-symbol box (or `/`) injects a
-  ticker onto the watchlist for the rest of the session.
-- **Activity Feed** — submissions and fills as they happen.
-- **Order Lifecycle** — per-order status from the event store.
-- **Connection dot** (top right) — green pulse = live SSE stream; it falls
-  back to 4 s polling automatically if the stream drops.
-- **Sound** — toggle the button or press `S` for an audio + toast alert when a
-  new signal goes ready. Press `?` for the shortcut list.
-
-Streams from `GET /api/stream` (SSE); also exposes `/api/snapshots`,
-`/api/criteria?symbol=`, `/api/bars?symbol=&minutes=`, and
-`POST /api/watch/add`. Charts require the research DB, which the orchestrator
-wires in automatically; the standalone `dashboard_api.py` has no bars source,
-so sparklines there show "no bars".
-
----
-
-## 5. Troubleshooting
-
-| Symptom | Likely cause / fix |
-|---|---|
-| `watch: evaluated=0` all day | No minute bars for today. Check keys; run `python research_cli.py symbols`. Outside market hours there are no new bars — backfill with `fetch_minute_bars.py --lookback 1440`. |
-| Candidates exist but never `ready` | Score < 60%. Watch the `Watch States` panel score column; criteria details are in `criteria_evaluated` events. |
-| `ingest: … 403` errors | Free keys are IEX-only — ensure `ALPACA_DATA_FEED=iex` (default) and that the request window ends ≥30 s in the past (handled automatically). |
-| Approve button does nothing | You're on `dashboard_api.py` (read-only). Use `run_live_paper.py`. |
-| Port 8010 busy | Set `DASHBOARD_PORT`, or `--no-dashboard`. |
-| Schwab shows `SCHWAB-UNAUTH` | Expected without OAuth. Run the flow in `schwab/auth/oauth2_flow_manager.py`; verify with `milestone3_verify.py`. |
-| Start fresh | Delete `data/momentum.duckdb` (events) and/or `data/research/` (bars). Both rebuild automatically. |
-
----
-
-## 6. Going live on Schwab (when ready)
-
-Paper-trade until the stats earn it. Then: complete OAuth
-(`milestone3_verify.py` all green), set `TRADING_EXECUTION_MODE=schwab_live`,
-keep `TRADING_AUTO_APPROVE=0`, and start with the smallest viable
-`TRADING_RISK_PER_TRADE_PCT`. The dashboard badge turns red in live mode.
+- **Everything queryable**: PostgreSQL (`DATABASE_URL`) — events, bars,
+  research, shadow tracks, caches.
+- **Local disk / app-data volume**: logs (rotated nightly), transcripts,
+  `learned_params.json`, `overrides.env` (admin panel), `profiles/`.
+- **Historical plans/era docs**: `docs/history/` — don't follow them.