momentum-system/docs/history/RUNBOOK_pre2026-07.md

RUNBOOK

Operational guide for running the momentum system. This file replaces the original MILESTONE*.md documents, which were unrecoverable during the rebuild.

---

1. Why live paper trading never worked (root causes, now fixed)

1. Minute bars were never ingested. The watcher reads 1-minute bars from data/research/market.duckdb, but the old ingestion only wrote daily bars for a handful of mega-caps. Fix: research/ingestion/market_data.py ingests 1-minute Alpaca IEX bars continuously, and run_live_paper.py schedules it every LIVE_BARS_INTERVAL_SECONDS.

2. Session flags were hardcoded. Every old row had is_regular_hours = TRUE. Fix: flags are derived from the bar timestamp in US/Eastern (premarket 04:00–09:30, regular 09:30–16:00, afterhours 16:00–20:00) — see classify_session() and its tests.

3. The candidate filter excluded everything ingested. The query had a hardcoded close BETWEEN 1 AND 20 while only mega-caps (price ≫ 20) were in the database, so the watchlist was always empty. Fix: the band is configurable (WATCHER_PRICE_MIN / WATCHER_PRICE_MAX) and the default symbol list now actually gets ingested.

4. No orchestrator. Ingestion, watcher, broker sync, and execution were never wired into one loop. Fix: run_live_paper.py.

5. Broken source files. runtime/providers/registry.py, schwab/market/models.py, option_chain_service.py, and schwab/streaming/client.py had syntax errors that crashed imports. All repaired/rewritten.

6. Schwab tokens were committed to git. data/schwab_tokens.json is now git-ignored and written chmod 0600. Action required: if that file exists anywhere in your repo history, revoke the refresh token in the Schwab developer portal and re-run the OAuth flow. Scrub history with git filter-repo --path data/schwab_tokens.json --invert-paths if the repo was ever shared.

---

2. Environment setup

pip install -r requirements.txt
cp .env.example .env

Minimum for paper trading: ALPACA_API_KEY, ALPACA_SECRET_KEY (free keys are fine — the system uses the IEX feed). Everything else has a sensible default; see .env.example for the full catalog.

Key knobs:

Variable	Default	Meaning
TRADING_EXECUTION_MODE	alpaca_paper	schwab_live routes to real money — leave on paper
TRADING_AUTO_APPROVE	0	1 = orders submit without dashboard approval (or pass --auto-approve)
TRADING_MAX_ORDERS_PER_TICK	1	throttle new entries
TRADING_MAX_CONCURRENT_POSITIONS	3	cap on simultaneous open positions
TRADING_RISK_PER_TRADE_PCT	0.01	1% of equity risked per trade (sizing = risk / (entry − stop))
TRADING_REWARD_MULTIPLE	2.0	profit target = entry + this × risk (2R by default)
TRADING_ENTRY_ORDER_TYPE	limit	limit rests at the trigger so an unfilled entry can be cancelled; market fills instantly
TRADING_ENTRY_TIMEOUT_BARS	2	cancel an unfilled entry after this many minutes resting (wall-clock; 0 disables the time box)
TRADING_ENTRY_INVALIDATE_PCT	0.0	cancel an unfilled entry if the live last-trade price trades this fraction below the trigger (0.0 = any break below entry; 0.005 = 0.5% below; negative disables)
TRADING_ENTRY_GUARD_INTERVAL_SECONDS	5	how often the fast guard re-checks armed entries against the live price — lower = faster break detection, more quote calls
TRADING_MAX_DAILY_LOSS_PCT	0.03	daily-loss circuit breaker
WATCHER_SYMBOLS	AAPL,TSLA,…	static watchlist; --discover adds screener actives
WATCHER_PRICE_MIN/MAX	1 / 20	candidate price band
WATCHER_EVENT_DB_PATH	./data/momentum.duckdb	the event store
DASHBOARD_HOST/PORT	127.0.0.1 / 8010	UI address

---

3. Start commands

Full live paper loop (recommended):

python run_live_paper.py
# dashboard:  http://127.0.0.1:8010
# flags:      --once  --symbols SNDL,COSM  --auto-approve  --discover  --no-dashboard

Each tick prints one status line:

[09:31:02] ingest: 240 minute rows | watch: evaluated=5 ready=['SNDL'] blocked=3 | sync: ok | execute: approvals_requested=['…'] auto_executed=0

Standalone read-only dashboard (e.g. to review a past session):

python dashboard_api.py

Approve/reject/exit buttons require the orchestrator (it attaches the execution service to the same server); the standalone runner returns 503 on actions by design.

Other tools:

python fetch_minute_bars.py SNDL COSM --lookback 390   # ad-hoc ingestion
python research_cli.py symbols --date 2026-06-11       # what's in the DB
python research_cli.py gappers --date 2026-06-11       # gap scan → scanner_snapshots
python backtest_cli.py --date 2026-06-10               # backtest a stored session
python milestone3_verify.py                            # Schwab health walk
python -m pytest                                       # full test suite

---

4. The trading day, end to end

1. Ingest — every 60 s, 1-minute IEX bars for the watchlist land in data/research/market.duckdb with correct session flags. On boot, a 30-day daily backfill gives gap % and relative volume a baseline.
2. Watch — every 30 s the watcher pulls in-band candidates, evaluates nine weighted criteria (gap, RVOL, impulse, pullback, pullback volume, VWAP, candle quality, breakout, data sufficiency) plus structure detection, and drives discovered → watching → ready | blocked | late. A signal_ready is emitted at most once per symbol per session.
3. Sync — Alpaca account, positions, and orders snapshot into the event store so projections and risk guards see broker truth.
4. Execute — ready signals become orders sized at risk_per_trade_pct of equity, with the stop from the setup and the target at entry + reward_multiple × risk (2R by default). Manual mode parks them in the dashboard queue; auto mode (--auto-approve) submits immediately, no approval. Guards: max concurrent positions, no re-entry into held symbols, one request per symbol per session.
5. Manage — approve/reject from the dashboard; Exit sends a market sell for the synced quantity.

The entry lifecycle (auto-arm → fill or back out)

The earlier system would "see" a setup as something that had already happened rather than a trade about to happen, and never pull the trigger. The execution model now separates recognition from conviction from discipline:

1. Recognize — when all the criteria pass, a signal_ready fires. The setup is evaluated against the last bar's timestamp, not the wall clock, so a live 9:40 breakout is judged against the 9:40 cutoff instead of being dismissed as hours stale.
2. Arm with conviction — in auto mode the order is submitted right away as a bracket (entry + stop + target placed atomically). With TRADING_ENTRY_ORDER_TYPE=limit (default) the entry rests at the trigger, so "unfilled" is a real, observable state.
3. Back out at a defined point — a fast guard loop (TRADING_ENTRY_GUARD_INTERVAL_SECONDS, default 5 s) re-checks every armed-but-unfilled entry and cancels it if either:
   • the live last-trade price (pulled tick-by-tick from Alpaca's data API, not a bar close) trades back below the trigger by TRADING_ENTRY_INVALIDATE_PCT, or
   • it has rested longer than TRADING_ENTRY_TIMEOUT_BARS minutes (wall-clock — running the guard often does not shorten this window). A cancel frees the risk budget and the concurrent-position slot, and emits a risk_rule_triggered (entry_backout) you can see in the activity feed. Once the order fills, it stops being tracked and the bracket's stop and target manage the open position — no second-guessing.

Tuning cheat-sheet: react to a fakeout faster → lower TRADING_ENTRY_GUARD_INTERVAL_SECONDS (e.g. 2); more patience on the fill → raise TRADING_ENTRY_TIMEOUT_BARS; give the breakout room to wobble → set TRADING_ENTRY_INVALIDATE_PCT to something like 0.005; chase fills instead of resting → TRADING_ENTRY_ORDER_TYPE=market (note: market fills can't time out); bigger winners → raise TRADING_REWARD_MULTIPLE.

Quote rate limits. The guard makes one latest-trade call per armed symbol per pass (a ~2 s cache collapses duplicates). At the 5 s default with a few open entries you're well inside Alpaca's free-tier 200 req/min. If you drop the interval to 1 s with several armed entries, watch the budget — bars ingestion and account sync also draw from it.

Reading the dashboard

• Badge — blue PAPER · ALPACA vs red LIVE · SCHWAB. If it's red and you didn't mean it, stop and check TRADING_EXECUTION_MODE.
• P&L strip (top) — session realized / unrealized / total, win rate, average R, open/closed counts. Green/red signed.
• Approval Queue — pending entries with a risk preview ($ risk, shares, notional, stop %) and Approve/Reject. Keyboard: A / R act on the top row.
• Ready Signals / Open Positions — each row has a live sparkline (last ~40 one-minute closes) with dashed entry (blue) and stop (red) lines.
• Watch States — click any row to expand the nine-criteria breakdown (✓ pass / ✕ fail / · not-evaluated). The add-symbol box (or /) injects a ticker onto the watchlist for the rest of the session.
• Activity Feed — submissions and fills as they happen.
• Order Lifecycle — per-order status from the event store.
• Connection dot (top right) — green pulse = live SSE stream; it falls back to 4 s polling automatically if the stream drops.
• Sound — toggle the button or press S for an audio + toast alert when a new signal goes ready. Press ? for the shortcut list.

Streams from GET /api/stream (SSE); also exposes /api/snapshots, /api/criteria?symbol=, /api/bars?symbol=&minutes=, and POST /api/watch/add. Charts require the research DB, which the orchestrator wires in automatically; the standalone dashboard_api.py has no bars source, so sparklines there show "no bars".

---

5. Troubleshooting

Symptom	Likely cause / fix
watch: evaluated=0 all day	No minute bars for today. Check keys; run python research_cli.py symbols. Outside market hours there are no new bars — backfill with fetch_minute_bars.py --lookback 1440.
Candidates exist but never ready	Score < 60%. Watch the Watch States panel score column; criteria details are in criteria_evaluated events.
ingest: … 403 errors	Free keys are IEX-only — ensure ALPACA_DATA_FEED=iex (default) and that the request window ends ≥30 s in the past (handled automatically).
Approve button does nothing	You're on dashboard_api.py (read-only). Use run_live_paper.py.
Port 8010 busy	Set DASHBOARD_PORT, or --no-dashboard.
Schwab shows SCHWAB-UNAUTH	Expected without OAuth. Run the flow in schwab/auth/oauth2_flow_manager.py; verify with milestone3_verify.py.
Start fresh	Delete data/momentum.duckdb (events) and/or data/research/ (bars). Both rebuild automatically.

---

6. Going live on Schwab (when ready)

Paper-trade until the stats earn it. Then: complete OAuth (milestone3_verify.py all green), set TRADING_EXECUTION_MODE=schwab_live, keep TRADING_AUTO_APPROVE=0, and start with the smallest viable TRADING_RISK_PER_TRADE_PCT. The dashboard badge turns red in live mode.