momentum devandClaude Opus 4.8
007cd95c79
Snapshot before the DuckDB->PostgreSQL migration. data/, archive/, the 232MB rebuild zip, the Windows .venv, and .env are excluded via .gitignore. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
228 lines
11 KiB
Markdown
228 lines
11 KiB
Markdown
# RUNBOOK
|
||
|
||
Operational guide for running the momentum system. This file replaces the
|
||
original MILESTONE*.md documents, which were unrecoverable during the
|
||
rebuild.
|
||
|
||
---
|
||
|
||
## 1. Why live paper trading never worked (root causes, now fixed)
|
||
|
||
1. **Minute bars were never ingested.** The watcher reads 1-minute bars from
|
||
`data/research/market.duckdb`, but the old ingestion only wrote *daily*
|
||
bars for a handful of mega-caps. Fix: `research/ingestion/market_data.py`
|
||
ingests 1-minute Alpaca IEX bars continuously, and
|
||
`run_live_paper.py` schedules it every `LIVE_BARS_INTERVAL_SECONDS`.
|
||
|
||
2. **Session flags were hardcoded.** Every old row had
|
||
`is_regular_hours = TRUE`. Fix: flags are derived from the bar timestamp
|
||
in US/Eastern (premarket 04:00–09:30, regular 09:30–16:00, afterhours
|
||
16:00–20:00) — see `classify_session()` and its tests.
|
||
|
||
3. **The candidate filter excluded everything ingested.** The query had a
|
||
hardcoded `close BETWEEN 1 AND 20` while only mega-caps (price ≫ 20) were
|
||
in the database, so the watchlist was always empty. Fix: the band is
|
||
configurable (`WATCHER_PRICE_MIN` / `WATCHER_PRICE_MAX`) and the default
|
||
symbol list now actually gets ingested.
|
||
|
||
4. **No orchestrator.** Ingestion, watcher, broker sync, and execution were
|
||
never wired into one loop. Fix: `run_live_paper.py`.
|
||
|
||
5. **Broken source files.** `runtime/providers/registry.py`,
|
||
`schwab/market/models.py`, `option_chain_service.py`, and
|
||
`schwab/streaming/client.py` had syntax errors that crashed imports.
|
||
All repaired/rewritten.
|
||
|
||
6. **Schwab tokens were committed to git.** `data/schwab_tokens.json` is now
|
||
git-ignored and written `chmod 0600`. **Action required:** if that file
|
||
exists anywhere in your repo history, revoke the refresh token in the
|
||
Schwab developer portal and re-run the OAuth flow. Scrub history with
|
||
`git filter-repo --path data/schwab_tokens.json --invert-paths` if the
|
||
repo was ever shared.
|
||
|
||
---
|
||
|
||
## 2. Environment setup
|
||
|
||
```bash
|
||
pip install -r requirements.txt
|
||
cp .env.example .env
|
||
```
|
||
|
||
Minimum for paper trading: `ALPACA_API_KEY`, `ALPACA_SECRET_KEY`
|
||
(free keys are fine — the system uses the IEX feed). Everything else has a
|
||
sensible default; see `.env.example` for the full catalog.
|
||
|
||
Key knobs:
|
||
|
||
| Variable | Default | Meaning |
|
||
|---|---|---|
|
||
| `TRADING_EXECUTION_MODE` | `alpaca_paper` | `schwab_live` routes to real money — leave on paper |
|
||
| `TRADING_AUTO_APPROVE` | `0` | `1` = orders submit without dashboard approval (or pass `--auto-approve`) |
|
||
| `TRADING_MAX_ORDERS_PER_TICK` | `1` | throttle new entries |
|
||
| `TRADING_MAX_CONCURRENT_POSITIONS` | `3` | cap on simultaneous open positions |
|
||
| `TRADING_RISK_PER_TRADE_PCT` | `0.01` | 1% of equity risked per trade (sizing = risk / (entry − stop)) |
|
||
| `TRADING_REWARD_MULTIPLE` | `2.0` | profit target = entry + this × risk (2R by default) |
|
||
| `TRADING_ENTRY_ORDER_TYPE` | `limit` | `limit` rests at the trigger so an unfilled entry can be cancelled; `market` fills instantly |
|
||
| `TRADING_ENTRY_TIMEOUT_BARS` | `2` | cancel an unfilled entry after this many minutes resting (wall-clock; `0` disables the time box) |
|
||
| `TRADING_ENTRY_INVALIDATE_PCT` | `0.0` | cancel an unfilled entry if the **live last-trade price** trades this fraction below the trigger (`0.0` = any break below entry; `0.005` = 0.5% below; negative disables) |
|
||
| `TRADING_ENTRY_GUARD_INTERVAL_SECONDS` | `5` | how often the fast guard re-checks armed entries against the live price — lower = faster break detection, more quote calls |
|
||
| `TRADING_MAX_DAILY_LOSS_PCT` | `0.03` | daily-loss circuit breaker |
|
||
| `WATCHER_SYMBOLS` | `AAPL,TSLA,…` | static watchlist; `--discover` adds screener actives |
|
||
| `WATCHER_PRICE_MIN/MAX` | `1` / `20` | candidate price band |
|
||
| `WATCHER_EVENT_DB_PATH` | `./data/momentum.duckdb` | the event store |
|
||
| `DASHBOARD_HOST/PORT` | `127.0.0.1` / `8010` | UI address |
|
||
|
||
---
|
||
|
||
## 3. Start commands
|
||
|
||
**Full live paper loop (recommended):**
|
||
|
||
```bash
|
||
python run_live_paper.py
|
||
# dashboard: http://127.0.0.1:8010
|
||
# flags: --once --symbols SNDL,COSM --auto-approve --discover --no-dashboard
|
||
```
|
||
|
||
Each tick prints one status line:
|
||
|
||
```
|
||
[09:31:02] ingest: 240 minute rows | watch: evaluated=5 ready=['SNDL'] blocked=3 | sync: ok | execute: approvals_requested=['…'] auto_executed=0
|
||
```
|
||
|
||
**Standalone read-only dashboard** (e.g. to review a past session):
|
||
|
||
```bash
|
||
python dashboard_api.py
|
||
```
|
||
|
||
Approve/reject/exit buttons require the orchestrator (it attaches the
|
||
execution service to the same server); the standalone runner returns 503 on
|
||
actions by design.
|
||
|
||
**Other tools:**
|
||
|
||
```bash
|
||
python fetch_minute_bars.py SNDL COSM --lookback 390 # ad-hoc ingestion
|
||
python research_cli.py symbols --date 2026-06-11 # what's in the DB
|
||
python research_cli.py gappers --date 2026-06-11 # gap scan → scanner_snapshots
|
||
python backtest_cli.py --date 2026-06-10 # backtest a stored session
|
||
python milestone3_verify.py # Schwab health walk
|
||
python -m pytest # full test suite
|
||
```
|
||
|
||
---
|
||
|
||
## 4. The trading day, end to end
|
||
|
||
1. **Ingest** — every 60 s, 1-minute IEX bars for the watchlist land in
|
||
`data/research/market.duckdb` with correct session flags. On boot, a
|
||
30-day daily backfill gives gap % and relative volume a baseline.
|
||
2. **Watch** — every 30 s the watcher pulls in-band candidates, evaluates
|
||
nine weighted criteria (gap, RVOL, impulse, pullback, pullback volume,
|
||
VWAP, candle quality, breakout, data sufficiency) plus structure
|
||
detection, and drives `discovered → watching → ready | blocked | late`.
|
||
A `signal_ready` is emitted at most once per symbol per session.
|
||
3. **Sync** — Alpaca account, positions, and orders snapshot into the event
|
||
store so projections and risk guards see broker truth.
|
||
4. **Execute** — ready signals become orders sized at `risk_per_trade_pct`
|
||
of equity, with the stop from the setup and the target at
|
||
entry + `reward_multiple` × risk (2R by default). Manual mode parks them
|
||
in the dashboard queue; **auto mode (`--auto-approve`) submits
|
||
immediately, no approval.** Guards: max concurrent positions, no re-entry
|
||
into held symbols, one request per symbol per session.
|
||
5. **Manage** — approve/reject from the dashboard; **Exit** sends a market
|
||
sell for the synced quantity.
|
||
|
||
### The entry lifecycle (auto-arm → fill or back out)
|
||
|
||
The earlier system would "see" a setup as something that had *already
|
||
happened* rather than a trade about to happen, and never pull the trigger.
|
||
The execution model now separates recognition from conviction from
|
||
discipline:
|
||
|
||
1. **Recognize** — when all the criteria pass, a `signal_ready` fires. The
|
||
setup is evaluated against the **last bar's timestamp**, not the wall
|
||
clock, so a live 9:40 breakout is judged against the 9:40 cutoff instead
|
||
of being dismissed as hours stale.
|
||
2. **Arm with conviction** — in auto mode the order is submitted right away
|
||
as a **bracket** (entry + stop + target placed atomically). With
|
||
`TRADING_ENTRY_ORDER_TYPE=limit` (default) the entry rests at the trigger,
|
||
so "unfilled" is a real, observable state.
|
||
3. **Back out at a defined point** — a fast **guard loop**
|
||
(`TRADING_ENTRY_GUARD_INTERVAL_SECONDS`, default 5 s) re-checks every
|
||
armed-but-unfilled entry and cancels it if either:
|
||
- the **live last-trade price** (pulled tick-by-tick from Alpaca's data
|
||
API, not a bar close) trades back below the trigger by
|
||
`TRADING_ENTRY_INVALIDATE_PCT`, or
|
||
- it has rested longer than `TRADING_ENTRY_TIMEOUT_BARS` minutes
|
||
(wall-clock — running the guard often does **not** shorten this window).
|
||
A cancel frees the risk budget and the concurrent-position slot, and emits
|
||
a `risk_rule_triggered` (`entry_backout`) you can see in the activity feed.
|
||
Once the order **fills**, it stops being tracked and the bracket's stop and
|
||
target manage the open position — no second-guessing.
|
||
|
||
Tuning cheat-sheet: react to a fakeout faster → lower
|
||
`TRADING_ENTRY_GUARD_INTERVAL_SECONDS` (e.g. `2`); more patience on the fill →
|
||
raise `TRADING_ENTRY_TIMEOUT_BARS`; give the breakout room to wobble → set
|
||
`TRADING_ENTRY_INVALIDATE_PCT` to something like `0.005`; chase fills instead
|
||
of resting → `TRADING_ENTRY_ORDER_TYPE=market` (note: market fills can't time
|
||
out); bigger winners → raise `TRADING_REWARD_MULTIPLE`.
|
||
|
||
> **Quote rate limits.** The guard makes one latest-trade call per armed
|
||
> symbol per pass (a ~2 s cache collapses duplicates). At the 5 s default with
|
||
> a few open entries you're well inside Alpaca's free-tier 200 req/min. If you
|
||
> drop the interval to 1 s with several armed entries, watch the budget —
|
||
> bars ingestion and account sync also draw from it.
|
||
|
||
### Reading the dashboard
|
||
|
||
- **Badge** — blue `PAPER · ALPACA` vs red `LIVE · SCHWAB`. If it's red and
|
||
you didn't mean it, stop and check `TRADING_EXECUTION_MODE`.
|
||
- **P&L strip** (top) — session realized / unrealized / total, win rate,
|
||
average R, open/closed counts. Green/red signed.
|
||
- **Approval Queue** — pending entries with a **risk preview** ($ risk,
|
||
shares, notional, stop %) and Approve/Reject. Keyboard: `A` / `R` act on the
|
||
top row.
|
||
- **Ready Signals / Open Positions** — each row has a live **sparkline** (last
|
||
~40 one-minute closes) with dashed entry (blue) and stop (red) lines.
|
||
- **Watch States** — click any row to expand the **nine-criteria breakdown**
|
||
(✓ pass / ✕ fail / · not-evaluated). The add-symbol box (or `/`) injects a
|
||
ticker onto the watchlist for the rest of the session.
|
||
- **Activity Feed** — submissions and fills as they happen.
|
||
- **Order Lifecycle** — per-order status from the event store.
|
||
- **Connection dot** (top right) — green pulse = live SSE stream; it falls
|
||
back to 4 s polling automatically if the stream drops.
|
||
- **Sound** — toggle the button or press `S` for an audio + toast alert when a
|
||
new signal goes ready. Press `?` for the shortcut list.
|
||
|
||
Streams from `GET /api/stream` (SSE); also exposes `/api/snapshots`,
|
||
`/api/criteria?symbol=`, `/api/bars?symbol=&minutes=`, and
|
||
`POST /api/watch/add`. Charts require the research DB, which the orchestrator
|
||
wires in automatically; the standalone `dashboard_api.py` has no bars source,
|
||
so sparklines there show "no bars".
|
||
|
||
---
|
||
|
||
## 5. Troubleshooting
|
||
|
||
| Symptom | Likely cause / fix |
|
||
|---|---|
|
||
| `watch: evaluated=0` all day | No minute bars for today. Check keys; run `python research_cli.py symbols`. Outside market hours there are no new bars — backfill with `fetch_minute_bars.py --lookback 1440`. |
|
||
| Candidates exist but never `ready` | Score < 60%. Watch the `Watch States` panel score column; criteria details are in `criteria_evaluated` events. |
|
||
| `ingest: … 403` errors | Free keys are IEX-only — ensure `ALPACA_DATA_FEED=iex` (default) and that the request window ends ≥30 s in the past (handled automatically). |
|
||
| Approve button does nothing | You're on `dashboard_api.py` (read-only). Use `run_live_paper.py`. |
|
||
| Port 8010 busy | Set `DASHBOARD_PORT`, or `--no-dashboard`. |
|
||
| Schwab shows `SCHWAB-UNAUTH` | Expected without OAuth. Run the flow in `schwab/auth/oauth2_flow_manager.py`; verify with `milestone3_verify.py`. |
|
||
| Start fresh | Delete `data/momentum.duckdb` (events) and/or `data/research/` (bars). Both rebuild automatically. |
|
||
|
||
---
|
||
|
||
## 6. Going live on Schwab (when ready)
|
||
|
||
Paper-trade until the stats earn it. Then: complete OAuth
|
||
(`milestone3_verify.py` all green), set `TRADING_EXECUTION_MODE=schwab_live`,
|
||
keep `TRADING_AUTO_APPROVE=0`, and start with the smallest viable
|
||
`TRADING_RISK_PER_TRADE_PCT`. The dashboard badge turns red in live mode.
|