Docs cleanup: current-state rewrite + three deployment modes + history/
- 10 era documents (phase plans, build summaries, redesign writeups, old runbooks, completed consolidation plan) moved to docs/history/ with a do-not-follow README. - Root README: current validated population ($2-20, 10-35% static gap + squeeze exception, SIP), Deployment section, automated-maintenance reality, corrected repo map (admin panel, youtube_claims, deploy/). - RUNBOOK rewritten for the running system: daily rhythm, health checks per deployment, and the production failure-mode table earned this week (env drift, pgdata mount traps, uid 10001, zombie approvals, PG_SKIP_ SCHEMA, timezone artifacts). - deploy/README.md is now the three-mode index: compose (canonical), systemd (local dev — unit templates now committed), helm (refresh staged as issue #8). Old k8s doc moved to deploy/helm/README.md. - docker/README.md: six TrueNAS gotchas, each from a real outage. - Admin panel improvements staged as GitHub issues #4-#7. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
momentum devandClaude Opus 4.8 committed
1 parent
d4c695e213
commit
5ac2cc7b89
23 files changed
+680
-396
No files matched your search
@@ -1,15 +1,17 @@
|
|||||||
# Momentum
|
# Momentum
|
||||||
|
|
||||||
Event-sourced small-cap **momentum day-trading** system (Ross Cameron–style
|
Event-sourced small-cap **momentum day-trading** system (Ross Cameron–style
|
||||||
gap-and-go / opening-range breakout). It screens the day's most-active gappers
|
gap-and-go / opening-range breakout). It screens the day's top overnight
|
||||||
($1–5, gapping ≥3% on ≥2× relative volume), arms an opening-range-breakout
|
gappers — the year-scale-validated population: **$2–20, static open gap
|
||||||
trigger the moment the range forms, fires a bracket order on the live cross,
|
10–35%, real volume**, plus a volume-verified **>150% squeeze exception** —
|
||||||
and manages the exit with a trailing stop + profit-lock ladder. Execution runs
|
arms an opening-range-breakout trigger the moment the range forms, fires a
|
||||||
on an **Alpaca paper** account; everything is recorded in a **PostgreSQL**
|
bracket order on the live cross (regular session only), and manages the exit
|
||||||
append-only event store and replayable.
|
with a breakeven ratchet + trailing profit-lock ladder. Execution runs on an
|
||||||
|
**Alpaca paper** account over the **SIP consolidated feed**; everything is
|
||||||
|
recorded in a **PostgreSQL** append-only event store and replayable.
|
||||||
|
|
||||||
```
|
```
|
||||||
Alpaca IEX bars ─► research DBs (minute+daily) ─► screen gappers ─► ArmedTriggerBook
|
Alpaca SIP bars ─► research DBs (minute+daily) ─► screen gappers ─► ArmedTriggerBook
|
||||||
(1-min, live) │ live price
|
(1-min, live) │ live price
|
||||||
▼ crosses trigger
|
▼ crosses trigger
|
||||||
PostgreSQL event store ◄────────────────────── TradingExecutionService ─► Alpaca paper
|
PostgreSQL event store ◄────────────────────── TradingExecutionService ─► Alpaca paper
|
||||||
@@ -36,6 +38,15 @@ python run_live_paper.py # full loop (+ embedded dashboard)
|
|||||||
|
|
||||||
No keys? `python run_live_paper.py --once` runs a single dry pass to verify wiring.
|
No keys? `python run_live_paper.py --once` runs a single dry pass to verify wiring.
|
||||||
|
|
||||||
|
## Deployment
|
||||||
|
|
||||||
|
Three supported shapes — **Docker Compose is canonical** (production runs the
|
||||||
|
full stack self-contained on TrueNAS SCALE: postgres, loop, dashboard :8765,
|
||||||
|
admin panel :8767, telegram bot, transcripts, nightly scheduler). Local
|
||||||
|
systemd units remain the development shape; a Helm refresh is planned.
|
||||||
|
**See [deploy/README.md](deploy/README.md)** for all three, including the
|
||||||
|
behavior flags every deployment must set explicitly.
|
||||||
|
|
||||||
## Commands
|
## Commands
|
||||||
|
|
||||||
Everything is driven from the repo root. The `./momentum` wrapper runs the
|
Everything is driven from the repo root. The `./momentum` wrapper runs the
|
||||||
@@ -111,8 +122,11 @@ them at session end.
|
|||||||
All behavior is environment-driven — see **`.env.example`** for every key with
|
All behavior is environment-driven — see **`.env.example`** for every key with
|
||||||
the *why*. The knobs that matter most:
|
the *why*. The knobs that matter most:
|
||||||
|
|
||||||
- **Entry:** `WATCHER_PRICE_MIN/MAX` (1–5), `TRIGGER_GAP_MIN/MAX`, `TRIGGER_RVOL_MIN`,
|
- **Entry population** (year-scale validated 2026-07): `TRADING_PRICE_MIN/MAX`
|
||||||
`TRIGGER_MIN_DOLLAR_VOL` (absolute liquidity floor — skip thin spikes).
|
(2–20), `TRADING_MIN_ENTRY_GAP_PCT` (0.10, static open gap), `TRIGGER_GAP_MAX`
|
||||||
|
(35 — the 35-150% band tested dead), `TRIGGER_GAP_SQUEEZE_MIN/RVOL` (150/5 —
|
||||||
|
re-admits real squeezes), `TRADING_MIN_DAY_VOLUME`, `TRADING_SESSION_ONLY`
|
||||||
|
(bars after-hours entries).
|
||||||
- **Sizing:** `TRADING_RISK_PER_TRADE_PCT` (1%), `TRADING_MAX_POSITION_PCT`,
|
- **Sizing:** `TRADING_RISK_PER_TRADE_PCT` (1%), `TRADING_MAX_POSITION_PCT`,
|
||||||
`TRADING_LIQUIDITY_MAX_VOLUME_PCT` (size down thin names).
|
`TRADING_LIQUIDITY_MAX_VOLUME_PCT` (size down thin names).
|
||||||
- **Exits:** `TRADING_EXIT_TRAIL_MODE=prior_low`, `TRADING_EXIT_TRAIL_AFTER_R`,
|
- **Exits:** `TRADING_EXIT_TRAIL_MODE=prior_low`, `TRADING_EXIT_TRAIL_AFTER_R`,
|
||||||
@@ -124,8 +138,9 @@ the *why*. The knobs that matter most:
|
|||||||
|
|
||||||
## Database & performance
|
## Database & performance
|
||||||
|
|
||||||
The single datastore is **PostgreSQL** (`DATABASE_URL`); research bars live in
|
The single datastore is **PostgreSQL** (`DATABASE_URL`) — bars, events,
|
||||||
DuckDB under `data/research/`. The event store is one append-only `events`
|
research tables, everything (the DuckDB era is over; `data/` holds only logs,
|
||||||
|
transcripts, and boot-time state). The event store is one append-only `events`
|
||||||
table that every projection reads.
|
table that every projection reads.
|
||||||
|
|
||||||
**Hot-path index.** Projections overwhelmingly run
|
**Hot-path index.** Projections overwhelmingly run
|
||||||
@@ -149,10 +164,10 @@ For ongoing tuning:
|
|||||||
+ time partitioning + compression). Not required at current scale, and must be
|
+ time partitioning + compression). Not required at current scale, and must be
|
||||||
installed on the server.
|
installed on the server.
|
||||||
|
|
||||||
**Maintenance.** ~80% of `events` rows are high-volume diagnostics
|
**Maintenance is automated.** The nightly job (scheduler service / systemd
|
||||||
(`criteria_evaluated`, `signal_blocked`, `module_tick`). A retention/prune of old
|
timer) prunes diagnostic events older than 14 days, rotates logs, refreshes the
|
||||||
diagnostic events keeps the table small and every query fast. Run `ANALYZE events`
|
day's bars to SIP truth, and updates the float/dilution shadow caches.
|
||||||
after large ingests so the planner has fresh stats.
|
Autovacuum is tuned per-table; run `ANALYZE` manually only after bulk imports.
|
||||||
|
|
||||||
## Repository map
|
## Repository map
|
||||||
|
|
||||||
@@ -166,8 +181,13 @@ storage/ event schema + store (PostgreSQL), projections
|
|||||||
research/ research DBs, ingestion (bars, RSS, gappers), screeners
|
research/ research DBs, ingestion (bars, RSS, gappers), screeners
|
||||||
alpaca_paper/ Alpaca paper client, executor, account/order sync
|
alpaca_paper/ Alpaca paper client, executor, account/order sync
|
||||||
schwab/ Schwab OAuth, market data, positions/orders, health
|
schwab/ Schwab OAuth, market data, positions/orders, health
|
||||||
api/ + dashboard_api.py dashboard JSON API + static UI
|
api/ + dashboard_api.py dashboard JSON API + static UI (:8765)
|
||||||
|
admin_api.py admin panel (:8767): health, config overrides, profiles, restarts
|
||||||
|
youtube_claims/ trader-video transcript -> claim extraction pipeline
|
||||||
|
controlroom/ NiceGUI control room (disabled by owner preference)
|
||||||
|
deploy/ compose (canonical) / systemd (local) / helm (planned)
|
||||||
eod_replay.py end-of-day whole-market replay
|
eod_replay.py end-of-day whole-market replay
|
||||||
|
docs/history/ point-in-time plans from earlier eras — don't follow them
|
||||||
tests/ pytest suite (unit + integration)
|
tests/ pytest suite (unit + integration)
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|||||||
+49
-209
@@ -1,227 +1,67 @@
|
|||||||
# RUNBOOK
|
# RUNBOOK
|
||||||
|
|
||||||
Operational guide for running the momentum system. This file replaces the
|
Operating the Momentum stack day to day. Deployment shapes and setup live in
|
||||||
original MILESTONE*.md documents, which were unrecoverable during the
|
[deploy/README.md](deploy/README.md); this is "it's running — now what."
|
||||||
rebuild.
|
|
||||||
|
|
||||||
---
|
## Daily rhythm (all times ET)
|
||||||
|
|
||||||
## 1. Why live paper trading never worked (root causes, now fixed)
|
| When | What happens | Verify |
|
||||||
|
|
||||||
1. **Minute bars were never ingested.** The watcher reads 1-minute bars from
|
|
||||||
`data/research/market.duckdb`, but the old ingestion only wrote *daily*
|
|
||||||
bars for a handful of mega-caps. Fix: `research/ingestion/market_data.py`
|
|
||||||
ingests 1-minute Alpaca IEX bars continuously, and
|
|
||||||
`run_live_paper.py` schedules it every `LIVE_BARS_INTERVAL_SECONDS`.
|
|
||||||
|
|
||||||
2. **Session flags were hardcoded.** Every old row had
|
|
||||||
`is_regular_hours = TRUE`. Fix: flags are derived from the bar timestamp
|
|
||||||
in US/Eastern (premarket 04:00–09:30, regular 09:30–16:00, afterhours
|
|
||||||
16:00–20:00) — see `classify_session()` and its tests.
|
|
||||||
|
|
||||||
3. **The candidate filter excluded everything ingested.** The query had a
|
|
||||||
hardcoded `close BETWEEN 1 AND 20` while only mega-caps (price ≫ 20) were
|
|
||||||
in the database, so the watchlist was always empty. Fix: the band is
|
|
||||||
configurable (`WATCHER_PRICE_MIN` / `WATCHER_PRICE_MAX`) and the default
|
|
||||||
symbol list now actually gets ingested.
|
|
||||||
|
|
||||||
4. **No orchestrator.** Ingestion, watcher, broker sync, and execution were
|
|
||||||
never wired into one loop. Fix: `run_live_paper.py`.
|
|
||||||
|
|
||||||
5. **Broken source files.** `runtime/providers/registry.py`,
|
|
||||||
`schwab/market/models.py`, `option_chain_service.py`, and
|
|
||||||
`schwab/streaming/client.py` had syntax errors that crashed imports.
|
|
||||||
All repaired/rewritten.
|
|
||||||
|
|
||||||
6. **Schwab tokens were committed to git.** `data/schwab_tokens.json` is now
|
|
||||||
git-ignored and written `chmod 0600`. **Action required:** if that file
|
|
||||||
exists anywhere in your repo history, revoke the refresh token in the
|
|
||||||
Schwab developer portal and re-run the OAuth flow. Scrub history with
|
|
||||||
`git filter-repo --path data/schwab_tokens.json --invert-paths` if the
|
|
||||||
repo was ever shared.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 2. Environment setup
|
|
||||||
|
|
||||||
```bash
|
|
||||||
pip install -r requirements.txt
|
|
||||||
cp .env.example .env
|
|
||||||
```
|
|
||||||
|
|
||||||
Minimum for paper trading: `ALPACA_API_KEY`, `ALPACA_SECRET_KEY`
|
|
||||||
(free keys are fine — the system uses the IEX feed). Everything else has a
|
|
||||||
sensible default; see `.env.example` for the full catalog.
|
|
||||||
|
|
||||||
Key knobs:
|
|
||||||
|
|
||||||
| Variable | Default | Meaning |
|
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `TRADING_EXECUTION_MODE` | `alpaca_paper` | `schwab_live` routes to real money — leave on paper |
|
| ~04:00+ | premarket ingest + discovery begin | trigger board fills with gappers |
|
||||||
| `TRADING_AUTO_APPROVE` | `0` | `1` = orders submit without dashboard approval (or pass `--auto-approve`) |
|
| 09:30 | session gate opens; entries allowed 09:30–15:55 | first `TRIGGER FIRED` / auto-approved orders |
|
||||||
| `TRADING_MAX_ORDERS_PER_TICK` | `1` | throttle new entries |
|
| 09:35 | opening ranges complete → triggers arm | board states flip waiting→armed |
|
||||||
| `TRADING_MAX_CONCURRENT_POSITIONS` | `3` | cap on simultaneous open positions |
|
| 15:55 | fresh entries barred | — |
|
||||||
| `TRADING_RISK_PER_TRADE_PCT` | `0.01` | 1% of equity risked per trade (sizing = risk / (entry − stop)) |
|
| 16:00± | EOD flatten of open positions | positions = 0 |
|
||||||
| `TRADING_REWARD_MULTIPLE` | `2.0` | profit target = entry + this × risk (2R by default) |
|
| 20:00 Mon–Fri | nightly job: SIP re-fetch, event retention, log rotation, shadow tracks, float/dilution caches | `appended:` line in scheduler log |
|
||||||
| `TRADING_ENTRY_ORDER_TYPE` | `limit` | `limit` rests at the trigger so an unfilled entry can be cancelled; `market` fills instantly |
|
|
||||||
| `TRADING_ENTRY_TIMEOUT_BARS` | `2` | cancel an unfilled entry after this many minutes resting (wall-clock; `0` disables the time box) |
|
|
||||||
| `TRADING_ENTRY_INVALIDATE_PCT` | `0.0` | cancel an unfilled entry if the **live last-trade price** trades this fraction below the trigger (`0.0` = any break below entry; `0.005` = 0.5% below; negative disables) |
|
|
||||||
| `TRADING_ENTRY_GUARD_INTERVAL_SECONDS` | `5` | how often the fast guard re-checks armed entries against the live price — lower = faster break detection, more quote calls |
|
|
||||||
| `TRADING_MAX_DAILY_LOSS_PCT` | `0.03` | daily-loss circuit breaker |
|
|
||||||
| `WATCHER_SYMBOLS` | `AAPL,TSLA,…` | static watchlist; `--discover` adds screener actives |
|
|
||||||
| `WATCHER_PRICE_MIN/MAX` | `1` / `20` | candidate price band |
|
|
||||||
| `WATCHER_EVENT_DB_PATH` | `./data/momentum.duckdb` | the event store |
|
|
||||||
| `DASHBOARD_HOST/PORT` | `127.0.0.1` / `8010` | UI address |
|
|
||||||
|
|
||||||
---
|
## Health checks
|
||||||
|
|
||||||
## 3. Start commands
|
|
||||||
|
|
||||||
**Full live paper loop (recommended):**
|
|
||||||
|
|
||||||
|
**Compose (production):**
|
||||||
```bash
|
```bash
|
||||||
python run_live_paper.py
|
docker ps # all momentum containers Up, loop (healthy)
|
||||||
# dashboard: http://127.0.0.1:8010
|
docker logs -f <loop> # discover/ingest/watch/execute cycle lines
|
||||||
# flags: --once --symbols SNDL,COSM --auto-approve --discover --no-dashboard
|
curl -s localhost:8765/api/snapshots | head -c 200 # dashboard serves
|
||||||
|
# admin panel: http://<host>:8767 — services, DB, Alpaca, Ollama cards
|
||||||
```
|
```
|
||||||
|
|
||||||
Each tick prints one status line:
|
**Local (systemd):** `systemctl --user status 'momentum-*'` + the same curls.
|
||||||
|
|
||||||
|
**Database truth (any shape):**
|
||||||
|
```sql
|
||||||
|
SELECT max(timestamp) FROM events; -- heartbeat: < ~2 min during any hour
|
||||||
|
SELECT count(*) FROM minute_bars; -- history intact: ~19M+ (2026-07)
|
||||||
```
|
```
|
||||||
[09:31:02] ingest: 240 minute rows | watch: evaluated=5 ready=['SNDL'] blocked=3 | sync: ok | execute: approvals_requested=['…'] auto_executed=0
|
Timestamps are **ET-naive**: compare against ET now, not UTC (a constant
|
||||||
```
|
"4-hour stale" reading is a timezone artifact, not an outage).
|
||||||
|
|
||||||
**Standalone read-only dashboard** (e.g. to review a past session):
|
## Failure modes seen in production (and the fix)
|
||||||
|
|
||||||
```bash
|
| Symptom | Cause | Fix |
|
||||||
python dashboard_api.py
|
|---|---|---|
|
||||||
```
|
| Entries queue as "manual" in the dashboard | `TRADING_AUTO_APPROVE` missing from the loop's env | add `=1` to the loop service env; stale queue rows are zombies that clear with the session |
|
||||||
|
| Orders submitted after hours | (fixed 2026-07-10) missing session gate | `TRADING_SESSION_ONLY=1` — default on; verify if reintroduced |
|
||||||
|
| Duplicate resting orders after a loop crash/restart | armed/submitted state is in-memory | diff Alpaca open orders after any loop restart; cancel dupes (`get_orders` unscoped — `symbols=` can silently return empty) |
|
||||||
|
| `PermissionError` on `/app/data/...` | host dir not owned by container uid | `chown -R 10001:10001 <data dir>` |
|
||||||
|
| Postgres serving a tiny DB when history should exist | wrong pgdata mount (path typo/case, nested cluster, stale container) | `docker inspect <postgres> --format '{{range .Mounts}}...'`; verify by **size at destination**; recreate containers after volume edits (restart is not enough) |
|
||||||
|
| Autovacuum cancelled every 60s on a big table | a frequent connector re-applying schema DDL | healthchecks/one-shot scripts must set `PG_SKIP_SCHEMA=1` |
|
||||||
|
| Env edits in the YAML do nothing | TrueNAS expanded the anchors; `x-app` is decorative | put env in each **service's** block; verify with `docker exec <svc> sh -c 'echo $VAR'` |
|
||||||
|
| Loop trading names it shouldn't | check gates in order: static open gap 10–35% (or ≥150% + rvol≥5), $2–20, day-vol, quality ≥0.60, VWAP, session window | `./momentum inspect criteria SYM` or the events `risk_rule_triggered` payloads |
|
||||||
|
|
||||||
Approve/reject/exit buttons require the orchestrator (it attaches the
|
## Safety posture
|
||||||
execution service to the same server); the standalone runner returns 503 on
|
|
||||||
actions by design.
|
|
||||||
|
|
||||||
**Other tools:**
|
- Paper account; bracket orders carry the protective stop; the exit manager
|
||||||
|
enforces stops on anything naked, ratchets breakeven at +5%, catastrophe-
|
||||||
|
stops at −10%/1.5×R.
|
||||||
|
- **When in doubt, stop the loop container** — positions stay protected by
|
||||||
|
broker-side stop legs; a missed day costs nothing (the no-trade day is a
|
||||||
|
legitimate outcome). Never run two loops against one account.
|
||||||
|
- Config experiments: use the admin panel's profiles; treat each switch as a
|
||||||
|
hypothesis and compare the day against `validated-core` behavior.
|
||||||
|
|
||||||
```bash
|
## Where things live
|
||||||
python fetch_minute_bars.py SNDL COSM --lookback 390 # ad-hoc ingestion
|
|
||||||
python research_cli.py symbols --date 2026-06-11 # what's in the DB
|
|
||||||
python research_cli.py gappers --date 2026-06-11 # gap scan → scanner_snapshots
|
|
||||||
python backtest_cli.py --date 2026-06-10 # backtest a stored session
|
|
||||||
python milestone3_verify.py # Schwab health walk
|
|
||||||
python -m pytest # full test suite
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
- **Everything queryable**: PostgreSQL (`DATABASE_URL`) — events, bars,
|
||||||
|
research, shadow tracks, caches.
|
||||||
## 4. The trading day, end to end
|
- **Local disk / app-data volume**: logs (rotated nightly), transcripts,
|
||||||
|
`learned_params.json`, `overrides.env` (admin panel), `profiles/`.
|
||||||
1. **Ingest** — every 60 s, 1-minute IEX bars for the watchlist land in
|
- **Historical plans/era docs**: `docs/history/` — don't follow them.
|
||||||
`data/research/market.duckdb` with correct session flags. On boot, a
|
|
||||||
30-day daily backfill gives gap % and relative volume a baseline.
|
|
||||||
2. **Watch** — every 30 s the watcher pulls in-band candidates, evaluates
|
|
||||||
nine weighted criteria (gap, RVOL, impulse, pullback, pullback volume,
|
|
||||||
VWAP, candle quality, breakout, data sufficiency) plus structure
|
|
||||||
detection, and drives `discovered → watching → ready | blocked | late`.
|
|
||||||
A `signal_ready` is emitted at most once per symbol per session.
|
|
||||||
3. **Sync** — Alpaca account, positions, and orders snapshot into the event
|
|
||||||
store so projections and risk guards see broker truth.
|
|
||||||
4. **Execute** — ready signals become orders sized at `risk_per_trade_pct`
|
|
||||||
of equity, with the stop from the setup and the target at
|
|
||||||
entry + `reward_multiple` × risk (2R by default). Manual mode parks them
|
|
||||||
in the dashboard queue; **auto mode (`--auto-approve`) submits
|
|
||||||
immediately, no approval.** Guards: max concurrent positions, no re-entry
|
|
||||||
into held symbols, one request per symbol per session.
|
|
||||||
5. **Manage** — approve/reject from the dashboard; **Exit** sends a market
|
|
||||||
sell for the synced quantity.
|
|
||||||
|
|
||||||
### The entry lifecycle (auto-arm → fill or back out)
|
|
||||||
|
|
||||||
The earlier system would "see" a setup as something that had *already
|
|
||||||
happened* rather than a trade about to happen, and never pull the trigger.
|
|
||||||
The execution model now separates recognition from conviction from
|
|
||||||
discipline:
|
|
||||||
|
|
||||||
1. **Recognize** — when all the criteria pass, a `signal_ready` fires. The
|
|
||||||
setup is evaluated against the **last bar's timestamp**, not the wall
|
|
||||||
clock, so a live 9:40 breakout is judged against the 9:40 cutoff instead
|
|
||||||
of being dismissed as hours stale.
|
|
||||||
2. **Arm with conviction** — in auto mode the order is submitted right away
|
|
||||||
as a **bracket** (entry + stop + target placed atomically). With
|
|
||||||
`TRADING_ENTRY_ORDER_TYPE=limit` (default) the entry rests at the trigger,
|
|
||||||
so "unfilled" is a real, observable state.
|
|
||||||
3. **Back out at a defined point** — a fast **guard loop**
|
|
||||||
(`TRADING_ENTRY_GUARD_INTERVAL_SECONDS`, default 5 s) re-checks every
|
|
||||||
armed-but-unfilled entry and cancels it if either:
|
|
||||||
- the **live last-trade price** (pulled tick-by-tick from Alpaca's data
|
|
||||||
API, not a bar close) trades back below the trigger by
|
|
||||||
`TRADING_ENTRY_INVALIDATE_PCT`, or
|
|
||||||
- it has rested longer than `TRADING_ENTRY_TIMEOUT_BARS` minutes
|
|
||||||
(wall-clock — running the guard often does **not** shorten this window).
|
|
||||||
A cancel frees the risk budget and the concurrent-position slot, and emits
|
|
||||||
a `risk_rule_triggered` (`entry_backout`) you can see in the activity feed.
|
|
||||||
Once the order **fills**, it stops being tracked and the bracket's stop and
|
|
||||||
target manage the open position — no second-guessing.
|
|
||||||
|
|
||||||
Tuning cheat-sheet: react to a fakeout faster → lower
|
|
||||||
`TRADING_ENTRY_GUARD_INTERVAL_SECONDS` (e.g. `2`); more patience on the fill →
|
|
||||||
raise `TRADING_ENTRY_TIMEOUT_BARS`; give the breakout room to wobble → set
|
|
||||||
`TRADING_ENTRY_INVALIDATE_PCT` to something like `0.005`; chase fills instead
|
|
||||||
of resting → `TRADING_ENTRY_ORDER_TYPE=market` (note: market fills can't time
|
|
||||||
out); bigger winners → raise `TRADING_REWARD_MULTIPLE`.
|
|
||||||
|
|
||||||
> **Quote rate limits.** The guard makes one latest-trade call per armed
|
|
||||||
> symbol per pass (a ~2 s cache collapses duplicates). At the 5 s default with
|
|
||||||
> a few open entries you're well inside Alpaca's free-tier 200 req/min. If you
|
|
||||||
> drop the interval to 1 s with several armed entries, watch the budget —
|
|
||||||
> bars ingestion and account sync also draw from it.
|
|
||||||
|
|
||||||
### Reading the dashboard
|
|
||||||
|
|
||||||
- **Badge** — blue `PAPER · ALPACA` vs red `LIVE · SCHWAB`. If it's red and
|
|
||||||
you didn't mean it, stop and check `TRADING_EXECUTION_MODE`.
|
|
||||||
- **P&L strip** (top) — session realized / unrealized / total, win rate,
|
|
||||||
average R, open/closed counts. Green/red signed.
|
|
||||||
- **Approval Queue** — pending entries with a **risk preview** ($ risk,
|
|
||||||
shares, notional, stop %) and Approve/Reject. Keyboard: `A` / `R` act on the
|
|
||||||
top row.
|
|
||||||
- **Ready Signals / Open Positions** — each row has a live **sparkline** (last
|
|
||||||
~40 one-minute closes) with dashed entry (blue) and stop (red) lines.
|
|
||||||
- **Watch States** — click any row to expand the **nine-criteria breakdown**
|
|
||||||
(✓ pass / ✕ fail / · not-evaluated). The add-symbol box (or `/`) injects a
|
|
||||||
ticker onto the watchlist for the rest of the session.
|
|
||||||
- **Activity Feed** — submissions and fills as they happen.
|
|
||||||
- **Order Lifecycle** — per-order status from the event store.
|
|
||||||
- **Connection dot** (top right) — green pulse = live SSE stream; it falls
|
|
||||||
back to 4 s polling automatically if the stream drops.
|
|
||||||
- **Sound** — toggle the button or press `S` for an audio + toast alert when a
|
|
||||||
new signal goes ready. Press `?` for the shortcut list.
|
|
||||||
|
|
||||||
Streams from `GET /api/stream` (SSE); also exposes `/api/snapshots`,
|
|
||||||
`/api/criteria?symbol=`, `/api/bars?symbol=&minutes=`, and
|
|
||||||
`POST /api/watch/add`. Charts require the research DB, which the orchestrator
|
|
||||||
wires in automatically; the standalone `dashboard_api.py` has no bars source,
|
|
||||||
so sparklines there show "no bars".
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 5. Troubleshooting
|
|
||||||
|
|
||||||
| Symptom | Likely cause / fix |
|
|
||||||
|---|---|
|
|
||||||
| `watch: evaluated=0` all day | No minute bars for today. Check keys; run `python research_cli.py symbols`. Outside market hours there are no new bars — backfill with `fetch_minute_bars.py --lookback 1440`. |
|
|
||||||
| Candidates exist but never `ready` | Score < 60%. Watch the `Watch States` panel score column; criteria details are in `criteria_evaluated` events. |
|
|
||||||
| `ingest: … 403` errors | Free keys are IEX-only — ensure `ALPACA_DATA_FEED=iex` (default) and that the request window ends ≥30 s in the past (handled automatically). |
|
|
||||||
| Approve button does nothing | You're on `dashboard_api.py` (read-only). Use `run_live_paper.py`. |
|
|
||||||
| Port 8010 busy | Set `DASHBOARD_PORT`, or `--no-dashboard`. |
|
|
||||||
| Schwab shows `SCHWAB-UNAUTH` | Expected without OAuth. Run the flow in `schwab/auth/oauth2_flow_manager.py`; verify with `milestone3_verify.py`. |
|
|
||||||
| Start fresh | Delete `data/momentum.duckdb` (events) and/or `data/research/` (bars). Both rebuild automatically. |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 6. Going live on Schwab (when ready)
|
|
||||||
|
|
||||||
Paper-trade until the stats earn it. Then: complete OAuth
|
|
||||||
(`milestone3_verify.py` all green), set `TRADING_EXECUTION_MODE=schwab_live`,
|
|
||||||
keep `TRADING_AUTO_APPROVE=0`, and start with the smallest viable
|
|
||||||
`TRADING_RISK_PER_TRADE_PCT`. The dashboard badge turns red in live mode.
|
|
||||||
+37
-171
@@ -1,182 +1,48 @@
|
|||||||
# Kubernetes deployment
|
# Deploying Momentum
|
||||||
|
|
||||||
A container image and Helm chart that run the whole Momentum System — including
|
Three supported shapes, one image/codebase. Pick by where it runs:
|
||||||
its **PostgreSQL datastore** — inside a single Kubernetes namespace.
|
|
||||||
|
|
||||||
## What gets deployed
|
| Mode | Status | Use when | Docs |
|
||||||
|
|---|---|---|---|
|
||||||
|
| **Docker Compose** | ✅ **canonical** (production since 2026-07-09) | Any always-on Linux host; TrueNAS SCALE | [docker/README.md](docker/README.md), `docker-compose.yml` (root), [docker/truenas-compose.yml](docker/truenas-compose.yml) |
|
||||||
|
| **Local (systemd)** | ✅ maintained | Development machine, WSL, bare metal without Docker | [systemd/README.md](systemd/README.md) + unit templates in `deploy/systemd/` |
|
||||||
|
| **Helm / Kubernetes** | 🚧 planned refresh | Future k8s hosting | [helm/](helm/) — the chart predates the 2026-07 architecture; starting point, not deployable |
|
||||||
|
|
||||||
| Workload | Kind | Notes |
|
## Docker Compose (canonical)
|
||||||
| --- | --- | --- |
|
|
||||||
| PostgreSQL | StatefulSet + PVC | Single source of truth. Schema auto-creates on the app's first connection from `storage/pg_schema.sql`. |
|
|
||||||
| App loop + dashboard | Deployment (`replicas: 1`, `Recreate`) | `run_live_paper.py` with the **embedded** dashboard on :8010. Stateful singleton — never scale up. |
|
|
||||||
| journal / eod-replay / nightly-tune | CronJobs | US/Eastern (`spec.timeZone`), weekdays. Share the app's `data/` PVC. |
|
|
||||||
| Grafana | Deployment + Service | Optional (`grafana.enabled`). Datasource auto-points at the in-cluster Postgres. |
|
|
||||||
|
|
||||||
The app loop and the CronJobs run the **same image** (built from the repo
|
The root `docker-compose.yml` runs the full stack self-contained: postgres 18,
|
||||||
`Dockerfile`), just with different commands.
|
trading loop, dashboard (:8765), admin panel (:8767), telegram bot, transcripts
|
||||||
|
pipeline, nightly scheduler, on-demand jobs, optional Grafana. The image is
|
||||||
|
published to `docker.io/fez5587/momentum-app` and `ghcr.io/fez5587/momentum-app`
|
||||||
|
by CI on every master push.
|
||||||
|
|
||||||
## Local development with Docker Compose
|
For TrueNAS SCALE use `docker/truenas-compose.yml` (pull-only, dataset host
|
||||||
|
paths, inline env). **Read the gotchas in docker/README.md before editing the
|
||||||
|
YAML** — anchor expansion, per-service env, and path case-sensitivity have each
|
||||||
|
caused an outage.
|
||||||
|
|
||||||
For a single-machine stack (same components, no Kubernetes) use the root
|
## Local (systemd user services)
|
||||||
`docker-compose.yml`. It builds the app from the `Dockerfile`, brings up
|
|
||||||
PostgreSQL with a persistent volume, and runs the loop with its embedded
|
|
||||||
dashboard. Config comes from a local `.env` (optional — without `ALPACA_*` keys
|
|
||||||
the loop runs in dry mode; the dashboard and DB still work). `DATABASE_URL` is
|
|
||||||
overridden to the in-network `postgres` service, so the stack is self-contained.
|
|
||||||
|
|
||||||
```bash
|
The original deployment: unit templates in `deploy/systemd/` running a git
|
||||||
cp .env.example .env # optional: add ALPACA_* keys for live data
|
checkout + venv, config from `.env`. Still the best loop for development —
|
||||||
|
tests, backtests, and research scripts run here. Keep the units **disabled**
|
||||||
|
while the compose stack owns production: two loops against one Alpaca account
|
||||||
|
create duplicate orders.
|
||||||
|
|
||||||
docker compose up -d --build # postgres + app loop + dashboard (:8010)
|
## Helm (future)
|
||||||
docker compose logs -f app # watch the loop
|
|
||||||
open http://localhost:8010 # dashboard
|
|
||||||
|
|
||||||
docker compose --profile monitoring up -d # also start Grafana (:3000)
|
`deploy/helm/` holds the original chart (single app deployment + CronJobs). It
|
||||||
```
|
predates the service split, the session gate, and the env-flag inventory; the
|
||||||
|
refresh is tracked in GitHub issues. Until then, compose is the way.
|
||||||
|
|
||||||
The scheduled jobs sit under the `jobs` profile so they don't auto-start; run
|
## Config in all modes
|
||||||
them on demand (or from host cron / a scheduler on the ET times in `.env`):
|
|
||||||
|
|
||||||
```bash
|
- `.env.example` (committed, sanitized) is the full variable inventory and
|
||||||
docker compose run --rm nightly-tune
|
says which layer each belongs in.
|
||||||
docker compose run --rm eod-replay
|
- Behavior flags that MUST be set explicitly in any deployment — silent
|
||||||
docker compose run --rm journal
|
defaults bite: `TRADING_AUTO_APPROVE=1` (else entries queue for manual
|
||||||
```
|
approval), `ALPACA_DATA_FEED=sip`, `OLLAMA_ENABLED=true` (catalyst engine),
|
||||||
|
`FMP_API_KEY` (float cache). `TRADING_SESSION_ONLY` defaults ON in
|
||||||
Postgres credentials default to `momentum`/`momentum`/`momentum`; override with
|
deployed configs (bars after-hours entries).
|
||||||
`POSTGRES_USER` / `POSTGRES_PASSWORD` / `POSTGRES_DB` in `.env`. Named volumes
|
- Runtime overrides: the admin panel writes `data/overrides.env`, which every
|
||||||
`pg-data`, `app-data`, and `grafana-data` persist across restarts (the jobs share
|
service loads over its baked env at boot (`runtime/env_overrides.py`).
|
||||||
`app-data` with the app so `nightly_tune.py`'s `learned_params.json` reaches the
|
|
||||||
loop). The Compose Grafana uses its own datasource override at
|
|
||||||
`deploy/compose/grafana/datasources/momentum.yaml` (pointed at the `postgres`
|
|
||||||
service); the legacy `grafana/docker-compose.yml` is unrelated and left for
|
|
||||||
Grafana-only use against an external DB.
|
|
||||||
|
|
||||||
## Build & publish the image
|
|
||||||
|
|
||||||
Pushes to the working branch / `main` / `v*` tags trigger
|
|
||||||
`.github/workflows/docker-publish.yml`, which builds and pushes to
|
|
||||||
`ghcr.io/fez5587/momentum-system`. Point the chart at a pushed tag via
|
|
||||||
`image.tag` (a commit SHA, or `latest` on the default branch).
|
|
||||||
|
|
||||||
Local build:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
docker build -t momentum:dev .
|
|
||||||
```
|
|
||||||
|
|
||||||
## Install
|
|
||||||
|
|
||||||
```bash
|
|
||||||
helm install momentum ./deploy/helm/momentum \
|
|
||||||
-n momentum --create-namespace \
|
|
||||||
--set image.tag=latest \
|
|
||||||
--set secrets.alpacaApiKey=YOUR_KEY \
|
|
||||||
--set secrets.alpacaSecretKey=YOUR_SECRET
|
|
||||||
```
|
|
||||||
|
|
||||||
Without Alpaca keys the loop runs in **dry mode** (no live data/orders); the
|
|
||||||
dashboard and database still work.
|
|
||||||
|
|
||||||
### Secrets
|
|
||||||
|
|
||||||
The chart builds `DATABASE_URL` from the in-cluster Postgres service and a
|
|
||||||
password that is generated on first install (or set `postgres.password` to pin
|
|
||||||
it). Broker keys come from `secrets.*`.
|
|
||||||
|
|
||||||
For real keys, prefer an existing Secret over `values.yaml`:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
kubectl -n momentum create secret generic momentum-broker \
|
|
||||||
--from-literal=DATABASE_URL=... \
|
|
||||||
--from-literal=ALPACA_API_KEY=... --from-literal=ALPACA_SECRET_KEY=... \
|
|
||||||
--from-literal=APCA_API_KEY_ID=... --from-literal=APCA_API_SECRET_KEY=...
|
|
||||||
helm install momentum ./deploy/helm/momentum -n momentum \
|
|
||||||
--set secrets.existingSecret=momentum-broker
|
|
||||||
```
|
|
||||||
|
|
||||||
When `secrets.existingSecret` is set the chart renders no Secret of its own and
|
|
||||||
expects that one to carry `DATABASE_URL`, the `ALPACA_*`/`APCA_*` keys, and
|
|
||||||
`POSTGRES_PASSWORD`.
|
|
||||||
|
|
||||||
## Reach the dashboard
|
|
||||||
|
|
||||||
```bash
|
|
||||||
kubectl -n momentum port-forward svc/momentum-app 8010:8010
|
|
||||||
open http://localhost:8010
|
|
||||||
```
|
|
||||||
|
|
||||||
Set `ingress.enabled=true` (with `ingress.host`/`className`) to expose it
|
|
||||||
instead.
|
|
||||||
|
|
||||||
## Verify
|
|
||||||
|
|
||||||
```bash
|
|
||||||
helm lint deploy/helm/momentum
|
|
||||||
helm template momentum deploy/helm/momentum | less
|
|
||||||
|
|
||||||
# After install:
|
|
||||||
kubectl -n momentum get pods
|
|
||||||
kubectl -n momentum exec sts/momentum-postgres -- psql -U momentum -d momentum -c '\dt'
|
|
||||||
kubectl -n momentum create job --from=cronjob/momentum-nightly-tune tune-test
|
|
||||||
```
|
|
||||||
|
|
||||||
## Notes & constraints
|
|
||||||
|
|
||||||
- **Singleton loop.** The trading loop holds in-memory state and one broker
|
|
||||||
session; `replicas` must stay `1` and the strategy is `Recreate`.
|
|
||||||
- **Shared `data/` PVC is ReadWriteOnce.** The CronJob pods mount the same PVC
|
|
||||||
as the app (so `nightly_tune.py` can write `learned_params.json` for the loop
|
|
||||||
to read), which schedules them onto the app pod's node. For multi-node
|
|
||||||
clusters, use an RWX StorageClass or split the volumes.
|
|
||||||
- **CronJob `spec.timeZone`** requires Kubernetes ≥ 1.27.
|
|
||||||
- The legacy `grafana/docker-compose.yml` is left in place for local use; the
|
|
||||||
chart provisions its own Grafana against the in-cluster Postgres.
|
|
||||||
|
|
||||||
## Future hardening (roadmap)
|
|
||||||
|
|
||||||
The package deploys and runs, but the items below separate "it comes up" from
|
|
||||||
"runs reliably for a trading system." They are **not implemented yet** — captured
|
|
||||||
here so they can be picked up later. Each notes what, why, and the rough approach.
|
|
||||||
|
|
||||||
### Reliability
|
|
||||||
|
|
||||||
- **Loop-heartbeat liveness** *(needs an app-code change).* Today `/api/health`
|
|
||||||
(`api/main.py:343`) returns a static `{"ok": true}`, which only proves the
|
|
||||||
dashboard *thread* is alive. The trading loop runs in the main thread
|
|
||||||
(`run_live_paper.py:328`), so a **crashed** loop exits the process and Kubernetes
|
|
||||||
restarts it (fine), but a **hung** loop (stuck network call, deadlock) keeps
|
|
||||||
answering health checks `200` and runs undetected. Approach: have the loop stamp a
|
|
||||||
last-tick timestamp into `DashboardState` each iteration, make `/api/health` (or a
|
|
||||||
new `/healthz`) return `503` when the last tick is older than a threshold, and
|
|
||||||
point the Deployment's `livenessProbe` at it. Highest-value item.
|
|
||||||
- **Postgres backup CronJob.** The event store is the source of truth and nothing
|
|
||||||
backs it up. Approach: a chart CronJob running `pg_dump` (postgres image, creds
|
|
||||||
from the Secret) to the app `data` PVC with N-day retention; optionally offload to
|
|
||||||
object storage.
|
|
||||||
- **Schema bootstrap hook.** The schema auto-applies on the app's first connection,
|
|
||||||
so a first-run race between the app and a CronJob is possible. Approach: a
|
|
||||||
`pre-install`/`pre-upgrade` Helm-hook Job that applies `storage/pg_schema.sql`
|
|
||||||
deterministically, with `PG_SKIP_SCHEMA=1` set on the app and CronJobs.
|
|
||||||
|
|
||||||
### Security
|
|
||||||
|
|
||||||
- **Pod hardening.** Add a `securityContext` across workloads: `runAsNonRoot`, drop
|
|
||||||
ALL capabilities, `seccompProfile: RuntimeDefault`, and `readOnlyRootFilesystem`
|
|
||||||
where feasible (the app needs `/app/data` writable — already a mounted volume).
|
|
||||||
- **NetworkPolicy.** Default-deny in the namespace; allow only the app, CronJobs, and
|
|
||||||
Grafana to reach Postgres:5432, and restrict who can reach the dashboard.
|
|
||||||
- **Grafana auth.** The chart inherits anonymous-admin from the original compose
|
|
||||||
(`templates/grafana.yaml` env). Make it off by default in-cluster, with a real
|
|
||||||
admin password sourced from the Secret (`GF_SECURITY_ADMIN_PASSWORD`).
|
|
||||||
|
|
||||||
### CI / operational
|
|
||||||
|
|
||||||
- **Chart CI validation.** Add a CI job running `helm lint` + `helm template` +
|
|
||||||
`kubeconform` on every push — catches template/schema errors automatically (the
|
|
||||||
chart could not be rendered in the sandbox where it was authored, since image-blob
|
|
||||||
CDNs were egress-blocked).
|
|
||||||
- **Image digest pinning.** For production installs, pin `image` by SHA digest rather
|
|
||||||
than the mutable `latest` tag.
|
|
||||||
- **PodDisruptionBudget / singleton guard.** A PDB for the singleton loop plus a
|
|
||||||
guard so it is never scaled above `1` or evicted to zero unexpectedly.
|
|
||||||
@@ -94,3 +94,24 @@ NOTE (2026-07-09): the live `.env` contains two commented-out PowerShell lines
|
|||||||
embedding the real Alpaca credentials — harmless there, but they nearly leaked
|
embedding the real Alpaca credentials — harmless there, but they nearly leaked
|
||||||
into the committed `.env.example` (caught by the regeneration leak check).
|
into the committed `.env.example` (caught by the regeneration leak check).
|
||||||
Consider deleting them from `.env`.
|
Consider deleting them from `.env`.
|
||||||
|
|
||||||
|
## TrueNAS gotchas (each one caused a real outage — read before editing)
|
||||||
|
|
||||||
|
1. **Anchors get expanded.** TrueNAS renders `x-app:`/`<<: *app` into inline
|
||||||
|
per-service blocks. After the first save, the `x-app` block is decorative —
|
||||||
|
env/volume edits there reach nothing. Edit each service's own block.
|
||||||
|
2. **Verify env landed** with `docker exec <svc> sh -c 'echo $VAR'` — not by
|
||||||
|
reading the YAML, and not via the admin panel (it reports its own container).
|
||||||
|
3. **Paths are case-sensitive** and Docker auto-creates missing bind sources:
|
||||||
|
`/mnt/Volume` vs `/mnt/volume` silently forked the database into two worlds.
|
||||||
|
After any data move, verify by **size at destination** (`du -sh`), never `ls`.
|
||||||
|
4. **Volume changes need container recreation** — Stop/Start or Update the app;
|
||||||
|
a container restart keeps the old mounts (and a running postgres keeps
|
||||||
|
writing to *deleted* files if you move its dir underneath it).
|
||||||
|
5. **First deploy initializes postgres** — `POSTGRES_PASSWORD` only applies to
|
||||||
|
an empty pgdata. A failed first attempt leaves a cluster with the old
|
||||||
|
password; fix with `ALTER USER` via `docker exec ... psql -U momentum` or
|
||||||
|
wipe before the real deploy.
|
||||||
|
6. **uid 10001**: the app containers run non-root; `chown -R 10001:10001` the
|
||||||
|
host data dir or every file-writer (scheduler TSVs, admin profiles,
|
||||||
|
transcripts) hits `PermissionError`.
|
||||||
@@ -0,0 +1,182 @@
|
|||||||
|
# Kubernetes deployment
|
||||||
|
|
||||||
|
A container image and Helm chart that run the whole Momentum System — including
|
||||||
|
its **PostgreSQL datastore** — inside a single Kubernetes namespace.
|
||||||
|
|
||||||
|
## What gets deployed
|
||||||
|
|
||||||
|
| Workload | Kind | Notes |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| PostgreSQL | StatefulSet + PVC | Single source of truth. Schema auto-creates on the app's first connection from `storage/pg_schema.sql`. |
|
||||||
|
| App loop + dashboard | Deployment (`replicas: 1`, `Recreate`) | `run_live_paper.py` with the **embedded** dashboard on :8010. Stateful singleton — never scale up. |
|
||||||
|
| journal / eod-replay / nightly-tune | CronJobs | US/Eastern (`spec.timeZone`), weekdays. Share the app's `data/` PVC. |
|
||||||
|
| Grafana | Deployment + Service | Optional (`grafana.enabled`). Datasource auto-points at the in-cluster Postgres. |
|
||||||
|
|
||||||
|
The app loop and the CronJobs run the **same image** (built from the repo
|
||||||
|
`Dockerfile`), just with different commands.
|
||||||
|
|
||||||
|
## Local development with Docker Compose
|
||||||
|
|
||||||
|
For a single-machine stack (same components, no Kubernetes) use the root
|
||||||
|
`docker-compose.yml`. It builds the app from the `Dockerfile`, brings up
|
||||||
|
PostgreSQL with a persistent volume, and runs the loop with its embedded
|
||||||
|
dashboard. Config comes from a local `.env` (optional — without `ALPACA_*` keys
|
||||||
|
the loop runs in dry mode; the dashboard and DB still work). `DATABASE_URL` is
|
||||||
|
overridden to the in-network `postgres` service, so the stack is self-contained.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cp .env.example .env # optional: add ALPACA_* keys for live data
|
||||||
|
|
||||||
|
docker compose up -d --build # postgres + app loop + dashboard (:8010)
|
||||||
|
docker compose logs -f app # watch the loop
|
||||||
|
open http://localhost:8010 # dashboard
|
||||||
|
|
||||||
|
docker compose --profile monitoring up -d # also start Grafana (:3000)
|
||||||
|
```
|
||||||
|
|
||||||
|
The scheduled jobs sit under the `jobs` profile so they don't auto-start; run
|
||||||
|
them on demand (or from host cron / a scheduler on the ET times in `.env`):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker compose run --rm nightly-tune
|
||||||
|
docker compose run --rm eod-replay
|
||||||
|
docker compose run --rm journal
|
||||||
|
```
|
||||||
|
|
||||||
|
Postgres credentials default to `momentum`/`momentum`/`momentum`; override with
|
||||||
|
`POSTGRES_USER` / `POSTGRES_PASSWORD` / `POSTGRES_DB` in `.env`. Named volumes
|
||||||
|
`pg-data`, `app-data`, and `grafana-data` persist across restarts (the jobs share
|
||||||
|
`app-data` with the app so `nightly_tune.py`'s `learned_params.json` reaches the
|
||||||
|
loop). The Compose Grafana uses its own datasource override at
|
||||||
|
`deploy/compose/grafana/datasources/momentum.yaml` (pointed at the `postgres`
|
||||||
|
service); the legacy `grafana/docker-compose.yml` is unrelated and left for
|
||||||
|
Grafana-only use against an external DB.
|
||||||
|
|
||||||
|
## Build & publish the image
|
||||||
|
|
||||||
|
Pushes to the working branch / `main` / `v*` tags trigger
|
||||||
|
`.github/workflows/docker-publish.yml`, which builds and pushes to
|
||||||
|
`ghcr.io/fez5587/momentum-system`. Point the chart at a pushed tag via
|
||||||
|
`image.tag` (a commit SHA, or `latest` on the default branch).
|
||||||
|
|
||||||
|
Local build:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker build -t momentum:dev .
|
||||||
|
```
|
||||||
|
|
||||||
|
## Install
|
||||||
|
|
||||||
|
```bash
|
||||||
|
helm install momentum ./deploy/helm/momentum \
|
||||||
|
-n momentum --create-namespace \
|
||||||
|
--set image.tag=latest \
|
||||||
|
--set secrets.alpacaApiKey=YOUR_KEY \
|
||||||
|
--set secrets.alpacaSecretKey=YOUR_SECRET
|
||||||
|
```
|
||||||
|
|
||||||
|
Without Alpaca keys the loop runs in **dry mode** (no live data/orders); the
|
||||||
|
dashboard and database still work.
|
||||||
|
|
||||||
|
### Secrets
|
||||||
|
|
||||||
|
The chart builds `DATABASE_URL` from the in-cluster Postgres service and a
|
||||||
|
password that is generated on first install (or set `postgres.password` to pin
|
||||||
|
it). Broker keys come from `secrets.*`.
|
||||||
|
|
||||||
|
For real keys, prefer an existing Secret over `values.yaml`:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl -n momentum create secret generic momentum-broker \
|
||||||
|
--from-literal=DATABASE_URL=... \
|
||||||
|
--from-literal=ALPACA_API_KEY=... --from-literal=ALPACA_SECRET_KEY=... \
|
||||||
|
--from-literal=APCA_API_KEY_ID=... --from-literal=APCA_API_SECRET_KEY=...
|
||||||
|
helm install momentum ./deploy/helm/momentum -n momentum \
|
||||||
|
--set secrets.existingSecret=momentum-broker
|
||||||
|
```
|
||||||
|
|
||||||
|
When `secrets.existingSecret` is set the chart renders no Secret of its own and
|
||||||
|
expects that one to carry `DATABASE_URL`, the `ALPACA_*`/`APCA_*` keys, and
|
||||||
|
`POSTGRES_PASSWORD`.
|
||||||
|
|
||||||
|
## Reach the dashboard
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl -n momentum port-forward svc/momentum-app 8010:8010
|
||||||
|
open http://localhost:8010
|
||||||
|
```
|
||||||
|
|
||||||
|
Set `ingress.enabled=true` (with `ingress.host`/`className`) to expose it
|
||||||
|
instead.
|
||||||
|
|
||||||
|
## Verify
|
||||||
|
|
||||||
|
```bash
|
||||||
|
helm lint deploy/helm/momentum
|
||||||
|
helm template momentum deploy/helm/momentum | less
|
||||||
|
|
||||||
|
# After install:
|
||||||
|
kubectl -n momentum get pods
|
||||||
|
kubectl -n momentum exec sts/momentum-postgres -- psql -U momentum -d momentum -c '\dt'
|
||||||
|
kubectl -n momentum create job --from=cronjob/momentum-nightly-tune tune-test
|
||||||
|
```
|
||||||
|
|
||||||
|
## Notes & constraints
|
||||||
|
|
||||||
|
- **Singleton loop.** The trading loop holds in-memory state and one broker
|
||||||
|
session; `replicas` must stay `1` and the strategy is `Recreate`.
|
||||||
|
- **Shared `data/` PVC is ReadWriteOnce.** The CronJob pods mount the same PVC
|
||||||
|
as the app (so `nightly_tune.py` can write `learned_params.json` for the loop
|
||||||
|
to read), which schedules them onto the app pod's node. For multi-node
|
||||||
|
clusters, use an RWX StorageClass or split the volumes.
|
||||||
|
- **CronJob `spec.timeZone`** requires Kubernetes ≥ 1.27.
|
||||||
|
- The legacy `grafana/docker-compose.yml` is left in place for local use; the
|
||||||
|
chart provisions its own Grafana against the in-cluster Postgres.
|
||||||
|
|
||||||
|
## Future hardening (roadmap)
|
||||||
|
|
||||||
|
The package deploys and runs, but the items below separate "it comes up" from
|
||||||
|
"runs reliably for a trading system." They are **not implemented yet** — captured
|
||||||
|
here so they can be picked up later. Each notes what, why, and the rough approach.
|
||||||
|
|
||||||
|
### Reliability
|
||||||
|
|
||||||
|
- **Loop-heartbeat liveness** *(needs an app-code change).* Today `/api/health`
|
||||||
|
(`api/main.py:343`) returns a static `{"ok": true}`, which only proves the
|
||||||
|
dashboard *thread* is alive. The trading loop runs in the main thread
|
||||||
|
(`run_live_paper.py:328`), so a **crashed** loop exits the process and Kubernetes
|
||||||
|
restarts it (fine), but a **hung** loop (stuck network call, deadlock) keeps
|
||||||
|
answering health checks `200` and runs undetected. Approach: have the loop stamp a
|
||||||
|
last-tick timestamp into `DashboardState` each iteration, make `/api/health` (or a
|
||||||
|
new `/healthz`) return `503` when the last tick is older than a threshold, and
|
||||||
|
point the Deployment's `livenessProbe` at it. Highest-value item.
|
||||||
|
- **Postgres backup CronJob.** The event store is the source of truth and nothing
|
||||||
|
backs it up. Approach: a chart CronJob running `pg_dump` (postgres image, creds
|
||||||
|
from the Secret) to the app `data` PVC with N-day retention; optionally offload to
|
||||||
|
object storage.
|
||||||
|
- **Schema bootstrap hook.** The schema auto-applies on the app's first connection,
|
||||||
|
so a first-run race between the app and a CronJob is possible. Approach: a
|
||||||
|
`pre-install`/`pre-upgrade` Helm-hook Job that applies `storage/pg_schema.sql`
|
||||||
|
deterministically, with `PG_SKIP_SCHEMA=1` set on the app and CronJobs.
|
||||||
|
|
||||||
|
### Security
|
||||||
|
|
||||||
|
- **Pod hardening.** Add a `securityContext` across workloads: `runAsNonRoot`, drop
|
||||||
|
ALL capabilities, `seccompProfile: RuntimeDefault`, and `readOnlyRootFilesystem`
|
||||||
|
where feasible (the app needs `/app/data` writable — already a mounted volume).
|
||||||
|
- **NetworkPolicy.** Default-deny in the namespace; allow only the app, CronJobs, and
|
||||||
|
Grafana to reach Postgres:5432, and restrict who can reach the dashboard.
|
||||||
|
- **Grafana auth.** The chart inherits anonymous-admin from the original compose
|
||||||
|
(`templates/grafana.yaml` env). Make it off by default in-cluster, with a real
|
||||||
|
admin password sourced from the Secret (`GF_SECURITY_ADMIN_PASSWORD`).
|
||||||
|
|
||||||
|
### CI / operational
|
||||||
|
|
||||||
|
- **Chart CI validation.** Add a CI job running `helm lint` + `helm template` +
|
||||||
|
`kubeconform` on every push — catches template/schema errors automatically (the
|
||||||
|
chart could not be rendered in the sandbox where it was authored, since image-blob
|
||||||
|
CDNs were egress-blocked).
|
||||||
|
- **Image digest pinning.** For production installs, pin `image` by SHA digest rather
|
||||||
|
than the mutable `latest` tag.
|
||||||
|
- **PodDisruptionBudget / singleton guard.** A PDB for the singleton loop plus a
|
||||||
|
guard so it is never scaled above `1` or evicted to zero unexpectedly.
|
||||||
@@ -0,0 +1,30 @@
|
|||||||
|
# Local deployment (systemd user services)
|
||||||
|
|
||||||
|
Unit templates for running the stack from a git checkout + venv (the original
|
||||||
|
production shape; now the development shape). Paths inside the units assume
|
||||||
|
`/mnt/c/code/momentum` + `/home/philip/.venvs/momentum` — adjust to your box.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cp deploy/systemd/momentum-*.service ~/.config/systemd/user/
|
||||||
|
systemctl --user daemon-reload
|
||||||
|
systemctl --user enable --now momentum-loop momentum-dashboard momentum-qabot momentum-transcripts
|
||||||
|
systemctl --user enable --now momentum-premarket-eval.timer # 20:00 ET weekdays
|
||||||
|
loginctl enable-linger $USER # start on boot without login
|
||||||
|
```
|
||||||
|
|
||||||
|
| Unit | What |
|
||||||
|
|---|---|
|
||||||
|
| momentum-loop | trading loop (`run_live_paper.py --no-dashboard`) |
|
||||||
|
| momentum-dashboard | web dashboard :8765 (`dashboard_api.py`) |
|
||||||
|
| momentum-qabot | Telegram Q&A bot (read-only) |
|
||||||
|
| momentum-transcripts | YouTube → transcript → claims pipeline |
|
||||||
|
| momentum-premarket-eval(.timer) | nightly SIP refetch + housekeeping + shadow-track re-validation |
|
||||||
|
| momentum-testpg | local Postgres for test isolation (`/home/philip/pgrun` socket) — keep even when production is elsewhere |
|
||||||
|
| momentum-controlroom | NiceGUI control room :8766 — **disabled by owner preference**; kept for reference |
|
||||||
|
|
||||||
|
Config comes from `.env` in the repo root (see `.env.example`). The same
|
||||||
|
behavior-flag warnings as compose apply (`TRADING_AUTO_APPROVE=1` etc. — see
|
||||||
|
`deploy/README.md`).
|
||||||
|
|
||||||
|
**Never run these enabled while the compose stack owns production** — two loops
|
||||||
|
on one Alpaca account duplicate orders.
|
||||||
@@ -0,0 +1,21 @@
|
|||||||
|
[Unit]
|
||||||
|
Description=Momentum dashboard (read-only web UI on :8765)
|
||||||
|
# Self-heal forever: a DB blip or crash must not leave the website down. Decoupled
|
||||||
|
# from the loop on purpose -- a loop restart never takes the dashboard down and vice versa.
|
||||||
|
StartLimitIntervalSec=0
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
Type=simple
|
||||||
|
WorkingDirectory=/mnt/c/code/momentum
|
||||||
|
# dashboard_api.py loads .env (DATABASE_URL -> the LAN Postgres, with the host fallback)
|
||||||
|
# and binds DASHBOARD_HOST (default 0.0.0.0) : DASHBOARD_PORT (.env=8765) so it is reachable
|
||||||
|
# from the Windows browser via the netsh portproxy.
|
||||||
|
ExecStart=/home/philip/.venvs/momentum/bin/python -u dashboard_api.py
|
||||||
|
Restart=always
|
||||||
|
RestartSec=15
|
||||||
|
StandardOutput=append:/mnt/c/code/momentum/data/dashboard.log
|
||||||
|
StandardError=append:/mnt/c/code/momentum/data/dashboard.log
|
||||||
|
SyslogIdentifier=momentum-dashboard
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=default.target
|
||||||
@@ -0,0 +1,24 @@
|
|||||||
|
[Unit]
|
||||||
|
Description=Momentum live paper-trading loop
|
||||||
|
# Self-heal: if the loop crashes or the box reboots, systemd brings it back.
|
||||||
|
# The DB host (192.168.1.5) can go fully unreachable (a network/host outage) and
|
||||||
|
# the loop crashes on connect — so retry FOREVER (no give-up limit) and it
|
||||||
|
# auto-reconnects the moment the host returns. StartLimitIntervalSec=0 disables
|
||||||
|
# the "give up after N starts" cap entirely.
|
||||||
|
StartLimitIntervalSec=0
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
Type=simple
|
||||||
|
WorkingDirectory=/mnt/c/code/momentum
|
||||||
|
# the loop calls load_dotenv() -> finds .env in WorkingDirectory -> DATABASE_URL
|
||||||
|
# and all secrets load from there; no secrets in this unit file.
|
||||||
|
ExecStart=/home/philip/.venvs/momentum/bin/python -u run_live_paper.py --no-dashboard
|
||||||
|
Restart=always
|
||||||
|
RestartSec=30
|
||||||
|
# keep the same log file the operator + the review crons already tail
|
||||||
|
StandardOutput=append:/mnt/c/code/momentum/data/live_run.log
|
||||||
|
StandardError=append:/mnt/c/code/momentum/data/live_run.log
|
||||||
|
SyslogIdentifier=momentum-loop
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=default.target
|
||||||
@@ -0,0 +1,17 @@
|
|||||||
|
[Unit]
|
||||||
|
Description=Momentum Telegram Q&A bot
|
||||||
|
# retry forever through a DB-host outage (see momentum-loop.service)
|
||||||
|
StartLimitIntervalSec=0
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
Type=simple
|
||||||
|
WorkingDirectory=/mnt/c/code/momentum
|
||||||
|
ExecStart=/home/philip/.venvs/momentum/bin/python -u telegram_bot.py
|
||||||
|
Restart=always
|
||||||
|
RestartSec=30
|
||||||
|
StandardOutput=append:/mnt/c/code/momentum/data/telegram_bot.log
|
||||||
|
StandardError=append:/mnt/c/code/momentum/data/telegram_bot.log
|
||||||
|
SyslogIdentifier=momentum-qabot
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=default.target
|
||||||
@@ -0,0 +1,11 @@
|
|||||||
|
[Unit]
|
||||||
|
Description=Local Postgres for momentum test isolation (socket /home/philip/pgrun)
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
Type=forking
|
||||||
|
ExecStart=/home/philip/pg/bin/pg_ctl -D /home/philip/pgdata -o "-k /home/philip/pgrun" -l /home/philip/pgdata/logfile start
|
||||||
|
ExecStop=/home/philip/pg/bin/pg_ctl -D /home/philip/pgdata stop -m fast
|
||||||
|
Restart=on-failure
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=default.target
|
||||||
@@ -0,0 +1,19 @@
|
|||||||
|
[Unit]
|
||||||
|
Description=Momentum YouTube transcript -> claim-extraction pipeline
|
||||||
|
StartLimitIntervalSec=0
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
Type=simple
|
||||||
|
WorkingDirectory=/mnt/c/code/momentum
|
||||||
|
# secrets (DATABASE_URL, YOUTUBE_API_KEY, YT_WATCHLIST) load from the repo's gitignored .env
|
||||||
|
# via youtube_claims.config (python-dotenv). Whisper is option A (CPU) by default; flip to the
|
||||||
|
# LAN GPU (option B) with YT_WHISPER_MODEL=large-v3 YT_WHISPER_DEVICE=cuda YT_WHISPER_COMPUTE_TYPE=float16.
|
||||||
|
ExecStart=/home/philip/.venvs/momentum/bin/python -u -m youtube_claims.run run
|
||||||
|
Restart=always
|
||||||
|
RestartSec=30
|
||||||
|
StandardOutput=append:/mnt/c/code/momentum/data/transcripts_pipeline.log
|
||||||
|
StandardError=append:/mnt/c/code/momentum/data/transcripts_pipeline.log
|
||||||
|
SyslogIdentifier=momentum-transcripts
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=default.target
|
||||||
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
@@ -0,0 +1,6 @@
|
|||||||
|
# Historical documents
|
||||||
|
|
||||||
|
Point-in-time plans, phase writeups, and runbooks from earlier eras of the
|
||||||
|
project (DuckDB backend, WSL/systemd-only deployment, pre-migration
|
||||||
|
architecture pushes). Kept for archaeology; **do not follow them** — the
|
||||||
|
living docs are the root README.md, RUNBOOK.md, and deploy/README.md.
|
||||||
File renamed without changes.
@@ -0,0 +1,227 @@
|
|||||||
|
# RUNBOOK
|
||||||
|
|
||||||
|
Operational guide for running the momentum system. This file replaces the
|
||||||
|
original MILESTONE*.md documents, which were unrecoverable during the
|
||||||
|
rebuild.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Why live paper trading never worked (root causes, now fixed)
|
||||||
|
|
||||||
|
1. **Minute bars were never ingested.** The watcher reads 1-minute bars from
|
||||||
|
`data/research/market.duckdb`, but the old ingestion only wrote *daily*
|
||||||
|
bars for a handful of mega-caps. Fix: `research/ingestion/market_data.py`
|
||||||
|
ingests 1-minute Alpaca IEX bars continuously, and
|
||||||
|
`run_live_paper.py` schedules it every `LIVE_BARS_INTERVAL_SECONDS`.
|
||||||
|
|
||||||
|
2. **Session flags were hardcoded.** Every old row had
|
||||||
|
`is_regular_hours = TRUE`. Fix: flags are derived from the bar timestamp
|
||||||
|
in US/Eastern (premarket 04:00–09:30, regular 09:30–16:00, afterhours
|
||||||
|
16:00–20:00) — see `classify_session()` and its tests.
|
||||||
|
|
||||||
|
3. **The candidate filter excluded everything ingested.** The query had a
|
||||||
|
hardcoded `close BETWEEN 1 AND 20` while only mega-caps (price ≫ 20) were
|
||||||
|
in the database, so the watchlist was always empty. Fix: the band is
|
||||||
|
configurable (`WATCHER_PRICE_MIN` / `WATCHER_PRICE_MAX`) and the default
|
||||||
|
symbol list now actually gets ingested.
|
||||||
|
|
||||||
|
4. **No orchestrator.** Ingestion, watcher, broker sync, and execution were
|
||||||
|
never wired into one loop. Fix: `run_live_paper.py`.
|
||||||
|
|
||||||
|
5. **Broken source files.** `runtime/providers/registry.py`,
|
||||||
|
`schwab/market/models.py`, `option_chain_service.py`, and
|
||||||
|
`schwab/streaming/client.py` had syntax errors that crashed imports.
|
||||||
|
All repaired/rewritten.
|
||||||
|
|
||||||
|
6. **Schwab tokens were committed to git.** `data/schwab_tokens.json` is now
|
||||||
|
git-ignored and written `chmod 0600`. **Action required:** if that file
|
||||||
|
exists anywhere in your repo history, revoke the refresh token in the
|
||||||
|
Schwab developer portal and re-run the OAuth flow. Scrub history with
|
||||||
|
`git filter-repo --path data/schwab_tokens.json --invert-paths` if the
|
||||||
|
repo was ever shared.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Environment setup
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pip install -r requirements.txt
|
||||||
|
cp .env.example .env
|
||||||
|
```
|
||||||
|
|
||||||
|
Minimum for paper trading: `ALPACA_API_KEY`, `ALPACA_SECRET_KEY`
|
||||||
|
(free keys are fine — the system uses the IEX feed). Everything else has a
|
||||||
|
sensible default; see `.env.example` for the full catalog.
|
||||||
|
|
||||||
|
Key knobs:
|
||||||
|
|
||||||
|
| Variable | Default | Meaning |
|
||||||
|
|---|---|---|
|
||||||
|
| `TRADING_EXECUTION_MODE` | `alpaca_paper` | `schwab_live` routes to real money — leave on paper |
|
||||||
|
| `TRADING_AUTO_APPROVE` | `0` | `1` = orders submit without dashboard approval (or pass `--auto-approve`) |
|
||||||
|
| `TRADING_MAX_ORDERS_PER_TICK` | `1` | throttle new entries |
|
||||||
|
| `TRADING_MAX_CONCURRENT_POSITIONS` | `3` | cap on simultaneous open positions |
|
||||||
|
| `TRADING_RISK_PER_TRADE_PCT` | `0.01` | 1% of equity risked per trade (sizing = risk / (entry − stop)) |
|
||||||
|
| `TRADING_REWARD_MULTIPLE` | `2.0` | profit target = entry + this × risk (2R by default) |
|
||||||
|
| `TRADING_ENTRY_ORDER_TYPE` | `limit` | `limit` rests at the trigger so an unfilled entry can be cancelled; `market` fills instantly |
|
||||||
|
| `TRADING_ENTRY_TIMEOUT_BARS` | `2` | cancel an unfilled entry after this many minutes resting (wall-clock; `0` disables the time box) |
|
||||||
|
| `TRADING_ENTRY_INVALIDATE_PCT` | `0.0` | cancel an unfilled entry if the **live last-trade price** trades this fraction below the trigger (`0.0` = any break below entry; `0.005` = 0.5% below; negative disables) |
|
||||||
|
| `TRADING_ENTRY_GUARD_INTERVAL_SECONDS` | `5` | how often the fast guard re-checks armed entries against the live price — lower = faster break detection, more quote calls |
|
||||||
|
| `TRADING_MAX_DAILY_LOSS_PCT` | `0.03` | daily-loss circuit breaker |
|
||||||
|
| `WATCHER_SYMBOLS` | `AAPL,TSLA,…` | static watchlist; `--discover` adds screener actives |
|
||||||
|
| `WATCHER_PRICE_MIN/MAX` | `1` / `20` | candidate price band |
|
||||||
|
| `WATCHER_EVENT_DB_PATH` | `./data/momentum.duckdb` | the event store |
|
||||||
|
| `DASHBOARD_HOST/PORT` | `127.0.0.1` / `8010` | UI address |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Start commands
|
||||||
|
|
||||||
|
**Full live paper loop (recommended):**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python run_live_paper.py
|
||||||
|
# dashboard: http://127.0.0.1:8010
|
||||||
|
# flags: --once --symbols SNDL,COSM --auto-approve --discover --no-dashboard
|
||||||
|
```
|
||||||
|
|
||||||
|
Each tick prints one status line:
|
||||||
|
|
||||||
|
```
|
||||||
|
[09:31:02] ingest: 240 minute rows | watch: evaluated=5 ready=['SNDL'] blocked=3 | sync: ok | execute: approvals_requested=['…'] auto_executed=0
|
||||||
|
```
|
||||||
|
|
||||||
|
**Standalone read-only dashboard** (e.g. to review a past session):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python dashboard_api.py
|
||||||
|
```
|
||||||
|
|
||||||
|
Approve/reject/exit buttons require the orchestrator (it attaches the
|
||||||
|
execution service to the same server); the standalone runner returns 503 on
|
||||||
|
actions by design.
|
||||||
|
|
||||||
|
**Other tools:**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python fetch_minute_bars.py SNDL COSM --lookback 390 # ad-hoc ingestion
|
||||||
|
python research_cli.py symbols --date 2026-06-11 # what's in the DB
|
||||||
|
python research_cli.py gappers --date 2026-06-11 # gap scan → scanner_snapshots
|
||||||
|
python backtest_cli.py --date 2026-06-10 # backtest a stored session
|
||||||
|
python milestone3_verify.py # Schwab health walk
|
||||||
|
python -m pytest # full test suite
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. The trading day, end to end
|
||||||
|
|
||||||
|
1. **Ingest** — every 60 s, 1-minute IEX bars for the watchlist land in
|
||||||
|
`data/research/market.duckdb` with correct session flags. On boot, a
|
||||||
|
30-day daily backfill gives gap % and relative volume a baseline.
|
||||||
|
2. **Watch** — every 30 s the watcher pulls in-band candidates, evaluates
|
||||||
|
nine weighted criteria (gap, RVOL, impulse, pullback, pullback volume,
|
||||||
|
VWAP, candle quality, breakout, data sufficiency) plus structure
|
||||||
|
detection, and drives `discovered → watching → ready | blocked | late`.
|
||||||
|
A `signal_ready` is emitted at most once per symbol per session.
|
||||||
|
3. **Sync** — Alpaca account, positions, and orders snapshot into the event
|
||||||
|
store so projections and risk guards see broker truth.
|
||||||
|
4. **Execute** — ready signals become orders sized at `risk_per_trade_pct`
|
||||||
|
of equity, with the stop from the setup and the target at
|
||||||
|
entry + `reward_multiple` × risk (2R by default). Manual mode parks them
|
||||||
|
in the dashboard queue; **auto mode (`--auto-approve`) submits
|
||||||
|
immediately, no approval.** Guards: max concurrent positions, no re-entry
|
||||||
|
into held symbols, one request per symbol per session.
|
||||||
|
5. **Manage** — approve/reject from the dashboard; **Exit** sends a market
|
||||||
|
sell for the synced quantity.
|
||||||
|
|
||||||
|
### The entry lifecycle (auto-arm → fill or back out)
|
||||||
|
|
||||||
|
The earlier system would "see" a setup as something that had *already
|
||||||
|
happened* rather than a trade about to happen, and never pull the trigger.
|
||||||
|
The execution model now separates recognition from conviction from
|
||||||
|
discipline:
|
||||||
|
|
||||||
|
1. **Recognize** — when all the criteria pass, a `signal_ready` fires. The
|
||||||
|
setup is evaluated against the **last bar's timestamp**, not the wall
|
||||||
|
clock, so a live 9:40 breakout is judged against the 9:40 cutoff instead
|
||||||
|
of being dismissed as hours stale.
|
||||||
|
2. **Arm with conviction** — in auto mode the order is submitted right away
|
||||||
|
as a **bracket** (entry + stop + target placed atomically). With
|
||||||
|
`TRADING_ENTRY_ORDER_TYPE=limit` (default) the entry rests at the trigger,
|
||||||
|
so "unfilled" is a real, observable state.
|
||||||
|
3. **Back out at a defined point** — a fast **guard loop**
|
||||||
|
(`TRADING_ENTRY_GUARD_INTERVAL_SECONDS`, default 5 s) re-checks every
|
||||||
|
armed-but-unfilled entry and cancels it if either:
|
||||||
|
- the **live last-trade price** (pulled tick-by-tick from Alpaca's data
|
||||||
|
API, not a bar close) trades back below the trigger by
|
||||||
|
`TRADING_ENTRY_INVALIDATE_PCT`, or
|
||||||
|
- it has rested longer than `TRADING_ENTRY_TIMEOUT_BARS` minutes
|
||||||
|
(wall-clock — running the guard often does **not** shorten this window).
|
||||||
|
A cancel frees the risk budget and the concurrent-position slot, and emits
|
||||||
|
a `risk_rule_triggered` (`entry_backout`) you can see in the activity feed.
|
||||||
|
Once the order **fills**, it stops being tracked and the bracket's stop and
|
||||||
|
target manage the open position — no second-guessing.
|
||||||
|
|
||||||
|
Tuning cheat-sheet: react to a fakeout faster → lower
|
||||||
|
`TRADING_ENTRY_GUARD_INTERVAL_SECONDS` (e.g. `2`); more patience on the fill →
|
||||||
|
raise `TRADING_ENTRY_TIMEOUT_BARS`; give the breakout room to wobble → set
|
||||||
|
`TRADING_ENTRY_INVALIDATE_PCT` to something like `0.005`; chase fills instead
|
||||||
|
of resting → `TRADING_ENTRY_ORDER_TYPE=market` (note: market fills can't time
|
||||||
|
out); bigger winners → raise `TRADING_REWARD_MULTIPLE`.
|
||||||
|
|
||||||
|
> **Quote rate limits.** The guard makes one latest-trade call per armed
|
||||||
|
> symbol per pass (a ~2 s cache collapses duplicates). At the 5 s default with
|
||||||
|
> a few open entries you're well inside Alpaca's free-tier 200 req/min. If you
|
||||||
|
> drop the interval to 1 s with several armed entries, watch the budget —
|
||||||
|
> bars ingestion and account sync also draw from it.
|
||||||
|
|
||||||
|
### Reading the dashboard
|
||||||
|
|
||||||
|
- **Badge** — blue `PAPER · ALPACA` vs red `LIVE · SCHWAB`. If it's red and
|
||||||
|
you didn't mean it, stop and check `TRADING_EXECUTION_MODE`.
|
||||||
|
- **P&L strip** (top) — session realized / unrealized / total, win rate,
|
||||||
|
average R, open/closed counts. Green/red signed.
|
||||||
|
- **Approval Queue** — pending entries with a **risk preview** ($ risk,
|
||||||
|
shares, notional, stop %) and Approve/Reject. Keyboard: `A` / `R` act on the
|
||||||
|
top row.
|
||||||
|
- **Ready Signals / Open Positions** — each row has a live **sparkline** (last
|
||||||
|
~40 one-minute closes) with dashed entry (blue) and stop (red) lines.
|
||||||
|
- **Watch States** — click any row to expand the **nine-criteria breakdown**
|
||||||
|
(✓ pass / ✕ fail / · not-evaluated). The add-symbol box (or `/`) injects a
|
||||||
|
ticker onto the watchlist for the rest of the session.
|
||||||
|
- **Activity Feed** — submissions and fills as they happen.
|
||||||
|
- **Order Lifecycle** — per-order status from the event store.
|
||||||
|
- **Connection dot** (top right) — green pulse = live SSE stream; it falls
|
||||||
|
back to 4 s polling automatically if the stream drops.
|
||||||
|
- **Sound** — toggle the button or press `S` for an audio + toast alert when a
|
||||||
|
new signal goes ready. Press `?` for the shortcut list.
|
||||||
|
|
||||||
|
Streams from `GET /api/stream` (SSE); also exposes `/api/snapshots`,
|
||||||
|
`/api/criteria?symbol=`, `/api/bars?symbol=&minutes=`, and
|
||||||
|
`POST /api/watch/add`. Charts require the research DB, which the orchestrator
|
||||||
|
wires in automatically; the standalone `dashboard_api.py` has no bars source,
|
||||||
|
so sparklines there show "no bars".
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Troubleshooting
|
||||||
|
|
||||||
|
| Symptom | Likely cause / fix |
|
||||||
|
|---|---|
|
||||||
|
| `watch: evaluated=0` all day | No minute bars for today. Check keys; run `python research_cli.py symbols`. Outside market hours there are no new bars — backfill with `fetch_minute_bars.py --lookback 1440`. |
|
||||||
|
| Candidates exist but never `ready` | Score < 60%. Watch the `Watch States` panel score column; criteria details are in `criteria_evaluated` events. |
|
||||||
|
| `ingest: … 403` errors | Free keys are IEX-only — ensure `ALPACA_DATA_FEED=iex` (default) and that the request window ends ≥30 s in the past (handled automatically). |
|
||||||
|
| Approve button does nothing | You're on `dashboard_api.py` (read-only). Use `run_live_paper.py`. |
|
||||||
|
| Port 8010 busy | Set `DASHBOARD_PORT`, or `--no-dashboard`. |
|
||||||
|
| Schwab shows `SCHWAB-UNAUTH` | Expected without OAuth. Run the flow in `schwab/auth/oauth2_flow_manager.py`; verify with `milestone3_verify.py`. |
|
||||||
|
| Start fresh | Delete `data/momentum.duckdb` (events) and/or `data/research/` (bars). Both rebuild automatically. |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Going live on Schwab (when ready)
|
||||||
|
|
||||||
|
Paper-trade until the stats earn it. Then: complete OAuth
|
||||||
|
(`milestone3_verify.py` all green), set `TRADING_EXECUTION_MODE=schwab_live`,
|
||||||
|
keep `TRADING_AUTO_APPROVE=0`, and start with the smallest viable
|
||||||
|
`TRADING_RISK_PER_TRADE_PCT`. The dashboard badge turns red in live mode.
|
||||||
Reference in new issue
Block a user