Docs cleanup: current-state rewrite + three deployment modes + history/
- 10 era documents (phase plans, build summaries, redesign writeups, old runbooks, completed consolidation plan) moved to docs/history/ with a do-not-follow README. - Root README: current validated population ($2-20, 10-35% static gap + squeeze exception, SIP), Deployment section, automated-maintenance reality, corrected repo map (admin panel, youtube_claims, deploy/). - RUNBOOK rewritten for the running system: daily rhythm, health checks per deployment, and the production failure-mode table earned this week (env drift, pgdata mount traps, uid 10001, zombie approvals, PG_SKIP_ SCHEMA, timezone artifacts). - deploy/README.md is now the three-mode index: compose (canonical), systemd (local dev — unit templates now committed), helm (refresh staged as issue #8). Old k8s doc moved to deploy/helm/README.md. - docker/README.md: six TrueNAS gotchas, each from a real outage. - Admin panel improvements staged as GitHub issues #4-#7. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
1 parent
d4c695e213
commit
5ac2cc7b89
23 files changed
+680
-396
No files matched your search
+37
-171
@@ -1,182 +1,48 @@
|
||||
# Kubernetes deployment
|
||||
# Deploying Momentum
|
||||
|
||||
A container image and Helm chart that run the whole Momentum System — including
|
||||
its **PostgreSQL datastore** — inside a single Kubernetes namespace.
|
||||
Three supported shapes, one image/codebase. Pick by where it runs:
|
||||
|
||||
## What gets deployed
|
||||
| Mode | Status | Use when | Docs |
|
||||
|---|---|---|---|
|
||||
| **Docker Compose** | ✅ **canonical** (production since 2026-07-09) | Any always-on Linux host; TrueNAS SCALE | [docker/README.md](docker/README.md), `docker-compose.yml` (root), [docker/truenas-compose.yml](docker/truenas-compose.yml) |
|
||||
| **Local (systemd)** | ✅ maintained | Development machine, WSL, bare metal without Docker | [systemd/README.md](systemd/README.md) + unit templates in `deploy/systemd/` |
|
||||
| **Helm / Kubernetes** | 🚧 planned refresh | Future k8s hosting | [helm/](helm/) — the chart predates the 2026-07 architecture; starting point, not deployable |
|
||||
|
||||
| Workload | Kind | Notes |
|
||||
| --- | --- | --- |
|
||||
| PostgreSQL | StatefulSet + PVC | Single source of truth. Schema auto-creates on the app's first connection from `storage/pg_schema.sql`. |
|
||||
| App loop + dashboard | Deployment (`replicas: 1`, `Recreate`) | `run_live_paper.py` with the **embedded** dashboard on :8010. Stateful singleton — never scale up. |
|
||||
| journal / eod-replay / nightly-tune | CronJobs | US/Eastern (`spec.timeZone`), weekdays. Share the app's `data/` PVC. |
|
||||
| Grafana | Deployment + Service | Optional (`grafana.enabled`). Datasource auto-points at the in-cluster Postgres. |
|
||||
## Docker Compose (canonical)
|
||||
|
||||
The app loop and the CronJobs run the **same image** (built from the repo
|
||||
`Dockerfile`), just with different commands.
|
||||
The root `docker-compose.yml` runs the full stack self-contained: postgres 18,
|
||||
trading loop, dashboard (:8765), admin panel (:8767), telegram bot, transcripts
|
||||
pipeline, nightly scheduler, on-demand jobs, optional Grafana. The image is
|
||||
published to `docker.io/fez5587/momentum-app` and `ghcr.io/fez5587/momentum-app`
|
||||
by CI on every master push.
|
||||
|
||||
## Local development with Docker Compose
|
||||
For TrueNAS SCALE use `docker/truenas-compose.yml` (pull-only, dataset host
|
||||
paths, inline env). **Read the gotchas in docker/README.md before editing the
|
||||
YAML** — anchor expansion, per-service env, and path case-sensitivity have each
|
||||
caused an outage.
|
||||
|
||||
For a single-machine stack (same components, no Kubernetes) use the root
|
||||
`docker-compose.yml`. It builds the app from the `Dockerfile`, brings up
|
||||
PostgreSQL with a persistent volume, and runs the loop with its embedded
|
||||
dashboard. Config comes from a local `.env` (optional — without `ALPACA_*` keys
|
||||
the loop runs in dry mode; the dashboard and DB still work). `DATABASE_URL` is
|
||||
overridden to the in-network `postgres` service, so the stack is self-contained.
|
||||
## Local (systemd user services)
|
||||
|
||||
```bash
|
||||
cp .env.example .env # optional: add ALPACA_* keys for live data
|
||||
The original deployment: unit templates in `deploy/systemd/` running a git
|
||||
checkout + venv, config from `.env`. Still the best loop for development —
|
||||
tests, backtests, and research scripts run here. Keep the units **disabled**
|
||||
while the compose stack owns production: two loops against one Alpaca account
|
||||
create duplicate orders.
|
||||
|
||||
docker compose up -d --build # postgres + app loop + dashboard (:8010)
|
||||
docker compose logs -f app # watch the loop
|
||||
open http://localhost:8010 # dashboard
|
||||
## Helm (future)
|
||||
|
||||
docker compose --profile monitoring up -d # also start Grafana (:3000)
|
||||
```
|
||||
`deploy/helm/` holds the original chart (single app deployment + CronJobs). It
|
||||
predates the service split, the session gate, and the env-flag inventory; the
|
||||
refresh is tracked in GitHub issues. Until then, compose is the way.
|
||||
|
||||
The scheduled jobs sit under the `jobs` profile so they don't auto-start; run
|
||||
them on demand (or from host cron / a scheduler on the ET times in `.env`):
|
||||
## Config in all modes
|
||||
|
||||
```bash
|
||||
docker compose run --rm nightly-tune
|
||||
docker compose run --rm eod-replay
|
||||
docker compose run --rm journal
|
||||
```
|
||||
|
||||
Postgres credentials default to `momentum`/`momentum`/`momentum`; override with
|
||||
`POSTGRES_USER` / `POSTGRES_PASSWORD` / `POSTGRES_DB` in `.env`. Named volumes
|
||||
`pg-data`, `app-data`, and `grafana-data` persist across restarts (the jobs share
|
||||
`app-data` with the app so `nightly_tune.py`'s `learned_params.json` reaches the
|
||||
loop). The Compose Grafana uses its own datasource override at
|
||||
`deploy/compose/grafana/datasources/momentum.yaml` (pointed at the `postgres`
|
||||
service); the legacy `grafana/docker-compose.yml` is unrelated and left for
|
||||
Grafana-only use against an external DB.
|
||||
|
||||
## Build & publish the image
|
||||
|
||||
Pushes to the working branch / `main` / `v*` tags trigger
|
||||
`.github/workflows/docker-publish.yml`, which builds and pushes to
|
||||
`ghcr.io/fez5587/momentum-system`. Point the chart at a pushed tag via
|
||||
`image.tag` (a commit SHA, or `latest` on the default branch).
|
||||
|
||||
Local build:
|
||||
|
||||
```bash
|
||||
docker build -t momentum:dev .
|
||||
```
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
helm install momentum ./deploy/helm/momentum \
|
||||
-n momentum --create-namespace \
|
||||
--set image.tag=latest \
|
||||
--set secrets.alpacaApiKey=YOUR_KEY \
|
||||
--set secrets.alpacaSecretKey=YOUR_SECRET
|
||||
```
|
||||
|
||||
Without Alpaca keys the loop runs in **dry mode** (no live data/orders); the
|
||||
dashboard and database still work.
|
||||
|
||||
### Secrets
|
||||
|
||||
The chart builds `DATABASE_URL` from the in-cluster Postgres service and a
|
||||
password that is generated on first install (or set `postgres.password` to pin
|
||||
it). Broker keys come from `secrets.*`.
|
||||
|
||||
For real keys, prefer an existing Secret over `values.yaml`:
|
||||
|
||||
```bash
|
||||
kubectl -n momentum create secret generic momentum-broker \
|
||||
--from-literal=DATABASE_URL=... \
|
||||
--from-literal=ALPACA_API_KEY=... --from-literal=ALPACA_SECRET_KEY=... \
|
||||
--from-literal=APCA_API_KEY_ID=... --from-literal=APCA_API_SECRET_KEY=...
|
||||
helm install momentum ./deploy/helm/momentum -n momentum \
|
||||
--set secrets.existingSecret=momentum-broker
|
||||
```
|
||||
|
||||
When `secrets.existingSecret` is set the chart renders no Secret of its own and
|
||||
expects that one to carry `DATABASE_URL`, the `ALPACA_*`/`APCA_*` keys, and
|
||||
`POSTGRES_PASSWORD`.
|
||||
|
||||
## Reach the dashboard
|
||||
|
||||
```bash
|
||||
kubectl -n momentum port-forward svc/momentum-app 8010:8010
|
||||
open http://localhost:8010
|
||||
```
|
||||
|
||||
Set `ingress.enabled=true` (with `ingress.host`/`className`) to expose it
|
||||
instead.
|
||||
|
||||
## Verify
|
||||
|
||||
```bash
|
||||
helm lint deploy/helm/momentum
|
||||
helm template momentum deploy/helm/momentum | less
|
||||
|
||||
# After install:
|
||||
kubectl -n momentum get pods
|
||||
kubectl -n momentum exec sts/momentum-postgres -- psql -U momentum -d momentum -c '\dt'
|
||||
kubectl -n momentum create job --from=cronjob/momentum-nightly-tune tune-test
|
||||
```
|
||||
|
||||
## Notes & constraints
|
||||
|
||||
- **Singleton loop.** The trading loop holds in-memory state and one broker
|
||||
session; `replicas` must stay `1` and the strategy is `Recreate`.
|
||||
- **Shared `data/` PVC is ReadWriteOnce.** The CronJob pods mount the same PVC
|
||||
as the app (so `nightly_tune.py` can write `learned_params.json` for the loop
|
||||
to read), which schedules them onto the app pod's node. For multi-node
|
||||
clusters, use an RWX StorageClass or split the volumes.
|
||||
- **CronJob `spec.timeZone`** requires Kubernetes ≥ 1.27.
|
||||
- The legacy `grafana/docker-compose.yml` is left in place for local use; the
|
||||
chart provisions its own Grafana against the in-cluster Postgres.
|
||||
|
||||
## Future hardening (roadmap)
|
||||
|
||||
The package deploys and runs, but the items below separate "it comes up" from
|
||||
"runs reliably for a trading system." They are **not implemented yet** — captured
|
||||
here so they can be picked up later. Each notes what, why, and the rough approach.
|
||||
|
||||
### Reliability
|
||||
|
||||
- **Loop-heartbeat liveness** *(needs an app-code change).* Today `/api/health`
|
||||
(`api/main.py:343`) returns a static `{"ok": true}`, which only proves the
|
||||
dashboard *thread* is alive. The trading loop runs in the main thread
|
||||
(`run_live_paper.py:328`), so a **crashed** loop exits the process and Kubernetes
|
||||
restarts it (fine), but a **hung** loop (stuck network call, deadlock) keeps
|
||||
answering health checks `200` and runs undetected. Approach: have the loop stamp a
|
||||
last-tick timestamp into `DashboardState` each iteration, make `/api/health` (or a
|
||||
new `/healthz`) return `503` when the last tick is older than a threshold, and
|
||||
point the Deployment's `livenessProbe` at it. Highest-value item.
|
||||
- **Postgres backup CronJob.** The event store is the source of truth and nothing
|
||||
backs it up. Approach: a chart CronJob running `pg_dump` (postgres image, creds
|
||||
from the Secret) to the app `data` PVC with N-day retention; optionally offload to
|
||||
object storage.
|
||||
- **Schema bootstrap hook.** The schema auto-applies on the app's first connection,
|
||||
so a first-run race between the app and a CronJob is possible. Approach: a
|
||||
`pre-install`/`pre-upgrade` Helm-hook Job that applies `storage/pg_schema.sql`
|
||||
deterministically, with `PG_SKIP_SCHEMA=1` set on the app and CronJobs.
|
||||
|
||||
### Security
|
||||
|
||||
- **Pod hardening.** Add a `securityContext` across workloads: `runAsNonRoot`, drop
|
||||
ALL capabilities, `seccompProfile: RuntimeDefault`, and `readOnlyRootFilesystem`
|
||||
where feasible (the app needs `/app/data` writable — already a mounted volume).
|
||||
- **NetworkPolicy.** Default-deny in the namespace; allow only the app, CronJobs, and
|
||||
Grafana to reach Postgres:5432, and restrict who can reach the dashboard.
|
||||
- **Grafana auth.** The chart inherits anonymous-admin from the original compose
|
||||
(`templates/grafana.yaml` env). Make it off by default in-cluster, with a real
|
||||
admin password sourced from the Secret (`GF_SECURITY_ADMIN_PASSWORD`).
|
||||
|
||||
### CI / operational
|
||||
|
||||
- **Chart CI validation.** Add a CI job running `helm lint` + `helm template` +
|
||||
`kubeconform` on every push — catches template/schema errors automatically (the
|
||||
chart could not be rendered in the sandbox where it was authored, since image-blob
|
||||
CDNs were egress-blocked).
|
||||
- **Image digest pinning.** For production installs, pin `image` by SHA digest rather
|
||||
than the mutable `latest` tag.
|
||||
- **PodDisruptionBudget / singleton guard.** A PDB for the singleton loop plus a
|
||||
guard so it is never scaled above `1` or evicted to zero unexpectedly.
|
||||
- `.env.example` (committed, sanitized) is the full variable inventory and
|
||||
says which layer each belongs in.
|
||||
- Behavior flags that MUST be set explicitly in any deployment — silent
|
||||
defaults bite: `TRADING_AUTO_APPROVE=1` (else entries queue for manual
|
||||
approval), `ALPACA_DATA_FEED=sip`, `OLLAMA_ENABLED=true` (catalyst engine),
|
||||
`FMP_API_KEY` (float cache). `TRADING_SESSION_ONLY` defaults ON in
|
||||
deployed configs (bars after-hours entries).
|
||||
- Runtime overrides: the admin panel writes `data/overrides.env`, which every
|
||||
service loads over its baked env at boot (`runtime/env_overrides.py`).
|
||||
@@ -94,3 +94,24 @@ NOTE (2026-07-09): the live `.env` contains two commented-out PowerShell lines
|
||||
embedding the real Alpaca credentials — harmless there, but they nearly leaked
|
||||
into the committed `.env.example` (caught by the regeneration leak check).
|
||||
Consider deleting them from `.env`.
|
||||
|
||||
## TrueNAS gotchas (each one caused a real outage — read before editing)
|
||||
|
||||
1. **Anchors get expanded.** TrueNAS renders `x-app:`/`<<: *app` into inline
|
||||
per-service blocks. After the first save, the `x-app` block is decorative —
|
||||
env/volume edits there reach nothing. Edit each service's own block.
|
||||
2. **Verify env landed** with `docker exec <svc> sh -c 'echo $VAR'` — not by
|
||||
reading the YAML, and not via the admin panel (it reports its own container).
|
||||
3. **Paths are case-sensitive** and Docker auto-creates missing bind sources:
|
||||
`/mnt/Volume` vs `/mnt/volume` silently forked the database into two worlds.
|
||||
After any data move, verify by **size at destination** (`du -sh`), never `ls`.
|
||||
4. **Volume changes need container recreation** — Stop/Start or Update the app;
|
||||
a container restart keeps the old mounts (and a running postgres keeps
|
||||
writing to *deleted* files if you move its dir underneath it).
|
||||
5. **First deploy initializes postgres** — `POSTGRES_PASSWORD` only applies to
|
||||
an empty pgdata. A failed first attempt leaves a cluster with the old
|
||||
password; fix with `ALTER USER` via `docker exec ... psql -U momentum` or
|
||||
wipe before the real deploy.
|
||||
6. **uid 10001**: the app containers run non-root; `chown -R 10001:10001` the
|
||||
host data dir or every file-writer (scheduler TSVs, admin profiles,
|
||||
transcripts) hits `PermissionError`.
|
||||
@@ -0,0 +1,182 @@
|
||||
# Kubernetes deployment
|
||||
|
||||
A container image and Helm chart that run the whole Momentum System — including
|
||||
its **PostgreSQL datastore** — inside a single Kubernetes namespace.
|
||||
|
||||
## What gets deployed
|
||||
|
||||
| Workload | Kind | Notes |
|
||||
| --- | --- | --- |
|
||||
| PostgreSQL | StatefulSet + PVC | Single source of truth. Schema auto-creates on the app's first connection from `storage/pg_schema.sql`. |
|
||||
| App loop + dashboard | Deployment (`replicas: 1`, `Recreate`) | `run_live_paper.py` with the **embedded** dashboard on :8010. Stateful singleton — never scale up. |
|
||||
| journal / eod-replay / nightly-tune | CronJobs | US/Eastern (`spec.timeZone`), weekdays. Share the app's `data/` PVC. |
|
||||
| Grafana | Deployment + Service | Optional (`grafana.enabled`). Datasource auto-points at the in-cluster Postgres. |
|
||||
|
||||
The app loop and the CronJobs run the **same image** (built from the repo
|
||||
`Dockerfile`), just with different commands.
|
||||
|
||||
## Local development with Docker Compose
|
||||
|
||||
For a single-machine stack (same components, no Kubernetes) use the root
|
||||
`docker-compose.yml`. It builds the app from the `Dockerfile`, brings up
|
||||
PostgreSQL with a persistent volume, and runs the loop with its embedded
|
||||
dashboard. Config comes from a local `.env` (optional — without `ALPACA_*` keys
|
||||
the loop runs in dry mode; the dashboard and DB still work). `DATABASE_URL` is
|
||||
overridden to the in-network `postgres` service, so the stack is self-contained.
|
||||
|
||||
```bash
|
||||
cp .env.example .env # optional: add ALPACA_* keys for live data
|
||||
|
||||
docker compose up -d --build # postgres + app loop + dashboard (:8010)
|
||||
docker compose logs -f app # watch the loop
|
||||
open http://localhost:8010 # dashboard
|
||||
|
||||
docker compose --profile monitoring up -d # also start Grafana (:3000)
|
||||
```
|
||||
|
||||
The scheduled jobs sit under the `jobs` profile so they don't auto-start; run
|
||||
them on demand (or from host cron / a scheduler on the ET times in `.env`):
|
||||
|
||||
```bash
|
||||
docker compose run --rm nightly-tune
|
||||
docker compose run --rm eod-replay
|
||||
docker compose run --rm journal
|
||||
```
|
||||
|
||||
Postgres credentials default to `momentum`/`momentum`/`momentum`; override with
|
||||
`POSTGRES_USER` / `POSTGRES_PASSWORD` / `POSTGRES_DB` in `.env`. Named volumes
|
||||
`pg-data`, `app-data`, and `grafana-data` persist across restarts (the jobs share
|
||||
`app-data` with the app so `nightly_tune.py`'s `learned_params.json` reaches the
|
||||
loop). The Compose Grafana uses its own datasource override at
|
||||
`deploy/compose/grafana/datasources/momentum.yaml` (pointed at the `postgres`
|
||||
service); the legacy `grafana/docker-compose.yml` is unrelated and left for
|
||||
Grafana-only use against an external DB.
|
||||
|
||||
## Build & publish the image
|
||||
|
||||
Pushes to the working branch / `main` / `v*` tags trigger
|
||||
`.github/workflows/docker-publish.yml`, which builds and pushes to
|
||||
`ghcr.io/fez5587/momentum-system`. Point the chart at a pushed tag via
|
||||
`image.tag` (a commit SHA, or `latest` on the default branch).
|
||||
|
||||
Local build:
|
||||
|
||||
```bash
|
||||
docker build -t momentum:dev .
|
||||
```
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
helm install momentum ./deploy/helm/momentum \
|
||||
-n momentum --create-namespace \
|
||||
--set image.tag=latest \
|
||||
--set secrets.alpacaApiKey=YOUR_KEY \
|
||||
--set secrets.alpacaSecretKey=YOUR_SECRET
|
||||
```
|
||||
|
||||
Without Alpaca keys the loop runs in **dry mode** (no live data/orders); the
|
||||
dashboard and database still work.
|
||||
|
||||
### Secrets
|
||||
|
||||
The chart builds `DATABASE_URL` from the in-cluster Postgres service and a
|
||||
password that is generated on first install (or set `postgres.password` to pin
|
||||
it). Broker keys come from `secrets.*`.
|
||||
|
||||
For real keys, prefer an existing Secret over `values.yaml`:
|
||||
|
||||
```bash
|
||||
kubectl -n momentum create secret generic momentum-broker \
|
||||
--from-literal=DATABASE_URL=... \
|
||||
--from-literal=ALPACA_API_KEY=... --from-literal=ALPACA_SECRET_KEY=... \
|
||||
--from-literal=APCA_API_KEY_ID=... --from-literal=APCA_API_SECRET_KEY=...
|
||||
helm install momentum ./deploy/helm/momentum -n momentum \
|
||||
--set secrets.existingSecret=momentum-broker
|
||||
```
|
||||
|
||||
When `secrets.existingSecret` is set the chart renders no Secret of its own and
|
||||
expects that one to carry `DATABASE_URL`, the `ALPACA_*`/`APCA_*` keys, and
|
||||
`POSTGRES_PASSWORD`.
|
||||
|
||||
## Reach the dashboard
|
||||
|
||||
```bash
|
||||
kubectl -n momentum port-forward svc/momentum-app 8010:8010
|
||||
open http://localhost:8010
|
||||
```
|
||||
|
||||
Set `ingress.enabled=true` (with `ingress.host`/`className`) to expose it
|
||||
instead.
|
||||
|
||||
## Verify
|
||||
|
||||
```bash
|
||||
helm lint deploy/helm/momentum
|
||||
helm template momentum deploy/helm/momentum | less
|
||||
|
||||
# After install:
|
||||
kubectl -n momentum get pods
|
||||
kubectl -n momentum exec sts/momentum-postgres -- psql -U momentum -d momentum -c '\dt'
|
||||
kubectl -n momentum create job --from=cronjob/momentum-nightly-tune tune-test
|
||||
```
|
||||
|
||||
## Notes & constraints
|
||||
|
||||
- **Singleton loop.** The trading loop holds in-memory state and one broker
|
||||
session; `replicas` must stay `1` and the strategy is `Recreate`.
|
||||
- **Shared `data/` PVC is ReadWriteOnce.** The CronJob pods mount the same PVC
|
||||
as the app (so `nightly_tune.py` can write `learned_params.json` for the loop
|
||||
to read), which schedules them onto the app pod's node. For multi-node
|
||||
clusters, use an RWX StorageClass or split the volumes.
|
||||
- **CronJob `spec.timeZone`** requires Kubernetes ≥ 1.27.
|
||||
- The legacy `grafana/docker-compose.yml` is left in place for local use; the
|
||||
chart provisions its own Grafana against the in-cluster Postgres.
|
||||
|
||||
## Future hardening (roadmap)
|
||||
|
||||
The package deploys and runs, but the items below separate "it comes up" from
|
||||
"runs reliably for a trading system." They are **not implemented yet** — captured
|
||||
here so they can be picked up later. Each notes what, why, and the rough approach.
|
||||
|
||||
### Reliability
|
||||
|
||||
- **Loop-heartbeat liveness** *(needs an app-code change).* Today `/api/health`
|
||||
(`api/main.py:343`) returns a static `{"ok": true}`, which only proves the
|
||||
dashboard *thread* is alive. The trading loop runs in the main thread
|
||||
(`run_live_paper.py:328`), so a **crashed** loop exits the process and Kubernetes
|
||||
restarts it (fine), but a **hung** loop (stuck network call, deadlock) keeps
|
||||
answering health checks `200` and runs undetected. Approach: have the loop stamp a
|
||||
last-tick timestamp into `DashboardState` each iteration, make `/api/health` (or a
|
||||
new `/healthz`) return `503` when the last tick is older than a threshold, and
|
||||
point the Deployment's `livenessProbe` at it. Highest-value item.
|
||||
- **Postgres backup CronJob.** The event store is the source of truth and nothing
|
||||
backs it up. Approach: a chart CronJob running `pg_dump` (postgres image, creds
|
||||
from the Secret) to the app `data` PVC with N-day retention; optionally offload to
|
||||
object storage.
|
||||
- **Schema bootstrap hook.** The schema auto-applies on the app's first connection,
|
||||
so a first-run race between the app and a CronJob is possible. Approach: a
|
||||
`pre-install`/`pre-upgrade` Helm-hook Job that applies `storage/pg_schema.sql`
|
||||
deterministically, with `PG_SKIP_SCHEMA=1` set on the app and CronJobs.
|
||||
|
||||
### Security
|
||||
|
||||
- **Pod hardening.** Add a `securityContext` across workloads: `runAsNonRoot`, drop
|
||||
ALL capabilities, `seccompProfile: RuntimeDefault`, and `readOnlyRootFilesystem`
|
||||
where feasible (the app needs `/app/data` writable — already a mounted volume).
|
||||
- **NetworkPolicy.** Default-deny in the namespace; allow only the app, CronJobs, and
|
||||
Grafana to reach Postgres:5432, and restrict who can reach the dashboard.
|
||||
- **Grafana auth.** The chart inherits anonymous-admin from the original compose
|
||||
(`templates/grafana.yaml` env). Make it off by default in-cluster, with a real
|
||||
admin password sourced from the Secret (`GF_SECURITY_ADMIN_PASSWORD`).
|
||||
|
||||
### CI / operational
|
||||
|
||||
- **Chart CI validation.** Add a CI job running `helm lint` + `helm template` +
|
||||
`kubeconform` on every push — catches template/schema errors automatically (the
|
||||
chart could not be rendered in the sandbox where it was authored, since image-blob
|
||||
CDNs were egress-blocked).
|
||||
- **Image digest pinning.** For production installs, pin `image` by SHA digest rather
|
||||
than the mutable `latest` tag.
|
||||
- **PodDisruptionBudget / singleton guard.** A PDB for the singleton loop plus a
|
||||
guard so it is never scaled above `1` or evicted to zero unexpectedly.
|
||||
@@ -0,0 +1,30 @@
|
||||
# Local deployment (systemd user services)
|
||||
|
||||
Unit templates for running the stack from a git checkout + venv (the original
|
||||
production shape; now the development shape). Paths inside the units assume
|
||||
`/mnt/c/code/momentum` + `/home/philip/.venvs/momentum` — adjust to your box.
|
||||
|
||||
```bash
|
||||
cp deploy/systemd/momentum-*.service ~/.config/systemd/user/
|
||||
systemctl --user daemon-reload
|
||||
systemctl --user enable --now momentum-loop momentum-dashboard momentum-qabot momentum-transcripts
|
||||
systemctl --user enable --now momentum-premarket-eval.timer # 20:00 ET weekdays
|
||||
loginctl enable-linger $USER # start on boot without login
|
||||
```
|
||||
|
||||
| Unit | What |
|
||||
|---|---|
|
||||
| momentum-loop | trading loop (`run_live_paper.py --no-dashboard`) |
|
||||
| momentum-dashboard | web dashboard :8765 (`dashboard_api.py`) |
|
||||
| momentum-qabot | Telegram Q&A bot (read-only) |
|
||||
| momentum-transcripts | YouTube → transcript → claims pipeline |
|
||||
| momentum-premarket-eval(.timer) | nightly SIP refetch + housekeeping + shadow-track re-validation |
|
||||
| momentum-testpg | local Postgres for test isolation (`/home/philip/pgrun` socket) — keep even when production is elsewhere |
|
||||
| momentum-controlroom | NiceGUI control room :8766 — **disabled by owner preference**; kept for reference |
|
||||
|
||||
Config comes from `.env` in the repo root (see `.env.example`). The same
|
||||
behavior-flag warnings as compose apply (`TRADING_AUTO_APPROVE=1` etc. — see
|
||||
`deploy/README.md`).
|
||||
|
||||
**Never run these enabled while the compose stack owns production** — two loops
|
||||
on one Alpaca account duplicate orders.
|
||||
@@ -0,0 +1,21 @@
|
||||
[Unit]
|
||||
Description=Momentum dashboard (read-only web UI on :8765)
|
||||
# Self-heal forever: a DB blip or crash must not leave the website down. Decoupled
|
||||
# from the loop on purpose -- a loop restart never takes the dashboard down and vice versa.
|
||||
StartLimitIntervalSec=0
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
WorkingDirectory=/mnt/c/code/momentum
|
||||
# dashboard_api.py loads .env (DATABASE_URL -> the LAN Postgres, with the host fallback)
|
||||
# and binds DASHBOARD_HOST (default 0.0.0.0) : DASHBOARD_PORT (.env=8765) so it is reachable
|
||||
# from the Windows browser via the netsh portproxy.
|
||||
ExecStart=/home/philip/.venvs/momentum/bin/python -u dashboard_api.py
|
||||
Restart=always
|
||||
RestartSec=15
|
||||
StandardOutput=append:/mnt/c/code/momentum/data/dashboard.log
|
||||
StandardError=append:/mnt/c/code/momentum/data/dashboard.log
|
||||
SyslogIdentifier=momentum-dashboard
|
||||
|
||||
[Install]
|
||||
WantedBy=default.target
|
||||
@@ -0,0 +1,24 @@
|
||||
[Unit]
|
||||
Description=Momentum live paper-trading loop
|
||||
# Self-heal: if the loop crashes or the box reboots, systemd brings it back.
|
||||
# The DB host (192.168.1.5) can go fully unreachable (a network/host outage) and
|
||||
# the loop crashes on connect — so retry FOREVER (no give-up limit) and it
|
||||
# auto-reconnects the moment the host returns. StartLimitIntervalSec=0 disables
|
||||
# the "give up after N starts" cap entirely.
|
||||
StartLimitIntervalSec=0
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
WorkingDirectory=/mnt/c/code/momentum
|
||||
# the loop calls load_dotenv() -> finds .env in WorkingDirectory -> DATABASE_URL
|
||||
# and all secrets load from there; no secrets in this unit file.
|
||||
ExecStart=/home/philip/.venvs/momentum/bin/python -u run_live_paper.py --no-dashboard
|
||||
Restart=always
|
||||
RestartSec=30
|
||||
# keep the same log file the operator + the review crons already tail
|
||||
StandardOutput=append:/mnt/c/code/momentum/data/live_run.log
|
||||
StandardError=append:/mnt/c/code/momentum/data/live_run.log
|
||||
SyslogIdentifier=momentum-loop
|
||||
|
||||
[Install]
|
||||
WantedBy=default.target
|
||||
@@ -0,0 +1,17 @@
|
||||
[Unit]
|
||||
Description=Momentum Telegram Q&A bot
|
||||
# retry forever through a DB-host outage (see momentum-loop.service)
|
||||
StartLimitIntervalSec=0
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
WorkingDirectory=/mnt/c/code/momentum
|
||||
ExecStart=/home/philip/.venvs/momentum/bin/python -u telegram_bot.py
|
||||
Restart=always
|
||||
RestartSec=30
|
||||
StandardOutput=append:/mnt/c/code/momentum/data/telegram_bot.log
|
||||
StandardError=append:/mnt/c/code/momentum/data/telegram_bot.log
|
||||
SyslogIdentifier=momentum-qabot
|
||||
|
||||
[Install]
|
||||
WantedBy=default.target
|
||||
@@ -0,0 +1,11 @@
|
||||
[Unit]
|
||||
Description=Local Postgres for momentum test isolation (socket /home/philip/pgrun)
|
||||
|
||||
[Service]
|
||||
Type=forking
|
||||
ExecStart=/home/philip/pg/bin/pg_ctl -D /home/philip/pgdata -o "-k /home/philip/pgrun" -l /home/philip/pgdata/logfile start
|
||||
ExecStop=/home/philip/pg/bin/pg_ctl -D /home/philip/pgdata stop -m fast
|
||||
Restart=on-failure
|
||||
|
||||
[Install]
|
||||
WantedBy=default.target
|
||||
@@ -0,0 +1,19 @@
|
||||
[Unit]
|
||||
Description=Momentum YouTube transcript -> claim-extraction pipeline
|
||||
StartLimitIntervalSec=0
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
WorkingDirectory=/mnt/c/code/momentum
|
||||
# secrets (DATABASE_URL, YOUTUBE_API_KEY, YT_WATCHLIST) load from the repo's gitignored .env
|
||||
# via youtube_claims.config (python-dotenv). Whisper is option A (CPU) by default; flip to the
|
||||
# LAN GPU (option B) with YT_WHISPER_MODEL=large-v3 YT_WHISPER_DEVICE=cuda YT_WHISPER_COMPUTE_TYPE=float16.
|
||||
ExecStart=/home/philip/.venvs/momentum/bin/python -u -m youtube_claims.run run
|
||||
Restart=always
|
||||
RestartSec=30
|
||||
StandardOutput=append:/mnt/c/code/momentum/data/transcripts_pipeline.log
|
||||
StandardError=append:/mnt/c/code/momentum/data/transcripts_pipeline.log
|
||||
SyslogIdentifier=momentum-transcripts
|
||||
|
||||
[Install]
|
||||
WantedBy=default.target
|
||||
Reference in new issue
Block a user