From 4173acc9d5de1579202eeb0e826f22a73e67549d Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 29 Jun 2026 21:35:38 +0000 Subject: [PATCH] docs: add future-hardening roadmap to deploy README Document reliability, security, and CI/operational improvements for the Kubernetes package as an actionable backlog (not yet implemented). Notably flags the loop-heartbeat liveness gap: /api/health proves only the dashboard thread is alive, so a hung trading loop would go undetected. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01JCVMm2AbmZvvxwLTX6UpNR --- deploy/README.md | 48 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) diff --git a/deploy/README.md b/deploy/README.md index 829189e..c87dd7f 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -95,3 +95,51 @@ kubectl -n momentum create job --from=cronjob/momentum-nightly-tune tune-test - **CronJob `spec.timeZone`** requires Kubernetes ≥ 1.27. - The legacy `grafana/docker-compose.yml` is left in place for local use; the chart provisions its own Grafana against the in-cluster Postgres. + +## Future hardening (roadmap) + +The package deploys and runs, but the items below separate "it comes up" from +"runs reliably for a trading system." They are **not implemented yet** — captured +here so they can be picked up later. Each notes what, why, and the rough approach. + +### Reliability + +- **Loop-heartbeat liveness** *(needs an app-code change).* Today `/api/health` + (`api/main.py:343`) returns a static `{"ok": true}`, which only proves the + dashboard *thread* is alive. The trading loop runs in the main thread + (`run_live_paper.py:328`), so a **crashed** loop exits the process and Kubernetes + restarts it (fine), but a **hung** loop (stuck network call, deadlock) keeps + answering health checks `200` and runs undetected. Approach: have the loop stamp a + last-tick timestamp into `DashboardState` each iteration, make `/api/health` (or a + new `/healthz`) return `503` when the last tick is older than a threshold, and + point the Deployment's `livenessProbe` at it. Highest-value item. +- **Postgres backup CronJob.** The event store is the source of truth and nothing + backs it up. Approach: a chart CronJob running `pg_dump` (postgres image, creds + from the Secret) to the app `data` PVC with N-day retention; optionally offload to + object storage. +- **Schema bootstrap hook.** The schema auto-applies on the app's first connection, + so a first-run race between the app and a CronJob is possible. Approach: a + `pre-install`/`pre-upgrade` Helm-hook Job that applies `storage/pg_schema.sql` + deterministically, with `PG_SKIP_SCHEMA=1` set on the app and CronJobs. + +### Security + +- **Pod hardening.** Add a `securityContext` across workloads: `runAsNonRoot`, drop + ALL capabilities, `seccompProfile: RuntimeDefault`, and `readOnlyRootFilesystem` + where feasible (the app needs `/app/data` writable — already a mounted volume). +- **NetworkPolicy.** Default-deny in the namespace; allow only the app, CronJobs, and + Grafana to reach Postgres:5432, and restrict who can reach the dashboard. +- **Grafana auth.** The chart inherits anonymous-admin from the original compose + (`templates/grafana.yaml` env). Make it off by default in-cluster, with a real + admin password sourced from the Secret (`GF_SECURITY_ADMIN_PASSWORD`). + +### CI / operational + +- **Chart CI validation.** Add a CI job running `helm lint` + `helm template` + + `kubeconform` on every push — catches template/schema errors automatically (the + chart could not be rendered in the sandbox where it was authored, since image-blob + CDNs were egress-blocked). +- **Image digest pinning.** For production installs, pin `image` by SHA digest rather + than the mutable `latest` tag. +- **PodDisruptionBudget / singleton guard.** A PDB for the singleton loop plus a + guard so it is never scaled above `1` or evicted to zero unexpectedly.