Post-migration DB health: healthcheck DDL storm + broken-pipe fixes

The loop healthcheck opened a fresh connection every 60s with schema
auto-apply enabled; the CREATE-IF-NOT-EXISTS lock bursts cancelled
minute_bars autovacuum every single cycle (visible as 'canceling
autovacuum task' each minute post-migration), so the 19M-row table
never got analyzed, snapshot queries crawled, browsers timed out, and
the dashboard log filled with double BrokenPipeError tracebacks.

- healthchecks now set PG_SKIP_SCHEMA=1 (both compose files)
- api/main.py _send_json swallows BrokenPipe/ConnectionReset (client
  hung up; nothing to tell it)
- live DB: ALTER SYSTEM max_wal_size=4GB, shared_buffers=1GB,
  effective_cache_size=3GB, checkpoint_completion_target=0.9 (the
  checkpoint-every-24s warning under migration load on NAS storage)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
momentum devandClaude Opus 4.8 committed 2026-07-09 18:41:12 -04:00
1 parent f0fb91a069
commit befcec2965
3 files changed
+19 -8

No files matched your search

+4 -1
View File
@@ -78,7 +78,10 @@ services:
<<: *app
command: ["python", "-u", "run_live_paper.py", "--no-dashboard"]
healthcheck:
test: ["CMD", "python", "-c", "import datetime,sys; from research.multi_schema import open_research_db; r=open_research_db('market').execute('SELECT max(timestamp) FROM events').fetchall()[0][0]; sys.exit(0 if r and (datetime.datetime.now()-r).total_seconds()<300 else 1)"]
# PG_SKIP_SCHEMA=1: a healthcheck must NOT re-apply schema DDL — the 60s
# CREATE-IF-NOT-EXISTS lock burst cancelled minute_bars autovacuum forever
# (2026-07-09 post-migration: stats rotted, dashboard queries crawled)
test: ["CMD", "python", "-c", "import os; os.environ['PG_SKIP_SCHEMA']='1'; import datetime,sys; from research.multi_schema import open_research_db; r=open_research_db('market').execute('SELECT max(timestamp) FROM events').fetchall()[0][0]; sys.exit(0 if r and (datetime.datetime.now()-r).total_seconds()<300 else 1)"]
interval: 60s
timeout: 20s
retries: 5