Post-migration DB health: healthcheck DDL storm + broken-pipe fixes
The loop healthcheck opened a fresh connection every 60s with schema auto-apply enabled; the CREATE-IF-NOT-EXISTS lock bursts cancelled minute_bars autovacuum every single cycle (visible as 'canceling autovacuum task' each minute post-migration), so the 19M-row table never got analyzed, snapshot queries crawled, browsers timed out, and the dashboard log filled with double BrokenPipeError tracebacks. - healthchecks now set PG_SKIP_SCHEMA=1 (both compose files) - api/main.py _send_json swallows BrokenPipe/ConnectionReset (client hung up; nothing to tell it) - live DB: ALTER SYSTEM max_wal_size=4GB, shared_buffers=1GB, effective_cache_size=3GB, checkpoint_completion_target=0.9 (the checkpoint-every-24s warning under migration load on NAS storage) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
1 parent
f0fb91a069
commit
befcec2965
3 files changed
+19
-8
No files matched your search
@@ -78,7 +78,10 @@ services:
|
||||
<<: *app
|
||||
command: ["python", "-u", "run_live_paper.py", "--no-dashboard"]
|
||||
healthcheck:
|
||||
test: ["CMD", "python", "-c", "import datetime,sys; from research.multi_schema import open_research_db; r=open_research_db('market').execute('SELECT max(timestamp) FROM events').fetchall()[0][0]; sys.exit(0 if r and (datetime.datetime.now()-r).total_seconds()<300 else 1)"]
|
||||
# PG_SKIP_SCHEMA=1: a healthcheck must NOT re-apply schema DDL — the 60s
|
||||
# CREATE-IF-NOT-EXISTS lock burst cancelled minute_bars autovacuum forever
|
||||
# (2026-07-09 post-migration: stats rotted, dashboard queries crawled)
|
||||
test: ["CMD", "python", "-c", "import os; os.environ['PG_SKIP_SCHEMA']='1'; import datetime,sys; from research.multi_schema import open_research_db; r=open_research_db('market').execute('SELECT max(timestamp) FROM events').fetchall()[0][0]; sys.exit(0 if r and (datetime.datetime.now()-r).total_seconds()<300 else 1)"]
|
||||
interval: 60s
|
||||
timeout: 20s
|
||||
retries: 5
|
||||
|
||||
Reference in new issue
Block a user