Dashboard backtest sources sessions from setup_events v1, which only the
labeler writes — and the labeler was never in the nightly scheduler, so
setups (and therefore backtests) froze at the last manual run (07-17).
Root blocker was _insert crashing on duplicate setup_ids on any re-run;
now ON CONFLICT DO NOTHING, so incremental nightly builds are safe.
Scheduler gains 'labeler build' + 'build-tod1' after eod_replay in both
compose files.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Day-22 forensic (BIYA session): 6 entry submissions, 0 fills — IREX missed
+1.7R, IRE +1.65R by one cent. Two root causes fixed:
1. Slow-path limit rested AT the trigger while price ran 2% past it, and
retries reused the same stale price. Marketable model now prices from
max(trigger, live last) via price_provider; the anti-chase extension
gate gets the live ref too (was an entry_ref==trigger no-op on this
path). TRADING_ENTRY_FILL_MODEL flipped to marketable.
2. Fast trigger path armed nothing all day: legacy rvol (cum-so-far /
full-day MEAN) is ~0.1-0.3 minutes after the open for a normal heavy
name and its mean is poisoned by reverse-split/prior-squeeze days
(BIYA legacy 1.27 vs real activity 481x median). New rvol_tod =
cum / (MEDIAN(20d) * elapsed-session fraction) drives arm eligibility
and the scanner keep-filter; legacy rvol untouched for validated
boundaries (gap-squeeze rvol>=5).
Also: stagnation time-stop (TRADING_EXIT_STAGNATION_MIN/_R, default OFF)
in simulate_exit + manage_live with sweep_exits.py variants for
validation before any live enable.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Telemetry: discovery took 6-30s ON the single scheduler, freezing ingest/arm
so the trigger read stale bars — the real 'late edge detection'. Now discovery
runs on its own daemon thread with a dedicated DB connection; the watchlist it
mutates is a lock-guarded Watchlist (list-like drop-in, no torn reads); the
scheduler discover step auto-disables (single-writer). Loop is now 3 fast
threads (entries/exits/discovery) + slow scheduler for tolerant work.
Also: nightly_tune writes a heartbeat every run, so the boot alert distinguishes
'tuner not running' (real) from 'ran, no profitable ORB config, kept params'
(expected — the strategy has no edge). Kills the misleading Telegram alarm.
543 tests green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Exits move off the single-threaded scheduler onto their own daemon thread at
1s (mirrors the fast trigger thread). Dedicated research connection (shared
facade isn't thread-safe); scheduler exits step auto-disabled when the thread
runs (single-caller invariant — no racing instance state, no double-submit);
manage() already idempotent (stop ratchets >1c up only). 3 driver tests + 539
suite green. On fast small-caps this is the 'get out fast' the owner asked for.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Owner: 'make all decisions as fast as possible.' The real entry-detection
ceiling was ingest — bars refreshed every 20s, so the 4s trigger thread read
data up to 20s stale and detected breakouts late. Ingest is incremental+cheap,
safe at 5s. Fresher bars + faster arm (trigger levels) + 2s trigger check +
the 4s exits shipped earlier = the whole ingest->detect->decide->exit loop
tightened. One batched data call per cycle so Alpaca rate limits are fine.
Sub-second exit management on a dedicated thread is the next (code) step.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Owner's real complaint: late exits gave back gains. Exits ran on the slow
scheduler at 12s while entries run on the 4s fast thread — exits were 3x
slower than entries. A small-cap can spike and round-trip inside 12s. Drop
TRADING_EXIT_MANAGE_INTERVAL_SECONDS to 4s (config, reversible, both compose
files). Deeper fix (exit mgmt on the sub-second trigger thread) planned
separately. Entry chase-tail (24% fill >5% above break) is the next
investigation.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
One playlist per add-playlist CLI call made adding a second playlist a chore.
Now YT_PLAYLIST_URLS takes a comma-separated list of playlist URLs (list=
param extracted) or bare ids; the transcripts service seeds the playlists
table from it at boot (insert-if-missing — hand-set labels are never
clobbered), and `python -m youtube_claims.run sync-playlists` does the same
on demand. Documented in both compose files; 3 tests.
Also: queue the two executor-surfaced defects (OrderFilledEvent side,
/api/attribution env hardcode) + the flaky nightly regime build-features
step in the plans index; regime_features 07-16 backfilled supervised.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Wires the regime signal into the money path — fresh entries are BLOCKED on
sessions the walk-forward model scores unfavorable. Session-level: evaluated
once per day, cached. Enabled in both compose templates (paper).
Correctness (blocker 1 resolved): the LIVE feature set drops tape_regime (the
only non-strictly-open feature — AND measured 0.000 importance) + breadth_gap
(also 0.000). The 5 remaining features are all non-look-ahead at entry time:
breadth_rvol = median(relative_volume=vol_upto/avg_vol_20d) over the session's
setups so far (the SAME up-to-trigger calc the live bot runs, NOT day-end),
spy_* lagged to prior close, trailing from PRIOR sessions. The live single-
session fit reproduces the batch meta_v1 verdict exactly on the last 6 sessions.
Safety: fail-open everywhere — disabled, unfit model (<60 prior sessions),
missing breadth_rvol (pre-setup), or ANY exception returns None and never
blocks. The gate can only SUBTRACT trades on bad-regime days.
- research/regime.py: live_session_favorable() (refit on priors <today, predict
today; writes regime_scores['meta_live'] for forward validation) + LIVE_FEATURE_COLS
+ the breadth_rvol-missing fail-open guard.
- trading_execution.py: TRADING_REGIME_GATE_ENABLED / _P_MIN settings + _regime_reject()
(cached per session) + a REGIME gate after SESSION in _evaluate_entry_gates.
- docker-compose.yml + truenas-compose.yml: TRADING_REGIME_GATE_ENABLED=1.
- tests/unit/test_regime_gate_live.py: block / pass / fail-open(x3) / scored-once-per-day.
KNOWN train/serve skew (documented, validate forward via meta_live vs EOD label):
model trains on full-day breadth_rvol, live uses setups-so-far. Small edge on a
~266-session sample; paper only. Suite: 504 passed, 15 pre-existing loopback fails.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The session-level metalabel P(our ORB profits today) — the one lever that
survives out-of-sample. Backtested on 266 prod sessions (walk-forward):
trade-always (trained subset): -0.111R/session, 36% hit
meta_v1 GATED (trades best ~32% of days): -0.004R/session, 48% hit
delta +0.107R in-sample, +0.128R OOS (recent 80 sessions), improves across folds
It does NOT make the bot profitable — it takes a bleeding strategy to roughly
flat by sitting out the worst ~2/3 of sessions. threshold_v1 rule gates too
little to matter; meta_v1 (walk-forward GBM) is the signal. Leaky full-fit
(+0.61R) >> walk-forward confirms the discipline was load-bearing; the OOS
holding is the evidence the +0.078R is real, not look-ahead.
Changes:
- research/regime.py: batch minute-bar load per session (64k per-fire round-trips
-> ~266 queries; fixed a >5min build to ~3min) + incremental label pass (skip
already-labeled sessions; --rebuild forces full). Nightly cost now ~one new day.
- docker-compose.yml + truenas-compose.yml: scheduler runs build-features + score
(both models) before `experiments run`, nightly. ALL-SHADOW — does NOT gate live
trading (that stays owner-gated Step 5).
- admin_api.py + admin.html: POST /api/regime/run (detached) + GET /api/regime/status
+ a Run/Status button — the panel had no way to trigger this.
- plans/011 + docs/research/regime_notes.md (6-book distillation) + plans 008/009 docs.
Prod state (shadow, rollback = DROP regime_features/regime_scores + DELETE
experiments WHERE id LIKE 'regime%'): 266 feature rows, 532 scores, 2 experiments
registered.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Step 3 of plan 009 (config + code; no live-path change):
- research/ingestion/market_context.py: pulls SPY/QQQ full-session minute bars
(SIP, post-close embargo-safe) + daily bars via the existing upsert helpers, so
the labeler's market-context features have a FORWARD source (index ETFs never
enter the gapper universe the live loop ingests). Mirrors sip_backfill's shape.
- both compose scheduler blocks: run it after premarket_eval, guarded || echo.
- NEWS_ENRICH_ENABLED on: added to docker-compose x-app-env anchor (loop +
scheduler inherit it); truenas-compose + .env.example already had it. Catalyst
is a FEATURE source only, never a live gate; enabled-but-down never blocks
trading. Enrichment OUTPUT is verified by the orchestrator (needs live Ollama).
Compose statically validated (yaml.safe_load). Ingest self-checked via a
fake-client :memory: round-trip through _preload_spy.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every POST endpoint (approve/reject/exit/watch) crashed with NameError:
do_POST references os.environ + hmac.compare_digest but neither was imported.
Adds the imports, a non-loopback regression test for the token gate (no token
-> routes; token set -> 401 without/with-wrong header, 200 with correct), and
plumbs DASHBOARD_TOKEN + PG_SKIP_SCHEMA=1 into both compose dashboard services
(+ .env.example docs for DASHBOARD_TOKEN / COMMAND_BUS_ENABLED).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Plan 006 step 5. python -m research.experiments run && adjudicate after
nightly_tune in both scheduler blocks (guarded with FAILED echoes). ALL-SHADOW:
emits verdicts, never touches live gates/sizing/exits.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The tuner was fiction in production: the 20:00 ET scheduler ran only
premarket_eval (+ eod_reconcile), so learned_params.json sat 26 days stale and
the whole-market dataset never grew. Both scheduler loops now run, in order,
after the eval/reconcile pass:
python eod_replay.py || echo "[scheduler] eod_replay FAILED"
python nightly_tune.py || echo "[scheduler] nightly_tune FAILED"
Order matters: replay grows the dataset first, the tuner fits after. 20:00 ET is
after close, so replaying "today" is complete data — NOT look-ahead (replay is a
labeling job, not a live gate). Deletes eod_replay.sh (dead cron-era wrapper
pointing at a nonexistent venv; compose is the runner).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Plan 003 step 5 (completes plan 003). research/eod_reconcile.py diffs broker
truth (open positions, today's live orders) against event-store truth (today's
submits/fills/closes) and emits a loud eod_reconcile event + Telegram alert on
any drift — the duplicate-resting-order-after-restart class (2026-07-06) caught
automatically instead of by hand. Read-only (reports, never auto-fixes). Wired
into both scheduler blocks after premarket_eval. Pure diff core is unit-tested.
Suite: 373 passed (+22 plan-003 tests), 15 fail = WSL-loopback dashboard only.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Production Postgres had NO automated backup (pg_dump appeared only in the
one-time migration script). New db-backup service in both compose files fires
20:10 ET Mon-Fri: pg_dump | gzip with pipefail + gunzip -t + tmp-then-rename
(a mid-stream death can't masquerade as a backup), prunes past
BACKUP_KEEP_DAYS, writes last_backup_ok on success. postgres:18 image =
version-matched client (app image has none). TrueNAS host is pull-only, so
its copy inlines the script. Restore: gunzip -c dump | psql $DATABASE_URL.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The loop's Telegram notifier runs inside the loop process — loop-down means
alerts-down. The always-up qabot now reads max(events.timestamp) every ~120s
(ET-naive vs ET wall clock per RUNBOOK): stale > DEADMAN_STALE_MIN during
market hours (holiday-aware via runtime.market_calendar) alerts at most every
30 min; DEADMAN_DAILY_PING sends one OK-ping after 20:30 ET. Injected
clock/read/send; 6 unit tests, no network or DB.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Systematic diff of the old .env against the deployed loop env found 68
code-read keys running on silent defaults — including the risk cap
($500 vs $868 observed), the VWAP gate, the exit ladder, and the
NEWS_ENRICH_ENABLED flag that kept the catalyst engine dead despite
OLLAMA_ENABLED=true (enrichment requires BOTH, config.py:1235 gate).
Template env block regenerated with the full inventory, secrets as
<FILL-ME>; a ready-to-paste block with real values generated locally
(data/loop-env-block.yaml, gitignored).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- 10 era documents (phase plans, build summaries, redesign writeups, old
runbooks, completed consolidation plan) moved to docs/history/ with a
do-not-follow README.
- Root README: current validated population ($2-20, 10-35% static gap +
squeeze exception, SIP), Deployment section, automated-maintenance
reality, corrected repo map (admin panel, youtube_claims, deploy/).
- RUNBOOK rewritten for the running system: daily rhythm, health checks
per deployment, and the production failure-mode table earned this week
(env drift, pgdata mount traps, uid 10001, zombie approvals, PG_SKIP_
SCHEMA, timezone artifacts).
- deploy/README.md is now the three-mode index: compose (canonical),
systemd (local dev — unit templates now committed), helm (refresh
staged as issue #8). Old k8s doc moved to deploy/helm/README.md.
- docker/README.md: six TrueNAS gotchas, each from a real outage.
- Admin panel improvements staged as GitHub issues #4-#7.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The loop healthcheck opened a fresh connection every 60s with schema
auto-apply enabled; the CREATE-IF-NOT-EXISTS lock bursts cancelled
minute_bars autovacuum every single cycle (visible as 'canceling
autovacuum task' each minute post-migration), so the 19M-row table
never got analyzed, snapshot queries crawled, browsers timed out, and
the dashboard log filled with double BrokenPipeError tracebacks.
- healthchecks now set PG_SKIP_SCHEMA=1 (both compose files)
- api/main.py _send_json swallows BrokenPipe/ConnectionReset (client
hung up; nothing to tell it)
- live DB: ALTER SYSTEM max_wal_size=4GB, shared_buffers=1GB,
effective_cache_size=3GB, checkpoint_completion_target=0.9 (the
checkpoint-every-24s warning under migration load on NAS storage)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
New admin service — LAN-only, token-authed (ADMIN_TOKEN), card-based UI
matching the owner's preferred dashboard style. Capabilities:
- health: docker containers via /var/run/docker.sock (stdlib unix-socket
HTTP client), DB heartbeat, Alpaca account/clock, Ollama, feed
- config: whitelisted strategy knobs (TRADING_/TRIGGER_/WATCHER_/...)
edited via data/overrides.env, which every entrypoint now loads with
override=True at boot (runtime/env_overrides.py) — so changes apply on
container restart without touching the deployment YAML; secrets are
presence-only and rejected server-side if edited
- profiles: named knob presets (seeded: validated-core, conservative,
squeeze-hunter) applied in one click
- restart buttons per container; paper/live switch guarded behind
ALPACA_LIVE_* keys existing
Verified live in docker: auth 401s, status/config/profiles/write paths,
secret-edit rejection, page serves. Added to both compose files.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
CI now pushes fez5587/momentum-app to Docker Hub alongside GHCR (creds
via repo secrets). First push done manually and verified anonymously
pullable.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
truenas-compose.yml now uses /mnt/volume/Apps/momentum and publishes the
new postgres on 5433 so it can coexist with the existing instance during
migration. deploy/docker/migrate_db.sh: dump / restore / verify steps
(pg_dump -Fc read-only source, parallel pg_restore, ANALYZE, row-count
diff) with env-overridable URLs and PG18 client autodetection —
connectivity smoke-tested against the live DB.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
TrueNAS custom apps cannot build: the main compose's 'build: .' (plus
relative .env/bind mounts) fails app 'up'. Added a GitHub Actions
workflow publishing ghcr.io/fez5587/momentum-app on every master push,
and deploy/docker/truenas-compose.yml — a pull-only variant with
dataset host-paths, inline environment (secrets as <FILL-ME>
placeholders, current validated strategy knobs pre-filled), and the
same healthchecks. Validated with docker compose config.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
.env.example was stale (pre-migration keys, missing all current
TRADING_*/TRIGGER_* knobs). Regenerated from the live config with
secrets <REDACTED>; a leak check caught live Alpaca credentials
embedded in commented-out PowerShell lines and scrubbed them before
commit. README now documents the three config layers (protected
secrets vs promotable strategy knobs vs infra addresses) and the
external-Ollama networking expectations.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Brings the pre-migration compose file up to the current architecture:
python 3.13 image with ffmpeg + faster-whisper/yt-dlp (transcripts),
postgres 16 -> 18 (prod parity), and the real service set — loop
(--no-dashboard, heartbeat healthcheck), dashboard_api.py on :8765,
telegram qabot, youtube transcripts, and a 20:00-ET weekday scheduler
replacing the premarket-eval systemd timer. Jobs (journal/eod-replay/
nightly-tune) and Grafana keep their profiles. Self-contained by
default (in-stack DATABASE_URL); deploy/docker/README.md documents the
pg_dump migration, the transition mode against the LAN DB, and the
cutover checklist. Validated: image builds, compose config passes, and
a smoke container served the dashboard (HTTP 200) against the live
LAN Postgres.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>