Commit Graph
26 Commits
Author SHA1 Message Date
momentum devandClaude Fable 5 ed176b3712 fix: backtest starvation — schedule the labeler nightly, make inserts idempotent
Dashboard backtest sources sessions from setup_events v1, which only the
labeler writes — and the labeler was never in the nightly scheduler, so
setups (and therefore backtests) froze at the last manual run (07-17).
Root blocker was _insert crashing on duplicate setup_ids on any re-run;
now ON CONFLICT DO NOTHING, so incremental nightly builds are safe.
Scheduler gains 'labeler build' + 'build-tod1' after eod_replay in both
compose files.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-22 06:51:56 -05:00
momentum devandClaude Fable 5 5597250ff1 fix: marketable entries price from live last + time-of-day rvol arms fast path
Day-22 forensic (BIYA session): 6 entry submissions, 0 fills — IREX missed
+1.7R, IRE +1.65R by one cent. Two root causes fixed:

1. Slow-path limit rested AT the trigger while price ran 2% past it, and
   retries reused the same stale price. Marketable model now prices from
   max(trigger, live last) via price_provider; the anti-chase extension
   gate gets the live ref too (was an entry_ref==trigger no-op on this
   path). TRADING_ENTRY_FILL_MODEL flipped to marketable.

2. Fast trigger path armed nothing all day: legacy rvol (cum-so-far /
   full-day MEAN) is ~0.1-0.3 minutes after the open for a normal heavy
   name and its mean is poisoned by reverse-split/prior-squeeze days
   (BIYA legacy 1.27 vs real activity 481x median). New rvol_tod =
   cum / (MEDIAN(20d) * elapsed-session fraction) drives arm eligibility
   and the scanner keep-filter; legacy rvol untouched for validated
   boundaries (gap-squeeze rvol>=5).

Also: stagnation time-stop (TRADING_EXIT_STAGNATION_MIN/_R, default OFF)
in simulate_exit + manage_live with sweep_exits.py variants for
validation before any live enable.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 15:40:15 -05:00
momentum devandClaude Fable 5 b3a47d775d feat(019): discovery on its own thread + tuner heartbeat (honest stale alert)
Telemetry: discovery took 6-30s ON the single scheduler, freezing ingest/arm
so the trigger read stale bars — the real 'late edge detection'. Now discovery
runs on its own daemon thread with a dedicated DB connection; the watchlist it
mutates is a lock-guarded Watchlist (list-like drop-in, no torn reads); the
scheduler discover step auto-disables (single-writer). Loop is now 3 fast
threads (entries/exits/discovery) + slow scheduler for tolerant work.

Also: nightly_tune writes a heartbeat every run, so the boot alert distinguishes
'tuner not running' (real) from 'ran, no profitable ORB config, kept params'
(expected — the strategy has no edge). Kills the misleading Telegram alarm.
543 tests green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-19 08:44:10 -05:00
momentum devandClaude Fable 5 5896d291d8 feat(018): sub-second exits on a dedicated thread + fast decision pipeline
Exits move off the single-threaded scheduler onto their own daemon thread at
1s (mirrors the fast trigger thread). Dedicated research connection (shared
facade isn't thread-safe); scheduler exits step auto-disabled when the thread
runs (single-caller invariant — no racing instance state, no double-submit);
manage() already idempotent (stop ratchets >1c up only). 3 driver tests + 539
suite green. On fast small-caps this is the 'get out fast' the owner asked for.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-19 08:32:46 -05:00
momentum devandClaude Fable 5 680ddc2174 loop: faster decision pipeline — ingest 20->5s, arm 20->8s, watch 15->8s, trigger 4->2s
Owner: 'make all decisions as fast as possible.' The real entry-detection
ceiling was ingest — bars refreshed every 20s, so the 4s trigger thread read
data up to 20s stale and detected breakouts late. Ingest is incremental+cheap,
safe at 5s. Fresher bars + faster arm (trigger levels) + 2s trigger check +
the 4s exits shipped earlier = the whole ingest->detect->decide->exit loop
tightened. One batched data call per cycle so Alpaca rate limits are fine.
Sub-second exit management on a dedicated thread is the next (code) step.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-18 22:04:29 -05:00
momentum devandClaude Fable 5 c3828eff15 exits: manage every 4s not 12s — kill the give-back on fast small-cap round-trips
Owner's real complaint: late exits gave back gains. Exits ran on the slow
scheduler at 12s while entries run on the 4s fast thread — exits were 3x
slower than entries. A small-cap can spike and round-trip inside 12s. Drop
TRADING_EXIT_MANAGE_INTERVAL_SECONDS to 4s (config, reversible, both compose
files). Deeper fix (exit mgmt on the sub-second trigger thread) planned
separately. Entry chase-tail (24% fill >5% above break) is the next
investigation.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-18 21:53:57 -05:00
momentum devandClaude Fable 5 92488b72bc youtube: YT_PLAYLIST_URLS — comma-separated playlists via env, synced at boot
One playlist per add-playlist CLI call made adding a second playlist a chore.
Now YT_PLAYLIST_URLS takes a comma-separated list of playlist URLs (list=
param extracted) or bare ids; the transcripts service seeds the playlists
table from it at boot (insert-if-missing — hand-set labels are never
clobbered), and `python -m youtube_claims.run sync-playlists` does the same
on demand. Documented in both compose files; 3 tests.

Also: queue the two executor-surfaced defects (OrderFilledEvent side,
/api/attribution env hardcode) + the flaky nightly regime build-features
step in the plans index; regime_features 07-16 backfilled supervised.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 09:14:22 -05:00
momentum devandClaude Opus 4.8 acdf0019c3 Regime gate: document TRADING_REGIME_GATE_P_MIN in both compose anchors
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-15 15:21:49 -05:00
momentum devandClaude Opus 4.8 0ac09d3574 Regime gate LIVE (plan 011): gate fresh entries on the session-level metalabel
Wires the regime signal into the money path — fresh entries are BLOCKED on
sessions the walk-forward model scores unfavorable. Session-level: evaluated
once per day, cached. Enabled in both compose templates (paper).

Correctness (blocker 1 resolved): the LIVE feature set drops tape_regime (the
only non-strictly-open feature — AND measured 0.000 importance) + breadth_gap
(also 0.000). The 5 remaining features are all non-look-ahead at entry time:
breadth_rvol = median(relative_volume=vol_upto/avg_vol_20d) over the session's
setups so far (the SAME up-to-trigger calc the live bot runs, NOT day-end),
spy_* lagged to prior close, trailing from PRIOR sessions. The live single-
session fit reproduces the batch meta_v1 verdict exactly on the last 6 sessions.

Safety: fail-open everywhere — disabled, unfit model (<60 prior sessions),
missing breadth_rvol (pre-setup), or ANY exception returns None and never
blocks. The gate can only SUBTRACT trades on bad-regime days.

- research/regime.py: live_session_favorable() (refit on priors <today, predict
  today; writes regime_scores['meta_live'] for forward validation) + LIVE_FEATURE_COLS
  + the breadth_rvol-missing fail-open guard.
- trading_execution.py: TRADING_REGIME_GATE_ENABLED / _P_MIN settings + _regime_reject()
  (cached per session) + a REGIME gate after SESSION in _evaluate_entry_gates.
- docker-compose.yml + truenas-compose.yml: TRADING_REGIME_GATE_ENABLED=1.
- tests/unit/test_regime_gate_live.py: block / pass / fail-open(x3) / scored-once-per-day.

KNOWN train/serve skew (documented, validate forward via meta_live vs EOD label):
model trains on full-day breadth_rvol, live uses setups-so-far. Small edge on a
~266-session sample; paper only. Suite: 504 passed, 15 pre-existing loopback fails.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-15 14:11:50 -05:00
momentum devandClaude Opus 4.8 092d6072e8 Regime gate (plan 011): deploy-wire the shadow pipeline + on-demand admin trigger + prod backtest
The session-level metalabel P(our ORB profits today) — the one lever that
survives out-of-sample. Backtested on 266 prod sessions (walk-forward):

  trade-always (trained subset): -0.111R/session, 36% hit
  meta_v1 GATED (trades best ~32% of days): -0.004R/session, 48% hit
  delta +0.107R in-sample, +0.128R OOS (recent 80 sessions), improves across folds

It does NOT make the bot profitable — it takes a bleeding strategy to roughly
flat by sitting out the worst ~2/3 of sessions. threshold_v1 rule gates too
little to matter; meta_v1 (walk-forward GBM) is the signal. Leaky full-fit
(+0.61R) >> walk-forward confirms the discipline was load-bearing; the OOS
holding is the evidence the +0.078R is real, not look-ahead.

Changes:
- research/regime.py: batch minute-bar load per session (64k per-fire round-trips
  -> ~266 queries; fixed a >5min build to ~3min) + incremental label pass (skip
  already-labeled sessions; --rebuild forces full). Nightly cost now ~one new day.
- docker-compose.yml + truenas-compose.yml: scheduler runs build-features + score
  (both models) before `experiments run`, nightly. ALL-SHADOW — does NOT gate live
  trading (that stays owner-gated Step 5).
- admin_api.py + admin.html: POST /api/regime/run (detached) + GET /api/regime/status
  + a Run/Status button — the panel had no way to trigger this.
- plans/011 + docs/research/regime_notes.md (6-book distillation) + plans 008/009 docs.

Prod state (shadow, rollback = DROP regime_features/regime_scores + DELETE
experiments WHERE id LIKE 'regime%'): 266 feature rows, 532 scores, 2 experiments
registered.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-15 14:05:34 -04:00
momentum devandClaude Fable 5 5b885a719f ingest(009): nightly SPY/QQQ ingest + catalyst enrichment ON as feature source
Step 3 of plan 009 (config + code; no live-path change):

- research/ingestion/market_context.py: pulls SPY/QQQ full-session minute bars
  (SIP, post-close embargo-safe) + daily bars via the existing upsert helpers, so
  the labeler's market-context features have a FORWARD source (index ETFs never
  enter the gapper universe the live loop ingests). Mirrors sip_backfill's shape.
- both compose scheduler blocks: run it after premarket_eval, guarded || echo.
- NEWS_ENRICH_ENABLED on: added to docker-compose x-app-env anchor (loop +
  scheduler inherit it); truenas-compose + .env.example already had it. Catalyst
  is a FEATURE source only, never a live gate; enabled-but-down never blocks
  trading. Enrichment OUTPUT is verified by the orchestrator (needs live Ollama).

Compose statically validated (yaml.safe_load). Ingest self-checked via a
fake-client :memory: round-trip through _preload_spy.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-14 20:27:49 -04:00
momentum devandClaude Fable 5 07570fb0e4 operator-plane: fix dashboard POST NameError (import os/hmac)
Every POST endpoint (approve/reject/exit/watch) crashed with NameError:
do_POST references os.environ + hmac.compare_digest but neither was imported.
Adds the imports, a non-loopback regression test for the token gate (no token
-> routes; token set -> 401 without/with-wrong header, 200 with correct), and
plumbs DASHBOARD_TOKEN + PG_SKIP_SCHEMA=1 into both compose dashboard services
(+ .env.example docs for DASHBOARD_TOKEN / COMMAND_BUS_ENABLED).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-14 00:04:01 -04:00
momentum devandClaude Fable 5 1036c19448 experiment-os: wire harness into both compose schedulers
Plan 006 step 5. python -m research.experiments run && adjudicate after
nightly_tune in both scheduler blocks (guarded with FAILED echoes). ALL-SHADOW:
emits verdicts, never touches live gates/sizing/exits.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-13 23:49:31 -04:00
momentum devandClaude Fable 5 3e2d690b21 flywheel: schedule eod_replay + nightly_tune in both compose scheduler blocks
The tuner was fiction in production: the 20:00 ET scheduler ran only
premarket_eval (+ eod_reconcile), so learned_params.json sat 26 days stale and
the whole-market dataset never grew. Both scheduler loops now run, in order,
after the eval/reconcile pass:

  python eod_replay.py    || echo "[scheduler] eod_replay FAILED"
  python nightly_tune.py  || echo "[scheduler] nightly_tune FAILED"

Order matters: replay grows the dataset first, the tuner fits after. 20:00 ET is
after close, so replaying "today" is complete data — NOT look-ahead (replay is a
labeling job, not a live gate). Deletes eod_replay.sh (dead cron-era wrapper
pointing at a nonexistent venv; compose is the runner).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-13 22:46:06 -04:00
momentum devandClaude Fable 5 23b3d01a00 money-path: nightly cage-count reconcile (broker vs event store) + schedule it
Plan 003 step 5 (completes plan 003). research/eod_reconcile.py diffs broker
truth (open positions, today's live orders) against event-store truth (today's
submits/fills/closes) and emits a loud eod_reconcile event + Telegram alert on
any drift — the duplicate-resting-order-after-restart class (2026-07-06) caught
automatically instead of by hand. Read-only (reports, never auto-fixes). Wired
into both scheduler blocks after premarket_eval. Pure diff core is unit-tested.

Suite: 373 passed (+22 plan-003 tests), 15 fail = WSL-loopback dashboard only.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-13 22:29:00 -04:00
momentum devandClaude Fable 5 caee1d5655 backup: nightly pg_dump service on the postgres:18 image
Production Postgres had NO automated backup (pg_dump appeared only in the
one-time migration script). New db-backup service in both compose files fires
20:10 ET Mon-Fri: pg_dump | gzip with pipefail + gunzip -t + tmp-then-rename
(a mid-stream death can't masquerade as a backup), prunes past
BACKUP_KEEP_DAYS, writes last_backup_ok on success. postgres:18 image =
version-matched client (app image has none). TrueNAS host is pull-only, so
its copy inlines the script. Restore: gunzip -c dump | psql $DATABASE_URL.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-13 16:39:26 -04:00
momentum devandClaude Fable 5 5b95a9712c qabot: dead-man's switch on the loop's event heartbeat
The loop's Telegram notifier runs inside the loop process — loop-down means
alerts-down. The always-up qabot now reads max(events.timestamp) every ~120s
(ET-naive vs ET wall clock per RUNBOOK): stale > DEADMAN_STALE_MIN during
market hours (holiday-aware via runtime.market_calendar) alerts at most every
30 min; DEADMAN_DAILY_PING sends one OK-ping after 20:30 ET. Injected
clock/read/send; 6 unit tests, no network or DB.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-13 16:37:01 -04:00
momentum devandClaude Opus 4.8 cad9051a96 truenas template: complete 86-key env block (kills env-drift class)
Systematic diff of the old .env against the deployed loop env found 68
code-read keys running on silent defaults — including the risk cap
($500 vs $868 observed), the VWAP gate, the exit ladder, and the
NEWS_ENRICH_ENABLED flag that kept the catalyst engine dead despite
OLLAMA_ENABLED=true (enrichment requires BOTH, config.py:1235 gate).
Template env block regenerated with the full inventory, secrets as
<FILL-ME>; a ready-to-paste block with real values generated locally
(data/loop-env-block.yaml, gitignored).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 12:26:25 -04:00
momentum devandClaude Opus 4.8 5ac2cc7b89 Docs cleanup: current-state rewrite + three deployment modes + history/
- 10 era documents (phase plans, build summaries, redesign writeups, old
  runbooks, completed consolidation plan) moved to docs/history/ with a
  do-not-follow README.
- Root README: current validated population ($2-20, 10-35% static gap +
  squeeze exception, SIP), Deployment section, automated-maintenance
  reality, corrected repo map (admin panel, youtube_claims, deploy/).
- RUNBOOK rewritten for the running system: daily rhythm, health checks
  per deployment, and the production failure-mode table earned this week
  (env drift, pgdata mount traps, uid 10001, zombie approvals, PG_SKIP_
  SCHEMA, timezone artifacts).
- deploy/README.md is now the three-mode index: compose (canonical),
  systemd (local dev — unit templates now committed), helm (refresh
  staged as issue #8). Old k8s doc moved to deploy/helm/README.md.
- docker/README.md: six TrueNAS gotchas, each from a real outage.
- Admin panel improvements staged as GitHub issues #4-#7.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-10 10:56:53 -04:00
momentum devandClaude Opus 4.8 befcec2965 Post-migration DB health: healthcheck DDL storm + broken-pipe fixes
The loop healthcheck opened a fresh connection every 60s with schema
auto-apply enabled; the CREATE-IF-NOT-EXISTS lock bursts cancelled
minute_bars autovacuum every single cycle (visible as 'canceling
autovacuum task' each minute post-migration), so the 19M-row table
never got analyzed, snapshot queries crawled, browsers timed out, and
the dashboard log filled with double BrokenPipeError tracebacks.

- healthchecks now set PG_SKIP_SCHEMA=1 (both compose files)
- api/main.py _send_json swallows BrokenPipe/ConnectionReset (client
  hung up; nothing to tell it)
- live DB: ALTER SYSTEM max_wal_size=4GB, shared_buffers=1GB,
  effective_cache_size=3GB, checkpoint_completion_target=0.9 (the
  checkpoint-every-24s warning under migration load on NAS storage)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-09 18:41:12 -04:00
momentum devandClaude Opus 4.8 f0fb91a069 Admin panel (:8767): config, profiles, service control, paper/live switch
New admin service — LAN-only, token-authed (ADMIN_TOKEN), card-based UI
matching the owner's preferred dashboard style. Capabilities:
- health: docker containers via /var/run/docker.sock (stdlib unix-socket
  HTTP client), DB heartbeat, Alpaca account/clock, Ollama, feed
- config: whitelisted strategy knobs (TRADING_/TRIGGER_/WATCHER_/...)
  edited via data/overrides.env, which every entrypoint now loads with
  override=True at boot (runtime/env_overrides.py) — so changes apply on
  container restart without touching the deployment YAML; secrets are
  presence-only and rejected server-side if edited
- profiles: named knob presets (seeded: validated-core, conservative,
  squeeze-hunter) applied in one click
- restart buttons per container; paper/live switch guarded behind
  ALPACA_LIVE_* keys existing
Verified live in docker: auth 401s, status/config/profiles/write paths,
secret-edit rejection, page serves. Added to both compose files.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-09 18:37:18 -04:00
momentum devandClaude Opus 4.8 f4198cda5c Publish image to Docker Hub too; TrueNAS yaml pulls from docker.io
CI now pushes fez5587/momentum-app to Docker Hub alongside GHCR (creds
via repo secrets). First push done manually and verified anonymously
pullable.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-09 15:34:13 -04:00
momentum devandClaude Opus 4.8 7427d14301 TrueNAS migration: dataset paths, 5433 coexistence port, dump/restore script
truenas-compose.yml now uses /mnt/volume/Apps/momentum and publishes the
new postgres on 5433 so it can coexist with the existing instance during
migration. deploy/docker/migrate_db.sh: dump / restore / verify steps
(pg_dump -Fc read-only source, parallel pg_restore, ANALYZE, row-count
diff) with env-overridable URLs and PG18 client autodetection —
connectivity smoke-tested against the live DB.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-09 14:21:21 -04:00
momentum devandClaude Opus 4.8 e563380f28 TrueNAS SCALE deployment: GHCR image publishing + pull-only compose
TrueNAS custom apps cannot build: the main compose's 'build: .' (plus
relative .env/bind mounts) fails app 'up'. Added a GitHub Actions
workflow publishing ghcr.io/fez5587/momentum-app on every master push,
and deploy/docker/truenas-compose.yml — a pull-only variant with
dataset host-paths, inline environment (secrets as <FILL-ME>
placeholders, current validated strategy knobs pre-filled), and the
same healthchecks. Validated with docker compose config.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-09 13:55:24 -04:00
momentum devandClaude Opus 4.8 8ba4899108 Sanitized .env.example regenerated + config-layering guide
.env.example was stale (pre-migration keys, missing all current
TRADING_*/TRIGGER_* knobs). Regenerated from the live config with
secrets <REDACTED>; a leak check caught live Alpaca credentials
embedded in commented-out PowerShell lines and scrubbed them before
commit. README now documents the three config layers (protected
secrets vs promotable strategy knobs vs infra addresses) and the
external-Ollama networking expectations.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-09 13:41:40 -04:00
momentum devandClaude Opus 4.8 8864041066 Docker stack modernized: full service set, PG 18, validated end-to-end
Brings the pre-migration compose file up to the current architecture:
python 3.13 image with ffmpeg + faster-whisper/yt-dlp (transcripts),
postgres 16 -> 18 (prod parity), and the real service set — loop
(--no-dashboard, heartbeat healthcheck), dashboard_api.py on :8765,
telegram qabot, youtube transcripts, and a 20:00-ET weekday scheduler
replacing the premarket-eval systemd timer. Jobs (journal/eod-replay/
nightly-tune) and Grafana keep their profiles. Self-contained by
default (in-stack DATABASE_URL); deploy/docker/README.md documents the
pg_dump migration, the transition mode against the LAN DB, and the
cutover checklist. Validated: image builds, compose config passes, and
a smoke container served the dashboard (HTTP 200) against the live
LAN Postgres.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-09 13:06:27 -04:00