diff --git a/docs/CONSOLIDATION_PLAN.md b/docs/CONSOLIDATION_PLAN.md new file mode 100644 index 0000000..d46167d --- /dev/null +++ b/docs/CONSOLIDATION_PLAN.md @@ -0,0 +1,177 @@ +# Momentum consolidation plan + +_Scope-only plan produced 2026-06-23 by a read-only 4-agent mapping workflow (wf_3fb5d523). No code was changed. Items are for operator approval; execute in P0→P2 order._ + +## TL;DR + +The single-Postgres migration is functionally COMPLETE at the connection layer (all live code flows through storage/db.py -> db_pg -> one DATABASE_URL Postgres; no live duckdb.connect exists), so the datastore work is cleanup, not architecture. The genuine open architectural item is the two entry chokepoints — but the maps are slightly stale in our favor: the VWAP gate (commit e3ac89c) and daily-entry cap (ac1ac50) are now ALSO on the slow/auto path, so the remaining fast-only gaps are narrower than stated: extension, day-extension, halt, liquidity sizing, and rank-concentration. The whole entry-path effort hinges on one unverified empirical claim — whether live fills originate from the fast/trigger path (which has all gates) or the slow auto path (which lacks the five above) — so the plan opens by PROVING that via the existing approved_by tag ('auto' vs 'orb_trigger'), since it determines whether those gaps are live or theoretical. Sequencing favors reversibility: P0 = the empirical fill-source check plus the zero-risk, zero-blast-radius wins (delete the untracked gitignored archive/ tree and 232MB zip; drop the orphaned mem_ test schema confirming the known PG-isolation landmine and close its recurrence). P1 = behaviour-neutral cleanups (delete the dead core/ and telemetry/ packages; remove vestigial .duckdb path-strings so a file backend can't be reintroduced; archive the two frozen pre-cutover .duckdb files AFTER a pg_dump; extract the fast path into a single _place_entry helper as a pure refactor). P2 = the higher-blast changes done last and behind a flag (enrich the signal_ready snapshot with live price/day_open/volume/rank — the structural root cause of the dead slow-path gates; route approve_order through the unified helper and choose ONE fill model; the Schwab delete-vs-repair product decision; and verifying the symbols/telemetry_events empty tables aren't silently-dead writers). Throughout, the non-negotiables are: keep every shared-state mutation (self._fresh_entries, self._armed, self._requested_symbols, self._exited_today, self._cooldown_until) inside @_locked methods, never touch the naked-stop protections in expire_stale_entries, preserve the auto-vs-manual approval exemption, and never block an entry on missing data (fail-open like the existing above_vwap handling). Key files: trading_execution.py (entry chokepoints + shared state), run_live_paper.py (fast-path call site + boot warning), storage/projections.py (ready-signal snapshot), research/schema.py + research/multi_schema.py (vestigial duckdb labels), storage/db_pg.py + tests/conftest.py (mem_ schema isolation), storage/pg_schema.sql (orphaned tables). + + +> **Safety note (answered):** the P0 'which path produces fills' question is already settled — on 2026-06-23 **47/47 live entries were `approved_by="auto"` (the slow path)**. So the fast-path-only gates the map flags as missing on the live path — **extension / day-extension / halt-guard / liquidity-sizing / rank** — are a REAL current gap, not theoretical. (VWAP gate + daily-cap are now on both paths as of today's commits.) + + +## Prioritized plan + +### P0 — Confirm empirically which entry path produces live fills (instrumentation only, no logic change) +- **blast radius:** low | **effort:** small +- **action:** Before touching any gate, tag every fill with its origin: the slow/auto path already submits approve_order(approved_by='auto') (trading_execution.py:1037) and the fast path submits approve_order(approved_by='orb_trigger') (:970). Add a single grep/SQL over the live events table (events filtered to order-submit / approval events) for the last N trading days, bucketed by approved_by, to prove whether real fills originate from 'auto' (slow) or 'orb_trigger' (fast). Record the result in the plan ticket. Do NOT change behaviour. +- **why:** The entire safety argument for the entry-path work hinges on an empirical claim the maps flag as unverified: 'whether live entries are protected depends on whether fills originate from the trigger thread.' If fills already flow through the fast/trigger path, the extension/halt/liquidity/rank gaps are theoretical and the high-risk unification can be deprioritized; if they flow through the slow auto path, those gaps are live and urgent. This one read-only check decides the priority of every downstream entry-path step and costs nothing. + +### P0 — Delete untracked, gitignored clutter: archive/ tree and momentum-rebuilt.zip +- **blast radius:** low | **effort:** small +- **action:** Delete /mnt/c/code/momentum/archive/ (6641 .py files, gitignored per .gitignore:59, git ls-files archive/ = 0) and /mnt/c/code/momentum/momentum-rebuilt.zip (232 MB, gitignored). Both are local-only and not recoverable from git, so first confirm with the operator that nothing local is needed, then rm. Neither is referenced by any live import (grep for 'from archive'/'import archive' = 0). +- **why:** Highest noise-reduction per unit risk. archive/ is a full duplicate codebase that inflates every repo-wide grep and is the single biggest source of troubleshooting noise; the 232 MB zip is pure disk clutter. Neither is in version control and nothing depends on them, so deletion cannot break the running system. This makes all subsequent investigation cleaner. + +### P0 — Drop the orphaned mem_44381ad9a97c test schema from production Postgres and verify test isolation holds +- **blast radius:** low | **effort:** small +- **action:** Run DROP SCHEMA IF EXISTS mem_44381ad9a97c CASCADE on 192.168.1.5/momentum. This is a throwaway test-isolation schema (db_pg.py:203 creates mem_ for ':memory:' connections) left undropped — evidence that a test/script touched production PG. The auto-cleanup already exists: db_pg.py:165 drops the temp schema on close(), and tests/conftest.py:16 sweeps leftover mem_% schemas on teardown — so a single orphan implies a crashed/killed run, not a missing guard. After dropping, add a recurring sweep (the conftest query at conftest.py:33 reused as a doctor-CLI check) and confirm CI/tests set MOMENTUM_PG_SCHEMA or a non-prod DATABASE_URL so a missed cleanup can never write to the live events/bars tables. +- **why:** This is the only finding in the maps rated medium risk in the datastore area, and it is the concrete confirmation of the known PG-isolation landmine: test code has at least once written to the production DB. Dropping the orphan is trivial and reversible; the real value is closing the recurrence path so a future un-isolated test cannot corrupt the 676k-row live events table. Low blast radius because dropping an unreferenced empty mem_ schema cannot affect the app's public-schema tables. + +### P1 — Delete the dead core/ package (contracts.py, telemetry.py, validation.py, empty __init__.py) and the dead telemetry/ package +- **blast radius:** low | **effort:** small +- **action:** Delete /mnt/c/code/momentum/core/ (telemetry.py, validation.py, contracts.py, empty __init__.py) and /mnt/c/code/momentum/telemetry/ (events.py, empty __init__.py). Verified: repo-wide grep (incl run_live_paper.py and all tests) for 'import telemetry'/'from telemetry'/'core.telemetry'/'core.validation'/'core.contracts' returns ZERO live callers; contracts.py's only importers are telemetry.py and validation.py, which are themselves dead. The live loop uses storage/event_store + storage/event_schema (run_live_paper.py:66-67), a parallel superseding implementation. Do this as its own commit so it is trivially revertable. Optionally drop the orphaned telemetry_events table from storage/pg_schema.sql:415 (the dead telemetry/events.py was its only writer; the table has 0 rows) in a SEPARATE follow-up after the code delete is confirmed safe. +- **why:** Pure dead-by-disuse code (imports cleanly but nobody calls it), so removing it cannot change runtime behaviour. It removes two competing telemetry implementations that are a constant source of 'which telemetry is real?' confusion during troubleshooting. Keep the code delete and the schema drop as separate steps so the reversible code change ships first and the schema change (slightly higher blast radius) is isolated. + +### P1 — Collapse the vestigial DuckDB path-strings and logical-DB labels so a file backend can't be reintroduced +- **blast radius:** low | **effort:** medium +- **action:** In research/schema.py remove the literal DEFAULT_RESEARCH_DB='data/research/market.duckdb' (line 9) used as a default arg in get_research_connection (line 12) — it is never used as a real path because get_connection ignores db_path. In research/multi_schema.py collapse the DB_FILES map (line 19) and open_research_db (line 32)/research_db_path so callers route directly to storage.db.get_connection() with no '.duckdb' filename. Keep the function names/signatures stable (callers like run_live_paper.py:150-152 pass 'market'/'momentum') so this is a no-op at the call sites — only the dead path-string plumbing is removed. Confirm no duckdb.connect anywhere in live code first (already verified empty outside archive/tests). +- **why:** Every datastore access already flows through storage/db.py -> db_pg -> the single Postgres; the 'market'/'news'/'signals' labels and '.duckdb' strings are vestigial and actively misleading (they imply a file backend that no longer exists). Removing the literals prevents someone from 'fixing' them back into a real file path and re-fragmenting the datastore. Low blast radius because the strings are inert today — but test thoroughly since these helpers are on the boot path (research_con in run_live_paper.py). + +### P1 — Archive the two frozen live-dir .duckdb files after confirming Postgres covers their history +- **blast radius:** low | **effort:** medium +- **action:** data/momentum.duckdb (5.0MB, the pre-migration event store) and data/research/market.duckdb (5.5MB, pre-migration bars), both frozen at mtime 2026-06-16 00:24 (the cutover) and opened by no live code. Step 1: verify PG coverage — confirm PG events (676k rows) includes the date range of the duckdb events table, and PG daily_bars (38k)/minute_bars (1.46M) cover market.duckdb's range, by comparing min/max timestamps. Step 2: take a pg_dump of the relevant tables as the durable backup. Step 3: move both files to data/_archive/ (do NOT delete yet — they are the only on-disk copy of pre-2026-06-16 data). Delete only after the pg_dump exists and coverage is confirmed. +- **why:** These are genuinely orphaned (no code path opens them post-cutover), so moving them cannot affect the running system. But they are the sole on-disk copy of pre-migration history, so the order matters: prove PG coverage and create a dump BEFORE removing them. Archiving rather than deleting keeps the step reversible. Lower urgency than the noise-clutter deletes because they are small and harmless where they sit. + +### P1 — Refactor the fast path into a single internal _place_entry() helper (pure refactor, behaviour-preserving) +- **blast radius:** medium | **effort:** medium +- **action:** Extract a private _place_entry(symbol, entry_ref, trigger, stop, ctx) from the body of submit_breakout_now (trading_execution.py:868-1005), where ctx carries day_open, cum_volume, rank, halted, above_vwap, last_price. _place_entry runs the full fast-path gate set (dilution, geometry, extension :887-896, day-extension, halt :899-900, VWAP, rank :917/:922, liquidity sizing :903-904, gross-notional) and the shared post-submit arming. submit_breakout_now becomes a thin wrapper that builds ctx from its args and calls it. CRITICAL: keep all mutations of shared state (self._fresh_entries, self._requested_symbols, self._armed, self._exited_today, self._cooldown_until) inside @_locked methods under self._lock; do not touch expire_stale_entries (:1269) or the naked-stop grace/broker_held protections. Cover with existing fast-path tests; assert no behaviour change. +- **why:** This is STEP 1 of the entry-unification and the prerequisite for closing the gate gaps, but on its own it changes NOTHING at runtime — it just relocates code into a callable helper. Doing it as an isolated, test-covered refactor de-risks the higher-blast-radius steps that follow (routing the auto path through the same helper). Marked P1 not P0 because P0 instrumentation must first confirm whether the slow path even produces live fills; if it doesn't, the gate gaps are not live. + +### P2 — Enrich signal_ready / query_ready_signals_snapshot with trigger, day_open, cum_volume, rank, last_price +- **blast radius:** medium | **effort:** large +- **action:** Add trigger_price, day_open, cum_volume, rank, and a live/last price to the signal_ready event payload and to storage/projections.py query_ready_signals_snapshot (:223-233). The signal producer (watcher) must populate the new fields; until it does, the snapshot returns None and downstream gates must fail-open (never block on missing data, matching the existing above_vwap None-handling at trading_execution.py:660). This is the structural root cause of the dead slow-path gates: the slow snapshot has no live price/day_open/volume/rank, so extension/halt/liquidity/rank CANNOT be computed there regardless of code. +- **why:** Without this, the auto/slow path is structurally incapable of running the anti-chase, halt, liquidity, and rank gates — so this is the true root cause of the dead gates, not the missing helper. It is P2 (not earlier) because it requires changing the signal producer and the projection contract, which is more invasive and must be validated end-to-end; and because if P0 shows live fills already flow through the fast/trigger path (which already has all this context), this enrichment is lower urgency. Risk is in the producer change, not the read side. + +### P2 — Route approve_order through the unified _place_entry and pick ONE fill model behind a flag +- **blast radius:** high | **effort:** large +- **action:** After the snapshot is enriched (prior step), make approve_order (trading_execution.py:740) build a ctx from the enriched approval payload and call the SAME _place_entry instead of its inline executor.execute, so both paths share one gate set. PRESERVE the auto-vs-manual distinction: manual dashboard approvals (approved_by!='auto') intentionally bypass the daily cap (:754) and may bypass other gates — keep that exemption explicit in the helper. Then decide the single fill model (resting limit at signal entry, slow :713, vs marketable limit a hair above trigger, fast :929-931) and gate the change behind an env flag so it can be A/B'd against current behaviour and rolled back. Only after production logs confirm fills flow exclusively through _place_entry, delete the divergent inline blocks and the boot WARNING (run_live_paper.py:812-815). Note: VWAP gate (e3ac89c) and daily-cap (ac1ac50) are ALREADY on the auto path, so those two no longer need plumbing — the remaining gaps are extension, day-extension, halt, liquidity sizing, and rank. +- **why:** This is the genuine architectural win — one entry chokepoint with one gate set — but it is the highest-risk change in the plan: it alters how live orders are placed and changes fill economics (resting vs marketable limit). It must come last, behind a flag, after the helper exists and the snapshot carries the needed context, and after P0 has shown where fills actually originate. The flag + A/B + 'delete only after production confirmation' sequencing is what keeps 'don't break what we have' true. Honest risk: get the shared-state locking or the manual-override exemption wrong and you either double-count the daily cap or strip a protection. + +### P2 — Decide the fate of the dead Schwab integration (delete vs. repair) as an explicit product call +- **blast radius:** medium | **effort:** medium +- **action:** Present the operator a binary decision on schwab/ (23 files, ~1121 LOC) + milestone3_verify.py + tests/unit/schwab/ + the schwab_live Literal branch in trading_mode.py:8,21 + README/RUNBOOK Schwab rows. Facts: it is read-only (no order-placement code anywhere in schwab/orders or schwab/positions), it is isolated from the live loop (run_live_paper.py has zero schwab refs), and its credentials never load due to a confirmed env-var mismatch (schwab/settings.py:25-28 reads SCHWAB_MARKET_DATA_APP_KEY/SCHWAB_BROKER_APP_KEY with no fallback, but .env supplies SCHWAB_Market_APP_KEY/SCHWAB_Trading_APP_KEY). Option A (delete): remove all the above as one revertable commit. Option B (keep as future path): fix the env-var names and add a real order executor before it counts as anything. Recommend A unless Schwab is a committed roadmap item. +- **why:** Schwab is the largest single block of troubleshooting noise after archive/, and it cannot trade (no executor, broken creds), so it is functionally dead today. But unlike the core/telemetry deletes it has a test file and is documented in README/RUNBOOK, so deletion is a product decision rather than mechanical cleanup — hence medium blast radius and P2. The cost of leaving it is ongoing confusion ('do we support Schwab?'); the cost of deleting wrongly is re-doing 1.1k LOC if Schwab is genuinely planned. Surface it as a decision rather than silently removing. + +### P2 — Verify the two unpopulated PG tables (symbols=0, telemetry_events=0) are intentionally empty, not silently-dead writers +- **blast radius:** low | **effort:** small +- **action:** For storage/pg_schema.sql tables with 0 rows: telemetry_events is explained (its only writer is the dead telemetry/events.py being removed in the P1 dead-code step — drop the table there). For symbols=0, grep for any INSERT/writer against the symbols table; if a live code path is supposed to populate it (e.g. a symbol-universe loader) and isn't, that is a silently-dead writer to fix; if nothing should write it, drop the table from pg_schema.sql. Report which case applies; do not drop until the writer status is confirmed. +- **why:** An empty table can be benign (reserved) or a symptom of a dead writer that is silently failing — the same failure class the observability overhaul exists to catch. Cheap to investigate (one grep + one row count) and it either closes a loose end or surfaces a real bug. Low blast radius and low urgency, so P2; bundle the telemetry_events drop with the P1 dead-code removal since they share a cause. + + +## Appendix — Datastores map + +- **[live / low risk] PostgreSQL (single canonical datastore)** + - where: 192.168.1.5:5432/momentum via DATABASE_URL in /mnt/c/code/momentum/.env; connection layer /mnt/c/code/momentum/storage/db_pg.py; schema /mnt/c/code/momentum/storage/pg_schema.sql (24 public tables) + - recommend: This IS the single-Postgres target; the migration is effectively done at the connection layer. Keep it. Note symbols=0 and telemetry_events=0 rows (tables exist but unpopulated) — verify those are intentionally empty vs a silently-dead writer. +- **[live / low risk] get_connection / EventStore('momentum') / open_research_db('market') unification** + - where: /mnt/c/code/momentum/storage/db.py -> db_pg.get_connection; /mnt/c/code/momentum/research/multi_schema.py (DB_FILES market/news/signals); /mnt/c/code/momentum/research/schema.py (DEFAULT_RESEARCH_DB) + - recommend: Logical namespaces (market/news/signals) and the 'momentum' db_path are vestigial labels that all collapse to one DB — harmless but misleading. Optional cleanup: collapse open_research_db/get_research_connection to a single get_connection() call and drop the DB_FILES/DEFAULT_RESEARCH_DB '.duckdb' path strings so no one re-introduces a file backend. +- **[dead / low risk] data/momentum.duckdb (root events DB)** + - where: /mnt/c/code/momentum/data/momentum.duckdb + - recommend: Orphaned leftover. Move to data/_archive/ or delete after confirming PG events table contains its history (PG events already has 676k rows). Keep one backup copy until a PG dump exists, since it is the only on-disk copy of pre-2026-06-16 events. +- **[dead / low risk] data/research/market.duckdb (research/bars DB)** + - where: /mnt/c/code/momentum/data/research/market.duckdb + - recommend: Orphaned. Same disposition as root duckdb: archive/delete after confirming PG daily_bars+minute_bars cover its date range. Remove the literal '.duckdb' default in research/schema.py to prevent confusion. +- **[dead / low risk] archive/data/*.duckdb (legacy multi-file research split)** + - where: /mnt/c/code/momentum/archive/data/ (momentum.duckdb, momentum_events.duckdb, research/{market,news,signals,sec,rss,economic,evaluations,log}.duckdb, several test_*_events.duckdb, research.duckdb.wal) + - recommend: Pure historical leftover. Safe to delete wholesale (or keep the archive/ dir as the migration snapshot). Not part of any single-Postgres step. +- **[live / low risk] data/learned_params.json (nightly-tuner params cache)** + - where: /mnt/c/code/momentum/data/learned_params.json + - recommend: Legitimately a flat-file param cache, fine to leave as JSON (it is config, not transactional data). If you want truly single-Postgres, you could store it in a params table, but the file is simpler and the risk of putting boot-critical params behind a network DB (loop can't start if PG is down) argues for keeping it on local disk. +- **[live / low risk] news / catalyst caches (now PG tables, not files)** + - where: PG tables news_catalyst_cache, raw_news_items, raw_fetch_attempts, news_events (in /mnt/c/code/momentum/storage/pg_schema.sql); writers research/ingestion/news_enrichment.py; readers research/labeler.py, storage/projections.py + - recommend: Already single-Postgres — nothing to migrate. No action. +- **[live / low risk] Grafana datasource (read-only consumer)** + - where: /mnt/c/code/momentum/grafana/provisioning/datasources/momentum.yaml + dashboards/momentum.json + - recommend: Correct as-is. This is the intended end-state (Grafana + doctor CLI over one PG). No action. +- **[dead / medium risk] Leftover mem_* test schema in production PG** + - where: schema mem_44381ad9a97c inside the live 192.168.1.5/momentum database + - recommend: Drop it: DROP SCHEMA mem_44381ad9a97c CASCADE. Then guard against recurrence — ensure tests set MOMENTUM_PG_SCHEMA or use a dedicated non-prod DATABASE_URL so a missed cleanup can never pollute production. Periodically sweep for mem_* schemas (the conftest already attempts this on teardown). + +_notes:_ Concrete single-Postgres status: the migration is functionally COMPLETE at the connection layer. Every datastore access in live code (run_live_paper.py, trading_execution.py, runtime/triggers, research ingestion, projections, dashboard api, telegram_bot) flows through storage/db.py -> storage/db_pg.py -> psycopg2 to the one DATABASE_URL Postgres. No live code calls duckdb.connect (verified by grep excluding archive+tests). The remaining 'work' is cleanup, not architecture: (1) delete/archive the two orphaned live-dir .duckdb files (data/momentum.duckdb, data/research/market.duckdb, both frozen 2026-06-16 at cutover) and the archive/data/*.duckdb legacy split; (2) remove the vestigial '.duckdb' path-string defaults in research/schema.py and the logical DB_FILES map in research/multi_schema.py so no one re-introduces a file backend; (3) DROP the orphaned mem_* test schema in prod and harden test isolation. Two flat files remain on-disk by design and are reasonable to keep: data/learned_params.json (boot params; local-disk so the loop can start even if PG is down) and the data/*.log files (live_run.log, trade_journal.log via journal.sh, dashboard.log — plain text logs, not a datastore). Risk note: making params/boot depend on the remote LAN Postgres would couple loop startup to network availability; the current split (transactional data in PG, a tiny boot-config JSON on disk) is defensible. The single real hazard found is the leftover mem_ schema confirming test code has at least once touched production PG. + +## Appendix — Entry-paths map + +- **[duplicate / high risk] Two entry chokepoints; live entries use the SLOW path (approve_order). FAST path (submit_breakout_now) is the trigger-thread counterpart carrying the richer anti-chase gates** + - where: trading_execution.py:607 request_approvals_for_ready_signals, :740 approve_order, :1031 process_auto_approvals (SLOW/LIVE) vs :809 submit_breakout_now (FAST) + - recommend: Pick ONE submit chokepoint. Keep the trigger thread as signal source (it has live price/day_open/rank/cum_volume) but route both paths through a single internal _place_entry(symbol, entry_ref, trigger, stop, ctx) helper so gates live in one function. Verify in production logs which path actually produces fills before deleting either. +- **[live / low risk] Daily fresh-entry cap (max_fresh_entries_per_day=5)** + - where: FAST trading_execution.py:844-852; SLOW :754-761; increments :793-799 and :989-991 + - recommend: Already unified via shared state. Use as the model: a unified helper should keep the auto-vs-manual distinction. +- **[dead / medium risk] Rank-concentration / rank risk-haircut (concentrate_top_n, rank_risk_factor)** + - where: FAST only: trading_execution.py:841-843 skip-if-rank_factor<=0, :917 risk*rank_factor, :922 max_risk_dollars*rank_factor. SLOW has no rank concept. + - recommend: To make it live: emit rank onto signal_ready, add it to query_ready_signals_snapshot, apply rank_risk_factor inside the unified helper. Until then keep TRADING_CONCENTRATE_TOP_N=0 and keep the boot warning. +- **[partial / medium risk] VWAP selection gate (require_above_vwap=1)** + - where: SLOW only: trading_execution.py:659-680. FAST has no VWAP check. + - recommend: Plumb above_vwap (or session VWAP) into the trigger path, or funnel all live entries through the unified chokepoint. Easiest: compute above_vwap in the unified helper from live price + a vwap provider. +- **[live / low risk] Dilution veto (_dilution_vetoed, NEWS_DILUTION_VETO_ENABLED=1)** + - where: BOTH: SLOW trading_execution.py:644-648; FAST :864-867 + - recommend: No change. Template for unification: one shared predicate called from both paths. +- **[partial / low risk] Bad-geometry check (stop>=entry, trigger<=0, bad price)** + - where: SLOW: trading_execution.py:651 (not entry or not stop or stop>=entry). FAST: :872-878 (bad_price, then stop>=entry_ref or trigger<=0). + - recommend: In the unified helper validate geometry against the same reference (the price the order will actually use) and keep the bad_price guard. Low risk to converge. +- **[dead / high risk] Extension / anti-chase guard (extension_max_pct=0.15) + day-extension guard (day_extension_max_pct=0.30)** + - where: FAST only: trading_execution.py:887-896 (over_extended / over_extended_day). SLOW has neither. + - recommend: Most safety-relevant gap. Either ensure live fills happen on the fast/trigger path, or carry trigger+day_open+live price into the unified chokepoint and apply the same extension math. Do not leave the live entry point without anti-chase protection. +- **[dead / high risk] Halt guard (halt_guard_enabled, halt_max_bar_gap_min)** + - where: FAST only: trading_execution.py:899-900 (skipped='halted_symbol'). SLOW has no halt check. + - recommend: Compute or pass the halt flag inside the unified chokepoint. Until unified, slow-path entries can buy into a halt. +- **[partial / medium risk] Sizing: liquidity cap (liquidity_max_volume_pct=0.025)** + - where: FAST only: trading_execution.py:903-904 max_shares=int(liq*cum_volume) -> calculate_position_size(max_shares=...). SLOW :687-697 passes no max_shares. + - recommend: Carry cum_volume (or a recent-volume estimate) onto the signal/projection so the unified helper can pass max_shares. +- **[live / low risk] Sizing core: equity / risk% / max_position_pct / max_gross_notional / max_risk_dollars** + - where: BOTH: SLOW trading_execution.py:682-697; FAST :902-923 + - recommend: Keep the shared sizing core. Move rank_factor and max_shares into the unified helper's context so the single calculate_position_size call covers all paths. +- **[duplicate / medium risk] Fill mechanism diverges: resting limit (slow) vs marketable limit (fast)** + - where: SLOW: trading_execution.py:713 order_type=self.settings.entry_order_type (default 'limit' resting at signal entry). FAST: :929-931 marketable limit = max(entry_ref,trigger)*(1+trigger_slippage_pct). + - recommend: Decide one fill model. The marketable-limit (fast) model is intended for runners and needs live price, which is another reason the unified chokepoint should keep the trigger thread as price source. +- **[live / low risk] Shared post-submit arming + backout (expire_stale_entries) already unified** + - where: BOTH arm into self._armed (SLOW :781-792, FAST :976-988); expire_stale_entries :1269, _note_position_exits :1232, _register_backout :1207 use shared state. + - recommend: Model the entry unification on this already-unified exit side: bring the entry decision+submit into one _place_entry the same way. + +_notes:_ UNIFICATION PLAN (concrete, ordered, per-step risk). Goal: one entry chokepoint _place_entry(symbol, entry_ref, trigger, stop, ctx) that both the trigger thread and the auto-approval flow call, eliminating the dead/partial gates. ctx carries day_open, cum_volume, rank, halted, above_vwap, last_price. STEP 1 (low risk): Extract a private _place_entry from submit_breakout_now's body (lines 868-1005) that takes a context object and runs ALL gates (dilution, geometry, extension, day-extension, halt, VWAP, rank, liquidity sizing, gross-notional). submit_breakout_now becomes a thin wrapper that builds ctx from its args and calls it. Pure refactor, no behaviour change; covered by existing fast-path tests. STEP 2 (medium risk): Enrich the signal_ready event payload + query_ready_signals_snapshot (storage/projections.py:223-233) to carry trigger_price, day_open, cum_volume, rank, and a live/last price. Without this the slow path structurally cannot run the extension/halt/liquidity/rank gates (root cause of the dead gates). Risk: signal producer (watcher) must populate the new fields; until it does, gates fail-open. STEP 3 (medium): Make approve_order build a ctx from the enriched approval payload and call the SAME _place_entry instead of the inline executor.execute (trading_execution.py:774). Now both paths share one gate set. Keep approve_order's manual-override exemption (manual dashboard approvals skip the daily cap and possibly other gates by design). STEP 4 (high risk, do last and behind a flag): Decide the single fill model. If the marketable-limit model wins, the slow path must price off live price (needs STEP 2's last price). Gate behind an env flag so you can A/B against the current resting-limit behaviour and roll back. STEP 5 (cleanup): Once verified in production that fills flow only through the unified helper, delete the divergent inline blocks and the boot WARNING at run_live_paper.py:810-815. What MUST NOT break: shared state under self._lock (self._fresh_entries, self._requested_symbols, self._armed, self._exited_today, self._cooldown_until, self._held_ret) — keep all mutations inside @_locked methods; the naked-stop protections in expire_stale_entries (grace window, broker_held check) — leave untouched; the auto-vs-manual approval distinction. SAFETY NOTE: today, because live entries take the slow path, the anti-chase extension guards, day-extension guard, halt guard, VWAP gate (on fast), liquidity sizing, and rank-concentration are NOT all applied to the actual live entry. The trigger thread DOES apply them when it fires, so whether live entries are protected depends on whether fills originate from the trigger thread or the slow auto-approval flow — confirm this empirically before relying on those guards. Files: /mnt/c/code/momentum/trading_execution.py, /mnt/c/code/momentum/run_live_paper.py, /mnt/c/code/momentum/storage/projections.py, /mnt/c/code/momentum/strategy/risk/position_sizing.py, /mnt/c/code/momentum/.env. + +## Appendix — Dead-code map + +- **[dead / low risk] telemetry/ package (telemetry/events.py + empty telemetry/__init__.py)** + - where: /mnt/c/code/momentum/telemetry/events.py, /mnt/c/code/momentum/telemetry/__init__.py + - recommend: Delete telemetry/events.py and telemetry/__init__.py (both git-tracked). Optionally drop the now-orphaned `telemetry_events` table from storage/pg_schema.sql:415 (no live writer/reader). +- **[dead / low risk] core/telemetry.py** + - where: /mnt/c/code/momentum/core/telemetry.py + - recommend: Delete core/telemetry.py (git-tracked). It is a parallel, unused telemetry implementation superseded by storage/event_store + storage/event_schema. +- **[dead / low risk] core/validation.py** + - where: /mnt/c/code/momentum/core/validation.py + - recommend: Delete core/validation.py (git-tracked). +- **[dead / low risk] core/contracts.py** + - where: /mnt/c/code/momentum/core/contracts.py + - recommend: Delete core/contracts.py together with core/telemetry.py and core/validation.py. The entire core/ package (3 files + empty core/__init__.py) can be removed; __init__.py is 0 bytes with no re-exports. +- **[dead / medium risk] Schwab integration (schwab/ package, 23 files / ~1121 LOC)** + - where: /mnt/c/code/momentum/schwab/ (auth/, market/, orders/, positions/, health/, streaming/, settings.py) + - recommend: Non-functional and isolated from the live loop. Either (a) delete schwab/ + milestone3_verify.py + tests/unit/schwab/ + the schwab_live branch in trading_mode.py + Schwab rows in README/RUNBOOK, or (b) if Schwab is a deliberate future path, fix the env-var names in schwab/settings.py and add an actual order executor before keeping it. As-is it is pure troubleshooting noise. +- **[dead / medium risk] milestone3_verify.py (Schwab health-walk script)** + - where: /mnt/c/code/momentum/milestone3_verify.py + - recommend: Delete alongside the schwab/ package (they live and die together). Remove its README/RUNBOOK rows. +- **[live / high risk] Standalone research/sweep CLIs (diagnose_entries.py, sweep_exits.py, sweep_filters.py, comprehensive_backtest.py, fetch_minute_bars.py, backtest_cli.py)** + - where: /mnt/c/code/momentum/diagnose_entries.py, sweep_exits.py, sweep_filters.py, comprehensive_backtest.py, fetch_minute_bars.py, backtest_cli.py + - recommend: KEEP. Do not delete -- these are documented operator/research entry points, just not part of the trading runtime. +- **[dead / low risk] archive/ directory (6641 .py files, ~older full copies of the codebase)** + - where: /mnt/c/code/momentum/archive/ + - recommend: Safe to delete the local archive/ directory entirely -- it is gitignored (not in version control, nothing depends on it). Removing it dramatically cleans up search/grep results. Verify nothing local is needed first since it is not recoverable from git. +- **[dead / low risk] momentum-rebuilt.zip (232 MB stray artifact)** + - where: /mnt/c/code/momentum/momentum-rebuilt.zip + - recommend: Safe to delete the local file (gitignored, not tracked, no references). +- **[dead / low risk] data/_backtest_multiday.py, data/_sip_backfill.py (underscore-prefixed scratch modules)** + - where: /mnt/c/code/momentum/data/_backtest_multiday.py, /mnt/c/code/momentum/data/_sip_backfill.py + - recommend: Safe to delete (local-only, untracked, no callers). Confirm they aren't a backfill you still need to run, since they're not in git. + +_notes:_ Method: read entry points (run_live_paper.py imports at lines 38-68 use alpaca_paper + research.ingestion + runtime + storage + trading_execution -- NOT core/ or telemetry/ or schwab/), then grepped the whole repo (excluding the gitignored archive/ and .venv) for each suspect, explicitly including run_live_paper.py and all of tests/. Verified imports actually load with system python3 (core.contracts/validation/telemetry and telemetry.events all import OK), so the memory's 'reported broken' label is stale -- these are dead-by-disuse, not syntactically broken. Schwab is the higher-risk delete: it's a 23-file/~1.1k-LOC read-only integration with a confirmed env-var mismatch (settings.py reads SCHWAB_MARKET_DATA_APP_KEY/SCHWAB_BROKER_APP_KEY with no fallback; .env supplies SCHWAB_Market_APP_KEY/SCHWAB_Trading_APP_KEY) and no order-placement code, so it can't trade live -- but it has a test file and README/RUNBOOK docs, so deletion is a product decision (medium risk). The sweep/diagnose CLIs are LIVE research tools (documented, not runtime) and must NOT be deleted (marked high risk to prevent accidental removal). Biggest noise reducers with lowest risk: delete the untracked, gitignored archive/ tree (6641 .py files) and momentum-rebuilt.zip (232MB). I MODIFIED NOTHING -- read-only inspection only. \ No newline at end of file diff --git a/run_live_paper.py b/run_live_paper.py index c4ecc8f..eabe0e0 100644 --- a/run_live_paper.py +++ b/run_live_paper.py @@ -802,17 +802,19 @@ def main(argv: list[str] | None = None) -> int: if client else None ) print(f"[boot] exit management: {exit_cfg.describe()}") - print( - f"[boot] entry gates: VWAP {'ENFORCED' if exec_settings.require_above_vwap else 'shadow-only'}" - f", daily-cap {exec_settings.max_fresh_entries_per_day or 'off'}" - f", risk-cap ${exec_settings.max_risk_dollars:.0f}" - ) - if exec_settings.concentrate_top_n > 0: + _es = rt["execution"].settings if rt["execution"] is not None else None + if _es is not None: print( - "[boot] WARNING: TRADING_CONCENTRATE_TOP_N>0, but rank-concentration is " - "enforced ONLY on the dead fast-trigger path, NOT the live auto-approval " - "path — it will not affect live entries (see entry-path-fast-vs-auto)." + f"[boot] entry gates: VWAP {'ENFORCED' if _es.require_above_vwap else 'shadow-only'}" + f", daily-cap {_es.max_fresh_entries_per_day or 'off'}" + f", risk-cap ${_es.max_risk_dollars:.0f}" ) + if _es.concentrate_top_n > 0: + print( + "[boot] WARNING: TRADING_CONCENTRATE_TOP_N>0, but rank-concentration is " + "enforced ONLY on the dead fast-trigger path, NOT the live auto-approval " + "path — it will not affect live entries (see entry-path-fast-vs-auto)." + ) # Telegram push alerts — gated, decoupled (polls the event store), and # fire-and-forget so it can never impede trading.