momentum-system/plans/019-discovery-off-scheduler.md

Plan 019: Move discovery off the scheduler thread — it blocks 6–30s and staleness the whole loop

Executor instructions: Live money path. TDD, honor STOP conditions, update plans/README.md when done.

Drift check: git diff --stat 5896d29..HEAD -- run_live_paper.py runtime/ tests/

Status

• Priority: P1 (the telemetry-proven edge-detection bottleneck)
• Effort: M
• Risk: MED-HIGH (shared-watchlist mutation across threads)
• Depends on: 018 (the dedicated-thread + dedicated-connection pattern to mirror)
• Category: latency / money-path
• Planned at: commit 5896d29, 2026-07-19

Why this matters (telemetry, not theory)

Per-module duration_ms from live module_tick events (2026-07-17→19):

module	avg ms	max ms
triggers (detect)	0.0	0.0
ingestion	0.0	0.0
discovery	5,921	30,789

Breakout detection and bar ingestion are instant. Discovery takes ~6s average, up to 30s — and it runs as a step on the single-threaded main scheduler. While it churns, the scheduler cannot run ingest (bar refresh), arm (trigger levels), or watch. The fast trigger thread keeps checking at 2s, but it reads bars that ingest couldn't refresh — so during a 6–30s discovery pass the bot detects breakouts on stale prices. This is the hidden cause of "late edge detection": not the trigger (instant), the scheduler being blocked by discovery.

018 already removed exits from the scheduler (own thread). Discovery is the last time-critical blocker on the scheduler thread.

The fix

Move discovery to its OWN daemon thread (mirror trigger_loop / the plan-018 exit_loop), so its 6–30s never blocks ingest/arm. Discovery itself is NOT time-critical (finding new names to watch can lag a few seconds); it must simply stop blocking the things that ARE.

The hard part is the SHARED WATCHLIST. step_discover mutates the module-level symbols list (symbols.extend(added), del symbols[:N]) which step_ingest / step_arm / step_watch read every cycle. A background thread mutating a list another thread iterates = "list changed during iteration" / half-updated reads. And run_discovery uses the shared rt["research_con"] (not thread-safe — needs its own, like 018's exit connection).

Steps

Step 1: Make the watchlist a thread-safe object

Replace the bare symbols list with a tiny Watchlist helper (new runtime/watchlist.py): a lock-guarded list with snapshot() -> tuple (readers use this), add(names), and trim(max). All scheduler readers (step_ingest, step_arm, step_watch, anywhere iterating symbols) switch to watchlist.snapshot(); discovery calls add/trim. Atomic, no torn reads.

Verify: unit test — concurrent add/trim from one thread while another loops snapshot() 10k times never raises and never yields a torn list.

Step 2: Discovery on its own thread with a dedicated connection

• def discover_loop(): open its OWN open_research_db("market") (like 018's exit thread), then run_exit_loop-style while not stop_event.wait(interval): run_discovery(...); watchlist.add(...). Reuse a shared generic loop driver if clean (the 018 run_exit_loop is actually generic — consider run_interval_loop(fn, stop, interval) and have both use it).
• Remove/disable the scheduler discover step when the thread runs (same single-writer gate 018 uses for exits: enabled = ... and not _flag("DISCOVER_THREAD_ENABLED","1")).
• Interval: keep ~120s (discovery cadence unchanged — the point is it no longer BLOCKS, not that it runs more often).

Verify: scheduler no longer registers discover when the thread is on; discovery still folds new names into the watchlist (integration check).

Step 3: Confirm ingest freshness under a slow discovery

With discovery off the scheduler, a 30s discovery pass must NOT delay ingest.

Verify: with the discover thread sleeping/slow (monkeypatched run_discovery that sleeps 10s), the scheduler's ingest step still runs on its interval (no starvation). Measure: bar-age stays within the ingest interval.

Optional Step 4: also speed discovery itself

Only if easy — profile WHY run_discovery takes 6–30s (the screener call? per- symbol processing?). If a cheap win exists (batch a query, cache the universe), take it. Not required — the thread move solves the blocking regardless.

Done criteria

• Watchlist is a lock-guarded object; all readers use snapshot()
• Discovery runs on its own thread with its own DB connection; scheduler discover step disabled when the thread is on
• Ingest freshness proven independent of discovery duration
• New tests green; full suite green; ruff --select F clean
• DISCOVER_THREAD_ENABLED (default 1) in the deploy configs

STOP conditions

• The watchlist is read in more places than can be cleanly switched to snapshot() in one pass → STOP, list them, do it as its own change first.
• run_discovery writes shared state beyond the watchlist that would race → STOP, audit and report.
• Any change touches entry/exit/sizing decision logic → out of scope.

Maintenance notes

• After this, the loop shape is: three fast daemon threads (trigger=entries, exits, discovery) + the scheduler for genuinely non-time-critical steps (sync, news, notify, eod). That's the target architecture — anything time-critical gets its own thread; the scheduler is for the slow, tolerant stuff.
• This is the LAST latency blocker in the telemetry. After 019, ingest→detect→ decide→exit is bounded by the fast-thread intervals (2s trigger / 1s exit / 5s bars), not by any multi-second scheduler step. Beyond this, remaining P&L is edge, not speed — and gap-ORB's edge was exhaustively refuted.