momentum-system/plans/018-subsecond-exit-thread.md

Plan 018: Sub-second exit management on a dedicated thread

Executor instructions: Live money path — TDD every change, run the exit tests before/after, honor STOP conditions. When done, update plans/README.md.

Drift check: git diff --stat 680ddc2..HEAD -- run_live_paper.py strategy/ tests/

Status

• Priority: P1 (owner's core ask: get out of trades fast; exits leak the P&L on fast small-caps)
• Effort: M
• Risk: MED-HIGH (touches the live exit execution path)
• Depends on: none (config speedups already shipped 680ddc2)
• Category: latency / money-path
• Planned at: commit 680ddc2, 2026-07-18

Why this matters

Exit management (LiveExitManager.manage() — breakeven ratchet, trail, tier scale-out, catastrophe/first-red) runs as a step on the single-threaded main scheduler. Even at the new 4s interval it (a) queues behind slower steps (ingest, discover, news) that share the thread, so real reaction can drift well past 4s, and (b) is bounded by the scheduler tick. On a $2–20 small-cap that spikes and round-trips in seconds, a stop or profit-tier that reacts "sometime in the next 4–10s" gives back the gain. This is the owner's #1 felt problem ("late exits, gave back gains"), and unlike entry timing (verified fine, memory execution-plus-judgment-mode), exit latency is real and measurable.

The fix: run exit management on its own dedicated thread (like the fast trigger thread already does for entries), at a sub-second-to-1s cadence, independent of the scheduler queue. Entries already got this treatment (trigger_loop on its own thread); exits deserve the symmetric fix.

Current state

• run_live_paper.py:
  • trigger_loop() (~line 1078) — the PATTERN to mirror: own daemon thread, while not stop_event.wait(interval): _trigger_pass(), wrapped in try/except so one error never kills the loop. Started ~line 1339.
  • exit_mgr = LiveExitManager(client, store, _rth_bars, cfg, session_id) (~line 1097) — the object whose .manage() does all exit work.
  • step_manage_exits() (~line 1128) calls exit_mgr.manage() on the scheduler at TRADING_EXIT_MANAGE_INTERVAL_SECONDS (now 4s).
• strategy/.../LiveExitManager.manage() — reads current bars/price per position, computes desired stop/target/scale, submits broker orders. MUST be audited for thread-safety before running it concurrently with the trigger thread and scheduler (shared broker client, shared store).

Steps

Step 1: Audit thread-safety of the shared resources

Before adding a thread, confirm what manage() touches that the trigger thread / scheduler also touch: the Alpaca client (HTTP — usually fine, confirm no shared mutable cursor), rt["store"] (EventStore — has a _lock around emit, good), research_con (the facade — NOT thread-safe; if manage() reads bars via a shared research_con, it needs its OWN connection like step_quotes/catalyst provider do, per the memory note entry-path-fast-vs-auto about dedicated connections). Document findings; if manage() shares research_con, Step 2 must give the exit thread its own.

Verify: written audit; grep every attribute manage() reads.

Step 2: Move exit management to its own thread

• Add exit_interval = float(os.environ.get("TRADING_EXIT_THREAD_INTERVAL_SECONDS", "1.0")).
• def exit_loop(): while not stop_event.wait(exit_interval): try: exit_mgr.manage() except: log — mirror trigger_loop exactly.
• Give the exit thread its OWN research DB connection if Step 1 found sharing (open it inside the thread, single-threaded use, like the enrich/quote paths).
• Start it next to the trigger thread (~line 1339), gated on _flag("TRADING_EXECUTION_ENABLED","1") and client is not None.
• REMOVE the scheduler.add("exits", ...) step (or gate it off) so exits don't run on BOTH the thread and the scheduler (double-submit risk).
• Keep TRADING_EXIT_MANAGE_INTERVAL_SECONDS honored as a fallback only if the thread is disabled.

Verify: unit test — a fake exit_mgr whose .manage() counts calls, driven by a short-interval loop, is invoked repeatedly off the scheduler; the scheduler no longer registers an "exits" step (grep the scheduler.add list).

Step 3: Idempotency / double-submit guard

At 1s cadence the manager runs ~4× more often. Confirm manage() is idempotent — it must not re-submit a stop/scale it already placed (check for an existing resting order before submitting; get_orders(symbols=...) scoped, per the get-orders-truncation memory). If it isn't already idempotent, add the guard. This is the highest-risk correctness point — a fast loop that re-fires orders is worse than a slow one.

Verify: test that two back-to-back manage() calls on the same position with an already-resting protective stop submit the stop ONCE.

Step 4: Backpressure / cost check

1s cadence × ~10 positions = more broker calls. Confirm within Alpaca limits (scope reads by symbols=, reuse the position snapshot the sync step already fetches rather than re-polling per position). If manage() is heavy, cap the effective rate or reuse cached quotes.

Verify: measure manage() wall-time; at 1s interval it must finish well inside the interval for the real position count.

Done criteria

• Exit management runs on its own daemon thread at ~1s, off the scheduler
• manage() thread-safe (own DB connection if needed) + idempotent (no double-submit)
• Scheduler no longer runs a competing "exits" step
• Exit tests green; full suite green; ruff --select F clean
• TRADING_EXIT_THREAD_INTERVAL_SECONDS in the deploy configs (default 1.0)

STOP conditions

• manage() shares mutable state with the trigger thread that can't be cleanly isolated → STOP, report; a data race in the exit path is worse than 4s latency.
• manage() is NOT idempotent and making it so requires restructuring the exit engine → STOP, plan that separately; do not ship a fast loop that double-fires.
• Any change would touch entry/sizing logic → out of scope.

Maintenance notes

• Symmetry: entries on trigger_loop, exits on exit_loop — two fast daemon threads + the slow scheduler for everything non-time-critical. Keep that shape.
• If the owner later wants tick-level exits, the thread is the seam to feed a streaming quote instead of a polled one — don't shorten the poll below ~0.5s without moving to a push feed.
• This is the last measurable execution-latency lever. Beyond it, remaining P&L is strategy (edge), not speed — and the exhaustive search says the edge isn't there for gap-ORB (see orderflow-shelf-verdicts). Speed makes the good days cleaner; it does not create an edge.