plans(018): sub-second exit thread — the last real execution-latency lever

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

plans/018-subsecond-exit-thread.md

@@ -0,0 +1,122 @@
+# Plan 018: Sub-second exit management on a dedicated thread
+
+> **Executor instructions**: Live money path — TDD every change, run the exit
+> tests before/after, honor STOP conditions. When done, update
+> `plans/README.md`.
+>
+> **Drift check**: `git diff --stat 680ddc2..HEAD -- run_live_paper.py strategy/ tests/`
+
+## Status
+- **Priority**: P1 (owner's core ask: get out of trades fast; exits leak the P&L on fast small-caps)
+- **Effort**: M
+- **Risk**: MED-HIGH (touches the live exit execution path)
+- **Depends on**: none (config speedups already shipped 680ddc2)
+- **Category**: latency / money-path
+- **Planned at**: commit `680ddc2`, 2026-07-18
+
+## Why this matters
+
+Exit management (`LiveExitManager.manage()` — breakeven ratchet, trail, tier
+scale-out, catastrophe/first-red) runs as a step on the **single-threaded main
+scheduler**. Even at the new 4s interval it (a) queues behind slower steps
+(ingest, discover, news) that share the thread, so real reaction can drift well
+past 4s, and (b) is bounded by the scheduler tick. On a $2–20 small-cap that
+spikes and round-trips in seconds, a stop or profit-tier that reacts "sometime
+in the next 4–10s" gives back the gain. This is the owner's #1 felt problem
+("late exits, gave back gains"), and unlike entry timing (verified fine, memory
+`execution-plus-judgment-mode`), exit latency is real and measurable.
+
+The fix: run exit management on its **own dedicated thread** (like the fast
+trigger thread already does for entries), at a sub-second-to-1s cadence,
+independent of the scheduler queue. Entries already got this treatment
+(`trigger_loop` on its own thread); exits deserve the symmetric fix.
+
+## Current state
+
+- `run_live_paper.py`:
+  - `trigger_loop()` (~line 1078) — the PATTERN to mirror: own daemon thread,
+    `while not stop_event.wait(interval): _trigger_pass()`, wrapped in
+    try/except so one error never kills the loop. Started ~line 1339.
+  - `exit_mgr = LiveExitManager(client, store, _rth_bars, cfg, session_id)`
+    (~line 1097) — the object whose `.manage()` does all exit work.
+  - `step_manage_exits()` (~line 1128) calls `exit_mgr.manage()` on the
+    scheduler at `TRADING_EXIT_MANAGE_INTERVAL_SECONDS` (now 4s).
+- `strategy/.../LiveExitManager.manage()` — reads current bars/price per
+  position, computes desired stop/target/scale, submits broker orders. MUST be
+  audited for thread-safety before running it concurrently with the trigger
+  thread and scheduler (shared broker client, shared store).
+
+## Steps
+
+### Step 1: Audit thread-safety of the shared resources
+Before adding a thread, confirm what `manage()` touches that the trigger
+thread / scheduler also touch: the Alpaca `client` (HTTP — usually fine,
+confirm no shared mutable cursor), `rt["store"]` (EventStore — has a `_lock`
+around emit, good), `research_con` (the facade — NOT thread-safe; if
+`manage()` reads bars via a shared `research_con`, it needs its OWN connection
+like `step_quotes`/catalyst provider do, per the memory note
+`entry-path-fast-vs-auto` about dedicated connections). Document findings; if
+`manage()` shares `research_con`, Step 2 must give the exit thread its own.
+
+**Verify**: written audit; grep every attribute `manage()` reads.
+
+### Step 2: Move exit management to its own thread
+- Add `exit_interval = float(os.environ.get("TRADING_EXIT_THREAD_INTERVAL_SECONDS", "1.0"))`.
+- `def exit_loop(): while not stop_event.wait(exit_interval): try: exit_mgr.manage() except: log` — mirror `trigger_loop` exactly.
+- Give the exit thread its OWN research DB connection if Step 1 found sharing
+  (open it inside the thread, single-threaded use, like the enrich/quote paths).
+- Start it next to the trigger thread (~line 1339), gated on
+  `_flag("TRADING_EXECUTION_ENABLED","1")` and `client is not None`.
+- REMOVE the `scheduler.add("exits", ...)` step (or gate it off) so exits don't
+  run on BOTH the thread and the scheduler (double-submit risk).
+- Keep `TRADING_EXIT_MANAGE_INTERVAL_SECONDS` honored as a fallback only if the
+  thread is disabled.
+
+**Verify**: unit test — a fake exit_mgr whose `.manage()` counts calls, driven
+by a short-interval loop, is invoked repeatedly off the scheduler; the
+scheduler no longer registers an "exits" step (grep the scheduler.add list).
+
+### Step 3: Idempotency / double-submit guard
+At 1s cadence the manager runs ~4× more often. Confirm `manage()` is idempotent
+— it must not re-submit a stop/scale it already placed (check for an existing
+resting order before submitting; `get_orders(symbols=...)` scoped, per the
+`get-orders-truncation` memory). If it isn't already idempotent, add the guard.
+This is the highest-risk correctness point — a fast loop that re-fires orders
+is worse than a slow one.
+
+**Verify**: test that two back-to-back `manage()` calls on the same position
+with an already-resting protective stop submit the stop ONCE.
+
+### Step 4: Backpressure / cost check
+1s cadence × ~10 positions = more broker calls. Confirm within Alpaca limits
+(scope reads by `symbols=`, reuse the position snapshot the sync step already
+fetches rather than re-polling per position). If `manage()` is heavy, cap the
+effective rate or reuse cached quotes.
+
+**Verify**: measure `manage()` wall-time; at 1s interval it must finish well
+inside the interval for the real position count.
+
+## Done criteria
+- [ ] Exit management runs on its own daemon thread at ~1s, off the scheduler
+- [ ] `manage()` thread-safe (own DB connection if needed) + idempotent (no double-submit)
+- [ ] Scheduler no longer runs a competing "exits" step
+- [ ] Exit tests green; full suite green; `ruff --select F` clean
+- [ ] `TRADING_EXIT_THREAD_INTERVAL_SECONDS` in the deploy configs (default 1.0)
+
+## STOP conditions
+- `manage()` shares mutable state with the trigger thread that can't be cleanly
+  isolated → STOP, report; a data race in the exit path is worse than 4s latency.
+- `manage()` is NOT idempotent and making it so requires restructuring the exit
+  engine → STOP, plan that separately; do not ship a fast loop that double-fires.
+- Any change would touch entry/sizing logic → out of scope.
+
+## Maintenance notes
+- Symmetry: entries on `trigger_loop`, exits on `exit_loop` — two fast daemon
+  threads + the slow scheduler for everything non-time-critical. Keep that shape.
+- If the owner later wants tick-level exits, the thread is the seam to feed a
+  streaming quote instead of a polled one — don't shorten the poll below ~0.5s
+  without moving to a push feed.
+- This is the last measurable execution-latency lever. Beyond it, remaining P&L
+  is strategy (edge), not speed — and the exhaustive search says the edge isn't
+  there for gap-ORB (see orderflow-shelf-verdicts). Speed makes the good days
+  cleaner; it does not create an edge.

plans/README.md

@@ -34,6 +34,7 @@ your row when done.
 | 015 | Backtest replay dashboard (/backtest) — theoretical entries/exits over past sessions under CURRENT settings, via Experiment-OS replay | P2 | L | 013 (chart component) | **DONE (2026-07-17, review-APPROVED first pass; commit 8ac222a on `worktree-agent-ae4d3f36e18c6e7a5` — awaiting owner merge). replay_window = assembly-only over harness primitives (same _entry_index/adverse-fill/R math); /api/backtest 45-session clamp + 10-min cache behind a dedicated lock; /backtest page with THEORETICAL banner + X-of-Y coverage line + per-trade drill-in (blue entry/purple exit markers). Deviations accepted: /api/chart gained backward-compatible date= anchor (72h cap couldn't reach old sessions); batching idiom taken from regime.py (plan's run() pointer was wrong).** |
 | 016 | EDGAR dilution enrichment — unbreak silent nightly refresh (UTC current_date bug + missing float_cache DDL) + pre-register dilution_veto_v1 keep-or-zero experiment | P2 | M | 006 | **DONE (2026-07-17, review-APPROVED first pass; f6999c7 merged). iv_dilution_gate keep-or-zero (fail-open on unfetched symbols, early results biased toward zero never inflated), dilution_veto_v1 self-registers via the experiments/*.json glob, nightly adjudication live. Prod backfill seeded same day (304 symbols swept, 245 filings / 91 symbols). FIRST ADJUDICATION 2026-07-17: **FAIL** (n=2044, expR_delta +0.0012 vs +0.05 bar) — but only 64/2069 fires (3%) had filing coverage (backfill covers recent actives only; historical fires read unflagged by fail-open design), AND the flagged fires ran BETTER than unflagged (baseline −0.038 vs −0.377, n=64): early evidence points AGAINST the dilution-trap thesis, not merely underpowered. Bar-design lesson: min_n counted ALL fires not flagged ones. Verdict terminal (no goalpost-moving); filings keep accruing nightly. IF revisited: dilution_veto_v2 with a flagged-fire min_n metric, after ~3-4 weeks coverage.** |
 | 017 | L2 order-book depth recorder **[VERDICT 2026-07-18: order-book overhang is REAL but too weak to trade. n=2191 under the real deployed exit: high-overhang fires -0.676R vs low -0.759R = +0.083R separation, SAME direction as the sniff (held from n=23 to n=2191 — genuinely real, not noise). BUT both buckets deeply negative; gating fails the random-gate null (delta +0.28 was mostly exposure-shrink). It reduces losses, does NOT create profit or beat a random gate. Live gap>=10% band: n=13/74, no signal. FAIL under the pre-registered bar. Same shape as the regime gate: real modest signal on a negative-EV base = less-bad, not positive. depth_features(2202) + code retained for a possible broader-population redesign; NOT wired live. Databento spend ~$16 free credits. REDESIGN LEAD REFUTED 2026-07-18: gap x overhang grid under real exit shows lower-gap is the WORST cell (0-3% -0.831R), not a hidden edge; NO positive cell exists anywhere. Exhaustive search complete — no positive subpopulation in gap-ORB.]** — the one untested selection axis; FREE-first (Phase 0-1 = $0 sniff test + recorder), paid feed only after the data earns it, pre-registered verdict before any live use | P1 | L | 006 | TODO (owner-approved to plan, no spend yet) |
+| 018 | Sub-second exit management on a dedicated thread — the last measurable execution-latency lever (exits leak P&L on fast small-caps; entries verified fine) | P1 | M | — | TODO |
 
 Status values: TODO | IN PROGRESS | DONE | BLOCKED (with one-line reason) | REJECTED (with one-line rationale)