From 375973fe21123dbd4d0f1701188ef39fa60dbda2 Mon Sep 17 00:00:00 2001 From: momentum dev Date: Sat, 18 Jul 2026 22:05:33 -0500 Subject: [PATCH] =?UTF-8?q?plans(018):=20sub-second=20exit=20thread=20?= =?UTF-8?q?=E2=80=94=20the=20last=20real=20execution-latency=20lever?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5 --- plans/018-subsecond-exit-thread.md | 122 +++++++++++++++++++++++++++++ plans/README.md | 1 + 2 files changed, 123 insertions(+) create mode 100644 plans/018-subsecond-exit-thread.md diff --git a/plans/018-subsecond-exit-thread.md b/plans/018-subsecond-exit-thread.md new file mode 100644 index 0000000..4d18f9b --- /dev/null +++ b/plans/018-subsecond-exit-thread.md @@ -0,0 +1,122 @@ +# Plan 018: Sub-second exit management on a dedicated thread + +> **Executor instructions**: Live money path — TDD every change, run the exit +> tests before/after, honor STOP conditions. When done, update +> `plans/README.md`. +> +> **Drift check**: `git diff --stat 680ddc2..HEAD -- run_live_paper.py strategy/ tests/` + +## Status +- **Priority**: P1 (owner's core ask: get out of trades fast; exits leak the P&L on fast small-caps) +- **Effort**: M +- **Risk**: MED-HIGH (touches the live exit execution path) +- **Depends on**: none (config speedups already shipped 680ddc2) +- **Category**: latency / money-path +- **Planned at**: commit `680ddc2`, 2026-07-18 + +## Why this matters + +Exit management (`LiveExitManager.manage()` — breakeven ratchet, trail, tier +scale-out, catastrophe/first-red) runs as a step on the **single-threaded main +scheduler**. Even at the new 4s interval it (a) queues behind slower steps +(ingest, discover, news) that share the thread, so real reaction can drift well +past 4s, and (b) is bounded by the scheduler tick. On a $2–20 small-cap that +spikes and round-trips in seconds, a stop or profit-tier that reacts "sometime +in the next 4–10s" gives back the gain. This is the owner's #1 felt problem +("late exits, gave back gains"), and unlike entry timing (verified fine, memory +`execution-plus-judgment-mode`), exit latency is real and measurable. + +The fix: run exit management on its **own dedicated thread** (like the fast +trigger thread already does for entries), at a sub-second-to-1s cadence, +independent of the scheduler queue. Entries already got this treatment +(`trigger_loop` on its own thread); exits deserve the symmetric fix. + +## Current state + +- `run_live_paper.py`: + - `trigger_loop()` (~line 1078) — the PATTERN to mirror: own daemon thread, + `while not stop_event.wait(interval): _trigger_pass()`, wrapped in + try/except so one error never kills the loop. Started ~line 1339. + - `exit_mgr = LiveExitManager(client, store, _rth_bars, cfg, session_id)` + (~line 1097) — the object whose `.manage()` does all exit work. + - `step_manage_exits()` (~line 1128) calls `exit_mgr.manage()` on the + scheduler at `TRADING_EXIT_MANAGE_INTERVAL_SECONDS` (now 4s). +- `strategy/.../LiveExitManager.manage()` — reads current bars/price per + position, computes desired stop/target/scale, submits broker orders. MUST be + audited for thread-safety before running it concurrently with the trigger + thread and scheduler (shared broker client, shared store). + +## Steps + +### Step 1: Audit thread-safety of the shared resources +Before adding a thread, confirm what `manage()` touches that the trigger +thread / scheduler also touch: the Alpaca `client` (HTTP — usually fine, +confirm no shared mutable cursor), `rt["store"]` (EventStore — has a `_lock` +around emit, good), `research_con` (the facade — NOT thread-safe; if +`manage()` reads bars via a shared `research_con`, it needs its OWN connection +like `step_quotes`/catalyst provider do, per the memory note +`entry-path-fast-vs-auto` about dedicated connections). Document findings; if +`manage()` shares `research_con`, Step 2 must give the exit thread its own. + +**Verify**: written audit; grep every attribute `manage()` reads. + +### Step 2: Move exit management to its own thread +- Add `exit_interval = float(os.environ.get("TRADING_EXIT_THREAD_INTERVAL_SECONDS", "1.0"))`. +- `def exit_loop(): while not stop_event.wait(exit_interval): try: exit_mgr.manage() except: log` — mirror `trigger_loop` exactly. +- Give the exit thread its OWN research DB connection if Step 1 found sharing + (open it inside the thread, single-threaded use, like the enrich/quote paths). +- Start it next to the trigger thread (~line 1339), gated on + `_flag("TRADING_EXECUTION_ENABLED","1")` and `client is not None`. +- REMOVE the `scheduler.add("exits", ...)` step (or gate it off) so exits don't + run on BOTH the thread and the scheduler (double-submit risk). +- Keep `TRADING_EXIT_MANAGE_INTERVAL_SECONDS` honored as a fallback only if the + thread is disabled. + +**Verify**: unit test — a fake exit_mgr whose `.manage()` counts calls, driven +by a short-interval loop, is invoked repeatedly off the scheduler; the +scheduler no longer registers an "exits" step (grep the scheduler.add list). + +### Step 3: Idempotency / double-submit guard +At 1s cadence the manager runs ~4× more often. Confirm `manage()` is idempotent +— it must not re-submit a stop/scale it already placed (check for an existing +resting order before submitting; `get_orders(symbols=...)` scoped, per the +`get-orders-truncation` memory). If it isn't already idempotent, add the guard. +This is the highest-risk correctness point — a fast loop that re-fires orders +is worse than a slow one. + +**Verify**: test that two back-to-back `manage()` calls on the same position +with an already-resting protective stop submit the stop ONCE. + +### Step 4: Backpressure / cost check +1s cadence × ~10 positions = more broker calls. Confirm within Alpaca limits +(scope reads by `symbols=`, reuse the position snapshot the sync step already +fetches rather than re-polling per position). If `manage()` is heavy, cap the +effective rate or reuse cached quotes. + +**Verify**: measure `manage()` wall-time; at 1s interval it must finish well +inside the interval for the real position count. + +## Done criteria +- [ ] Exit management runs on its own daemon thread at ~1s, off the scheduler +- [ ] `manage()` thread-safe (own DB connection if needed) + idempotent (no double-submit) +- [ ] Scheduler no longer runs a competing "exits" step +- [ ] Exit tests green; full suite green; `ruff --select F` clean +- [ ] `TRADING_EXIT_THREAD_INTERVAL_SECONDS` in the deploy configs (default 1.0) + +## STOP conditions +- `manage()` shares mutable state with the trigger thread that can't be cleanly + isolated → STOP, report; a data race in the exit path is worse than 4s latency. +- `manage()` is NOT idempotent and making it so requires restructuring the exit + engine → STOP, plan that separately; do not ship a fast loop that double-fires. +- Any change would touch entry/sizing logic → out of scope. + +## Maintenance notes +- Symmetry: entries on `trigger_loop`, exits on `exit_loop` — two fast daemon + threads + the slow scheduler for everything non-time-critical. Keep that shape. +- If the owner later wants tick-level exits, the thread is the seam to feed a + streaming quote instead of a polled one — don't shorten the poll below ~0.5s + without moving to a push feed. +- This is the last measurable execution-latency lever. Beyond it, remaining P&L + is strategy (edge), not speed — and the exhaustive search says the edge isn't + there for gap-ORB (see orderflow-shelf-verdicts). Speed makes the good days + cleaner; it does not create an edge. diff --git a/plans/README.md b/plans/README.md index 54da646..ffd458c 100644 --- a/plans/README.md +++ b/plans/README.md @@ -34,6 +34,7 @@ your row when done. | 015 | Backtest replay dashboard (/backtest) — theoretical entries/exits over past sessions under CURRENT settings, via Experiment-OS replay | P2 | L | 013 (chart component) | **DONE (2026-07-17, review-APPROVED first pass; commit 8ac222a on `worktree-agent-ae4d3f36e18c6e7a5` — awaiting owner merge). replay_window = assembly-only over harness primitives (same _entry_index/adverse-fill/R math); /api/backtest 45-session clamp + 10-min cache behind a dedicated lock; /backtest page with THEORETICAL banner + X-of-Y coverage line + per-trade drill-in (blue entry/purple exit markers). Deviations accepted: /api/chart gained backward-compatible date= anchor (72h cap couldn't reach old sessions); batching idiom taken from regime.py (plan's run() pointer was wrong).** | | 016 | EDGAR dilution enrichment — unbreak silent nightly refresh (UTC current_date bug + missing float_cache DDL) + pre-register dilution_veto_v1 keep-or-zero experiment | P2 | M | 006 | **DONE (2026-07-17, review-APPROVED first pass; f6999c7 merged). iv_dilution_gate keep-or-zero (fail-open on unfetched symbols, early results biased toward zero never inflated), dilution_veto_v1 self-registers via the experiments/*.json glob, nightly adjudication live. Prod backfill seeded same day (304 symbols swept, 245 filings / 91 symbols). FIRST ADJUDICATION 2026-07-17: **FAIL** (n=2044, expR_delta +0.0012 vs +0.05 bar) — but only 64/2069 fires (3%) had filing coverage (backfill covers recent actives only; historical fires read unflagged by fail-open design), AND the flagged fires ran BETTER than unflagged (baseline −0.038 vs −0.377, n=64): early evidence points AGAINST the dilution-trap thesis, not merely underpowered. Bar-design lesson: min_n counted ALL fires not flagged ones. Verdict terminal (no goalpost-moving); filings keep accruing nightly. IF revisited: dilution_veto_v2 with a flagged-fire min_n metric, after ~3-4 weeks coverage.** | | 017 | L2 order-book depth recorder **[VERDICT 2026-07-18: order-book overhang is REAL but too weak to trade. n=2191 under the real deployed exit: high-overhang fires -0.676R vs low -0.759R = +0.083R separation, SAME direction as the sniff (held from n=23 to n=2191 — genuinely real, not noise). BUT both buckets deeply negative; gating fails the random-gate null (delta +0.28 was mostly exposure-shrink). It reduces losses, does NOT create profit or beat a random gate. Live gap>=10% band: n=13/74, no signal. FAIL under the pre-registered bar. Same shape as the regime gate: real modest signal on a negative-EV base = less-bad, not positive. depth_features(2202) + code retained for a possible broader-population redesign; NOT wired live. Databento spend ~$16 free credits. REDESIGN LEAD REFUTED 2026-07-18: gap x overhang grid under real exit shows lower-gap is the WORST cell (0-3% -0.831R), not a hidden edge; NO positive cell exists anywhere. Exhaustive search complete — no positive subpopulation in gap-ORB.]** — the one untested selection axis; FREE-first (Phase 0-1 = $0 sniff test + recorder), paid feed only after the data earns it, pre-registered verdict before any live use | P1 | L | 006 | TODO (owner-approved to plan, no spend yet) | +| 018 | Sub-second exit management on a dedicated thread — the last measurable execution-latency lever (exits leak P&L on fast small-caps; entries verified fine) | P1 | M | — | TODO | Status values: TODO | IN PROGRESS | DONE | BLOCKED (with one-line reason) | REJECTED (with one-line rationale)