From the multi-agent audit (ship-now, low-overfit-risk cluster):
INGESTION RESILIENCY
- client._request: 429/5xx now retry with Retry-After + exponential backoff
(previously hard-failed, feeding silent-stale ingest).
- SIP backfill feed (settings.backfill_feed='sip'): live realtime stays IEX, but
backtest/backfill ingestion uses the full consolidated tape — IEX sees only
~2-5% of true volume (verified: AAPL 22,797 vs 452,704), which was silently
corrupting every rvol/$-volume decision and backtest. eod_replay wired to it.
- step_ingest freshness watchdog: emits a STALE event + board note when the
newest bar is older than INGEST_STALE_SECONDS during RTH (45% of passes
historically returned 0 rows with no error). Optional hard arm-block
(INGEST_STALE_BLOCK, off by default).
RISK
- Portfolio gross-notional cap (TRADING_MAX_GROSS_NOTIONAL_PCT=0.60): both sizing
paths now cap each entry to the remaining gross budget, so 6 x 16% can't pile
the book into ~96% gross of correlated small-caps. Blocks/shrinks late entries.
- Fixed the duplicate TRADING_MAX_CONCURRENT_POSITIONS in .env.example.
MONITORING (UI)
- Naked/unprotected-position alarm: _unprotected_positions joins the latest
position + order snapshots (order sync now flattens legs) and the TUI board
shows a red UNPROTECTED banner the instant a held long lacks stop coverage —
the exact blind spot when APWC/NOWL went naked. Verified it flags today's
3 overnight positions.
Tests: gross-notional cap blocks when budget full; 50 pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Today's-trades reconstruction showed normally-exited positions as falsely
"open" and understated realized P&L. Two causes:
1. sync_orders recorded only TOP-LEVEL orders. A position closed by its bracket
STOP or TAKE-PROFIT leg has that exit fill as a child LEG, so every stop/TP
exit was invisible — QTEX/BBD/UWMC/etc. looked open when they were flat, and
their realized P&L was dropped. Now flatten parent + filled legs (legs inherit
the parent symbol).
2. get_orders only fetched the most-recent page; Alpaca's `limit` truncates by
recency (legs count toward the cap), so a busy day could drop the morning's
fills entirely. Unscoped get_orders now pages backwards with `until` (dedup by
id) to capture the full day; the symbol-scoped path stays single-page.
Verified live: journal open set now matches the broker exactly
(ATPC/GRAB/LNKS/NOWL/WPRT) and every reconstructed round-trip ties to a real
broker fill. NB: the realized total is FIFO-lot based, so while positions are
partially open it won't equal the broker's average-cost day P&L — the broker
figure remains the authoritative headline.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Alpaca's orders `limit` keeps only the most-RECENT orders and counts child
legs toward the cap. On a busy day the order list runs to hundreds (every
backed-out entry is an order), so the exit manager's limit=100 snapshot
silently dropped the stop legs of older OPEN positions. A stop-leg lookup then
read "no stop" on a fully-protected position -> broken trailing/profit-lock for
that name (and false-naked alarms in monitoring).
Fix: get_orders gains a `symbols` filter; the exit manager scopes its snapshot
to the handful of held names (no truncation possible). Verified live: an
unscoped limit=100 query reported PLUG stop=None while the scoped query
correctly surfaced PLUG's stop @2.65.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The get_positions TTL cache (added for the speed fix) silently broke the
naked-stop guard: _broker_held_symbols read 2.5s-stale positions, so a
just-filled entry wasn't seen as held, the guard thought it was unfilled, and
cancelled its bracket on a price dip -> NAKED position (APWC bled -12.8%
unprotected today). Fixes:
- client.get_positions(fresh=True) bypasses the cache; the guard uses it.
- entry_grace_seconds (default 5): never cancel an entry within N seconds of
arming, so a marketable fill can confirm before the bracket could be stripped.
Regression test added.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The 2.5s value-cache + fewer calls only halved the slowness; the real cause was
flaky WSL DNS (~5s on ~1 in 5 resolves) and urllib re-resolving on EVERY
request. Cache successful DNS lookups process-wide (5min TTL) so each Alpaca
host is resolved at most once per window. Measured: get_orders went from
intermittent 5s spikes to a steady ~200ms. Opt out via ALPACA_DNS_CACHE=0.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The single-threaded passes (and the trigger thread) ballooned to 60-120s.
Cause: flaky WSL DNS makes ~1 in 5 Alpaca calls take ~5s to resolve, and each
pass fired ~20 calls (breaker, sync, guard, exit manager all re-reading
account/positions/orders). Fixes:
- client: 2.5s TTL cache for get_account/get_positions (invalidated on any
order mutation) — collapses the redundant per-pass reads to one.
- exit manager: fetch the (heavy) status=all orders snapshot ONCE per pass and
reuse it for every position's stop-leg + entry-time lookup (was 2 calls/pos).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A transient WSL DNS outage made 3 consecutive equity reads fail, and the
fail-CLOSED breaker then PERMANENTLY halted new entries for the whole session
even after DNS recovered seconds later. Fixes:
- split the breaker into a permanent _halted (real daily-loss breach) and a
recoverable _data_halt (equity unreadable) that clears on the next good read;
raise the pause threshold to 5 (TRADING_EQUITY_FAIL_LIMIT).
- AlpacaPaperClient._request retries transient URLError/gaierror (not HTTP
4xx/5xx) a couple times, so brief blips don't even register as failures.
- sync logs network failures as one-line warnings, not full tracebacks (the
log was spamming stacks).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the static -1R/+2R bracket with active, parameterised exit management
shared by the backtest and live (strategy/exits.py): move to breakeven, trail
under prior-bar lows or by %, scale out, or first-red exit.
Swept all variants over 187 sessions with a held-out train/test split
(sweep_exits.py), then an adversarial 3-judge workflow picked the robust winner
OUT-OF-SAMPLE: trailLow_a1 (trail under the prior bar's low after +1R) — test
avgR 0.128 vs the old static_2R 0.067 (~1.9x), win 49% vs 38%. The naive
in-sample winner (trailPct10) was correctly rejected as overfit
(train 0.684 -> test 0.085).
- strategy/exits.py: ExitConfig + simulate_exit (backtest) + manage_live (live)
- strategy/backtest/engine.py: simulate the managed exit on entry; Trade.entry_index
- runtime/exit_manager.py: live manager — ratchets the broker stop UP via
REPLACE/PATCH (never cancel = never naked), scales out, first-red exit
- alpaca_paper/client.py: replace_order (PATCH) + partial close_position
- run_live_paper.py: 'exits' step (12s); TRIGGER_MAX_ARMED 6 -> 10
- tests: exits sim/live + manager ratchet-never-cancel
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
HIGH:
- Circuit breaker fails CLOSED: a repeated unreadable equity (API 401/429/5xx or
missing field) now halts new entries after 3 consecutive failures instead of
silently reporting 'no breach' and continuing to trade through a drawdown.
- execute(): map order statuses. Terminal-fail (rejected/expired/...) -> ok=False
(never armed as a resting entry); record ANY fill (partial too), not only an
exact 'filled'; require a real broker order id. approve_order only arms
genuinely OPEN statuses.
- sync: account/positions/orders failures now emit a BrokerHealthChangedEvent
(degraded) and step_sync surfaces 'DEGRADED', instead of a success-shaped None
that left risk gates reading stale snapshots as current.
MEDIUM:
- _flatten_all records a position closed only on a CONFIRMED close; the breaker
event reports 'halted_flatten_incomplete' + the unclosed symbols when a close
fails, instead of claiming the book is flat.
- submit_exit_order prefers close_position (handles short sign + fractional qty;
a plain sell of abs(qty) would double a short).
LOW:
- get_orders(nested=true) so bracket child legs aren't dropped from the snapshot.
+ tests/unit/test_execution_safety.py (4 regressions). 75/75 green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The live 'symbols' list only ever grew (discover appends every 5 min, dashboard
injects add more) and was comma-joined into ONE bars request; once >~100 symbols
Alpaca 400s the whole batch -> ALL minute-bar ingestion silently stops mid-session.
- client.get_minute_bars / get_daily_bars / get_latest_trades batch symbols in
chunks of 100 (cap-safe) and merge; get_daily_bars also paginates now.
- run_live_paper bounds the watchlist (WATCHER_MAX_UNIVERSE, default 80) so it
rotates toward current movers instead of growing all day.
Found by the silent-failure audit workflow. Verified _chunked(250)->[100,100,50],
live calls intact. 71/71 tests green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Snapshot before the DuckDB->PostgreSQL migration. data/, archive/, the 232MB
rebuild zip, the Windows .venv, and .env are excluded via .gitignore.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
