Breaker: recoverable data-halt + network retries (DNS blip ended the day)

A transient WSL DNS outage made 3 consecutive equity reads fail, and the
fail-CLOSED breaker then PERMANENTLY halted new entries for the whole session
even after DNS recovered seconds later. Fixes:
- split the breaker into a permanent _halted (real daily-loss breach) and a
  recoverable _data_halt (equity unreadable) that clears on the next good read;
  raise the pause threshold to 5 (TRADING_EQUITY_FAIL_LIMIT).
- AlpacaPaperClient._request retries transient URLError/gaierror (not HTTP
  4xx/5xx) a couple times, so brief blips don't even register as failures.
- sync logs network failures as one-line warnings, not full tracebacks (the
  log was spamming stacks).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
momentum devandClaude Opus 4.8 committed 2026-06-17 15:03:35 -04:00
1 parent 9cc02e01ac
commit 45d5484dd7
4 files changed
+72 -28

No files matched your search

+12 -4
View File
@@ -58,15 +58,23 @@ def test_partial_fill_is_recorded():
assert len(store.query_events(event_type="order_filled", limit=None)) == 1
def test_breaker_fails_closed_when_equity_unreadable():
def test_breaker_pauses_then_recovers_when_equity_unreadable():
"""Equity-read outage PAUSES new entries (recoverable), not a permanent halt:
a transient DNS/network blip must not end the trading day."""
store = EventStore(":memory:")
client = _FakeClient(raise_account=True)
svc = TradingExecutionService(
store,
executor=AlpacaPaperExecutor(store, client=_FakeClient(raise_account=True)),
executor=AlpacaPaperExecutor(store, client=client),
settings=ExecutionSettings(max_daily_loss_pct=0.03),
session_id="cb",
)
svc._equity_fail_limit = 3 # tighten for the test
assert svc._daily_loss_breach() is False # 1 failure
assert svc._daily_loss_breach() is False # 2 failures
assert svc._daily_loss_breach() is True # 3 -> fail CLOSED
assert svc._halted is True
assert svc._daily_loss_breach() is True # 3 -> PAUSE (recoverable)
assert svc._data_halt is True and svc._halted is False # NOT a permanent halt
# equity readable again -> resume
client.raise_account = False
assert svc._daily_loss_breach() is False
assert svc._data_halt is False