Incident: entry-timeout cancels raced lazy sandbox fills -> untracked
positions; naked-stop guard then force-sold but every sell rejected
('more shares than your current long position') for hours.
1. translate_order: Tradier OTOCO/OTO parents carry no top-level
symbol/side/qty (legs only) — inherit from the entry leg so
symbol-scoped lookups can see brackets; get_orders symbols filter
also matches by leg symbols. Without this the stop-leg search read
a fully-protected position as naked.
2. close_position: cancel the symbol's resting sell orders/legs first —
Tradier reserves held shares against open sells, so selling without
releasing them always rejects.
3. cancel_entry (broker-agnostic): after cancelling, re-check the
broker's final word by client_order_id; journal a fill for any
executed shares instead of orphaning a real position (2026-07-06
class, same race exists on Alpaca).
49 tests pass incl. new coverage for all three.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Exit manager reads position current_price for breakeven/trail ratchets and
the stop-breach guard; Tradier sandbox quotes are 15-min delayed, so marks
lagged reality. Prefer the Alpaca data client's latest trades (real-time),
fall back to Tradier /markets/quotes per missing symbol.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
eod_reconcile, backfill_journal and the admin health block were hardwired
to AlpacaPaperClient — on a BROKER=tradier stack they would silently
reconcile/report against the wrong broker. make_trading_client() honors
BROKER env with no silent cross-broker fallback (missing Tradier keys =>
skip with reason, never fall back to Alpaca). Admin UI shows the active
broker label; JSON key stays 'alpaca' for compatibility.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Badge text was hardcoded to ALPACA; _broker_block now reports the active
broker name and derives paper/live from the Tradier sandbox host when
BROKER=tradier, keeping the base-URL-truth rule from plan 007.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fresh DB has no config_overrides row, so without the compose-level pin the
parallel stack falls back to the thin IEX feed and its rvol/volume decisions
diverge from the main stack (which carries feed=sip in the DB).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Self-contained shakeout stack beside the paper stack: own postgres (host
5434, disposable pgdata-tradier volume), own dashboard (host 8768), loop
with BROKER=tradier (sandbox). Data plane stays Alpaca; Telegram/deadman
disabled so main-stack alerts stay unambiguous; main data dir mounted
read-only (inherits learned_params/overrides.env). Filled local copy is
git-excluded; bootstrap_live_db.sh playbook reused for DB parity.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
/api/backtest now annotates each replay trade with the live loop's realized
P&L for the same (session, symbol) and returns a live_split summary (taken vs
not-taken n/sumR/avgR). Page grows two summary cells, a Live P&L column with
an amber ≠ marker when sim and live disagree on a trade's sign, and a footer
caveat: minute-bar granularity + adverse fills flip individual rows — trust
aggregates, not rows. Read-only annotation; cached replay result unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Read-only nightly-scheduler job (Friday-gated): buckets realized P&L by entry
time (open/late-morning/midday) and, for each market_exit close, computes the
post-exit run-up capturable under a breakeven stop (0 if price undercuts entry
first; intrabar ties count as stopped). Emits a weekly_tally event + Telegram
summary. Evidence accumulator for the dead no_new_entries_after cutoff and the
weakness-exit winner-clipping question — no live trading path touched.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The DB-config wiring was unnecessary there anyway — market_context reads
zero trading knobs; the guard test (plan 009) rightly rejected the import.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Version: CI now tags images YYYY.MM.DD-<shortsha> alongside :latest/:sha
and passes GIT_SHA/BUILD_TAG build-args; the running app reports them via
dashboard /api/health and admin /api/status (TrueNAS custom apps only show
'custom/1.0.0', so the app self-reports what it actually runs).
Config parity: every scheduler-run script (nightly_tune, eod_replay,
premarket_eval, regime, market_context, experiments cli) now loads
config_overrides at startup like the live services do — a bare env would
replay under DEFAULT exits (the documented 4.5x-baseline landmine).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Owner preference: the bulk of TRADING_*/TRIGGER_* logic switches should live
in the DB, not the compose file (keys + connection info stay in compose —
the DB address cannot live inside the DB, and secrets don't belong in a
table the stack can SELECT). Precedence at boot: compose env < overrides.env
< config_overrides table; every entrypoint already routes through
env_overrides.load(), which now applies the table last. Admin /api/config
and profile-apply write the table (secrets whitelist unchanged); the config
page shows source=database. DB down = silent fall-through to file/compose,
boot never blocks on config. One-shot migration:
python -m runtime.env_overrides seed-from-env
run in any stack container to copy its current effective knobs into the
table, after which the compose lines are dead weight strippable at the next
redeploy.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The 07-22 'tuner NOT running' alert could not be diagnosed remotely: the
heartbeat is a file on the scheduler container's volume, invisible to
everything else, and indistinguishable from a stale scheduler image or a
DATA_DIR mismatch. _write_heartbeat now also inserts a telemetry_events
row (best-effort); the boot staleness check falls back to the DB row when
the file is absent, so 'tuner ran, no profitable config' stops looking
like 'tuner dead'.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
docs/LIVE-CUTOVER.md: full flip-day procedure — staged pieces, discovered
dependencies (second paper key pair, DB seeding, learned_params copy,
telegram token, ports/dirs), 15-min cutover with the paper stack running
throughout, verification list, rollback. bootstrap_live_db.sh seeds the
live stack's Postgres from the paper nightly dump and truncates the
money-record tables so research history carries over while live P&L
starts at zero.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
TRADING_EQUITY_FLOOR (0=off): equity at/below an absolute dollar floor
permanently halts the session and flattens the book — the total-drawdown
backstop the per-day breaker can't provide (a slow bleed of -2.9% days
never trips it). Same flatten path, loud risk event, bounce does not
re-open.
TRADING_BOOT_RECONCILE (default on): at boot, diff the broker's open
orders against today's submitted events — an open BUY on an unheld symbol
this process never submitted is a crash-restart orphan (the 2026-07-06
duplicate-order incident) and is cancelled. Orders on held symbols are
never touched (cancelling a bracket parent strips its protective stop).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Prices each closed round-trip against the nearest quote_snapshots NBBO:
per-trade spread cost, book-walk flag (order size > displayed ask), and a
cost-adjusted P&L total. Measured this week: 0.3-0.5% median spreads on
gate-passing names, ~12% profit drag — far below the static 2-4% microcap
proxy. Coverage gaps (quote recorder only follows tracked names) reported
per-trade, not silently dropped.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The stored minute-bar history is 273 sessions (~13 months); the 45-session
clamp blocked months-scale ranges now that the nightly labeler keeps
setup_events current. Requests stay serialized behind _backtest_lock; a
full-history run takes minutes by operator choice.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Dashboard backtest sources sessions from setup_events v1, which only the
labeler writes — and the labeler was never in the nightly scheduler, so
setups (and therefore backtests) froze at the last manual run (07-17).
Root blocker was _insert crashing on duplicate setup_ids on any re-run;
now ON CONFLICT DO NOTHING, so incremental nightly builds are safe.
Scheduler gains 'labeler build' + 'build-tod1' after eod_replay in both
compose files.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The prior commit imported query_loop_session but a botched idempotency
guard skipped inserting the snapshot field (the import string itself
matched the 'already present' check) — ruff unused-import failed CI and
the stale-session banner had no data. Field added; banner live.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The 07-21 morning outage (loop booted post-EOD on the 20th, survived
midnight, silently blocked every entry while signals went READY) had zero
UI surface. New query_loop_session projection parses the live session id's
embedded date; a red banner fires when it isn't today's — the exact day-2
poison signature — telling the operator to restart the loop. Armed
Triggers rows now also show rvol_tod (the arming signal since 5597250)
next to legacy rvol.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The 07-21 misses (CPHI +417%, DFNS +94%, GREE/WGRX +40%) were all static-gap
<10% names the validated gapper population excludes by design, but they share
a signature no refuted detector (liq1/run1) measured: rvol_tod (cum volume vs
median-20d prorated by elapsed session) at 50-1000x on a flat open. New
labeler setup tod1 fires on the rising edge of rvol_tod>=50 AND price>=
prev_close*1.20, restricted to static gap <10%, stop = 5-bar swing low,
scored via the shared _forward_labels. Shadow-only; pre-registered promotion
bar locked before forward data (median fwdmax60 >=+10%, median dd60 >-6%,
+1R>50%, top stratum rvol_tod>=200 clears alone).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
SNXX 2026-07-21: 18s after a mid-morning restart the fast path bought a
15-minute-old break +1.6% above the trigger — fires() only checked
price >= trigger, so any blind window (boot, halt, feed outage) fired
instantly on already-extended moves. A trigger now fires only if the live
price was seen BELOW the trigger since arming (a real cross), with a
small at-the-break tolerance (TRIGGER_FRESH_CROSS_TOL, default 1.5%)
so a fast break between ticks still fires; beyond it the name must pull
back below the level and re-cross. Gap-through runners stay covered by
the slow marketable path.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A process booted AFTER the day's EOD flatten never gets exit-after-flatten,
survives midnight, and every memory-only daily flag (_session_closed, entry
caps, benches) stays stale — the documented day-2 limitation, hit live at the
2026-07-21 open (booted 17:11 the day before; signals READY + book armed but
zero gate evaluations, both entry paths silently blocked by _session_closed).
Main loop now exits cleanly when the session date rolls over; compose restarts
a fresh process that rebuilds the new day's state.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Day-22 forensic (BIYA session): 6 entry submissions, 0 fills — IREX missed
+1.7R, IRE +1.65R by one cent. Two root causes fixed:
1. Slow-path limit rested AT the trigger while price ran 2% past it, and
retries reused the same stale price. Marketable model now prices from
max(trigger, live last) via price_provider; the anti-chase extension
gate gets the live ref too (was an entry_ref==trigger no-op on this
path). TRADING_ENTRY_FILL_MODEL flipped to marketable.
2. Fast trigger path armed nothing all day: legacy rvol (cum-so-far /
full-day MEAN) is ~0.1-0.3 minutes after the open for a normal heavy
name and its mean is poisoned by reverse-split/prior-squeeze days
(BIYA legacy 1.27 vs real activity 481x median). New rvol_tod =
cum / (MEDIAN(20d) * elapsed-session fraction) drives arm eligibility
and the scanner keep-filter; legacy rvol untouched for validated
boundaries (gap-squeeze rvol>=5).
Also: stagnation time-stop (TRADING_EXIT_STAGNATION_MIN/_R, default OFF)
in simulate_exit + manage_live with sweep_exits.py variants for
validation before any live enable.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Owner asks on the candlesticks:
- VOLUME: histogram series (bottom 20% overlay), coloured by candle direction.
- BOLLINGER BANDS: 20-period 2-sigma, computed client-side from closes; price
riding the upper band = strong, break back inside = fading (momentum context).
- EXIT markers were NEVER wired — the fills feed carries only entries
(order_submitted), and the loop fills brackets via position-diffing so no
order_filled fires. Exits live in position_closed; now surfaced as square
markers with reason + P&L (green win / red loss). Entry 'expected' (unfilled
submit) markers labelled distinctly. Fixed a self-inflicted bug: queried the
event store AFTER closing it — moved the position_closed fetch inside the
open-store block. 11 chart tests green incl. new exit-marker test.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The self-heal commit (a00f967) inserted the _research method INSIDE __init__ at
method-body indent, truncating it so _change_signal/_backtest_lock/_backtest_
cache never got set -> AttributeError on backtest. Moved _research to a sibling
method; __init__ now sets all attrs. Two chart/sparkline tests updated: a None
research_con now REOPENS lazily, so 'no research db attached' is reached only
when the reopen also fails (they monkeypatch open to fail). 22 api tests green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The charts weren't broken — the data pipe was empty. dashboard_api opened the
research connection ONCE at boot; if the DB was unreachable then (container
started before Postgres, or a blip) it stayed None forever, so /api/chart and
/api/bars returned 'no research db attached' and every candlestick + sparkline
rendered blank. Live-verified: GET /api/chart returned bars:[] reason:'no
research db attached' on the deployed dashboard. Now DashboardState._research()
opens the connection LAZILY and reopens if None — same self-heal as the
event-store emit fix. 2 tests; chart/backtest endpoint tests still green.
No library change: Lightweight Charts v5.2 is fine, it just had nothing to draw.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The test compared len(w) and list(w) as two SEPARATE locked reads, so the
churn thread could mutate between them (30 vs 31) — a bug in the test, not the
Watchlist (each read is individually consistent; there is no torn-read
guarantee ACROSS two calls). Now snapshots once and asserts that snapshot never
tears + concurrent contains/slice never raise. 5/5 green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Telemetry: discovery took 6-30s ON the single scheduler, freezing ingest/arm
so the trigger read stale bars — the real 'late edge detection'. Now discovery
runs on its own daemon thread with a dedicated DB connection; the watchlist it
mutates is a lock-guarded Watchlist (list-like drop-in, no torn reads); the
scheduler discover step auto-disables (single-writer). Loop is now 3 fast
threads (entries/exits/discovery) + slow scheduler for tolerant work.
Also: nightly_tune writes a heartbeat every run, so the boot alert distinguishes
'tuner not running' (real) from 'ran, no profitable ORB config, kept params'
(expected — the strategy has no edge). Kills the misleading Telegram alarm.
543 tests green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Exits move off the single-threaded scheduler onto their own daemon thread at
1s (mirrors the fast trigger thread). Dedicated research connection (shared
facade isn't thread-safe); scheduler exits step auto-disabled when the thread
runs (single-caller invariant — no racing instance state, no double-submit);
manage() already idempotent (stop ratchets >1c up only). 3 driver tests + 539
suite green. On fast small-caps this is the 'get out fast' the owner asked for.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Owner: 'make all decisions as fast as possible.' The real entry-detection
ceiling was ingest — bars refreshed every 20s, so the 4s trigger thread read
data up to 20s stale and detected breakouts late. Ingest is incremental+cheap,
safe at 5s. Fresher bars + faster arm (trigger levels) + 2s trigger check +
the 4s exits shipped earlier = the whole ingest->detect->decide->exit loop
tightened. One batched data call per cycle so Alpaca rate limits are fine.
Sub-second exit management on a dedicated thread is the next (code) step.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Owner's real complaint: late exits gave back gains. Exits ran on the slow
scheduler at 12s while entries run on the 4s fast thread — exits were 3x
slower than entries. A small-cap can spike and round-trip inside 12s. Drop
TRADING_EXIT_MANAGE_INTERVAL_SECONDS to 4s (config, reversible, both compose
files). Deeper fix (exit mgmt on the sub-second trigger thread) planned
separately. Entry chase-tail (24% fill >5% above break) is the next
investigation.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sliced 2191 depth fires gap x overhang under real exit: lower-gap is WORST
(0-3% -0.831R), inverting the low-gap-edge hypothesis. Overhang helps a little
everywhere, nothing crosses zero. Exhaustive: 13 levers + regime + depth +
this grid = zero positive. No positive subpopulation in gap-ORB $2-20.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Definitive test, n=2191 real deployed exit: ask-overhang separates +0.083R
(high -0.676 vs low -0.759, direction held sniff->scale = genuinely real) but
fails the random-gate null — it shrinks losses, never creates profit. Live
gap>=10% band n=13/74 = no signal. Same shape as regime gate on a negative-EV
base. 13 conversion levers + regime + depth tested, zero make it positive.
Real-money rec: NO. Depth edge argues for a broader-population redesign as
future research, not a live fix.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The 2026-07-17 20:23 postgres blip left the live loop writing zero events for
hours: emit() executed on a RAW cursor, bypassing PgConnection._run's
reconnect. cursor() doesn't round-trip so the dead conn went unnoticed until
the raw execute failed with no retry, and self._conn was never rebuilt.
Routing emit through the facade execute() wraps the round-trip in _run, so a
DB blip reconnects on the next write instead of running blind. 2 reconnect
tests + 34 storage tests green. (Deeper 'halt trading when persistently
unable to write' safety guard queued separately.)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Plan 017 Phase 0->3. Databento depth backfilled for 602 fires across 21
sessions -> depth_features table. Reweighted to true 23% population win rate,
gating on ask-overhang-above-break (keep>=0.034) lifts win rate 23->28% and
clears the random-gate null (+0.057 vs +0.05 on the conservative +1/-1
bracket). iv_depth_overhang_gate = keep-or-zero, skip-on-no-coverage (the
dilution-v1 lesson). This commit registers the REAL-exit harness run
(simulate_exit asymmetric trail) for the true live delta. Counter-intuitive
mechanism: more resting supply above the break = MORE likely to win
(absorption/fuel), invisible in bar data — the owner's 'size stacked above'
instinct, proven.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
First genuine selection signal in the project. 40 real fires, Databento
XNAS.ITCH book at each trigger: naive imbalance = noise (AUC 0.45/0.50), but
ask-size resting within 1% ABOVE the break separates winners at AUC 0.80
(winner median 0.276 vs loser 0.005; median split 27% vs 67% win). Inverted
from supply-trap intuition = absorption/fuel mechanism, invisible in OHLCV.
n=23 single-day sniff -> GO to Phase 1 recorder, not to trading. Cost: pennies.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The gate is damage-control, not an edge source: it can only allocate the
strategy's existing (negative) edge onto least-bad days — best case flat,
never gain. Its value is CONDITIONAL: a holding pattern until an entry edge
(plan 017 L2 depth) is found, at which point it flips to force-multiplier
(good names x good days). Owner's 'it just turns the bot off' critique is
correct at the current no-edge base.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Cost-checked (no data spent): DATABENTO_API_KEY reaches XNAS.ITCH TotalView
10-level depth; every real gapper (EHGO/ATPC/MUD/LASE/AEON) covered at
$0.0001-0.0066/fire, full 20-fire sniff <$1 inside free credits. Catalogs
all 13 provisioned-but-unwired keys with a wire-on-demand triage (Databento
= game-changer; 4 news APIs = skip; rest redundant).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The one untested selection axis (order-book depth = the owner's 'size stacked
above the break' instinct). Staged so $0 is spent until a free sniff test +
recorder prove the signal exists and covers the universe; paid feed is an
explicit owner gate; the signal faces the same pre-registered bar + random-gate
null as every refuted lever before it touches the live entry. Honest ceiling
stated: a PASS nudges odds, is not an autonomous oracle.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Owner saw a blank /backtest — it was CORRECT (the regime gate sat out every
recent session) but unexplained. Now: ?regime=0|1 override on /api/backtest,
an 'apply regime gate (as deployed)' checkbox, and an empty-state message
saying WHY there are no trades and what to click. Gated view = what the bot
would do; ungated = the raw simple criteria.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
