Owner: 'make all decisions as fast as possible.' The real entry-detection
ceiling was ingest — bars refreshed every 20s, so the 4s trigger thread read
data up to 20s stale and detected breakouts late. Ingest is incremental+cheap,
safe at 5s. Fresher bars + faster arm (trigger levels) + 2s trigger check +
the 4s exits shipped earlier = the whole ingest->detect->decide->exit loop
tightened. One batched data call per cycle so Alpaca rate limits are fine.
Sub-second exit management on a dedicated thread is the next (code) step.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Owner's real complaint: late exits gave back gains. Exits ran on the slow
scheduler at 12s while entries run on the 4s fast thread — exits were 3x
slower than entries. A small-cap can spike and round-trip inside 12s. Drop
TRADING_EXIT_MANAGE_INTERVAL_SECONDS to 4s (config, reversible, both compose
files). Deeper fix (exit mgmt on the sub-second trigger thread) planned
separately. Entry chase-tail (24% fill >5% above break) is the next
investigation.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sliced 2191 depth fires gap x overhang under real exit: lower-gap is WORST
(0-3% -0.831R), inverting the low-gap-edge hypothesis. Overhang helps a little
everywhere, nothing crosses zero. Exhaustive: 13 levers + regime + depth +
this grid = zero positive. No positive subpopulation in gap-ORB $2-20.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Definitive test, n=2191 real deployed exit: ask-overhang separates +0.083R
(high -0.676 vs low -0.759, direction held sniff->scale = genuinely real) but
fails the random-gate null — it shrinks losses, never creates profit. Live
gap>=10% band n=13/74 = no signal. Same shape as regime gate on a negative-EV
base. 13 conversion levers + regime + depth tested, zero make it positive.
Real-money rec: NO. Depth edge argues for a broader-population redesign as
future research, not a live fix.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The 2026-07-17 20:23 postgres blip left the live loop writing zero events for
hours: emit() executed on a RAW cursor, bypassing PgConnection._run's
reconnect. cursor() doesn't round-trip so the dead conn went unnoticed until
the raw execute failed with no retry, and self._conn was never rebuilt.
Routing emit through the facade execute() wraps the round-trip in _run, so a
DB blip reconnects on the next write instead of running blind. 2 reconnect
tests + 34 storage tests green. (Deeper 'halt trading when persistently
unable to write' safety guard queued separately.)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Plan 017 Phase 0->3. Databento depth backfilled for 602 fires across 21
sessions -> depth_features table. Reweighted to true 23% population win rate,
gating on ask-overhang-above-break (keep>=0.034) lifts win rate 23->28% and
clears the random-gate null (+0.057 vs +0.05 on the conservative +1/-1
bracket). iv_depth_overhang_gate = keep-or-zero, skip-on-no-coverage (the
dilution-v1 lesson). This commit registers the REAL-exit harness run
(simulate_exit asymmetric trail) for the true live delta. Counter-intuitive
mechanism: more resting supply above the break = MORE likely to win
(absorption/fuel), invisible in bar data — the owner's 'size stacked above'
instinct, proven.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
First genuine selection signal in the project. 40 real fires, Databento
XNAS.ITCH book at each trigger: naive imbalance = noise (AUC 0.45/0.50), but
ask-size resting within 1% ABOVE the break separates winners at AUC 0.80
(winner median 0.276 vs loser 0.005; median split 27% vs 67% win). Inverted
from supply-trap intuition = absorption/fuel mechanism, invisible in OHLCV.
n=23 single-day sniff -> GO to Phase 1 recorder, not to trading. Cost: pennies.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The gate is damage-control, not an edge source: it can only allocate the
strategy's existing (negative) edge onto least-bad days — best case flat,
never gain. Its value is CONDITIONAL: a holding pattern until an entry edge
(plan 017 L2 depth) is found, at which point it flips to force-multiplier
(good names x good days). Owner's 'it just turns the bot off' critique is
correct at the current no-edge base.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Cost-checked (no data spent): DATABENTO_API_KEY reaches XNAS.ITCH TotalView
10-level depth; every real gapper (EHGO/ATPC/MUD/LASE/AEON) covered at
$0.0001-0.0066/fire, full 20-fire sniff <$1 inside free credits. Catalogs
all 13 provisioned-but-unwired keys with a wire-on-demand triage (Databento
= game-changer; 4 news APIs = skip; rest redundant).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The one untested selection axis (order-book depth = the owner's 'size stacked
above the break' instinct). Staged so $0 is spent until a free sniff test +
recorder prove the signal exists and covers the universe; paid feed is an
explicit owner gate; the signal faces the same pre-registered bar + random-gate
null as every refuted lever before it touches the live entry. Honest ceiling
stated: a PASS nudges odds, is not an autonomous oracle.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Owner saw a blank /backtest — it was CORRECT (the regime gate sat out every
recent session) but unexplained. Now: ?regime=0|1 override on /api/backtest,
an 'apply regime gate (as deployed)' checkbox, and an empty-state message
saying WHY there are no trades and what to click. Gated view = what the bot
would do; ungated = the raw simple criteria.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
admin_api.py called load_dotenv() unconditionally at module level, so
merely importing it (e.g. pytest collecting
tests/unit/test_admin_effective_config.py) ran python-dotenv's upward
find_dotenv() search, loaded the real repo .env (secrets included),
and polluted os.environ for the rest of the test session.
Gate the load_dotenv() call behind `if __name__ == "__main__":` at the
same source location, so it still runs before the module-level
os.environ.get(...) constants when run as the entrypoint (`python
admin_api.py` is unchanged) but never fires on a plain import.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
research/attribution.py's _connect() read os.environ["DATABASE_URL"]
directly, bypassing the storage facade fallback chain (DATABASE_URL,
PG* vars, LAN host retries) every other module goes through — and
KeyError-ing (uncaught by name, though still 500'd by the endpoint's
broad except) whenever DATABASE_URL was unset.
Delegate to storage.db.get_connection().raw instead, so DB resolution
matches EventStore/research.multi_schema, and a genuinely unreachable
DB fails with a catchable psycopg2.OperationalError rather than a
KeyError.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
OrderFilledEvent had no side field, forcing api/main.py's chart() to
backfill it from the sibling order_submitted row. Add an additive-only
`side: str | None = None` field (existing payloads without it still
parse) and populate it at the sole production emit site
(alpaca_paper/execution.py _finalize) from the ExecutionRequest already
in scope. The api/main.py backfill workaround is left in place as a
harmless fallback for pre-existing events.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Null (the H2 lesson): zeroing share f of a negative baseline mechanically
yields expR_delta = f*|mean baseline|; a PASS on expR_delta_min now also
requires expR_delta_vs_null >= the same margin whenever zeroed_share >= 5%.
Exposure-shrinking can no longer masquerade as selection; non-gate
experiments unaffected. Honest disclosure: regime_meta_v1's FIRE-weighted
pass would not clear this bar either — its standing rests on the session-
level walk-forward OOS evidence, which no random gate reproduces.
H6: CKS order-flow-imbalance from quote_snapshots bid/ask SIZES —
size-weighted book imbalance over the 10-min pre-trigger window, zero
ask-tilted books; skip-on-no-coverage; judged under the null from day one.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
H2 exposed an adjudicator gap: on a -0.21R/fire baseline any keep-or-zero
gate zeroing ~44% of fires clears +0.05 mechanically; its own pre-registered
strata check inverts (choppy beats persistent within both gap buckets =
Simpson's). Queued: random-gate null for keep-or-zero bars. H3 accruing as
designed (n=29, skip-on-no-coverage working).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
H4 (3-book): keep-or-zero on trigger-bar volume concentration vs prior 5-bar
sum (>1.0 = completed one-shot burst). H3 (5-book, direction disputed):
fragility-camp primary — zero fires whose fill-time relative spread exceeds
2x the name's own 10-min median from quote_snapshots (which carry bid/ask
SIZES since 07-02 — the note's data-upgrade #1 was already satisfied).
Coverage design learns the dilution_veto_v1 lesson: no-quote fires are
SKIPPED (variant None, excluded from n), so thin coverage cannot false-FAIL.
H5 deliberately deferred until H1's verdict (its incremental value is defined
net of H1's catches). 6 tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
From docs/research/orderflow_notes.md — the order-flow shelf distillation.
H1 (5-book convergence): exit on first post-fill 1-min close back below the
trigger within 5 bars — the transitory-impact/recross signature — else the
standing exit stack. H2 (4-book): keep-or-zero on pre-trigger volume-weighted
bar-sign persistence (<0.30 over 15 bars = choppy one-shot burst, zeroed);
strictly-before-entry window, fail-open on thin history. Both pre-registered
at the house bar (+0.05R, n>=300, standard fidelity); nightly adjudication
decides. 7 tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
9 sources (Hasbrouck, Bouchaud x2, Cartea, Dalton, Bellafiore, Neill, CKS,
Cetin-Danilova) read by parallel agents + adversarial synthesis, constrained
to computable data (1-min SIP bars, 15s quotes) and fenced from the 8 refuted
levers. Companion to regime_notes.md. H1 (trigger-erasure fast invalidation,
5-book convergence) is the lead candidate.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
replay_window reconstructed fires from ALL setup versions, so liq1/vr1/run1/pm1
(refuted shadow tracks, never traded live, 8:1 volume vs v1, tight-stop
artifacts) dominated /backtest: 17k trades / 45 sessions at -1.8R/trade —
garbage scale vs the harness-validated -0.08R/fire. reconstruct_fires_from_setups
gains an optional setup_version filter (default None: experiments unchanged);
replay_window pins it to v1.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
premarket_eval's shadow enrichment call site queried session_date = current_date
(wrong on UTC-server Postgres at night, so it never matched any symbols) and
shared one try-block for float_cache + dilution filings (a float_cache failure
silently skipped the dilution refresh too). Fix: compute the ET session date
in Python, split into two independent try-blocks, lift into a testable
_refresh_shadow_enrichment(con, session_date). float_cache now applies its own
DDL unconditionally on use instead of only when an API key is present.
Adds iv_dilution_gate (additive, mirrors iv_regime_gate's keep-or-zero shape
keyed on symbol instead of session_date, fail-open on unfetched symbols) plus
its once-per-run preload, and registers the pre-registered dilution_veto_v1
experiment via the existing experiments/*.json self-registering cohort. Live
veto stays OFF; this only wires the shadow measurement.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
One playlist per add-playlist CLI call made adding a second playlist a chore.
Now YT_PLAYLIST_URLS takes a comma-separated list of playlist URLs (list=
param extracted) or bare ids; the transcripts service seeds the playlists
table from it at boot (insert-if-missing — hand-set labels are never
clobbered), and `python -m youtube_claims.run sync-playlists` does the same
on demand. Documented in both compose files; 3 tests.
Also: queue the two executor-surfaced defects (OrderFilledEvent side,
/api/attribution env hardcode) + the flaky nightly regime build-features
step in the plans index; regime_features 07-16 backfilled supervised.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds a read-only /backtest page showing how the bot WOULD have traded past
sessions under the CURRENTLY configured live settings: theoretical ORB fires
from setup_events, replayed through the shared exit engine (simulate_exit +
ExitConfig.from_env(), same adverse-fill accounting the exit-experiment
harness uses), screened through the shared population gate and (optionally)
the live regime gate. No new simulation math — replay.replay_window() is pure
assembly of existing pieces (reconstruct_fires_from_setups, _fire_in_population,
_load_regime_favorable, simulate_exit); api/main.py adds the /backtest and
/api/backtest routes plus a dedicated-lock 10-min cache so a slow replay never
blocks the live dashboard's SSE stream. Also extends /api/chart with an
optional date= anchor so the trade drill-in chart modal can reach sessions
older than the existing 72h-from-now window, and adds a one-line BACKTEST nav
link to the live dashboard header.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Surfaces the entry_gate_evaluated decision receipts (plan 005) that were
write-only until now: a Decision Trace panel (per-gate value->threshold
chips, symbol filter, lazy-load + SSE refresh) and collapsed reject markers
merged into the plan-013 symbol chart's existing markers primitive.
- storage/projections.py: query_gate_receipts (case-insensitive symbol
filter, newest-first, same _hist_window idiom as query_fills_feed)
- api/main.py: GET /api/decisions (clamped limit<=500, date-scoped);
DashboardState.decisions() adds a server-computed epoch per row; factored
chart()'s inline to_epoch into a shared _to_epoch_et helper
- api/static/dashboard.html: Decision Trace panel + refreshChart now also
fetches /api/decisions and merges collapsed reject markers (same-reason
bursts within 5min -> one marker) into the single setMarkers call
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
createSeriesMarkers(series, markers) attaches a NEW primitive to the series
on every call — refreshChart() called it on every 5s SSE-driven refresh, so
a modal left open accumulated one primitive per refresh (identical markers
overdraw so it looked fine, but it's an unbounded render leak on long-open
modals). v5 pattern: create the markers primitive once in openChart, mutate
it via markersApi.setMarkers(markers) on refresh. chart.remove() in
closeChart already tears the primitive down with the chart.
Click any symbol cell to open a 24h candlestick chart (vendored TradingView
Lightweight Charts v5.2, Apache-2.0) with buy/sell arrows at the bot's fills,
muted arrows for submitted-unfilled orders, and entry/stop/target price
lines. New read-only /api/chart endpoint (DashboardState.chart) converts
naive-ET minute_bars/event timestamps to UTC epoch seconds; query_fills_feed
gained an optional symbol filter. Chart refreshes off the existing SSE
snapshot signal (5s throttle, full replace) and tears down on close. Also
backfills order_filled markers' missing `side` (OrderFilledEvent's schema
has no side field) via an order_id join to the sibling order_submitted row.
Merged plan 012 (advisor/012-dashboard-order-limits) first — this plan
depends on its stopWorkingBy/targetBy/entryLimitBy maps.
sync.py fetches status="all" so the legacy stopBy map (kept for the
R-bar/spark stop line, unchanged) can pick up a cancelled stop leg.
Add stopWorkingBy, filtered to the same WORKING statuses as
targetBy/entryLimitBy, and point the new Positions Stop column at it
so a cancelled leg never renders as live protection.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Positions table gains Stop/Target columns from resting bracket legs
(targetBy/entryLimitBy maps, filtered to working order status). Order
Lifecycle splits the generic Price column into Type/Limit/Stop/Fill via
a client-side join to the broker order snapshot (client_order_id ==
the internal order_id we send Alpaca) since the lifecycle events
themselves never captured limit/stop/type; the truncated order-id
column is dropped in favor of a title attribute on the Symbol cell.
Approval queue drops the always-"auto" Mode column and shows explicit
entry/stop prices in the risk chip instead.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Owner-requested UI work, advisor-planned for executor handoff:
- 012: show working limit/stop/target prices on positions + order lifecycle,
cut dead columns (order-id, approval Mode)
- 013: symbol drill-in — 24h candlestick + live buy/sell fill markers
(TradingView Lightweight Charts v5.2, vendored, Apache-2.0)
- 014: live decision trace — surface entry_gate_evaluated receipts as a feed
(value→threshold per gate) + chart reject markers
- 015: /backtest replay page — theoretical entries/exits over past sessions
under CURRENT settings, assembled strictly from Experiment-OS primitives
All four are read-path only; zero live-loop surface.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The 07-15 live gate never made a real decision: live_session_favorable read
today's breadth_rvol from today's OWN setup_events rows, which only the
NIGHTLY labeler writes — always None intraday, fail-open every session (the
07-15 "validation block" was computed pre-guard on a zero-filled feature).
Worse, breadth_rvol was unit-corrupted: _session_context pooled all setup
versions and pm1 stores RAW cumulative premarket volume in relative_volume
(median ~261k) vs the ratio tracks (~0.8) — ~240/267 historical sessions'
breadth was actually median premarket volume, and plan 009's v1 RVOL fallback
was about to flip forward sessions to ratio units.
Fixes (TDD, 21 regime tests):
- breadth_rvol is now pm1-ONLY: median premarket volume — the dominant
historical source, unit-consistent, and strictly pre-open knowable.
- _live_breadth_rvol recomputes today's pm1 fires IN MEMORY from premarket
minute_bars (pure compute_premarket_setups, no insert); previous_close
falls back to the latest prior daily close when today's daily_bars row is
absent intraday (else the gap basis silently flips to the premarket open).
- _regime_reject: verdicts before 9:50 ET are PROVISIONAL (re-scored ~5 min;
ingest lag can truncate the premarket frame) and the first verdict at/after
9:50 is FINAL even when None — caps the scorer at a handful of runs/day
instead of ~78 on quiet days (hot-path stall guard). Throttle arithmetic
moved inside the fail-open try (naive/aware mixing must not escape).
- Live fit excludes breadth_rvol-NULL training rows (zero-fill taught a
"zero premarket volume" class the serve path can never produce).
Prod re-validation on the cleaned feature (regime_features rebuilt, scores
rewritten): regime_meta_v1 fire-weighted expR_delta +0.0570 (pre-registered
bar +0.05, n=2058, PASS — the contaminated +0.0965 was inflated), keeps 37%;
recent-80-session delta +0.149R/session (always −0.141 → gated +0.008).
First genuine live verdict 2026-07-16: UNFAVORABLE (pm breadth 612,819).
Known accepted skew: live scaled-IEX volumes vs nightly SIP tape — the
meta_live-vs-EOD comparison is the standing forward measure.
Also backfilled the missed 07-15 nightly job supervised (SIP refetch, 50 v1
+ 6 pm1 setups, SPY/QQQ ingest, regime row) — plan 009's accrual clock is
unbroken.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The docker-image workflow's test gate has aborted at collection since sklearn
was introduced (test_conversion_probe.py, plan 008), so no image published.
Regime gate (plan 011) also needs sklearn at runtime in the image, else the
live gate silently fail-opens forever. Adds scikit-learn to requirements.txt
(installed by both the CI test job and the Dockerfile).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Wires the regime signal into the money path — fresh entries are BLOCKED on
sessions the walk-forward model scores unfavorable. Session-level: evaluated
once per day, cached. Enabled in both compose templates (paper).
Correctness (blocker 1 resolved): the LIVE feature set drops tape_regime (the
only non-strictly-open feature — AND measured 0.000 importance) + breadth_gap
(also 0.000). The 5 remaining features are all non-look-ahead at entry time:
breadth_rvol = median(relative_volume=vol_upto/avg_vol_20d) over the session's
setups so far (the SAME up-to-trigger calc the live bot runs, NOT day-end),
spy_* lagged to prior close, trailing from PRIOR sessions. The live single-
session fit reproduces the batch meta_v1 verdict exactly on the last 6 sessions.
Safety: fail-open everywhere — disabled, unfit model (<60 prior sessions),
missing breadth_rvol (pre-setup), or ANY exception returns None and never
blocks. The gate can only SUBTRACT trades on bad-regime days.
- research/regime.py: live_session_favorable() (refit on priors <today, predict
today; writes regime_scores['meta_live'] for forward validation) + LIVE_FEATURE_COLS
+ the breadth_rvol-missing fail-open guard.
- trading_execution.py: TRADING_REGIME_GATE_ENABLED / _P_MIN settings + _regime_reject()
(cached per session) + a REGIME gate after SESSION in _evaluate_entry_gates.
- docker-compose.yml + truenas-compose.yml: TRADING_REGIME_GATE_ENABLED=1.
- tests/unit/test_regime_gate_live.py: block / pass / fail-open(x3) / scored-once-per-day.
KNOWN train/serve skew (documented, validate forward via meta_live vs EOD label):
model trains on full-day breadth_rvol, live uses setups-so-far. Small edge on a
~266-session sample; paper only. Suite: 504 passed, 15 pre-existing loopback fails.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Both regime experiments PASS pre-registered bars over the 2041 v1-population
fires — first PASS of the milestone. meta_v1 gated expR +0.0117R/fire (from
-0.085R baseline). Adjudicated live via the admin /api/regime/run pipeline.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The session-level metalabel P(our ORB profits today) — the one lever that
survives out-of-sample. Backtested on 266 prod sessions (walk-forward):
trade-always (trained subset): -0.111R/session, 36% hit
meta_v1 GATED (trades best ~32% of days): -0.004R/session, 48% hit
delta +0.107R in-sample, +0.128R OOS (recent 80 sessions), improves across folds
It does NOT make the bot profitable — it takes a bleeding strategy to roughly
flat by sitting out the worst ~2/3 of sessions. threshold_v1 rule gates too
little to matter; meta_v1 (walk-forward GBM) is the signal. Leaky full-fit
(+0.61R) >> walk-forward confirms the discipline was load-bearing; the OOS
holding is the evidence the +0.078R is real, not look-ahead.
Changes:
- research/regime.py: batch minute-bar load per session (64k per-fire round-trips
-> ~266 queries; fixed a >5min build to ~3min) + incremental label pass (skip
already-labeled sessions; --rebuild forces full). Nightly cost now ~one new day.
- docker-compose.yml + truenas-compose.yml: scheduler runs build-features + score
(both models) before `experiments run`, nightly. ALL-SHADOW — does NOT gate live
trading (that stays owner-gated Step 5).
- admin_api.py + admin.html: POST /api/regime/run (detached) + GET /api/regime/status
+ a Run/Status button — the panel had no way to trigger this.
- plans/011 + docs/research/regime_notes.md (6-book distillation) + plans 008/009 docs.
Prod state (shadow, rollback = DROP regime_features/regime_scores + DELETE
experiments WHERE id LIKE 'regime%'): 266 feature rows, 532 scores, 2 experiments
registered.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
research/regime.py: Step 0 probe (labelable-session/SPY-coverage STOP gate),
Step 1 build-features (regime_features: pre-open breadth/tape + prior-close
SPY momentum/vol + trailing bot expR, label = mean _sim_baseline_r over the
session's setup fires) + walk-forward threshold_v1 rule, Step 2 walk-forward
meta_v1 (shallow sklearn GBM), Step 4 OOS/leakage-audit helpers
(oos_split/sequential_folds/session_expr_delta/full_sample_fit_threshold).
Step 3 (HMM) deferred per scope — statsmodels stays absent.
replay.py: new iv_regime_gate intervention (keep-or-zero a fire on a
pre-computed session regime score, fail-open on no score row) wired into
_compute_fire + run(), mirroring rank_one_bullet's shape.
experiments/regime_threshold_v1.json + regime_meta_v1.json register the two
model variants against the shared population/bars/fidelity contract.
All new code is unit-tested against isolated EventStore(":memory:") schemas
only — no production DB read or written. Live-trading paths untouched.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Step 1 measured: pm1+fat-tail expR -0.50R@0.5% fill -> -1.27R@1% (realistic) ->
-2.74R@2%, vs ORB -0.11R. The 22-min fat tail can't outrun the 1-2% premarket
spread. Every entry lever this session (gap bands + premarket) is negative after
honest fills; the gated ORB is least-bad. Keep TRIGGER_GAP_MAX=35.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The one unexplored setup (premarket/pm1). Step-1 go/no-go: does pm1 + the
validated fat-tail exit beat the ORB AFTER honest thin-book fills (real
spread_pct, fat premarket adverse)? If not, STOP — premarket is a paper-fill
mirage. Shadow-first; TRADING_SESSION_ONLY stays ON; no live order until a
forward shadow edge survives realistic slippage (owner-gated).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Root anchor lacked the TRADING_EXIT_* block the truenas template already has, so
a fresh root deploy ran no-trail defaults (scratches v1 winners at breakeven).
prior_low trail after +1R (counterfactual: v1 win 17%->24%) + step0.25 + profit
tiers + BE@1R. Only the loop acts on these.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
iv_exit_config replays ANY target/breakeven/trail combo through the shared exit
engine vs the live-exit baseline — reusable way to adjudicate exit numbers.
Counterfactual over setup fires: v1 (slow, 1.37R median winners) gains most from
prior_low trailing (win 17->24%, +0.036R); pm1 (fast, 2.64R median, 10R tail)
gains most from NO target + step0.5 (+0.273R, win 24->34%) — the fixed target
throws away pm1's fat tail. Registered both as tracked experiments. +test.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The 0.514 was the liq1-dominated mix. Scoped to live v1: CV AUC 0.641, driven
~entirely by above_vwap_flag (above-VWAP +1R 35.4% vs below 12.7%, 2.8x). The
bot already enforces above-VWAP; residual within that population = 0.526 ≈ noise.
No new alpha in current features; revisit after 009 fills rvol/context/catalyst.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
research/conversion/{__init__,dataset,signal_probe}.py — the leakage-safe probe
plan 009 Step 5 re-runs scoped to a live setup_version:
- dataset.load_dataset: setup_events JOIN outcome_labels; label =
reached_1r_before_minus_1r; features = trigger-time-knowable setup_events
columns ONLY, dropping >70%-null and <2-distinct columns. HARD leakage rule —
every outcome_labels column is blocklisted AND asserted absent from X, so a
schema change can't leak the answer in.
- signal_probe.run_probe: expanding-window CV over ~5 sequential session-date
blocks (whole days, train strictly precedes test), HistGradientBoostingClassifier
(max_iter=250, lr=0.05, max_depth=4, l2=1.0, categorical_features=indices),
per-fold + mean ROC-AUC, top-decile +1R lift vs base, permutation importance on
the last fold. Drops columns constant within a train slice (HistGBM binning).
CLI: python3 -m research.conversion.signal_probe [--setup-version v1].
Tests: the critical leakage guard (zero outcome columns in features), AUC>0.5 on a
synthetic signal + ~0.5 on pure noise, and a static live-path guard. All :memory:.
No live-path change.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Step 3 of plan 009 (config + code; no live-path change):
- research/ingestion/market_context.py: pulls SPY/QQQ full-session minute bars
(SIP, post-close embargo-safe) + daily bars via the existing upsert helpers, so
the labeler's market-context features have a FORWARD source (index ETFs never
enter the gapper universe the live loop ingests). Mirrors sip_backfill's shape.
- both compose scheduler blocks: run it after premarket_eval, guarded || echo.
- NEWS_ENRICH_ENABLED on: added to docker-compose x-app-env anchor (loop +
scheduler inherit it); truenas-compose + .env.example already had it. Catalyst
is a FEATURE source only, never a live gate; enabled-but-down never blocks
trading. Enrichment OUTPUT is verified by the orchestrator (needs live Ollama).
Compose statically validated (yaml.safe_load). Ingest self-checked via a
fake-client :memory: round-trip through _preload_spy.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Steps 1-2 of plan 009 — fill two fillable feature gaps so a conversion model
gets a fair, instrumented test:
- market context: _spy_context computes spy_intraday_return (SPY RTH open ->
trigger-minute return) + a coarse tape_regime {trend_up,trend_down,chop} from
SPY's sign + higher-highs structure. Replaces the hardcoded
tape_regime=None/spy_intraday_return=None; wired into ALL five build paths
(v1/vr1/run1/liq1/pm1) via _preload_spy. Null when SPY has no RTH bars at/before
the trigger (pre-30d history, or a premarket fire before SPY's RTH open).
- thin-name RVOL fallback (v1): when avg_vol is 0/null (thin IPO-class, no 20d
baseline) fall back to the median of prior sessions' total volume (real
daily_bars.volume, knowable pre-open); flagged in new setup_events column
relative_volume_source ('avg_vol_20d' | 'prior_days_median_volume' | NULL).
Never fabricates a baseline.
A feature to TEST, not a signal to trust. No live gate/sizing/exit change.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
eod_replay was the wrong tool to grow the harness sample — it only PRINTS a
backtest report (persists no fires) and get_most_actives is today-biased for
historical dates. The right source was already in the DB: 18,164 setup_events,
each carrying the ORB entry (entry_reference_price) + stop (invalidation_price).
reconstruct_fires_from_setups turns every setup into a simulated fire; the exit
experiments (volume_fade, time_stop, asym_exit) replay over its minute-bar path
with baseline = the LIVE exit config simulated on the same path (_sim_baseline_r,
same adverse-fill accounting) — a clean same-path delta needing no live P&L.
Gated by EXPERIMENTS_INCLUDE_SETUP_SIM (default on), capped by
EXPERIMENTS_SETUP_SIM_LIMIT for chunked backfill; rank_one_bullet excluded (it
needs live receipts). Grows every exit experiment's n from ~15 to thousands.
+test.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The live loop fills resting-limit brackets via position-snapshot diffing and
emits NO order_filled (verified: 0 all-time in prod), so reconstruct_fires
skipped every fire and the ENTIRE harness replayed nothing (0 run rows).
Reconstruction now falls back to the order_submitted timestamp for entry timing
and reads entry/stop/exit from position_closed — restores the whole flagship
(0 -> 324 run rows on prod; pm1's overdue FAIL verdict now posts).
Adds the volume_fade intervention (exit to lock the gain when a bar's volume
drops below fade_frac x the entry-burst average, once in profit; original stop
still caps downside; adverse-tick fills on entry AND exit) + volume_fade_exit_v1
(pre-registered bars/futility/fidelity). Interim n=15 read: -0.68R vs baseline
(INCONCLUSIVE, min_n=100) — the harness answers 'does volume-fade help?' from
data before any live change. +6 tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The scheduler ran 'experiments run/adjudicate' but nothing registered the five
experiments/*.json into the experiments table, so on a fresh DB the harness
replayed over an empty set and the ledger stayed empty forever (owner hit this
on first deploy). registry.register_cohort() registers every cohort JSON
idempotently (skips already-registered, tolerates a malformed file); 'run'
calls it at startup so a fresh deployment self-bootstraps with no manual
register step. +test.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
compute_ratios bails when UTC now-16min precedes today's 08:00Z, i.e. nightly
~00:00-04:16 ET; the test's hardcoded 2026-07-06 bars made it KeyError at the
date rollover independent of any code change. Freeze the module clock to a
mid-session UTC time so the window is valid deterministically.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Step 6: tests/smoke_dashboard_headless.py renders the real page in a headless
browser and fails on any JS console/pageerror (standalone script, not
pytest-collected; CI non-blocking first). admin_api gains GET /api/config/effective
(per-key masked value + source: overrides.env vs compose, missing .env.example
keys flagged as drift) + an admin.html table (values XSS-escaped). dashboard.html
gains the Experiment Ledger panel reading the plan-006 verdicts from the snapshot
(empty-state until the tables exist). Removes the last CI ruff per-file pin
(api/main.py:F821) now that os/hmac are imported; scoped ruff is green unpinned.
Marks plan 007 DONE in plans/README.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
DashboardState.bars() short-circuits to [] unless a research_con is attached; the
embedded orchestrator passed one but the standalone dashboard_api.py never built
it, so production sparklines were permanently dead. The missing wire was purely
the entrypoint (DATABASE_URL already reaches the market schema) — now it opens
open_research_db("market") read-only and passes it, degrading to None (charts
show "no research db") if it can't connect. Verified live: bars() returns real
minute rows through the standalone reader.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Badge now derives PAPER/LIVE from broker truth (base-URL host) instead of the
schwab_live literal that read PAPER while go-live swapped only ALPACA_BASE_URL;
live flips body.live-mode (top shimmer -> solid red) with a tooltip of host +
account. Adds the header FLATTEN ALL (two-step arm/confirm, 5s auto-disarm) and
PAUSE ENTRIES toggle (+ amber banner), per-position EXIT buttons (two-step), and
an operator-token field sent as X-Dashboard-Token on every POST (also fixes
approve/reject when a token is set). Action controls gate on
has_execution || command_bus; kill-switch controls on command_bus. No keyboard
shortcut for any destructive action. New styling is :root-token-only.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds OperatorCommandEvent + a closed command vocabulary (flatten_all,
flatten_symbol, pause_entries, resume_entries). POST /api/command emits it
(token-required: 503 when DASHBOARD_TOKEN unset — commands never run
unauthenticated); the live loop's new 5s "command" step consumes them via
OperatorCommandConsumer with three replay-safety layers (in-session nonce set,
persisted operator_command_done reload, 10-min staleness window).
flatten_all routes through close_session DIRECTLY (never step_eod_flatten) so it
cannot trip plan 003's daily-restart exit. Pause is checked at the top of BOTH
request_approvals_for_ready_signals AND submit_breakout_now (missing either
leaks pause on that path). qabot gains an authorized-chat-only /flatten slash
command (prose "flatten" still refuses via _ACTION_WORDS). Snapshot exposes
broker truth + command_bus + a defensive experiment ledger.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every SSE client polled COUNT(*) on a fresh EventStore connection each ~1s
(re-applying the 473-line schema unless PG_SKIP_SCHEMA=1). Replaced with one
process-wide _ChangeSignal: a single long-lived connection reads max(timestamp)
(idx_events_timestamp tail seek) at most once per second GLOBALLY behind a lock,
so N clients cost one query/sec. EventStore.max_timestamp() backs it; reader is
injectable for the query-budget test.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every POST endpoint (approve/reject/exit/watch) crashed with NameError:
do_POST references os.environ + hmac.compare_digest but neither was imported.
Adds the imports, a non-loopback regression test for the token gate (no token
-> routes; token set -> 401 without/with-wrong header, 200 with correct), and
plumbs DASHBOARD_TOKEN + PG_SKIP_SCHEMA=1 into both compose dashboard services
(+ .env.example docs for DASHBOARD_TOKEN / COMMAND_BUS_ENABLED).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Plan 006 step 5. python -m research.experiments run && adjudicate after
nightly_tune in both scheduler blocks (guarded with FAILED echoes). ALL-SHADOW:
emits verdicts, never touches live gates/sizing/exits.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Plan 006 step 4. experiments/*.json for time_stop_10m_v1, probe_add_v1,
asym_exit_v1, rank_one_bullet_v1, pm1_v1 — bars verbatim from the plan table,
production population, fidelity floors {0.25, 3, 1} from the research doc. Also
hardened evaluate(): n==0 and any uncomputable bar metric -> INCONCLUSIVE (not a
false FAIL). Cohort registers + survives run->adjudicate end-to-end. 25 tests green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Plan 006 step 3. evaluate() scores runs against pre-registered bars
(_min/_max/min_n vocabulary) -> PASS/FAIL/KILLED/INCONCLUSIVE; terminal
verdicts flip status, emit an experiment_verdict event, and Telegram-ping ONCE
(terminal experiments are never re-adjudicated). Metric vocabulary covers
expR_delta/rate_1r plus pm1's median_up_60m/median_dd_60m/big_gap_rate_1r.
backfill_pm1 bridges pm1 from outcome_labels JOIN setup_events (variant +/-1,
baseline NULL). premarket_eval dual-writes the verdict through the harness while
keeping the legacy TSV. 8 tests green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Plan 006 step 2. Reconstructs fires from the event stream (submit[buy] -> fill
-> close), population-filters via strategy/population.py, and applies three pure
interventions counterfactually on the bar path: time_stop, probe_add, asym_exit
(the last through strategy.exits.simulate_exit ONLY). Hard fidelity: +1-tick
adverse fills on entry AND stop, spread_proxy = median quiet-bar high-low, stop
floor = max(atr_frac*ATR14, spread_mult*spread_proxy). rank_one_bullet is
receipts-only and starves (documented) without plan-005 rank. Idempotent via PK
+ ON CONFLICT. Anti-slippage-artifact regression (test #2) green: a
zero-slippage-profitable path LOSES under adverse fills.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rank concentration now runs on BOTH paths via the shared gate, so the warning
that it was 'enforced ONLY on the dead fast-trigger path' is false. Removed it
and folded rank top-N state into the [boot] entry-gates summary line.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New EntryGateEvaluatedEvent (symbol, source, passed, reject_reason, gates map)
emitted once per _evaluate_entry_gates decision on both paths, behind
TRADING_GATE_RECEIPTS (default on; dataclass default off so unit tests opt in).
Added to the 14-day nightly purge tuple (high-volume diagnostic) and .env.example.
Turns 'why didn't it trade X?' into a query and is the substrate plan 006 replays.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
One ordered gate sequence (session, confirmation-speed, population, VWAP,
quality, concentrate-by-rank, daily cap [fast-only], dedup/cooldown/reentry,
dilution veto, anti-chase) that BOTH request_approvals_for_ready_signals and
submit_breakout_now call. Gate bodies lifted verbatim from whichever path
owned them; slow-path risk events preserved source-guarded so the fast path
gains no new events. The two documented asymmetries now close:
- fast path routes through the VWAP gate (was ungated),
- slow path honours concentrate-by-rank via a rank_map threaded from the
trigger book into tick()/request_approvals (no-op at concentrate_top_n=0,
which is the production config — STOP #3 does not apply).
Price-sanity/entry_ref coercion stays path-native (different data sources);
slow-path dedup pre-filter kept first so per-tick re-scans don't re-emit.
16 characterization tests pin both paths; the only decision diffs vs
pre-extraction are the two closures above.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Pins today's slow-path (approval/event) and fast-path (skipped string)
decisions across single-condition signals, incl. the two asymmetries that
step 2 will close (fast has no VWAP gate; slow has no rank gate). Green
against unmodified trading_execution.py.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
from_env merges os.environ, and importing the .env-loading consumers populates
TRADING_MIN_ENTRY_GAP_PCT — so the "bare shell disables the gap gate" assertion
must delenv the key first. Test-only fix; the module behaviour is unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The tuner was fiction in production: the 20:00 ET scheduler ran only
premarket_eval (+ eod_reconcile), so learned_params.json sat 26 days stale and
the whole-market dataset never grew. Both scheduler loops now run, in order,
after the eval/reconcile pass:
python eod_replay.py || echo "[scheduler] eod_replay FAILED"
python nightly_tune.py || echo "[scheduler] nightly_tune FAILED"
Order matters: replay grows the dataset first, the tuner fits after. 20:00 ET is
after close, so replaying "today" is complete data — NOT look-ahead (replay is a
labeling job, not a live gate). Deletes eod_replay.sh (dead cron-era wrapper
pointing at a nonexistent venv; compose is the runner).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- nightly_tune: write learned_params.json via tmp -> flush -> fsync ->
os.replace so a crash mid-dump can't leave the live loop parsing half a JSON
on its next boot (it reads this file on every restart).
- run_live_paper._load_learned: warns (boot log + one gated Telegram note) when
the file is older than LEARNED_PARAMS_MAX_AGE_DAYS (default 3) — the missing
signal that let the params sit 26 days stale. Loads anyway; never blocks boot.
Staleness math factored into _learned_params_age_days (naive ts = UTC).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Both offline consumers now screen through strategy.population.population_reject
instead of divergent, look-ahead admits:
- learn_params.load_sessions: replaces the `gap>=0.03 OR day_rvol>=2.0` admit,
whose day_rvol was a FULL-DAY volume sum (outcome-time look-ahead, the
2026-07-08 class). gap is already a FRACTION (open vs prev close) — passed
as-is; the OPEN price is passed for the price band. Stored daily bars expose
no open-knowable volume, so volume=None (fail-open on the day-volume floor) —
no day-end volume is smuggled into admission.
- eod_replay._daily_gappers: screens on the OPEN price + STATIC open gap (was
t_close look-ahead) via the shared gate; the local gap is converted to a
FRACTION before the call, so it inherits the >=150% squeeze exception. Deletes
the stale $1-5 / 3-60% WATCHER_/TRIGGER_ constants (KEEPS RISK_PCT). The
second-stage scan_gappers is made permissive (membership already decided by
the shared gate), removing the `gap_pct <= 60%` cap that wrongly dropped
squeezes. Prod thresholds passed explicitly (bare from_env would disable gap).
Unit test replaces the live-DB smoke run: $15/+12% qualifies, $4/+45% rejected
unless >=150%+rvol5. research/labeler.py deferred to plan 006 (no locatable
price/gap admission screen); sweep_exits/sweep_filters likewise deferred.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The tradeable population is now defined ONCE. trading_execution._population_reject
becomes a thin wrapper building a PopulationConfig from its settings and calling
strategy.population.population_reject — logic moved verbatim (dead-band 0.35/1.50
and squeeze rvol 5.0 are module CONSTANTS, gap floor / price band / volume floor
from config). Behaviour byte-identical: the pre-extraction characterization suite
stays green through the new public seam.
from_env mirrors ExecutionSettings.from_env (same 4 keys + the [0,1] _qfloat clamp)
so a live-shaped shell reconstructs the identical gate. gap_pct FRACTION contract
and the as_of open-knowable contract are documented for offline consumers.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
11 table-driven cases pin the CURRENT behaviour of the live population gate
before plan 004 extracts it to strategy/population.py: in-band pass, below-gap,
dead band, squeeze pass/fail, both price-band sides, volume floor, missing-data
fail-open, the gap-is-a-FRACTION unit pin, and the gap-gate-disabled nesting.
These must stay green byte-for-byte across the extraction.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Plan 003 step 5 (completes plan 003). research/eod_reconcile.py diffs broker
truth (open positions, today's live orders) against event-store truth (today's
submits/fills/closes) and emits a loud eod_reconcile event + Telegram alert on
any drift — the duplicate-resting-order-after-restart class (2026-07-06) caught
automatically instead of by hand. Read-only (reports, never auto-fixes). Wired
into both scheduler blocks after premarket_eval. Pure diff core is unit-tested.
Suite: 373 passed (+22 plan-003 tests), 15 fail = WSL-loopback dashboard only.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Plan 003 step 4. A transport error AFTER the POST landed leaves a live order
resting at the broker; journaling it 'cancelled' breeds the duplicate/naked
order (2026-07-06 incident). Add client.get_order_by_client_order_id + an
after= passthrough on get_orders; on the generic-exception path, look up the
order by the client_order_id we already send and, if live, emit the normal
submitted sequence (ok=True) instead of cancelled. AlpacaApiError (real
rejection) is NOT reconciled. Fills the empty test_alpaca_client.py.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Plan 003 steps 1-3.
- Loop exit(0) after a SUCCESSFUL EOD flatten (TRADING_EXIT_AFTER_EOD_FLATTEN,
default on); compose restarts fresh. Restart-churn guard: _rebuild_daily_state
detects today's eod_flatten event and seeds _session_closed/_eod_done so a
post-flatten restart neither re-flattens nor re-exits. SIGTERM = clean drain.
- Exit manager: _FETCH_FAILED sentinel distinguishes a failed get_orders from a
genuinely-empty snapshot. A failed fetch skips naked-stop enforcement (catastrophe
still exits, re-fetching); an empty-but-successful snapshot needs TWO consecutive
reads before 'naked' is trusted (Alpaca silently returns [] for symbols=). Costs
~one 12s pass on the all-naked case only; isolated naked positions unaffected.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
pg_schema.sql: drop CREATE TABLE for 10 zero-reference tables (operator DROP
list in plans/002-deletions.log — production tables untouched until run).
projections.py: remove 4 zero-consumer query functions.
Clear all 'plan 002' ruff pins from the CI gate (files now F-clean);
api/main.py:F821 pin remains for plan 007. README repo-map/test-count fixed;
pydantic-settings removed (zero imports). Full receipts: plans/002-deletions.log
Suite: 351 passed (16 schwab/dead-module tests removed), 15 failures = the
known WSL-loopback dashboard set only.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
grafana/ pointed at the decommissioned pre-migration DB with committed dev
credentials (rotation note in plans/002-deletions.log); its dashboards
provisioning MOVED to deploy/compose/grafana/dashboards (docker-compose.yml
bind-mount re-pointed — the in-compose grafana service keeps working).
docker-publish.yml fired on a nonexistent branch pushing a wrong image name.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
