focus read everything, brainstorm enhancements, pick the highest-leverage build
48
raw ideas · 5 frames
14
merged candidates
14/14
verified sound (fresh-context spot-checks)
7
survivors · 5/5 axes covered
Grounding Context
Codebase context. Event-sourced small-cap momentum day-trading system (gap-and-go / opening-range breakout). Alpaca paper account over the SIP feed; single PostgreSQL datastore (append-only events table + projections, ~19M minute bars); production is a Docker Compose stack on TrueNAS SCALE (postgres 18, trading loop, dashboard :8765, admin :8767, telegram bot, nightly scheduler); 376 pytest tests against throwaway mem_* schemas. Solo owner-operator.
Grounding came from an 8-agent recon sweep (2026-07-13) that read every subsystem — live loop, strategy, storage, research/data, dashboard, ops/deploy, brokers, tests — and produced ~95 evidence-cited findings, plus the in-repo external research pass docs/research/entry_conversion_research.md (academic + backtest literature on breakout entry conversion, with a ranked experiment shortlist). A fresh-context verifier then spot-checked every load-bearing claim against HEAD with ~25 targeted reads; every checked citation matched.
Settled constraints honored throughout: the validated entry population stays ($2–20, static open gap 10–35%, real volume, ≥150%-gap & rvol≥5 squeeze exception); seven selection levers are refuted at year scale and are not re-proposed; scale-out partials and retest/pullback entries are refuted; the :8765 dashboard is the only UI to extend; the system's proven bottleneck is entry-conversion, not selection.
1. Experiment OS — pre-registered, auto-adjudicated counterfactual experiments over logged fires
entry-conversionConfidence 75%Complexity High
Description
One parametric experiment harness replacing the four hand-built shadow-track triples in research/labeler.py (vwap_reclaim / runner / liquid / premarket — ~1,467 lines of structurally near-identical build/report/validate code). An experiment becomes a config row: population, intervention (an entry or exit policy variant), pre-registered pass/fail bars, minimum n, and futility criteria. A replay engine runs every logged live fire counterfactually against its recorded bar path — the Tesla shadow-mode pattern — including a counterfactual exit ledger (every real trade re-run under each exit variant). A verdict engine auto-adjudicates at the pre-registered thresholds (the clinical-trial DSMB pattern, with futility stops) and emits PASS / FAIL / INCONCLUSIVE verdict events that flow to Telegram and a dashboard experiment-ledger panel. First cohort ships with the harness: the three literature-backed entry-conversion experiments (idea card behind them in docs/research/entry_conversion_research.md): time-boxed entries + 10-minute time-stop on unconfirmed positions, probe-then-add 0.5×+0.5× sizing, asymmetric exit (volatility-fractional stop with a spread floor, no target, trail to EOD) — plus the rank-concentration "one-bullet" counterfactual and an immediate adjudication of pm1.
Basis
direct:data/premarket_eval_history.tsv latest row (2026-07-09) shows pm1 at n=2377 with rate_1r=0.443 vs the pre-registered >0.50 bar and median_up_60m=0.0433 vs the ≥0.10 bar — failing 3 of 4 pre-registered criteria while the code's own docstring still frames it as "n=21 is far too small… watch n climb". Seven prior levers each required a human to notice a threshold crossing in a log file. external: the three first-cohort experiments carry academic + replicated-backtest support (Zarattini/Barbon/Aziz SSRN 4729284, Osler NY Fed SR150, Mesfin arXiv 2605.04004) per the in-repo research doc. Verifier confirmed none of the harness/experiment machinery exists yet.
Rationale
The system's bottleneck is proven to be entry-conversion, and the three highest-priority interventions are already researched, pre-registered, and testable for free against fires the system has already logged — zero live risk. The blocker is machinery, not ideas: every new experiment today costs a hand-built ~300-line triple plus a human reading a TSV. Build the machinery once and every future experiment (premarket module included) drops to a config row with an automatic verdict.
Downsides
Biggest build in the set; replay fidelity is a real risk (fills, spreads, halts must be modeled honestly — the tight-stop zero-slippage artifact has already fooled two shadow tracks here, so the harness must enforce adverse-fill floors by construction). Requires the population module from idea 2 to avoid baking look-ahead into every experiment.
Every logged fire becomes reusable evidence; every experiment becomes a config row with an automatic verdict.
2. Learning-flywheel resurrection — one population function + scheduled tune/replay
learning-flywheelConfidence 90%Complexity Medium
Description
Extract the live population gate into one shared pure module with an as-of-time contract (open-knowable inputs only: static open gap, price band, premarket-knowable volume, squeeze exception) and make every offline consumer — learn_params.py, the sweeps, eod_replay.py, the labeler — consume it instead of re-implementing it. Then actually schedule the flywheel in production: eod_replay after each close (unbiased whole-market dataset growth), nightly_tune after it (atomic tmp+rename write, staleness alert if learned_params.json ages past 2 days).
Basis
direct: verifier-confirmed at HEAD: the TrueNAS scheduler runs onlyresearch.premarket_eval; data/learned_params.json is stamped as_of 2026-06-17 — 26 days stale — while the boot log still claims self-tuning; learn_params.py:61,64 admits day-end day_rvol (outcome-time look-ahead); eod_replay.py:44-48 screens $1–5 under a "live verified config" comment while production trades $2–20 (truenas-compose.yml:105-106); eod_replay.sh points at a dead venv path; nightly_tune.py:91-92 writes non-atomically.
Rationale
The project memory says "nightly_tune self-tunes params, applied on boot" — that story is currently fiction in production, and the offline population drift means even a re-enabled tuner would fit a universe the bot doesn't trade. This is the data-integrity substrate under idea 1: experiments adjudicated against a look-ahead population are worthless. The owner has already been burned twice by exactly this class (the 2026-07-08 look-ahead correction).
Downsides
Population-module extraction touches the live gate — needs characterization tests before refactor. A deliberate alternative is to delete the tuner and stop claiming self-tuning; the wrong option is the current silent limbo.
Every backtest, tune, and experiment verdict becomes trustworthy by construction when all consumers share the live gate.
3. Entry-path unification + decision receipts
money-path-correctnessConfidence 85%Complexity Medium
Description
Collapse the two drifted entry paths — slow auto-approval (request_approvals_for_ready_signals) and fast trigger (submit_breakout_now) — onto one shared pure gate function. Every gate evaluation emits a compact decision receipt event (gate name, inputs, pass/fail), so "why did/didn't it trade X?" becomes a query instead of a forensic session.
Basis
direct: verifier-confirmed: _population_reject/_outside_session/_anti_chase_skip are already shared by both paths, but the VWAP check exists only in the slow path (~trading_execution.py:938) and rank_risk_factor only in the fast path (:1228); the boot log itself warns "rank-concentration is enforced ONLY on the dead fast-trigger path, NOT the live auto-approval path" (run_live_paper.py:1026-1028). The daily-cap counted 44-vs-5 incident (fixed ac1ac50) was this same drift class.
Rationale
Half the convergence is already done — this finishes it so every future gate (including any Experiment-OS winner promoted to live) lands on both paths by construction. Receipts double as the raw material Experiment OS replays and the operator reads on the dashboard.
Downsides
Touches the live money path — needs the CI gate (idea 5) and characterization tests first. Receipt events add volume to the events table (bounded; they replace ad-hoc diagnostic events that retention already prunes).
Three small, high-severity fixes sharing one principle — never act on assumed state. (a) Daily self-restart: exit(0) after a successful EOD flatten and let compose's restart: unless-stopped resurrect a fresh process — the process is the trading session. (b) Positive-evidence contract: no destructive market action on an ambiguous broker read — the exit manager must distinguish "orders call failed / silently empty" from "position confirmed naked" before market-flattening, and a lost order-POST must reconcile via client_order_id lookup before journaling "cancelled". (c) EOD cage count: a nightly reconcile that diffs broker positions/orders/fills against event-store truth and alerts on any discrepancy, plus SIGTERM handled as clean shutdown (docker stop currently skips the shutdown path).
Basis
direct: verifier-confirmed: _eod_done/_catchup_done (run_live_paper.py:1064,1113) and _session_closed/_daily_loss_breach (trading_execution.py:398,612) never reset — an uninterrupted day-2 takes zero entries and has no EOD flatten, masked today only by frequent redeploys; exit_manager.py:71-72 returns None on any exception and treats it identically to "naked", flattening after ~24s grace; execution.py:124-129 journals lost POSTs as cancelled with no reconcile (no by_client_order_id method exists in the client); the 2026-07-06 duplicate-resting-order incident is corroborated by d54a2a9's own commit message; recovery-on-boot (_rebuild_daily_state) already exists and is tested.
Rationale
Highest severity-to-effort ratio in the whole set. (a) alone converts an entire recurring bug class (session gate, after-hours submissions, bench state — each patched separately before) into a non-issue by construction, in ~5 lines plus tests.
Downsides
Self-restart must fire only after a successful flatten (exit-on-failure would orphan positions — though broker-side stop legs keep them protected); the positive-evidence branch adds one more state to exit-manager logic that is already subtle.
5. Production safety rails — CI gate, dead-man's switch, automated backup
operability-deliveryConfidence 95%Complexity Low
Description
The "vacation test" bundle: (a) CI runs the 376-test suite (plus ruff check --select F for the untested script layer) before the image workflow pushes :latest to GHCR/Docker Hub; (b) the always-up qabot container gets a dead-man's switch — alert when the event-store heartbeat goes stale during market hours, plus a nightly OK-ping (the loop can't report its own death; the qabot is the independent observer); (c) a scheduled pg_dump (or documented ZFS snapshot task) — production currently has no automated backup.
Basis
direct: verifier-confirmed: .github/workflows/docker.yml has zero test step and ships on every master push; api/main.py at HEAD references os/hmac without importing them (AST-confirmed) — an untested regression that already shipped; loop-down currently means alerts-down (the notifier lives inside the loop process, run_live_paper.py:1034); pg_dump appears only in the one-time migration script.
Rationale
This is the lever that makes every other lever safe to ship. Every subsequent idea in this list lands as an image push; today each push is an untested bet against the production money path.
Downsides
CI needs a Postgres service container for the suite (~3–4 min per push). Dead-man thresholds need market-calendar awareness to avoid weekend false alarms — the holiday-aware calendar already exists in runtime/market_calendar.py.
Make the production :8765 surface tell the truth and accept commands. Repairs: fix the POST NameError (unimported os/hmac); wire the research DB into the standalone dashboard so sparklines work in production; gate action buttons on the has_execution flag the API already ships; derive the LIVE badge from broker truth (it currently keys on a schwab_live literal the admin go-live flow never sets — it would read PAPER while trading real money). Capability: manual flatten (all / per-symbol) and pause-entries as guarded operator-command events through the store — the loop polls its own 5s guard cadence and executes with the existing cancel_protective_and_close helper; dashboard and qabot become thin command emitters. Efficiency: replace the per-client per-second fresh-connection + full-DDL + COUNT(*) SSE change detector with one shared cached signal. Visibility: an effective-config panel showing the merged config per service with per-key provenance (baked env → compose env → overrides.env → learned params) and drift-vs-inventory warnings.
Basis
direct: verifier-confirmed: telegram_bot.py:29 hard-refuses action words; dashboard_api.py:36 builds the production dashboard with execution_service=None — no manual flatten path exists anywhere in the production UI; dashboard.html:1061 keys LIVE on schwab_live while admin_api.py:276-295 only swaps Alpaca URL/keys; SSE poll confirmed at api/main.py:355; concurrency-gauge default drift (3 vs 6) confirmed at trading_execution.py:75 vs projections.py:869.
Rationale
The operator's two moments of need — "something is wrong, get me flat NOW" and "what is this thing actually running?" — both currently require shelling into a container. Command-as-event reuses the store as the bus (no new API surface into the loop) and leaves an audit trail by construction.
Downsides
A kill switch is a live-money control: needs HMAC/token guard, idempotency, and a test that a replayed or stale command cannot re-fire. Command latency is bounded by the 5s guard cadence (acceptable; broker stops remain the fast layer).
7. Dead-weight deletion sweep — every table names its reader or dies
operability-deliveryConfidence 90%Complexity Low
Description
Finish the deletion d54a2a9 claimed: schwab/ is still fully tracked (23 files) despite the commit message. Then the rest of the confirmed-dead surface: comprehensive_backtest.py (crashes on import — research/connection.py doesn't exist), strategy/regime.py, universe.py, backtest/outcomes.py, the ignition detector, the grafana/ stack (points at the decommissioned pre-migration DB with committed plaintext dev credentials), the telemetry/ husk, docker-publish.yml (fires on a nonexistent main branch, pushes a wrong image name), stale uv.lock, the empty test_alpaca_client.py, 12 zero-reference tables and 6 dead projections. Standing rule going forward: every table names its reader or dies — quote_snapshots gets retention or deletion; float_cache/dilution_events get wired into the labeler's features or stop being collected nightly.
Basis
direct: verifier-confirmed each item at HEAD, including git ls-files schwab/ (23 tracked files) vs d54a2a9's message, the import crash, the grafana datasource credentials, and grep-confirmed zero readers for the write-only tables.
Rationale
Every dead artifact taxes every future session ("which one is real?") and the grafana credentials are a small standing security smell. Deletion is the cheapest enhancement in the set and makes every subsequent plan smaller.
Downsides
The schwab/ decision should be explicit (its only non-delete rationale is the dormant human-vs-bot comparison); deleting tables needs a backup first (idea 5c) and a one-session soak.
Rejection Summary
Nothing was refuted: the fresh-context verifier confirmed all 14 merged candidates sound, and all 48 raw ideas were absorbed into the 7 survivors. The table records the merges and the judgment calls.