momentum-system/plans/006-experiment-os.md

Plan 006: Experiment OS — pre-registered, auto-adjudicated counterfactual experiments over logged fires

Executor instructions: Follow this plan step by step. Run every verification command and confirm the expected result before moving to the next step. If anything in the "STOP conditions" section occurs, stop and report — do not improvise. When done, update the status row for this plan in plans/README.md.

Drift check (run first): git diff --stat d54a2a9..HEAD -- research/labeler.py research/premarket_eval.py strategy/ storage/pg_schema.sql data/premarket_eval_history.tsv On any in-scope drift, compare "Current state" excerpts before proceeding; mismatch = STOP. Diffs introduced by plans 001–005 are EXPECTED — see plans/README.md "Expected drift".

Status

• Priority: P1 (flagship — attacks the proven entry-conversion bottleneck)
• Effort: L
• Risk: MED (all-shadow: zero live-trading changes; the risk is silent replay-fidelity bugs, mitigated by pre-registered fidelity checks)
• Depends on: 004 (population module — every experiment population MUST come from it), 005 recommended (receipts enrich replay inputs but bar-path replay works without them)
• Category: direction / research-infrastructure
• Planned at: commit d54a2a9, 2026-07-13

Why this matters

The system's bottleneck is proven to be ENTRY-CONVERSION (7 refuted selection levers; 76–84% of ORB fires poke-then-stop). The three most promising interventions are already researched with external evidence (docs/research/entry_conversion_research.md, ranked shortlist): (1) time-boxed entries + 10-min time-stop on unconfirmed positions, (2) probe-then-add 0.5×+0.5× sizing, (3) asymmetric exit (volatility-fractional stop with a spread floor, NO profit target, trail to EOD). All three are testable for free against fires already logged — zero live risk. What blocks them is machinery, not ideas: today each shadow track is a hand-built ~300-line build/report/validate triple in research/labeler.py (1,467 lines: build_vwap_reclaim:428, build_runner:630, build_liquid:870, build_premarket:1053), and verdicts rot in a TSV nobody reads — data/premarket_eval_history.tsv's latest row (2026-07-09) shows pm1 at n=2377 with rate_1r=0.443 vs the pre-registered >0.50 bar and median_up_60m=0.0433 vs >=0.10, passed=False — failing 3 of 4 pre-registered criteria while the code's docstring still frames it as underpowered. Build the harness once: experiment = config row; replay engine runs interventions counterfactually over logged fires; verdict engine auto-adjudicates at pre-registered thresholds and emits events → Telegram + dashboard. Every future experiment drops to a config row.

The owner provisions the Postgres database/tables once this plan is approved — the DDL below is the deliverable for that.

Current state

• Shadow-track pattern (to be superseded, not deleted yet): research/labeler.py — four structurally near-identical triples; pre-registered bars as module constants, e.g. lines 120–122:

  PREMARKET_PROMO_MEDIAN_FWDMAX = 0.10       # (1) median max-upside (within pm) >= +10%
  PREMARKET_PROMO_MEDIAN_ADVERSE = -0.06     # (2) median max-drawdown > -6%
  PREMARKET_PROMO_MIN_1R = 0.50              # (3) +1R-before-(-1R) > 50%
  

  validate_premarket (1132–1152) checks them; research/premarket_eval.py appends a TSV row nightly.
• Fire history: every live fire and its context is in the event store (order_submitted, order_filled, position_closed, risk_rule_triggered; plan 005 adds entry_gate_evaluated receipts). Minute bars for fired symbols/days are in minute_bars (~19M rows). eod_replay.py (plan 004 fixes its population) demonstrates the replay-from-bars idiom.
• Exit policy source of truth: ExitConfig (strategy/exits.py) — eod_replay.py:79-84 shows _exit_cfg() construction from env. The live exit is runtime/exit_manager.py; strategy/exits.py has the simulation functions (simulate_exit family). KNOWN DIVERGENCE (recon): sweeps/replay exit variants have drifted from live exit behavior — the harness must simulate through ONE exit implementation and declare which (see step 2 fidelity rules).
• Wired-but-unread data feeds usable as experiment features: float_cache, dilution_events (written nightly, zero readers — plan 002 logged them as "disposition here").
• Slippage honesty (non-negotiable, from the research doc): the tightest-stop-wins result is a documented zero-slippage artifact that already fooled two in-house tracks (vr1, liq1). Every experiment must apply stop_width >= max(k_ATR × ATR14, m × spread_proxy) floors and +1-tick adverse fills on entry AND stop, with the floor parameters recorded in the experiment row.
• Telegram: runtime/notifier.py send_telegram (used at run_live_paper.py:1037); the qabot is read-only — verdict pings go through the send helper from the nightly job.
• Population: strategy/population.py (plan 004) with as_of contract.
• Scheduler: both compose scheduler blocks (see plan 004 step 5 for the pattern).

Database DDL (the owner provisions this — also the executor's migration file)

Create storage/experiments_schema.sql (applied by the harness on first run, same idiom as pg_schema.sql; ALSO handed to the owner to run manually if preferred):

-- An experiment is a pre-registered, immutable contract. Rows are never
-- updated after registration except status/verdict fields.
CREATE TABLE IF NOT EXISTS experiments (
    id            TEXT PRIMARY KEY,          -- slug, e.g. 'time_stop_10m_v1'
    registered_at TIMESTAMP NOT NULL,
    hypothesis    TEXT NOT NULL,             -- one sentence, human-readable
    population    JSONB NOT NULL,            -- PopulationConfig dump + extra filters
    intervention  JSONB NOT NULL,            -- typed params, e.g. {"kind":"time_stop","minutes":10,"confirm":"new_high_or_0.5R"}
    bars          JSONB NOT NULL,            -- pre-registered pass/fail criteria, e.g. {"expR_delta_min":0.05,"min_n":300,"p_max":0.05}
    futility      JSONB,                     -- early-kill rule, e.g. {"min_n":500,"expR_delta_max":-0.02}
    fidelity      JSONB NOT NULL,            -- slippage floors: {"atr_frac":0.25,"spread_mult":3,"adverse_ticks":1}
    status        TEXT NOT NULL DEFAULT 'running',  -- running|passed|failed|inconclusive|killed
    verdict_at    TIMESTAMP,
    verdict_note  TEXT
);

-- One row per (experiment, replayed fire): the counterfactual outcome.
CREATE TABLE IF NOT EXISTS experiment_runs (
    experiment_id TEXT NOT NULL REFERENCES experiments(id),
    fire_id       TEXT NOT NULL,             -- order_id / receipt correlation id
    symbol        TEXT NOT NULL,
    session_date  DATE NOT NULL,
    baseline_r    DOUBLE PRECISION,          -- what actually happened (R units)
    variant_r     DOUBLE PRECISION,          -- counterfactual outcome (R units)
    detail        JSONB,                     -- entry/exit stamps, fills, floor hits
    computed_at   TIMESTAMP NOT NULL,
    PRIMARY KEY (experiment_id, fire_id)
);
CREATE INDEX IF NOT EXISTS idx_exp_runs_exp_date ON experiment_runs (experiment_id, session_date);

-- Verdict history (append-only; the dashboard ledger reads this).
CREATE TABLE IF NOT EXISTS experiment_verdicts (
    experiment_id TEXT NOT NULL REFERENCES experiments(id),
    as_of         TIMESTAMP NOT NULL,
    n             INTEGER NOT NULL,
    metrics       JSONB NOT NULL,            -- observed values vs each bar
    verdict       TEXT NOT NULL,             -- PASS|FAIL|INCONCLUSIVE|KILLED
    PRIMARY KEY (experiment_id, as_of)
);

Commands you will need

Purpose	Command	Expected
Suite	DATABASE_URL=<pg url> python -m pytest -q	all pass
Harness run	DATABASE_URL=<pg url> python -m research.experiments run	replays new fires for all running experiments
Adjudicate	DATABASE_URL=<pg url> python -m research.experiments adjudicate	prints/emits verdicts
Register	DATABASE_URL=<pg url> python -m research.experiments register experiments/time_stop_10m_v1.json	row inserted, immutable

Scope

In scope:

• research/experiments/ (new package: __init__.py, registry.py, replay.py, adjudicate.py, cli.py as python -m research.experiments)
• storage/experiments_schema.sql (the DDL above)
• experiments/ (repo-root dir of experiment JSON definitions — the first cohort)
• research/premarket_eval.py (emit pm1's overdue verdict via the new adjudicator; keep the legacy TSV append until the owner retires it)
• Both compose scheduler blocks (add python -m research.experiments run && … adjudicate after eod_replay/nightly_tune)
• Tests as listed
• docs/research/entry_conversion_research.md — read-only input; do NOT edit

Out of scope:

• ANY live-trading change. No experiment result may alter gates, sizing, or exits in this plan. Promotion of a PASS verdict to live behavior is a separate, owner-approved change (lands on plan 005's unified gate).
• Deleting the legacy labeler triples (follow-up after two clean harness cycles).
• The dashboard ledger panel UI (plan 007 renders experiment_verdicts; this plan only fills the tables).
• New market-data feeds.

Git workflow

• Branch: advisor/006-experiment-os
• Commits: schema → registry/CLI → replay engine (+fidelity tests) → adjudicator → first cohort registrations → scheduler wiring.
• Do NOT push or open a PR.

Steps

Step 0: Precondition check (plan 004 must have landed)

test -f strategy/population.py && grep -q "nightly_tune FAILED" docker-compose.yml deploy/docker/truenas-compose.yml && echo OK

If either check fails, plan 004 has not landed — STOP and report. (At the planned-at commit neither exists; this plan is unexecutable before 004.)

Step 1: Schema + registry

storage/experiments_schema.sql as above; registry.py applies it idempotently (respect PG_SKIP_SCHEMA=1, mirroring storage/db_pg.py:275-277 semantics) and implements register(json_path) (validates required keys incl. fidelity floors; refuses to overwrite an existing id — immutability), list(), get(id).

Verify: register a fixture experiment twice → first succeeds, second errors with "already registered". Tables exist (\dt or information_schema query).

Step 2: Replay engine

replay.py: for each running experiment, find fires not yet in experiment_runs (source: order_filled buy events joined to their submits — the _rebuild_daily_state payload idiom, trading_execution.py:412-446 — plus entry_gate_evaluated receipts when present), filter through strategy/population.py with the experiment's population JSON, load that symbol/day's minute_bars, and compute:

• baseline_r: what actually happened (from position_closed realized P&L in R units — the live truth, not a simulation).
• variant_r: the intervention applied counterfactually on the bar path. Intervention kinds for the first cohort, implemented as small pure functions in replay.py:
  • time_stop: exit at market (next bar open, +adverse tick) if no new post-entry high AND < +0.5R within N minutes.
  • probe_add: 0.5× at fill; add 0.5× at first new high above trigger-bar high within 10 min (fill at that bar's high + adverse tick); blended R.
  • asym_exit: replace the live exit with stop = max(atr_frac × ATR14, spread_mult × spread_proxy), no target, trail to EOD close. Simulate via strategy/exits.py simulate functions ONLY (declare in detail which code simulated; never mix live-manager logic in).
  • rank_one_bullet: portfolio-level — for each session, keep only the top-ranked fire. Rank source: plan 005's entry_gate_evaluated receipts ONLY — no historical event payload carries rank (the armed book is memory-only; do not hunt for an armed-book event type, it does not exist). Expect detail.rank_missing=true for effectively ALL fires predating plan 005; exclude those from n, register the experiment anyway, and report the starvation (its clock starts when receipts flow).
• FIDELITY RULES (hard-coded, not per-experiment-optional): +1-tick adverse fill on every counterfactual entry AND stop; spread_proxy from quiet-bar high-low as described in the research doc; a fidelity block recorded into detail per run.

Verify: python -m research.experiments run over a dev DB with ≥1 registered experiment and stored fires → experiment_runs rows appear; re-run → zero new rows (idempotent); unit fidelity tests pass (Test plan #2–4).

Step 3: Adjudicator

adjudicate.py: per running experiment, compute metrics over experiment_runs (n, mean/median R delta, +1R rate, win rate — whichever the experiment's bars reference), compare against bars/futility:

• all bars met AND n ≥ min_n → PASS
• futility triggered → KILLED
• any bar decisively failed at n ≥ min_n → FAIL
• else → INCONCLUSIVE (recorded, no spam)

Write experiment_verdicts, update experiments.status on terminal verdicts, emit an event (experiment_verdict type via EventStore) and Telegram ping on any TERMINAL verdict or status change only.

Then pm1 through the engine — the explicit bridge (pm1's data lives in the labeler's setup_events/outcome_labels tables, not in fires): backfill experiment_runs for pm1_v1 from outcome_labels joined to setup_events (one row per labeled premarket setup: variant_r = its reached-+1R-before-−1R outcome as ±1; max_upside/max_drawdown into detail; baseline_r NULL — pm1 is a detector track, not an intervention on fires), and extend the adjudicator's metric vocabulary to cover pm1's bars: median_up_60m, median_dd_60m, rate_1r, big_gap_rate_1r. Register pm1_v1 with bars exactly from research/labeler.py:120-122 + the validate function: {"median_up_60m_min": 0.10, "median_dd_60m_min": -0.06, "rate_1r_min": 0.50, "big_gap_rate_1r_min": 0.50}. Its FAIL verdict at n=2377 becomes the harness's first output. Keep the TSV append in premarket_eval.py for one release (dual-write).

Verify: python -m research.experiments adjudicate on the dev DB → pm1_v1 verdict row FAIL (or the truth of the data at run time), one event emitted, no repeat ping on second run.

Step 4: First cohort registrations

experiments/*.json — five files. The bars below are pre-registered HERE, in this plan (the research doc pre-registers the measures and fidelity floors, NOT numeric pass/fail thresholds — copy only the fidelity floors from it; rank_one_bullet does not appear in the doc at all). Use these bars verbatim:

id	intervention	bars (pass requires ALL)	futility (kill when)
time_stop_10m_v1	{"kind":"time_stop","minutes":10,"confirm":"new_high_or_0.5R"}	{"expR_delta_min": 0.05, "min_n": 300}	{"min_n": 500, "expR_delta_max": -0.02}
probe_add_v1	{"kind":"probe_add","probe_frac":0.5,"confirm_minutes":10}	{"expR_delta_min": 0.05, "min_n": 300}	{"min_n": 500, "expR_delta_max": -0.02}
asym_exit_v1	{"kind":"asym_exit","atr_frac":0.25,"no_target":true,"trail":"eod"}	{"expR_delta_min": 0.10, "min_n": 300} (higher bar — bigger structural change)	{"min_n": 500, "expR_delta_max": -0.02}
rank_one_bullet_v1	{"kind":"rank_one_bullet","top_n":1}	{"expR_delta_min": 0.10, "min_n": 100} (n = sessions with rank)	{"min_n": 200, "expR_delta_max": -0.05}
pm1_v1	(bridge — step 3)	{"median_up_60m_min": 0.10, "median_dd_60m_min": -0.06, "rate_1r_min": 0.50, "big_gap_rate_1r_min": 0.50} (from labeler.py:120-122)	none — adjudicate immediately at n=2377

expR_delta = mean(variant_r − baseline_r) over runs. All five carry hypothesis text, population (production thresholds via strategy/population.py — see plan 004's "pass production values explicitly" note), and fidelity floors {"atr_frac": 0.25, "spread_mult": 3, "adverse_ticks": 1} (these ARE from the research doc).

Verify: all five register; run + adjudicate complete end-to-end on stored history; verdicts land in the table.

Step 5: Schedule

Add to both compose scheduler blocks after nightly_tune (plan 004's line):

python -m research.experiments run || echo "[scheduler] experiments run FAILED"
python -m research.experiments adjudicate || echo "[scheduler] experiments adjudicate FAILED"

Verify: grep -n "research.experiments" docker-compose.yml deploy/docker/truenas-compose.yml → both present, ordered after tune.

Test plan

1. Registry: register/immutability/validation errors (missing fidelity → refuse).
2. Fidelity: a synthetic bar path where zero-slippage would profit but adverse-tick fills lose → variant_r reflects the loss (the anti-artifact regression test — this is the single most important test in the plan).
3. time_stop on a synthetic poke-then-die path → scratch loss smaller than full stop; on a confirm-then-run path → unchanged from baseline.
4. probe_add blended-R arithmetic on a known path (hand-computed expected value in the test).
5. Adjudicator: PASS/FAIL/KILLED/INCONCLUSIVE each from constructed run rows; terminal-verdict ping fires once.
6. Idempotency: run twice → no duplicate experiment_runs (PK holds). Structure: model on tests/unit/ conventions; synthetic bars built inline (see tests/synthetic.py helpers if applicable).

• Verification: DATABASE_URL=… python -m pytest -q green, ~12 new tests.

Done criteria

• storage/experiments_schema.sql exists; three tables created idempotently; PG_SKIP_SCHEMA honored
• python -m research.experiments register|run|adjudicate all function end-to-end on a dev DB
• Five first-cohort experiments registered from experiments/*.json
• pm1 verdict emitted through the harness (event + verdict row)
• Fidelity floors hard-enforced (test #2 green)
• Scheduler wired in both compose files
• Full suite green; zero changes to live gates/sizing/exits (git diff --stat shows no trading_execution.py/runtime/ changes)
• plans/README.md updated

STOP conditions

• Fire history proves insufficient to reconstruct entries (missing order events for a large share of fills) — report the coverage number (fires reconstructable / total) before building interventions.
• strategy/exits.py simulate functions cannot express the asym_exit variant without modifying live-shared code — report; do NOT fork exit logic silently.
• Rank is unavailable for >50% of historical fires (rank_one_bullet starves) — register it anyway but report; its clock starts when plan 005 receipts flow.
• Anything requires touching live gates/sizing/exits — hard stop; that is promotion, which is owner-gated.

Maintenance notes

• The promotion path for a PASS verdict: owner reviews experiment_verdicts + detail distributions → implements the intervention behind a flag on plan 005's unified gate → ships as its own change. The harness NEVER auto-promotes.
• After two clean nightly cycles, the four legacy labeler triples (~1,200 lines) and the TSV append become deletable — log it as a follow-up in plans/README.md when reached.
• New experiment = one JSON file + (if a new intervention kind) one pure function in replay.py with its fidelity test. Review bar: no experiment registers without pre-registered bars and fidelity floors.
• float_cache/dilution_events become experiment FEATURES (population filters) — first use retires their "write-only" status from plan 002's log.