# Plan 017: Level-2 order-book depth recorder — the one untested selection axis, recorded free-first, spend only after the data earns it > **Executor / operator instructions**: This plan is deliberately staged so > that NO paid data is bought until a free evaluation proves the signal is > real. Phases 0–1 cost $0 and are the gate. Do NOT skip to a paid feed. > Follow each phase's verification before the next. When a phase lands, update > the status row in `plans/README.md`. > > **Drift check (run first)**: `git diff --stat 053c927..HEAD -- run_live_paper.py research/experiments/replay.py storage/` > On a mismatch with the "Current state" excerpts, STOP and reconcile. ## Status - **Priority**: P1 (this is the project's founding question — can premarket data give autonomous selection confidence — made into a testable experiment; every other lever is refuted) - **Effort**: L (multi-week, mostly passive data accrual) - **Risk**: LOW to live trading (recording-only; zero entry/exit/sizing change until a separate, owner-gated promotion plan). MED to wallet (Phase 2 is a paid feed — gated behind Phase 1 proof). - **Depends on**: 006 (Experiment OS harness), the `quote_snapshots` recorder pattern (already in prod) - **Category**: research-infrastructure / data-acquisition - **Planned at**: commit `053c927`, 2026-07-18 ## Why this matters (read fully — this is the strategic core) The bot enters and exits well; it cannot SELECT. Twelve entry-conversion levers computable from 1-min bars + top-of-book quotes have been refuted. The measured reason (2026-07-18, owner's own visual review confirmed it): at the trigger, ~80% of clean gap/RVOL/VWAP setups fail, and the winners and losers are **indistinguishable in the available data**. A model can only find signal that exists in its inputs; the free inputs are exhausted. The one axis never tested is the one the owner's own instinct names: **the order book — resting limit-order size stacked above/below the break.** Top-of- book (NBBO) is already recorded (`quote_snapshots`); full depth (all price levels, "Level 2" / Nasdaq TotalView) is not, because it needs a paid feed the current broker (Alpaca) does not sell. Depth is *intent* (what is waiting) vs. the tape's *history* (what happened) — genuinely new information, not a re-derivation of a refuted feature. **Honest priors, stated so nobody oversells this:** - L2 on thin small-caps is the noisiest, most-gamed data in the market (spoofing, hidden/iceberg size, HFT layering). The visible book ≠ the book that fills. This is a real reason it might show nothing. - Even if it works, best case nudges the hit rate (e.g. 20%→30% winnable), not to "autonomous certainty." Pros with full depth AND borrow data still run modest hit rates and win on discipline/sizing — which the bot already does. - It therefore MUST clear the same pre-registered bar as every other lever (`expR_delta ≥ +0.05`, `n ≥ 300`, adverse-fill fidelity, **and the random-gate null** added 2026-07-17), scoped to the live `v1` population, before one dollar of it touches the entry decision. The value of this plan is not "add L2 and win." It is: **spend $0 to find out whether the order book carries the selection signal, and only pay for the feed if the free evaluation says the signal is there.** That is the disciplined answer to "get to high confidence" — record the candidate, test it cold, believe only what clears the bar. ## Current state (what to mirror) - `run_live_paper.py:~218-230` — the DDL + `research_con` recorder pattern: ```python research_con.execute( """CREATE TABLE IF NOT EXISTS quote_snapshots ( symbol TEXT NOT NULL, ts TIMESTAMPTZ NOT NULL DEFAULT now(), bid_price DOUBLE PRECISION, bid_size DOUBLE PRECISION, ask_price DOUBLE PRECISION, ask_size DOUBLE PRECISION)""") ``` - `run_live_paper.py:~972` `step_quotes()` — the exact recorder idiom to copy: gather tracked-trigger + held symbols, fetch, insert per symbol, never raise, return a one-line status string. Registered on the scheduler with an interval (see the `scheduler.add("quotes", ...)` line, ~15 s cadence). The L2 recorder is a sibling step with the same shape and a wider row. - `research/experiments/replay.py` — the harness that consumes recorded features. The order-flow gate interventions from `orderflow_notes.md` (`iv_spread_expansion_gate`, `iv_quote_imbalance_gate`) already read `quote_snapshots` with a **skip-on-no-coverage** design (a fire without data → `variant_r = None`, excluded from `n`, never silently kept). The L2 hypothesis reuses this exact coverage discipline (the dilution_veto_v1 false-FAIL lesson). - `storage/db_pg.py` `PgConnection` — the facade auto-reconnects on a dropped connection (`_run`/`_reconnect`); the recorder inherits that for free. - Config convention: every recorder/feature is env-flagged (`_flag("X_ENABLED", "1")`), default-on, fail-open. The L2 recorder ships **default-OFF** until Phase 2 (no free depth source exists to record from in Phases 0–1). ## Phased plan (each phase gates the next) > **UPDATE 2026-07-18 — Phase 0 is DE-RISKED, cost verified ~$0.** The owner's > `DATABENTO_API_KEY` (already in `.env`) is live and reaches **XNAS.ITCH > (Nasdaq TotalView, MBP-10 = 10-level order-book depth)**. Cost-checked (no > data spent, `metadata.get_cost`): 5 min of AAPL depth = $0.026; and every > real recent bot gapper tested — EHGO, ATPC, MUD, LASE, AEON — resolves with > full depth at **$0.0001–0.0066 per fire** (thin books = tiny payloads). The > full 20-fire Phase-0 sniff is **< $1, inside Databento's free signup > credits.** The "will we spend to find out" risk is gone for Phase 0 — it is > now a same-day free experiment. Databento covers the $2–20 Nasdaq-listed > universe the bot trades; NYSE/ARCA-listed gappers may need `DBEQ.BASIC` > (consolidated) — check per-symbol in 0b. `pip install databento` (research > layer only; NOT the loop). ### Phase 0 — FREE evaluation: does any usable depth exist, and does it look predictive at all? ($0 — DATABENTO KEY IN HAND) Goal: decide whether L2 is worth a paid feed WITHOUT buying one, using free trials and free tiers only. - **0a. Vendor scan (no purchase).** Identify 2–3 sources that expose full-depth (Nasdaq TotalView / MBO or aggregated depth) for the $2–20 small-cap universe, and their FREE trial / free-tier terms. Candidates to evaluate (verify current terms, do not assume): Polygon.io (has an aggregated quotes tier; confirm whether true depth is included and at what plan), Databento (pay-as-you-go with a free credit allotment — TotalView/MBO available; good for a bounded historical pull), Alpaca (confirm it still does NOT sell depth, only NBBO — the assumption this plan rests on), a DAS/ broker-terminal trial. Deliverable: a one-page comparison — provider × (depth available? y/n) × (free trial size) × ($/mo after) × (symbols covered) × (latency). **No card entered that auto-charges.** - **0b. Free historical probe.** Using a free-credit/trial pull (e.g. Databento's free allotment or a Polygon trial), fetch full-depth book snapshots for **~20 of the bot's actual recent gapper fires** (pull the symbols+session_dates from `setup_events WHERE setup_version='v1'` on days with known outcomes). For each fire, compute one candidate feature at the trigger minute: **depth imbalance = (Σ bid size top N levels − Σ ask size top N levels) / (Σ both)**, plus **ask-stack-above = resting ask size within X% above the break / recent traded volume**. Join to the known `reached_1r_before_minus_1r` outcome. - **Verify / GO-NO-GO:** a crude eyeball on ~20 fires (winners vs losers) shows a *visible* separation in depth imbalance or ask-stack — even noisy. This is NOT the real test (n far too small); it is the cheap sniff test that decides whether the paid recording phase is worth starting. **If 20 fires show zero hint of separation AND the vendor scan shows depth is expensive → STOP here, write the negative result in `plans/README.md`, recommend the human-selector fork instead. This is a legitimate, cheap, correct place to end.** > **PHASE 0 RESULT — 2026-07-18 — GO. First genuine signal in the project.** > Ran the sniff on 40 real v1 fires ($2-20, 07-17), pulled the actual Databento > XNAS.ITCH order book at each trigger (23/40 covered; 17 missing = NYSE names > needing DBEQ.BASIC — a Phase-1 coverage fix). Total spend: pennies. > - Naive book imbalance (full + near-touch): **NOISE** (AUC 0.45 / 0.50) — the > obvious "bid-heavy = good" idea is dead, as it was in bar data. > - **Ask-overhang above the break: AUC 0.80.** Winners' median resting ask-size > within 1% above the break = 0.276 vs losers' 0.005 (~50x). Median split: > low-overhang fires won 27%, high-overhang 67% (+40pts, robust to outliers). > - Direction is INVERTED from the supply-trap intuition: a thick ask wall above > the break predicts WINNING — the absorption/fuel mechanism (a break that > clears visible resting supply ignites; a break into thin air fizzles). This > is invisible in OHLCV — the first feature that needed the order book to see. > CAVEATS: n=23, single day (regime confound); must rule out price/liquidity > proxying at Phase 3 (stratify by price/rvol); a sniff, NOT a verdict. GO to > Phase 1 (build recorder), NOT to trading. The lead feature to record and test: > `ask_overhang = Σ ask_sz at px in (break, break*1.01] / Σ bid_sz`. ### Phase 1 — Build the recorder against the free/trial feed, log-only ($0 or trial credit) If Phase 0 passed the sniff test, build the pipe — still on free/trial data where possible, recording only. - **1a. Schema.** New table `depth_snapshots` (research DB), mirroring the `quote_snapshots` DDL idiom but wider: `symbol TEXT, ts TIMESTAMPTZ DEFAULT now(), provider TEXT, bids JSONB, asks JSONB, mid DOUBLE PRECISION, spread_bps DOUBLE PRECISION` where `bids`/`asks` are the top-N `[price, size]` ladder (N≈10). JSONB keeps it one row per snapshot without a level table; the harness reads the ladder back and computes imbalance features. Index `(symbol, ts)`. - **1b. Recorder step.** New `research/ingestion/depth_recorder.py` with a `snapshot_depth(con, client, symbols) -> str` function shaped exactly like `step_quotes()` (fetch top-N depth per symbol, insert, never raise, return a status line). Wire a `scheduler.add("depth", ...)` step in `run_live_paper.py` behind `_flag("DEPTH_RECORD_ENABLED", "0")` (**default OFF**), symbols = same tracked-trigger + held set as `step_quotes`. Cadence: ~5 s around the open (depth is fast), configurable via `DEPTH_RECORD_INTERVAL_SECONDS`. A provider adapter module isolates the vendor API so Phase 2 swaps the feed without touching the step. - **1c. Feature extractor.** `research/depth_features.py`: `depth_imbalance(ladder_bids, ladder_asks, levels)`, `ask_overhang(asks, break_price, pct_window)`, `book_thinness(bids, asks)` — all pure, unit-tested on synthetic ladders. - **Verify:** recorder run against the trial feed writes well-formed `depth_snapshots` rows for live/replayed gappers; `depth_features` unit tests pass; `python3 -m pytest tests/unit/ -q` green; recording is provably zero-impact on the trade loop (grep: no import of depth code in any gate/exit/sizing path). **STOP condition:** if the trial feed's coverage of $2–20 small-caps is poor (the names the bot trades aren't covered), report — a feed that doesn't cover the universe is worthless regardless of price. ### Phase 2 — Paid feed + accrual (THE SPEND — owner-gated, explicit) Only after Phases 0–1 prove the data exists, covers the universe, and sniff- tests positive. This is the single go/no-go where money is committed. - **2a. Owner decision point.** Present: the chosen vendor, exact $/mo, the Phase-0 sniff result, and the expected accrual timeline to `n ≥ 300` covered fires (at ~5–15 qualifying v1 fires/session with depth coverage, ~4–8 weeks). **The executor does not buy anything; the owner does, or declines.** - **2b. Flip `DEPTH_RECORD_ENABLED=1`** in the deployed compose (both files), pointed at the paid feed via the Phase-1 adapter. Recording only. Let it run. - **Verify:** `depth_snapshots` accrues daily for the live gapper set; a weekly coverage line reports covered-fires / total-fires (the accrual clock). ### Phase 3 — Pre-registered experiment (the real verdict) When covered fires reach the bar, test it like every other lever — no exceptions, no special pleading. - Register `experiments/depth_imbalance_v1.json`: keep-or-zero gate that zeros fires whose trigger-minute depth imbalance is ask-heavy (supply overhang above the break), population `{gap≥0.10, vol≥300k, $2–20}`, bar `{expR_delta_min: 0.05, min_n: 300}` **with the random-gate null applied**, fidelity `{atr_frac:0.25, spread_mult:3, adverse_ticks:1}`. Coverage design: fires without a depth snapshot are SKIPPED (variant None), never kept. - Add `iv_depth_imbalance_gate` to `replay.py` (additive, mirrors `iv_quote_imbalance_gate`), reading `depth_snapshots`. Record the raw imbalance in `detail` per fire for tercile/stratification autopsy. - **Verify / VERDICT:** nightly `experiments run`+`adjudicate` produces a verdict. PASS (clears the bar AND the null AND survives gap%/RVOL stratification) → a SEPARATE owner-gated promotion plan wires it into the live entry gate. FAIL/INCONCLUSIVE → recorded honestly; the founding question is answered "no even with depth," and the human-selector fork is the remaining path. Either way, the project gets a real answer instead of a hope. ## Scope **In scope**: `storage/` (new `depth_snapshots` DDL), `research/ingestion/ depth_recorder.py` + provider adapter (new), `research/depth_features.py` (new), `run_live_paper.py` (one flag-gated scheduler step), `research/ experiments/replay.py` (one additive intervention, Phase 3), `experiments/ depth_imbalance_v1.json` (Phase 3), `tests/`. **Out of scope**: ANY change to the live entry/exit/sizing path (promotion is a separate plan after a PASS); buying data before the Phase-2 owner gate; trading on depth in any form during Phases 0–3. ## Done criteria (per phase, not all at once) - [ ] Phase 0: vendor comparison one-pager + 20-fire sniff result recorded in `plans/README.md`; GO or STOP decision explicit - [ ] Phase 1: `depth_snapshots` table + flag-OFF recorder + pure feature extractors, all tested, zero live-path imports - [ ] Phase 2: owner-gated spend decision documented; if GO, recording accrues with a weekly coverage line - [ ] Phase 3: pre-registered experiment adjudicated; verdict recorded; promotion is a NEW plan only on PASS - [ ] `git status` clean per phase; suite green; `ruff check --select F` clean ## STOP conditions - Phase 0 sniff shows no separation hint AND depth is costly → STOP, record the negative, recommend the human-selector fork. (Cheapest correct ending.) - Any trial/paid feed fails to cover the $2–20 small-cap universe the bot actually trades → STOP; a feed that misses the names is worthless. - Any temptation to trade on depth before the Phase-3 verdict → STOP (violates the positive-evidence contract; this is exactly the discipline that killed 12 false levers cheaply). - Phase 3 experiment cannot reach `n ≥ 300` covered fires within ~3 months → park as accrual-blocked, do not lower the bar. ## Maintenance notes - Depth data is large. `depth_snapshots` JSONB at 5 s cadence over the watched set will grow fast — add it to the nightly retention purge (mirror the `entry_gate_evaluated` 14-day purge in `research/premarket_eval.py`) keeping only fires' trigger-window snapshots long-term, spam-purge the rest. - The provider adapter is the seam: keep vendor specifics behind it so a feed swap (or drop) is one module, never a loop change. - If Phase 3 PASSES: the promotion plan must re-verify latency — a signal that works in recorded snapshots must survive being read and acted on inside the live trigger thread's timing budget (the same fast-path discipline the regime gate needed). - Honest ceiling reminder for whoever reads this later: a PASS nudges the odds; it is not the autonomous oracle. Size and discipline still carry the P&L.