momentum-system/plans/017-l2-depth-recorder.md

Plan 017: Level-2 order-book depth recorder — the one untested selection axis, recorded free-first, spend only after the data earns it

Executor / operator instructions: This plan is deliberately staged so that NO paid data is bought until a free evaluation proves the signal is real. Phases 0–1 cost $0 and are the gate. Do NOT skip to a paid feed. Follow each phase's verification before the next. When a phase lands, update the status row in plans/README.md.

Drift check (run first): git diff --stat 053c927..HEAD -- run_live_paper.py research/experiments/replay.py storage/ On a mismatch with the "Current state" excerpts, STOP and reconcile.

Status

• Priority: P1 (this is the project's founding question — can premarket data give autonomous selection confidence — made into a testable experiment; every other lever is refuted)
• Effort: L (multi-week, mostly passive data accrual)
• Risk: LOW to live trading (recording-only; zero entry/exit/sizing change until a separate, owner-gated promotion plan). MED to wallet (Phase 2 is a paid feed — gated behind Phase 1 proof).
• Depends on: 006 (Experiment OS harness), the quote_snapshots recorder pattern (already in prod)
• Category: research-infrastructure / data-acquisition
• Planned at: commit 053c927, 2026-07-18

Why this matters (read fully — this is the strategic core)

The bot enters and exits well; it cannot SELECT. Twelve entry-conversion levers computable from 1-min bars + top-of-book quotes have been refuted. The measured reason (2026-07-18, owner's own visual review confirmed it): at the trigger, ~80% of clean gap/RVOL/VWAP setups fail, and the winners and losers are indistinguishable in the available data. A model can only find signal that exists in its inputs; the free inputs are exhausted.

The one axis never tested is the one the owner's own instinct names: the order book — resting limit-order size stacked above/below the break. Top-of- book (NBBO) is already recorded (quote_snapshots); full depth (all price levels, "Level 2" / Nasdaq TotalView) is not, because it needs a paid feed the current broker (Alpaca) does not sell. Depth is intent (what is waiting) vs. the tape's history (what happened) — genuinely new information, not a re-derivation of a refuted feature.

Honest priors, stated so nobody oversells this:

• L2 on thin small-caps is the noisiest, most-gamed data in the market (spoofing, hidden/iceberg size, HFT layering). The visible book ≠ the book that fills. This is a real reason it might show nothing.
• Even if it works, best case nudges the hit rate (e.g. 20%→30% winnable), not to "autonomous certainty." Pros with full depth AND borrow data still run modest hit rates and win on discipline/sizing — which the bot already does.
• It therefore MUST clear the same pre-registered bar as every other lever (expR_delta ≥ +0.05, n ≥ 300, adverse-fill fidelity, and the random-gate null added 2026-07-17), scoped to the live v1 population, before one dollar of it touches the entry decision.

The value of this plan is not "add L2 and win." It is: spend $0 to find out whether the order book carries the selection signal, and only pay for the feed if the free evaluation says the signal is there. That is the disciplined answer to "get to high confidence" — record the candidate, test it cold, believe only what clears the bar.

Current state (what to mirror)

• run_live_paper.py:~218-230 — the DDL + research_con recorder pattern:

  research_con.execute(
      """CREATE TABLE IF NOT EXISTS quote_snapshots (
             symbol TEXT NOT NULL, ts TIMESTAMPTZ NOT NULL DEFAULT now(),
             bid_price DOUBLE PRECISION, bid_size DOUBLE PRECISION,
             ask_price DOUBLE PRECISION, ask_size DOUBLE PRECISION)""")
  

• run_live_paper.py:~972 step_quotes() — the exact recorder idiom to copy: gather tracked-trigger + held symbols, fetch, insert per symbol, never raise, return a one-line status string. Registered on the scheduler with an interval (see the scheduler.add("quotes", ...) line, ~15 s cadence). The L2 recorder is a sibling step with the same shape and a wider row.

• research/experiments/replay.py — the harness that consumes recorded features. The order-flow gate interventions from orderflow_notes.md (iv_spread_expansion_gate, iv_quote_imbalance_gate) already read quote_snapshots with a skip-on-no-coverage design (a fire without data → variant_r = None, excluded from n, never silently kept). The L2 hypothesis reuses this exact coverage discipline (the dilution_veto_v1 false-FAIL lesson).

• storage/db_pg.py PgConnection — the facade auto-reconnects on a dropped connection (_run/_reconnect); the recorder inherits that for free.

• Config convention: every recorder/feature is env-flagged (_flag("X_ENABLED", "1")), default-on, fail-open. The L2 recorder ships default-OFF until Phase 2 (no free depth source exists to record from in Phases 0–1).

Phased plan (each phase gates the next)

UPDATE 2026-07-18 — Phase 0 is DE-RISKED, cost verified ~$0. The owner's DATABENTO_API_KEY (already in .env) is live and reaches XNAS.ITCH (Nasdaq TotalView, MBP-10 = 10-level order-book depth). Cost-checked (no data spent, metadata.get_cost): 5 min of AAPL depth = $0.026; and every real recent bot gapper tested — EHGO, ATPC, MUD, LASE, AEON — resolves with full depth at $0.0001–0.0066 per fire (thin books = tiny payloads). The full 20-fire Phase-0 sniff is < $1, inside Databento's free signup credits. The "will we spend to find out" risk is gone for Phase 0 — it is now a same-day free experiment. Databento covers the $2–20 Nasdaq-listed universe the bot trades; NYSE/ARCA-listed gappers may need DBEQ.BASIC (consolidated) — check per-symbol in 0b. pip install databento (research layer only; NOT the loop).

Phase 0 — FREE evaluation: does any usable depth exist, and does it look predictive at all? ($0 — DATABENTO KEY IN HAND)

Goal: decide whether L2 is worth a paid feed WITHOUT buying one, using free trials and free tiers only.

• 0a. Vendor scan (no purchase). Identify 2–3 sources that expose full-depth (Nasdaq TotalView / MBO or aggregated depth) for the $2–20 small-cap universe, and their FREE trial / free-tier terms. Candidates to evaluate (verify current terms, do not assume): Polygon.io (has an aggregated quotes tier; confirm whether true depth is included and at what plan), Databento (pay-as-you-go with a free credit allotment — TotalView/MBO available; good for a bounded historical pull), Alpaca (confirm it still does NOT sell depth, only NBBO — the assumption this plan rests on), a DAS/ broker-terminal trial. Deliverable: a one-page comparison — provider × (depth available? y/n) × (free trial size) × ($/mo after) × (symbols covered) × (latency). No card entered that auto-charges.
• 0b. Free historical probe. Using a free-credit/trial pull (e.g. Databento's free allotment or a Polygon trial), fetch full-depth book snapshots for ~20 of the bot's actual recent gapper fires (pull the symbols+session_dates from setup_events WHERE setup_version='v1' on days with known outcomes). For each fire, compute one candidate feature at the trigger minute: depth imbalance = (Σ bid size top N levels − Σ ask size top N levels) / (Σ both), plus ask-stack-above = resting ask size within X% above the break / recent traded volume. Join to the known reached_1r_before_minus_1r outcome.
• Verify / GO-NO-GO: a crude eyeball on ~20 fires (winners vs losers) shows a visible separation in depth imbalance or ask-stack — even noisy. This is NOT the real test (n far too small); it is the cheap sniff test that decides whether the paid recording phase is worth starting. If 20 fires show zero hint of separation AND the vendor scan shows depth is expensive → STOP here, write the negative result in plans/README.md, recommend the human-selector fork instead. This is a legitimate, cheap, correct place to end.

PHASE 0 RESULT — 2026-07-18 — GO. First genuine signal in the project. Ran the sniff on 40 real v1 fires ($2-20, 07-17), pulled the actual Databento XNAS.ITCH order book at each trigger (23/40 covered; 17 missing = NYSE names needing DBEQ.BASIC — a Phase-1 coverage fix). Total spend: pennies.

• Naive book imbalance (full + near-touch): NOISE (AUC 0.45 / 0.50) — the obvious "bid-heavy = good" idea is dead, as it was in bar data.
• Ask-overhang above the break: AUC 0.80. Winners' median resting ask-size within 1% above the break = 0.276 vs losers' 0.005 (~50x). Median split: low-overhang fires won 27%, high-overhang 67% (+40pts, robust to outliers).
• Direction is INVERTED from the supply-trap intuition: a thick ask wall above the break predicts WINNING — the absorption/fuel mechanism (a break that clears visible resting supply ignites; a break into thin air fizzles). This is invisible in OHLCV — the first feature that needed the order book to see. CAVEATS: n=23, single day (regime confound); must rule out price/liquidity proxying at Phase 3 (stratify by price/rvol); a sniff, NOT a verdict. GO to Phase 1 (build recorder), NOT to trading. The lead feature to record and test: ask_overhang = Σ ask_sz at px in (break, break*1.01] / Σ bid_sz.

Phase 1 — Build the recorder against the free/trial feed, log-only ($0 or trial credit)

If Phase 0 passed the sniff test, build the pipe — still on free/trial data where possible, recording only.

• 1a. Schema. New table depth_snapshots (research DB), mirroring the quote_snapshots DDL idiom but wider: symbol TEXT, ts TIMESTAMPTZ DEFAULT now(), provider TEXT, bids JSONB, asks JSONB, mid DOUBLE PRECISION, spread_bps DOUBLE PRECISION where bids/asks are the top-N [price, size] ladder (N≈10). JSONB keeps it one row per snapshot without a level table; the harness reads the ladder back and computes imbalance features. Index (symbol, ts).
• 1b. Recorder step. New research/ingestion/depth_recorder.py with a snapshot_depth(con, client, symbols) -> str function shaped exactly like step_quotes() (fetch top-N depth per symbol, insert, never raise, return a status line). Wire a scheduler.add("depth", ...) step in run_live_paper.py behind _flag("DEPTH_RECORD_ENABLED", "0") (default OFF), symbols = same tracked-trigger + held set as step_quotes. Cadence: ~5 s around the open (depth is fast), configurable via DEPTH_RECORD_INTERVAL_SECONDS. A provider adapter module isolates the vendor API so Phase 2 swaps the feed without touching the step.
• 1c. Feature extractor. research/depth_features.py: depth_imbalance(ladder_bids, ladder_asks, levels), ask_overhang(asks, break_price, pct_window), book_thinness(bids, asks) — all pure, unit-tested on synthetic ladders.
• Verify: recorder run against the trial feed writes well-formed depth_snapshots rows for live/replayed gappers; depth_features unit tests pass; python3 -m pytest tests/unit/ -q green; recording is provably zero-impact on the trade loop (grep: no import of depth code in any gate/exit/sizing path). STOP condition: if the trial feed's coverage of $2–20 small-caps is poor (the names the bot trades aren't covered), report — a feed that doesn't cover the universe is worthless regardless of price.

Phase 2 — Paid feed + accrual (THE SPEND — owner-gated, explicit)

Only after Phases 0–1 prove the data exists, covers the universe, and sniff- tests positive. This is the single go/no-go where money is committed.

• 2a. Owner decision point. Present: the chosen vendor, exact $/mo, the Phase-0 sniff result, and the expected accrual timeline to n ≥ 300 covered fires (at ~5–15 qualifying v1 fires/session with depth coverage, ~4–8 weeks). The executor does not buy anything; the owner does, or declines.
• 2b. Flip DEPTH_RECORD_ENABLED=1 in the deployed compose (both files), pointed at the paid feed via the Phase-1 adapter. Recording only. Let it run.
• Verify: depth_snapshots accrues daily for the live gapper set; a weekly coverage line reports covered-fires / total-fires (the accrual clock).

Phase 3 — Pre-registered experiment (the real verdict)

When covered fires reach the bar, test it like every other lever — no exceptions, no special pleading.

• Register experiments/depth_imbalance_v1.json: keep-or-zero gate that zeros fires whose trigger-minute depth imbalance is ask-heavy (supply overhang above the break), population {gap≥0.10, vol≥300k, $2–20}, bar {expR_delta_min: 0.05, min_n: 300} with the random-gate null applied, fidelity {atr_frac:0.25, spread_mult:3, adverse_ticks:1}. Coverage design: fires without a depth snapshot are SKIPPED (variant None), never kept.
• Add iv_depth_imbalance_gate to replay.py (additive, mirrors iv_quote_imbalance_gate), reading depth_snapshots. Record the raw imbalance in detail per fire for tercile/stratification autopsy.
• Verify / VERDICT: nightly experiments run+adjudicate produces a verdict. PASS (clears the bar AND the null AND survives gap%/RVOL stratification) → a SEPARATE owner-gated promotion plan wires it into the live entry gate. FAIL/INCONCLUSIVE → recorded honestly; the founding question is answered "no even with depth," and the human-selector fork is the remaining path. Either way, the project gets a real answer instead of a hope.

Scope

In scope: storage/ (new depth_snapshots DDL), research/ingestion/ depth_recorder.py + provider adapter (new), research/depth_features.py (new), run_live_paper.py (one flag-gated scheduler step), research/ experiments/replay.py (one additive intervention, Phase 3), experiments/ depth_imbalance_v1.json (Phase 3), tests/.

Out of scope: ANY change to the live entry/exit/sizing path (promotion is a separate plan after a PASS); buying data before the Phase-2 owner gate; trading on depth in any form during Phases 0–3.

Done criteria (per phase, not all at once)

• Phase 0: vendor comparison one-pager + 20-fire sniff result recorded in plans/README.md; GO or STOP decision explicit
• Phase 1: depth_snapshots table + flag-OFF recorder + pure feature extractors, all tested, zero live-path imports
• Phase 2: owner-gated spend decision documented; if GO, recording accrues with a weekly coverage line
• Phase 3: pre-registered experiment adjudicated; verdict recorded; promotion is a NEW plan only on PASS
• git status clean per phase; suite green; ruff check --select F clean

STOP conditions

• Phase 0 sniff shows no separation hint AND depth is costly → STOP, record the negative, recommend the human-selector fork. (Cheapest correct ending.)
• Any trial/paid feed fails to cover the $2–20 small-cap universe the bot actually trades → STOP; a feed that misses the names is worthless.
• Any temptation to trade on depth before the Phase-3 verdict → STOP (violates the positive-evidence contract; this is exactly the discipline that killed 12 false levers cheaply).
• Phase 3 experiment cannot reach n ≥ 300 covered fires within ~3 months → park as accrual-blocked, do not lower the bar.

Maintenance notes

• Depth data is large. depth_snapshots JSONB at 5 s cadence over the watched set will grow fast — add it to the nightly retention purge (mirror the entry_gate_evaluated 14-day purge in research/premarket_eval.py) keeping only fires' trigger-window snapshots long-term, spam-purge the rest.
• The provider adapter is the seam: keep vendor specifics behind it so a feed swap (or drop) is one module, never a loop change.
• If Phase 3 PASSES: the promotion plan must re-verify latency — a signal that works in recorded snapshots must survive being read and acted on inside the live trigger thread's timing budget (the same fast-path discipline the regime gate needed).
• Honest ceiling reminder for whoever reads this later: a PASS nudges the odds; it is not the autonomous oracle. Size and discipline still carry the P&L.