plans(017): L2 order-book depth recorder — free-first, spend-gated, pre-registered

The one untested selection axis (order-book depth = the owner's 'size stacked
above the break' instinct). Staged so $0 is spent until a free sniff test +
recorder prove the signal exists and covers the universe; paid feed is an
explicit owner gate; the signal faces the same pre-registered bar + random-gate
null as every refuted lever before it touches the live entry. Honest ceiling
stated: a PASS nudges odds, is not an autonomous oracle.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

plans/017-l2-depth-recorder.md

@@ -0,0 +1,236 @@
+# Plan 017: Level-2 order-book depth recorder — the one untested selection axis, recorded free-first, spend only after the data earns it
+
+> **Executor / operator instructions**: This plan is deliberately staged so
+> that NO paid data is bought until a free evaluation proves the signal is
+> real. Phases 0–1 cost $0 and are the gate. Do NOT skip to a paid feed.
+> Follow each phase's verification before the next. When a phase lands, update
+> the status row in `plans/README.md`.
+>
+> **Drift check (run first)**: `git diff --stat 053c927..HEAD -- run_live_paper.py research/experiments/replay.py storage/`
+> On a mismatch with the "Current state" excerpts, STOP and reconcile.
+
+## Status
+
+- **Priority**: P1 (this is the project's founding question — can premarket data give autonomous selection confidence — made into a testable experiment; every other lever is refuted)
+- **Effort**: L (multi-week, mostly passive data accrual)
+- **Risk**: LOW to live trading (recording-only; zero entry/exit/sizing change until a separate, owner-gated promotion plan). MED to wallet (Phase 2 is a paid feed — gated behind Phase 1 proof).
+- **Depends on**: 006 (Experiment OS harness), the `quote_snapshots` recorder pattern (already in prod)
+- **Category**: research-infrastructure / data-acquisition
+- **Planned at**: commit `053c927`, 2026-07-18
+
+## Why this matters (read fully — this is the strategic core)
+
+The bot enters and exits well; it cannot SELECT. Twelve entry-conversion
+levers computable from 1-min bars + top-of-book quotes have been refuted. The
+measured reason (2026-07-18, owner's own visual review confirmed it): at the
+trigger, ~80% of clean gap/RVOL/VWAP setups fail, and the winners and losers
+are **indistinguishable in the available data**. A model can only find signal
+that exists in its inputs; the free inputs are exhausted.
+
+The one axis never tested is the one the owner's own instinct names: **the
+order book — resting limit-order size stacked above/below the break.** Top-of-
+book (NBBO) is already recorded (`quote_snapshots`); full depth (all price
+levels, "Level 2" / Nasdaq TotalView) is not, because it needs a paid feed the
+current broker (Alpaca) does not sell. Depth is *intent* (what is waiting) vs.
+the tape's *history* (what happened) — genuinely new information, not a
+re-derivation of a refuted feature.
+
+**Honest priors, stated so nobody oversells this:**
+- L2 on thin small-caps is the noisiest, most-gamed data in the market
+  (spoofing, hidden/iceberg size, HFT layering). The visible book ≠ the book
+  that fills. This is a real reason it might show nothing.
+- Even if it works, best case nudges the hit rate (e.g. 20%→30% winnable),
+  not to "autonomous certainty." Pros with full depth AND borrow data still
+  run modest hit rates and win on discipline/sizing — which the bot already
+  does.
+- It therefore MUST clear the same pre-registered bar as every other lever
+  (`expR_delta ≥ +0.05`, `n ≥ 300`, adverse-fill fidelity, **and the
+  random-gate null** added 2026-07-17), scoped to the live `v1` population,
+  before one dollar of it touches the entry decision.
+
+The value of this plan is not "add L2 and win." It is: **spend $0 to find out
+whether the order book carries the selection signal, and only pay for the feed
+if the free evaluation says the signal is there.** That is the disciplined
+answer to "get to high confidence" — record the candidate, test it cold,
+believe only what clears the bar.
+
+## Current state (what to mirror)
+
+- `run_live_paper.py:~218-230` — the DDL + `research_con` recorder pattern:
+
+  ```python
+  research_con.execute(
+      """CREATE TABLE IF NOT EXISTS quote_snapshots (
+             symbol TEXT NOT NULL, ts TIMESTAMPTZ NOT NULL DEFAULT now(),
+             bid_price DOUBLE PRECISION, bid_size DOUBLE PRECISION,
+             ask_price DOUBLE PRECISION, ask_size DOUBLE PRECISION)""")
+  ```
+
+- `run_live_paper.py:~972` `step_quotes()` — the exact recorder idiom to
+  copy: gather tracked-trigger + held symbols, fetch, insert per symbol, never
+  raise, return a one-line status string. Registered on the scheduler with an
+  interval (see the `scheduler.add("quotes", ...)` line, ~15 s cadence). The
+  L2 recorder is a sibling step with the same shape and a wider row.
+- `research/experiments/replay.py` — the harness that consumes recorded
+  features. The order-flow gate interventions from `orderflow_notes.md`
+  (`iv_spread_expansion_gate`, `iv_quote_imbalance_gate`) already read
+  `quote_snapshots` with a **skip-on-no-coverage** design (a fire without
+  data → `variant_r = None`, excluded from `n`, never silently kept). The L2
+  hypothesis reuses this exact coverage discipline (the dilution_veto_v1
+  false-FAIL lesson).
+- `storage/db_pg.py` `PgConnection` — the facade auto-reconnects on a dropped
+  connection (`_run`/`_reconnect`); the recorder inherits that for free.
+- Config convention: every recorder/feature is env-flagged
+  (`_flag("X_ENABLED", "1")`), default-on, fail-open. The L2 recorder ships
+  **default-OFF** until Phase 2 (no free depth source exists to record from
+  in Phases 0–1).
+
+## Phased plan (each phase gates the next)
+
+### Phase 0 — FREE evaluation: does any usable depth exist, and does it look predictive at all? ($0)
+
+Goal: decide whether L2 is worth a paid feed WITHOUT buying one, using free
+trials and free tiers only.
+
+- **0a. Vendor scan (no purchase).** Identify 2–3 sources that expose
+  full-depth (Nasdaq TotalView / MBO or aggregated depth) for the $2–20
+  small-cap universe, and their FREE trial / free-tier terms. Candidates to
+  evaluate (verify current terms, do not assume): Polygon.io (has an aggregated
+  quotes tier; confirm whether true depth is included and at what plan),
+  Databento (pay-as-you-go with a free credit allotment — TotalView/MBO
+  available; good for a bounded historical pull), Alpaca (confirm it still does
+  NOT sell depth, only NBBO — the assumption this plan rests on), a DAS/
+  broker-terminal trial. Deliverable: a one-page comparison —
+  provider × (depth available? y/n) × (free trial size) × ($/mo after) ×
+  (symbols covered) × (latency). **No card entered that auto-charges.**
+- **0b. Free historical probe.** Using a free-credit/trial pull (e.g.
+  Databento's free allotment or a Polygon trial), fetch full-depth book
+  snapshots for **~20 of the bot's actual recent gapper fires** (pull the
+  symbols+session_dates from `setup_events WHERE setup_version='v1'` on days
+  with known outcomes). For each fire, compute one candidate feature at the
+  trigger minute: **depth imbalance = (Σ bid size top N levels − Σ ask size top
+  N levels) / (Σ both)**, plus **ask-stack-above = resting ask size within X%
+  above the break / recent traded volume**. Join to the known
+  `reached_1r_before_minus_1r` outcome.
+- **Verify / GO-NO-GO:** a crude eyeball on ~20 fires (winners vs losers) shows
+  a *visible* separation in depth imbalance or ask-stack — even noisy. This is
+  NOT the real test (n far too small); it is the cheap sniff test that decides
+  whether the paid recording phase is worth starting. **If 20 fires show zero
+  hint of separation AND the vendor scan shows depth is expensive → STOP here,
+  write the negative result in `plans/README.md`, recommend the human-selector
+  fork instead. This is a legitimate, cheap, correct place to end.**
+
+### Phase 1 — Build the recorder against the free/trial feed, log-only ($0 or trial credit)
+
+If Phase 0 passed the sniff test, build the pipe — still on free/trial data
+where possible, recording only.
+
+- **1a. Schema.** New table `depth_snapshots` (research DB), mirroring the
+  `quote_snapshots` DDL idiom but wider:
+  `symbol TEXT, ts TIMESTAMPTZ DEFAULT now(), provider TEXT,
+   bids JSONB, asks JSONB, mid DOUBLE PRECISION, spread_bps DOUBLE PRECISION`
+  where `bids`/`asks` are the top-N `[price, size]` ladder (N≈10). JSONB keeps
+  it one row per snapshot without a level table; the harness reads the ladder
+  back and computes imbalance features. Index `(symbol, ts)`.
+- **1b. Recorder step.** New `research/ingestion/depth_recorder.py` with a
+  `snapshot_depth(con, client, symbols) -> str` function shaped exactly like
+  `step_quotes()` (fetch top-N depth per symbol, insert, never raise, return a
+  status line). Wire a `scheduler.add("depth", ...)` step in `run_live_paper.py`
+  behind `_flag("DEPTH_RECORD_ENABLED", "0")` (**default OFF**), symbols =
+  same tracked-trigger + held set as `step_quotes`. Cadence: ~5 s around the
+  open (depth is fast), configurable via `DEPTH_RECORD_INTERVAL_SECONDS`.
+  A provider adapter module isolates the vendor API so Phase 2 swaps the feed
+  without touching the step.
+- **1c. Feature extractor.** `research/depth_features.py`:
+  `depth_imbalance(ladder_bids, ladder_asks, levels)`,
+  `ask_overhang(asks, break_price, pct_window)`,
+  `book_thinness(bids, asks)` — all pure, unit-tested on synthetic ladders.
+- **Verify:** recorder run against the trial feed writes well-formed
+  `depth_snapshots` rows for live/replayed gappers; `depth_features` unit tests
+  pass; `python3 -m pytest tests/unit/ -q` green; recording is provably
+  zero-impact on the trade loop (grep: no import of depth code in any
+  gate/exit/sizing path). **STOP condition:** if the trial feed's coverage of
+  $2–20 small-caps is poor (the names the bot trades aren't covered), report —
+  a feed that doesn't cover the universe is worthless regardless of price.
+
+### Phase 2 — Paid feed + accrual (THE SPEND — owner-gated, explicit)
+
+Only after Phases 0–1 prove the data exists, covers the universe, and sniff-
+tests positive. This is the single go/no-go where money is committed.
+
+- **2a. Owner decision point.** Present: the chosen vendor, exact $/mo, the
+  Phase-0 sniff result, and the expected accrual timeline to `n ≥ 300` covered
+  fires (at ~5–15 qualifying v1 fires/session with depth coverage, ~4–8 weeks).
+  **The executor does not buy anything; the owner does, or declines.**
+- **2b. Flip `DEPTH_RECORD_ENABLED=1`** in the deployed compose (both files),
+  pointed at the paid feed via the Phase-1 adapter. Recording only. Let it run.
+- **Verify:** `depth_snapshots` accrues daily for the live gapper set; a weekly
+  coverage line reports covered-fires / total-fires (the accrual clock).
+
+### Phase 3 — Pre-registered experiment (the real verdict)
+
+When covered fires reach the bar, test it like every other lever — no
+exceptions, no special pleading.
+
+- Register `experiments/depth_imbalance_v1.json`: keep-or-zero gate that zeros
+  fires whose trigger-minute depth imbalance is ask-heavy (supply overhang
+  above the break), population `{gap≥0.10, vol≥300k, $2–20}`, bar
+  `{expR_delta_min: 0.05, min_n: 300}` **with the random-gate null applied**,
+  fidelity `{atr_frac:0.25, spread_mult:3, adverse_ticks:1}`. Coverage design:
+  fires without a depth snapshot are SKIPPED (variant None), never kept.
+- Add `iv_depth_imbalance_gate` to `replay.py` (additive, mirrors
+  `iv_quote_imbalance_gate`), reading `depth_snapshots`. Record the raw
+  imbalance in `detail` per fire for tercile/stratification autopsy.
+- **Verify / VERDICT:** nightly `experiments run`+`adjudicate` produces a
+  verdict. PASS (clears the bar AND the null AND survives gap%/RVOL
+  stratification) → a SEPARATE owner-gated promotion plan wires it into the
+  live entry gate. FAIL/INCONCLUSIVE → recorded honestly; the founding
+  question is answered "no even with depth," and the human-selector fork is the
+  remaining path. Either way, the project gets a real answer instead of a hope.
+
+## Scope
+
+**In scope**: `storage/` (new `depth_snapshots` DDL), `research/ingestion/
+depth_recorder.py` + provider adapter (new), `research/depth_features.py`
+(new), `run_live_paper.py` (one flag-gated scheduler step), `research/
+experiments/replay.py` (one additive intervention, Phase 3), `experiments/
+depth_imbalance_v1.json` (Phase 3), `tests/`.
+
+**Out of scope**: ANY change to the live entry/exit/sizing path (promotion is a
+separate plan after a PASS); buying data before the Phase-2 owner gate;
+trading on depth in any form during Phases 0–3.
+
+## Done criteria (per phase, not all at once)
+
+- [ ] Phase 0: vendor comparison one-pager + 20-fire sniff result recorded in `plans/README.md`; GO or STOP decision explicit
+- [ ] Phase 1: `depth_snapshots` table + flag-OFF recorder + pure feature extractors, all tested, zero live-path imports
+- [ ] Phase 2: owner-gated spend decision documented; if GO, recording accrues with a weekly coverage line
+- [ ] Phase 3: pre-registered experiment adjudicated; verdict recorded; promotion is a NEW plan only on PASS
+- [ ] `git status` clean per phase; suite green; `ruff check --select F` clean
+
+## STOP conditions
+
+- Phase 0 sniff shows no separation hint AND depth is costly → STOP, record the
+  negative, recommend the human-selector fork. (Cheapest correct ending.)
+- Any trial/paid feed fails to cover the $2–20 small-cap universe the bot
+  actually trades → STOP; a feed that misses the names is worthless.
+- Any temptation to trade on depth before the Phase-3 verdict → STOP (violates
+  the positive-evidence contract; this is exactly the discipline that killed
+  12 false levers cheaply).
+- Phase 3 experiment cannot reach `n ≥ 300` covered fires within ~3 months →
+  park as accrual-blocked, do not lower the bar.
+
+## Maintenance notes
+
+- Depth data is large. `depth_snapshots` JSONB at 5 s cadence over the watched
+  set will grow fast — add it to the nightly retention purge (mirror the
+  `entry_gate_evaluated` 14-day purge in `research/premarket_eval.py`) keeping
+  only fires' trigger-window snapshots long-term, spam-purge the rest.
+- The provider adapter is the seam: keep vendor specifics behind it so a feed
+  swap (or drop) is one module, never a loop change.
+- If Phase 3 PASSES: the promotion plan must re-verify latency — a signal that
+  works in recorded snapshots must survive being read and acted on inside the
+  live trigger thread's timing budget (the same fast-path discipline the
+  regime gate needed).
+- Honest ceiling reminder for whoever reads this later: a PASS nudges the odds;
+  it is not the autonomous oracle. Size and discipline still carry the P&L.

plans/README.md

@@ -33,6 +33,7 @@ your row when done.
 | 014 | Live decision trace — gate-receipt feed (value→threshold per gate) + chart reject markers | P2 | M | 013 (chart overlay half; feed panel standalone) | **DONE (2026-07-17, review-APPROVED first pass, zero revisions; commit dd81dbb on `worktree-agent-acb00535650deac1a` — awaiting owner merge). Decision Trace panel (lazy-load + SSE-throttled), /api/decisions (clamped 500, case-insensitive, server-side epoch), collapsed reject markers merged into the chart's single markers primitive.** |
 | 015 | Backtest replay dashboard (/backtest) — theoretical entries/exits over past sessions under CURRENT settings, via Experiment-OS replay | P2 | L | 013 (chart component) | **DONE (2026-07-17, review-APPROVED first pass; commit 8ac222a on `worktree-agent-ae4d3f36e18c6e7a5` — awaiting owner merge). replay_window = assembly-only over harness primitives (same _entry_index/adverse-fill/R math); /api/backtest 45-session clamp + 10-min cache behind a dedicated lock; /backtest page with THEORETICAL banner + X-of-Y coverage line + per-trade drill-in (blue entry/purple exit markers). Deviations accepted: /api/chart gained backward-compatible date= anchor (72h cap couldn't reach old sessions); batching idiom taken from regime.py (plan's run() pointer was wrong).** |
 | 016 | EDGAR dilution enrichment — unbreak silent nightly refresh (UTC current_date bug + missing float_cache DDL) + pre-register dilution_veto_v1 keep-or-zero experiment | P2 | M | 006 | **DONE (2026-07-17, review-APPROVED first pass; f6999c7 merged). iv_dilution_gate keep-or-zero (fail-open on unfetched symbols, early results biased toward zero never inflated), dilution_veto_v1 self-registers via the experiments/*.json glob, nightly adjudication live. Prod backfill seeded same day (304 symbols swept, 245 filings / 91 symbols). FIRST ADJUDICATION 2026-07-17: **FAIL** (n=2044, expR_delta +0.0012 vs +0.05 bar) — but only 64/2069 fires (3%) had filing coverage (backfill covers recent actives only; historical fires read unflagged by fail-open design), AND the flagged fires ran BETTER than unflagged (baseline −0.038 vs −0.377, n=64): early evidence points AGAINST the dilution-trap thesis, not merely underpowered. Bar-design lesson: min_n counted ALL fires not flagged ones. Verdict terminal (no goalpost-moving); filings keep accruing nightly. IF revisited: dilution_veto_v2 with a flagged-fire min_n metric, after ~3-4 weeks coverage.** |
+| 017 | L2 order-book depth recorder — the one untested selection axis; FREE-first (Phase 0-1 = $0 sniff test + recorder), paid feed only after the data earns it, pre-registered verdict before any live use | P1 | L | 006 | TODO (owner-approved to plan, no spend yet) |
 
 Status values: TODO | IN PROGRESS | DONE | BLOCKED (with one-line reason) | REJECTED (with one-line rationale)