diff --git a/plans/017-l2-depth-recorder.md b/plans/017-l2-depth-recorder.md new file mode 100644 index 0000000..608f36b --- /dev/null +++ b/plans/017-l2-depth-recorder.md @@ -0,0 +1,236 @@ +# Plan 017: Level-2 order-book depth recorder — the one untested selection axis, recorded free-first, spend only after the data earns it + +> **Executor / operator instructions**: This plan is deliberately staged so +> that NO paid data is bought until a free evaluation proves the signal is +> real. Phases 0–1 cost $0 and are the gate. Do NOT skip to a paid feed. +> Follow each phase's verification before the next. When a phase lands, update +> the status row in `plans/README.md`. +> +> **Drift check (run first)**: `git diff --stat 053c927..HEAD -- run_live_paper.py research/experiments/replay.py storage/` +> On a mismatch with the "Current state" excerpts, STOP and reconcile. + +## Status + +- **Priority**: P1 (this is the project's founding question — can premarket data give autonomous selection confidence — made into a testable experiment; every other lever is refuted) +- **Effort**: L (multi-week, mostly passive data accrual) +- **Risk**: LOW to live trading (recording-only; zero entry/exit/sizing change until a separate, owner-gated promotion plan). MED to wallet (Phase 2 is a paid feed — gated behind Phase 1 proof). +- **Depends on**: 006 (Experiment OS harness), the `quote_snapshots` recorder pattern (already in prod) +- **Category**: research-infrastructure / data-acquisition +- **Planned at**: commit `053c927`, 2026-07-18 + +## Why this matters (read fully — this is the strategic core) + +The bot enters and exits well; it cannot SELECT. Twelve entry-conversion +levers computable from 1-min bars + top-of-book quotes have been refuted. The +measured reason (2026-07-18, owner's own visual review confirmed it): at the +trigger, ~80% of clean gap/RVOL/VWAP setups fail, and the winners and losers +are **indistinguishable in the available data**. A model can only find signal +that exists in its inputs; the free inputs are exhausted. + +The one axis never tested is the one the owner's own instinct names: **the +order book — resting limit-order size stacked above/below the break.** Top-of- +book (NBBO) is already recorded (`quote_snapshots`); full depth (all price +levels, "Level 2" / Nasdaq TotalView) is not, because it needs a paid feed the +current broker (Alpaca) does not sell. Depth is *intent* (what is waiting) vs. +the tape's *history* (what happened) — genuinely new information, not a +re-derivation of a refuted feature. + +**Honest priors, stated so nobody oversells this:** +- L2 on thin small-caps is the noisiest, most-gamed data in the market + (spoofing, hidden/iceberg size, HFT layering). The visible book ≠ the book + that fills. This is a real reason it might show nothing. +- Even if it works, best case nudges the hit rate (e.g. 20%→30% winnable), + not to "autonomous certainty." Pros with full depth AND borrow data still + run modest hit rates and win on discipline/sizing — which the bot already + does. +- It therefore MUST clear the same pre-registered bar as every other lever + (`expR_delta ≥ +0.05`, `n ≥ 300`, adverse-fill fidelity, **and the + random-gate null** added 2026-07-17), scoped to the live `v1` population, + before one dollar of it touches the entry decision. + +The value of this plan is not "add L2 and win." It is: **spend $0 to find out +whether the order book carries the selection signal, and only pay for the feed +if the free evaluation says the signal is there.** That is the disciplined +answer to "get to high confidence" — record the candidate, test it cold, +believe only what clears the bar. + +## Current state (what to mirror) + +- `run_live_paper.py:~218-230` — the DDL + `research_con` recorder pattern: + + ```python + research_con.execute( + """CREATE TABLE IF NOT EXISTS quote_snapshots ( + symbol TEXT NOT NULL, ts TIMESTAMPTZ NOT NULL DEFAULT now(), + bid_price DOUBLE PRECISION, bid_size DOUBLE PRECISION, + ask_price DOUBLE PRECISION, ask_size DOUBLE PRECISION)""") + ``` + +- `run_live_paper.py:~972` `step_quotes()` — the exact recorder idiom to + copy: gather tracked-trigger + held symbols, fetch, insert per symbol, never + raise, return a one-line status string. Registered on the scheduler with an + interval (see the `scheduler.add("quotes", ...)` line, ~15 s cadence). The + L2 recorder is a sibling step with the same shape and a wider row. +- `research/experiments/replay.py` — the harness that consumes recorded + features. The order-flow gate interventions from `orderflow_notes.md` + (`iv_spread_expansion_gate`, `iv_quote_imbalance_gate`) already read + `quote_snapshots` with a **skip-on-no-coverage** design (a fire without + data → `variant_r = None`, excluded from `n`, never silently kept). The L2 + hypothesis reuses this exact coverage discipline (the dilution_veto_v1 + false-FAIL lesson). +- `storage/db_pg.py` `PgConnection` — the facade auto-reconnects on a dropped + connection (`_run`/`_reconnect`); the recorder inherits that for free. +- Config convention: every recorder/feature is env-flagged + (`_flag("X_ENABLED", "1")`), default-on, fail-open. The L2 recorder ships + **default-OFF** until Phase 2 (no free depth source exists to record from + in Phases 0–1). + +## Phased plan (each phase gates the next) + +### Phase 0 — FREE evaluation: does any usable depth exist, and does it look predictive at all? ($0) + +Goal: decide whether L2 is worth a paid feed WITHOUT buying one, using free +trials and free tiers only. + +- **0a. Vendor scan (no purchase).** Identify 2–3 sources that expose + full-depth (Nasdaq TotalView / MBO or aggregated depth) for the $2–20 + small-cap universe, and their FREE trial / free-tier terms. Candidates to + evaluate (verify current terms, do not assume): Polygon.io (has an aggregated + quotes tier; confirm whether true depth is included and at what plan), + Databento (pay-as-you-go with a free credit allotment — TotalView/MBO + available; good for a bounded historical pull), Alpaca (confirm it still does + NOT sell depth, only NBBO — the assumption this plan rests on), a DAS/ + broker-terminal trial. Deliverable: a one-page comparison — + provider × (depth available? y/n) × (free trial size) × ($/mo after) × + (symbols covered) × (latency). **No card entered that auto-charges.** +- **0b. Free historical probe.** Using a free-credit/trial pull (e.g. + Databento's free allotment or a Polygon trial), fetch full-depth book + snapshots for **~20 of the bot's actual recent gapper fires** (pull the + symbols+session_dates from `setup_events WHERE setup_version='v1'` on days + with known outcomes). For each fire, compute one candidate feature at the + trigger minute: **depth imbalance = (Σ bid size top N levels − Σ ask size top + N levels) / (Σ both)**, plus **ask-stack-above = resting ask size within X% + above the break / recent traded volume**. Join to the known + `reached_1r_before_minus_1r` outcome. +- **Verify / GO-NO-GO:** a crude eyeball on ~20 fires (winners vs losers) shows + a *visible* separation in depth imbalance or ask-stack — even noisy. This is + NOT the real test (n far too small); it is the cheap sniff test that decides + whether the paid recording phase is worth starting. **If 20 fires show zero + hint of separation AND the vendor scan shows depth is expensive → STOP here, + write the negative result in `plans/README.md`, recommend the human-selector + fork instead. This is a legitimate, cheap, correct place to end.** + +### Phase 1 — Build the recorder against the free/trial feed, log-only ($0 or trial credit) + +If Phase 0 passed the sniff test, build the pipe — still on free/trial data +where possible, recording only. + +- **1a. Schema.** New table `depth_snapshots` (research DB), mirroring the + `quote_snapshots` DDL idiom but wider: + `symbol TEXT, ts TIMESTAMPTZ DEFAULT now(), provider TEXT, + bids JSONB, asks JSONB, mid DOUBLE PRECISION, spread_bps DOUBLE PRECISION` + where `bids`/`asks` are the top-N `[price, size]` ladder (N≈10). JSONB keeps + it one row per snapshot without a level table; the harness reads the ladder + back and computes imbalance features. Index `(symbol, ts)`. +- **1b. Recorder step.** New `research/ingestion/depth_recorder.py` with a + `snapshot_depth(con, client, symbols) -> str` function shaped exactly like + `step_quotes()` (fetch top-N depth per symbol, insert, never raise, return a + status line). Wire a `scheduler.add("depth", ...)` step in `run_live_paper.py` + behind `_flag("DEPTH_RECORD_ENABLED", "0")` (**default OFF**), symbols = + same tracked-trigger + held set as `step_quotes`. Cadence: ~5 s around the + open (depth is fast), configurable via `DEPTH_RECORD_INTERVAL_SECONDS`. + A provider adapter module isolates the vendor API so Phase 2 swaps the feed + without touching the step. +- **1c. Feature extractor.** `research/depth_features.py`: + `depth_imbalance(ladder_bids, ladder_asks, levels)`, + `ask_overhang(asks, break_price, pct_window)`, + `book_thinness(bids, asks)` — all pure, unit-tested on synthetic ladders. +- **Verify:** recorder run against the trial feed writes well-formed + `depth_snapshots` rows for live/replayed gappers; `depth_features` unit tests + pass; `python3 -m pytest tests/unit/ -q` green; recording is provably + zero-impact on the trade loop (grep: no import of depth code in any + gate/exit/sizing path). **STOP condition:** if the trial feed's coverage of + $2–20 small-caps is poor (the names the bot trades aren't covered), report — + a feed that doesn't cover the universe is worthless regardless of price. + +### Phase 2 — Paid feed + accrual (THE SPEND — owner-gated, explicit) + +Only after Phases 0–1 prove the data exists, covers the universe, and sniff- +tests positive. This is the single go/no-go where money is committed. + +- **2a. Owner decision point.** Present: the chosen vendor, exact $/mo, the + Phase-0 sniff result, and the expected accrual timeline to `n ≥ 300` covered + fires (at ~5–15 qualifying v1 fires/session with depth coverage, ~4–8 weeks). + **The executor does not buy anything; the owner does, or declines.** +- **2b. Flip `DEPTH_RECORD_ENABLED=1`** in the deployed compose (both files), + pointed at the paid feed via the Phase-1 adapter. Recording only. Let it run. +- **Verify:** `depth_snapshots` accrues daily for the live gapper set; a weekly + coverage line reports covered-fires / total-fires (the accrual clock). + +### Phase 3 — Pre-registered experiment (the real verdict) + +When covered fires reach the bar, test it like every other lever — no +exceptions, no special pleading. + +- Register `experiments/depth_imbalance_v1.json`: keep-or-zero gate that zeros + fires whose trigger-minute depth imbalance is ask-heavy (supply overhang + above the break), population `{gap≥0.10, vol≥300k, $2–20}`, bar + `{expR_delta_min: 0.05, min_n: 300}` **with the random-gate null applied**, + fidelity `{atr_frac:0.25, spread_mult:3, adverse_ticks:1}`. Coverage design: + fires without a depth snapshot are SKIPPED (variant None), never kept. +- Add `iv_depth_imbalance_gate` to `replay.py` (additive, mirrors + `iv_quote_imbalance_gate`), reading `depth_snapshots`. Record the raw + imbalance in `detail` per fire for tercile/stratification autopsy. +- **Verify / VERDICT:** nightly `experiments run`+`adjudicate` produces a + verdict. PASS (clears the bar AND the null AND survives gap%/RVOL + stratification) → a SEPARATE owner-gated promotion plan wires it into the + live entry gate. FAIL/INCONCLUSIVE → recorded honestly; the founding + question is answered "no even with depth," and the human-selector fork is the + remaining path. Either way, the project gets a real answer instead of a hope. + +## Scope + +**In scope**: `storage/` (new `depth_snapshots` DDL), `research/ingestion/ +depth_recorder.py` + provider adapter (new), `research/depth_features.py` +(new), `run_live_paper.py` (one flag-gated scheduler step), `research/ +experiments/replay.py` (one additive intervention, Phase 3), `experiments/ +depth_imbalance_v1.json` (Phase 3), `tests/`. + +**Out of scope**: ANY change to the live entry/exit/sizing path (promotion is a +separate plan after a PASS); buying data before the Phase-2 owner gate; +trading on depth in any form during Phases 0–3. + +## Done criteria (per phase, not all at once) + +- [ ] Phase 0: vendor comparison one-pager + 20-fire sniff result recorded in `plans/README.md`; GO or STOP decision explicit +- [ ] Phase 1: `depth_snapshots` table + flag-OFF recorder + pure feature extractors, all tested, zero live-path imports +- [ ] Phase 2: owner-gated spend decision documented; if GO, recording accrues with a weekly coverage line +- [ ] Phase 3: pre-registered experiment adjudicated; verdict recorded; promotion is a NEW plan only on PASS +- [ ] `git status` clean per phase; suite green; `ruff check --select F` clean + +## STOP conditions + +- Phase 0 sniff shows no separation hint AND depth is costly → STOP, record the + negative, recommend the human-selector fork. (Cheapest correct ending.) +- Any trial/paid feed fails to cover the $2–20 small-cap universe the bot + actually trades → STOP; a feed that misses the names is worthless. +- Any temptation to trade on depth before the Phase-3 verdict → STOP (violates + the positive-evidence contract; this is exactly the discipline that killed + 12 false levers cheaply). +- Phase 3 experiment cannot reach `n ≥ 300` covered fires within ~3 months → + park as accrual-blocked, do not lower the bar. + +## Maintenance notes + +- Depth data is large. `depth_snapshots` JSONB at 5 s cadence over the watched + set will grow fast — add it to the nightly retention purge (mirror the + `entry_gate_evaluated` 14-day purge in `research/premarket_eval.py`) keeping + only fires' trigger-window snapshots long-term, spam-purge the rest. +- The provider adapter is the seam: keep vendor specifics behind it so a feed + swap (or drop) is one module, never a loop change. +- If Phase 3 PASSES: the promotion plan must re-verify latency — a signal that + works in recorded snapshots must survive being read and acted on inside the + live trigger thread's timing budget (the same fast-path discipline the + regime gate needed). +- Honest ceiling reminder for whoever reads this later: a PASS nudges the odds; + it is not the autonomous oracle. Size and discipline still carry the P&L. diff --git a/plans/README.md b/plans/README.md index 472d410..a150d4b 100644 --- a/plans/README.md +++ b/plans/README.md @@ -33,6 +33,7 @@ your row when done. | 014 | Live decision trace — gate-receipt feed (value→threshold per gate) + chart reject markers | P2 | M | 013 (chart overlay half; feed panel standalone) | **DONE (2026-07-17, review-APPROVED first pass, zero revisions; commit dd81dbb on `worktree-agent-acb00535650deac1a` — awaiting owner merge). Decision Trace panel (lazy-load + SSE-throttled), /api/decisions (clamped 500, case-insensitive, server-side epoch), collapsed reject markers merged into the chart's single markers primitive.** | | 015 | Backtest replay dashboard (/backtest) — theoretical entries/exits over past sessions under CURRENT settings, via Experiment-OS replay | P2 | L | 013 (chart component) | **DONE (2026-07-17, review-APPROVED first pass; commit 8ac222a on `worktree-agent-ae4d3f36e18c6e7a5` — awaiting owner merge). replay_window = assembly-only over harness primitives (same _entry_index/adverse-fill/R math); /api/backtest 45-session clamp + 10-min cache behind a dedicated lock; /backtest page with THEORETICAL banner + X-of-Y coverage line + per-trade drill-in (blue entry/purple exit markers). Deviations accepted: /api/chart gained backward-compatible date= anchor (72h cap couldn't reach old sessions); batching idiom taken from regime.py (plan's run() pointer was wrong).** | | 016 | EDGAR dilution enrichment — unbreak silent nightly refresh (UTC current_date bug + missing float_cache DDL) + pre-register dilution_veto_v1 keep-or-zero experiment | P2 | M | 006 | **DONE (2026-07-17, review-APPROVED first pass; f6999c7 merged). iv_dilution_gate keep-or-zero (fail-open on unfetched symbols, early results biased toward zero never inflated), dilution_veto_v1 self-registers via the experiments/*.json glob, nightly adjudication live. Prod backfill seeded same day (304 symbols swept, 245 filings / 91 symbols). FIRST ADJUDICATION 2026-07-17: **FAIL** (n=2044, expR_delta +0.0012 vs +0.05 bar) — but only 64/2069 fires (3%) had filing coverage (backfill covers recent actives only; historical fires read unflagged by fail-open design), AND the flagged fires ran BETTER than unflagged (baseline −0.038 vs −0.377, n=64): early evidence points AGAINST the dilution-trap thesis, not merely underpowered. Bar-design lesson: min_n counted ALL fires not flagged ones. Verdict terminal (no goalpost-moving); filings keep accruing nightly. IF revisited: dilution_veto_v2 with a flagged-fire min_n metric, after ~3-4 weeks coverage.** | +| 017 | L2 order-book depth recorder — the one untested selection axis; FREE-first (Phase 0-1 = $0 sniff test + recorder), paid feed only after the data earns it, pre-registered verdict before any live use | P1 | L | 006 | TODO (owner-approved to plan, no spend yet) | Status values: TODO | IN PROGRESS | DONE | BLOCKED (with one-line reason) | REJECTED (with one-line rationale)