From d8d37700aaa09615462a0e9cadb0012b188d5063 Mon Sep 17 00:00:00 2001 From: momentum dev Date: Fri, 17 Jul 2026 09:29:40 -0500 Subject: [PATCH] =?UTF-8?q?plans(016):=20EDGAR=20dilution=20enrichment=20?= =?UTF-8?q?=E2=80=94=20unbreak=20nightly=20refresh=20+=20pre-register=20fi?= =?UTF-8?q?ling-veto=20experiment?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5 --- plans/016-edgar-dilution-enrichment.md | 288 +++++++++++++++++++++++++ plans/README.md | 1 + 2 files changed, 289 insertions(+) create mode 100644 plans/016-edgar-dilution-enrichment.md diff --git a/plans/016-edgar-dilution-enrichment.md b/plans/016-edgar-dilution-enrichment.md new file mode 100644 index 0000000..0c3af62 --- /dev/null +++ b/plans/016-edgar-dilution-enrichment.md @@ -0,0 +1,288 @@ +# Plan 016: EDGAR dilution enrichment — unbreak the silent shadow pipeline, seed it, and pre-register the filing-veto experiment + +> **Executor instructions**: Follow this plan step by step. Run every +> verification command and confirm the expected result before moving to the +> next step. If anything in the "STOP conditions" section occurs, stop and +> report — do not improvise. When done, update the status row for this plan +> in `plans/README.md` — unless a reviewer dispatched you and told you they +> maintain the index. +> +> **Drift check (run first)**: `git diff --stat 92488b7..HEAD -- research/premarket_eval.py research/ingestion/sec_dilution.py research/ingestion/float_cache.py research/experiments/ experiments/ tests/unit/` +> On mismatch with the "Current state" excerpts, STOP. + +## Status + +- **Priority**: P2 +- **Effort**: M +- **Risk**: LOW-MED (nightly-job + shadow-experiment code; ZERO live entry-path change) +- **Depends on**: none (Experiment OS = plan 006, long DONE) +- **Category**: research-infrastructure / data-quality +- **Planned at**: commit `92488b7`, 2026-07-17 + +## Why this matters + +A fresh S-1/S-3 shelf or 424B offering on a runner is the classic dilution +trap the source trader warns about. The repo ALREADY has an EDGAR filings +fetcher (`research/ingestion/sec_dilution.py`) wired into the nightly job — +but `dilution_events` has **0 rows in production, ever**, because the call +site is silently broken two independent ways (verified 2026-07-17): + +1. The symbol query uses `session_date = current_date`, and production + Postgres runs in **Etc/UTC** — at 20:00 EDT (= 00:00 UTC *next day*) + `current_date` is tomorrow, so the query returns zero symbols every night. +2. `refresh_float_cache` runs first in the SAME try-block, and the + `float_cache` table does not exist in production (its `CREATE TABLE IF NOT + EXISTS` DDL sits in the module but is never executed) — the + `UndefinedTable` exception skips the dilution refresh too. + +The fetcher itself works (live-tested 2026-07-17: ATPC → 2 filings, EHGO → 2, +false-positive guard via `display_names` correct). This plan: fix the call +site, apply the missing DDL, seed a backfill, and — the payoff — register a +**pre-registered keep-or-zero experiment** (`dilution_veto_v1`) that measures +on logged fires whether skipping filing-flagged names raises expR. No LLM +needed: the filing TYPE is deterministic. The live veto stays OFF until the +experiment passes its bar (owner-gated promotion, out of scope here). + +## Current state + +- `research/ingestion/sec_dilution.py` (87 lines) — WORKING fetcher: + - `fetch_filings(symbol)` → EDGAR full-text search, forms + `S-1,S-3,424B5,424B4,424B3`, 180-day lookback, `display_names` contains + `(SYMBOL)` guard, `USER_AGENT = "momentum-research fez5587@gmail.com"`, + 0.4 s pause between symbols. + - `refresh_dilution_events(con, symbols)` → idempotent insert into + `dilution_events` (id = accession number). + - `recent_dilution(con, symbol, days=90)` → filing count reader. +- `dilution_events` DDL exists in `storage/pg_schema.sql:131` (id, symbol, + event_type, announced_at, …). Table exists in prod; 0 rows. +- Broken call site: `research/premarket_eval.py:122-133`: + + ```python + try: + from research.ingestion.float_cache import refresh_float_cache + from research.ingestion.sec_dilution import refresh_dilution_events + syms = [r[0] for r in con.execute( + "SELECT symbol FROM minute_bars WHERE session_date = current_date " + "GROUP BY symbol ORDER BY sum(volume) DESC LIMIT 30").fetchall()] + if syms: + print("float cache:", refresh_float_cache(con, syms)) + print("dilution filings:", refresh_dilution_events(con, syms)) + except Exception as exc: # noqa: BLE001 — enrichment never breaks the eval + print(f"float/dilution enrichment skipped: {exc}") + ``` + +- ET-session-date convention: the codebase never trusts DB-server local time; + it computes the US/Eastern date in Python — e.g. `api/main.py:89` + `_eastern_session_date()` → `datetime.now(ZoneInfo("America/New_York")).date()`. + `research/labeler.py` and `research/regime.py` pass explicit dates the same + way. Match this: compute the ET date in Python, pass it as a `?` parameter. +- `research/ingestion/float_cache.py:27` — `CREATE TABLE IF NOT EXISTS + float_cache (...)` DDL string exists in the module; nothing executes it. + FMP key env: `FMP_API_KEY` (absent in prod .env → refresh returns an + error-count dict; that's fine — it must just not THROW before dilution runs). +- Experiment OS pattern to copy (keep-or-zero on fires): + - `research/experiments/replay.py:429` `iv_regime_gate(fire, baseline, + favorable, model_version)` → variant_r = baseline if favorable else 0.0; + no-data = KEEP (fail-open); detail dict records the decision. + - Dispatch: `_compute_fire` at replay.py:~503 (`if kind == "regime_gate":`) + and the run-loop prefetch at ~729 (`_load_regime_favorable` loaded ONCE + per run, not per fire). A dilution gate needs the same shape: preload + `{(symbol, session_date_window) …}` once. + - Experiment JSONs live in `experiments/*.json` — copy + `experiments/regime_meta_v1.json`'s shape: population + `{min_entry_gap_pct: 0.10, min_day_volume: 300000, price_min: 2.0, + price_max: 20.0}`, bars `{expR_delta_min: 0.05, min_n: 300}`, fidelity + `{atr_frac: 0.25, spread_mult: 3, adverse_ticks: 1}`. + - Registration: `research.experiments run` self-registers the declared + cohort (16f880d) — check `research/experiments/registry.py` for where the + declared experiment list lives and add the new JSON there. +- Nightly ordering (docker-compose.yml:167-183 scheduler block): + premarket_eval (which refreshes dilution_events) runs BEFORE + `research.experiments run` — so the experiment replays over fresh filings + the same night. No compose change needed. +- Tests: `tests/unit/experiments/` — existing regime-gate experiment tests + show the store/seeding fixtures (`EventStore(":memory:")`); model the new + intervention tests on `tests/unit/experiments/test_regime_gate.py`. + +## Commands you will need + +| Purpose | Command | Expected on success | +|---------|---------|---------------------| +| Experiments tests | `python3 -m pytest tests/unit/experiments/ -q` | all pass (35 existing + new) | +| Full suite | `python3 -m pytest tests/ -q --ignore=tests/unit/api` | all pass | +| API tests (unaffected) | `python3 -m pytest tests/unit/api/ -q` | 15 pre-existing urllib failures only | +| Lint (CI gate) | `ruff check --select F api/ storage/ research/` | exit 0 | + +## Scope + +**In scope**: +- `research/premarket_eval.py` (ONLY the float/dilution block, lines ~122-133) +- `research/ingestion/float_cache.py` (apply its own DDL idempotently) +- `research/ingestion/sec_dilution.py` (if a helper is needed; fetcher logic unchanged) +- `research/experiments/replay.py` (new `iv_dilution_gate` + its preload — + additive only, existing functions untouched) +- `research/experiments/registry.py` (declare the new experiment, matching how + the regime experiments are declared) +- `experiments/dilution_veto_v1.json` (create) +- `tests/unit/experiments/`, `tests/unit/research/` (new tests) + +**Out of scope — do NOT touch**: +- `trading_execution.py`, `run_live_paper.py`, `alpaca_paper/` — the LIVE + dilution veto (`catalyst_veto_enabled`, headline-based) stays exactly as is; + promotion of a filing-based veto is an OWNER decision after the experiment + passes, not this plan. +- `simulate_exit` / fidelity math / population logic. +- Any LLM/Ollama call — filings are classified by FORM TYPE, deterministically. +- compose files (ordering already correct). + +## Git workflow + +- Match repo style; suggested: + `dilution(016): unbreak nightly EDGAR refresh + pre-register filing-veto experiment`. +- Do NOT push or open a PR. + +## Steps + +### Step 1: Fix the nightly call site + +In `research/premarket_eval.py`, replace the block quoted above: + +- Compute the session date in Python (ET): reuse the file's existing imports — + `from zoneinfo import ZoneInfo` idiom as in `api/main.py:_eastern_session_date` + — and pass it as a parameter: + `"SELECT symbol FROM minute_bars WHERE session_date = ? GROUP BY symbol ORDER BY sum(volume) DESC LIMIT 30"`. +- Split into TWO independent try-blocks (float failing must never skip + dilution, and vice versa). Keep each block's failure print. + +**Verify**: new unit test (Test plan case 1) passes. + +### Step 2: float_cache applies its own DDL + +In `research/ingestion/float_cache.py`, execute the module's existing +`CREATE TABLE IF NOT EXISTS` DDL at the top of `refresh_float_cache` (and of +its reader if one is public) — same idempotent apply-on-use idiom as +`research/regime.py:apply_schema`. No schema content change. + +**Verify**: unit test creates a fresh `EventStore(":memory:")`, calls +`refresh_float_cache(con, [], api_key="x")` (empty symbols → no HTTP) → no +exception, `SELECT count(*) FROM float_cache` → 0 rows, table exists. + +### Step 3: `iv_dilution_gate` intervention + preload + +In `research/experiments/replay.py`, additive: + +- `_load_dilution_flags(con, days=90) -> set[tuple[str, date]]` — one query: + for every `dilution_events` row, emit `(symbol, session_date)` pairs the + flag covers… **do NOT materialize 90 days × filings**; instead load + `{symbol: [announced_date, ...]}` once and provide + `def _dilution_flagged(loaded, symbol, session_date, days) -> bool` + (any filing with `0 <= session_date - announced_date <= days`). +- `iv_dilution_gate(fire, baseline, flags, params)` → keep-or-zero mirroring + `iv_regime_gate`: flagged → `variant_r = 0.0`, detail + `{"kind": "dilution_gate", "flagged": True, "lookback_days": N}`; not + flagged (or symbol absent from dilution_events — the no-data case) → KEEP + baseline, `{"flagged": False}` (fail-open: an unfetched symbol must not + count as vetoed). +- Wire into `_compute_fire`'s dispatch and the run-loop's once-per-run preload + exactly where `regime_gate` does both (replay.py:~503 and ~729). + `params` carries `{"lookback_days": 90}` from the experiment JSON. + +**Verify**: new intervention tests (Test plan cases 2-3) pass; +`python3 -m pytest tests/unit/experiments/ -q` all green. + +### Step 4: register `dilution_veto_v1` + +Create `experiments/dilution_veto_v1.json`: + +```json +{ + "id": "dilution_veto_v1", + "hypothesis": "Zeroing ORB fires on names with an S-1/S-3/424B filing in the trailing 90 days raises whole-population expR — the dilution-trap gaps fade as new shares hit.", + "population": {"min_entry_gap_pct": 0.10, "min_day_volume": 300000, "price_min": 2.0, "price_max": 20.0}, + "intervention": {"kind": "dilution_gate", "lookback_days": 90}, + "bars": {"expR_delta_min": 0.05, "min_n": 300}, + "fidelity": {"atr_frac": 0.25, "spread_mult": 3, "adverse_ticks": 1} +} +``` + +Declare it wherever the regime experiments are declared in +`research/experiments/registry.py` (the self-registering cohort) so the +nightly `experiments run` picks it up with zero operator action. + +**Verify**: `python3 -m pytest tests/unit/experiments/ -q` green (registry +tests, if any, still pass); a unit test loads the JSON and asserts the +registry declares it. + +### Step 5: honesty note in the JSON's hypothesis vicinity + adjudication expectations + +Nothing extra to build: `experiments adjudicate` already handles the verdict. +Add one comment where `iv_dilution_gate` is defined: with dilution_events +seeded only from ~now forward (plus the fetcher's 180-day filing lookback per +symbol), early adjudications run with PARTIAL flag coverage — fires on +never-fetched symbols read unflagged (fail-open), so early expR_delta is +biased TOWARD zero, never inflated. The experiment can only understate the +veto's value until coverage accrues — acceptable, self-correcting, worth the +comment so nobody reads an early INCONCLUSIVE as refutation. + +**Verify**: comment present (code review). + +## Test plan + +1. `tests/unit/research/test_premarket_eval_enrichment.py` (new, small): + monkeypatch `refresh_float_cache` to RAISE and `refresh_dilution_events` + to record its call → run the extracted enrichment block (refactor it into + a small function `_refresh_shadow_enrichment(con, session_date)` in + premarket_eval so it's testable without the whole main()) → dilution still + called with the explicit ET date, symbols from a seeded minute_bars store. +2. `tests/unit/experiments/test_dilution_gate.py` (new; model on + `test_regime_gate.py`): seeded `dilution_events` row for SYM on D-10 with + `lookback_days=90` → fire on SYM@D zeroed (`variant_r == 0.0`, + `flagged: True`); fire on OTHER@D kept (baseline passthrough, + `flagged: False`); filing OLDER than lookback → kept. +3. End-to-end registry: after `run` self-registration, `dilution_veto_v1` + exists in the experiments table and a replay over seeded fires writes + `experiment_runs` rows with `detail.kind == "dilution_gate"`. +4. float_cache DDL test (Step 2's verify). + +**Verify**: `python3 -m pytest tests/unit/experiments/ tests/unit/research/ -q` +→ all pass. + +## Done criteria + +- [ ] Nightly enrichment block uses an explicit ET session date and isolated + try-blocks (grep: `current_date` no longer appears in + `research/premarket_eval.py`) +- [ ] `float_cache` DDL applied on use; empty-symbol call is a no-op, not a crash +- [ ] `iv_dilution_gate` additive in replay.py, fail-open on missing data, + preloaded once per run; `experiments/dilution_veto_v1.json` registered + in the self-declaring cohort +- [ ] All new tests pass; `tests/unit/experiments/` fully green; full suite + passes; `ruff check --select F api/ storage/ research/` exit 0 +- [ ] `git status` clean outside scope + +## STOP conditions + +- The registry's self-registration mechanism doesn't match the description + (can't find where regime experiments are declared) — report, don't invent a + second registration path. +- Wiring `iv_dilution_gate` requires modifying `iv_regime_gate` or any + existing intervention — everything here must be additive. +- The premarket_eval refactor (extracting `_refresh_shadow_enrichment`) pulls + in more than the float/dilution block — the function must be a pure lift of + those ~12 lines plus the date fix. +- Any temptation to touch the LIVE veto path in trading_execution.py. + +## Maintenance notes + +- SUPERVISED FOLLOW-UP (reviewer/owner, NOT the executor — it must not touch + prod): seed the backfill once — + `python3 -c "...refresh_dilution_events(con, )"` + against production, so the experiment starts with coverage instead of + waiting weeks. Then watch `experiments adjudicate` output nightly. +- Promotion path if `dilution_veto_v1` PASSES: extend the existing gated + `_dilution_vetoed` (trading_execution.py) to consult `recent_dilution` — + that is a NEW plan with its own live-path review, never a drive-by. +- The 180-day filing lookback (fetcher) vs 90-day veto lookback (experiment) + are deliberately different: fetch wide, test the pre-registered window. +- Deferred, explicitly: LLM classification of 8-K bodies / ATM language — + only worth it if the deterministic form-type veto shows signal first. diff --git a/plans/README.md b/plans/README.md index 3b313d9..d2fa4c0 100644 --- a/plans/README.md +++ b/plans/README.md @@ -32,6 +32,7 @@ your row when done. | 013 | Symbol drill-in: 24h candlestick + live buy/sell trade markers (TradingView Lightweight Charts v5.2, vendored) | P2 | M | 012 (soft — reuses its bracket-leg maps) | **DONE (2026-07-17, review-APPROVED after 1 revision round; commits b4ad3d3+709e7a4 on `worktree-agent-ae2ee0000d5e3a7eb` — awaiting owner merge). Headless-browser render-verified. Executor surfaced 2 pre-existing out-of-scope defects: OrderFilledEvent has no `side` field (worked around via order_submitted sibling join in /api/chart) and GET /api/attribution 500s when DATABASE_URL unset (research/attribution.py hardcodes os.environ lookup).** | | 014 | Live decision trace — gate-receipt feed (value→threshold per gate) + chart reject markers | P2 | M | 013 (chart overlay half; feed panel standalone) | **DONE (2026-07-17, review-APPROVED first pass, zero revisions; commit dd81dbb on `worktree-agent-acb00535650deac1a` — awaiting owner merge). Decision Trace panel (lazy-load + SSE-throttled), /api/decisions (clamped 500, case-insensitive, server-side epoch), collapsed reject markers merged into the chart's single markers primitive.** | | 015 | Backtest replay dashboard (/backtest) — theoretical entries/exits over past sessions under CURRENT settings, via Experiment-OS replay | P2 | L | 013 (chart component) | **DONE (2026-07-17, review-APPROVED first pass; commit 8ac222a on `worktree-agent-ae4d3f36e18c6e7a5` — awaiting owner merge). replay_window = assembly-only over harness primitives (same _entry_index/adverse-fill/R math); /api/backtest 45-session clamp + 10-min cache behind a dedicated lock; /backtest page with THEORETICAL banner + X-of-Y coverage line + per-trade drill-in (blue entry/purple exit markers). Deviations accepted: /api/chart gained backward-compatible date= anchor (72h cap couldn't reach old sessions); batching idiom taken from regime.py (plan's run() pointer was wrong).** | +| 016 | EDGAR dilution enrichment — unbreak silent nightly refresh (UTC current_date bug + missing float_cache DDL) + pre-register dilution_veto_v1 keep-or-zero experiment | P2 | M | 006 | IN PROGRESS (2026-07-17) | Status values: TODO | IN PROGRESS | DONE | BLOCKED (with one-line reason) | REJECTED (with one-line rationale)