plans(016): EDGAR dilution enrichment — unbreak nightly refresh + pre-register filing-veto experiment

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

plans/016-edgar-dilution-enrichment.md

@@ -0,0 +1,288 @@
+# Plan 016: EDGAR dilution enrichment — unbreak the silent shadow pipeline, seed it, and pre-register the filing-veto experiment
+
+> **Executor instructions**: Follow this plan step by step. Run every
+> verification command and confirm the expected result before moving to the
+> next step. If anything in the "STOP conditions" section occurs, stop and
+> report — do not improvise. When done, update the status row for this plan
+> in `plans/README.md` — unless a reviewer dispatched you and told you they
+> maintain the index.
+>
+> **Drift check (run first)**: `git diff --stat 92488b7..HEAD -- research/premarket_eval.py research/ingestion/sec_dilution.py research/ingestion/float_cache.py research/experiments/ experiments/ tests/unit/`
+> On mismatch with the "Current state" excerpts, STOP.
+
+## Status
+
+- **Priority**: P2
+- **Effort**: M
+- **Risk**: LOW-MED (nightly-job + shadow-experiment code; ZERO live entry-path change)
+- **Depends on**: none (Experiment OS = plan 006, long DONE)
+- **Category**: research-infrastructure / data-quality
+- **Planned at**: commit `92488b7`, 2026-07-17
+
+## Why this matters
+
+A fresh S-1/S-3 shelf or 424B offering on a runner is the classic dilution
+trap the source trader warns about. The repo ALREADY has an EDGAR filings
+fetcher (`research/ingestion/sec_dilution.py`) wired into the nightly job —
+but `dilution_events` has **0 rows in production, ever**, because the call
+site is silently broken two independent ways (verified 2026-07-17):
+
+1. The symbol query uses `session_date = current_date`, and production
+   Postgres runs in **Etc/UTC** — at 20:00 EDT (= 00:00 UTC *next day*)
+   `current_date` is tomorrow, so the query returns zero symbols every night.
+2. `refresh_float_cache` runs first in the SAME try-block, and the
+   `float_cache` table does not exist in production (its `CREATE TABLE IF NOT
+   EXISTS` DDL sits in the module but is never executed) — the
+   `UndefinedTable` exception skips the dilution refresh too.
+
+The fetcher itself works (live-tested 2026-07-17: ATPC → 2 filings, EHGO → 2,
+false-positive guard via `display_names` correct). This plan: fix the call
+site, apply the missing DDL, seed a backfill, and — the payoff — register a
+**pre-registered keep-or-zero experiment** (`dilution_veto_v1`) that measures
+on logged fires whether skipping filing-flagged names raises expR. No LLM
+needed: the filing TYPE is deterministic. The live veto stays OFF until the
+experiment passes its bar (owner-gated promotion, out of scope here).
+
+## Current state
+
+- `research/ingestion/sec_dilution.py` (87 lines) — WORKING fetcher:
+  - `fetch_filings(symbol)` → EDGAR full-text search, forms
+    `S-1,S-3,424B5,424B4,424B3`, 180-day lookback, `display_names` contains
+    `(SYMBOL)` guard, `USER_AGENT = "momentum-research fez5587@gmail.com"`,
+    0.4 s pause between symbols.
+  - `refresh_dilution_events(con, symbols)` → idempotent insert into
+    `dilution_events` (id = accession number).
+  - `recent_dilution(con, symbol, days=90)` → filing count reader.
+- `dilution_events` DDL exists in `storage/pg_schema.sql:131` (id, symbol,
+  event_type, announced_at, …). Table exists in prod; 0 rows.
+- Broken call site: `research/premarket_eval.py:122-133`:
+
+  ```python
+      try:
+          from research.ingestion.float_cache import refresh_float_cache
+          from research.ingestion.sec_dilution import refresh_dilution_events
+          syms = [r[0] for r in con.execute(
+              "SELECT symbol FROM minute_bars WHERE session_date = current_date "
+              "GROUP BY symbol ORDER BY sum(volume) DESC LIMIT 30").fetchall()]
+          if syms:
+              print("float cache:", refresh_float_cache(con, syms))
+              print("dilution filings:", refresh_dilution_events(con, syms))
+      except Exception as exc:  # noqa: BLE001 — enrichment never breaks the eval
+          print(f"float/dilution enrichment skipped: {exc}")
+  ```
+
+- ET-session-date convention: the codebase never trusts DB-server local time;
+  it computes the US/Eastern date in Python — e.g. `api/main.py:89`
+  `_eastern_session_date()` → `datetime.now(ZoneInfo("America/New_York")).date()`.
+  `research/labeler.py` and `research/regime.py` pass explicit dates the same
+  way. Match this: compute the ET date in Python, pass it as a `?` parameter.
+- `research/ingestion/float_cache.py:27` — `CREATE TABLE IF NOT EXISTS
+  float_cache (...)` DDL string exists in the module; nothing executes it.
+  FMP key env: `FMP_API_KEY` (absent in prod .env → refresh returns an
+  error-count dict; that's fine — it must just not THROW before dilution runs).
+- Experiment OS pattern to copy (keep-or-zero on fires):
+  - `research/experiments/replay.py:429` `iv_regime_gate(fire, baseline,
+    favorable, model_version)` → variant_r = baseline if favorable else 0.0;
+    no-data = KEEP (fail-open); detail dict records the decision.
+  - Dispatch: `_compute_fire` at replay.py:~503 (`if kind == "regime_gate":`)
+    and the run-loop prefetch at ~729 (`_load_regime_favorable` loaded ONCE
+    per run, not per fire). A dilution gate needs the same shape: preload
+    `{(symbol, session_date_window) …}` once.
+  - Experiment JSONs live in `experiments/*.json` — copy
+    `experiments/regime_meta_v1.json`'s shape: population
+    `{min_entry_gap_pct: 0.10, min_day_volume: 300000, price_min: 2.0,
+    price_max: 20.0}`, bars `{expR_delta_min: 0.05, min_n: 300}`, fidelity
+    `{atr_frac: 0.25, spread_mult: 3, adverse_ticks: 1}`.
+  - Registration: `research.experiments run` self-registers the declared
+    cohort (16f880d) — check `research/experiments/registry.py` for where the
+    declared experiment list lives and add the new JSON there.
+- Nightly ordering (docker-compose.yml:167-183 scheduler block):
+  premarket_eval (which refreshes dilution_events) runs BEFORE
+  `research.experiments run` — so the experiment replays over fresh filings
+  the same night. No compose change needed.
+- Tests: `tests/unit/experiments/` — existing regime-gate experiment tests
+  show the store/seeding fixtures (`EventStore(":memory:")`); model the new
+  intervention tests on `tests/unit/experiments/test_regime_gate.py`.
+
+## Commands you will need
+
+| Purpose | Command | Expected on success |
+|---------|---------|---------------------|
+| Experiments tests | `python3 -m pytest tests/unit/experiments/ -q` | all pass (35 existing + new) |
+| Full suite | `python3 -m pytest tests/ -q --ignore=tests/unit/api` | all pass |
+| API tests (unaffected) | `python3 -m pytest tests/unit/api/ -q` | 15 pre-existing urllib failures only |
+| Lint (CI gate) | `ruff check --select F api/ storage/ research/` | exit 0 |
+
+## Scope
+
+**In scope**:
+- `research/premarket_eval.py` (ONLY the float/dilution block, lines ~122-133)
+- `research/ingestion/float_cache.py` (apply its own DDL idempotently)
+- `research/ingestion/sec_dilution.py` (if a helper is needed; fetcher logic unchanged)
+- `research/experiments/replay.py` (new `iv_dilution_gate` + its preload —
+  additive only, existing functions untouched)
+- `research/experiments/registry.py` (declare the new experiment, matching how
+  the regime experiments are declared)
+- `experiments/dilution_veto_v1.json` (create)
+- `tests/unit/experiments/`, `tests/unit/research/` (new tests)
+
+**Out of scope — do NOT touch**:
+- `trading_execution.py`, `run_live_paper.py`, `alpaca_paper/` — the LIVE
+  dilution veto (`catalyst_veto_enabled`, headline-based) stays exactly as is;
+  promotion of a filing-based veto is an OWNER decision after the experiment
+  passes, not this plan.
+- `simulate_exit` / fidelity math / population logic.
+- Any LLM/Ollama call — filings are classified by FORM TYPE, deterministically.
+- compose files (ordering already correct).
+
+## Git workflow
+
+- Match repo style; suggested:
+  `dilution(016): unbreak nightly EDGAR refresh + pre-register filing-veto experiment`.
+- Do NOT push or open a PR.
+
+## Steps
+
+### Step 1: Fix the nightly call site
+
+In `research/premarket_eval.py`, replace the block quoted above:
+
+- Compute the session date in Python (ET): reuse the file's existing imports —
+  `from zoneinfo import ZoneInfo` idiom as in `api/main.py:_eastern_session_date`
+  — and pass it as a parameter:
+  `"SELECT symbol FROM minute_bars WHERE session_date = ? GROUP BY symbol ORDER BY sum(volume) DESC LIMIT 30"`.
+- Split into TWO independent try-blocks (float failing must never skip
+  dilution, and vice versa). Keep each block's failure print.
+
+**Verify**: new unit test (Test plan case 1) passes.
+
+### Step 2: float_cache applies its own DDL
+
+In `research/ingestion/float_cache.py`, execute the module's existing
+`CREATE TABLE IF NOT EXISTS` DDL at the top of `refresh_float_cache` (and of
+its reader if one is public) — same idempotent apply-on-use idiom as
+`research/regime.py:apply_schema`. No schema content change.
+
+**Verify**: unit test creates a fresh `EventStore(":memory:")`, calls
+`refresh_float_cache(con, [], api_key="x")` (empty symbols → no HTTP) → no
+exception, `SELECT count(*) FROM float_cache` → 0 rows, table exists.
+
+### Step 3: `iv_dilution_gate` intervention + preload
+
+In `research/experiments/replay.py`, additive:
+
+- `_load_dilution_flags(con, days=90) -> set[tuple[str, date]]` — one query:
+  for every `dilution_events` row, emit `(symbol, session_date)` pairs the
+  flag covers… **do NOT materialize 90 days × filings**; instead load
+  `{symbol: [announced_date, ...]}` once and provide
+  `def _dilution_flagged(loaded, symbol, session_date, days) -> bool`
+  (any filing with `0 <= session_date - announced_date <= days`).
+- `iv_dilution_gate(fire, baseline, flags, params)` → keep-or-zero mirroring
+  `iv_regime_gate`: flagged → `variant_r = 0.0`, detail
+  `{"kind": "dilution_gate", "flagged": True, "lookback_days": N}`; not
+  flagged (or symbol absent from dilution_events — the no-data case) → KEEP
+  baseline, `{"flagged": False}` (fail-open: an unfetched symbol must not
+  count as vetoed).
+- Wire into `_compute_fire`'s dispatch and the run-loop's once-per-run preload
+  exactly where `regime_gate` does both (replay.py:~503 and ~729).
+  `params` carries `{"lookback_days": 90}` from the experiment JSON.
+
+**Verify**: new intervention tests (Test plan cases 2-3) pass;
+`python3 -m pytest tests/unit/experiments/ -q` all green.
+
+### Step 4: register `dilution_veto_v1`
+
+Create `experiments/dilution_veto_v1.json`:
+
+```json
+{
+  "id": "dilution_veto_v1",
+  "hypothesis": "Zeroing ORB fires on names with an S-1/S-3/424B filing in the trailing 90 days raises whole-population expR — the dilution-trap gaps fade as new shares hit.",
+  "population": {"min_entry_gap_pct": 0.10, "min_day_volume": 300000, "price_min": 2.0, "price_max": 20.0},
+  "intervention": {"kind": "dilution_gate", "lookback_days": 90},
+  "bars": {"expR_delta_min": 0.05, "min_n": 300},
+  "fidelity": {"atr_frac": 0.25, "spread_mult": 3, "adverse_ticks": 1}
+}
+```
+
+Declare it wherever the regime experiments are declared in
+`research/experiments/registry.py` (the self-registering cohort) so the
+nightly `experiments run` picks it up with zero operator action.
+
+**Verify**: `python3 -m pytest tests/unit/experiments/ -q` green (registry
+tests, if any, still pass); a unit test loads the JSON and asserts the
+registry declares it.
+
+### Step 5: honesty note in the JSON's hypothesis vicinity + adjudication expectations
+
+Nothing extra to build: `experiments adjudicate` already handles the verdict.
+Add one comment where `iv_dilution_gate` is defined: with dilution_events
+seeded only from ~now forward (plus the fetcher's 180-day filing lookback per
+symbol), early adjudications run with PARTIAL flag coverage — fires on
+never-fetched symbols read unflagged (fail-open), so early expR_delta is
+biased TOWARD zero, never inflated. The experiment can only understate the
+veto's value until coverage accrues — acceptable, self-correcting, worth the
+comment so nobody reads an early INCONCLUSIVE as refutation.
+
+**Verify**: comment present (code review).
+
+## Test plan
+
+1. `tests/unit/research/test_premarket_eval_enrichment.py` (new, small):
+   monkeypatch `refresh_float_cache` to RAISE and `refresh_dilution_events`
+   to record its call → run the extracted enrichment block (refactor it into
+   a small function `_refresh_shadow_enrichment(con, session_date)` in
+   premarket_eval so it's testable without the whole main()) → dilution still
+   called with the explicit ET date, symbols from a seeded minute_bars store.
+2. `tests/unit/experiments/test_dilution_gate.py` (new; model on
+   `test_regime_gate.py`): seeded `dilution_events` row for SYM on D-10 with
+   `lookback_days=90` → fire on SYM@D zeroed (`variant_r == 0.0`,
+   `flagged: True`); fire on OTHER@D kept (baseline passthrough,
+   `flagged: False`); filing OLDER than lookback → kept.
+3. End-to-end registry: after `run` self-registration, `dilution_veto_v1`
+   exists in the experiments table and a replay over seeded fires writes
+   `experiment_runs` rows with `detail.kind == "dilution_gate"`.
+4. float_cache DDL test (Step 2's verify).
+
+**Verify**: `python3 -m pytest tests/unit/experiments/ tests/unit/research/ -q`
+→ all pass.
+
+## Done criteria
+
+- [ ] Nightly enrichment block uses an explicit ET session date and isolated
+      try-blocks (grep: `current_date` no longer appears in
+      `research/premarket_eval.py`)
+- [ ] `float_cache` DDL applied on use; empty-symbol call is a no-op, not a crash
+- [ ] `iv_dilution_gate` additive in replay.py, fail-open on missing data,
+      preloaded once per run; `experiments/dilution_veto_v1.json` registered
+      in the self-declaring cohort
+- [ ] All new tests pass; `tests/unit/experiments/` fully green; full suite
+      passes; `ruff check --select F api/ storage/ research/` exit 0
+- [ ] `git status` clean outside scope
+
+## STOP conditions
+
+- The registry's self-registration mechanism doesn't match the description
+  (can't find where regime experiments are declared) — report, don't invent a
+  second registration path.
+- Wiring `iv_dilution_gate` requires modifying `iv_regime_gate` or any
+  existing intervention — everything here must be additive.
+- The premarket_eval refactor (extracting `_refresh_shadow_enrichment`) pulls
+  in more than the float/dilution block — the function must be a pure lift of
+  those ~12 lines plus the date fix.
+- Any temptation to touch the LIVE veto path in trading_execution.py.
+
+## Maintenance notes
+
+- SUPERVISED FOLLOW-UP (reviewer/owner, NOT the executor — it must not touch
+  prod): seed the backfill once —
+  `python3 -c "...refresh_dilution_events(con, <last-30-sessions' top symbols>)"`
+  against production, so the experiment starts with coverage instead of
+  waiting weeks. Then watch `experiments adjudicate` output nightly.
+- Promotion path if `dilution_veto_v1` PASSES: extend the existing gated
+  `_dilution_vetoed` (trading_execution.py) to consult `recent_dilution` —
+  that is a NEW plan with its own live-path review, never a drive-by.
+- The 180-day filing lookback (fetcher) vs 90-day veto lookback (experiment)
+  are deliberately different: fetch wide, test the pre-registered window.
+- Deferred, explicitly: LLM classification of 8-K bodies / ATM language —
+  only worth it if the deterministic form-type veto shows signal first.

plans/README.md

@@ -32,6 +32,7 @@ your row when done.
 | 013 | Symbol drill-in: 24h candlestick + live buy/sell trade markers (TradingView Lightweight Charts v5.2, vendored) | P2 | M | 012 (soft — reuses its bracket-leg maps) | **DONE (2026-07-17, review-APPROVED after 1 revision round; commits b4ad3d3+709e7a4 on `worktree-agent-ae2ee0000d5e3a7eb` — awaiting owner merge). Headless-browser render-verified. Executor surfaced 2 pre-existing out-of-scope defects: OrderFilledEvent has no `side` field (worked around via order_submitted sibling join in /api/chart) and GET /api/attribution 500s when DATABASE_URL unset (research/attribution.py hardcodes os.environ lookup).** |
 | 014 | Live decision trace — gate-receipt feed (value→threshold per gate) + chart reject markers | P2 | M | 013 (chart overlay half; feed panel standalone) | **DONE (2026-07-17, review-APPROVED first pass, zero revisions; commit dd81dbb on `worktree-agent-acb00535650deac1a` — awaiting owner merge). Decision Trace panel (lazy-load + SSE-throttled), /api/decisions (clamped 500, case-insensitive, server-side epoch), collapsed reject markers merged into the chart's single markers primitive.** |
 | 015 | Backtest replay dashboard (/backtest) — theoretical entries/exits over past sessions under CURRENT settings, via Experiment-OS replay | P2 | L | 013 (chart component) | **DONE (2026-07-17, review-APPROVED first pass; commit 8ac222a on `worktree-agent-ae4d3f36e18c6e7a5` — awaiting owner merge). replay_window = assembly-only over harness primitives (same _entry_index/adverse-fill/R math); /api/backtest 45-session clamp + 10-min cache behind a dedicated lock; /backtest page with THEORETICAL banner + X-of-Y coverage line + per-trade drill-in (blue entry/purple exit markers). Deviations accepted: /api/chart gained backward-compatible date= anchor (72h cap couldn't reach old sessions); batching idiom taken from regime.py (plan's run() pointer was wrong).** |
+| 016 | EDGAR dilution enrichment — unbreak silent nightly refresh (UTC current_date bug + missing float_cache DDL) + pre-register dilution_veto_v1 keep-or-zero experiment | P2 | M | 006 | IN PROGRESS (2026-07-17) |
 
 Status values: TODO | IN PROGRESS | DONE | BLOCKED (with one-line reason) | REJECTED (with one-line rationale)