momentum-system/plans/016-edgar-dilution-enrichment.md

Plan 016: EDGAR dilution enrichment — unbreak the silent shadow pipeline, seed it, and pre-register the filing-veto experiment

Executor instructions: Follow this plan step by step. Run every verification command and confirm the expected result before moving to the next step. If anything in the "STOP conditions" section occurs, stop and report — do not improvise. When done, update the status row for this plan in plans/README.md — unless a reviewer dispatched you and told you they maintain the index.

Drift check (run first): git diff --stat 92488b7..HEAD -- research/premarket_eval.py research/ingestion/sec_dilution.py research/ingestion/float_cache.py research/experiments/ experiments/ tests/unit/ On mismatch with the "Current state" excerpts, STOP.

Status

• Priority: P2
• Effort: M
• Risk: LOW-MED (nightly-job + shadow-experiment code; ZERO live entry-path change)
• Depends on: none (Experiment OS = plan 006, long DONE)
• Category: research-infrastructure / data-quality
• Planned at: commit 92488b7, 2026-07-17

Why this matters

A fresh S-1/S-3 shelf or 424B offering on a runner is the classic dilution trap the source trader warns about. The repo ALREADY has an EDGAR filings fetcher (research/ingestion/sec_dilution.py) wired into the nightly job — but dilution_events has 0 rows in production, ever, because the call site is silently broken two independent ways (verified 2026-07-17):

1. The symbol query uses session_date = current_date, and production Postgres runs in Etc/UTC — at 20:00 EDT (= 00:00 UTC next day) current_date is tomorrow, so the query returns zero symbols every night.
2. refresh_float_cache runs first in the SAME try-block, and the float_cache table does not exist in production (its CREATE TABLE IF NOT EXISTS DDL sits in the module but is never executed) — the UndefinedTable exception skips the dilution refresh too.

The fetcher itself works (live-tested 2026-07-17: ATPC → 2 filings, EHGO → 2, false-positive guard via display_names correct). This plan: fix the call site, apply the missing DDL, seed a backfill, and — the payoff — register a pre-registered keep-or-zero experiment (dilution_veto_v1) that measures on logged fires whether skipping filing-flagged names raises expR. No LLM needed: the filing TYPE is deterministic. The live veto stays OFF until the experiment passes its bar (owner-gated promotion, out of scope here).

Current state

• research/ingestion/sec_dilution.py (87 lines) — WORKING fetcher:

  • fetch_filings(symbol) → EDGAR full-text search, forms S-1,S-3,424B5,424B4,424B3, 180-day lookback, display_names contains (SYMBOL) guard, USER_AGENT = "momentum-research fez5587@gmail.com", 0.4 s pause between symbols.
  • refresh_dilution_events(con, symbols) → idempotent insert into dilution_events (id = accession number).
  • recent_dilution(con, symbol, days=90) → filing count reader.

• dilution_events DDL exists in storage/pg_schema.sql:131 (id, symbol, event_type, announced_at, …). Table exists in prod; 0 rows.

• Broken call site: research/premarket_eval.py:122-133:

      try:
          from research.ingestion.float_cache import refresh_float_cache
          from research.ingestion.sec_dilution import refresh_dilution_events
          syms = [r[0] for r in con.execute(
              "SELECT symbol FROM minute_bars WHERE session_date = current_date "
              "GROUP BY symbol ORDER BY sum(volume) DESC LIMIT 30").fetchall()]
          if syms:
              print("float cache:", refresh_float_cache(con, syms))
              print("dilution filings:", refresh_dilution_events(con, syms))
      except Exception as exc:  # noqa: BLE001 — enrichment never breaks the eval
          print(f"float/dilution enrichment skipped: {exc}")
  

• ET-session-date convention: the codebase never trusts DB-server local time; it computes the US/Eastern date in Python — e.g. api/main.py:89 _eastern_session_date() → datetime.now(ZoneInfo("America/New_York")).date(). research/labeler.py and research/regime.py pass explicit dates the same way. Match this: compute the ET date in Python, pass it as a ? parameter.

• research/ingestion/float_cache.py:27 — CREATE TABLE IF NOT EXISTS float_cache (...) DDL string exists in the module; nothing executes it. FMP key env: FMP_API_KEY (absent in prod .env → refresh returns an error-count dict; that's fine — it must just not THROW before dilution runs).

• Experiment OS pattern to copy (keep-or-zero on fires):

  • research/experiments/replay.py:429 iv_regime_gate(fire, baseline, favorable, model_version) → variant_r = baseline if favorable else 0.0; no-data = KEEP (fail-open); detail dict records the decision.
  • Dispatch: _compute_fire at replay.py:~503 (if kind == "regime_gate":) and the run-loop prefetch at ~729 (_load_regime_favorable loaded ONCE per run, not per fire). A dilution gate needs the same shape: preload {(symbol, session_date_window) …} once.
  • Experiment JSONs live in experiments/*.json — copy experiments/regime_meta_v1.json's shape: population {min_entry_gap_pct: 0.10, min_day_volume: 300000, price_min: 2.0, price_max: 20.0}, bars {expR_delta_min: 0.05, min_n: 300}, fidelity {atr_frac: 0.25, spread_mult: 3, adverse_ticks: 1}.
  • Registration: research.experiments run self-registers the declared cohort (16f880d) — check research/experiments/registry.py for where the declared experiment list lives and add the new JSON there.

• Nightly ordering (docker-compose.yml:167-183 scheduler block): premarket_eval (which refreshes dilution_events) runs BEFORE research.experiments run — so the experiment replays over fresh filings the same night. No compose change needed.

• Tests: tests/unit/experiments/ — existing regime-gate experiment tests show the store/seeding fixtures (EventStore(":memory:")); model the new intervention tests on tests/unit/experiments/test_regime_gate.py.

Commands you will need

Purpose	Command	Expected on success
Experiments tests	python3 -m pytest tests/unit/experiments/ -q	all pass (35 existing + new)
Full suite	python3 -m pytest tests/ -q --ignore=tests/unit/api	all pass
API tests (unaffected)	python3 -m pytest tests/unit/api/ -q	15 pre-existing urllib failures only
Lint (CI gate)	ruff check --select F api/ storage/ research/	exit 0

Scope

In scope:

• research/premarket_eval.py (ONLY the float/dilution block, lines ~122-133)
• research/ingestion/float_cache.py (apply its own DDL idempotently)
• research/ingestion/sec_dilution.py (if a helper is needed; fetcher logic unchanged)
• research/experiments/replay.py (new iv_dilution_gate + its preload — additive only, existing functions untouched)
• research/experiments/registry.py (declare the new experiment, matching how the regime experiments are declared)
• experiments/dilution_veto_v1.json (create)
• tests/unit/experiments/, tests/unit/research/ (new tests)

Out of scope — do NOT touch:

• trading_execution.py, run_live_paper.py, alpaca_paper/ — the LIVE dilution veto (catalyst_veto_enabled, headline-based) stays exactly as is; promotion of a filing-based veto is an OWNER decision after the experiment passes, not this plan.
• simulate_exit / fidelity math / population logic.
• Any LLM/Ollama call — filings are classified by FORM TYPE, deterministically.
• compose files (ordering already correct).

Git workflow

• Match repo style; suggested: dilution(016): unbreak nightly EDGAR refresh + pre-register filing-veto experiment.
• Do NOT push or open a PR.

Steps

Step 1: Fix the nightly call site

In research/premarket_eval.py, replace the block quoted above:

• Compute the session date in Python (ET): reuse the file's existing imports — from zoneinfo import ZoneInfo idiom as in api/main.py:_eastern_session_date — and pass it as a parameter: "SELECT symbol FROM minute_bars WHERE session_date = ? GROUP BY symbol ORDER BY sum(volume) DESC LIMIT 30".
• Split into TWO independent try-blocks (float failing must never skip dilution, and vice versa). Keep each block's failure print.

Verify: new unit test (Test plan case 1) passes.

Step 2: float_cache applies its own DDL

In research/ingestion/float_cache.py, execute the module's existing CREATE TABLE IF NOT EXISTS DDL at the top of refresh_float_cache (and of its reader if one is public) — same idempotent apply-on-use idiom as research/regime.py:apply_schema. No schema content change.

Verify: unit test creates a fresh EventStore(":memory:"), calls refresh_float_cache(con, [], api_key="x") (empty symbols → no HTTP) → no exception, SELECT count(*) FROM float_cache → 0 rows, table exists.

Step 3: iv_dilution_gate intervention + preload

In research/experiments/replay.py, additive:

• _load_dilution_flags(con, days=90) -> set[tuple[str, date]] — one query: for every dilution_events row, emit (symbol, session_date) pairs the flag covers… do NOT materialize 90 days × filings; instead load {symbol: [announced_date, ...]} once and provide def _dilution_flagged(loaded, symbol, session_date, days) -> bool (any filing with 0 <= session_date - announced_date <= days).
• iv_dilution_gate(fire, baseline, flags, params) → keep-or-zero mirroring iv_regime_gate: flagged → variant_r = 0.0, detail {"kind": "dilution_gate", "flagged": True, "lookback_days": N}; not flagged (or symbol absent from dilution_events — the no-data case) → KEEP baseline, {"flagged": False} (fail-open: an unfetched symbol must not count as vetoed).
• Wire into _compute_fire's dispatch and the run-loop's once-per-run preload exactly where regime_gate does both (replay.py:~503 and ~729). params carries {"lookback_days": 90} from the experiment JSON.

Verify: new intervention tests (Test plan cases 2-3) pass; python3 -m pytest tests/unit/experiments/ -q all green.

Step 4: register dilution_veto_v1

Create experiments/dilution_veto_v1.json:

{
  "id": "dilution_veto_v1",
  "hypothesis": "Zeroing ORB fires on names with an S-1/S-3/424B filing in the trailing 90 days raises whole-population expR — the dilution-trap gaps fade as new shares hit.",
  "population": {"min_entry_gap_pct": 0.10, "min_day_volume": 300000, "price_min": 2.0, "price_max": 20.0},
  "intervention": {"kind": "dilution_gate", "lookback_days": 90},
  "bars": {"expR_delta_min": 0.05, "min_n": 300},
  "fidelity": {"atr_frac": 0.25, "spread_mult": 3, "adverse_ticks": 1}
}

Declare it wherever the regime experiments are declared in research/experiments/registry.py (the self-registering cohort) so the nightly experiments run picks it up with zero operator action.

Verify: python3 -m pytest tests/unit/experiments/ -q green (registry tests, if any, still pass); a unit test loads the JSON and asserts the registry declares it.

Step 5: honesty note in the JSON's hypothesis vicinity + adjudication expectations

Nothing extra to build: experiments adjudicate already handles the verdict. Add one comment where iv_dilution_gate is defined: with dilution_events seeded only from ~now forward (plus the fetcher's 180-day filing lookback per symbol), early adjudications run with PARTIAL flag coverage — fires on never-fetched symbols read unflagged (fail-open), so early expR_delta is biased TOWARD zero, never inflated. The experiment can only understate the veto's value until coverage accrues — acceptable, self-correcting, worth the comment so nobody reads an early INCONCLUSIVE as refutation.

Verify: comment present (code review).

Test plan

1. tests/unit/research/test_premarket_eval_enrichment.py (new, small): monkeypatch refresh_float_cache to RAISE and refresh_dilution_events to record its call → run the extracted enrichment block (refactor it into a small function _refresh_shadow_enrichment(con, session_date) in premarket_eval so it's testable without the whole main()) → dilution still called with the explicit ET date, symbols from a seeded minute_bars store.
2. tests/unit/experiments/test_dilution_gate.py (new; model on test_regime_gate.py): seeded dilution_events row for SYM on D-10 with lookback_days=90 → fire on SYM@D zeroed (variant_r == 0.0, flagged: True); fire on OTHER@D kept (baseline passthrough, flagged: False); filing OLDER than lookback → kept.
3. End-to-end registry: after run self-registration, dilution_veto_v1 exists in the experiments table and a replay over seeded fires writes experiment_runs rows with detail.kind == "dilution_gate".
4. float_cache DDL test (Step 2's verify).

Verify: python3 -m pytest tests/unit/experiments/ tests/unit/research/ -q → all pass.

Done criteria

• Nightly enrichment block uses an explicit ET session date and isolated try-blocks (grep: current_date no longer appears in research/premarket_eval.py)
• float_cache DDL applied on use; empty-symbol call is a no-op, not a crash
• iv_dilution_gate additive in replay.py, fail-open on missing data, preloaded once per run; experiments/dilution_veto_v1.json registered in the self-declaring cohort
• All new tests pass; tests/unit/experiments/ fully green; full suite passes; ruff check --select F api/ storage/ research/ exit 0
• git status clean outside scope

STOP conditions

• The registry's self-registration mechanism doesn't match the description (can't find where regime experiments are declared) — report, don't invent a second registration path.
• Wiring iv_dilution_gate requires modifying iv_regime_gate or any existing intervention — everything here must be additive.
• The premarket_eval refactor (extracting _refresh_shadow_enrichment) pulls in more than the float/dilution block — the function must be a pure lift of those ~12 lines plus the date fix.
• Any temptation to touch the LIVE veto path in trading_execution.py.

Maintenance notes

• SUPERVISED FOLLOW-UP (reviewer/owner, NOT the executor — it must not touch prod): seed the backfill once — python3 -c "...refresh_dilution_events(con, <last-30-sessions' top symbols>)" against production, so the experiment starts with coverage instead of waiting weeks. Then watch experiments adjudicate output nightly.
• Promotion path if dilution_veto_v1 PASSES: extend the existing gated _dilution_vetoed (trading_execution.py) to consult recent_dilution — that is a NEW plan with its own live-path review, never a drive-by.
• The 180-day filing lookback (fetcher) vs 90-day veto lookback (experiment) are deliberately different: fetch wide, test the pre-registered window.
• Deferred, explicitly: LLM classification of 8-K bodies / ATM language — only worth it if the deterministic form-type veto shows signal first.