ingest(009): nightly SPY/QQQ ingest + catalyst enrichment ON as feature source

Step 3 of plan 009 (config + code; no live-path change):

- research/ingestion/market_context.py: pulls SPY/QQQ full-session minute bars
  (SIP, post-close embargo-safe) + daily bars via the existing upsert helpers, so
  the labeler's market-context features have a FORWARD source (index ETFs never
  enter the gapper universe the live loop ingests). Mirrors sip_backfill's shape.
- both compose scheduler blocks: run it after premarket_eval, guarded || echo.
- NEWS_ENRICH_ENABLED on: added to docker-compose x-app-env anchor (loop +
  scheduler inherit it); truenas-compose + .env.example already had it. Catalyst
  is a FEATURE source only, never a live gate; enabled-but-down never blocks
  trading. Enrichment OUTPUT is verified by the orchestrator (needs live Ollama).

Compose statically validated (yaml.safe_load). Ingest self-checked via a
fake-client :memory: round-trip through _preload_spy.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
momentum devandClaude Fable 5 committed 2026-07-14 20:27:49 -04:00
1 parent 51f095a0c3
commit 5b885a719f
4 files changed
+124

No files matched your search

@@ -0,0 +1,40 @@
"""SPY/QQQ market-context ingest (plan 009): fake-client bars land as minute_bars
and round-trip through the labeler's _preload_spy. Isolated :memory: schema."""
from datetime import date
from research.ingestion.market_context import ingest_index_session
from research.labeler import _preload_spy
from storage.db import get_connection
D = date(2026, 6, 2)
class _FakeClient:
"""Minimal AlpacaPaperClient stand-in for SPY/QQQ minute + daily bars."""
def get_minute_bars(self, symbols, start_iso, end_iso, feed=None):
# two RTH minute bars (13:30/13:31Z = 09:30/09:31 ET) per requested symbol
out = {}
for s in symbols:
out[s] = [
{"t": f"{D.isoformat()}T13:30:00Z", "o": 400, "h": 401, "l": 399, "c": 400.5, "v": 1000, "vw": 400.3},
{"t": f"{D.isoformat()}T13:31:00Z", "o": 400.5, "h": 402, "l": 400, "c": 401.5, "v": 1200, "vw": 401.0},
]
return out
def get_daily_bars(self, symbols, start_iso, feed=None):
return {s: [{"t": f"{D.isoformat()}T00:00:00Z", "o": 399, "h": 403, "l": 398, "c": 401.5, "v": 5_000_000, "vw": 400.8}]
for s in symbols}
def test_index_ingest_lands_and_preloads():
con = get_connection(":memory:")
try:
m, dr = ingest_index_session(con, _FakeClient(), D)
assert m == 4 and dr == 2 # 2 symbols x 2 minute bars, x1 daily each
spy = _preload_spy(con) # the labeler's market-context source
assert D in spy and len(spy[D]) == 2
assert set(spy[D].columns) >= {"timestamp", "open", "high", "low", "close"}
finally:
con.close()