ingest(009): nightly SPY/QQQ ingest + catalyst enrichment ON as feature source
Step 3 of plan 009 (config + code; no live-path change): - research/ingestion/market_context.py: pulls SPY/QQQ full-session minute bars (SIP, post-close embargo-safe) + daily bars via the existing upsert helpers, so the labeler's market-context features have a FORWARD source (index ETFs never enter the gapper universe the live loop ingests). Mirrors sip_backfill's shape. - both compose scheduler blocks: run it after premarket_eval, guarded || echo. - NEWS_ENRICH_ENABLED on: added to docker-compose x-app-env anchor (loop + scheduler inherit it); truenas-compose + .env.example already had it. Catalyst is a FEATURE source only, never a live gate; enabled-but-down never blocks trading. Enrichment OUTPUT is verified by the orchestrator (needs live Ollama). Compose statically validated (yaml.safe_load). Ingest self-checked via a fake-client :memory: round-trip through _preload_spy. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
1 parent
51f095a0c3
commit
5b885a719f
4 files changed
+124
No files matched your search
@@ -0,0 +1,40 @@
|
||||
"""SPY/QQQ market-context ingest (plan 009): fake-client bars land as minute_bars
|
||||
and round-trip through the labeler's _preload_spy. Isolated :memory: schema."""
|
||||
|
||||
from datetime import date
|
||||
|
||||
from research.ingestion.market_context import ingest_index_session
|
||||
from research.labeler import _preload_spy
|
||||
from storage.db import get_connection
|
||||
|
||||
D = date(2026, 6, 2)
|
||||
|
||||
|
||||
class _FakeClient:
|
||||
"""Minimal AlpacaPaperClient stand-in for SPY/QQQ minute + daily bars."""
|
||||
|
||||
def get_minute_bars(self, symbols, start_iso, end_iso, feed=None):
|
||||
# two RTH minute bars (13:30/13:31Z = 09:30/09:31 ET) per requested symbol
|
||||
out = {}
|
||||
for s in symbols:
|
||||
out[s] = [
|
||||
{"t": f"{D.isoformat()}T13:30:00Z", "o": 400, "h": 401, "l": 399, "c": 400.5, "v": 1000, "vw": 400.3},
|
||||
{"t": f"{D.isoformat()}T13:31:00Z", "o": 400.5, "h": 402, "l": 400, "c": 401.5, "v": 1200, "vw": 401.0},
|
||||
]
|
||||
return out
|
||||
|
||||
def get_daily_bars(self, symbols, start_iso, feed=None):
|
||||
return {s: [{"t": f"{D.isoformat()}T00:00:00Z", "o": 399, "h": 403, "l": 398, "c": 401.5, "v": 5_000_000, "vw": 400.8}]
|
||||
for s in symbols}
|
||||
|
||||
|
||||
def test_index_ingest_lands_and_preloads():
|
||||
con = get_connection(":memory:")
|
||||
try:
|
||||
m, dr = ingest_index_session(con, _FakeClient(), D)
|
||||
assert m == 4 and dr == 2 # 2 symbols x 2 minute bars, x1 daily each
|
||||
spy = _preload_spy(con) # the labeler's market-context source
|
||||
assert D in spy and len(spy[D]) == 2
|
||||
assert set(spy[D].columns) >= {"timestamp", "open", "high", "low", "close"}
|
||||
finally:
|
||||
con.close()
|
||||
Reference in new issue
Block a user