SIP re-backfill: fix the 2-5% IEX volume distortion across the whole dataset

MEASURED 2026-07-02: our entire bar history was IEX-only. IEX sees ~2-5% of consolidated
volume (SNDQ 7/01: IEX 188 bars/1.5M shares vs SIP 391 bars/60.1M — 39x) and misses whole
minutes on thin names. Every volume-based signal (RVOL, liquidity floors, volume bursts,
dollar-volume strata) and several research verdicts ("untradeable dust", "thin premarket")
were computed on that sample. Daily bars had the same defect. Historical SIP is FREE on our
keys (15-min embargo) — only real-time SIP needs the paid plan.

- client.get_daily_bars: feed param (None -> settings.feed; pass "sip" for consolidated).
- research/sip_backfill.py: re-fetch minute sessions (04:00-20:00 ET) + dailies from SIP
  via the existing idempotent upserts; embargo-caps the end when refetching today;
  refresh_rolling_volume rebuild after (this morning's 20d averages were IEX-scaled too).
  CLI: full history < today, or --date D for one session.
- premarket_eval.py: nightly job now SIP-upgrades today's bars BEFORE the pm1 rebuild, so
  every finished session lands on true volume automatically (live loop stays IEX real-time
  by design — free-key SIP is embargoed 15 min).

Chain running: full backfill -> verification -> rebuild liquid/premarket/runner shadow
tracks + re-validate (their volume gates were IEX-calibrated; verdicts may change).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
momentum devandClaude Opus 4.8 committed 2026-07-02 13:56:01 -04:00
1 parent 97b318dd6e
commit f8f85b3963
3 files changed
+119 -3

No files matched your search

+7 -3
View File
@@ -397,9 +397,13 @@ class AlpacaPaperClient:
return payload.get("news") or []
def get_daily_bars(
self, symbols: list[str], start_iso: str, end_iso: str | None = None
self, symbols: list[str], start_iso: str, end_iso: str | None = None,
feed: str | None = None,
) -> dict[str, list[dict]]:
"""Daily bars, batching symbols (cap-safe) + following pagination."""
"""Daily bars, batching symbols (cap-safe) + following pagination.
`feed`: None -> settings.feed (iex on free keys, ~2-5% of true volume).
Pass "sip" for consolidated-tape volume (free keys allow historical SIP)."""
all_bars: dict[str, list[dict]] = {}
for chunk in self._chunked(symbols, 100):
page_token = None
@@ -413,7 +417,7 @@ class AlpacaPaperClient:
"start": start_iso,
"end": end_iso,
"limit": 10_000,
"feed": self.settings.feed,
"feed": feed or self.settings.feed,
"adjustment": "raw",
"page_token": page_token,
},