SIP re-backfill: fix the 2-5% IEX volume distortion across the whole dataset
MEASURED 2026-07-02: our entire bar history was IEX-only. IEX sees ~2-5% of consolidated
volume (SNDQ 7/01: IEX 188 bars/1.5M shares vs SIP 391 bars/60.1M — 39x) and misses whole
minutes on thin names. Every volume-based signal (RVOL, liquidity floors, volume bursts,
dollar-volume strata) and several research verdicts ("untradeable dust", "thin premarket")
were computed on that sample. Daily bars had the same defect. Historical SIP is FREE on our
keys (15-min embargo) — only real-time SIP needs the paid plan.
- client.get_daily_bars: feed param (None -> settings.feed; pass "sip" for consolidated).
- research/sip_backfill.py: re-fetch minute sessions (04:00-20:00 ET) + dailies from SIP
via the existing idempotent upserts; embargo-caps the end when refetching today;
refresh_rolling_volume rebuild after (this morning's 20d averages were IEX-scaled too).
CLI: full history < today, or --date D for one session.
- premarket_eval.py: nightly job now SIP-upgrades today's bars BEFORE the pm1 rebuild, so
every finished session lands on true volume automatically (live loop stays IEX real-time
by design — free-key SIP is embargoed 15 min).
Chain running: full backfill -> verification -> rebuild liquid/premarket/runner shadow
tracks + re-validate (their volume gates were IEX-calibrated; verdicts may change).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
1 parent
97b318dd6e
commit
f8f85b3963
3 files changed
+119
-3
No files matched your search
@@ -397,9 +397,13 @@ class AlpacaPaperClient:
|
||||
return payload.get("news") or []
|
||||
|
||||
def get_daily_bars(
|
||||
self, symbols: list[str], start_iso: str, end_iso: str | None = None
|
||||
self, symbols: list[str], start_iso: str, end_iso: str | None = None,
|
||||
feed: str | None = None,
|
||||
) -> dict[str, list[dict]]:
|
||||
"""Daily bars, batching symbols (cap-safe) + following pagination."""
|
||||
"""Daily bars, batching symbols (cap-safe) + following pagination.
|
||||
|
||||
`feed`: None -> settings.feed (iex on free keys, ~2-5% of true volume).
|
||||
Pass "sip" for consolidated-tape volume (free keys allow historical SIP)."""
|
||||
all_bars: dict[str, list[dict]] = {}
|
||||
for chunk in self._chunked(symbols, 100):
|
||||
page_token = None
|
||||
@@ -413,7 +417,7 @@ class AlpacaPaperClient:
|
||||
"start": start_iso,
|
||||
"end": end_iso,
|
||||
"limit": 10_000,
|
||||
"feed": self.settings.feed,
|
||||
"feed": feed or self.settings.feed,
|
||||
"adjustment": "raw",
|
||||
"page_token": page_token,
|
||||
},
|
||||
|
||||
Reference in new issue
Block a user