diff --git a/docs/research/data_sources_inventory.md b/docs/research/data_sources_inventory.md new file mode 100644 index 0000000..b1d7b64 --- /dev/null +++ b/docs/research/data_sources_inventory.md @@ -0,0 +1,29 @@ +# Data-source key inventory (2026-07-18) + +All keys below are in `.env` (gitignored). Most were provisioned but never +wired into code — the system uses only Alpaca (bars/quotes), FMP (float), +EDGAR/SEC full-text (filings), and RSS+Ollama (news). This catalogs what's +available so a future need pulls the right one instead of assuming absence. + +**Triage — what's worth wiring, and what's redundant.** Do NOT wire all of +these; more sources = more complexity, not more edge (news was already only +weakly useful). Wire a source only when a specific hypothesis needs it. + +| Key | What it is | Verdict | +|---|---|---| +| `DATABENTO_API_KEY` | **Nasdaq TotalView L2 depth (MBP-10)**, pay-as-you-go + free credits | **GAME-CHANGER — unblocks plan 017 Phase 0 at ~$0. Verified live 2026-07-18: covers the real gapper universe, fractions of a cent per fire.** | +| `POLYGON_API_KEY` | aggregated bars/quotes, some depth | Nice-to-have: cross-check / fallback for Alpaca bars; not urgent | +| `SEC_API_KEY` | sec-api.io — STRUCTURED filings (vs raw EDGAR full-text the dilution feed uses now) | Minor upgrade to `research/ingestion/sec_dilution.py` if that feed ever needs form-subtype precision; the raw feed already works | +| `Financial_Modeling_Prep_Api_Key` | FMP — float/fundamentals | **Already wired** (float_cache) | +| `FINNHUB_API_KEY` | news + fundamentals + some float | Redundant with FMP/news pipe | +| `Alpha_Vantage_Api_Key`, `Twelve_Data_Api_Key`, `Marketstack_Api_Key`, `EOD_Historical_Data_Api_Key` | bars/fundamentals/float, free tiers | Redundant — float refuted as a gate; bars come from Alpaca SIP | +| `NewsAPI_Api_Key`, `NewsData.io_Api_Key`, `Marketaux_Api_Key`, `MediaStack_Api_Key` | news feeds, free tiers | SKIP for now — 4 news APIs is complexity, not edge; RSS+Ollama already covers catalyst. Revisit only if a news-LATENCY hypothesis is registered | + +**Bottom line:** one key (Databento) materially advances the project's founding +question (can premarket data give autonomous selection confidence — via the +order book). The rest are redundancy or unproven-need. Discipline over +accumulation: wire on demand, not on availability. + +**Security:** these keys were pasted into a chat session 2026-07-18 — rotate +Databento + Polygon (billing feeds) when convenient. `.env` is gitignored; +never commit this file's real values (it holds none — names only). diff --git a/plans/017-l2-depth-recorder.md b/plans/017-l2-depth-recorder.md index 608f36b..781e3f2 100644 --- a/plans/017-l2-depth-recorder.md +++ b/plans/017-l2-depth-recorder.md @@ -87,7 +87,20 @@ believe only what clears the bar. ## Phased plan (each phase gates the next) -### Phase 0 — FREE evaluation: does any usable depth exist, and does it look predictive at all? ($0) +> **UPDATE 2026-07-18 — Phase 0 is DE-RISKED, cost verified ~$0.** The owner's +> `DATABENTO_API_KEY` (already in `.env`) is live and reaches **XNAS.ITCH +> (Nasdaq TotalView, MBP-10 = 10-level order-book depth)**. Cost-checked (no +> data spent, `metadata.get_cost`): 5 min of AAPL depth = $0.026; and every +> real recent bot gapper tested — EHGO, ATPC, MUD, LASE, AEON — resolves with +> full depth at **$0.0001–0.0066 per fire** (thin books = tiny payloads). The +> full 20-fire Phase-0 sniff is **< $1, inside Databento's free signup +> credits.** The "will we spend to find out" risk is gone for Phase 0 — it is +> now a same-day free experiment. Databento covers the $2–20 Nasdaq-listed +> universe the bot trades; NYSE/ARCA-listed gappers may need `DBEQ.BASIC` +> (consolidated) — check per-symbol in 0b. `pip install databento` (research +> layer only; NOT the loop). + +### Phase 0 — FREE evaluation: does any usable depth exist, and does it look predictive at all? ($0 — DATABENTO KEY IN HAND) Goal: decide whether L2 is worth a paid feed WITHOUT buying one, using free trials and free tiers only. diff --git a/requirements.txt b/requirements.txt index 3c07ee9..96c27f7 100644 --- a/requirements.txt +++ b/requirements.txt @@ -10,3 +10,4 @@ apscheduler>=3.10 youtube-transcript-api>=0.6 rich>=13.0 typer>=0.9 +databento # L2 depth (plan 017, research layer)