Why: the pilot in data/ross showed that Ross Cameron's daily recaps, joined to
the SIP tape and to what our bot saw, answer "which of his trades could we have
taken and why didn't we". It was a set of one-off scripts run by hand; the
server-side youtube_claims pipeline has been dead since ~08-24 (YouTube blocks
the TrueNAS IP) and its captions fallback hid that. This makes the pilot a
nightly, local, cumulative loop without a paid LLM API.
youtube_claims/recap/ (python -m youtube_claims.recap.nightly):
- channel: yt-dlp flat listing of @DaytradeWarrior, m4a audio, faster-whisper
small.en on CPU, pilot-format bundles (meta + our scanner list + transcript).
- extract: Ollama structured output (JSON schema) per EXTRACT_SPEC, chunked
with overlap for long transcripts and merged. The verbatim rule is enforced in
code: every quote must be found (token-level, one-word tolerance) and is
snapped to the transcript's words; unquoted items, trades with no surviving
quote, prices never spoken, and passed-on names never said are dropped.
- llm: endpoint pool over RECAP_OLLAMA_HOSTS (round-robin + failover, per-host
model/ctx/think pins) since the owner runs several Ollama boxes.
- tape/join/ourside: ported join.py/ourside.py -- tickers are price-verified on
the Alpaca SIP tape (plus an order-free check for his own named ticker), then
paper (:5433) and live (:5434) event views are attached. Bars cached on disk.
- store + schema.sql: idempotent recap_* tables in the transcripts schema,
upserted per video_id. report: the pilot's sections + "what changed since last
run" against the previous day's metrics JSON.
- evaluate: scores any model/prompt against the 36 gold extractions.
Model choice (36 gold videos, final code path, P/R):
gpt-oss:20b on the workstation GPU, two focused calls, think=low:
trades P .84 R .70; tickers P .82 R .77; entry +-2% P .54 R .50;
passed-on P .66 R .76; temperature 16/31; kind 30/36; ~49 s/video.
tools:latest (4B, server) one call: trades P .81 R .72, tickers P .83 R .85,
passed-on P .80 R .75, temperature 20/31, but it invents prices
(entry P .42) and occasionally runs away (5-9 min). Prices drive the tape
verification, so the more precise local model is the default.
Iterations: prompt v2 + spoken-price rescale (699 -> 6.99) + split calls took
gpt-oss trades F1 .74 -> .76-.78 and tickers .78 -> .79-.80; think=medium
blew the token budget. qwen3-30b-a3b on mason only finished 5 videos (host
saturated) -- inconclusive.
Also: youtube_claims/transcription.py::_captions_fallback now picks the
youtube-transcript-api shape by feature detection instead of retrying the
removed get_transcript on any error, and transcribe() reports both the whisper
and captions causes -- the IpBlocked error was being masked as AttributeError.
No trading code touched.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The systemd user service runs with a minimal PATH that excludes /usr/bin, so yt-dlp's audio
postprocessor reported 'ffprobe and ffmpeg not found' and every video silently fell back to
captions. Pass ffmpeg_location=config.FFMPEG_LOCATION (shutil.which with /usr/bin fallback) to
yt-dlp so the Whisper path works under systemd. Verified: yt-dlp download + ffmpeg extract now
produces a wav. Reset the 4 caption-fallback videos to pending so the backlog gets Whisper.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A descriptive ingestion layer feeding the downstream reasoning model: monitors YouTube
playlists, transcribes new uploads, and extracts structured, verifiable CLAIM records
(with provenance + a mandatory verbatim quote). It records WHAT was claimed; it never
judges correctness or tradeability -- all judgment stays downstream, in one place.
Built per the operator's spec. Two clocks coordinated through Postgres (§2): a 30-min
detection poll + a paced concurrency-1 transcription/extraction worker. Reuses the app's
infra: shared Postgres (isolated `transcripts` schema), the app's Ollama for extraction
(free, on-LAN, private -- transcripts never leave the network), the config.py host resolver.
youtube_claims/schema.sql playlists / videos / claims (§7-8); claims.verbatim_quote NOT NULL
youtube_claims/db.py psycopg2; claim-by-atomic-status-flip under FOR UPDATE SKIP LOCKED
youtube_claims/detection.py playlistItems.list over urllib; dedupe by video_id (§5)
youtube_claims/extraction.py local Ollama -> descriptive claims; drops any claim lacking an
exact verbatim quote present in the source window (anti-hallucination)
youtube_claims/transcription.py yt-dlp -> faster-whisper (medium/cpu/int8, option A;
initial_prompt=watchlist) with youtube-transcript-api captions fallback
youtube_claims/worker.py paced drain; per-video state machine + retries/backoff
youtube_claims/run.py APScheduler poll + worker loop in one process; CLI for setup
VERIFIED LIVE: schema applied; playlist "Anatomy of a Trade" (Ross Cameron) added; poll
ingested all 12 videos (1 API unit); Ollama extraction pulled clean descriptive claims
(TC/DXST/CF) with exact verbatim quotes from a real transcript. Whisper transcription pending
ffmpeg (`apt install ffmpeg`, needs sudo) + the yt-dlp/faster-whisper deps (installing via uv).
Secrets (YOUTUBE_API_KEY) live in gitignored .env only. Whisper is option A (CPU); env-flippable
to option B (LAN GPU large-v3) later.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
