One playlist per add-playlist CLI call made adding a second playlist a chore.
Now YT_PLAYLIST_URLS takes a comma-separated list of playlist URLs (list=
param extracted) or bare ids; the transcripts service seeds the playlists
table from it at boot (insert-if-missing — hand-set labels are never
clobbered), and `python -m youtube_claims.run sync-playlists` does the same
on demand. Documented in both compose files; 3 tests.
Also: queue the two executor-surfaced defects (OrderFilledEvent side,
/api/attribution env hardcode) + the flaky nightly regime build-features
step in the plans index; regime_features 07-16 backfilled supervised.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New admin service — LAN-only, token-authed (ADMIN_TOKEN), card-based UI
matching the owner's preferred dashboard style. Capabilities:
- health: docker containers via /var/run/docker.sock (stdlib unix-socket
HTTP client), DB heartbeat, Alpaca account/clock, Ollama, feed
- config: whitelisted strategy knobs (TRADING_/TRIGGER_/WATCHER_/...)
edited via data/overrides.env, which every entrypoint now loads with
override=True at boot (runtime/env_overrides.py) — so changes apply on
container restart without touching the deployment YAML; secrets are
presence-only and rejected server-side if edited
- profiles: named knob presets (seeded: validated-core, conservative,
squeeze-hunter) applied in one click
- restart buttons per container; paper/live switch guarded behind
ALPACA_LIVE_* keys existing
Verified live in docker: auth 401s, status/config/profiles/write paths,
secret-edit rejection, page serves. Added to both compose files.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two data-quality fixes surfaced by inspecting the first 94 claims:
- _clean_ticker: coerce the literal 'NULL'/'NONE'/'$AAPL' strings the LLM sometimes returns to
a real None, and collapse ASR space-mangling ('B VC' -> 'BVC').
- require_asset filter: drop claims with neither a ticker nor an asset_name -- generic trading
commentary ('buy the dip') from tutorial content is not an asset claim. On the backlog this
cut 94 -> 58 claims (36 generic), leaving clean per-asset signal.
Added the $2,000 Small Account Challenge recap playlist (the daily per-asset trade breakdowns) --
far higher claim density than the evergreen 'Anatomy of a Trade' tutorials. 7 tests green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The systemd user service runs with a minimal PATH that excludes /usr/bin, so yt-dlp's audio
postprocessor reported 'ffprobe and ffmpeg not found' and every video silently fell back to
captions. Pass ffmpeg_location=config.FFMPEG_LOCATION (shutil.which with /usr/bin fallback) to
yt-dlp so the Whisper path works under systemd. Verified: yt-dlp download + ffmpeg extract now
produces a wav. Reset the 4 caption-fallback videos to pending so the backlog gets Whisper.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Extraction fix: the anti-hallucination check (a claim's verbatim_quote must appear in the
source window) was too brittle against auto-captions — fragmented segments joined with
newlines meant a valid quote spanning a join was dropped (0 claims on a 623-segment captions
transcript). Now whitespace-normalized matching (_norm + _quote_supported, ~40-char prefix):
the SAME transcript yields 6 real claims (WETG +160% reverse-split+news; 4 AURC trade claims
with prices; a rejection concern), each quote verifiably present. Whisper output is cleaner
still; this just makes the captions fallback usable too.
- tests/unit/test_youtube_extraction.py: 5 pure-helper tests (norm, quote-support across joins,
hallucination rejection, verbatim-quote-mandatory drop, chunk time-spans).
- youtube_claims/momentum-transcripts.service: reboot-surviving systemd user unit (Restart=always,
loads secrets from .env; Whisper option A/CPU, env-flippable to B/GPU).
END-TO-END VERIFIED (captions path, no ffmpeg): detection -> claim -> transcribe -> extract ->
6 claim rows in Postgres. Full Whisper path needs `sudo apt install ffmpeg`, then
`systemctl --user enable --now momentum-transcripts` to process the 11-video backlog at
Whisper quality + poll for new uploads.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A descriptive ingestion layer feeding the downstream reasoning model: monitors YouTube
playlists, transcribes new uploads, and extracts structured, verifiable CLAIM records
(with provenance + a mandatory verbatim quote). It records WHAT was claimed; it never
judges correctness or tradeability -- all judgment stays downstream, in one place.
Built per the operator's spec. Two clocks coordinated through Postgres (§2): a 30-min
detection poll + a paced concurrency-1 transcription/extraction worker. Reuses the app's
infra: shared Postgres (isolated `transcripts` schema), the app's Ollama for extraction
(free, on-LAN, private -- transcripts never leave the network), the config.py host resolver.
youtube_claims/schema.sql playlists / videos / claims (§7-8); claims.verbatim_quote NOT NULL
youtube_claims/db.py psycopg2; claim-by-atomic-status-flip under FOR UPDATE SKIP LOCKED
youtube_claims/detection.py playlistItems.list over urllib; dedupe by video_id (§5)
youtube_claims/extraction.py local Ollama -> descriptive claims; drops any claim lacking an
exact verbatim quote present in the source window (anti-hallucination)
youtube_claims/transcription.py yt-dlp -> faster-whisper (medium/cpu/int8, option A;
initial_prompt=watchlist) with youtube-transcript-api captions fallback
youtube_claims/worker.py paced drain; per-video state machine + retries/backoff
youtube_claims/run.py APScheduler poll + worker loop in one process; CLI for setup
VERIFIED LIVE: schema applied; playlist "Anatomy of a Trade" (Ross Cameron) added; poll
ingested all 12 videos (1 API unit); Ollama extraction pulled clean descriptive claims
(TC/DXST/CF) with exact verbatim quotes from a real transcript. Whisper transcription pending
ffmpeg (`apt install ffmpeg`, needs sudo) + the yt-dlp/faster-whisper deps (installing via uv).
Secrets (YOUTUBE_API_KEY) live in gitignored .env only. Whisper is option A (CPU); env-flippable
to option B (LAN GPU large-v3) later.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
