Why: the pilot in data/ross showed that Ross Cameron's daily recaps, joined to
the SIP tape and to what our bot saw, answer "which of his trades could we have
taken and why didn't we". It was a set of one-off scripts run by hand; the
server-side youtube_claims pipeline has been dead since ~08-24 (YouTube blocks
the TrueNAS IP) and its captions fallback hid that. This makes the pilot a
nightly, local, cumulative loop without a paid LLM API.
youtube_claims/recap/ (python -m youtube_claims.recap.nightly):
- channel: yt-dlp flat listing of @DaytradeWarrior, m4a audio, faster-whisper
small.en on CPU, pilot-format bundles (meta + our scanner list + transcript).
- extract: Ollama structured output (JSON schema) per EXTRACT_SPEC, chunked
with overlap for long transcripts and merged. The verbatim rule is enforced in
code: every quote must be found (token-level, one-word tolerance) and is
snapped to the transcript's words; unquoted items, trades with no surviving
quote, prices never spoken, and passed-on names never said are dropped.
- llm: endpoint pool over RECAP_OLLAMA_HOSTS (round-robin + failover, per-host
model/ctx/think pins) since the owner runs several Ollama boxes.
- tape/join/ourside: ported join.py/ourside.py -- tickers are price-verified on
the Alpaca SIP tape (plus an order-free check for his own named ticker), then
paper (:5433) and live (:5434) event views are attached. Bars cached on disk.
- store + schema.sql: idempotent recap_* tables in the transcripts schema,
upserted per video_id. report: the pilot's sections + "what changed since last
run" against the previous day's metrics JSON.
- evaluate: scores any model/prompt against the 36 gold extractions.
Model choice (36 gold videos, final code path, P/R):
gpt-oss:20b on the workstation GPU, two focused calls, think=low:
trades P .84 R .70; tickers P .82 R .77; entry +-2% P .54 R .50;
passed-on P .66 R .76; temperature 16/31; kind 30/36; ~49 s/video.
tools:latest (4B, server) one call: trades P .81 R .72, tickers P .83 R .85,
passed-on P .80 R .75, temperature 20/31, but it invents prices
(entry P .42) and occasionally runs away (5-9 min). Prices drive the tape
verification, so the more precise local model is the default.
Iterations: prompt v2 + spoken-price rescale (699 -> 6.99) + split calls took
gpt-oss trades F1 .74 -> .76-.78 and tickers .78 -> .79-.80; think=medium
blew the token budget. qwen3-30b-a3b on mason only finished 5 videos (host
saturated) -- inconclusive.
Also: youtube_claims/transcription.py::_captions_fallback now picks the
youtube-transcript-api shape by feature detection instead of retrying the
removed get_transcript on any error, and transcribe() reports both the whisper
and captions causes -- the IpBlocked error was being masked as AttributeError.
No trading code touched.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
95 lines
3.9 KiB
SQL
95 lines
3.9 KiB
SQL
-- Ross Cameron recap learning loop (youtube_claims.recap). Idempotent; {SCHEMA} is substituted
|
|
-- at apply time (default: transcripts). Every child table is keyed by video_id and rewritten
|
|
-- whole on re-ingest, so re-running a video never duplicates rows.
|
|
|
|
CREATE SCHEMA IF NOT EXISTS {SCHEMA};
|
|
|
|
-- one row per recap video the loop has processed (or tried to)
|
|
CREATE TABLE IF NOT EXISTS {SCHEMA}.recap_videos (
|
|
video_id TEXT PRIMARY KEY,
|
|
upload_date DATE,
|
|
session_date DATE,
|
|
title TEXT,
|
|
duration_s INT,
|
|
video_kind TEXT, -- daily_recap|watchlist|education|challenge_summary|other
|
|
transcript_source TEXT, -- whisper-small|whisper-medium|captions
|
|
extractor TEXT, -- 'gold' (strong-model pilot file) or the Ollama model name
|
|
status TEXT NOT NULL DEFAULT 'pending', -- pending|done|failed|skipped
|
|
last_error TEXT,
|
|
attempts INT NOT NULL DEFAULT 0, -- failed attempts (nightly retries up to a cap)
|
|
run_meta JSONB, -- chunks, seconds, quote-drop counts
|
|
record JSONB, -- the full extraction record (EXTRACT_SPEC)
|
|
first_seen_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
|
processed_at TIMESTAMPTZ,
|
|
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
|
);
|
|
CREATE INDEX IF NOT EXISTS recap_videos_session_idx ON {SCHEMA}.recap_videos(session_date);
|
|
|
|
-- the day as he described it (one per video; several videos can describe one session)
|
|
CREATE TABLE IF NOT EXISTS {SCHEMA}.recap_day_summary (
|
|
video_id TEXT PRIMARY KEY REFERENCES {SCHEMA}.recap_videos(video_id) ON DELETE CASCADE,
|
|
session_date DATE,
|
|
net_pnl NUMERIC,
|
|
green BOOLEAN,
|
|
market_temperature TEXT, -- hot|cold|mixed
|
|
market_quote TEXT,
|
|
max_loss_hit BOOLEAN,
|
|
daily_goal_comment TEXT,
|
|
sizing_comment TEXT
|
|
);
|
|
|
|
-- trades he described, joined to the tape and to what OUR bot saw
|
|
CREATE TABLE IF NOT EXISTS {SCHEMA}.recap_trades (
|
|
video_id TEXT NOT NULL REFERENCES {SCHEMA}.recap_videos(video_id) ON DELETE CASCADE,
|
|
seq INT NOT NULL,
|
|
session_date DATE,
|
|
ticker_heard TEXT,
|
|
ticker_guess TEXT,
|
|
symbol TEXT, -- after price verification (NULL if unresolved)
|
|
resolve TEXT, -- price_verified|price_matched_scanner|ambiguous:..|no_price_match|unverified
|
|
setup TEXT,
|
|
setup_quote TEXT,
|
|
entry_price NUMERIC,
|
|
exit_price NUMERIC,
|
|
stated_high NUMERIC,
|
|
stated_low NUMERIC,
|
|
shares NUMERIC,
|
|
pnl NUMERIC,
|
|
outcome TEXT,
|
|
add_on BOOLEAN,
|
|
entry_time_hint TEXT,
|
|
entry_time_tape TEXT, -- HH:MM ET the tape first traded his entry price
|
|
stock_context JSONB,
|
|
why_entered JSONB,
|
|
tape_reads JSONB,
|
|
why_exited JSONB,
|
|
mistake_or_lesson JSONB,
|
|
our_view JSONB, -- discovery/gates/orders on paper+live, tape stats, float, news
|
|
PRIMARY KEY (video_id, seq)
|
|
);
|
|
CREATE INDEX IF NOT EXISTS recap_trades_symbol_idx ON {SCHEMA}.recap_trades(symbol, session_date);
|
|
|
|
-- stocks he looked at and did NOT trade
|
|
CREATE TABLE IF NOT EXISTS {SCHEMA}.recap_passed_on (
|
|
video_id TEXT NOT NULL REFERENCES {SCHEMA}.recap_videos(video_id) ON DELETE CASCADE,
|
|
seq INT NOT NULL,
|
|
session_date DATE,
|
|
ticker_heard TEXT,
|
|
ticker_guess TEXT,
|
|
symbol TEXT,
|
|
resolve TEXT,
|
|
reason TEXT,
|
|
quote TEXT,
|
|
our_view JSONB,
|
|
PRIMARY KEY (video_id, seq)
|
|
);
|
|
|
|
-- general principles he stated, verbatim-quoted
|
|
CREATE TABLE IF NOT EXISTS {SCHEMA}.recap_rules (
|
|
video_id TEXT NOT NULL REFERENCES {SCHEMA}.recap_videos(video_id) ON DELETE CASCADE,
|
|
seq INT NOT NULL,
|
|
rule TEXT,
|
|
quote TEXT NOT NULL,
|
|
PRIMARY KEY (video_id, seq)
|
|
);
|