Why: the pilot in data/ross showed that Ross Cameron's daily recaps, joined to
the SIP tape and to what our bot saw, answer "which of his trades could we have
taken and why didn't we". It was a set of one-off scripts run by hand; the
server-side youtube_claims pipeline has been dead since ~08-24 (YouTube blocks
the TrueNAS IP) and its captions fallback hid that. This makes the pilot a
nightly, local, cumulative loop without a paid LLM API.
youtube_claims/recap/ (python -m youtube_claims.recap.nightly):
- channel: yt-dlp flat listing of @DaytradeWarrior, m4a audio, faster-whisper
small.en on CPU, pilot-format bundles (meta + our scanner list + transcript).
- extract: Ollama structured output (JSON schema) per EXTRACT_SPEC, chunked
with overlap for long transcripts and merged. The verbatim rule is enforced in
code: every quote must be found (token-level, one-word tolerance) and is
snapped to the transcript's words; unquoted items, trades with no surviving
quote, prices never spoken, and passed-on names never said are dropped.
- llm: endpoint pool over RECAP_OLLAMA_HOSTS (round-robin + failover, per-host
model/ctx/think pins) since the owner runs several Ollama boxes.
- tape/join/ourside: ported join.py/ourside.py -- tickers are price-verified on
the Alpaca SIP tape (plus an order-free check for his own named ticker), then
paper (:5433) and live (:5434) event views are attached. Bars cached on disk.
- store + schema.sql: idempotent recap_* tables in the transcripts schema,
upserted per video_id. report: the pilot's sections + "what changed since last
run" against the previous day's metrics JSON.
- evaluate: scores any model/prompt against the 36 gold extractions.
Model choice (36 gold videos, final code path, P/R):
gpt-oss:20b on the workstation GPU, two focused calls, think=low:
trades P .84 R .70; tickers P .82 R .77; entry +-2% P .54 R .50;
passed-on P .66 R .76; temperature 16/31; kind 30/36; ~49 s/video.
tools:latest (4B, server) one call: trades P .81 R .72, tickers P .83 R .85,
passed-on P .80 R .75, temperature 20/31, but it invents prices
(entry P .42) and occasionally runs away (5-9 min). Prices drive the tape
verification, so the more precise local model is the default.
Iterations: prompt v2 + spoken-price rescale (699 -> 6.99) + split calls took
gpt-oss trades F1 .74 -> .76-.78 and tickers .78 -> .79-.80; think=medium
blew the token budget. qwen3-30b-a3b on mason only finished 5 videos (host
saturated) -- inconclusive.
Also: youtube_claims/transcription.py::_captions_fallback now picks the
youtube-transcript-api shape by feature detection instead of retrying the
removed get_transcript on any error, and transcribe() reports both the whisper
and captions causes -- the IpBlocked error was being masked as AttributeError.
No trading code touched.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
127 lines
7.6 KiB
Python
127 lines
7.6 KiB
Python
"""The extraction instructions (EXTRACT_SPEC.md, adapted for a local model) and the JSON
|
|
schema handed to Ollama's structured-output mode. The spec file itself lives in the
|
|
git-ignored data dir; this is the versioned copy the pipeline actually uses."""
|
|
|
|
from __future__ import annotations
|
|
|
|
SETUPS = ["micro_pullback", "bull_flag", "first_pullback", "breakout_whole_dollar",
|
|
"break_of_premarket_high", "halt_resume", "dip_and_curl", "vwap_break", "abcd",
|
|
"red_to_green", "other"]
|
|
KINDS = ["daily_recap", "watchlist", "education", "challenge_summary", "other"]
|
|
|
|
SYSTEM = """You extract a structured record from ONE transcript of Ross Cameron's (Warrior Trading)
|
|
day-trading video. Record what he SAID, not what you think is right.
|
|
|
|
QUOTES ARE CHECKED BY A PROGRAM. Every `quote`, `setup_quote`, `catalyst_quote` and `market_quote`
|
|
must be copied word-for-word from the TRANSCRIPT (a contiguous run of 6-30 words, ASR noise
|
|
included; do not fix grammar or numbers). Anything whose quote is not found verbatim is deleted.
|
|
If you cannot quote it, leave the item out.
|
|
|
|
The transcript is speech-to-text: tickers and prices are often misheard. Put the ticker AS HEARD in
|
|
`ticker_heard` and your best guess in `ticker_guess`, preferring a symbol from OUR SCANNER LIST when
|
|
it sounds alike and the price fits. Spoken prices: "550" usually means $5.50, "750" means $7.50,
|
|
"a dollar twenty" = 1.20, "four and a half" = 4.50. Prices are verified against the tape later, so
|
|
only give a price he actually said.
|
|
|
|
What goes where:
|
|
- video_kind: daily_recap (he recaps his own trading of a session -- even if he mentions a challenge
|
|
or account milestone), watchlist (mainly stocks to watch for the next session), challenge_summary
|
|
(ONLY when the video summarises a multi-day/multi-week challenge as a whole), education (a lesson
|
|
or story not about one session), other.
|
|
- session_date: the trading day described (usually the upload date; the prior Friday for a weekend
|
|
watchlist; null if no specific day).
|
|
- trades: walk through the transcript IN ORDER and create ONE entry every time he says he bought,
|
|
got in, took a trade, added, or got back in ("I got in", "I bought", "I was in", "I added",
|
|
"I took a trade on", "I jumped in"). Several trades on the same stock are normal -- each re-entry
|
|
or add is its own entry (add_on=true). Trades he took but mentions only briefly still count, and
|
|
watchlist or challenge videos can contain trades too. Do NOT create trades for stocks he only
|
|
watched, showed on a chart, or talks about hypothetically.
|
|
setup: one of micro_pullback, bull_flag, first_pullback, breakout_whole_dollar,
|
|
break_of_premarket_high, halt_resume, dip_and_curl, vwap_break, abcd, red_to_green, other.
|
|
outcome: win / loss / scratch as he describes it, else null. pnl/shares only if stated.
|
|
- passed_on: EVERY other stock he names that he did NOT trade -- top gainers he skipped, stocks on
|
|
his watchlist, names he says he missed or avoided -- with the reason (or what he said about it) and
|
|
a quote naming or describing it. Usually 3-10 per video.
|
|
- ticker_guess: when the heard name is close to a symbol in OUR SCANNER LIST (sound-alike, spelled
|
|
letters, or the price he mentions fits), use that scanner symbol.
|
|
- rules_stated: general principles he states (risk, sizing, stock selection, when to stop,
|
|
statistics about his accuracy or P&L), each with a quote.
|
|
- day.market_temperature: hot (lots of big clean movers, strong momentum), cold (slow, few good
|
|
setups, choppy, tough month), mixed, or null if he gives no read. market_quote: the sentence where
|
|
he characterises the market.
|
|
- day.net_pnl: the day's $ result as stated (negative for red); green = true if the day is positive.
|
|
|
|
Keep paraphrases (reason, rule, lesson, read) short. Skip sponsor segments, course promotion and
|
|
filler. Return ONLY the JSON object."""
|
|
|
|
USER_TMPL = """{part_note}Upload metadata and the day's context follow, then the transcript.
|
|
|
|
{bundle}"""
|
|
|
|
PART_NOTE = ("This is PART {i} of {n} of one transcript (parts overlap slightly). Extract only what "
|
|
"is said in this part; fill `day` fields only if this part states them.\n\n")
|
|
|
|
|
|
def _nstr():
|
|
return {"type": ["string", "null"]}
|
|
|
|
|
|
def _nnum():
|
|
return {"type": ["number", "null"]}
|
|
|
|
|
|
def _quoted(key: str) -> dict:
|
|
return {"type": "array", "items": {"type": "object", "properties": {
|
|
key: {"type": "string"}, "quote": {"type": "string"}}, "required": [key, "quote"]}}
|
|
|
|
|
|
SCHEMA = {
|
|
"type": "object",
|
|
"properties": {
|
|
"video_kind": {"type": "string", "enum": KINDS},
|
|
"session_date": _nstr(),
|
|
"day": {"type": "object", "properties": {
|
|
"net_pnl": _nnum(), "green": {"type": ["boolean", "null"]},
|
|
"market_temperature": {"type": ["string", "null"], "enum": ["hot", "cold", "mixed", None]},
|
|
"market_quote": _nstr(), "max_loss_hit": {"type": ["boolean", "null"]},
|
|
"daily_goal_comment": _nstr(), "sizing_comment": _nstr()},
|
|
"required": ["net_pnl", "green", "market_temperature", "market_quote"]},
|
|
"trades": {"type": "array", "items": {"type": "object", "properties": {
|
|
"ticker_heard": {"type": "string"}, "ticker_guess": {"type": "string"},
|
|
"entry_time_hint": _nstr(),
|
|
"setup": {"type": "string", "enum": SETUPS}, "setup_quote": {"type": "string"},
|
|
"entry_price": _nnum(), "exit_price": _nnum(), "stated_high": _nnum(), "stated_low": _nnum(),
|
|
"shares": _nnum(), "pnl": _nnum(),
|
|
"outcome": {"type": ["string", "null"], "enum": ["win", "loss", "scratch", None]},
|
|
"add_on": {"type": "boolean"},
|
|
"stock_context": {"type": "object", "properties": {
|
|
"catalyst": _nstr(), "catalyst_quote": _nstr(), "float": _nstr(),
|
|
"pct_up_at_entry": _nstr(), "leading_gainer_rank": _nnum(),
|
|
"other": {"type": "array", "items": {"type": "string"}}}},
|
|
"why_entered": _quoted("reason"), "tape_reads": _quoted("read"),
|
|
"why_exited": _quoted("reason"), "mistake_or_lesson": _quoted("lesson")},
|
|
"required": ["ticker_heard", "ticker_guess", "setup", "setup_quote", "entry_price",
|
|
"exit_price", "outcome", "why_entered"]}},
|
|
"passed_on": {"type": "array", "items": {"type": "object", "properties": {
|
|
"ticker_heard": {"type": "string"}, "ticker_guess": {"type": "string"},
|
|
"reason": {"type": "string"}, "quote": {"type": "string"}},
|
|
"required": ["ticker_heard", "ticker_guess", "reason", "quote"]}},
|
|
"rules_stated": _quoted("rule"),
|
|
},
|
|
"required": ["video_kind", "session_date", "day", "trades", "passed_on", "rules_stated"],
|
|
}
|
|
|
|
|
|
_TASK_A = ("TASK FOR THIS CALL: your main job is `trades` -- list EVERY position he took, in order, "
|
|
"with its quotes and prices. Also fill video_kind, session_date and day (net P&L, "
|
|
"market_temperature). Passed-on stocks and rules are extracted in a separate call.\n\n")
|
|
_TASK_B = ("TASK FOR THIS CALL: fill passed_on (every stock he names but did NOT trade) and "
|
|
"rules_stated (every general principle, statistic or lesson he states -- usually 5-12). "
|
|
"Trades are extracted in a separate call.\n\n")
|
|
SCHEMA_A = {"type": "object", "properties": {k: SCHEMA["properties"][k] for k in
|
|
("video_kind", "session_date", "day", "trades")},
|
|
"required": ["video_kind", "session_date", "day", "trades"]}
|
|
SCHEMA_B = {"type": "object", "properties": {k: SCHEMA["properties"][k] for k in ("passed_on", "rules_stated")},
|
|
"required": ["passed_on", "rules_stated"]}
|
|
SPLIT_TASKS = [(_TASK_A, SCHEMA_A), (_TASK_B, SCHEMA_B)]
|