feat: nightly Ross-recap learning loop on the owner's local Ollama
Why: the pilot in data/ross showed that Ross Cameron's daily recaps, joined to
the SIP tape and to what our bot saw, answer "which of his trades could we have
taken and why didn't we". It was a set of one-off scripts run by hand; the
server-side youtube_claims pipeline has been dead since ~08-24 (YouTube blocks
the TrueNAS IP) and its captions fallback hid that. This makes the pilot a
nightly, local, cumulative loop without a paid LLM API.
youtube_claims/recap/ (python -m youtube_claims.recap.nightly):
- channel: yt-dlp flat listing of @DaytradeWarrior, m4a audio, faster-whisper
small.en on CPU, pilot-format bundles (meta + our scanner list + transcript).
- extract: Ollama structured output (JSON schema) per EXTRACT_SPEC, chunked
with overlap for long transcripts and merged. The verbatim rule is enforced in
code: every quote must be found (token-level, one-word tolerance) and is
snapped to the transcript's words; unquoted items, trades with no surviving
quote, prices never spoken, and passed-on names never said are dropped.
- llm: endpoint pool over RECAP_OLLAMA_HOSTS (round-robin + failover, per-host
model/ctx/think pins) since the owner runs several Ollama boxes.
- tape/join/ourside: ported join.py/ourside.py -- tickers are price-verified on
the Alpaca SIP tape (plus an order-free check for his own named ticker), then
paper (:5433) and live (:5434) event views are attached. Bars cached on disk.
- store + schema.sql: idempotent recap_* tables in the transcripts schema,
upserted per video_id. report: the pilot's sections + "what changed since last
run" against the previous day's metrics JSON.
- evaluate: scores any model/prompt against the 36 gold extractions.
Model choice (36 gold videos, final code path, P/R):
gpt-oss:20b on the workstation GPU, two focused calls, think=low:
trades P .84 R .70; tickers P .82 R .77; entry +-2% P .54 R .50;
passed-on P .66 R .76; temperature 16/31; kind 30/36; ~49 s/video.
tools:latest (4B, server) one call: trades P .81 R .72, tickers P .83 R .85,
passed-on P .80 R .75, temperature 20/31, but it invents prices
(entry P .42) and occasionally runs away (5-9 min). Prices drive the tape
verification, so the more precise local model is the default.
Iterations: prompt v2 + spoken-price rescale (699 -> 6.99) + split calls took
gpt-oss trades F1 .74 -> .76-.78 and tickers .78 -> .79-.80; think=medium
blew the token budget. qwen3-30b-a3b on mason only finished 5 videos (host
saturated) -- inconclusive.
Also: youtube_claims/transcription.py::_captions_fallback now picks the
youtube-transcript-api shape by feature detection instead of retrying the
removed get_transcript on any error, and transcribe() reports both the whisper
and captions causes -- the IpBlocked error was being masked as AttributeError.
No trading code touched.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
42ca523053
commit
91af4c2e93
20 files changed
+2775
-4
No files matched your search
@@ -0,0 +1,206 @@
|
||||
"""Pure pieces of the Ross recap extractor: verbatim-quote verification, chunking, and the
|
||||
merge of per-chunk results. The Ollama call itself is exercised by the gold-set evaluation."""
|
||||
|
||||
from collections import Counter
|
||||
|
||||
from youtube_claims.recap.extract import (chunk_bundle, clean_ticker, fix_spoken_prices, merge_records,
|
||||
scanner_prices, verify_record)
|
||||
from youtube_claims.recap.quotes import Transcript, transcript_text_from_bundle
|
||||
|
||||
TX = Transcript(
|
||||
"So I got in on that dip right there, and I got a nice little trade as it rallied back up. "
|
||||
"Then we come back up right here looking for a curl through VWAP. It fails. Doesn't happen. "
|
||||
"I took a small loss right there. GLND. This is not an easy one to day trade because it's very thickly traded. "
|
||||
"Generally, the right approach that I have is to avoid stocks that are giving us those big topping tails.")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- quotes
|
||||
def test_exact_quote_is_found_and_snapped_to_transcript_words():
|
||||
got = TX.find("looking for a curl through vwap")
|
||||
assert got == "looking for a curl through VWAP."
|
||||
|
||||
|
||||
def test_quote_matching_ignores_punctuation_and_case():
|
||||
assert TX.find("It fails, doesn't happen, I took a small loss") == "It fails. Doesn't happen. I took a small loss"
|
||||
|
||||
|
||||
def test_one_miscopied_word_in_a_long_quote_is_tolerated():
|
||||
q = "This is not an easy one to trade because it's very thickly traded" # dropped "day"
|
||||
got = TX.find(q)
|
||||
assert got is not None and got.startswith("This is not an easy one") and "thickly traded" in got
|
||||
|
||||
|
||||
def test_invented_quote_is_rejected():
|
||||
assert TX.find("I made ten thousand dollars on the open with a huge short squeeze") is None
|
||||
# a real 40-char prefix with an invented tail is NOT enough (the old prefix check accepted it)
|
||||
assert TX.find("So I got in on that dip right there, and then I sold the whole thing at twelve dollars") is None
|
||||
|
||||
|
||||
def test_too_short_quote_is_not_evidence():
|
||||
assert TX.find("it fails") is None
|
||||
assert TX.find("") is None
|
||||
|
||||
|
||||
def test_transcript_text_from_bundle_strips_timestamps():
|
||||
b = "video_id: x\nTRANSCRIPT (mm:ss text):\n[00:01] Hello there.\n[00:05] Second line."
|
||||
assert transcript_text_from_bundle(b) == "Hello there. Second line."
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- verification
|
||||
def _raw():
|
||||
return {
|
||||
"video_kind": "daily_recap", "session_date": "2026-09-24",
|
||||
"day": {"net_pnl": "$2,992.62", "green": True, "market_temperature": "Cold",
|
||||
"market_quote": "totally made up sentence about the market being slow today"},
|
||||
"trades": [
|
||||
{"ticker_heard": "a pus", "ticker_guess": "$apus", "setup": "dip_and_curl",
|
||||
"setup_quote": "I got in on that dip right there", "entry_price": "5.50", "outcome": "WIN",
|
||||
"why_entered": [{"reason": "dip", "quote": "So I got in on that dip right there"},
|
||||
{"reason": "fake", "quote": "because the level two was stacked with buyers everywhere"}],
|
||||
"stock_context": {"catalyst": "headline", "catalyst_quote": "a quote that does not exist anywhere here",
|
||||
"other": ["topping tails", "rocket ship"]}},
|
||||
{"ticker_heard": "ZZZ", "ticker_guess": "ZZZ", "setup": "bull_flag",
|
||||
"setup_quote": "I bought the bull flag at the high of day for a big win",
|
||||
"why_entered": [{"reason": "x", "quote": "entirely invented reason for entering this trade"}]},
|
||||
],
|
||||
"passed_on": [
|
||||
{"ticker_heard": "glnd", "ticker_guess": "GLND", "reason": "grinder",
|
||||
"quote": "not an easy one to day trade because it's very thickly traded"},
|
||||
{"ticker_heard": "pmax", "ticker_guess": "PMAX", "reason": "halts",
|
||||
"quote": "you cannot possibly manage your risk on these halting stocks"},
|
||||
],
|
||||
"rules_stated": [{"rule": "avoid topping tails",
|
||||
"quote": "avoid stocks that are giving us those big topping tails"},
|
||||
{"rule": "fake", "quote": "always buy the first red candle after a halt resume"}],
|
||||
}
|
||||
|
||||
|
||||
def test_verify_drops_unquoted_items_and_snaps_the_rest():
|
||||
st = Counter()
|
||||
rec = verify_record(_raw(), TX, st)
|
||||
assert [t["ticker_guess"] for t in rec["trades"]] == ["APUS"] # ZZZ had no real quote -> dropped
|
||||
t = rec["trades"][0]
|
||||
assert t["entry_price"] is None and t["outcome"] == "win" # 5.50 is never said -> dropped
|
||||
assert [w["reason"] for w in t["why_entered"]] == ["dip"]
|
||||
assert t["stock_context"]["catalyst"] is None # its quote was invented
|
||||
assert t["stock_context"]["other"] == ["topping tails"]
|
||||
assert [p["ticker_guess"] for p in rec["passed_on"]] == ["GLND"]
|
||||
assert rec["passed_on"][0]["quote"].startswith("not an easy one to day trade")
|
||||
assert len(rec["rules_stated"]) == 1
|
||||
assert rec["day"]["net_pnl"] == 2992.62 and rec["day"]["market_temperature"] == "cold"
|
||||
assert rec["day"]["market_quote"] is None
|
||||
assert st["dropped_trades"] == 1 and st["dropped_passed_on"] == 1 and st["dropped_rules"] == 1
|
||||
|
||||
|
||||
def test_clean_ticker():
|
||||
assert clean_ticker("$apus") == "APUS"
|
||||
assert clean_ticker("K I T T") == "KITT"
|
||||
assert clean_ticker("null") is None and clean_ticker("Greenland stocks") is None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- chunking
|
||||
def test_short_bundle_is_one_chunk():
|
||||
b = "video_id: x\nTRANSCRIPT (mm:ss text):\n[00:01] hi"
|
||||
assert chunk_bundle(b, 10_000) == [b]
|
||||
|
||||
|
||||
def test_long_bundle_chunks_keep_header_and_overlap():
|
||||
head = "video_id: x\nupload_date: 20260924\nTRANSCRIPT (mm:ss text):"
|
||||
lines = [f"[{i // 60:02d}:{i % 60:02d}] line number {i} " + "x" * 80 for i in range(200)]
|
||||
b = head + "\n" + "\n".join(lines)
|
||||
chunks = chunk_bundle(b, 4000, overlap_lines=3)
|
||||
assert len(chunks) > 3
|
||||
assert all(c.startswith(head) for c in chunks)
|
||||
body = [c.split("TRANSCRIPT (mm:ss text):\n", 1)[1].splitlines() for c in chunks]
|
||||
assert body[0][-3:] == body[1][:3] # overlap carried over
|
||||
seen = {ln for part in body for ln in part}
|
||||
assert seen == set(lines) # nothing lost
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- merge
|
||||
def _trade(tk, setup_q, reasons=(), **kw):
|
||||
return {"ticker_heard": tk, "ticker_guess": tk, "setup": kw.pop("setup", "other"), "setup_quote": setup_q,
|
||||
"entry_price": kw.pop("entry_price", None), "exit_price": None, "outcome": kw.pop("outcome", None),
|
||||
"stock_context": {"catalyst": None, "other": kw.pop("other", [])},
|
||||
"why_entered": [{"reason": r, "quote": q} for r, q in reasons], "tape_reads": [], "why_exited": [],
|
||||
"mistake_or_lesson": []}
|
||||
|
||||
|
||||
def test_merge_dedupes_the_overlap_and_unions_fields():
|
||||
a = {"video_kind": "daily_recap", "session_date": "2026-09-24",
|
||||
"day": {"net_pnl": 100.0, "market_temperature": "cold", "market_quote": None},
|
||||
"trades": [_trade("APUS", "i bought the dip right there", [("dip", "i bought the dip right there")],
|
||||
other=["grinder"])],
|
||||
"passed_on": [{"ticker_guess": "GLND", "ticker_heard": "glnd", "reason": "r", "quote": "q1"},
|
||||
{"ticker_guess": "PFSA", "ticker_heard": "pfsa", "reason": "r", "quote": "q2"}],
|
||||
"rules_stated": [{"rule": "a", "quote": "cut losses quickly and let winners run"}]}
|
||||
b = {"video_kind": "daily_recap", "session_date": None,
|
||||
"day": {"net_pnl": None, "market_temperature": "cold", "market_quote": "slow market today"},
|
||||
"trades": [_trade("APUS", "i bought the dip right there and it rallied", entry_price=5.5, outcome="win",
|
||||
other=["sympathy"]),
|
||||
_trade("PFSA", "curl through vwap", setup="vwap_break")],
|
||||
"passed_on": [{"ticker_guess": "GLND", "ticker_heard": "glnd", "reason": "r", "quote": "q1 again"}],
|
||||
"rules_stated": [{"rule": "a", "quote": "cut losses quickly and let winners run"},
|
||||
{"rule": "b", "quote": "size down when the market is cold"}]}
|
||||
m = merge_records([a, b])
|
||||
assert [t["ticker_guess"] for t in m["trades"]] == ["APUS", "PFSA"]
|
||||
apus = m["trades"][0]
|
||||
assert apus["entry_price"] == 5.5 and apus["outcome"] == "win" # filled from the later chunk
|
||||
assert apus["stock_context"]["other"] == ["grinder", "sympathy"]
|
||||
assert [p["ticker_guess"] for p in m["passed_on"]] == ["GLND", "PFSA"] # GLND deduped across chunks
|
||||
assert len(m["rules_stated"]) == 2
|
||||
assert m["day"] == {"net_pnl": 100.0, "market_temperature": "cold", "market_quote": "slow market today"}
|
||||
assert m["session_date"] == "2026-09-24"
|
||||
|
||||
|
||||
def test_merge_keeps_distinct_trades_on_the_same_ticker():
|
||||
a = {"video_kind": "daily_recap", "day": {}, "passed_on": [], "rules_stated": [],
|
||||
"trades": [_trade("APUS", "first entry on the dip at the open")]}
|
||||
b = {"video_kind": "daily_recap", "day": {}, "passed_on": [], "rules_stated": [],
|
||||
"trades": [_trade("APUS", "then i got back in for the break through seven")]}
|
||||
assert len(merge_records([a, b])["trades"]) == 2
|
||||
|
||||
|
||||
def test_merge_temperature_conflict_becomes_mixed():
|
||||
parts = [{"video_kind": "daily_recap", "day": {"market_temperature": t}, "trades": [], "passed_on": [],
|
||||
"rules_stated": []} for t in ("hot", "cold")]
|
||||
assert merge_records(parts)["day"]["market_temperature"] == "mixed"
|
||||
|
||||
|
||||
def test_spoken_prices_are_rescaled_for_small_caps_only():
|
||||
b = ("OUR SCANNER LIST (symbol, gap %, price) by session date:\n"
|
||||
" 2026-06-23: ICCM 80.1% $7.2, BIG 5.0% $150.0\nTRANSCRIPT (mm:ss text):\n")
|
||||
px = scanner_prices(b)
|
||||
assert px == {"ICCM": 7.2, "BIG": 150.0}
|
||||
rec = {"trades": [{"ticker_guess": "ICCM", "entry_price": 699.0, "exit_price": 7.5, "stated_high": None,
|
||||
"stated_low": None},
|
||||
{"ticker_guess": "BIG", "entry_price": 151.0}]}
|
||||
assert fix_spoken_prices(rec, px) == 1
|
||||
assert rec["trades"][0]["entry_price"] == 6.99 and rec["trades"][1]["entry_price"] == 151.0
|
||||
|
||||
|
||||
def test_paragraph_quotes_are_verified_then_cut_to_a_pointer():
|
||||
from youtube_claims.recap.extract import MAX_QUOTE_WORDS, _snap
|
||||
words = [f"w{i}" for i in range(120)]
|
||||
tx = Transcript(" ".join(words))
|
||||
got = _snap(tx, " ".join(words[5:100]))
|
||||
assert got == " ".join(words[5:5 + MAX_QUOTE_WORDS])
|
||||
|
||||
|
||||
def test_passed_on_names_must_be_said():
|
||||
from youtube_claims.recap.extract import name_mentioned
|
||||
tx = Transcript("Then K I T T popped up and Greenland stocks ran but I passed on PMAX")
|
||||
assert name_mentioned(tx, "Kit", "KITT") # spelled out
|
||||
assert name_mentioned(tx, "Greenland stocks", None) # heard phrase
|
||||
assert name_mentioned(tx, "p max", "PMAX")
|
||||
assert not name_mentioned(tx, "benf", "BENF") # only on the scanner list
|
||||
|
||||
|
||||
def test_prices_must_be_spoken_in_the_transcript():
|
||||
from youtube_claims.recap.extract import price_mentioned
|
||||
tx = Transcript("I got in at 550 and sold at $6.25, the high was 7 and it traded 1,200 shares")
|
||||
assert price_mentioned(tx, 5.5) and price_mentioned(tx, 6.25) and price_mentioned(tx, 7.0)
|
||||
assert not price_mentioned(tx, 8.4)
|
||||
rec = verify_record({"trades": [{"ticker_guess": "ABC", "setup_quote": "I got in at 550 and sold at",
|
||||
"entry_price": 5.5, "exit_price": 9.1}]}, tx)
|
||||
assert rec["trades"][0]["entry_price"] == 5.5 and rec["trades"][0]["exit_price"] is None
|
||||
@@ -0,0 +1,228 @@
|
||||
"""Ross recap loop: ticker/price verification against fake tapes, the report diff, the gold
|
||||
scorer, and the captions-fallback error surfacing."""
|
||||
|
||||
import datetime as dt
|
||||
import sys
|
||||
import types
|
||||
|
||||
import pytest
|
||||
|
||||
from youtube_claims.recap.evaluate import pair_trades, score_video
|
||||
from youtube_claims.recap.report import compute, diff_metrics, render
|
||||
from youtube_claims.recap.tape import entry_bucket, entry_time, path_score, resolve, tape_stats
|
||||
|
||||
|
||||
def bars(path, start="07:00"):
|
||||
"""Fake 1-min tape from a list of (high, low); open/close at the midpoint, volume 1000."""
|
||||
t0 = dt.datetime.fromisoformat(f"2026-09-24T{start}")
|
||||
return [(t0 + dt.timedelta(minutes=i), (h + l) / 2, h, l, (h + l) / 2, 1000.0) for i, (h, l) in enumerate(path)]
|
||||
|
||||
|
||||
RUNNER = bars([(3.0, 2.9), (3.6, 3.0), (4.2, 3.5), (5.1, 4.1), (4.6, 4.0), (4.3, 3.9)]) # 3 -> 5.1 -> 3.9
|
||||
FLAT = bars([(10.0, 9.8), (10.1, 9.9), (10.2, 9.9)])
|
||||
OTHER_RUNNER = bars([(3.4, 3.3), (3.6, 3.4), (5.3, 3.5), (5.0, 4.6)])
|
||||
|
||||
|
||||
def test_path_score_accepts_the_story_the_tape_can_tell():
|
||||
assert path_score(RUNNER, 3.5, None, None) == 0.5 # entry traded, no high named
|
||||
s = path_score(RUNNER, 3.5, 5.0, None) # entry then a high near 5.0 (5.1: 2%)
|
||||
assert s is not None and s < 0.025
|
||||
assert path_score(RUNNER, 3.5, 5.0, 3.9) is not None # ...then a low near 3.9
|
||||
|
||||
|
||||
def test_path_score_rejects_impossible_stories():
|
||||
assert path_score(FLAT, 3.5, None, None) is None # never traded 3.5
|
||||
assert path_score(RUNNER, 3.5, 7.0, None) is None # never reached 7
|
||||
assert path_score(RUNNER, 3.5, 5.0, 2.0) is None # never came back to 2
|
||||
assert path_score([], 3.5, None, None) is None
|
||||
assert path_score(RUNNER, None, None, None) is None # no price -> nothing to verify
|
||||
|
||||
|
||||
def _bars_for(tapes):
|
||||
return lambda s: tapes.get(s, [])
|
||||
|
||||
|
||||
def test_resolve_verifies_his_ticker_when_the_tape_fits():
|
||||
item = {"ticker_heard": "a pus", "ticker_guess": "APUS", "entry_price": 3.5, "stated_high": 5.0}
|
||||
assert resolve(item, _bars_for({"APUS": RUNNER}), ["FLAT"]) == ("APUS", "price_verified")
|
||||
|
||||
|
||||
def test_resolve_prefers_his_name_when_levels_fit_out_of_order():
|
||||
# he says "squeezed to 5.1 ... it had come from 2.95": the low came FIRST on the tape
|
||||
item = {"ticker_heard": "apus", "ticker_guess": "APUS", "entry_price": None, "stated_high": 5.1,
|
||||
"stated_low": 2.95}
|
||||
other = bars([(5.2, 5.0), (3.1, 2.9)]) # a scanner name with high-then-low in order
|
||||
assert resolve(item, _bars_for({"APUS": RUNNER, "OTH": other}), ["OTH"]) == ("APUS", "price_verified_loose")
|
||||
|
||||
|
||||
def test_resolve_finds_the_unique_scanner_name_when_his_ticker_is_misheard():
|
||||
item = {"ticker_heard": "kit", "ticker_guess": "KIT", "entry_price": 3.5, "stated_high": 5.0}
|
||||
tapes = {"KIT": FLAT, "KITT": RUNNER, "BIG": FLAT}
|
||||
assert resolve(item, _bars_for(tapes), ["BIG", "KITT"]) == ("KITT", "price_matched_scanner")
|
||||
|
||||
|
||||
def test_resolve_reports_ambiguity_instead_of_guessing():
|
||||
item = {"ticker_heard": "x", "ticker_guess": "XX", "entry_price": 3.5, "stated_high": 5.1}
|
||||
got = resolve(item, _bars_for({"AAA": RUNNER, "BBB": RUNNER}), ["AAA", "BBB"])
|
||||
assert got[0] is None and got[1].startswith("ambiguous:") and "AAA" in got[1] and "BBB" in got[1]
|
||||
|
||||
|
||||
def test_resolve_no_price_match_and_unverified():
|
||||
item = {"ticker_heard": "zz", "ticker_guess": "ZZ", "entry_price": 50.0, "stated_high": None}
|
||||
assert resolve(item, _bars_for({"ZZ": RUNNER}), ["ZZ"]) == ("ZZ", "no_price_match")
|
||||
item = {"ticker_heard": "zz", "ticker_guess": "ZZ", "entry_price": None, "stated_high": None}
|
||||
assert resolve(item, _bars_for({}), []) == ("ZZ", "unverified")
|
||||
assert resolve({"ticker_heard": "Greenland", "ticker_guess": ""}, _bars_for({}), []) == (None, "unverified")
|
||||
|
||||
|
||||
def test_entry_time_is_first_touch_before_his_stated_high():
|
||||
t = entry_time(RUNNER, 4.1, 5.1)
|
||||
assert t.strftime("%H:%M") == "07:02"
|
||||
assert entry_time(RUNNER, 9.9, None) is None
|
||||
assert entry_bucket(t) == "7:00-9:30" and entry_bucket(dt.datetime(2026, 9, 24, 9, 31)) == "9:30-9:45"
|
||||
|
||||
|
||||
def test_tape_stats_splits_premarket_and_rth():
|
||||
tape = bars([(2.0, 1.9), (2.8, 2.0)], start="09:28") + bars([(3.0, 2.5), (2.6, 2.2)], start="09:30")
|
||||
s = tape_stats(tape, prev_close=2.0)
|
||||
assert s["pm_high"] == 2.8 and s["hod"] == 3.0 and s["hod_time"] == "09:30"
|
||||
assert s["open"] == pytest.approx(2.75) and s["gap_pct"] == pytest.approx(0.375)
|
||||
assert s["first_30pct_time"] == "09:29"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- report
|
||||
def _view(disc=None, orders=None, oop=None):
|
||||
side = {"discovered": [[disc, "Discovered"]] if disc else [], "orders": orders or [],
|
||||
"states": [["08:00", f"x out_of_population: price 1.5 < 2"]] if oop else [],
|
||||
"top_blocks": [["spread too wide", 3]]}
|
||||
return {"paper": side, "live": None, "tape": {"prev_close": 2.0, "open": 3.0, "close": 3.3,
|
||||
"open_to_hod_pct": 0.4}}
|
||||
|
||||
|
||||
def _rows(n_videos=2):
|
||||
videos = [{"video_id": f"v{i}", "session_date": f"2026-09-2{i}", "video_kind": "daily_recap",
|
||||
"extractor": "gold", "market_temperature": "cold"} for i in range(n_videos)]
|
||||
trades = [{"video_id": "v0", "session_date": "2026-09-20", "symbol": "APUS", "resolve": "price_verified",
|
||||
"setup": "micro_pullback", "outcome": "win", "pnl": 1200, "entry_price": 3.0,
|
||||
"entry_time_tape": "07:05", "stock_context": {"catalyst": "headline"},
|
||||
"our_view": _view(disc="06:59", orders=[["07:10", "buy"]])},
|
||||
{"video_id": "v1", "session_date": "2026-09-21", "symbol": None, "resolve": "ambiguous:A,B",
|
||||
"setup": "other", "outcome": None, "pnl": None, "entry_price": 5.0, "entry_time_tape": None,
|
||||
"stock_context": {}, "our_view": None}]
|
||||
passed = [{"video_id": "v0", "symbol": "GLND", "our_view": _view()}]
|
||||
return videos, trades, passed
|
||||
|
||||
|
||||
def test_compute_aggregates_coverage_and_buckets():
|
||||
m = compute(*_rows())
|
||||
assert m["videos"] == 2 and m["trades"] == 2 and m["verified"] == 1
|
||||
assert m["resolution"] == {"price_verified": 1, "ambiguous": 1}
|
||||
assert m["entry_buckets"]["7:00-9:30"] == 1
|
||||
assert m["coverage"] == {"verified": 1, "discovered": 1, "discovered_before_entry": 1, "we_ordered": 1,
|
||||
"out_of_population": 0}
|
||||
assert m["passed_tape"]["with_tape"] == 1 and m["passed_tape"]["median_open_to_close"] == pytest.approx(0.1)
|
||||
md = render(m, ["x"], today="2026-09-25")
|
||||
assert "## What changed since last run" in md and "## Our coverage of his verified trades" in md
|
||||
|
||||
|
||||
def test_diff_first_run_and_no_change():
|
||||
m = compute(*_rows())
|
||||
assert diff_metrics(None, m) == ["First report -- no previous run to compare with."]
|
||||
assert diff_metrics(m, m) == ["No change since the previous run."]
|
||||
|
||||
|
||||
def test_diff_reports_new_videos_trades_and_metric_moves():
|
||||
prev = compute(*_rows())
|
||||
videos, trades, passed = _rows(3)
|
||||
trades.append({"video_id": "v2", "session_date": "2026-09-22", "symbol": "KITT", "resolve": "price_verified",
|
||||
"setup": "bull_flag", "outcome": "loss", "pnl": -500, "entry_price": 6.0,
|
||||
"entry_time_tape": "09:31", "stock_context": {}, "our_view": _view()})
|
||||
cur = compute(videos, trades, passed)
|
||||
lines = diff_metrics(prev, cur)
|
||||
text = "\n".join(lines)
|
||||
assert "New videos (1): v2" in text
|
||||
assert "trades: 2 -> 3 (+1)" in text and "verified: 1 -> 2 (+1)" in text
|
||||
assert "entry_buckets.9:30-9:45: 0 -> 1" in text
|
||||
assert "New trades (1): KITT|bull_flag|6.0" in text
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- evaluation scorer
|
||||
def test_score_video_perfect_and_partial():
|
||||
gold = {"video_kind": "daily_recap", "day": {"market_temperature": "hot", "net_pnl": 1000},
|
||||
"trades": [{"ticker_guess": "APUS", "setup": "micro_pullback", "entry_price": 5.0, "exit_price": 5.5},
|
||||
{"ticker_guess": "APUS", "setup": "vwap_break", "entry_price": 4.0, "exit_price": None},
|
||||
{"ticker_guess": "PFSA", "setup": "other", "entry_price": None, "exit_price": None}],
|
||||
"passed_on": [{"ticker_guess": "GLND"}, {"ticker_guess": "PMAX"}], "rules_stated": []}
|
||||
s = score_video(gold, gold)
|
||||
assert s["matched_trades"] == 3 and s["setup_ok"] == 3 and s["tick_tp"] == 2 and s["pass_tp"] == 2
|
||||
pred = {"video_kind": "daily_recap", "day": {"market_temperature": "cold", "net_pnl": 1010},
|
||||
"trades": [{"ticker_guess": "APUS", "setup": "vwap_break", "entry_price": 4.05, "exit_price": 9.0},
|
||||
{"ticker_guess": "ZZZ", "setup": "other"}],
|
||||
"passed_on": [{"ticker_guess": "GLND"}, {"ticker_guess": "BENF"}], "rules_stated": []}
|
||||
s = score_video(gold, pred)
|
||||
assert s["matched_trades"] == 1 and s["setup_ok"] == 1 # paired with the vwap_break trade
|
||||
assert s["entry_price_ok"] == 1 and s["exit_price_ok"] == 0 and s["exit_price_pred"] == 1
|
||||
assert (s["tick_tp"], s["tick_pred"], s["tick_gold"]) == (1, 2, 2)
|
||||
assert (s["pass_tp"], s["pass_pred"], s["pass_gold"]) == (1, 2, 2)
|
||||
assert s["temp_ok"] == 0 and s["pnl_ok"] == 1
|
||||
|
||||
|
||||
def test_pair_trades_matches_each_trade_once():
|
||||
g = [{"ticker_guess": "A", "setup": "x"}]
|
||||
p = [{"ticker_guess": "A", "setup": "x"}, {"ticker_guess": "A", "setup": "x"}]
|
||||
assert len(pair_trades(g, p)) == 1
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- captions fallback
|
||||
def _fake_api(monkeypatch, cls):
|
||||
mod = types.ModuleType("youtube_transcript_api")
|
||||
mod.YouTubeTranscriptApi = cls
|
||||
monkeypatch.setitem(sys.modules, "youtube_transcript_api", mod)
|
||||
|
||||
|
||||
class IpBlocked(Exception):
|
||||
pass
|
||||
|
||||
|
||||
def test_captions_fallback_surfaces_the_real_error(monkeypatch):
|
||||
from youtube_claims import transcription
|
||||
|
||||
class NewApi: # >=1.0 shape, no get_transcript
|
||||
def fetch(self, vid):
|
||||
raise IpBlocked("YouTube is blocking requests from your IP")
|
||||
|
||||
_fake_api(monkeypatch, NewApi)
|
||||
with pytest.raises(IpBlocked, match="blocking requests"):
|
||||
transcription._captions_fallback("abc")
|
||||
|
||||
|
||||
def test_captions_fallback_old_api_still_works(monkeypatch):
|
||||
from youtube_claims import transcription
|
||||
|
||||
class OldApi:
|
||||
@staticmethod
|
||||
def get_transcript(vid):
|
||||
return [{"start": 1.0, "duration": 2.0, "text": "hi"}]
|
||||
|
||||
_fake_api(monkeypatch, OldApi)
|
||||
assert transcription._captions_fallback("abc") == [{"start": 1.0, "end": 3.0, "text": "hi"}]
|
||||
|
||||
|
||||
def test_transcribe_reports_both_causes(monkeypatch, tmp_path):
|
||||
from youtube_claims import config, transcription
|
||||
|
||||
class NewApi:
|
||||
def fetch(self, vid):
|
||||
raise IpBlocked("blocked")
|
||||
|
||||
_fake_api(monkeypatch, NewApi)
|
||||
monkeypatch.setattr(config, "TRANSCRIPT_DIR", str(tmp_path))
|
||||
|
||||
def no_audio(vid, td):
|
||||
raise RuntimeError("Sign in to confirm you're not a bot")
|
||||
|
||||
monkeypatch.setattr(transcription, "_download_audio", no_audio)
|
||||
with pytest.raises(RuntimeError) as ei:
|
||||
transcription.transcribe("abc", watchlist=[])
|
||||
msg = str(ei.value)
|
||||
assert "not a bot" in msg and "IpBlocked: blocked" in msg and "get_transcript" not in msg
|
||||
@@ -0,0 +1,65 @@
|
||||
"""Round-robin + failover across the owner's Ollama endpoints (no network: fake transport)."""
|
||||
|
||||
import urllib.error
|
||||
|
||||
import pytest
|
||||
|
||||
from youtube_claims.recap.llm import OllamaPool, parse_hosts
|
||||
|
||||
|
||||
def test_parse_hosts_with_per_host_model_and_options():
|
||||
eps = parse_hosts(" http://a:11434/ , http://b:11434=qwen3:30b#ctx=8192#think=false,, ")
|
||||
assert [e.host for e in eps] == ["http://a:11434", "http://b:11434"]
|
||||
assert eps[0].model is None and eps[0].think == "default"
|
||||
assert eps[1].model == "qwen3:30b" and eps[1].num_ctx == 8192 and eps[1].think is False
|
||||
|
||||
|
||||
def _pool(spec, models, fail=(), default="m1"):
|
||||
calls = []
|
||||
|
||||
def call(system, user, *, model, host, num_ctx, think, **kw):
|
||||
calls.append((host, model, num_ctx, think))
|
||||
if host in fail:
|
||||
raise urllib.error.URLError("connection refused")
|
||||
return "{}", {"seconds": 1}
|
||||
|
||||
pool = OllamaPool(parse_hosts(spec), default, default_ctx=24576, default_think="low", call=call,
|
||||
list_models=lambda h: models[h])
|
||||
return pool, calls
|
||||
|
||||
|
||||
def test_round_robin_over_hosts_that_have_the_model():
|
||||
pool, calls = _pool("http://a,http://b,http://c", {"http://a": {"m1"}, "http://b": {"other"},
|
||||
"http://c": {"m1:latest"}})
|
||||
for _ in range(4):
|
||||
pool.chat("s", "u")
|
||||
assert [c[0] for c in calls] == ["http://a", "http://c", "http://c", "http://a"]
|
||||
|
||||
|
||||
def test_failover_parks_a_dead_host():
|
||||
pool, calls = _pool("http://a,http://b", {"http://a": {"m1"}, "http://b": {"m1"}}, fail={"http://a"})
|
||||
_, st = pool.chat("s", "u")
|
||||
assert st["host"] == "http://b"
|
||||
pool.chat("s", "u")
|
||||
assert [c[0] for c in calls] == ["http://a", "http://b", "http://b"] # a parked after failing
|
||||
|
||||
|
||||
def test_per_host_model_and_options_are_used():
|
||||
pool, calls = _pool("http://a=qwen#ctx=8192#think=false", {"http://a": {"qwen"}})
|
||||
_, st = pool.chat("s", "u")
|
||||
assert calls == [("http://a", "qwen", 8192, False)] and st["model"] == "qwen"
|
||||
|
||||
|
||||
def test_no_endpoint_with_the_model_is_an_error():
|
||||
pool, _ = _pool("http://a", {"http://a": {"x"}})
|
||||
with pytest.raises(RuntimeError, match="no reachable Ollama endpoint"):
|
||||
pool.chat("s", "u")
|
||||
|
||||
|
||||
def test_client_errors_are_not_retried_elsewhere():
|
||||
def call(*a, **kw):
|
||||
raise RuntimeError("ollama 400: invalid think value")
|
||||
|
||||
pool = OllamaPool(parse_hosts("http://a,http://b"), "m1", call=call, list_models=lambda h: {"m1"})
|
||||
with pytest.raises(RuntimeError, match="400"):
|
||||
pool.chat("s", "u")
|
||||
Reference in new issue
Block a user