Claude
a7f2e0aa79
Accuracy: news_enrichment now sends a JSON schema to Ollama's `format` (enum-locked catalyst_type, typed fields) so decoding is constrained at generation time, and build_prompt carries 3 few-shot examples (offering, FDA, earnings) — the biggest accuracy lever for a small local model. classify_headline gains use_schema (default True; False = plain json). Validation: - research/ingestion/catalyst_eval.py — hand-labeled fixture + pure score_predictions (type accuracy, is_dilutive precision/recall, sentiment sign). catalyst_eval.py CLI A/Bs models (--model) against the same set. - validate_catalysts.py — joins enriched labels to each stored gapper's open->close / open->high outcome; pure outcome_correlation splits by dilution flag and bullish-score bucket. Reports coverage honestly (news is live-only) and writes a CSV. Run before flipping the Phase 2 flags. Tests: schema-in-request assertions, pure eval/aggregator coverage. 212 pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01M5HTMkaadGDnAPtXe2hWwg
42 lines
1.5 KiB
Python
42 lines
1.5 KiB
Python
#!/usr/bin/env python3
|
|
"""A/B a local Ollama model on catalyst-labeling accuracy against a fixed set.
|
|
|
|
Usage:
|
|
PYTHONPATH=. python catalyst_eval.py # default model
|
|
PYTHONPATH=. python catalyst_eval.py --model qwen3:14b # A/B a bigger model
|
|
PYTHONPATH=. python catalyst_eval.py --no-schema # plain JSON format
|
|
|
|
Runs the hand-labeled fixture in ``research/ingestion/catalyst_eval.py`` through
|
|
the live model and prints type accuracy, is_dilutive precision/recall (the veto's
|
|
basis), and the type mistakes. Pick the model that wins here for OLLAMA_MODEL.
|
|
"""
|
|
|
|
import argparse
|
|
|
|
from research.ingestion.catalyst_eval import format_report, run_eval
|
|
|
|
|
|
def main(argv=None) -> int:
|
|
p = argparse.ArgumentParser(description=__doc__)
|
|
p.add_argument("--model", default="qwen2.5:7b-instruct")
|
|
p.add_argument("--host", default="http://localhost:11434")
|
|
p.add_argument("--timeout", type=int, default=30)
|
|
p.add_argument("--no-schema", action="store_true",
|
|
help="send plain 'format: json' instead of the JSON schema")
|
|
args = p.parse_args(argv)
|
|
|
|
metrics, _ = run_eval(
|
|
host=args.host, model=args.model, timeout=args.timeout,
|
|
use_schema=not args.no_schema,
|
|
)
|
|
print(format_report(metrics, model=args.model))
|
|
if metrics["scored"] == 0:
|
|
print(f"\nNo predictions scored — is Ollama running at {args.host} "
|
|
f"with model '{args.model}' pulled?")
|
|
return 1
|
|
return 0
|
|
|
|
|
|
if __name__ == "__main__":
|
|
raise SystemExit(main())
|