Commit Graph
1 Commits
Author SHA1 Message Date
KoalaTrapLord bff7b3a81e feat(eval): add forge_evaluator, eval_policy, compare_eval for issue #153
- ml/eval/forge_evaluator.py: reusable eval harness wrapping
  MultiThreadBatchRunner + PolicyInferenceService; collects win rate,
  avg turns, life differential, commander damage, illegal action rate,
  policy entropy per game; writes results/eval-{run_id}.json
- scripts/eval_policy.py: CLI to run N Forge eval games with any
  checkpoint vs forge-builtin or policy-server opponent; supports
  --threads, --clock, --seed, --greedy, --temperature
- scripts/compare_eval.py: loads two result JSONs, computes absolute
  win-rate delta + 95% CI, runs Fisher's exact test for significance,
  prints formatted comparison table and pass/fail verdict
2026-03-29 04:49:43 -04:00