Commit Graph
4 Commits
Author SHA1 Message Date
rgriffith 574bb79a0b fix(eval): correct indentation for load/reload in _load_policy_service 2026-03-30 10:36:43 -04:00
KoalaTrapLord 5554ef4dd7 fix(eval): call load() for embeddings before reload() for checkpoint 2026-03-30 09:53:56 -04:00
KoalaTrapLord cf2d7ed806 fix(eval): use reload() with explicit checkpoint path instead of load()
load() hardcodes best_policy.pt but PPO trainer saves as best_ppo.pt, causing the model to never load during eval. reload() accepts an explicit path.
2026-03-30 09:51:22 -04:00
KoalaTrapLord bff7b3a81e feat(eval): add forge_evaluator, eval_policy, compare_eval for issue #153
- ml/eval/forge_evaluator.py: reusable eval harness wrapping
  MultiThreadBatchRunner + PolicyInferenceService; collects win rate,
  avg turns, life differential, commander damage, illegal action rate,
  policy entropy per game; writes results/eval-{run_id}.json
- scripts/eval_policy.py: CLI to run N Forge eval games with any
  checkpoint vs forge-builtin or policy-server opponent; supports
  --threads, --clock, --seed, --greedy, --temperature
- scripts/compare_eval.py: loads two result JSONs, computes absolute
  win-rate delta + 95% CI, runs Fisher's exact test for significance,
  prints formatted comparison table and pass/fail verdict
2026-03-29 04:49:43 -04:00