commander-ai-lab/docs/PHASE1-SELF-PLAY-INTEGRATION-CHECKLIST.md

Phase 1 — Self-Play Integration Checklist

Issue: #65 — PPO Decision Exporter
Ref: docs/CLOSED-LOOP-DISTILLATION.md
Status: In Progress

---

Completed

• Create ml/training/decision_exporter.py with DecisionExporter class
• ExporterConfig dataclass (output_dir, only_wins, model_version, min_episode_steps)
• build_decision_record() — converts PPO step data into Forge-compatible JSONL format
• DecisionExporter.begin_episode() / record_step() / end_episode() / flush() lifecycle
• Write to data/results/ml-decisions-ppo-{batch_id}.jsonl
• Include PPO metadata fields: source, model_version, reward, episode_return
• Win-only filtering (only export decisions from winning games)
• Short-episode gating (min_episode_steps threshold)
• _macro_action_to_forge_type() mapping (MacroAction enum → Forge action type strings)
• Export stats tracking (stats property)

---

Remaining — Integrate into ml/training/self_play.py

Import

• Add import: from ml.training.decision_exporter import DecisionExporter

Modify run_self_play_episode()

• Add exporter: DecisionExporter = None parameter to function signature
• After each trajectory.append(...) block (3 locations), add:

  if exporter is not None:
      exporter.record_step(
          game_state_snapshot=prev_snapshot,
          action_idx=action_idx,
          reward=reward,
      )
  

Modify collect_rollouts()

• Add exporter: DecisionExporter = None parameter to function signature
• Before each episode, call:

  if exporter is not None:
      exporter.begin_episode(agent_seat=agent_seat, playstyle=playstyle)
  

• Pass exporter=exporter to run_self_play_episode()
• After outcome tracking, call:

  if exporter is not None:
      won = trajectory and trajectory[-1]["done"] and trajectory[-1]["reward"] > 0.5
      ep_return = sum(s["reward"] for s in trajectory)
      exporter.end_episode(won=won, episode_return=ep_return)
  

Commit

• git add ml/training/self_play.py
• git commit -m "feat(ml): integrate DecisionExporter into self-play rollouts (#65)"
• git push origin main

---

Integration Pattern

collect_rollouts(exporter=exporter)
  └─ for each episode:
       ├─ exporter.begin_episode(seat, playstyle)
       ├─ run_self_play_episode(..., exporter=exporter)
       │    └─ for each step:
       │         ├─ trajectory.append(step_data)
       │         └─ exporter.record_step(snapshot, action, reward)
       └─ exporter.end_episode(won, episode_return)
  # Caller does: exporter.flush()

---

Terminal Commands

cd commander-ai-lab
git pull origin main
# Make edits to ml/training/self_play.py per checklist above
git add ml/training/self_play.py
git commit -m "feat(ml): integrate DecisionExporter into self-play rollouts (#65)"
git push origin main