diff --git a/docs/PHASE1-SELF-PLAY-INTEGRATION-CHECKLIST.md b/docs/PHASE1-SELF-PLAY-INTEGRATION-CHECKLIST.md new file mode 100644 index 0000000..f772668 --- /dev/null +++ b/docs/PHASE1-SELF-PLAY-INTEGRATION-CHECKLIST.md @@ -0,0 +1,93 @@ +# Phase 1 — Self-Play Integration Checklist + +> **Issue:** #65 — PPO Decision Exporter +> **Ref:** `docs/CLOSED-LOOP-DISTILLATION.md` +> **Status:** In Progress + +--- + +## Completed + +- [x] Create `ml/training/decision_exporter.py` with `DecisionExporter` class +- [x] `ExporterConfig` dataclass (output_dir, only_wins, model_version, min_episode_steps) +- [x] `build_decision_record()` — converts PPO step data into Forge-compatible JSONL format +- [x] `DecisionExporter.begin_episode()` / `record_step()` / `end_episode()` / `flush()` lifecycle +- [x] Write to `data/results/ml-decisions-ppo-{batch_id}.jsonl` +- [x] Include PPO metadata fields: `source`, `model_version`, `reward`, `episode_return` +- [x] Win-only filtering (only export decisions from winning games) +- [x] Short-episode gating (`min_episode_steps` threshold) +- [x] `_macro_action_to_forge_type()` mapping (MacroAction enum → Forge action type strings) +- [x] Export stats tracking (`stats` property) + +--- + +## Remaining — Integrate into `ml/training/self_play.py` + +### Import + +- [ ] Add import: `from ml.training.decision_exporter import DecisionExporter` + +### Modify `run_self_play_episode()` + +- [ ] Add `exporter: DecisionExporter = None` parameter to function signature +- [ ] After each `trajectory.append(...)` block (3 locations), add: + ```python + if exporter is not None: + exporter.record_step( + game_state_snapshot=prev_snapshot, + action_idx=action_idx, + reward=reward, + ) + ``` + +### Modify `collect_rollouts()` + +- [ ] Add `exporter: DecisionExporter = None` parameter to function signature +- [ ] Before each episode, call: + ```python + if exporter is not None: + exporter.begin_episode(agent_seat=agent_seat, playstyle=playstyle) + ``` +- [ ] Pass `exporter=exporter` to `run_self_play_episode()` +- [ ] After outcome tracking, call: + ```python + if exporter is not None: + won = trajectory and trajectory[-1]["done"] and trajectory[-1]["reward"] > 0.5 + ep_return = sum(s["reward"] for s in trajectory) + exporter.end_episode(won=won, episode_return=ep_return) + ``` + +### Commit + +- [ ] `git add ml/training/self_play.py` +- [ ] `git commit -m "feat(ml): integrate DecisionExporter into self-play rollouts (#65)"` +- [ ] `git push origin main` + +--- + +## Integration Pattern + +``` +collect_rollouts(exporter=exporter) + └─ for each episode: + ├─ exporter.begin_episode(seat, playstyle) + ├─ run_self_play_episode(..., exporter=exporter) + │ └─ for each step: + │ ├─ trajectory.append(step_data) + │ └─ exporter.record_step(snapshot, action, reward) + └─ exporter.end_episode(won, episode_return) + # Caller does: exporter.flush() +``` + +--- + +## Terminal Commands + +```bash +cd commander-ai-lab +git pull origin main +# Make edits to ml/training/self_play.py per checklist above +git add ml/training/self_play.py +git commit -m "feat(ml): integrate DecisionExporter into self-play rollouts (#65)" +git push origin main +```