docs: add closed-loop iterative distillation implementation roadmap

docs/CLOSED-LOOP-DISTILLATION.md

@@ -0,0 +1,129 @@
+# Closed-Loop Iterative Distillation — Implementation Roadmap
+
+> **Goal:** Create a feedback loop where the PPO self-play agent exports its game decisions
+> as JSONL files, which the supervised training pipeline incorporates alongside Forge
+> heuristic data. The supervised baseline gradually improves as the RL agent discovers
+> better play patterns — the RL agent teaches the supervised model.
+
+---
+
+## Phase 1 — PPO Decision Exporter
+
+**Goal:** PPO self-play writes decisions in the same JSONL format as Forge batch sims.
+
+- [ ] Add a `DecisionExporter` class in `ml/training/self_play.py` that captures each
+      (state, action, reward, game_context) during PPO rollouts
+- [ ] Write to `data/results/ml-decisions-ppo-{batch_id}.jsonl` using the exact same
+      schema as the Forge `DecisionSnapshot` files
+- [ ] Include metadata fields to distinguish PPO-generated data:
+  - `"source": "ppo"`
+  - `"model_version"`
+  - `"reward"`
+  - `"episode_return"`
+- [ ] Only export decisions from **winning** games initially (higher signal-to-noise)
+
+**Files touched:** `ml/training/self_play.py`, new `ml/training/decision_exporter.py`
+
+---
+
+## Phase 2 — Dataset Builder Source Awareness
+
+**Goal:** `dataset_builder.py` can ingest both Forge and PPO decision files with
+configurable mixing.
+
+- [ ] Update `build_dataset()` to scan for both `ml-decisions-sim-*.jsonl` (Forge) and
+      `ml-decisions-ppo-*.jsonl` (PPO) files
+- [ ] Add a `source_weights` config param — e.g., `{"forge": 1.0, "ppo": 0.5}` — to
+      control sampling ratios so PPO data doesn't overwhelm the Forge baseline early on
+- [ ] Add a `min_reward_threshold` filter for PPO data — only include decisions from
+      games above a certain reward score
+- [ ] Tag each sample with its source for downstream analysis
+
+**Files touched:** `ml/data/dataset_builder.py`, `ml/config/scope.py`
+
+---
+
+## Phase 3 — Quality Gate & Validation
+
+**Goal:** Prevent bad PPO data from degrading the supervised model.
+
+- [ ] Add a validation split in `dataset_builder.py` (hold out 10-15% of Forge data as
+      a fixed eval set)
+- [ ] After each supervised training run, evaluate on the held-out Forge set — if
+      accuracy drops below a threshold compared to the previous model, reject the PPO
+      data batch
+- [ ] Track win rate of the PPO agent — only export decisions once the PPO model exceeds
+      a configurable win rate (e.g., >30% in 4-player pods, above the 25% random baseline)
+- [ ] Log data quality metrics: action distribution entropy, reward distribution of
+      exported games
+
+**Files touched:** `ml/training/trainer.py`, `ml/eval/`, new `ml/data/quality_gate.py`
+
+---
+
+## Phase 4 — Distillation Loop Orchestrator
+
+**Goal:** Automate the full cycle: supervised train -> PPO self-play -> export -> retrain.
+
+- [ ] New `ml/training/distillation_loop.py` that orchestrates:
+  1. Supervised train on current dataset (Forge + any existing PPO data)
+  2. Run N episodes of PPO self-play using the new supervised model as starting policy
+  3. Export winning PPO decisions to JSONL
+  4. Quality gate check
+  5. If passed, merge into dataset and go to step 1
+  6. If failed, discard PPO batch, adjust hyperparameters, retry
+- [ ] Add `max_iterations` and `convergence_check` (stop if win rate plateaus across
+      iterations)
+- [ ] Each iteration is a "generation" — tag data and models with generation number
+
+**Files touched:** new `ml/training/distillation_loop.py`, `ml/config/scope.py`
+
+---
+
+## Phase 5 — UI Integration & Monitoring
+
+**Goal:** Expose the distillation loop in the Commander AI Lab web UI.
+
+- [ ] Add a "Distillation" tab or section in `ui/ai-lab.js` showing:
+  - Current generation number
+  - Per-generation win rate trend chart
+  - Data composition pie chart (% Forge vs % PPO per generation)
+  - Start / stop / pause controls for the loop
+- [ ] API endpoints in `routes/ml.py`:
+  - `POST /api/ml/distill/start`
+  - `GET /api/ml/distill/status`
+  - `POST /api/ml/distill/stop`
+- [ ] History of each generation's metrics stored as JSON for the trends view
+
+**Files touched:** `routes/ml.py`, `ui/ai-lab.js`, `ui/ai-lab.css`
+
+---
+
+## Phase 6 — Advanced: ELO Tracking & Model Tournament
+
+**Goal:** Quantify improvement across generations.
+
+- [ ] Pit each generation's model against previous generations and the Forge heuristic
+      in batch sims
+- [ ] Compute ELO ratings per generation
+- [ ] Store in `data/elo_history.json` and visualize in the UI as a line chart
+- [ ] Use ELO delta as a convergence signal — stop the loop when ELO gain per generation
+      drops below a threshold
+
+**Files touched:** new `ml/eval/elo_tracker.py`, `routes/ml.py`, `ui/ai-lab.js`
+
+---
+
+## Priority
+
+| Phase | Priority | Description |
+|-------|----------|-------------|
+| 1     | **Critical** | PPO decision export — foundation for everything |
+| 2     | **Critical** | Dataset builder ingests both sources |
+| 3     | **Critical** | Quality gate prevents model regression |
+| 4     | High     | Automation of the full loop |
+| 5     | Medium   | UI visibility and controls |
+| 6     | Low      | ELO tracking and tournament system |
+
+> **Phases 1-3** are the core — get those working and you have a functional closed loop.
+> **Phase 4** automates it. **Phases 5-6** are polish and observability.