# Closed-Loop Iterative Distillation — Implementation Roadmap

> **Goal:** Create a feedback loop where the PPO self-play agent exports its game decisions
> as JSONL files, which the supervised training pipeline incorporates alongside Forge
> heuristic data. The supervised baseline gradually improves as the RL agent discovers
> better play patterns — the RL agent teaches the supervised model.

---

## Phase 1 — PPO Decision Exporter

**Goal:** PPO self-play writes decisions in the same JSONL format as Forge batch sims.

- [ ] Add a `DecisionExporter` class in `ml/training/self_play.py` that captures each
      (state, action, reward, game_context) during PPO rollouts
- [ ] Write to `data/results/ml-decisions-ppo-{batch_id}.jsonl` using the exact same
      schema as the Forge `DecisionSnapshot` files
- [ ] Include metadata fields to distinguish PPO-generated data:
  - `"source": "ppo"`
  - `"model_version"`
  - `"reward"`
  - `"episode_return"`
- [ ] Only export decisions from **winning** games initially (higher signal-to-noise)

**Files touched:** `ml/training/self_play.py`, new `ml/training/decision_exporter.py`

---

## Phase 2 — Dataset Builder Source Awareness

**Goal:** `dataset_builder.py` can ingest both Forge and PPO decision files with
configurable mixing.

- [ ] Update `build_dataset()` to scan for both `ml-decisions-sim-*.jsonl` (Forge) and
      `ml-decisions-ppo-*.jsonl` (PPO) files
- [ ] Add a `source_weights` config param — e.g., `{"forge": 1.0, "ppo": 0.5}` — to
      control sampling ratios so PPO data doesn't overwhelm the Forge baseline early on
- [ ] Add a `min_reward_threshold` filter for PPO data — only include decisions from
      games above a certain reward score
- [ ] Tag each sample with its source for downstream analysis

**Files touched:** `ml/data/dataset_builder.py`, `ml/config/scope.py`

---

## Phase 3 — Quality Gate & Validation

**Goal:** Prevent bad PPO data from degrading the supervised model.

- [ ] Add a validation split in `dataset_builder.py` (hold out 10-15% of Forge data as
      a fixed eval set)
- [ ] After each supervised training run, evaluate on the held-out Forge set — if
      accuracy drops below a threshold compared to the previous model, reject the PPO
      data batch
- [ ] Track win rate of the PPO agent — only export decisions once the PPO model exceeds
      a configurable win rate (e.g., >30% in 4-player pods, above the 25% random baseline)
- [ ] Log data quality metrics: action distribution entropy, reward distribution of
      exported games

**Files touched:** `ml/training/trainer.py`, `ml/eval/`, new `ml/data/quality_gate.py`

---

## Phase 4 — Distillation Loop Orchestrator

**Goal:** Automate the full cycle: supervised train -> PPO self-play -> export -> retrain.

- [ ] New `ml/training/distillation_loop.py` that orchestrates:
  1. Supervised train on current dataset (Forge + any existing PPO data)
  2. Run N episodes of PPO self-play using the new supervised model as starting policy
  3. Export winning PPO decisions to JSONL
  4. Quality gate check
  5. If passed, merge into dataset and go to step 1
  6. If failed, discard PPO batch, adjust hyperparameters, retry
- [ ] Add `max_iterations` and `convergence_check` (stop if win rate plateaus across
      iterations)
- [ ] Each iteration is a "generation" — tag data and models with generation number

**Files touched:** new `ml/training/distillation_loop.py`, `ml/config/scope.py`

---

## Phase 5 — UI Integration & Monitoring

**Goal:** Expose the distillation loop in the Commander AI Lab web UI.

- [ ] Add a "Distillation" tab or section in `ui/ai-lab.js` showing:
  - Current generation number
  - Per-generation win rate trend chart
  - Data composition pie chart (% Forge vs % PPO per generation)
  - Start / stop / pause controls for the loop
- [ ] API endpoints in `routes/ml.py`:
  - `POST /api/ml/distill/start`
  - `GET /api/ml/distill/status`
  - `POST /api/ml/distill/stop`
- [ ] History of each generation's metrics stored as JSON for the trends view

**Files touched:** `routes/ml.py`, `ui/ai-lab.js`, `ui/ai-lab.css`

---

## Phase 6 — Advanced: ELO Tracking & Model Tournament

**Goal:** Quantify improvement across generations.

- [ ] Pit each generation's model against previous generations and the Forge heuristic
      in batch sims
- [ ] Compute ELO ratings per generation
- [ ] Store in `data/elo_history.json` and visualize in the UI as a line chart
- [ ] Use ELO delta as a convergence signal — stop the loop when ELO gain per generation
      drops below a threshold

**Files touched:** new `ml/eval/elo_tracker.py`, `routes/ml.py`, `ui/ai-lab.js`

---

## Priority

| Phase | Priority | Description |
|-------|----------|-------------|
| 1     | **Critical** | PPO decision export — foundation for everything |
| 2     | **Critical** | Dataset builder ingests both sources |
| 3     | **Critical** | Quality gate prevents model regression |
| 4     | High     | Automation of the full loop |
| 5     | Medium   | UI visibility and controls |
| 6     | Low      | ELO tracking and tournament system |

> **Phases 1-3** are the core — get those working and you have a functional closed loop.
> **Phase 4** automates it. **Phases 5-6** are polish and observability.