From 3cca49c5a9b8a1dd65e352cfdfca5bc887449bd5 Mon Sep 17 00:00:00 2001 From: KoalaTrapLord <265133236+KoalaTrapLord@users.noreply.github.com> Date: Sat, 21 Mar 2026 17:55:18 -0400 Subject: [PATCH] docs: add closed-loop iterative distillation implementation roadmap --- docs/CLOSED-LOOP-DISTILLATION.md | 129 +++++++++++++++++++++++++++++++ 1 file changed, 129 insertions(+) create mode 100644 docs/CLOSED-LOOP-DISTILLATION.md diff --git a/docs/CLOSED-LOOP-DISTILLATION.md b/docs/CLOSED-LOOP-DISTILLATION.md new file mode 100644 index 0000000..550b02d --- /dev/null +++ b/docs/CLOSED-LOOP-DISTILLATION.md @@ -0,0 +1,129 @@ +# Closed-Loop Iterative Distillation — Implementation Roadmap + +> **Goal:** Create a feedback loop where the PPO self-play agent exports its game decisions +> as JSONL files, which the supervised training pipeline incorporates alongside Forge +> heuristic data. The supervised baseline gradually improves as the RL agent discovers +> better play patterns — the RL agent teaches the supervised model. + +--- + +## Phase 1 — PPO Decision Exporter + +**Goal:** PPO self-play writes decisions in the same JSONL format as Forge batch sims. + +- [ ] Add a `DecisionExporter` class in `ml/training/self_play.py` that captures each + (state, action, reward, game_context) during PPO rollouts +- [ ] Write to `data/results/ml-decisions-ppo-{batch_id}.jsonl` using the exact same + schema as the Forge `DecisionSnapshot` files +- [ ] Include metadata fields to distinguish PPO-generated data: + - `"source": "ppo"` + - `"model_version"` + - `"reward"` + - `"episode_return"` +- [ ] Only export decisions from **winning** games initially (higher signal-to-noise) + +**Files touched:** `ml/training/self_play.py`, new `ml/training/decision_exporter.py` + +--- + +## Phase 2 — Dataset Builder Source Awareness + +**Goal:** `dataset_builder.py` can ingest both Forge and PPO decision files with +configurable mixing. + +- [ ] Update `build_dataset()` to scan for both `ml-decisions-sim-*.jsonl` (Forge) and + `ml-decisions-ppo-*.jsonl` (PPO) files +- [ ] Add a `source_weights` config param — e.g., `{"forge": 1.0, "ppo": 0.5}` — to + control sampling ratios so PPO data doesn't overwhelm the Forge baseline early on +- [ ] Add a `min_reward_threshold` filter for PPO data — only include decisions from + games above a certain reward score +- [ ] Tag each sample with its source for downstream analysis + +**Files touched:** `ml/data/dataset_builder.py`, `ml/config/scope.py` + +--- + +## Phase 3 — Quality Gate & Validation + +**Goal:** Prevent bad PPO data from degrading the supervised model. + +- [ ] Add a validation split in `dataset_builder.py` (hold out 10-15% of Forge data as + a fixed eval set) +- [ ] After each supervised training run, evaluate on the held-out Forge set — if + accuracy drops below a threshold compared to the previous model, reject the PPO + data batch +- [ ] Track win rate of the PPO agent — only export decisions once the PPO model exceeds + a configurable win rate (e.g., >30% in 4-player pods, above the 25% random baseline) +- [ ] Log data quality metrics: action distribution entropy, reward distribution of + exported games + +**Files touched:** `ml/training/trainer.py`, `ml/eval/`, new `ml/data/quality_gate.py` + +--- + +## Phase 4 — Distillation Loop Orchestrator + +**Goal:** Automate the full cycle: supervised train -> PPO self-play -> export -> retrain. + +- [ ] New `ml/training/distillation_loop.py` that orchestrates: + 1. Supervised train on current dataset (Forge + any existing PPO data) + 2. Run N episodes of PPO self-play using the new supervised model as starting policy + 3. Export winning PPO decisions to JSONL + 4. Quality gate check + 5. If passed, merge into dataset and go to step 1 + 6. If failed, discard PPO batch, adjust hyperparameters, retry +- [ ] Add `max_iterations` and `convergence_check` (stop if win rate plateaus across + iterations) +- [ ] Each iteration is a "generation" — tag data and models with generation number + +**Files touched:** new `ml/training/distillation_loop.py`, `ml/config/scope.py` + +--- + +## Phase 5 — UI Integration & Monitoring + +**Goal:** Expose the distillation loop in the Commander AI Lab web UI. + +- [ ] Add a "Distillation" tab or section in `ui/ai-lab.js` showing: + - Current generation number + - Per-generation win rate trend chart + - Data composition pie chart (% Forge vs % PPO per generation) + - Start / stop / pause controls for the loop +- [ ] API endpoints in `routes/ml.py`: + - `POST /api/ml/distill/start` + - `GET /api/ml/distill/status` + - `POST /api/ml/distill/stop` +- [ ] History of each generation's metrics stored as JSON for the trends view + +**Files touched:** `routes/ml.py`, `ui/ai-lab.js`, `ui/ai-lab.css` + +--- + +## Phase 6 — Advanced: ELO Tracking & Model Tournament + +**Goal:** Quantify improvement across generations. + +- [ ] Pit each generation's model against previous generations and the Forge heuristic + in batch sims +- [ ] Compute ELO ratings per generation +- [ ] Store in `data/elo_history.json` and visualize in the UI as a line chart +- [ ] Use ELO delta as a convergence signal — stop the loop when ELO gain per generation + drops below a threshold + +**Files touched:** new `ml/eval/elo_tracker.py`, `routes/ml.py`, `ui/ai-lab.js` + +--- + +## Priority + +| Phase | Priority | Description | +|-------|----------|-------------| +| 1 | **Critical** | PPO decision export — foundation for everything | +| 2 | **Critical** | Dataset builder ingests both sources | +| 3 | **Critical** | Quality gate prevents model regression | +| 4 | High | Automation of the full loop | +| 5 | Medium | UI visibility and controls | +| 6 | Low | ELO tracking and tournament system | + +> **Phases 1-3** are the core — get those working and you have a functional closed loop. +> **Phase 4** automates it. **Phases 5-6** are polish and observability.