130 lines
5.2 KiB
Markdown
130 lines
5.2 KiB
Markdown
# Closed-Loop Iterative Distillation — Implementation Roadmap
|
|
|
|
> **Goal:** Create a feedback loop where the PPO self-play agent exports its game decisions
|
|
> as JSONL files, which the supervised training pipeline incorporates alongside Forge
|
|
> heuristic data. The supervised baseline gradually improves as the RL agent discovers
|
|
> better play patterns — the RL agent teaches the supervised model.
|
|
|
|
---
|
|
|
|
## Phase 1 — PPO Decision Exporter
|
|
|
|
**Goal:** PPO self-play writes decisions in the same JSONL format as Forge batch sims.
|
|
|
|
- [ ] Add a `DecisionExporter` class in `ml/training/self_play.py` that captures each
|
|
(state, action, reward, game_context) during PPO rollouts
|
|
- [ ] Write to `data/results/ml-decisions-ppo-{batch_id}.jsonl` using the exact same
|
|
schema as the Forge `DecisionSnapshot` files
|
|
- [ ] Include metadata fields to distinguish PPO-generated data:
|
|
- `"source": "ppo"`
|
|
- `"model_version"`
|
|
- `"reward"`
|
|
- `"episode_return"`
|
|
- [ ] Only export decisions from **winning** games initially (higher signal-to-noise)
|
|
|
|
**Files touched:** `ml/training/self_play.py`, new `ml/training/decision_exporter.py`
|
|
|
|
---
|
|
|
|
## Phase 2 — Dataset Builder Source Awareness
|
|
|
|
**Goal:** `dataset_builder.py` can ingest both Forge and PPO decision files with
|
|
configurable mixing.
|
|
|
|
- [ ] Update `build_dataset()` to scan for both `ml-decisions-sim-*.jsonl` (Forge) and
|
|
`ml-decisions-ppo-*.jsonl` (PPO) files
|
|
- [ ] Add a `source_weights` config param — e.g., `{"forge": 1.0, "ppo": 0.5}` — to
|
|
control sampling ratios so PPO data doesn't overwhelm the Forge baseline early on
|
|
- [ ] Add a `min_reward_threshold` filter for PPO data — only include decisions from
|
|
games above a certain reward score
|
|
- [ ] Tag each sample with its source for downstream analysis
|
|
|
|
**Files touched:** `ml/data/dataset_builder.py`, `ml/config/scope.py`
|
|
|
|
---
|
|
|
|
## Phase 3 — Quality Gate & Validation
|
|
|
|
**Goal:** Prevent bad PPO data from degrading the supervised model.
|
|
|
|
- [ ] Add a validation split in `dataset_builder.py` (hold out 10-15% of Forge data as
|
|
a fixed eval set)
|
|
- [ ] After each supervised training run, evaluate on the held-out Forge set — if
|
|
accuracy drops below a threshold compared to the previous model, reject the PPO
|
|
data batch
|
|
- [ ] Track win rate of the PPO agent — only export decisions once the PPO model exceeds
|
|
a configurable win rate (e.g., >30% in 4-player pods, above the 25% random baseline)
|
|
- [ ] Log data quality metrics: action distribution entropy, reward distribution of
|
|
exported games
|
|
|
|
**Files touched:** `ml/training/trainer.py`, `ml/eval/`, new `ml/data/quality_gate.py`
|
|
|
|
---
|
|
|
|
## Phase 4 — Distillation Loop Orchestrator
|
|
|
|
**Goal:** Automate the full cycle: supervised train -> PPO self-play -> export -> retrain.
|
|
|
|
- [ ] New `ml/training/distillation_loop.py` that orchestrates:
|
|
1. Supervised train on current dataset (Forge + any existing PPO data)
|
|
2. Run N episodes of PPO self-play using the new supervised model as starting policy
|
|
3. Export winning PPO decisions to JSONL
|
|
4. Quality gate check
|
|
5. If passed, merge into dataset and go to step 1
|
|
6. If failed, discard PPO batch, adjust hyperparameters, retry
|
|
- [ ] Add `max_iterations` and `convergence_check` (stop if win rate plateaus across
|
|
iterations)
|
|
- [ ] Each iteration is a "generation" — tag data and models with generation number
|
|
|
|
**Files touched:** new `ml/training/distillation_loop.py`, `ml/config/scope.py`
|
|
|
|
---
|
|
|
|
## Phase 5 — UI Integration & Monitoring
|
|
|
|
**Goal:** Expose the distillation loop in the Commander AI Lab web UI.
|
|
|
|
- [ ] Add a "Distillation" tab or section in `ui/ai-lab.js` showing:
|
|
- Current generation number
|
|
- Per-generation win rate trend chart
|
|
- Data composition pie chart (% Forge vs % PPO per generation)
|
|
- Start / stop / pause controls for the loop
|
|
- [ ] API endpoints in `routes/ml.py`:
|
|
- `POST /api/ml/distill/start`
|
|
- `GET /api/ml/distill/status`
|
|
- `POST /api/ml/distill/stop`
|
|
- [ ] History of each generation's metrics stored as JSON for the trends view
|
|
|
|
**Files touched:** `routes/ml.py`, `ui/ai-lab.js`, `ui/ai-lab.css`
|
|
|
|
---
|
|
|
|
## Phase 6 — Advanced: ELO Tracking & Model Tournament
|
|
|
|
**Goal:** Quantify improvement across generations.
|
|
|
|
- [ ] Pit each generation's model against previous generations and the Forge heuristic
|
|
in batch sims
|
|
- [ ] Compute ELO ratings per generation
|
|
- [ ] Store in `data/elo_history.json` and visualize in the UI as a line chart
|
|
- [ ] Use ELO delta as a convergence signal — stop the loop when ELO gain per generation
|
|
drops below a threshold
|
|
|
|
**Files touched:** new `ml/eval/elo_tracker.py`, `routes/ml.py`, `ui/ai-lab.js`
|
|
|
|
---
|
|
|
|
## Priority
|
|
|
|
| Phase | Priority | Description |
|
|
|-------|----------|-------------|
|
|
| 1 | **Critical** | PPO decision export — foundation for everything |
|
|
| 2 | **Critical** | Dataset builder ingests both sources |
|
|
| 3 | **Critical** | Quality gate prevents model regression |
|
|
| 4 | High | Automation of the full loop |
|
|
| 5 | Medium | UI visibility and controls |
|
|
| 6 | Low | ELO tracking and tournament system |
|
|
|
|
> **Phases 1-3** are the core — get those working and you have a functional closed loop.
|
|
> **Phase 4** automates it. **Phases 5-6** are polish and observability.
|