docs: rewrite README to reflect full current feature set

- anthropic_client.py: all 4 methods (chat_structured, chat_plain, achat_structured, achat_plain) now use .messages.stream() context manager
- coach_service.py: inline aclient.messages.create() replaced with async streaming via aclient.messages.stream()
- _parse_response() updated to accept pre-extracted text string (since stream gives us text directly)
- check_status() left as non-streaming (max_tokens=10, safe from the 10min rule)

README.md

@@ -1,381 +1,180 @@
 # Commander AI Lab
 
-Headless batch simulation environment for 3-AI MTG Commander pods.
-Run 100+ games, analyze win rates, and stress-test your decks — all from a web UI or CLI.
+A full-stack MTG Commander AI research platform. Simulate hundreds of games overnight, train ML models on the results, and let simulation data automatically shape the AI deck builder — all from a web UI or CLI.
+
+## What It Does
+
+| Layer | What it is |
+|---|---|
+| **Monte Carlo Engine** | Pure-Python headless sim — runs thousands of games in minutes, no JVM needed |
+| **Forge Engine** | Java/Forge rules-complete sim for ground-truth validation |
+| **Overnight Runner** | Unattended batch runner — runs sims, updates sim weights, trains neural net |
+| **ML Pipeline** | Outcome-weighted RL nudges card scoring weights; neural net trains on decision logs |
+| **Deck Builder** | 7-step LLM pipeline (EDHrec → Scryfall → Ollama) now injected with sim-learned insights |
+| **AI Coach** | LLM-powered coaching sessions (Anthropic/Ollama) with streaming support |
+| **Web UI** | React + FastAPI dashboard for decks, sims, training, and deck building |
 
 ## Architecture
 
 ```
-┌─────────────────────────────────────────────────────────────┐
-│  Web Browser (ai-lab.js + ai-lab.css)                       │
-│    Select decks → Start batch → Poll progress → View stats  │
-└─────────────────────┬───────────────────────────────────────┘
-                      │ HTTP (REST)
-┌─────────────────────┴───────────────────────────────────────┐
-│  Python FastAPI Server (lab_api.py, port 8080)              │
-│    /api/lab/start  — spawn Java batch runner subprocess     │
-│    /api/lab/status — poll progress (parses stdout)          │
-│    /api/lab/result — return completed JSON results          │
-│    /api/lab/decks  — list decks from Forge user data        │
-│    /api/lab/history— browse past batch runs                 │
-│    /api/lab/log    — stream live console output             │
-└─────────────────────┬───────────────────────────────────────┘
-                      │ subprocess (java -jar)
-┌─────────────────────┴───────────────────────────────────────┐
-│  Java CLI (LabCli.java + BatchRunner + MultiThreadRunner)   │
-│    Parses Forge sim output → GameResult objects → JSON      │
-└─────────────────────┬───────────────────────────────────────┘
-                      │ subprocess per game
-┌─────────────────────┴───────────────────────────────────────┐
-│  Card Forge Engine (forge-gui-desktop sim mode)             │
-│    java -jar forge.jar sim -d "deck1" "deck2" "deck3"       │
-│    -f commander -n 1 -q                                     │
-└─────────────────────────────────────────────────────────────┘
+┌─────────────────────────────────────────────────────────────────┐
+│  Web UI  (React/TypeScript — frontend/)                         │
+│    Deck Builder · Sim Dashboard · Training · Coach              │
+└────────────────────────┬────────────────────────────────────────┘
+                         │ HTTP/REST + WebSocket
+┌────────────────────────┴────────────────────────────────────────┐
+│  FastAPI Server  (lab_api.py / routes/ — port 8080)             │
+│    /api/lab/*   /api/ml/*   /api/coach/*   /api/deck-builder/*  │
+└────────┬──────────────────┬──────────────────┬──────────────────┘
+         │                  │                  │
+   Java subprocess    Python sim engine    Ollama (local LLM)
+   (Forge rules)      (Monte Carlo)        qwen2.5:7b + others
+         │                  │
+┌────────┴───────┐  ┌───────┴────────────────────────────────────┐
+│  Forge Engine  │  │  src/commander_ai_lab/                      │
+│  (rules-       │  │    sim/   — engine, models, rules           │
+│   complete)    │  │    deck_builder/ — 7-step LLM pipeline      │
+└────────────────┘  │    coach/ — streaming LLM coach             │
+                    └────────────────────────────────────────────┘
 ```
 
 ## Quick Start (Windows)
 
-### Prerequisites
-- **Java 17+** (`java -version`)
-- **Python 3.10+** (`python --version`)
-- **Maven 3.8+** (`mvn --version`)
-- **Forge repo** cloned and built at `D:\ForgeCommander\forge-repo`
+**Prerequisites:** Java 17+, Python 3.10+, Maven 3.8+, [Forge](https://github.com/Card-Forge/forge) built at `D:\ForgeCommander\forge-repo`
 
-### 1. Build Forge (one-time)
 ```powershell
+# 1. Build Forge (one-time)
 cd D:\ForgeCommander\forge-repo
 mvn package -pl forge-gui-desktop "-DskipTests" "-Drevision=2.0.12-SNAPSHOT" -am
-```
 
-### 2. Build Commander AI Lab
-```powershell
+# 2. Build Commander AI Lab
 cd D:\ForgeCommander\commander-ai-lab
 mvn package "-DskipTests"
-```
-This produces `target/commander-ai-lab-1.0.0-SNAPSHOT.jar` (shaded fat JAR).
 
-### 3. Install Python dependencies
-```powershell
-pip install fastapi uvicorn
-```
+# 3. Install Python deps
+pip install -e .
 
-### 4. Launch the Web UI
-**Option A — One-click launcher:**
-```powershell
+# 4. Launch
 .\start-lab.bat
+# → open http://localhost:8080
 ```
-Edit the paths at the top of `start-lab.bat` if needed.
 
-**Option B — Manual launch:**
+## Overnight AFK Runner
+
+Runs batch sims unattended, then updates card weights and trains the neural net:
+
 ```powershell
-python lab_api.py `
-  --forge-jar "D:\ForgeCommander\forge-repo\forge-gui-desktop\target\forge-gui-desktop-2.0.12-SNAPSHOT-jar-with-dependencies.jar" `
-  --forge-dir "D:\ForgeCommander\forge-repo\forge-gui" `
-  --port 8080
+python overnight-run.py                          # default: 11.5h DeepSeek sims
+python overnight-run.py --batches 20 --games 50  # fixed batch count
+python overnight-run.py --no-update-weights      # skip weight update
+python overnight-run.py --no-train               # skip neural net training
 ```
 
-Open `http://localhost:8080` — select 3 decks, set game count, click **Run AI Sims**.
+**What happens after sims complete:**
+1. `update_weights.py` — nudges `learned_weights.json` from decision logs (feeds deck builder)
+2. `/api/ml/train` — trains the neural net on the full decision dataset
+
+## Sim → Deck Builder Feedback Loop
+
+Simulation outcomes automatically improve deck recommendations:
+
+```
+Overnight sims → decisions_*.jsonl
+       ↓
+update_weights.py → learned_weights.json
+       ↓
+sim_insights.py → natural-language insight block
+       ↓
+Ollama prompts (suggest_cards + enforce_deck_ratios)
+       ↓
+Better card recommendations
+```
+
+When active, `BuildResult.sources_consulted` includes `"sim_weights"`. No-op until real sim data exists.
+
+## Deck Builder Pipeline
+
+The `POST /api/deck-builder/build` endpoint runs 7 steps:
+
+1. Resolve commander via Scryfall
+2. Fetch EDHrec recommendations
+3. Fetch Scryfall candidates by category (ramp, removal, draw, lands)
+4. Ollama suggests synergy/protection/wincon cards ← **sim insights injected here**
+5. Filter by color identity, collection, ban list
+6. Ollama enforces ratios (trim/expand to 99) ← **sim insights injected here**
+7. Assemble final deck
+
+## ML Pipeline
 
-### 5. CLI-only usage (no web server)
 ```powershell
-cd D:\ForgeCommander\forge-repo\forge-gui
+# Update sim card weights from decision logs (runs automatically in overnight-run.py)
+python -m ml.scripts.update_weights --min-games 10 --lr 0.05 --dry-run
 
-java -jar "D:\ForgeCommander\commander-ai-lab\target\commander-ai-lab-1.0.0-SNAPSHOT.jar" `
-  --forge-jar "..\forge-gui-desktop\target\forge-gui-desktop-2.0.12-SNAPSHOT-jar-with-dependencies.jar" `
-  --forge-dir "." `
-  --deck1 "Edgar Markov" `
-  --deck2 "Grimgrin" `
-  --deck3 "Xyris" `
-  --games 100 `
-  --threads 4 `
-  --output results/batch-100.json
+# Reset weights to defaults
+python -m ml.scripts.update_weights --reset
+
+# Generate synthetic training data
+python ml/scripts/generate_synthetic_data.py
 ```
 
-## Scaling Tests
+Learned weights are clamped to `[-20.0, 20.0]` and stored in `src/commander_ai_lab/sim/learned_weights.json`.
 
-### Single-threaded baseline (10 games)
-```powershell
-java -jar target\commander-ai-lab-1.0.0-SNAPSHOT.jar `
-  --forge-jar "..." --forge-dir "..." `
-  --deck1 "Edgar Markov" --deck2 "Grimgrin" --deck3 "Xyris" `
-  --games 10 --threads 1 --output results/baseline-10.json
+## Python Simulator API
+
+```python
+from commander_ai_lab.lab.experiments import build_deck, run_simulation
+
+deck = build_deck(["Sol Ring", "Cultivate", "Lightning Bolt"] + ["Forest"] * 36)
+stats = run_simulation(deck, num_games=100, name_a="My Deck")
+print(f"Win rate: {stats['win_rate']:.1%}, Avg turns: {stats['avg_turns']:.1f}")
 ```
 
-### Multi-threaded (10 games, 2 threads)
-```powershell
-java -jar target\commander-ai-lab-1.0.0-SNAPSHOT.jar `
-  --forge-jar "..." --forge-dir "..." `
-  --deck1 "Edgar Markov" --deck2 "Grimgrin" --deck3 "Xyris" `
-  --games 10 --threads 2 --output results/mt-10-t2.json
-```
-
-### Scale to 100 games (4 threads)
-```powershell
-java -jar target\commander-ai-lab-1.0.0-SNAPSHOT.jar `
-  --forge-jar "..." --forge-dir "..." `
-  --deck1 "Edgar Markov" --deck2 "Grimgrin" --deck3 "Xyris" `
-  --games 100 --threads 4 --output results/scale-100-t4.json
-```
-
-### Maximum throughput test (100 games, 8 threads)
-```powershell
-java -jar target\commander-ai-lab-1.0.0-SNAPSHOT.jar `
-  --forge-jar "..." --forge-dir "..." `
-  --deck1 "Edgar Markov" --deck2 "Grimgrin" --deck3 "Xyris" `
-  --games 100 --threads 8 --seed 42 --output results/scale-100-t8-seed42.json
-```
-
-### Performance comparison script
-Run different thread counts and compare sims/sec:
-```powershell
-# Baseline: 1 thread
-java -jar target\commander-ai-lab-1.0.0-SNAPSHOT.jar --forge-jar "..." --forge-dir "..." --deck1 "Edgar Markov" --deck2 "Grimgrin" --deck3 "Xyris" --games 20 --threads 1 --seed 42 --output results/perf-t1.json
-
-# 2 threads
-java -jar target\commander-ai-lab-1.0.0-SNAPSHOT.jar --forge-jar "..." --forge-dir "..." --deck1 "Edgar Markov" --deck2 "Grimgrin" --deck3 "Xyris" --games 20 --threads 2 --seed 42 --output results/perf-t2.json
-
-# 4 threads
-java -jar target\commander-ai-lab-1.0.0-SNAPSHOT.jar --forge-jar "..." --forge-dir "..." --deck1 "Edgar Markov" --deck2 "Grimgrin" --deck3 "Xyris" --games 20 --threads 4 --seed 42 --output results/perf-t4.json
-
-# 8 threads
-java -jar target\commander-ai-lab-1.0.0-SNAPSHOT.jar --forge-jar "..." --forge-dir "..." --deck1 "Edgar Markov" --deck2 "Grimgrin" --deck3 "Xyris" --games 20 --threads 8 --seed 42 --output results/perf-t8.json
-```
-
-Then compare the `simsPerSecond` values in each output JSON to see scaling behavior.
+The Monte Carlo engine models: combat keywords (flying, trample, deathtouch, lifelink, menace, reach, haste), summoning sickness, tapped/untap state, commander zone, multiplayer attack targeting, and board wipes. It does **not** model the stack, instants, or continuous type-changing effects.
 
 ## API Reference
 
 | Endpoint | Method | Description |
 |---|---|---|
 | `/api/lab/decks` | GET | List available Commander decks |
-| `/api/lab/start` | POST | Start a batch run (returns batchId) |
-| `/api/lab/status?batchId=X` | GET | Poll progress for a running batch |
-| `/api/lab/result?batchId=X` | GET | Get completed batch result JSON |
-| `/api/lab/log?batchId=X` | GET | Get live log lines from running batch |
-| `/api/lab/history` | GET | List past batch result files |
-
-### POST /api/lab/start
-```json
-{
-  "decks": ["Edgar Markov", "Grimgrin", "Xyris"],
-  "numGames": 100,
-  "threads": 4,
-  "seed": null,
-  "clock": 120
-}
-```
-
-### GET /api/lab/status response
-```json
-{
-  "batchId": "abc123",
-  "running": true,
-  "completed": 42,
-  "total": 100,
-  "threads": 4,
-  "elapsedMs": 156000,
-  "error": null
-}
-```
-
-## Python Simulator (Monte Carlo Engine)
-
-A pure-Python deck simulator for rapid Monte Carlo analysis — no Forge JVM required.
-Ported from the [mtg-commander-lan](https://github.com/) Node.js simulator.
-
-### Location
-
-```
-src/commander_ai_lab/
-├── __init__.py
-├── sim/                         # Core simulation engine
-│   ├── __init__.py              # Public exports
-│   ├── models.py                # Card, Player, SimState, GameResult, PlayerStats
-│   ├── engine.py                # GameEngine — turn loop, combat (flying/trample/deathtouch/lifelink/menace/reach)
-│   └── rules.py                 # AI_DEFAULT_WEIGHTS, enrich_card(), score_card(), parse_decklist()
-└── lab/                         # High-level experiment helpers
-    ├── __init__.py
-    └── experiments.py           # build_deck(), run_single_game(), run_simulation(), _cli_main()
-```
-
-### Install as editable package
-
-```powershell
-cd D:\ForgeCommander\commander-ai-lab
-pip install -e .
-```
-
-This installs `commander_ai_lab` as an importable package and registers the `commander-sim` CLI command.
-
-### Quick smoke test
-
-```powershell
-python -m commander_ai_lab.lab.experiments
-```
-
-Expected output — a 10-game Monte Carlo run of a sample "Korvold" deck:
-
-```
-══════════════════════════════════════════
-  Commander AI Lab — Experiment Runner
-══════════════════════════════════════════
-
---- Demo: Korvold Aristocrats (10 games) ---
-
-  Win rate:          XX.X%
-  Avg turns:         XX.X
-  Avg damage dealt:  XX.X
-  Avg spells cast:   XX.X
-  Avg creatures:     XX.X
-  Elapsed:           0.02s
-
-  Done.
-```
-
-### Python API
-
-```python
-from commander_ai_lab.sim.models import Card
-from commander_ai_lab.sim.engine import GameEngine
-from commander_ai_lab.lab.experiments import build_deck, run_simulation
-
-# Build a deck from card names
-deck = build_deck(["Sol Ring", "Cultivate", "Lightning Bolt", "Forest"] + ["Forest"] * 36)
-
-# Monte Carlo simulation (100 games)
-stats = run_simulation(deck, num_games=100, name_a="My Deck")
-print(f"Win rate: {stats['win_rate']:.1%}")
-print(f"Avg turns: {stats['avg_turns']:.1f}")
-```
-
-Key functions:
-
-| Function | Description |
-|---|---|
-| `build_deck(card_names)` | Create enriched `Card` list from names |
-| `build_deck_from_text(decklist)` | Parse `"1 Sol Ring\n37 Forest"` format |
-| `run_single_game(deck_a, deck_b)` | One game, returns winner/turns/stats |
-| `run_simulation(deck_a, deck_b, n)` | Monte Carlo over `n` games, aggregated stats |
-
-### CLI entry point
-
-After `pip install -e .`:
-
-```powershell
-commander-sim
-```
-
-Runs the same demo as `python -m commander_ai_lab.lab.experiments`.
-
-### How the engine works
-
-1. **Deck enrichment** — `enrich_card()` infers type, CMC, keywords, and power/toughness from card name heuristics when Scryfall data is unavailable.
-2. **AI scoring** — `score_card()` uses weighted categories (removal, ramp, draw, aggro, etc.) to pick the best play each turn.
-3. **Game loop** — Each turn: untap → draw → play lands → cast spells (AI scored) → combat (attack/block decisions) → end step. Full keyword support for flying, trample, deathtouch, lifelink, menace, and reach.
-4. **Monte Carlo** — `run_simulation()` runs N independent games and aggregates win rate, avg turns, damage dealt/received, spells cast, creatures played, removal used, ramp played, cards drawn, and max board size.
+| `/api/lab/start` | POST | Start Forge batch (returns batchId) |
+| `/api/lab/start-deepseek` | POST | Start Python/DeepSeek batch |
+| `/api/lab/status?batchId=X` | GET | Poll batch progress |
+| `/api/lab/result?batchId=X` | GET | Get completed batch JSON |
+| `/api/lab/history` | GET | List past batch runs |
+| `/api/ml/train` | POST | Start neural net training |
+| `/api/ml/train/status` | GET | Poll training progress |
+| `/api/coach/chat` | POST | LLM coaching session (streaming) |
+| `/api/deck-builder/build` | POST | Build a 99-card Commander deck |
 
 ## Project Structure
 
 ```
 commander-ai-lab/
-├── lab_api.py                    # Python FastAPI server (web UI backend)
+├── overnight-run.py              # AFK batch runner
 ├── start-lab.bat                 # Windows one-click launcher
-├── pom.xml                       # Maven build (Java 17, shaded JAR)
-├── pyproject.toml                # Python package config (src/ layout)
-├── ui/
-│   ├── index.html                # Web UI entry point
-│   ├── ai-lab.js                 # UI controller (deck selection, progress, results)
-│   ├── ai-lab.css                # Styles (dark theme, responsive)
-│   ├── collection.html           # Card collection manager
-│   ├── collection.js             # Collection controller
-│   ├── collection.css            # Collection styles
-│   ├── deckbuilder.html          # Deck builder + EDH advisor
-│   ├── deckbuilder.js            # Deck builder controller
-│   └── deckbuilder.css           # Deck builder styles
-├── src/commander_ai_lab/         # Python simulator package
-│   ├── __init__.py
-│   ├── sim/                      # Core engine (models, engine, rules)
-│   └── lab/                      # Experiment helpers
-├── src/main/java/commanderailab/ # Java batch runner (Forge integration)
-│   ├── cli/LabCli.java           # CLI entry point (picocli)
-│   ├── batch/
-│   │   ├── BatchRunner.java      # Single-thread runner (parses Forge output)
-│   │   └── MultiThreadBatchRunner.java  # Multi-thread runner (ExecutorService)
-│   ├── bridge/WebApiBridge.java  # Java-side web API bridge
-│   ├── schema/
-│   │   ├── BatchResult.java      # Data model POJOs
-│   │   └── JsonExporter.java     # JSON serialization + validation
-│   ├── stats/StatsAggregator.java # Win rates, turn counts, breakdowns
-│   └── ai/
-│       ├── AiPolicy.java         # Policy interface
-│       └── ForgeBuiltinPolicy.java # v1 Forge AI wrapper
-├── scanner/                      # Card scanner (OCR pipeline)
-│   ├── __init__.py
-│   ├── ocr.py                    # Tesseract OCR wrapper
-│   ├── pipeline.py               # Image preprocessing pipeline
-│   └── preprocess.py             # Image preprocessing helpers
-├── docs/
-│   ├── PROJECT_VISION.md
-│   ├── batch-result-schema.json
-│   └── game-log-schema.json
-├── sample-decks/                 # 3 sample .dck files
-├── precon-decks/                 # Preconstructed decks + index
-├── results/                      # Batch output JSON files (gitignored)
-└── src/test/java/                # Unit tests
+├── lab_api.py                    # FastAPI server entry point
+├── frontend/                     # React + TypeScript web UI
+├── ui/                           # Legacy static HTML UI
+├── src/commander_ai_lab/
+│   ├── sim/                      # Monte Carlo engine (engine, models, rules)
+│   ├── deck_builder/             # LLM deck builder pipeline
+│   │   ├── api/                  # Scryfall, EDHrec, Ollama, sim_insights
+│   │   ├── core/                 # Models, rules engine, assembler
+│   │   └── pipeline/             # build_deck.py (7-step orchestrator)
+│   └── coach/                    # LLM coaching (Anthropic + Ollama)
+├── ml/
+│   ├── scripts/                  # update_weights.py, ml_cli.py
+│   ├── training/                 # Neural net trainer
+│   ├── encoder/                  # Feature encoding
+│   └── models/                   # Saved model checkpoints
+├── scanner/                      # Card OCR pipeline
+├── logs/decisions/               # ML decision logs (gitignored)
+└── src/main/java/commanderailab/ # Java/Forge batch runner
 ```
 
-## Output Format
-
-Each batch run produces JSON conforming to `docs/batch-result-schema.json`:
-```json
-{
-  "metadata": {
-    "schemaVersion": "1.0.0",
-    "batchId": "uuid",
-    "timestamp": "2026-03-06T...",
-    "totalGames": 100,
-    "completedGames": 100,
-    "format": "commander",
-    "podSize": 3,
-    "engineVersion": "forge-2.0.12-SNAPSHOT",
-    "masterSeed": 42,
-    "threads": 4,
-    "elapsedMs": 260000
-  },
-  "decks": [...],
-  "games": [...],
-  "summary": {
-    "perDeck": [
-      { "deckName": "Edgar Markov", "winRate": 0.60, "wins": 60, ... }
-    ],
-    "avgGameTurns": 12.3,
-    "simsPerSecond": 0.38
-  }
-}
-```
-
-## Frontend Development
-
-The UI is a React + TypeScript SPA in the `frontend/` directory.
-
-### Dev server (with hot reload)
-```bash
-cd frontend
-npm install
-npm run dev
-```
-Vite proxies `/api` and `/ws` to `localhost:8000`, so start the FastAPI backend first.
-
-### Production build
-```bash
-cd frontend
-npm run build
-```
-FastAPI serves the built assets from `frontend/dist/`.
-
 ## Known Limitations
-- Winner's life total not captured (Forge quiet mode only reports losers)
-- Win condition classification is heuristic-based on Forge loss reason strings
-- Each game spawns a new JVM process (~2-3s overhead per game)
-- Max throughput ~0.04 sims/sec single-threaded (Forge sim is CPU-heavy)
-- Multithreading spawns parallel subprocesses — throughput scales linearly with threads up to CPU core count
+
+- **Monte Carlo engine** — no stack, no instants, no continuous type changes (animated lands don't attack)
+- **Forge engine** — each game spawns a new JVM (~2-3s overhead), ~0.04 sims/sec single-threaded
+- **Forge winner life total** not captured (quiet mode only reports losers)
+- **Deck builder** — Ollama suggestions are only as good as the local model; `qwen2.5:7b` works well

coach/clients/anthropic_client.py

@@ -4,6 +4,9 @@ Commander AI Lab — Anthropic Claude Structured Output Client
 Uses the anthropic SDK to call Claude's messages endpoint
 with structured JSON output enforced via system prompt.
 Supports both sync and native async operation.
+
+All substantive calls use .messages.stream() to avoid the SDK's
+10-minute non-streaming timeout restriction on large max_tokens.
 """
 import json
 import logging
@@ -53,6 +56,8 @@ class AnthropicClient:
     Anthropic Claude API client.
 
     Drop-in replacement for PerplexityClient.
+    All substantive calls use .messages.stream() so the SDK never
+    enforces the 10-minute non-streaming timeout.
 
     Usage:
         client = AnthropicClient(api_key="sk-ant-...")
@@ -101,37 +106,24 @@ class AnthropicClient:
     ) -> AnthropicResponse:
         """
         Send a chat request expecting JSON output.
-
-        The system_prompt should instruct the model to return JSON matching
-        the expected schema. Claude Opus follows JSON instructions reliably.
-
-        Args:
-            system_prompt: System message (should instruct JSON output)
-            user_prompt: User message
-            json_schema: (optional) Kept for call-site compatibility; appended
-                         as a schema hint in the system prompt when provided
-            schema_name: (optional) Label used in the schema hint
-            model: Override model
-            temperature: Sampling temperature (0.0-1.0)
-            max_tokens: Max output tokens
-
-        Returns:
-            AnthropicResponse with parsed_json extracted from model output
+        Uses streaming to avoid the Anthropic SDK 10-minute timeout restriction.
         """
         use_model = model or self.model
         effective_system = self._build_system(system_prompt, json_schema, schema_name)
 
         try:
-            response = self._client.messages.create(
+            with self._client.messages.stream(
                 model=use_model,
                 system=effective_system,
                 messages=[{"role": "user", "content": user_prompt}],
                 temperature=temperature,
                 max_tokens=max_tokens,
-            )
-            return self._parse_response(response, use_model)
+            ) as stream:
+                content = stream.get_final_text()
+                final_msg = stream.get_final_message()
+            return self._parse_streamed(content, final_msg, use_model)
         except Exception as e:
-            logger.error("[anthropic] API call failed: %s", e)
+            logger.error("[anthropic] chat_structured failed: %s", e)
             raise
 
     def chat_plain(
@@ -144,21 +136,23 @@ class AnthropicClient:
     ) -> AnthropicResponse:
         """
         Send a plain chat request without strict structured output.
-        Used for substitution fallback queries.
+        Uses streaming to avoid the Anthropic SDK 10-minute timeout restriction.
         """
         use_model = model or self.model
 
         try:
-            response = self._client.messages.create(
+            with self._client.messages.stream(
                 model=use_model,
                 system=system_prompt,
                 messages=[{"role": "user", "content": user_prompt}],
                 temperature=temperature,
                 max_tokens=max_tokens,
-            )
-            return self._parse_response(response, use_model)
+            ) as stream:
+                content = stream.get_final_text()
+                final_msg = stream.get_final_message()
+            return self._parse_streamed(content, final_msg, use_model)
         except Exception as e:
-            logger.error("[anthropic] Plain chat failed: %s", e)
+            logger.error("[anthropic] chat_plain failed: %s", e)
             raise
 
     async def achat_structured(
@@ -171,21 +165,23 @@ class AnthropicClient:
         temperature: float = 0.2,
         max_tokens: int = 8192,
     ) -> AnthropicResponse:
-        """Native async structured chat — no run_in_executor needed."""
+        """Native async structured chat using streaming."""
         use_model = model or self.model
         effective_system = self._build_system(system_prompt, json_schema, schema_name)
 
         try:
-            response = await self._async_client.messages.create(
+            async with self._async_client.messages.stream(
                 model=use_model,
                 system=effective_system,
                 messages=[{"role": "user", "content": user_prompt}],
                 temperature=temperature,
                 max_tokens=max_tokens,
-            )
-            return self._parse_response(response, use_model)
+            ) as stream:
+                content = await stream.get_final_text()
+                final_msg = await stream.get_final_message()
+            return self._parse_streamed(content, final_msg, use_model)
         except Exception as e:
-            logger.error("[anthropic] Async structured chat failed: %s", e)
+            logger.error("[anthropic] achat_structured failed: %s", e)
             raise
 
     async def achat_plain(
@@ -196,24 +192,29 @@ class AnthropicClient:
         temperature: float = 0.2,
         max_tokens: int = 4096,
     ) -> AnthropicResponse:
-        """Native async plain chat."""
+        """Native async plain chat using streaming."""
         use_model = model or self.model
 
         try:
-            response = await self._async_client.messages.create(
+            async with self._async_client.messages.stream(
                 model=use_model,
                 system=system_prompt,
                 messages=[{"role": "user", "content": user_prompt}],
                 temperature=temperature,
                 max_tokens=max_tokens,
-            )
-            return self._parse_response(response, use_model)
+            ) as stream:
+                content = await stream.get_final_text()
+                final_msg = await stream.get_final_message()
+            return self._parse_streamed(content, final_msg, use_model)
         except Exception as e:
-            logger.error("[anthropic] Async plain chat failed: %s", e)
+            logger.error("[anthropic] achat_plain failed: %s", e)
             raise
 
     def check_status(self) -> dict:
-        """Quick health check — try a minimal API call."""
+        """
+        Quick health check — tiny max_tokens=10 call, safe to use non-streaming
+        since it will never approach the 10-minute threshold.
+        """
         try:
             response = self._client.messages.create(
                 model=self.model,
@@ -249,13 +250,9 @@ class AnthropicClient:
             f"{schema_str}"
         )
 
-    def _parse_response(self, response, use_model: str) -> AnthropicResponse:
-        """Extract content and attempt JSON parsing from an Anthropic response."""
-        content = ""
-        if response.content:
-            content = response.content[0].text or ""
-
-        usage = response.usage
+    def _parse_streamed(self, content: str, final_msg, use_model: str) -> AnthropicResponse:
+        """Build an AnthropicResponse from streamed text + final message metadata."""
+        usage = getattr(final_msg, "usage", None)
         prompt_tokens = usage.input_tokens if usage else 0
         completion_tokens = usage.output_tokens if usage else 0
 

coach/coach_service.py

@@ -65,10 +65,8 @@ class CoachService:
 
     def load_deck_report(self, deck_id: str) -> Optional[DeckReport]:
         """Load a deck report from disk."""
-        # Try exact filename first, then case-insensitive search
         report_path = DECK_REPORTS_DIR / f"{deck_id}.json"
         if not report_path.exists():
-            # Try case-insensitive match
             for f in DECK_REPORTS_DIR.glob("*.json"):
                 if re.sub(r'[^a-z0-9]', '', f.stem.lower()) == re.sub(r'[^a-z0-9]', '', deck_id.lower()):
                     report_path = f
@@ -96,12 +94,10 @@ class CoachService:
 
     def save_session(self, session: CoachSession) -> Path:
         """Persist a coaching session to disk and SQLite."""
-        # 1. Write JSON file (backward compat)
         COACH_SESSIONS_DIR.mkdir(parents=True, exist_ok=True)
         path = COACH_SESSIONS_DIR / f"{session.sessionId}.json"
         with open(path, "w") as f:
             f.write(session.model_dump_json(indent=2))
-        # 2. Write to SQLite
         if _DB_AVAILABLE:
             try:
                 _db_save_session({
@@ -124,10 +120,8 @@ class CoachService:
                 logger.warning("Failed to persist session to SQLite: %s", e)
         return path
 
-
     def load_session(self, session_id: str) -> Optional[CoachSession]:
         """Load a coaching session from SQLite, falling back to disk."""
-        # Try SQLite first
         if _DB_AVAILABLE:
             try:
                 row = _db_load_session(session_id)
@@ -150,7 +144,6 @@ class CoachService:
                     )
             except Exception as e:
                 logger.warning("SQLite load failed for %s: %s", session_id, e)
-        # Fallback to disk
         path = COACH_SESSIONS_DIR / f"{session_id}.json"
         if not path.exists():
             return None
@@ -162,10 +155,8 @@ class CoachService:
             logger.error("Failed to load session %s: %s", session_id, e)
             return None
 
-
     def list_sessions(self, deck_id: str = None) -> List[dict]:
         """List coaching sessions from SQLite, falling back to disk."""
-        # Try SQLite first
         if _DB_AVAILABLE:
             try:
                 rows = _db_list_sessions(deck_id)
@@ -173,7 +164,6 @@ class CoachService:
                     return rows
             except Exception as e:
                 logger.warning("SQLite list_sessions failed: %s", e)
-        # Fallback to disk
         if not COACH_SESSIONS_DIR.exists():
             return []
         sessions = []
@@ -195,20 +185,21 @@ class CoachService:
                 continue
         return sessions
 
-
     # ── Main Coaching Pipeline ─────────────────────────────────
 
-    async def run_coaching_session(self, deck_id: str,
-                                    goals: Optional[CoachGoals] = None,
-                                    fallback_report: Optional[DeckReport] = None,
-                                    ) -> CoachSession:
+    async def run_coaching_session(
+        self,
+        deck_id: str,
+        goals: Optional[CoachGoals] = None,
+        fallback_report: Optional[DeckReport] = None,
+    ) -> CoachSession:
         """
         Full coaching pipeline:
         1. Load deck report (or use fallback from DB card list)
         2. Identify underperformers
         3. Find replacement candidates via embeddings
         4. Build prompt
-        5. Call LLM
+        5. Call LLM (streaming for Anthropic to avoid 10-min SDK timeout)
         6. Parse response
         7. Persist session
         """
@@ -224,7 +215,7 @@ class CoachService:
             except Exception as e:
                 logger.warning("Embeddings auto-load failed: %s", e)
 
-        # 1. Load deck report — try simulation report first, then fallback
+        # 1. Load deck report
         report = self.load_deck_report(deck_id)
         if report is None and fallback_report is not None:
             report = fallback_report
@@ -237,20 +228,15 @@ class CoachService:
 
         # 3. Find replacement candidates for underperformers
         candidates: Dict[str, List[dict]] = {}
-
         underperformers = report.underperformers[:MAX_UNDERPERFORMERS]
         if not underperformers:
-            # If no explicit underperformers, pick lowest impact cards
             sorted_cards = sorted(report.cards, key=lambda c: c.impactScore)
             underperformers = [
                 c.name for c in sorted_cards[:MAX_UNDERPERFORMERS]
                 if c.impactScore < UNDERPERFORMER_IMPACT_THRESHOLD
             ]
-
-        # If still no underperformers and we have cards, pick bottom N by impact
         if not underperformers and report.cards:
             sorted_cards = sorted(report.cards, key=lambda c: c.impactScore)
-            # Take the bottom 5 non-land cards as candidates for cuts
             underperformers = [
                 c.name for c in sorted_cards[:5]
                 if not any(t in (c.tags or []) for t in [])
@@ -273,29 +259,33 @@ class CoachService:
         system_prompt = build_system_prompt(report, goals)
         user_prompt = build_user_prompt(report, candidates)
 
-        # 5. Call LLM (Anthropic or local Ollama)
+        # 5. Call LLM
         logger.info("Calling LLM for deck: %s (provider=%s)", deck_id, COACH_PROVIDER)
         if COACH_PROVIDER == "anthropic":
-            import anthropic
+            import anthropic as _anthropic
             anthropic_key = ANTHROPIC_API_KEY
             if not anthropic_key:
                 raise ConnectionError("Anthropic API key not configured. Set ANTHROPIC_API_KEY env var.")
-            aclient = anthropic.AsyncAnthropic(api_key=anthropic_key)
-            resp = await aclient.messages.create(
+
+            # Use streaming to avoid the Anthropic SDK 10-minute non-streaming timeout.
+            aclient = _anthropic.AsyncAnthropic(api_key=anthropic_key)
+            async with aclient.messages.stream(
                 model=ANTHROPIC_MODEL,
                 system=system_prompt,
                 messages=[{"role": "user", "content": user_prompt}],
                 max_tokens=DEFAULT_MAX_TOKENS,
                 temperature=0.7,
-            )
-            content = resp.content[0].text if resp.content else ""
-            usage = resp.usage
+            ) as stream:
+                content = await stream.get_final_text()
+                final_msg = await stream.get_final_message()
+
+            usage = getattr(final_msg, "usage", None)
 
             # Build an LLMResponse-compatible object
             from types import SimpleNamespace
             llm_response = SimpleNamespace(
                 content=content,
-                model=resp.model,
+                model=final_msg.model,
                 prompt_tokens=usage.input_tokens if usage else 0,
                 completion_tokens=usage.output_tokens if usage else 0,
                 parsed_json=None,
@@ -305,7 +295,6 @@ class CoachService:
                 cleaned = re.sub(r'\s*```$', '', cleaned.strip())
                 llm_response.parsed_json = json.loads(cleaned)
             except (json.JSONDecodeError, ValueError):
-                # Try to find JSON object in the text
                 match = re.search(r'\{[\s\S]*\}', content)
                 if match:
                     try:
@@ -317,7 +306,6 @@ class CoachService:
 
         # 6. Parse response into CoachSession
         session_id = f"sess-{datetime.now(timezone.utc).strftime('%Y%m%d-%H%M%S')}-{uuid.uuid4().hex[:6]}"
-
         session = CoachSession(
             sessionId=session_id,
             deckId=deck_id,
@@ -331,14 +319,12 @@ class CoachService:
         if llm_response.parsed_json:
             self._populate_session_from_json(session, llm_response.parsed_json)
         else:
-            # Fallback: store raw text
             session.rawTextExplanation = llm_response.content
             session.summary = "LLM returned non-JSON response. See raw explanation."
 
         # 7. Persist
         self.save_session(session)
         logger.info("Coaching session saved: %s", session_id)
-
         return session
 
     def _populate_session_from_json(self, session: CoachSession, data: dict):
@@ -347,11 +333,9 @@ class CoachService:
         session.rawTextExplanation = data.get("rawTextExplanation", "")
         session.manaBaseAdvice = data.get("manaBaseAdvice")
 
-        # Heuristic hints
         hints = data.get("heuristicHints", [])
         session.heuristicHints = hints if isinstance(hints, list) else [str(hints)]
 
-        # Suggested cuts
         for cut_data in data.get("suggestedCuts", []):
             if isinstance(cut_data, dict):
                 session.suggestedCuts.append(SuggestedCut(
@@ -360,7 +344,6 @@ class CoachService:
                     replacementOptions=cut_data.get("replacementOptions", []),
                 ))
 
-        # Suggested adds
         for add_data in data.get("suggestedAdds", []):
             if isinstance(add_data, dict):
                 session.suggestedAdds.append(SuggestedAdd(
@@ -376,7 +359,6 @@ class CoachService:
         """Check the health of all coach subsystems."""
         status = CoachStatus()
 
-        # Check LLM / Anthropic provider
         if COACH_PROVIDER == "anthropic":
             anthropic_key = ANTHROPIC_API_KEY
             if anthropic_key:
@@ -396,11 +378,8 @@ class CoachService:
             if not status.llmConnected:
                 status.error = llm_status.get("error", "Local LLM not reachable")
 
-        # Check embeddings
         status.embeddingsLoaded = self.embeddings.loaded
         status.embeddingCards = self.embeddings.card_count
-
-        # Check deck reports
         status.deckReportsAvailable = len(self.list_deck_reports())
 
         return status

overnight-run.py

@@ -25,6 +25,7 @@ import json
 import logging
 import logging.handlers
 import os
+import subprocess
 import sys
 import time
 import urllib.request
@@ -42,6 +43,9 @@ DEFAULT_CONFIG = {
     "time_limit_hours": 11.5,   # stop after this many hours (leave buffer for training)
     "pause_between_batches": 10,  # seconds to pause between batches
     "auto_train": True,         # trigger ML training after all batches
+    "auto_update_weights": True,  # run update_weights.py after sims
+    "weights_min_games": 10,    # minimum games required for weight update
+    "weights_lr": 0.05,         # learning rate for weight update
     "train_epochs": 100,        # training epochs
     "train_lr": 0.001,          # learning rate
     "train_batch_size": 256,    # training batch size
@@ -229,6 +233,52 @@ def wait_for_batch(base_url, batch_id, engine="deepseek", deadline=None):
         time.sleep(poll_interval)
 
 
+def run_update_weights(cfg):
+    """
+    Run ml.scripts.update_weights to nudge learned_weights.json from sim
+    decision logs.  This feeds the deck builder's sim_insights injection.
+
+    Returns True on success, False on failure (non-fatal — overnight run
+    continues regardless).
+    """
+    log.info("")
+    log.info("── Auto-update sim weights ─────────────────────────────────")
+    log.info("Running update_weights.py to refresh learned_weights.json...")
+
+    cmd = [
+        sys.executable, "-m", "ml.scripts.update_weights",
+        "--min-games", str(cfg["weights_min_games"]),
+        "--lr", str(cfg["weights_lr"]),
+    ]
+
+    try:
+        result = subprocess.run(
+            cmd,
+            capture_output=True,
+            text=True,
+            timeout=120,  # 2 min ceiling — this should complete in seconds
+        )
+        if result.returncode == 0:
+            # Print the last ~800 chars of the weight report to the log
+            output = (result.stdout or "").strip()
+            if output:
+                for line in output.splitlines()[-20:]:
+                    log.info(f"  {line}")
+            log.info("Sim weights updated — deck builder will use new insights on next build.")
+            return True
+        else:
+            log.warning(f"update_weights.py exited with code {result.returncode}")
+            if result.stderr:
+                log.warning(result.stderr.strip()[-300:])
+            return False
+    except subprocess.TimeoutExpired:
+        log.warning("update_weights.py timed out after 120s — skipping")
+        return False
+    except Exception as e:
+        log.warning(f"update_weights.py failed: {e}")
+        return False
+
+
 def start_training(base_url, epochs, lr, batch_size, patience):
     """Trigger ML training and return immediately."""
     resp = api_post(base_url, "/api/ml/train", {
@@ -364,6 +414,7 @@ def run_overnight(cfg):
     else:
         log.info(f"  Mode: Run for {cfg['time_limit_hours']}h, {games_per_batch} games/batch")
     log.info(f"  Engine: {engine}")
+    log.info(f"  Auto-update weights: {'Yes' if cfg['auto_update_weights'] else 'No'}")
     log.info(f"  Auto-train after: {'Yes' if cfg['auto_train'] else 'No'}")
     log.info("")
 
@@ -440,6 +491,17 @@ def run_overnight(cfg):
     log.info(f"  Elapsed time:      {str(timedelta(seconds=int(total_elapsed)))}")
     log.info("")
 
+    # ── Auto-update sim weights ───────────────────────────────
+    # Runs update_weights.py to nudge learned_weights.json from the
+    # decision logs produced during this run.  This feeds sim_insights
+    # into the deck builder's Ollama prompts on the next deck build.
+    # Runs before neural net training so both pipelines benefit from
+    # the freshest possible data.
+    if cfg["auto_update_weights"]:
+        run_update_weights(cfg)
+    else:
+        log.info("Skipping sim weight update (--no-update-weights flag set)")
+
     # ── Auto-train ────────────────────────────────────────────
     if cfg["auto_train"]:
         if decisions_after < 100:
@@ -482,17 +544,17 @@ def main():
         formatter_class=argparse.RawDescriptionHelpFormatter,
         epilog="""
 Examples:
-  # Run DeepSeek sims for 11.5 hours, then train:
+  # Run DeepSeek sims for 11.5 hours, then update weights + train:
   python overnight-run.py
 
-  # Run 20 batches of 50 games each, then train:
+  # Run 20 batches of 50 games each, then update weights + train:
   python overnight-run.py --batches 20 --games 50
 
   # Run for 6 hours with Java engine:
   python overnight-run.py --engine java --hours 6 --decks "Deck A" "Deck B" "Deck C"
 
-  # Just run sims, no training:
-  python overnight-run.py --no-train
+  # Just run sims, skip both weight update and training:
+  python overnight-run.py --no-train --no-update-weights
 
   # Custom training settings:
   python overnight-run.py --epochs 200 --lr 0.0005 --patience 20
@@ -514,6 +576,12 @@ Examples:
                         help="Deck names (auto-detected if not specified)")
     parser.add_argument("--no-train", action="store_true",
                         help="Skip auto-training after sims")
+    parser.add_argument("--no-update-weights", action="store_true",
+                        help="Skip updating learned_weights.json after sims")
+    parser.add_argument("--weights-min-games", type=int, default=10,
+                        help="Minimum games required for weight update (default: 10)")
+    parser.add_argument("--weights-lr", type=float, default=0.05,
+                        help="Learning rate for sim weight update (default: 0.05)")
     parser.add_argument("--epochs", type=int, default=100,
                         help="Training epochs (default: 100)")
     parser.add_argument("--lr", type=float, default=0.001,
@@ -536,6 +604,9 @@ Examples:
     cfg["time_limit_hours"] = args.hours
     cfg["pause_between_batches"] = args.pause
     cfg["auto_train"] = not args.no_train
+    cfg["auto_update_weights"] = not args.no_update_weights
+    cfg["weights_min_games"] = args.weights_min_games
+    cfg["weights_lr"] = args.weights_lr
     cfg["train_epochs"] = args.epochs
     cfg["train_lr"] = args.lr
     cfg["train_batch_size"] = args.batch_size