Commit Graph
572 Commits
Author SHA1 Message Date
KoalaTrapLord bbca0b7985 fix(#126): add batch insert retry logic with exponential backoff
- Add BATCH_RETRY_LIMIT and BATCH_RETRY_DELAY constants after DEFAULT_BATCH_SIZE
- Add _insert_batch_with_retry() helper with exponential backoff
- Replace both bare try/except insert blocks in _build_collection_from_bulk
  with calls to the retry helper so transient ChromaDB errors are retried
  up to BATCH_RETRY_LIMIT times before a batch is counted as failed
2026-03-29 04:24:58 -04:00
KoalaTrapLord da8de77704 feat(bridge): add buildStateVector() emitting state_vector float[] in snapshots (#83 Step 1)
Adds a buildStateVector(int seat) method to GameSession that produces
a float[29] global scalar vector matching STATE_DIMS layout in
ml/config/scope.py:

  Per-player block (14 features × 2 players = 28):
    [0]  life_total           / 40.0
    [1]  commander_damage     / 21.0  (stub 0.0 until real Forge tracks it)
    [2]  mana_available       / 20.0  (stub 0.0 until mana pool wired)
    [3]  commander_tax        / 10.0
    [4]  cards_in_hand        / 15.0
    [5]  cards_in_graveyard   / 100.0
    [6]  creatures_on_battlefield / 30.0
    [7]  total_power_on_board / 100.0  (stub = creature count until P/T data)
    [8]  is_active_player     (0 or 1)
    [9]  phase one-hot[0]  main1
    [10] phase one-hot[1]  combat
    [11] phase one-hot[2]  main2
    [12] phase one-hot[3]  end
    [13] (padding / reserved)

  Global:
    [28] turn_number / max_turns (25)

buildStateSnapshot() now includes "state_vector": float[] so that
PolicyClient.decide(JsonObject) sends a machine-readable vector the
Python /api/policy/decide endpoint can pass directly to the encoder
without re-parsing the nested human-readable fields.

Stub fields (mana_available, commander_damage, total_power_on_board)
are zero until real Forge board state is wired in Step 4.
2026-03-29 04:24:55 -04:00
KoalaTrapLord 11990c2fd3 feat(ml): add --data-source forge flag to PPO trainer (issue #83 phase 1.3)
Wires forge_episode_generator into the PPO training loop:
- New --data-source flag: synthetic (default) | forge | mixed
- New --forge-results-dir, --forge-batch-id, --forge-weight flags
- ForgeDataSource class: runs run_forge_pipeline() in background thread,
  drains episodes into RolloutBuffer via episodes_to_buffer()
- Mixed mode: blends Forge episodes + synthetic self-play per iteration
  using --forge-weight ratio (0.0–1.0)
- Synthetic path unchanged — no regression for existing training runs
2026-03-29 03:02:30 -04:00
KoalaTrapLord aefd0cace6 feat(rag): Phase 1 — rewrite build_card_text to natural-language format (#137)
* feat(rag): Phase 1 — rewrite build_card_text to natural-language format

* feat(rag): Phase 2 — Forge-enriched ChromaDB RAG store (2a–2e)

Phase 2a: Forge lazy import guard (_try_import_forge_lookup)
Phase 2b: _build_document appends Forge trigger/replacement/static/keyword tags
Phase 2c: _build_metadata adds has_trigger, has_replacement, has_static, trigger_modes fields
Phase 2d: query_cards extended with require_trigger, require_replacement, require_static params + ChromaDB where clause support
Phase 2e: get_rag_stats reports forge_enriched and forge_coverage; _write_build_meta tracks coverage; build loop counts enriched cards

* feat(rag): Phase 3 — add mtgrules second ChromaDB collection

Adds build_rules_index() and query_rules() operating on a separate
'mtgrules' ChromaDB collection. Chunks MagicCompRules.txt by rule-number
boundaries, embeds each chunk, and exposes semantic search. Completely
independent from the mtg_oracle card collection.

* feat(coach): Phase 4 — behavior-aware RAG + rules context in coach_service

Phase 4a: run_coaching_session uses dual RAG query
  - Primary: commander synergy (n=10)
  - Secondary: require_replacement=True for graveyard/protection cards (n=5)
  - Deduplicates by name before merging
Phase 4b: query_rules() fetches 3 Commander rules chunks to ground the prompt
  - Passed as rules_context kwarg to build_user_prompt()

* feat(rag): Phase 5 — add /api/rag/build-rules and /api/rag/query-rules endpoints
2026-03-29 02:11:38 -04:00
KoalaTrapLord ac536999eb Update lab_api.pyfeat: wire validate_forge_config into lifespan startup 2026-03-28 14:48:43 -04:00
KoalaTrapLord 2e43511d87 feat: add validate_forge_config() startup check
Validates FORGE_DIR at startup with clear warning if cardsfolder not found.
Checks both forge-gui/res/cardsfolder and res/cardsfolder layouts.
2026-03-28 14:34:50 -04:00
KoalaTrapLord 1bebe38e66 fix: forge_card_loader path resolution + reset_forge_cache helper
- _get_cards_folder() now tries both forge-gui/res/cardsfolder and res/cardsfolder
  so FORGE_DIR can point to either the repo root or the forge-gui subdirectory
- Add reset_forge_cache() to allow hot-reloading without server restart
2026-03-28 14:31:42 -04:00
KoalaTrapLord eca1653024 fix: backfill basic lands on card_entries to guarantee 99 cards
The previous backfill operated on seen_names dict which silently
ignored basics already present (dict keys are unique). Now backfills
directly on card_entries list after Scryfall fetch, incrementing
quantity on existing basics or appending new CardEntry objects.
Distributes shortfall evenly across commander color identity basics.
2026-03-28 00:26:42 -04:00
KoalaTrapLord 08f1cb7581 fix(deckgen): backfill basic lands after cross-category dedup to reach 99 cards 2026-03-28 00:10:52 -04:00
KoalaTrapLord 9d9e8675a9 fix: restore missing # on line 27 section banner in rules.py (SyntaxError on import) 2026-03-27 23:36:02 -04:00
KoalaTrapLord 64526b7bee fix: exclude lands from cmc_buckets in _build_deck_report_from_db (matches deck_service behaviour) 2026-03-27 20:38:49 -04:00
KoalaTrapLord e9732d292f fix: expand _land_kw, hoist to module level, guard has_real_data for cmc=0 lands after Forge lookup 2026-03-27 20:36:22 -04:00
KoalaTrapLord fa5349ea7c fix: basic land name fallback in _build_deck_report_from_db when type_line is missing from collection join 2026-03-27 20:32:15 -04:00
KoalaTrapLord 1c491bc2b0 fix: add Land role detection to _detect_card_roles (fixes basics showing as Unknown) 2026-03-27 20:28:35 -04:00
KoalaTrapLord 81fe5108d2 fix: coach_apply rollback on partial write, 4-space reindent; ws_game hoist inline imports 2026-03-27 20:24:55 -04:00
KoalaTrapLord 61fa37dc48 fix: lifespan indent error, get_running_loop, remove double rag init, register policy routes 2026-03-27 20:22:42 -04:00
KoalaTrapLord 0118950939 fix: add anthropic_api_key to Config with env-var default (fixes AttributeError in ASGI mode) 2026-03-27 20:20:45 -04:00
KoalaTrapLord 22cf23b24d fix: correct query_cards kwarg names in coach_service (n= -> n_results=, colors= -> color_identity=) 2026-03-27 20:17:14 -04:00
KoalaTrapLord 0010b0f394 refactor: RAG improvements — shared card_text, Chroma fallback, staleness monitor (fixes #133, #134, #135) (#136)
* refactor: add shared build_card_text() utility

* refactor: unify _build_document via shared card_text + add is_chroma_available/check_staleness (fixes #133, #134, #135)

* refactor: delegate embeddings._build_card_text to shared utility

* feat: add background RAG staleness monitor to lab_api lifespan (fixes #135)
2026-03-27 16:46:33 -04:00
KoalaTrapLord d0cdafa66a feat(rag): Phase 3 — inject RAG cards into coach prompts (#132)
* feat(rag): add RAG card injection to prompt_template.py

* feat(rag): wire RAG query into coach_service.py

* test(rag): add Phase 3 RAG injection tests
2026-03-27 15:20:49 -04:00
KoalaTrapLord bcf558418a fix(rag): squash all RAG/ChromaDB bugs #123–#130
Fixes critical API contract mismatch, batch_size wiring, lifespan init, failure tracking, build_meta persistence, mana_cost in embeddings, pyproject.toml deps, and adds full test suite.

Closes #123, #124, #125, #126, #127, #128, #129, #130
2026-03-27 13:24:41 -04:00
KoalaTrapLord 61ac349445 fix: re-encode requirements.txt as UTF-8 (fixes pip install failure) 2026-03-27 11:14:34 -04:00
KoalaTrapLord 06ce5cc91a fix(rules): add nonbasic land guard before creature fallback (fixes #118) (#122) 2026-03-27 10:22:05 -04:00
KoalaTrapLord fc2e9d88d5 feat(rag): wire ensure_rag_store into main() startup (#121) 2026-03-27 09:13:03 -04:00
KoalaTrapLord 4b0cd34eda feat(rag): Phase 2 — ChromaDB vector index over bulk card DB (#120)
* feat(rag): Phase 2 — ChromaDB vector index over bulk card DB

* feat(rag): add routes/rag.py with build, search, status endpoints

* Update lab_api.pyfeat(rag): register rag_router in lab_api.py, remove old inline rag_status

* Update requirements.txtfeat(rag): add chromadb>=0.5.0 to requirements.txtfeat(rag): add chromadb to requirements.txt
2026-03-27 08:39:02 -04:00
KoalaTrapLord 83502ce726 feat: Phase 3/5 deep analysis + Quick Digest endpoint (issue #108) (#119)
* feat(coach): Phase 3/5 JSON schema + QUICK_DIGEST prompt (issue #108)

- Append removalCoverage, rampQuality, drawEngineProfile, winConditions,
  antiSynergyFlags, podPresence, tempoAssessment, metaMatchups to schema
- Add 8 new RULES for Phase 3/5 analysis
- Add QUICK_DIGEST_SYSTEM_PROMPT constant
- Add build_quick_system_prompt() function

* feat(coach): Phase 3/5 JSON parsing + run_quick_digest() (issue #108)

* feat(routes): add POST /api/coach/decks/{deck_id}/quick endpoint (issue #108)

* feat(coach): Phase 3/5 parsing + run_quick_digest() (issue #108)

* feat(coach): Phase 3/5 parsing + run_quick_digest() (issue #108)

- Import RemovalCoverage, RampQuality, DrawEngineProfile, WinCondition,
  AntiSynergyFlag, PodPresence, TempoAssessment, MetaMatchup
- Add Phase 3/5 blocks to _populate_session_from_json()
- Add run_quick_digest() method

* feat(ui): Top Priority card + Quick/Deep buttons in coach.html (issue #108)

* feat(coach): Phase 3/5 parsing + run_quick_digest() (issue #108)

* feat(ui): wire Quick Digest button + displayTopPriority() (issue #108)
2026-03-27 07:51:01 -04:00
KoalaTrapLord 8df5900ff9 feat(rag): Phase 1 — Scryfall bulk offline card database (SQLite FTS5) (#117)
* feat(rag): add scryfall_bulk.py — offline SQLite FTS5 card database (RAG Phase 1)

* feat(rag): wire ensure_bulk_db() into startup and add /api/rag/status endpoint (RAG Phase 1)

* fix: migrate on_event to lifespan handler, fix /api/rag/status route ordering
2026-03-27 07:16:55 -04:00
KoalaTrapLord 864cefb8d1 feat(coach): Phase 5 - add PodPresence, TempoAssessment, MetaMatchup models + isQuickDigest flag (#108) 2026-03-27 04:22:23 -04:00
KoalaTrapLord 6c03e56794 fix(#116): dual card_records fallback JOIN, card_type/card_category fields, simulator land detection
- deck_service._compute_deck_analysis: add LEFT JOIN card_records with COALESCE on all fields
- deck_service: add _classify_card_category() with oracle-text keyword matching
- deckbuilder.GET /api/decks/{id}/cards: add card_records fallback JOIN + COALESCE, emit card_type/card_category per card
- deepseek._fetch_deck_card_data: add card_records fallback JOIN so lands get correct type_line before enrich_card()

Closes #116
2026-03-27 04:10:17 -04:00
KoalaTrapLord 4b92a5f364 Update models.pyfeat(coach): Phase 3 - add RemovalCoverage, RampQuality, DrawEngineProfile, WinCondition, AntiSynergyFlag models (#108) 2026-03-27 04:01:25 -04:00
KoalaTrapLord da0193eb11 Update coach_service.pyfeat(coach): Phase 2 - parse upgradePriority, commanderDependency, mulliganAnalysis in coach_service (#108)feat(coach): Phase 2 - parse new fields in coach_service (#108) 2026-03-27 03:45:32 -04:00
KoalaTrapLord 71e0db4985 feat(coach): Phase 2 - add upgradePriority, commanderDependency, mulliganAnalysis to prompt schema (#108) 2026-03-27 03:32:03 -04:00
KoalaTrapLord 825946ae75 fix(deckgen): Phase 1 - non-destructive smart substitution + 99-card guarantee (#111)
- Add SubstitutionMethod enum to track how each card was resolved
- smart_substitute() now always returns a valid card string (never None/drops)
  fallback chain: exact -> fuzzy -> functional analog -> original kept -> basic land
- apply_substitutions() loop no longer silently drops cards on failure
- _pad_to_99() circuit breaker added before BuildResult validation
- Pre-validation count logging per category for fast regression diagnosis
- Startup assertion: sum(SLOT_BUDGET) == 99 raises on app boot if misconfigured
- Prompt injection: slot budget + exact card count requirement sent to LLM
- run_substitution() anthropic fallback failure now pads with basic land instead of leaving slot missing
- SubstitutionBatchResult gains padded_slots field (> 0 = circuit breaker fired)
2026-03-27 03:27:52 -04:00
KoalaTrapLord bd4df0c263 Update models.pyfeat(coach): Phase 2 - add UpgradePriority, CommanderDependency, MulliganAnalysis models (#108) 2026-03-27 02:52:01 -04:00
KoalaTrapLord 113b5c63f0 docs: rewrite README to reflect full current feature set (#115) 2026-03-27 02:51:12 -04:00
KoalaTrapLord 3f6d71ecbc fix(coach): switch all Anthropic calls to streaming to prevent 10-min timeout error (closes #109)
- anthropic_client.py: all 4 methods (chat_structured, chat_plain, achat_structured, achat_plain) now use .messages.stream() context manager
- coach_service.py: inline aclient.messages.create() replaced with async streaming via aclient.messages.stream()
- _parse_response() updated to accept pre-extracted text string (since stream gives us text directly)
- check_status() left as non-streaming (max_tokens=10, safe from the 10min rule)
2026-03-27 02:36:22 -04:00
KoalaTrapLord 4246555228 feat: auto-run update_weights.py after overnight sims (#114)
Completes the sim → weights → deck builder feedback loop.

- Add run_update_weights() to overnight-run.py
- Add auto_update_weights config key (default True)
- Add --no-update-weights, --weights-min-games, --weights-lr CLI flags
- Runs before neural net training so both pipelines get fresh data
2026-03-27 02:35:08 -04:00
KoalaTrapLord 8dd9623417 feat: inject simulation-learned weights into deck builder LLM prompts (#113)
Closes #112

- Add sim_insights.py to translate learned_weights.json into LLM prompt text
- Add sim_context param to suggest_cards() and enforce_deck_ratios() in ollama_client.py
- Wire get_sim_insights() into build_deck pipeline (Steps 4 & 6)
- Fully backwards compatible — no-op until real sim data exists
2026-03-27 02:27:42 -04:00
KoalaTrapLord 817b71f69d feat(ui): add tip-icon tooltips to distillation, simulator, coach, deckgenerator pages
- distillation.html: tooltips on all 10 params + ELO Ratings header
- simulator.html: tooltips on Opponent, Games, Record logs, API Endpoint, Model, Temp, Timeout
- coach.html: tooltips on Power Level, Meta Focus, Budget, Focus Areas, Similar Cards Results
- deckgenerator.html: tooltips on Target Bracket, Use my collection, Smart Substitution, AI Model
- All pages now share tooltip styles from global.css (no per-page duplication needed)
2026-03-27 02:06:58 -04:00
KoalaTrapLord e71cac9dd1 feat(ui): add tooltips to all pages — distillation, simulator, coach, deckgenerator
- Move .tip-icon tooltip CSS from training.css into global.css so all pages share it
- distillation.html: 8 param tooltips (Max Generations, Convergence Window, Supervised Epochs, Supervised LR, PPO Iterations, PPO Episodes/Iter, PPO LR, Min Win Rate Gate, Opponent, Playstyle) + ELO Ratings header
- simulator.html: tooltips on Opponent, Games, Record turn logs, API Endpoint, Model, Temp, Timeout
- coach.html: tooltips on Target Power Level, Meta Focus, Budget, Focus Areas, Find Similar Cards Results
- deckgenerator.html: tooltips on Target Bracket, Use my collection, Smart Substitution, AI Model
2026-03-27 01:57:37 -04:00
KoalaTrapLord cb38081a8b fix: add top_n cutoff to ELO tournament (default top 10 generations) 2026-03-27 01:48:10 -04:00
KoalaTrapLord cd9d2eebd8 feat(coach): Phase 1 - pass richer card stats to Claude prompt (#108)
- _format_underperformers: add synergyScore, clunkinessScore, avgTurnCast, keptInOpeningHandRate
- _format_top_cards: add synergyScore and avgTurnCast
- _format_matchups: append perArchetypeWinRates block from DeckMeta
- USER_PROMPT_TEMPLATE: add ### Archetype Win Rates section placeholder
2026-03-27 01:32:51 -04:00
KoalaTrapLord e6f18d4a61 feat(training): add intro banner and parameter tooltips to ML Training page 2026-03-27 00:18:21 -04:00
KoalaTrapLord 4e90337f54 feat: extend card hover preview to all card names in coach UI 2026-03-26 23:48:27 -04:00
KoalaTrapLord a01a70f725 Use DEFAULT_MAX_TOKENS config for Anthropic max_tokens 2026-03-26 22:50:10 -04:00
KoalaTrapLord 4da58681a4 Enhance coach prompt for more detailed reports 2026-03-26 22:41:23 -04:00
KoalaTrapLord 224b4b5d5b feat: increase coach detail limits (MAX_UNDERPERFORMERS=8, MAX_TOKENS=16384) 2026-03-26 22:33:18 -04:00
KoalaTrapLord 1a34f00574 Update .env.examplefix: correct model ID in .env.example (claude-opus-4-0 -> claude-opus-4) 2026-03-26 15:50:53 -04:00
KoalaTrapLord 8ec7db1612 fix: correct malformed string literals in config.py model defaults 2026-03-26 15:42:34 -04:00
KoalaTrapLord e0a75e2d04 fix: correct Anthropic model ID from claude-opus-4-0 to claude-opus-4 2026-03-26 15:36:23 -04:00