momentum-system/docs/history/INSPECTION_ROADMAP.md

Inspection & Debugging Roadmap

What You'll Be Able to See (When All Phases Complete)

Right now, when you run python run_live_paper.py, you see:

• ❌ No indication bars are being collected
• ❌ No visibility into research data
• ❌ Symbols appear/disappear silently
• ❌ No way to test individual modules
• ❌ Can't verify calculation accuracy

After Phases 1-4, you'll have:

---

1. Real-Time Module Dashboard (Port 8011)

Open http://127.0.0.1:8011 to see live module execution:

Data Pipeline Diagram (Auto-Generated)

┌──────────────┐    ┌──────────────┐    ┌────────────────┐    ┌───────────────┐
│ INGESTION    │───→│ RESEARCH     │───→│ EVALUATION     │───→│ EXECUTION     │
├──────────────┤    ├──────────────┤    ├────────────────┤    ├───────────────┤
│ 500 rows in  │    │ 5 researched │    │ 5 evaluated    │    │ 2 approved    │
│ 245ms        │    │ 156ms        │    │ 1823ms         │    │ 89ms          │
│ ✓ validated  │    │ ✓ all found  │    │ 2 ready, 3 blocked    │ 0 rejected    │
└──────────────┘    └──────────────┘    └────────────────┘    └───────────────┘

Each box shows:

• Input count → "500 rows in"
• Latency → "245ms"
• Status → "✓ validated"
• Click to expand → see detailed metrics

Module Performance Grid

┌───────────┬──────────────┬──────────┬────────────────┬─────────────────┐
│ Module    │ Last Run (s) │ Avg (ms) │ Error Rate     │ Input → Output  │
├───────────┼──────────────┼──────────┼────────────────┼─────────────────┤
│ Ingestion │ 2.3s ago     │ 245      │ 0.0%           │ 500 → 500       │
│ Research  │ 2.1s ago     │ 156      │ 0.0%           │ 5 → 5           │
│ Evaluation│ 0.8s ago     │ 1823     │ 0.0%           │ 5 → 2           │
│ Execution │ 0.5s ago     │ 89       │ 0.0%           │ 2 → 2           │
│ Broker    │ 0.3s ago     │ 342      │ 0.0%           │ 2 → 2           │
└───────────┴──────────────┴──────────┴────────────────┴─────────────────┘

Click on each row to:

• See last 10 executions (histogram of latencies)
• View error log if failed
• Inspect metrics

---

2. Symbol Inspector (Search Box)

Type a symbol name to see its complete evaluation journey:

 🔍 Search symbol... [AAPL]
────────────────────────────────────────────────────────────────

AAPL (state: blocked, score: 48%)
├─ INGESTION ✓
│  ├─ bars collected: 12
│  ├─ validation: passed ✓
│  ├─ quality_score: 0.82
│  └─ collected_at: 09:35:12
│
├─ RESEARCH ✓
│  ├─ gaps: 2.1%
│  ├─ rvol: 1.2x ✗ (need 1.5x)
│  ├─ daily avg volume: 500k
│  ├─ previous close: $10.50
│  └─ researched_at: 09:35:13
│
├─ EVALUATION ✓ bars_loaded
│                ├─ gap_checked ✓ (2.1%)
│                ├─ rvol_calculated ✗ (1.2x < 1.5x) ← BLOCKING REASON
│                ├─ bull_flag ✓ (confidence 0.75)
│                ├─ 1st_candle ✗ (weak)
│                ├─ momentum ✓
│                ├─ quality: 0.62
│                ├─ score: 48% (threshold: 60%)
│                └─ evaluated_at: 09:35:15
│
├─ EXECUTION (blocked - not evaluated)
│  └─ blocked_reason: "criteria score 48% < 60%"
│
└─ BROKER (N/A - not submitted)

Click on each section to see:

• Exact metric values
• Why it passed/failed
• Timestamp of calculation
• Raw data (for debugging)

Sparkline Chart (OHLCV)

AAPL: 09:30 → 09:35 (12 bars)

  11.00│                    ╭─╮
  10.80│          ╭─╮  ╭─╮╭─╯ ╰─╮
  10.60│ ╭─╮ ╭─╮ ╭─╯╰─╮
  10.40│╭─╯ ╰─╮
  10.20│╯
       └────────────────────────────
        Volume: avg=125k, last=85k
        
  Entry: $10.52 ━━━━━━━━━━━━━━
  Stop:  $10.20 ━━━━━━━━━━━━━━

Interactive:

• Hover over bars to see OHLCV
• Click to zoom in/out
• See entry/stop/target reference lines

---

3. Data Quality Report

📊 DATA QUALITY SNAPSHOT
────────────────────────────────────────────────────────────────

Ingestion Validation
├─ Rows validated: 500 ✓
├─ Schema failures: 0 ✓
├─ Checksum failures: 0 ✓
├─ Business rules failures: 0 ✓
└─ Warnings: 2 (zero-volume bars)

Research Data
├─ Market data freshness: 23 seconds old ✓
├─ Daily history freshness: 1 day old ✓
├─ Symbols missing data: 0 ✓
└─ DB integrity check: PASSED ✓

Evaluation Quality
├─ Criteria evaluated: 5 of 5 ✓
├─ Blocking reasons captured: 3 ✓
├─ Score distribution: [48%, 52%, 58%, 64%] (avg 55%)
└─ Scores < 60%: 3 symbols

Broker Connectivity
├─ API health: healthy ✓
├─ Last account sync: 2s ago ✓
├─ Last order: 45s ago ✓
└─ Latency: 342ms avg ✓

---

4. Event Timeline (Live Stream)

╔════════════════════════════════════════════════════════════════╗
║ EVENT TIMELINE (newest first, filter by module/symbol/type)    ║
╚════════════════════════════════════════════════════════════════╝

09:35:15.347  [evaluation]  ✓ completed    duration 1823ms, 5 evaluated, 2 ready
09:35:14.512  [evaluation]  AAPL: rvol_calculated (1.2x < 1.5x) ✗
09:35:14.381  [evaluation]  AAPL: gap_checked (2.1%) ✓
09:35:13.856  [evaluation]  TSLA: bull_flag_detected (conf 0.85) ✓
09:35:13.245  [data_validation] ingestion: schema validation PASSED (500 rows)
09:35:13.122  [research]  ✓ completed    duration 156ms, 5 researched
09:35:12.856  [ingestion]  ✓ completed    duration 245ms, 500 rows inserted
09:35:12.123  [ingestion]  ↦ started      (source: alpaca, 500 bars)
09:35:10.234  [execution]  AAPL: blocked (score 48% < 60%)
09:35:09.856  [approval_queue] → empty (all processed)
09:35:09.453  [broker]  ✓ positions synced  3 open positions
09:35:09.123  [broker]  ✓ account updated   equity $99,234

Click on any event to see:

• Full event JSON
• Correlation ID (trace all events for one signal)
• Copy to clipboard

---

5. Dry-Run Mode (Phase 7)

# Test with synthetic bull-flag setups (no API calls, no broker)
$ python run_live_paper.py --dry-run \
    --symbols AAPL,TSLA,AMD \
    --bars-provider synthetic:bull_flag \
    --num-ticks 5 \
    --inspection-dashboard

Starting dry-run mode (no API calls, no broker)
Using synthetic bull-flag bars: 50 bars/symbol
Inspection dashboard: http://127.0.0.1:8011

Tick 1 (09:30-09:35)
  Ingestion:  ✓ 150 bars collected
  Research:   ✓ 3 symbols researched
  Evaluation: ✓ 3 evaluated → 2 ready (AAPL, TSLA)
  Execution:  ✓ 2 approved
  Broker:     ✓ 2 orders submitted (simulated)

Tick 2 (09:35-09:40)
  Ingestion:  ✓ 150 bars collected
  Research:   ✓ 3 symbols researched
  Evaluation: ✓ 3 evaluated → 1 ready (AMD)
  Execution:  ✓ 1 approved
  Broker:     ✓ 1 fill (AAPL +$250), 2 pending

... (Dashboard shows all metrics live)

---

6. Module-Level Testing (Phase 6)

# Test just the evaluation module with deterministic data
$ pytest tests/modules/test_evaluation_module.py -v

tests/modules/test_evaluation_module.py::test_evaluation_ready ✓
  - Bull-flag bars pass evaluation
  - Score: 85%
  - Entry: $150.50
  - Stop: $149.00

tests/modules/test_evaluation_module.py::test_evaluation_blocked ✓
  - Fading bars fail evaluation
  - Score: 35%
  - Blocking: low volume
  - Reason: RVOL 0.8x < 1.5x

tests/modules/test_evaluation_module.py::test_evaluation_dry_run ✓
  - Can run without research DB
  - Works with injected bars
  - Full metrics returned

tests/modules/test_evaluation_module.py::test_evaluation_validation ✓
  - Invalid bars rejected
  - Validation failures captured
  - Error messages clear

tests/modules/test_evaluation_module.py::test_evaluation_contract ✓
  - Accepts EvaluationRequest
  - Returns EvaluationResult
  - All fields populated

5 passed in 0.23s

---

7. Rich TUI Mode (Phase 8, Optional)

# Real-time terminal UI (useful for SSH, CI/CD logs)
$ python run_live_paper.py --tui

╔═══════════════════════════════════════════════════════════════╗
║ MOMENTUM TRADING SYSTEM - TUI DASHBOARD                       ║
╚═══════════════════════════════════════════════════════════════╝

[Ingestion]    ████████████░░░░░░░░░░░░  50% (250/500 bars)
[Research]     ████████████████████░░░░  85% (4/5 symbols)
[Evaluation]   ████████████████████░░░░  85% (4/5 evaluated)
[Execution]    ████████████░░░░░░░░░░░░  50% (1/2 approved)
[Broker]       ███░░░░░░░░░░░░░░░░░░░░░   15% (1/5 submitted)

┌─ Module Latencies ──────────────────────────────┐
│ Ingestion: 245ms                                │
│ Research:  156ms                                │
│ Evaluation: 1823ms  ← slow (5 symbols)          │
│ Execution: 89ms                                 │
│ Broker:    342ms                                │
└─────────────────────────────────────────────────┘

┌─ Recent Events ─────────────────────────────────┐
│ [09:35:15] evaluation ✓ TSLA ready              │
│ [09:35:14] evaluation ✗ AAPL blocked (rvol)     │
│ [09:35:13] research ✓ 5 symbols researched      │
│ [09:35:12] ingestion ✓ 500 bars collected       │
└─────────────────────────────────────────────────┘

[press 'q' to quit, 's' to toggle details, 'c' to copy logs]

---

Phase Completion Checklist

Phase	What Gets Built	When You Can Use It
✅ 1	Core contracts + telemetry	After Phase 1
📍 2	Module instrumentation	After Phase 2
🔲 3	Inspection API endpoints	After Phase 3
🔲 4	Web dashboard frontend	After Phase 4
🔲 5	Module refactoring	After Phase 5
🔲 6	Module tests + fixtures	After Phase 6
🔲 7	--dry-run mode + synthetic data	After Phase 7
🔲 8	Rich TUI (optional)	After Phase 8

---

Usage Scenarios

"I want to debug why AAPL isn't signaling"

With inspection dashboard:

1. Open http://127.0.0.1:8011
2. Search "AAPL" in symbol inspector
3. Expand each stage to see:
   • Bars collected: 12 ✓
   • Gap: 2.1% ✓
   • RVOL: 1.2x ✗ (blocked here!)
4. Click "RVOL" stage to see:
   • Current volume: 85k
   • 20-day average: 500k
   • Ratio: 0.17x (need 1.5x)
5. Understand exactly why it's blocked

Time saved: 10 seconds vs. 30 minutes digging through logs

---

"I want to test the evaluation module alone"

With Phase 6:

pytest tests/modules/test_evaluation_module.py -v

• No database needed
• No API calls
• Deterministic test data (bull flags, fading bars)
• See exactly what each criterion does
• Change thresholds and re-test instantly

---

"I want to see if my strategy works with dry-run data"

With Phase 7:

python run_live_paper.py --dry-run \
  --bars-provider synthetic:bull_flag \
  --num-ticks 5

• No API keys needed
• No broker interaction
• Dashboard still works
• See signals, approval, execution all flow through
• Test configuration changes risk-free

---

Summary

Before Phases 1-4: Black box ("did it work?")

After Phases 1-4: Transparent pipeline ("here's exactly what happened at each step, why it passed/failed, and how long it took")

This is the foundation for:

• ✅ Debugging blocked signals in seconds
• ✅ Testing modules independently
• ✅ Verifying data accuracy at each stage
• ✅ Dry-running strategies without real APIs
• ✅ Understanding your system's behavior

Next: Phase 2 instruments the existing modules. Then Phases 3-4 build the visual dashboard to make all this data human-readable.