momentum-system/BACKTEST_GUIDE.md

Comprehensive Backtest Guide

This guide will help you run a multi-week backtest to assess the profitability and risk profile of the momentum strategy before trading live.

Quick Start (Windows Terminal)

Run a 1-week backtest on small-cap momentum names (the strategy's sweet spot):

uv run comprehensive_backtest.py --start 2026-06-05 --end 2026-06-12

This analyzes every trading day in that range, pulling minute bars for the specified symbols and reporting:

• Daily P&L — win/loss pattern by day
• Symbol breakdown — which tickers traded well/poorly
• Trade-by-trade details — entry/exit, profit/loss, reason for exit
• Statistical summary — win rate, average R-multiple, profit factor, risk metrics
• Loss analysis — patterns in losses (gap-throughs, wide stops, etc.)

What the Output Tells You

Summary Metrics

• Win Rate: % of trades that were profitable. 40%+ is acceptable; 50%+ is strong.
• Profit Factor: Total wins ÷ absolute value of total losses.
  • < 1.3: marginal edge or slippage issues (investigate before live)
  • 1.3–2.0: decent edge

  • 2.0: strong edge

• Avg R-Multiple: Average return per unit of risk. 1.5+ suggests the strategy rewards you for the risk taken.
• Max Loss: Largest single loss. If it's 2–3× worse than your average loss, stops may be too wide or slippage is severe.

Red Flags to Watch For

1. Profit factor < 1.3 — your losses are eating most of your wins
2. Win rate < 40% — even with 2:1 R ratio, this struggles to be profitable
3. Max loss >> average loss — suggests gap-throughs or wide stops on losers
4. Inconsistent daily results — if one bad day wipes out a week's gains, volatility is too high

Advanced Options

# Test a different date range (e.g., last 2 weeks)
uv run comprehensive_backtest.py --start 2026-05-27 --end 2026-06-12

# Test on your watched symbols instead of the defaults
uv run comprehensive_backtest.py --start 2026-06-05 --end 2026-06-12 --symbols SNDL,MARA,RIOT

# Test a full month (if available)
uv run comprehensive_backtest.py --start 2026-05-13 --end 2026-06-12

Important Caveats

1. This is NOT the live execution path — the backtest engine shows signal→entry→exit on historical bars, but it does NOT exercise:

   • The new auto-arm resting-limit logic
   • The tick-by-tick entry invalidation (live prices)
   • The approval queue or manual trading decisions

   All of those features only run in uv run run_live_paper.py --auto-approve. This backtest proves the signal-generation and position-management logic works; it's not a full replay of the live system.

2. IEX feed gaps — free tier only sees IEX trades (a subset of consolidated). On illiquid small-caps there may be minutes with no prints, causing simulated fills to happen on stale data. The paid SIP feed (ALPACA_DATA_FEED=sip) would see more ticks.

3. Minute-bar vs tick-by-tick — the backtest assumes fills happen at the bar open of the next bar. Live, entries/exits are much more granular (actual filled prices as they happen). This is usually conservative (against you) but can help on some exits.

4. Slippage assumptions — the engine includes configurable slippage (base_spread_pct in config). The default assumes ~0.05% slippage; real fills vary by size, liquidity, and market conditions.

Interpretation Guide

Strong Strategy (Should Trade)

• Win rate ≥ 45%
• Profit factor ≥ 1.8
• Avg R ≥ 1.5R
• Max loss ≤ 2× average loss
• Consistent week-to-week results

Marginal Strategy (Investigate Further)

• Win rate 40–45%
• Profit factor 1.3–1.8
• Avg R ≈ 1.0–1.5R
• Max loss 2–3× average loss
• High daily variance

Weak Strategy (Do Not Trade)

• Win rate < 40%
• Profit factor < 1.3
• Avg R < 0.5R
• Max loss > 3× average loss
• Losing weeks common

Next Steps

1. Run the backtest for the most recent 2–4 weeks using small-cap symbols (SNDL, MARA, RIOT, BBAI, SOUN, PLUG).
2. Review the loss analysis — click "Loss Analysis" section and understand why you're losing. Is it:
   • Entries are wrong (enter on fakeouts)? → Tighten entry criteria
   • Stops are too wide (gap-throughs)? → Use a tighter stop or accept wider slippage
   • Exits are leaving money? → Adjust target or exit rules
3. Check the daily pattern — if one symbol dominates losses, exclude it (maybe illiquid on IEX).
4. If strong → uv run run_live_paper.py --auto-approve on the next trading day.
5. If marginal/weak → dig into the top losers to understand the pattern before risking real capital.

Debugging a Bad Backtest

If you see 0 trades despite seeing signals:

# First, make sure minute bars were fetched
uv run fetch_minute_bars.py SNDL MARA RIOT --lookback 5000

# Then try a single-symbol, single-day backtest to see detailed output
uv run backtest_cli.py --symbol SNDL --date 2026-06-12

If you get "no symbols with minute bars," the date may not have tradeable data (check it's a weekday) or the lookback was too short.

Questions?

• The backtest engine: strategy/backtest/engine.py
• Setup evaluation (what defines a "signal"): strategy/evaluation/setup_evaluator.py
• Risk/position sizing: strategy/risk/position_sizing.py
• Config defaults: config.py (BacktestConfig section)