momentum-system/docs/research/regime_notes.md

Regime-detection research notes — distilled for the momentum bot

Date: 2026-07-15 Method: 6 books in books/market_regime_state_detection/ distilled by parallel readers (Ang & Timmermann NBER survey; Kritzman Regime Shifts; Zucchini/MacDonald/Langrock HMM for Time Series; Chan Hands-On AI Trading & Quantitative Trading; Kakkar et al Trading Decoded). Hamilton 1989 is a scanned image (no extractable text); its Markov-switching method is covered by the others. Problem this feeds: the bot is ~breakeven with no entry edge after honest fills; the one thing untried is regime awareness — it did well 07-08/09 (a runner regime) and sits out / bleeds otherwise. Goal: a FREE-data daily regime signal that gates/sizes ORB entries, harness-backtestable over ~260 sessions.

---

The one insight that reframes everything (Chan's metalabeling)

"The key to successfully apply AI/ML to finance is to focus on metalabeling — finding the probability of profit of your own simple trading strategy, and not to use it to predict the market directly. Your own strategy's past track record is private; no one else is trying to predict it." — Chan, Quantitative Trading

Don't detect abstract "market regime." Predict P(our ORB makes money today) from free pre-open features, using our OWN per-session expR (which the plan-006 harness already computes) as the label. This targets exactly what matters (does our edge work today), sidesteps the "high-vol lumps runner-days with crash-days" problem, reuses the harness, is free, and is a private signal nobody arbs away. Every practical book (Chan CPO, Trading Decoded RF) converges here; the academic survey (Ang-Timmermann) calls the same thing "regime-conditioned position sizing."

The method stack (what to build, in escalation order — ponytail)

1. Threshold baseline first (Kritzman, Trading Decoded both insist): a trivial rule — e.g. SPY 5d momentum > 0 AND SPY 20d vol < median, or Kritzman Financial Turbulence (sqrt((r−μ)Σ⁻¹(r−μ)') on ~11 sector-ETF returns) below its 75th percentile. If the fancy model can't beat this net-of-costs out-of-sample, ship the rule.
2. Metalabel classifier (the primary): GradientBoostingClassifier / RandomForest (free sklearn — the books' predictnow.ai/gbdt replaced) predicting P(profitable ORB session) from ~5–8 free features. Gate/size on the probability.
3. HMM regime as a FEATURE (not the whole model): a 2-state Markov-switching state on SPY (statsmodels MarkovRegression(spy_returns, k_regimes=2, switching_variance=True)) or on the turbulence series — one column feeding the classifier, or a standalone gate to compare.

Free features (all pre-open, all we have or can ingest free)

Feature	Source	Note
SPY 5d/20d momentum + 20d realized vol (VIX proxy)	already ingesting SPY (plan 009)	the level-1 input every book uses
IWM (small-cap index) + sector-ETF momentum	Alpaca/yfinance, free	small-caps specifically; sector rotation
Kritzman turbulence (Mahalanobis on ~11 sector ETFs)	sector-ETF daily returns, free	"prices behaving uncharacteristically"
Breadth = % of the gapper universe green / held-VWAP yesterday	our own stored data	the SIGNED feature that breaks vol-symmetry
Bot's own trailing-N-day realized expR + hit-rate	our harness	the most on-target feature — a hot/cold-streak detector on our OWN edge
VIX level + 1d change	CBOE/yfinance, free	the vol axis directly

Keep to ≤5–8 features — only ~260 sessions, ~a third fewer "runner days." More features = curve-fit.

The load-bearing discipline (or the whole thing is a lie)

• Filtered / walk-forward / purged only. Only a regime label computed from data ≤ today is tradeable. Smoothed HMM probs and full-sample fits peek at the future. Sanity check every book demanded: if a smoothed-gated backtest crushes the filtered-gated one, your edge is look-ahead. Use purged/embargoed walk-forward CV (drop the boundary day) — the same leakage-safe split used in plans/008.
• Label is same-day-outcome, features are strictly pre-open. The session's expR (label) uses open-knowable exits; every feature is lagged to the prior close.
• Out-of-sample split is mandatory. Fit on the older half, apply to the recent half; confirm the favorable-regime expR advantage survives. "In-sample regimes need not persist out-of-sample" (Ang-Timmermann) — if the edge evaporates OOS, the regimes were in-sample noise.
• Acceptance = NET expR improves across EVERY walk-forward fold, not AUC. A 52.8% classifier can still be net-negative (Chan Hands-On's own OOS accuracy was 52.77%).
• Let the DATA pick the sign. Ang-Timmermann finds low-vol = high-Sharpe for a long-index investor — but a long-dispersion small-cap momentum strategy likely inverts this (runners need volatility). Measure conditional expR per regime; never assume high-vol = bad.
• Small-sample humility. 2 states (don't LR-test the count), shallow trees (max_depth ≤ 3), report bootstrap/posterior bands on the regime probs, deflate any OOS Sharpe by the number of tweaks tried (Deflated Sharpe Ratio).
• Regimes lag. A daily gate enters the runner regime a day or two late and exits late — structural. It sits out some good days and takes some bad ones. The bar is net improvement, not perfection.

Honest prior

This is the most promising lever of the session — it targets the bot's own edge, uses free data, reuses the harness, and directly explains the 07-08/09-vs-today swing. But 260 sessions is a small sample, so the real risk is curve-fitting a regime that doesn't persist. The threshold-baseline-first rule, purged walk-forward, and OOS split are the guards. If, after all that, the gated expR doesn't clearly beat trade-always out-of-sample → the honest finding is "no exploitable regime signal in free data at this sample size," and we've learned it cheaply.

Sources (in books/market_regime_state_detection/)

• Ang & Timmermann (2011), Regime Changes and Financial Markets, NBER WP 17182 — the survey + the sizing template + every caveat.
• Kritzman et al, Regime Shifts: Implications for Dynamic Strategies — Financial Turbulence + absorption ratio.
• Zucchini, MacDonald, Langrock, Hidden Markov Models for Time Series — the estimation recipe (forward-filter, BIC state selection, why filtered ≠ smoothed).
• Chan et al, Hands-On AI Trading — the exact statsmodels MarkovRegression daily-gate recipe (Example 4) + the 4-year-lookback calibration.
• Chan, Quantitative Trading — Conditional Parameter Optimization + the metalabeling principle.
• Kakkar, Sahni, Shanmugamani, Trading Decoded — the RandomForest regime-as-feature recipe + purged walk-forward CV + the "beat the trivial rule or ship the rule" discipline.