momentum-system/plans/008-conversion-signal-model.md

Plan 008: Measure, then model — does the labeled data contain a separable conversion signal, and can a model (+ an Ollama catalyst feature) find it?

Executor instructions: This plan is GATED. Step 1 is a measurement whose result decides whether Steps 2–4 run at all. Do not build the model before Step 1's go/no-go passes. Run every verification command; if a STOP condition fires, stop and report. Update the status row in plans/README.md when done.

Drift check: git diff --stat d28fe25..HEAD -- research/ storage/ — the experiment harness (research/experiments/) and labeler tables are the substrate; if they moved, re-confirm the schema in Step 1 before proceeding.

Status

• Priority: P1 (this is the one entry-conversion lever never tested)
• Effort: L (but Step 1 alone is S and may end the plan)
• Risk: LOW to the live system — this is all SHADOW (predict + log, never gate) until an experiment verdict promotes it
• Depends on: 006 (experiment harness — the promotion path), 004 (population module)
• Category: direction / research-infrastructure
• Planned at: commit d28fe25, 2026-07-14

MEASURED RESULT (Step 1 ran 2026-07-14) — verdict: NO-GO on the current data

Leakage-safe expanding-window CV (5 date-blocks, whole-day splits) over 63,803 labeled setups, HistGradientBoostingClassifier on the trigger-time features that are actually populated:

• Mean CV AUC = 0.514 (folds: 0.489 / 0.538 / 0.528 / 0.526 / 0.487) — noise. The largest fold (62k rows) is 0.487, below random.
• Top-decile +1R = 0.456 vs base 0.428 — a ~+2.8pp lift, inconsistent across folds (some below base). Not tradeable.
• Top feature by far: session_minute_number (+0.010 AUC), then relative_volume (+0.003); everything else ≈ 0. The only faint signal is enter early + higher RVOL — both already known and already in the live gate.

Two data-quality findings that gate any future attempt:

1. The rich features are mostly NULL — gap_pct 85% null, impulse_pct 94%, pullback_depth_pct 98%, catalyst_freshness_at_trigger 76%, float_rotation/tape_regime/spy_return ~all null. The labeler populates different fields per track, so the "24-feature vector" is really ~8 dense features for most rows. A model can't find signal in columns that aren't filled.
2. The dataset is 85% liq1 (53,992/63,803 — the refuted liquid-breakout track); the live ORB v1 is only 6,217 rows and converts WORSE (+1R rate 0.241 vs liq1's 0.437). So this isn't even a clean test of the live strategy.

Conclusion: on the data as it exists, there is no separable joint signal — the model confirms at the multi-feature level what the 7 single-lever tests found. This is NOT a clean rejection of "could a model ever work," because the potentially-predictive features (gap, structure, catalyst, tape regime) are unpopulated. The honest next move is a prerequisite data-plumbing fix (populate those features for live ORB setups + turn on catalyst enrichment so catalyst_freshness fills), then re-run this probe — with a strong negative prior. Absent that, the lever is a different entry (premarket axis), not a better filter on ORB.

Why this matters (read before building anything)

The project has refuted 7+ single-indicator selection levers (float, RVOL beyond a floor, liquidity, VWAP-reclaim, leading-gainer, premarket-volume, runner-propensity) and, this week, every exit variant (volume_fade wash, time_stop worse, asym_exit killed). The consistent finding: no lone indicator separates the ~50% of ORB setups that convert from the ~50% that stop out. The bottleneck is entry-conversion.

But every lever was tested in isolation. The move never tried: a model over the full feature vector. There are now 18,164 labeled setups (setup_events JOIN outcome_labels) with ~24 features (gap, rvol, VWAP position, float rotation, impulse/pullback structure, session minute, tape regime, catalyst freshness, SPY return) and a clean binary label (reached_1r_before_minus_1r). A model asks "does the combination separate winners?" and its feature importances are literally "which indicators matter, jointly" — the disciplined version of the operator's "surely some indicator works." The ML approach was correctly deferred before because n was tiny; it is no longer tiny.

The "humanistic / LLM / Alpaca-data" idea has a disciplined home too: the catalyst engine (Ollama + Alpaca news) is already built and gated off. The wrong use is a live gate (that is the refuted-folklore class). The right use is a feature: classify each setup's catalyst via Ollama over Alpaca news, add it to the vector, and let the model say whether the qualitative "story" separates winners.

The hard discipline (this project has been burned by look-ahead 3+ times): purged/embargoed time-series splits, strict out-of-sample validation, and shadow-first promotion through the experiment harness — never a live gate on an unvalidated score.

Current state

• Labeled dataset: setup_events (features) JOIN outcome_labels (labels) on setup_id, ~18,164 rows.
  • Feature columns (setup_events): entry_reference_price, invalidation_price, target_r_multiple, impulse_pct, pullback_depth_pct, pullback_bars, pullback_volume_ratio, above_vwap_flag, vwap_at_trigger, gap_pct, relative_volume, float_rotation_at_trigger, catalyst_freshness_at_trigger, session_minute_number, tape_regime_at_trigger, spy_intraday_return_pct_at_trigger (+ symbol, session_date, setup_time, setup_name, setup_version as keys/metadata — NOT features).
  • Label columns (outcome_labels): reached_1r_before_minus_1r (primary target), reached_2r_before_minus_1r, max_upside_next_{5,15,60}m, max_drawdown_next_{5,15,60}m, held_vwap_{5,15}m, trend_day_flag, failed_breakout_flag, time_to_max_upside_minutes.
• LEAKAGE RULE (non-negotiable): the model may use ONLY features knowable at setup_time. Every *_at_trigger field and gap/rvol/vwap/impulse/pullback/session-minute qualify. No outcome_labels column may be a feature (they are the future). target_r_multiple is a config, fine. Confirm each feature is trigger-time-knowable before including it.
• Experiment harness (research/experiments/) is the promotion path: a conversion-score gate becomes a registered experiment replayed over logged fires before it ever gates live.
• Ollama catalyst engine: config.py OllamaConfig; enrichment gated by NEWS_ENRICH_ENABLED (currently off). Alpaca news via the existing news ingestion. catalyst_freshness_at_trigger already exists as a feature (mostly null until enrichment runs).
• Tooling: scikit-learn is available in the analysis environment (1.9.0). No sklearn dependency exists in requirements.txt yet — Step 3 adds it only if the model ships.
• DB: DATABASE_URL → the production Postgres. Analysis is READ-ONLY. Extract with python3 -c "…" reading DATABASE_URL; never source .env.

Commands you will need

Purpose	Command	Expected
DB reachable	python3 -c "import os,psycopg2;psycopg2.connect(os.environ['DATABASE_URL']).close();print('ok')"	ok
Suite	DATABASE_URL=<url> python3 -m pytest -q	green (baseline: 15 WSL-loopback dashboard failures only)
ruff	ruff check --select F research/	clean

Scope

In scope:

• research/conversion/ (new package: dataset.py, signal_probe.py, model.py, cli.py)
• research/experiments/ — a new intervention/experiment that gates on the model's score (shadow), reusing the harness
• config.py / catalyst wiring — ONLY to expose the catalyst classification as a feature (Step 4), no live-gate change
• requirements.txt — add scikit-learn ONLY if Step 3 ships a model
• tests under tests/unit/conversion/
• plans/README.md (status row)

Out of scope:

• ANY live gate/sizing/exit change. The model produces a SHADOW score; promotion to live is a separate, owner-approved change through the harness (like every other lever).
• Turning NEWS_ENRICH_ENABLED on as a live gate. Catalyst is a feature only.
• Deep-learning / new heavy deps. Gradient-boosted trees (sklearn HistGradientBoostingClassifier) is the right tool for imbalanced tabular data and is already available.
• Predicting anything other than trigger-time-knowable → reached_1r_before_minus_1r (and optionally reached_2r). No regression on P&L.

Steps

Step 1 — GO/NO-GO: does separable signal exist at all? (leakage-safe CV AUC)

Build research/conversion/dataset.py (load the JOINed dataset, drop leakage columns, coerce types, one-hot tape_regime/setup_name) and research/conversion/signal_probe.py:

• Split by TIME, not randomly: sort by session_date; use sklearn.model_selection.TimeSeriesSplit (5 folds) with a 1-day embargo between train and test (drop the boundary day) so no same-day leakage.
• Train HistGradientBoostingClassifier (handles NaNs natively — important, many features are sparse) on each fold; score test-fold ROC-AUC and precision@top-decile (if you only took the model's top 10% most-confident setups, what's the +1R rate vs the base rate?).
• Report per-fold and mean AUC, the top-decile lift, and permutation feature importances on the held-out folds.

Verify: DATABASE_URL=<url> python3 -m research.conversion.signal_probe prints mean CV AUC, top-decile +1R rate vs base rate, and ranked feature importances.

GO/NO-GO gate (this decides the plan):

• Mean CV AUC ≥ ~0.55 AND top-decile +1R rate materially above base rate → signal exists. Proceed to Step 2.
• AUC ≈ 0.50 (0.47–0.53) and no top-decile lift → STOP and report: the data contains no trigger-time-knowable signal that separates conversion. That is the definitive answer to "is there an indicator" — there isn't one in this feature set, and the lever is a different setup (the premarket axis), not more modeling of this one. Do NOT proceed to build a model that fits noise.

Step 2 — Honest out-of-sample evaluation + calibration

Only if Step 1 passed. research/conversion/model.py: train on the first ~80% of sessions, hold out the most recent ~20% as a true forward test (never touched in tuning). Report forward AUC, a calibration curve (CalibratedClassifierCV), and the top-decile forward +1R rate. Persist the trained model + the exact feature list + the training date range to data/conversion_model.pkl (atomic write, like learned_params.json).

Verify: forward-test AUC within ~0.03 of CV AUC (no overfit blowup); calibration curve roughly diagonal. If forward AUC collapses to ~0.50 while CV was high → STOP, that is leakage or overfit; report which feature leaked.

Step 3 — Shadow score through the experiment harness (NOT a live gate)

Only if Step 2 held. Register a harness experiment conversion_gate_v1: intervention = "take only setups whose model score ≥ threshold τ"; replay over logged fires/setups (the harness already does this); pre-register the bar (expR_delta_min vs taking all). Add scikit-learn to requirements.txt. The loop may COMPUTE and LOG a conversion score per candidate (a new shadow field on the decision receipt) but MUST NOT gate on it.

Verify: python3 -m research.experiments run && … adjudicate shows conversion_gate_v1 accruing; score logged on receipts; zero change to live gate code (git diff touches no live-path gate).

Step 4 — The catalyst/LLM feature (the "humanistic" layer), as a feature only

Only if Steps 1–3 showed promise AND catalyst_freshness/tape features ranked meaningfully (i.e. qualitative context looks predictive). Wire the Ollama catalyst classification (news type, freshness, dilution risk from Alpaca news) into the labeler so it lands as setup_events features at build time; rebuild the dataset; re-run Step 1's probe WITH the catalyst features and compare AUC. If catalyst features lift AUC out-of-sample, that is the evidence to enrich; if not, leave enrichment off.

Verify: probe AUC with-vs-without catalyst features, reported side by side. Promotion of any of this to a live gate remains a separate owner decision.

Test plan

• tests/unit/conversion/test_dataset_no_leakage.py — assert the feature list contains ZERO outcome_labels columns and only trigger-time-knowable fields (the single most important test; a leaked label is how you get a fake AUC of 0.9).
• test_signal_probe_time_split — the CV uses time-ordered folds with an embargo (no shuffle); a synthetic dataset with a known signal recovers AUC > 0.5, pure noise recovers ~0.5.
• test_model_persist_roundtrip — save/load preserves the feature list + date range.
• Verification: DATABASE_URL=<url> python3 -m pytest -q green; non-loopback failures = 0.

Done criteria

• Step 1 probe runs and prints mean CV AUC + top-decile lift + feature importances
• The go/no-go was recorded (AUC value + decision) in plans/README.md
• If NO-GO: plan stops here with the honest "no separable signal" finding logged — this is a valid, complete outcome
• If GO: forward-test AUC + calibration reported; model persisted; harness experiment registered; live gate code unchanged (git diff proves it)
• Leakage test green; suite green
• scikit-learn in requirements.txt only if a model shipped

STOP conditions

• DB unreachable (the current blocker) — wait; do not fabricate results.
• Step 1 AUC ≈ 0.50 — STOP, report "no signal", do not model.
• Step 2 forward AUC collapses vs CV — STOP, report leakage/overfit and which feature.
• Any step requires touching a live gate — STOP, that is promotion (owner-gated).
• A feature turns out to be trigger-time-UNknowable (leak) — remove it, re-run, report.

Maintenance notes

• The model is a SHADOW predictor until the harness proves its score separates forward and the owner promotes it. Same discipline as every refuted lever — the graveyard is full of things that looked good in-sample.
• Feature importances are the real deliverable even if the model never gates: they tell the owner which of the 24 measured things actually co-vary with conversion, ending the one-indicator-at-a-time guessing.
• If Step 1 says no-signal, that is genuinely useful: it redirects effort from re-modeling this setup to the premarket axis (the one unexplored setup), which is a new entry, not a better filter on this one.
• Retrain cadence + drift: if promoted, the model retrains on the same nightly scheduler as nightly_tune, with the same staleness alarm.