Version: CI now tags images YYYY.MM.DD-<shortsha> alongside :latest/:sha
and passes GIT_SHA/BUILD_TAG build-args; the running app reports them via
dashboard /api/health and admin /api/status (TrueNAS custom apps only show
'custom/1.0.0', so the app self-reports what it actually runs).
Config parity: every scheduler-run script (nightly_tune, eod_replay,
premarket_eval, regime, market_context, experiments cli) now loads
config_overrides at startup like the live services do — a bare env would
replay under DEFAULT exits (the documented 4.5x-baseline landmine).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
premarket_eval's shadow enrichment call site queried session_date = current_date
(wrong on UTC-server Postgres at night, so it never matched any symbols) and
shared one try-block for float_cache + dilution filings (a float_cache failure
silently skipped the dilution refresh too). Fix: compute the ET session date
in Python, split into two independent try-blocks, lift into a testable
_refresh_shadow_enrichment(con, session_date). float_cache now applies its own
DDL unconditionally on use instead of only when an API key is present.
Adds iv_dilution_gate (additive, mirrors iv_regime_gate's keep-or-zero shape
keyed on symbol instead of session_date, fail-open on unfetched symbols) plus
its once-per-run preload, and registers the pre-registered dilution_veto_v1
experiment via the existing experiments/*.json self-registering cohort. Live
veto stays OFF; this only wires the shadow measurement.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Plan 006 step 3. evaluate() scores runs against pre-registered bars
(_min/_max/min_n vocabulary) -> PASS/FAIL/KILLED/INCONCLUSIVE; terminal
verdicts flip status, emit an experiment_verdict event, and Telegram-ping ONCE
(terminal experiments are never re-adjudicated). Metric vocabulary covers
expR_delta/rate_1r plus pm1's median_up_60m/median_dd_60m/big_gap_rate_1r.
backfill_pm1 bridges pm1 from outcome_labels JOIN setup_events (variant +/-1,
baseline NULL). premarket_eval dual-writes the verdict through the harness while
keeping the legacy TSV. 8 tests green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New EntryGateEvaluatedEvent (symbol, source, passed, reject_reason, gates map)
emitted once per _evaluate_entry_gates decision on both paths, behind
TRADING_GATE_RECEIPTS (default on; dataclass default off so unit tests opt in).
Added to the 14-day nightly purge tuple (high-volume diagnostic) and .env.example.
Turns 'why didn't it trade X?' into a query and is the substrate plan 006 replays.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two new nightly enrichments for the day's tracked names, both SHADOW
ONLY (float was refuted as a gate; a filings-based dilution veto needs
its own pre-registered bar):
- research/ingestion/float_cache.py: FMP stable shares-float ->
float_cache table (free-plan budget guards: 7-day refresh, 200
calls/run). First real float data in the system — LHSW's +346% ran
on a 1.1M float; BJDX is 0.4M.
- research/ingestion/sec_dilution.py: SEC EDGAR full-text search (free,
keyless, fair-use paced) -> populates the previously-empty
dilution_events table with S-1/S-3/424B filings; TVRD + BJDX flag
immediately. recent_dilution() helper for future veto work.
Wired into premarket_eval nightly. Not wired: Polygon/AlphaVantage/
TwelveData/Marketstack/EODHD (redundant vs paid Alpaca SIP) and the
generic news aggregators (slower than Alpaca/Benzinga, dedup noise).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Everything hot already lives in Postgres — local data/ was 25 MB of
pre-migration relics and unbounded logs. Moved the two dead .duckdb
files (untouched since 2026-06-16) to archive/, added nightly log
rotation (copy-truncate >20 MB, safe with systemd append mode), and
updated the last vestigial "market.duckdb" docstrings to say Postgres,
closing the CONSOLIDATION_PLAN cleanup items.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The events table hit 3.5 GB / 1.8M rows, ~80% of it module_tick /
criteria_evaluated / signal_blocked / account_* snapshots that are only
useful for a few days of debugging. Trading-decision events are kept
forever. One-time purge (353k rows) + missing hot-path indexes
(raw_news_items payload_hash + fetched_at: 911k seq scans -> index-only)
and per-table autovacuum tuning applied directly to the DB.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
With the nightly SIP backfill making all BASELINES consolidated-scale, the still-IEX live
ingest would make Monday's RVOL read iex_intraday/sip_baseline ~= 25-50x too LOW and choke
the scanner. Free-key SIP is only 15-min DELAYED, so each morning the true per-name capture
ratio is directly measurable:
- research/ingestion/volume_ratio.py: compute_ratios fetches BOTH feeds for the identical
embargo-capped window (never vs our DB bars — those are scaled; that would double-correct);
guardrails (min samples, clamp [1,300]); persist_ratios records per-day ratios.
- ingest_live_minute_bars(volume_scale=...): scales bar volumes per symbol at ingest, marks
source_provider=alpaca_iex_scaled. Missing symbol -> 1.0 (conservative: reads LOW, blocks).
Constant per-symbol scale leaves cumulative VWAP unchanged (cancels in the weighted mean).
- run_live_paper: "vratio" step every 300s (VOLUME_RATIO_ENABLED/INTERVAL) merges fresh
ratios (a name gone thin keeps its last good ratio); step_ingest applies them.
- premarket_eval nightly audit: snapshots the day's scaled ESTIMATES before the SIP truth
overwrites, then logs median/worst error to data/volume_estimator_audit.tsv — the
estimator's accuracy is self-measured daily, which is exactly the evidence that decides
whether the $99/mo real-time SIP plan is ever needed.
Live check (2026-07-02 full session): ratios are hugely name-dependent — DSY x300(clamped),
CETX x286, SOFI x45, SNDQ x42, NOK x17 — validating per-name daily measurement over any
global constant. 3 pure unit tests (ratio math + thin-sample skip; ingest scaling + provider
marking; no-ratio unchanged).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
MEASURED 2026-07-02: our entire bar history was IEX-only. IEX sees ~2-5% of consolidated
volume (SNDQ 7/01: IEX 188 bars/1.5M shares vs SIP 391 bars/60.1M — 39x) and misses whole
minutes on thin names. Every volume-based signal (RVOL, liquidity floors, volume bursts,
dollar-volume strata) and several research verdicts ("untradeable dust", "thin premarket")
were computed on that sample. Daily bars had the same defect. Historical SIP is FREE on our
keys (15-min embargo) — only real-time SIP needs the paid plan.
- client.get_daily_bars: feed param (None -> settings.feed; pass "sip" for consolidated).
- research/sip_backfill.py: re-fetch minute sessions (04:00-20:00 ET) + dailies from SIP
via the existing idempotent upserts; embargo-caps the end when refetching today;
refresh_rolling_volume rebuild after (this morning's 20d averages were IEX-scaled too).
CLI: full history < today, or --date D for one session.
- premarket_eval.py: nightly job now SIP-upgrades today's bars BEFORE the pm1 rebuild, so
every finished session lands on true volume automatically (live loop stays IEX real-time
by design — free-key SIP is embargoed 15 min).
Chain running: full backfill -> verification -> rebuild liquid/premarket/runner shadow
tracks + re-validate (their volume gates were IEX-calibrated; verdicts may change).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The pm1 verdict is promising-but-underpowered (n=21): the forward moves are real (+11-14%) but the
sample is too small to conclude. The live loop ingests premarket bars every trading day, so this
automates the "accumulate more days then re-validate" step:
- research/premarket_eval.py: rebuild pm1 from the grown DB, append a timestamped metrics row
(n, days, +1R rate, Wilson-95 LB, median upside/adverse, big-gap +1R, pass/fail) to
data/premarket_eval_history.tsv, and print the current validate output. Watch n climb + the
Wilson bound tighten until the locked bar is decisively cleared or failed.
- deploy/systemd/momentum-premarket-eval.{service,timer}: weekday-evening (Mon-Fri 20:00) oneshot,
Persistent=true (catches reboots). Installed to ~/.config/systemd/user and enabled.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
