α
AlphaSteve
← Brain index

Market-Data Ingestion Standard (T0)

Adopted 2026-06-23. Companion to data-integrity-gate (which hardened the fundamentals path on 2026-06-10) and benchmark-sourcing-discipline (the single-closes-file convention). This file closes the remaining hole: the price/rate surface was still being read from the news and hand-typed, which is the root cause of audit entries #003, #005–#010 and the recurring closes-file P1s.

The principle

The agent never types a number it could look up. Market figures enter the vault through a deterministic tool that pulls from a sourced API, writes a dated artifact, and records provenance. Agent prose reasons over those figures; it does not transcribe them. This is the same pattern the edgar-fundamentals-standard already proves for fundamentals — now extended to the close surface.

This is a T0 (ingestion-time) control. Conflicts and staleness are resolved when the data is pulled, before any note is written — not discovered the next morning by the optimization audit, and not narrated as a "judgment call." A resolved conflict is a manifest entry, not a sentence.

Architecture (and why the puller runs on the host)

The agent sandbox cannot reach market-data APIs. Verified 2026-06-23: bash networking is firewalled; the agent web-fetch allowlist serves data.sec.gov (SEC EDGAR) but returns empty for data.nasdaq.com (Sharadar) and api.stlouisfed.org (FRED API). A scheduled agent task therefore cannot pull prices. So the deterministic pull runs on the user's machine (real internet) via Task Scheduler, after the cash close, and writes into the vault. The agent reads the artifact.

  [user machine cron, 18:15 ET]                     [agent scheduled tasks]
  _Tools/marketdata/market_close.py   ──writes──▶   13-Research/Daily-Scans/<DATE>-closes.md
     ├─ Sharadar SEP  (equities)                          + <DATE>-closes.json  (manifest)
     ├─ Sharadar SFP  (ETF index/commodity proxies)              │
     └─ FRED CSV      (rates, VIX, oil)                           ▼
                                                    PM scan / portfolio / AM scan READ it
                                                    (no independent web pull — ever)

Source map

Surface Source Notes
Equity closes (watchlist, portfolio, theses) Sharadar SEP closeunadj = headline close; day % from prior closeunadj
Index / commodity level Sharadar SFP ETF proxies: SPY, QQQ, DIA, IWM, GLD Proxies, not literal index prints — cleaner and sourced; labelled as proxies
Rates, VIX, oil FRED official daily series (keyless CSV) DGS10, DGS2, VIXCLS, DCOILWTICO, DCOILBRENTEU
Forward event dates (earnings, Fed) issuer 8-K / IR (EDGAR), not a week-ahead calendar the GIS-class fix; route through the issuer, never an aggregator calendar

The provenance manifest

<DATE>-closes.json carries, for every figure: value, source, table, asof, lastupdated, status, note. This is the machine-checkable record the optimization audit reads to verify the gate ran, instead of re-deriving numbers from a vault that is already internally consistent around them.

Hard rules

  1. No fabrication. If the source is unreachable or the key fails, the puller writes nothing and exits non-zero. Closes are never backfilled from a web guess.
  2. Settlement gate. No same-day close is asserted before 16:15 ET. A pre-close run yields status: not-yet-settled. (This is the structural fix for the afternoon tasks that fire before the cash close.)
  3. Staleness is explicit. A source value older than the requested date is STALE; a missing ticker is MISSING; overall status degrades to partial. Readers use the prior settled close and note the gap — never a fresh independent pull.
  4. Readers do not re-pull. Per benchmark-sourcing-discipline, the PM scan, portfolio-daily, and AM scan READ the closes file. They never pull prices from the web independently.
  5. Event dates carry a T1 issuer source. A forward earnings/Fed date written as fact must cite the issuer's announcement (8-K / IR), not a calendar aggregator.

How the audit changes

The daily optimization run stops trying to re-discover price errors (it can't reach the data anyway) and instead audits the manifest: confirm <DATE>-closes.json exists, status is acceptable, no figure is MISSING/STALE without the reader noting the gap, and every market number cited in the day's notes traces to the manifest. Audit moves from "is the vault consistent with itself" to "does the vault match the sourced ground truth."

Operational

Setup, scheduling, and the connectivity finding: _Tools/marketdata/README.md. Universe is edited in _Tools/marketdata/config.json (no code change to add a ticker).

Linked