# BDTLab batch backtesting — schema 1

## Human workflow
Sign in and open **Batch Test** at `/batch-test`. **Canvas** (`/canvas`) and **Single Test** (`/backtest`, also called solo) are separate pages in the left navigation. Select **Underlying SPY / options SPXW** or another downloaded mapping. Batch defaults to automatic dates, with an optional Manual · indexed dates range. The Manual From/Through choices are saved in this browser per signed-in user and ticker and revalidated on reload. Drop several JSON files or ZIP archives, or select them together. Parsing starts automatically, with Reading X/N progress; **Parse & validate files** explicitly rechecks the current selection without starting jobs. Review every row and the visible invalid-file error list, then explicitly click **Run all valid strategies**. This never invokes AI generation, market downloads, broker orders, or paid provider APIs.

100 strategies maximum; 10 MiB combined UTF-8 JSON. Files are parsed as JSON text, not paths, Python or JavaScript. Malformed individual JSON files and unsupported file envelopes remain visible as invalid rows with filenames and reasons; semantic errors remain per strategy while valid entries can run. Invalid entries are never converted into runnable DSL. If a whole upload fails (network, proxy, unsafe ZIP, or aggregate limit), the old preview is reference-only and Run stays blocked until revalidation. The visible Run explanation identifies reading, active job, import failure, unavailable dates, or no valid entries. **Retry dates** retries failed scope/catalog loading. Requests time out after 60 seconds and unlock validation; a timed-out submission must be checked in saved batches before retrying. Duplicate names are allowed and never overwrite: IDs combine ordered position and original recipe hash. Raw recipes without names get `Strategy N`. Atomic composition 1 imports retain their original supported semantics.

Single Test's user-chosen ticker/dates persist separately and are never overwritten by Batch Test. **Automatic** Batch uses the oldest validated downloaded market date through the newest available market date minus **one calendar day** (America/New_York session dates, not UTC fetch timestamps and not today minus one). Only actual downloaded dates at or before that cutoff are eligible: latest Monday gives a Sunday cutoff and typically a Friday actual end. The newest downloaded date is excluded even if complete. Zero/one-date datasets have no eligible automatic history. The automatic UI shows the cutoff separately from the actual last eligible date and count. Submission reads the catalog once; later downloads cannot add dates to an active batch. All accepted batch entries receive the same sorted downloaded date vector, frozen local route choices, source identity snapshot and cost settings. Missing dates between endpoints are not synthesized. Strategy-specific missing evidence remains unavailable. Each recipe has independent capital and chronological equity.

## Manual indexed date scope
Manual accepts explicit inclusive From/Through endpoints only when both are present in the selected ticker's downloaded catalog. Only actual indexed dates inside the interval run; gaps are not fabricated. The newest indexed date can be included only by explicitly selecting it. A single indexed date can be run manually. The preview reports count and endpoints, but submit independently freezes the catalog and routes, and the worker revalidates file identities at execution. A changed/missing source fails rather than fetching or assuming a trade. Stale saved browser dates remain visible with Run disabled until corrected. These are session dates, not new option-chain or continuous-NBBO evidence.

## Worker performance and verification
The Batch worker checks the complete frozen dataset at start and completion, and verifies only the current date's frozen file identities and manifest hashes before and after that date's replay. A changed or newly resolved source fails the final complete comparison; a changed current-date file/manifest fails during that date. This avoids rescanning every indexed date twice *per date*. Running progress writes contain only the small progress object; full per-recipe results, source identities and costs are committed on successful completion. A failed job records its failure rather than claiming partial results are complete. A service restart marks unfinished jobs interrupted without resubmitting. The production Batch worker reads the frozen source directly rather than rewriting a prepared per-symbol tape; a service-identity ABBA replay on two real NAS-backed dates measured the direct path faster for the tested 13 recipes, with identical result hashes. Two bounded market readers load independent recipes per date; simulation, capital, output ordering and before/after date provenance checks remain sequential. This is an observed workload-specific optimization, not a guarantee for all future recipe mixes. The optional explicit `PreparedTape` utility remains testable but is not used by production Batch. Single Test independently uses a bounded two-day market-read look-ahead; its capital and simulation remain chronological. A 12-date real-data service-account ABBA check matched complete result hashes and measured the two-reader runs faster, but no full-range throughput guarantee is implied.

## ZIP limits and safety
ZIPs may contain JSON recipes, arrays, or schema 1 envelopes in nested folders. Filenames/member paths are shown as text; unnamed recipes use the member name. Duplicate names remain separate ordered IDs. Malformed JSON members and invalid recipes stay visible as invalid rows; valid entries may run only after explicit confirmation. Files are never extracted to disk or executed.

Limits: 10 MiB compressed aggregate uploads, 10 MiB uncompressed aggregate, 1 MiB per ZIP member, 250 members, 100 total strategies, 12 path components and compression ratio <=200. Stored/deflate only. Encrypted, corrupt/CRC-failed, unsafe absolute/drive/backslash/traversal paths, symlinks and unsupported methods reject explicitly. Directory entries are ignored. Non-JSON files and __MACOSX/.DS_Store metadata are ignored and counted; nested ZIP files are not expanded. Every JSON member uses the same compiler/envelope validation. No CDN ZIP library or filesystem extraction.

## Canvas sessions and theme
Canvas keeps a versioned, account-scoped **tab-local sessionStorage** draft, including incomplete graph, committed recipe, viewport, open card editor draft and undo. Navigation/reload restores it; Cancel retains original card semantics. This never overwrites the explicitly saved browser recipe. Tabs are isolated; an older tab cannot overwrite another. Sign out clears that account's session draft. Quota/storage failures show NOT saved and provide Export session draft; no false persistence claims.

Check my recipe compiles on the server. Failure stays on Canvas. Success stores the exact checked input snapshot with revision and SHA-256 (sorted-key JavaScript canonical JSON), then opens Single Test without starting a job. Single Test revalidates the snapshot and waits for downloaded coverage before Run is enabled. JSON import there is independent of the canvas draft. Shared top-right keyboard-accessible Light/Dark switch changes semantic surfaces immediately and persists a browser preference separately from session drafts. First visit defaults to light.

## AI import
Each file can contain one existing v2 recipe, an array of raw recipes / named entries, or:

```json
{"schema_version":1,"strategies":[{"name":"Morning A","recipe":{"version":2,"name":"Morning A","start":"read","nodes":[{"id":"read","kind":"read_candle","params":{"symbol":"SPY","timeframe":"5m","time":"09:30","evaluate_at":"09:35","timezone":"America/New_York","variable":"candle"},"next":"buy"},{"id":"buy","kind":"buy_option","params":{"symbol":"SPXW","side":"call","time":"09:35","cutoff":"09:36","deadline":"09:40","timezone":"America/New_York","target_pct":50}}]}}]}
```

Illustrative syntax only, not a profitable recommendation or promise of complete data. Batch adds **no new DSL, indicators or market fields**. Existing compiler rejects unknown fields/parameters, unsupported indicators, loops, paths and code. Error text is attached to the relevant row. Envelope accepts only schema_version and strategies; named entries accept only name and recipe.

## Deployment boundary
The HTTPS proxy must allow 10 MiB for `/api/batches` and its child paths, matching the existing bounded API. Keep unrelated route limits unchanged. ZIP preview is followed by a combined JSON preview, so allowing only compressed ZIP upload is insufficient. HTML 413 responses identify a proxy/body-size rejection, not invalid strategy DSL. Public deployment probes must exercise 28 JSON files and a 28-member ZIP, not only loopback or two-file fixtures.

## Authenticated API
- `POST /api/batches/preview`: body is the import document. Returns ordered strategies with valid/invalid, errors, stable strategy_id and original_strategy_hash (SHA256 of canonical original recipe JSON).
- `POST /api/batches`: `{ "document": <import document>, "config": {"ticker":"SPY/SPXW","date_scope":"auto","initial_capital":10000,"contracts":1,"fee_per_contract":0.65,"slippage":0,"evidence_mode":"trade_nbbo"}, "run_all_valid":true }`. Explicit true required. Returns **202** `{id,status:"queued"}` after local admission; no synchronous full replay. Unknown config fields reject. Recipe symbol roles are not rewritten.
- `GET /api/batches/scope?ticker=SPY%2FSPXW`: authenticated authoritative automatic scope preview. `date_scope=manual&start_date=YYYY-MM-DD&end_date=YYYY-MM-DD` returns the independently validated inclusive indexed selection; missing or nonindexed endpoints return 422. Automatic preview returns cutoff_date; both return selected_dates, local_routes, date_count and policy. Submission reconfirms against one frozen catalog.
- `POST /api/batches/preview-zip`: raw `application/zip` body; authenticated, rate-limited, in-memory bounded standard-library reader. Returns document (normalized import envelope), strategies (same compiler validation), per-row members, ignored_files and uncompressed_bytes. Does not create a job.
- Automatic submission rejects explicit start_date/end_date; manual submission requires both indexed endpoints. Unknown `date_scope` and forged selected_dates/local_routes reject with 422. The result's date_selection records the chosen policy, selected dates and count; auto_scope remains for automatic runs and is null for manual runs. Historical saved jobs and exports remain readable.
- `GET /api/batches`: signed-in account's recent saved batches.
- `GET /api/batches/{id}`: owned progress/results; other accounts get 404.
- `GET /api/batches/{id}/download`: UTF-8 JSON attachment, safe UUID filename. Same ownership guard.

Result schema_version 1: batch_id, status, evidence_class, synthetic=false, ticker_pair, requested_dates, resolved_dates, dataset_snapshot (file identities plus fingerprint), cost_provenance, timestamps, progress, cache_metrics, ordered results. Each result retains name, strategy_id, original_strategy_hash, validated recipe/config, status, stats, errors and final engine result. Stats use existing names: **net_pnl, trade_count, win_rate, max_drawdown, max_drawdown_pct, unavailable_days, initial_capital, ending_capital, evidence_complete, synthetic**. Win rate is null without trades. Failed/invalid stats are null, never invented zero winners. A completed no-trade or unavailable engine audit may contain zero net P&L with evidence_complete=false; that is not positive market evidence. Full days/trades/equity audit is included.

## Runtime and reproducibility
Shared bounded worker pool/queue with ordinary backtests. One chronological engine iterator per recipe, advanced **date first, then strategies**. Capital never crosses recipes. Full-tape batch preparation filters the selected option product's raw full-day events to a streaming temporary parquet, not an entry-window or selected-contract cache. Key includes resolved source identity (path/size/mtime_ns/ctime_ns/device/inode), date, route, product and preparation version. Every recipe applies its own complete candle/evaluation/window/side/selection filters afterward; future events do not enter earlier decisions. No reuse of a FUN RUN-only subset for morning strategies.

Only one day/product prepared file is retained; 2 GiB disk budget, streaming row groups 32,768. A source above the conservative budget falls back to its original lazy reader (no warm-reuse promise for oversized files). No full-year RAM cache. Temporary files are removed on normal completion/failure; hard process kill can leave OS-temp files. Existing per-window materialization and final trade/audit output still consume memory; this is not a constant-memory output guarantee or a benchmark for 100 strategies × years. New independent batches do not retain previous batch temp files. Committed checksum validation uses the existing identity-bound verification index; no forced whole-lake checksum rebuild.

No cancellation and no automatic resubmission. Browser refresh reattaches durable jobs. Server restart marks active batches interrupted; it does not resume partial engines. Diagnostics are separate **dbo.BacktestBatches**, never immutable AIResearchResults or user strategy records.

## Actual time/market capabilities — batch first
Full-tape v2 time cards genuinely drive read_candle time/timeframe/evaluate_at and buy time/cutoff/deadline. They are not restricted to the FUN RUN afternoon window; the example uses 09:30 ET. Only completed actual underlying minutes and available option trade-event bid/ask evidence are accepted. SPY is the signal underlying, SPXW the actual option product; no SPY-share substitution.

This release does **not** introduce rolling all-session evaluation, automatic 09:30–16:00 / early-close schedule defaults, multiple opens per recipe/day, post-entry candle indicators, premarket/after-hours tradability, continuous NBBO, verified full option chain/Greeks, non-ATM or arbitrary expiry selection. Recipes retain explicit times; blank settings are not silently assigned FUN RUN or full-session defaults. Legacy SPXW data retains its explicit original limited windows. Full-session/early-close evaluator is the next dedicated scope, not an unverified claim of this batch release.

## Verification
Verified live batch `553742e3-9f03-4547-9bbe-851b071cbbe5`: two valid duplicate-named recipes with different morning/afternoon schedules plus one intentionally invalid entry, fixed C: dates 2023-10-17–18. POST returned 202; request-to-response-start 1.737s; sampled engine completion 2.672s. Two cold preparations / two cache hits; 882,274,312 source bytes, 78,651,130 cumulative prepared bytes, peak temporary disk 39,908,402 bytes; preparation 2.187s. The two subsequently isolated, OS-warm lazy-reader runs took approximately 0.281s and 0.187s. **This small batch demonstrates safe reuse, not a measured overall speedup**; preparation overhead dominated. Larger-scope latency and peak RSS have not been benchmarked.

Both real recipes reported two unavailable dates and no fills. Investigation of 2023-10-17 found morning and afternoon SPXW events with correct sides and same-day expiries, but attached `underlying_price` was null throughout both probed windows. The selector cannot establish contemporaneous near-ATM evidence; widening the clock or substituting SPY price would not fix that legitimately. Browser displays no-trade P&L as unavailable, not a winner. Isolated engines exactly matched both summaries, days, trades and equity. Saved JSON export is `artifacts/batch-export-553742e3-9f03-4547-9bbe-851b071cbbe5.json`.

See `artifacts/batch-backtester-progress.md` and `artifacts/batch-live-verification.json` for release status, saved diagnostic ID, real timing/cache counts, source paths, screenshots and limitations. Fixture equivalence proves chronological accounting and different-window cache safety, not profitability. Live diagnostic remains fixed two recipes × two C: dates; never a broad research sweep.