I used to be a better software engineer than AI. Claude Opus 4.7 changed that.

I was quite proud of my backtesting system.

Lightning fast. Deterministic to the bit. And expensive.

Very expensive. At peak, over $1,000 a month across compute and disk, with $230 of that going to duplicated market history on rent even when the machines were idle. Every machine that served a backtest carried its own private 200 GB copy: prices, options chains, fundamentals. Two machines meant 200 GB of identical bytes billed twice. It was the most optimal design I could build with Claude Opus 4.1 and Gemini 2.5 Pro, over about a week of focused work last September.

Then, last week, Anthropic released Opus 4.7 at the same $5/$25 per million token price as Opus 4.6, with 90% off warm reads via prompt caching. On the API, Bedrock, Vertex, and Copilot on day one. The numbers:

Ten points over Opus 4.6 in six months. Seven-point lead over GPT-5.4. Anthropic also reports 3× the production-engineering task resolution rate of Opus 4.6 on Rakuten's internal SWE-bench. Benchmarks only go so far.

What I actually cared about was if it could implement a large, complicated feature end-to-end. I wrote out the plan: the target architecture, the fingerprint test as the correctness gate, the data types to migrate, the staged cutover. Then I handed it to Opus 4.7 and stepped back.

Every thirty minutes or so I'd check in: diffs looked right, the fingerprint held, commits read coherently. Otherwise I trusted the automated code reviews and my own review pass at the end to catch anything I missed.

Unlike the week I spent driving the mmap build as the lead architect, this time I mostly let it cook.

Every load-bearing architectural decision came from Opus: the three pruning layers, the concurrent chunk fetcher sharing one prefetched metadata, the condvar-backed appendable reader, the four nested retry layers. I read the diffs and ran the fingerprint. That's the difference this time.

And honestly, I was shocked.

The NexusTrade backtest engine, briefly

NexusTrade is the AI trading platform I've been building since 2020. Describe a trading idea in plain English. Aurora (our AI agent) creates a full algorithmic trading strategy: entry rules, exit rules, position sizing, options legs when relevant. The backtest engine runs that portfolio over real market history and reports what would have happened. Determinism is non-negotiable: identical inputs must produce identical outputs, bit-for-bit. Sub-second backtests feel magical; ten-second ones feel broken.

The magic is the goal.

That engine is what I handed Opus 4.7 this weekend. By Sunday night the architecture I'd been running since last September was gone.

200 GB of data on every worker, billed every time

For about seven months, the engine ran an mmap architecture: a Rust hydrator downloaded Parquet shards from GCS on startup, decoded them into consolidated .bin files on a local persistent disk, and the backtest process memory-mapped those files at request time. The hot path was fast because, after the first read of each file, the OS kept those bytes in RAM, so every subsequent read hit memory, not disk. Every worker carried the same 200 GB of market history.

The deployment fought back at every stage. Live trading had to be split off on day one (a long-running backtest can't share a CPU with a process that needs to place real orders in milliseconds), so two machines with their own 200 GB copies was the starting point. Then optimizations started hogging the CPU on the backtest box, so optimization moved onto its own Render machine. As user load grew, I had to stand up additional backtest Render instances by hand, each with its own 200 GB disk, because Render persistent disks can't be shared or auto-scaled.

By the time I'd moved compute to Fly, the fleet was three Fly backtest workers (one always-on primary + two reserves), three Fly optimizer reserves (no primary: optimization was never on the hot path for user traffic), and one Render live-trading instance. Seven machines, each with its own 200 GB disk: 1.4 TB of duplicated market history on rent. At Fly's $0.15/GB/month for volumes and Render's $0.25/GB/month for SSDs, that came out to $230/month purely for duplicated disk. Scale-to-zero saved nothing on that line: volumes bill whether the machine is running or not.

The bill got untenable. I eventually dropped the always-on backtest primary and one optimizer reserve just to survive, trading peak throughput for a cost I could stomach. That's where the architecture sat when Opus 4.7 dropped.

How mmap served a backtest

Why that stopped working

Four infra stages tried to scale the design without dropping the "fast hot path" property. Every stage ran into the same wall: the 200 GB footprint was per-machine, and machines were multiplying.

• Stage 1 · Render · day one
  400 GB
  1 Render box for backtest + optimization, 1 for live trading. Optimizations hogged CPU and starved user backtests.
• Stage 2 · Render · split + scaled
  800 GB
  Optimization gets its own box. To handle more users I manually stood up additional backtest instances, each with its own 200 GB disk (Render persistent disks can't be shared or auto-scaled). 4 Render instances, same data × 4.
• Stage 3 · Fly + mmap · peak
  1.4 TB
  3 Fly backtest (1 primary + 2 reserves), 3 Fly optimizer reserves, 1 Render live-trade. Seven 200 GB disks, each one its own copy of the same data.
• Stage 4 · Fly + mmap · cost cuts
  1.0 TB
  Dropped the always-on backtest primary and one optimizer reserve to survive Fly's volume bill. Worse throughput at peak load, survivable cost.
• Stage 5 · Today
  0 GB
  Stateless workers. Scale-to-zero on compute AND disk. One shared source on Tigris, streamed on demand.

So how did Opus 4.7 replace it? Start with the shape.

One shared data layer, no required local disk

Every Fly worker now boots with no required persistent state. Tigris (S3-compatible, Fly-hosted) is the source of truth for Parquet shards. No mandatory binary cache on disk, no hydration at startup. All thirteen data types (stock intraday, daily OHLC, options intraday, options daily, fundamentals, dividends, splits, economic, index signals, reports, financials, earnings, crypto) go through the streaming path.

Warm data loads in pointer-bump time. A new worker serves its first backtest in seconds, not hours of hydration. One always-on primary carries an optional disk-LRU tier for second-touch wins on options and intraday workloads (covered in the postscript to the trade-offs section below); every other worker runs with zero local disk. But how is any of that possible?

The new architecture, end to end

Here's the full path of a cold backtest on the streaming stack: a fresh worker that has never served a request, no local disk, nothing cached. Click through the tabs to walk the request from "strategy + time window" all the way to "backtest result."

That's the operational walkthrough. To understand why each of those steps is fast enough to add up to a sub-second warm backtest, four levers do the work: cut bytes, hide the network, keep the warm path instant, survive failures.

The first lever: don't fetch bytes you don't need

Four sections of deep mechanics follow. If that's not what you're here for, skip ahead to the takeaway.

A minute of SPY intraday in 2025 is a few hundred thousand option contracts. A month is tens of millions. The reader that tries to download all of it for a four-day backtest has already lost. Three mechanisms attack the same problem from three directions: change the storage layout so filters read 80× less data, prune whole row groups before they leave Tigris, and narrow the columns on data types where the backtest only needs one.

Why the decoder stores columns, not rows

The biggest single performance characteristic of the new decoder: every field gets its own tight buffer instead of 80-byte interleaved row structs. The options resolver's filter chains (find all calls in a DTE window with strike near $450) touch one column at a time. In the old row-form layout a single 1-byte option_type check pulled 80 bytes per row into cache. In columnar, it pulls one. Same data, 80× less memory bandwidth.

Three layers of row-group pruning

A Parquet shard on Tigris is split into dozens of row groups, each a few thousand rows. The reader's job is to avoid downloading the ones it doesn't need. Three independent filters run before any bytes leave the bucket, and each catches a different access pattern the others miss.

Timestamp min/max. Every row group's footer records the min and max timestamp it contains. A backtest on March 5th skips every row group whose range doesn't overlap the day. This is the coarse filter; it's what makes "give me one day out of a ten-year shard" free.

Symbol min/max. The writer emits shards with ORDER BY (ticker, date) for daily types and ORDER BY (underlying, timestamp) for intraday options. That ordering turns the per-row-group ticker min/max into a real filter: a {SPY} backtest reads the row groups whose ticker range touches SPY and skips the rest. Without the writer's ORDER BY, every row group's min/max straddles the full alphabet and the filter is a no-op. The reader and the writer have to agree for this to work.

Bloom filter. Min/max is a range check: it admits row groups whose range contains the ticker you asked for even if that specific ticker never appears in the group. A Bloom filter in the row-group metadata answers the precise question "could this ticker be here?" with false positives but no false negatives. For narrow universes on fat shards, this is the layer that drops the last unnecessary download.

A row group that survives all three filters still might contain rows for tickers outside the universe, because min/max and bloom are group-granular, not row-granular. So one more filter runs after the decode, in memory: a ColumnFamily::filter_rows_by_asset_ids step that permutes each Arc<[T]> column through a kept-index list. It sits after the LRU publish, not before it, so a {SPY} backtest and a {QQQ} backtest on the same machine share one cached row-group buffer and each applies its own filter to the same Arc<Cols>. The cache-key space stays universe-agnostic.

stock_intraday narrows to a lookup table for options DAY backtests

One last byte-reduction trick, narrower but meaningful. An options backtest at daily granularity uses stock_intraday only to resolve the underlying's close at decision time. It doesn't need open, high, low, or volume. Loading all five columns for the sole purpose of reading one is 40–60% wasted bytes.

When the backtest is options-only at DAY granularity, the reader narrows the projection to ticker, timestamp, and close. The other OHLCV columns are NaN-poisoned rather than decoded. MINUTE and HOUR backtests still read the full schema; the narrowing only kicks in where it's semantically safe. Same fingerprint, less bandwidth.

Reducing work alone isn't enough. Even after pruning to two row groups, those two groups still have to come over the network. The next cluster is about making that network cost invisible.

The second lever: overlap the network with everything else

The bytes we do need still have to travel from Tigris to the worker. Two mechanisms keep that travel from showing up as wall time: fan the fetches out in parallel, and let the backtest start running before the full dataset has arrived.

Concurrent row-group fetches per shard

parquet-rs's ParquetRecordBatchStream reads row groups sequentially. It asks for group 1, waits, asks for group 2, waits. For a shard with 40 surviving row groups on a 50 ms RTT link, that's two seconds of pure round-trip time even if each individual fetch is free.

The reader fans the work out itself. After pruning, the surviving row groups for a shard are split into N chunks. N builders are constructed against the same Arc<ArrowReaderMetadata> (the footer is prefetched once and shared), and each builder streams its chunk on its own task. HTTP/2 multiplexes the concurrent requests onto a single connection to Tigris; the bucket sees the fan-out, the process sees N times less wall time. Decoded row groups land in the LRU in whatever order they arrive; the singleflight (covered in the next cluster) means a late arrival for a group another task already published is a cheap no-op. Default N is 4, tunable by env var.

Pipelined feeding: data loads while the backtest runs

A twelve-month intraday backtest reads data in timestamp order. The iterator consumes month 1 entirely before it touches a single row in month 2. So month 2 doesn't need to be fully decoded before exec starts; it just needs to be ready by the time the cursor crosses into its range. The architecture exploits that.

AppendableColumnarReader is the primitive. It implements the same IndexedReader trait the merge iterator already knows how to consume, but its internal state is a Mutex<ReaderState> + Condvar. When the iterator asks for a row past the currently-loaded tail, it parks on the condvar. When a background fetch task calls append_chunk, the condvar wakes. A 120-second watchdog fails loud if a producer dies.

// Consumer side: called by KWayMergeIterator, blocks until data lands
fn visit_point_at(&self, idx: usize, visitor: &mut impl Visitor<T>) {
    let mut state = self.state.lock().unwrap();
    loop {
        if let Some(block) = state.block_for(idx) { return block.visit(idx, visitor); }
        if state.is_complete { return; }             // producer said "no more"
        if let Some(err) = &state.error { panic!(...); }
        let (g, t) = self.cond.wait_timeout(state, DEADLOCK_GUARD).unwrap();
        if t.timed_out() { panic!("appendable reader stuck >120s"); }
        state = g;
    }
}

// Producer side: called from the tokio::spawn feeder task
pub fn append_chunk(&self, cols: C)       { /* push + notify_all */ }
pub fn mark_complete(&self)               { /* set flag + notify_all */ }
pub fn mark_error(&self, msg: String)     { /* set err + notify_all */ }

The seed-then-spawn pattern is the orchestrator. The first month's fetch is awaited synchronously because the iterator needs something to emit. For every subsequent month, the reader wraps an empty AppendableColumnarReader whose min_timestamp is parsed directly from the "YYYY-MM" key string, so the MultiMonthReader's chronological sort is correct before any chunk has landed. Then it spawns the feeder and returns.

// seed: first month blocks, iterator needs it to start emitting ticks
let (seed_key, seed_cols) = fetch_one_key_cols_owned(keys[0], ...).await?;
let seed_reader = build_columnar_reader(seed_cols, ...);
let mut readers = vec![(seed_key, seed_reader)];

// pending: every other month. declare min_timestamp from the key string,
//          wrap an empty appendable reader, spawn the feeder
for key in &keys[1..] {
    let min_ts = min_timestamp_from_key(key)?;      // parses "2024-03" → ns
    let reader = AppendableColumnarReader::new_empty(min_ts);
    let handle  = reader.clone_handle();
    tokio::spawn(async move {
        match fetch_one_key_cols_owned(key, ...).await {
            Ok((_, cols)) => { handle.append_chunk(cols); handle.mark_complete(); }
            Err(e)        => handle.mark_error(e.to_string()),
        }
    });
    readers.push((key.clone(), AnyReader::AppendableColumnar(reader)));
}
MultiMonthReader::from_sorted(readers)   // sorts on declared min_timestamp

The wait function changes shape. Instead of Σ(per-month), the backtest waits max(yearly_types, seed_month) plus whatever the iterator races against as it walks forward in time. On a twelve-month intraday bench that's roughly twenty seconds of serial decode collapsed to one or two seconds of seed wait, with everything else hidden under exec. Options pipelining works the same way, with NBBO enrichment moved inside each per-month task so there's no cross-key post-decode barrier.

Cold is the first backtest on a fresh worker. Everything after that should feel free. The next cluster is about why.