Partition ownership and burst capacity
Give each workload its own scaling policy
At scale, the workloads stop looking alike. Web traffic is fairly steady. Data arrives in bursts. An optimization can launch thousands of backtests and then sit idle. Live trading still needs one owner per user, and adding machines does not increase brokerage API quotas.
Back-of-the-envelope interview assumptions
| Workload | Illustrative load | Verified implementation and design response |
|---|---|---|
|
Minute bars Interview assumption |
At 5,000 instruments, each minute closes 5,000 bars: 1.95 million per session and 83 events per second on average. Provider batches sharpen the boundary burst, while concurrent backtests request overlapping historical slices. |
Publish one immutable columnar snapshot, then reuse decoded
market columns from a memory-bounded least-recently-used cache.
NexusTrade stores the snapshot as Parquet and implements the
cache as UnifiedColumnarLru. Its key includes the
dataset, partition, time range, selected columns, and symbol
filter.
|
|
Strategy evaluation hot path Interview assumption |
At 100,000 portfolios, 5 ms per evaluation requires 500 CPU seconds per cycle. Recursive condition trees, repeated enum dispatch, asset hashing, and rebuilding equivalent indicator state multiply that cost before another machine helps. |
CompiledCondition and
CompiledIndicator flatten trees into stack-machine
programs; CachedIndicatorMap stores dense per-asset
state. Profile release builds with
cargo flamegraph, DTrace plus Inferno, or
perf. Benchmark the widest frames, then partition
the remaining work.
|
|
Optimization Interview assumption |
A 10,000-candidate sweep produces thousands of backtest work items inside one durable optimizer job. Candidate runtimes vary, so stragglers determine tail latency and heavy candidates may exhaust memory before CPU. If a worker disappears, the unfinished part of the sweep may need to run again. | Let one worker reserve the optimizer job with an expiring lease, then reject writes from any older owner. NexusTrade stores the lease, ownership generation, and reservation token in MongoDB. A bounded Rust pool separates light and heavy candidates, while short-lived Fly Machines add capacity as demand grows. Dead worker reservations can be requeued up to three times. Live trading keeps separate capacity. |
|
Persisted live events Measured event shape plus capacity model |
A live evaluation produces a small event set rather than one
generic update. In a 48-hour sample of seven active
constant-frequency
RebalanceOption portfolios, a normal event-bearing
second persisted three rows per portfolio. The median and 95th
percentile were both three. Order activity briefly raised one
live portfolio to 14 rows in a second. The five paper and two
live-brokerage portfolios averaged 18.4 to 20.1 rows per
event-bearing minute because Constant mode evaluated several
times per minute. This supplies a concrete per-evaluation event
shape for capacity planning.
|
Show both planning rates. At the target cadence of one evaluation per wall-clock minute, 100,000 portfolios create 300,000 rows per minute, or 5,000 per second. Scaling the observed Constant-mode rate linearly produces 1.84 to 2.01 million rows per minute, or about 30,700 to 33,500 per second. Constant mode is a stress profile rather than the target once-per-minute rebalance cadence. Stagger evaluation clocks, partition ingestion and export, and keep browser projection outside the trading loop. |
|
Broker submission Current product contract |
Strategy evaluation can create many order intents, while a live
order still requires durable approval before any broker call.
Unapproved orders wait in PendingUserApproval, so
throughput depends on human review as well as venue quotas.
|
Persist intent first and collect one explicit user approval per order or rebalance unit. A current portfolio policy can approve eligible actions after owner authorization and current consent, within its daily trade-action cap. Stable client IDs, per-account limits, and reconciliation protect the final broker boundary. |
The 5,000-instrument, 100,000-portfolio, and 10,000-candidate figures are system-design capacity targets. The per-evaluation event shape was measured from anonymous persisted metadata between August 27 and August 29, 2026. Named classes and batching constants come from the current NexusTrade implementation; the larger workload shows how to turn those measurements into partition counts and scaling decisions.
Production target cadence
A normal evaluation persisted three typed rows: market context, option-close evaluation, and the decision outcome. Normalized to one wall-clock evaluation per minute, each portfolio contributes three events per minute. At 100,000 portfolios, the design target is 300,000 durable events per minute plus burst headroom.
Observed stress reference
The current source defines independent Rust backtest and optimizer roles that reserve durable jobs stored in MongoDB. The checked-in Fly configuration defines three live-trading shards, and the routing code maps each user to one shard.
High-throughput backtesting decisions
The online request creates one durable optimization job and returns. Workers reserve bounded candidate work, reuse one immutable data generation and compiled strategy semantics, then persist evidence. The optimization workspace exposes this lifecycle without giving research workers brokerage authority.
Sharding live trading
Live execution needs one answer to a basic question: which process is allowed to evaluate this user's portfolios? A stable routing function turns the same user ID into the same number on every service. Modulo then reduces that number to one valid shard. NexusTrade uses FNV-1a-64 as the stable routing function.
The checked-in live-trading shard count is
3. Three shards have ordinals 0,
1, and 2, so any user must map to
exactly one of those three values.
Changing the divisor changes ownership. The
example user maps to shard 2 when N = 3. The same
user maps to shard 1 when N = 4. Node and every Rust
machine must change the shard count together during a
market-closed operation, and every ordinal must have exactly one
running owner. A count mismatch can send a command to a process
that rejects ownership for the user's portfolios.
Single-owner shard continuity
The current design favors duplicate prevention over automatic reassignment. Restarting the existing Fly machine preserves its configured ordinal. A separate coverage worker checks every 15 minutes whether each shard that owns active portfolios has written a recent portfolio snapshot. This detects a missing or wedged shard; it is separate from the market-data freshness gate used by strategy evaluation. Recovery verifies the full ordinal set, restarts or reassigns exactly one machine to the missing ordinal, hydrates its portfolios from durable state, and then resumes evaluation.
This is an active-passive continuity model for each shard ordinal. It deliberately preserves one trading owner while recovery verifies the ordinal, restores durable state, and resumes evaluation. A requirement for automatic market-hours reassignment triggers the next design: a versioned consistent-hash ring with virtual nodes, routing epochs, drain-and-hydrate handoff, and fenced writes from the former owner. Consistent hashing limits remapping; the epoch and handoff protocol preserve the single-owner invariant during migration.
Do not combine these freshness controls. The coverage monitor detects a missing live-trading owner. The evaluation path separately rejects stale quotes and incomplete market state before creating order intent. Faster shard recovery requires quicker detection plus a fenced handoff protocol.
Market state is an execution dependency
A senior design classifies each timestamp by exchange session and instrument state before the evaluation path admits an order.
Use the venue calendar and timezone. Suppress scheduled evaluation outside the strategy's allowed session.
Block new submissions for the affected instrument while order status and fills continue to reconcile.
Version adjusted data, positions, and identifiers together so cached history preserves consistent pre-event and post-event units.
Record the event effective time and cash or position effect, then rebuild any derived portfolio snapshot that depends on it.
Other senior-level decisions
- Recoverable job ownership: reclaim stale research and agent jobs after a worker dies.
- Conditional transitions: one worker wins the right to submit an approved order.
- Stable external identity: reuse a client order ID after ambiguous timeouts and reconcile before replay.
- Backpressure: limit work per provider, brokerage, account, and machine. Internal capacity leaves external quotas unchanged.
- Degraded modes: keep read-only research available while live execution fails closed on stale prices or uncertain order state.
- Cost-aware capacity: keep the web tier warm, suspend reserve web machines, and let bursty research compute scale to zero.
Each brokerage adapter has a named product boundary. NexusTrade presents separate connection and capability surfaces for Alpaca, TradeStation, Public, and Tradier. Each adapter normalizes venue-specific authentication, order states, rate limits, and reconciliation behind the same intent contract.
Senior interview signal: discuss how ownership changes during a rebalance, how an ambiguous broker timeout is reconciled, how stale market data blocks execution, and which workloads scale on CPU versus external rate limits.
Production extensions
Open the detail the interview calls for
Most interviews will stop at the core design above. Open these extensions when the interviewer asks about secrets, recovery, retention, storage, latency, or strategy configuration.
Senior interviewer probes Secrets, recovery, retention, and storage choices The questions that usually arrive after the main diagram
A senior answer should state the current boundary, then name the condition that would force a different design. These probes do not require another 20 boxes on the main diagram.
Brokerage secrets and OAuth rotation
Brokerage access and refresh tokens are stored encrypted and decrypted inside the adapter path. The model and research workers receive no brokerage credentials. A rotation design must support overlapping key versions, refresh-token updates, revocation, and an auditable reconnect path without exposing plaintext tokens to general workers.
Backup, restore, and recovery targets
The current backup job writes nightly compressed MongoDB dumps to Tigris, retains 30 days, and excludes regenerable market histories. Backup frequency is an implementation fact. An RPO or RTO requires a restore drill that measures data loss, restore time, index rebuilds, and reconciliation with brokers.
Region loss and live ownership
Stateless web and elastic research roles can be recreated from durable state. Live execution has the harder rule: one ordinal owns each user. A disaster-recovery design must fence the former owner, advance a routing epoch, hydrate the new owner, and reconcile brokerage state before evaluation resumes.
Audit identity and retention
Persist the observation time, market-data generation, strategy revision, decision, approval, stable client order ID, broker response, and reconciliation result. Retention is set per record class and jurisdiction. Do not turn an architectural event log into an unsupported compliance claim.
Choose storage by access pattern
Start with the workload: transactional updates, historical scans, or interactive analytics. The products below show how NexusTrade fills each role.
| Workload | Current choice | Why it fits | When another store wins |
|---|---|---|---|
| Historical simulation | Versioned Parquet in Tigris | Immutable columnar partitions are cheap to publish, cache, replay, and share across elastic Rust workers. | ClickHouse becomes attractive when continuously ingested events need low-latency aggregates across many concurrent users. |
| Interactive research SQL | MotherDuck analytical mirror | DuckDB semantics fit columnar research and ad hoc screening without placing those scans on MongoDB. | A dedicated time-series service wins when streaming windows, continuous materialization, and subsecond operational queries dominate the workload. |
| Online product state | MongoDB plus Redis | MongoDB owns durable product and workflow state. Redis owns derived cache entries and low-latency delivery. | Keep this state outside the historical lake because user mutations, job reservations, and approvals need current ownership. |
Strong interview answer: name the workload first, choose the store second, and finish with the threshold that would make you reconsider it.
Operations Monitoring, alerts, and financial-boundary latency SystemAlert flow, recovery signals, and trace points
A green HTTP health check proves that one route answers. Separate signals verify portfolio evaluation, data freshness, and order reconciliation across the user's financial workflow.
Producers persist a severity, source, stable alert key, subject, and diagnostic body. Notification happens downstream. The worker records whether the alert was sent, suppressed by key, suppressed by the global breaker, acknowledged, or failed. That audit trail becomes part of recovery.
| Signal | Alert condition | First response |
|---|---|---|
| Data freshness | Latest expected partition or quote exceeds its freshness budget. | Block affected evaluations and identify provider or publication lag. |
| Scheduler lag | A portfolio exceeds its evaluation cadence and grace period. | Inspect shard ownership, loop heartbeat, and durable command state. |
| Ownership lease age | A research, agent, or order worker is nearing the end of its reserved ownership window. | Recover the owner or quarantine the job before replay. |
| Order uncertainty | A submitted client order ID lacks a resolved broker state. | Freeze duplicate submission and reconcile with the venue. |
| Broker health | Latency, rejection rate, authentication errors, or rate limiting exceeds the adapter budget. | Open the circuit for new submissions while reconciliation continues. |
| User risk | Exposure, concentration, drawdown, or buying-power checks cross product policy. | Require review, reduce authority, or stop the affected portfolio. |
Measure latency at the financial boundary
Report evaluation, approval wait, adapter time, and total latency separately. Split paper and live execution by brokerage and order type. Manual approval time is user wait time, so combining it with machine latency produces a misleading percentile.
Logs should carry the same identity chain across decision, portfolio, strategy, dataset generation, order, client order ID, brokerage account, and fill. Metrics show that a class of work is unhealthy. Traces and durable records explain one affected run.
Strategy interface A portable DSL across research and live execution Natural language, SDK, MCP, and REST entry points
A strategy should describe intent in a portable, typed model. The runtime decides whether that model is evaluated against historical data, a paper portfolio, or a live portfolio. This keeps the strategy authoring surface separate from execution authority.
Strategy {
condition: And([
RSI("SPY", 14 days) < 35,
Price("SPY") > SMA("SPY", 200 days)
])
action: OpenOption({
underlying: "SPY",
expiration: 30 to 45 days,
targetDelta: 0.30,
allocation: 5 percent of buying power
})
}The example is educational and excludes investment advice. The architectural point is that indicators, comparisons, composite conditions, actions, contract selection, and allocation live in a structured model. Arbitrary callback code stays outside the strategy contract.
Natural language
An agent or assisted builder translates intent into the typed model, validates it, and returns errors as structured feedback.
SDKs
The published Python SDK and TypeScript SDK create the same portfolios, conditions, actions, backtests, and data requests from code. Their public Python and TypeScript mirrors contain only the client libraries. The platform implementation remains proprietary.
MCP
External assistants call the permissioned MCP tool catalog and receive typed results. Raw database and brokerage access stay inside the product boundary.
REST API
Applications integrate through the versioned API contract, stable identifiers, validation errors, and asynchronous job status.
One internal model prevents the browser builder, SDK, MCP server, agent, backtester, and live runtime from inventing different strategy semantics.
Research validity Walk-forward analysis and independent holdouts Point-in-time data, validation windows, and paper trading
An optimizer can search thresholds, lookbacks, allocations, and contract parameters faster than a person. That speed also makes overfitting easier. The research contract should separate discovery, validation, and final holdout periods before paper trading.
- Use only data knowable at each simulated timestamp.
- Apply splits, dividends, delistings, option-contract availability, fees, and slippage consistently.
- Rank candidates on multiple objectives such as return, drawdown, turnover, and stability.
- Record the dataset generation, strategy revision, parameter set, and engine version with every result.
- Treat a failed out-of-sample result as a stop signal. Preserve the holdout for an independent decision.
NexusTrade exposes these stages separately: the corpus identifies the available evidence, the Backtest Explorer runs a fixed strategy, and Optimization searches parameter candidates.
Conclusion
The platform makes the agent useful
The progression is now complete: prove one correct loop, separate the competing workloads, then add ownership, recovery, and elastic capacity where the measurements require them.
Those boundaries give an autonomous agent somewhere safe to operate. The model proposes the next research step. The platform supplies the tools, data, durable execution, approval, and brokerage controls. You can follow that split in the NexusTrade agent.
The companion article traces the agent runtime: Router V5, Agent V6, durable ReAct steps, parallel tools, subagents, Run Compute, crash recovery, and the boundary between creating order intent and receiving authority to submit it.
The companion article, How to Design an AI Trading Bot, follows that runtime from the router through execution and recovery.
Try your own task with NexusTrade ↗
No comments yet.