Nemotron scored lower across four jobs.

TL;DR

Nemotron 3.5 Super VL scored below the saved results for our selected models across four full NexusTrade benchmark suites on October 15, 2026. The 298 attempts covered execution, planning, strategy creation and backtest configuration. Accounted generation cost was 1.3 to 9.1 times as high.

The baselines were recorded between September 22 and October 8. Prompts, context and strategy-repair behavior changed between runs, so these gaps describe a historical comparison.

Watch all four jobs move through the experiment

Each job asks the model to produce something different. The executor chooses the next tool action from a frozen conversation. The planner turns a request into a sequence of work. Strategy creation produces executable portfolio rules. Backtest configuration specifies the dates, portfolio and other inputs for a test.

The full run preserved every failed attempt. Four executor responses failed JSON parsing. One planner call hit the ten-minute deadline. Another strategy response contained no usable code.

The score gaps appeared in all four jobs

Muse Spark’s saved executor result was 89.8 out of 100, compared with Nemotron’s 81.6. At the benchmark’s 75-point correctness threshold, Muse passed 185 of 207 attempts and Nemotron passed 154. Nemotron’s mean grade was 81.6, while its attempt pass rate was 74.4%.

The saved GPT 5.6 Luna planner averaged 89.0, versus Nemotron’s 80.0. Strategy creation averaged 96.6 for Luna and 90.3 for Nemotron. Backtest configuration used deterministic checks: GPT 6 Luna passed 22 of 22 cases, while Nemotron passed 19.

A one-time plan left an ongoing request unfinished

One request called for recurring monitoring and future agent work. Nemotron’s plan fetched the paper portfolio, requested prices, evaluated conditions, staged a conditional order and ended with a final answer. It provided no recurring mechanism to launch the later work.

The missing return path matters after the first check. A plan that completes one evaluation cannot observe a condition that changes tomorrow. The normal planner judge scored this response 20 out of 100.

Three configurations failed: one answer emitted a configuration when it should have refused a request without a portfolio. Two others corrupted the supplied portfolio identifiers.

The lower grades came with higher generation cost

The executor suite cost about $0.96 for Muse and $1.44 for Nemotron. The Nemotron total includes $0.259 recovered from the provider records for its four malformed responses. Leaving failed outputs out would understate the cost of the run.

Planning rose from $0.110 to $0.141 across 29 cases. Strategy generation rose from $0.127 to $0.249 across 40. Configuration rose from $0.0054 to $0.0488 across 22, roughly nine times the incumbent’s recorded amount.

These totals combine recorded provider-response amounts and catalog calculations. They are accounted benchmark amounts rather than verified invoices. Input lengths, caching and routing changed between the historical and new runs.

Planning took more than twice as long

Executor latency increased modestly: the median was 14.1 seconds for Muse and 15.7 for Nemotron. Planning increased from 15.2 to 33.4 seconds. Configuration increased from 3.2 to 9.0 seconds.

For strategy creation, the available generation-only records put Nemotron at an 8.7-second median across all 40 calls. Historical timing records could be uniquely matched to 38 of the incumbent’s 40 generation receipts; that subset’s median was 8.5 seconds. The partial comparison suggests similar generation speed.

The portfolio harness also reports a roughly 38-second median for both models, but that clock includes grading. Separating those two calls prevents a slow grader from being mistaken for a slow strategy model.

The grader also needed an audit

One strategy request specified software or energy stocks. Nemotron used the supported INCLUDE_ONLY industry filter shown below. The language-model grader gave the draft full credit, but a deterministic checker lowered the result to 0.4 because it searched for an older IsIndustry representation.

That 0.6 deduction on one of 40 cases subtracts 1.5 points from the suite’s mean. A second industry-filter deduction is also questionable and affected both Nemotron and the incumbent. The reported 90.3 score preserves the normal grades unchanged.

The portfolio evaluator also asks for a detailed audit while its strict response format allows only a score and explanation. These findings make the strategy score less conclusive about exact rule behavior. They are a reason to inspect the generated rules alongside the grade.

{
  "stockEligibility": {
    "industryFilter": {
      "mode": "INCLUDE_ONLY",
      "match": "ANY",
      "industries": [
        "software",
        "energy"
      ]
    }
  }
}

How the full evaluation was scored

Inspect the conditions before testing the returns

A useful next step is to make the assistant expose a strategy’s entry and exit conditions. Inspect one condition that should pass and one that should fail. This makes an incorrect AND/OR relationship or a missing exit easier to spot than a polished explanation.

Open Aurora and review the following request before sending it. The expected output is a plain-English rule check with two worked observations.

Inspect the recorded comparison

The downloadable table contains the eight model/task rows used in these figures, including dates, sample counts, grade units, generation cost and median latency. It preserves the partial strategy-timing sample rather than filling the two missing incumbent matches.

Continue exploring