TL;DR
Muse Spark 1.2 scored 86.1 on the August 2026 next-action evaluation, with 99.5% valid structured output. Historical billing was $44.23 per 1,000 decisions and production median latency was 7.2 seconds. The replay used 69 frozen decisions with three samples each, grading proposed actions without executing their tools. These measurements describe this model version, not the separate Muse personal-agent product; valid structure alone does not establish that an action is correct.
- Exact tested ID
- meta/muse-spark-1.2
- Published studies
- 1
- Provider
- meta
Next-action execution model bakeoff: compare the tested outputs
Published August 8, 2026. Muse scored 86.1 with 99.5% valid output and 7.2s median latency; the listed Luna Pro arm scored 91.7 with 100.0% validity and 9.9s latency.
Show the table · 2 rows, 5 columns
Scroll sideways to compare all columns. Select a heading to sort.
| meta/muse-spark-1.2 | 86.1 | 99.5% | $44.23 | 7.2s |
| openai/gpt-5.6-luna-pro | 91.7 | 100.0% | $25.05 | 9.9s |
No matching rows. Clear the filter to see all records.
23-model campaign; 69 frozen decisions × 3 samples; 12 published model rows. Published 2026-08-08. Values retain their original units and grading scale.
meta/muse-spark-1.2 was tested on 2026-08-08. 23-model campaign; 69 frozen decisions × 3 samples; 12 published model rows. Its row and the other published arm use the same task and measurement units.
What this version helps you choose
Against the Luna Pro arm listed above, Muse Spark 1.2 had a lower judge score and structured-output validity, higher historical billing, and shorter median production response time. These are separate measurements from the same next-action replay.
Choose the next allowed agent action: follow the published task
A well-formed action can still be the wrong decision. Keep output validity and judged decision quality separate.
Published task summary
The existing plan, conversation and results returned by tools.
Choose one next allowed action, or finish with an answer when the evidence is sufficient.
Read what came back
A structured action. The replay records the proposed action without executing its tools.
Replay 69 saved decisions three times each. Check the required output structure separately from the judge score.
Which model was tested
Muse Spark is the underlying model tested here. Muse is a separate personal-agent product; these measurements describe the tested model version and task.
Next-action execution model bakeoff
For the next allowed action, Muse Spark 1.2 recorded a judge mean of 86.1 and produced valid structured output 99.5% of the time. Historical billing was $44.23 per 1,000 decisions; production median response time was 7.2s. Decision quality, readable output and operating cost answer different questions.
Show the table · 4 rows, 2 columns
| Published metric | Value |
|---|---|
| Mean score | 86.1 |
| Schema valid | 99.5% |
| $ / 1k decisions | $44.23 |
| Prod p50 | 7.2s |
No matching rows. Clear the filter to see all records.
Next-action execution model bakeoff: read these measurements
Read the complete rubric, sample and billing notes
23-model campaign; 69 frozen decisions × 3 samples; 12 published model rows
Choose one next action in an unfinished trading-agent workflow. This replay measures individual decisions, not a fresh complete trading run.
Scoring
Mean action score comes from the judge. Schema validity is the share of actions matching the required output structure; a valid structure can still contain the wrong action. Switching judges reduced the Luna-versus-DeepSeek score gap from 13.9 to 1.5 points.
Cost units
Cost uses historical provider billing per 1,000 decisions. Reused prefixes earned cache discounts; the cached-token split was not retained.
Time units
Prod p50 is median production response latency, separate from replay quality and complete agent wall time.
No comments yet.