# Blender MCP vs Angelo: latency and correctness

Generated by `tests/python/angelo_agent/bench_mcp_vs_angelo.py` on 2026-10-15T08:01:01+0900. n = 1 trials per side, alternating.

Gate set version 2.

Grading contract: 4.

## Head-to-head (5 tasks, same prompts, default-cube start)

Wall seconds, median (p75) [min–max]; every attempted run, failed checks included.

| Task | MCP wall s | Angelo wall s | MCP pass | Angelo pass | MCP API calls | Angelo API calls | MCP tools | Angelo tools |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| read_only_question | – (–) [–––] | 3.0 (3.0) [3.0–3.0] | 0/0 | 1/1 | – | 1 | – | 0 |
| cube_creation | – (–) [–––] | 3.1 (3.1) [3.1–3.1] | 0/0 | 1/1 | – | 1 | – | 1 |
| transform | – (–) [–––] | 3.3 (3.3) [3.3–3.3] | 0/0 | 1/1 | – | 1 | – | 1 |
| bevel_and_shade | – (–) [–––] | 3.9 (3.9) [3.9–3.9] | 0/0 | 1/1 | – | 1 | – | 1 |
| material | – (–) [–––] | 1.8 (1.8) [1.8–1.8] | 0/0 | 1/1 | – | 1 | – | 1 |

| Per-trial total of the 5 tasks | MCP | Angelo |
| --- | --- | --- |
| Wall s, median (p75) | – (–) | 15.1 (15.1) |
| Model API calls, median | – (–) | 5 (5) |
| Claude Code cost estimate USD, median | – (–) | 0.090 (0.090) |
| Head-to-head pass rate | 0.0 | 1.0 |
| Trials with all 5 passed | 0/0 | 1/1 |
| Graded (a run is passed only when a check read its result) | – | passed 5, failed 0, ungraded 0, skipped 0 of 5 |
| Answers read, of them parsed leniently | – | 1, 0 |

## Where the time went (medians per task)

| Task | Side | Model API calls | Input tokens per call | Model API s | Output tokens | Wall s | First feedback s |
| --- | --- | --- | --- | --- | --- | --- | --- |
| read_only_question | mcp | – | – | – | – | – | – |
| read_only_question | angelo | 1 | 36274 | 2.0 | 122 | 3.0 | 2.2 |
| cube_creation | mcp | – | – | – | – | – | – |
| cube_creation | angelo | 1 | 36691 | 1.9 | 159 | 3.1 | – |
| transform | mcp | – | – | – | – | – | – |
| transform | angelo | 1 | 37307 | 2.2 | 196 | 3.3 | – |
| bevel_and_shade | mcp | – | – | – | – | – | – |
| bevel_and_shade | angelo | 1 | 37825 | 3.8 | 396 | 3.9 | 3.1 |
| material | mcp | – | – | – | – | – | – |
| material | angelo | 1 | 38645 | 1.6 | 167 | 1.8 | – |

First feedback: MCP = first streamed text or tool call; Angelo = first reply text of its plan.

## Observations from this run

- Model API time as a share of head-to-head wall time: MCP –, Angelo 77%.
- Model API calls per trial (5 tasks): MCP [0]; Angelo [5].
- Wall s per trial (5 tasks): MCP [0.0]; Angelo [15.1].
- MCP add-on socket round trips per trial, s (includes its 0.25/1 s polling): [0.0].
- Angelo opening before the first prompt, per trial: ['none'].
- MCP cold start (process spawn to Claude Code init), ms: [].
- MCP first-message prompt-cache writes per trial (tokens): [].
- MCP permission denials 0, API retries 0, add-on script errors 0; Angelo declined tools 0, Python fallbacks 0.

## Tokens per task (median, all attempted runs)

| Task | Side | Input (incl. cache) | Cached read | Cache write | Output | API s | Cost USD |
| --- | --- | --- | --- | --- | --- | --- | --- |
| read_only_question | angelo | 36274 | 34543 | 1729 | 122 | 2.0 | 0.0230 |
| cube_creation | angelo | 36691 | 36272 | 417 | 159 | 1.9 | 0.0140 |
| transform | angelo | 37307 | 36689 | 616 | 196 | 2.2 | 0.0160 |
| bevel_and_shade | angelo | 37825 | 37305 | 518 | 396 | 3.8 | 0.0200 |
| material | angelo | 38645 | 37823 | 820 | 167 | 1.6 | 0.0170 |
| preview_accept | angelo | 39323 | 38643 | 678 | 130 | 1.5 | 0.0160 |

## Angelo-only: staged preview then accept (not in the head-to-head)

Wall s median (p75): 1.6 (1.6); passed 1/1. Blender MCP has no staged preview/accept.

## Where the steps went (tool calls summed over all trials)

- angelo / cube_creation: angelo_scene_program ×1
- angelo / transform: angelo_scene_program ×1
- angelo / bevel_and_shade: angelo_apply_recipe ×1
- angelo / material: angelo_apply_recipe ×1
- angelo / preview_accept: angelo_scene_program ×1

## Method and environment

- Same Claude account on both sides: the Angelo connection's native Claude Code profile (CLAUDE_CONFIG_DIR).
- Same pinned model, --effort low and Fast off on both sides; the model each side actually used is read back from Claude Code's own output (MCP: init/result; Angelo: its session transcript).
- Same five prompts from eval_live_tasks.build_tasks(), one continuing chat per trial, factory default-cube scene.
- MCP side: stock Blender with a private copy of Blender Lab's MCP add-on (default 0.25 s active / 1 s idle socket polling), Blender Lab's installed MCP server, headless Claude Code with its default system prompt and built-in tools; --allowedTools mcp__blender (all of that server's tools), --permission-prompts none (any other prompt is denied and reported), --strict-mcp-config, --no-session-persistence.
- Angelo side: the unchanged live-eval harness (approvals on Ask; its test-only rule approves typed bounded scene tools and declines reviewed Python, file writes and the rest).
- Wall time = prompt sent -> final result (MCP: Claude Code's result line; Angelo: the request receipt's total_ms). The first prompt of each trial includes the runtime launch on both sides: by default Angelo readies nothing before it (ANGELO_EVAL_OPENING_WAIT 0; each trial's report records its opening).
- Before every MCP prompt the driver waits 6 s after its own last add-on request, so the add-on is on its default idle polling, as after a human pause; Angelo sends its next prompt immediately (it does not poll).
- Correctness: after each turn the driver (not the model) grades the scene with the Angelo harness's own predicates (verify() + eval_live_tasks checks + the session's scene_digest/bind_checks/smooth_shaded); MCP grading runs over the add-on socket. Failed checks count as failures; their time is still reported.
- Both sides start from a minimal environment (PATH, HOME, USER, locale; DISABLE_AUTOUPDATER=1) so no inherited ANTHROPIC_*/CLAUDE_CODE_* routing or session variable reaches either runtime.
- Trials alternate MCP, Angelo, MCP, Angelo...; each trial is a fresh process pair, scene and session; nothing is retried.
- Questions (gate set version 2; a side whose trials are reused from a version-1 run is named above and was not graded this way): each question keeps its wording and ends with one sentence that asks for an answer line. The driver, never the model, computes the truth from the scene before it sends the prompt and reads the turn's answer line against it (eval_answers); a question passes only on the right answer with the scene unchanged. So that neither answer can be given without reading the scene, the driver adds one small mesh before the opening question and moves the cube by a fixed offset before each later one, and undoes both after grading. An answer read with a liberty (units among the numbers, a count before the names) is counted as parsed leniently.

```json
{
  "runtime": "claude",
  "claude_code": "2.1.293 (Claude Code)",
  "codex": null,
  "codex_account": null,
  "models_reported": {
    "mcp": [],
    "angelo": [
      "claude-opus-5-5"
    ]
  },
  "requested_model": "claude-opus-5-5",
  "effort": "low",
  "fast_mode": false,
  "mcp_init_model": [],
  "mcp_server": {
    "extension_manifest_version": "1.0.1",
    "path": "<local profile>"
  },
  "mcp_addon": null,
  "stock_blender": null,
  "angelo_binary": [
    "Blender 5.2.2 LTS",
    "build date: 2026-10-14",
    "build hash: 3092c5ccba45",
    "build branch: angelo-v1.0-release",
    "build type: Release"
  ],
  "angelo_checkout_commit": "3092c5ccba457d380943ce52221596b3f19434c7",
  "angelo_checkout_branch": "",
  "angelo_checkout_dirty_files": 0,
  "angelo_python_source": "the add-on and harness Python load from this checkout (<local path>)",
  "machine": {
    "platform": "macOS-26.3-arm64-arm-64bit-Mach-O",
    "cpu": "Apple M4 Pro",
    "cores": 14,
    "memory_bytes": "51539607552"
  }
}
```

## Threats to validity

- Safety asymmetry: the MCP side executes model-written raw Python in the live scene with no staging, verification gate, approval or single-step Undo; Angelo stages typed operations, verifies and commits one Undo step. Equal latency would not mean equal risk, and Angelo's checks cost time the MCP side does not spend.
- Prompt wording: the prompts were written for Angelo's harness (explicit names, exact numbers). Other phrasings, vaguer requests or larger scenes may favour either side; five short tasks are not a general workload.
- Small sample: n = 5 trials per side and task; medians and p75 are reported with min/max, but differences of a few seconds are within run-to-run spread. No significance test is claimed.
- Network and service variance: both sides call the same hosted model over the internet; queueing, retries and time-of-day load move wall times. Alternation spreads drift across sides but does not remove it.
- Shared machine: other agents' builds/tests ran concurrently; load averages are recorded per trial.
- Context asymmetry by design: Angelo attaches a bounded scene observation to each prompt; the MCP side must call tools to see the scene. Angelo's Claude Code runs with a restricted tool set and structured output, the MCP side with Claude Code's full default tool set and system prompt. This is each product as shipped.
- Stock Blender 5.2.0 (MCP) vs the Angelo 5.2.2 build: different binaries; Blender-side execution is milliseconds on both, so this does not drive end-to-end time, but window focus behaviour differs (stock Blender may come to the front).
- The 6 s settle and the add-on's 1 s idle polling are the MCP add-on's defaults; a user who lowers the idle interval would see somewhat faster MCP tool round trips.
- Cost figures are Claude Code's own estimates, not a bill; this is a subscription account.
- Prompt cache: the MCP side reuses one empty working directory, so after its first trial its system prompt is served from the prompt cache. Each Angelo trial has a new workspace path (Claude's working directory) unless the run used --angelo-stable-workspace (see plan.angelo); see the first-message cache writes under Observations. Without it this mainly lowers the MCP cost estimate.
- Round-count variance: Angelo's number of model rounds depends on the model's own verify choice per plan, and Claude Code defers MCP tools behind ToolSearch on first use; both are counted as they happened, and n = 5 may under-sample how often each occurs.
- Wall-clock boundaries differ slightly: the MCP clock ends when the driver reads Claude Code's result line; Angelo's receipt total_ms ends inside the controller (the harness's send-to-idle time is also recorded).