Should the AI click directly? We tested it.

StrangerTest’s hands work in two steps. The AI user describes what it wants to act on, in words. A separate grounding step finds that target on the rendered screen, and the click, typing or drag happens there. The extra step costs time and money, and it is one more place for an error to enter.

Current vision models can output screen coordinates themselves. So the obvious question was whether the grounding layer had become unnecessary: let the user point directly, and remove a step.

• ·Frozen: the user model’s reasoning and instructions (except its list of available actions), perception, safety rules, the judge, and the models.
• ·Changed: only the hand. In the direct arm the user gave coordinates on a 0–1000 grid of the screenshot for clicks and drags, with no grounding call.
• ·Products: four live public products — a form builder, a web analytics product, an AI research tool and a canvas drawing app.
• ·Users: one AI user per product in each arm. Each direct run reused the persona and goal of an existing grounded baseline run of the same product.
• ·Measured: whether each hand action reached its intended target on the first attempt, findings caused by the hand, cost, and time. Hand success was judged by us from each step’s record — the target the user described, the element the hand actually reached, and the recorded outcome.

Direct control removed the grounding calls, but the user’s own replies became longer — it now had to reason about where to point — averaging nearly three times the output of the grounded arm. One reply ran to the model’s output limit and ended a direct session after four actions. Net, the direct arm cost about 25% more ($0.523 against $0.418).

At this size, 85% and 86% are the same number. The experiment does not show that grounded execution is more accurate, and we do not claim it.

What it shows is narrower and more useful: removing a whole layer did not produce an end-to-end advantage. Accuracy did not improve, cost went up, and the errors changed kind. Direct pointing was worse in dense text navigation, where rows are close together. It was sometimes better in free-space canvas work, where there is no labelled element to ground to — a case the grounded hand already covers with a coordinate fallback.

• ·Grounded execution stays the default hand. The direct hand was not merged or deployed.
• ·The canvas case is noted as the place where direct pointing might earn a role later, as a fallback rather than a replacement.
• ·Separately, the canvas run surfaced an AI user claiming a finished result the screen did not show. That became its own note: When the AI said it was done — and the browser disagreed.

• ·Very small sample: four products, one user per product per arm. This is not a statistical benchmark and no significance is claimed.
• ·Different denominators (27 and 22 actions), because the users acted differently.
• ·One direct session ended early after a single model-output failure, which shortened that arm.
• ·The grounded arm used existing baseline runs of the same personas, not runs interleaved with the direct arm.
• ·Hand success was judged by us, not by an independent adjudicator. The recorded outcome we judged from includes the user’s own description of the screen; screenshots were not separately reviewed.
• ·Neither arm published any product finding, so “0 findings caused by the hand” in both arms says little.