When the AI said it was done — and the browser disagreed

An AI user was working in a canvas-style drawing product, with the goal of sketching a simple multi-step flow. It stopped, saying it had what it came for: the completed flow was on the canvas, with arrows between the steps. Its closing takeaway said the same.

The final browser evidence showed one unfinished shape and an empty text box with an active text cursor. There were no arrows and no second step.

The research subject here is StrangerTest, not the drawing product. Nothing in this session suggests the product was at fault; the problem was that our AI user described an outcome that had not happened.

StrangerTest already had a consistency guard, which corrects a user that claims to have typed, clicked or submitted something the visit never did. It caught one phrasing of the claim. A differently worded outcome claim got past it.

Downstream, the claim travelled further than it should have. The session summary handed to the judge said the user got what they came for, and the takeaway was positioned to be quoted to the founder under “In their words”.

A false success is as misleading as a false failure. A founder told that a first-time user completed setup believes the setup works. And a system whose narration can be wrong in one direction can be wrong in the other.

The escaped paraphrase also showed that a list of wordings cannot fix this. Whatever list we wrote, the model could say it differently.

• ·User narration is explicitly subjective. A user who stops because they are satisfied is recorded as choosing to stop, saying they had what they came for — their account, not ours.
• ·Outcome claims are checked against evidence. At the end of each visit, the user’s takeaway and the last thing they said they saw are split into sentences, and every claim to have made, submitted or completed something is checked against the recorded actions and the final screen.
• ·Founder-facing quotes are filtered. A claim the record does not support never becomes a quote, is marked in the session record as not established, and cannot support a finding. If the check cannot run, nothing is quoted.
• ·Reactions, decisions and what the user read are always kept. They are the user’s own experience and need no corroboration.
• ·When the narration and the browser record disagree, the record wins.

The new check was run on stored records rather than new visits: the session that contained the unsupported claims, and four control sessions from other visits, with the model the check uses in production. The visits were not re-run; the check read the same recorded actions and final screens the judge sees.

Re-run on the stored sessions: both unsupported completion claims from the canvas session were dropped, and every sentence of four control sessions was kept.

• ·One session with the failure and four controls. This shows the check working on the case that prompted it; it does not measure how often it errs.
• ·The check itself reads screens with a model, so it can be wrong. When it cannot run, the design fails closed: no quote.
• ·It checks claims about outcomes, not every sentence a user writes.
• ·Checked on stored sessions, not a new cohort of live runs.