When the tester is wrong, the product should not get blamed

An AI user that misses a button, types into the wrong element or is turned away by bot protection experiences exactly what a confused person would: it tried, and it did not work. Its account of the visit will say so, sincerely. Early versions of StrangerTest could turn those experiences into findings about the product.

The products below are not named. In each case the product was fine; the subject is our tester.

We went back through development runs between 2026-09-22 and 2026-09-26 for cases where a finding, or a user’s quote, was later traced to our test rather than the product. Each was checked by opening the live page or reading the browser record. Case 5 comes from an earlier version of the engine and is told from our internal validation notes of the time; its figures are not published here because we could not re-derive them from the raw records for this note.

1. A working button, reported broken by three users

The first live scan reported, as critical, that a product’s free-trial links did not work — observed by more than one user. We checked the live page: every link worked. Our element references had gone stale on a single-page site, so the click landed on a navigation link higher up the page. Each user arrived somewhere unexpected and concluded the button was broken.

The lesson was about independence as much as about clicking. Several users meeting the same defect of ours are one defect observed several times, not several witnesses — and our corroboration logic had promoted it to high confidence.

2. Text that would not enter

A user tried to type into a text area headed by a question. The grounding step placed the target on the heading label, which carried those words, rather than on the text area, which had only a placeholder. Several actions went on refused attempts to type into a label, and the user’s takeaway said it could not get text into the first field — true for them, caused by us.

3. A redirect that was about the test, not the product

For a person, a product’s main action opened a research step. For our automated browser, the server sent that step straight back to the homepage. Nothing on screen explained it, and an earlier run reported the product’s main action as dead.

4. A timeout blamed on the page

A typing step ran out of time inside our own grounding step, and the user was told the page had stopped responding. A link click returned before acting. Neither was marked as ours, so either could have supported a finding.

5. Corroboration that was not independent

An earlier design sent a second, blind user to re-check single observations. When we checked its headline findings by hand against the live pages, the false ones were observations the re-check had “confirmed”; issues that recurred naturally across users held up. Two agents getting stuck on the same page for unrelated reasons had looked like agreement. The re-check was removed.

Each case produced a specific fix — stale references cleared and element identity verified before acting; typing into the field a label belongs to, or not typing at all; the manufactured corroboration removed. Together they also produced a rule that now runs through the judge:

• ·Our hand’s uncertainty — a target not found, a click or drag that failed, a step that timed out in our own machinery — is recorded as ours.
• ·Environment limits — bot protection, a server sending the automated browser back where it started, and the attempts straight after — are limits of the test.
• ·Our safety refusals are ours. A refused action is explained to the user so it records the boundary rather than reporting a broken product.
• ·Steps the recorded screens contradict, and addresses the user guessed, are not evidence about the product.
• ·None of these can support a finding. They are stripped from every finding’s evidence; a finding left with none is set aside, and all of them are listed in the report under “What we didn’t count”.
• ·A user’s sentence that rests on one of these steps is kept in the record as their account of what our test caused, and never shown to the founder as a quote about the product.

• ·These are incidents, not a rate. We found them because we looked; we cannot claim there are no others.
• ·Case 5 was adjudicated by us, on an earlier version of the engine, and its figures are withheld because we could not re-derive them from the raw records for this note.
• ·Exclusion can err the other way: a real product problem met in the same step as our own uncertainty will not be reported from that step. We accept that cost.