Research
StrangerTest Research
Engineering an AI user is easy to demo. Making its conclusions trustworthy is harder.
We publish selected internal experiments that materially changed how StrangerTest works. Each note separates what we observed, how we interpret it, what limits it, and what we decided — and says so when a result did not support the approach we expected.
The subject of every note is StrangerTest. Where an experiment ran on a live third-party product, the product is described by kind rather than named, because nothing here is a finding about it.
- Exploratory engineering experiment
Should the AI click directly? We tested it.
Removing the grounding layer made clicks no more accurate and total cost about 25% higher. We kept grounded execution.
Sample 4 live public products. 1 AI user per product in each arm. 27 measured hand actions in the grounded arm, 22 in the direct arm.
- Controlled engineering observation
When the AI said it was done — and the browser disagreed
An AI user reported a finished result the screen did not show. One phrasing was caught; another escaped. Outcome claims are now checked against the browser record.
Sample 1 session containing the unsupported completion claims, and 4 control sessions, re-checked from their stored records.
- Engineering retrospective
When the tester is wrong, the product should not get blamed
A browser agent that cannot operate a page looks, from inside the loop, exactly like a user who cannot. Five cases from our own development, and what excludes them now.
Sample Incidents from our own development runs against live products. Not a measured rate.
In progress
Reliability Benchmark 1.0
We are developing a controlled, held-out benchmark for StrangerTest itself. It will measure product-defect precision, false-blame rate, true-issue recall, abstention quality, unsupported-claim rejection and repeatability.
It has not been run. There are no results yet, and this page will not show any until there are.
Will a stranger actually get it?
Paste your product’s address. Three independent first-time users try it while you watch. Free, and no account.