# StrangerTest Reliability Benchmark 1.0 — pre-registered protocol

Status: **pre-registered**. Committed before any held-out case was run. The commit that
adds this file and `benchmark_v1_protocol_hash.txt` is the pre-registration timestamp.
Nothing in this file, `scoring` (api/benchmark/score.py), `cases.json` or
`packages.json` changes after the held-out run begins.

## 1. Research question

When StrangerTest tells a founder that their product caused a problem, how often does the
evidence actually support blaming the product?

This benchmark measures **attribution and evidence integrity** on controlled cases whose
ground truth is known independently of any model. It does not measure, and must not be
presented as measuring, how real people behave, product quality in general, conversion,
or agreement with human preferences.

## 2. What is tested

The production engine, unchanged:

| | |
|---|---|
| Engine code | `api/app` at commit `e849140c1addf7130cc4a95fd679a6561e00505e` (last engine change; identical to what production runs) |
| User model | Natural Agent (`AGENT_STYLE=natural`) |
| Perception | V2 visual perception (`PERCEPTION_ENGINE=v2`) |
| Hand | grounded (the only hand on this commit) |
| Safety, screen checks, ending checks, reviewer, evidence filters | production code paths, production settings |
| Follow-up user | enabled, as in production (`ENABLE_BLIND_VERIFICATION=true`) |
| Models | master `openai:gpt-5-mini` (effort medium); browser agent `openai:gpt-5.6-terra` (medium); aggregation `openai:gpt-5.6-terra` (high); grounding `openai:gpt-5-mini` (low) |
| Budgets per user | 10 actions, 130 seconds (production Free Scan values); other limits at production defaults |
| Users per case | 1 (end-to-end) or as scripted (packages). The planner is not asked; the case's persona is saved before the run |
| Fixture | fictional sites at `https://fixture.strangertest.app/v/<slug>`, commit given in §9 |
| Runner / scorer | `api/benchmark/run.py`, `api/benchmark/score.py` (not engine code) |

The engine receives only a case's start URL and persona. Ground truth, target terms,
classes and descriptions are never passed to it (`tests/test_benchmark_v1.py`
`test_engine_sees_only_url_and_personas`), never appear in fixture pages or scripts
(`test_fixture_route_is_live_noindexed_and_label_free`), and route slugs are random.

## 3. Case sets

**Held-out** (published): 40 end-to-end browser cases + 20 evidence packages.
**Calibration** (never published as results): 12 end-to-end cases + 8 packages, used only
to validate fixtures, runner, scoring, provenance capture and report generation.

End-to-end held-out cases, 10 per class:

| Class | Ground truth | What it contains |
|---|---|---|
| product_failure | PRODUCT | dead primary button; valid input rejected; link routes to the wrong page; state lost; confirmation never appears; primary action covered and inert; product error blocks the goal; retry does nothing; pagination repeats; promised information absent |
| tester_trap | NOT_PRODUCT | hover menu; label not associated with its field; dense navigation; delayed interface; canvas drag; subtle save confirmation; custom collapsed FAQ; control below the fold; edit behind a pencil button; custom dropdown |
| environment | NOT_PRODUCT | redirect for automated browsers; automation-only 403; external-domain boundary; upload refused by our safety; email required (refused); message field (refused); subscription (refused); external sign-in; automation-only 429; automation-only 28 s response |
| epistemic | NOT_PRODUCT (9), INCONCLUSIVE_EXPECTED (1) | content exists but collapsed; on a nearby route; behind a guessed 404; in a tab; genuinely absent (INCONCLUSIVE_EXPECTED); ambiguous no-visible-effect control; below the fold; similar labels; scope too wide for one visit; positive evidence in plain view |

Each case gives the user a natural goal (see `cases.json`). No goal names what is broken,
which control is being tested, the expected result, or that this is a test.

Evidence packages (20 held-out): a scripted user (fixed actions and narration) acts in a
real browser on a fixture site; grounding, screen checks, ending checks and the reviewer
are the real models. They test the judge separately from browser-control variance. Kinds:
unsupported submit / creation / count claims; completion claimed after our hand failed;
blame after our safety refusal; absence after a guessed URL; can't-find after reasonable
and after insufficient search; environment failure blamed on the product; genuine product
failure; subjective reaction only; valid success; stale narration; one shared cause across
two users; two independent reproductions; collapsed content never opened; explicit
product error; delayed success; hand miss then recovery; unsupported claim mixed with a
valid reaction.

## 4. Repeatability subset (fixed now)

Held-out end-to-end cases run **three** times: P01, P05, P07, T01, T04, T05, E01, E04,
X01, X03. All other held-out cases run once. Total: 40 first runs + 20 repeat runs = 60
end-to-end sessions. Primary and all secondary metrics other than repeatability use the
first run of every case only.

## 5. Scoring (implemented in api/benchmark/score.py, version 1.0)

Machine scoring only. A manual-review note may be attached to any case
(`manual_review.json`); it is displayed beside the machine score and never changes it.

**Published finding**: an entry in the final report's `findings` (after every evidence
filter), as returned by `ScanRunner.synthesise`.

**Supported finding**: the case's ground truth is PRODUCT, or INCONCLUSIVE_EXPECTED where
the case allows a finding; the finding mentions one of the case's target terms (case- and
quote-insensitive substring over headline, what happened, why it matters, confidence note,
expectation and evidence text); and its `claim_scope` is in the case's allowed scopes.
Every other published finding is **unsupported**, including any finding on a case whose
product works and any off-target finding.

**Metrics** (each reported with numerator, denominator and a 95% Wilson interval):

1. **Product-finding precision (primary)** = supported / published, over held-out E2E
   first runs.
2. **False-blame rate** = NOT_PRODUCT E2E cases with ≥1 published finding / NOT_PRODUCT
   E2E cases.
3. **True-issue recall** = PRODUCT E2E cases with ≥1 supported finding / PRODUCT E2E
   cases. Also reported over cases where the user demonstrably met the target
   (an observation whose target, reached element or outcome text contains an encounter
   term, or whose page URL matches an encounter path).
4. **Abstention quality** = among NOT_PRODUCT and INCONCLUSIVE_EXPECTED E2E cases where the
   system recorded a candidate problem (any set-aside item or published finding), the
   share with no unsupported published finding.
5. **Unsupported-claim rejection** = planted unsupported sentences (packages) that reach
   neither a founder-facing quote nor any published finding's text / all planted
   unsupported sentences. A quote is founder-facing exactly when the product would show
   it: a user's checked takeaway, under a published finding whose evidence cites that
   user (`api/app/main.py` `_visible_finding`). Reported beside it: the stricter
   diagnostic *removed from the quote pool* (the ending check dropped the sentence
   whether or not any finding could have shown it), and planted valid sentences kept.
6. **False-absence rate** = cases whose target content exists (E2E and packages) with a
   published finding scoped `product_absence` / all such cases.
7. **Repeatability** = repeat-subset cases whose attribution (product blamed or not) is
   the same in all three runs / cases with three runs. Also exact-outcome agreement
   (blamed / set aside / nothing reported) and pairwise attribution agreement.

Also reported: per-class results, a confusion matrix (truth × outcome), package-level
precision / recall / false blame, cost, latency, and every miss.

## 6. Inclusion and exclusion

Every held-out case is included in every applicable denominator. No case is removed after
results are seen. A case that could not be run at all is reported as not run and counted
as a miss (no finding) in every denominator.

## 7. Failures

- **Engine runtime or model errors** (scan ends failed/cancelled, a session ends with a
  model or internal error, a model output cannot be parsed): part of the result. The case
  is scored on whatever the engine published (usually nothing) and listed as a runtime
  failure. Not re-run.
- **Benchmark infrastructure errors**: an exception raised by the benchmark runner itself;
  the fixture start page not answering 200 to an ordinary request immediately before the
  run (checked, retried once after 30 s); a local machine or database failure; the model
  provider rejecting authentication or quota before any model call succeeds. The failed
  attempt is kept in the run directory under an `infra/` name and the case is re-run once.
  If infrastructure fails again, the run stops, the failure is documented, and only
  infrastructure is repaired. If a repair changes what a score means, this protocol is
  versioned (1.0.1) and the change is published with the results.

## 8. Order and procedure

1. Calibration set run; infrastructure fixed only. (Done before this commit.)
2. Deterministic tests pass (`tests/test_benchmark_v1.py`).
3. This protocol, the scorer and the held-out manifests are committed and hashed.
4. Held-out run, once: E2E first runs in case order (two runner processes, started 10 s
   apart), then repeat rounds 2 and 3, then the 20 packages.
5. Score; publish every number from `results.json`.
6. No engine, prompt, fixture or scoring change between steps 3 and 5. A weakness found in
   the held-out run is recorded, not fixed; fixes belong to Benchmark 1.1 with a new
   held-out set.

## 9. Frozen artefacts

- Fixture commit: see `run_meta.json` (`fixture_commit`), written at pre-registration.
- SHA-256 of this protocol, `api/benchmark/score.py`, `api/benchmark/run.py`,
  `api/benchmark/scripted.py`, `api/benchmark/engine_env.py`, `cases.json` and
  `packages.json`: `benchmark_v1_protocol_hash.txt`.

## 10. Planned analysis and reporting

Descriptive only. No significance tests between classes; intervals are shown so small
denominators read as small. The public page leads with the primary metric, shows every
secondary metric with its denominator, lists every miss with the engine's own output, and
states prominently: *These controlled cases measure attribution reliability, not
prediction of real-human behavior.*

## 11. Calibration and what it changed

Calibration (12 E2E cases, 8 packages; $0.97 of model spend) exercised every fixture
type once. Changes made as a result — infrastructure and scoring definitions only; no
engine, prompt or setting change:

- The grounding model is pinned in the benchmark configuration at production's default
  (`GROUNDING_MODEL=openai:gpt-5-mini`, effort low); it had been left implicit.
- Runner processes applying the database schema at the same moment deadlocked; the two
  held-out processes start 10 s apart.
- In packages, a follow-up user requested by the reviewer is unscripted and uses the real
  model, as the engine's own independent user does in production.
- Metric 5 now defines founder-facing quotes as the product surfaces them (above); the
  first definition counted a quote in the stored pool even when no finding could show it.
- A preflight check and an `infra/` record for infrastructure failures (§7).

Calibration results are not published as Benchmark 1.0 results.

## 12. Cost

Hard ceiling for Benchmark 1.0: $20 total model spend, calibration included.

Projected from calibration spend per session: product-failure E2E sessions averaged
$0.172, other E2E sessions $0.032, packages $0.021 (calibration maxima $0.272 and
$0.031). Held-out: 16 product-failure sessions, 44 other E2E sessions, 20 packages —
**expected $4.56; worst case $17.57** (every E2E session at the calibration maximum and
every package at twice its maximum). With calibration's $0.97, the worst case is $18.54,
inside the ceiling.