A verdict on the pull request

The first post ended with a record: every step a test took, with the name of the control it touched, what it expected and what the page actually had. This one is about what reads that record when a run fails, and how it turns into one short comment on the pull request.

There are two ways to get it. razo-analyze is an open source CLI and GitHub Action that runs with your own model key. razo.ar is the hosted dashboard, which also reads the pull request's diff and description. They share the same idea: give the model the story of the test, not the stack trace.

What the model reads

A stack trace says where the test gave up. The narration says what the test was trying to do. This is a failed test as the analyzer renders it into the prompt:

## Test: export a model
File: tests/export.spec.ts — status: failed

Narrated steps:
- [passed] Type "mi-llavero" into field "Filename"
- [passed] Choose "3MF" in select "Format"
- [passed] Click button "Export"
- [failed] Assert label "Export status" has text "Exported mi-llavero.3mf (Standard)"
  expected: "Exported mi-llavero.3mf (Standard)"
  actual:   "Error: missing fields"

And this is what came back:

What failed: the final verification Assert label "Export status" has text "Exported mi-llavero.3mf (Standard)". The app answered "Error: missing fields" instead.

Root cause: missing test precondition. The narration shows the filename was typed and the format chosen, but no step ever selected a quality before Click button "Export". The app correctly rejected the export.

Fix: add quality.select('Standard') before the export, or if quality became optional, update the expected status text.

The model did not have to guess what getByTestId('status') meant or which click mattered. The answer is in what the steps say, and in a step that is not there.

The prompt carries more than steps when there is more to say. A step that healed says which locator stopped resolving and which one worked, and the model is told that is locator drift, not a bug. A step that could not heal lists the same-role elements still on the page, so the model can propose the replacement locator as code. Since razo 0.4.0, the page's own errors are listed too, with the one that came first before the failure marked: a 500 right after a click points at the app, not the test.

Six answers, and one of them is "I can't tell"

On razo.ar, a failed run gets one classification for the whole run:

The hard pair is the first two. A diff tells you what changed, never whether it was meant. A button hidden on purpose and a button hidden by a bad merge produce the same diff.

So razo only answers regression or intentional change when someone has stated the intent: the pull request description, a comment in the code, a changelog entry. Without that, the answer is Undetermined, together with an exact description of what the diff did and what would settle it. An empty description is the normal case in real repositories, and a confident wrong cause in a one-line comment costs a reviewer more than an honest "this PR hid the Place order button; was that intended?".

Joining the test to the diff

The analysis on razo.ar reads the pull request, but not all of it. For each failing test it takes the controls the test drove, and the strings that identify each one in source: its human name and anything quoted in its selector, like a testid or a role name. Then it looks for those strings in the lines the pull request added or removed.

When one matches, the comment shows it under the test:

razo — 2 failed · 2 passed · 02b1f81

Intentional change — PR intentionally hides the Place order button and removes the Mouse row; both failing tests assert the old UI.

  • placing the order confirms it — tests/checkout.spec.ts ↳ touched button "Place order", which this PR changed in tests/checkout-page.ts
  • the cart lists both items — tests/checkout.spec.ts

Next: Update or skip these two tests until the express-checkout summary (follow-up PR) lands, per the stated plan.

That is the real comment on razohq/razo-demo#1. Two details are deliberate.

A test shows at most one match. The comment is evidence for a verdict, not an inventory of the diff.

A test with no match, like the cart lists both items, shows nothing. No match does not mean the pull request left the test alone; it means the diff never named one of its controls. Silence is the only honest way to render that.

A comment you can read in ten seconds

The comment is the classification, one sentence of verdict and one next step. The reasoning, the per-test diagnoses and the page signals live on the run's page on razo.ar, one click away. Re-running CI replaces the comment instead of adding another one.

With razo-analyze the shape is per test (what failed, the likely cause, a fix) and it goes wherever you point it: the terminal, a file, or the pull request with --pr-comment.

What leaves your machine

Both the prompt and the upload to razo.ar pass through the same masking first. Known credential shapes are replaced with [REDACTED] anywhere they appear: authorization headers, bearer tokens, JWTs, credentials in URLs, cloud and API keys. So are the values of environment variables whose names look like secrets, and what a test typed into a control named like a password, a token or a PIN. The razo-steps.json files on disk are never changed.

Size is handled the same way: never by failing, never silently. A huge suite is split into several model calls. A huge test keeps its failed steps and its first and last passed ones, and the middle becomes a visible "N steps omitted" marker.

What it costs

With your own key, one analysis is a single model call of a few thousand tokens, roughly $0.05 to $0.15 a run with the default Claude model. The open source path has no limit on the razo side, because it never talks to razo.ar at all.

The hosted analysis comes from a quota, and a single run analyzes at most five failed tests, so one suite that breaks everywhere does not spend a month of it in one night. The dashboard and the run history work without it.

No model inside the tests

The model runs after the run, on its record. Nothing it says changes whether a test passed. Healing during the run is deterministic, and so is the next piece: the one that looks at every failure from last night and groups them by cause, without calling a model. That is the next post.

To try the analyzer:

npm install -D @razohq/razo-analyzer
npx razo-analyze --dry-run   # prints the prompt, calls nothing

The CI setup guide has both recipes: your own key, or razo.ar.