Morning triage
The last post was about one pull request: a run fails and a comment explains why. Most teams have a second question that no pull request owns. The nightly run on main went red. Four tests failed, or forty. Which of them are the same problem? Did something break, or is it the flaky one again? Whose commit was it?
Somebody answers that every morning, usually by opening CI and reading logs until the coffee is cold. @razohq/triage is an open source engine that does that reading for you and leaves a report: last night's failures grouped by cause, what each group is, how sure it is, and which commit probably did it.
It calls no model. Every part of it is a rule over data you can check.
One morning
This is a real morning from razohq/razo-demo, razo's public demo project. On 7 October a pull request hid the Place order button and changed the SAVE10 discount; the next day it was reverted. The triage that morning saw the red runs and the green revert in the same window:
triage 2026-10-08T19:00:00.000Z: 4 tests, 4 failures, 2 clusters
regression · high · a valid coupon applies a discount
stale-test · medium · placing the order confirms itAnd in the report, for the second group:
## stale-test · medium
`timeouterror: locator.click: timeout 5000ms exceeded.`
failed from 09f567e, green again since 012a04b
Suspects:
- aabb49e Checkout: hide Place order until pricing lands and apply
the new SAVE10 amount — button "Place order"
Evidence:
- [sha-range] c799daa..09f567e (2 commit(s))
- [commit] aabb49e … touches button "Place order"
- [history] button "Place order" stopped responding at 09f567e;
aabb49e (Checkout: hide Place order …) changed itTwo failures, two different causes, the commit that caused both, and the fact that it is already fixed. Nobody had to open CI.
Grouping
Forty timeouts on one button are usually one problem. The engine groups failed attempts by a signature of their error message: the first line, with the machine-generated noise collapsed, like timestamps, UUIDs, long ids and ports. Everything a person wrote or asserted stays, because two failures that differ in a control name or an expected value are two different problems, and merging them would give the report a group nobody can act on.
Five categories, checked in order
Each group gets one category, from the first rule that applies:
- Environment. Several different files fail within minutes of each other with network errors, navigation timeouts or 5xx answers. The CI or a backend was down, not the app.
- Flaky. The test passed and failed on the same commit, including passing on a retry. Or its recent history alternates without any commit in between.
- Stale test. The test could not find or see an element, and a commit in the range changed a control the test touched.
- Regression. The test passed in its last runs before a commit and failed on every attempt since.
- Unknown. None of the above, or not enough history to say.
Each comes with a confidence, and the confidence is earned, not estimated. A regression is high only when a suspect commit actually touched one of the test's controls. A stale test is never high: telling an outdated test from a real regression needs a person, so the engine says medium and shows its evidence. The rule from the first two posts holds here too: better "I don't know" than a confident mistake.
Suspects come from the controls
Because every test is narrated by razo, the engine knows which controls each test drove: button "Place order", field "Coupon". It takes the commits between the last green run and the first red one, and looks for those controls' names and testids in the lines each commit added or removed.
The matching is strict on purpose. A name has to appear as a whole word, so order does not match reorder. Short and generic words like button or item are ignored, since they would make every commit a suspect. A control whose step actually failed weighs more than one the test only passed through. A file the engine cannot read, like a binary or a truncated diff, is reported as "could not evaluate" instead of being silently skipped.
It remembers
Run every morning, the engine keeps a small state file: the groups it has seen, when each started failing and what people decided about them. In CI, that file travels between mornings as a workflow artifact.
So a failure that was there yesterday says recurring · open for 2 days instead of looking new. One that stays green for three runs on main becomes resolved, and if it fails again it comes back as reopened, at the top of the report. A group someone marked as flaky or ignored moves to a short "Known" section instead of being triaged again every day. A group marked flaky three times proposes quarantine.
It only reads
The engine reads CI artifacts and commits. It writes its report and its own state file, and nothing else. It does not open issues, post to chat or comment on pull requests.
When a group deserves an issue, the report includes a link to GitHub's new-issue form, already filled in, for a person to submit. With a tracker configured, an open issue carrying the group's id marks the group as ticketed, and the report stops proposing a new one.
On razo.ar, the Triage tab shows the same report, day by day, and lets you take those decisions with a click: acknowledge, ignore, mark as flaky. The next morning the engine reads them back. Nothing is computed on razo.ar and no analysis credit is spent; the dashboard is one place the report can go, next to a Markdown file or any server that implements the same small HTTP contract.
Tested against real mornings, and corrected by them
The thresholds were not picked once and left alone. For weeks the engine ran every morning against razo-demo, which breaks its own checkout on purpose and reverts it, and every report was compared with what actually happened. Each finding is in the design document's calibration log.
Some of them were wrong verdicts. The morning shown above originally came out as unknown · low with no suspects, because the revert had already closed the red streak and the engine only looked for a streak that was still open. It also said open for 10 days about a failure that had started the day before, because it counted from the first failure it had ever seen, two incidents earlier. Both are fixed, and those real runs are now a test fixture that replays the morning on every change.
Which green tests are about to go red
Since 0.2.0, the report has one more section. The rules only look at what failed. The flake forecast looks at the tests that passed, and flags the ones whose recent history is trending unstable: results that flip, attempts that needed a retry, durations that spread or grow. Recent runs count more than old ones.
The calibration shaped it too. A commit that breaks main and the revert that fixes it make a test flip twice without the test being flaky, so a flip across commits counts for a quarter of a flip on the same commit. Without that, the forecast flagged razo-demo's deliberately broken tests. And a test that runs in a couple of hundred milliseconds was flagged once because its timing wobbled by about 60 ms on a shared runner; changes under 250 ms are now treated as noise. Replayed over 21 real mornings, it flags nothing, which is right for a suite with no flaky test.
The forecast is advisory. It never changes a verdict and proposes no action. It is adapted from playwright-smart-reporter's flake forecast, MIT licensed.
Try it
npm install -D @razohq/triage
npx triage pull --since 14d # download the last two weeks of CI runs
npx triage run --since 24h # triage the last dayIt needs a test suite that uses razo, and a read-only GitHub token. The package README has the configuration, and the Triage docs have the workflow that runs it every morning and sends the report to razo.ar.