Morning triage
The Triage tab answers the first question of the day: what broke overnight, and why. Each morning it shows your base branch's failures grouped by cause, a category and a confidence for each group, the commits that probably caused it, and a next step.
The triage is not computed by razo.ar. It runs in your own CI with the open-source @razohq/triage engine, which reads the razo-steps.json artifacts of your recent runs and sends its report here. Grouping, classifying and naming suspects is deterministic: no AI model is involved, and no analysis credit is spent.
Set it up
You need a project on razo.ar with its ingest token in the repository secret RAZO_INGEST_TOKEN (see Dashboard & plans), and a test workflow that uploads test-results/ as an artifact on every run, green ones included, under a name that starts with razo-test-results-:
- uses: actions/upload-artifact@v4
if: always()
with:
name: razo-test-results-${{ github.run_id }}-${{ github.run_attempt }}
path: test-results/
retention-days: 30Install the engine:
npm i -D @razohq/triageAdd .github/triage/triage.config.yaml, replacing <owner>/<repo>:
baseBranch: main
source:
plugin: razo-source
config: { dataDir: .razo }
pull:
repo: <owner>/<repo>
token: "${GITHUB_TOKEN}"
workflow: e2e.yml # the workflow that runs your tests
branch: main
code:
plugin: github
config: { repo: <owner>/<repo>, token: "${GITHUB_TOKEN}" }
notifiers:
- plugin: http
config: { url: "https://razo.ar/api/triage/reports", token: "${RAZO_INGEST_TOKEN}" }
store:
plugin: json-file
config: { path: .razo/triage-state.json }
# The decisions you take on the Triage tab, read back every morning.
decisions:
url: "https://razo.ar/api/triage/decisions"
token: "${RAZO_INGEST_TOKEN}"
# Optional: link failures to the GitHub issues that track them.
tracker:
plugin: github
config: { repo: <owner>/<repo>, token: "${GITHUB_TOKEN}" }Then add .github/workflows/triage.yml. It runs after each nightly run of the test workflow on main, and keeps the triage's memory between mornings as an artifact:
name: triage
on:
workflow_run:
workflows: [e2e] # the name of your test workflow
types: [completed]
branches: [main]
workflow_dispatch:
concurrency:
group: razo-triage
cancel-in-progress: false
permissions:
actions: read # list runs, download their artifacts and the previous state
contents: read # compare commits and read their diffs
issues: read # only with the tracker: find the issues that track each failure
jobs:
triage:
# Only after the scheduled run, not after every push or pull request.
if: github.event_name == 'workflow_dispatch' || github.event.workflow_run.event == 'schedule'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with: { node-version: 22, cache: npm }
- run: npm ci
env: { PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD: '1' }
- run: npx triage pull --config .github/triage/triage.config.yaml --since 14d
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
RAZO_INGEST_TOKEN: ${{ secrets.RAZO_INGEST_TOKEN }}
- run: npx triage run --config .github/triage/triage.config.yaml --since 24h --lookback 14d
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
RAZO_INGEST_TOKEN: ${{ secrets.RAZO_INGEST_TOKEN }}
- uses: actions/upload-artifact@v4
with:
name: razo-triage-state
path: .razo/triage-state.json
retention-days: 90Run it once by hand (workflow_dispatch) to check the setup; the morning appears in the Triage tab. Until the first one arrives, the tab says so. The engine's README covers every config option, the token permissions in detail, and running it locally.
Reading a morning
The top of the tab lists the last 14 mornings, with the number of failure groups next to each date, and the selected morning's window: how many tests ran, how many failures, how many groups, and how many came back after being fixed.
Below it, one card per group of failures that share a cause (the same error, normalized), most urgent category first:
| Category | Meaning |
|---|---|
regression | The test passed in its last runs and fails on every attempt since one commit. |
stale-test | An element the test uses is missing or hidden, and a recent commit changed it: the test may need updating, not the app. |
environment | Many test files failed with network, timeout or 5xx errors within minutes of each other. |
flaky | The test passed and failed on the same commit, or passed on retry. |
unknown | Not enough history to tell yet. |
The confidence pill says how firmly the rules decided: high, medium or low. A stale-test is never high: telling an outdated test from a real regression takes a person. Each card also shows:
- Novelty and age:
new,recurring, orreopened(fixed before, failing again, with a red border), how many days the current failure streak has lasted, and how many failures the group has in total. - The error signature, a one-line summary and a Next step. A failure that is already green again names the commit that fixed it and the one that broke it.
- Tests in the group, the Range of commits between the last green run and the first red one, and the Suspects: commits in that range whose changes touch the controls the test used.
- Evidence, folded: the facts behind the verdict (run history, retries, the commit range, the suspects).
At risk
At the bottom, At risk lists tests that passed this morning on their first attempt but whose recent history on the base branch looks unstable. Each one shows a risk from 0 to 100% and why:
- Pass/fail flips: it alternates between passing and failing. A flip that came with a new commit counts a quarter of one on the same commit, so a commit that broke the test and the revert that fixed it do not make it look flaky.
- Retries: it needed more than one attempt in recent runs, even if it ended green.
- Duration: its run time became erratic or keeps rising. Changes under 250 ms are ignored as CI noise.
It is advice, not a verdict: nothing is proposed and nothing about the failure groups changes. It needs @razohq/triage 0.2.0 or later and at least 5 runs of history; the thresholds are under rules.forecast in the config, and maxTests: 0 turns it off.
Decisions
Each card has buttons to record what you decided. They change nothing in your repository or CI: the next morning's triage pull reads them and the engine applies them.
- Acknowledge: you have seen it. The group stays in the list.
- Ignore: a failure you don't plan to act on. It leaves the list.
- Mark flaky: the engine treats the group as flaky. It leaves the list too. After a few marks the morning proposes Quarantine.
Ignored and flaky groups move to the Known section at the bottom, with the date of the decision. They stay there even if they go green and fail again, marked reopened, so an ignored failure does not come back by itself. To undo, use Stop ignoring or Not flaky: the group returns to the list the next time it fails. An acknowledge recorded before a group reopened no longer applies, and the card says so.
Issues
With the tracker section in the config, a regression or stale-test card offers Create issue in GitHub. It opens GitHub's new-issue form prefilled with the title, tests, range, suspects and evidence. You review it and submit it yourself: neither razo.ar nor the engine ever writes to GitHub.
The issue body carries a line razo-triage: <id> that identifies the group. Keep it: the next morning finds the issue through it, links the card to the issue, and stops proposing a new one. To link an issue that already exists, add that line to its body. A group that is already green again still offers the button, for when the fix was a temporary revert and you want to keep a record.
Next steps
- Dashboard & plans: projects, ingest tokens and the per-run analysis.
- CI setup: the workflow step that uploads runs to the dashboard.