This is preview documentation for the upcoming release. Publication is pending live Jev validation. The demo uses a live navigator with mocked Jev element selection.
Quality checks and visual baselines
Available in the 0.2.0-beta.1 candidate. Check your installed version before using these options.
Bounded HTML and agent JSON reports expose separate accessibility, Web Vitals, and visual results. Each check records its policy, status, reason, measurements, and evidence. Disabled and unavailable checks never appear as passes. A failed journey skips end-of-test accessibility and vitals checks.
Configure checks explicitly:
{
"execution": { "quality": {
"a11y": { "mode": "fail", "maxSerious": 0 },
"vitals": { "mode": "warn", "lcpMaxMs": 2500, "clsMax": 0.1 }
} },
"visual": { "mode": "fail", "baselineDir": "visual-baselines", "updateBaselines": false }
}Modes are off, warn, and fail. Threshold failures in fail mode fail the test; unavailable required evidence makes it inconclusive. Warning mode preserves the functional verdict and records findings. Accessibility uses axe; vitals measure lab LCP and CLS, not INP or field Core Web Vitals compliance.
Add Visual: checkout under a goal's Success section, or Capture visual checkpoint "checkout" in a scripted test. The navigator cannot replace this assertion. Action screenshots alone do not perform visual regression.
Visual checkpoints compare exact image bytes, with separate baseline and actual images and a hash difference record. They do not offer pixel tolerances or heatmaps. Use stable data, fonts, viewport, and browser versions. Baselines are scoped by suite, test, target, browser, and checkpoint name.
Missing baselines are unavailable and are never created automatically. Set visual.updateBaselines: true explicitly to capture candidates. This returns review-required and an inconclusive verdict. Review the images, commit approved baselines, then rerun with updates disabled. Keep baseline updates outside unattended release gates.
The legacy HTML reporter remains active for legacy execution and retains its compatibility coverage. Bounded execution uses the new reporter; both retain the full JSON artifact.
Reading the artifacts
| Artifact | Purpose |
|---|---|
| HTML | Human review of verdicts, findings, metrics, baseline and actual images |
| Agent JSON | Structured quality.a11y, quality.vitals, and quality.visual results for agents and CI |
| Full JSON | Complete execution and attempt evidence, including retained compatibility fields |
| Calls NDJSON | Metered browser and model operations; quality scans and checkpoint captures appear as tool calls |
| Demo manifest | Paths to the latest demo reports and run summary |
passed means the configured check met its threshold. warning preserves the functional verdict. failed identifies a breached gate. disabled means the policy is off; not-run means it did not execute; unavailable means required evidence could not be obtained. review-required marks an explicitly captured baseline candidate. None of these last four states is a quality pass.
The Everyday Goods demo enables accessibility and vitals in warning mode. It does not implicitly add visual assertions. An action screenshot documents a moment in the run; only an explicit visual checkpoint compares against a baseline.