Phase one: Markdown goals

A test supplies intent, observable success conditions and optional input data. The runtime opens the suite's base URL, observes the page and asks a configured language model for one next action. It repeats until the agent finishes, becomes blocked, violates a bound or reaches the deadline. Every success condition is then checked through the browser. There is no generated JavaScript, persisted action script or model-owned verdict.

---
suite: Shop smoke
baseUrl: http://127.0.0.1:4010
data:
  customerName: Demo Customer
  customerEmail: [email protected]
---

# Buy a cup

## Goal
Buy one Studio cup with standard delivery using the supplied customer details.

## Success
- Text: Order confirmed
- Text: Order total: $24.00

Authoring boundaries

Goal is plain language. Success contains one or more Text:, Not text:, URL:, or Visual: conditions. Text checks use normalized visible text; URL checks use substring matching. Visual conditions name a reviewed screenshot baseline; see quality checks. All final conditions must hold when the agent finishes. Hidden text cannot satisfy a positive check. A negative text condition alone is a weak oracle: pair it with a positive result.

Do not mix Steps or hooks into goal tests. Existing Steps/Expected suites retain their previous behavior. Free-form semantic success claims are not supported in this phase: the runtime must know what evidence to check without letting the executor rewrite its acceptance criteria. A PASS proves these conditions, not every unstated business invariant. Inspect the recorded journey for requirements such as reloading or preserving other state, and encode critical observable outcomes explicitly.

Use Data for customer details and environment-backed credentials. A key such as password: env TEST_PASSWORD is resolved only by the browser runtime. The navigator receives available key names, not their values. Shared nonsecret fixture data can live in frontmatter. Models may provide literal search terms and select-option values from the goal and current page. Synthetic details are appropriate for the demo.

Configure a real agent

Copy examples/config/goal-agent.config.json, set the model ID to one your Ollama server has loaded, then set WEBTEST_AI_MODEL_CONFIG to that file and run npm run demo:local. Model names in examples are editable deployment placeholders or baseline configurations, not benchmark recommendations.

For a served OpenAI-compatible navigator with Jev element selection, use examples/config/goal-agent-jev.config.json, set TYPESAFE_API_KEY, and run npm run demo:live. For your own app, put the agent configuration in webtest-ai.config.json and run the Markdown suite with the normal run command.

For Intel Macs or CPU inference, use examples/config/goal-agent-cpu.config.json and start with node src/cli/index.js demo local headed smoke. See demo setup for readiness checks, concurrency and deadlines.

The two roles are independent:

• agent.profile names an Ollama or OpenAI-compatible model that chooses the next action as JSON.
• selector grounds the requested target. Unique observed labels and explicit observed form context resolve locally; ambiguity invokes Jev or a configured selector model.

Jev cannot act as the journey planner. Phase one sends text observations, visible control names, roles, context, fill/checked state and select options. It does not send screenshots to models. Provider-specific Responses, Anthropic and Gemini protocols are not native adapters. A compatible gateway must support the configured chat completion interface and JSON output.

The navigator supports click, fill, navigation within configured origins, scope selection, explicit frame/page switching, bounded wait, done and blocked. See limits and login reuse for configuration and supported contexts. It cannot execute code, choose arbitrary browser methods, change assertions or navigate to an arbitrary origin. Every interaction re-observes eligible elements and validates the bound node immediately before acting. Up to two stale, hidden or occluded element failures can return to the agent for a new decision; they remain visible in the report. There is no automatic retry of a whole mutated test.

Limits and failure meanings

Set execution.bounds for maximum steps, tool calls, model calls and candidates, plus execution.timeouts.testTimeout. Model calls have their own request deadline. Three repeated identical actions on unchanged state stop the loop. Model and selector calls consume the same per-test model-call allowance. Navigation and verification count toward the step allowance. Cleanup has its separate bounded allowance.

• PASS: the agent finished and the browser verified every fixed success condition.
• FAIL: a fixed condition mismatched or a configured quality gate failed.
• INCONCLUSIVE: the agent could not finish safely, returned invalid output, exceeded a limit, or encountered an unavailable runtime/provider.

A product defect may prevent the journey from completing and appear as INCONCLUSIVE rather than a verified assertion failure. Both fail the command. Do not equate INCONCLUSIVE with a healthy application. Model confidence values require calibration; generative selector probability output is not a calibrated probability of correctness.

Evidence and cost

Set execution.artifacts.screenshot to every-step for the demo. Each completed navigation/click/fill gets a PNG, plus the final screenshot. Screenshots are captured immediately after actions and may show an intermediate loading state; the next observation supplies subsequent page state. Other screenshot modes remain supported.

Each agent turn saves an observation JSON containing the bounded visible page state. The full report stores decisions, action explanations, browser assertion evidence, screenshots and the call ledger. The HTML report exposes these as a timeline and screenshot filmstrip. Compact agent JSON includes the goal, steps, decisions, verdict, structured quality results, and artifact references; NDJSON provides one event per call for ingestion. artifacts/demo-latest.json points to the demo's reports.

Screenshots can contain visible customer data; use synthetic data or authenticated fixture environments appropriate for artifact retention. Structured logs redact declared input values. Screenshots are not pixel-redacted.

Navigator and selector requests record provider/model, usage, latency, status and configured pricing. Unknown usage or rates remain unknown. Set profile pricing.inputUsdPerMillion and pricing.outputUsdPerMillion for your actual tariff. Self-hosting has no vendor token bill but still consumes compute. Optional worker/browser hourly rates appear as a separate execution estimate. Call counts cap inference requests; they are not a dollar-denominated hard spending cap.

Demo and release gates

The default npm run demo intentionally uses scripted HTTP fixtures. It demonstrates the real execution and reporting pipeline without a model dependency. Every simulated model call is labelled, and no billed usage is invented. It is not a live AI demo or an accuracy benchmark. demo:local and demo:live require real configuration and never silently substitute a fixture.

npm run test:browser runs all twelve goal scenarios and injects a checkout-total defect into both the goal and scripted paths. Even a simulated agent returning done cannot override the wrong total. npm run test:models:live runs the twelve goals against the configured real model and rejects simulated calls. Run that gate on the exact demo machine and model version before presenting live agent behavior.

Agent factories can invoke the existing CLI or MCP suite runner with the Markdown suite and model config, then ingest the compact report and call ledger. Run a worker per isolated job and preserve the suite hash, application build and model version with its artifacts.

Later phases

Image/GIF/video authoring, vision-based actions and assertions, richer state oracles, calibrated model routing, a shared job queue and hosted multi-tenant reporting remain later work. This phase deliberately concentrates on inspectable Markdown goals and a working browser loop.