Bounded browser testing with Jev

For the new goal-based execution path, start with Markdown goals. A Goal plus Success conditions now runs through a bounded agent loop; the Steps/Expected form below remains supported.

The default runner executes versioned Markdown contracts through native Chrome DevTools Protocol (CDP). It observes the current document before each interaction, selects a concrete element, checks that the same element is still usable, and performs the action. A model never writes executable browser code or changes the assertions during a run.

This branch changes the default execution engine. The existing V2 planner and Playwright flows are available with execution.engine: "legacy". See migration before moving existing suites.

Quick start

Install dependencies with npm ci in a checkout. Install Chrome or Chromium separately. Set WEBTEST_AI_CDP_CHROME if the executable is outside the standard locations. The bounded runner does not use Playwright for browser actions; the package retains Playwright for compatibility, authoring, and existing tools.

Set TYPESAFE_API_KEY in your environment to enable Jev. Never put the key in a spec or commit it to configuration. No model call is made when an eligible element has an exact, unique label.

Create webtest-ai.config.json:

{
  "driver": { "name": "cdp" },
  "execution": {
    "engine": "bounded",
    "timeouts": { "testTimeout": 30000, "expectTimeout": 5000 },
    "bounds": { "maxSteps": 50, "maxToolCalls": 200, "maxModelCalls": 12, "maxCandidates": 80 },
    "artifacts": { "screenshot": "only-on-failure", "trace": "off" },
    "quality": { "a11y": { "mode": "off" }, "vitals": { "mode": "off" } }
  },
  "selector": {
    "provider": "jev",
    "model": "jev-1.13.0",
    "minConfidence": 0.9,
    "minMargin": 0.2
  },
  "reporting": {
    "costs": { "workerUsdPerHour": 0.12, "browserUsdPerHour": 0 }
  }
}

The hourly prices are illustrative allocation rates. Replace them with your own rates or omit them to keep infrastructure cost unknown. A zero browser rate explicitly means browser cost is included elsewhere or intentionally excluded from the estimate. Shared startup, storage, and network charges are not allocated by this runner.

Create a spec under specs/:

---
suite: Address book
baseUrl: http://localhost:3000
---

# Test: Save shipping details

## Steps
- Open "/addresses"
- Fill intent "Full name" with data "name"
- Click action "Save the shipping address"

## Data
- name: Ada

## Expected
- Assert text "Shipping address saved"

Run npx webtest-ai run. The existing run command also accepts a suite path, tags, targets, and worker count. Results appear in artifacts/reports.

Contract vocabulary

Click element and Click text are compatibility spellings for grounded interactions with actionable elements, not arbitrary CSS selectors. URL assertions use substring matching. Text assertions compare normalized visible text in the selected document, excluding input values and hidden text. They do not infer business success from a toast unless the contract explicitly asserts that toast.

Every contract must finish its main steps and expected conditions with an explicit assertion (an optional screenshot may follow). Expected sections accept assertions only. Unrecognized prose, unsupported actions, unnamed submit buttons, and goals without observable success conditions return INCONCLUSIVE. The compiler never silently drops an expected condition. The contract hash includes the actions, approved data declarations, and limits.

Before hooks run before steps. After hooks are attempted after a failure while the deadline and budgets permit. A failed cleanup cannot erase the original failure. Destroying the isolated browser context is always attempted and has a separate two-second cleanup allowance.

Element decisions

1. Collect a bounded, text-only surface from the selected document and open shadow roots. Preserve accessible labels, roles, form or section context, visibility, and editability. Input values are omitted.
2. Resolve an exact, unique eligible label locally. Otherwise ask the configured selector to choose a reference or NONE.
3. Validate the response against the actual candidate set, complete probability distribution, confidence threshold, and separation from the next choice. Confidence is a model signal, not a calibrated probability of correctness.
4. Revalidate the selected DOM node, its relevant attributes and context, visibility, enabled state, and hit target immediately before acting. Replaced, hidden, covered, or changed elements stop the step. There is no automatic alternate journey or model repair.
5. Run the next instruction against a fresh observation. The independent assertion path supplies PASS or FAIL; uncertainty produces INCONCLUSIVE.

Jev uses TypeSafe's Choice API, not a chat-completions endpoint. This implementation pins jev-1.13.0 and checks the returned version. It sends candidate text only. Jev does not receive screenshots, GIFs, or video in this path. Media ingestion belongs in a separate authoring workflow that produces an inspectable contract.

The hosted API and model rates were checked against TypeSafe API documentation and model documentation on September 21, 2026. The selected version lists $0.042 per million input tokens and no output token charge. A different version requires its own pricing configuration. The current checks use a local service implementing that API contract; live Jev quality and latency still need calibration with a real key and representative applications.

Open source model servers

Use an Ollama, vLLM, LM Studio, llama.cpp, or other supported OpenAI-compatible profile for the same finite selection task:

{
  "selector": { "provider": "profile", "profile": "local-selector", "minConfidence": 0.9, "minMargin": 0.2 },
  "models": {
    "profiles": {
      "local-selector": {
        "provider": "ollama",
        "model": "your-installed-model",
        "endpoint": "http://localhost:11434",
        "responseFormat": "json",
        "limits": { "maxOutputTokens": 2048, "timeoutMs": 10000 },
        "pricing": { "inputUsdPerMillion": 0, "outputUsdPerMillion": 0 }
      }
    }
  }
}

Explicit zero token prices mean no token invoice, not free hardware. Supply worker rates separately if applicable. Omit prices when unknown. OpenAI-compatible servers can set responseFormat: "text" when they do not support JSON response mode; the returned content must still be valid JSON. Function calling, vision, and long-horizon planning capabilities are unnecessary. Local models must be evaluated for selection quality; schema compliance alone does not establish reliability.

Costs and machine reports

Each test and each attempt retains its tool timeline, model/provider/version, response usage, configured price snapshot, latency, errors, assertion evidence, and artifact paths. Invalid model responses still retain billable usage. Automatic selector retries are disabled. Suite totals include earlier test attempts.

• cost contains model token cost. totalUsd is null when any charged call has missing usage or rates; knownUsd preserves the known portion.
• runCost estimates model, worker, and browser cost using elapsed time and configured hourly rates. Missing infrastructure rates make the total unknown. Concurrent tests are allocated their own active elapsed time, so choose per-session rates rather than a whole shared host rate.
• HTML provides searchable tests, status filters, expected versus observed evidence, attempt history, and expandable calls.
• RUN.agent.json is the compact schema-v2 result document.
• RUN.calls.ndjson contains one tool event per line, with run, test, step, attempt, and call IDs.
• RUN.json retains the full evidence. Legacy reports are marked as incompletely instrumented rather than reporting zero cost.

For an illustration, 1,000 tests with three Jev requests each at 1,200 input tokens per request would have $0.1512 in model input charges at the documented rate. This is arithmetic, not a measured suite cost. Actual prompt size, server latency, browser execution, and infrastructure charges must be measured.

Screenshots are raw browser evidence and are not automatically redacted. Text logs remove approved field values and use the existing redaction rules. Keep sensitive artifacts in controlled storage or turn screenshots off. Prompts, credentials, and raw fill values are not included in model-call logs.

Agent factory integration

Use webtest-ai.spec.parse to read Markdown, webtest-ai.run.suite to execute it, and webtest-ai.report.test to inspect a particular result with paginated calls. The last tool accepts report, testId, offset, and limit (1 through 100). Its nextOffset is null on the final page. Use the JSON files for durable ingestion into your factory's run store. The MCP resource webtest-ai://schemas/agent-report-v2 publishes the report and tool-event JSON Schema.

Factory workers should consume immutable specs and data fixtures. Store the application build, environment, browser version, model version, feature flags, and contract hash alongside each run. Keep specification repair and contract approval outside execution. Treat INCONCLUSIVE as an unsuccessful gate, and retain flaky: true when a retry passes.

The CLI exits unsuccessfully for failed, inconclusive, or blocked tests. Mutating tests do not automatically repeat unless their retry policy explicitly sets safeToRepeat: true; test retries are capped at three. Mark this only for resettable or idempotent fixtures. An uncertain model decision is never automatically retried.

This MCP server runs suites synchronously. Calls are serialized around process-wide working-directory state; suites can still use internal workers. A production factory should allocate one worker process per job. Durable queueing, cancellation APIs, tenant isolation, callback delivery, and remote browser capacity are roadmap work.

Current boundary

Bounded interactions support Chromium, open shadow roots, scoped discovery, nested frames, approved cross-origin frames, and explicit tab/popup switching. Assertions inspect the selected document. Configure origins, budgets, and login validation using the limits and authentication guide. Closed shadow roots, canvas controls, transformed frames, browser traces, and video-to-contract authoring remain outside this grounded action surface. Explicit visual checkpoints are supported as described in the quality guide.