Demo: Everyday Goods

Everyday Goods is a compact Node HTTP server with three frontend files, a session-scoped bag and a server-calculated checkout total. No frontend dependencies, external assets, database or build step are needed. Nothing is charged or shipped.

Installed-package users should use npx webtest-ai demo from their own project. Commands starting with npm run or node src/ below assume a source checkout; installed helpers live under node_modules/@asserthive/webtest-ai/. See the isolated beta trial for the candidate version and complete setup.

Rehearse the platform

Install dependencies with npm ci, then run npm run demo. Chrome must be installed; WEBTEST_AI_CDP_CHROME can point to a custom binary. The command chooses local ports, opens isolated browser contexts, runs twelve Markdown goals, writes reports and shuts down its services. Failed or inconclusive results produce a failing exit code.

This default mode is a scripted rehearsal. A local HTTP fixture chooses actions from observed browser state. A second fixture demonstrates the selector protocol. Neither is a language model. Both are explicitly labelled in the console, manifest and report. This mode verifies the pipeline and is useful for setup and CI; it must not be presented as live agent intelligence.

Demonstrate a real agent

1. Start an Ollama or OpenAI-compatible model endpoint.
2. Copy examples/config/goal-agent.config.json and select a model that is actually loaded. Set WEBTEST_AI_MODEL_CONFIG to its absolute path.
3. Run npm run demo:local. The model now chooses each action from the goal and current page.
4. To use Jev for ambiguous elements, use goal-agent-jev.config.json, configure the navigator, set TYPESAFE_API_KEY, and run npm run demo:live.

Jev is an element selector and still needs a navigator model for goal execution. Missing configuration fails explicitly. Live modes never fall back to scripted fixtures. Model latency, quality and price need verification on your chosen model before a live presentation.

Run the session demo with a real ChatGPT model

The session mode lets an agent available in the current ChatGPT session supply navigator decisions while the harness runs the real browser. It is useful when you have a ChatGPT subscription but no provider API key or local inference server. The harness opens an authenticated, local bridge and waits for one JSON action for each fresh observation:

WEBTEST_AI_CDP_CHROME=/path/to/chromium \\
node src/cli/index.js demo session headed

The command prints a private control-file path. A controller reads the next observation and submits an action through demo-site/sessionControl.js:

export WEBTEST_AI_SESSION_CONTROL=/absolute/path/from/the/command
node demo-site/sessionControl.js next
printf '%s\n' '{"id":"<request-id>","action":{"action":"click","target":"Add Field notebook to bag","reason":"The observed product control starts the goal."}}' | node demo-site/sessionControl.js reply

The session bridge forwards the model messages and returns the supplied action. It has no goal rules, fallback planner, or generated answer. Jev is mocked in this mode so the run proves the real navigator and browser protocol without requiring a Typesafe seat. Reports label both facts explicitly. The report shows Token usage: Not reported and Model cost: Usage unavailable because a ChatGPT subscription session does not expose API billing metadata to this harness. Use an API endpoint such as OpenRouter, Ollama, or a compatible gateway when you need measured tokens and cost.

One key for hosted models

OpenRouter offers a hosted model catalog behind one API endpoint and OpenRouter key. The existing openrouter-compatible adapter uses this interface; no per-provider connector is required for models that support our Chat Completions and JSON-output contract. This does not guarantee compatibility with every model or provider feature.

1. Create an API key in https://openrouter.ai/settings/keys and arrange API credits separately from any ChatGPT subscription.
2. Copy examples/config/goal-agent-openrouter.config.json. Replace REPLACE_WITH_OPENROUTER_MODEL_ID with an explicit model ID from the catalog that supports JSON output. Avoid changing aliases or automatic model routing when comparing runs.
3. Export OPENROUTER_API_KEY privately in your shell and set WEBTEST_AI_MODEL_CONFIG to the config file.
4. Run node src/cli/index.js demo local headed smoke, then remove smoke for the full suite. Here local means the locally running demo browser with a configured model endpoint; the inference endpoint can be remote.

The same profile can drive navigation and ambiguous-element selection. Our runtime still owns actions, limits and browser assertions. The current readiness warmup is Ollama-specific; hosted calls begin with the test's first observation. The example needs a model ID and credential and has not been live-validated.

Cost reporting uses provider returned token usage and explicitly configured per-million-token pricing rates. If the provider returns tokens but the profile has no rates, the report says Rate unavailable. If the provider does not return usage, it says Usage unavailable. Provider-side billing is not reconciled into our ledger. A model name alone does not pin the upstream serving provider; strict provider routing and fallback provenance need additional gateway options before provider-level benchmarking.

For a self-hosted organizational gateway, LiteLLM Proxy can expose the same OpenAI-compatible interface and a virtual key while managing upstream credentials centrally. Use openai-compatible and the gateway's endpoint, model alias and key environment variable. Hosted provider credentials are still configured at the gateway. Native Ollama, Jev's specialized selector API and future Codex subscription access remain separate integration paths where useful.

References: OpenRouter quickstart, structured output support, LiteLLM.

Intel Macs and CPU inference

Use examples/config/goal-agent-cpu.config.json for an initial Qwen3 4B experiment. This is a starting configuration, not a model accuracy or latency guarantee. It uses one worker, a 180-second limit per navigator/selector request, a 15-minute limit per test, an 8192-token context, at most 512 output tokens, and non-thinking mode for Qwen3. The model stays loaded for 15 minutes between calls. The enclosing test deadline can still interrupt an inference request.

Start Ollama in a separate terminal with ollama serve, then:

ollama pull qwen3:4b
export WEBTEST_AI_MODEL_CONFIG="$PWD/examples/config/goal-agent-cpu.config.json"
node src/cli/index.js demo local headed smoke

smoke runs just Find a product. Its manifest is artifacts/demo-smoke.json, so a one-test success never replaces the full-suite manifest. Remove smoke after inspecting the real-model result:

node src/cli/index.js demo local headed

Local and live demos default to one worker. Override with workers=2 or demo.workers in the configuration after measuring your machine. Request timeoutMs is honored for both navigator and selector, supports 1 through 600000 milliseconds, and is recorded on each call. Invalid timeout values fail explicitly. The generic model profile retains its shorter defaults.

Before an Ollama demo starts a browser, a small real JSON inference warms the model and checks availability. It has its own request deadline and usage ledger in artifacts/demo-readiness-<timestamp>.json. Failure stops startup, without manufacturing twelve inconclusive browser tests. Passing this small probe does not establish that larger page observations will fit the latency budget or that the model can solve a test. Cold loading, prompt evaluation, generation and queued work may all affect response time.

Startup inference usage and cost are recorded separately in that file and in the demo manifest's readiness field; they are excluded from per-test cost and the HTML test totals. No model is silently substituted. Local/live demos reject profiles explicitly marked simulated.

If readiness times out, check ollama ps, the Ollama server log, and a direct model request. Capture the readiness JSON and test call ledger for diagnosis. Unknown cost is expected when rates or usage are unavailable; a timeout is not a free successful call. CPU and electricity costs are not automatically measured.

What to show

Open specs/storefront.md: each scenario has a goal and a few observable success conditions, with no click sequence. Shared customer details live at suite level. The scenarios cover product search, standard checkout, express delivery, item removal, checkout validation, bag persistence after reload, and updating Shipping without saving Billing.

Open the generated HTML report. Show the original goal, chosen actions and explanations, independently verified success conditions, action screenshots, observations, target decisions, model calls and costs. Explain that an agent's done response requests verification; it does not supply the verdict.

The app provides a Reset demo control for manual exploration. Each automated scenario starts in a new context, so tests do not depend on reset ordering or another test's bag.

Artifacts

PNG capture is enabled after each action in demo goal mode. It can show an intermediate loading state. The next observation records the subsequent page state. Text observations are redacted; screenshots are not pixel-redacted, so use synthetic data.

Model costs use measured usage and configured rates. Fixtures have zero usage and are marked simulated. Missing live pricing remains unknown. Infrastructure estimates are separate.

Failure demonstration

npm run test:demo also injects a one-dollar checkout error. Both the goal agent path and retained scripted path must report FAIL with expected $42.00 and observed $43.00. The harness succeeds only when the runtime detects that regression. The agent cannot change the fixed total or report itself successful.

See Markdown goals for runtime limits, supported actions, model configuration and phase-one scope.

Model identity and token usage in reports

The HTML report has a Models and token usage table for each test and a run-wide summary. Agent JSON and full JSON expose the same models array. Rows group actual requests by role, provider, requested model, recorded model, identity source and simulation status, including retry attempts. A configured selector that was never called does not create a usage row.

Input, output and total token counts come from recorded provider usage. Missing totals are null, with a separate known subtotal for partial records. Mocked Jev calls are shown separately so their zero usage cannot be mistaken for live navigator usage. Cached input is included in input tokens and is available separately in JSON.

A provider's returned model ID is recorded with source response; a configured fallback is labeled configuration. Older call ledgers use recorded when the original identity source was not captured. The external bridge uses session-label: a label such as ChatGPT-session is not proof of an exact model version. Its existing protocol supplies no token usage, so the report shows Not reported. This reporting change does not add a Codex CLI usage collector.

The demo collects accessibility and lab LCP/CLS in warning mode. Expand Quality checks in the report to inspect findings and thresholds. INP is not collected. Visual regression runs only for explicit checkpoints with reviewed baselines; ordinary action screenshots are evidence. See quality configuration and baseline review.