Live Model Testing

Live testing verifies provider connectivity and model behavior separately from deterministic unit and release tests. A live run must be identified as live in its report; it must never silently fall back to fixtures.

Provider roles

Role	Provider examples	Responsibility
Navigator	Ollama, OpenRouter, OpenAI-compatible server, Codex session	Chooses the next bounded browser action
Selector	Typesafe Jev or a configured compatible model	Chooses one element from fresh candidates when local matching is ambiguous
Runtime	Native CDP or Playwright driver	Executes actions, checks assertions, and writes evidence

Ollama

Ollama is the first local path for open-source models. Start the service and pull a model:

ollama serve
ollama pull qwen3:8b

Point the navigator profile at the local endpoint:

{
  "agent": { "profile": "local-agent" },
  "models": {
    "profiles": {
      "local-agent": {
        "provider": "ollama",
        "model": "qwen3:8b",
        "endpoint": "http://127.0.0.1:11434",
        "capabilities": { "reasoning": true },
        "limits": { "maxOutputTokens": 2048 }
      }
    }
  }
}

WEBTEST_AI_MODEL_CONFIG=examples/config/goal-agent.config.json
npm run demo:local

OpenAI-compatible gateways

OpenAI-compatible profiles make it possible to switch hosted and self-hosted navigator models without changing the test contract. OpenRouter can provide one gateway key for multiple model families. The selected provider still determines usage and pricing metadata.

{
  "agent": { "profile": "navigator" },
  "models": {
    "profiles": {
      "navigator": {
        "provider": "openrouter-compatible",
        "model": "provider/model-name",
        "endpoint": "https://openrouter.ai/api/v1",
        "apiKeyEnv": "OPENROUTER_API_KEY",
        "limits": { "maxOutputTokens": 1024 }
      }
    }
  }
}

Typesafe Jev

Jev is a separate selector role. It is not the navigator and it does not replace the model that plans the journey. WebTest AI sends Jev an intent, operation, and bounded fresh candidate list. Jev returns a choice, confidence, and probability distribution. The runtime rejects missing, unsafe, or insufficiently distinct choices.

{
  "agent": { "profile": "navigator" },
  "selector": { "provider": "jev", "model": "jev-1.13.0" },
  "models": {
    "profiles": {
      "navigator": {
        "provider": "openai-compatible",
        "model": "your-served-navigator-model",
        "endpoint": "http://127.0.0.1:8000/v1",
        "limits": { "maxOutputTokens": 512 }
      }
    }
  }
}

export TYPESAFE_API_KEY=your-key
export WEBTEST_AI_MODEL_CONFIG=examples/config/goal-agent-jev.config.json
npm run demo:live

Exact unique element names bypass a selector model call. Ambiguous choices must meet the configured confidence and margin thresholds or the test is INCONCLUSIVE.

Codex subscription session

A Codex subscription can drive the navigator through the external session bridge used by the demo. This is different from an API provider profile. The bridge can execute browser decisions without exposing subscription token billing metadata, so reports may show the model identity with usage unavailable.

WEBTEST_AI_SESSION_MODEL=Codex-subscription
node src/cli/index.js demo session headed

For a controlled demonstration, use the Codex navigator with mocked Jev. This proves the browser protocol, goal loop, and reporting without claiming Jev accuracy or pricing.

Evaluate live models

Run a small calibration set before expanding coverage. Track:

goal completion rate
wrong-target and unsafe-action rate
INCONCLUSIVE rate and reason
p95 action latency
navigator calls and selector calls per goal
recorded token usage and cost per correctly completed goal

Keep deterministic fixture runs in CI for release gates. Use live runs as a separate credentialed evaluation gate, and fail the gate when live configuration is missing instead of silently reporting a fixture pass.