qavo: a QA agent built on Jev's typed choices

LLMs are pretty much prediction machines, predicting the next token over and over until they’ve given you back a wall of text. Jev is different, because it doesn’t give you text at all.

Jev is the first of what TypeSafe calls System One models, which they describe as models “built to make fast, structured decisions that software can use directly.” You send it some state as JSON along with a set of questions, and it sends back typed answers with probabilities attached. There’s no prose in the response and no explanation of how it got there. The questions come in three kinds:

• Choice, which picks one option from a fixed list
• Noul, which gives the probability that a statement is true, from 0 to 1
• Score, which places the state along a set of ordered levels that you define

Those three are the entire interface. If you ask a Choice question, this is what you get back:

{
  "choice": "TYPE_TEXT",
  "confidence": 0.99,
  "probabilities": { "TYPE_TEXT": 0.99, "CLICK": 0.01, "SELECT": 0, "WAIT": 0, "DONE": 0, "BLOCKED": 0 }
}

The answer is always one of the options you offered. The probabilities are trained to be calibrated against real outcomes, so a 0.99 is supposed to mean something quite different from a 0.55. Confidence is a measure of how concentrated that distribution is: if all of the probability lands on one option you get 1.0, and it drops as the probability spreads across the others.

The first time I saw this I wanted to know what you could build with a model that can only say “option 3, and I’m 95% sure.”

So I built qavo (shouts out Migos). Part of it was that I wanted an excuse to try Jev on something real, and part of it was that I have a handful of apps in production and I want CI to tell me they still work. The unit tests passing doesn’t tell me much about whether a person can still log in, fill out a form, hit save, and land on the right screen afterward, and that’s the thing I actually care about.

qavo reads a scenario written in plain language, drives a real browser through it, and reports pass, fail, blocked, or unclear for each step, along with the evidence behind that result.

A model that can only pick from a list turns out to fit that job well. A test runner needs an answer it can act on, and it needs some way to know when that answer shouldn’t be trusted. Because Jev always answers with one of the options I gave it, there’s no output to parse and no way for it to invent an action outside the list. The probability gives me a reason to stop and report unclear when Jev isn’t sure, instead of pushing ahead on a guess. And since every answer has the same shape, the whole run can be written to a report, and I can send any single decision again later to see whether it still comes out the same way.

The stack is small:

• TypeScript on Node 24, since everything else I write is TypeScript and so is the TypeSafe SDK
• playwright-core for the browser, with my own in-page snapshot script in place of Playwright’s locators so that code never builds a selector from model output
• @typesafe-ai/sdk for talking to Jev
• zod on every answer, so that a choice outside the offered labels or a probability outside 0 to 1 throws an error instead of getting acted on
• Vitest with static fixture pages and a stubbed Jev client, so pnpm test runs offline

The idea of asking one operation question plus one target question per operation comes from browser-use/jev-ultrafast, a Python project that showed a browser agent can run on Jev choices alone. qavo ports that idea to TypeScript and adds what a QA tool needs on top of it, which is scenarios, expect checks, and reports.

Running it against Realty Pilot Suite

The first real app I pointed it at was Realty Pilot Suite, the back-office tool I build for property managers. It’s a Vite app behind Supabase auth, and it’s exactly the kind of app I don’t click through by hand after every change.

Here’s one of its scenarios. It lives in the app’s repo under qa/ rather than in qavo:

{
  "name": "Request status updates toggle",
  "url": "/dashboard",
  "steps": [
    {
      "intent": "Go to settings then notifications, then turn off the Request Status Updates in-app notifications switch",
      "expect": "The Request Status Updates in-app notifications switch is off"
    },
    {
      "intent": "Turn on the Request Status Updates in-app notifications switch",
      "expect": "The Request Status Updates in-app notifications switch is on"
    }
  ]
}

You won’t find any selectors or test IDs in there. The intent is what I’d tell a person to do, and the expect is what I’d ask them to check afterward. qavo starts the run already logged in, because qavo login opens a browser once, lets me sign in by hand, and saves that session for later runs.

One turn, from the inside

Each step in a scenario runs as a loop, and code handles every part of that loop except for one question per turn:

1. Wait until the page is quiet, meaning there are no open fetch or XHR requests and the controls and text have stayed the same for 250 ms.
2. Snapshot the page with one in-page script that reads the URL, the visible text, and every visible, enabled control along with its role, name, and state (checked, expanded, and so on).
3. Ask Jev what to do next.
4. Check the guards, then act on the element Jev picked.
5. Go around again until Jev says DONE or BLOCKED, or until a limit stops the step.

Step 3 is a single API call. This is roughly what it looks like, trimmed down from qavo’s decide.ts:

import { TypeSafeClient, choice } from "@typesafe-ai/sdk";

const jev = new TypeSafeClient();

const { answers } = await jev.systemOne({
  state: {
    page: { url: snapshot.url, title: snapshot.title, text: snapshot.text },
    elements: snapshot.elements, // [{ element: "[7] Request Status Updates", role: "switch", checked: true }, ...]
    recent_actions: history.slice(-10),
  },
  questions: {
    operation: choice(
      { step: intent, rules: OPERATION_RULES },
      {
        CLICK: "Click a button, link, tab, checkbox, menu item, or suggestion.",
        WAIT: "Wait for the page to update.",
        DONE: "The step is visibly complete.",
        BLOCKED: "No offered operation can make progress on the step.",
      },
    ),
    CLICK_target: choice(
      { step: intent, operation: "CLICK", rules: TARGET_RULES },
      Object.fromEntries(clickable.map((e) => [String(e.index), describeElement(e)])),
    ),
  },
});

The list of operations changes from page to page, because code only offers what the page allows. TYPE_TEXT shows up only when there’s an editable field, and SCROLL_DOWN only when there’s more of the page below. There’s also one target question for every operation that has candidates, and they all go out in the same request. Jev answers them in parallel, and none of the questions can see how the others were answered. In practice that means Jev picks the operation while also picking the best click target in case the next move is a click, the best field in case it’s typing, and so on. Code keeps the target that belongs to the winning operation and ignores the rest, so each action costs one round trip.

The confidence for a turn is the lower of the two answers that were actually used. If it falls under 0.5, the step ends as unclear, since I’d rather have the test tell me it couldn’t decide than have it guess and move on.

When Jev picks DONE, one more request checks the step’s expect against the page, and this time the question is a Noul:

const { answers } = await jev.systemOne({
  state: { page, elements },
  questions: { expect: noul({ expected: step.expect, rules: EXPECT_RULES }) },
});

A probability of 0.7 or higher passes the step, 0.3 or lower fails it, and anything in between comes back as unclear.

Here’s every decision from the run in the video:

Step	Turn	Jev chose	Confidence	Latency
1	1	CLICK “Change in settings”	0.82	478 ms
1	2	CLICK “Notification Preferences”	0.96	195 ms
1	3	CLICK “Request Status Updates in-app notifications”	0.99	177 ms
1	4	DONE, expect check 0.97	0.99	163 ms
2	1	CLICK “Request Status Updates in-app notifications”	0.99	195 ms
2	2	DONE, expect check 0.97	0.98	181 ms

The whole scenario passed in 7.4 seconds, using 8 Jev requests and no calls to a text model. The least confident decision was the very first one, which I think makes sense. The dashboard has no visible “Settings” link, so Jev went for the “Change in settings” link inside a billing card, and 0.82 seems like a reasonable amount of doubt for taking a shortcut like that (it was right, for what it’s worth).

What the real app taught it

The first runs against RPS didn’t go nearly that smoothly. Each of the bugs below got past the test suite, because the suite only ran against small static HTML fixtures, and a real React app behind auth behaves differently in ways I hadn’t thought about.

The first problem was that Jev saw an empty page. The very first run against the real app ended as blocked on turn one. The snapshot ran as soon as the page loaded, but a React app draws after load and then shows a skeleton while it fetches its data, so Jev was handed a page with zero controls on it and correctly decided there was nothing it could do. The fixtures draw everything before goto returns, which is why they never caught it. The fix is the wait at the top of the loop, where qavo holds off until there are no open requests and nothing on the page has changed for 250 ms. I also added a fixture, spa.html, that behaves like a real app on purpose, with an empty root, then a skeleton, then a slow fetch.

The second was that the Settings link was hiding in a menu. Like in most apps, the Settings link in RPS lives in the account menu behind a button with my initials on it. The links in a closed menu aren’t visible, so they never make it into the snapshot, and Jev had no way to pick one. It chose BLOCKED instead. The snapshot now records aria-haspopup on each control, and the operation rules tell Jev to open a likely menu before it gives up.

The third was the scenario itself. My first version was a single step that turned the switch off and then back on. That can’t pass reliably, because when the step ends the switch looks exactly the way it did at the start, and nothing on the page shows that anything happened. Splitting it into two steps, each changing the state once and each with its own expect, fixed it. That one wasn’t a code change so much as a rule for writing scenarios, and it’s in the README now.

After the first of these bugs I added a rule to qavo’s CLAUDE.md saying an iteration isn’t done until one run passes against a real app. The fixture tests are still useful, but they only prove that qavo works on pages I wrote myself.

Getting it into CI

The whole reason for qavo is to have a check that runs on every change, so most of the work after the loop went into making it behave like something a CI job can call.

The exit code carries the result. 0 means every step passed, 1 means something failed, was blocked, or came back unclear, and 2 means the run finished but the upload failed. I made unclear fail the build on purpose, because I’d rather open a report and look than have CI wave through something Jev wasn’t sure about.

Runs write report.json and screenshots to a private temp directory outside the checkout, so nothing ends up in the app’s repo. If the QAVO_S3_* variables are set, qavo uploads the report and its screenshots to a private R2 bucket, sends the report last, and prints where everything went. Each report holds the exact request Jev saw on every turn, which means I can pull a failure from CI and pick it apart later on my laptop.

Secrets come from the environment. A value like "password": "@env:QA_PASSWORD" in a step gets resolved from the CI secret store before the browser starts, and if the variable is missing the run fails before anything happens. Those values never reach a model, and reports show *** in their place. Runs that use them also skip screenshots, since an app can show a secret anywhere on the page and I’d have no way of knowing where.

I also didn’t want writing scenarios to mean writing JSON by hand. qavo scenario new takes QA instructions I already have, either pasted in or from a .txt or .docx file, and turns headings into scenarios, list items into steps, and Expected: lines into checks. It doesn’t call a model and it copies my wording exactly, then shows me the drafts so I can approve or edit them before it writes anything to disk.

What qavo doesn’t do yet is run in Realty Pilot Suite’s CI. Everything it needs is in place, but I haven’t wired it up, and that’s what I’m working on next.

TODO

• Stage 0 on Realty Pilot Suite: 5 to 10 real scenarios, each run 3 times, with every Jev decision labeled right or wrong, so I can set the confidence threshold from real data instead of the 0.5 I picked
• A check-pr job in RPS’s CI that runs the scenarios against a preview deploy
• A way to log in inside CI, since today the session comes from qavo login by hand and that doesn’t work on a runner
• An HTML report, so a failure is readable without opening JSON
• Exporting a passed run as a plain Playwright test
• A Claude “rescuer” for steps that end unclear or blocked, but only if the Stage 0 numbers show that Jev needs one