Introduction
Desktop apps are the hardest apps to test from the outside. A web page has a DOM and a dozen good test tools. A desktop app has a window.
So desktop UI tests usually go one of two ways. A script clicks at screen coordinates and compares pictures. Or you learn a different test framework for every operating system.
A person does neither. A person reads the window, finds the button that says "Unlock", and clicks it.
tamesu (試す, Japanese for to try, to test) is an open-source CLI written in Go that tests software that way. Version 1.22 adds a desktop target for macOS, Windows and Linux. You write a step in plain language. An agent reads the app through the accessibility tree, the same tree a screen reader uses, and acts on the elements it finds there. There are no screenshots to interpret and no coordinates.
This is the fourth article about tamesu. The first three cover web apps, AI agents and A2A agents. This one covers desktop apps.
Why Desktop UI Tests Break
| Coordinate and picture scripts | tamesu | |
|---|---|---|
| A step | A click at x, y, or a match on a picture | A sentence that says what the user wants |
| What the test reads | Pixels | The accessibility tree, as text |
| A renamed or moved button | The test fails | The agent finds the element by its role and name |
| The mouse pointer | Moves across the screen | Stays where it is: a click is an accessibility press |
| Cost of AI steps | Every run pays for every step | Plans are cached, so a stable window replays with no model calls for its goals |
| Passwords | In the script | Typed at the last moment, never sent to the model |
| macOS, Windows, Linux | One tool per system | One YAML format for the three |
A tamesu test mixes three kinds of steps:
-
goal: a sentence. The agent acts until it is met. -
expect: a sentence. The model says if the window satisfies it. -
action: an exact, deterministic step, such as a click on a named button or a check on a text. No model is involved.
Use a goal where a person would say what they want. Use an action where the test must be exact.
How tamesu Reaches a Desktop App
tamesu does not talk to the app itself. It runs agent-desktop, a CLI from Vercel Labs that speaks the accessibility API of each system:
- the AX API on macOS,
- UI Automation on Windows,
- AT-SPI on Linux.
tamesu pins one version of that CLI (1.2.0 in tamesu 1.22.0). On first use it downloads the binary into ~/.cache/tamesu/agent-desktop. It checks the file against a SHA-256 that ships inside tamesu before it runs it. To use your own binary, set TAMESU_AGENT_DESKTOP.
A goal step runs in a loop of three stages:
- Observe. The window becomes one line per element. A line with a number is something the agent can act on. A line without a number is text it can read.
-
Plan. The agent gets the goal and those lines. It calls one tool:
click,type,select,press,waitordone. - Act. The tool runs on the app. The agent gets the new window and plans again.
The loop ends when the agent calls done. It also ends after max_goal_steps actions (15 by default), and then the step fails.
This is what the agent reads when the demo app of this guide starts:
window "Mochi Stock"
text "Mochi Stock"
text "Enter the staff PIN to open the stock room"
[1] textfield "PIN" (focused, secure)
[2] button "Unlock"
[3] button "Close"
[4] button "Full Screen"
[5] button "Minimize"
text "Mochi Stock"
That is the whole window: nine short lines of text. A small, fast model can drive a desktop app from that.
What You Will Build
You will test Mochi Stock, a small native Mac app for the stock room of Mochi Shoes, the demo shop of the earlier articles. It has a PIN screen, a product list with a search box and a form to receive stock. A command in the menu bar exports a CSV file.
The suite has four tests:
- Unlock the app and receive stock, written as goals.
- A wrong PIN keeps the app locked.
- The search box narrows the list.
- The export command in the menu bar shows a confirmation.
Then you will change the app twice. First you rename a button, and the test still passes. Then you add a real bug, and the test fails with the reason.
What You Need
- A Mac with macOS 13 or newer, for the demo app of this guide. The
desktoptarget also runs on Windows 10 (1809) or newer and on Linux (Ubuntu 24.04 or newer). - tamesu 1.22.0 or newer.
- The Xcode Command Line Tools (
xcode-select --install), only to build the demo app. - A Gemini API key in
GEMINI_API_KEY. tamesu calls models through Genkit, so OpenAI, Anthropic and local models work too.
Step 1: Install tamesu and Give It Access
Install the CLI
brew install xavidop/tap/tamesu # macOS
go install github.com/xavidop/tamesu/cmd/tamesu@latest # any platform
Linux packages and a Scoop bucket for Windows are on the install page.
Give the terminal access
A desktop test needs a desktop: the session of a signed-in user, where the app can open a window. Each system asks for something more:
| System | What it takes |
|---|---|
| macOS | The Accessibility permission for the app that runs tamesu (your terminal, your editor or your CI agent). It is in System Settings, Privacy & Security. Add Screen Recording to get screenshots. |
| Windows | The interactive session of the signed-in user: not a service, not a locked screen. |
| Linux | A graphical session with a window manager and the accessibility bus (at-spi2-core), on X11 or Wayland. |
On macOS, grant the two permissions now. You will check them with tamesu doctor in Step 3.
Step 2: Build the App Under Test
Mochi Stock is one Swift file. Create a folder named mochi-stock and save this as MochiStock.swift:
import SwiftUI
struct Product: Identifiable {
let id: Int
let name: String
var units: Int
}
final class Store: ObservableObject {
@Published var unlocked = false
@Published var wrongPIN = false
@Published var exported = false
@Published var pin = ""
@Published var query = ""
@Published var selected = 0
@Published var units = ""
@Published var products = [
Product(id: 0, name: "Trail runner", units: 12),
Product(id: 1, name: "City sneaker", units: 30),
Product(id: 2, name: "Canvas slip-on", units: 3),
Product(id: 3, name: "Leather boot", units: 8),
]
var total: Int { products.reduce(0) { $0 + $1.units } }
func unlock() {
unlocked = pin == "2468"
wrongPIN = !unlocked
}
func receive() {
products[selected].units += Int(units) ?? 0
units = ""
}
}
@main
struct MochiStockApp: App {
@StateObject private var store = Store()
var body: some Scene {
WindowGroup("Mochi Stock") {
Group {
if store.unlocked { StockView() } else { LockView() }
}
.environmentObject(store)
.frame(width: 560, height: 440)
}
.commands {
CommandMenu("Stock") {
Button("Export CSV") { store.exported = true }
.disabled(!store.unlocked)
}
}
}
}
struct LockView: View {
@EnvironmentObject var store: Store
var body: some View {
VStack(spacing: 12) {
Text("Mochi Stock").font(.largeTitle)
Text("Enter the staff PIN to open the stock room")
SecureField("PIN", text: $store.pin).frame(width: 160)
Button("Unlock") { store.unlock() }
if store.wrongPIN {
Text("Wrong PIN. Try again.").foregroundColor(.red)
}
}
.padding()
}
}
struct StockView: View {
@EnvironmentObject var store: Store
var shown: [Product] {
store.query.isEmpty ? store.products : store.products.filter {
$0.name.localizedCaseInsensitiveContains(store.query)
}
}
var body: some View {
VStack(alignment: .leading, spacing: 12) {
TextField("Search products", text: $store.query)
List(shown) { product in
HStack {
Text(product.name)
Spacer()
if product.units < 5 { Text("Low stock").foregroundColor(.orange) }
Text("\(product.units) in stock")
}
}
HStack {
Picker("Product", selection: $store.selected) {
ForEach(store.products) { Text($0.name).tag($0.id) }
}
TextField("Units", text: $store.units).frame(width: 80)
Button("Receive") { store.receive() }
}
Text("Total: \(store.total) units")
}
.padding()
.alert("Exported 4 products to stock.csv", isPresented: $store.exported) {
Button("OK") {}
}
}
}
Three details matter for the tests:
- The PIN is
2468, and the PIN field is a secure field. It never shows its value. - "Canvas slip-on" starts with 3 units, so it shows a Low stock label. The label goes away at 5 units or more.
- All the state lives in memory. Every launch starts locked, with 53 units in total.
Build and install it
Save this next to the Swift file as build.sh:
#!/bin/sh
set -eu
APP="Mochi Stock.app"
rm -rf "$APP"
mkdir -p "$APP/Contents/MacOS"
swiftc -O -parse-as-library MochiStock.swift -o "$APP/Contents/MacOS/MochiStock"
cat > "$APP/Contents/Info.plist" <<'EOF'
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>CFBundleName</key><string>Mochi Stock</string>
<key>CFBundleIdentifier</key><string>dev.mochi.stock</string>
<key>CFBundleExecutable</key><string>MochiStock</string>
<key>CFBundlePackageType</key><string>APPL</string>
<key>LSMinimumSystemVersion</key><string>13.0</string>
</dict>
</plist>
EOF
mkdir -p "$HOME/Applications"
rm -rf "$HOME/Applications/$APP"
cp -R "$APP" "$HOME/Applications/"
echo "installed $HOME/Applications/$APP"
chmod +x build.sh
./build.sh
installed ~/Applications/Mochi Stock.app
The script copies the app to ~/Applications for a reason. tamesu starts an app by its name or its bundle id, not by a path. So the app must live where macOS looks for apps.
Step 3: Configure the Project
Create a second folder for the tests. Add a file named tamesu.yaml:
version: 1
suites: ["suites/*.yaml"]
genkit:
plugins: [googlegenai]
models:
default: googleai/gemini-3.8-flash
judge: googleai/gemini-3.8-flash
browser_agent: googleai/gemini-3.8-flash
defaults: { timeout: 3m, max_goal_steps: 10 }
budgets: { max_cost_usd: 2 }
reporters: [console, junit, html]
secrets:
GEMINI_API_KEY: env:GEMINI_API_KEY
-
models.browser_agentis the model that drives the app in a goal step. The name comes from the browser target; the desktop target uses the same setting. -
models.judgegrades the whole session at the end. -
budgetsstops the run when the model spend passes the limit. -
secretsare references, never values.env:GEMINI_API_KEYtells tamesu where the key lives.
Step 4: Write the First Test with Goals
Create suites/stock.yaml:
suite: stock
tags: [desktop]
secrets:
STOCK_PIN: env:STOCK_PIN
targets:
stock:
type: desktop
app: Mochi Stock
tests:
- name: unlock and receive stock
target: stock
steps:
- goal: "Unlock the app with the PIN {{secrets.STOCK_PIN}}"
- expect: "The stock list is visible and the Canvas slip-on is marked as low stock"
- goal: "Receive 5 units of the Canvas slip-on"
- expect: "The Canvas slip-on has 8 units in stock and no low stock warning"
- action: { assert_text: "Total: 58 units" }
assert:
- judge: "The steps show the app being unlocked and 5 units of the Canvas slip-on being received"
- metric: { name: goal_steps, lte: 8 }
The target is two lines: the type and the name of the app. The test reads like a note to a colleague. Two goals do the work, two expect steps check the meaning, and one action pins the exact total.
The assert block runs after the steps:
-
judgeasks a model to grade the whole session against one sentence. -
metricchecks numbers that tamesu measured. Here the agent may use 8 actions at most.
Check the setup
export STOCK_PIN=2468
tamesu validate
tamesu doctor
1 suite, 1 test, 0 errors, 0 warnings
ok config: tamesu.yaml (version 1)
ok suites: 1 file, 1 test
ok secret GEMINI_API_KEY: env (link 1 of 1)
ok secret STOCK_PIN: env (link 1 of 1)
ok models: default=googleai/gemini-3.8-flash (GEMINI_API_KEY resolved), judge=googleai/gemini-3.8-flash (GEMINI_API_KEY resolved), browser_agent=googleai/gemini-3.8-flash (GEMINI_API_KEY resolved)
skip speech: no speech models configured (speech.tts / speech.stt); audio mode and sip need them
ok chromium: Google Chrome 157.0.8103.79 (/Applications/Google Chrome.app/Contents/MacOS/Google Chrome); no browser target configured
ok agent-desktop: 1.2.0 (~/.cache/tamesu/agent-desktop/1.2.0/agent-desktop)
ok desktop permissions: the desktop can be read and captured
skip livekit: no livekit target configured
skip sip: no sip target configured
The last two ok lines are the desktop checks. agent-desktop shows the binary a run will use. desktop permissions confirms that macOS lets tamesu read and capture the desktop.
Before the first run, the agent-desktop line says that version 1.2.0 "is not in the cache yet". The first tamesu run downloads it.
Run it
tamesu run
tamesu run: 1 test, output in .tamesu/runs/2026-11-17T14-43-00
PASS stock / unlock and receive stock 1m13s
1 passed, 0 failed, 0 errors in 1m13s · 5 model calls, 6801 tokens, $0.0000 · llm cache: 0 hits, 1 misses
Mochi Stock opens, unlocks, receives the stock and closes. Nobody touched the mouse.
What the agent did
Every step is in steps.json, in the run directory. This command lists the actions of the two goals:
jq -c '.[] | select(.kind == "goal") | .actions[] | [.name, .args.element, .args.text]' \
.tamesu/runs/*/tests/unlock-and-receive-stock/run-1/steps.json
["type","textfield \"PIN\"","{{secrets.STOCK_PIN}}"]
["click","button \"Unlock\"",null]
["click","combobox \"Product\"",null]
["click","menuitem \"Canvas slip-on\"",null]
["type","textfield \"Units\"","5"]
["click","button \"Receive\"",null]
Two things to notice:
- The agent found everything by role and name. Nobody told it that the product is picked in a dropdown named "Product".
-
The PIN is still a placeholder. The model planned with
{{secrets.STOCK_PIN}}and never saw2468. tamesu puts the value in at the moment of typing, and passes it to agent-desktop on standard input, never as a command-line argument.
After the unlock, the window is richer. This is what the agent reads before the second goal:
window "Mochi Stock" (focused)
[6] textfield "Search products"
[7] scrollarea
[8] outline
[9] treeitem "Trail runner 12 in stock"
[10] cell "Trail runner 12 in stock" (focused)
text "Trail runner"
text "12 in stock"
[11] treeitem "City sneaker 30 in stock"
[12] cell "City sneaker 30 in stock" (focused)
text "City sneaker"
text "30 in stock"
[13] treeitem "Canvas slip-on Low stock 3 in stock"
[14] cell "Canvas slip-on Low stock 3 in stock" (focused)
text "Canvas slip-on"
text "Low stock"
text "3 in stock"
[15] treeitem "Leather boot 8 in stock"
[16] cell "Leather boot 8 in stock" (focused)
text "Leather boot"
text "8 in stock"
text "Product"
[17] combobox "Product" value="Trail runner" (readonly)
[18] textfield "Units"
[19] button "Receive"
text "Total: 53 units"
[3] button "Close"
[4] button "Full Screen"
[5] button "Minimize"
text "Mochi Stock"
A row of the list is a treeitem. The dropdown is a combobox with its current value. The total is plain text.
Good to know:
- A click presses the element through the accessibility API. The pointer does not move.
- Text is normally set the same way, with no key pressed. Some fields take text from the keyboard only. tamesu then focuses the field and types with the real keyboard, and the run records a
keyboardevent inresult.json. The three text fields of Mochi Stock are of that kind. - The real keyboard types into whatever has the focus. Leave the keyboard and the mouse alone while a suite runs.
- An
expectstep sees what the window shows. The secure field shows no value:textfield "PIN" (focused, secure).
Step 5: Run It Again: the Plan Cache
Run the same command a second time:
tamesu run
tamesu run: 1 test, output in .tamesu/runs/2026-11-17T14-44-13
PASS stock / unlock and receive stock 31.3s
1 passed, 0 failed, 0 errors in 31.3s · 3 model calls, 1660 tokens, $0.0000 · llm cache: 0 hits, 1 misses
The first run took 1m13s, 5 model calls and 6,801 tokens. This one took 31 seconds, 3 model calls and 1,660 tokens.
When a goal succeeds, tamesu saves its actions as a plan in .tamesu/cache/actions. This is the heart of the plan of the first goal:
{
"goal": "Unlock the app with the PIN {{secrets.STOCK_PIN}}",
"url_path": "desktop://Mochi Stock",
"key": "1487a403320f0602b9600cc0",
"actions": [
{
"name": "type",
"args": {
"element": "textfield \"PIN\"",
"ref": 1,
"text": "{{secrets.STOCK_PIN}}"
}
},
{
"name": "click",
"args": {
"element": "button \"Unlock\"",
"ref": 2
}
}
]
}
The next run replays the plan and does not call the agent. The three calls that remain are the two expect steps and the judge, the steps that need judgment. The metrics of the second run say the same:
{
"cache_hits": 2,
"cache_misses": 0,
"desktop_actions": 6,
"goal_steps": 6,
"llm_cost_usd": 0,
"llm_tokens": 1660,
"observations": 10,
"turns": 0
}
The key of a plan is the goal, the app and the set of elements in the window (their roles and names). When the window changes, the key changes, and the agent plans again. You will see that in Step 9.
Two flags control the cache:
-
tamesu run --update-cacheignores the saved plans and plans every goal again. -
tamesu run --replayuses only saved plans. A goal with no plan fails. This is the mode for CI.
Commit the .tamesu/cache folder with the suite.
Step 6: Pin Exact Behavior with Actions
A goal is flexible. Some checks must not be. Add two tests to suites/stock.yaml:
- name: wrong PIN keeps the stock room closed
target: stock
steps:
- action: { fill: { name: PIN, text: "0000" } }
- action: { click: { role: button, name: Unlock } }
- action: { assert_text: "Wrong PIN. Try again." }
- expect: "The app is still locked and shows no products"
- name: search narrows the list
target: stock
steps:
- action: { fill: { name: PIN, text: "{{secrets.STOCK_PIN}}" } }
- action: { click: Unlock }
- action: { wait: { text: "Total: 53 units" } }
- action: { fill: { name: "Search products", text: "boot" } }
- action: { assert_text: "Leather boot" }
- expect: "The list shows the Leather boot and no other product"
No goal here, so no agent drives the app. Each action names its element the way a person would: by the text on it. The expect at the end of each test still asks the model one question.
The desktop target has six actions:
| Action | Argument |
|---|---|
click |
An accessible name (click: Unlock), or { role, name }
|
fill |
{ name, text }: empties the field, then types |
press |
A key or a combination: Enter, Escape, Tab, cmd+s
|
wait |
Milliseconds, or { text } to wait until the window holds a text |
screenshot |
A name: the window, saved under screenshots/
|
assert_text |
A text the window must hold, or { contains }, { equals }, { matches }
|
Good to know:
- Names are matched exactly, but upper and lower case do not matter.
- Add the
rolewhen two elements share a name. - A
filltakes secrets too.{{secrets.STOCK_PIN}}is replaced at the moment of typing. -
wait: { text: ... }is the safe way to wait for a screen change. The search test uses it before it types. - A desktop app has no URL, no page script and no network of its own to control.
navigate,assert_url,eval_js,mock,a11yandvisualbelong to the browser target only.
Step 7: Drive the Menu Bar and a Native Alert
Mochi Stock has a Stock menu in the menu bar with one command, Export CSV. It answers with a native alert. Add the fourth test:
- name: export from the menu bar
target: stock
steps:
- action: { fill: { name: PIN, text: "{{secrets.STOCK_PIN}}" } }
- action: { click: Unlock }
- action: { click: Stock }
- action: { click: "Export CSV" }
- action: { assert_text: "Exported 4 products to stock.csv" }
- action: { screenshot: export-alert }
- action: { wait: 700 }
- action: { press: Enter }
- expect: "No alert is open and the stock list is visible"
A menu item only answers when it is showing. So the test opens the menu first, then clicks the item. A name alone is the portable way to name a menu, because its role differs between systems.
The alert is not part of the window. tamesu lists what is open on top of the window after it, and searches it first. This is the end of the observation while the alert is open:
text "Total: 53 units"
[3] button "Close"
[4] button "Full Screen"
[5] button "Minimize"
text "Mochi Stock"
sheet "alert" (focused)
text "Exported 4 products to stock.csv"
[20] button "OK"
The screenshot action saves what the test saw:
Good to know:
-
Use actions for the menu bar. With tamesu 1.22.0 on macOS 27, the agent does not get the titles of the menu bar in a goal step. A goal such as "export from the Stock menu" fails there. Two
clickactions work. -
Close an alert with a key.
press: Enteranswers the default button. Aclickon the alert's own button ends in an error: agent-desktop reports that it could not verify the press. -
Wait a moment before the key. Without the short
wait, the key press can fail with aSTALE_REFerror.
Step 8: Run the Whole Suite
tamesu run
tamesu run: 4 tests, output in .tamesu/runs/2026-11-17T14-44-45
PASS stock / export from the menu bar 22.2s
PASS stock / wrong PIN keeps the stock room closed 16.1s
PASS stock / unlock and receive stock 30.7s
PASS stock / search narrows the list 18.2s
4 passed, 0 failed, 0 errors in 1m27s · 6 model calls, 2685 tokens, $0.0000 · llm cache: 0 hits, 1 misses
Four tests, 1m27s and six model calls. Desktop tests take turns. A machine has one desktop, so tamesu runs them one at a time, whatever --parallel says.
Tags and filters select a part of the suite:
tamesu run --filter "wrong PIN"
tamesu run --tags desktop
The app starts clean every time
Each test launches the app, and tamesu quits it at the end of the test. That is why every test of this suite starts at the lock screen with 53 units.
- An app that was already running is yours. tamesu joins it and leaves it open at the end.
- An app that refuses to quit, because it asks to save, is terminated.
- State that an app keeps on disk is yours to reset. Use
desktop.argson the target, orbefore_eachandafter_eachhooks.
What a run leaves behind
| File | What it holds |
|---|---|
screenshots/step-NN.png |
One screenshot of the window per step |
steps.json |
Each step, its result, its reason and the agent's actions |
agent-desktop.jsonl |
agent-desktop's own trace of what it read and did |
transcript.json |
The steps as a conversation, for judge, contains and regex assertions |
result.json |
Assertions, metrics and events |
report.html |
The whole run as one page, next to junit.xml
|
The screenshots and the trace cover the app under test only, never the rest of your desktop.
Step 9: Rename a Button
Now change the app the way a designer would. Rename the Receive button:
- Button("Receive") { store.receive() }
+ Button("Add units") { store.receive() }
Build the app again and run the first test:
./build.sh
tamesu run --filter "unlock and receive"
tamesu run: 1 test, output in .tamesu/runs/2026-11-17T14-47-51
PASS stock / unlock and receive stock 56.5s
1 passed, 0 failed, 0 errors in 56.5s · 4 model calls, 5300 tokens, $0.0000 · llm cache: 0 hits, 1 misses
The test passes, and nobody edited the suite. Look at the model calls: four, not three.
The lock screen did not change, so the first goal replayed its plan. The stock screen has a new button name, so the key of the second goal changed. The agent planned that goal again, clicked "Add units", and tamesu saved the new plan. The metrics show it:
{
"cache_hits": 1,
"cache_misses": 1,
"desktop_actions": 6,
"goal_steps": 6,
"llm_cost_usd": 0,
"llm_tokens": 5300,
"observations": 11,
"turns": 0
}
The next run replays both plans again.
Step 10: Catch a Real Bug
A rename is not a bug. This is one. In Store.receive(), a += becomes a =:
- products[selected].units += Int(units) ?? 0
+ products[selected].units = Int(units) ?? 0
The app now replaces the stock with the units you receive. Build and run:
./build.sh
tamesu run --filter "unlock and receive"
echo "exit code: $?"
tamesu run: 1 test, output in .tamesu/runs/2026-11-17T14-48-52
FAIL stock / unlock and receive stock 32.5s
x step[4] expect: The Canvas slip-on currently has 5 units in stock according to treeitem "Canvas slip-on 5 in stock", not 8.
x judge "The steps show the app being unlocked and 5 units of the Canvas slip-on being received": Although the app was unlocked, 5 units were not properly received; starting from 3 units in stock, receiving 5 units should have resulted in 8 units, but only 5 units in stock were recorded.
last turns:
agent: replayed 4 cached actions (plan 60274b032ed3361120575328)
user: The Canvas slip-on has 8 units in stock and no low stock warning
agent: The Canvas slip-on currently has 5 units in stock according to treeitem "Canvas slip-on 5 in stock", not 8.
0 passed, 1 failed, 0 errors in 32.5s · 3 model calls, 1578 tokens, $0.0000 · llm cache: 0 hits, 1 misses
exit code: 1
Both goals replayed without a problem. The buttons are all there, and every click works. A test that only clicks would pass.
The expect step reads the window and fails: the Canvas slip-on has 5 units, not 8. The judge says the same about the whole session. The exit code is 1, so a pipeline stops here.
Put the += back and build again before you go on.
Step 11: Run It in CI
Replay in the pipeline
tamesu run --replay --out out
tamesu run: 4 tests, output in out
PASS stock / export from the menu bar 22.2s
PASS stock / wrong PIN keeps the stock room closed 12.1s
PASS stock / unlock and receive stock 32.5s
PASS stock / search narrows the list 16.3s
4 passed, 0 failed, 0 errors in 1m23s · 6 model calls, 2724 tokens, $0.0000 · llm cache: 0 hits, 1 misses
With --replay, every goal comes from the saved plans. The six model calls are the five expect steps and the judge, so the job still needs the API key. The exit codes are the same as for browser tests: 0 passed, 1 a test failed, 2 config error, 3 infrastructure error, 4 over budget.
Give the runner a desktop
The runner needs what your laptop needed in Step 1:
- macOS runner: grant Accessibility to the process that starts tamesu.
- Windows runner: run the job in the interactive session of the signed-in user.
- Linux runner: most have no screen, so start a session first.
This is the Linux recipe from the tamesu documentation:
sudo apt-get install -y xvfb openbox dbus x11-utils at-spi2-core libatk-bridge2.0-0
export DISPLAY=:99
Xvfb :99 -screen 0 1280x800x24 -nolisten tcp &
openbox &
dbus-run-session -- tamesu run
tamesu's own pipeline tests the desktop target this way on every pull request: against a GTK text editor in a headless Linux container, and against Notepad on a hosted Windows runner of GitHub Actions.
Split a large suite
Desktop tests run one at a time on one machine. To go faster, give each job its own machine and split the suite:
tamesu run --replay --shard 1/2 # job 1
tamesu run --replay --shard 2/2 # job 2
Also New in 1.22: agent-browser for Web Tests
The same release adds a second engine to the browser target. Set browser.engine: agent-browser, and the agent-browser CLI reads the page and acts for the agent. The browser stays tamesu's own Chromium. Mocks, the accessibility audit, visual baselines and recordings work the same under both engines, and chromium stays the default.
Conclusion
A desktop test should say what the user wants, and fail only when the user would fail. The accessibility tree makes that possible. It turns a window into a few lines of text that a model can read and a test can assert on.
The rename and the bug show the difference. A renamed button costs one extra model call, once. A wrong number in the stock list fails the test with a sentence that says what is wrong.
Start small. Take the one flow of your app that someone still checks by hand before every release. Write it as two goals and two expect steps, and run it on your own machine first.
Further reading
- tamesu on GitHub and the documentation
- The desktop target: what each system needs, every action, artifacts and metrics
- The desktop example in the tamesu repository: a note in TextEdit, or in the editor you name
- agent-desktop, the CLI that reads the accessibility tree
-
The browser target, with the new
agent-browserengine - Agentic Browser Testing with tamesu, the first article of this set
- Testing AI Agents with tamesu, the second article of this set
- Testing A2A Agents with tamesu, the third article of this set






Top comments (0)