Skip to content

Tests with AI agents

AI coding agents (Claude Code, Cursor, Codex and others) are good at writing code and bad at knowing whether the product still works. Sisyphus closes that gap. Through its MCP server, an agent can explore your app, write tests, run them after every change, and read results that say exactly what broke.

AI helps write the tests. The tests themselves are plain YAML that runs deterministically, with no AI and no per-run cost, forever.

In Studio, Settings → AI agents (MCP) shows a ready-made config for the open project, with a copy button. It looks like this:

.mcp.json
{
"mcpServers": {
"sisyphus": { "command": "/path/to/sis", "args": ["mcp", "-C", "/path/to/project"] }
}
}

For Claude Code, put it in .mcp.json in your repository. Other agents have their own MCP settings; the command and arguments are the same.

Tool Does
list_tests The tests, flows (with parameters and how often they’re used) and suites
run_tests Run tests (same options as sis run) and get a compact result with the failure screenshot
get_run, get_artifact Look at a previous run, a screenshot or an accessibility tree
validate_tests Check files without running them, with suggested fixes
test_stats Flaky and slow tests over recent runs
session_start → snapshot → act → session_save Explore an app step by step and save what worked as a test
screenshot, session_end See the app during a session; close it
list_apps, capabilities Apps it can test; what this build supports
run_load Run a load test

The agent doesn’t control your screen. It drives the app through Sisyphus, one step at a time, using the same steps a test uses:

  1. session_start opens the app.
  2. snapshot returns what’s on screen, as selectors.
  3. act runs one step (click: button "New project"). Every step that works is added to a draft test.
  4. session_save writes the draft as a .sis.yaml file.

By the time the agent understands the feature, the test is written. From then on it just calls run_tests.

A passing run is a few lines. A failure carries the step and line, the error code, the nearest matching selectors and the paths to the screenshot and tree, so the agent can usually fix the test, or spot the regression, in one go. Secrets are masked, so they never reach the model.

  • Ask for one behaviour at a time: “write a test that a wrong password shows an error”.
  • The agent is told Sisyphus’s rules: list existing tests and flows first, reuse flows, one behaviour per test, check what the user sees.
  • Review its tests like code. Is the name a behaviour? Does it stand alone? Does it check what matters, or just click around?
  • Ask it to run the suite after its changes, and to fix what it broke, not the tests that caught it.
Made by Bussin Bytes© 2026 Bussin Bytes Ltd · sisyphus.bussinbytes.io