Tests with AI agents
AI coding agents (Claude Code, Cursor, Codex and others) are good at writing code and bad at knowing whether the product still works. Sisyphus closes that gap. Through its MCP server, an agent can explore your app, write tests, run them after every change, and read results that say exactly what broke.
AI helps write the tests. The tests themselves are plain YAML that runs deterministically, with no AI and no per-run cost, forever.
Connect your agent over MCP
Section titled “Connect your agent over MCP”In Studio, Settings → AI agents (MCP) shows a ready-made config for the open project, with a copy button. It looks like this:
{ "mcpServers": { "sisyphus": { "command": "/path/to/sis", "args": ["mcp", "-C", "/path/to/project"] } }}For Claude Code, put it in .mcp.json in your repository. Other agents have their own MCP settings; the command and arguments are the same.
What the agent can do
Section titled “What the agent can do”| Tool | Does |
|---|---|
list_tests |
The tests, flows (with parameters and how often they’re used) and suites |
run_tests |
Run tests (same options as sis run) and get a compact result with the failure screenshot |
get_run, get_artifact |
Look at a previous run, a screenshot or an accessibility tree |
validate_tests |
Check files without running them, with suggested fixes |
test_stats |
Flaky and slow tests over recent runs |
session_start → snapshot → act → session_save |
Explore an app step by step and save what worked as a test |
screenshot, session_end |
See the app during a session; close it |
list_apps, capabilities |
Apps it can test; what this build supports |
run_load |
Run a load test |
Exploring is writing
Section titled “Exploring is writing”The agent doesn’t control your screen. It drives the app through Sisyphus, one step at a time, using the same steps a test uses:
session_startopens the app.snapshotreturns what’s on screen, as selectors.actruns one step (click: button "New project"). Every step that works is added to a draft test.session_savewrites the draft as a.sis.yamlfile.
By the time the agent understands the feature, the test is written. From then on it just calls run_tests.
Results made for agents
Section titled “Results made for agents”A passing run is a few lines. A failure carries the step and line, the error code, the nearest matching selectors and the paths to the screenshot and tree, so the agent can usually fix the test, or spot the regression, in one go. Secrets are masked, so they never reach the model.
Working with an agent well
Section titled “Working with an agent well”- Ask for one behaviour at a time: “write a test that a wrong password shows an error”.
- The agent is told Sisyphus’s rules: list existing tests and flows first, reuse flows, one behaviour per test, check what the user sees.
- Review its tests like code. Is the name a behaviour? Does it stand alone? Does it check what matters, or just click around?
- Ask it to run the suite after its changes, and to fix what it broke, not the tests that caught it.
