Skip to content

Flaky tests

A flaky test passes sometimes and fails sometimes, with no change to the app. Flaky tests are the main reason teams stop trusting end-to-end tests, so Sisyphus is built to make them rare and visible.

The classic cause is timing: the test clicked before the button appeared, or checked before the list refreshed. In Sisyphus every step waits for its target to be ready, and every expect retries until it passes. As long as you never sleep and always check what the user sees, timing is handled for you.

Terminal window
sis run --retries 1

or retries: in sisyphus.yaml, a suite, or one test. A failing test reruns from scratch. If a rerun passes, the test counts as passed but flaky, and the failed attempts’ errors, screenshots and trees are kept in the result. You get a green build and the evidence.

Terminal window
sis stats --flaky # tests that flaked or flip between pass and fail
sis stats # also the slowest tests

In Studio, Runs → Flaky and slow tests shows the same over the recorded runs, and the Tests list has a Flaky filter.

Symptom Likely cause Fix
Fails only when run with others Tests share data Give each its own data (${worker}, ${random}); reset in setup
Fails only after another test A test depends on what an earlier one left Make it set up its own state; try --no-reuse
Times out now and then The app is sometimes slower than the timeout Raise that step’s timeout:, or find out why the app is slow
Clicks the wrong thing sometimes An animation moves things Wait for the end state with an expect first; or step_delay for that app
Depends on the date or time Real clock Fix the data, or check a pattern, not an exact value

A flaky test is either a test bug or an app bug (a real race condition users hit too). Fix them as they appear; don’t let them pile up.

Made by Bussin Bytes© 2026 Bussin Bytes Ltd · sisyphus.bussinbytes.io