Flaky tests
A flaky test passes sometimes and fails sometimes, with no change to the app. Flaky tests are the main reason teams stop trusting end-to-end tests, so Sisyphus is built to make them rare and visible.
Why Sisyphus tests are rarely flaky
Section titled “Why Sisyphus tests are rarely flaky”The classic cause is timing: the test clicked before the button appeared, or checked before the list refreshed. In Sisyphus every step waits for its target to be ready, and every expect retries until it passes. As long as you never sleep and always check what the user sees, timing is handled for you.
Retries that don’t hide anything
Section titled “Retries that don’t hide anything”sis run --retries 1or retries: in sisyphus.yaml, a suite, or one test. A failing test reruns from scratch. If a rerun passes, the test counts as passed but flaky, and the failed attempts’ errors, screenshots and trees are kept in the result. You get a green build and the evidence.
Finding them
Section titled “Finding them”sis stats --flaky # tests that flaked or flip between pass and failsis stats # also the slowest testsIn Studio, Runs → Flaky and slow tests shows the same over the recorded runs, and the Tests list has a Flaky filter.
Common causes and fixes
Section titled “Common causes and fixes”| Symptom | Likely cause | Fix |
|---|---|---|
| Fails only when run with others | Tests share data | Give each its own data (${worker}, ${random}); reset in setup |
| Fails only after another test | A test depends on what an earlier one left | Make it set up its own state; try --no-reuse |
| Times out now and then | The app is sometimes slower than the timeout | Raise that step’s timeout:, or find out why the app is slow |
| Clicks the wrong thing sometimes | An animation moves things | Wait for the end state with an expect first; or step_delay for that app |
| Depends on the date or time | Real clock | Fix the data, or check a pattern, not an exact value |
Treat them as bugs
Section titled “Treat them as bugs”A flaky test is either a test bug or an app bug (a real race condition users hit too). Fix them as they appear; don’t let them pile up.
