Skip to content

Organising a suite

Piece What it is Runs alone?
Step One action or check: click: button "Save" No
Flow Reusable steps with parameters: “log in as X” No, tests call it
Test One behaviour: “a wrong password shows an error” Yes: passes or fails
Suite A named selection of tests: “smoke”, “nightly” Yes

The smallest thing you run is a test. The smallest thing you reuse is a flow. A suite is a selection of tests, not a big test made of small ones.

sisyphus.yaml
flows/
login.sis.yaml
create-project.sis.yaml
tests/
auth/ sign-in.sis.yaml, sign-out.sis.yaml, …
checkout/ pay-by-card.sis.yaml, refund.sis.yaml, …
layout/ phone.sis.yaml, …

Folders are features. Someone looking for the checkout tests should find them in tests/checkout.

name: Paying by card charges the card
tags: [smoke, payments]

Useful tags:

  • smoke: the 5 to 10 journeys that must never break. Run on every pull request.
  • slow: tests you skip when you’re in a hurry.
  • manual: tests that need something you don’t have in CI.
  • A team or area name, so each team can run its own.
sisyphus.yaml
suites:
smoke: { description: Every pull request, tags: [smoke] }
nightly: { description: Everything, paths: [tests], skip_tags: [manual], workers: 4, retries: 1 }

A suite picks tests by paths (default: all), keeps those with any of its tags, then drops those with any skip_tags. It can set its own workers and retries.

Terminal window
sis run --suite smoke
sis list --suites # each suite and how many tests it has

In Studio, suites run from the play button above the Tests list.

A test that only makes sense on some systems says so with os: (macos, windows, linux):

name: The Finder shows the export
os: [macos]

On any other system it’s reported as skipped, with the reason (runs on macos only), never failed. The same suite can run on every OS in CI.

When only a few steps differ, keep one test and branch on ${os}:

- if: { equals: ["${os}", windows] }
then:
- press: Alt+F4
else:
- press: Mod+Q

For a run: command, give the other OS its own version instead (macos, linux, windows, or unix for both macOS and Linux):

- run:
script: scripts/seed.sh
windows: { script: scripts/seed.ps1 }

What each OS supports is on Platform support.

If every test of an app signs in first, don’t repeat it: put it in the app’s setup (see Flows). A test that must start signed out sets app_setup: false.

  1. Every test stands alone.
  2. One behaviour per test, 5 to 20 steps.
  3. Shared steps go in flows.
  4. Setup every test of an app needs goes on the app.
  5. Same test, many inputs: data:, not copies.
  6. Keep cross-app tests few, and about the interaction.
  7. Check what the user sees; never sleep.
  8. Tests that run in parallel don’t share data they change.

The reasons are in Writing good tests.

Made by Bussin Bytes© 2026 Bussin Bytes Ltd · sisyphus.bussinbytes.io