Organising a suite
The pieces, small to big
Section titled “The pieces, small to big”| Piece | What it is | Runs alone? |
|---|---|---|
| Step | One action or check: click: button "Save" |
No |
| Flow | Reusable steps with parameters: “log in as X” | No, tests call it |
| Test | One behaviour: “a wrong password shows an error” | Yes: passes or fails |
| Suite | A named selection of tests: “smoke”, “nightly” | Yes |
The smallest thing you run is a test. The smallest thing you reuse is a flow. A suite is a selection of tests, not a big test made of small ones.
Layout
Section titled “Layout”sisyphus.yamlflows/ login.sis.yaml create-project.sis.yamltests/ auth/ sign-in.sis.yaml, sign-out.sis.yaml, … checkout/ pay-by-card.sis.yaml, refund.sis.yaml, … layout/ phone.sis.yaml, …Folders are features. Someone looking for the checkout tests should find them in tests/checkout.
Tags cut across folders
Section titled “Tags cut across folders”name: Paying by card charges the cardtags: [smoke, payments]Useful tags:
smoke: the 5 to 10 journeys that must never break. Run on every pull request.slow: tests you skip when you’re in a hurry.manual: tests that need something you don’t have in CI.- A team or area name, so each team can run its own.
Suites name a selection once
Section titled “Suites name a selection once”suites: smoke: { description: Every pull request, tags: [smoke] } nightly: { description: Everything, paths: [tests], skip_tags: [manual], workers: 4, retries: 1 }A suite picks tests by paths (default: all), keeps those with any of its tags, then drops those with any skip_tags. It can set its own workers and retries.
sis run --suite smokesis list --suites # each suite and how many tests it hasIn Studio, suites run from the play button above the Tests list.
Tests for one platform
Section titled “Tests for one platform”A test that only makes sense on some systems says so with os: (macos, windows, linux):
name: The Finder shows the exportos: [macos]On any other system it’s reported as skipped, with the reason (runs on macos only), never failed. The same suite can run on every OS in CI.
When only a few steps differ, keep one test and branch on ${os}:
- if: { equals: ["${os}", windows] } then: - press: Alt+F4 else: - press: Mod+QFor a run: command, give the other OS its own version instead (macos, linux, windows, or unix for both macOS and Linux):
- run: script: scripts/seed.sh windows: { script: scripts/seed.ps1 }What each OS supports is on Platform support.
Shared setup lives on the app
Section titled “Shared setup lives on the app”If every test of an app signs in first, don’t repeat it: put it in the app’s setup (see Flows). A test that must start signed out sets app_setup: false.
Rules that keep a big suite healthy
Section titled “Rules that keep a big suite healthy”- Every test stands alone.
- One behaviour per test, 5 to 20 steps.
- Shared steps go in flows.
- Setup every test of an app needs goes on the app.
- Same test, many inputs:
data:, not copies. - Keep cross-app tests few, and about the interaction.
- Check what the user sees; never sleep.
- Tests that run in parallel don’t share data they change.
The reasons are in Writing good tests.
