Running in CI
End-to-end tests pay off when they run on every change, automatically. A common setup:
- Pull requests: the
smokesuite. A few minutes; blocks merging if it fails. - Main branch or nightly: everything, in parallel and split across machines.
sis run --suite smoke --junit reports/junit.xmlsis run --suite nightly --junit reports/junit.xmlExit codes
Section titled “Exit codes”| Code | Means |
|---|---|
0 |
Everything passed (flaky tests too; they’re listed) |
1 |
A test failed |
2 |
The run couldn’t start: invalid file, unknown suite, missing runtime |
GitHub Actions
Section titled “GitHub Actions”name: e2eon: [pull_request, push]jobs: smoke: if: github.event_name == 'pull_request' runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 # install `sis` for your platform here - run: sis setup - run: sis run --suite smoke --junit reports/junit.xml env: PASSWORD: ${{ secrets.E2E_PASSWORD }} - uses: actions/upload-artifact@v4 if: always() with: name: e2e-smoke path: | reports/ .sisyphus/runs/
nightly: if: github.event_name == 'push' runs-on: ubuntu-latest strategy: fail-fast: false matrix: { shard: [1, 2, 3] } # split the suite across 3 machines steps: - uses: actions/checkout@v4 - run: sis setup - run: sis run --suite nightly --shard ${{ matrix.shard }}/3 --junit reports/junit-${{ matrix.shard }}.xml - uses: actions/upload-artifact@v4 if: always() with: name: e2e-${{ matrix.shard }} path: | reports/ .sisyphus/runs/Upload .sisyphus/runs/ as an artifact: it holds each failure’s screenshot and accessibility tree, so you can see what happened without rerunning. Feed reports/*.xml to any JUnit reporter (GitHub Actions reporters, GitLab’s artifacts:reports:junit, Jenkins’ JUnit plugin) to see failures on the pull request.
Secrets in CI
Section titled “Secrets in CI”Store passwords and tokens in your CI’s secret store and pass them as environment variables. Tests read them with ${secret.NAME}, and Sisyphus masks them in results.
--workers Nruns N tests at once on one machine;--shard i/ngives each of n machines its own slice. Both need independent tests.serve:dev servers start once per run, not per test.- Slow desktop apps:
reuse. Run with--no-reusenightly to catch tests that lean on each other.
Desktop apps in CI
Section titled “Desktop apps in CI”Web and Electron tests run on Linux, macOS and Windows runners. Native macOS app tests need a macOS runner with the Accessibility permission granted to the runner process, and they run one at a time, since there’s one screen.
Flaky tests
Section titled “Flaky tests”With --retries 1, a test that fails then passes is reported as flaky: it doesn’t fail the build, but it’s listed in the summary, the JSON, the JUnit report and Studio’s Runs. See Flaky tests.
