Skip to content

Running in CI

End-to-end tests pay off when they run on every change, automatically. A common setup:

  • Pull requests: the smoke suite. A few minutes; blocks merging if it fails.
  • Main branch or nightly: everything, in parallel and split across machines.
Terminal window
sis run --suite smoke --junit reports/junit.xml
sis run --suite nightly --junit reports/junit.xml
Code Means
0 Everything passed (flaky tests too; they’re listed)
1 A test failed
2 The run couldn’t start: invalid file, unknown suite, missing runtime
.github/workflows/e2e.yml
name: e2e
on: [pull_request, push]
jobs:
smoke:
if: github.event_name == 'pull_request'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
# install `sis` for your platform here
- run: sis setup
- run: sis run --suite smoke --junit reports/junit.xml
env:
PASSWORD: ${{ secrets.E2E_PASSWORD }}
- uses: actions/upload-artifact@v4
if: always()
with:
name: e2e-smoke
path: |
reports/
.sisyphus/runs/
nightly:
if: github.event_name == 'push'
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix: { shard: [1, 2, 3] } # split the suite across 3 machines
steps:
- uses: actions/checkout@v4
- run: sis setup
- run: sis run --suite nightly --shard ${{ matrix.shard }}/3 --junit reports/junit-${{ matrix.shard }}.xml
- uses: actions/upload-artifact@v4
if: always()
with:
name: e2e-${{ matrix.shard }}
path: |
reports/
.sisyphus/runs/

Upload .sisyphus/runs/ as an artifact: it holds each failure’s screenshot and accessibility tree, so you can see what happened without rerunning. Feed reports/*.xml to any JUnit reporter (GitHub Actions reporters, GitLab’s artifacts:reports:junit, Jenkins’ JUnit plugin) to see failures on the pull request.

Store passwords and tokens in your CI’s secret store and pass them as environment variables. Tests read them with ${secret.NAME}, and Sisyphus masks them in results.

  • --workers N runs N tests at once on one machine; --shard i/n gives each of n machines its own slice. Both need independent tests.
  • serve: dev servers start once per run, not per test.
  • Slow desktop apps: reuse. Run with --no-reuse nightly to catch tests that lean on each other.

Web and Electron tests run on Linux, macOS and Windows runners. Native macOS app tests need a macOS runner with the Accessibility permission granted to the runner process, and they run one at a time, since there’s one screen.

With --retries 1, a test that fails then passes is reported as flaky: it doesn’t fail the build, but it’s listed in the summary, the JSON, the JUnit report and Studio’s Runs. See Flaky tests.

Made by Bussin Bytes© 2026 Bussin Bytes Ltd · sisyphus.bussinbytes.io