snapshot of https://github.com/mattermost/mattermost.git (master, 2026-09-16; blobs > 131072B omitted)
Some checks failed
Server CI / Postgres (shard 0) (push) Has been skipped
Server CI / Postgres (shard 3) (push) Has been skipped
BuildEnv Docker Image / build-image (push) Failing after 4s
BuildEnv Docker Image / build-image-fips (push) Failing after 4s
Migration Review & Release Notes / Generate migration review & release notes (push) Failing after 3s
Server CI / Build Environment (push) Failing after 3s
Server CI / Postgres (shard 1) (push) Has been skipped
Tools CI / check-style (mattermost-govet) (push) Failing after 4s
Server CI / Postgres (shard 2) (push) Has been skipped
Tools CI / check-style (mmgotool) (push) Failing after 3s
Server CI / Elasticsearch v8 Compatibility (push) Has been skipped
Server CI / OpenSearch v2 Compatibility (push) Has been skipped
Server CI / Postgres FIPS (shard 0) (push) Has been skipped
Tools CI / Test (mattermost-govet) (push) Failing after 3s
Server CI / Postgres FIPS (shard 1) (push) Has been skipped
Server CI / Postgres FIPS (shard 2) (push) Has been skipped
Tools CI / Test (mmgotool) (push) Failing after 3s
Server CI / Postgres FIPS (shard 3) (push) Has been skipped
Server CI / Run mmctl tests (push) Has been skipped
Server CI / Run mmctl tests (FIPS) (push) Has been skipped
Server CI / Merge Postgres Test Results (push) Has been skipped
Server CI / Merge Postgres FIPS Test Results (push) Has been skipped
Server CI / Check backport migrations (push) Has been skipped
Server CI / Check generated files (push) Has been skipped
Server CI / check-style (push) Has been skipped
Server CI / Vet API (push) Has been skipped
Server CI / Build mattermost server app (push) Has been skipped
Server CI / Server CI Complete (push) Failing after 2s
Server CI / Server CI Artifacts (push) Has been skipped
Server CI / Sentry Upload (push) Has been skipped
Server CI / Server CI Report (push) Failing after 5m1s
Server CI Nightly Race / Race Detector (push) Has been skipped
CodeQL / Analyze (go) (push) Has been cancelled
CodeQL / Analyze (javascript) (push) Has been cancelled
Webapp CI Cache Warm / Warm Webapp node_modules Cache (push) Has been cancelled
E2E CI Cache Warm / Warm E2E npm Registry Cache (push) Has been cancelled
API / build (push) Has been cancelled
Web App CI / check-types (push) Has been cancelled
Web App CI / check-lint (push) Has been cancelled
Web App CI / check-i18n (push) Has been cancelled
Web App CI / check-external-links (push) Has been cancelled
Web App CI / test (platform) (push) Has been cancelled
Web App CI / test (mattermost-redux) (push) Has been cancelled
Web App CI / test (channels shard 1/4) (push) Has been cancelled
Web App CI / test (channels shard 2/4) (push) Has been cancelled
Web App CI / test (channels shard 3/4) (push) Has been cancelled
Web App CI / test (channels shard 4/4) (push) Has been cancelled
Web App CI / upload-coverage (push) Has been cancelled
Web App CI / build (push) Has been cancelled
YAML Lint / yamllint (push) Has been cancelled
Push mirrored docker images / cd/Push mirrored docker images (push) Has been cancelled
docs-cd / Build and deploy docs (push) Has been cancelled
Docs CI / build (push) Has been cancelled
E2E Cache Warm: Cypress deps / Warm Cypress deps cache (push) Has been cancelled
Scorecards supply-chain security / Scorecard analysis (push) Has been cancelled
Server CI Weekly / Postgres FIPS (push) Has been skipped
Server CI Weekly / Run mmctl tests (FIPS) (push) Has been skipped
Server CI Weekly / Postgres with binary parameters (push) Has been skipped
Server CI Weekly / Build Environment (push) Failing after 1s
Server CI Cache Warm / Warm Go Module and Build Cache (push) Failing after 2s
Server CI Nightly Race / Build Environment (push) Failing after 1s
Some checks failed
Server CI / Postgres (shard 0) (push) Has been skipped
Server CI / Postgres (shard 3) (push) Has been skipped
BuildEnv Docker Image / build-image (push) Failing after 4s
BuildEnv Docker Image / build-image-fips (push) Failing after 4s
Migration Review & Release Notes / Generate migration review & release notes (push) Failing after 3s
Server CI / Build Environment (push) Failing after 3s
Server CI / Postgres (shard 1) (push) Has been skipped
Tools CI / check-style (mattermost-govet) (push) Failing after 4s
Server CI / Postgres (shard 2) (push) Has been skipped
Tools CI / check-style (mmgotool) (push) Failing after 3s
Server CI / Elasticsearch v8 Compatibility (push) Has been skipped
Server CI / OpenSearch v2 Compatibility (push) Has been skipped
Server CI / Postgres FIPS (shard 0) (push) Has been skipped
Tools CI / Test (mattermost-govet) (push) Failing after 3s
Server CI / Postgres FIPS (shard 1) (push) Has been skipped
Server CI / Postgres FIPS (shard 2) (push) Has been skipped
Tools CI / Test (mmgotool) (push) Failing after 3s
Server CI / Postgres FIPS (shard 3) (push) Has been skipped
Server CI / Run mmctl tests (push) Has been skipped
Server CI / Run mmctl tests (FIPS) (push) Has been skipped
Server CI / Merge Postgres Test Results (push) Has been skipped
Server CI / Merge Postgres FIPS Test Results (push) Has been skipped
Server CI / Check backport migrations (push) Has been skipped
Server CI / Check generated files (push) Has been skipped
Server CI / check-style (push) Has been skipped
Server CI / Vet API (push) Has been skipped
Server CI / Build mattermost server app (push) Has been skipped
Server CI / Server CI Complete (push) Failing after 2s
Server CI / Server CI Artifacts (push) Has been skipped
Server CI / Sentry Upload (push) Has been skipped
Server CI / Server CI Report (push) Failing after 5m1s
Server CI Nightly Race / Race Detector (push) Has been skipped
CodeQL / Analyze (go) (push) Has been cancelled
CodeQL / Analyze (javascript) (push) Has been cancelled
Webapp CI Cache Warm / Warm Webapp node_modules Cache (push) Has been cancelled
E2E CI Cache Warm / Warm E2E npm Registry Cache (push) Has been cancelled
API / build (push) Has been cancelled
Web App CI / check-types (push) Has been cancelled
Web App CI / check-lint (push) Has been cancelled
Web App CI / check-i18n (push) Has been cancelled
Web App CI / check-external-links (push) Has been cancelled
Web App CI / test (platform) (push) Has been cancelled
Web App CI / test (mattermost-redux) (push) Has been cancelled
Web App CI / test (channels shard 1/4) (push) Has been cancelled
Web App CI / test (channels shard 2/4) (push) Has been cancelled
Web App CI / test (channels shard 3/4) (push) Has been cancelled
Web App CI / test (channels shard 4/4) (push) Has been cancelled
Web App CI / upload-coverage (push) Has been cancelled
Web App CI / build (push) Has been cancelled
YAML Lint / yamllint (push) Has been cancelled
Push mirrored docker images / cd/Push mirrored docker images (push) Has been cancelled
docs-cd / Build and deploy docs (push) Has been cancelled
Docs CI / build (push) Has been cancelled
E2E Cache Warm: Cypress deps / Warm Cypress deps cache (push) Has been cancelled
Scorecards supply-chain security / Scorecard analysis (push) Has been cancelled
Server CI Weekly / Postgres FIPS (push) Has been skipped
Server CI Weekly / Run mmctl tests (FIPS) (push) Has been skipped
Server CI Weekly / Postgres with binary parameters (push) Has been skipped
Server CI Weekly / Build Environment (push) Failing after 1s
Server CI Cache Warm / Warm Go Module and Build Cache (push) Failing after 2s
Server CI Nightly Race / Build Environment (push) Failing after 1s
This commit is contained in:
492
.github/e2e-tests-workflows.md
vendored
Normal file
492
.github/e2e-tests-workflows.md
vendored
Normal file
@@ -0,0 +1,492 @@
|
||||
# E2E Test Pipelines
|
||||
|
||||
Three automated E2E test pipelines cover different stages of the development lifecycle.
|
||||
|
||||
## Pipelines
|
||||
|
||||
| Pipeline | Trigger | Editions Tested | Image Source |
|
||||
|----------|---------|----------------|--------------|
|
||||
| **PR** (`e2e-tests-ci.yml`) | Argo Events on `Enterprise CI/docker-image` status | enterprise | `mattermostdevelopment/**` |
|
||||
| **Merge to master/release** (`e2e-tests-on-merge.yml`) | Platform delivery after docker build (`delivery-platform/.github/workflows/mattermost-platform-delivery.yaml`) | enterprise, fips | `mattermostdevelopment/**` |
|
||||
| **Release cut** (`e2e-tests-on-release.yml`) | Platform release after docker build (`delivery-platform/.github/workflows/release-mattermost-platform.yml`) | enterprise, fips, team (future) | `mattermost/**` |
|
||||
|
||||
All pipelines follow the **smoke-then-full** pattern: smoke tests run first, full tests only run if smoke passes.
|
||||
|
||||
## Workflow Files
|
||||
|
||||
```
|
||||
.github/workflows/
|
||||
├── e2e-tests-ci.yml # PR orchestrator
|
||||
├── e2e-tests-on-merge.yml # Merge orchestrator (master/release branches)
|
||||
├── e2e-tests-on-release.yml # Release cut orchestrator
|
||||
├── e2e-tests-cypress.yml # Shared wrapper: calls the cypress template
|
||||
├── e2e-tests-playwright.yml # Shared wrapper: calls the playwright template
|
||||
├── e2e-tests-playwright-rolling-upgrades.yml # resolves the from-version matrix, calls the template per version
|
||||
├── e2e-tests-cypress-template.yml # cypress + test-system-io dispatch
|
||||
├── e2e-tests-playwright-template.yml # playwright + test-system-io dispatch
|
||||
└── e2e-tests-playwright-rolling-upgrades-template.yml # one from-version: upgrade-from/to, then dispatch on the upgraded server
|
||||
```
|
||||
|
||||
### Call hierarchy
|
||||
|
||||
```
|
||||
e2e-tests-ci.yml ─────────────────┐
|
||||
e2e-tests-on-merge.yml ───────────┤──► e2e-tests-cypress.yml ────► e2e-tests-cypress-template.yml
|
||||
e2e-tests-on-release.yml ─────────┘ e2e-tests-playwright.yml ─► e2e-tests-playwright-template.yml
|
||||
│ └► e2e-tests-playwright-rolling-upgrades.yml
|
||||
│ └► e2e-tests-playwright-rolling-upgrades-template.yml
|
||||
│ (once per resolved from-version)
|
||||
```
|
||||
|
||||
`e2e-tests-playwright-rolling-upgrades.yml` is also directly runnable from the Actions UI, so it has no caller in that case.
|
||||
|
||||
---
|
||||
|
||||
## Workflow Architecture
|
||||
|
||||
The template splits into five jobs — `prepare-run`, `prep-deps`, `dispatch-begin`, `workers` (matrix), and `report` — and pushes spec-level execution to [Test System IO](https://github.com/mattermost/mattermost-test-system-io) so workers stay thin and identical.
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────────────────────────────┐
|
||||
│ Template: e2e-tests-{cypress,playwright}-template.yml │
|
||||
│ │
|
||||
│ ┌───────────────────┐ ┌──────────────────────────────┐ │
|
||||
│ │ prepare-run │ │ prep-deps │ │
|
||||
│ │ (1 runner) │ parallel │ (1 runner) │ │
|
||||
│ │ │ ◄────────────► │ │ │
|
||||
│ │ • build workers │ │ Cypress: │ │
|
||||
│ │ matrix [1..N] │ │ • cypress/node_modules │ │
|
||||
│ │ • compute commit │ │ • ~/.cache/Cypress (binary)│ │
|
||||
│ │ status context │ │ │ │
|
||||
│ │ • emit composite │ │ Playwright: │ │
|
||||
│ │ identity │ │ • webapp/platform/{client, │ │
|
||||
│ │ │ │ types}/{lib,node_mod} │ │
|
||||
│ │ │ │ • playwright/node_modules │ │
|
||||
│ │ │ │ • playwright/lib/dist │ │
|
||||
│ │ │ │ • ~/.cache/ms-playwright │ │
|
||||
│ │ │ │ (chromium only) │ │
|
||||
│ │ │ │ │ │
|
||||
│ │ │ │ → saved to actions/cache │ │
|
||||
│ └─────────┬─────────┘ └───────────────┬──────────────┘ │
|
||||
│ │ │ │
|
||||
│ │ ▼ │
|
||||
│ │ ┌──────────────────────────────┐ │
|
||||
│ │ │ dispatch-begin │ │
|
||||
│ │ │ • register run with │ │
|
||||
│ │ │ Test System IO │ │
|
||||
│ │ │ • runs immediately before │ │
|
||||
│ │ │ workers to minimise the │ │
|
||||
│ │ │ inactivity-timeout window │ │
|
||||
│ │ └───────────────┬──────────────┘ │
|
||||
│ │ │ │
|
||||
│ └────────────────────┬─────────────────────┘ │
|
||||
│ ▼ │
|
||||
│ ┌─────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ workers (matrix, fail-fast: false) │ │
|
||||
│ │ Cypress full: N=40 | Playwright full: N=10 │ │
|
||||
│ │ │ │
|
||||
│ │ each worker, in parallel: │ │
|
||||
│ │ 1. sparse-checkout actions + full checkout-repo │ │
|
||||
│ │ 2. setup-node │ │
|
||||
│ │ 3. restore caches ◄─── actions/cache (from prep-deps) │ │
|
||||
│ │ (fail-on-cache-miss: true) │ │
|
||||
│ │ 4. cloud-init + start-server (docker compose stack) │ │
|
||||
│ │ 5. prepare-cypress | prepare-playwright (run setup project) │ │
|
||||
│ │ 6. dispatch-run ──────────────────────────────────┐ │ │
|
||||
│ │ (pulls specs from Test System IO, runs locally, │ │ │
|
||||
│ │ posts result, loops until queue is empty) │ │ │
|
||||
│ │ 7. cloud-teardown │ │ │
|
||||
│ └────────────────────┬────────────────────────────────────┼───────┘ │
|
||||
│ │ │ │
|
||||
│ ▼ │ │
|
||||
│ ┌─────────────────────────────────────────────────┐ │ │
|
||||
│ │ report │ │ │
|
||||
│ │ • pull aggregated results from Test System IO │ ◄────┘ │
|
||||
│ │ • post commit status │ │
|
||||
│ │ • send webhook notification │ │
|
||||
│ └─────────────────────────────────────────────────┘ │
|
||||
└────────────────────────────────────────────────────────────────────────┬─┘
|
||||
│
|
||||
┌───────────────────────────▼───┐
|
||||
│ Test System IO (external) │
|
||||
│ • spec-level dispatch │
|
||||
│ • result aggregation │
|
||||
│ • retry orchestration │
|
||||
└───────────────────────────────┘
|
||||
```
|
||||
|
||||
### Key properties
|
||||
|
||||
- **Spec-level vs. job-level parallelism.** The matrix sizes the runner pool; Test System IO does the spec assignment. Slow specs don't block a worker — fast workers keep pulling the next spec from the queue.
|
||||
- **Cache-only workers.** `prep-deps` installs once per workflow run and saves to `actions/cache`. Every worker restores with `fail-on-cache-miss: true` and runs zero `npm ci`. Eliminates the 40-way `EEXIST/ENOENT` race in npm's shared cacache writer.
|
||||
- **dispatch-begin runs late.** It depends on `prep-deps` so the gap between Test System IO run registration and the first worker calling `dispatch-run` is just per-worker setup (~3–5 min). Registering earlier risks the run timing out before any worker checks in, bulk-failing every spec.
|
||||
- **Playwright slim slice.** Playwright only consumes `@mattermost/client` and `@mattermost/types` from webapp, so prep-deps caches just those two packages' built `lib/` and `node_modules` (~10–30 MB) instead of the full `webapp/node_modules` tree (~1–2 GB).
|
||||
- **Browser/binary caches.** Cypress caches `~/.cache/Cypress` (cypress binary lives outside node_modules); playwright caches `~/.cache/ms-playwright` (chromium only). Both keyed on the framework's lockfile so they invalidate on version bumps.
|
||||
- **No retry plumbing in the template.** Test System IO handles per-spec retries; the workflow only sees aggregated results.
|
||||
|
||||
### Playwright rolling upgrades (separate from full suite)
|
||||
|
||||
Two workflows, split on the question each answers:
|
||||
|
||||
- **`e2e-tests-playwright-rolling-upgrades.yml`** — *which* versions to upgrade from. Resolves the matrix via `script/resolve_upgrade_matrix.mjs` (last 3 minors + active ESR; ESR entries carry `isESR: true` and a `contextLabel` ending in `-esr`), then calls the template once per entry in a `strategy.matrix` of reusable-workflow invocations.
|
||||
- **`e2e-tests-playwright-rolling-upgrades-template.yml`** — *how* to test one version, given two images: `upgrade_from_server_image` (that entry's `dockerTag`) and `SERVER_IMAGE` (the to-image). Same job shape as `e2e-tests-playwright-template.yml` — `prepare-run` → `prep-deps` → `dispatch-begin` → `workers` → `report` — with the harness prepended to each worker.
|
||||
|
||||
Per worker, within one from-version:
|
||||
|
||||
1. `upgrade-from` → `upgrade-to`
|
||||
2. `dispatch-run` against the just-upgraded server
|
||||
|
||||
So the upgraded server does not stop at the harness — it goes on to run the normal suite, which is the coverage this pipeline exists for. Nothing sits between the two: no `--project=setup`, no re-patching, no restart, since an upgrade that needed the suite's setup re-run before the server was usable would not be a passing upgrade. The upgrade specs never enter the queue themselves: `upgrade-specs/` sits outside the Playwright `testDir`, so `dispatch-begin` cannot enqueue them.
|
||||
|
||||
**Statuses match the matrix.** A 4-entry matrix posts 4 contexts, `e2e-test/playwright-full/{edition}/upgrade-from-{contextLabel}`, each covering that version's harness *and* its post-upgrade suite — posted pending by its `dispatch-begin` and resolved by its `report`. There is no aggregate context. An empty matrix is the exception: the workflow posts a single `upgrade-from-none` context and never invokes the template.
|
||||
|
||||
Each worker upgrades its own server (a server cannot be shared across runners), so the harness runs `workers` times per from-version and the pipeline costs `matrix size × workers` runners. `workers` defaults to 20, matching the full suite, and is an input; workers get 60m (vs the full suite's 30m) to cover the harness's two image pulls and swap on top of their share of the suite.
|
||||
|
||||
There is deliberately no `testcontainers:down` first — the runner is fresh, so `upgrade-from`'s clean-slate guard has nothing to trip over. `PW_TESTCONTAINERS_REUSE=true` is what lets `upgrade-to` adopt the stack, survive the swap, and keep every per-spec dispatch on the same upgraded server. Step 2 is skipped when the harness fails; that worker then never reports, the run closes short of `total-reports-expected`, and the version's context goes red.
|
||||
|
||||
**Commit status description** — written by the Test System IO summary, same shape as the full suite (`image_tag:` is the target/to image). The from-version needs no description field of its own: it is already in the context name, one context per version.
|
||||
|
||||
**When it runs.** `e2e-tests-ci.yml`'s `check-changes` turns it on for a PR whose diff touches `e2e-tests/playwright/upgrade-specs/`, `e2e-tests/playwright/script/resolve_upgrade_matrix.mjs`, either `e2e-tests-playwright-rolling-upgrades*` workflow, or a server migration (`server/channels/db/migrations/`, `server/config/migrations/`, `server/channels/app/migrations.go`, `server/channels/app/permissions_migrations.go`, `server/public/model/version.go`). Everything else — including `e2e-tests/playwright/lib/` and `playwright.config.ts` — is unmatched and needs the manual dispatch's opt-in checkbox to run within this workflow. That decision is the only gate: unlike the full suite, this pipeline does not also consult `should_run`, because `should_run` matches `^e2e-tests/.*\.(ts|tsx|js|jsx)$` and would veto a run whose only change is the `.mjs` matrix resolver. Consequently there is no "skipped" status — when it is not requested, nothing is posted.
|
||||
|
||||
Failed rolling-upgrade contexts are included when applying **E2E Tests/verified** or the override-status workflow (discovered by pattern `e2e-test/playwright-full/{edition}/upgrade-from-*`, same principle as full-suite contexts).
|
||||
|
||||
This pipeline is invoked from `e2e-tests-playwright.yml` when `run_rolling_upgrades: "true"`. It is **not** embedded in `e2e-tests-playwright-template.yml`.
|
||||
|
||||
| Trigger | `run_rolling_upgrades` |
|
||||
|---------|------------------------|
|
||||
| PR (Argo / automated) | `false` (default) |
|
||||
| PR (`workflow_dispatch`, manual) | Opt-in: **Run rolling upgrades** on `e2e-tests-ci.yml` |
|
||||
| Merge to `master` / `release-*` | `false` — too expensive per merge |
|
||||
| Release cut | `true` |
|
||||
| Ad-hoc | n/a — **Run workflow** directly on `e2e-tests-playwright-rolling-upgrades.yml` |
|
||||
|
||||
`e2e-tests-playwright-rolling-upgrades.yml` carries its own `workflow_dispatch` so the coverage can be run from the Actions UI without a PR: `commit_sha` defaults to the selected ref's HEAD, `server_image_tag` to `master`, and the license / webhook secrets fall back to the repo secrets the calling workflows would have forwarded.
|
||||
|
||||
When `resolve_upgrade_matrix.mjs` returns `[]` (no supported from-versions), the entry workflow posts success on `e2e-test/playwright-full/{edition}/upgrade-from-none` and never invokes the template.
|
||||
|
||||
---
|
||||
|
||||
## Pipeline 1: PR (`e2e-tests-ci.yml`)
|
||||
|
||||
Runs E2E tests for every PR commit after the enterprise docker image is built. Fails if the commit is not associated with an open PR.
|
||||
|
||||
**Trigger chain:**
|
||||
```
|
||||
PR commit ─► Enterprise CI builds docker image
|
||||
─► Argo Events detects "Enterprise CI/docker-image" status
|
||||
─► dispatches e2e-tests-ci.yml
|
||||
```
|
||||
|
||||
For PRs from forks, `body.branches` may be empty so the workflow falls back to `master` for workflow files (trusted code), while `commit_sha` still points to the fork's commit.
|
||||
|
||||
**Jobs:** 2 (cypress + playwright), each does smoke -> full
|
||||
|
||||
**Commit statuses (4 total):**
|
||||
|
||||
| Context | Description (pending) | Description (result) |
|
||||
|---------|----------------------|---------------------|
|
||||
| `e2e-test/cypress-smoke\|enterprise` | `tests running, image_tag:abc1234` | `100% passed (1313), 440 specs, image_tag:abc1234` |
|
||||
| `e2e-test/cypress-full\|enterprise` | `tests running, image_tag:abc1234` | `100% passed (1313), 440 specs, image_tag:abc1234` |
|
||||
| `e2e-test/playwright-smoke\|enterprise` | `tests running, image_tag:abc1234` | `100% passed (200), 50 specs, image_tag:abc1234` |
|
||||
| `e2e-test/playwright-full\|enterprise` | `tests running, image_tag:abc1234` | `99.5% passed (199/200), 1 failed, 50 specs, image_tag:abc1234` |
|
||||
|
||||
**Manual trigger (CLI):**
|
||||
```bash
|
||||
gh workflow run e2e-tests-ci.yml \
|
||||
--repo mattermost/mattermost \
|
||||
--field pr_number="35171"
|
||||
```
|
||||
|
||||
**Manual trigger (GitHub UI):**
|
||||
1. Go to **Actions** > **E2E Tests (smoke-then-full)**
|
||||
2. Click **Run workflow**
|
||||
3. Fill in `pr_number` (e.g., `35171`)
|
||||
4. Click **Run workflow**
|
||||
|
||||
### On-demand testing
|
||||
|
||||
For on-demand E2E testing, the existing triggers still work:
|
||||
- **Comment triggers**: `/e2e-test`, `/e2e-test fips`, or with `MM_ENV` parameters
|
||||
- **Label trigger**: `E2E/Run`
|
||||
|
||||
These are separate from the automated workflow and can be used for custom test configurations or re-runs.
|
||||
|
||||
---
|
||||
|
||||
## Pipeline 2: Merge (`e2e-tests-on-merge.yml`)
|
||||
|
||||
Runs E2E tests after every push/merge to `master` or `release-*` branches.
|
||||
|
||||
**Trigger chain:**
|
||||
```
|
||||
Push to master/release-*
|
||||
─► Argo Events (mattermost-platform-package sensor)
|
||||
─► delivery-platform/.github/workflows/mattermost-platform-delivery.yaml
|
||||
─► builds docker images (enterprise + fips)
|
||||
─► trigger-e2e-tests job dispatches e2e-tests-on-merge.yml
|
||||
```
|
||||
|
||||
**Jobs:** 4 (cypress + playwright) x (enterprise + fips), smoke skipped, full tests only
|
||||
|
||||
**Commit statuses (4 total):**
|
||||
|
||||
| Context | Description example |
|
||||
|---------|-------------------|
|
||||
| `e2e-test/cypress-full\|enterprise` | `100% passed (1313), 440 specs, image_tag:abc1234_def5678` |
|
||||
| `e2e-test/cypress-full\|fips` | `100% passed (1313), 440 specs, image_tag:abc1234_def5678` |
|
||||
| `e2e-test/playwright-full\|enterprise` | `100% passed (200), 50 specs, image_tag:abc1234_def5678` |
|
||||
| `e2e-test/playwright-full\|fips` | `100% passed (200), 50 specs, image_tag:abc1234_def5678` |
|
||||
|
||||
**Manual trigger (CLI):**
|
||||
```bash
|
||||
# For master
|
||||
gh workflow run e2e-tests-on-merge.yml \
|
||||
--repo mattermost/mattermost \
|
||||
--field branch="master" \
|
||||
--field commit_sha="<full_commit_sha>" \
|
||||
--field server_image_tag="<image_tag>"
|
||||
|
||||
# For release branch
|
||||
gh workflow run e2e-tests-on-merge.yml \
|
||||
--repo mattermost/mattermost \
|
||||
--field branch="release-11.4" \
|
||||
--field commit_sha="<full_commit_sha>" \
|
||||
--field server_image_tag="<image_tag>"
|
||||
```
|
||||
|
||||
**Manual trigger (GitHub UI):**
|
||||
1. Go to **Actions** > **E2E Tests (master/release - merge)**
|
||||
2. Click **Run workflow**
|
||||
3. Fill in:
|
||||
- `branch`: `master` or `release-11.4`
|
||||
- `commit_sha`: full 40-char SHA
|
||||
- `server_image_tag`: e.g., `abc1234_def5678`
|
||||
4. Click **Run workflow**
|
||||
|
||||
---
|
||||
|
||||
## Pipeline 3: Release Cut (`e2e-tests-on-release.yml`)
|
||||
|
||||
Runs E2E tests after a release cut against the published release images.
|
||||
|
||||
**Trigger chain:**
|
||||
```
|
||||
Manual release cut
|
||||
─► delivery-platform/.github/workflows/release-mattermost-platform.yml
|
||||
─► builds and publishes release docker images
|
||||
─► trigger-e2e-tests job dispatches e2e-tests-on-release.yml
|
||||
```
|
||||
|
||||
**Jobs:** 4 (cypress + playwright) x (enterprise + fips), smoke skipped, full tests only. Team edition planned for future.
|
||||
|
||||
**Commit statuses (4 total, 6 when team is enabled):**
|
||||
|
||||
Descriptions include alias tags showing which rolling docker tags point to the same image.
|
||||
|
||||
RC example (11.4.0-rc3):
|
||||
|
||||
| Context | Description example |
|
||||
|---------|-------------------|
|
||||
| `e2e-test/cypress-full\|enterprise` | `100% passed (1313), 440 specs, image_tag:11.4.0-rc3 (release-11.4, release-11)` |
|
||||
| `e2e-test/cypress-full\|fips` | `100% passed (1313), 440 specs, image_tag:11.4.0-rc3 (release-11.4, release-11)` |
|
||||
| `e2e-test/cypress-full\|team` (future) | `100% passed (1313), 440 specs, image_tag:11.4.0-rc3 (release-11.4, release-11)` |
|
||||
|
||||
Stable example (11.4.0) — includes `MAJOR.MINOR` alias:
|
||||
|
||||
| Context | Description example |
|
||||
|---------|-------------------|
|
||||
| `e2e-test/cypress-full\|enterprise` | `100% passed (1313), 440 specs, image_tag:11.4.0 (release-11.4, release-11, 11.4)` |
|
||||
| `e2e-test/cypress-full\|fips` | `100% passed (1313), 440 specs, image_tag:11.4.0 (release-11.4, release-11, 11.4)` |
|
||||
| `e2e-test/cypress-full\|team` (future) | `100% passed (1313), 440 specs, image_tag:11.4.0 (release-11.4, release-11, 11.4)` |
|
||||
|
||||
**Manual trigger (CLI):**
|
||||
```bash
|
||||
gh workflow run e2e-tests-on-release.yml \
|
||||
--repo mattermost/mattermost \
|
||||
--field branch="release-11.4" \
|
||||
--field commit_sha="<full_commit_sha>" \
|
||||
--field server_image_tag="11.4.0" \
|
||||
--field server_image_aliases="release-11.4, release-11, 11.4"
|
||||
```
|
||||
|
||||
**Manual trigger (GitHub UI):**
|
||||
1. Go to **Actions** > **E2E Tests (release cut)**
|
||||
2. Click **Run workflow**
|
||||
3. Fill in:
|
||||
- `branch`: `release-11.4`
|
||||
- `commit_sha`: full 40-char SHA
|
||||
- `server_image_tag`: e.g., `11.4.0` or `11.4.0-rc3`
|
||||
- `server_image_aliases`: e.g., `release-11.4, release-11, 11.4` (optional)
|
||||
4. Click **Run workflow**
|
||||
|
||||
---
|
||||
|
||||
## Commit Status Format
|
||||
|
||||
**Context name:** `e2e-test/<phase>|<edition>`
|
||||
|
||||
Where `<phase>` is `cypress-smoke`, `cypress-full`, `playwright-smoke`, or `playwright-full`.
|
||||
|
||||
**Description format:**
|
||||
- All passed: `100% passed (<count>), <specs> specs, image_tag:<tag>[ (<aliases>)]`
|
||||
- With failures: `<rate>% passed (<passed>/<total>), <failed> failed, <specs> specs, image_tag:<tag>[ (<aliases>)]`
|
||||
- Pending: `tests running, image_tag:<tag>[ (<aliases>)]`
|
||||
- Rolling upgrades: identical shape — the from-version lives in the context
|
||||
(`…/upgrade-from-release-11.9`), one per resolved version, not in the description
|
||||
|
||||
- Pass rate: `100%` if all pass, otherwise one decimal (e.g., `99.5%`)
|
||||
- Aliases only present for release cuts
|
||||
- `image_tag:` is the target/to server image. Rolling upgrades put the from-version in the context name only, not in the description.
|
||||
|
||||
### Failure behavior
|
||||
|
||||
1. **Smoke test fails**: Full tests are skipped, only smoke commit status shows failure
|
||||
2. **Full test fails**: Full commit status shows failure with pass rate
|
||||
3. **Both pass**: Both smoke and full commit statuses show success
|
||||
4. **No PR found** (PR pipeline only): Workflow fails immediately
|
||||
|
||||
---
|
||||
|
||||
## Smoke-then-Full Pattern
|
||||
|
||||
Each wrapper (Cypress/Playwright) follows this flow:
|
||||
|
||||
```
|
||||
generate-build-variables (branch, build_id, server_image)
|
||||
─► smoke tests (1 worker, minimal docker services)
|
||||
─► if smoke passes ─► full tests (20 workers cypress / 1 worker playwright, all docker services)
|
||||
─► report (aggregate results, update commit status)
|
||||
```
|
||||
|
||||
### Test filtering
|
||||
|
||||
| Framework | Smoke | Full |
|
||||
|-----------|-------|------|
|
||||
| **Cypress** | `--stage=@prod --group=@smoke` | `--stage="@prod" --excludeGroup="@te_only,@cloud_only,@high_availability" --sortFirst=... --sortLast=...` |
|
||||
| **Playwright** | `--grep @smoke` | `--grep-invert "@smoke\|@visual"` |
|
||||
|
||||
### Worker configuration
|
||||
|
||||
| Framework | Smoke Workers | Full Workers |
|
||||
|-----------|---------------|--------------|
|
||||
| **Cypress** | 1 | 20 |
|
||||
| **Playwright** | 1 | 1 (uses internal parallelism via `PW_WORKERS`) |
|
||||
|
||||
### Docker services
|
||||
|
||||
| Test Phase | Docker Services |
|
||||
|------------|-----------------|
|
||||
| Smoke | `postgres inbucket` |
|
||||
| Full | `postgres inbucket minio openldap elasticsearch keycloak` |
|
||||
|
||||
---
|
||||
|
||||
## Tagging Smoke Tests
|
||||
|
||||
### Cypress
|
||||
|
||||
Add `@smoke` to the Group comment at the top of spec files:
|
||||
|
||||
```javascript
|
||||
// Stage: @prod
|
||||
// Group: @channels @messaging @smoke
|
||||
```
|
||||
|
||||
### Playwright
|
||||
|
||||
Add `@smoke` to the test tag option:
|
||||
|
||||
```typescript
|
||||
test('critical login flow', {tag: ['@smoke', '@login']}, async ({pw}) => {
|
||||
// ...
|
||||
});
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Shared Wrapper Inputs
|
||||
|
||||
The wrappers (`e2e-tests-cypress.yml`, `e2e-tests-playwright.yml`) accept these inputs:
|
||||
|
||||
| Input | Default | Description |
|
||||
|-------|---------|-------------|
|
||||
| `server_edition` | `enterprise` | Edition: `enterprise`, `fips`, or `team` |
|
||||
| `server_image_repo` | `mattermostdevelopment` | Docker namespace: `mattermostdevelopment` or `mattermost` |
|
||||
| `server_image_tag` | derived from `commit_sha` | Docker image tag |
|
||||
| `server_image_aliases` | _(empty)_ | Alias tags shown in commit status description |
|
||||
| `ref_branch` | _(empty)_ | Source branch name for webhook messages (e.g., `master` or `release-11.4`) |
|
||||
|
||||
The automation dashboard branch name is derived from context:
|
||||
- PR: `pr-<pr_number>` (e.g., `pr-35205`)
|
||||
- Master merge: `master`
|
||||
- Release merge: `release-<version>` (e.g., `release-11.7`)
|
||||
- Fallback: `commit-<short_sha>` (e.g., `commit-abc1234`)
|
||||
|
||||
Master and release runs use the real branch verbatim so the
|
||||
`/reports/{repo}/{branch}` view aggregates every build on that branch
|
||||
into one history. PR runs use a `pr-<n>` token because there's no real
|
||||
branch to use; the commit-only fallback uses a `commit-<short_sha>`
|
||||
token (defensive — no current caller hits this path).
|
||||
|
||||
The test type suffix (`-smoke` or `-full`) is appended by the template.
|
||||
|
||||
The server image is derived as:
|
||||
```
|
||||
{server_image_repo}/{edition_image_name}:{server_image_tag}
|
||||
```
|
||||
|
||||
Where `edition_image_name` maps to:
|
||||
- `enterprise` -> `mattermost-enterprise-edition`
|
||||
- `fips` -> `mattermost-enterprise-fips-edition`
|
||||
- `team` -> `mattermost-team-edition`
|
||||
|
||||
---
|
||||
|
||||
## Webhook Message Format
|
||||
|
||||
After full tests complete, a webhook notification is sent to the configured `REPORT_WEBHOOK_URL`. The results line uses the same `commit_status_message` as the GitHub commit status. The source line varies by pipeline using `report_type` and `ref_branch`.
|
||||
|
||||
**Report types:** `PR`, `MASTER`, `RELEASE`, `RELEASE_CUT`
|
||||
|
||||
### PR
|
||||
|
||||
```
|
||||
:open-pull-request: mattermost-pr-35205
|
||||
:docker: mattermostdevelopment/mattermost-enterprise-edition:abc1234
|
||||
100% passed (1313), 440 specs | full report
|
||||
```
|
||||
|
||||
### Merge to master
|
||||
|
||||
```
|
||||
:git_merge: abc1234 on master
|
||||
:docker: mattermostdevelopment/mattermost-enterprise-edition:abc1234_def5678
|
||||
100% passed (1313), 440 specs | full report
|
||||
```
|
||||
|
||||
### Merge to release branch
|
||||
|
||||
```
|
||||
:git_merge: abc1234 on release-11.4
|
||||
:docker: mattermostdevelopment/mattermost-enterprise-edition:abc1234_def5678
|
||||
100% passed (1313), 440 specs | full report
|
||||
```
|
||||
|
||||
### Release cut
|
||||
|
||||
```
|
||||
:github_round: abc1234 on release-11.4
|
||||
:docker: mattermost/mattermost-enterprise-edition:11.4.0-rc3
|
||||
100% passed (1313), 440 specs | full report
|
||||
```
|
||||
|
||||
The commit short SHA links to the commit on GitHub. The PR number links to the pull request.
|
||||
|
||||
---
|
||||
|
||||
## Related Files
|
||||
|
||||
- `e2e-tests/cypress/` - Cypress test suite
|
||||
- `e2e-tests/playwright/` - Playwright test suite
|
||||
- `e2e-tests/.ci/` - CI configuration and environment files
|
||||
- `e2e-tests/Makefile` - Makefile with targets for running tests, generating cycles, and reporting
|
||||
Reference in New Issue
Block a user