upstream-mattermost/e2e-tests/playwright

Local development

1. Start local server in a separate terminal.

There are two ways to run the local server:

Option 1: Run from source

# Typically run the local server with:
cd server && make run

# Or run webapp and server on separate terminals for better performance
# First terminal: Build and run the webapp
cd webapp && make run
# Second terminal: Run the server
cd server && make run-server

Option 2: Testcontainers (recommended for testing, and what CI uses)

No separate terminal or setup step needed — Playwright brings up Postgres, Inbucket, and the Mattermost server itself via Testcontainers, then tears them down after the run.

# Run with defaults (Postgres, Inbucket, Mattermost server, minio, openldap, keycloak, elasticsearch)
PW_USE_TESTCONTAINERS=true npm run test -- login

# Change which additional services start, comma-separated (or "" to start none)
PW_USE_TESTCONTAINERS=true PW_TESTCONTAINERS_SERVICES=minio,openldap npm run test

# Pin a specific server image (defaults to mattermostdevelopment/mattermost-enterprise-edition:master)
PW_USE_TESTCONTAINERS=true SERVER_IMAGE=mattermostdevelopment/mattermost-enterprise-edition:<tag> npm run test

# Pass arbitrary MM_* config overrides as comma-separated KEY=VALUE pairs
PW_USE_TESTCONTAINERS=true MM_ENV=MM_LICENSE='<your-license-key>' npm run test

Containers are reused across invocations by default (PW_TESTCONTAINERS_REUSE=true) instead of being recreated every run — tear the stack down explicitly when you're done with npm run testcontainers:down. Set PW_TESTCONTAINERS_REUSE=false for a one-off run that tears itself down when it finishes. Use npm run testcontainers:up to just bring the stack up (or confirm an existing one's still reachable) without running any tests.

See lib/README.md for every available environment variable.

2. Install dependencies and run the test.

# Install npm packages
npm i

# Install browser binaries as prompted if Playwright is just installed or updated
# See https://playwright.dev/docs/browsers
npx playwright install

# Run a specific test of all projects -- Chrome, Firefox, iPhone and iPad.
# See https://playwright.dev/docs/test-cli.
npm run test -- login

# Run a specific test of a project
npm run test -- login --project=chrome

# Run all tests (including visual tests)
npm run test

# Run CI tests (excludes visual tests, runs only in Chrome)
# Note: visual tests run in a separate workflow
npm run test:ci

3. Inspect test results at /results/output folder when something fails unexpectedly.

Run tests in UI mode

Check out https://playwright.dev/docs/test-ui-mode for detailed guide on UI Mode to learn more about its features.

npm run playwright-ui

Note: If no tests appear in the UI, check your filter settings:

• Test name filters
• Project filters (setup, ipad, chrome, firefox)
• Tag filters (@tag)
• Execution status filters

The "setup" project runs the initial configuration tests in specs/test_setup.ts (ensuring plugins are loaded and server deployment is correct). These setup tests are typically run only once before other tests and may be unchecked for subsequent runs, though they can remain checked if needed.

Upgrade-path testing

Boots an older server image, seeds it, then replaces only the Mattermost container with a newer image while the same Postgres keeps running, and re-verifies through the Client4 API that migrations completed and prior data survived. Testcontainers mode only, since that in-place container swap is the whole mechanism.

The upgrade projects make no UI assertions: older images ship older webapps, so master POMs and testids cannot be trusted against release-X.Y. UI coverage comes from the normal suite, which CI runs afterwards against the upgraded server.

How it works

Three projects in playwright.config.ts, each selecting its specs by directory (upgrade-specs/from, upgrade-specs/to). The @upgrade-from / @upgrade-to tags are bookkeeping for reports and ad-hoc --grep; no project filters on them.

Project	Does
upgrade-from	Boots fresh on PW_UPGRADE_FROM_SERVER_IMAGE, runs setup, seeds content via Client4, writes .upgrade_baseline.json
upgrade-swap-to	Restarts the container on SERVER_IMAGE with the same boot env as from, including host MM_LICENSE
upgrade-to	Re-verifies every baseline slice against the upgraded server

• The swap. pw.upgradeServerImage(image) (lib/src/server/version.ts) repoints testConfig.serverImage and calls the same restartMattermostContainer() used by pw.ensureMinio() — stop the Mattermost container, start a fresh one on the same network. Postgres is never touched.
• State that must survive lives outside the container. Config is stored in Postgres (MM_CONFIG set to the SQL datasource in lib/src/containers/mattermost_container.ts, a supported production mode) and /mattermost/data is bind-mounted to .mattermost_data/ (lib/src/containers/constants.ts). Both are Docker volumes by default, so the swap would otherwise factory-reset the config and discard uploads.
• API-first harness. upgrade-specs/{from,to}/*.spec.ts use Client4 / PlaywrightClient4 — no POMs. Authenticated binary downloads use Playwright's request fixture; browser login only via loginByAPI when a real session is needed. Helpers in upgrade-specs/upgrade_fixtures.ts.
• Continuity across phases. Fixed (non-random) team/user/channel names are looked up idempotently, plus .upgrade_baseline.json for version identity, license, plugin state, and post/channel IDs. The from-phase container uses Testcontainers withReuse() so Ryuk does not reap it when that process exits.
• Plugins are driven by API, not env. upgrade-from enables playbooks and installs the demo plugin, upgrade-to asserts both survived, and global setup's plugin reset is skipped for those phases. The first normal spec afterwards deactivates the demo plugin, which otherwise overrides the webapp's attachment button and breaks every UI file-upload spec.
• Phase logs. pw.saveUpgradePhaseLogs('from'|'to') writes Mattermost/Postgres docker logs to logs/upgrade/, plus a migration/license/panic highlights file. The from-image has to be captured before swap-to replaces the container. CI uploads logs/ via ci/upload-debug-artifacts.

Run locally

The stack stays up between the two commands so you can inspect it. test:upgrade:from requires a clean slate — tear down any leftover stack first. upgrade-to adopts that stack, which needs testcontainers.reuse.enable=true in ~/.testcontainers.properties (see lib/README.md).

npm run testcontainers:down

# Phase 1 — boots fresh on the older from-image. Use a release-* tag; patch tags like 11.9.1 are
# not published. `node script/resolve_upgrade_matrix.mjs` prints the tags CI uses.
# MM_LICENSE is optional; when set, the baseline records license details.
MM_LICENSE='<your-license-key>' \
  PW_UPGRADE_FROM_SERVER_IMAGE=mattermostdevelopment/mattermost-enterprise-edition:release-11.9 \
  npm run test:upgrade:from

# Phase 2 — swaps to SERVER_IMAGE (defaults to :master) with the same env as from.
MM_LICENSE='<your-license-key>' \
  SERVER_IMAGE=mattermostdevelopment/mattermost-enterprise-edition:master \
  npm run test:upgrade:to

npm run testcontainers:down

Both env vars can live in a local .env file instead of the command line.

CI

CI tests a rolling matrix rather than one fixed version: the last 3 minor releases plus any release still in its Extended Support (ESR) window. script/resolve_upgrade_matrix.mjs resolves it at run time — last-3-minors from this checkout's server/public/model/version.go, support-end dates and ESR status fetched live from the published releases table on master, since those lapse with calendar time and a stale branch should not be trusted for them.

node script/resolve_upgrade_matrix.mjs
# [{"dockerTag":"release-10.11","minor":"10.11","patch":"10.11.22","isESR":true,"contextLabel":"release-10.11-esr"}, ...]

Rolling upgrades run in a separate pipeline from the normal full suite, not inside e2e-tests-playwright-template.yml:

• e2e-tests-playwright-rolling-upgrades.yml resolves the matrix and calls the template once per from-version, in parallel.
• e2e-tests-playwright-rolling-upgrades-template.yml tests one from-version, and owns that version's Test System IO run, dispatch workers, and commit status.

Each of a version's workers (default 20, same as the full suite) runs --project=upgrade-from, then --project=upgrade-to, then dispatch-run for its share of the normal suite. Nothing is re-prepared in between — no setup, no re-patching, no restart — so the suite exercises a server that reached the to-image by upgrading. An upgrade that needed the suite's setup re-run to be usable would not be a passing upgrade, which is why that step is absent. The upgrade specs themselves live in upgrade-specs/, outside testDir, so dispatch-begin never sees them.

Every worker upgrades its own server, since a server cannot be shared across runners: the harness runs workers times per from-version and the pipeline costs matrix size × workers runners, each on a 60m timeout. dispatch-run is skipped when the harness fails. No testcontainers:down first — the runner is fresh, so the clean-slate guard has nothing to trip over.

When it runs

Pipeline	Rolling upgrades
PR (automated)	On when the diff touches upgrade-specs/, script/resolve_upgrade_matrix.mjs, a rolling-upgrade workflow, or a server migration (server/channels/db/migrations/, server/config/migrations/, server/channels/app/migrations.go, server/channels/app/permissions_migrations.go, server/public/model/version.go)
PR (manual workflow_dispatch)	Opt-in via Run rolling upgrades checkbox
Merge to master / release-*	Off — too expensive per merge
Release cut	On automatically
Ad-hoc	Run workflow on E2E Tests - Playwright Rolling Upgrades; no PR needed — pick a ref, to-image tag, edition, and worker count

Commit statuses — one per matrix entry and nothing else, so a 4-entry matrix produces exactly 4 contexts. When the resolver returns [], no matrix jobs run and the workflow posts a single upgrade-from-none context instead. All sit under e2e-test/playwright-full/{edition}/upgrade-from-*, which is how the verified-label and override-status workflows discover them:

Context	Covers
upgrade-from-release-11.9	that from-version's harness and post-upgrade suite (ESR entries append -esr)
upgrade-from-none	resolver returned an empty matrix; nothing ran

There is no "skipped" context: when rolling upgrades are not requested the job never starts and posts nothing. The pipeline is deliberately not gated on the generic should_run — its own triggers already decide, and the .mjs resolver falls outside should_run's ^e2e-tests/.*\.(ts|tsx|js|jsx)$ pattern, so deferring to it would drop requested runs. The normal full suite keeps using e2e-test/playwright-full/enterprise, with no /upgrade-from-… suffix.

Visual Testing

All visual tests must be placed in the specs/visual/ directory and tagged with @visual in the test tags array. This organization ensures proper test discovery and execution patterns.

Visual tests are used to verify the UI appearance is consistent across browsers and remains stable across code changes. There are two types of visual tests supported:

1. Built-in snapshot testing: Uses Playwright's built-in snapshot comparison
2. Percy integration: Uses the Percy service for more advanced visual testing and reporting

CI Pipeline for Visual Tests

In CI environments, visual tests run in a separate dedicated pipeline:

• Regular tests run with npm run test:ci which excludes all tests with the @visual tag
• Visual tests run in a separate workflow using the Playwright Docker container
• This separation prevents visual tests from slowing down the main test pipeline
• It also ensures visual tests always run in a consistent environment

Writing Visual Tests

When creating visual tests:

1. Follow the test documentation format like other tests:

   • Include JSDoc with @objective tag
   • Use action-oriented test title
   • Add proper comment prefixes (// # for actions, // * for verifications)

2. Place in the correct location:

   • Put visual tests in the specs/visual/ directory, organized by feature area
   • Example: specs/visual/channels/intro_channel.spec.ts

3. Add required tags:

   • Always include @visual tag
   • Add feature-specific tags as needed (e.g., @login_page, @channel_page)

4. Manage dynamic content:

   • Use pw.hideDynamicChannelsContent() to hide elements that could change between runs
   • Take snapshots only after UI is fully loaded and stable

Example:

/**
 * @objective Capture visual snapshot of the landing/login page
 */
test(
    'displays landing page with login options',
    {tag: ['@visual', '@landing_page']},
    async ({pw, page, browserName, viewport}, testInfo) => {
        // # Go to landing login page
        await pw.landingLoginPage.goto();
        await pw.landingLoginPage.toBeVisible();

        // * Verify landing page appears as expected
        await pw.matchSnapshot(testInfo, {page, browserName, viewport});
    },
);

Updating screenshots is done strictly via Playwright's docker container for consistency

1. Run Playwright's docker container

Change to the ./ project directory, then run the docker container. (See https://playwright.dev/docs/docker for reference.)

docker run -it --rm -v "$(pwd):/mattermost/" --ipc=host mcr.microsoft.com/playwright:v1.62.0-noble /bin/bash

2. Inside the docker container

export PW_BASE_URL=http://host.docker.internal:8065
export PW_HEADLESS=true
cd mattermost/e2e-tests/playwright

# Install npm packages. Use "npm ci" to match the automated environment
export PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD=1 npm ci

# Run specific test. See https://playwright.dev/docs/test-cli.
npm run test -- login --project=chrome

# Or run all tests
npm run test

# Run visual tests (must be run inside Docker for consistency)
npm run test -- specs/visual

# Update snapshots of visual tests (must be run inside Docker)
npm run test -- specs/visual --update-snapshots

# Run Percy visual tests (requires PERCY_TOKEN environment variable)
export PERCY_TOKEN=<your-percy-token>
npm run percy:docker

Accessibility Testing

Accessibility tests ensure Mattermost meets WCAG 2.1 AA compliance standards. Tests are located in specs/accessibility/ and cover keyboard navigation, screen reader support, focus management, and automated accessibility scanning.

For comprehensive guidelines on writing accessibility tests, aria snapshots, and folder structure, see docs/accessibility/.

Accessibility Locators

Playwright's accessibility locators should be the preferred approach for all tests, not just accessibility tests. These locators query elements based on how users and assistive technologies perceive them, making tests more resilient to implementation changes and ensuring better accessibility by design.

Why Use Accessibility Locators?

• Resilient to changes: Tests won't break when CSS classes or data-testid attributes change
• Encourages accessibility: Forces proper ARIA roles, labels, and semantic HTML
• Better readability: page.getByRole('button', {name: 'Save'}) is clearer than page.locator('[data-testid="save-btn"]')
• Aligns with user experience: Tests what users actually perceive, not implementation details

Preferred Locators (in order of preference)

1. Role-based: page.getByRole('button', {name: 'Save'}), page.getByRole('textbox', {name: 'Email'})
2. Label-based: page.getByLabel('Email address')
3. Text-based: page.getByText('Welcome'), page.getByPlaceholder('Enter email')
4. Test IDs: page.locator('[data-testid="..."]') - Use only when accessibility locators aren't possible
5. CSS selectors: page.locator('.class') - Avoid unless absolutely necessary

When Test IDs Are Acceptable

Use data-testid only when:

• Element has no semantic role (e.g., decorative divs)
• Multiple identical elements need distinction
• Component is not interactive or visible to assistive tech

For all test examples, see docs/accessibility/ for comprehensive patterns and best practices.

Page/Component Object Model

See https://playwright.dev/docs/test-pom.

Page and component abstractions are in shared library located at ./lib/src/ui. They should be established before writing a spec file so that any future changes in the DOM structure will be made in one place only. No static UI text or fixed locator should be written in the spec file.