snapshot of https://github.com/mattermost/mattermost.git (master, 2026-09-16; blobs > 131072B omitted)

docs/main/administration-guide/scale/collect-performance-metrics.mdx

@@ -0,0 +1,69 @@
+---
+title: "Collect performance metrics"
+---
+import useBaseUrl from '@docusaurus/useBaseUrl';
+
+<PlanAvailability slug="entry-ent" />
+
+System admins can collect and store the [same performance monitoring metrics](/administration-guide/scale/performance-monitoring-metrics) as Prometheus, without having to deploy these third-party tools. Data is collected every minute and is stored in a path you configure. The data is synchronized to either a cloud-based or local file store every hour, and retained for 15 days.
+
+Download and share the collected data with Mattermost to understand application performance, troubleshoot system stability and performance, as well as inform route cause analysis.
+
+<Tip>
+
+Already have Prometheus and Grafana deployed? You can [use these tools to monitor performance of your Mattermost deployment](/administration-guide/scale/deploy-prometheus-grafana-for-performance-monitoring).
+
+</Tip>
+
+## Mattermost configuration
+
+<Note>
+
+For Mattermost Cloud deployments, no setup is required. See the [usage](#usage) section below for details on collecting performance metrics.
+
+</Note>
+
+For a self-hosted Mattermost deployment, a Mattermost system admin must perform the following steps in Mattermost.
+
+1.  Log in to your Mattermost [workspace](/end-user-guide/end-user-guide-index) as a system admin.
+2.  From Mattermost v10.1, you can install the Metrics plugin from the in-product Mattermost Marketplace by selecting the **Product** <img src={useBaseUrl('/img/ui/products_E82F.svg')} alt="Navigate between Channels, collaborative playbooks, and boards using the product menu icon." className="theme-icon" /> icon and selecting **App Marketplace**. Search for **Metrics** and select **Install**.
+3.  Go to **System Console \> Plugins \> Plugin Management**. In the **Installed Plugins** section, scroll to **Mattermost Metrics Plugin**, and select **Enable Plugin**.
+4.  Specify the path of the time-series database, and select **Save**.
+5.  Go to **System Console \> Environment \> Performance Monitoring**, and set **Enable Performance Monitoring** to **true**. Select **Save**.
+6.  Go to **System Console \> Environment \> Performance Monitoring**, and set **Enable Client Performance Monitoring** to **true**. This setting is required for the system administrator's performance monitoring product experience. Select **Save**.
+
+<Note>
+
+For Mattermost deployments prior to v10.1, you must download the latest version of [the plugin binary release](https://github.com/mattermost/mattermost-plugin-metrics/releases), compatible with Mattermost v8.0.1 and later. Go to **System Console \> Plugins \> Plugin Management \> Upload Plugin**, and upload the plugin binary you downloaded.
+
+</Note>
+
+## Upgrade
+
+We recommend upgrading this feature as new versions are released. Generally, updates are seamless and don't interrupt the user experience in Mattermost. Visit the [Releases page](https://github.com/mattermost/mattermost-plugin-metrics/releases) for the latest release, available releases, and compatibiilty considerations.
+
+## Usage
+
+You need to be a Mattermost system admin to collect performance metrics. Select **Create Dump** to generate dump files.
+
+To use the generated dump file, you can simply clone the [Dockprom](https://github.com/stefanprodan/dockprom) repository. Change the Prometheus data volume to point to the dump that you just downloaded. The downloaded file is compressed, so to be able to use it, you need to decompress it first.
+
+The volume configuration for Prometheus should look like the code below in the `docker-compose.yml` file:
+
+``` yaml
+volumes:
+  - ./prometheus:/etc/prometheus
+  - /Path/To/Dump/Directory:/prometheus/data
+```
+
+Once you set this up, run `docker-compose` as described in [Dockprom Repository](https://github.com/stefanprodan/dockprom?tab=readme-ov-file#install).
+
+You can also use our [Mattermost Performance Monitoring v2](https://grafana.com/grafana/dashboards/15582-mattermost-performance-monitoring-v2/) dashboard by simply importing it into Grafana.
+
+1.  Open Grafana (`<localhost>:3000` by default) and then log into it.
+2.  Once you log in, go to the **Plus** <img src={useBaseUrl('/img/ui/plus_F0415.svg')} alt="The Plus icon provides access to channel and direct message functionality." className="theme-icon" /> icon on the left sidebar, and then select **Import**.
+3.  Enter the dashboard ID (`15582`) in the **Grafana.com Dashboard** field, and then select **Load** to fetch the dashboard.
+
+## What's collected?
+
+Mattermost provides [custom metrics](/administration-guide/scale/performance-monitoring-metrics#custom-mattermost-metrics) and [standard Go metrics](/administration-guide/scale/performance-monitoring-metrics#standard-go-metrics) that can be used to monitor your system's performance. Additionally Enterprise customers can use the Metrics plugin to collect [host/system metrics](/administration-guide/scale/performance-monitoring-metrics#hostsystem-metrics) from [node exporter](https://github.com/prometheus/node_exporter) targets to monitor network-related panels for Mattermost Calls.

docs/main/administration-guide/scale/deploy-grafana-loki-for-centralized-logging.mdx

@@ -0,0 +1,445 @@
+---
+title: "Deploy Grafana Loki for centralized logging"
+---
+<PlanAvailability slug="entry-ent" />
+
+This guide extends your existing [Prometheus and Grafana performance monitoring deployment](/administration-guide/scale/deploy-prometheus-grafana-for-performance-monitoring) by adding [Grafana Loki](https://grafana.com/oss/loki/) for centralized log aggregation. While Prometheus collects **metrics** (CPU, request latency, goroutine counts), Loki collects **logs** — giving your operations team the ability to search, filter, and correlate Mattermost application logs across all servers from a single Grafana interface.
+
+With Loki in place you can:
+
+- Search Mattermost logs across all application servers from Grafana.
+- Filter by log level, HTTP status code, or free-text search.
+- Identify the top recurring errors and warnings.
+- Correlate log events with Prometheus metric spikes on the same timeline.
+- Optionally collect PostgreSQL database logs alongside application logs.
+
+<Tip>
+
+This guide assumes you have already deployed Prometheus and Grafana by following the [performance monitoring guide](/administration-guide/scale/deploy-prometheus-grafana-for-performance-monitoring). Loki and the OpenTelemetry Collector will be added to that existing infrastructure.
+
+</Tip>
+
+## Architecture overview
+
+The deployment adds two components to your monitoring stack:
+
+- **Loki** — the log aggregation engine, installed on your existing Grafana/Prometheus monitoring server.
+- **OpenTelemetry Collector** — a versatile, industry-standard agent installed on each Mattermost application server to ship logs.
+
+``` text
+┌──────────────────────┐       ┌──────────────────────┐
+│  Mattermost App 01   │       │  Mattermost App 02   │
+│                      │       │                      │
+│  /opt/mattermost/    │       │  /opt/mattermost/    │
+│   logs/mattermost.log│       │   logs/mattermost.log│
+│         │            │       │         │            │
+│    [ OTel Col ]      │       │    [ OTel Col ]      │
+└─────────┬────────────┘       └─────────┬────────────┘
+          │ push (:3100/otlp)            │
+          ▼                              ▼
+┌─────────────────────────────────────────────────────┐
+│           Monitoring Server                         │
+│                                                     │
+│   [ Prometheus :9090 ]    [ Loki :3100 ]            │
+│           │                     │                   │
+│           └──────┬──────────────┘                   │
+│                  ▼                                  │
+│           [ Grafana :3000 ]                         │
+│            Metrics + Logs                           │
+└─────────────────────────────────────────────────────┘
+
+Optional: Add the OpenTelemetry Collector on your PostgreSQL server to also
+ship database logs to Loki.
+```
+
+## Prerequisites
+
+Before starting, confirm the following:
+
+- Prometheus and Grafana are deployed and collecting Mattermost metrics per the [performance monitoring guide](/administration-guide/scale/deploy-prometheus-grafana-for-performance-monitoring).
+- You have **SSH access** with sudo privileges to the monitoring server and each Mattermost application server.
+- Mattermost is installed at `/opt/mattermost` (the default location). If your installation path differs, substitute your path wherever `/opt/mattermost` appears.
+- **Network connectivity**: Each Mattermost server can reach the monitoring server on **TCP port 3100** (Loki's HTTP API).
+
+## Step 1: Verify Mattermost JSON logging
+
+Mattermost must be writing JSON-formatted file logs for Loki queries to work. This is the default on all plans — verify it on each application server:
+
+``` bash
+tail -1 /opt/mattermost/logs/mattermost.log
+```
+
+A JSON-formatted line looks like:
+
+``` json
+{"timestamp":"2025-01-15T14:32:01.123Z","level":"info","msg":"Server is listening on [::]:8065","caller":"app/server.go:482"}
+```
+
+If you see plain-text output instead, go to **System Console \> Environment \> Logging** and set **Output file logs as JSON** to `true`, then restart Mattermost:
+
+``` bash
+sudo systemctl restart mattermost
+```
+
+## Step 2: Install Loki on the monitoring server
+
+Install Loki on the same server that runs Grafana and Prometheus. All commands in this section are run **on the monitoring server**.
+
+<Important>
+
+The commands below use Loki version **3.4.2**. Check the [Loki releases page](https://github.com/grafana/loki/releases) for the latest version and substitute the version number as needed.
+
+</Important>
+
+1.  Create the Loki user, directories, and download the binary:
+
+    ``` bash
+    # Create a dedicated system user
+    sudo useradd --system --no-create-home --shell /bin/false loki
+
+    # Create directories
+    sudo mkdir -p /opt/loki/data /opt/loki/bin
+
+    # Download and extract Loki
+    sudo apt-get install -y unzip
+    cd /tmp
+    curl -LO https://github.com/grafana/loki/releases/download/v3.4.2/loki-linux-amd64.zip
+    unzip loki-linux-amd64.zip
+    sudo mv loki-linux-amd64 /opt/loki/bin/loki
+    sudo chmod +x /opt/loki/bin/loki
+    ```
+
+2.  Download and install the production configuration file:
+
+    - [Download loki-config.yaml](/files/samples/loki/loki-config.yaml)
+
+    ``` bash
+    sudo cp loki-config.yaml /opt/loki/loki-config.yaml
+    ```
+
+    <div class="tip">
+
+    <div class="title">
+
+    Tip
+
+    </div>
+
+    **Log retention** is set to **14 days** by default. To change this, edit `/opt/loki/loki-config.yaml` and update the `retention_period` value under `limits_config`. Loki requires this value in hours — common values:
+
+    - `336h` = 14 days (default)
+    - `720h` = 30 days
+    - `2160h` = 90 days
+    - `8760h` = 365 days
+
+    Longer retention increases disk usage. As a rough guide, expect 1–3 GB per day for a moderately active Mattermost deployment (varies with log volume). Monitor `/opt/loki/data/` after the first week to project storage needs. The `compactor.retention_enabled` setting must remain `true` for retention enforcement to work.
+
+    </div>
+
+3.  Set ownership:
+
+    ``` bash
+    sudo chown -R loki:loki /opt/loki
+    ```
+
+4.  Create a systemd service file:
+
+    ``` bash
+    sudo tee /etc/systemd/system/loki.service > /dev/null <<'EOF'
+    [Unit]
+    Description=Grafana Loki Log Aggregation
+    Documentation=https://grafana.com/docs/loki/latest/
+    After=network-online.target
+    Wants=network-online.target
+
+    [Service]
+    Type=simple
+    User=loki
+    Group=loki
+    ExecStart=/opt/loki/bin/loki -config.file=/opt/loki/loki-config.yaml
+    Restart=on-failure
+    RestartSec=5
+    LimitNOFILE=65536
+    StandardOutput=journal
+    StandardError=journal
+
+    [Install]
+    WantedBy=multi-user.target
+    EOF
+    ```
+
+5.  Start Loki:
+
+    ``` bash
+    sudo systemctl daemon-reload
+    sudo systemctl enable --now loki
+    ```
+
+6.  Verify Loki is running:
+
+    ``` bash
+    sudo systemctl status loki
+    curl -s http://localhost:3100/ready
+    ```
+
+    The `/ready` endpoint should return `ready`. Loki may take 15–20 seconds after startup before it reports ready — this warmup delay is normal.
+
+## Step 3: Install OpenTelemetry Collector on each Mattermost server
+
+The OpenTelemetry (OTel) Collector runs on each Mattermost application server and pushes logs to Loki. Repeat these steps on **every Mattermost application server**.
+
+<Important>
+
+Replace `<LOKI_HOST>` with the IP address or hostname of your monitoring server, `<HOSTNAME>` with this server's hostname (e.g., `mm-app-01`), and `<SERVICE_NAME>` with the service type (e.g., `mattermost` or `postgres`).
+
+</Important>
+
+1.  Install the OTel Collector Contrib distribution:
+
+    ``` bash
+    # Add OTel Collector repository and install
+    # Commands below are for Ubuntu/Debian
+    wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.145.0/otelcol-contrib_0.145.0_linux_amd64.deb
+    sudo dpkg -i otelcol-contrib_0.145.0_linux_amd64.deb
+    ```
+
+    <div class="note">
+
+    <div class="title">
+
+    Note
+
+    </div>
+
+    The installation process automatically creates a system user and group named `otelcol-contrib`.
+
+    </div>
+
+2.  Download and edit the OpenTelemetry Collector configuration:
+
+    - [Download otel-collector-config-mattermost.yaml](/files/samples/loki/otel-collector-config-mattermost.yaml)
+
+    ``` bash
+    sudo cp otel-collector-config-mattermost.yaml /etc/otelcol-contrib/config.yaml
+    ```
+
+    Edit `/etc/otelcol-contrib/config.yaml` and replace the placeholders:
+
+    - Replace `<LOKI_HOST>` with your monitoring server's address.
+    - Replace `<HOSTNAME>` with this server's hostname.
+    - Replace `<SERVICE_NAME>` with the service type (e.g., `mattermost`).
+
+    ``` bash
+    sudo vi /etc/otelcol-contrib/config.yaml
+    ```
+
+3.  Grant the collector read access to the Mattermost log directory:
+
+    ``` bash
+    # Grant the collector access to read Mattermost logs.
+    # Substitute 'mattermost' for the group that owns your log file.
+    sudo usermod -aG mattermost otelcol-contrib
+
+    # Ensure logs are group-readable
+    sudo chmod 640 /opt/mattermost/logs/mattermost.log
+    sudo chmod g+rx /opt/mattermost/logs
+    ```
+
+4.  Restart the service:
+
+    ``` bash
+    sudo systemctl restart otelcol-contrib
+    sudo systemctl enable otelcol-contrib
+    ```
+
+5.  Verify the collector is running and shipping logs:
+
+    ``` bash
+    sudo systemctl status otelcol-contrib
+    sudo journalctl -u otelcol-contrib -f
+    ```
+
+## Step 4: Add Loki as a data source in Grafana
+
+1.  Log in to Grafana (default: `http://<monitoring-server>:3000`).
+2.  Navigate to **Administration \> Data Sources \> Add data source**.
+3.  Select **Loki** from the list.
+4.  Configure the connection:
+    - **Name**: `Loki`
+    - **URL**: `http://localhost:3100` (since Loki is colocated with Grafana)
+5.  Select **Save & test**. Grafana should confirm the data source is working.
+
+## Step 5: Import the Mattermost Loki dashboard
+
+A pre-built Grafana dashboard is recommended for monitoring your logs. This dashboard is provided as a basic example:
+
+- [Download the Mattermost Loki Logs dashboard JSON](/files/samples/grafana-dashboards/mattermost-loki-logs.json)
+
+<Note>
+
+While the dashboard provides a useful starting point, users can gain full query flexibility using the **Explore** tab in Grafana to build custom LogQL queries on the fly.
+
+</Note>
+
+The dashboard provides:
+
+- **Log Volume by Level** — stacked bar chart showing log rates by severity.
+- **Error / Warning / Total counters** — stat panels for the selected time range.
+- **HTTP 4xx/5xx count** — quick visibility into application-level HTTP errors.
+- **Error Rate Over Time** — time series per instance for spotting error spikes.
+- **Top Error Messages** — table ranking the most frequent errors.
+- **Log Browser** — searchable, filterable log panel with level and free-text variables.
+
+To import the dashboard:
+
+1.  In Grafana, go to **Dashboards \> New \> Import**.
+2.  Select **Upload JSON file** and choose the downloaded `mattermost-loki-logs.json` file.
+3.  On the import screen, select your **Loki** data source from the dropdown.
+4.  Select **Import**.
+
+## Step 6: Useful LogQL queries
+
+You can run these queries directly in **Grafana Explore** (select the Loki data source) or use them to build additional dashboard panels.
+
+**Search all Mattermost logs:**
+
+``` text
+{service_name="mattermost"}
+```
+
+**Filter by log level:**
+
+``` text
+{service_name="mattermost"} | json | detected_level="error"
+```
+
+**Search for all HTTP 4xx responses:**
+
+``` text
+{service_name="mattermost"} | json | status_code >= 400
+```
+
+**Search for all HTTP 5xx responses:**
+
+``` text
+{service_name="mattermost"} | json | status_code >= 500
+```
+
+**Top 5 error messages over 5-minute windows:**
+
+``` text
+topk(5, sum(count_over_time({service_name="mattermost"} | json | detected_level="error" [5m])) by (msg))
+```
+
+**Count errors by message over 5-minute windows:**
+
+``` text
+sum(count_over_time({service_name="mattermost"} | json | detected_level="error" [5m])) by (msg)
+```
+
+**Free-text search (e.g., plugin errors):**
+
+``` text
+{service_name="mattermost"} |~ "plugin"
+```
+
+**Authentication-related log lines:**
+
+``` text
+{service_name="mattermost"} |~ "(?i)(auth|login|token|session)"
+```
+
+**Logs from a specific server instance:**
+
+``` text
+{service_name="mattermost", service_instance_id="mm-app-01"}
+```
+
+<Tip>
+
+For a full LogQL reference, see the [Grafana Loki LogQL documentation](https://grafana.com/docs/loki/latest/query/).
+
+</Tip>
+
+## Optional: Add PostgreSQL log collection
+
+If your PostgreSQL database runs on a dedicated server (not RDS), you can ship its logs to Loki as well. This is useful for correlating slow queries or database errors with Mattermost application events.
+
+<Note>
+
+A separate configuration file is provided for PostgreSQL log collection. Install the OpenTelemetry Collector on the PostgreSQL server using the same steps from **Step 3**, then use this configuration file instead of the Mattermost one:
+
+- [Download otel-collector-config-postgres.yaml](/files/samples/loki/otel-collector-config-postgres.yaml)
+
+The PostgreSQL configuration handles common Ubuntu/Debian JSON log paths (`/var/log/postgresql/*.json`). Replace the same `<LOKI_HOST>`, `<HOSTNAME>`, and `<SERVICE_NAME>` placeholders as described in Step 3.
+
+</Note>
+
+Once running, PostgreSQL logs appear in Grafana under the label `&#123;service_name="postgres"&#125;`:
+
+``` text
+{service_name="postgres"} | json | detected_level="error"
+```
+
+## Verification and troubleshooting
+
+**End-to-end verification checklist:**
+
+1.  Loki is healthy:
+
+    ``` bash
+    curl -s http://<monitoring-server>:3100/ready
+    # Expected: "ready"
+    ```
+
+2.  OpenTelemetry Collector is shipping logs (run on each Mattermost server):
+
+    ``` bash
+    sudo journalctl -u otelcol-contrib -n 100
+    # Look for export successful messages or lack of errors
+    ```
+
+3.  Logs are visible in Grafana: Go to **Explore**, select the **Loki** data source, and run `&#123;service_name="mattermost"&#125;`. You should see recent log lines.
+
+4.  The dashboard loads: Open the **Mattermost Log Aggregation** dashboard and confirm panels are populated. (Note: You may need to update dashboard queries to use <code>service_name</code> and <code>service_instance_id</code>).
+
+**Common issues:**
+
+<table>
+<colgroup>
+<col style={{width: '40%'}} />
+<col style={{width: '60%'}} />
+</colgroup>
+<thead>
+<tr>
+<th>Symptom</th>
+<th>Resolution</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>OTel Col target shows errors</td>
+<td>Verify the file paths in the config match the actual log file location. Check file permissions (<code>otelcol-contrib</code> user must be able to read the file).</td>
+</tr>
+<tr>
+<td><code>connection refused</code> on port 3100</td>
+<td>Ensure Loki is running (<code>systemctl status loki</code>). Check firewall rules: <code>sudo iptables -L -n | grep 3100</code> or check your AWS security group allows TCP 3100 from the Mattermost servers.</td>
+</tr>
+<tr>
+<td>Logs appear in Loki but fields aren't parsed</td>
+<td>Confirm Mattermost is writing JSON-formatted logs (see Step 1). Check that the <code>json_parser</code> operator (for Postgres) or OTel processors are correctly configured.</td>
+</tr>
+<tr>
+<td><code>/ready</code> returns an error</td>
+<td>Check Loki logs: <code>sudo journalctl -u loki -f</code>. Common cause: permissions on <code>/opt/loki/data/</code> — ensure the <code>loki</code> user owns the directory.</td>
+</tr>
+<tr>
+<td>Old logs are not being deleted</td>
+<td>Verify <code>compactor.retention_enabled: true</code> and <code>limits_config.retention_period</code> are both set in the Loki config. The compactor runs on the <code>compaction_interval</code> (default: 10 minutes) and applies a <code>retention_delete_delay</code> (default: 2 hours) before actually removing data.</td>
+</tr>
+<tr>
+<td>High disk usage on the monitoring server</td>
+<td>Review the retention period in the configuration. Consider reducing it or adding more disk. Check <code>du -sh /opt/loki/data/</code> to see current usage.</td>
+</tr>
+</tbody>
+</table>

docs/main/administration-guide/scale/deploy-prometheus-grafana-for-performance-monitoring.mdx

@@ -0,0 +1,139 @@
+---
+title: "Deploy Prometheus and Grafana for performance monitoring"
+---
+<PlanAvailability slug="entry-ent" />
+
+Performance monitoring support enables admins to track system health for large Enterprise deployments through integrations with [Prometheus](https://prometheus.io/) and [Grafana](https://grafana.com/). These integrations support data collection from several Mattermost servers, which is particularly useful if you're running Mattermost [in high availability mode](/deployment-guide/server/high-availability-cluster-based-deployment). Once you're tracking system health, you can [set up performance alerts](/administration-guide/scale/performance-alerting) on your Grafana dashboard.
+
+Admins can collect and store various data points from the Mattermost application in an [OpenMetrics](https://openmetrics.io) format by [deploying Prometheus](#install-prometheus) and [Grafana](#install-grafana).
+
+<Tip>
+
+Don't want to deploy Prometheus and Grafana? You can also [collect performance metrics using the Mattermost Metrics plugin](/administration-guide/scale/collect-performance-metrics).
+
+</Tip>
+
+## Install Prometheus
+
+<Important>
+
+While Prometheus and Grafana may be installed on the same server as Mattermost, we recommend installing these integrations on separate servers, and configure Prometheus to pull all metrics from Mattermost and other connected servers.
+
+</Important>
+
+1.  [Download a precompiled binary for Prometheus](https://prometheus.io/download/). Binaries are provided for many popular distributions, including Darwin, Linux, and Windows. For installation instructions, see the [Prometheus install guides](https://prometheus.io/docs/prometheus/latest/getting_started/).
+
+2.  The following settings are recommended in the Prometheus configuration file named `prometheus.yml`:
+
+    ``` yaml
+    # my global config
+    global:
+       scrape_interval:     5s # Set to 5 seconds for optimal performance monitoring.
+       evaluation_interval: 5s # Set to 5 seconds for optimal performance monitoring.
+       # scrape_timeout is set to the global default (10s).
+
+      # Attach these labels to any time series or alerts when communicating with
+      # external systems (federation, remote storage, Alertmanager).
+      external_labels:
+        monitor: 'mattermost-monitor'
+
+    # Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
+    rule_files:
+      # - "first.rules"
+      # - "second.rules"
+
+    # A scrape configuration containing exactly one endpoint to scrape:
+    # Here it's Prometheus itself.
+    scrape_configs:
+      # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
+      - job_name: 'prometheus'
+
+       # The scrape_interval can be overridden per job if needed (5s matches the global default).
+       # scrape_interval: 5s
+
+      # metrics_path defaults to '/metrics'
+      # scheme defaults to 'http'.
+
+        static_configs:
+          - targets: ["<hostname1>:<port>", "<hostname2>:<port>"]
+    ```
+
+3.  Replace the `<hostname1>:<port>` parameter with your Mattermost host IP address and port to scrape the data. It connects to `/metrics` using HTTP.
+
+4.  In the Mattermost System Console, go to **Environment \> Performance Monitoring** to set **Enable Performance Monitoring** to **true**, then specify the **Listen Address** (port-only, e.g., `8067`), and select **Save**. See our [Configuration Settings](/administration-guide/configure/environment-configuration-settings#performance-monitoring) documentation for details.
+
+    ![Enable performance monitoring options in the System Console by going to Environment \> Performance Monitoring, then specifying a listen address.](/images/perf_monitoring_system_console.png)
+
+5.  To test that the server is running, go to `<ip>:<port>/metrics`.
+
+<Note>
+
+A Mattermost Enterprise license is required to connect to `/metrics` using HTTP.
+
+</Note>
+
+6.  Finally, run `vi prometheus.yml` to finish configuring Prometheus. For starting the Prometheus service, read the [comprehensive guides provided by Prometheus](https://prometheus.io/docs/prometheus/latest/getting_started/).
+7.  Once the service has started, you can access the data in `<localhost>:<port>/graph`. While you can use the Prometheus service to create graphs, we'll focus on creating metric and analytics dashboards in Grafana.
+
+<Tip>
+
+For troubleshooting advice, check the [Prometheus FAQ page](https://prometheus.io/docs/introduction/faq/).
+
+</Tip>
+
+## Install Grafana
+
+<Important>
+
+While Prometheus and Grafana may be installed on the same server as Mattermost, we recommend installing these integrations on separate servers, and configure Prometheus to pull all metrics from Mattermost and other connected servers.
+
+</Important>
+
+1.  [Download a precompiled binary for Grafana](https://grafana.com/docs/grafana/latest/setup-grafana/installation/debian/) on Ubuntu or Debian. Binaries are also available for other distributions, including Redhat, Windows and Mac. For install instructions, see [Grafana install guides](https://grafana.com/docs/grafana/latest/setup-grafana/installation/debian/)
+
+2.  The Grafana package is installed as a service, so it is easy to start the server. See their [install guides](https://grafana.com/docs/grafana/latest/setup-grafana/installation/debian/) to learn more.
+
+3.  The default HTTP port is `3000` and default username and password are `admin`.
+
+4.  Add a Mattermost data source with the following settings as defined in the screenshot below
+
+    ![Mattermost data source configuration settings for a Grafana installation.](/images/mattermost_datasource.png)
+
+<Tip>
+
+- For troubleshooting advice, check the [Grafana Troubleshooting page](https://grafana.com/docs/grafana/latest/troubleshooting/).
+- For user guides and tutorials, check the [Grafana documentation to learn more](https://grafana.com/docs/grafana/latest/).
+
+</Tip>
+
+## Getting started
+
+To help you get started, you can download three sample dashboards shared in Grafana:
+
+<Tip>
+
+See [this Grafana guide](https://grafana.com/docs/grafana/v7.5/dashboards/export-import/) to learn how to import Grafana dashboards either from the UI or from the HTTP API.
+
+</Tip>
+
+- [Mattermost Performance Monitoring v2](https://grafana.com/grafana/dashboards/15582-mattermost-performance-monitoring-v2/), which contains detailed charts for performance monitoring including application, cluster, job server, and system metrics.
+- [Mattermost Notification Health Monitoring](https://grafana.com/grafana/dashboards/21305-mattermost-notification-health/), which can be used to track different types of notifications sent from Mattermost. System admins can [disable notification monitoring data collection](/administration-guide/configure/site-configuration-settings#enable-notification-monitoring) through the System Console.
+- [Mattermost Web App Performance Metrics](https://grafana.com/grafana/dashboards/21460-web-app-metrics/), which contains detailed metrics for client-side performance, including web vitals and Mattermost-specifc metrics.
+- [Mattermost Desktop App Performance Metrics](https://grafana.com/grafana/dashboards/22736-desktop-app-metrics/), which contains detailed metrics for client-side desktop performance, including CPU and memory usage metrics.
+- [Mattermost Mobile App Performance Metrics](https://grafana.com/grafana/dashboards/21695-mobile-performance-metrics/), which contains detailed metrics for client-side mobile performance, including web vitals and Mattermost-specifc metrics.
+- [Mattermost Threaded Discussion Metrics](https://grafana.com/grafana/dashboards/15581-collapsed-reply-threads-performance/), which contains detailed metrics on the queries involved in our threaded discussions feature.
+- [Mattermost Performance KPI Metrics](https://grafana.com/grafana/dashboards/2539-mattermost-performance-kpi-metrics/), which contains key metrics for monitoring performance and system health.
+- [Mattermost Performance Monitoring (Bonus Metrics)](https://grafana.com/grafana/dashboards/2545-mattermost-performance-monitoring-bonus-metrics/), which contains additional metrics such as emails sent or files uploaded, which may be important to monitor in some deployments.
+
+## What's collected?
+
+Mattermost provides [custom metrics](/administration-guide/scale/performance-monitoring-metrics#custom-mattermost-metrics) and [standard Go metrics](/administration-guide/scale/performance-monitoring-metrics#standard-go-metrics) that can be used to monitor your system's performance.
+
+When the `AggregatePluginMetrics` feature flag is enabled, plugin-provided metrics are included in the same `/metrics` scrape target and can be filtered by the `plugin_id` label.
+
+## Next steps
+
+Once you've set up performance monitoring, you may want to:
+
+- [Set up performance alerts](/administration-guide/scale/performance-alerting) to be notified when metrics exceed thresholds.
+- [Deploy Grafana Loki for centralized logging](/administration-guide/scale/deploy-grafana-loki-for-centralized-logging) to search and correlate logs with metric spikes.

docs/main/administration-guide/scale/ensuring-releases-perform-at-scale.mdx

@@ -0,0 +1,40 @@
+---
+title: "Ensure releases perform at scale"
+---
+<PlanAvailability slug="entry-ent" />
+
+To ensure each release of Mattermost upholds our high standards for performance at scale, the Mattermost Engineering team performs thorough load testing, develops features with scale in mind, and follows strict guidelines for database schema migrations.
+
+## Monthly release load tests
+
+Each month, before being approved for distribution, a release candidate of Mattermost is load tested via a comprehensive and mature set of [load test tooling](https://github.com/mattermost/mattermost-load-test-ng) using simulated data that matches real-world, high-scale usage patterns.
+
+Multiple tests covering different configurations are run against highly-available deployments of Mattermost with thousands of users and millions of posts. Both PostgreSQL and MySQL are tested, although MySQL tests will be dropped when [MySQL goes out of support with Mattermost v11](/product-overview/deprecated-features#mattermost-server-v1000).
+
+The load tests generate a report detailing average API request times, database I/O, memory usage, concurrency, requests per second, and more. This performance report on the release candidate is then compared to the report of the latest previous stable version. Any deviations are investigated and remedied before the release candidate is promoted to the final release.
+
+Each report, along with analysis, is posted into the public [Developers: Performance](https://community.mattermost.com/core/channels/developers-performance) channel.
+
+## Developing scalable features and systems
+
+Scale is a major consideration during the development of new features and systems, and it's included in technical specifications from the beginning of the software design process.
+
+As a part of implementation, the load test agent is updated to include coverage for the feature or system. A load test is then run with results being compared to baselines.
+
+The code changes are then reviewed by at least two developers and a SDET/QA analyst before being merged.
+
+Once merged, a new build is created and deployed the next day to the [Mattermost Community Server](https://community.mattermost.com) where any impacts on performance are monitored for 3-4 weeks before the changes are included in a release candidate.
+
+## Database schema changes
+
+Database schema changes are kept to a minimum to reduce risk on upgrades and impact on performance. When a change in schema is required, Mattermost follows [strict migration guidelines](https://developers.mattermost.com/contribute/more-info/server/schema-migration-guide/) that minimize risk and prevents performance impact during and after migrations.
+
+If a more involved migration is required, detailed analysis is performed and published with guidance. An example analysis is the [Mattermost v6.0 schema migration](https://gist.github.com/streamer45/59b3582118913d4fc5e8ff81ea78b055).
+
+Additionally, all database schema changes are load tested as a part of our monthly release process.
+
+## Monitoring post-release
+
+After a new version of Mattermost is released, it is rolled out to Mattermost Cloud customers over a multi-week period. During roll out, performance and error rate metrics are monitored for any unexpected changes. If any user-impacting changes are observed, the release is reverted and the deviations are investigated with any fixes being delivered as a part of patch release.
+
+In addition, Mattermost channels, user forums, and support tickets are closely monitored for reports of any issues. Any reports are investigated and resolved appropriately.

docs/main/administration-guide/scale/performance-alerting.mdx

@@ -0,0 +1,158 @@
+---
+title: "Performance alerting"
+---
+<PlanAvailability slug="entry-ent" />
+
+Mattermost recommends using [Prometheus](https://prometheus.io/) and [Grafana](https://grafana.com/) to track performance metrics of the Mattermost application servers. The purpose of this guide is to help you set up alerts on your Grafana dashboard once you've [set up system health tracking](/administration-guide/scale/deploy-prometheus-grafana-for-performance-monitoring).
+
+<Note>
+
+We highly recommend setting up performance alerting for deployments above 5,000 users, where additional servers have been added for performance load-balancing.
+
+</Note>
+
+## Prerequisites
+
+Set up performance monitoring for Mattermost. See our [Performance Monitoring](/administration-guide/scale/deploy-prometheus-grafana-for-performance-monitoring) documentation to learn more.
+
+To get alerts, first set up a Notification Channel in Grafana. Here’s how you can set it up to automatically post alerts in Mattermost:
+
+1.  In Mattermost:
+
+> 1.  Create an Alerts channel.
+> 2.  Create an [incoming webhook](https://developers.mattermost.com/integrate/webhooks/incoming/) for the Alerts channel and copy the URL.
+
+2.  In Grafana:
+
+> 1.  Under the alert icon in the sidebar, select **Contact points**.
+> 2.  Select **Create contact point**.
+> 3.  Enter **Mattermost Alerts Channel** as the name.
+> 4.  For type, select **Slack**.
+> 5.  Paste your webhook URL into the URL field.
+> 6.  Include an @ mention in the mention field, if you want to send mentions when an alert is posted to Mattermost.
+> 7.  Press **Send Test** to test the alert.
+
+If you would also like to get email alerts, you can follow [these instructions](https://grafana.com/docs/grafana/latest/alerting/) to set that up.
+
+## Configure alerts
+
+The [Mattermost dashboards](https://grafana.com/grafana/dashboards/?search=mattermost) for Grafana come with some partially pre-configured alerts on the following charts:
+
+- CPU Utilization Rate
+- Memory Usage
+- Number of Goroutines
+- Number of API Errors per Second
+- Mean API Request Time
+
+To configure alerts, set an appropriate threshold and enable notifications. Enabling notifications is the same for each chart, but setting the correct threshold can have some variances that are better handled on a per-chart basis.
+
+1.  For each chart, select the chart name, then select **Edit**:
+
+![Configure Grafana dashboard alerts for each chart by selecting the chart name then selecting Edit.](/images/perf-1.png)
+
+2.  Select the **Alert** tab:
+
+![Switch to the Alert tab to access and configure Grafana dashboard alerts for the current chart.](/images/perf-2.png)
+
+3.  The alert threshold, which will be discussed in the sections below, is the last field under **Conditions** (the one set to 600 in the screenshot above).
+
+See the sections below for how to set the threshold for each individual chart. If you would like to add your own custom alert conditions, configure them here.
+
+![When configuring Grafana dashboard alerts, set custom threshold conditions in the Conditions section of the screen.](/images/perf-3.png)
+
+4.  To enable the notifications for any alerts, select the **Notification** tab on the left, then select **Mattermost Alerts Channel** under **Send to**:
+
+![When configuring Grafana dashboard alerts, set notifications for any alerts by switching to the Notification tab, selecting Send to, then selecting by Mattermost Alerts Channel.](/images/perf-4.png)
+
+Enter a message if you would like to add more context to the alert.
+
+By default, the alerts are configured to check the average of a chart over the last minute to see if that value is above a threshold. If it’s above the threshold, the alert will be triggered. Since it’s an average over the last minute, small spikes that go past the threshold won’t necessarily cause an alert. This helps prevent false positives that would result from natural spikes in usage. The alert state for each chart is evaluated every minute.
+
+## Available charts
+
+The sections below describe each chart in more detail.
+
+### CPU utilization rate
+
+CPU Utilization Rate is fairly straightforward. CPU Utilization Rate tracks the CPU usage of the app servers as a percentage. The maximum percentage is based on the number of CPU cores or vCPUs your app server has. For example, if you have four CPU cores and your app server was at 100% utilization rate on all four cores, the graph would show 400% for that app server.
+
+It’s best to set the alert threshold based on your average CPU utilization and how many cores/vCPUs your app servers have. Take a look at the chart over the last seven days. You’ll want to set the threshold somewhere between your maximum CPU usage (cores \* 100) and the CPU usage you see. The lower you set the threshold, the more alerts, and therefore the more false positives, you will get. Set the threshold too high and you may not receive an alert for an incident until it’s gotten worse. The same principle applies to all alerts, regardless of the chart.
+
+For example, on our community server, we have the threshold set to 15%:
+
+![Example CPU utilization rate metrics for the Mattermost Community Server, where the threshold is set to 15%. System admins should set the threshold between the maximum CPU usage and the CPU usage observed in metrics.](/images/perf-5.png)
+
+This value is below our maximum CPU usage and above our average usage at peak times. Therefore, we will get alerts if we begin experiencing unusually high CPU usage.
+
+### Memory usage
+
+Memory Usage tracks the megabytes of RAM that your app servers are using. Set the threshold similar to the CPU Utilization Rate: below maximum available memory and above your average usage during peak times.
+
+Here’s how we have the alert set on our Community server:
+
+![Example memory usage metrics for the Mattermost Community Server, where the threshold is configured similarly to the CPU utilization rate.](/images/perf-6.png)
+
+### Number of Goroutines
+
+Goroutines are functions or methods that run concurrently with other functions and methods. Goroutines are like lightweight threads with low-creation costs. A rising number of goroutines can be a good measure of the performance of your app servers. A continuous increase indicates your app server can't keep up and is creating goroutines faster than they can complete their tasks and stop.
+
+Set the threshold somewhere above the average number of goroutines you see during peak load times. Small spikes are usually nothing to worry about. It’s the uncontrolled climbing of goroutines that you want to watch out for.
+
+Here’s how we have it set on our Community server:
+
+![Example Goroutines metrics for the Mattermost Community Server, where the threshold is configured above the average number of Goroutines observed during peak load times.](/images/perf-7.png)
+
+### Number of API errors per second
+
+Any 4xx or 5xx HTTP response status codes are counted as a REST API error. API errors themselves are not necessarily a problem. There are many legitimate reasons for an API error to occur, such as users’ sessions expiring or clients requesting to see if a resource exists and is being given a `404 Not Found` response. It is normal to have some API errors that scale with your installation base.
+
+That said, errors against the REST API can be indicative of deployment and other issues. For example, if one of your app servers did not deploy correctly for whatever reason, it may begin returning a high number of API errors. Another example would be a rogue bot spamming the API with bad requests. Alerts on API errors per second would help catch these and other issues.
+
+Here’s how it’s set on our Community server:
+
+![Example metrics of the number of API errors per second for the Mattermost Community Server, where it's normal to have some API errors that scale with an installation base, but that can be indicative of deployment issues or other issues.](/images/perf-8.png)
+
+### Mean API request time
+
+The Mean API Request Time is the average amount of time a REST API request to the Mattermost app server takes to complete. If an app server starts to perform poorly, you’ll likely see a rise in the mean request time as it takes longer to complete requests. This could also happen if your database can’t sustain the load from the app servers. It may also be indicative of an issue between the app servers and your proxy.
+
+You’ll want to set the alert threshold a little above what the mean request time is during your peak load times.
+
+Here’s how it’s set on our community server:
+
+![Example mean API request time metrics for the Mattermost Community Server, where the alert threshold is configured a bit above the mean request time during peak load times.](/images/perf-9-b.png)
+
+### Plugin hooks
+
+You can trace hooks and plugin API calls with Prometheus. Below are some examples of hooks and API Prometheus metrics that you may want to be aware of when troubleshooting or monitoring your server's performance.
+
+``` text
+# HELP mattermost_plugin_hook_time Time to execute plugin hook handler in seconds.
+# TYPE mattermost_plugin_hook_time histogram
+mattermost_plugin_hook_time_bucket{hook_name="ChannelHasBeenCreated",plugin_id="com.mattermost.demo-plugin",success="true",le="0.005"} 0
+mattermost_plugin_hook_time_bucket{hook_name="ChannelHasBeenCreated",plugin_id="com.mattermost.demo-plugin",success="true",le="0.01"} 0
+```
+
+``` text
+# HELP mattermost_plugin_multi_hook_time Time to execute multiple plugin hook handler in seconds.
+# TYPE mattermost_plugin_multi_hook_time histogram
+mattermost_plugin_multi_hook_time_bucket{plugin_id="com.mattermost.custom-attributes",le="0.005"} 100
+mattermost_plugin_multi_hook_time_bucket{plugin_id="com.mattermost.custom-attributes",le="0.01"} 100
+```
+
+``` text
+# HELP mattermost_plugin_multi_hook_server_time Time for the server to execute multiple plugin hook handlers in seconds.
+# TYPE mattermost_plugin_multi_hook_server_time histogram
+mattermost_plugin_multi_hook_server_time_bucket{le="0.005"} 1043
+```
+
+``` text
+# HELP mattermost_plugin_api_time Time to execute plugin API handlers in seconds.
+# TYPE mattermost_plugin_api_time histogram
+mattermost_plugin_api_time_bucket{api_name="AddUserToChannel",plugin_id="com.mattermost.plugin-incident-response",success="true",le="0.005"} 0
+mattermost_plugin_api_time_bucket{api_name="AddUserToChannel",plugin_id="com.mattermost.plugin-incident-response",success="true",le="0.01"} 0
+```
+
+## Other alerts
+
+If you want more alerts, you can set them up on any of the Grafana charts you'd like. We recommend reviewing custom metrics listed on our [Performance Monitoring feature documentation](/administration-guide/scale/performance-monitoring-metrics).

docs/main/administration-guide/scale/performance-monitoring-metrics.mdx

@@ -0,0 +1,266 @@
+---
+title: "Performance monitoring metrics"
+---
+<PlanAvailability slug="entry-ent" />
+
+Mattermost provides the following performance monitoring statistics to integrate with Prometheus and Grafana.
+
+## Custom Mattermost metrics
+
+The following is a list of custom Mattermost metrics that can be used to monitor your system's performance:
+
+### API metrics
+
+- `mattermost_api_time`: The total time in seconds to execute a given API handler.
+
+### Caching metrics
+
+- `mattermost_cache_etag_hit_total`: The total number of ETag cache hits for a specific cache.
+- `mattermost_cache_etag_miss_total`: The total number of ETag cache misses for an API call.
+- `mattermost_cache_mem_hit_total`: The total number of memory cache hits for a specific cache.
+- `mattermost_cache_mem_invalidation_total`: The total number of memory cache invalidations for a specific cache.
+- `mattermost_cache_mem_miss_total`: The total number of cache misses for a specific cache.
+
+The above metrics can be used to calculate ETag and memory cache hit rates over time.
+
+![Example caching metrics, including Etag hit rate and mem cache hit rate, in a self-hosted Mattermost deployment.](/images/perf_monitoring_caching_metrics.png)
+
+### Cluster metrics
+
+- `mattermost_cluster_cluster_request_duration_seconds`: The total duration in seconds of the inter-node cluster requests.
+- `mattermost_cluster_cluster_health_score`: A score that gives an idea of how well it is meeting the soft-real time requirements of the gossip protocol.
+- `mattermost_cluster_cluster_requests_total`: The total number of inter-node requests.
+- `mattermost_cluster_cluster_event_type_totals`: The total number of cluster requests sent for any type.
+- `mattermost_cluster_reliable_fallback_tcp`: The total length in bytes of the `SendBestEffort` calls (UDP) that had to fallback to `SendReliable` (TCP) because of the message length. This is a histogram with 8 buckets, from 32KiB to 4MiB. Ideally, the count of this metric should be always 0, but plugins may be misusing the websocket API and attempting to send large messages via UDP; in that case, plugins should be fixed to set [ReliableClusterSend](https://github.com/mattermost/mattermost/blob/4fe11a7d5d14616f3c31363762617ddfd99895df/server/public/model/websocket_message.go#L155-L157) to true.
+
+### Database metrics
+
+- `mattermost_db_active_users`: The total number of active users.
+- `mattermost_db_cache_time`: Time to execute the cache handler.
+- `mattermost_db_master_connections_total`: The total number of connections to the master database.
+- `mattermost_db_read_replica_connections_total`: The total number of connections to all the read replica databases.
+- `mattermost_db_search_replica_connections_total`: The total number of connections to all the search replica databases.
+- `mattermost_db_store_time`: The total time in seconds to execute a given database store method.
+- `mattermost_db_replica_lag_abs`: Absolute lag time based on binlog distance/transaction queue length.
+- `mattermost_db_replica_lag_time`: The time taken for the replica to catch up.
+
+### Database connection metrics
+
+- `go_sql_max_open_connections`: Maximum number of open connections to the database.
+- `go_sql_open_connections`: The number of established connections both in use and idle.
+- `go_sql_in_use_connections`: The number of connections currently in use.
+- `go_sql_idle_connections`: The number of idle connections.
+- `go_sql_wait_count_total`: The total number of connections waited for.
+- `go_sql_wait_duration_seconds_total`: The total time blocked waiting for a new connection.
+- `go_sql_max_idle_closed_total`: The total number of connections closed due to SetMaxIdleConns.
+- `go_sql_max_idle_time_closed_total`: The total number of connections closed due to SetConnMaxIdleTime.
+- `go_sql_max_lifetime_closed_total`: The total number of connections closed due to SetConnMaxLifetime.
+
+### HTTP metrics
+
+- `mattermost_http_errors_total`: The total number of http API errors.
+- `mattermost_http_requests_total`: The total number of http API requests.
+- `mattermost_http_websockets_total`: The total number of websocket connections to this server.
+
+<Note>
+
+From Mattermost version v9.9, this value includes any potentially unauthenticated connections. Furthermore, this metric comes with an `origin_client` label that can be used to see the distribution of connections from different client types (i.e. web, mobile, and desktop).
+
+</Note>
+
+![Example HTTP metrics, including number of API errors per minute, number of API requests per minute, and mean request time per minute, in a self-hosted Mattermost deployment.](/images/perf_monitoring_http_metrics.png)
+
+### Login and session metrics
+
+- `mattermost_login_logins_fail_total`: The total number of failed logins.
+- `mattermost_login_logins_total`: The total number of successful logins.
+
+### Mattermost channels metrics
+
+- `mattermost_post_broadcasts_total`: The total number of websocket broadcasts sent because a post was created.
+- `mattermost_post_emails_sent_total`: The total number of emails sent because a post was created.
+- `mattermost_post_file_attachments_total`: The total number of file attachments created because a post was created.
+- `mattermost_post_pushes_sent_total`: The total number of mobile push notifications sent because a post was created.
+- `mattermost_post_total`: The total number of posts created.
+- `mattermost_post_webhooks_total`: Total number of webhook posts created.
+
+![Example Mattermost channels metrics, including messages per minute, broadcasts per minute, emails sent per minute, mobile push notifications per minute, and number of file attachments per minute, in a self-hosted Mattermost deployment.](/images/perf_monitoring_messaging_metrics.png)
+
+### Process metrics
+
+- `mattermost_process_cpu_seconds_total`: Total user and system CPU time spent in seconds.
+- `mattermost_process_max_fds`: Maximum number of open file descriptors.
+- `mattermost_process_open_fds`: Number of open file descriptors.
+- `mattermost_process_resident_memory_bytes`: Resident memory size in bytes.
+- `mattermost_process_start_time_seconds`: Start time of the process since unix epoch in seconds.
+- `mattermost_process_virtual_memory_bytes`: Virtual memory size in bytes.
+- `mattermost_process_virtual_memory_max_bytes`: Maximum amount of virtual memory available in bytes.
+
+### Search metrics
+
+- `mattermost_search_posts_searches_duration_seconds`: The total duration in seconds of post searches.
+- `mattermost_search_channel_index_total`: The total number of channel indexes carried out.
+- `mattermost_search_file_index_total`: The total number of files indexes carried out.
+- `mattermost_search_files_searches_duration_seconds`: The total duration in seconds of file searches.
+- `mattermost_search_files_searches_total`: The total number of file searches carried out.
+- `mattermost_search_post_index_total`: The total number of posts indexes carried out.
+- `mattermost_search_posts_searches_total`: The total number of post searches carried out.
+- `mattermost_search_user_index_total`: The total number of user indexes carried out.
+- `mattermost_search_engine_status`: Status of the configured search engine: `1` = healthy or not configured, `0` = configured but unavailable. Use this metric to set up alerts for search engine outages.
+
+### WebSocket metrics
+
+- `mattermost_websocket_broadcast_buffer_size`: Number of events in the websocket broadcasts buffer waiting to be processed.
+- `mattermost_websocket_broadcast_buffer_users_registered`: Number of users registered in a broadcast buffer hub.
+- `mattermost_websocket_broadcasts_total`: The total number of websocket broadcasts sent for any type.
+- `mattermost_websocket_event_total`: Total number of websocket events.
+- `mattermost_websocket_reconnects_total`: Total number of websocket reconnect attempts.
+
+### Logging metrics
+
+- `mattermost_logging_logger_queue_used`: Number of records in log target queue.
+- `mattermost_logging_logger_logged_total`: The total number of records logged.
+- `mattermost_logging_logger_error_total`: The total number of logger errors.
+- `mattermost_logging_logger_dropped_total`: The total number of dropped log records.
+- `mattermost_logging_logger_blocked_total`: The total number of log records that were blocked/delayed.
+
+### Debugging metrics - system
+
+- `mattermost_system_server_start_time`: The time the server started.
+
+Use `mattermost_system_server_start_time` to dynamically add an annotation corresponding to the event.
+
+![Example debugging metrics, including number of messages per second, in a self-hosted Mattermost deployment.](/images/mattermost_system_server_start_time.png)
+
+### Debugging metrics - jobs
+
+- `mattermost_jobs_active`: Number of active jobs.
+
+Use `mattermost_jobs_active` to display an active jobs chart.
+
+![Example debugging metrics, including active jobs, in a self-hosted Mattermost deployment.](/images/mattermost_active_jobs_chart.png)
+
+Or, use `mattermost_jobs_active` to dynamically add a range annotation corresponding to jobs being active.
+
+![Example debugging metrics, including number of messages per second, in a self-hosted Mattermost deployment.](/images/mattermost_dynamic_range_annotation.png)
+
+Use annotations to streamline analysis when a job is long running, such as an LDAP synchronization job.
+
+<Note>
+
+Jobs where the runtime is less than the Prometheus polling interval are unlikely to be visible because Grafana is performing range queries over the raw Prometheus timeseries data, and rendering an event each time the value changes.
+
+</Note>
+
+### Plugin metrics
+
+- `mattermost_plugin_api_time`: Time to execute plugin API handlers in seconds.
+- `mattermost_plugin_hook_time`: Time to execute plugin hook handler in seconds.
+- `mattermost_plugin_multi_hook_server_time`: Time for the server to execute multiple plugin hook handlers in seconds.
+- `mattermost_plugin_multi_hook_time`: Time to execute multiple plugin hook handler in seconds.
+
+The metrics above are measured by the Mattermost server as it executes plugin code. From Mattermost v11.8.0, plugin-provided Prometheus metrics can also be exposed through the standard Mattermost `/metrics` endpoint when the `AggregatePluginMetrics` feature flag is enabled. Aggregated plugin metrics include a `plugin_id` label, based on the plugin's manifest ID, so admins can identify which plugin produced each metric.
+
+<Note>
+
+`AggregatePluginMetrics` is disabled by default and must be enabled before plugin-provided metrics are included in the `/metrics` response.
+
+</Note>
+
+### Shared metrics
+
+- `mattermost_shared_channels_sync_collection_duration_seconds`: Duration tasks spend collecting sync data (seconds).
+- `mattermost_shared_channels_sync_collection_step_duration_seconds`: Duration tasks spend in each step collecting data (seconds).
+- `mattermost_shared_channels_sync_count`: Count of sync events processed for each remote.
+- `mattermost_shared_channels_sync_send_duration_seconds`: Duration tasks spend sending sync data (seconds).
+- `mattermost_shared_channels_sync_send_step_duration_seconds`: Duration tasks spend in each step sending data (seconds).
+- `mattermost_shared_channels_task_in_queue_duration_seconds`: Duration tasks spend in queue (seconds).
+- `mattermost_shared_channels_task_queue_size`: Current number of tasks in queue.
+
+### Remote cluster metrics
+
+- `mattermost_remote_cluster_clock_skew`: An approximated value for clock skew between clusters.
+- `mattermost_remote_cluster_conn_state_change_total`: Total number of connection state changes.
+- `mattermost_remote_cluster_msg_errors_total`: Total number of message errors.
+- `mattermost_remote_cluster_msg_received_total`: Total number of messages received from the remote cluster.
+- `mattermost_remote_cluster_msg_sent_total`: Total number of messages sent to the remote cluster.
+- `mattermost_remote_cluster_ping_time`: The ping roundtrip times to the remote cluster.
+
+### Notification metrics
+
+- `mattermost_notifications_error`: Total number of errors that stop the notification flow.
+- `mattermost_notifications_not_sent`: Total number of notifications the system deliberately did not send.
+- `mattermost_notifications_success`: Total number of successfully sent notifications.
+- `mattermost_notifications_total`: Total number of notification events.
+- `mattermost_notifications_total_ack`: Total number of notification events acknowledged.
+- `mattermost_notifications_unsupported`: Total number of untrackable notifications due to an unsupported app version.
+
+### Mobile app metrics
+
+- `mattermost_mobileapp_mobile_channel_switch`: Duration of the time taken from when a user clicks on a channel name, and the full channel sreen is loaded (seconds).
+- `mattermost_mobileapp_mobile_load`: Duration of the time taken from when a user opens the app and the app finally loads all relevant information (seconds).
+- `mattermost_mobileapp_mobile_team_switch`: Duration of the time taken from when a user clicks on a team, and the full categories screen is loaded (seconds).
+
+### Web app metrics
+
+- `mattermost_webapp_channel_switch`: Duration of the time taken from when a user clicks on a channel in the LHS to when posts in that channel become visible (seconds).
+- `mattermost_webapp_cumulative_layout_shift`: Measure of how much a page's content shifts unexpectedly.
+- `mattermost_webapp_first_contentful_paint`: Duration of how long it takes for any content to be displayed on screen to a user (seconds).
+- `mattermost_webapp_global_threads_load`: Duration of the time taken from when a user clicks to open Threads in the LHS until when the global threads view becomes visible (milliseconds).
+- `mattermost_webapp_interaction_to_next_paint`: Measure of how long it takes for a user to see the effects of clicking with a mouse, tapping with a touchscreen, or pressing a key on the keyboard (seconds).
+- `mattermost_webapp_largest_contentful_paint`: Duration of how long it takes for large content to be displayed on screen to a user (seconds).
+- `mattermost_webapp_long_tasks`: Counter of the number of times that the browser's main UI thread is blocked for more than 50ms by a single task.
+- `mattermost_webapp_page_load`: The amount of time from when the browser starts loading the web app until when the web app's load event has finished (seconds).
+- `mattermost_webapp_rhs_load`: Duration of the time taken from when a user clicks to open a thread in the RHS until when posts in that thread become visible (seconds).
+- `mattermost_webapp_team_switch`: Duration of the time taken from when a user clicks on a team in the LHS to when posts in that team become visible (seconds).
+- `mattermost_webapp_time_to_first_byte`: Duration from when a browser starts to request a page from a server until when it starts to receive data in response (seconds).
+
+## Standard Go metrics
+
+<PlanAvailability slug="all-commercial" />
+
+The performance monitoring feature provides standard Go metrics for HTTP server runtime profiling data and system monitoring, such as:
+
+- `go_memstats_alloc_bytes` for memory usage
+- `go_goroutines` for number of goroutines
+- `go_gc_duration_seconds` for garbage collection duration
+- `go_memstats_heap_objects` for object tracking on the heap
+
+To learn how to set up runtime profiling, see the [pprof package Go documentation](https://pkg.go.dev/net/http/pprof). You can also visit the `ip:port` page for a complete list of metrics with descriptions.
+
+<Note>
+
+A Mattermost Enterprise license is required to connect to `/metrics` using HTTP.
+
+</Note>
+
+If enabled, you can run the profiler by
+
+`go tool pprof http://localhost:<port>/debug/pprof/profile?seconds=<duration>`
+
+where you can replace `localhost` with the server name. The profiling reports are available at `<ip>:<port>`, which include:
+
+- `/debug/pprof/profile?seconds=30` for CPU profiling
+- `/debug/pprof/cmdline` for command line profiling
+- `/debug/pprof/symbol` for symbol profiling
+- `/debug/pprof/trace` for trace profiling
+- `/debug/pprof/goroutine` for Go routine profiling
+- `/debug/pprof/heap` for heap profiling
+- `/debug/pprof/threadcreate` for threads profiling
+- `/debug/pprof/block` for block profiling
+- `/debug/pprof/allocs` for past memory allocations
+- `/debug/pprof/mutex` for logs of the holders of contended mutexes
+
+![Example Go metrics for HTTP server runtime profiling data and system monitoring, including memory usage, Go routines, and garbage collection duration, in a self-hosted Mattermost deployment.](/images/perf_monitoring_go_metrics.png)
+
+## Host/system metrics
+
+From Metrics Plugin v0.7.0, you can pull metrics from [node exporter](https://github.com/prometheus/node_exporter) targets to access network-related panels for Mattermost Calls.
+
+## Frequently asked questions
+
+### Why are chart labels difficult to distinguish?
+
+The chart labels used in server filters and legends are based on the hostname of your machines. If the hostnames are similar, then it will be difficult to distinguish the labels.
+
+You can either set more descriptive hostnames for your machines or change the display name with a `relabel_config` in [Prometheus configuration](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config).

docs/main/administration-guide/scale/push-notification-health-targets.mdx

@@ -0,0 +1,74 @@
+---
+title: "Push notification health targets"
+---
+<PlanAvailability slug="entry-ent" />
+
+When using the [Mattermost Notification Health](https://grafana.com/grafana/dashboards/21305-mattermost-notification-health) Grafana dashboard to track different types of notifications sent from Mattermost, we recommend adhering to the following mobile push notification health targets to ensure a performant production deployment of Mattermost.
+
+<Note>
+
+- Accessing and enabling Mattermost Notification Health Monitoring requires `MetricsSettings.Enable` set to `true`.
+- `MetricsSettings.EnableNotificationMetrics` must be enabled in the [Performance Monitoring](/administration-guide/configure/environment-configuration-settings#enable-notification-monitoring) configuration.
+- System admins can [disable notification monitoring data collection](/administration-guide/configure/site-configuration-settings#enable-notification-monitoring) through the System Console.
+
+</Note>
+
+## Push Proxy Delivery Rate
+
+The **Push Proxy Delivery Rate** panel indicates the percentage of push notifications that have been successfully delivered to the Push Notification Service (i.e., Apple Push Notification service (APNs) or Firebase Cloud Messaging (FCM)) out of the total number of push notifications sent.
+
+Monitor this delivery rate to ensure that users are receiving their notifications in a timely manner. This value should always be 100%. Anything below 99% should be investigated further to identify issues with the push notification configuration, connectivity problems, or possible service interruptions with the push proxy provider.
+
+![The Push Notifications Errors by Reason panel provides admins with a breakdown of the errors encountered when sending push notifications, categorized by the specific reason for each error.](/images/push-proxy-delivery-rate.png)
+
+## Push Notifications Errors by Reason
+
+The **Push Notifications Errors by Reason** panel is a critical tool for maintaining the reliability and effectiveness of the push notification system. These values provide a breakdown of the errors encountered when sending push notifications, categorized by the specific reason for each error.
+
+This information helps admins to pinpoint common problems affecting push notification delivery, diagnose and fix issues more efficiently, detect patterns and trends, identify intermittent issues before they become more serious, and enhance the overall user experience with the Mattermost platform.
+
+Missing profile errors are generally safe to ignore. However, if you see occurences of this error increase, verify logs for `push_proxy_send_error`.
+
+## Push Notifications Not Sent Distribution
+
+The **Push Notifications Not Sent Distribution** panel helps admins ensure the robustness and reliability of the push notification system by providing detailed insights into the specific reasons why notifications weren't sent.
+
+This information helps admins understand the reasons why push notifications may not be sent, (i.e., the channel is muted), identify underlying system or configuration problems that need attention, prioritize areas for troubleshooting and improvement by focusing on the most significant issues affecting notification delivery rates, and monitor any trends or spikes in push notifications not sent, resulting in proactive maintenance and quicker resolution of issues as they arise.
+
+![The Push Notifications Not Sent Distribution panel provides admins with detailed insights into the specific reasons for notification delivery failures.](/images/push-notifications-not-sent-distribution.png)
+
+## Push Notifications Stats
+
+The **Push Notifications Stats** panel provides a high-level view of the number of notifications sent and acknowledged.
+
+An acknowledged push notification indicates successful communication between the service and the device because the recipient's mobile device successfully received the push notification, and sent back a confirmation of delivery to the push notification service.
+
+When a push notification isn't acknowledged, this indicates that the notification wasn't received, or that the device didn't send back a confirmation to the push notification service.
+
+Closer alignment between the send and acknowledge lines in this graph indicates better performance.
+
+![The Push Notifications Stats panel provides admins with a high-level view of the number of notifications sent and acknowledged.](/images/push-notification-stats.png)
+
+## Total Acked Push Notifications
+
+The **Total Acked Push Notifications** value indicates the total number of push notifications that have been acknowledged by mobile devices. This metric helps admins understand how many push notifications sent from the Mattermost server were successfully received and acknowledged by mobile devices. It's an important indicator of the reliability and effectiveness of the push notification system, as it ensures that messages are being delivered and received as intended. Any value over 80% is considered normal.
+
+![The Total Acked Push Notifications value helps admins understand how many push notifications sent from the Mattermost server were successfully received and acknowledged by mobile devices.](/images/total-acked-push-notifications.png)
+
+There are several scenarios where notifications may not be acknowledged which cover the remaining ~10-20% of notifications, as follows:
+
+Servers removed without users logging out
+
+- When a server is removed from the mobile app, and you can no longer contact the server using the mobile app, the server never logs the user session out. Therefore, Mattermost continues to send notifications to the mobile device. These sent notifications, can't be acknowledged because Mattermost would be missing the information about the server URL.
+
+iOS notifications are disabled
+
+- When notifications are disabled on iOS devices, notifications aren't processed by the mobile app. Therefore, we cannot acknowledge those notifications. Mattermost tracks whether the device has notifications disabled, but this information is only updated when the app starts. So any change without re-starting the app can lead to notifications not being acknowledged.
+
+Notifications are dropped at the Push Notification Service
+
+- APNs and FCM can drop notifications in several circumstances, especially when the mobile device doesn't have a connection. The notifications are marked as sent to the Push Proxy, but are never sent to the devices from the Push Notification Service. Therefore, the mobile device cannot acknowledge the notification.
+
+Extreme battery save mode
+
+- On Android, users can activate an extreme battery save mode. In this mode, notifications are not acknowledged.