snapshot of https://github.com/mattermost/mattermost.git (master, 2026-09-16; blobs > 131072B omitted)

docs/main/administration-guide/scale/deploy-grafana-loki-for-centralized-logging.mdx

@@ -0,0 +1,445 @@
+---
+title: "Deploy Grafana Loki for centralized logging"
+---
+<PlanAvailability slug="entry-ent" />
+
+This guide extends your existing [Prometheus and Grafana performance monitoring deployment](/administration-guide/scale/deploy-prometheus-grafana-for-performance-monitoring) by adding [Grafana Loki](https://grafana.com/oss/loki/) for centralized log aggregation. While Prometheus collects **metrics** (CPU, request latency, goroutine counts), Loki collects **logs** — giving your operations team the ability to search, filter, and correlate Mattermost application logs across all servers from a single Grafana interface.
+
+With Loki in place you can:
+
+- Search Mattermost logs across all application servers from Grafana.
+- Filter by log level, HTTP status code, or free-text search.
+- Identify the top recurring errors and warnings.
+- Correlate log events with Prometheus metric spikes on the same timeline.
+- Optionally collect PostgreSQL database logs alongside application logs.
+
+<Tip>
+
+This guide assumes you have already deployed Prometheus and Grafana by following the [performance monitoring guide](/administration-guide/scale/deploy-prometheus-grafana-for-performance-monitoring). Loki and the OpenTelemetry Collector will be added to that existing infrastructure.
+
+</Tip>
+
+## Architecture overview
+
+The deployment adds two components to your monitoring stack:
+
+- **Loki** — the log aggregation engine, installed on your existing Grafana/Prometheus monitoring server.
+- **OpenTelemetry Collector** — a versatile, industry-standard agent installed on each Mattermost application server to ship logs.
+
+``` text
+┌──────────────────────┐       ┌──────────────────────┐
+│  Mattermost App 01   │       │  Mattermost App 02   │
+│                      │       │                      │
+│  /opt/mattermost/    │       │  /opt/mattermost/    │
+│   logs/mattermost.log│       │   logs/mattermost.log│
+│         │            │       │         │            │
+│    [ OTel Col ]      │       │    [ OTel Col ]      │
+└─────────┬────────────┘       └─────────┬────────────┘
+          │ push (:3100/otlp)            │
+          ▼                              ▼
+┌─────────────────────────────────────────────────────┐
+│           Monitoring Server                         │
+│                                                     │
+│   [ Prometheus :9090 ]    [ Loki :3100 ]            │
+│           │                     │                   │
+│           └──────┬──────────────┘                   │
+│                  ▼                                  │
+│           [ Grafana :3000 ]                         │
+│            Metrics + Logs                           │
+└─────────────────────────────────────────────────────┘
+
+Optional: Add the OpenTelemetry Collector on your PostgreSQL server to also
+ship database logs to Loki.
+```
+
+## Prerequisites
+
+Before starting, confirm the following:
+
+- Prometheus and Grafana are deployed and collecting Mattermost metrics per the [performance monitoring guide](/administration-guide/scale/deploy-prometheus-grafana-for-performance-monitoring).
+- You have **SSH access** with sudo privileges to the monitoring server and each Mattermost application server.
+- Mattermost is installed at `/opt/mattermost` (the default location). If your installation path differs, substitute your path wherever `/opt/mattermost` appears.
+- **Network connectivity**: Each Mattermost server can reach the monitoring server on **TCP port 3100** (Loki's HTTP API).
+
+## Step 1: Verify Mattermost JSON logging
+
+Mattermost must be writing JSON-formatted file logs for Loki queries to work. This is the default on all plans — verify it on each application server:
+
+``` bash
+tail -1 /opt/mattermost/logs/mattermost.log
+```
+
+A JSON-formatted line looks like:
+
+``` json
+{"timestamp":"2025-01-15T14:32:01.123Z","level":"info","msg":"Server is listening on [::]:8065","caller":"app/server.go:482"}
+```
+
+If you see plain-text output instead, go to **System Console \> Environment \> Logging** and set **Output file logs as JSON** to `true`, then restart Mattermost:
+
+``` bash
+sudo systemctl restart mattermost
+```
+
+## Step 2: Install Loki on the monitoring server
+
+Install Loki on the same server that runs Grafana and Prometheus. All commands in this section are run **on the monitoring server**.
+
+<Important>
+
+The commands below use Loki version **3.4.2**. Check the [Loki releases page](https://github.com/grafana/loki/releases) for the latest version and substitute the version number as needed.
+
+</Important>
+
+1.  Create the Loki user, directories, and download the binary:
+
+    ``` bash
+    # Create a dedicated system user
+    sudo useradd --system --no-create-home --shell /bin/false loki
+
+    # Create directories
+    sudo mkdir -p /opt/loki/data /opt/loki/bin
+
+    # Download and extract Loki
+    sudo apt-get install -y unzip
+    cd /tmp
+    curl -LO https://github.com/grafana/loki/releases/download/v3.4.2/loki-linux-amd64.zip
+    unzip loki-linux-amd64.zip
+    sudo mv loki-linux-amd64 /opt/loki/bin/loki
+    sudo chmod +x /opt/loki/bin/loki
+    ```
+
+2.  Download and install the production configuration file:
+
+    - [Download loki-config.yaml](/files/samples/loki/loki-config.yaml)
+
+    ``` bash
+    sudo cp loki-config.yaml /opt/loki/loki-config.yaml
+    ```
+
+    <div class="tip">
+
+    <div class="title">
+
+    Tip
+
+    </div>
+
+    **Log retention** is set to **14 days** by default. To change this, edit `/opt/loki/loki-config.yaml` and update the `retention_period` value under `limits_config`. Loki requires this value in hours — common values:
+
+    - `336h` = 14 days (default)
+    - `720h` = 30 days
+    - `2160h` = 90 days
+    - `8760h` = 365 days
+
+    Longer retention increases disk usage. As a rough guide, expect 1–3 GB per day for a moderately active Mattermost deployment (varies with log volume). Monitor `/opt/loki/data/` after the first week to project storage needs. The `compactor.retention_enabled` setting must remain `true` for retention enforcement to work.
+
+    </div>
+
+3.  Set ownership:
+
+    ``` bash
+    sudo chown -R loki:loki /opt/loki
+    ```
+
+4.  Create a systemd service file:
+
+    ``` bash
+    sudo tee /etc/systemd/system/loki.service > /dev/null <<'EOF'
+    [Unit]
+    Description=Grafana Loki Log Aggregation
+    Documentation=https://grafana.com/docs/loki/latest/
+    After=network-online.target
+    Wants=network-online.target
+
+    [Service]
+    Type=simple
+    User=loki
+    Group=loki
+    ExecStart=/opt/loki/bin/loki -config.file=/opt/loki/loki-config.yaml
+    Restart=on-failure
+    RestartSec=5
+    LimitNOFILE=65536
+    StandardOutput=journal
+    StandardError=journal
+
+    [Install]
+    WantedBy=multi-user.target
+    EOF
+    ```
+
+5.  Start Loki:
+
+    ``` bash
+    sudo systemctl daemon-reload
+    sudo systemctl enable --now loki
+    ```
+
+6.  Verify Loki is running:
+
+    ``` bash
+    sudo systemctl status loki
+    curl -s http://localhost:3100/ready
+    ```
+
+    The `/ready` endpoint should return `ready`. Loki may take 15–20 seconds after startup before it reports ready — this warmup delay is normal.
+
+## Step 3: Install OpenTelemetry Collector on each Mattermost server
+
+The OpenTelemetry (OTel) Collector runs on each Mattermost application server and pushes logs to Loki. Repeat these steps on **every Mattermost application server**.
+
+<Important>
+
+Replace `<LOKI_HOST>` with the IP address or hostname of your monitoring server, `<HOSTNAME>` with this server's hostname (e.g., `mm-app-01`), and `<SERVICE_NAME>` with the service type (e.g., `mattermost` or `postgres`).
+
+</Important>
+
+1.  Install the OTel Collector Contrib distribution:
+
+    ``` bash
+    # Add OTel Collector repository and install
+    # Commands below are for Ubuntu/Debian
+    wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.145.0/otelcol-contrib_0.145.0_linux_amd64.deb
+    sudo dpkg -i otelcol-contrib_0.145.0_linux_amd64.deb
+    ```
+
+    <div class="note">
+
+    <div class="title">
+
+    Note
+
+    </div>
+
+    The installation process automatically creates a system user and group named `otelcol-contrib`.
+
+    </div>
+
+2.  Download and edit the OpenTelemetry Collector configuration:
+
+    - [Download otel-collector-config-mattermost.yaml](/files/samples/loki/otel-collector-config-mattermost.yaml)
+
+    ``` bash
+    sudo cp otel-collector-config-mattermost.yaml /etc/otelcol-contrib/config.yaml
+    ```
+
+    Edit `/etc/otelcol-contrib/config.yaml` and replace the placeholders:
+
+    - Replace `<LOKI_HOST>` with your monitoring server's address.
+    - Replace `<HOSTNAME>` with this server's hostname.
+    - Replace `<SERVICE_NAME>` with the service type (e.g., `mattermost`).
+
+    ``` bash
+    sudo vi /etc/otelcol-contrib/config.yaml
+    ```
+
+3.  Grant the collector read access to the Mattermost log directory:
+
+    ``` bash
+    # Grant the collector access to read Mattermost logs.
+    # Substitute 'mattermost' for the group that owns your log file.
+    sudo usermod -aG mattermost otelcol-contrib
+
+    # Ensure logs are group-readable
+    sudo chmod 640 /opt/mattermost/logs/mattermost.log
+    sudo chmod g+rx /opt/mattermost/logs
+    ```
+
+4.  Restart the service:
+
+    ``` bash
+    sudo systemctl restart otelcol-contrib
+    sudo systemctl enable otelcol-contrib
+    ```
+
+5.  Verify the collector is running and shipping logs:
+
+    ``` bash
+    sudo systemctl status otelcol-contrib
+    sudo journalctl -u otelcol-contrib -f
+    ```
+
+## Step 4: Add Loki as a data source in Grafana
+
+1.  Log in to Grafana (default: `http://<monitoring-server>:3000`).
+2.  Navigate to **Administration \> Data Sources \> Add data source**.
+3.  Select **Loki** from the list.
+4.  Configure the connection:
+    - **Name**: `Loki`
+    - **URL**: `http://localhost:3100` (since Loki is colocated with Grafana)
+5.  Select **Save & test**. Grafana should confirm the data source is working.
+
+## Step 5: Import the Mattermost Loki dashboard
+
+A pre-built Grafana dashboard is recommended for monitoring your logs. This dashboard is provided as a basic example:
+
+- [Download the Mattermost Loki Logs dashboard JSON](/files/samples/grafana-dashboards/mattermost-loki-logs.json)
+
+<Note>
+
+While the dashboard provides a useful starting point, users can gain full query flexibility using the **Explore** tab in Grafana to build custom LogQL queries on the fly.
+
+</Note>
+
+The dashboard provides:
+
+- **Log Volume by Level** — stacked bar chart showing log rates by severity.
+- **Error / Warning / Total counters** — stat panels for the selected time range.
+- **HTTP 4xx/5xx count** — quick visibility into application-level HTTP errors.
+- **Error Rate Over Time** — time series per instance for spotting error spikes.
+- **Top Error Messages** — table ranking the most frequent errors.
+- **Log Browser** — searchable, filterable log panel with level and free-text variables.
+
+To import the dashboard:
+
+1.  In Grafana, go to **Dashboards \> New \> Import**.
+2.  Select **Upload JSON file** and choose the downloaded `mattermost-loki-logs.json` file.
+3.  On the import screen, select your **Loki** data source from the dropdown.
+4.  Select **Import**.
+
+## Step 6: Useful LogQL queries
+
+You can run these queries directly in **Grafana Explore** (select the Loki data source) or use them to build additional dashboard panels.
+
+**Search all Mattermost logs:**
+
+``` text
+{service_name="mattermost"}
+```
+
+**Filter by log level:**
+
+``` text
+{service_name="mattermost"} | json | detected_level="error"
+```
+
+**Search for all HTTP 4xx responses:**
+
+``` text
+{service_name="mattermost"} | json | status_code >= 400
+```
+
+**Search for all HTTP 5xx responses:**
+
+``` text
+{service_name="mattermost"} | json | status_code >= 500
+```
+
+**Top 5 error messages over 5-minute windows:**
+
+``` text
+topk(5, sum(count_over_time({service_name="mattermost"} | json | detected_level="error" [5m])) by (msg))
+```
+
+**Count errors by message over 5-minute windows:**
+
+``` text
+sum(count_over_time({service_name="mattermost"} | json | detected_level="error" [5m])) by (msg)
+```
+
+**Free-text search (e.g., plugin errors):**
+
+``` text
+{service_name="mattermost"} |~ "plugin"
+```
+
+**Authentication-related log lines:**
+
+``` text
+{service_name="mattermost"} |~ "(?i)(auth|login|token|session)"
+```
+
+**Logs from a specific server instance:**
+
+``` text
+{service_name="mattermost", service_instance_id="mm-app-01"}
+```
+
+<Tip>
+
+For a full LogQL reference, see the [Grafana Loki LogQL documentation](https://grafana.com/docs/loki/latest/query/).
+
+</Tip>
+
+## Optional: Add PostgreSQL log collection
+
+If your PostgreSQL database runs on a dedicated server (not RDS), you can ship its logs to Loki as well. This is useful for correlating slow queries or database errors with Mattermost application events.
+
+<Note>
+
+A separate configuration file is provided for PostgreSQL log collection. Install the OpenTelemetry Collector on the PostgreSQL server using the same steps from **Step 3**, then use this configuration file instead of the Mattermost one:
+
+- [Download otel-collector-config-postgres.yaml](/files/samples/loki/otel-collector-config-postgres.yaml)
+
+The PostgreSQL configuration handles common Ubuntu/Debian JSON log paths (`/var/log/postgresql/*.json`). Replace the same `<LOKI_HOST>`, `<HOSTNAME>`, and `<SERVICE_NAME>` placeholders as described in Step 3.
+
+</Note>
+
+Once running, PostgreSQL logs appear in Grafana under the label `&#123;service_name="postgres"&#125;`:
+
+``` text
+{service_name="postgres"} | json | detected_level="error"
+```
+
+## Verification and troubleshooting
+
+**End-to-end verification checklist:**
+
+1.  Loki is healthy:
+
+    ``` bash
+    curl -s http://<monitoring-server>:3100/ready
+    # Expected: "ready"
+    ```
+
+2.  OpenTelemetry Collector is shipping logs (run on each Mattermost server):
+
+    ``` bash
+    sudo journalctl -u otelcol-contrib -n 100
+    # Look for export successful messages or lack of errors
+    ```
+
+3.  Logs are visible in Grafana: Go to **Explore**, select the **Loki** data source, and run `&#123;service_name="mattermost"&#125;`. You should see recent log lines.
+
+4.  The dashboard loads: Open the **Mattermost Log Aggregation** dashboard and confirm panels are populated. (Note: You may need to update dashboard queries to use <code>service_name</code> and <code>service_instance_id</code>).
+
+**Common issues:**
+
+<table>
+<colgroup>
+<col style={{width: '40%'}} />
+<col style={{width: '60%'}} />
+</colgroup>
+<thead>
+<tr>
+<th>Symptom</th>
+<th>Resolution</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>OTel Col target shows errors</td>
+<td>Verify the file paths in the config match the actual log file location. Check file permissions (<code>otelcol-contrib</code> user must be able to read the file).</td>
+</tr>
+<tr>
+<td><code>connection refused</code> on port 3100</td>
+<td>Ensure Loki is running (<code>systemctl status loki</code>). Check firewall rules: <code>sudo iptables -L -n | grep 3100</code> or check your AWS security group allows TCP 3100 from the Mattermost servers.</td>
+</tr>
+<tr>
+<td>Logs appear in Loki but fields aren't parsed</td>
+<td>Confirm Mattermost is writing JSON-formatted logs (see Step 1). Check that the <code>json_parser</code> operator (for Postgres) or OTel processors are correctly configured.</td>
+</tr>
+<tr>
+<td><code>/ready</code> returns an error</td>
+<td>Check Loki logs: <code>sudo journalctl -u loki -f</code>. Common cause: permissions on <code>/opt/loki/data/</code> — ensure the <code>loki</code> user owns the directory.</td>
+</tr>
+<tr>
+<td>Old logs are not being deleted</td>
+<td>Verify <code>compactor.retention_enabled: true</code> and <code>limits_config.retention_period</code> are both set in the Loki config. The compactor runs on the <code>compaction_interval</code> (default: 10 minutes) and applies a <code>retention_delete_delay</code> (default: 2 hours) before actually removing data.</td>
+</tr>
+<tr>
+<td>High disk usage on the monitoring server</td>
+<td>Review the retention period in the configuration. Consider reducing it or adding more disk. Check <code>du -sh /opt/loki/data/</code> to see current usage.</td>
+</tr>
+</tbody>
+</table>