upstream-mattermost/docs/main/administration-guide/scale/deploy-grafana-loki-for-centralized-logging.mdx

---
title: "Deploy Grafana Loki for centralized logging"
---
<PlanAvailability slug="entry-ent" />

This guide extends your existing [Prometheus and Grafana performance monitoring deployment](/administration-guide/scale/deploy-prometheus-grafana-for-performance-monitoring) by adding [Grafana Loki](https://grafana.com/oss/loki/) for centralized log aggregation. While Prometheus collects **metrics** (CPU, request latency, goroutine counts), Loki collects **logs** — giving your operations team the ability to search, filter, and correlate Mattermost application logs across all servers from a single Grafana interface.

With Loki in place you can:

- Search Mattermost logs across all application servers from Grafana.
- Filter by log level, HTTP status code, or free-text search.
- Identify the top recurring errors and warnings.
- Correlate log events with Prometheus metric spikes on the same timeline.
- Optionally collect PostgreSQL database logs alongside application logs.

<Tip>

This guide assumes you have already deployed Prometheus and Grafana by following the [performance monitoring guide](/administration-guide/scale/deploy-prometheus-grafana-for-performance-monitoring). Loki and the OpenTelemetry Collector will be added to that existing infrastructure.

</Tip>

## Architecture overview

The deployment adds two components to your monitoring stack:

- **Loki** — the log aggregation engine, installed on your existing Grafana/Prometheus monitoring server.
- **OpenTelemetry Collector** — a versatile, industry-standard agent installed on each Mattermost application server to ship logs.

``` text
┌──────────────────────┐       ┌──────────────────────┐
│  Mattermost App 01   │       │  Mattermost App 02   │
│                      │       │                      │
│  /opt/mattermost/    │       │  /opt/mattermost/    │
│   logs/mattermost.log│       │   logs/mattermost.log│
│         │            │       │         │            │
│    [ OTel Col ]      │       │    [ OTel Col ]      │
└─────────┬────────────┘       └─────────┬────────────┘
          │ push (:3100/otlp)            │
          ▼                              ▼
┌─────────────────────────────────────────────────────┐
│           Monitoring Server                         │
│                                                     │
│   [ Prometheus :9090 ]    [ Loki :3100 ]            │
│           │                     │                   │
│           └──────┬──────────────┘                   │
│                  ▼                                  │
│           [ Grafana :3000 ]                         │
│            Metrics + Logs                           │
└─────────────────────────────────────────────────────┘

Optional: Add the OpenTelemetry Collector on your PostgreSQL server to also
ship database logs to Loki.
```

## Prerequisites

Before starting, confirm the following:

- Prometheus and Grafana are deployed and collecting Mattermost metrics per the [performance monitoring guide](/administration-guide/scale/deploy-prometheus-grafana-for-performance-monitoring).
- You have **SSH access** with sudo privileges to the monitoring server and each Mattermost application server.
- Mattermost is installed at `/opt/mattermost` (the default location). If your installation path differs, substitute your path wherever `/opt/mattermost` appears.
- **Network connectivity**: Each Mattermost server can reach the monitoring server on **TCP port 3100** (Loki's HTTP API).

## Step 1: Verify Mattermost JSON logging

Mattermost must be writing JSON-formatted file logs for Loki queries to work. This is the default on all plans — verify it on each application server:

``` bash
tail -1 /opt/mattermost/logs/mattermost.log
```

A JSON-formatted line looks like:

``` json
{"timestamp":"2025-01-15T14:32:01.123Z","level":"info","msg":"Server is listening on [::]:8065","caller":"app/server.go:482"}
```

If you see plain-text output instead, go to **System Console \> Environment \> Logging** and set **Output file logs as JSON** to `true`, then restart Mattermost:

``` bash
sudo systemctl restart mattermost
```

## Step 2: Install Loki on the monitoring server

Install Loki on the same server that runs Grafana and Prometheus. All commands in this section are run **on the monitoring server**.

<Important>

The commands below use Loki version **3.4.2**. Check the [Loki releases page](https://github.com/grafana/loki/releases) for the latest version and substitute the version number as needed.

</Important>

1.  Create the Loki user, directories, and download the binary:

    ``` bash
    # Create a dedicated system user
    sudo useradd --system --no-create-home --shell /bin/false loki

    # Create directories
    sudo mkdir -p /opt/loki/data /opt/loki/bin

    # Download and extract Loki
    sudo apt-get install -y unzip
    cd /tmp
    curl -LO https://github.com/grafana/loki/releases/download/v3.4.2/loki-linux-amd64.zip
    unzip loki-linux-amd64.zip
    sudo mv loki-linux-amd64 /opt/loki/bin/loki
    sudo chmod +x /opt/loki/bin/loki
    ```

2.  Download and install the production configuration file:

    - [Download loki-config.yaml](/files/samples/loki/loki-config.yaml)

    ``` bash
    sudo cp loki-config.yaml /opt/loki/loki-config.yaml
    ```

    <div class="tip">

    <div class="title">

    Tip

    </div>

    **Log retention** is set to **14 days** by default. To change this, edit `/opt/loki/loki-config.yaml` and update the `retention_period` value under `limits_config`. Loki requires this value in hours — common values:

    - `336h` = 14 days (default)
    - `720h` = 30 days
    - `2160h` = 90 days
    - `8760h` = 365 days

    Longer retention increases disk usage. As a rough guide, expect 1–3 GB per day for a moderately active Mattermost deployment (varies with log volume). Monitor `/opt/loki/data/` after the first week to project storage needs. The `compactor.retention_enabled` setting must remain `true` for retention enforcement to work.

    </div>

3.  Set ownership:

    ``` bash
    sudo chown -R loki:loki /opt/loki
    ```

4.  Create a systemd service file:

    ``` bash
    sudo tee /etc/systemd/system/loki.service > /dev/null <<'EOF'
    [Unit]
    Description=Grafana Loki Log Aggregation
    Documentation=https://grafana.com/docs/loki/latest/
    After=network-online.target
    Wants=network-online.target

    [Service]
    Type=simple
    User=loki
    Group=loki
    ExecStart=/opt/loki/bin/loki -config.file=/opt/loki/loki-config.yaml
    Restart=on-failure
    RestartSec=5
    LimitNOFILE=65536
    StandardOutput=journal
    StandardError=journal

    [Install]
    WantedBy=multi-user.target
    EOF
    ```

5.  Start Loki:

    ``` bash
    sudo systemctl daemon-reload
    sudo systemctl enable --now loki
    ```

6.  Verify Loki is running:

    ``` bash
    sudo systemctl status loki
    curl -s http://localhost:3100/ready
    ```

    The `/ready` endpoint should return `ready`. Loki may take 15–20 seconds after startup before it reports ready — this warmup delay is normal.

## Step 3: Install OpenTelemetry Collector on each Mattermost server

The OpenTelemetry (OTel) Collector runs on each Mattermost application server and pushes logs to Loki. Repeat these steps on **every Mattermost application server**.

<Important>

Replace `<LOKI_HOST>` with the IP address or hostname of your monitoring server, `<HOSTNAME>` with this server's hostname (e.g., `mm-app-01`), and `<SERVICE_NAME>` with the service type (e.g., `mattermost` or `postgres`).

</Important>

1.  Install the OTel Collector Contrib distribution:

    ``` bash
    # Add OTel Collector repository and install
    # Commands below are for Ubuntu/Debian
    wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.145.0/otelcol-contrib_0.145.0_linux_amd64.deb
    sudo dpkg -i otelcol-contrib_0.145.0_linux_amd64.deb
    ```

    <div class="note">

    <div class="title">

    Note

    </div>

    The installation process automatically creates a system user and group named `otelcol-contrib`.

    </div>

2.  Download and edit the OpenTelemetry Collector configuration:

    - [Download otel-collector-config-mattermost.yaml](/files/samples/loki/otel-collector-config-mattermost.yaml)

    ``` bash
    sudo cp otel-collector-config-mattermost.yaml /etc/otelcol-contrib/config.yaml
    ```

    Edit `/etc/otelcol-contrib/config.yaml` and replace the placeholders:

    - Replace `<LOKI_HOST>` with your monitoring server's address.
    - Replace `<HOSTNAME>` with this server's hostname.
    - Replace `<SERVICE_NAME>` with the service type (e.g., `mattermost`).

    ``` bash
    sudo vi /etc/otelcol-contrib/config.yaml
    ```

3.  Grant the collector read access to the Mattermost log directory:

    ``` bash
    # Grant the collector access to read Mattermost logs.
    # Substitute 'mattermost' for the group that owns your log file.
    sudo usermod -aG mattermost otelcol-contrib

    # Ensure logs are group-readable
    sudo chmod 640 /opt/mattermost/logs/mattermost.log
    sudo chmod g+rx /opt/mattermost/logs
    ```

4.  Restart the service:

    ``` bash
    sudo systemctl restart otelcol-contrib
    sudo systemctl enable otelcol-contrib
    ```

5.  Verify the collector is running and shipping logs:

    ``` bash
    sudo systemctl status otelcol-contrib
    sudo journalctl -u otelcol-contrib -f
    ```

## Step 4: Add Loki as a data source in Grafana

1.  Log in to Grafana (default: `http://<monitoring-server>:3000`).
2.  Navigate to **Administration \> Data Sources \> Add data source**.
3.  Select **Loki** from the list.
4.  Configure the connection:
    - **Name**: `Loki`
    - **URL**: `http://localhost:3100` (since Loki is colocated with Grafana)
5.  Select **Save & test**. Grafana should confirm the data source is working.

## Step 5: Import the Mattermost Loki dashboard

A pre-built Grafana dashboard is recommended for monitoring your logs. This dashboard is provided as a basic example:

- [Download the Mattermost Loki Logs dashboard JSON](/files/samples/grafana-dashboards/mattermost-loki-logs.json)

<Note>

While the dashboard provides a useful starting point, users can gain full query flexibility using the **Explore** tab in Grafana to build custom LogQL queries on the fly.

</Note>

The dashboard provides:

- **Log Volume by Level** — stacked bar chart showing log rates by severity.
- **Error / Warning / Total counters** — stat panels for the selected time range.
- **HTTP 4xx/5xx count** — quick visibility into application-level HTTP errors.
- **Error Rate Over Time** — time series per instance for spotting error spikes.
- **Top Error Messages** — table ranking the most frequent errors.
- **Log Browser** — searchable, filterable log panel with level and free-text variables.

To import the dashboard:

1.  In Grafana, go to **Dashboards \> New \> Import**.
2.  Select **Upload JSON file** and choose the downloaded `mattermost-loki-logs.json` file.
3.  On the import screen, select your **Loki** data source from the dropdown.
4.  Select **Import**.

## Step 6: Useful LogQL queries

You can run these queries directly in **Grafana Explore** (select the Loki data source) or use them to build additional dashboard panels.

**Search all Mattermost logs:**

``` text
{service_name="mattermost"}
```

**Filter by log level:**

``` text
{service_name="mattermost"} | json | detected_level="error"
```

**Search for all HTTP 4xx responses:**

``` text
{service_name="mattermost"} | json | status_code >= 400
```

**Search for all HTTP 5xx responses:**

``` text
{service_name="mattermost"} | json | status_code >= 500
```

**Top 5 error messages over 5-minute windows:**

``` text
topk(5, sum(count_over_time({service_name="mattermost"} | json | detected_level="error" [5m])) by (msg))
```

**Count errors by message over 5-minute windows:**

``` text
sum(count_over_time({service_name="mattermost"} | json | detected_level="error" [5m])) by (msg)
```

**Free-text search (e.g., plugin errors):**

``` text
{service_name="mattermost"} |~ "plugin"
```

**Authentication-related log lines:**

``` text
{service_name="mattermost"} |~ "(?i)(auth|login|token|session)"
```

**Logs from a specific server instance:**

``` text
{service_name="mattermost", service_instance_id="mm-app-01"}
```

<Tip>

For a full LogQL reference, see the [Grafana Loki LogQL documentation](https://grafana.com/docs/loki/latest/query/).

</Tip>

## Optional: Add PostgreSQL log collection

If your PostgreSQL database runs on a dedicated server (not RDS), you can ship its logs to Loki as well. This is useful for correlating slow queries or database errors with Mattermost application events.

<Note>

A separate configuration file is provided for PostgreSQL log collection. Install the OpenTelemetry Collector on the PostgreSQL server using the same steps from **Step 3**, then use this configuration file instead of the Mattermost one:

- [Download otel-collector-config-postgres.yaml](/files/samples/loki/otel-collector-config-postgres.yaml)

The PostgreSQL configuration handles common Ubuntu/Debian JSON log paths (`/var/log/postgresql/*.json`). Replace the same `<LOKI_HOST>`, `<HOSTNAME>`, and `<SERVICE_NAME>` placeholders as described in Step 3.

</Note>

Once running, PostgreSQL logs appear in Grafana under the label `&#123;service_name="postgres"&#125;`:

``` text
{service_name="postgres"} | json | detected_level="error"
```

## Verification and troubleshooting

**End-to-end verification checklist:**

1.  Loki is healthy:

    ``` bash
    curl -s http://<monitoring-server>:3100/ready
    # Expected: "ready"
    ```

2.  OpenTelemetry Collector is shipping logs (run on each Mattermost server):

    ``` bash
    sudo journalctl -u otelcol-contrib -n 100
    # Look for export successful messages or lack of errors
    ```

3.  Logs are visible in Grafana: Go to **Explore**, select the **Loki** data source, and run `&#123;service_name="mattermost"&#125;`. You should see recent log lines.

4.  The dashboard loads: Open the **Mattermost Log Aggregation** dashboard and confirm panels are populated. (Note: You may need to update dashboard queries to use <code>service_name</code> and <code>service_instance_id</code>).

**Common issues:**

<table>
<colgroup>
<col style={{width: '40%'}} />
<col style={{width: '60%'}} />
</colgroup>
<thead>
<tr>
<th>Symptom</th>
<th>Resolution</th>
</tr>
</thead>
<tbody>
<tr>
<td>OTel Col target shows errors</td>
<td>Verify the file paths in the config match the actual log file location. Check file permissions (<code>otelcol-contrib</code> user must be able to read the file).</td>
</tr>
<tr>
<td><code>connection refused</code> on port 3100</td>
<td>Ensure Loki is running (<code>systemctl status loki</code>). Check firewall rules: <code>sudo iptables -L -n | grep 3100</code> or check your AWS security group allows TCP 3100 from the Mattermost servers.</td>
</tr>
<tr>
<td>Logs appear in Loki but fields aren't parsed</td>
<td>Confirm Mattermost is writing JSON-formatted logs (see Step 1). Check that the <code>json_parser</code> operator (for Postgres) or OTel processors are correctly configured.</td>
</tr>
<tr>
<td><code>/ready</code> returns an error</td>
<td>Check Loki logs: <code>sudo journalctl -u loki -f</code>. Common cause: permissions on <code>/opt/loki/data/</code> — ensure the <code>loki</code> user owns the directory.</td>
</tr>
<tr>
<td>Old logs are not being deleted</td>
<td>Verify <code>compactor.retention_enabled: true</code> and <code>limits_config.retention_period</code> are both set in the Loki config. The compactor runs on the <code>compaction_interval</code> (default: 10 minutes) and applies a <code>retention_delete_delay</code> (default: 2 hours) before actually removing data.</td>
</tr>
<tr>
<td>High disk usage on the monitoring server</td>
<td>Review the retention period in the configuration. Consider reducing it or adding more disk. Check <code>du -sh /opt/loki/data/</code> to see current usage.</td>
</tr>
</tbody>
</table>