--- title: "Calls Offloader Setup and Configuration" --- This guide provides detailed instructions for setting up, configuring, and validating the Mattermost calls-offloader service used for call recording and transcription features. - [Overview](#overview) - [Prerequisites](#prerequisites) - [Installation and deployment](#installation-and-deployment) - [Configuration](#configuration) - [Validation and testing](#validation-and-testing) - [Integration with Mattermost](#integration-with-mattermost) - [Troubleshooting](#troubleshooting) - [Air-Gapped Deployments](#air-gapped-installation-of-calls-offloader) ## Overview The calls-offloader service is a dedicated microservice that handles resource-intensive tasks for Mattermost Calls, including: - **Call recording**: Captures audio and screen sharing content from calls - **Call transcription**: Provides automated transcription of recorded calls - **Live captions** (Experimental): Real-time transcription during active calls By offloading these tasks to a dedicated service, the main Mattermost server and RTCD service can focus on core functionality while maintaining optimal performance. ## Prerequisites Before deploying calls-offloader, ensure you have: - A Mattermost Enterprise license - A properly configured Mattermost Calls deployment (either integrated or with RTCD) - A moderately powerful server with Docker installed and running - Sufficient storage space for recordings (see [Storage Requirements](#storage-requirements)) ### System Requirements For detailed system requirements and performance recommendations, refer to the [calls-offloader performance documentation](https://github.com/mattermost/calls-offloader/blob/master/docs/performance.md). ### Storage Requirements Call recordings can consume significant storage space. Based on average recording sizes with screen sharing on (including one audio track), storage usage by quality chosen is approximately: - **Low**: ~0.5GB/hour (or ~8MB/minute) - **Medium**: ~0.7GB/hour (or ~12MB/minute) - **High**: ~1.2GB/hour (or ~20MB/minute) *Note: Audio-only recordings consume approximately 1MB per minute per participant.* ## Installation and Deployment ### Bare Metal or VM Deployment Looking for an automated setup? Check out these community-maintained [Calls Installation Scripts](https://github.com/bgardner8008/calls-install-scripts) for quick provisioning of the Calls Offloader service on Ubuntu/Debian systems. 1. Download the latest release from the [calls-offloader GitHub repository](https://github.com/mattermost/calls-offloader/releases) 2. Create the necessary directories: ```bash sudo mkdir -p /opt/calls-offloader/data/db sudo useradd --system --home /opt/calls-offloader calls-offloader sudo chown -R calls-offloader:calls-offloader /opt/calls-offloader ``` 3. Create a configuration file (`/opt/calls-offloader/config.toml`): ```toml [api] http.listen_address = ":4545" http.tls.enable = false http.tls.cert_file = "" http.tls.cert_key = "" security.allow_self_registration = true security.enable_admin = false security.admin_secret_key = "" security.session_cache.expiration_minutes = 1440 [store] data_source = "/opt/calls-offloader/data/db" [jobs] api_type = "docker" max_concurrent_jobs = 100 failed_jobs_retention_time = "30d" image_registry = "mattermost" [logger] enable_console = true console_json = false console_level = "INFO" enable_file = true file_json = true file_level = "DEBUG" file_location = "/opt/calls-offloader/calls-offloader.log" enable_color = true ``` 4. Create a systemd service file (`/etc/systemd/system/calls-offloader.service`): ```ini [Unit] Description=Mattermost Calls Offloader Service After=network.target docker.service Requires=docker.service [Service] Type=simple User=calls-offloader WorkingDirectory=/opt/calls-offloader ExecStart=/opt/calls-offloader/calls-offloader --config /opt/calls-offloader/config.toml Restart=always RestartSec=10 LimitNOFILE=65536 [Install] WantedBy=multi-user.target ``` 5. Enable and start the service: ```bash sudo systemctl daemon-reload sudo systemctl enable calls-offloader sudo systemctl start calls-offloader ``` 6. Check the service status: ```bash sudo systemctl status calls-offloader ``` 7. Verify the service is responding: ```bash curl http://localhost:4545/version # Example output: # {"buildDate":"2025-03-10 19:13","buildVersion":"v0.9.2","buildHash":"a4bd418","goVersion":"go1.23.6"} ``` ## Configuration ### API Configuration The API section controls how the service accepts requests: - **http.listen_address**: The address and port where the service listens (default: `:4545`) - **http.tls.enable**: Whether to use TLS encryption for the API - **security.allow_self_registration**: Allow clients to self-register for job management - **security.enable_admin**: Enable admin functionality - **security.admin_secret_key**: Secret key for admin authentication (change from default!) ### Store Configuration Controls persistent data storage: - **data_source**: Path to directory for storing client (i.e., connecting Mattermost nodes) IDs and credentials ### Jobs Configuration Controls job processing behavior: - **api_type**: Job execution backend (`docker` or `kubernetes`) - **max_concurrent_jobs**: Maximum number of simultaneous recording/transcription jobs - **failed_jobs_retention_time**: How long to keep failed job data before cleanup - **image_registry**: Docker registry for job runner images (typically `mattermost`) ### Logger Configuration Controls logging output: - **enable_console**: Log to console output - **console_json**: Use JSON format for console logs - **console_level**: Log level for console (DEBUG, INFO, WARN, ERROR) - **enable_file**: Log to file - **file_location**: Path to log file - **enable_color**: Use colored output for console logs ### Private Network Configuration When the Mattermost deployment is running in a private network, additional configuration may be necessary for the jobs spawned by the calls-offloader service to reach the Mattermost server. In such cases, you can override the site URL used by recorder jobs or transcriber jobs to connect to Mattermost by setting the following environment variables on the Mattermost server: - **MM_CALLS_RECORDER_SITE_URL**: Override the site URL used by recording jobs - **MM_CALLS_TRANSCRIBER_SITE_URL**: Override the site URL used by transcription jobs When these Site URL overrides are used, the `ServiceSettings.AllowCorsFrom` setting on your Mattermost server may need to be adjusted accordingly to ensure CORS does not block requests. This override configuration lets the recorder and transcriber jobs connect to mattermost server using HTTP instead of HTTPS, which should only be used in a private network. Example configuration: Create or edit the Mattermost environment file (`/opt/mattermost/config/mattermost.environment`): ```bash MM_CALLS_RECORDER_SITE_URL="http://internal-mattermost-server:8065" MM_CALLS_TRANSCRIBER_SITE_URL="http://internal-mattermost-server:8065" ``` Then ensure your Mattermost systemd service references this environment file: ```ini [Unit] Description=Mattermost After=network.target [Service] Type=notify EnvironmentFile=/opt/mattermost/config/mattermost.environment ExecStart=/opt/mattermost/bin/mattermost TimeoutStartSec=3600 KillMode=mixed Restart=always RestartSec=10 WorkingDirectory=/opt/mattermost User=mattermost Group=mattermost [Install] WantedBy=multi-user.target ``` This is particularly useful when: - The calls-offloader service runs in a different network segment than clients - Internal DNS resolution differs from external URLs - You need to use internal load balancer endpoints for job communication ## Validation and Testing After deploying calls-offloader, validate the installation: 1. **Check service status**: ```bash # For systemd sudo systemctl status calls-offloader ``` 2. **Test API connectivity**: **From the calls-offloader server (localhost test)**: ```bash curl http://localhost:4545/version # Should return version information # Example: {"buildDate":"2025-03-10 19:13","buildVersion":"v0.9.2","buildHash":"a4bd418","goVersion":"go1.23.6"} ``` **From the Mattermost server**: ```bash curl http://YOUR_CALLS_OFFLOADER_SERVER:4545/version # Should return the same version information # This confirms network connectivity from Mattermost to calls-offloader ``` If the localhost test works but the Mattermost server test fails, check: - Firewall rules or SELinux policies on the calls-offloader server (port 4545 must be accessible) - Network connectivity between Mattermost and calls-offloader servers 3. **Verify Docker service** (if using docker api_type): ```bash # Check that system user running calls-offloader can access Docker sudo -u calls-offloader docker ps ``` ## Integration with Mattermost Once calls-offloader is properly set up and validated, configure Mattermost to use it: 1. Go to **System Console > Plugins > Calls** 2. In the **Job Service** section: - Set **Job Service URL** to your calls-offloader service (e.g., `http://calls-offloader-server:4545`) 3. Enable recording and transcription features as needed: - **Enable Call Recordings**: Toggle to allow call recordings - **Enable Call Transcriptions**: Toggle to allow call transcriptions - **Enable Live Captions** (Experimental): Toggle to allow real-time transcription 4. Save the configuration 5. Restart the Calls plugin to re-establish state: - Go to **System Console > Plugins > Plugin Management** - Find the **Calls** plugin and click **Disable** - Wait a few seconds, then click **Enable** 6. Test by starting a call and starting a recording ## Troubleshooting ### Common Issues **"failed to create recording job: max concurrent jobs reached"** This error occurs when the calls-offloader service has reached its configured job limit, and it will usually result in a failure message on the Mattermost Calls plugin side (such as a timeout). Solutions: - Increase `max_concurrent_jobs` in the configuration - Check if jobs are hanging and restart the service - Monitor system resources and scale up if needed **Jobs not processing** Check the following: - Verify the calls-offloader service is running: `sudo systemctl status calls-offloader` - Ensure network connectivity between Mattermost and calls-offloader - Check Docker daemon is running and accessible by the user running the Calls Offloader service (E.g., user: ``calls-offloader``) - Verify authentication configuration matches between services - Review service logs for specific error messages **Docker permission issues** If using Docker API and seeing permission errors: ```bash # Add calls-offloader user to docker group sudo usermod -a -G docker calls-offloader sudo systemctl restart calls-offloader ``` ### Debugging Commands Monitor calls-offloader job containers: ```bash # View running job containers docker ps --format "{{.ID}} {{.Image}}" | grep "calls" # Follow logs for debugging docker ps --format "{{.ID}} {{.Image}}" | grep "calls" | awk '{print $1}' | xargs -I {} docker logs -f {} # View completed job containers docker ps -a --filter "status=exited" ``` Monitor service health: ```bash # Check service version and health curl http://localhost:4545/version ``` Check service logs: ```bash # View recent logs sudo journalctl -u calls-offloader -f # View log file (if file logging enabled) tail -f /opt/calls-offloader/calls-offloader.log ``` ### Performance Monitoring Monitor calls-offloader performance and resource usage to ensure optimal operation. See [Calls Metrics and Monitoring](calls-metrics-monitoring.mdx) for details on setting up metrics and observability. ## Air-Gapped Installation of `calls-offloader` This guide covers deploying `calls-offloader` in an environment without internet access. The process uses scripts from the [calls-install-scripts](https://github.com/bgardner8008/calls-install-scripts) repository and follows a two-phase workflow: preparing a transfer bundle on an internet-connected machine, then deploying it on the isolated target machine. ### Overview Because `calls-offloader` relies on Docker images for the recorder and transcriber jobs, an air-gapped deployment requires that those images be pre-pulled and packaged alongside the `calls-offloader` binary before being transferred to the target environment. The install scripts handle this in two stages: 1. **Prepare** (internet-connected machine) — `setup-airgap-offloader.sh` pulls the required Docker images and downloads the `calls-offloader` binary, then packages everything into a transfer bundle and generates a ready-to-run deployment script. 2. **Deploy** (air-gapped machine) — Transfer the bundle and run the generated `deploy-airgap-offloader.sh` script, which loads the Docker images into a local registry and installs the service. ### Prerequisites - An internet-connected Linux machine with Docker installed (for the preparation phase) - A target air-gapped Linux machine with: - systemd - Docker installed - Root or sudo access - The [calls-install-scripts](https://github.com/bgardner8008/calls-install-scripts) repository cloned on the internet-connected machine ### Phase 1: Prepare the Transfer Bundle On the **internet-connected machine**, run `setup-airgap-offloader.sh` specifying the versions of each component to package: ```bash ./setup-airgap-offloader.sh \ --offloader v0.9.5 \ --recorder v0.9.0 \ --transcriber v0.3.0 \ --arch amd64 ``` | Flag | Description | Default | |------|-------------|---------| | `--offloader VERSION` | `calls-offloader` binary version (e.g. `v0.9.5`) | required | | `--recorder VERSION` | `calls-recorder` Docker image version | required | | `--transcriber VERSION` | `calls-transcriber` Docker image version | required | | `--arch amd64\|arm64` | Target CPU architecture | `amd64` | The script will: 1. Pull the `calls-recorder` and `calls-transcriber` Docker images from Docker Hub 2. Download the `calls-offloader` binary from GitHub releases 3. Save the Docker images as `.tar` archives 4. Generate a `deploy-airgap-offloader.sh` deployment script configured for the selected versions 5. Produce a transfer bundle containing all of the above ### Phase 2: Transfer to the Air-Gapped Machine Copy the generated bundle to the target machine using whatever transfer mechanism is available in your environment (USB drive, secure file transfer, etc.): ```bash scp calls-offloader-airgap-bundle.tar.gz user@airgap-host:/tmp/ ``` On the air-gapped machine, extract the bundle: ```bash tar -xzf calls-offloader-airgap-bundle.tar.gz cd calls-offloader-airgap-bundle/ ``` ### Phase 3: Deploy on the Air-Gapped Machine Run the generated deployment script with root or sudo privileges: ```bash sudo ./deploy-airgap-offloader.sh ``` This script will: 1. Configure the local Docker daemon to use a local image registry 2. Load the packaged Docker images into that registry 3. Install the `calls-offloader` binary to `/usr/local/bin/` 4. Create the `mattermost` system user and add it to the `docker` group 5. Generate and enable a systemd service unit 6. Start the service and verify it is running Once complete, verify the service is up: ```bash curl http://localhost:4545/version ``` ### Connecting to Mattermost Configure the Mattermost Calls plugin to use the offloader service via **System Console > Plugins > Calls > Job service URL**, setting it to `http://:4545`. > [!NOTE] > The first time Mattermost connects to the offloader it will self-register and store its authentication key in the database, provided `API_SECURITY_ALLOWSELFREGISTRATION=true` is set (the default in the deployment script). ### Private Network Considerations In air-gapped environments the recorder and transcriber containers typically need to reach the Mattermost server via an internal URL. Set the following environment variables on the **Mattermost server** to override the site URL used by spawned jobs: ``` MM_CALLS_RECORDER_SITE_URL=http://internal-mattermost-server:8065 MM_CALLS_TRANSCRIBER_SITE_URL=http://internal-mattermost-server:8065 ``` You may also need to add the internal URL to [`ServiceSettings.AllowCorsFrom`](https://docs.mattermost.com/configure/integrations-configuration-settings.html#enable-cross-origin-requests-from) in the Mattermost server configuration. > [!NOTE] > In particularly restrictive environments (e.g., VMs with strict network isolation), set `DOCKER_NETWORK=host` in the `calls-offloader` service environment so that job containers can reach the Mattermost server via its local address. ### Custom Docker Registry If your air-gapped environment already has an internal Docker registry, you can point `install-offloader.sh` at it directly instead of using the local registry set up by the deployment script: ```bash sudo ./install-offloader.sh \ --binary ./calls-offloader-linux-amd64 \ --image-registry registry.internal.example.com/mattermost \ --arch amd64 ``` The `--image-registry` flag sets the registry prefix used when the offloader pulls recorder and transcriber images for each job. ## Other Calls Documentation - [Calls Deployment Guide](calls-deployment-guide.mdx): Overview of deployment options and architecture - [RTCD Setup and Configuration](calls-rtcd-setup.mdx): Comprehensive guide for setting up the dedicated RTCD service - [Calls Metrics and Monitoring](calls-metrics-monitoring.mdx): Guide to monitoring Calls performance using metrics and observability - [Calls Deployment on Kubernetes](calls-kubernetes.mdx): Detailed guide for deploying Calls in Kubernetes environments - [Calls Logging](calls-logging.mdx): Detailed guidance for collecting Calls logs and client diagnostics - [calls-offloader performance documentation](https://github.com/mattermost/calls-offloader/blob/master/docs/performance.md): Detailed performance tuning and monitoring recommendations