Skip to content

Monitoring/Observability ​

This guide will help you to install and configure tools to monitor your PeerTube platform to store and analyze the following components:

  • Metrics (PeerTube stats, OS CPU, PostgreSQL queries etc)
  • PeerTube Logs
  • PeerTube traces (details of CPU time spent in PeerTube functions, more suited for developers)

It will use the following software that can be installed on another server:

  • Grafana to visualize data from Prometheus, Tempo and Loki
  • Prometheus to store and query OS, PeerTube, PostgreSQL and Redis metrics
  • Tempo to store and query PeerTube traces
  • Loki to store and query PeerTube logs
  • Grafana Alloy to send PeerTube logs to Loki

Installing Prometheus, Tempo, Loki and Grafana is out of the scope of this guide.

OS/PeerTube metrics in Prometheus ​

PeerTube follows OpenTelemetry standard and can export its metrics in Prometheus format. A Prometheus compatible scraper can then fetch these metrics.

Here is a non exhaustive list of metrics you may find interesting:

  • Viewers count on local/remote videos
  • Viewer network stats (HTTP download and P2P download/upload)
  • Viewer playback errors
  • HTTP requests (timing, count etc)
  • Job queue information (waiting/active jobs etc)
  • ...

Installing other Prometheus exporters like PostgreSQL exporter, Redis exporter and Node (OS metrics) exporter will allow us to create a detailed Grafana dashboard.

You can choose to only install one of these exporters or just use metrics exported by PeerTube if you want.

Enable PeerTube metrics exporter ​

Update your PeerTube configuration and set open_telemetry.metrics.enabled to true. After a restart, your PeerTube platform will expose its metrics on http://127.0.0.1:9091/metrics (see open_telemetry.metrics.prometheus_exporter to change the hostname/port). We'll expose them to your Prometheus server with authentication using NGINX later in this guide.

HTTP request metrics (requests count, duration, status codes etc.) are disabled by default because they can have a high cardinality. Set open_telemetry.metrics.http_request_duration.enabled to true to enable them.

Install other Prometheus exporters ​

Install and configure PostgreSQL, Redis and Node exporters on your PeerTube server. For example on Debian Trixie:

bash
apt install prometheus-postgres-exporter prometheus-node-exporter prometheus-redis-exporter apache2-utils

Configure the exporters:

  • prometheus-postgres-exporter

    • Set the following in /etc/default/prometheus-postgres-exporter
    bash
    DATA_SOURCE_NAME='user=prometheus host=/run/postgresql dbname=postgres'
    ARGS='--web.listen-address="127.0.0.1:9187"'
    • Configure PostgreSQL using sudo -u postgres psql
    sql
    -- To use IF statements, hence to be able to check if the user exists before
    -- attempting creation, we need to switch to procedural SQL (PL/pgSQL)
    -- instead of standard SQL.
    -- More: https://www.postgresql.org/docs/9.3/plpgsql-overview.html
    -- To preserve compatibility with <9.0, DO blocks are not used; instead,
    -- a function is created and dropped.
    CREATE OR REPLACE FUNCTION __tmp_create_user() returns void as $$
    BEGIN
      IF NOT EXISTS (
              SELECT                       -- SELECT list can stay empty for this
              FROM   pg_catalog.pg_user
              WHERE  usename = 'prometheus') THEN
        CREATE USER prometheus;
      END IF;
    END;
    $$ language plpgsql;
    
    SELECT __tmp_create_user();
    DROP FUNCTION __tmp_create_user();
    
    ALTER USER prometheus SET SEARCH_PATH TO prometheus,pg_catalog;
    
    -- If deploying as non-superuser (for example in AWS RDS), uncomment the GRANT
    -- line below and replace <MASTER_USER> with your root user.
    -- GRANT prometheus TO <MASTER_USER>;
    
    GRANT CONNECT ON DATABASE postgres TO prometheus;
    GRANT pg_monitor to prometheus;
  • prometheus-node-exporter

    • Set the following in /etc/default/prometheus-node-exporter
bash
ARGS='--web.listen-address="127.0.0.1:9200"'
  • prometheus-redis-exporter
    • Set the following in /etc/default/prometheus-redis-exporter
bash
ARGS='--web.listen-address="127.0.0.1:9121"'

NGINX configuration ​

Now we'll configure the reverse proxy to expose these exporter metrics with authentication.

First, we generate the htpasswd file. Run as root:

bash
mkdir /etc/nginx/htaccess
htpasswd -c /etc/nginx/htaccess/peertube_prometheus scraper

Add the following NGINX configuration at the beginning of your /etc/nginx/sites-enabled/peertube file:

# PeerTube Prometheus exporter
server {
  listen 19091;
  listen [::]:19091;

  auth_basic           "Auth";
  auth_basic_user_file /etc/nginx/htaccess/peertube_prometheus;

  location / {
    proxy_pass http://localhost:9091;
  }
}

# PostgreSQL Prometheus exporter
server {
  listen 19187;
  listen [::]:19187;

  auth_basic           "Auth";
  auth_basic_user_file /etc/nginx/htaccess/peertube_prometheus;

  location / {
    proxy_pass http://localhost:9187;
  }
}

# Node server Prometheus exporter
server {
  listen 19200;
  listen [::]:19200;

  auth_basic           "Auth";
  auth_basic_user_file /etc/nginx/htaccess/peertube_prometheus;

  location / {
    proxy_pass http://localhost:9200;
  }
}

# Redis Prometheus exporter
server {
  listen 19121;
  listen [::]:19121;

  auth_basic           "Auth";
  auth_basic_user_file /etc/nginx/htaccess/peertube_prometheus;

  location / {
    proxy_pass http://localhost:9121;
  }
}

INFO

These ports don't conflict with the ones used by horizontal scalability secondaries (9100+ for listen.port and 9092+ for their Prometheus exporters). If you run secondaries, expose and scrape the Prometheus exporter of each PeerTube process.

Test your metrics ​

Run your exporters and reload nginx as root:

bash
systemctl restart peertube prometheus-postgres-exporter.service prometheus-redis-exporter.service prometheus-node-exporter.service
nginx -t && systemctl reload nginx

You can check metrics are correctly exposed using the following endpoints:

Configure Prometheus ​

Now your metrics are correctly exported, you can configure your Prometheus server to scrape them. Add the following scrape configuration in your Prometheus configuration (replace peertube.example.com and your-password strings):

yaml
scrape_configs:
  - job_name: 'peertube'

    basic_auth:
      username: scraper
      password: your-password

    static_configs:
      - targets: [ 'peertube.example.com:19091' ]
        labels:
          instance: 'peertube.example.com'


  - job_name: 'postgresql'

    basic_auth:
      username: scraper
      password: your-password

    static_configs:
      - targets: [ 'peertube.example.com:19187' ]
        labels:
          instance: 'peertube.example.com'


  - job_name: 'node'

    basic_auth:
      username: scraper
      password: your-password

    static_configs:
      - targets: [ 'peertube.example.com:19200' ]
        labels:
          instance: 'peertube.example.com'


  - job_name: 'redis'

    basic_auth:
      username: scraper
      password: your-password

    static_configs:
      - targets: [ 'peertube.example.com:19121' ]
        labels:
          instance: 'peertube.example.com'

Save and restart prometheus:

systemctl restart prometheus

Your metrics can now be queried on your Prometheus instance.

INFO

The Grafana dashboard provided at the end of this guide expects these exact job names (peertube, postgresql, node and redis) and the same instance label on every job.

PeerTube logs in Loki ​

PeerTube is not able to send its logs to a third-party yet.

To send PeerTube logs to Loki, you will need Grafana Alloy on your PeerTube server.

INFO

Alloy replaces Promtail, which is deprecated. If you still use Promtail, Grafana provides a migration guide.

Install Alloy on your PeerTube server following the official documentation (it provides a Debian package and a systemd service).

Then replace the content of /etc/alloy/config.alloy with the following configuration (replace peertube.example.com, loki.example.com and your-password strings):

local.file_match "peertube" {
  path_targets = [{
    __path__ = "/var/www/peertube/storage/logs/peertube*.log",
    job      = "peertube",
    instance = "peertube.example.com",
  }]
}

loki.source.file "peertube" {
  targets    = local.file_match.peertube.targets
  forward_to = [loki.process.peertube.receiver]
}

loki.process "peertube" {
  stage.json {
    expressions = {
      timestamp = "timestamp",
      level     = "level",
    }
  }

  // Audit logs are also written in peertube-audit.log
  stage.drop {
    source = "level"
    value  = "audit"
  }

  stage.labels {
    values = {
      level = "",
    }
  }

  stage.timestamp {
    source = "timestamp"
    format = "RFC3339"
  }

  forward_to = [loki.write.default.receiver]
}

loki.write "default" {
  endpoint {
    url = "http://loki.example.com:3100/loki/api/v1/push"

    // If you use basic auth on your Loki instance using a reverse proxy for example
    basic_auth {
      username = "loki"
      password = "your-password"
    }
  }
}

The alloy system user must be able to read PeerTube log files. If it's not the case, add it to the peertube group as root:

bash
usermod -a -G peertube alloy

Enable and restart Alloy as root:

bash
systemctl enable alloy
systemctl restart alloy

Your logs are now sent to your Loki instance.

PeerTube traces in Tempo ​

Traces are interesting to debug slow PeerTube platforms and to help developers to find slow paths in PeerTube code. They can be automatically sent by PeerTube to an OpenTelemetry (OTLP over HTTP) compatible endpoint (like Tempo in this guide, or Jaeger).

Configure PeerTube to send traces ​

Update your PeerTube configuration:

  • Set open_telemetry.tracing.enabled to true
  • Set your OTLP endpoint in open_telemetry.tracing.otlp_exporter.endpoint, for example http://tempo.example.com:4318/v1/traces (the OTLP HTTP receiver must be enabled in your Tempo configuration)

PeerTube must also be started with the --experimental-loader=@opentelemetry/instrumentation/hook.mjs node option. If you use the PeerTube systemd service, run systemctl edit peertube as root and add:

[Service]
Environment=NODE_OPTIONS=--experimental-loader=@opentelemetry/instrumentation/hook.mjs

After a PeerTube restart, your traces can now be queried on your Tempo instance.

Visualize data in Grafana ​

Grafana can be easily configured to display graphs, logs, information from Prometheus, Loki and Tempo. Since creating these dashboard could be long, you can import this dashboard in your Grafana instance (Grafana >= 10 is required): monitoring.json.

When importing it, Grafana will ask you to select your Prometheus, Loki and Tempo data sources. Then select your PeerTube instance at the top of the dashboard, and set the Database name variable if your PeerTube database is not peertube_prod.

Some panels need additional configuration and stay empty otherwise:

  • HTTP requests panels: open_telemetry.metrics.http_request_duration.enabled must be set to true in PeerTube configuration
  • PostgreSQL and Redis rows: the corresponding Prometheus exporter must be installed
  • Read replica panels (in the PostgreSQL row): database read replicas must be configured
  • Remote runners row: you must use remote runners
  • Akismet comments panel: the peertube-plugin-akismet plugin must be installed
  • Player metrics row: viewers must play videos using the PeerTube web player