English

Français

English

Français

Appearance

Monitoring/Observability ​

This guide will help you to install and configure tools to monitor your PeerTube instance to store and analyze the following components:

  • Metrics (PeerTube stats, OS CPU, PostgreSQL queries etc)
  • PeerTube Logs
  • PeerTube traces (details of CPU time spent in PeerTube functions, more suited for developers)

It will use the following software that can be installed on another server:

  • Grafana to visualize data from Prometheus, Tempo and Loki
  • Prometheus to store and query OS, PeerTube, PostgreSQL and Redis metrics
  • Tempo to store and query PeerTube traces
  • Loki to store and query PeerTube logs
  • Promtail to send PeerTube logs to Loki

Installing Prometheus, Tempo, Loki and Grafana is out of the scope of this guide.

OS/PeerTube metrics in Prometheus ​

PeerTube follows OpenTelemetry standard and can export its metrics in Prometheus format. A Prometheus compatible scraper can then fetch these metrics.

Here is a non exhaustive list of metrics you may find interesting:

  • Viewers count on local/remote videos
  • Viewer network stats (HTTP download and P2P download/upload)
  • Viewer playback errors
  • HTTP requests (timing, count etc)
  • Job queue information (waiting/active jobs etc)
  • ...

Installing other Prometheus exporters like PostgreSQL exporter, Redis exporter and Node (OS metrics) exporter will allow us to create a detailed Grafana dashboard.

You can choose to only install one of these exporters or just use metrics exported by PeerTube if you want.

Enable PeerTube metrics exporter ​

Update your PeerTube configuration and set open_telemetry.metrics.enabled to true. After a restart, your instance will expose its metrics on http://peertube.example.com:9091/metrics.

Install other Prometheus exporters ​

Install and configure PostgreSQL, Redis and Node exporters on your PeerTube server. For example on Debian Bullseye:

bash
apt install prometheus-postgres-exporter prometheus-node-exporter prometheus-redis-exporter

Configure the exporters:

  • prometheus-postgres-exporter

    • Set the following in /etc/default/prometheus-postgres-exporter
    bash
    DATA_SOURCE_NAME='user=prometheus host=/run/postgresql dbname=postgres'
ARGS='--web.listen-address="127.0.0.1:9187"'
    • Configure PostgreSQL using sudo -u postgres psql
    sql
    -- To use IF statements, hence to be able to check if the user exists before
-- attempting creation, we need to switch to procedural SQL (PL/pgSQL)
-- instead of standard SQL.
-- More: https://www.postgresql.org/docs/9.3/plpgsql-overview.html
-- To preserve compatibility with <9.0, DO blocks are not used; instead,
-- a function is created and dropped.
CREATE OR REPLACE FUNCTION __tmp_create_user() returns void as $$
BEGIN
  IF NOT EXISTS (
          SELECT                       -- SELECT list can stay empty for this
          FROM   pg_catalog.pg_user
          WHERE  usename = 'postgres_exporter') THEN
    CREATE USER postgres_exporter;
  END IF;
END;
$$ language plpgsql;

SELECT __tmp_create_user();
DROP FUNCTION __tmp_create_user();

ALTER USER postgres_exporter WITH PASSWORD 'password';
ALTER USER postgres_exporter SET SEARCH_PATH TO postgres_exporter,pg_catalog;

-- If deploying as non-superuser (for example in AWS RDS), uncomment the GRANT
-- line below and replace <MASTER_USER> with your root user.
-- GRANT postgres_exporter TO <MASTER_USER>;

GRANT CONNECT ON DATABASE postgres TO postgres_exporter;
GRANT pg_monitor to postgres_exporter;

  • prometheus-node-exporter

    • Set the following in /etc/default/prometheus-node-exporter
bash
ARGS='--web.listen-address="127.0.0.1:9100"'
  • prometheus-redis-exporter
    • Set the following in /etc/default/prometheus-redis-exporter
bash
ARGS='--web.listen-address="127.0.0.1:9121"'

NGINX configuration ​

Now we'll configure the reverse proxy to expose these exporter metrics with authentication.

First, we generate the htpasswd file. Run as root:

bash
mkdir /etc/nginx/htaccess
htpasswd -c /etc/nginx/htaccess/peertube_prometheus scraper

Add the following NGINX configuration at the beginning of your /etc/nginx/site-enabled/peertube file:

# PeerTube Prometheus exporter
server {
  listen 9092;
  listen [::]:9092;

  auth_basic           "Auth";
  auth_basic_user_file /etc/nginx/htaccess/peertube_prometheus;

  location / {
    proxy_pass http://localhost:9091;
  }
}

# PostgreSQL Prometheus exporter
server {
  listen 9188;
  listen [::]:9188;

  auth_basic           "Auth";
  auth_basic_user_file /etc/nginx/htaccess/peertube_prometheus;

  location / {
    proxy_pass http://localhost:9187;
  }
}

# Node server Prometheus exporter
server {
  listen 9101;
  listen [::]:9101;

  auth_basic           "Auth";
  auth_basic_user_file /etc/nginx/htaccess/peertube_prometheus;

  location / {
    proxy_pass http://localhost:9100;
  }
}

# Redis Prometheus exporter
server {
  listen 9122;
  listen [::]:9122;

  auth_basic           "Auth";
  auth_basic_user_file /etc/nginx/htaccess/peertube_prometheus;

  location / {
    proxy_pass http://localhost:9121;
  }
}

Test your metrics ​

Run your exporters and reload nginx as root:

systemctl restart peertube prometheus-postgres-exporter.service prometheus-redis-exporter.service prometheus-node-exporter.service
nginx -t && systemctl reload nginx

You can check metrics are correctly exposed using the following endpoints:

Configure Prometheus ​

Now your metrics are correctly exported, you can configure your Prometheus server to scrape them. Add the following scrape configuration in your Prometheus configuration (replace peertube.example.com and your-password strings):

yaml
scrape_configs:
  - job_name: 'peertube'

    basic_auth:
      username: scraper
      password: your-password

    static_configs:
      - targets: [ 'peertube.example.com:9092' ]
        labels:
          instance: 'peertube.example.com'


  - job_name: 'postgresql'

    basic_auth:
      username: scraper
      password: your-password

    static_configs:
      - targets: [ 'peertube.example.com:9188' ]
        labels:
          instance: 'peertube.example.com'


  - job_name: 'node'

    basic_auth:
      username: scraper
      password: your-password

    static_configs:
      - targets: [ 'peertube.example.com:9101' ]
        labels:
          instance: 'peertube.example.com'


  - job_name: 'redis'

    basic_auth:
      username: scraper
      password: your-password

    static_configs:
      - targets: [ 'peertube.example.com:9122' ]
        labels:
          instance: 'peertube.example.com'

Save and restart prometheus:

systemctl restart prometheus

You metrics can now be queried on your Prometheus instance.

PeerTube logs in Loki ​

PeerTube is not able to send its logs to a third-party yet.

To send PeerTube logs to Loki, you will need the promtail software.

Install promtail locally by downloading the binary on your PeerTube server: https://grafana.com/docs/loki/latest/clients/promtail/installation/#binary

Use the following configuration as promtail.yaml:

yaml
server:
  http_listen_port: 3101

clients:
  -
    basic_auth: # If you use basic auth on your Loki instance using a reverse proxy for example
      username: loki
      password: your-password
    url: http://loki.example.com:3100/loki/api/v1/push

positions:
  filename: /tmp/positions.yaml

target_config:
   sync_period: 10s

scrape_configs:
  - job_name: peertube
    static_configs:
      - targets:
          - localhost
        labels:
          job: peertube
          instance: peertube.example.com
          __path__: /var/www/peertube/storage/logs/peertube*.log
    pipeline_stages:
      - json:
          expressions:
            timestamp: timestamp
            level: level
      - drop:
          source: "level"
          value: "audit"
      - labels:
          level:
      - timestamp:
          format: RFC3339
          source: timestamp

Now we need to automatically start promtail. We can use systemd for that purpose. Create /etc/systemd/system/peertube-promtail.service file with the following content (replace your-user and /path/to/promtail-directory):

[Unit]
Description=Send logs to Loki
After=peertube.service

[Service]
Type=simple
User=your-user
Group=your-user
StandardOutput=journal
ExecStart=/path/to/promtail-directory/promtail -config.file=/path/to/promtail-directory/promtail.yaml
Restart=always

[Install]
WantedBy=multi-user.target

Enable this service as root:

systemctl daemon-reload
systemctl enable peertube-promtail
systemctl start peertube-promtail

Your logs are now sent to your Loki instance.

PeerTube traces in Tempo ​

Traces are interesting to debug slow PeerTube instances and to help developers to find slow paths in PeerTube code. They can be automatically sent by PeerTube to a Jaeger compatible endpoint (like Tempo in this guide).

Configure PeerTube to send traces ​

Update your PeerTube configuration and set open_telemetry.tracing.enabled to true and configure your Jaeger endpoint in open_telemetry.tracing.jaeger_exporter.endpoint (for example http://tempo.example.com:14268/api/traces).

After a PeerTube restart, your logs can now be queried on your Tempo instance.

Visualize data in Grafana ​

Grafana can be easily configured to display graphs, logs, information from Prometheus, Loki and Tempo. Since creating these dashboard could be long, you can import this dashboard in your Grafana instance: monitoring.json.