Plugins

• Prometheus plugin
  • Overview
  • Configuration
  • Build and Run plugin in demo environment
  • Integration tests
  • Labels
  • Available BlazingMQ metrics
  • System metrics
    • CPU
    • Operating System
    • Memory
    • Network metrics
  • Broker metrics
  • Cluster metrics
  • Cluster partitions metrics
  • Domain metrics
  • Queue metrics

Prometheus plugin

Overview

This plugin gathers BlazingMQ broker statistic and sends it to Prometheus monitoring system. Both push and pool modes of interaction with Prometheus are supported.

Configuration

By default, plugin is disabled. To enable and configure it, edit bmqbrkcfg.json file as follows:

1. Enable plugin and provide path to plugin library

   "appConfig": {
       ...
       "plugins": {
           "libraries": ["<path-to-prometheus-plugin-library-folder>"],
           "enabled": ["PrometheusStatConsumer"]
       }
   }
   

2. Provide plugin configuration

   "appConfig": {
       "stats": {
           "snapshotInterval": 1,
           "plugins": [
               ...
               {
                   "name": "PrometheusStatConsumer",
                   "publishInterval": 10,
                   "prometheusSpecific": {
                       "host": "localhost",
                       "port": 9091,
                       "mode": "E_PUSH"
                   }
               }
           ],
           ...
       }
   }
   

where

1. Common plugins configuration

   • snapshotInterval: represents how often stats are computed by broker internally, in seconds (typically every 1s);

2. Prometheus plugin configuration

   • name: plugin name, must be “PrometheusStatConsumer”;
   • publishInterval: Specified as a number of seconds. Must be a multiple of the snapshotInterval.
     • in push mode: it is the time period (in seconds) to send statistic to Prometheus Push Gateway;
     • in pull mode: it is time (in seconds) to update statistic;
   • host
     • in push mode: Prometheus Push Gateway URL;
     • in pull mode: Prometheus exposer (local http server) URL that should be accessible by Prometheus to pull the statistic, usually Host IP address;
   • port
     • in push mode: Prometheus Push Gateway port, usually 9091;
     • in pull mode: Prometheus exposer port that should be accessible by Prometheus to pull the statistic;
   • mode: interaction with Prometheus mode: E_PUSH or E_PULL;

Build and Run plugin in demo environment

To build Prometheus plugin, pass ‘–plugins prometheus’ argument to the build script, e.g.

bin/build-ubuntu.sh --plugins prometheus

To run plugin in demo environment, perform the following steps:

1. Set plugin configuration:
   • For push mode:

       {
         "host": "localhost",
         "port": 9091,
         "mode": "E_PUSH"
       }
     

   • For pull mode:

       {
         "host": "localhost",
         "port": 8080,
         "mode": "E_PULL"
       }
     

2. Run BlazingMQ broker, it will automatically load and configure the plugin;

3. Run Prometheus and Grafana services in Docker:

      docker compose -f docker/plugins/prometheus/docker-compose.yml up
   

4. In browser open link http://localhost:9090/ with Prometheus UI to analyze available metrics;

5. [Optional] In browser open link http://localhost:3000/ with Grafana UI to analyze available metrics:
   • Select Prometrheus data source;
   • Set http://prometheus:9090 as Prometheus server URL;

Integration tests

Test plan:

1. Test Prometheus plugin in ‘push’ mode:
   • Run Prometheus (in docker);
   • Run broker with local cluster and enabled Prometheus plugin in sandbox (temp folder);
   • Put several messages into different queues;
   • Request metrics from Prometheus and compare them with expected metric values.
2. Test Prometheus plugin in ‘pull’ mode:
   • Run Prometheus (in docker);
   • Run broker with local cluster and enabled Prometheus plugin in sandbox (temp folder);
   • Put several messages into different queues;
   • Request metrics from Prometheus and compare them with expected metric values.

Prerequisites:

1. bmqbroker, bmqtool and prometheus plugin library should be built;
2. Python3 should be installed;
3. Docker should be installed, user launching the test script must be included into the group ‘docker’.

   Usage: ./src/plugins/bmqprometheus/tests/bmqprometheus_prometheusstatconsumer_test.py [-h] -p PATH
   options:
     -h, --help            show this help message and exit
     -p PATH, --path PATH  path to BlazingMQ build folder, e.g. './build/blazingmq'
   

Labels

The following labels may be used by each metric, whenever applicable. Refer to each individual metric documentation for restrictions which may apply.

Available BlazingMQ metrics

This section lists all the metrics reported to Prometheus by BlazingMQ brokers. Note that any metric whose value is 0 is not published. The only exception to this rule is queue.heartbeat which is always published.

In this section, report interval refers to how often stats are being published in push mode (typically every 30s) and snapshot represents how often stats are computed internally (typically every 1s).

Note that all metrics published to Prometheus are also dumped by BlazingMQ broker on the local machine in a file, which is located at this location:

<broker_path>/localBMQ/logs/stat.*

These stat files can come in handy when Prometheus is unavailable or when metrics during certain time interval are missing from Prometheus for some reason.

System metrics

System metrics represent BlazingMQ broker’s overall operating system metrics. Every broker reports them, with ‘Instance’ label which is always set.

CPU

Operating System

Memory

Network metrics

Broker metrics

Broker summary metrics represent high level aggregated view of the activity on that broker. Every broker reports them, with ‘Instance’ label.

Cluster metrics

Every broker reports this metric for each of the clusters it has created, with the following tags: ‘Instance’, ‘Cluster’ and ‘Role’. If the ‘Role’ is ‘PROXY’, the ‘RemoteHost’ tag is also set (but can have the value ‘none’).

Metrics in the following section are reported only by the leader node of the cluster.

Cluster partitions metrics

The following metrics are reported for each partition, only by the primary of the partition, with the ‘Cluster’ and ‘Instance’ tags set. Note that in the following metrics name <X> represents the partition id (0 based integer).

Domain metrics

Domain metrics represent high level metrics related to a domain. Only the leader node of the cluster reports them, with the ‘Cluster’, ‘Domain’, ‘Tier’ and ‘Instance’ tags set.

Queue metrics

Queue metrics represent detailed, per queue, metrics. For each of them, the following tags are set: ‘Cluster’, ‘Domain’, ‘Tier’, ‘Queue’, ‘Role’ (which value can only be one of ‘PRIMARY’, ‘REPLICA’ or ‘PROXY’) and instanceName. ‘AppId’ tag could be applied on ‘queue.confirm_time_max’ and ‘queue.queue_time_max’ metrics in fanout mode if this feature is configured (see ‘appIdTagDomains’ setting).

Metrics in the following section are reported only by the node primary of the queue.