Metrics
  • 15 Nov 2021
  • 3 Minutes to read
  • Dark
    Light

Metrics

  • Dark
    Light

Standard Metrics

Graylog collects metrics throughout its operation and stores them in-memory on each Graylog node. The metrics include event counts, timers and statistics for many parts of the Graylog application and its subsystems. Standard metrics are viewable within Graylog on the System > Nodes page through the Metrics button.
standardmetrics.png

Standard metrics can also be queried through the Graylog REST API Browser.

Prometheus Metric Exporting

Starting in Graylog 4.1, metrics can be exported to other systems via the Prometheus Metrics Exporting feature.This feature can be optionally enabled. When enabled, Graylog will export metrics via a standard Prometheus HTTP exporter on port 9833. Prometheuscan then scrape and ingest the metrics.

Configuration

To begin exporting Prometheus metrics, enable it with the following configuration property in the server.conf file.

prometheus_exporter_enabled = true

Once enabled, metrics are exported in the standard export format on the following URI by default.

http://127.0.0.1:9833/api/metrics/prometheus

To specify a custom export hostname or IP address, the following configuration can be used.We suggest leaving the default export port , since it is already registered with Prometheus.

prometheus_exporter_bind_address = 10.0.0.1:9090 

Depending on the Prometheus scrape_config setting, an explicit scrape target might need to be specified in your Prometheus target hosts configuration.

Default Core Metric Mappings

A pre-defined core set of Graylog-to-Prometheus metric mappings are enabled by default.

All metric names are prefixed with gl_...
to clearly indicate that the metrics originated from Graylog. For example,the metric name gl_input_throughput.
prometheusmetricnames.png

All metrics are automatically assigned a nodemetric label,which contains the Graylog Node ID where the metric originated from. This can be useful for visualizingthe same metric across Graylog nodes.
nodeidmetric.png
Many metrics are exported with a consolidated name, which allows breakouts by labels to allow easier visualization of metrics.

See the pre-defined metric mappings for a full list of metric mappings.

Customized Metric Mappings

We understand that metric mapping requirements can be very specific the each per-customer use case, so we providethe ability to override the default core mappings. We also support the ability to specify any additional needed mappings.

Graylog metrics exports heavily utilize Prometheus labels to help you effectively visualize and analyze your metricsdata.

To completely replace the default core Prometheus mappings, provide the path to a core replacement mapping file YMLfilecontaining just the desired mappings with the following configuration property. This file is monitored for changes atruntime, so you can customize as needed without restarting your Graylog server. The file path is interpretedrelative to the Graylog server working directory. Absolute paths are also supported.

prometheus_exporter_mapping_file_path_core = prometheus-exporter-mapping-core.yml

To provide mappings in addition to the core defaults, provide the path to an additional mapping file yaml
filecontaining the additional desired mappings with the following configuration property. This file is also monitored forchanges at runtime.

prometheus_exporter_mapping_file_path_custom = prometheus-exporter-mapping-custom.yml

Custom Mappings Format Example 1

This example mapping produces the gl_stream_incoming_messages metric in Prometheus, which shows the number ofmessages received by for each stream in Graylog.

The standard metric name in Graylog contains the ID of the stream, and one distinct metric is recorded for each stream.

For example:

org.graylog2.plugin.streams.Stream.stream-1.incomingMessages org.graylog2.plugin.streams.Stream.stream-2.incomingMessages org.graylog2.plugin.streams.Stream.stream-3.incomingMessages 

The mapping definition that follows provides a match_pattern with a wildcard *for the stream-id, whichprovides one label for each stream id automatically. This allows for the visualization of messages received for allstreams together, but broken-out by stream-id via the labels functionality. The wildcard_extract_labels is an orderedarray, which provides the label names for any specified wildcards in the match_pattern.

Mapping Definition:

- metric_name : "stream_incoming_messages" match_pattern : "org.graylog2.plugin.streams.Stream.*.incomingMessages" wildcard_extract_labels : - "stream" 

Custom Mappings Example 2

This example mapping produces the gl_buffer_usage
metric in Prometheus, which provides the current usage state ofthe Graylog input, processand output
buffers.

Note that three separate mappings are provided to consolidate three independent buffer Graylog metrics into a singlePrometheus metric with three unique labels (input, process, output) that correspond to each Graylog metric.

The additional_labels property allows for the assignment of an explicit label corresponding to the metric defined inthe match_pattern.

Mapping Definition:

- metric_name: "buffer_usage"
  match_pattern: "org.graylog2.buffers.input.usage"
  additional_labels:
    type: "input"

- metric_name: "buffer_usage"
  match_pattern: "org.graylog2.buffers.output.usage"
  additional_labels:
    type: "output"

- metric_name: "buffer_usage"
  match_pattern: "org.graylog2.buffers.process.usage"
  additional_labels:
    type: "process"

Custom Mappings Refresh Interval

By default, custom and core mapping files are refreshed (re-read from disk) every 30 seconds. You can override thiswith a custom duration if desired. Use the standard Graylog duration notation (eg 60s, 5m or 1h).

prometheus_exporter_mapping_file_refresh_interval = 5m 

Was this article helpful?

What's Next