- 15 Nov 2021
- 3 Minutes to read
- Updated on 15 Nov 2021
- 3 Minutes to read
Graylog collects metrics throughout its operation and stores them in-memory on each Graylog node. The metrics include event counts, timers and statistics for many parts of the Graylog application and its subsystems. Standard metrics are viewable within Graylog on the System > Nodes page through the Metrics button.
Standard metrics can also be queried through the Graylog REST API Browser.
Prometheus Metric Exporting
Starting in Graylog 4.1, metrics can be exported to other systems via the Prometheus Metrics Exporting feature.This feature can be optionally enabled. When enabled, Graylog will export metrics via a standard Prometheus HTTP exporter on port
9833. Prometheuscan then scrape and ingest the metrics.
To begin exporting Prometheus metrics, enable it with the following configuration property in the
prometheus_exporter_enabled = true
Once enabled, metrics are exported in the standard export format on the following URI by default.
To specify a custom export hostname or IP address, the following configuration can be used.We suggest leaving the default export port , since it is already registered with Prometheus.
prometheus_exporter_bind_address = 10.0.0.1:9090
Depending on the Prometheus scrape_config setting, an explicit scrape target might need to be specified in your Prometheus target hosts configuration.
Default Core Metric Mappings
A pre-defined core set of Graylog-to-Prometheus metric mappings are enabled by default.
All metric names are prefixed with
to clearly indicate that the metrics originated from Graylog. For example,the metric name
All metrics are automatically assigned a
nodemetric label,which contains the Graylog Node ID where the metric originated from. This can be useful for visualizingthe same metric across Graylog nodes.
Many metrics are exported with a consolidated name, which allows breakouts by labels to allow easier visualization of metrics.
See the pre-defined metric mappings for a full list of metric mappings.
Customized Metric Mappings
We understand that metric mapping requirements can be very specific the each per-customer use case, so we providethe ability to override the default core mappings. We also support the ability to specify any additional needed mappings.
Graylog metrics exports heavily utilize Prometheus labels to help you effectively visualize and analyze your metricsdata.
To completely replace the default core Prometheus mappings, provide the path to a core replacement mapping file
YMLfilecontaining just the desired mappings with the following configuration property. This file is monitored for changes atruntime, so you can customize as needed without restarting your Graylog server. The file path is interpretedrelative to the Graylog server working directory. Absolute paths are also supported.
prometheus_exporter_mapping_file_path_core = prometheus-exporter-mapping-core.yml
To provide mappings in addition to the core defaults, provide the path to an additional mapping file
filecontaining the additional desired mappings with the following configuration property. This file is also monitored forchanges at runtime.
prometheus_exporter_mapping_file_path_custom = prometheus-exporter-mapping-custom.yml
Custom Mappings Format Example 1
This example mapping produces the
gl_stream_incoming_messages metric in Prometheus, which shows the number ofmessages received by for each stream in Graylog.
The standard metric name in Graylog contains the ID of the stream, and one distinct metric is recorded for each stream.
org.graylog2.plugin.streams.Stream.stream-1.incomingMessages org.graylog2.plugin.streams.Stream.stream-2.incomingMessages org.graylog2.plugin.streams.Stream.stream-3.incomingMessages
The mapping definition that follows provides a
match_pattern with a wildcard
stream-id, whichprovides one label for each stream id automatically. This allows for the visualization of messages received for allstreams together, but broken-out by stream-id via the labels functionality. The
wildcard_extract_labels is an orderedarray, which provides the label names for any specified wildcards in the
- metric_name : "stream_incoming_messages" match_pattern : "org.graylog2.plugin.streams.Stream.*.incomingMessages" wildcard_extract_labels : - "stream"
Custom Mappings Example 2
This example mapping produces the
metric in Prometheus, which provides the current usage state ofthe Graylog
Note that three separate mappings are provided to consolidate three independent buffer Graylog metrics into a singlePrometheus metric with three unique labels (
output) that correspond to each Graylog metric.
additional_labels property allows for the assignment of an explicit label corresponding to the metric defined inthe
- metric_name: "buffer_usage" match_pattern: "org.graylog2.buffers.input.usage" additional_labels: type: "input" - metric_name: "buffer_usage" match_pattern: "org.graylog2.buffers.output.usage" additional_labels: type: "output" - metric_name: "buffer_usage" match_pattern: "org.graylog2.buffers.process.usage" additional_labels: type: "process"
Custom Mappings Refresh Interval
By default, custom and core mapping files are refreshed (re-read from disk) every 30 seconds. You can override thiswith a custom duration if desired. Use the standard Graylog duration notation (eg
prometheus_exporter_mapping_file_refresh_interval = 5m