Graylog collects metrics throughout its operation and stores them in-memory on each Graylog node. The metrics include event counts, timers and statistics for many parts of the Graylog application and its subsystems. Standard metrics are viewable within Graylog on the System > Nodes page through the Metrics button.
Standard metrics can also be queried through the Graylog REST API Browser.
Prometheus Metric Exporting¶
Starting in Graylog 4.1, metrics can be exported to other systems via the Prometheus Metrics Exporting feature.
This feature can be optionally enabled. When enabled, Graylog will export metrics via a standard
Prometheus HTTP exporter on port
can then scrape and ingest the metrics.
To begin exporting Prometheus metrics, enable it with the following configuration property in the
prometheus_exporter_enabled = true
Once enabled, metrics are exported in the standard export format on the following URI by default.
To specify a custom export hostname or IP address, the following configuration can be used. We suggest leaving the default export port 9833, since it is already registered with Prometheus.
prometheus_exporter_bind_address = 10.0.0.1:9090
Depending on the Prometheus scrape_config setting, an explicit scrape target might need to be specified in your Prometheus target hosts configuration.
Default Core Metric Mappings¶
A pre-defined core set of Graylog-to-Prometheus metric mappings are enabled by default.
All metric names are prefixed with
gl_... to clearly indicate that the metrics originated from Graylog. For example,
the metric name
All metrics are automatically assigned a
node metric label,
which contains the Graylog Node ID where the metric originated from. This can be useful for visualizing
the same metric across Graylog nodes.
Many metrics are exported with a consolidated name, which allows breakouts by labels to allow easier visualization of metrics.
See the pre-defined metric mappings for a full list of metric mappings.
Customized Metric Mappings¶
We understand that metric mapping requirements can be very specific the each per-customer use case, so we provide the ability to override the default core mappings. We also support the ability to specify any additional needed mappings.
Graylog metrics exports heavily utilize Prometheus labels to help you effectively visualize and analyze your metrics data.
To completely replace the default core Prometheus mappings, provide the path to a core replacement mapping file
containing just the desired mappings with the following configuration property. This file is monitored for changes at
runtime, so you can customize as needed without restarting your Graylog server. The file path is interpreted
relative to the Graylog server working directory. Absolute paths are also supported.
prometheus_exporter_mapping_file_path_core = prometheus-exporter-mapping-core.yml
To provide mappings in addition to the core defaults, provide the path to an additional mapping file
containing the additional desired mappings with the following configuration property. This file is also monitored for
changes at runtime.
prometheus_exporter_mapping_file_path_custom = prometheus-exporter-mapping-custom.yml
Custom Mappings Format Example 1¶
This example mapping produces the
gl_stream_incoming_messages metric in Prometheus, which shows the number of
messages received by for each stream in Graylog.
The standard metric name in Graylog contains the ID of the stream, and one distinct metric is recorded for each stream.
org.graylog2.plugin.streams.Stream.stream-1.incomingMessages org.graylog2.plugin.streams.Stream.stream-2.incomingMessages org.graylog2.plugin.streams.Stream.stream-3.incomingMessages
The mapping definition that follows provides a
match_pattern with a wildcard
* for the
provides one label for each stream id automatically. This allows for the visualization of messages received for all
streams together, but broken-out by stream-id via the labels functionality. The
wildcard_extract_labels is an ordered
array, which provides the label names for any specified wildcards in the
- metric_name: "stream_incoming_messages" match_pattern: "org.graylog2.plugin.streams.Stream.*.incomingMessages" wildcard_extract_labels: - "stream"
Custom Mappings Example 2¶
This example mapping produces the
gl_buffer_usage metric in Prometheus, which provides the current usage state of
Note that three separate mappings are provided to consolidate three independent buffer Graylog metrics into a single
Prometheus metric with three unique labels (
output) that correspond to each Graylog metric.
additional_labels property allows for the assignment of an explicit label corresponding to the metric defined in
- metric_name: "buffer_usage" match_pattern: "org.graylog2.buffers.input.usage" additional_labels: type: "input" - metric_name: "buffer_usage" match_pattern: "org.graylog2.buffers.output.usage" additional_labels: type: "output" - metric_name: "buffer_usage" match_pattern: "org.graylog2.buffers.process.usage" additional_labels: type: "process"
Custom Mappings Refresh Interval¶
By default, custom and core mapping files are refreshed (re-read from disk) every 30 seconds. You can override this
with a custom duration if desired. Use the standard Graylog duration notation (eg
prometheus_exporter_mapping_file_refresh_interval = 5m