Sending in log data

A Graylog setup is pretty worthless without any data in it. This page explains the basic principles of getting your data into the system and also explains common fallacies.

What are Graylog message inputs?

Message inputs are the Graylog parts responsible for accepting log messages. Some default messages types are available by default in Graylog. But it might be needed to install additional plugins to enable Graylog to receive particular messages.

After choosing the input type in the Graylog web interface at System / Inputs, the input is launched without a restart of Graylog. Most environments will use the defaults for the inputs, but most inputs have a granular configuration available. Some can use TLS or authentication via certificates, and others can make use of a queuing system.

Most environments will have one input of each type and ingest all messages of that type to the one input. But it might be needed to have multiple inputs of the same kind to allow some extractors to work only on specific messages. But the processing pipeline would allow us to work on ingested messages from different sources on the same input in different ways. But as every environment is different, Graylog is flexible and will enable you to adjust everything to your need and not let you change your setting to fit Graylog.

The section about the different log sources describes how you ingest those types of logs; besides, some logs have individual inputs. Some of them are listed below in the individual inputs section. In addition to that, the Graylog Marketplace contains some inputs created by the community.

Log sources

We can’t cover all possible options and device configurations in this section. The following describes the fundamentals of those inputs and the current best practice to ingest those kinds of messages into Graylog. Some can be done native others need some collector. The section will include one or two example collectors, but the free and open world has multiple more to offer, we can’t cover all solutions.

Individual Inputs

These listed inputs are not all native available in Graylog. Most are dedicated to an individual product or framework. The default inputs like Syslog, CEF, GELF, or the RAW/Plaintext are not listed individually.

Using Apache Kafka as transport queue

Graylog supports Apache Kafka as a transport for various inputs such as GELF, syslog, and Raw/Plaintext inputs. The Kafka topic can be filtered by a regular expression and depending on the input, various additional settings can be configured.

Learn how to use rsyslog and Apache Kafka in the Sending syslog via Kafka into Graylog guide.

Using RabbitMQ (AMQP) as transport queue

Graylog supports AMQP as a transport for various inputs such as GELF, syslog, and Raw/Plaintext inputs. It can connect to any AMQP broker supporting AMQP 0-9-1 such as RabbitMQ.

Learn how to use rsyslog and RabbitMQ in the Sending syslog via AMQP into Graylog guide.

Input Throttling

Throttling allows certain Graylog Inputs to slow their message intake rates (by temporarily pausing intake processing) if contention occurs in the Graylog Journal.

Graylog Inputs that support throttling

  • AWS Flow Logs

  • AWS Cloudtrail

  • AWS Logs

  • CEF AMQP Input

  • CEF Kafka Input

  • GELF AMQP

  • GELF Kafka

  • JSON path from HTTP API

  • Raw/Plaintext AMQP

  • Raw/Plaintext Kafka

  • Syslog AMQP

  • Syslog Kafka

Enabling throttling

To enable throttling for one of these inputs, edit it in System > Inputs and check the Allow throttling this input checkbox.

Throttling criteria

When enabled, the following criteria will be used to determine if throttling will occur:

  1. If there are zero uncommitted entries in the Graylog Journal, throttling will not occur. No further checks will be performed.

  2. Throttling will occur if the Journal has more than 100k uncommitted entries.

  3. Throttling will occur if the Journal is growing in size rapidly (approximately 20k entries per second or greater).

  4. Throttling will occur if the process ring buffer is full.

  5. Nothing is currently being written to the Journal; throttling will not occur. No further checks will be performed.

  6. Throttling will occur if the Journal is more than 90% full.

  7. Throttling will occur if the Journal write rate is more than twice as high as the read rate.