Indexer and Processing Failures
  • 12 Oct 2021
  • 2 Minutes to read
  • Dark
    Light

Indexer and Processing Failures

  • Dark
    Light

Graylog 4.2 allows for storing indexing and processing failure notifications in a dedicated Elasticsearch failure index.

When a failure occurs while indexing or processing log data, it is important not only to receive the notification but to be able to analyze the failure message to understand why the error occurred--just like any log data. With a dedicated stream for failure notification, these messages can be logged and aggregated meaningfully into a dashboard, used to set up alert notifications, and more.

Configuration

This feature is disabled by default. To enable, navigate to System > Configurations and scroll down to the Failure Processing section. Here, you will be able to activate each failure processing feature individually.

Indexer_1

  • Log Indexing Failures
    • Allows indexer failure notifications to be stored in Elasticsearch and logged in a dedicated Graylog stream.
  • Log Processing Failures
    • Allows processing failure notifications to be stored in Elasticsearch and logged in a dedicated Graylog stream.
  • Include Failed Messages
    • Displays the full log message in the failure notification, allowing for a more thorough investigation of the failure reason. Note that either Log Indexing Failures or Log Processing Failures must be enabled to make this selection.
  • Continue Processing on Error
    • Permits the original message to be stored alongside a new field (gl2_processing_error) containing specific error details while a failure message with all the error details is stored in the dedicatedGraylog stream. Note that Log Processing Failures must be enabled to make this selection.

Once enabled, the widget in System Overview will display a counter of your failure messages.

Indexer_2

Common Indexer Failure Reasons

The most common indexer failure is classified as a “MapperParsingException.” This type of notification may look something like this:

Indexer_3

For additional information on this type of failure, review Common Indexer Failure Reasons.

Common Processing Failure Reasons

A processing failure, which can occur within the Graylog processing stack, may have multiple reasons for occurring. The following is a list of the most common reasons and may give you an indication of why the failure has occurred:

  • “RuleStatementEvaluationError”
    • Occurs when there is an error in the statement between the “then” and “end” value of the pipeline rule.
  • “RuleConditionEvaluationError”
    • Occurs when there is an error in the statement between the “when” and “then” value of the pipeline rule.
  • “ExtractorException”
    • Occurs when an extractor or converter incorrectly reads or extrapolates a message.
  • “MessageFilterException”
    • Occurs when there is a backend system failure involving the Graylog application; further troubleshooting with Graylog support may be required.
  • “InvalidTimestampException”
    • Occurs when there is a failure during an attempt to set or extract a value in the timestamp field, e.g a pipeline rule failed while attempting to extract a timestamp from a string and attempted to assign this null timestamp to a message.
  • “UNKNOWN”
    • The reason for this error is unknown and will require further investigation into the log data.

Was this article helpful?

What's Next