Usage
  • 09 Sep 2021
  • 3 Minutes to read
  • Dark
    Light

Usage

  • Dark
    Light

Overview

Once you understand the concepts explained in Pipelines , Rules , and Stream connections, you’re ready to start creating your own processing pipelines. Thispage gives you the information you need to get started with the user interface.

Configuration

Configure the message processor

Before start using the processing pipelines you need to ensure the Pipeline Processor message processor is enabledand correctly configured. You can do so by going to the System -> Configurations page, andchecking the configuration in the Message Processors Configuration section.

pipelines_message_processor

On the Configurations page, you need to enable the Pipeline Processor message processor and, if you want your pipelinesto have access to static fields set on inputs and/or fields set by extractors, set the Pipeline Processor after the Message Filter Chain .

Manage rules

You can create, edit, and delete your pipeline rules in the page, under . System -> Pipelines.

pipelines_manage_rules

Clicking on Create Rule or Edit or in one of the rules will open a page where you can writeyour own rule. The page lists available functions and their details to make the task a bit more manageable.

pipelines_edit_rule

Managing pipelines

Once there are some rules in Graylog, you can create pipelines that use them to modify and enrichyour messages.

To manage your pipelines, access page under .This page is where you can create, edit, and delete pipelines.

pipelines_manage_pipelines

In order to create or edit pipelines, and as explained in Pipelines , you need to add yourrules to a stage, which has a certain priority. The Web interface will let you add rules to the defaultstage (priority 0), and to create new stages with potentially different priorities.

pipelines_show_pipeline

A pipeline can have more than one stage, and when you create or edit a stage you need to select howto proceed to the next stage in the pipeline:

All rules on this stage match the message
This option will only consider further stages in the pipeline when all conditions in rulesevaluated in this stage are true. This is equivalent to match allin the Pipelines section.

At least one of the rules on this stage matches the message
Selecting this option will continue to further stages in the pipeline when one or more of theconditions in rules evaluated in this stage are true. This is equivalent to match eitherin the Pipelines section.

Connect pipelines to streams

You can decide which streams are connected to a pipeline from the pipeline details page. Under System -> Pipelines, click on the title of the pipeline you want to connect to a stream, andthen click on the button.

pipelines_manage_connections

You can assign many pipelines to the same stream, in which case all connected pipelines will process messages routed into that streambased upon the overall order of stage priorities.

pipelines_edit_connections

Remember, as mentioned in the Stream connections documentation, All messages the stream is where all messages areinitially routed, and is therefore a good place to apply pipelines applicable to all of your messages. Such pipelines might be responsible for stream routing, blacklisting, field manipulation, etc.

Simulate your changes

After performing some changes in a processing pipeline, you most likely want to see how they areapplied to incoming messages. This is what the pipeline simulator is for.

Click the button under System -> Pipelines or in the pipeline details page to access the pipeline simulator.

pipelines_simulation_1

In order to test the message processing you need to provide a raw message that will be routed intothe stream you want to simulate. The raw message should use the same format Graylog willreceive. For example: you can type a GELF message, in the same format your GELF library would send, in the field.Don’t forget to select the correct codec for the message you provide.

After specifying the message and codec, click to start the simulation and display the results.

pipelines_simulation_2

The simulation provides the following results:

Changes summary
Provides a summary of modified fields in the original message, as well as a list of added and droppedmessages.

Results preview
Shows all fields in the processed message.

Simulation trace
Displays a trace of the processing, indicating which rules were evaluated and which were executed.It also includes a timeline, in microseconds, to allow you to see which rules and pipelines aretaking up the most time during message processing.


Was this article helpful?

What's Next