Functions

Overview

Functions are the means of how to interact with the messages Graylog processes.

They are written in Java and are pluggable, allowing extending the capabilities of Graylog in a simple manner.

Conceptually a function receives parameters, the current message context and returns a value. The data types of its return value and parameters determines where in the rules it can be used. Graylog makes sure the rules are sound from a data type perspective.

A function’s parameters can be passed as named pairs or by position, as long as optional parameters are declared as coming last. The function’s documentation below indicates which parameters are optional.

Let’s look at a small example to illustrate these properties:

rule "function howto"
when
    has_field("transaction_date")
then
    // the following date format assumes there's no time zone in the string
    let new_date = parse_date(to_string($message.transaction_date), "yyyy-MM-dd HH:mm:ss");
    set_field("transaction_year", new_date.year);
end

In this example, we check if the current message contains the field some_date and then, after converting it to a string, try to parse it according to the format string yyyy-MM-dd HH:mm:ss, so for example the string 2016-03-05 14:45:02 would match. Parse date returns a DateTime object from the Java Joda-Time library, allowing easier access to the date’s components.

We then add the transaction’s year as a new field to the message.

You’ll note that we haven’t said in which time zone the timestamp is in, but still Graylog had to pick one (Graylog never relies on the local time of your server as that makes it nearly impossible to figure out why date handling came up with its result).

The reason Graylog knows which timezone to pick is because parse_date actually takes three parameters rather than the two we’ve given it in this example. The third one is a String called timezone and is optional. It has a default value of "UTC".

Let’s assume we have another field in the message, called transaction_timezone. It is send by the application and contains the time zone ID the transaction was done in (hopefully no application in the world sends its data like this, though):

rule "function howto"
when
    has_field("transaction_date") && has_field("transaction_timezone")
then
    // the following date format assumes there's no time zone in the string
    let new_date = parse_date(
                        to_string($message.transaction_date),
                        "yyyy-MM-dd HH:mm:ss",
                        to_string($message.transaction_timezone)
                );
    set_field("transaction_year", new_date.year);
end

Now, if the time zone is specified in the other field, we pass it to parse_date.

In this case we only have a single optional parameter, which makes it easy to simply omit it from the end of the function call. However, if there are multiple ones, or if there are many parameters and it gets difficult to keep track of which positions correspond to which parameters, you can also use the named parameter variant of function calls. In this mode the order of the parameters does not matter, but all required ones still need to be there.

In our case the alternative version of calling parse_date would look like this:

rule "function howto"
when
    has_field("transaction_date") && has_field("transaction_timezone")
then
    // the following date format assumes there's no time zone in the string
    let new_date = parse_date(
                        value: to_string($message.transaction_date),
                        pattern: "yyyy-MM-dd HH:mm:ss",
                        timezone: to_string($message.transaction_timezone)
                );
    set_field("transaction_year", new_date.year);
end

All parameters in Graylog’s processing functions are named, please refer to the function index.

Function Index

The following list describes the built in functions that ship with Graylog. Additional third party functions are available via other plugins in the marketplace.

Built-in Functions
Name Description
to_bool Converts the single parameter to a boolean value using its string value.
to_double Converts the first parameter to a double floating point value.
to_long Converts the first parameter to a long integer value.
to_string Converts the first parameter to its string representation.
to_url Converts a value to a valid URL using its string representation.
is_null Checks whether a value is ‘null’.
is_not_null Checks whether a value is not ‘null’.
abbreviate Abbreviates a String using ellipses.
capitalize Capitalizes a String changing the first letter to title case.
uncapitalize Uncapitalizes a String changing the first letter to lower case.
uppercase Converts a String to upper case.
lowercase Converts a String to lower case.
swapcase Swaps the case of a String.
contains Checks if a string contains another string.
substring Returns a substring of value with the given start and end offsets.
concat Concatenates two strings.
split Split a string around matches of this pattern (Java syntax).
regex Match a regular expression against a string, with matcher groups.
grok Applies a Grok pattern to a string.
key_value Extracts key/value pairs from a string.
crc32 Returns the hex encoded CRC32 digest of the given string.
crc32c Returns the hex encoded CRC32C (RFC 3720, Section 12.1) digest of the given string.
md5 Returns the hex encoded MD5 digest of the given string.
murmur3_32 Returns the hex encoded MurmurHash3 (32-bit) digest of the given string.
murmur3_128 Returns the hex encoded MurmurHash3 (128-bit) digest of the given string.
sha1 Returns the hex encoded SHA1 digest of the given string.
sha256 Returns the hex encoded SHA256 digest of the given string.
sha512 Returns the hex encoded SHA512 digest of the given string.
parse_json Parse a string into a JSON tree.
select_jsonpath Selects one or more named JSON Path expressions from a JSON tree.
to_ip Converts the given string to an IP object.
cidr_match Checks whether the given IP matches a CIDR pattern.
from_input Checks whether the current message was received by the given input.
route_to_stream Assigns the current message to the specified stream.
create_message Currently incomplete Creates a new message which will be evaluated by the entire processing pipeline.
clone_message Clones a message.
drop_message This currently processed message will be removed from the processing pipeline after the rule finishes.
has_field Checks whether the currently processed message contains the named field.
remove_field Removes the named field from the currently processed message.
set_field Sets the name field to the given value in the currently processed message.
set_fields Sets multiple fields to the given values in the currently processed message.
rename_field Rename a message field.
syslog_facility Converts a syslog facility number to its string representation.
syslog_level Converts a syslog level number to its string representation.
expand_syslog_priority Converts a syslog priority number to its level and facility.
expand_syslog_priority_as_string Converts a syslog priority number to its level and facility string representations.
now Returns the current date and time.
parse_date Parses a date and time from the given string, according to a strict pattern.
flex_parse_date Attempts to parse a date and time using the Natty date parser.
format_date Formats a date and time according to a given formatter pattern.
to_date Converts a type to a date.
years Create a period with a specified number of years.
months Create a period with a specified number of months.
weeks Create a period with a specified number of weeks.
days Create a period with a specified number of days.
hours Create a period with a specified number of hours.
minutes Create a period with a specified number of minutes.
seconds Create a period with a specified number of seconds.
millis Create a period with a specified number of millis.
period Parses an ISO 8601 period from the specified string.

to_bool

to_bool(value: any)

Converts the single parameter to a boolean value using its string value.

to_double

to_double(value: any, [default: double])

Converts the first parameter to a double floating point value.

to_long

to_long(value: any, [default: long])

Converts the first parameter to a long integer value.

to_string

to_string(value: any, [default: string])

Converts the first parameter to its string representation.

to_url

to_url(url: any, [default: string])

Converts the given url to a valid URL.

is_null

is_null(value: any)

Checks if the given value is null.

is_not_null

is_not_null(value: any)

Checks if the given value is not null.

abbreviate

abbreviate(value: string, width: long)

Abbreviates a String using ellipses, the width defines the maximum length of the resulting string.

capitalize

capitalize(value: string)

Capitalizes a String changing the first letter to title case.

uncapitalize

uncapitalize(value: string)

Uncapitalizes a String changing the first letter to lower case.

uppercase

uppercase(value: string, [locale: string])

Converts a String to upper case. The locale (IETF BCP 47 language tag) defaults to “en”.

lowercase

lowercase(value: string, [locale: string])

Converts a String to lower case. The locale (IETF BCP 47 language tag) defaults to “en”.

swapcase

swapcase(value: string)

Swaps the case of a String changing upper and title case to lower case, and lower case to upper case.

contains

contains(value: string, search: string, [ignore_case: boolean])

Checks if value contains search, optionally ignoring the case of the search pattern.

substring

substring(value: string, start: long, [end: long])

Returns a substring of value starting at the start offset (zero based indices), optionally ending at the end offset. Both offsets can be negative, indicating positions relative to the end of value.

concat

concat(first: string, second: string)

Returns a new string combining the text of first and second.

split

split(pattern: string, value: string, [limit: int])

Split a value around matches of pattern. Use limit to indicate the number of times the pattern should be applied.

Note: Patterns have to be valid Java String literals, please ensure you escape any backslashes in your regular expressions!

regex

regex(pattern: string, value: string, [group_names: array[string])

Match the regular expression in pattern against value. Returns a match object, with the boolean property matches to indicate whether the regular expression matched and, if requested, the matching groups as groups. The groups can optionally be named using the group_names array. If not named, the groups names are strings starting with "0".

Note: Patterns have to be valid Java String literals, please ensure you escape any backslashes in your regular expressions!

grok

grok(pattern: string, value: string, [only_named_captures: boolean])

Applies the grok pattern grok to value. Returns a match object, containing a Map of field names and values. You can set only_named_captures to true to only return matches using named captures.

Tip: The result of executing the grok function can be passed as argument for set_fields to set the extracted fields into a message.

key_value

key_value(
  value: string,
  [delimiters: string],
  [kv_delimiters: string],
  [ignore_empty_values: boolean],
  [allow_dup_keys: boolean],
  [handle_dup_keys: string],
  [trim_key_chars: string],
  [trim_value_chars: string]
)

Extracts key-value pairs from the given value and returns them as a Map of field names and values. You can optionally specify:

delimiters
Characters used to separate pairs. We will use each character in the string, so you do not need to separate them. Default value: <whitespace>.
kv_delimiters
Characters used to separate keys from values. Again, there is no need to separate each character. Default value: =.
ignore_empty_values
Ignores keys containing empty values. Default value: true.
allow_dup_keys
Indicates if duplicated keys are allowed. Default value: true.
handle_dup_keys
How to handle duplicated keys (if allow_dup_keys is set). It can take the values take_first, which will only use the first value for the key; or take_last, which will only use the last value for the key. Setting this option to any other value will change the handling to concatenate, which will combine all values given to the key, separating them with the value set in this option. For example, setting handle_dup_keys: ",", would combine all values given to a key a, separating them with a comma, such as 1,2,foo. Default value: take_first.
trim_key_chars
Characters to trim (remove from the beginning and end) from keys. Default value: no trim.
trim_value_chars
Characters to trim (remove from the beginning and end) from values. Default value: no trim.

Tip: The result of executing the key_value function can be passed as argument for set_fields to set the extracted fields into a message.

crc32

crc32(value: string)

Creates the hex encoded CRC32 digest of the value.

crc32c

crc32c(value: string)

Creates the hex encoded CRC32C (RFC 3720, Section 12.1) digest of the value.

md5

md5(value: string)

Creates the hex encoded MD5 digest of the value.

murmur3_32

murmur3_32(value: string)

Creates the hex encoded MurmurHash3 (32-bit) digest of the value.

murmur3_128

murmur3_128(value: string)

Creates the hex encoded MurmurHash3 (128-bit) digest of the value.

sha1

sha1(value: string)

Creates the hex encoded SHA1 digest of the value.

sha256

sha256(value: string)

Creates the hex encoded SHA256 digest of the value.

sha512

sha512(value: string)

Creates the hex encoded SHA512 digest of the value.

parse_json

parse_json(value: string)

Parses the value string as JSON, returning the resulting JSON tree.

select_jsonpath

select_jsonpath(json: JsonNode, paths: Map<string, string>)

Evaluates the given paths against the json tree and returns the map of the resulting values.

to_ip

to_ip(ip: string)

Converts the given ip string to an IpAddress object.

cidr_match

cidr_match(cidr: string, ip: IpAddress)

Checks whether the given ip address object matches the cidr pattern.

from_input

from_input(id: string | name: string)

Checks whether the currently processed message was received on the given input. The input can be looked up by either specifying its name (the comparison ignores the case) or the id.

route_to_stream

route_to_stream(id: string | name: string, [message: Message])

Routes the messsage to the given stream. The stream can be looked up by either specifying its name or the id.

If message is omitted, this function uses the currently processed message.

This causes the message to be evaluated on the pipelines connected to that stream, unless the stream has already been processed for this message.

create_message

create_message([message: string], [source: string], [timestamp: DateTime])

Creates a new message with from the given parameters. If any of them is omitted, its value is taken from the corresponding fields of the currently processed message. If timestamp is omitted, the timestamp of the created message will be the timestamp at that moment.

clone_message

clone_message([message: Message])

Clones a message. If message is omitted, this function uses the currently processed message.

drop_message

drop_message(message: Message)

The processing pipeline will remove the given message after the rule is finished executing.

If message is omitted, this function uses the currently processed message.

This can be used to implement flexible blacklisting based on various conditions.

has_field

has_field(field: string, [message: Message])

Checks whether the given message contains a field with the name field.

If message is omitted, this function uses the currently processed message.

remove_field

remove_field(field: string, [message: Message])

Removes the given field with the name field from the given message, unless the field is reserved.

If message is omitted, this function uses the currently processed message.

set_field

set_field(field: string, value: any, [message: Message])

Sets the given field named field to the new value. The field name must be valid, and specifically cannot include a . character. It is trimmed of leading and trailing whitespace. String values are trimmed of whitespace as well.

If message is omitted, this function uses the currently processed message.

set_fields

set_fields(fields: Map<string, any>, [message: Message])

Sets all of the given name-value pairs in field in the given message. This is a convenience function acting like set_field. It can be helpful for using the result of a function like select_jsonpath or regex in the currently processed message especially when the key names are the result of a regular expression.

If message is omitted, this function uses the currently processed message.

rename_field

rename_field(old_field: string, new_field: string, [message: Message])

Modifies the field name old_field to new_field in the given message, keeping the field value unchanged.

syslog_facility

syslog_facility(value: any)

Converts the syslog facility number in value to its string representation.

syslog_level

syslog_level(value: any)

Converts the syslog severity number in value to its string representation.

expand_syslog_priority

expand_syslog_priority(value: any)

Converts the syslog priority number in value to its numeric severity and facility values.

expand_syslog_priority_as_string

expand_syslog_priority_as_string(value: any)

Converts the syslog priority number in value to its severity and facility string representations.

now

now([timezone: string])

Returns the current date and time. Uses the default time zone UTC.

parse_date

parse_date(value: string, pattern: string, [timezone: string])

Parses the value into a date and time object, using the pattern. If no timezone is detected in the pattern, the optional timezone parameter is used as the assumed timezone. If omitted the timezone defaults to UTC.

The format used for the pattern parameter is identical to the pattern of the Joda-Time DateTimeFormat.

Symbol Meaning Presentation Examples
G era text AD
C century of era (>=0) number 20
Y year of era (>=0) year 1996
x weekyear year 1996
w week of weekyear number 27
e day of week number 2
E day of week text Tuesday; Tue
y year year 1996
D day of year number 189
M month of year month July; Jul; 07
d day of month number 10
a halfday of day text PM
K hour of halfday (0~11) number 0
h clockhour of halfday (1~12) number 12
H hour of day (0~23) number 0
k clockhour of day (1~24) number 24
m minute of hour number 30
s second of minute number 55
S fraction of second millis 978
z time zone text Pacific Standard Time; PST
Z time zone offset/id zone -0800; -08:00; America/Los_Angeles
' escape for text delimiter  
'' single quote literal

flex_parse_date

flex_parse_date(value: string, [default: DateTime], [timezone: string])

Uses the Natty date parser to parse a date and time value. If no timezone is detected in the pattern, the optional timezone parameter is used as the assumed timezone. If omitted the timezone defaults to UTC.

In case the parser fails to detect a valid date and time the default date and time is being returned, otherwise the expression fails to evaluate and will be aborted.

format_date

format_date(value: DateTime, format: string, [timezone: string])

Returns the given date and time value formatted according to the format string. If no timezone is given, it defaults to UTC.

to_date

to_date(value: any, [timezone: string])

Converts value to a date. If no timezone is given, it defaults to UTC.

years

years(value: long)

Create a period with value number of years.

months

months(value: long)

Create a period with value number of months.

weeks

weeks(value: long)

Create a period with value number of weeks.

days

days(value: long)

Create a period with value number of days.

hours

hours(value: long)

Create a period with value number of hours.

minutes

minutes(value: long)

Create a period with value number of minutes.

seconds

seconds(value: long)

Create a period with value number of seconds.

millis

millis(value: long)

Create a period with value number of milliseconds.

period

period(value: string)

Parses an ISO 8601 period from value.