Upgrading to Graylog 3.0.x

Elasticsearch Version Requirements

Graylog 3.0 drops support for Elasticsearch versions before 5.6.x. That means you have to upgrade Elasticsearch to at least version 5.6.13 before upgrading Graylog to version 3.0. Make sure to read the Elasticsearch upgrade guides before doing that.

Simplified HTTP interface configuration

Graylog used to have a lot of different settings regarding the various HTTP interfaces it provides, namely the Graylog REST API and the Graylog web interface.

This mostly originates from the fact that Graylog used to consist of two components before Graylog 2.0.0, a server component and a separate web interface.

The changes in this release finally merge the HTTP listeners for the Graylog REST API and web interface into a single HTTP listener, which should make the initial configuration of Graylog simpler and reduce errors caused by conflicting settings.

The path of the Graylog REST API is now hard-coded to /api, so if you’re still using the legacy URI on port 12900/tcp or have been using a custom path (via the rest_listen_uri or rest_transport_uri settings), you’ll have to update the URI used to access the Graylog REST API.

This might also affect your Graylog Collector Sidecars. Make sure to check each collector_sidecar.yml and update the server_url accordingly.

If you are using a reverse proxy in front of Graylog (like nginx) and configured it to set the X-Graylog-Server-URL HTTP header, you have to remove the api/ suffix because that is now the default. (as mentioned above)


# This nginx setting in Graylog <3.0 ...
header_upstream X-Graylog-Server-URL http://{host}/api

# ... needs to be changed to the following with Graylog 3.0
header_upstream X-Graylog-Server-URL http://{host}/

For a more detailed description of the new HTTP settings, please consult the annotated Graylog configuration file.

Overview of removed Graylog REST API settings:

Removed Setting New Setting Default
rest_listen_uri http_bind_address
rest_transport_uri http_publish_uri http://$http_bind_address/
web_endpoint_uri http_external_uri $http_publish_uri
rest_enable_cors http_enable_cors true
rest_enable_gzip http_enable_gzip true
rest_max_header_size http_max_header_size 8192
rest_max_initial_line_length http_max_initial_line_length 4096
rest_thread_pool_size http_thread_pool_size 16
rest_enable_tls http_enable_tls false
rest_tls_cert_file http_tls_cert_file Empty
rest_tls_key_file http_tls_key_file Empty
rest_tls_key_password http_tls_key_password Empty

Overview of removed Graylog web interface settings:

Removed Setting New Setting Default
web_enable None  
web_listen_uri http_bind_address
web_enable_cors http_enable_cors true
web_enable_gzip http_enable_gzip true
web_max_header_size http_max_header_size 8192
web_max_initial_line_length http_max_initial_line_length 4096
web_thread_pool_size http_thread_pool_size 16
web_enable_tls http_enable_tls false
web_tls_cert_file http_tls_cert_file Empty
web_tls_key_file http_tls_key_file Empty
web_tls_key_password http_tls_key_password Empty

Plugins merged into the Graylog server

Starting with Graylog 3.0.0, the following official plugins were merged into the Graylog server:

That means these plugins are not available as separate plugins anymore. If you manually update your Graylog installation (without using operating system packages), make sure to remove all old plugin files from the plugin_dir folder.

The old issues in these repositories are still available for reference but new issues should only be created in the Graylog server issue tracker.

The following HTTP API paths changed due to the plugin merge:

Old Path New Path
/plugins/org.graylog.plugins.map/mapdata /search/mapdata
/plugins/org.graylog.plugins.pipelineprocessor/system/pipelines/pipeline /system/pipelines/pipeline
/plugins/org.graylog.plugins.pipelineprocessor/system/pipelines/pipeline/parse /system/pipelines/pipeline/parse
/plugins/org.graylog.plugins.pipelineprocessor/system/pipelines/rule /system/pipelines/rule
/plugins/org.graylog.plugins.pipelineprocessor/system/pipelines/rule/functions /system/pipelines/rule/functions
/plugins/org.graylog.plugins.pipelineprocessor/system/pipelines/rule/multiple /system/pipelines/rule/multiple
/plugins/org.graylog.plugins.pipelineprocessor/system/pipelines/rule/parse /system/pipelines/rule/parse
/plugins/org.graylog.plugins.pipelineprocessor/system/pipelines/connections /system/pipelines/connections
/plugins/org.graylog.plugins.pipelineprocessor/system/pipelines/connections/to_stream /system/pipelines/connections/to_stream
/plugins/org.graylog.plugins.pipelineprocessor/system/pipelines/connections/to_pipeline /system/pipelines/connections/to_pipeline
/plugins/org.graylog.plugins.pipelineprocessor/system/pipelines/simulate /system/pipelines/simulate

New “bin_dir” and “data_dir” configuration parameters

We introduced two new configuration parameters related to file system paths.

  • bin_dir config option points to the directory that contains scripts like graylogctl.
  • data_dir option configures the base directory for Graylog server state.

Please check the updated default graylog.conf configuration file for required changes to your existing file.

Removed support for Drools-based filters

For a long time, Graylog allowed to use Drools to filter messages. Unfortunately, using Drools to perform complex filter logic came with a performance penalty and wasn’t as flexible as we would have liked it to be.

Starting with Graylog 3.0.0, the support for Drools-based message filters has been removed from Graylog. The rules_file configuration setting has been removed accordingly.

We recommend migrating the Drools-based logic to Processing Pipelines.

Drools-based blacklist

Graylog provided undocumented blacklist-functionality based on Drools. This blacklist could only be modified via the Graylog REST API on the /filters/blacklist resource.

If you’ve been using this functionality, you’ll have to migrate these blacklist rules to the Processing Pipelines.

To check if you’re using the Drools-based blacklist in Graylog prior to version 3.0.0, you can run the following command:

# curl -u admin:password -H 'Accept: application/json' 'http://graylog.example.com/api/filters/blacklist?pretty=true'

String-based blacklist rule

Old blacklist rule:

   "id" : "54e300001234123412340001",
   "type" : "string",
   "name" : "String Blacklist",
   "description" : "Drop messages based on case-insensitive string comparison",
   "fieldName" : "custom_field",
   "pattern" : "EXAMPLE pattern",
   "creator_user_id" : "admin",
   "created_at" : "2018-04-04T12:00:00.000Z"

New pipeline rule:

rule "string-blacklist"
  has_field("custom_field") &&
  lowercase(to_string($message.custom_field)) == "example pattern"

See also:

Regex-based blacklist rule

Old blacklist rule:

   "id" : "54e300001234123412340002",
   "type" : "regex",
   "name" : "Regex Blacklist",
   "description" : "Drop messages based on regular expression",
   "fieldName" : "custom_field",
   "pattern" : "^EXAMPLE.*",
   "creator_user_id" : "admin",
   "created_at" : "2018-04-04T12:00:00.000Z"

New pipeline rule:

rule "regex-blacklist"
  has_field("custom_field") &&
  regex("^EXAMPLE.*", to_string($message.custom_field)).matches == true

See also:

IP Range-based blacklist rule

Old blacklist rule:

   "id" : "54e300001234123412340003",
   "type" : "iprange",
   "name" : "IP Blacklist",
   "description" : "Drop messages based on IP address",
   "fieldName" : "custom_field",
   "pattern" : "",
   "creator_user_id" : "admin",
   "created_at" : "2018-04-04T12:00:00.000Z"

New pipeline rule:

rule "ip-blacklist"
  has_field("custom_field") &&
  cidr_match("", to_ip($message.custom_field))

See also:

Changed metrics name for stream rules

The name of the metrics for stream rules have been changed to include the stream ID which helps identifying the actual stream they are related to.

Old metric name:


New metric name:


Email alarm callback default settings

The defaults of the configuration settings for the email alarm callback with regard to encrypted connections have been changed.

Setting Old default New default
transport_email_use_tls false true
transport_email_use_ssl true false

Furthermore, it’s not possible anymore to enable both settings (SMTP with STARTTLS and SMTP over SSL) at the same time because this led to errors at runtime when Graylog tried to upgrade the connection to TLS with STARTTLS in an already existing SMTPS connection.

Most SMTP services prefer SMTP with STARTTLS to provide an encrypted connection.

Collector Sidecar is deprecated

Graylog 3.0 comes with a new Sidecar implementation. We still support the old Collector Sidecars, which can be found in the System / Collectors (legacy) menu entry. For more information check the Sidecar documentation and the Upgrade guide.

Legacy Content Packs

The implementation of content packs where fundamentally reworked. Parameters were added and checks implemented to give the user better usability. This rework did come with the cost that old content packs might not work any longer and stop the new content packs from loading. If the content packs page does not finish loading we recommend to remove the old content packs from your MongoDB. For that, please connect to your MongoDB shell and remove all content packs with the following command:

> db.content_packs.deleteMany({})

This command will only remove the content packs, it will not remove the installed configurations.

Elasticsearch 6 changes

There is a breaking change in Elasticsearch 6 that may affect some queries on your searches and dashboards:

Before Elasticsearch 6, queries for keyword fields were split by whitespaces and combined with OR operators resulting, for example, in type:(ssh login) and type:(ssh OR login) being equivalent. This is no longer the case in Elasticsearch 6 and now those queries are different: the former looking for the ssh login value, the second for either ssh or login values.

Please ensure to look for those queries in your Graylog setup before upgrading to Elasticsearch 6 and add the OR operators where needed.