- 09 Sep 2021
- 5 Minutes to read
- Updated on 09 Sep 2021
- 5 Minutes to read
This guide doesn’t provide a step-by-step tutorial for building a multi-node Graylog cluster but does simply give some advice for questions that might arise during the setup.
It’s important for such a project that you understand each step in the setup process and do some planning upfront. Without a proper roadmap of all the things you want to achieve with a Graylog cluster, you will be lost on the way.
Graylog should be the last component you install in this setup. Its dependencies, namely MongoDB and Elasticsearch, have to be up and running first.
This guide doesn’t include instructions for running a multi-node Graylog cluster in an untrusted network. We assume that the connection between the hosts is trusted and doesn’t have to be secured individually.
Every server which is part of this setup should have the software requirements installed to run the targeted software. All software requirements can be found in the installation manual.
We highly recommend that the system time on all systems is kept in sync via NTP or a similar mechanism. Needless to say that DNS resolution must be working, too. Because everything is a freaking DNS problem.
In order to simplify the installation process, the servers should have a working Internet connection.
MongoDB replica set
We recommend to deploy a MongoDB replica set.
MongoDB doesn’t have to run on dedicated servers for the workload generated by Graylog, but you should follow the recommendations given in the MongoDB documentation about architecture. Most important is that you have an odd number of MongoDB servers in the replica set.
In most setups, each Graylog server will also host an instance of MongoDB which is part of the same replica set and shares the data with all other nodes in the cluster.
To avoid unauthorized access to your MongoDB database, the MongoDB replica set should be setup with authentication.
The correct order of working steps should be as follows:
- Create the replica set (
- Create the database (
- Create a user account for accessing the database, which has the roles
If your MongoDB needs to be reachable over network you should set the IP with
bind_ipin the configuration.
The Elasticsearch setup documentation should help you to install Elasticsearch with a robust base configuration.
It is important to name the Elasticsearch cluster not simply named to avoid accidental conflicts with Elasticsearch nodes using the default configuration. Just choose anything else (we recommend ), because this is the default name and any Elasticsearch instance that is started in the same network will try to connect to this cluster.
The Elasticsearch servers need one IP that can be reached over network set in
network.hostand some participants of the cluster in
discovery.zen.ping.unicast.hosts. That is enough to have a minimal cluster setup.
After the installation of Graylog, you should take care that only one Graylog node is configured to be master with theconfiguration setting
is_master = true.
http_bind_addressconfigured address needs to be reachable by all Graylog nodes in the cluster.The
http_publish_uriis normally auto-generated from the
http_bind_address.This URI is used for the inter-node communication.
If the http_bind_address is configured with
0.0.0.0you must configure
http_publish_uri.Otherwise Graylog will use the first non loopback IP, what might not fit into your desired design.All Graylog nodes need to reach all other Graylog nodes via their configured
http_publish_urifor inter-nodecommunication. If you use TLS in your Graylog configuration, this includes https as protocol.
Graylog to MongoDB connection
mongodb_uriconfiguration setting must include all MongoDB nodes forming the replica set, the name of the replica set, as well as the previously configured user account with access to the replica set. The configuration setting is a normal MongoDB connection string .
Finally, the MongoDB connection string in the Graylog configuration file should look like this:
mongodb_uri = mongodb://USERNAME:PASSWORD@mongodb-node01:27017,mongodb-node02:27017,mongodb-node03:27017/graylog?replicaSet=rs01
Graylog to Elasticsearch connection
Graylog will connect to the Elasticsearch REST API .
To avoid issues with the connection to the Elasticsearch cluster you should add some of the network addresses of the Elasticsearch nodes to
Graylog web interface
It’s possible to use a loadbalancer in front of all Graylog servers, please refer to Making the web interface work with load balancers/proxies for more details.
Each component in this multi-node setup can be scaled on the individual needs.
Depending on the amount of messages ingested and how long messages should be available for direct search, the Elasticsearch cluster will need most of the resources on your setup.
Keep an eye on the Metrics of each part of the cluster. One option is to use telegraf to fetch importand metrics and store them in your favorite metric system (e. g. Graphite, Prometheus or Influx).
Graylog Metrics can be monitored with the Graylog Metrics Reporter plugins which are able to send the internal Graylog metrics to your favorite metrics collector (e. g. Graphite or Prometheus).
Up until today, we have almost never faced the issue that the MongoDB replica set needed special attention. But of course you should still monitor it and store its metrics - just to be sure.
After every configuration change or service restart, watch the logfile of the applications you have worked on. Sometimes other log files can also give you hints about what went wrong. For example if you’re configuring Graylog and try to find out why the connection to the MongoDB isn’t working, the MongoDB logs can help to identify the problem.
If HTTPS has been enabled for the Graylog REST API , it need to be setup for the Graylog web interface, too.