When you are planning to upgrade your Elasticsearch software to version 7.x, there are a few things to consider.

First, determine whether any existing Elasticsearch indices need to be re-indexed to upgrade their versions. Indices created by Elasticsearch are versioned. When an Elasticsearch index is created, the version of that index is the same version as the Elasticsearch cluster that created it. New versions of Elasticsearch software typically introduce a new version for indices as changes to underlying data structures are introduced while maintaining one full-version of backwards compatibility. It is important to note that the versions of existing Elasticsearch indices are not automatically upgraded when the Elasticsearch software itself is upgraded. So if you are preparing to upgrade the Elasticsearch software, it is necessary to first upgrade the versions of indices themselves. To upgrade versions of existing Elasticsearch indices, those indices simply must be re-indexed.

If all of your currently existing indices have been created on your current major version and you are upgrading from at least Elasticsearch 6.x, then re-indexing indices is optional due to the backwards compatibility. If you do know that you have older indices or you are unsure, it is best to check before doing the upgrade.

So, let's say you want to upgrade to Elasticsearch 7. The oldest index version it supports is from Elasticsearch 6.0.0. The internal version representation for this is 6000000 (major version * 1000000).

Now we can check if we have any indices created by a version older than that:

curl -X GET "http://localhost:9200/_settings?pretty=true" | jq '.[] | select(.settings.index.version.created <= 6000000) | [.settings.index.provided_name, .settings.index.version.created]'

The above example uses the tools httpie and jq to query the Elasticsearch API at localhost port 9200. Please adjust the command to your Elasticsearch server’s URL and the required version.

If this command returns any index names, then those are indices that will not be readable after the upgrade and that will need a re-index before they work with the next major version.

Upgrade without Re-Index

When no re-index is needed, the easiest way is to follow the elastic upgrade guide for Elasticsearch this gives all needed commands. Please use the guide corresponding to your version.

Upgrade with Re-Index

If you have identified any incompatible indices in the previous step, those need to be reindexed. First a brief overview what steps need to be performed followed by the list of commands. Once you started the process of re-index your data you need to finish all steps to get a working Graylog and Elasticsearch cluster again.

1. Upgrade to the latest patch release of your current Graylog.
2. If you have not been on Elasticsearch 6 before, please follow the Elasticsearch 6 Upgrade Notes.
3. Upgrade to Graylog 4.x. All of your pre-existing custom index mapping templates should still exist.
4. (Optional) Reindex indices on ES7 that were created on ES6, so the upgrade to ES8 will be painless. Use the Reindexing Procedure steps.