Cluster Upgrade from 4.39 to 4.40

This guide will lead you through the steps specific for upgrading a NetEye Cluster installation from version 4.39 to 4.40.

During the upgrade, individual nodes will be put into standby mode. Thus, overall performance will be degraded until the upgrade is completed and all nodes are revoked from standby mode. Granted the environment connectivity is seamless, the upgrade procedure may take up to 30 minutes per node.

Warning

Remember that you must upgrade sequentially without skipping versions, therefore an upgrade to 4.40 is possible only from 4.39; for example, if you have version 4.27, you must first upgrade to the 4.28, then 4.29, and so on.

Breaking Changes

Manual Elasticsearch service restart

From NetEye 4.40 onwards, manually restarting the elasticsearch systemd service (i.e. executing the command systemctl restart elasticsearch.service) may cause the shards of the Elasticsearch node to be reallocated to other nodes, since the elasticsearch.service will not anymore take care of pausing the reallocation of shards when restarting the service. This does not affect the NetEye Update and Upgrade procedures, which will still automatically take care of avoiding unneeded reallocations when restarting the Elasticsearch nodes.

For this reason, if you want to make sure that no reallocation happens when you manually restart an Elasticsearch node, you can temporarily disable shard allocation via the cluster.routing.allocation.enable setting and re-enable it afterwards, in a similar way as described in the official rolling restart procedure

Note

Additionally, the variable names used in the NetEye upgrade process to manage the cluster status waiting time have been updated. For more details, refer to the dedicated section.

Elastic 8.17

In NetEye 4.40, Elastic Stack is upgraded from version 8.16.0 to version 8.17.0.

To ensure the full compatibility of your installation, please review the official release notes, focusing in particular on the breaking changes of the Elastic Stack components.

Among the known issues and breaking changes listed by Elastic, we would like to emphasize those that may have more impact on NetEye installations than the rest:

Breaking changes:

1. Elasticsearch: Change to synthetic _source licensing. It is important to note that, thanks to a grandfathering policy, this change will not immediately impact NetEye installations. The feature will continue to be supported under the currently active Platinum License, which is set to expire in October 2026.

2. Beats: Drop support for Debian 10

3. Elastic Agent: Drop support for Debian 10

Known issues:

1. Beats: Standalone Beats docker image will not start if -e option is not added

2. Elastic Agent: Elastic Agents with the Defend integration may not be upgradable through Fleet

3. Security: Exception tab does not load if comments contain newlines

4. Security: Duplicate alerts can be produced from manually running threshold rules

5. Security: Manually running custom query rules with suppression could suppress more alerts than expected

Prerequisites

Before starting the upgrade, you should read very carefully the latest release notes on NetEye’s blog and check out the features that will be changed or deprecated after the upgrade.

1. All NetEye packages installed on a currently running version must be updated according to the update procedure prior to running the upgrade.

2. NetEye must be up and running in a healthy state.

3. Disk Space required:

   • 3GB for / and /var

   • 150MB for /boot

4. If the SIEM module is installed:

   • The rubygems.org domain should be reachable by the NetEye Master only during the update/upgrade procedure. This domain is needed to update additional Logstash plugins and thus is required only if you manually installed any Logstash plugin that is not present by default.

1. Run the Upgrade

The Cluster Upgrade is carried out by running the following command:

cluster# (nohup neteye upgrade &) && tail --retry -f nohup.out

Warning

If the SIEM feature module is installed and a new version of Elasticsearch is available, please note that the procedure will upgrade one node at the time and wait for the Elasticsearch cluster health status to turn green before proceeding with the next node. For more information, please consult the dedicated section.

After the command was executed, the output will inform if the upgrade was successful or not:

• In case of successful upgrade you might need to restart the nodes to properly apply the upgrades. If the reboot is not needed, please skip the next step.

• In case the command fails refer to the troubleshooting section.

2. Reboot Nodes

Restart each node, one at a time, to apply the upgrades correctly.

1. Run the reboot command

   cluster-node-N# neteye node reboot
   

2. In case of a standard NetEye node, put it back online once the reboot is finished

   cluster-node-N# pcs node unstandby --wait=300
   

You can now reboot the next node.

3. Cluster Reactivation

At this point you can proceed to restore the cluster to high availability operation.

1. Bring all cluster nodes back out of standby with this command on the last standard node

   cluster# pcs node unstandby --all --wait=300
   cluster# echo $?
   

   0
   

   If the exit code is different from 0, some nodes have not been reactivated, so please make sure that all nodes are active before proceeding.

2. Run the checks in the section Checking that the Cluster Status is Normal. If any of the above checks fail, please call our service and support team before proceeding.

3. Re-enable fencing on the last standard node, if it was enabled prior to the upgrade:

   cluster# pcs property set stonith-enabled=true
   

4. Additional Tasks