Cluster Upgrade from 4.32 to 4.33¶

This guide will lead you through the steps specific for upgrading a NetEye Cluster installation from version 4.32 to 4.33.

During the upgrade, individual nodes will be put into standby mode. Thus, overall performance will be degraded until the upgrade is completed and all nodes are revoked from standby mode. Granted the environment connectivity is seamless, the upgrade procedure may take up to 30 minutes per node.

Warning

Remember that you must upgrade sequentially without skipping versions, therefore an upgrade to 4.33 is possible only from 4.32; for example, if you have version 4.27, you must first upgrade to the 4.28, then 4.29, and so on.

Breaking Changes¶

Upgrade of Elastic Stack to version 8.10.2¶

In NetEye 4.33, we upgraded the Elastic Stack from version 8.8.2 to version 8.10.2.

To ensure the full compatibility of your installation, please review the official release notes, focusing in particular on the breaking changes of the Elastic Stack components: Elasticsearch, Logstash, Kibana and Beats.

Please note that, for a complete review, the release notes of all versions between 8.8.2 and 8.10.2 should be reviewed.

APM Server migration to Elastic Agent¶

With the new NetEye 4.33 version, we will introduce the new implementation of the APM Server which will be hosted in the Elastic Agents enrolled in the Operative nodes of the NetEye system.

With the new implementation, APM Server as well as Elastic Agents will work out-of-the-box without the need for prior configuration, thus allowing a wide range of customization features, available after following the instructions in the Elastic Agent section.

The Elastic Agent package will be moved out from the preview stage, being officially supported by NetEye. Taking into consideration this change, in order to ensure consistent upgrade path, also to future versions, it is important to distinguish among the following two scenarios:

I haven’t used Elastic Agent yet¶

Since Elastic Agents will work right away after the NetEye upgrade to 4.33, there is pretty much nothing to configure and you can freely use the new feature, explained in Elastic Agent. Make sure that the system still satisfy the dedicated requirements.

I am already using Elastic Agent and I already configured it to work in NetEye¶

NetEye installations that were already using Elastic Agent functionality prior to NetEye version 4.33 should perform a migration process after the upgrade of NetEye. This operation is needed since official NetEye-managed policies will be used from version 4.34 onwards for NetEye nodes. In order to proceed with the migration process please follow the dedicated section in Cluster Additional Tasks or Single Node Additional Tasks, depending on your installation.

Note

After the upgrade, APM Server will not be available until you migrate the Elastic Agents running on the Operative Nodes to the NetEye-managed policies. To minimize the downtime of APM Server, please perform the migration of the Elastic Agents right after finishing the upgrade.

Reporting users read-only permission¶

We upgraded the Reporting module. The new version adds more granular permissions. From NetEye 4.33, the users that have only the General Module Access permission, cannot edit the reports anymore. To allow those users to edit the reports, it is mandatory to give them the dedicated reporting/reports permission. The users with the Full Module Access permission are still able to manage the whole Reporting module.

Eventhandler Deprecation Notice¶

In NetEye 4.34, the Eventhandler module will be permanently removed from each NetEye installation.

Refer to the migration procedure described in Migration to Tornado SMS Collector in order to correctly migrate all you Eventhandler rules to Tornado.

Prerequisites¶

Before starting the upgrade, you should read very carefully the latest release notes on NetEye’s blog and check out the features that will be changed or deprecated after the upgrade.

All NetEye packages installed on a currently running version must be updated according to the update procedure prior to running the upgrade.
NetEye must be up and running in a healthy state.
Disk Space required:
- 3GB for / and /var
- 150MB for /boot

1. Run the Upgrade¶

The Cluster Upgrade is carried out by running the following command:

cluster# (nohup neteye upgrade &) && tail --retry -f nohup.out

Warning

If the SIEM feature module is installed and a new version of Elasticsearch is available, please note that the procedure will upgrade one node at the time and wait for the Elasticsearch cluster health status to turn green before proceeding with the next node. For more information, please consult the dedicated section.

After the command was executed, the output will inform if the upgrade was successful or not:

In case of successful upgrade you might need to restart the nodes to properly apply the upgrades. If the reboot is not needed, please skip the next step.
In case the command fails refer to the troubleshooting section.

2. Reboot Nodes¶

Restart each node, one at a time, to apply the upgrades correctly.

Run the reboot command
cluster-node-N# neteye node reboot
In case of a standard NetEye node, put it back online once the reboot is finished
cluster-node-N# pcs node unstandby --wait=300

You can now reboot the next node.

3. Cluster Reactivation¶

At this point you can proceed to restore the cluster to high availability operation.

Bring all cluster nodes back out of standby with this command on the last standard node
cluster# pcs node unstandby --all --wait=300 cluster# echo $?
0
If the exit code is different from 0, some nodes have not been reactivated, so please make sure that all nodes are active before proceeding.
Run the checks in the section Checking that the Cluster Status is Normal. If any of the above checks fail, please call our service and support team before proceeding.
Re-enable fencing on the last standard node, if it was enabled prior to the upgrade:
```
cluster# pcs property set stonith-enabled=true
```

4. Additional Tasks¶

Elastic Agent migration¶

This section applies only to NetEye installations that were using Elastic Agent before the upgrade to NetEye 4.33. In order to proceed with the migration of Elastic Agent to the official supported configuration, the following points can be considered:

The new NetEye version will not apply any change in the configuration of the current Elastic Agent system: the upgrade process will only create two new policies without applying them to any agent.

The new policies, explained in detail in the Elastic Agent section, will be official policies used by NetEye that should be applied to every Elastic Agent in the NetEye system.
In order to maintain the current status of Elastic Agent, the configuration in use should be migrated to the official NetEye policies that are present but not used by any agent. Please follow the instructions in Elastic Agent section to perform the migration of the settings.
After the configuration of the official NetEye policies for Elastic Agents has been completed, the policies should be applied to the Elastic Agents running on the nodes of the NetEye Master, as follows:
1. Firstly, the official NetEye policy needs to be applied to the Elastic Agents running on the NetEye Operative nodes. Since the script affects only the node on which it is executed, allowing so a fine-grained migration, it should be run on every NetEye Operative node as follows:
  sh /usr/share/neteye/elastic-agent/scripts/switch_elastic_agent_to_neteye_policy.sh
2. Secondly, the Nginx configuration of the Fleet Server integration needs to be applied. For this, a dedicated script will apply the official Fleet Server configuration file (fleet-server-loadbalanced.conf) in the Nginx configuration folder (/neteye/shared/nginx/conf/conf.d/).
  
  In order to avoid failures during the migration process, it is important to make sure that the following points are satisfied:
  - The port 8220 is available on the external interface
  - Nginx is free from any custom configuration related to Fleet Server so, if you configured any custom rule for Fleet Server under /neteye/shared/nginx/conf/conf.d/, please remove it before proceeding.
  To finalize this step, the following command can be executed on any node:
  sh /usr/share/neteye/elastic-agent/scripts/apply_standard_nginx_config_for_fleet_server.sh
3. Lastly, the official NetEye policy needs to be applied to the Elastic Agents running on the NetEye Single-Purpose nodes. In order to do this, the already used script can be executed on each NetEye Single-Purpose node, as follows:
  sh /usr/share/neteye/elastic-agent/scripts/switch_elastic_agent_to_neteye_policy.sh
It is important to mention that with NetEye 4.34, the system will force the adoption of the official NetEye policies for Elastic Agents.