Cluster Upgrade from 4.50 to 4.51

This guide will lead you through the steps specific for upgrading a NetEye Cluster installation from version 4.50 to 4.51.

Granted the environment connectivity is seamless, the upgrade procedure may take up to 30 minutes per node.

Warning

Remember that you must upgrade sequentially without skipping versions, therefore an upgrade to 4.51 is possible only from 4.50; for example, if you have version 4.27, you must first upgrade to the 4.28, then 4.29, and so on.

Breaking Changes

NetEye Kubernetes Operator

This release will introduce a brand new NetEye Operator which is the component responsible for managing the NetEye deployment in Kubernetes. All configuration and management is made by the standard NetEye CLI procedures such as neteye install and neteye upgrade, but you can configure and override the default configuration by editing the /etc/neteye-environment.yaml file. NetEye CLI procedures will automatically source this configuration file and apply desired changes to the Operator. This upgrade does not require users to update the configuration beforehand since default values will be applied. For the full list of configurable parameters, please refer to the /usr/share/neteye/setup/neteye-environment.yaml.tpl file. To apply the configuration changes during normal operation, you can use the neteye install --restrict-services-to neteye-operator command. This command will apply the configuration changes to the Operator without affecting the other services.

Prerequisites

Before starting the upgrade, carefully read the latest release notes on NetEye’s blog and check the features that will change or be deprecated.

1. All NetEye packages installed on a currently running version must be updated according to the update procedure prior to running the upgrade.

2. NetEye must be up and running in a healthy state.

3. Disk Space required:

   • 3GB for / and /var

   • 150MB for /boot

4. If the NetEye Elastic Stack module is installed:

   1. The rubygems.org domain should be reachable by the NetEye Master only during the update/upgrade procedure. This domain is needed to update additional Logstash plugins and thus is required only if you manually installed any Logstash plugin that is not present by default.

   2. There is a number of configuration items that should not be modified in order to avoid issues during the update/upgrade of your instance. Please check out Protected Configuration Items for details.

5. NetEye services are progressively moving to Kubernetes to improve scalability, security, resource management and the speed of updates.

   Access to the following domains is strictly required for upgrading to NetEye 4.51: Please ensure that all NetEye nodes can reach these domains over HTTPS (TCP port 443):

   Domain	Port	Intended Use
   ghcr.io	443 TCP	GitHub container images
   api.github.com	443 TCP	GitHub container images
   pkg-containers.githubusercontent.com	443 TCP	GitHub container images
   quay.io	443 TCP	Quay container images
   cdn01.quay.io	443 TCP	Quay container images
   docker.io	443 TCP	Docker Hub container images
   hub.docker.com	443 TCP	Docker Hub container images
   auth.docker.io	443 TCP	Docker Hub container images
   index.docker.io	443 TCP	Docker Hub container images
   registry-1.docker.io	443 TCP	Docker Hub container images
   production.cloudfront.docker.com	443 TCP	Docker Hub container images
   *.cloudflarestorage.com	443 TCP	Container images
   rpm.rancher.io	443 TCP	Rancher packages
   docker.elastic.co	443 TCP	Elastic container images (only with the Elastic Stack)
   docker-auth.elastic.co	443 TCP	Elastic container images (only with the Elastic Stack)

1. Run the Upgrade

The Cluster Upgrade is carried out by running the following command:

cluster# (nohup neteye upgrade &) && tail --retry -f nohup.out

Warning

If the NetEye Elastic Stack feature module is installed and a new version of Elasticsearch is available, please note that the procedure may take a while to upgrade the Elasticsearch cluster. For more information on the Elasticsearch cluster upgrade and how to customize the upgrade process, please consult the dedicated section.

After the command was executed, the output will inform if the upgrade was successful or not:

• In case of successful upgrade you might need to restart the nodes to properly apply the upgrades. If the reboot is not needed, please skip the next step.

• In case the command fails refer to the troubleshooting section.

2. Reboot Nodes

Restart each node, one at a time, to apply the upgrades correctly.

1. Run the reboot command

   cluster-node-N# neteye node reboot
   

2. In case of a standard NetEye node, put it back online once the reboot is finished

   cluster-node-N# pcs node unstandby --wait=300
   

You can now reboot the next node.

3. Cluster Reactivation

At this point you can proceed to restore the cluster to high availability operation.

1. Run the checks in the section Checking that the Cluster Status is Normal. If any of the above checks fail, please contact our service and support team before proceeding.

2. Re-enable fencing on the last standard node, if it was enabled prior to the upgrade:

   cluster# pcs property set stonith-enabled=true
   

4. Additional Tasks

Keycloak in Kubernetes and the NetEye Operator

After a successful upgrade, you will be required to perform and verify the migration of the Keycloak systemd/pcs instance to the new Keycloak deployment in Kubernetes, which will be an upgrade prerequisite check for future releases. To automate this process, the product ships 3 utility tools and scripts that allow you to perform the validation and migration.

1. neteye config auth switch-to-kube: This command will switch the authentication configuration from the systemd/pcs instance to the new Keycloak deployment in Kubernetes by updating the necessary configuration files. After successful execution, the systemd/pcs instance will be disabled, and you will be required to test the new environment to ensure that authentication is working as expected.

2. neteye config auth rollback-from-kube: Should any issues arise with the new Keycloak deployment in Kubernetes, this command will allow you to roll back the authentication configuration to the previous systemd/pcs instance by restoring the necessary configuration files.

3. neteye config auth cleanup-keycloak-pcs: Finally, once you are satisfied with the new Keycloak deployment, this command will clean up the old systemd/pcs instance by removing any remaining configuration files and data related to the previous Keycloak deployment. Backups of the old configuration files will be created in the /root/keycloak_upgrade_<timestamp>.bck directory. This step is non-reversible, and you will not be able to roll back to the previous systemd/pcs instance after executing this command.