Cluster Upgrade from 4.37 to 4.38¶

This guide will lead you through the steps specific for upgrading a NetEye Cluster installation from version 4.37 to 4.38.

During the upgrade, individual nodes will be put into standby mode. Thus, overall performance will be degraded until the upgrade is completed and all nodes are revoked from standby mode. Granted the environment connectivity is seamless, the upgrade procedure may take up to 30 minutes per node.

Warning

Remember that you must upgrade sequentially without skipping versions, therefore an upgrade to 4.38 is possible only from 4.37; for example, if you have version 4.27, you must first upgrade to the 4.28, then 4.29, and so on.

Breaking Changes¶

New Authentication Method¶

The authentication method has been updated to enable new features such as login in NetEye with federated identity providers through protocols such as OIDC (OAuth 2.0), SAML 2.0, or LDAP/Active Directory.

NetEye 4.38 introduces Keycloak as its access manager, enhancing security by enabling single sign-on (SSO) and role-based access control (RBAC) across NetEye’s advanced monitoring and threat protection features.

Keycloak is an open-source identity and access management (IAM) solution that provides authentication, authorization, and user management for modern applications. It is effectively the new hub for managing users and groups in NetEye.

This integration ensures that only authenticated and authorized users can access NetEye’s critical services, streamlining user management while maintaining robust security. Keycloak’s flexible integration with various identity providers further strengthens NetEye’s secure and unified monitoring environment.

More information about the new authentication method can be found in the Authentication section. This important change requires some additional steps to be taken before and after upgrading to the new version.

Warning

The local IcingaWeb2 users will be migrated to Keycloak during the upgrade process. The migration will be performed automatically, and the users will be able to log in with the new method using their existing credentials. Please note that those users cannot have usernames shorter than 3 characters. If you have users with a username shorter than 3 characters, you will need to update the username before running the upgrade.

Upgrade of Elastic Stack to version 8.15¶

In NetEye 4.38, Elastic Stack is upgraded from version 8.14.3 to version 8.15.2.

To ensure the full compatibility of your installation, please review the official release notes, focusing in particular on the breaking changes of the Elastic Stack components.

Among the known issues and breaking changes listed by Elastic, we would like to emphasize those that may have more impact on NetEye installations than the rest:

input-snmp and input-snmptrap Logstash plugins are now bundled in the new Logstash SNMP integration plugin, so a migration may be needed

Drop-in directory for El Proxy configuration¶

In NetEye 4.38, the configuration of the El Proxy service has been moved to a drop-in directory logic. This change allows you to customize the El Proxy configuration by creating .toml files in the drop-in directory without modifying the default configuration file /neteye/shared/elastic-blockchain-proxy/conf/elastic_blockchain_proxy.toml, which from now on will be entirely managed by NetEye.

Due to this change, if you modified the default configuration file /neteye/shared/elastic-blockchain-proxy/conf/elastic_blockchain_proxy.toml, during the upgrade to NetEye 4.38 an .rpmsave file containing your custom configuration will be created under /neteye/shared/elastic-blockchain-proxy/conf/elastic_blockchain_proxy.toml.rpmsave.

If this occurs, you can migrate your custom configuration as follows:

Move in the directory /neteye/shared/elastic-blockchain-proxy/conf/
Analyze the differences between the default configuration file elastic_blockchain_proxy.toml and your custom configuration file elastic_blockchain_proxy.toml.rpmsave and understand which custom settings you want to keep
Create a new .toml file in the drop-in directory elastic_blockchain_proxy.d, for example named elastic_blockchain_proxy.d/custom-settings.toml, and port in the new file the custom settings from the .rpmsave that you want to keep
Finally, remove the .rpmsave file elastic_blockchain_proxy.toml.rpmsave

Prerequisites¶

Before starting the upgrade, you should read very carefully the latest release notes on NetEye’s blog and check out the features that will be changed or deprecated after the upgrade.

All NetEye packages installed on a currently running version must be updated according to the update procedure prior to running the upgrade.
NetEye must be up and running in a healthy state.
Disk Space required:
- 3GB for / and /var
- 150MB for /boot
If the SIEM module is installed:
- The rubygems.org domain should be reachable by the NetEye Master only during the update/upgrade procedure. This domain is needed to update additional Logstash plugins and thus is required only if you manually installed any Logstash plugin that is not present by default.

Cluster - Configuration of Keycloak PCS resource¶

In case you are upgrading a NetEye Cluster, before proceeding with the upgrade, you must:

Configure the Keycloak PCS resource located at /usr/share/neteye/keycloak/cluster/Services-core-keycloak.conf with the correct values in accordance with your cluster configuration.
Confirm that the Keycloak PCS resource is correctly configured by enabling the switch Keycloak PCS file is configured located in Configuration > Modules > neteye > Configuration.

This is needed to ensure that the PCS resource will be correctly created during the upgrade process.

1. Run the Upgrade¶

The Cluster Upgrade is carried out by running the following command:

cluster# (nohup neteye upgrade &) && tail --retry -f nohup.out

Warning

If the SIEM feature module is installed and a new version of Elasticsearch is available, please note that the procedure will upgrade one node at the time and wait for the Elasticsearch cluster health status to turn green before proceeding with the next node. For more information, please consult the dedicated section.

After the command was executed, the output will inform if the upgrade was successful or not:

In case of successful upgrade you might need to restart the nodes to properly apply the upgrades. If the reboot is not needed, please skip the next step.
In case the command fails refer to the troubleshooting section.

2. Reboot Nodes¶

Restart each node, one at a time, to apply the upgrades correctly.

Run the reboot command
cluster-node-N# neteye node reboot
In case of a standard NetEye node, put it back online once the reboot is finished
cluster-node-N# pcs node unstandby --wait=300

You can now reboot the next node.

3. Cluster Reactivation¶

At this point you can proceed to restore the cluster to high availability operation.

Bring all cluster nodes back out of standby with this command on the last standard node
cluster# pcs node unstandby --all --wait=300 cluster# echo $?
0
If the exit code is different from 0, some nodes have not been reactivated, so please make sure that all nodes are active before proceeding.
Run the checks in the section Checking that the Cluster Status is Normal. If any of the above checks fail, please call our service and support team before proceeding.
Re-enable fencing on the last standard node, if it was enabled prior to the upgrade:
```
cluster# pcs property set stonith-enabled=true
```

4. Additional Tasks¶

New Authentication Method¶

After upgrading the new authentication method will be enabled along with the legacy one. This approach allows users to switch to the new authentication method at their own pace and test it before the final transition.

Login page with a greyed-out background indicates that you are using the legacy authentication method. Below the login form you will find a red button that will open the new login page.

After the upgrade, the management of users and groups should be handled through Keycloak rather than NetEye. Any users or groups created after the upgrade will need to be manually migrated to Keycloak.

At this point you need to log-in as a keycloak administrator and test the new login page as well as verifying that everything works as expected.

Note

By default the root user has access to the Keycloak admin console. The password for the root user is saved in the file /root/.pwd_icingaweb2_root unless it has been previously changed. All other users that were previously allowed to access the console continue to be authorized for this in case the migration was successful.

We suggest to

Get inside the Keycloak admin console by visiting the Configuration > Authentication in the NetEye menu
Once inside, verify that the users are correctly imported from the legacy authentication method by clicking the Users link
Verify that the groups are correctly imported from the legacy authentication method by clicking the Groups link
Verify the configuration of migrated LDAP/AD by clicking the User federation link, and then select the LDAP or Active Directory item in the list
- If an LDAP/AD group backend was configured before the upgrade, verify in the Mapper tab that a group-ldap-mapper type exists and that its configuration is correct
Verify roles defined in Keycloak and assigned to users and groups by visiting the Users/Groups > Role mapping page
Test the login with different users and groups

During the upgrade process, users and groups are automatically migrated; however, we strongly recommend verifying that everything functions as expected.

Due to the complexity of the operation, it is possible that, depending on the current configuration, some errors may occur, leading to certain users or groups not being migrated correctly.

In such cases, manual migration may be necessary.

Warning

If errors occur during the automatic migration of LDAP/AD settings from NetEye to Keycloak, you will find all errors details in the log file /root/upgrade_4.38/failed_ldap_configuration.log and you’ll have to correct the configuration after the upgrade as described in the Test your LDAP/AD configuration section.

Warning

If errors occur during the procedure of replicating admin permissions of any local/LDAP users in keycloak, you will find all errors details in log files located in /root/upgrade_4.38/ and you’ll have to migrate the permissions manually after the upgrade.

Note

We are only migrating admin permissions automatically during the upgrade, any other auth-related permissions will have to be migrated manually.

If something is not working as expected you can always use to the legacy login page for the time being, and adjust the configuration of the new authentication method by visiting the Configuration > Authentication page. Refer to the Authentication section for more information about the new authentication method.

Warning

Before continuing with the definitive switch to the new authentication method, make sure that the login and the users are working as expected.

Until the new authentication method is switched as default, a new service check in the monitoring will be in critical in order to remind you that you are still using the legacy authentication method and the migration is not completed yet. The service check is called auth-finalization-check-neteyelocal and is part of the host NetEye Local Self Monitoring.

../../_images/auth-finalization-check.png — Fig. 249 The service check that reminds you that the migration is not completed yet¶

Warning

Until the new authentication method is switched as default, all the new users and groups created in Configuration > Access Control > Users / User Groups will also have to be created in the new authentication method in Configuration > Authentication.

Once you are sure that everything is working as expected you can switch to the new authentication method by visiting the Configuration > Modules > neteye > Configuration page and enabling the switch Finalize Auth Migration. This step is required before upgrading to the next version.

Warning

All Icingaweb2 roles (with permissions) will still be handled by Icinga Web 2 and will not be migrated or delegated to the new authentication method.

Warning

The switch to the new authentication method is irreversible and once you have switched to the new authentication method you cannot go back to the legacy one.

More information about the new authentication method can be found in the Authentication section.