Cluster Upgrade from 4.41 to 4.42¶
This guide will lead you through the steps specific for upgrading a NetEye Cluster installation from version 4.41 to 4.42.
During the upgrade, individual nodes will be put into standby mode. Thus, overall performance will be degraded until the upgrade is completed and all nodes are revoked from standby mode. Granted the environment connectivity is seamless, the upgrade procedure may take up to 30 minutes per node.
Warning
Remember that you must upgrade sequentially without skipping versions, therefore an upgrade to 4.42 is possible only from 4.41; for example, if you have version 4.27, you must first upgrade to the 4.28, then 4.29, and so on.
Breaking Changes¶
NetEye SIEM feature module renaming¶
From NetEye 4.42 onwards, the SIEM feature module is now called Elastic Stack. This change better reflects the module’s functionalities, which encompass the entire Elastic Stack, including Observability and not only SIEM.
This change includes:
In the User Guide, the feature module will be referred to as Elastic Stack instead of SIEM
To install the feature module on systems without it, use the dnf group
neteye-elastic-stack
instead ofneteye-siem
, as explained :ref:here <neteye-modules>.If you have NetEye SIEM installed, upgrading to 4.42 will automatically install the dnf group
neteye-elastic-stack
.
To ensure a smooth transition, the old dnf group neteye-siem
will remain installed after the upgrade but will be removed
in future NetEye versions. Any custom integration or automation relying on the presence of the old
neteye-siem
dnf group will continue to function as before.
However, you should update it to refer to the new neteye-elastic-stack
dnf group to prepare for the
removal of the old group in future versions.
MariaDB Galera Cluster¶
During the upgrade to NetEye 4.42, MariaDB will be migrated to a MariaDB Galera Cluster. This improvement will enhance the reliability of the database system on NetEye clusters since the service will be distributed across multiple nodes instead of relying on a single instance. This will not only enhance the general reliability of the database system but also improve and minimize the downtime of the database and all the services that rely on it in case of a node failure or a service restart. For system consistency, the MariaDB Galera migration will be performed also on NetEye single-node installations, where the MariaDB instance will be migrated to a single-node Galera Cluster.
Deprecated Settings¶
With the transition to MariaDB Galera Cluster in NetEye 4.42, several default settings and functionalities have been updated or deprecated. Please review the following changes to ensure compatibility and optimal performance:
Directory Location Change¶
With the migration to MariaDB Galera Cluster, the MariaDB directory has been moved from
/neteye/shared/mysql
to /neteye/local/mariadb
. This is a breaking change that affects
any custom backup scripts targeting the old location, monitoring configurations checking the old path,
and custom integrations or automations accessing data/configuration in the old location.
All custom MariaDB configuration files placed in the my.cnf.d
folder will be automatically
migrated to the new directory during the upgrade process.
If you have any scripts, monitoring, or configurations that reference the old path, please update them to use the new location to ensure proper functionality after the upgrade.
User Privileges Management¶
In Galera Cluster, modifying user privileges requires the use of the GRANT
statement. Directly inserting into the
mysql.user table (e.g., INSERT INTO mysql.user ...
) is not supported and will not propagate changes across the cluster.
Always use GRANT
to ensure consistent and reliable privilege updates.
Unsupported Locking Mechanisms¶
The following locking operations are not supported in Galera Cluster and may lead to errors or undefined behavior:
LOCK TABLES
andUNLOCK TABLES
GET_LOCK
andRELEASE_LOCK
If you have custom applications interacting with the MariaDB database, please ensure they do not perform these operations and consider using alternative methods for synchronization and locking, such as transactions or application-level locks.
More information about the differences between MariaDB and Galera Cluster can be found in the Official Galera documentation
Large Pages Support¶
If you have previously configured or wish to configure large pages for MariaDB performance optimization
(using innodb_large_prefix
setting), please note that with the migration to Galera Cluster,
you will need to create a systemd drop-in file for the mariadb-galera.service
to grant the
necessary CAP_IPC_LOCK
capability to the service.
This configuration is not automatically migrated during the upgrade and requires manual intervention only if
you need large pages support.
Upgrade and Migration procedure¶
The NetEye upgrade procedure to 4.42 will automatically migrate the existing MariaDB instance, that is managed by PCS to a brand new architecture based on a Galera Cluster, Nginx and Keepalived to maximize the availability of the database service. This migration will be performed automatically during the upgrade process, no manual intervention is required and no additional disk space is needed. Here is a brief overview of the migration steps:
Preparation of Galera Cluster: A new Galera Cluster is set up, with the existing PCS-managed MariaDB instance serving as the first node in this cluster.
Node-by-Node Migration: Each node, one at a time, except the one hosting the current MariaDB instance, undergoes the following steps:
DRBD Volume Removal: The existing DRBD volume assigned to MariaDB is deleted.
New Volume Creation: A new storage volume is created for Galera with the same size as the old DRBD volume and mounted at
/neteye/local/mariadb
.Galera Node Initialization: The Galera instance is started on the node and joins the cluster.
Data Synchronization: The joining node synchronizes its data with the cluster. Depending on the database size, this process may take some time.
Finalization: Once all database service nodes have joined the Galera Cluster, the following final steps are performed:
Pacemaker Resource Removal: The Pacemaker resource managing the MariaDB instance is deleted.
DRBD Volume Deletion: The last remaining DRBD volume associated with MariaDB is removed.
Final Node Integration: The last node joins the Galera Cluster, with its data volume correctly placed at
/neteye/local/mariadb
.
The migration process is designed to minimize the downtimes and to ensure a smooth transition to the new Galera Cluster architecture.
Prerequisites¶
Before starting the upgrade, you should read very carefully the latest release notes on NetEye’s blog and check out the features that will be changed or deprecated after the upgrade.
All NetEye packages installed on a currently running version must be updated according to the update procedure prior to running the upgrade.
NetEye must be up and running in a healthy state.
Disk Space required:
3GB for
/
and/var
150MB for
/boot
If the NetEye Elastic Stack module is installed:
The rubygems.org domain should be reachable by the NetEye Master only during the update/upgrade procedure. This domain is needed to update additional Logstash plugins and thus is required only if you manually installed any Logstash plugin that is not present by default.
We’ve deprecated the default option in Proxy IP Mode under here
. You can check how to configure it
To prepare for the MariaDB Galera migration, ensure that the port 3307 is free and available on all the NetEye nodes interface. In order to grant a smooth upgrade, consider launching the command
neteye node check-upgrade-prerequisites
before the upgrade to check if the prerequisites are met.
1. Run the Upgrade¶
The Cluster Upgrade is carried out by running the following command:
cluster# (nohup neteye upgrade &) && tail --retry -f nohup.out
Warning
If the NetEye Elastic Stack feature module is installed and a new version of Elasticsearch is available, please note that the procedure will upgrade one node at the time and wait for the Elasticsearch cluster health status to turn green before proceeding with the next node. For more information, please consult the dedicated section.
After the command was executed, the output will inform if the upgrade was successful or not:
In case of successful upgrade you might need to restart the nodes to properly apply the upgrades. If the reboot is not needed, please skip the next step.
In case the command fails refer to the troubleshooting section.
2. Reboot Nodes¶
Restart each node, one at a time, to apply the upgrades correctly.
Run the reboot command
cluster-node-N# neteye node reboot
In case of a standard NetEye node, put it back online once the reboot is finished
cluster-node-N# pcs node unstandby --wait=300
You can now reboot the next node.
3. Cluster Reactivation¶
At this point you can proceed to restore the cluster to high availability operation.
Bring all cluster nodes back out of standby with this command on the last standard node
cluster# pcs node unstandby --all --wait=300 cluster# echo $?
0
If the exit code is different from 0, some nodes have not been reactivated, so please make sure that all nodes are active before proceeding.
Run the checks in the section Checking that the Cluster Status is Normal. If any of the above checks fail, please contact our service and support team before proceeding.
Re-enable fencing on the last standard node, if it was enabled prior to the upgrade:
cluster# pcs property set stonith-enabled=true
4. Additional Tasks¶
Enabling of the Content-Security-Policy (CSP) header¶
With the upgrade to IcingaWeb2 2.12.x we allow you to enable the Content-Security-Policy (CSP) header for the NetEye web interface. The CSP header helps protect your NetEye installation from cross-site scripting (XSS) attacks. The toggle for this can be found in the NetEye web interface under Configuration > Application > Enable strict content security policy.
Warning
The strict CSP header feature must be enabled before upgrading to the next release of NetEye (4.43).
IcingaWeb2 modules migration¶
Please make sure that all third-party icingaweb2 modules installed on the system are compatible with the CSP policies before enabling this feature to avoid any issues.
To do it you can follow the steps below:
Check and in case fix/upgrade a module.
Enable the strict CSP flag from the IcingaWeb2 settings.
Verify if everything is working.
In case of problems:
Disable the flag.
Fix the incompatibilities.
Repeat from step 1.
Repeat the process for the next module.
SLM report styling under the Content-Security-Policy (CSP) header¶
With the introduction of the CSP headers, we had to make minor breaking changes to the way styling is handled in the SLM reports. SLM reports can no longer use the inline style attributes. If any were used in a custom template, they now need to be migrated to the html style elements and applied with classes. Furthermore, if any html style elements are used in the custom template, they will now need to be updated to include the icingaweb2 style nonce to be compliant with the CSP header. Please refer to the appropriate section in the customizing templates section of the documentation for more information.