Cluster Upgrade from 4.16 to 4.17¶

This guide will lead you through the steps specific for upgrading from a NetEye Cluster installation from version 4.16 to 4.17.

Warning

Remember that you must upgrade sequentially without skipping versions, therefore an update to 4.17 is possible only from 4.16; if you have a 4.14 you must first upgrade to the 4.15, then 4.16. and so on.

Before starting an upgrade, you should very carefully read the the release notes on NetEye’s blog and check the feature changes and deprecations specific to the version being upgraded. You can check also the whole section Post Upgrade Steps below to verify if there are changes or specific steps that might significantly impact your NetEye Cluster installation.

Cluster Upgrade Prerequisites¶

Upgrading a cluster will take a nontrivial amount of time. During the cluster upgrade, individual nodes will be put into standby mode and so overall cluster performance will be degraded until the upgrade procedure is completed and all nodes are removed from standby mode.

An estimate for the time needed for a full upgrade (update + upgrade) when the cluster is healthy and there are no problems is approximately 30 minutes, plus 15 minutes per node. So for instance on an 3-node cluster it may take approximately 1 hour and 15 minutes (30 + 15*3). This estimate is a lower bound that does not include additional time should there be a kernel update or if you have additional modules installed.

This user guide uses the following conventions to highlight in which node you should execute the process:

(ALL) is the set of all cluster nodes
(N) indicates the last node
(OTHER) is the set of all nodes excluding (N)

For example if (ALL) is neteye01.wp, neteye02.wp, and neteye03.wp then:

(N) is neteye03.wp
(OTHER) is neteye01.wp and neteye02.wp

The order in which (OTHER) nodes are upgraded is not important. However, you should note that the last node (N) to be upgraded will require a slightly different process than the other nodes (see Post-upgrade steps for the last node (N) for details).

Cluster Upgrade Preparation¶

The Cluster Upgrade Preparation is carried out by running the command:

# nohup neteye upgrade

Warning

The neteye upgrade command can be run on a standard neteye node, but never on an Elastic-only or a Voting-only Node.

The neteye upgrade command will run a number of checks to make sure that:

The version on NetEye installed is eligible for upgrade, that is, it checks which is the installed version (i.e., 4.xx) and that the last upgrade was finalized, i.e., the neteye_finalize_installation script was carried out successfully.
NetEye is fully updated and there are no minor (bugfix) updates to be installed.

Warning

The neteye upgrade command may take a long time before it completes successfully, so please do not interrupt it until it exits.

If any of these tasks is unsuccessful, a message will explain where the command failed, allowing you to manually fix the corresponding step. For example, if the exit message is similar to the following one, you need to manually install the latest updates.

"Found updates not installed"
"Example: icingacli, version 2.8.2_neteye1.82.1"

Then, if needed, the command will:

Update all the NetEye repositories to the newer version (i.e., 4.yy, which is the next version to which it is possible to upgrade)
Install all the RPMs of the newer version (i.e., 4.yy)
Upgrade the NetEye’s yum groups

Warning

The neteye upgrade command will not restore stonith, therefore remember to execute it when necessary, that is, in step Cluster Reactivation (N) below.

If the command is successful, a message will inform you that it is possible to continue the cluster upgrade procedure.

Standby of Cluster Nodes (OTHER)¶

Put (OTHER) nodes in standby except the last one (N), by running the following command so that the current node is no longer able to host cluster resources:

# pcs node standby --wait=300
# echo $?

If the exit code is different from 0, the current node is not yet in standby, so please be sure that the current node is in standby before proceeding.

Status on all Cluster Nodes (ALL)¶

Repeat these upgrade steps for all nodes (ALL).

Run the following cluster command:
```
# pcs status
```
and please ensure that:
- Only the last node (N) MUST be active
- All cluster resources are marked “Started” on the last node (N)
- All cluster services under “Daemon Status” are marked active/enabled on the last node (N)
Check if the DRBD status is ok using the following command:
```
# drbdmon
```
You absolutely MUST remember to migrate ALL *.rpmnew and *.rpmsave files (see the RPM configuration migration page)

Post Upgrade Steps¶

This section describes all steps, necessary after the upgrade, that must be done on specific nodes.

Post-upgrade Scripts On (OTHER) nodes¶

Run the NetEye Secure Install on (OTHER) nodes but wait for the successful execution of the NetEye Secure Install before running it on another node:

# nohup neteye_secure_install

Warning

In case during the upgrade procedure a new cluster resource has been installed–as it is the case of Elastic Blockchain Proxy in this release–an error can be thrown. This error can be disregarded because it will be automatically fixed during the Post-upgrade steps for the last node (N) step.

Post-upgrade scripts on the Elastic-only nodes¶

Run the NetEye Secure Install on the Elastic-only nodes:
```
# nohup neteye_secure_install
```

Warning

In case during the upgrade procedure a new cluster resource has been installed–as it is the case of Elastic Blockchain Proxy in this release–an error can be thrown. This error can be disregarded because it will be automatically fixed during the Post-upgrade steps for the last node (N) step.

Post-upgrade steps for the last node (N)¶

If the SIEM module is installed, create the new Elastic Blockchain Proxy (EBP) cluster resource:
- Adapt the template /usr/share/neteye/cluster/templates/Services-ebp.conf.tpl to the settings of your cluster
- Save it to a file with the same name without the .tpl suffix
- Execute the following command:
```
# /usr/share/neteye/scripts/cluster/cluster_service_setup.pl -c /usr/share/neteye/cluster/templates/Services-ebp.conf
```
Run the NetEye Secure Install on the last node (N):
```
# nohup neteye_secure_install
```

Cluster Reactivation (N)¶

You can now restore the cluster to high availability operation.

Bring all cluster nodes back out of standby with this command on the last node (N):
```
# pcs node unstandby --all --wait=300
# echo $?
```
```
0
```
If the exit code is different from 0, some nodes have not been not reactivated, so please be sure that all nodes are active before proceeding.
Run the checks in the section Checking that the Cluster Status is Normal. If any of the above checks fail, please call our service and support team before proceeding.

Re-enable fencing on the last node (N):

# pcs property set stonith-enabled=true

Finalize the Cluster Upgrade (ALL)¶

You can now finalize the upgrade process by launching the following script on every node (ALL) one by one:

# neteye_finalize_installation

In this upgrade, no additional manual step is required.

Troubleshooting: Failing health checks, migration of modules¶

After the finalization procedure has successfully ended, you might notice in the Problems View that some health check fails and is in state WARNING. The reason is that you are using some module that needs to be migrated, because some breaking change has been introduced in the release.

Hence, you should go to the Problems View and check which health check is failing. There you will also find instructions for the correct migration of the module, which is in almost all cases amounts to enabling an option: the actual migration will then be executed manually.