Single Node Upgrade from 4.30 to 4.31¶

This guide will lead you through the steps specific for upgrading a NetEye Single Node installation from version 4.30 to 4.31.

Upgrading a NetEye Single Node takes a nontrivial amount of time. Granted the environment connectivity is seamless, the upgrade procedure may take up to 30 minutes.

Warning

Remember that you must upgrade sequentially without skipping versions, therefore an upgrade to 4.31 is possible only from 4.30; for example, if you have version 4.27, you must first upgrade to the 4.28, then 4.29, and so on.

Breaking Changes¶

Upgrade of Elastic Stack to version 8.8.2¶

In NetEye 4.31, we upgraded Elastic Stack from version 7.17.9 to version 8.8.2, which comes with several breaking changes.

In order to facilitate the upgrade, presented below is a list of breaking changes that might have impact on your NetEye installation if you have the SIEM module installed. Before proceeding with the upgrade, please read this section carefully as well as the Elastic official breaking changes documented in the Official Elastic Documentation

Logstash Deprecations and Warnings¶

With Logstash 8.8.2, all deprecations and warnings reported in Logstash 7.17 will be treated as errors, possibly leading to malfunctions. For this reason, we strongly recommend to review all deprecations and warnings reported by Logstash 7.17 to ensure they have been addressed in a proper way.

ECS Compatibility in Logstash¶

From NetEye 4.31, ECS Compatibility will be globally enabled in Logstash. For more information on this topic, please visit the Elastic Common Schema (ECS) Reference.

In particular, this change might concern your NetEye installation for the following reasons:
1. The Logstash index template is now ECS compatible. This means that any external agent sending data to the Logstash main pipeline must now send logs that are ECS compatible.
2. The Logstash default index has changed. The logs processed by the Logstash main pipeline will now be stored in the logs-logstash-master data stream, which is the new recommended way to store time series data. The default index was previously logstash-*.
  
  This means for example that you should migrate your Kibana visualizations to point to the new index, otherwise the visualizations will only display old Logstash data.
  
  To include both old data and new data in your Kibana Analytics objects (i.e. the objects shown in the screenshot below), you can create a new data view that includes old data as well as new data and use this index pattern in your visualizations. For example, the index pattern can be set to: logstash-*,logs-logstash-master
3. GeoIP Filter in ECS Compatibility mode requires a target when source is not an ip sub-field. For more information, please refer to the Geoip filter plugin documentation.
  
  If your installation contains custom Logstash filters in which the GeoIP Filter is used, please make sure that the filter either sets source to an ip sub-field or specifies a target.

Elasticsearch Curator¶

The Elasticsearch Curator, which could be used as a snapshot / backup utility, is no longer supported by NetEye.

In particular, the NetEye scripts that relied on the Elasticsearch Curator have been removed:
1. /usr/share/neteye/backup/elasticsearch/elasticsearch-backup.sh
2. /usr/share/neteye/backup/elasticsearch/elasticsearch-indices.sh
3. /usr/share/neteye/backup/elasticsearch/elasticsearch-restore.sh
If you were using the Elasticsearch Curator for your Elasticsearch data backups, please switch to the official backup functionality of Elastic. For more information please refer to Backup & Restore.

If you are not using the Elasticsearch Curator in any way, after the upgrade to NetEye 4.31 you can remove it from your Nodes with the command:
```
# dnf remove elasticsearch-curator
```
Warning

Please do not execute this command before upgrading to NetEye 4.31, otherwise this will remove other necessary RPMs

Beats data stream¶

Beats are now handled as data streams in Elasticsearch. This means that any Beats (or Logstash output) writing directly to Elasticsearch should now not include the date in the output index. They should instead just specify the name of the data stream.

For example if a Filebeat instance is writing in the index filebeat-%{[agent][version]}-%{[event][module]}-%{+YYYY}.%{+MM}.%{+dd}, after the upgrade it should write instead in filebeat-%{[agent][version]}-%{[event][module]}.

Elastic APM activation¶

Since NetEye 4.31, the Elastic APM functionality will not be enabled by default and needs a small manual procedure to enable it. If you use Elastic APM, please perform the following procedure, to add the APM integration, before upgrading to NetEye 4.31. Performing the procedure before upgrading allows you to avoid losing the APM data generated during the upgrade.

Warning

If you do not perform the manual procedure, also the APM data coming from Tornado will not be saved anymore.

Access Kibana home page by navigating to Log Analytics
Select Add integrations and then Elastic APM
Click on the APM Integration button
Click on Add Elastic APM button
Set the name of the integration to apm-neteye
Leave all other settings to their defaults and scroll down to section Apply to agent policy
Click on Create agent policy, set the policy name to apm-setup and remove the flag Collect system logs and metrics
Click on Create agent policy, then on Save and continue and afterwards on Add Elastic Agent later
If you are on a Single Node environment you should also adapt the Index Templates generated by the APM integration to allow APM indexes to have no replicas (you can skip this step in cluster environments). To do this you should:
1. Navigate to Log Analytics / Stack Management / Index Management / Component Templates
2. Search for the customizable Component Templates of APM with apm "@custom"
3. Edit each of the resulting Component Templates by inserting in Index Settings the following content:
```
{
  "index": {
    "auto_expand_replicas": "0-1"
  }
}
```

Elastic Security¶

During the upgrade, detection rules will be disabled and you need to re-enable them after the upgrade. For more information about the procedure and other notable changes related to the Elastic Security, including privileges, please consult the official documentation.

Prerequisites¶

Before starting the upgrade, you should read very carefully the latest release notes on NetEye’s blog and check out the features that will be changed or deprecated after the upgrade.

All NetEye packages installed on a currently running version must be updated according to the update procedure prior to running the upgrade.
NetEye must be up and running in a healthy state.
Disk Space required:
- 3GB for / and /var
- 150MB for /boot
If your NetEye has the SIEM feature module installed, the following prerequisites need to be satisfied to guarantee a smooth upgrade to the latest Elastic 8 version:
1. Elasticsearch version 8.8 is only compatible with Elastic Agent and Beats whose version is in the range 7.17.x - 8.8.x. For this reason, before upgrading please ensure that all your Elastic Agents and Beats are updated to version 7.17
2. The Elasticsearch disk watermarks for Single Node must be enabled. For more information, please refer to the official Elasticsearch documentation.
  
  This setting will let Elasticsearch monitor the available disk space also on Single Node installations. Before enabling it, you should ensure that the available disk space is enough by considering the low watermark setting.
  
  If you are unsure about the available disk space and the low watermark setting, the upgrade prerequisites checks can help you by printing out the relevant information in the following format:
  [!] Elasticsearch disk watermark for single node is not enabled. Before proceeding with the upgrade you need to enable the disk watermark for single node, which will avoids possible problems during the upgrade to Elastic 8 [i] Before enabling the disk watermark for single node, please take into consideration the following information about the current disk usage and the currently set watermark levels [i] Defaults low disk watermark is: 85% [i] No persistent low disk watermark set [i] No transient low disk watermark set [i] Disk size: 476G (510389125120 bytes) [i] Disk space used: 287G 63% (307746308096 bytes) [i] Please visit https://ne430-build.example.com/neteye/kibana/kibana/app/management/stack/upgrade_assistant/es_deprecations and fix the warning related to the issue 'Disk watermarks do not treat single-node clusters differently in versions 8.0 and later.' [i] For more information on this warning and its resolution, please visit https://neteye.guide/4.31/update-upgrade/upgrade/upgrade.html#prerequisites
  To see this information, you can either run the upgrade as explained in section Run the Upgrade or execute the relevant upgrade prerequisite check manually with the following command:
  bash /usr/share/neteye/scripts/upgrade/upgrade-prerequisite-checks/010_disk_watermark_enabled_for_single_node_check.sh
  From the example above it is clear that the Elasticsearch low disk watermark level is 85% and that we currently use 62% of the total disk space. Since we are using less than 85% of our disk space, enabling the disk watermarks for Single Node would be fine.
  
  Note that the format in which the low disk watermarks are reported (percentage in this case) could differ. Also note that the persistent and transient low disk watermarks are not set in this case. If your installation defines more than one low disk watermark, refer to the strictest one, which requires more free space on the disk.
  
  Once you are sure that the available disk space is enough considering the strictest low disk watermark level, you can proceed by enabling the disk watermarks for Single Node.
  
  The Cluster section of the Elasticsearch configuration file located under /neteye/local/elasticsearch/conf/elasticsearch.yml must be modified by adding the following line:
  cluster.routing.allocation.disk.watermark.enable_for_single_data_node: true
  After this modification, restart the Elasticsearch service for the changes to take effect with the following command:
  systemctl restart elasticsearch.service
3. The critical issues reported by the Kibana Upgrade Assistant must be resolved.
  
  You can inspect the issues reported by the Kibana Upgrade Assistant by visiting https://<hostname>/neteye/kibana/kibana/app/management/stack/upgrade_assistant/overview.
  
  Make sure to change <hostname> with your actual hostname.
  
  Warning
  
  The second step reported by the Kibana Upgrade Assistant, namely Migrate system indices, will be performed automatically by the NetEye upgrade procedure, to ensure it behaves correctly with the installed index templates. For this reason, it must not be performed manually.
4. The warnings reported by the Kibana Upgrade Assistant must be analyzed and resolved if possible.
  
  You can inspect the warnings reported by the Kibana Upgrade Assistant by visiting https://<hostname>/neteye/kibana/kibana/app/management/stack/upgrade_assistant/overview.
  
  Make sure <hostname> reflects your actual hostname.
  
  Note that Elastic does not prevent you from upgrading if there are unresolved warnings, but for the upgrade to be as smooth as possible, we introduced a feature flag that will prevent the NetEye upgrade if not enabled.
  
  Before enabling the feature flag, you should carefully analyze the warnings reported by the Kibana Upgrade Assistant, fix the warnings whenever possible, and understand the implications that the unresolved warnings will have on your installation after the upgrade.
  
  Once you inspect all the warnings, the feature flag that prevents the upgrade can be enabled via Configuration / Modules / kibana / Configuration.
  Note
  
  Due to the Kibana Upgrade Assistant implementation, some of the reported warnings will not disappear even after fixing them and some other warnings can be ignored. Here is an incomplete list of the warnings that can be ignored:
  
  File and/or native realms are enabled by default in next major release.
  
  Setting [xpack.monitoring.collection.enabled] is deprecated
  
  Setting "server.rewriteBasePath" should be set when using "server.basePath"
  
  Kibana's logging format is changing in 8.0
  
  The "xpack.reporting.roles" setting is deprecated
  
  "xpack.security.session.idleTimeout" is now 8 hours
  
  "xpack.security.session.lifespan" is now 30 days

1. Run the Upgrade¶

To perform the upgrade, run from the command line the following command:

neteye# (nohup neteye upgrade &) && tail --retry -f nohup.out

After the command was executed, the output will inform if the upgrade was successful or not:

In case of successful upgrade you might need to restart NetEye to properly apply the upgrades. If the reboot is not needed, please skip the next step.
In case the command fails refer to the troubleshooting section.

2. Reboot¶

Restart NetEye to apply the upgrades correctly.

neteye# neteye node reboot

3. Additional Tasks¶

In this release, no additional task is required.