Retention Policy for Log Manager

Warning

The Log Manager Retention Policy is now deprecated and is only applied to logs managed via rsyslog. The retention policy of logs managed via Real Time Log Signing must be configured with the Elasticsearch Index Lifecycle Management.

The NetEye Log Manager data retention policy defines the amount of time that information should be retained for operational or regulatory compliance needs. This policy can be defined separately for each host in NetEye. The retention policy applies to each host that has logs collected from NetEye Log Management. These logs can be stored both in the filesystem (by rsyslog) and in Elasticsearch.

The default retention policy for all hosts is 365 days (one year) and is automatically applied to all those hosts for which the user does not explicitly specify a retention policy. This value can be changed by modifying the “Default retention policy duration” field in the configuration form of the Log Manager module at Configuration > Modules > logmanager > Configuration that represents the time span (number of days) for which the logs of a host should be kept. The retention policy can also be set for either individual hosts or host templates via the “Retention policy days” field that can be found in the host creation/edit form (remember that the retention policy for a host has precedence over its host template).

To force retention on all hosts configured in Log Manager, you can use the following command:

# icingacli logmanager retentionpolicy apply

Note

There are two different systemd timers installed by default that periodically execute these tasks:

• icingaweb2-module-logmanager-retention.timer - every day after midnight - that enforces automatically the retention policies (see Log Manager’s technical overview) and verifies if the retention policies are correctly applied.

The main purpose of this command is to delete all logs generated before the time frame set by the retention policy. For example, if you set a retention policy of 10 days (i.e., setting the retention value to 10 whether on the host, host template or Log Manager default) and then launch the command on the 20th of May, it will delete all logs created before the 10th of May.

More precisely, the command first retrieves a list of all hosts in the monitoring module, and then for each one checks if a retention policy was set on the host or host template, using the default retention policy if none was otherwise set. The retention policies are then applied to logs from the single hosts, deleting them accordingly.

Retention Policy Verification

As part of the initial installation of NetEye, a default host (HOST_NAME=”neteye-local”) is created. The Logmanager module now provides the retention-policy-verify command, which is automatically attached as a service to the neteye-local host. In addition, the module also provides the JSON file that represents the action executed by the previously mentioned command. The action consists of calling this icingacli command:

# icingacli logmanager retentionpolicy verify

This check ensures that all log files saved in rsyslog’s data directory and all Elasticsearch logs older than the date computed with the specified retention policy have been correctly deleted. If all hosts reply that their logs were correctly deleted, the script will return ‘0’. Otherwise, it will return a ‘1’ and print the list of hosts that returned an error.

The output of the command can be seen in the Plugin Output section (under Overview > Services > Neteye Local Self Monitoring) of the default service to which the custom command relates to. This command checks whether the Log Manager retention policy was correctly applied to the logs generated by the hosts present in NetEye.

If the verification leads to positive results, the Plugin Output will be the message:

OK: Retention policy correctly applied.

On the other hand, if something went wrong during the deletion of the logs, the output will be the following:

CRITICAL - These are the hosts that returned an error when trying to delete their <log_type> logs: <host_name>

where log_type can be either rsyslog or elasticsearch depending on which kind of log the host has generated, and host_name is the name of the host that generated that log.