Agents configuration¶

Installation and Configuration of Beat Agents¶

Before being able to take fully advantage of the Beat feature, agents must be installed on the monitored hosts, along with the necessary certificates. On the hosts, any kind of Beat can be installed; for example, the Winlogbeat is available from the official download page; installation instructions are available as well. The agent configuration is stored in the YAML configuration file winlogbeat.yml. A description of the options available in the Beat’s configuration file can be found in the official documentation.

Note

You need to install a Beat whose version is compatible with the Elastic version installed on NetEye, which is 7.17. To find out which version of Beat you can install, please check the compatibility matrix

Relevant to the configuration are the following options:

ignore_older, which indicates how many hours/days it should gather data from. By default, indeed, the Beat collects all the data it finds, meaning it can act retroactively. This is the default option if not specified, so make sure to properly configure this option, to not overload the initial import of data and to avoid potential problems like crash of Logstash and ES disk space.
index: ”winlogbeat”, which is needed to match NetEye’s templates and ILM.

Filebeat Netflow module specific configuration¶

In NetEye, Filebeat is shipped with the NetFlow Module included. The module should be configured without directly modifying the configuration file. Any change in the configuration file will be overwritten during future updates and result in the reactivation of the module or losing particular customizations.

The correct configuration can be accomplished by setting the following three environment variables in the /neteye/shared/filebeat/conf/sysconfig/filebeat-user-customization environment file:

NETFLOW_ENABLED: which can be used to enable or disable the listening of logs on the specified host and port. Default: true
NETFLOW_HOST: The host to which the module will listen on. Default: 0.0.0.0 (all hosts)
NETFLOW_PORT: The port to listen on. Default: 2055

After changing the value of any of the aforementioned variables, the neteye install command should be executed to apply the changes:

neteye# neteye install

Warning

If the module is deactivated using the filebeat modules disable command and not using the associated environment variable, Filebeat will rename the configuration file and, at the next update, the module will be re-installed and re-activated.

Use of SSL certificates¶

Server certificates of Logstash allowing communication with Beats must be stored in the /neteye/shared/logstash/conf/certs/ directory, with names logstash-server.crt.pem and private/logstash-server.key. Additionally, also the root-ca.crt certificate must be available in the same directory.

The structure mentioned above for the certificates must be organised as:

certs/
   ├── logstash-server.crt.pem
   ├── root-ca.crt
   └── private/
          └── logstash-server.key

The certificates are stored under the logstash configuration directory, because it is indeed Logstash that listens for incoming Beat data flows.

As a consequence, all Beat clients must use a client certificate to send output data to Logstash. Please refer to the Elastic official documentation, for example the Filebeat SSL configuration is available here.

An example of Filebeat to Logstash SSL communication configuration is the following:

#--------- Logstash output ------------------------------------
    output.logstash:
      # The Logstash hosts
      hosts: ["yourNetEyeDomain.example:5044"]

      # List of root certificates for HTTPS server verifications
      ssl.certificate_authorities: ["/root/beat/root-ca.crt"]

      # Certificate for SSL client authentication
      ssl.certificate: "/root/beat/logstash-client.crt.pem"

      # Client Certificate Key
      ssl.key: "/root/beat/private/logstash-client.key.pem"

Self-signed certificates¶

Note

For production systems, you should upload your own certificates on NetEye. Moreover, you should use your own certificates also for all Beat clients. Self-signed certificates must never be used on production systems, but only for testing and demo purposes.

Self-signed certificates (logstash-server.crt.pem and private/logstash-server.key) and the Root CA (root-ca.crt) are shipped with NetEye for Logstash. Self-signed certificates for Beat clients can be generated from the CLI as follows:

you can run the script usr/share/neteye/scripts/security/generate_client_certs.sh using three suitable parameters:

The client name
The common name (CN) and information for the other certificate’s field
The output directory

An example of command line is the following:

/bin/bash /usr/share/neteye/scripts/security/generate_client_certs.sh \
    logstash-client \
    "/CN=logstash-client/OU=client/O=client/L=Bolzano/ST=Bolzano/C=IT" \
    "/root/beat/"

Inputs configuration¶

To set customer-specific filebeat inputs you can add a file with .yml extension in the directory /neteye/shared/filebeat/conf/inputs.d/. Configuration will be read and applied from .yml files only: any file with different extension will be ignored. To maintain a custom configuration saved but disabled, you should rename the file with a different extension, for example mqtt.yml can be disabled by renaming it to mqtt.yml.disable.

A sample configuration can be found in file /neteye/shared/filebeat/conf/inputs.d/mqtt.yml.sample.

Elastic Agent¶

Elastic Agent comes out-of-the-box in NetEye installations: every NetEye node hosts a running Elastic Agent with a preconfigured policy that depends on the role of the node in the NetEye system. Since the Elastic Agent implementation is officially supported, the Elastic Agents running on NetEye nodes should be assigned to the default NetEye policies and not to other custom policies.

In order to customize the policy parameters and the NetEye-managed integrations with the -neteye suffix:

Customize the configuration files in /neteye/local/elastic-agent/conf/fleet/templates/ specifying the parameters you want to change.
Apply the new configuration on all the NetEye nodes by running the following command on the node where the configuration file has been modified:
```
neteye tenant config apply master
```
This will synchronize the changes in the configuration file and apply the customization to the enrolled Elastic Agents in the system.

Fleet Server¶

By default, Fleet Server is exposed, through Nginx, using the certificate and key located at /neteye/local/elastic-agent/conf/fleet/certs/fleet-server-external.crt and /neteye/local/elastic-agent/conf/fleet/certs/private/fleet-server-external.key.

The certificate and key, which are the same as the ones used by Httpd, are automatically generated upon NetEye first installation procedure. However, it is recommended, as described in this procedure, to install a new trusted certificate as soon as possible, on all Operative Nodes.

Moreover, the cryptographic settings applied to the SSL connections replicate the settings applied at a system level, which may be changed as described in the official RHEL documentation.

Elastic Agent on Satellites¶

The installation of Elastic Agents on NetEye Satellites is not yet officially managed by the NetEye system. Elastic Agent can still be installed and configured manually in order to obtain a Tenant-related Agent.

The following guidelines will explain how to correctly configure an Elastic Agent on a NetEye Satellite:

Before adding the Elastic Agent to the NetEye system, a dedicated Policy is needed for the Satellite: Through the “Fleet” section in Kibana, an additional Policy called <tenant_display_name> Satellite can be created, using <tenant_id> as namespace for the Policy and the associated integrations.
After the installation of Elastic Agent from the NetEye repositories has been completed, the Agent can be enrolled to Elastic using the Fleet Server of the NetEye Master, reachable at <neteye_master_fqdn>:8220.
The Elastic Agent can use the already configured Output neteye-external-output, connected to the Elasticsearch server on the NetEye Master.
Once the Elastic Agent has been enrolled in the NetEye Fleet, additional integrations can be added and applied to the Tenant-related Elastic Agent running on the NetEye Satellite. Every integration belonging to the previously created Policy, associated with a specific Tenant, should have the -<tenant_id> suffix in order to maintain a good separation among different Tenants.

For example, a good choice for the APM integration of a Tenant having tenant_id = 12345 could be apm-12345

Signing Elastic Agent Logs¶

In order to sign documents collected by any Elastic Agent in the NetEye environment using El Proxy, it is possible to configure Elastic Agent to send logs to Logstash for the signing process. Once El Proxy has been correctly configured and enabled as described in Enabling El Proxy, the subsequent sections can be followed in order to apply the correct configuration to the specific scenario:

Note

If your Elastic Agents need to connect to the NetEye Logstash using an hostname different than the NetEye Hostname (declared in /etc/neteye-cluster for Clusters and hostname --fqdn for Single Nodes) in the case they run outside the company network, for example, you can substitute the certificate /neteye/shared/logstash/conf/certs/logstash-server-external.crt.pem and private key /neteye/shared/logstash/conf/certs/private/logstash-server-external.key with trusted certificates valid for the needed hostname.

Elastic Agent on the Master Tenant¶

If Elastic Agent belongs to the Master Tenant, it is enough to change the output of the Elastic Agent and send the logs to be signed to Logstash instead of sending them directly to Elasticsearch.

This operation can be executed under Fleet / Agent Policies / <selected_policy> / Settings by changing the Output for integrations field to the preconfigured neteye-external-logstash-output output.

Note that <selected_policy> should be the policy assigned to the Elastic Agent that wll then send the logs to be signed by El Proxy. Every other Elastic Agent connected to the same policy will change its output, starting to send logs to Logstash.

Elastic Agent on other Tenants¶

In the case Elastic Agent belongs to a Tenant different to the Master Tenant, the Elastic Agent should send the collected logs to the Logstash instance running on the NetEye Satellite of its Tenant.

In order to change the output of the Elastic Agent that should send logs to be signed, an additional output should be created in Fleet / Settings / Outputs defining as Logstash the output type of the configuration and filling the other parameters of the form.

Since the documents should be sent to the Logstash instance of the NetEye Satellite, the Satellite FQDN and the port 5045 should be specified as Logstash Host, for example <satellite_fqdn>:5045.

Note that the NetEye instance of Logstash already has a preconfigured pipeline dedicated to Elastic Agent input sources, no additional configurations are required in Logstash.

Once the new output options has been stored in Fleet, it is possible to assign the newly created output to the policy of the Elastic Agent that is collecting logs to be signed.