Concepts¶

Overview¶

Tornado is a high performance, scalable application, and is intended to handle millions of events each second on standard server hardware. Its overall architecture is depicted in Tornado architecture.

Tuning your infrastructure¶

As a part of the NetEye Core, Tornado module does not need additional installation. However, to start running passive monitoring of your infrastructure with Tornado software, some basic actions are to be taken.

Since passive monitoring process does not imply agent installation on the systems and devices to be monitored, the latter should be configured to send a particular type of data to NetEye. Tuning your own system to send events should be done taking into account its properties, architecture and setup.

Tornado Architecture¶

Tornado is structured as a library, and there are three main components of its architecture:

The Tornado Collector(s), or just Collector(s)
The Tornado Engine, or Engine
The Tornado Executor(s), or Executor(s)*

The term Tornado refers to the whole project or to a deployed system that includes all three components.

Architecturally, Tornado is organized as a processing pipeline, where input events move from Collectors to the Engine, to Executors, without branching or returning.

When the system receives an External Event, it first arrives at a Collector where it is converted into a Tornado Event. Then it is forwarded to the Tornado Engine where it is matched against user-defined, composable Rules. Finally, generated Actions are dispatched to the Executors.

Collectors¶

The purpose of a Collector is to receive and convert external events into the internal Tornado Event structure, and forward them to the Tornado Engine.

Collectors are Datasource-specific. For each datasource, there must be at least one Collector that knows how to manipulate the datasource’s Events and generate Tornado Events.

Out of the box, Tornado provides a number of Collectors for handling inputs from snmptrapd, rsyslog, JSON from Nats channels and generic Webhooks. All the collectors are pre-configured on NetEye, however, there are some that may eventually be configured manually in order to fit your purposes, e.g. Tornado Webhook or Icinga 2 Collectors.

Engine¶

The Engine is the second step of the pipeline. It receives and processes the events produced by the Collectors. The outcome of this step is fully defined by a Processing Tree composed of Filters, Iterators and Rulesets.

A Filter is a processing node that defines an access condition on the children nodes.

An Iterator can iterate over a field in the event, dispatching multiple events to be handled by it’s child nodes.

A Ruleset is a node that contains an ordered set of Rules, where each Rule determines:

The conditions a Tornado Event has to respect to match it
The actions to be executed in case of a match

The Processing Tree is parsed at startup from a configuration folder where the node definitions are stored in JSON format.

When an event matches one or more Rules, the Engine produces a set of Actions and forwards them to one or more Executors.

Executors¶

The Executors are the last element in the Tornado pipeline. They receive the Actions produced from the Engine and trigger the associated executable instructions.

An Action can be any command, process or operation. For example it can include:

Forwarding the events to a monitoring system
Logging events locally (e.g., as processed, discarded or matched) or remotely
Archiving events using software such as the Elastic Stack
Invoking a custom shell script

A single Executor usually takes care of a single Action type.

More information and example code snippets can be found in Section Tornado Actions.

Multitenancy¶

Tornado can be run on both single-tenant and multi-tenant environments. In case of a multi-tenant environment, each tenant has only got access to the part of a Processing Tree (starting from a filter) that contains filtered events received from this particular tenant. This way all the event streams are processed independently and securely.

Roles Configuration¶

If your NetEye installation is tenant aware, roles associated to each user must be configured to limit their access only to the Tornado configuration they are allowed to work with.

In the NetEye roles (Configuration / Access Control / Roles), add or edit the role related to the tenant limited users. In the detail of the role configuration you can find the tornado module section. You can set the tenant ID in the tornado/tenant_id restriction.

Hint

You can find the list of available Tenant IDs by reading the directory names in /etc/neteye-satellites.d/. You can use this command:

neteye# basename -a $(ls -d  /etc/neteye-satellite.d/*/)

Tenant-based Configuration¶

Tornado configuration is tenant-specifc. For single-tenant environments the configuration you will create will apply to the Master Tenant.

In case you would like to go with a multi-tenant environment, i.e. sending data to the NetEye from multiple tenants, you should create a tenant with neteye tenant config create if this has not been done yet. Consult The NetEye Command for more information on how to create a new tenant.

Please note that due to Implicit Lock mode only one user at a time from the pool of users belonging to all tenants within your NetEye installation can modify the configuration.

Tornado Collectors are preconfigured out of the box. However, for some of them additional steps are required in order to let a Collector receive events sent by your system or device, e.g. in case of sending SMS events to the Tornado SMS Collector the Modem and the smstools are to be additionally configured.

For more details on each particular type of Collectors please see Collecting Events.

Processing Tree¶

The Processing Tree presents all filters and rulesets within a Tenant. The order of rules within a ruleset defines the sequence of their execution.

In order to continuously improve UX and usability of the Tornado Instance, NetEye provides a GUI based on the Carbon Design System’s best practices.

../../_images/processing_tree.png — Fig. 140 Example Processing Tree.¶

Use drag & drop function focusing on a button to the left of the rule name to change the order.

For the start, select the tenant that you would like to work with in the toolbar. You can find more information on the relation between Tornado configuration and your tenants in Tenant-based Configuration.

More information on Processing Tree Configuration can be found in a dedicated section.

`Edit` Mode¶

Switch to Edit mode in order to modify Tornado configuration with the help of the Processing Tree.

When you start modifying the configuration, Tornado will continue to work with the existing configuration-thanks to the implicit lock mode, while the new changes are saved in a separate draft configuration. The new configuration then must be deployed to become operative.

Edit mode has other positive side effects: one does not need to complete the changes in one session, but can stop and then continue at a later point; another user can pick up the draft and complete it; in case of a disaster (like e.g., the abrupt end of the HTTPS connection to the GUI) it is possible to resume the draft from the point where it was left.

Enable ‘Edit’ Mode

Before you start modifying Tornado configuration with the help of a Processing Tree, make sure that editing permission is granted by the NetEye role.

In your NetEye installation, go to (Configuration / Access Control / Roles)
Select an existing role or add a new one for configuring the permission
Set the tornado/edit permission on the Tornado Module section of the role details to ‘On’

../../_images/edit-permission.png — Fig. 141 Tornado Module permissions¶

As a result, ‘Edit’ switcher would be available in the top right corner of the Tornado Configuration.

Implicit Lock Mode¶

Only one user at a time can modify the Processing Tree configuration. This prevents multiple users from changing the configuration simultaneously, which might lead to undesirable results and possibly to Tornado not working correctly due to incomplete or wrong configuration. When a user is editing the configuration, the actual, running configuration is left untouched: it continues to be operative and accepts incoming data to be processed.

Warning

Only one draft at a time is allowed; that is, editing of multiple draft is not supported!

Concepts