Concepts

Passive Monitoring

NetEye’s passive monitoring abilities count on Tornado, a Complex Event Processor that receives reports of events from data sources such as monitoring, email, and telegram, matches them against pre-configured rules, and executes the actions associated with those rules, which can include sending notifications, logging to files, and annotating events in a time series graphing system.

Tornado is a high performance, scalable application, and is intended to handle millions of events each second on standard server hardware. Its overall architecture is depicted in Fig. 146.

Fig. 146 Tornado architecture

Tornado Architecture

Tornado is structured as a library, with three example binaries included that show how it can be used. The three main components of the Tornado architecture are:

• The Tornado Collector(s), or just Collector(s)

• The Tornado Engine, or Engine

• The Tornado Executor(s), or Executor(s)*

The term Tornado refers to the whole project or to a deployed system that includes all three components.

Along with the main components, the following concepts are fundamental to the Tornado architecture:

• An Action: An operation performed by Tornado, usually on an external system. For example, writing to Elastic Search or setting a state in a monitoring system.

• A Tornado (or Internal) Event: The Tornado-specific Event format.

• A Datasource: A system that sends External Events to Tornado, or a system to which Tornado subscribes to receive External Events.

• An External Event: An input received from a datasource, whose format depends on its source. An example of input are events collected from rsyslog.

• A Rule: A group of conditions that an Internal Event must match to trigger a set of Actions.

See also

Events and Actions are described in more detail in the Tornado common API documentation.

Architecturally, Tornado is organized as a processing pipeline, where input events move from collectors to the engine, to executors, without branching or returning.

When the system receives an External Event, it first arrives at a Collector where it is converted into a Tornado Event. Then it is forwarded to the Tornado Engine where it is matched against user-defined, composable Rules. Finally, generated Actions are dispatched to the Executors.

The Tornado pipeline:

Datasources (e.g. rsyslog)
  |
  | External Events
  |
  \-> Tornado Collectors
        |
        | Tornado (or Internal) Events
        |
        \-> Tornado Engine (matches based on Rules)
              |
              | Actions
              |
              \-> Tornado Executors (execute the Actions)

Tornado Interaction with Icinga 2

The interaction between Tornado and Icinga 2 is explained in details in sections Icinga 2 Executor and Smart Monitoring Check Result Executor. In particular the Smart Monitoring Executor interacts with Icinga 2 to create objects and set their statuses. To ensure that the status of the Icinga 2 objects does not get lost, NetEye provides an automatism that stops the execution of Smart Monitoring Actions during any Icinga 2 restart or Icinga Director deployment.

The automatism keeps track of all Icinga 2 restarts (we consider also Icinga Director deployments as Icinga 2 restarts) in the icinga2_restarts table of the director database. As soon as an Icinga 2 restart takes place, a new entry with PENDING status is added in that table and at the same time the Tornado Smart Monitoring Executor is deactivated via API.

The icinga-director.service unit monitors the status of the Icinga 2 restarts that are in PENDING status and sets them to FINISHED as soon as the service recognizes that Icinga 2 completed the restart, then the Tornado Smart Monitoring Executor is activated.

Icinga 2 (including Icinga Director deployments) can not restart if for any reason there is another restart PENDING, i.e., ongoing or stuck, which in some corner case may result in Icinga 2 restarts being blocked. Should you encounter such unfortunate situation, you can manually unblock new restarts by issuing the following command, which will set the status of all PENDING restarts to OBSOLETE (i.e., restarts that are outdated):

neteye# icingacli director icinga2restart resetlock

Collectors

The purpose of a Collector is to receive and convert external events into the internal Tornado Event structure, and forward them to the Tornado Engine.

Collectors are Datasource-specific. For each datasource, there must be at least one collector that knows how to manipulate the datasource’s Events and generate Tornado Events.

Out of the box, Tornado provides a number of Collectors for handling inputs from snmptrapd, rsyslog, JSON from Nats channels and generic Webhooks.

Because all Collectors are defined with a simple format, Collectors for new event types can easily be added or extended from existing types for:

• Monitoring events

• Email messages

• Telegram

• DNS

• Cloud monitoring (AWS, Azure, Cisco/Meraki, etc.)

• Netflow

• Elastic Stack

• SMS

• Operating system and authorization events

Engine

The Engine is the second step of the pipeline. It receives and processes the events produced by the Collectors. The outcome of this step is fully defined by a processing tree composed of Filters and Rule Sets.

A Filter is a processing node that defines an access condition on the children nodes.

A Rule Set is a node that contains an ordered set of Rules, where each Rule determines:

• The conditions a Tornado Event has to respect to match it

• The actions to be executed in case of a match

The processing tree is parsed at startup from a configuration folder where the node definitions are stored in JSON format.

When an event matches one or more Rules, the Engine produces a set of Actions and forwards them to one or more Executors.

Executors

The Executors are the last element in the Tornado pipeline. They receive the Actions produced from the Engine and trigger the associated executable instructions.

An Action can be any command, process or operation. For example it can include:

• Forwarding the events to a monitoring system

• Logging events locally (e.g., as processed, discarded or matched) or remotely

• Archiving events using software such as the Elastic Stack

• Invoking a custom shell script

A single Executor usually takes care of a single Action type.

Tornado Crates Documentation Links

Tornado’s crate docs are produced according to the Rust documentation standards. The shortcuts below, organized thematically, will take you to the documentation for each module.

Note

Some of the link will bring you to the upstream Tornado repository, because they contain some advanced notion that are outside the scope of this User Guide.

Common Traits and Code

• tornado-common-api. (Github) The Common API page describes the API and defines the Event and Action structures.

• Common Logger. The Logger page describes how Tornado logs its own actions.

Collectors

• tornado-common-collector. (Github) This crate describes the commonalities of all Collector types.

• Email Collector. Describes a collector that receives a MIME email message and generates an Event.

• JMESPath Collector. This page illustrates the Collector for JSON events using the JMESPath JSON query language.

• tornado-json-collectors. (Github) Present the standard JSON collector that deserializes an unstructured JSON string into an Event.

Engine

• Matcher Engine. The Matcher page describes the structure of the rules used in matching.

Executors

• The tornado-executor-common crate (Github) describes the commonalities of all Executor types

• the Director Executor gathers data from Tornado Actions and sends it to the REST API of Icinga Director

• The Logger Executor outputs the whole Action body to the standard log at the info level

• The Script Executor launches shell scripts

• The Foreach Executor loops on a set of data and executes Actions on each entry

• The Archive Executor writes to log files on locally mounted file systems

• The Elasticsearch Executor is similar to the Director Executor, but sends data to Elasticsearch instead of the Director

• The Icinga 2 Executor forwards Tornado Actions to the Icinga 2 API

• The Smart Monitoring Check Result Executor carries out specific process checks against Icinga 2

Network

• tornado-network-common. (Github) This page contains high level traits not bound to any specific network technology.

• tornado-simple-network. (Github) Describes tests that dispatch Events and Actions on a single process without actually making network calls.

Executables

• Tornado Engine (Executable). Describes the structure of the Tornado binary executable, and the structure and configuration of many of its components.

• Tornado Email Collector (Executable). An executable that processes incoming emails and generates Tornado Events.

• Tornado Icinga 2 Collector (executable). An executable that subscribes to Icinga 2 Event Streams API and generates Tornado Events.

• Tornado Nats JSON Collector (executable). An executable that subscribes to Nats channels and generates Tornado Events.

• Tornado Rsyslog Collector (executable). The description of a binary executable that generates Tornado Events from rsyslog inputs.

• SNMP Trap Daemon Collectors. A Perl trap handler for Net-SNMP’s to subscribe to snmptrapd events.

• Tornado Webhook Collector (executable). A standalone HTTP server binary executable that listens for REST calls from a generic Webhook.

Tornado License

Tornado is licensed under the Apache License, Version 2.0 or the MIT license, at your choice. All files in the project carrying such notice may not be copied, modified, or distributed except according to those terms.