Concepts

Passive Monitoring

NetEye’s passive monitoring abilities count on Tornado, a Complex Event Processor that receives reports of events from data sources such as monitoring, email, and telegram, matches them against pre-configured rules, and executes the actions associated with those rules, which can include sending notifications, logging to files, and annotating events in a time series graphing system.

Tornado is a high performance, scalable application, and is intended to handle millions of events each second on standard server hardware. Its overall architecture is depicted in Fig. 150.

Fig. 150 Tornado architecture

Tornado Architecture

Tornado is structured as a library, with three example binaries included that show how it can be used. The three main components of the Tornado architecture are:

• The Tornado Collector(s), or just Collector(s)

• The Tornado Engine, or Engine

• The Tornado Executor(s), or Executor(s)*

The term Tornado refers to the whole project or to a deployed system that includes all three components.

Along with the main components, the following concepts are fundamental to the Tornado architecture:

• An Action: An operation performed by Tornado, usually on an external system. For example, writing to Elastic Search or setting a state in a monitoring system.

• A Tornado (or Internal) Event: The Tornado-specific Event format.

• A Datasource: A system that sends External Events to Tornado, or a system to which Tornado subscribes to receive External Events.

• An External Event: An input received from a datasource, whose format depends on its source. An example of input are events collected from rsyslog.

• A Rule: A group of conditions that an Internal Event must match to trigger a set of Actions.

See also

Events and Actions are described in more detail in the Tornado common API documentation.

Architecturally, Tornado is organized as a processing pipeline, where input events move from Collectors to the Engine, to Executors, without branching or returning.

When the system receives an External Event, it first arrives at a Collector where it is converted into a Tornado Event. Then it is forwarded to the Tornado Engine where it is matched against user-defined, composable Rules. Finally, generated Actions are dispatched to the Executors.

The Tornado pipeline:

Datasources (e.g. rsyslog)
  |
  | External Events
  |
  \-> Tornado Collectors
        |
        | Tornado (or Internal) Events
        |
        \-> Tornado Engine (matches based on Rules)
              |
              | Actions
              |
              \-> Tornado Executors (execute the Actions)

Collectors

The purpose of a Collector is to receive and convert external events into the internal Tornado Event structure, and forward them to the Tornado Engine.

Collectors are Datasource-specific. For each datasource, there must be at least one Collector that knows how to manipulate the datasource’s Events and generate Tornado Events.

Out of the box, Tornado provides a number of Collectors for handling inputs from snmptrapd, rsyslog, JSON from Nats channels and generic Webhooks.

Because all Collectors are defined with a simple format, Collectors for new event types can easily be added or extended from existing types for:

• Monitoring events

• Email messages

• Telegram

• DNS

• Cloud monitoring (AWS, Azure, Cisco/Meraki, etc.)

• Netflow

• Elastic Stack

• SMS

• Operating system and authorization events

More information on the Collectors can be found in Section Tornado Collectors, while configuration details for them are in section Tornado Collectors Configuration.

Engine

The Engine is the second step of the pipeline. It receives and processes the events produced by the Collectors. The outcome of this step is fully defined by a processing tree composed of Filters and Rule Sets.

A Filter is a processing node that defines an access condition on the children nodes.

A Rule Set is a node that contains an ordered set of Rules, where each Rule determines:

• The conditions a Tornado Event has to respect to match it

• The actions to be executed in case of a match

The processing tree is parsed at startup from a configuration folder where the node definitions are stored in JSON format.

When an event matches one or more Rules, the Engine produces a set of Actions and forwards them to one or more Executors.

Executors

The Executors are the last element in the Tornado pipeline. They receive the Actions produced from the Engine and trigger the associated executable instructions.

An Action can be any command, process or operation. For example it can include:

• Forwarding the events to a monitoring system

• Logging events locally (e.g., as processed, discarded or matched) or remotely

• Archiving events using software such as the Elastic Stack

• Invoking a custom shell script

A single Executor usually takes care of a single Action type.

More information and example code snippets can be found in Section Tornado Executors.

Tornado Interaction with Icinga 2

The interaction between Tornado and Icinga 2 is explained in details in sections Icinga 2 Executor and Smart Monitoring Check Result Executor. In particular the Smart Monitoring Executor interacts with Icinga 2 to create objects and set their statuses. To ensure that the status of the Icinga 2 objects does not get lost, NetEye provides an automatism that stops the execution of Smart Monitoring Actions during any Icinga 2 restart or Icinga Director deployment.

The automatism keeps track of all Icinga 2 restarts (we consider also Icinga Director deployments as Icinga 2 restarts) in the icinga2_restarts table of the director database. As soon as an Icinga 2 restart takes place, a new entry with PENDING status is added in that table and at the same time the Tornado Smart Monitoring Executor is deactivated via API.

The icinga-director.service unit monitors the status of the Icinga 2 restarts that are in PENDING status and sets them to FINISHED as soon as the service recognizes that Icinga 2 completed the restart, then the Tornado Smart Monitoring Executor is activated.

Icinga 2 (including Icinga Director deployments) can not restart if for any reason there is another restart PENDING, i.e., ongoing or stuck, which in some corner case may result in Icinga 2 restarts being blocked. Should you encounter such unfortunate situation, you can manually unblock new restarts by issuing the following command, which will set the status of all PENDING restarts to OBSOLETE (i.e., restarts that are outdated):

neteye# icingacli director icinga2restart resetlock

Tornado Crates Documentation Links

Tornado’s crate docs are produced according to the Rust documentation standards. The shortcuts below, organized thematically, will take you to the documentation for each module.

Note

Some of the link will bring you to the upstream Tornado repository, because they contain some advanced notion that are outside the scope of this User Guide.

Common Traits and Code

• tornado-common-api. (Github) The Common API page describes the API and defines the Event and Action structures.

• Common Logger. The Logger page describes how Tornado logs its own actions.

Collectors

• tornado-common-collector. (Github) This crate describes the commonalities of all Collector types.

• Email Collector. Describes a Collector that receives a MIME email message and generates an Event.

• JMESPath Collector. This page illustrates the Collector for JSON events using the JMESPath JSON query language.

• tornado-json-collectors. (Github) Present the standard JSON Collector that deserializes an unstructured JSON string into an Event.

Engine

• Matcher Engine Implementation Details. The Matcher page describes the structure of the rules used in matching.

Executors

• The tornado-executor-common crate (Github) describes the commonalities of all Executor types

• the Director Executor gathers data from Tornado Actions and sends it to the REST API of Icinga Director

• The Logger Executor outputs the whole Action body to the standard log at the info level

• The Script Executor launches shell scripts

• The Foreach Executor loops on a set of data and executes Actions on each entry

• The Archive Executor writes to log files on locally mounted file systems

• The Elasticsearch Executor is similar to the Director Executor, but sends data to Elasticsearch instead of the Director

• The Icinga 2 Executor forwards Tornado Actions to the Icinga 2 API

• The Smart Monitoring Check Result Executor carries out specific process checks against Icinga 2

Network

• tornado-network-common. (Github) This page contains high level traits not bound to any specific network technology.

• tornado-simple-network. (Github) Describes tests that dispatch Events and Actions on a single process without actually making network calls.

Executables

• Tornado Engine (Executable). Describes the structure of the Tornado binary executable, and the structure and configuration of many of its components.

• Tornado Email Collector (Executable). An executable that processes incoming emails and generates Tornado Events.

• Tornado Icinga 2 Collector (executable). An executable that subscribes to Icinga 2 Event Streams API and generates Tornado Events.

• Tornado Nats JSON Collector (executable). An executable that subscribes to Nats channels and generates Tornado Events.

• Tornado Rsyslog Collector (executable). The description of a binary executable that generates Tornado Events from rsyslog inputs.

• SNMP Trap Daemon Collectors. A Perl trap handler for Net-SNMP’s to subscribe to snmptrapd events.

• Tornado Webhook Collector (executable). A standalone HTTP server binary executable that listens for REST calls from a generic Webhook.

Tornado License

Tornado is licensed under the Apache License, Version 2.0 or the MIT license, at your choice. All files in the project carrying such notice may not be copied, modified, or distributed except according to those terms.

Processing Tree

Within the NetEye interface you can view the Tornado rule configuration, shown in Fig. 152, graphically instead of in a command shell. The Configuration Viewer (available after clicking Tornado in the left-hand side menu) shows the processing tree in a top-down format, allowing you to verify that the structure of the rules you have created is as you intended.

Definitions

While a more complete description of all Tornado elements is available in the official Tornado engine documentation, they are detailed below.

• Processing Tree: The entire set of filters and rules that creates the hierarchical structure where events are filtered and then matched against one or more rules.

• Filter: A node in the processing tree that contains (1) a filter definition and (2) a set of child nodes, each of which corresponds to a condition in the filter definition. You can use a Filter to create independent pipelines for different types of events, reducing the amount of time needed to process them.

• Implicit Filter: If a filter is not present in a node, a default filter is created which forwards an event to ALL child nodes, rather than a particular one that matched a filter condition.

• Ruleset: A leaf node that contains multiple rules to be matched against the event one by one once the filters in the parent nodes have let an event through.

• Rule: A single rule within a Ruleset, which can be matched against an event.

Details about the configuration and set up of the various Tornado elements is available in Section Processing Tree Configuration.

Tornado Rule Editor

The tornado Rule Editor allows to manage the single rules that are in a ruleset. On top of the window, all the parents of the current ruleset are shown, to allow you to quickly check on which leaf of the processing tree the rules shown are located. Like in the processing tree editor, a JSON validator assists you in checking that the syntax of the rules is correct.

In the main area, all the defined rules are shown, together with a number of information about them: name, action, and status (enabled or not).

Like in the Processing Tree Editor, available options differ depending whether the Edit mode is active or not:

When the Edit mode is not active, it is possible to click on the Open test window button on the top right-hand side of the window to check which events the current rule selection would match.

With active Edit mode, the Open test window button is disabled, but new rules can be added, modified, or deleted; each rule can also can be moved along the list with a simple drag and drop.

Tornado Carbon Interface Overview

In today’s world, the UX has become a key point in successful IT products and NetEye wants it to become one of its strongest features, providing a continuously improved GUI to support the users’s daily activities.

For this purpose, NetEye provides a totally redesigned, modern looking and attractive GUI for Tornado, based on solid design and usability guidelines defined by the Carbon Design System

While the new GUI is developed to completely replace the current UI, it is currently in preview and only some features are supported such as processing tree visualization, event-driven testing and multi-tenancy.

The Graphical User Interface allows you to explore the current configuration of your Tornado Instance.

The GUI can be divided in 3 logical blocks. At the top of the screen we find the toolbar, in this area it is possible to change the tenant and enable the edit mode.

Fig. 151 Tornado Carbon’s Toolbar.

Immediately below we find the most important part: the Processing Tree. All filters and rulesets of the processing tree are shown in hierarchical order. You can navigate the tree by expanding the child nodes of a filter by clicking on the arrow on the left side of the node. When clicking on a node, its details will be shown. In case of a ruleset, in addition to the details, also the list of rules will appear. To see the details of a rule, you can click on the entry in the list of rules.

Fig. 152 Example Processing Tree.

The test event window is a panel that can be opened using the icon in the top right corner, which allows you to send test Events. These Events can be created through a dedicated form and are composed by the following four fields:

• Event type: the type of the Event, such as trap, sms, email, etc.

• Creation time: the Event timestamp defined as an epoch in milliseconds.

• Tenant ID: The tenant ID that will be added to the event. Fill this field if there are filters or conditions related to the tenant ID, otherwise just leave it empty.

• Enable execution of actions: whether the actions of matching rules have to be executed or skipped.

• Payload: the event payload in JSON format.

Fig. 153 The test event window.

When a test is executed by clicking the “Run Test” button, the linked Event is sent to Tornado and the outcome of the operation will be reported in the Processing Tree.

Following the yellow line it is possible to see the path that the event has taken. The nodes that have matched the event are distinguishable by a full yellow lightning bolt while those partially matched have an empty bolt.

Fig. 154 A Processing Tree with an event result

At this point, a rule can be in one of the following states:

• matched: If a rule matched the Event.

• stopped: If a rule matched the Event and then stopped the. execution flow. This happens if the continue flag of the rule is set to false.

• partially matched: If the where condition of the Rule was matched but it was not possible to process the required extracted variables.

• not matched: If the Rule did not match the Event.

Fig. 155 Example of processed rules

• Matched rules: Extract_sender, Extract_subject, Archive_all

• Partially matched: Extract_message

• Not matched: Block_invalid_senders

For each rule in the table, the extracted variables and the generated Action payloads are shown. In addition, all these extracted variables are also shown in the Event Test form.

Fig. 156 Sample of extracted variables

Two other buttons are visible, one for cleaning all the fields of the form and one for cleaning the outcome of the test.

Tornado Processing Tree Editor

The Tornado GUI provides an edit mode that allows to modify the configuration of the Tornado rules’ processing tree directly from NetEye’s web interface. Two important principles have been used for the development of the edit mode and must be understood and taken into account when modifying Tornado’s configuration:

• Implicit Lock Mode. Only one user at a time can modify the processing tree configuration. This prevents multiple users from changing the configuration simultaneously, which might lead to unwanted results and possibly to Tornado not working correctly due to incomplete or wrong configuration. When a user is editing the configuration, the actual, running configuration is left untouched: it continues to be operative and accepts incoming data to be processed.

• Edit Mode. When starting to modify the configuration, Tornado will continue to work with the existing configuration–thanks to the implicit lock mode, while the new changes are saved in a separate draft configuration. The new configuration then must be deployed to become operative.

  This mode has other positive side effects: one does not need to complete the changes in one session, but can stop and then continue at a later point; another user can pick up the draft and complete it; in case of a disaster (like e.g., the abrupt end of the HTTPS connection to the GUI) it is possible to resume the draft from the point where it was left.

Warning

Only one draft at a time is allowed; that is, editing of multiple draft is not supported!

When a user enters the edit mode, a new draft is created on the fly if none is present, which will be an exact copy of the running Tornado configuration. If not present in the draft, a root node of type Filter will be automatically added to the draft.

To check for the correctness of a Draft, without impacting the deployed configuration, it is possible to open the test window also while in Edit Mode. The event will be processed using the Draft and the result will be displayed, while keeping the existing configuration running.

You can add a new node in two ways:

• by clicking on the “Add” button in the top right corner and then selecting the parent node to which you want to add the new node.

• by clicking on the icon with the three dots on each node that from now on we will call overflow menu.

Note

All nodes at the same level are ordered alphabetically

For each node, it is possible to define a name. For a filter, these three more options are available:

• a description

• whether it is active or not

• the filter that should match the event. A specific editor is available for the user to create a valid filter or alternatively it is possible to write it via JSON-based syntax, and examples can be found in the various How-tos present in the tornado section of the User Guide.

Moreover, in Edit mode, each node can be deleted by clicking the delete item available in the overflow menu.

Tornado Collectors

JMESPath Collector

This is a Collector that receives an input in JSON format and allows the creation of Events using the JMESPath JSON query language. It is used only internally by Tornado and should be never called directly.

Email Collector

When the Email Collector receives a valid MIME email message as input, it parses it and produces a Tornado Event with the extracted data.

For example, given the following input:

Subject: This is a test email
Content-Type: multipart/alternative; boundary=foobar
Date: Sun, 02 Oct 2016 07:06:22 -0700 (PDT)

--foobar
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

This is the plaintext version, in utf-8. Proof by Euro: =E2=82=AC
--foobar
Content-Type: text/html
Content-Transfer-Encoding: base64

PGh0bWw+PGJvZHk+VGhpcyBpcyB0aGUgPGI+SFRNTDwvYj4gdmVyc2lvbiwgaW4g
dXMtYXNjaWkuIFByb29mIGJ5IEV1cm86ICZldXJvOzwvYm9keT48L2h0bWw+Cg==
--foobar--

it will generate this Event:

{
  "type": "email",
  "created_ms": 1554130814854,
  "payload": {
    "date": 1475417182,
    "subject": "This is a test email",
    "to": "",
    "from": "",
    "cc": "",
    "body": "This is the plaintext version, in utf-8. Proof by Euro: €",
    "attachments": []
  }
}

If there are attachments, then attachments that are text files will be in plain text, otherwise they will be encoded in base64.

For example, passing this email with attachments:

From: "Francesco" <francesco@example.com>
Subject: Test for Mail Collector - with attachments
To: "Benjamin" <benjamin@example.com>,
 francesco <francesco@example.com>
Cc: thomas@example.com, francesco@example.com
Date: Sun, 02 Oct 2016 07:06:22 -0700 (PDT)
MIME-Version: 1.0
Content-Type: multipart/mixed;
 boundary="------------E5401F4DD68F2F7A872C2A83"
Content-Language: en-US

This is a multi-part message in MIME format.
--------------E5401F4DD68F2F7A872C2A83
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 7bit

<html>Test for Mail Collector with attachments</html>

--------------E5401F4DD68F2F7A872C2A83
Content-Type: application/pdf;
 name="sample.pdf"
Content-Transfer-Encoding: base64
Content-Disposition: attachment;
 filename="sample.pdf"

JVBERi0xLjMNCiXi48/TDQoNCjEgMCBvYmoNCjw8DQovVHlwZSAvQ2F0YWxvZw0KT0YNCg==

--------------E5401F4DD68F2F7A872C2A83
Content-Type: text/plain; charset=UTF-8;
 name="sample.txt"
Content-Transfer-Encoding: base64
Content-Disposition: attachment;
 filename="sample.txt"

dHh0IGZpbGUgY29udGV4dCBmb3IgZW1haWwgY29sbGVjdG9yCjEyMzQ1Njc4OTA5ODc2NTQz
MjEK
--------------E5401F4DD68F2F7A872C2A83--

will generate this Event:

{
  "type": "email",
  "created_ms": 1554130814854,
  "payload": {
    "date": 1475417182,
    "subject": "Test for Mail Collector - with attachments",
    "to": "\"Benjamin\" <benjamin@example.com>, francesco <francesco@example.com>",
    "from": "\"Francesco\" <francesco@example.com>",
    "cc": "thomas@example.com, francesco@example.com",
    "body": "<html>Test for Mail Collector with attachments</html>",
    "attachments": [
      {
        "filename": "sample.pdf",
        "mime_type": "application/pdf",
        "encoding": "base64",
        "content": "JVBERi0xLjMNCiXi48/TDQoNCjEgMCBvYmoNCjw8DQovVHlwZSAvQ2F0YWxvZw0KT0YNCg=="
      },
      {
        "filename": "sample.txt",
        "mime_type": "text/plain",
        "encoding": "plaintext",
        "content": "txt file context for email Collector\n1234567890987654321\n"
      }
    ]
  }
}

Within the Tornado Event, the filename and mime_type properties of each attachment are the values extracted from the incoming email.

Instead, the encoding property refers to the content encoding in the Event itself, which is one of two types:

• plaintext: The content is included in plain text

• base64: The content is encoded in base64

Special cases

The email Collector follows these rules to generate the Tornado Event:

• If more than one body is present in the email or its subparts, the first valid body found is used, while the others will be ignored

• A Content-Disposition different from Inline and Attachment is ignored

• Content Dispositions of type Inline are processed only if the content type is text/*

• The email subparts are not scanned recursively, thus only the subparts at the root level are evaluated

Tornado Email Collector (Executable)

The Email Collector Executable binary, built on actix, is an executable that generates Tornado Events from MIME email inputs.

On startup, it creates a UDS socket where it listens for incoming email messages. Each email published on the socket is processed by the embedded Email Collector to produce Tornado Events which are, finally, forwarded to the Tornado Engine’s TCP address.

The UDS socket is created with the same user and group as the tornado_email_collector process, with permissions set to 770 (read, write and execute for both the user and the group).

Each client that needs to write an email message to the socket should close the connection as soon as it completes its action. In fact, the Email Collector Executable will not even start processing that email until it receives an EOF signal. Only one email per connection is allowed.

Procmail Example

This client behavior can be obtained, for instance, by using procmail with the following configuration:

## .procmailrc file
MAILDIR=$HOME/Mail                 # You should make sure this exists
LOGFILE=$MAILDIR/procmail.log

# This is where we ask procmail to write to our UDS socket.
SHELL=/bin/sh
:0
| /usr/bin/socat - /var/run/tornado_email_collector/email.sock 2>&1

A precondition for procmail to work is that the mail server in use must be properly configured to notify procmail whenever it receives new email.

For additional information about how incoming email is processed and the structure of the generated Event, check the documentation specific to the embedded Email Collector.

Note that the Email Collector will support any email client that works with the MIME format and UDS sockets.

Tornado Rsyslog Collector (executable)

The rsyslog Collector binary is an executable that generates Tornado Events from rsyslog inputs.

This Collector is meant to be integrated with rsyslog’s own logging through the omprog module. Consequently, it is never started manually, but instead will be started, and managed, directly by rsyslog itself.

Here is an example rsyslog configuration template that pipes logs to the rsyslog-collector (the parameters are explained in section Tornado Rsyslog Collector Configuration) only logs with severity higher than warning:

module(load="omprog")

action(type="omprog"
       binary="/usr/lib64/tornado/bin/tornado_rsyslog_collector --logger=warn")

An example of a fully instantiated startup setup is:

module(load="omprog")

action(type="omprog"
       binary="/usr/lib64/tornado/bin/tornado_rsyslog_collector --config-dir=/tornado-rsyslog-collector/config --tornado-event-socket-ip=192.168.123.12 --tornado-event-socket-port=4747")

Note that all parameters for the binary option must be on the same line. You will need to place this configuration in a file in your rsyslog directory, for instance:

/etc/rsyslog.d/tornado.conf

In this example the Collector will:

• Reads the configuration from the /tornado-rsyslog-collector/config directory

• Write outgoing Events to the TCP socket at tornado_server_ip:4747

The Collector will need to be run in parallel with the Tornado Engine before any events will be processed, for example:

/opt/tornado/bin/tornado --tornado-event-socket-ip=tornado_server_ip

Under this configuration, rsyslog is in charge of starting the Collector when needed and piping the incoming logs to it. As the last stage, the Tornado Events generated by the Collector are forwarded to the Tornado Engine’s TCP socket.

This integration strategy is the best option for supporting high performance given massive amounts of log data.

Because the Collector expects the input to be in JSON format, rsyslog should be pre-configured to properly pipe its inputs in this form.

Tornado Webhook Collector (executable)

The Webhook Collector is a standalone HTTP server built on actix-web that listens for REST calls from a generic webhook, generates Tornado Events from the webhook JSON body, and sends them to the Tornado Engine.

On startup, it creates a dedicated REST endpoint for each configured webhook. Calls received by an endpoint are processed by the embedded JMESPath Collector that uses them to produce Tornado Events. In the final step, the Events are forwarded to the Tornado Engine through the configured connection type.

For each webhook, you must provide three values in order to successfully create an endpoint:

• id: The webhook identifier. This will determine the path of the endpoint; it must be unique per webhook.

• token: A security token that the webhook issuer has to include in the URL as part of the query string (see the example at the bottom of this page for details). If the token provided by the issuer is missing or does not match the one owned by the Collector, then the call will be rejected and an HTTP 401 code (UNAUTHORIZED) will be returned.

• collector_config: The transformation logic that converts a webhook JSON object into a Tornado Event. It consists of a JMESPath Collector configuration as described in its specific documentation.

Tornado Nats JSON Collector (executable)

The Nats JSON Collector is a standalone Collector that listens for JSON messages on Nats topics, generates Tornado Events, and sends them to the Tornado Engine.

The Nats JSON Collector executable is built on actix.

On startup, it connects to a set of topics on a Nats server. Calls received are then processed by the embedded jmespath Collector that uses them to produce Tornado Events. In the final step, the Events are forwarded to the Tornado Engine through the configured connection type.

For each topic, you must provide two values in order to successfully configure them:

• nats_topics: A list of Nats topics to which the Collector will subscribe.

• collector_config: (Optional) The transformation logic that converts a JSON object received from Nats into a Tornado Event. It consists of a JMESPath Collector configuration as described in its specific documentation.

Tornado Icinga 2 Collector (executable)

The Icinga 2 Collector subscribes to the Icinga 2 API event streams, generates Tornado Events from the Icinga 2 Events, and publishes them on the Tornado Engine TCP address.

The Icinga 2 Collector executable is built on actix.

On startup, it connects to an existing Icinga 2 Server API and subscribes to user defined Event Streams. Each Icinga 2 Event published on the stream, is processed by the embedded jmespath Collector that uses them to produce Tornado Events which are, finally, forwarded to the Tornado Engine’s TCP address.

More than one stream subscription can be defined. For each stream, you must provide two values in order to successfully create a subscription:

• stream: the stream configuration composed of:

  • types: An array of Icinga 2 Event types;

  • queue: A unique queue name used by Icinga 2 to identify the stream;

  • filter: An optional Event Stream filter. Additional information about the filter can be found in the official documentation.

• collector_config: The transformation logic that converts an Icinga 2 Event into a Tornado Event. It consists of a JMESPath Collector configuration as described in its specific documentation.

Note

Based on the Icinga 2 Event Streams documentation, multiple HTTP clients can use the same queue name as long as they use the same event types and filter.

SNMP Trap Daemon Collectors

The _snmptrapd_collector_s of this package are embedded Perl trap handlers for Net-SNMP’s snmptrapd. When registered as a subroutine in the Net-SNMP snmptrapd process, they receives snmptrap-specific inputs, transforms them into Tornado Events, and forwards them to the Tornado Engine.

There are two Collector implementations, the first one sends Events directly to the Tornado TCP socket and the second one forwards them to a NATS server.

The implementations rely on the Perl NetSNMP::TrapReceiver package. You can refer to its documentation for generic configuration examples and usage advice.

The _snmptrapd_collector_s receive snmptrapd messages, parse them, generate Tornado Events and, finally, sends them to Tornado using their specific communication channel.

The received messages are kept in an in-memory non-persistent buffer that makes the application resilient to crashes or temporary unavailability of the communication channel. When the connection to the channel is restored, all messages in the buffer will be sent. When the buffer is full, the Collectors will start discarding old messages. The buffer max size is set to 10000 messages.

Consider a snmptrapd message that contains the following information:

PDU INFO:
  version                        1
  errorstatus                    0
  community                      public
  receivedfrom                   UDP: [127.0.1.1]:41543->[127.0.2.2]:162
  transactionid                  1
  errorindex                     0
  messageid                      0
  requestid                      414568963
  notificationtype               TRAP
VARBINDS:
  iso.3.6.1.2.1.1.3.0            type=67 value=Timeticks: (1166403) 3:14:24.03
  iso.3.6.1.6.3.1.1.4.1.0        type=6  value=OID: iso.3.6.1.4.1.8072.2.3.0.1
  iso.3.6.1.4.1.8072.2.3.2.1     type=2  value=INTEGER: 123456

The Collector will produce this Tornado Event:

{
   "type":"snmptrapd",
   "created_ms":"1553765890000",
   "payload":{
      "protocol":"UDP",
      "src_ip":"127.0.1.1",
      "src_port":"41543",
      "dest_ip":"127.0.2.2",
      "PDUInfo":{
         "version":"1",
         "errorstatus":"0",
         "community":"public",
         "receivedfrom":"UDP: [127.0.1.1]:41543->[127.0.2.2]:162",
         "transactionid":"1",
         "errorindex":"0",
         "messageid":"0",
         "requestid":"414568963",
         "notificationtype":"TRAP"
      },
      "oids":{
         "iso.3.6.1.2.1.1.3.0":"67",
         "iso.3.6.1.6.3.1.1.4.1.0":"6",
         "iso.3.6.1.4.1.8072.2.3.2.1":"2"
      }
   }
}

The structure of the generated Event is not configurable.