Collecting Events

Overview

Tornado provides a number of preconfigured Collectors that handle inputs from various data sources:

1. Email Collector

2. Rsyslog Collector

3. Webhook Collector

4. Nats JSON Collector

5. Icinga 2 Collector

6. SNMP Trap Daemon Collector

7. SMS Collector

Most of the Tornado Collectors are functioning out of the box and do not require manual configuration. However, there are some, that may be configured to work in accordance with your needs.

The Event Type

All collectors convert the external events they collect to Tornado internal events that are then forwarded to the Tornado Processing Engine. These events share a similar base structure, but every collector adds their own data to the event payload and sets a custom event type. An event consists of the following properties:

Property	Type	Description
type	String	The type/source, of the event. Dependend on the collector that creates the event
created_ms	Integer	A timestamp in milliseconds, when the collector created the event
payload	Object	A JSON object that contains all the data for the event
metadata (Optional)	EventMetadata	Metadata for the event

Note

The Tornado iterator node can also add a new field iterator to the event, which is only present in the processing tree for the children of an iterator node.

EventMetadata

Property	Type	Description
tenant_id (Optional)	String	If the event is received over Nats, the Tornado collector will set this field, dependend on the sender.

Webhook

Out of the box the Webhook Collector is available on the NetEye Master and, if present, on Satellites, and the webhooks are to be configured.

On startup, it creates a dedicated REST endpoint for each configured webhook. Calls received by an endpoint are processed by the embedded JMESPath Collector that uses them to produce Tornado Events. In the final step, the Events are forwarded to the Tornado Engine through the configured connection type.

Webhooks in Tornado Webhook Collector are defined by JSON files, which are stored in the folder /neteye/shared/tornado_webhook_collector/conf/webhooks/.

For each webhook, you must provide the following values in order to successfully create an endpoint:

• id: The webhook identifier. This will determine the path of the endpoint; it must be unique per webhook.

• token: A security token that the webhook issuer has to include in the URL as part of the query string (see the example at the bottom of this page for details). If the token provided by the issuer is missing or does not match the one owned by the Collector, then the call will be rejected and an HTTP 401 code (UNAUTHORIZED) will be returned.

• collector_config: The transformation logic that converts a webhook JSON object into a Tornado Event. It consists of a JMESPath Collector configuration.

• event_type: A mandatory field to define the event type of the Tornado event generated based on a call received by an endpoint.

Assuming the webhook request is:

curl -v -X POST https://<neteye.master>/tornado/webhook/event/ups?token=my-secret-token -d '{"ups": "001", "type": "critical", "message": "Battery level critical"}'

the configuration file ups_notifications.conf would contain the following values:

{
 "id": "ups",
 "token": "my-secret-token",
 "collector_config": {
   "event_type": "ups_notification",
   "payload": {
     "source": "${@}"
    }
  }
}

the event received is:

{
 "created_ms": 1713266596490,
 "metadata": {
   "tenant_id": "master",
 },
 "payload": {
   "source":{
     "message":"Battery level critical",
     "type":"critical",
     "ups":"001"
     }
   }
 },
 "type": "ups_notification"
}

Detailed information on how to configure webhooks in Tornado can the found inside the official Tornado documentation; in particular, the Webhook Collector documentation describes the architecture of the Webhook Collector and its configuration parameters and options.

Icinga 2

The Icinga 2 Collector subscribes to the Icinga 2 API event streams, generates Tornado Events from the Icinga 2 Events, and publishes them on the Tornado Engine.

On startup, it connects to Icinga 2 Server API and subscribes to user defined Event Streams.

In order to configure the stream, create a JSON file in /neteye/shared/tornado_icinga2_collector/conf/streams/. More than one stream subscription can be defined.

For each stream, you must provide two values in order to successfully create a subscription:

• stream: the stream configuration composed of:

  • types: An array of Icinga 2 Event types;

  • queue: A unique queue name used by Icinga 2 to identify the stream;

  • filter: An optional Event Stream filter. Additional information about the filter can be found in the official documentation.

• collector_config: The transformation logic that converts an Icinga 2 Event into a Tornado Event.

Below you may find examples of valid content for a stream configuration JSON file:

For all Icinga 2 events

{
  "stream": {
    "types": ["CheckResult",
              "StateChange",
              "Notification",
              "AcknowledgementSet",
              "AcknowledgementCleared",
              "CommentAdded",
              "CommentRemoved",
              "DowntimeAdded",
              "DowntimeRemoved",
              "DowntimeStarted",
              "DowntimeTriggered"],
    "queue": "icinga2_AllEvents_all"
  },
  "collector_config": {
    "event_type": "icinga2_AllEvents_all",
    "payload": {
      "response": "${@}"
    }
  }
}

For check result events

{
  "stream": {
    "types": ["CheckResult"],
    "queue": "icinga2_CheckResult_all"
  },
  "collector_config": {
    "event_type": "icinga2_CheckResult_all",
    "payload": {
      "response": "${@}"
    }
  }
}

For notification events

{
  "stream": {
    "types": ["Notification"],
    "queue": "icinga2_Notification_all"
  },
  "collector_config": {
    "event_type": "icinga2_Notification_all",
    "payload": {
      "response": "${@}"
    }
  }
}

For statechange events

{
  "stream": {
    "types": ["StateChange"],
    "queue": "icinga2_StateChange_all"
   },
  "collector_config": {
    "event_type": "icinga2_StateChange_all",
    "payload": {
      "response": "${@}"
    }
  }
}

Note

Based on the Icinga 2 Event Streams documentation, multiple HTTP clients can use the same queue name as long as they use the same event types and filter.

After configuring the streams, restart tornado_icinga2_collector resource.

Email

The Email Collector generates Tornado Events from valid MIME email messages as inputs.

In NetEye, the incoming email messages are managed by the Postfix mail server. Only the emails sent to the local NetEye user eventgw are forwarded to the Tornado Email Collector. You or your Administrator will need to configure the NetEye Postfix mail server to receive emails for the user eventgw.

All the emails received to the before mentioned mailbox are then out of the box forwarded to the Tornado Email Collector that will parse them and convert in Tornado Events with the extracted data.

To check if the Email Collector is working properly, send an email to the dedicated eventgw user which will then be processed by Tornado: # echo "TestContent" | mail -s TestSubject eventgw@localhost

Now test that an email sent to that address makes it to Tornado (the timestamp reported by journalctl should be at most a second or two after you send the email):

# journalctl -u tornado_email_collector.service
Jun 21 15:11:59 host.example.com tornado_email_collector[12240]: [2019-06-21][15:11:59]
[tornado_common::actors::uds_server][INFO] UdsServerActor - new client connected to [/var/run/tornado/email.sock]``

With the attachments included, the ones that are text files will be in plain text, otherwise they will be encoded in base64.

For example, passing this email with attachments:

From: "Francesco" <francesco@example.com>
Subject: Test for Mail Collector - with attachments
To: "Benjamin" <benjamin@example.com>,
 francesco <francesco@example.com>
Cc: thomas@example.com, francesco@example.com
Date: Sun, 02 Oct 2016 07:06:22 -0700 (PDT)
MIME-Version: 1.0
Content-Type: multipart/mixed;
 boundary="------------E5401F4DD68F2F7A872C2A83"
Content-Language: en-US

This is a multi-part message in MIME format.
--------------E5401F4DD68F2F7A872C2A83
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 7bit

<html>Test for Mail Collector with attachments</html>

--------------E5401F4DD68F2F7A872C2A83
Content-Type: application/pdf;
 name="sample.pdf"
Content-Transfer-Encoding: base64
Content-Disposition: attachment;
 filename="sample.pdf"

JVBERi0xLjMNCiXi48/TDQoNCjEgMCBvYmoNCjw8DQovVHlwZSAvQ2F0YWxvZw0KT0YNCg==

--------------E5401F4DD68F2F7A872C2A83
Content-Type: text/plain; charset=UTF-8;
 name="sample.txt"
Content-Transfer-Encoding: base64
Content-Disposition: attachment;
 filename="sample.txt"

dHh0IGZpbGUgY29udGV4dCBmb3IgZW1haWwgY29sbGVjdG9yCjEyMzQ1Njc4OTA5ODc2NTQz
MjEK
--------------E5401F4DD68F2F7A872C2A83--

will generate this Event:

{
  "type": "email",
  "created_ms": 1554130814854,
  "payload": {
    "date": 1475417182,
    "subject": "Test for Mail Collector - with attachments",
    "to": "\"Benjamin\" <benjamin@example.com>, francesco <francesco@example.com>",
    "from": "\"Francesco\" <francesco@example.com>",
    "cc": "thomas@example.com, francesco@example.com",
    "body": "<html>Test for Mail Collector with attachments</html>",
    "attachments": [
      {
        "filename": "sample.pdf",
        "mime_type": "application/pdf",
        "encoding": "base64",
        "content": "JVBERi0xLjMNCiXi48/TDQoNCjEgMCBvYmoNCjw8DQovVHlwZSAvQ2F0YWxvZw0KT0YNCg=="
      },
      {
        "filename": "sample.txt",
        "mime_type": "text/plain",
        "encoding": "plaintext",
        "content": "txt file context for email Collector\n1234567890987654321\n"
      }
    ]
  }
}

Within the Tornado Event, the filename and mime_type properties of each attachment are the values extracted from the incoming email.

Instead, the encoding property refers to the content encoding in the Event itself, which is one of two types:

• plaintext: The content is included in plain text

• base64: The content is encoded in base64

Special cases

The email Collector follows these rules to generate the Tornado Event:

• If more than one body is present in the email or its subparts, the first valid body found is used, while the others will be ignored

• A Content-Disposition different from Inline and Attachment is ignored

• Content Dispositions of type Inline are processed only if the content type is text/*

• The email subparts are not scanned recursively, thus only the subparts at the root level are evaluated

SMS

Before Tornado can correctly catch SMS, you should configure your SMS modem to send events to Tornado. You can follow the configuration procedure at SMS Modem Setup.

Tornado SMS Collector will then receive then valid SMS like:

From: 39123456789
From_TOA: 91 international, ISDN/telephone
From_SMSC: 39123456789
Sent: 23-10-09 16:06:52
Received: 23-10-09 16:06:58
Subject: GSM1
Modem: GSM1
IMSI: 222018005102877
Report: no
Alphabet: ISO
Length: 20

Example text message

and based on that will create the following Tornado Event:

{
  "type": "sms",
  "created_ms": 155412314854,
  "payload": {
     "modem":"GSM1",
     "sender": "+39123456789",
     "text": "Example text message"
  }
}

Within the Tornado Event, the modem, sender and text properties are the values extracted from the incoming SMS.

Tornado Nats JSON

The Nats JSON Collector is a standalone Collector that listens for JSON messages on Nats topics, generates Tornado Events, and sends them to the Tornado Engine.

On startup, it connects to a set of topics on a Nats server. Calls received are then processed by the embedded JMESPath Collector that uses them to produce Tornado Events. In the final step, the Events are forwarded to the Tornado Engine through the configured connection type.

For each topic, you must provide two values in order to successfully configure them:

• nats_topics: A list of Nats topics to which the Collector will subscribe.

• collector_config: (Optional) The transformation logic that converts a JSON object received from Nats into a Tornado Event. It consists of a JMESPath Collector configuration.

Topics configuration

Two startup parameters config-dir and topics-dir determine the path to the topic configurations, and each topic is configured by providing nats_topics and collector_config. topics-dir is the folder where the topic configurations are saved in JSON format. This folder is relative to the config_dir. The default value is /topics/.

An example of valid content for a Topic configuration JSON file is:

 {
  "nats_topics": ["simple_test_one", "simple_test_two"],
  "collector_config": {
    "event_type": "${content.type}",
    "payload": {
      "ref": "${content.ref}",
      "repository_name": "${repository}"
    }
  }
}

With this configuration, two subscriptions are created to the Nats topics simple_test_one and simple_test_two. Messages received by those topics are processed using the collector_config that determines the content of the Tornado Event associated with them.

It is important to note that, if a Nats topic name is used more than once, then the collector will perfom multiple subscriptions accordingly. This can happen if a topic name is duplicated into the nats_topics array or in multiple JSON files. So for example, with this JSON message is received:

{
   "content": {
      "type": "content_type",
      "ref": "refs/heads/master"
   },
   "repository": {
      "id": 123456789,
      "name": "webhook-test"
   }
}

then the resulting Event will be:

{
   "type": "content_type",
   "created_ms": 1554130814854,
   "payload": {
      "ref": "refs/heads/master",
      "repository": {
         "id": 123456789,
         "name": "webhook-test"
      }
   }
}

Default values

The collector_config section and all of its internal entries are optional. If not provided explicitly, the collector will use these predefined values:

• When the collector_config.event_type is not provided, the name of the Nats topic that sent the message is used as Event type.

• When the collector_config.payload is not provided, the entire source message is included in the payload of the generated Event with the key data.

Consequently, the simplest valid topic configuration contains only the nats_topics:

{
   "nats_topics": ["subject_one", "subject_two"]
}

The above one is equivalent to:

{
   "nats_topics": ["subject_one", "subject_two"],
   "collector_config": {
      "payload": {
       "data": "${@}"
      }
   }
}

In this case the generated Tornado Events have type equals to the topic name and the whole source data in their payload.

Rsyslog

The rsyslog Collector binary is an executable that generates Tornado Events from rsyslog inputs.

The collector is pre-configured and is not to be started manually.

The example of the rsyslog event is:

{
   "type": "syslog",
   "created_ms": 1713881098196,
   "payload": {
      "@timestamp": "2024-04-23T16:04:58.016685+02:00",
      "facility": "daemon",
      "host": "myhostname",
      "message": "my-service.service: Failed with result exit-code.",
      "severity": "WARNING",
      "source": "systemd",
      "syslog-tag": "systemd[1]:"
   },
   "type": "syslog"
}

SNMP Trap Event

The SNMP Trap Collector receives and parses messages coming from snmptrapd. It will then pass those messages as Events through a specific communication channel to Tornado.

The snmptrap is configured out of the box on the NetEye Master and, if present, on Satellites. Thus, in case a trap message received by a tenant will be automatically sent to Tornado via through its dedicated Satellite via NATS Communication channel.

The received messages are kept in an in-memory non-persistent buffer that makes the application resilient to crashes or temporary unavailability of the communication channel. When the connection to the channel is restored, all messages in the buffer will be sent. When the buffer is full, the Collectors will start discarding old messages. The buffer max size is set to 10000 messages.

Consider a snmptrapd message that contains the following information:

PDU INFO:
  version                        1
  errorstatus                    0
  community                      public
  receivedfrom                   UDP: [127.0.1.1]:41543->[127.0.2.2]:162
  transactionid                  1
  errorindex                     0
  messageid                      0
  requestid                      414568963
  notificationtype               TRAP
VARBINDS:
  iso.3.6.1.2.1.1.3.0            type=67 value=Timeticks: (1166403) 3:14:24.03
  iso.3.6.1.6.3.1.1.4.1.0        type=6  value=OID: iso.3.6.1.4.1.8072.2.3.0.1
  iso.3.6.1.4.1.8072.2.3.2.1     type=2  value=INTEGER: 123456

The Collector will produce this Tornado Event:

{
   "type":"snmptrapd",
   "created_ms":"1553765890000",
   "payload":{
      "protocol":"UDP",
      "src_ip":"127.0.1.1",
      "src_port":"41543",
      "dest_ip":"127.0.2.2",
      "PDUInfo":{
         "version":"1",
         "errorstatus":"0",
         "community":"public",
         "receivedfrom":"UDP: [127.0.1.1]:41543->[127.0.2.2]:162",
         "transactionid":"1",
         "errorindex":"0",
         "messageid":"0",
         "requestid":"414568963",
         "notificationtype":"TRAP"
      },
      "oids":{
         "iso.3.6.1.2.1.1.3.0":"67",
         "iso.3.6.1.6.3.1.1.4.1.0":"6",
         "iso.3.6.1.4.1.8072.2.3.2.1":"2"
      }
   }
}

The structure of the generated Event is not configurable.