User Guide

Under the hood

Processing Tree Configuration

Note

The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and “OPTIONAL” in this document are to be interpreted as described in RFC 2119.

The location of configuration files in the file system is pre-configured in NetEye. NetEye automatically starts Tornado as follows:

  • Reads the configuration from the /neteye/shared/tornado/conf directory

  • Loads and validates the Matcher configuration from the directory /neteye/shared/tornado/conf/rules.d

  • Starts the Tornado Engine

The hierarchy of the subdirectories in the rules directory defines the structure of the Processing Tree, where each directory is a node in the Processing Tree. Each directory can be one of the following nodes:

  • A Filter

  • A Iterator

  • A Ruleset

Each node type MUST contain a JSON file that contains the node data in a defined format, as well as subdirectories that contain the node’s children.

Hint

Tornado will ignore all other file types.

Any Processing Node directory MUST NOT contain any JSON files, other than the one needed for the node definition. If another JSON file is present, the validation will fail and the Tornado Engine will fail to start up.

The names of all child nodes of a filter or iterator, as well as the names of all rules in a ruleset MUST be unique and MUST NOT contain any characters, other than letters, numbers and “_” (underscore).

  1. Loading the config version: First of all, Tornado will try to load the file version.json, which should contain the field version that defines the current config version. If the version is too old, please execute sudo -u tornado tornado rules-upgrade.

  2. Loading the Processing Tree Nodes: If the version matches the latest config version, Tornado will load all the subdirectories as Processing Tree nodes.

  3. Loading a Filter: A filter directory MUST contain a file called filter.json that contains the definition of the filter. The filter MAY contain subdirectories defining the filter’s child nodes. The definition MUST NOT contain any other fields as specified below. A filter file MUST contain the following fields:

    1. type: The type always needs to be "filter"

    2. name: The name of the filter node. It SHOULD be the same as the directory name.

    3. description: A quick description of the filter node. It MAY be just an empty string.

    4. active: If this field is set to false, the Processing Node will still be validated, but will not be part of the Processing Tree.

    5. filter: The filter field MAY contain an Operator otherwise it MUST be an empty object.

  4. Loading an Iterator: An Iterator directory MUST contain a file called iterator.json that describes the iterator. The iterator directory MAY contain subdirectories defining the iterators child nodes. An Iterator node MUST NOT have another iterator node as its child. The definition MUST NOT contain any other fields as specified below. An Iterator file MUST contain the following fields:

    1. type: The type always needs to be "iterator"

    2. name: The name of the iterator node. It SHOULD be the same as the directory name.

    3. description: A quick description of the iterator node. It MAY be just an empty string.

    4. active: If this field is set to false, the Processing Node will still be validated, but will not be part of the Processing Tree.

    5. target: The target field MUST be a valid accessor expression.

  5. Loading a Ruleset: A ruleset directory MUST contain a file called ruleset.json. Furthermore it MUST contain a directory rules, that contains all the rule definitions for the ruleset. The rules will be loaded from the filesystem in lexicographic order. The ruleset file MUST contain only the following fields:

    1. type: The type always needs to be "ruleset"

    2. name: The name of the iterator node. It SHOULD be the same as the directory name.

  6. Loading a Rule: A rule file MAY only exist in the rules directory of a ruleset node. It MUST only contain the following fields:

    1. name: The name of the iterator node. It SHOULD be the same as the directory name.

    2. description: A quick description of the iterator node. It MAY be just an empty string.

    3. continue: Defines, whether the processing of the rules should continue if the current rule was matched.

    4. active: If this field is set to false, the Processing Node will still be validated, but will not be part of the Processing Tree.

    5. constraint: The field constraint is an object and MUST only contain the following fields:

      1. WHERE: This field MAY contain an Operator or else be left out.

      2. WITH: This field MUST contain a JSON object with value names as a key and an Extractor as a value. The object MAY be left empty but MUST be present in the constraint definition.

    6. actions: The action field MUST be an array, containing a number of actions to perform. The array MAY be empty, but MUST NOT be left out of the rule definition. An action MUST have the following fields:

      1. id: The id MUST be a string that defines the executor, which will process the action.

      2. payload: The payload for the executor. This MUST be a JSON Object.

The directory names for the nodes MUST NOT be relied upon when interacting with the configuration on the filesystem and are considered implementation details. Directory names MAY change during a deploy or a rule migration.

For example, consider this directory structure:

/neteye/shared/tornado/conf/rules.d/
  ├ version.json
  └ master/
    ├ filter.json
    │ └ master_iterator/
    │   ├ iterator.json
    │   ├ iterator_ruleset/
    │   │ ├ ruleset.json
    │   │ └ rules/
    │   │   ├ 0000000010_archive.json
    │   │   └ 0000000020_director.json
    │   └ splitter_child_ruleset_node1/
    │     ├ ruleset.json
    │     └ rules/
    └ master_child_filter/
      └ filter.json

When Tornado is loading this configuration, the Processing Tree will be organized as follows:

  • The root node will have one child filter that is defined in the file master/filter.json

  • The node rules.d/master is a filter node with two child nodes:

    • The node rules.d/master/master_iterator is an iterator with one child.

      • The node rules.d/master/master_iterator/iterator_ruleset is a ruleset containing two rules.

    • The node rules.d/master/master_child_filter is a filter node without any children.

String Interpolation

An action payload can contain text with placeholders that Tornado will replace at runtime. The values to be used for the substitution are extracted from the incoming Events following the conventions mentioned in the previous section; for example, using that Event definition, this string in the action payload:

Received a ${event.type} with protocol ${event.payload.protocol}

produces:

*Received a trap with protocol UDP*

Note

Only values of type String, Number, Boolean and null are valid. Consequently, the interpolation will fail, and the action will not be executed, if the value associated with the placeholder extracted from the Event is an Array, a Map, or undefined.

Matcher Engine Implementation Details

The Matcher crate contains the core logic of the Tornado Engine. It is in charge of:

  • Receiving events from the Collectors

  • Processing incoming events

  • Detecting which Filters, Iterators and Rules an event matches

  • Triggering expected actions

Due to its strategic position, its performance is of utmost importance for global throughput. A matcher is stateless and thread-safe, thus a single instance can be used to serve the entire application load.

At a very high level view, when the matcher initializes, it follows these steps:

  • Configuration (see the code in the “config” module): The configuration phase loads a set of files from the file system. Each file is either a Filter, a Iterator or a Rule, defined in the JSON format. The loading generates an internal structure representing the hierarchy from the filesystem.

  • Validation (see the code in the “validator” module): The Validator receives the Processing Tree configuration and verifies that all nodes respect a set of predefined constraints (e.g., the identifiers cannot contain dots).

  • Match Preparation (see the code in the “matcher” module): The Matcher receives the Processing Tree configuration, and for each node:

    • if the node is a Filter:

      • Builds the Accessors for accessing the event properties using the AccessorBuilder (see the code in the “accessor” module).

      • Builds an Operator for evaluating whether an event matches the Filter itself (using the OperatorBuilder, code in the “operator” module).

      • Builds all the child nodes.

    • if the node is an Iterator:

      • Builds the accessor expression, defined in the target.

      • Builds all the child nodes.

    • if the node is a rule:

      • Builds the Accessors for accessing the event properties using AccessorBuilder (see the code in the “accessor” module).

      • Builds the Operator for evaluating whether an event matches the “WHERE” clause of the rule (using the OperatorBuilder, code in the “operator” module).

      • Builds the Extractors for generating the user-defined variables using the ExtractorBuilder (see the code in the “extractor” module).

  • Listening: Listen for incoming events and then match them against the stored Filters, Iterators and Rules.

Tornado Monitoring and Statistics

Tornado Engine performance metrics are exposed via Tornado APIs and periodically collected by a dedicated telegraf instance telegraf_tornado_monitoring.service. Metrics are stored into the database master_tornado_monitoring in InfluxDB.

Tornado Monitoring and Statistics gives an insight about what data Tornado is processing and how. These information can be useful in several scenarios, including workload inspection, identification of bottlenecks, and issue debugging. A common use case is to identify performance-related issues: for example a difference between the amount of events received and events processed by Tornado may identify a performance problem because Tornado does not have enough resources to handle the current workload.

Examples of collected metrics are:

  • events_processed_counter: total amount of event processed by Tornado Engine

  • events_received_counter: total amount of events received by Tornado Engine through all Collectors

  • actions_processed_counter: total amount of actions executed by Tornado Engine

Metrics will be automatically deleted according to the selected retention policy.

The user can configure Tornado Monitoring and Statistics via GUI under Configuration / Modules / Tornado / Configuration. Two parameters are available:

  • Tornado Monitoring Retention Policy: defines the number of days for which metrics are retained in InfluxDB and defaults to 7 days, after which data will be no longer available.

  • Tornado Monitoring Polling Interval: sets how often the Collector queries the Tornado APIs to gather metrics and defaults to 5 seconds.

To apply changes you have to either run neteye install for both options or execute /usr/share/neteye/tornado/scripts/apply_tornado_monitoring_retention_policy.sh or /usr/share/neteye/tornado/scripts/apply_tornado_monitoring_polling_interval.sh, according to the parameter changed.

Note

On a NetEye Cluster, execute the command on the node where icingaweb2 is active.

Tornado Engine (Executable)

This crate contains the Tornado Engine executable code, which is a configuration of the Engine based on actix and built as a portable executable.

Structure of Tornado Engine

This specific Tornado Engine executable is composed of the following components:

  • A JSON Collector

  • The Engine

  • The Archive Executor

  • The Elasticsearch Executor

  • The Foreach Executor

  • The Icinga 2 Executor

  • The Director Executor

  • The Monitoring Executor

  • The Logger Executor

  • The Script Executor

  • The Smart Monitoring Executor

Each component is wrapped in a dedicated actix actor.

This configuration is only one of many possible configurations. Each component has been developed as an independent library, allowing for greater flexibility in deciding whether and how to use it.

At the same time, there are no restrictions that force the use of the components into the same executable. While this is the simplest way to assemble them into a working product, the Collectors and Executors could reside in their own executables and communicate with the Tornado Engine via a remote call. This can be achieved either through a direct TCP or HTTP call, with an RPC technology (e.g., Protobuf, Flatbuffer, or CAP’n’proto), or with a message queue system (e.g., Nats.io or Kafka) in the middle for deploying it as a distributed system.

Tornado Interaction with Icinga 2

The interaction between Tornado and Icinga 2 is explained in details in sections Icinga 2 and Smart Monitoring Check Result. In particular the Smart Monitoring Executor interacts with Icinga 2 to create objects and set their statuses. To ensure that the status of the Icinga 2 objects does not get lost, NetEye provides an automatism that stops the execution of Smart Monitoring Actions during any Icinga 2 restart or Icinga Director deployment.

The automatism keeps track of all Icinga 2 restarts (we consider also Icinga Director deployments as Icinga 2 restarts) in the icinga2_restarts table of the director database. As soon as an Icinga 2 restart takes place, a new entry with PENDING status is added in that table and at the same time the Tornado Smart Monitoring Executor is deactivated via API.

The icinga-director.service unit monitors the status of the Icinga 2 restarts that are in PENDING status and sets them to FINISHED as soon as the service recognizes that Icinga 2 completed the restart, then the Tornado Smart Monitoring Executor is activated. In case of Icinga2 errors, see the troubleshooting page ::ref::icinga2-not-starting.