Advanced Topics¶

API¶

Self-Monitoring API¶

The monitoring endpoints allow you to monitor the health of Tornado. They provide information about the status, activities, logs and metrics of a running Tornado instance. Specifically, they return statistics about latency, traffic, and errors.

Available endpoints:

Ping endpoint

This endpoint returns a simple message “pong - “ followed by the current date in ISO 8601 format.

Details:

name : ping
path : /monitoring/ping
response type: JSON

response example:

{
  "message": "pong - 2019-04-12T10:11:31.300075398+02:00",
}

Metrics endpoint

This endpoint returns tornado metrics in the Prometheus text format Details:

name : metrics/prometheus
path : /monitoring/v1/metrics/prometheus
response type: Prometheus text format

response example:

# HELP events_processed_counter Events processed count
# TYPE events_processed_counter counter
events_processed_counter{app="tornado",event_type="icinga_process-check-result"} 1
# HELP events_processed_duration_seconds Events processed duration
# TYPE events_processed_duration_seconds histogram
events_processed_duration_seconds_bucket{app="tornado",event_type="icinga_process-check-result",le="0.5"} 1
events_processed_duration_seconds_bucket{app="tornado",event_type="icinga_process-check-result",le="0.9"} 1
events_processed_duration_seconds_bucket{app="tornado",event_type="icinga_process-check-result",le="0.99"} 1
events_processed_duration_seconds_bucket{app="tornado",event_type="icinga_process-check-result",le="+Inf"} 1
events_processed_duration_seconds_sum{app="tornado",event_type="icinga_process-check-result"} 0.000696327
events_processed_duration_seconds_count{app="tornado",event_type="icinga_process-check-result"} 1
# HELP events_received_counter Events received count
# TYPE events_received_counter counter
events_received_counter{app="tornado",event_type="icinga_process-check-result",source="http"} 1
# HELP http_requests_counter HTTP requests count
# TYPE http_requests_counter counter
http_requests_counter{app="tornado"} 1
# HELP http_requests_duration_secs HTTP requests duration
# TYPE http_requests_duration_secs histogram
http_requests_duration_secs_bucket{app="tornado",le="0.5"} 1
http_requests_duration_secs_bucket{app="tornado",le="0.9"} 1
http_requests_duration_secs_bucket{app="tornado",le="0.99"} 1
http_requests_duration_secs_bucket{app="tornado",le="+Inf"} 1
http_requests_duration_secs_sum{app="tornado"} 0.001695673
http_requests_duration_secs_count{app="tornado"} 1

The following metrics are provided:

events_received_counter : total number of received events grouped by source (nats, tcp, http) and event type
events_processed_counter : total number of processed events grouped by event type
events_processed_duration_seconds_sum : total time spent for event processing
events_processed_duration_seconds_count : total number of processed events
invalid_events_received_counter : total number of received event with a not valid format. This can be caused, for example, by a not valid JSON representation or by the missing of mandatory fields
actions_received_counter : total number of received actions grouped by type
actions_processed_counter : total number of processed actions grouped by type and outcome (failure or success)
actions_processing_attempts_counter : total number of attempts to execute an action grouped by type and outcome (failure or success). This number can be greater than the total number of received actions because the execution of a single action can be attempted multiple times based on the defined Retry Strategy

Many other metrics can be derived from these ones, for example:

events_processed_counter - events_received_counter = events waiting to be processed
events_processed_duration_seconds_sum / events_processed_duration_seconds_count = mean processing time for an event

Tornado Backend API v1¶

The Tornado Backend contains endpoints that allow you to interact with Tornado through REST endpoints.

In this section we describe the version 1 of the Tornado Backend APIs.

Tornado ‘Auth’ Backend API¶

The ‘auth’ APIs require the caller to pass an authorization token in the headers in the format:

Authorization : Bearer TOKEN_HERE

The token should be a base64 encoded JSON with this user data:

{
  "user": "THE_USER_IDENTIFIER",
  "roles": ["ROLE_1", "ROLE_2", "ROLE_2"]
}

In the coming releases the current token format will be replaced by a JSON Web Token (JWT).

Tornado ‘Config’ Backend API¶

The ‘config’ APIs require the caller to pass an authorization token in the headers as in the ‘auth’ API.

Working with configuration and drafts

These endpoints allow working with the configuration and the drafts

Endpoint: get the current Tornado configuration

HTTP Method: GET
path : /api/v1_beta/config/current
response type: JSON

response example:

{
  "type": "Rules",
  "rules": [
    {
      "name": "all_emails",
      "description": "This matches all emails",
      "continue": true,
      "active": true,
      "constraint": {
        "WHERE": {
          "type": "AND",
          "operators": [
            {
              "type": "equal",
              "first": "${event.type}",
              "second": "email"
            }
          ]
        },
        "WITH": {}
      },
      "actions": [
        {
          "id": "Logger",
          "payload": {
            "subject": "${event.payload.subject}",
            "type": "${event.type}"
          }
        }
      ]
    }
  ]
}

Endpoint: get list of draft ids

HTTP Method: GET
path : /api/v1_beta/config/drafts
response type: JSON
response: An array of String ids
response example:
```
["id1", "id2"]
```

Endpoint: get a draft by id

HTTP Method: GET
path : /api/v1_beta/config/drafts/{draft_id}
response type: JSON
response: the draft content

response example:

{
  "type": "Rules",
  "rules": [
    {
      "name": "all_emails",
      "description": "This matches all emails",
      "continue": true,
      "active": true,
      "constraint": {
        "WHERE": {},
        "WITH": {}
      },
      "actions": []
    }
  ]
}

Endpoint: create a new draft and return the draft id. The new draft is an exact copy of the current configuration; anyway, a root Filter node is added if not present.

HTTP Method: POST
path : /api/v1_beta/config/drafts
response type: JSON
response: the draft content
response example:
```
{
  "id": "id3"
}
```

Endpoint: update an existing draft

HTTP Method: PUT
path : /api/v1_beta/config/drafts/{draft_id}
request body type: JSON
request body: The draft content in the same JSON format returned by the GET /api/v1_beta/config/drafts/{draft_id} endpoint
response type: JSON
response: an empty json object

Endpoint: delete an existing draft

HTTP Method: DELETE
path : /api/v1_beta/config/drafts/{draft_id}
response type: JSON
response: an empty json object

Endpoint: take over an existing draft

HTTP Method: POST
path : /api/v1_beta/config/drafts/{draft_id}/take_over
response type: JSON
response: an empty json object

Endpoint: deploy an existing draft

HTTP Method: POST
path : /api/v1_beta/config/drafts/{draft_id}/deploy
response type: JSON
response: an empty json object

Tornado ‘Event’ Backend API¶

Send Test Event Endpoint

Endpoint: match an event on the current Tornado Engine configuration

HTTP Method: POST
path : /api/v1_beta/event/current/send
request type: JSON
request example:
```
{
    "event": {
      "type": "the_event_type",
      "created_ms": 123456,
      "payload": {
        "value_one": "something",
        "value_two": "something_else"
      }
    },
    "process_type": "SkipActions"
}
```
Where the event has the following structure:
- type: The Event type identifier
- created_ms: The Event creation timestamp in milliseconds since January 1, 1970 UTC
- payload: A Map<String, Value> with event-specific data
- process_type: Can be Full or SkipActions:
  - Full: The event is processed and linked actions are executed
  - SkipActions: The event is processed but actions are not executed
response type: JSON

response example:

{
 "event": {
   "type": "the_event_type",
   "created_ms": 123456,
   "payload": {
     "value_one": "something",
     "value_two": "something_else"
   }
 },
 "result": {
   "type": "Rules",
   "rules": {
     "rules": {
       "emails_with_temperature": {
         "rule_name": "emails",
         "status": "NotMatched",
         "actions": [],
         "message": null
       },
       "archive_all": {
         "rule_name": "archive_all",
         "status": "Matched",
         "actions": [
           {
             "id": "archive",
             "payload": {
               "archive_type": "one",
               "event": {
                 "created_ms": 123456,
                 "payload": {
                   "value_one": "something",
                   "value_two": "something_else"
                 },
                 "type": "the_event_type"
               }
             }
           }
         ],
         "message": null
       }
     },
     "extracted_vars": {}
   }
 }
}

Endpoint: match an event on a specific Tornado draft

HTTP Method: POST
path : /api/v1_beta/event/drafts/{draft_id}/send
request type: JSON
request/response example: same request and response of the /api/v1_beta/event/current/send endpoint

Tornado ‘RuntimeConfig’ Backend API¶

These endpoints allow inspecting and changing the tornado configuration at runtime. Please note that whatever configuration change performed with these endpoints will be lost when tornado is restarted.

Get the logger configuration

Endpoint: get the current logger level configuration

HTTP Method: GET
path: /api/v1_beta/runtime_config/logger
response type: JSON

response example:

{
  "level": "info",
  "stdout_enabled": true,
  "apm_enabled": false
}

Set the logger level

Endpoint: set the current logger level configuration

HTTP Method: POST
path: /api/v1_beta/runtime_config/logger/level
response: http status code 200 if the request was performed correctly
request body type: JSON
request body:
```
{
  "level": "warn,
  tornado=trace"
}
```

Set the logger stdout output

Endpoint: Enable or disable the logger stdout output

HTTP Method: POST
path: /api/v1_beta/runtime_config/logger/stdout
response: http status code 200 if the request was performed correctly
request body type: JSON
request body:
```
{
  "enabled": true
}
```

Set the logger output to Elastic APM

Endpoint: Enable or disable the logger output to Elastic APM

HTTP Method: POST
path: /api/v1_beta/runtime_config/logger/apm
response: http status code 200 if the request was performed correctly
request body type: JSON
request body:
```
{
  "enabled": true
}
```

Set the logger configuration with priority to Elastic APM

Endpoint: This will disable the stdout and enable the Elastic APM logger; in addition, the logger level will be set to the one provided, or to “info,tornado=debug” if not present.

HTTP Method: POST
path: /api/v1_beta/runtime_config/logger/set_apm_priority_configuration
response: http status code 200 if the request was performed correctly
request body type: JSON
request body:

{
  "logger_level": true
}

Set the logger configuration with priority to stdout

Endpoint: this will disable the Elastic APM logger and enable the stdout; in addition, the logger level will be set to the one provided in the configuration file.

HTTP Method: POST
path: /api/v1_beta/runtime_config/logger/set_stdout_priority_configuration
response: http status code 200 if the request was performed correctly
request body type: JSON
request body:
```
{}
```

Set the Smart Monitoring Executor status

Endpoint: this activates and deactivates the Smart Monitoring Executor. When the Executor is in active state, it will execute incoming Actions as soon as they are produced. When instead the Executor is in inactive state, all incoming Actions will be queued until the Executor returns in active state.

HTTP Method: POST
path: /api/v1_beta/runtime_config/executor/smart_monitoring
response: http status code 200 if the request was performed correctly
request body type: JSON
request body:
```
{
   "active": false
}
```

Tornado Backend APIv2¶

The version 2 of the APIs allows for a granular querying of the processing tree, reducing the end-to-end latency with respect to the version 1 of the APIs.

The Tornado Backend APIs v2 furthermore allow for granular authorization on the processing tree for different users.

Tornado ‘Auth’ Backend APIv2¶

The version 2 of the Tornado APIs require the caller to pass an authorization token in the headers in the format:

Authorization : Bearer TOKEN_HERE

The token should be a base64 encoded JSON with the following user data:

{
  "user": "THE_USER_IDENTIFIER",
  "auths": {
    "PARAM_AUTH_1": {
      "path": [
        "root"
      ],
      "roles": [
        "view",
        "edit",
        "test_event_execute_actions"
      ]
    },
    "PARAM_AUTH_2": {
      "path": [
        "root",
        "filter1",
        "filter2"
      ],
      "roles": [
        "view",
        "test_event_execute_actions"
      ]
    }
  },
  "preferences": {
    "language": "en_US"
  }
}

Tornado ‘Config’ Backend APIv2¶

The ‘config’ APIs require the caller to pass an authorization token in the headers as in the ‘auth’ API.

Reading the current configuration

These endpoints allow to read the current configuration tree

Endpoint: get the current configuration tree of the root node.

HTTP Method: GET
path : /api/v2_beta/config/active/tree/children/{param_auth}
- where {param_auth} is the key of one of the auth fields contained in the authentication header
response type: JSON
request permissions:
- To call this endpoint you need the ConfigView permission.

response example:

[
  {
    "type": "Filter",
    "name": "root",
    "rules_count": 60,
    "description": "This is the root node",
    "children_count": 2
  }
]

Endpoint: get the current configuration tree of a specific node. Node names must be separated by a comma.

HTTP Method: GET
path : /api/v2_beta/config/active/tree/children/{param_auth}/root,foo
- where {param_auth} is the key of one of the auth fields contained in the authentication header
response type: JSON
request permissions:
- To call this endpoint you need the ConfigView permission.

response example:

[
  {
    "type": "Filter",
    "name": "foo",
    "rules_count": 40,
    "description": "This is the foo node",
    "children_count": 4
  }
]

Tornado ‘Rule Details’ Backend APIv2¶

Reading a rule details

Endpoint: get single rule details giving the rule name and the ruleset path related to the current configuration tree. The ‘rule details’ APIs require the caller to pass an authorization token in the headers as in the ‘auth’ API.

HTTP Method: GET
path: /api/v2_beta/config/active/rule/details/admins/root,foo,rulesetA/foobar_rule
response type: JSON

response example:

{
    "name":"foobar_rule",
    "description":"foobar_rule description",
    "continue":true,
    "active":true,
    "constraint":{
        "type": "AND",
        "operators": [
            {
              "type": "equal",
              "first": "${event.type}",
              "second": "email"
            }
        ],
        "WITH":{}
    },
    "actions":[
        {
            "id": "Logger",
            "payload": {
                "subject": "${event.payload.subject}",
                "type": "${event.type}"
            }
        }
    ]
}

Tornado ‘Node Details’ Backend APIv2¶

Reading the current configuration details

Endpoint: The ‘node details’ APIs require the caller to pass an authorization token in the headers as in the ‘auth’ API.

HTTP Method: GET
path : /api/v2_beta/config/active/tree/details/{param_auth}/root,foo
- where {param_auth} is the key of one of the auth fields contained in the authentication header
response type: JSON
request permissions:
- To call this endpoint you need the ConfigView permission.

response example:

 {
    "type":"Filter",
    "name":"foo",
    "description":"This filter allows events for Linux hosts",
    "active":true,
    "filter":{
        "type":"equals",
        "first":"${event.metadata.os}",
        "second":"linux"
    }
}

Tornado ‘Event’ Backend APIv2¶

Send Test Event Endpoint V2

Endpoint: match an event on the current Tornado Engine configuration

HTTP Method: POST
path : /api/v2_beta/event/active/{param_auth}
- where {param_auth} is the key of one of the auth fields contained in the authentication header
request type: JSON
request permissions:
- To call this endpoint with a process_type equal to SkipActions you need at least one of the following permissions:
  - ConfigView
  - ConfigEdit
- To call this endpoint with a process_type equal to Full you need the TestEventExecuteActions permission in addition to the permissions needed for the SkipActions process_type.
request example:
```
{
    "event": {
      "type": "the_event_type",
      "created_ms": 123456,
      "payload": {
        "value_one": "something",
        "value_two": "something_else"
      }
    },
    "process_type": "SkipActions"
}
```
Where the event has the following structure:
- type: The Event type identifier
- created_ms: The Event creation timestamp in milliseconds since January 1, 1970 UTC
- payload: A Map<String, Value> with event-specific data
- process_type: Can be Full or SkipActions:
  - Full: The event is processed and linked actions are executed
  - SkipActions: The event is processed but actions are not executed
response type: JSON

response example:

{
 "event": {
   "type": "the_event_type",
   "created_ms": 123456,
   "payload": {
     "value_one": "something",
     "value_two": "something_else"
   }
 },
 "result": {
   "type": "Rules",
   "rules": {
     "rules": {
       "emails_with_temperature": {
         "rule_name": "emails",
         "status": "NotMatched",
         "actions": [],
         "message": null
       },
       "archive_all": {
         "rule_name": "archive_all",
         "status": "Matched",
         "actions": [
           {
             "id": "archive",
             "payload": {
               "archive_type": "one",
               "event": {
                 "created_ms": 123456,
                 "payload": {
                   "value_one": "something",
                   "value_two": "something_else"
                 },
                 "type": "the_event_type"
               }
             }
           }
         ],
         "message": null
       }
     },
     "extracted_vars": {}
   }
 }
}

Endpoint: match an event on a specific Tornado draft

HTTP Method: POST
path : /api/v2_beta/event/drafts/{draft_id}/{param_auth}
request type: JSON
request permissions:
- To call this endpoint with a process_type equal to SkipActions you need to be the owner of the draft and have the ConfigEdit permission.
- To call this endpoint with a process_type equal to Full you need the TestEventExecuteActions permission in addition to the permissions needed for the SkipActions process_type.
request/response example: same request and response of the /api/v2_beta/event/active/{param_auth} endpoint

Tornado ‘Tree Info’ Backend APIv2¶

Endpoint: The ‘tree info’ APIs require the caller to pass an authorization token in the headers as in the ‘auth’ API.

HTTP Method: GET
path : /api/v2_beta/config/active/tree/info/{param_auth}
- where {param_auth} is the key of one of the auth fields contained in the authentication header
response type: JSON
request permissions:
- To call this endpoint you need the ConfigView permission.

response example:

{
   "rules_count": 14,
   "filters_count": 2
}

Other¶

Advanced Filters and Operators¶

This section contains advanced filter operators that can be used in Tornado rules, together with their definition, use cases and sample code snippets. The list of operators is:

contains operator
containsIgnoreCase
comparison operators: equals, ge (equal to or greater than), gt (greater than), le (equal to or lower than), lt (lower than), and ne (not equal).
equalsIgnoreCase
regex
AND, OR, NOT operators
matchAll
the WITH clause, including
- a detailed introduction to the WITH clause
- Post-modifiers for the WITH clause.

The contains Operator

The contains operator (or it alias contain is used to check whether the first argument contains the second one.

It applies in three different situations:

The arguments are both strings: Returns true if the second string is a substring of the first one.
The first argument is an array: Returns true if the second argument is contained in the array.
The first argument is a map and the second is a string: Returns true if the second argument is an existing key in the map.

In any other case, it will return false.

Rule example:

{
  "description": "",
  "continue": true,
  "active": true,
  "constraint": {
    "WHERE": {
      "type": "contains",
      "first": "${event.payload.hostname}",
      "second": "linux"
    },
    "WITH": {}
  },
  "actions": []
}

An event matches this rule if in its payload appears an entry with key hostname and whose value is a string that contains linux.

A matching Event is:

{
    "type": "trap",
    "created_ms": 1554130814854,
    "payload":{
        "hostname": "linux-server-01"
    }
}

The ‘containsIgnoreCase’ Operator

The containsIgnoreCase operator is used to check whether the first argument contains the string passed as second argument, regardless of their capital and small letters. In other words, the arguments are compared in a case-insensitive way.

It applies in three different situations:

The arguments are both strings: Returns true if the second string is a case-insensitive substring of the first one
The first argument is an array: Returns true if the array passed as first parameter contains a (string) element which is equal to the string passed as second argument, regardless of uppercase and lowercase letters
The first argument is a map: Returns true if the second argument contains, an existing, case-insensitive, key of the map

In any other case, this operator will return false.

Rule example:

{
  "description": "",
  "continue": true,
  "active": true,
  "constraint": {
    "WHERE": {
      "type": "containsIgnoreCase",
      "first": "${event.payload.hostname}",
      "second": "Linux"
    },
    "WITH": {}
  },
  "actions": []
}

An event matches this rule if in its payload it has an entry with key “hostname” and whose value is a string that contains “linux”, ignoring the case of the strings.

A matching Event is:

{
    "type": "trap",
    "created_ms": 1554130814854,
    "payload":{
        "hostname": "LINUX-server-01"
    }
}

Additional values for hostname that match the rule include: linuX-SERVER-02, LInux-Host-12, Old-LiNuX-FileServer, and so on.

The ‘equals’, ‘ge’, ‘gt’, ‘le’, ‘lt’ and ‘ne’ Operators

The equals, ge, gt, le, lt, ne operators are used to compare two values.

All these operators can work with values of type Number, String, Bool, null and Array.

Warning

Please be extremely careful when using these operators with numbers of type float. The representation of floating point numbers is often slightly imprecise and can lead to unexpected results (for example, see https://www.floating-point-gui.de/errors/comparison/ ).

Example:

{
  "description": "",
  "continue": true,
  "active": true,
  "constraint": {
    "WHERE": {
      "type": "OR",
      "operators": [
        {
          "type": "equals",
          "first": "${event.payload.value}",
          "second": 1000
        },
        {
          "type": "AND",
          "operators": [
            {
              "type": "ge",
              "first": "${event.payload.value}",
              "second": 100
            },
            {
              "type": "le",
              "first": "${event.payload.value}",
              "second": 200
            },
            {
              "type": "ne",
              "first": "${event.payload.value}",
              "second": 150
            },
            {
              "type": "notEquals",
              "first": "${event.payload.value}",
              "second": 160
            }
          ]
        },
        {
          "type": "lt",
          "first": "${event.payload.value}",
          "second": 0
        },
        {
          "type": "gt",
          "first": "${event.payload.value}",
          "second": 2000
        }
      ]
    },
    "WITH": {}
  },
  "actions": []
}

An event matches this rule if event.payload.value exists and one or more of the following conditions hold:

It is equal to 1000
It is between 100 (inclusive) and 200 (inclusive), but not equal to 150 or to 160
It is less than 0 (exclusive)
It is greater than 2000 (exclusive)

A matching Event is:

{
    "type": "email",
    "created_ms": 1554130814854,
    "payload":{
      "value": 110
    }
}

Here are some examples showing how these operators behave:

[{"id":557}, {"one":"two"}] lt 3: false (cannot compare different types, e.g. here the first is an array and the second is a number)
{id: "one"} lt {id: "two"}: false (maps cannot be compared)
[["id",557], ["one"]] gt [["id",555], ["two"]]: true (elements in the array are compared recursively from left to right: so here “id” is first compared to “id”, then 557 to 555, returning true before attempting to match “one” and “two”)
[["id",557]] gt [["id",555], ["two"]]: true (elements are compared even if the length of the arrays is not the same)
true gt false: true (the value ‘true’ is evaluated as 1, and the value ‘false’ as 0; consequently, the expression is equivalent to “1 gt 0” which is true)
“twelve” gt “two”: false (strings are compared lexically, and ‘e’ comes before ‘o’, not after it)

The ‘equalsIgnoreCase’ Operator

The equalsIgnoreCase operator is used to check whether the strings passed as arguments are equal in a case-insensitive way.

It applies only if both the first and the second arguments are strings. In any other case, the operator will return false.

Rule example:

{
  "description": "",
  "continue": true,
  "active": true,
  "constraint": {
    "WHERE": {
      "type": "equalsIgnoreCase",
      "first": "${event.payload.hostname}",
      "second": "Linux"
    },
    "WITH": {}
  },
  "actions": []
}

An event matches this rule if in its payload it has an entry with key “hostname” and whose value is a string that is equal to “linux”, ignoring the case of the strings.

A matching Event is:

{
    "type": "trap",
    "created_ms": 1554130814854,
    "payload":{
        "hostname": "LINUX"
    }
}

The ‘regex’ Operator

The regex operator is used to check if a string matches a regular expression. The evaluation is performed with the Rust Regex library (see its github project home page )

Rule example:

{
  "description": "",
  "continue": true,
  "active": true,
  "constraint": {
    "WHERE": {
      "type": "regex",
      "regex": "[a-fA-F0-9]",
      "target": "${event.type}"
    },
    "WITH": {}
  },
  "actions": []
}

An event matches this rule if its type matches the regular expression [a-fA-F0-9].

A matching Event is:

{
    "type": "trap0",
    "created_ms": 1554130814854,
    "payload":{}
}

The ‘AND’, ‘OR’, and ‘NOT’ Operators

The and and or operators work on a set of operators, while the not operator works on one single operator. They can be nested recursively to define complex matching rules.

As you would expect:

The and operator evaluates to true if all inner operators match
The or operator evaluates to true if at least an inner operator matches
The not operator evaluates to true if the inner operator does not match, and evaluates to false if the inner operator matches

Example:

{
  "description": "",
  "continue": true,
  "active": true,
  "constraint": {
    "WHERE": {
      "type": "AND",
      "operators": [
        {
          "type": "equals",
          "first": "${event.type}",
          "second": "rsyslog"
        },
        {
          "type": "OR",
          "operators": [
            {
              "type": "equals",
              "first": "${event.payload.body}",
              "second": "something"
            },
            {
              "type": "equals",
              "first": "${event.payload.body}",
              "second": "other"
            }
          ]
        },
        {
          "type": "NOT",
          "operator": {
              "type": "equals",
              "first": "${event.payload.body}",
              "second": "forbidden"
          }
        }
      ]
    },
    "WITH": {}
  },
  "actions": []
}

An event matches this rule if in its payload:

The type is “rsyslog”
AND an entry with key body whose value is wither “something” OR “other”
AND an entry with key body is NOT “forbidden”

A matching Event is:

{
    "type": "rsyslog",
    "created_ms": 1554130814854,
    "payload":{
        "body": "other"
    }
}

A ‘Match all Events’ Rule

If the WHERE clause is not specified, the Rule evaluates to true for each incoming event.

For example, this Rule generates an “archive” Action for each Event:

{
    "description": "",
    "continue": true,
    "active": true,
    "constraint": {
      "WITH": {}
    },
    "actions": [
      {
        "id": "archive",
        "payload": {
          "event": "${event}",
          "archive_type": "one"
        }
      }
    ]
}

The ‘WITH’ Clause

The WITH clause generates variables extracted from the Event based on regular expressions. These variables can then be used to populate an Action payload.

All variables declared by a Rule must be resolved, or else the Rule will not be matched.

Two simple rules restrict the access and use of the extracted variables:

Because they are evaluated after the WHERE clause is parsed, any extracted variables declared inside the WITH clause are not accessible by the WHERE clause of the very same rule
A rule can use extracted variables declared by other rules, even in its WHERE clause, provided that:
- The two rules must belong to the same rule set
- The rule attempting to use those variables should be executed after the one that declares them
- The rule that declares the variables should also match the event

The syntax for accessing an extracted variable has the form:

_variables.[.RULE_NAME].VARIABLES_NAME

If the RULE_NAME is omitted, the current rule name is automatically selected.

Example:

{
  "description": "",
  "continue": true,
  "active": true,
  "constraint": {
    "WHERE": {
          "type": "equals",
          "first": "${event.type}",
          "second": "trap"
    },
    "WITH": {
      "sensor_description": {
        "from": "${event.payload.line_5}",
        "regex": {
          "match": "(.*)",
          "group_match_idx": 0
        }
      },
      "sensor_room": {
        "from": "${event.payload.line_6}",
        "regex": {
          "match": "(.*)",
          "group_match_idx": 0
        }
      }
    }
  },
  "actions": [
    {
      "id": "nagios",
      "payload": {
        "host": "bz-outsideserverroom-sensors",
        "service": "motion_sensor_port_4",
        "status": "Critical",
        "host_ip": "${event.payload.host_ip}",
        "room": "${_variables.sensor_room}",
        "message": "${_variables.sensor_description}"
      }
    }
  ]
}

This Rule matches only if its type is “trap” and it is possible to extract the two variables “sensor_description” and “sensor_room” defined in the WITH clause.

An Event that matches this Rule is:

{
  "type": "trap",
  "created_ms": 1554130814854,
  "payload":{
    "host_ip": "10.65.5.31",
    "line_1":  "netsensor-outside-serverroom.wp.lan",
    "line_2":  "UDP: [10.62.5.31]:161->[10.62.5.115]",
    "line_3":  "DISMAN-EVENT-MIB::sysUpTimeInstance 38:10:38:30.98",
    "line_4":  "SNMPv2-MIB::snmpTrapOID.0 SNMPv2-SMI::enterprises.14848.0.5",
    "line_5":  "SNMPv2-SMI::enterprises.14848.2.1.1.7.0 38:10:38:30.98",
    "line_6":  "SNMPv2-SMI::enterprises.14848.2.1.1.2.0 \"Outside Server Room\""
  }
}

It will generate this Action:

{
  "id": "nagios",
  "payload": {
    "host": "bz-outsideserverroom-sensors",
    "service": "motion_sensor_port_4",
    "status": "Critical",
    "host_ip": "10.65.5.31",
    "room": "SNMPv2-SMI::enterprises.14848.2.1.1.7.0 38:10:38:30.98",
    "message": "SNMPv2-SMI::enterprises.14848.2.1.1.2.0 \"Outside Server Room\""
  }
}

The ‘WITH’ Clause - Configuration details

As already seen in the previous section, the WITH clause generates variables extracted from the Event using regular expressions. There are multiple ways of configuring those regexes to obtain the desired result.

Common entries to all configurations:

from: An expression that determines to which value to apply the extractor regex;
modifiers_post: A list of String modifiers to post-process the extracted value. See following section for additional details.

In addition, three parameters combined will define the behavior of an extractor:

all_matches: whether the regex will loop through all the matches or only the first one will be considered. Accepted values are true and false. If omitted, it defaults to false
match, named_match or single_key_match: a string value representing the regex to be executed. In detail:
- match is used in case of an index-based regex,
- named_match is used when named groups are present.
- single_key_match is used to search in a map for a key that matches the regex. In case of a match, the extracted variable will be the value of the map associated with that key that matched the regex. This match will fail if more than one key matches the defined regex.
Note that all these values are mutually exclusive.
group_match_idx: valid only in case of an index-based regex. It is a positive numeric value that indicates which group of the match has to be extracted. If omitted, an array with all groups is returned.

To show how they work and what is the produced output, from now on, we’ll use this hypotetical email body as input:

A critical event has been received:

STATUS: CRITICAL HOSTNAME: MYVALUE2 SERVICENAME: MYVALUE3
STATUS: OK HOSTNAME: MYHOST SERVICENAME: MYVALUE41231

Our objective is to extract from it information about the host status and name, and the service name. We show how using different extractors leads to different results.

Option 1

{
  "WITH": {
      "server_info": {
        "from": "${event.payload.email.body}",
        "regex": {
          "all_matches": false,
          "match": "STATUS:\\s+(.*)\\s+HOSTNAME:\\s+(.*)SERVICENAME:\\s+(.*)",
          "group_match_idx": 1
        }
      }
  }
}

This extractor:

processes only the first match because all_matches is false
uses an index-based regex specified by match
returns the group of index 1

In this case the output will be the string “CRITICAL”.

Please note that, if the group_match_idx was 0, it would have returned “STATUS: CRITICAL HOSTNAME: MYVALUE2 SERVICENAME: MYVALUE3” as in any regex the group with index 0 always represents the full match.

Option 2

{
  "WITH": {
      "server_info": {
        "from": "${event.payload.email.body}",
        "regex": {
          "all_matches": false,
          "match": "STATUS:\\s+(.*)\\s+HOSTNAME:\\s+(.*)SERVICENAME:\\s+(.*)"
        }
      }
  }
}

This extractor:

processes only the first match because all_matches is false
uses an index-based regex specified by match
returns an array with all groups of the match because group_match_idx is omitted.

In this case the output will be an array of strings:

[
  "STATUS: CRITICAL HOSTNAME: MYVALUE2 SERVICENAME: MYVALUE3",
  "CRITICAL",
  "MYVALUE2",
  "MYVALUE3"
]

Option 3

{
  "WITH": {
      "server_info": {
        "from": "${event.payload.email.body}",
        "regex": {
          "all_matches": true,
          "match": "STATUS:\\s+(.*)\\s+HOSTNAME:\\s+(.*)SERVICENAME:\\s+(.*)",
          "group_match_idx": 2
        }
      }
  }
}

This extractor:

processes all matches because all_matches is true
uses an index-based regex specified by match
for each match, returns the group of index 2

In this case the output will be an array of strings:

[
  "MYVALUE2", <-- group of index 2 of the first match
  "MYHOST"    <-- group of index 2 of the second match
]

Option 4

{
  "WITH": {
      "server_info": {
        "from": "${event.payload.email.body}",
        "regex": {
          "all_matches": true,
          "match": "STATUS:\\s+(.*)\\s+HOSTNAME:\\s+(.*)SERVICENAME:\\s+(.*)"
        }
      }
  }
}

This extractor:

processes all matches because all_matches is true
uses an index-based regex specified by match
for each match, returns an array with all groups of the match because group_match_idx is omitted.

In this case the output will be an array of arrays of strings:

[
  [
    "STATUS: CRITICAL HOSTNAME: MYVALUE2 SERVICENAME: MYVALUE3",
    "CRITICAL",
    "MYVALUE2",
    "MYVALUE3"
  ],
  [
    "STATUS: OK HOSTNAME: MYHOST SERVICENAME: MYVALUE41231",
    "OK",
    "MYHOST",
    "MYVALUE41231"
  ]
]

The inner array, in position 0, contains all the groups of the first match while the one in position 1 contains the groups of the second match.

Option 5

{
  "WITH": {
      "server_info": {
        "from": "${event.payload.email.body}",
        "regex": {
          "named_match": "STATUS:\\s+(?P<STATUS>.*)\\s+HOSTNAME:\\s+(?P<HOSTNAME>.*)SERVICENAME:\\s+(?P<SERVICENAME>.*)"
        }
      }
  }
}

This extractor:

processes only the first match because all_matches is omitted
uses a regex with named groups specified by named_match

In this case the output is an object where the group names are the property keys:

{
  "STATUS": "CRITICAL",
  "HOSTNAME": "MYVALUE2",
  "SERVICENAME: "MYVALUE3"
}

Option 6

{
  "WITH": {
      "server_info": {
        "from": "${event.payload.email.body}",
        "regex": {
          "all_matches": true,
          "named_match": "STATUS:\\s+(?P<STATUS>.*)\\s+HOSTNAME:\\s+(?P<HOSTNAME>.*)SERVICENAME:\\s+(?P<SERVICENAME>.*)"
        }
      }
  }
 }

This extractor:

processes all matches because all_matches is true
uses a regex with named groups specified by named_match

In this case the output is an array that contains one object for each match:

[
  {
    "STATUS": "CRITICAL",
    "HOSTNAME": "MYVALUE2",
    "SERVICENAME: "MYVALUE3"
  },
  {
    "STATUS": "OK",
    "HOSTNAME": "MYHOST",
    "SERVICENAME: "MYVALUE41231"
  },
]

The ‘WITH’ Clause - Post Modifiers

The WITH clause can include a list of String modifiers to post-process the extracted value. The available modifiers are:

Lowercase: it converts the resulting String to lower case. Syntax:
```
{
    "type": "Lowercase"
}
```

Map: it maps a string to another string value. Syntax:

{
      "type": "Map",
      "mapping": {
        "Critical": "2",
        "Warning": "1",
        "Clear": "0",
        "Major": "2",
        "Minor": "1"
      },
      "default_value": "3"
}

The default_value is optional; when provided, it is used to map values that do not have a corresponding key in the mapping field. When not provided, the extractor will fail if a specific mapping is not found.

ReplaceAll: it returns a new string with all matches of a substring replaced by the new text; the find property is parsed as a regex if is_regex is true, otherwise it is evaluated as a static string. Syntax:
```
{
    "type": "ReplaceAll",
    "find": "the string to be found",
    "replace": "to be replaced with",
    "is_regex": false
}
```
In addition, when is_regex is true, is possible to interpolate the regex captured groups in the replace string, using the $<position> syntax, for example:
```
{
    "type": "ReplaceAll",
    "find": "(?P<lastname>[^,\\s]+),\\s+(?P<firstname>\\S+)",
    "replace": "firstname: $2, lastname: $1",
    "is_regex": true
}
```
Valid forms of the replace field are:
- extract from event: ${events.payload.hostname_ext}
- use named groups from regex: $digits and other
- use group positions from regex: $1 and other
ToNumber: it transforms the resulting String into a number. Syntax:
```
{
    "type": "ToNumber"
}
```
Trim: it trims the resulting String. Syntax:
```
{
    "type": "Trim"
}
```

A full example of a WITH clause using modifiers is:

{
  "WITH": {
      "server_info": {
       "from": "${event.payload.email.body}",
        "regex": {
          "all_matches": false,
          "match": "STATUS:\s+(.*)\s+HOSTNAME:\s+(.*)SERVICENAME:\s+(.*)",
          "group_match_idx": 1
        },
        "modifiers_post": [
            {
              "type": "Lowercase"
            },
            {
              "type": "ReplaceAll",
              "find": "to be found",
              "replace": "to be replaced with",
              "is_regex": false
            },
            {
              "type": "Trim"
            }
        ]
       }
     }
  }

This extractor has three modifiers that will be applied to the extracted value. The modifiers are applied in the order they are declared, so the extracted string will be transformed in lowercase, then some text replaced, and finally, the string will be trimmed.

Complete Rule Example 1

An example of a valid Rule in a JSON file is:

{
  "description": "This matches all emails containing a temperature measurement.",
  "continue": true,
  "active": true,
  "constraint": {
    "WHERE": {
      "type": "AND",
      "operators": [
        {
          "type": "equals",
          "first": "${event.type}",
          "second": "email"
        }
      ]
    },
    "WITH": {
      "temperature": {
        "from": "${event.payload.body}",
        "regex": {
          "match": "[0-9]+\\sDegrees",
          "group_match_idx": 0
        }
      }
    }
  },
  "actions": [
    {
      "id": "Logger",
      "payload": {
        "type": "${event.type}",
        "subject": "${event.payload.subject}",
        "temperature:": "The temperature is: ${_variables.temperature} degrees"
      }
    }
  ]
}

This creates a Rule with the following characteristics:

Its unique name is ‘emails_with_temperature’. There cannot be two rules with the same name.
An Event matches this Rule if, as specified in the WHERE clause, it has type “email”, and as requested by the WITH clause, it is possible to extract the “temperature” variable from the “event.payload.body” with a non-null value.
If an Event meets the previously stated requirements, the matcher produces an Action with id “Logger” and a payload with the three entries type, subject and temperature.

Matcher Engine Implementation Details¶

The Matcher crate contains the core logic of the Tornado Engine. It is in charge of: - Receiving events from the Collectors - Processing incoming events and detecting which Filter and Rule they satisfy - Triggering the expected actions

Due to its strategic position, its performance is of utmost importance for global throughput.

The code’s internal structure is kept simple on purpose, and the final objective is reached by splitting the global process into a set of modular, isolated and well-tested blocks of logic. Each “block” communicates with the others through a well-defined API, which at the same time hides its internal implementation.

This modularization effort is twofold; first, it minimizes the risk that local changes will have a global impact; and second, it separates functional from technical complexity, so that increasing functional complexity does not result in increasing code complexity. As a consequence, the maintenance and evolutionary costs of the code base are expected to be linear in the short, mid- and long term.

At a very high level view, when the matcher initializes, it follows these steps:

Configuration (see the code in the “config” module): The configuration phase loads a set of files from the file system. Each file is a Filter or a Rule in JSON format. The outcome of this step is a processing tree composed of Filter and Rule configurations created from the JSON files.
Validation (see the code in the “validator” module): The Validator receives the processing tree configuration and verifies that all nodes respect a set of predefined constraints (e.g., the identifiers cannot contain dots). The output is either the same processing tree as the input, or else an error.
Match Preparation (see the code in the “matcher” module): The Matcher receives the processing tree configuration, and for each node:
- if the node is a Filter:
  - Builds the Accessors for accessing the event properties using the AccessorBuilder (see the code in the “accessor” module).
  - Builds an Operator for evaluating whether an event matches the Filter itself (using the OperatorBuilder, code in the “operator” module).
- if the node is a rule:
  - Builds the Accessors for accessing the event properties using the AccessorBuilder (see the code in the “accessor” module).
  - Builds the Operator for evaluating whether an event matches the “WHERE” clause of the rule (using the OperatorBuilder, code in the “operator” module).
  - Builds the Extractors for generating the user-defined variables using the ExtractorBuilder (see the code in the “extractor” module). This step’s output is an instance of the Matcher that contains all the required logic to process an event against all the defined rules. A matcher is stateless and thread-safe, thus a single instance can be used to serve the entire application load.
Listening: Listen for incoming events and then match them against the stored Filters and Rules.

Tornado Monitoring and Statistics¶

Tornado Engine performance metrics are exposed via Tornado APIs and periodically collected by a dedicated telegraf instance telegraf_tornado_monitoring.service. Metrics are stored into the database master_tornado_monitoring in InfluxDB.

Tornado Monitoring and Statistics gives an insight about what data Tornado is processing and how. These information can be useful in several scenarios, including workload inspection, identification of bottlenecks, and issue debugging. A common use case is to identify performance-related issues: for example a difference between the amount of events received and events processed by Tornado may identify a performance problem because Tornado does not have enough resources to handle the current workload.

Examples of collected metrics are:

events_processed_counter: total amount of event processed by Tornado Engine
events_received_counter: total amount of events received by Tornado Engine through all Collectors
actions_processed_counter: total amount of actions executed by Tornado Engine

Metrics will be automatically deleted according to the selected retention policy.

The user can configure Tornado Monitoring and Statistics via GUI under Configuration / Modules / tornadocarbon / Configuration. Two parameters are available:

Tornado Monitoring Retention Policy: defines the number of days for which metrics are retained in InfluxDB and defaults to 7 days, after which data will be no longer available.
Tornado Monitoring Polling Interval: sets how often the Collector queries the Tornado APIs to gather metrics and defaults to 5 seconds.

To apply changes you have to either run neteye_secure_install for both options or execute /usr/share/neteye/tornadocarbon/scripts/apply_tornado_monitoring_retention_policy.sh or /usr/share/neteye/tornadocarbon/scripts/apply_tornado_monitoring_polling_interval.sh, according to the parameter changed.

Note

On a NetEye Cluster, execute the command on the node where icingaweb2 is active.

Tornado Engine (Executable)¶

This crate contains the Tornado Engine executable code, which is a configuration of the Engine based on actix and built as a portable executable.

Structure of Tornado Engine

This specific Tornado Engine executable is composed of the following components:

A JSON Collector
The Engine
The Archive Executor
The Elasticsearch Executor
The Foreach Executor
The Icinga 2 Executor
The Director Executor
The Monitoring Executor
The Logger Executor
The Script Executor
The Smart Monitoring Executor

Each component is wrapped in a dedicated actix actor.

This configuration is only one of many possible configurations. Each component has been developed as an independent library, allowing for greater flexibility in deciding whether and how to use it.

At the same time, there are no restrictions that force the use of the components into the same executable. While this is the simplest way to assemble them into a working product, the Collectors and Executors could reside in their own executables and communicate with the Tornado Engine via a remote call. This can be achieved either through a direct TCP or HTTP call, with an RPC technology (e.g., Protobuf, Flatbuffer, or CAP’n’proto), or with a message queue system (e.g., Nats.io or Kafka) in the middle for deploying it as a distributed system.