Advanced Topics¶

In this section we deal with some advanced topic, including passive monitoring, jobs, API, and Icinga Retention Policy.

Passive monitoring¶

When Tornado is installed, NetEye creates an Icinga Web 2 user neteye-tornado and an associated Icinga Web 2 role neteye_tornado_director_apis, which only gives access to the module Director, with limited authorizations on the actions that Tornado can perform.

Warning

These user and permission are required by the backend, for Tornado to call the Director API–and in particular for the authentication and authorization of the Tornado Director Executor to the APIs of the Icinga Director. Therefore neither the user, nor the associated role must be removed from the system.

In case you need it, for example to reconfigure the Tornado Director Executor, the password for the user neteye-tornado is stored in the file /root/.pwd_icingaweb2_neteye_tornado.

Processing Tree¶

Within the NetEye interface you can view the Tornado rule configuration graphically instead of in a command shell. The Configuration Viewer (available when click on Tornado in the left side menu) shows the processing tree in a top-down format, allowing you to verify that the structure of the rules you have created is as you intended.

Definitions¶

While a more complete description of all Tornado elements is available in the official documentation, they are summarized in enough detail here so that you can understanding this page without reading the full official documentation.

Filter: A node in the processing tree that contains (1) a filter definition and (2) a set of child nodes, each of which corresponds to a condition in the filter definition. You can use a Filter to create independent pipelines for different types of events, reducing the amount of time needed to process them.
Implicit Filter: If a filter is not present in a node, a default filter is created which forwards an event to ALL child nodes, rather than a particular one that matched a filter condition.
Rule Set: A leaf node that contains multiple rules to be matched against the event one by one once the filters in the parent nodes have let an event through.
Rule: A single rule within a rule set, which can be matched against an event.
Processing Tree: The entire set of filters and rules that creates the hierarchical structure where events are filtered and then matched against one or more rules.

Basic Configuration Information¶

The location of configuration files in the file system is pre-configured in NetEye. NetEye automatically starts Tornado as follows:

Reads the configuration from the /neteye/shared/tornado/conf/ directory
Starts the Tornado Engine
Searches for Filter and Rule definitions in /neteye/shared/tornado/conf/rules.d/

The structure of this last directory reflects the Processing Tree structure. Each subdirectory can contain either:

A Filter and a set of sub directories corresponding to the Filter’s children
A Rule Set

Each individual Rule or Filter to be included in the processing tree must be in its own file, in JSON format (Tornado will ignore all other file types). For instance, consider this directory structure:

/tornado/config/rules
                 |- node_0
                 |    |- 0001_rule_one.json
                 |    \- 0010_rule_two.json
                 |- node_1
                 |    |- inner_node
                 |    |    \- 0001_rule_one.json
                 |    \- filter_two.json
                 \- filter_one.json

In this example, the processing tree is organized as follows:

The root node is a filter named “filter_one”.
The filter filter_one* has two child nodes: node_0 and node_1.
- node_0 is a rule set that contains two rules called rule_one and rule_two, with an implicit filter that forwards all incoming events to both of its child rules.
- node_1 is a filter with a single child named “inner_node”. Its filter filter_two determines which incoming events are passed to its child node.
inner_node is a rule set with a single rule called rule_one.

Within a rule set, the alphanumeric order of the file names determines the execution order. The rule filename is composed of two parts separated by the first ‘*’ (underscore) symbol. The first part determines the rule execution order, and the second is the rule name. For example:

0001_rule_one.json -> 0001 determines the execution order, “rule_one” is the rule name
0010_rule_two.json -> 0010 determines the execution order, “rule_two” is the rule name

Rule names must be unique within their own rule set. There are no constraints on rule names in different rule sets.

Similar to what happens for Rules, Filter names are also derived from the filenames. However, in this case, the entire filename corresponds to the Filter name.

In the example above, the “filter_one” node is the entry point of the processing tree. When an Event arrives, the Matcher will evaluate whether it matches the filter condition, and will pass the Event to one (or more) of the filter’s children. Otherwise it will ignore it.

A node’s children are processed independently. Thus node_0 and node_1 will be processed in isolation and each of them will be unaware of the existence and outcome of the other. This process logic is applied recursively to every node.

Structure of a Filter¶

A Filter contains these properties, defined in its JSON file:

description: A string value providing a high-level description of the filter.
active: A boolean value; if false, the filter’s children will be ignored.
filter: An operator that, when applied to an event, returns true or false. The result determines how an Event will be processed by the filter’s inner nodes.

When the configuration is read from the file system, the filter name is automatically inferred from the filename by removing its ‘.json’ extension. It can be composed only of letters, numbers and the “_” (underscore) character.

Structure of a Rule Set¶

A Rule Set is simply a list of rules.

Interface Overview¶

The Graphical User Interface allows you to explore the current configuration of your Tornado Instance. The GUI has two main views. The first one is the Processing Tree View; this one allows you to examine the entire configuration processing tree of Tornado and to modify the configuration–please refer to the next section for more information. The second view, the Ruleset View, can be reached by clicking on whatever node is displayed in the Processing Tree View; from here, you can access the details of each Rule, send Events to Tornado, inspect the outcome of the Event execution, and modify the rules. In this view the information is organized into a table where each row represents a specific Rule.

In the Ruleset View, an Event Test panel is available to send simulated Events. These Events, can be created through a dedicated form and are composed by the following four fields:

event type: the type of the Event, such as trap, sms, email, etc.
creation time*: the Event timestamp defined as an epoch in milliseconds
enable execution of actions: whether the actions of matching rules have to be executed or skipped
payload: the event payload in JSON format

When a test is executed by clicking the “Run Test” button, the linked Event is sent to Tornado and the outcome of the operation will be reported in the rule table.

At this point, a rule can be in one of the following states:

matched: If a rule matched the Event
stopped: If a rule matched the Event and then stopped the execution flow. This happens if the continue flag of the rule is set to false
partially matched: If the where condition of the Rule was matched but it was not possible to process the required extracted variables
not matched: If the Rule did not match the Event

For each rule in the table, the extracted variables and the generated Action payloads are shown. In addition, all these extracted variables are also shown in the Event Test form.

Two other buttons are visible, one for cleaning all the fields of the form and one for cleaning the outcome of the test.

Tornado Processing Tree Editor¶

The Tornado GUI provides an edit mode that allows to modify the configuration of the Tornado rules’ processing tree directly from NetEye’s front-end. Rules must be written in JSON and the editor features a validator, that helps you checking that the rule is syntactically correct. If you are not acquainted with JSON, you can check one of the following tutorials:

Two important principles have been used for the development of the edit mode and must be understood and taken into account when modifying Tornado’s configuration, especially because these differ from other modules in Icinga Director:

Implicit Lock Mode. Only one user at a time can modify the processing tree configuration. This prevents multiple users from changing the configuration simultaneously, which might lead to unwanted results and possibly to Tornado not working correctly due to incomplete or wrong configuration. When a user is editing the configuration, the actual, running configuration is left untouched: it continues to be operative and accepts incoming data to be processed.
Edit Mode. When starting to modify the configuration, Tornado will continue to work with the existing configuration–thanks to the implicit lock mode, while the new changes are saved in a separate draft configuration. The new configuration then must be deployed to become operative.

Note

If the user logs out without deploying the draft, the next user that will log in–and start modifying tornado’s configuration–will have different possibilities: to check all the changes present in the draft by clicking on the Show Changes button and to continue editing and then deploying the draft, or to discard completely the existent draft and star editing a new one. Once the user deployed the changes, the action will be recorded in the Auditlog module (System > Auditlog). Here a detailed diff of what has been changed, together with the user who deployed those changes and the timestamp is displayed.

This mode has other positive side effects: one does not need to complete the changes in one session, but can stop and then continue at a later point; another user can pick up the draft and complete it; in case of a disaster (like e.g., the abrupt end of the HTTPS connection to the GUI) it is possible to resume the draft from the point where it was left.

Warning

Only one draft at a time is allowed; that is, editing of multiple draft is not supported!

When a user enters the edit mode, a new draft is created on the fly if none is present, which will be an exact copy of the running Tornado configuration. If not present in the draft, a root node of type Filter will be automatically added to the draft.

To check for the correctness of a Draft, without impacting the deployed configuration, it is possible to open the test window also while in Edit Mode. The event will be processed using the Draft and the result will be displayed, while keeping the existing configuration running.

In order to modify the processing tree, you can add nodes to each level of the processing tree, as well as inspecting and modifying each single rule. Moreover, from the main view it is also possible to add new filters or rulesets to the processing tree by clicking on the buttons on the right-hand side of the GUI.

In more details, the processing tree is shown in the main area of the GUI. The top filter node (nodes are containers of rules, please check the Definition in previous section for details) is the root one and therefore the first to be processed. Additional levels below contain further blocks with additional nodes, to allow for multiple processing of the events. Depending on the Edit mode being active or not, different interactions are possible with the processing tree.

When the Edit mode is disabled, the elements composing the Processing tree (Filters and rulesets) are shown in hierarchy. When clicking on a filter, its children–which can be other filters or rulesets–are shown; while a click on a ruleset will open the Rule Editor, to allow editing of the rules–refer to the next section for more information.

When clicking on Edit mode, it is possible to add new blocks in the processing tree or move them in other level. Using the buttons on the right-hand side of the Editor, new rulesets or filters can be added and placed within the hierarchy. For each node. it is possible to define:

a name and a description
its place in the hierarchy, by providing the name of the parent node

For a filter, these two more options are available.

whether it is active or not
the filter that should match the event. Syntax for the filter is JSON-based, and examples can be found in the various How-tos present in the tornado section of the User Guide.

Moreover, in Edit mode, in each box tree dots appear that when clicked, will open a small menu with two or three icons at the bottom of the box: Option are to edit or delete the ruleset or the filter, with the additional option, for ruleset only, to list rules:a click on the icon will open the rule GUI for editing the single rule of the ruleset.

Tornado Rule Editor¶

The tornado Rule Editor allows to manage the single rules that are in a ruleset. On top of the window, all the parents of the current ruleset are shown, to allow you to quickly check on which leaf of the processing tree the rules shown are located. Like in the processing tree editor, a JSON validator assists you in checking that the syntax of the rules is correct.

In the main area, all the defined rules are shown, together with a number of information about them: name, action, and status (enabled or not).

Like in the Processing Tree Editor, available options differ depending whether the Edit mode is active or not:

When the Edit mode is not active, it is possible to click on the Open test window button on the top right-hand side of the window to check which events the current rule selection would match.

With active Edit mode, the Open test window button is disabled, but new rules can be added, modified, or deleted; each rule can also can be moved along the list with a simple drag and drop.

Import and Export Configuration¶

The Tornado GUI provides multiple ways to import and export the whole configuration or just a subset of it.

Export Configuration¶

You have three possibilities to export Tornado configuration or part of it:

entire configuration: select the root node from the Processing Tree View and click on the export button to download the entire configuration
a node (either a ruleset or a filter): select the node from the Processing Tree View and click on the export button to download the node and its sub-nodes
a single rule: navigate to the rules table, select a rule, and click on the export button

Hint

You can backup and download Tornado configuration by exporting the entire configuration.

Import Configuration¶

You can use the import feature to upload to NetEye a previously downloaded configuration, new custom rules, or even the configuration from another NetEye instance.

When clicking on the import button a popup will appear with the following fields:

Node File: the file containing the configuration

Note

When importing a single rule the field will be labeled as Rule File.
Replace whole configuration?: If selected, the imported configuration will replace the root node and all of its sub-nodes.

Hint

You can restore a previous Tornado configuration by selecting this option.
Parent Node: The parent node where to add the imported configuration, by default it is set to the currently selected node.

Note

When a node or a rule with the same name of an already existing one is imported, the name of the new node/rule will be suffixed with _imported.

Jobs¶

The Icinga Director Background Daemon is responsible for running Jobs accoring our schedule. Director allows you to schedule eventually long-running tasks so that they can run in the background.

Currently this includes:

Import runs
Sync runs
Housekeeping tasks
Config rendering and deployment

This component is internally provided as a Hook. This allows other Icinga Web 2 modules to benefit from the Job Runner by providing their very own Job implementations.

Theory of operation¶

Jobs are configured via the Web frontend. You can create multiple definitions for the very same Job. Every single job will run with a configurable interval. Please do not expect this to behave like a scheduler or a cron daemon. Jobs are currently not executed in parallel. Therefore if one job takes longer, it might have an influence on the scheduling of other jobs.

Some of you might want actions like automated config deployment not to be executed all around the clock. That’s why you have the possibility to assign time periods to your jobs. Choose an Icinga timeperiod, the job will only be executed within that period.

Time periods¶

Icinga time periods can get pretty complex. You configure them with Director, but until now it didn’t have the necessity to “understand” them. This of course changed with Time Period support in our Job Runner. Director will try to fully “understand” periods in future, but right now it is only capable to interpret a limited subset of timeperiod range definitions.

API¶

Large parts of the Director’s functionality are also available as API on your CLI.

Manage Objects¶

Use icingacli director <type> <action> show, create modify or delete Icinga objects of a specific type:

Action	Description
`create`	Create a new object
`delete`	Delete a specific object
`exists`	Whether a specific object exists
`set`	Modify an existing objects properties
`show`	Show a specific object

Currently the following object types are available on CLI:

command
endpoint
host
hostgroup
notification
service
timeperiod
user
usergroup
zone

Create a new object¶

Use this command to create a new Icinga object

Usage

icingacli director <type> create [<name>] [options]

Options

Option	Description
`--<key> <value>`	Provide all properties as single command line options
`--json`	Otherwise provide all options as a JSON string

Examples

To create a new host you can provide all of its properties as command line parameters:

icingacli director host create localhost \
    --import generic-host \
    --address 127.0.0.1 \
    --vars.location 'My datacenter'

It would say:

Host 'localhost' has been created

Providing structured data could become tricky that way. Therefore you are also allowed to provide JSON formatted properties:

icingacli director host create localhost \
    --json '{ "address": "127.0.0.1", "vars": { "test": [ "one", "two" ] } }'

Delete a specific object¶

Use this command to delete a single Icinga object. Just run

icingacli director <type> delete <name>

That’s it. To delete the host created before, this would read

icingacli director host delete localhost

It will tell you whether your command succeeded:

Host 'localhost' has been deleted

Check whether a specific object exists¶

Use this command to find out whether a single Icinga object exists. Just run:

icingacli director <type> exists <name>

So if you run…

icingacli director host exists localhost

…it will either tell you …

Host 'localhost' exists

…or:

Host 'localhost' does not exist

When executed from custom scripts you could also just check the exit code, 0 means that the object exists, 1 that it doesn’t.

Modify an existing objects property¶

Use this command to modify specific properties of an existing Icinga object.

Usage

icingacli director <type> set <name> [options]

Options

Option	Description
`--<key> <value>`	Provide all properties as single command line options
`--append-<key> <value>`	Appends to array values, like `imports`,
	`groups` or `vars.system_owners`
`--remove-<key> [<value>]`	Remove a specific property, eventually only
	when matching `value`. In case the property is an
	array it will remove just `value` when given
`--json`	Otherwise provide all options as a JSON string
`--replace`	Replace all object properties with the given ones
`--auto-create`	Create the object in case it does not exist

Examples

icingacli director host set localhost \
    --address 127.0.0.2 \
    --vars.location 'Somewhere else'

It will either tell you

Host 'localhost' has been modified

or, when for example issued immediately a second time:

Host 'localhost' has not been modified

Like create, this also allows you to provide JSON-formatted properties:

icingacli director host set localhost --json '{ "address": "127.0.0.2" }'

This command will fail in case the specified object does not exist. This is when the --auto-create parameter comes in handy. Command output will tell you whether an object has either been created or (not) modified.

With set you only set the specified properties and do not touch the other ones. You could also want to completely override an object, purging all other eventually existing and unspecified parameters. Please use --replace if this is the desired behaviour.

Show a specific object¶

Use this command to show single objects rendered as Icinga 2 config or in JSON format.

Usage

icingacli director <type> show <name> [options]

Options

Option	Description
`--resolved`	Resolve all inherited properties and show a flat object
	object
`--json`	Use JSON format
`--no-pretty`	JSON is pretty-printed per default (for PHP >= 5.4)
	Use this flag to enforce unformatted JSON
`--no-defaults`	Per default JSON output skips null or default values
	With this flag you will get all properties

Clone an existing object¶

Use this command to clone a specific object.

Usage

icingacli director <type> clone <name> --from <original> [options]

Options

Option	Description
`--from <original>`	The name of the object you want to clone
`--<key> <value>`	Override specific properties while cloning
`--replace`	In case an object already exists replace it
	with the clone
`--flat`	Do no keep inherited properties but create a flat
	object with all resolved/inherited properties

Examples

icingacli director host clone localhost2 --from localhost

icingacli director host clone localhost3 --from localhost --address 127.0.0.3

Other interesting tasks

Rename objects¶

There is no rename command, but a simple set can easily accomplish this task:

icingacli director host set localhost --object_name localhost2

Please note that it is usually absolutely no problem to rename objects with the Director. Even renaming something essential as a template like the famous generic-host will not cause any trouble. At least not unless you have other components outside your Director depending on that template.

Disable an object¶

Objects can be disabled. That way they will still exist in your Director DB, but they will not be part of your next deployment. Toggling the disabled property is all you need:

icingacli director host set localhost --disabled

Valid values for booleans are y, n, 1 and 0. So to re-enable an object you could use:

icingacli director host set localhost --disabled n

Working with booleans¶

As we learned before, y, n, 1 and 0 are valid values for booleans. But custom variables have no data type. And even if there is such, you could always want to change or override this from CLI. So you usually need to provide booleans in JSON format in case you need them in a custom variable.

There is however one exception from this rule. CLI parameters without a given value are handled as boolean flags by the Icinga Web 2 CLI. That explains why the example disabling an object worked without passing y or 1. You could use this also to set a custom variable to boolean true:

icingacli director host set localhost --vars.some_boolean

Want to change it to false? No chance this way, you need to pass JSON:

icingacli director host set localhost --json '{ "vars.some_boolean": false }'

This example shows the dot-notation to set a specific custom variable. If we have had used { "vars": { "some_boolean": false } }, all other custom vars on this object would have been removed.

Change object types¶

The Icinga Director distincts between the following object types:

Type	Description
`object`	The default object type. A host, a command and similar
`template`	An Icinga template
`apply`	An apply rule. This allows for assign rules
`external_obj ect`	An external object. Can be referenced and used, will not be
	deployed

Example for creating a host template:

icingacli director host create 'Some template' \
    --object_type template \
    --check_command hostalive

Please take a lot of care when modifying object types, you should not do so for a good reason. The CLI allows you to issue operations that are not allowed in the web frontend. Do not use this unless you really understand its implications. And remember, with great power comes great responsibility.

Import/Export Director Objects¶

Some objects are not directly related to Icinga Objects but used by the Director to manage them. To make it easier for administrators to for example pre-fill an empty Director Instance with Import Sources and Sync Rules, related import/export commands come in handy.

Use icingacli director export <type> [options] to export objects of a specific type:

Type	Description
`datafields`	Export all DataField definitions
`datalists`	Export all DataList definitions
`hosttemplatechoices`	Export all IcingaTemplateChoiceHost definitions
`importsources`	Export all ImportSource definitions
`jobs`	Export all Job definitions
`syncrules`	Export all SyncRule definitions

Options

Option	Description
`--no-pretty`	JSON is pretty-printed per default. Use this flag to
	enforce unformatted JSON

Use icingacli director import <type> < exported.json to import objects of a specific type:

Type	Description
`importsources`	Import ImportSource definitions from STDIN
`syncrules`	Import SyncRule definitions from STDIN

This feature is available since v1.5.0.

Director Configuration Basket¶

A basket contains a set of Director Configuration objects (like Templates, Commands, Import/Sync definitions - but not single Hosts or Services). This CLI command allows you to integrate them into your very own workflows

Available Actions¶

Action	Description
`dump`	JSON-dump for objects related to the given Basket
`list`	List configured Baskets
`restore`	Restore a Basket from JSON dump provided on STDIN
`snapshot`	Take a snapshot for the given Basket

Options

Option	Description
`--name`	`dump` and `snapshot` require a specific object name

Use icingacli director basket restore < exported-basket.json to restore objects from a specific basket. Take a snapshot or a backup first to be on the safe side.

This feature is available since v1.6.0.

Health Check Plugin¶

You can use the Director CLI as an Icinga CheckPlugin and monitor your Director Health. This will run all or just one of the following test suites:

Name	Description
`config`	Configuration, Schema, Migrations
`sync`	All configured Sync Rules (pending changes are not a problem)
`import`	All configured Import Sources (pending changes are not a problem)
`jobs`	All configured Jobs (ignores disabled ones)
`deployment`	Deployment Endpoint, last deployment outcome

Usage

icingacli director health check [options]

Options

Option	Description
`--check <name>`	Run only a specific test suite
`--<db> <name>`	Use a specific Icinga Web DB resource

Examples

icingacli director health check

Example for running a check only for the configuration:

icingacli director health check --check config

Sample output:

Director configuration: 5 tests OK
[OK] Database resource 'Director DB' has been specified'
[OK] Make sure the DB schema exists
[OK] There are no pending schema migrations
[OK] Deployment endpoint is 'icinga.example.com'
[OK] There is a single un-deployed change

Kickstart and schema handling¶

The kickstart and the migration command are handled in the automation section of the upstream documentation.

Configuration handling¶

Render your configuration¶

The Director distincts between rendering and deploying your configuration. Rendering means that Icinga 2 config will be pre-rendered and stored to the Director DB. Nothing bad happens if you decide to render the current config thousands of times in a loop. In case a config with the same checksum already exists, it will store - nothing.

You can trigger config rendering by running

icingacli director config render

In case a new config has been created, it will tell you so:

New config with checksum b330febd0820493fb12921ad8f5ea42102a5c871 has been generated

Run it once again, and you’ll see that the output changes:

Config with checksum b330febd0820493fb12921ad8f5ea42102a5c871 already exists

Config deployment¶

You do not need to explicitely render your config before deploying it to your Icinga 2 master node. Just trigger a deployment, it will re-render the current config:

icingacli director config deploy

The output tells you which config has been shipped:

Config 'b330febd0820493fb12921ad8f5ea42102a5c871' has been deployed

Director tries to avoid needless deployments, so in case you immediately deploy again, the output changes:

Config matches active stage, nothing to do

You can override this by adding the --force parameter. It will then tell you:

Config matches active stage, deploying anyway

In case you want to do not want deploy to waste time to re-render your config or in case you decide to re-deploy a specific, eventually older config version the deploy command allows you to provide a specific checksum:

icingacli director config deploy --checksum b330febd0820493fb12921ad8f5ea42102a5c871

Deployments status¶

In case you want to fetch the information about the deployments status, you can call the following CLI command:

icingacli director config deploymentstatus

{
     "active_configuration": {
         "stage_name": "5c65cae0-4f1b-47b4-a890-766c82681622",
         "config": "617b9cbad9e141cfc3f4cb636ec684bd60073be1",
         "activity": "4f7bc6600dd50a989f22f82d3513e561ef333363"
     }
}

In case there is no active stage name related to the Director, active_configuration is set to null.

Another possibility is to pass a list of checksums to fetch the status of specific deployments and (activity log) activities. Following, you can see an example of how to do it:

icingacli director config deploymentstatus \
 --configs 617b9cbad9e141cfc3f4cb636ec684bd60073be1 \
 --activities 4f7bc6600dd50a989f22f82d3513e561ef333363

{
     "active_configuration": {
         "stage_name": "5c65cae0-4f1b-47b4-a890-766c82681622",
         "config": "617b9cbad9e141cfc3f4cb636ec684bd60073be1",
         "activity": "4f7bc6600dd50a989f22f82d3513e561ef333363"
     },
     "configs": {
         "617b9cbad9e141cfc3f4cb636ec684bd60073be1": "active"
     },
     "activities": {
         "4f7bc6600dd50a989f22f82d3513e561ef333363": "active"
     }
}

You can also decide to access directly to a value inside the result JSON by using the –key param:

icingacli director config deploymentstatus \
 --configs 617b9cbad9e141cfc3f4cb636ec684bd60073be1 \
 --activities 4f7bc6600dd50a989f22f82d3513e561ef333363 \
 --key active_configuration.config

617b9cbad9e141cfc3f4cb636ec684bd60073be1

Cronjob usage¶

You could decide to pre-render your config in the background quite often. As of this writing this has one nice advantage. It allows the GUI to find out whether a bunch of changes still results into the very same config. only one

Run sync and import jobs¶

Import Sources¶

List available Import Sources shows a table with your defined Import Sources, their IDs and current state. As triggering Imports requires an ID, this is where you can look up the desired ID:
```
icingacli director importsource list
```
Check a given Import Source for changes fetches data from the given Import Source and compares it to the most recently imported data:
```
icingacli director importsource check --id <id>
```
Options:

Option

Description

--id <id>

An Import Source ID. Use the list command to figure out

--benchmark

Show timing and memory usage details
Fetch data from a given Import Source fetches data from the given Import Source and outputs them as plain JSON:
```
icingacli director importsource fetch --id <id>
```
Options:

Option

Description

--id <id>

An Import Source ID. Use the list command to figure out

--benchmark

Show timing and memory usage details
Trigger an Import Run for a given Import Source fetches data from the given Import Source and stores it to the Director DB, so that the next related Sync Rule run can work with fresh data. In case data didn’t change, nothing is going to be stored:
```
icingacli director importsource run --id <id>
```
Options:

Option

Description

--id <id>

An Import Source ID. Use the list command to figure out

--benchmark

Show timing and memory usage details

Sync Rules¶

List defined Sync Rules shows a table with your defined Sync Rules, their IDs and current state. As triggering a Sync requires an ID, this is where you can look up the desired ID:
```
icingacli director syncrule list
```
Check a given Sync Rule for changes runs a complete Sync in memory but doesn’t persist eventual changes:
```
icingacli director syncrule check --id <id>
```
Options

Option

Description

--id <id>

A Sync Rule ID. Use the list command to figure out

--benchmark

Show timing and memory usage details
Trigger a Sync Run for a given Sync Rule builds new objects according your Sync Rule, compares them with existing ones and persists eventual changes:
```
icingacli director syncrule run --id <id>
```
Options

Option

Description

--id <id>

A Sync Rule ID. Use the list command to figure out

--benchmark

Show timing and memory usage details

Database housekeeping¶

Your database may grow over time and ask for various housekeeping tasks. You can usually store a lot of data in your Director DB before you would even notice a performance impact.

Still, we started to prepare some tasks that assist with removing useless garbage from your DB. You can show available tasks with:

icingacli director housekeeping tasks

The output might look as follows:

 Housekeeping task (name)                                  | Count
-----------------------------------------------------------|-------
 Undeployed configurations (oldUndeployedConfigs)          |     3
 Unused rendered files (unusedFiles)                       |     0
 Unlinked imported row sets (unlinkedImportedRowSets)      |     0
 Unlinked imported rows (unlinkedImportedRows)             |     0
 Unlinked imported properties (unlinkedImportedProperties) |     0

You could run a specific task with

icingacli director housekeeping run <taskName>

…like in:

icingacli director housekeeping run unlinkedImportedRows

Or you could also run all of them, that’s the preferred way of doing this:

icingacli director housekeeping run ALL

Please note that some tasks once issued create work for other tasks, as lost imported rows might appear once you remove lost row sets. So ALL is usually the best choice as it runs all of them in the best order.

The Icinga Director REST API¶

Icinga Director has been designed with a REST API in mind. Most URLs you can access with your browser will also act as valid REST url endpoints.

Base Headers¶

All your requests MUST have a valid accept header. The only acceptable variant right now is application/json, so please always append a header as follows to your requests:

Accept: application/json

Authentication¶

Please use HTTP authentication and any valid Icinga Web 2 user, granted enough permissions to accomplish the desired actions. The restrictions and permissions that have been assigned to web users will also be enforced for API users. In addition, the permission director/api is required for any API access.

Versioning¶

There are no version strings so far in the Director URLs. We will try hard to not break compatibility with future versions. Sure, sooner or later we also might be forced to introduce some kind of versioning. But who knows?

As a developer you can trust us to not remove any existing REST url or any provided property. However, you must always be ready to accept new properties.

URL scheme and supported methods¶

We support GET, POST, PUT and DELETE.

Method	Meaning
GET	Read / fetch data. Not allowed to run operations with the potential to cause any harm
POST	Trigger actions, create or modify objects. Can also be used to partially modify objects
PUT	Creates or replaces objects, cannot be used to modify single object properties
DELETE	Remove a specific object

POST director/host

gives 201 on success

GET director/host?name=hostname.example.com

PUT director/host?name=hostname.example.com

gives 200 ok on success and 304 not modified on no change

DELETE director/host?name=hostname.example.com

gives 200 on success

First example request with CURL¶

curl -H 'Accept: application/json' \
     -u 'username:password' \
     'https://icinga.example.com/icingaweb2/director/host?name=hostname.example.com'

CURL helper script¶

A script like the following makes it easy to play around with curl:

METHOD=$1
URL=$2
BODY="$3"
USERNAME="demo"
PASSWORD="***"
test -z "$PASSWORD" || USERNAME="$USERNAME:$PASSWORD"

test -z "$BODY" && curl -u "$USERNAME" \
  -i https://icingaweb/icingaweb/$URL \
  -H 'Accept: application/json' \
  -X $METHOD

test -z "$BODY" || curl -u "$USERNAME" \
  -i https://icingaweb/icingaweb/$URL \
  -H 'Accept: application/json' \
  -X $METHOD \
  -d "$BODY"

echo

It can be used as follows:

director-curl GET director/host?name=localhost

director-curl POST director/host '{"object_name": "host2", "... }'

Should I use HTTPS?¶

Sure, absolutely, no doubt. There is no, absolutely no reason to NOT use HTTPS these days. Especially not for a configuration tool allowing you to configure check commands that are going to be executed on all your servers.

Special parameters for Icinga Objects¶

Resolve object properties. In case you add the resolve parameter to your URL, all inherited object properties will be resolved. Such a URL could look as follows:
```
director/host?name=hostname.example.com&resolved
```

Retrieve all properties. Per default properties with null value are skipped when shipping a result. You can influence this behavior with the properties parameter. Just append properties=ALL to your URL:
```
director/host?name=hostname.example.com&properties=all
```
Retrieve only specific properties. The properties parameter also allows you to specify a list of specific properties. In that case, only the given properties will be returned, even when they have no (null) value:
```
director/host?name=hostname.example.com&properties=object_name,address,vars
```

Example

GET director/host?name=pe2015.example.com

{
  "address": "127.0.0.3",
  "check_command": null,
  "check_interval": null,
  "display_name": "pe2015 (example.com)",
  "enable_active_checks": null,
  "flapping_threshold": null,
  "groups": [ ],
  "imports": [
    "generic-host"
  ],
  "retry_interval": null,
  "vars": {
    "facts": {
      "aio_agent_build": "1.2.5",
      "aio_agent_version": "1.2.5",
      "architecture": "amd64",
      "augeas": {
        "version": "1.4.0"
      },
   ...
}

director/host?name=pe2015.example.com&resolved

{
    "address": "127.0.0.3",
    "check_command": "tom_ping",
    "check_interval": "60",
    "display_name": "pe2015 (example.com)",
    "enable_active_checks": true,
    "groups": [ ],
    "imports": [
      "generic-host"
    ],
    "retry_interval": "10",
    "vars": {
      "facts": {
        "aio_agent_build": "1.2.5",
        "aio_agent_version": "1.2.5",
        "architecture": "amd64",
        "augeas": {
          "version": "1.4.0"
        },
        ...
      }
    }
}

JSON is pretty-printed per default, at least for PHP >= 5.4

Error handling¶

Director tries hard to return meaningful output and error codes:

HTTP/1.1 400 Bad Request
Server: Apache
Content-Length: 46
Connection: close
Content-Type: application/json

{
    "error": "Invalid JSON: Syntax error"
}

Trigger actions¶

You can of course also use the API to trigger specific actions. Deploying the configuration is as simple as issuing::

POST director/config/deploy

Note

Currently we do not handle Last-Modified and ETag headers. This would involve some work, but could be a cool feature. Let us know your ideas!

Sample scenario¶

Let’s show you how the REST API works with a couple of practical examples:

Create a new host

POST director/host

{
  "object_name": "apitest",
  "object_type": "object",
  "address": "127.0.0.1",
  "vars": {
    "location": "Berlin"
  }
}

Response

HTTP/1.1 201 Created
Date: Tue, 01 Mar 2016 04:43:55 GMT
Server: Apache
Content-Length: 140
Content-Type: application/json

{
    "address": "127.0.0.1",
    "object_name": "apitest",
    "object_type": "object",
    "vars": {
        "location": "Berlin"
    }
}

The most important part of the response is the response code: 201, a resource has been created. Just for fun, let’s fire the same request again. The answer obviously changes:

HTTP/1.1 500 Internal Server Error
Date: Tue, 01 Mar 2016 04:45:04 GMT
Server: Apache
Content-Length: 60
Connection: close
Content-Type: application/json

{
    "error": "Trying to recreate icinga_host (apitest)"
}

So, let’s update this host. To work with existing objects, you must ship their name in the URL:

POST director/host?name=apitest

{
  "object_name": "apitest",
  "object_type": "object",
  "address": "127.0.0.1",
  "vars": {
    "location": "Berlin"
  }
}

Same body, so no change:

HTTP/1.1 304 Not Modified
Date: Tue, 01 Mar 2016 04:45:33 GMT
Server: Apache

So let’s now try to really change something:

POST director/host?name=apitest

{"address": "127.0.0.2", "vars.event": "Icinga CAMP" }

We get status 200, changes have been applied:

HTTP/1.1 200 OK
Date: Tue, 01 Mar 2016 04:46:25 GMT
Server: Apache
Content-Length: 172
Content-Type: application/json

{
    "address": "127.0.0.2",
    "object_name": "apitest",
    "object_type": "object",
    "vars": {
        "location": "Berlin",
        "event": "Icinga CAMP"
    }
}

The response always returns the full object on modification. This way you can immediately investigate the merged result. As you can see, POST requests only touch the parameters you passed - the rest remains untouched.

One more example to prove this:

POST director/host?name=apitest

{"address": "127.0.0.2", "vars.event": "Icinga CAMP" }

No modification, you get a 304. HTTP standards strongly discourage shipping a body in this case:

HTTP/1.1 304 Not Modified
Date: Tue, 01 Mar 2016 04:52:05 GMT
Server: Apache

As you might have noted, we only changed single properties in the vars dictionary. Now lets override the whole dictionary:

POST director/host?name=apitest

{"address": "127.0.0.2", "vars": { "event": [ "Icinga", "Camp" ] } }

The response shows that this works as expected:

HTTP/1.1 200 OK
Date: Tue, 01 Mar 2016 04:52:33 GMT
Server: Apache
Content-Length: 181
Content-Type: application/json

{
    "address": "127.0.0.2",
    "object_name": "apitest",
    "object_type": "object",
    "vars": {
        "event": [
            "Icinga",
            "Camp"
        ]
    }
}

If merging properties is not what you want, PUT comes to the rescue:

PUT director/host?name=apitest

{ "vars": { "event": [ "Icinga", "Camp" ] }

All other properties vanished, all but name and type:

HTTP/1.1 200 OK
Date: Tue, 01 Mar 2016 04:54:33 GMT
Server: Apache
Content-Length: 153
Content-Type: application/json

{
    "object_name": "apitest",
    "object_type": "object",
    "vars": {
        "event": [
            "Icinga",
            "Camp"
        ]
    }
}

Let’s put “nothing”:

PUT director/host?name=apitest

{}

Works as expected:

HTTP/1.1 200 OK
Date: Tue, 01 Mar 2016 04:57:35 GMT
Server: Apache
Content-Length: 62
Content-Type: application/json

{
    "object_name": "apitest",
    "object_type": "object"
}

Of course, PUT also supports 304, you can check this by sending the same request again.

Now let’s try to cheat:

KILL director/host?name=apitest

HTTP/1.1 400 Bad Request
Date: Tue, 01 Mar 2016 04:54:07 GMT
Server: Apache
Content-Length: 43
Connection: close
Content-Type: application/json

{
    "error": "Unsupported method KILL"
}

Ok, no way. So let’s use the correct method:

DELETE director/host?name=apitest

HTTP/1.1 200 OK
Date: Tue, 01 Mar 2016 05:59:22 GMT
Server: Apache
Content-Length: 109
Content-Type: application/json

{
    "imports": [
        "generic-host"
    ],
    "object_name": "apitest",
    "object_type": "object"
}

Service Apply Rules

Please note that Service Apply Rule names are not unique in Icinga 2. They are not real objects, they are creating other objects in a loop. This makes it impossible to distinct them by name. Therefore, a dedicated REST API endpoint director/serviceapplyrules ships all Service Apply Rules combined with their internal ID. This ID can then be used to modify or delete a Rule via director/service.

Deployment Status

In case you want to fetch the information about the deployments status, you can call the following API:

GET director/config/deployment-status

HTTP/1.1 200 OK
Date: Wed, 07 Oct 2020 13:14:33 GMT
Server: Apache
Content-Type: application/json

{
    "active_configuration": {
        "stage_name": "b191211d-05cb-4679-842b-c45170b96421",
        "config": "617b9cbad9e141cfc3f4cb636ec684bd60073be1",
        "activity": "028b3a19ca7457f5fc9dbb5e4ea527eaf61616a2"
    }
}

This throws a 500 in case Icinga isn’t reachable. In case there is no active stage name related to the Director, active_configuration is set to null.

Another possibility is to pass a list of checksums to fetch the status of specific deployments and (activity log) activities. Following, you can see an example of how to do it:

GET director/config/deployment-status?config_checksums=617b9cbad9e141cfc3f4cb636ec684bd60073be2,
617b9cbad9e141cfc3f4cb636ec684bd60073be1&activity_log_checksums=617b9cbad9e141cfc3f4cb636ec684bd60073be1,
028b3a19ca7457f5fc9dbb5e4ea527eaf61616a2

{
    "active_configuration": {
        "stage_name": "b191211d-05cb-4679-842b-c45170b96421",
        "config": "617b9cbad9e141cfc3f4cb636ec684bd60073be1",
        "activity": "028b3a19ca7457f5fc9dbb5e4ea527eaf61616a2"
    },
    "configs": {
        "617b9cbad9e141cfc3f4cb636ec684bd60073be2": "deployed",
        "617b9cbad9e141cfc3f4cb636ec684bd60073be1": "active"
    },
    "activities": {
        "617b9cbad9e141cfc3f4cb636ec684bd60073be1": "undeployed",
        "028b3a19ca7457f5fc9dbb5e4ea527eaf61616a2": "active"
    }
}

The list of possible status is:

active: whether this configuration is currently active
deployed: whether this configuration has ever been deployed
failed: whether the deployment of this configuration has failed
undeployed: whether this configuration has been rendered, but not yet deployed
unknown: whether no configurations have been found for this checksum

Agent Tickets

The Director is very helpful when it goes to manage your Icinga Agents. In case you want to fetch tickets through the API, please do as follows:

GET director/host/ticket?name=apitest

HTTP/1.1 200 OK
Date: Thu, 07 Apr 2016 22:19:24 GMT
Server: Apache
Content-Length: 43
Content-Type: application/json

"5de9883080e03278039bce57e4fbdbe8fd262c40"

Please expect an error in case the host does not exist or has not been configured to be an Icinga Agent.

Self Service API

Theory of operation

Icinga Director offers a Self Service API, allowing new Icinga nodes to register themselves. No credentials are required, authentication is based on API keys. There are two types of such keys:
- Host Template API keys
- Host Object API keys
Template keys basically grant the permission to:
- Create a new host based on that template
- Specify name and address properties for that host
This is a one-time operation and allows one to claim ownership of a specific host. Now, there are two possible scenarios:
- The host already exists
- The host is not known to Icinga Director
In case the host already exists, Director will check whether it’s API key matches the given one. [..]
Request processing for Host registration

A new node will POST to self-service/register-host, with two parameters in the URL:
- name: it’s desired object name, usually the FQDN
- key: a valid Host Template API key
In it’s body it is allowed to specify a specific set of properties. At the time of this writing, these are:
- display_name
- address
- address6
Director will validate the key and load the corresponding Host Template. In case no such is found, the request is rejected. Then it checks whether a Host with the given name exists. In case it does, the request is rejected unless:
- It inherits the loaded Host Template
- It already has an API key
If these conditions match, the request is processed. The following sketch roughly shows the decision tree (AFTER the key has been validated):

Self Service API¶

Icinga Director offers a Self Service API, allowing new Hosts running the Icinga Agent to register themselves in a secure way.

Windows Agents¶

Windows Agents are the main target audience for this feature. It allows you to generate a single Powershell Script based on the Icinga 2 Powershell Module. You can either use the same script for all of your Windows Hosts or generate different ones for different kind of systems.

This installation script could then be shipped with your base images, invoked remotely via PowerShell Remoting, distributed as a module via Group Policies and/or triggered via Run-Once (AD Policies).

Linux Agents¶

At the time of this writing, we do not ship a script with all the functionality you can find in the Windows Powershell script. Linux and Unix environments are mostly highly automated these days, and such a magic shell script is often not what people want.

Still, you can also benefit from this feature by directly using our Self Service REST API. It should be easy to integrate it into the automation tool of your choice.

Base Configuration¶

You have full control over the automation Script generated by the Icinga Director. Please got to the Infrastructure Dashboard and choose the Self Service API:

Fig. 106 Infrastructure Dashboard - Self Service API¶

This leads to the Self Service API Settings form. Most settings are self-explaining and come with detailled inline hints. The most important choice is whether the script should automatically install the Icinga Agent:

Fig. 107 Settings - Choose installation source¶

In case you opted for automated installation, more options will pop up:

Fig. 108 Settings - Installer Details¶

The Icinga Director “Live-Creation” experimental feature permits to create Icinga Objects in the Director and simultaneously also directly in Icinga2, without the need to deploy the Director configuration.

The Live-Creation is available both from icingacli and from the Director REST API.

Below you see a flowchart which gives you an idea of what happens when the Object Live-Creation is called from the Director REST API.

Fig. 109 Import source¶

CLI Commands¶

Fetch all available Virtual Machines¶

This command is mostly for test/debug reasons and gives you an output of all Virtual Machines with a default set of properties:

icingacli vsphere fetch virtualmachines [options]

The available options are:

Option	Description
`--<vhost> <host>`	IP, host or URL for your vCenter or ESX host
`--<username> <user>`	When authenticating, this username will be used
`--<password> <pass>`	The related password
`--lookup-ids`	Replace id-references with their name
`--no-ssl-verify-peer`	Accept certificates signed by an unknown CA
`--no-ssl-verify-host`	Accept certificates not matching the host name
`--use-insecure-http`	Use plaintext HTTP requests
`--proxy <proxy>`	Use the given Proxy (ip, host or host:port
`--proxy-type <type>`	HTTP (default) or SOCKS5
`--proxy-username <user>`	Username for authenticated HTTP proxy
`--proxy-password <pass>`	Password for authenticated HTTP proxy
`--benchmark`	Show resource usage summary
`--json`	Dump JSON output

Fetch all available Host Systems¶

This command is mostly for test/debug reasons and gives you an output of all Host Systems with a default set of properties:

icingacli vsphere fetch hostsystems [options]

The available options are:

Option	Description
`--<vhost> <host>`	IP, host or URL for your vCenter or ESX host
`--<username> <user>`	When authenticating, this username will be used
`--<password> <pass>`	The related password
`--no-ssl-verify-peer`	Accept certificates signed by an unknown CA
`--no-ssl-verify-host`	Accept certificates not matching the host name
`--use-insecure-http`	Use plaintext HTTP requests
`--proxy <proxy>`	Use the given Proxy (ip, host or host:port
`--proxy-type <type>`	HTTP (default) or SOCKS5
`--proxy-username <user>`	Username for authenticated HTTP proxy
`--proxy-password <pass>`	Password for authenticated HTTP proxy
`--benchmark`	Show resource usage summary
`--json`	Dump JSON output

Shutdown Management Rest API¶

Shutdown can be executed on an host using a REST API. Currently, the following calls are available.

Trigger Shutdown Definition¶

Endpoint: trigger-shutdown-definition

This endpoint enables you to trigger an asynchronous run of a shutdown definition via a REST API call. The call is non blocking and the shutdown will be performed in background.

Parameters: * id: The ID of the shutdown definition

Example:

curl -u root:xxx -H 'Accept: application/json' https://localhost/neteye/shutdownmanager/api/trigger-shutdown-definition?id=1

Configuring Icinga Monitoring Retention Policy¶

This section describes how to configure and applying the retention time for Icinga Data Output (IDO) DB.

The retention time is by default set to 550 days, meaning that the data stored in IDO DB are kept for 550 days and deleted afterwards. This will affect the monitoring data history, that are used for populating SLM reports. This value can be changed at a later point, customize it based on user’s needs.

Background¶

In the 4.13 release a new setting has been introduced, called retention policy. The reason for its introduction is that the all data (warnings, logs, output of checks, and so on) are stored in the Database, which can dramatically grow over time. This setting will be enabled by default in new installation only, while existing installation must configure it and enable it, following the steps described next. The default value given to this setting is 550 days, but can be changed at a later point.

How To Set The Retention Time¶

To configure or modify the value, go to Configuration > Modules > Neteye > Configuration.

Step 0. Click on the slider called Enable default retention policy to enable the feature. The slider will disappear afterwards. Indeed, once the feature is enabled, there is no rolling back, it will remain enabled forever.
Step 1. Insert a value in days for the Default retention policy age, which by default is 550, i.e., one year.
Step 2. Click on the Save Changes button.
Step 4. Go to the command line and run the neteye_secure_install script.

Warning

Only after Step 4 the setting will be applied and enabled, so make sure to complete successfully all the steps!

How To Disable the Retention Time¶

The retention time can be disabled, meaning that no data will ever be deleted. To do so, set the value of retention time to 0 (zero).

Warning

Disabling the retention time is discouraged, because the disk space required by the Databases might grow quickly if the monitoring activities on NetEye create a lot of input.

Option	Description
`--id <id>`	An Import Source ID. Use the list command to figure out
`--benchmark`	Show timing and memory usage details

Option	Description
`--id <id>`	A Sync Rule ID. Use the list command to figure out
`--benchmark`	Show timing and memory usage details