Blockchain Verification¶

How to Setup the Automatic Verification of Blockchains¶

This chapter illustrates the best practices for the setup of a secure and automatic verification of El Proxy blockchains. We will setup an environment where the verification of a specific blockchain is performed every day. Moreover, we will describe how we can notify of the blockchain verification results in Icinga 2 via Tornado Webhook Collectors and Tornado rules.

This guide is organized as follows: we first introduce the Prerequisites and then we describe the NetEye setup. After that, we provide indications on how to test the deployed configuration.

Prerequisites¶

For the start you will need to have the file containing the initial key of the blockchain to be verified

The automated verification needs to be set up on a machine which is external to the NetEye installation, accessible only by you, referred to as DPO machine. The DPO machine must meet the following requirements:

Run a Linux distribution

Support Docker

Warning

If you opt to use a Debian distribution, please verify that the AppArmor configuration is not interfering with Docker, as executing the automated neteye dpo setup could otherwise trigger a permission denied error.

The DPO machine must be reachable, from the NetEye Master, on port 22 for the SSH connection during the setup procedure

Note

The SSH connection will be necessary only when executing the setup command and will be performed using password authentication. Credentials to access the DPO machine are never stored inside NetEye.

The DPO machine must reach the NetEye Master on port 9200 for contacting Elasticsearch.

The DPO machine must reach the webhook_collector_host on port 443 to connect to the Tornado Webhook Collector.

The DPO machine must reach DockerHub to be able to pull El Proxy container images

The DPO machine must be a known host on the NetEye Master. This is needed for the setup procedure to safely connect to the DPO machine through SSH. Run the following command in order to add the DPO machine to the list of known hosts on the NetEye Master: ssh-keyscan <dpo-hostname> >> /root/.ssh/known_hosts.

The DPO machine and the NetEye installation must be configured in the same timezone

The backup of the DPO machine must be performed on the customer’s part

Note

If you would like to setup the automatic verification of the blockchain on a Windows system, you can download the Docker image from DockerHub and perform a manual setup of the automatic verification. For more information, please refer to the official channels: sales, consultants, or support portal.

Verification Setup¶

Step 1. Configure the blockchain verification

Configuration is to be carried out on the NetEye Master in the /etc/neteye-dpo JSON-formatted file.

As an example, we can configure, using the DPO root user on the dpo-host DPO machine, the verification of a blockchain having a 6_months retention. The blockchain belongs to the test_tenant and we would like the verification to run each day at 7PM, sending the results to a Tornado Webhook Collector hosted at satellite1 with the test_token token. In this case, the configuration in the /etc/neteye-dpo file is to be built as follows:

{
  "dpo_host": "dpo-host",
  "dpo_user": "root",
  "blockchains_verification": [
    {
      "tenant": "test_tenant",
      "retention": "6_months",
      "tag": "0",
      "webhook_host": "satellite1",
      "webhook_token": "test_token",
      "cron_scheduling": {
        "minute": "0",
        "hour": "19",
        "day": "*",
        "month": "*",
        "week_day": "*"
      }
    }
  ]
}

For looking up a full list of configuration file attributes please consult neteye dpo setup section.

Step 2. Run the neteye dpo setup command

The command should be run from the NetEye Master. Upon running the command, you will be prompted to provide the password of the user, specified in the configuration. This will allow you to connect to the DPO machine and the initial key of the blockchain.

Note

If you are not using the root account, you will be asked also for the password to be used to perform some actions with sudo. If the password equals to the one used to connect via SSH to the DPO machine, you can press enter to use it also for the commands execution.

The command will connect to the DPO machine and set up the verification launching a container for each verification, which will then be performed at the specified schedule.

Note

Among the different actions performed by the command, this also tries to install Docker on the system. However, the automatic installation may fail for some OS which require a more specific installation path. If the command fails due to this error, please manually install Docker on the system and then run the command with the –skip-tags install_docker_packages option:
neteye dpo setup --skip-tags install_docker_packages

Step 3. Configure a Webhook

Tornado Webhook Collector is to be configured on either the NetEye Master, or a NetEye Satellite. It will take care of receiving El Proxy blockchain verification result from the DPO machine and forwarding it to Tornado, which will then set an Icinga 2 status.

On the node where the Tornado Webhook Collector is running, create the file /neteye/shared/tornado_webhook_collector/conf/webhooks/elproxy_verification.json
Set its content to:
{
  "id": "elproxy_verification",
  "token": "<webhook_token>",
  "collector_config": {
    "event_type": "elproxy_verification",
    "payload": {
      "data": "${@}"
    }
  }
}
Restart the Tornado Webhook Collector service to load the webhook

Step 4. Configure a Rule in Tornado to set a status in Icinga 2

Via NetEye Tornado GUI, create a Rule that matches the Events with type elproxy_verification and executes an Action of type SMART_MONITORING_CHECK_RESULT, where we set as check_result -> exit_status the value event.payload.data.exit_status and as check_result -> plugin_output the value event.payload.data.output.

Moreover, if you are sending the results of the verification of multiple blockchains to the same Tornado Webhook Collector, you can filter them by using the event.payload.data.tenant, event.payload.data.retention and event.payload.data.tag values.

Configure the rest of the Rule as you prefer.

Testing the Configuration¶

Now everything should be configured correctly. To test your configuration, on the DPO machine, you can Manually trigger a pre-configured automatic verification.

Manually trigger a pre-configured automatic verification¶

As a user I want to be able to manually trigger a verification even though the automatic verification has already been configured, for example to immediately report the status of the blockchain to the management.

Connect to the DPO machine using the same credentials specified when performing the setup of the automatic verification.

Note

If you connected to the DPO machine with a user different from root, run the following commands using sudo
Open a shell in the verification container you would like to trigger manually. To do so, run the docker exec -it elproxy-verify-<tenant>-<retention>-<tag> bash where the tenant, retention and tag match the properties of your blockchain.

Hint

To check which verification containers are running on the DPO machine you can execute the docker ps command
Extract the verification command from the cron file defining its schedule, located at /etc/cron.d/elproxy-verify-<tenant>-<retention>-<tag> For example, for the blockchain defined in automatic verification, the file, located at /etc/cron.d/elproxy-verify-test_tenant-6_months-0 will have the following content:
```
0 19 * * * root /bin/elproxy_verify_blockchain_and_send_result test_tenant 6_months 0 satellite1 test_token 1000
```
In this case, the verification command will be /bin/elproxy_verify_blockchain_and_send_result test_tenant 6_months 0 satellite1 test_token 1000
Run the extracted verification command.

Expected results:

The verification is performed and the result is sent to the configured Tornado Webhook Collector. The output of the command is saved in the /root/elproxy-verification/logs folder, with the file being identified by the verified blockchain and the current timestamp.

Manual Blockchain Verification¶

This section aims to define the guidelines for the manual verification of a blockchain. The following naming convention is used while defining the guidelines for manual verification:

<tenant>: The tenant (also referred to as “customer” in the previous chapters) of the blockchain that needs to be verified
<retention>: the retention of the blockchain to be verified
<tag>: the tag of the blockchain to be verified

Note

While following this guide and executing commands, remember to always substitute the placeholders <…> with their real value.

The guide is based on the two examples which should cover most of the use cases.

For both examples, we assume the following:

El Proxy has been correctly set up by following the the NetEye Setup section
Every month has exactly 30 days (to facilitate the date calculations)

Manual verification with existing BSH file

In this example, we assume the following:

The current date is 2022-08-30
Stored on the NetEye machine, there is a non-empty blockchain created on 2022-05-01 with the following properties:
1. TENANT: exampletenant
2. RETENTION: 3_months
3. TAG: exampletag
The BSH file is stored under its default path on the DPO machine and contains 90 entries (one entry for each day) from 2022-05-01 to 2022-07-30

Note that, on the current date, the blockchain is already missing some logs. Since the retention policy is three months and the blockchain was created on 2022-05-01, the logs indexed from 2022-05-01 to 2022-05-30 have been automatically deleted by Elasticsearch.

In order to start the verification manually, the DPO admin needs to connect to the running verification container, see Manually trigger a pre-configured automatic verification, and then run:

/usr/bin/elastic_blockchain_proxy verify \
--key-file "/root/elproxy-verification/data/keys/elproxysigned-exampletenant-3_months-exampletag_key.json" \
--tenant "exampletenant" \
--retention "3_months" \
--tag "exampletag" \
--elasticsearch-authentication-method pemcertificatepath \
--elasticsearch-client-cert /root/elproxy-verification/conf/certs/neteye_ebp_verify_exampletenant.crt.pem \
--elasticsearch-client-private-key /root/elproxy-verification/conf/certs/private/neteye_ebp_verify_exampletenant.key.pem

Internally, the verify command tries to retrieve the iteration from which to start the verification; based on the current date and the retention policy of the blockchain, El Proxy expects the first iteration of the blockchain to be the first log indexed on 2022-06-01.

This date was calculated as follows:

date = current_date - retention_policy_in_days + 1

Where date is the resulting date, current_date is the current date, and retention_policy_in_days is the retention policy in days of the blockchain. For completeness, the final + 1 avoids the resulting date being the date on which Elasticsearch is applying the retention policy.

To retrieve the actual first iteration from which to start the verification, El Proxy fetches the BSH file searching for the entry corresponding to date. As an example, a section of the BSH file could look like the following one:

"2022-05-30": {
    "first_iteration": 15
    "last_iteration": 20
},
"2022-06-01": {
    "first_iteration": 21
    "last_iteration": 23
},
"2022-06-02": {
    "first_iteration": 24
    "last_iteration": 30
}

In this case, the first iteration from which to start the verification is iteration 21. At this point, El Proxy can start verifying the logs from the retrieved first iteration. If the verification is successful, the old BSH file will be overwritten with a new one containing one entry for each day from 2022-06-01 to the current date.

Manual verification of newly created blockchain

In this example, we suppose the following:

The current date is 2022-09-01
Stored on the NetEye machine, there is a non-empty blockchain created on the current date with the following properties:
1. TENANT: exampletenant
2. RETENTION: 3_months
3. TAG: exampletag
The BSH file does not exist

To start the verification manually, the DPO admin needs to connect to the running verification container, see Manually trigger a pre-configured automatic verification, and then run:

/usr/bin/elastic_blockchain_proxy verify \
--key-file "/root/elastic-blockchain-proxy/data/keys/elproxysigned-exampletenant-3_months-exampletag_key.json" \
--tenant "exampletenant" \
--retention "3_months" \
--tag "exampletag" \
--elasticsearch-authentication-method pemcertificatepath \
--elasticsearch-client-cert /root/elproxy-verification/conf/certs/neteye_ebp_verify_exampletenant.crt.pem \
--elasticsearch-client-private-key /root/elproxy-verification/conf/certs/private/neteye_ebp_verify_exampletenant.key.pem

Internally, the verify command tries to retrieve the iteration from which to start the verification. Since the BSH file is missing, El Proxy tries to get the first existing log by querying Elasticsearch.

In this example, since the blockchain is new and all its logs are still present, Elasticsearch will return the log with iteration zero as the first log in the blockchain.

El Proxy will then start the verification from the log with iteration zero. If the verification succeeds, a BSH file will be created and stored on the DPO machine.