User Guide Functional Overview Requirements Architecture System Installation NetEye Additional Components Installation Setup The neteye Command Director NetEye Self Monitoring Tornado Business Service Monitoring IT Operation Analytics - Telemetry Geo Maps NagVis Audit Log Shutdown Manager Reporting ntopng Visual Monitoring with Alyvix Elastic Stack IT Operations (Command Orchestrator) Asset Management Service Level Management Cyber Threat Intelligence - SATAYO NetEye Update & Upgrade How To NetEye Extension Packs Troubleshooting Security Policy Glossary
module icon Elastic Stack
Overview Architecture Authorization Elasticsearch Overview Enabling El Proxy Sending custom logs to El Proxy Configuration files Commands Elasticsearch Templates and Retentions El Proxy DLQ Blockchain Verification Handling Blockchain Corruptions El Proxy Metrics El Proxy Security El Proxy REST Endpoints Agents Logstash Elastic APM Elastic RUM Log Manager - Deprecated
ntopng Visual Monitoring with Alyvix Elastic Stack IT Operations (Command Orchestrator) Asset Management Service Level Management Cyber Threat Intelligence - SATAYO Introduction to NetEye Monitoring Business Service Monitoring IT Operation Analytics Visualization Network Visibility Log Management & Security Orchestrated Datacenter Shutdown Application Performance Monitoring User Experience Service Management Service Level Management & Reporting Requirements for a Node Cluster Requirements and Best Practices NetEye Satellite Requirements TCP and UDP Ports Requirements Additional Software Installation Introduction Single Node Cluster NetEye Master Master-Satellite Architecture Underlying Operating System Acquiring NetEye ISO Image Installing ISO Image Single Nodes and Satellites Cluster Nodes Configuration of Tenants Satellite Nodes Only Nodes behind a Proxy Additional NetEye Components Single Node Cluster Node Satellites Nodes only Verify if a module is running correctly Accessing the New Module Cluster Satellite Security Identity and Access Management External Identity Providers Configure federated LDAP/AD Emergency Reset of Keycloak Configuration Advanced Configuration Authorization Resources Tuning Advanced Topics Basic Concepts & Usage Advanced Topics Monitoring Environment Templates Monitored Objects Import Monitored Objects Data Fields Deployment Icinga 2 Agents Configuration Baskets Dashboard Monitoring Status VMD Permissions Notifications Jobs API Configuring Icinga Monitoring Retention Policy NetEye Self Monitoring 3b Concepts Collecting Events Add a Filter Node WHERE Conditions Iterating over Event fields Retrieving Payload of an Event Extract Variables Create a Rule Tornado Actions Test your Configuration Export and Import Configuration Example Under the hood Development Retry Strategy Configuration Thread Pool Configuration API Reference Configure a new Business Process Create your first Business Process Node Importing Processes Operators The ITOA Module Configuring User Permissions Telegraf Metrics in NetEye Telegraf Configuration Telegraf on Monitored Hosts Visualizing Dashboards Customizing Performance Graph The NetEye Geo Map Visualizer Map Viewer Configuring Geo Maps NagVis 3b Audit Log 3b Overview Shutdown Manager user Shutdown Manager GUI Shutdown Commands Advanced Topics Overview User Role Management Cube Use Cases ntopng and NetEye Integration Permissions Retention Advanced Topics Overview User Roles Nodes Test Cases Dashboard Use Cases Overview Architecture Authorization Elasticsearch Overview Enabling El Proxy Sending custom logs to El Proxy Configuration files Commands Elasticsearch Templates and Retentions El Proxy DLQ Blockchain Verification Handling Blockchain Corruptions El Proxy Metrics El Proxy Security El Proxy REST Endpoints Agents Logstash Elastic APM Elastic RUM Log Manager - Deprecated Overview Authorization in the Command Orchestrator Module Configuring CLI Commands Executing Commands Overview Permissions Installation Single Tenancy Multitenancy Communication through a Satellite Asset collection methods Display asset information in monitoring host page Overview Customers Availability Event Adjustment Outages Resource Advanced Topics Introduction Getting Started SATAYO Items Settings Managed Service Mitre Attack Coverage Changelog Before you start Update Procedure Single Node Upgrade from 4.41 to 4.42 Cluster Upgrade from 4.41 to 4.42 Satellite Upgrade from 4.41 to 4.42 DPO machine Upgrade from 4.41 to 4.42 Create a mirror of the RPM repository Sprint Releases Feature Troubleshooting Tornado Networking Service Management - Incident Response IT Operation Analytics - Telemetry Identity Provider (IdP) Configuration Introduction to NEP Getting Started with NEPs Online Resources Obtaining NEP Insights Available Packages Advanced Topics Upgrade to NetEye 4.31 Setup Configure swappiness Restarting Stopped Services Enable stack traces in web UI How to access standard logs Director does not deploy when services assigned to a host have the same name How to enable/disable debug logging Activate Debug Logging for Tornado Modules/Services do not start Sync Rule fails when trying to recreate Icinga object How to disable InfluxDB query logging Managing an Elasticsearch Cluster with a Full Disk Some logs are not indexed in Elasticsearch Elasticsearch is not functioning properly Reporting: Error when opening a report Debugging Logstash file input filter Bugfix Policy Reporting Vulnerabilities Glossary 3b

Handling Blockchain Corruptions

The correct state of a blockchain can be checked by running the verification process, which provides, at the end of its execution, a report about the consistency of the blockchain.

If everything is fine, the verification process will complete successfully, otherwise, the execution report will contain a list of all errors encountered. For example:

--------------------------------------------------------
--------------------------------------------------------
Verify command execution completed.
Blockchain verification process completed with 2 errors.
--------------------------------------------------------
Errors detail:
--------------------------------------------------------
Log verification error 0 details:
 - Error code: Log Verification Bad Iteration
 - Error type: Missing logs
 - Missing logs from iteration 0 (inclusive) to iteration 1 (inclusive)
 - CorruptionId: eyJmcm9tX2l0ZXJhdGlvbiI6MCwidG9faXRlcmF0aW9uIjoxLCJwcmV2aW91c19oYXNoIjpudWxsLCJuZXh0X2hhc2giOiI3YzQyNmU1OWM1ZGQ0MTkwODJkZGQ0ZWY1Mzk3ZDdhMGM5NmJiYjQxNmVkMWNkOTg3OTMyNDFiYjg5NTY4YjBjIiwiYWNrbm93bGVkZ2VfYmxvY2tjaGFpbl9pZCI6ImFja25vd2xlZGdlLWVscHJveHlzaWduZWQtbXljdXN0b21lcjItaW5maW5pdGUtMCJ9
--------------------------------------------------------
Log verification error 1 details:
- Error code: Log Verification Wrong Hash
- Error type: Wrong Log Hash
- Failed iteration_id: 16
- Expected hash: 34f5bd40d5042ba289d4c5032c75a426306a57e41c0703df4c7698df104f75ed
- Found hash   : 593bcd654fd80091105a21f548c1e6a8dd07c80380e72ceeeb1a3e7b126d26bb
- CorruptionId: eyJmcm9tX2l0ZXJhdGlvbiI6MTYsInRvX2l0ZXJhdGlvbiI6MTYsInByZXZpb3VzX2hhc2giOiIwNTE0YmY3YTBmNmRmNmNhMjg1YTIwYTM2OGFiNTA5M2I5NjgxMWZkZWFmMmQ1YThhYjFkOTYwYzgyNDRiNzJlIiwiYWNrbm93bGVkZ2VfYmxvY2tjaGFpbl9pZCI6ImFja25vd2xlZGdlLWVscHJveHlzaWduZWQtbmV0ZXllLW9uZV93ZWVrLTAifQ==
--------------------------------------------------------
--------------------------------------------------------

For more details about possible errors that can be reported by the El Proxy verification and information on the recommended instructions, please review the associated errors table.

Errors different from the Empty Blockchain Corruption error are provided with a CorruptionId that uniquely identifies them. The CorruptionId is a base64 encoded JSON that contains data to identify a specific corruption of a blockchain.

Once the admin has reviewed and investigated the conditions that led to the error, it is possible to fix the corruptions in one of the following ways:

  1. Recover the logs via the dlq recover command, in case the corruption was caused by logs that ended up in DLQ

  2. Acknowledge the corruptions via the acknowledge or acknowledge-range commands

Acknowledging Blockchain Corruptions

The acknowledge or acknowledge-range commands create an acknowledgment for a specific CorruptionId so that, when the verify command is executed next time, the linked error in the blockchain will be considered as resolved.

The acknowledgement data is persisted in a dedicated Elasticsearch index whose name is generated from the name of the corrupted data stream. For example, if the corrupted data stream is *-*-elproxysigned-neteye-one_week-0, then the name of the acknowledged blockchain will be acknowledge-elproxysigned-neteye-one_week-0.

For instance, we can acknowledge the first error reported in the above example by connecting to the tenant specific container on the DPO machine, as described here, and executing the following command:

elastic_blockchain_proxy acknowledge \
--key-file /path/to/secret/key \
--es-auth-method 'pemcertificatepath' \
--es-client-cert '/root/elproxy-verification/conf/certs/ElasticBlockchainProxyAcknowledge.crt.pem' \
--es-client-key '/root/elproxy-verification/conf/certs/private/ElasticBlockchainProxyAcknowledge.key.pem' \
--corruption-id=eyJmcm9tX2l0ZXJhdGlvbiI6MCwidG9faXRlcmF0aW9uIjoxLCJwcmV2aW91c19oYXNoIjpudWxsLCJuZXh0X2hhc2giOiI3YzQyNmU1OWM1ZGQ0MTkwODJkZGQ0ZWY1Mzk3ZDdhMGM5NmJiYjQxNmVkMWNkOTg3OTMyNDFiYjg5NTY4YjBjIiwiYWNrbm93bGVkZ2VfYmxvY2tjaGFpbl9pZCI6ImFja25vd2xlZGdlLWVscHJveHlzaWduZWQtbXljdXN0b21lcjItaW5maW5pdGUtMCJ9=

The acknowledge command is intended to be used for the acknowledgment of a single corruption and it is not optimized for the acknowledgment of multiple corruptions. If we want to acknowledge multiple corruptions, we can use the acknowledge-range command.

For example, assuming all the corruptions reported in the example above occurred from January 1st 2023 to July 31st 2023 (UTC time) on the blockchain of tenant mycustomer, retention 6_months and tag 0, we can run, always from the blockchain specific container on the DPO machine, the following command:

elastic_blockchain_proxy acknowledge-range \
--tenant 'mycustomer' \
--retention '6_months' \
--tag '0' \
--key-file '/path/to/secret/key' \
--from '2023-01-01T00:00:00Z' \
--to '2023-07-31T00:00:00Z' \
--es-read-auth-method 'pemcertificatepath' \
--es-read-client-cert '/root/elproxy-verification/conf/certs/neteye_elproxy_verify_mycustomer.crt.pem' \
--es-read-client-key '/root/elproxy-verification/conf/certs/private/neteye_elproxy_verify_mycustomer.key.pem' \
--es-write-auth-method 'pemcertificatepath' \
--es-write-client-cert '/root/elproxy-verification/conf/certs/ElasticBlockchainProxyAcknowledge.crt.pem' \
--es-write-client-key '/root/elproxy-verification/conf/certs/private/ElasticBlockchainProxyAcknowledge.key.pem'

For more information about the acknowledge and acknowledge-range commands please refer to the El Proxy Commands section.