Advanced Topics¶

Elasticsearch security helper tool¶

The secure communication provided by the X-Pack Security requires additional parameters such as authentication certificates to interact with the Elastic Stack APIs. We have developed a few helper tools, based on curl, to simplify your interaction with the APIs.

The Elasticsearch helper script lets you omit all the authentication parameters for the admin user, which would otherwise be required.

Location: /usr/share/neteye/elasticsearch/scripts/es_curl.sh

The NetEye helper script can be used instead if you only need read permission for the fields @timestamp and host on the Logstash index entries. This script is used by NetEye for self-monitoring activities.

Location: /usr/share/neteye/elasticsearch/scripts/es_neteye_curl.sh

Elasticsearch System indices on Single Nodes¶

Elasticsearch System indices are a type of indices meant to be used only internally by the components of the Elastic Stack. They may be associated with some System template which specifies their settings.

Some System templates may require System indices to have at least one replica, which in simple terms means that the index must be replicated on more than one node (note that this may apply also for non-System templates and indices). While having replicas is straightforward on a NetEye Cluster, on Single Node installations this is not possible, which causes these indices to go in yellow state due to the so-called unassigned shards warning.

Since on Single Node installations this warning cannot be solved by assigning shards to other nodes, the solution to the problem is to tell Elasticsearch that non-replicated indices should be allowed.

NetEye applies this solution by default during the neteye_secure_install on the templates and indices which have this problem. However, some templates and indices are created only when some features of the Elastic Stack are triggered by the user and must be manually fixed by the Elastic Stack administrator.

To facilitate the job of admin though, NetEye provides a script that given the name of an Elasticsearch template, modifies the settings of the template itself and of all the matching indices to allow zero replicas on Single Node installations.

This goal is achieved by the script by setting the option index.auto_expand_replicas to 0-1.

Suppose you identified that the Elasticsearch template that causes the index to have the number of replicas set to one (or more) is named .siem-signals-default. You can then call the script as follows:

neteye# bash /usr/share/neteye/elasticsearch/scripts/elasticsearch_set_autoexpand_replicas_to_index_templates_and_indexes.sh ".siem-signals-default"

Note

The script works only with the composable index templates introduced in Elasticsearch 7.8 and does not support the Legacy index templates.

Moreover, the script supports the update of multiple Index Templates at once. To perform such operation simply pass the multiple Index Templates names are arguments, like this:

neteye# bash /usr/share/neteye/elasticsearch/scripts/elasticsearch_set_autoexpand_replicas_to_index_templates_and_indexes.sh <index_template_name_1> <index_template_name_2> <index_template_name_3>

Kibana Keystore Usage¶

The Kibana Keystore feature comes with a keybana-keystore tool, which permits to manage the settings in the keystore.

If your installation is a NetEye Cluster, you are advised to use kibana-keystore tool only from the cluster nodes where the Kibana resource is active.

Using the keybana-keystore tool from nodes where Kibana is not running will have no effect on the Kibana Keystore configuration.

El Proxy Security¶

The process of log management provided within NetEye grants data integrity and inalterability due to the feature implementation methods, in order to comply with standard NetEye policies.

El Proxy Security section should serve as a proof of El Proxy being a reliable tool for signing your logs and safely sending them to Elasticsearch. The implementation of El Proxy encounters risks mitigations, so that the data breaches are preventable, or on the other hand, traceable.

The main aspects of security within mentioned software are covered in:

Secured methods of communication and authentication
Log signature flow, explaining the logic of how El Poxy actually works
Error handling strategies El Proxy uses to recover after failures
Healthchecks that are run to make sure log collecting process wasn’t error-prone

Additionally, this section highlights some details of the configuration and can be used to modify the setup if required.

Secure Communication¶

When installed on NetEye, the El Proxy automatically starts in secure mode using TLS. Additionally, authentication with Elasticsearch is protected by certificates.

TLS Configuration¶

Advanced users should be able to check the location of the configuration files or modify the setup.

The El Proxy server can start in HTTP or HTTPS mode; this is configured in the config web_server.tls section.

The available modes are:

None: The El Proxy server starts with TLS disabled. Example:
```
[web_server.tls]
type = "None"
```
PemCertificatePath: The El Proxy server starts with TLS enabled using the PEM certificates read from the local file system. When this method is used, the following information must be provided:
- certificate_path: path to the server public certificate
- private_key_path: path to the server private key
Example:
```
[web_server.tls]
type = "PemCertificatePath"
certificate_path = "/path/to/certs/ebp_server.crt.pem"
private_key_path = "/path/to/certs/private/ebp_server.key.pem"
```

Authentication to Elasticsearch¶

When the Elasticsearch client is created, the authentication method to be used to connect to Elasticsearch needs to be specified. The authentication method defined in the configuration file is only used for the serve command.

The available authentication methods are:

None: the client connects to Elasticsearch without authentication. Example:
```
[elasticsearch.auth]
type = "None"
```
BasicAuth: the client authenticates to Elasticsearch with username and password. When this method is used, the following information must be provided:
- username: name of the Elasticsearch user
- password: the password for the Elasticsearch user
```
[elasticsearch.auth]
type = "BasicAuth",
username = "myuser",
password = "mypassword"
```
PemCertificatePath: the client connects to Elasticsearch using the PEM certificates read from the local file system. When this method is used, the following information must be provided:
- certificate_path: path to the public certificate accepted by Elasticsearch
- private_key_path: path to the corresponding private key
Example:
```
[elasticsearch.auth]
type = "PemCertificatePath",
certificate_path = "/path/to/certs/ebp.crt.pem",
private_key_path = "/path/to/certs/private/ebp.key.pem",
```

Log Signature Flow¶

The El Proxy achieves secure logging by authentically encrypting each log record with an individual cryptographic key used only once and protects the integrity of the whole log archive by a cryptographic authentication code.

Generation of Signature Keys¶

The encryption keys are generated on-the-fly when needed and saved on the filesystem in the {data_dir} folder. Each key is bound to a specific customer, module, retention policy and blockchain tag; they are saved in the filesystem with the following naming convention: {data_dir}/{customer}/{module}-{customer}-{retention_policy}-{blockchain_tag}_key.json

When a new key is generated, a copy of the key file is created in the {data_backup_dir} folder with the same naming convention.

As soon as a new key file is created in the {data_backup_dir} folder, the Icinga2 service logmanager-blockchain-keys-neteyelocal will enter in CRITICAL state, indicating that a new key has been generated on the system. The new key must be moved in a safe place such as a password manager or a restricted access storage.

The content of the key file is in the form:

{
  "key": "initial_key",
  "iteration": 0
}

Where:

key is the encryption key to be used to sign the next incoming log
iteration: is the iteration number for which the signature key is valid.

Every time a log is signed, a new pair key/iteration is generated starting from the last one. The new pair will have the following values:

key equals to the SHA256 hash of the previous key
iteration equals to the previous iteration incremented by one

For example, if the key at iteration 10 is:

{
  "key": "abcdefghilmno",
  "iteration": 10
}

the next key will be:

{
  "key": "d1bf0c925ec44e073f18df0d70857be56578f43f6c150f119e931e85a3ae5cb4",
  "iteration": 11
}

This mechanism creates a blockchain of keys that cannot be altered without breaking the validity of the entire chain.

In addition, every time a set of logs is successfully sent to Elasticsearch, the specific key file is updated with the new value. Consequently, every reference to the previous keys is removed from the system making it impossible to recover and reuse old keys.

However, in case of unmanageable Elasticsearch errors, the El Proxy will reply with an error message to Logstash and will reuse the keys of the failed call for the next incoming logs.

Note

To be valid, the iteration values of signed logs in Elasticsearch should be incremental with no missing or duplicated values.

When the first log is received after its startup, the El Proxy calls Elasticsearch to query for the last indexed log iteration value to determine the correct iteration number for the next log. If the last log iteration value returned from Elasticsearch is greater than the value stored in the key file, the El Proxy will fail to process the log.

Usage of Signature Keys¶

For each incoming log, the El Proxy retrieves the first available encryption key, as described in the previous section, and then uses it to calculate the HMAC-SHA256 hash of the log.

The calculation of the HMAC hash takes into account:

the log itself as received from Logstash
the iteration number
the timestamp
the hash of the previous log

At this point, the signed log is a simple JSON object composed by the following fields:

All fields of the original log : all fields from the original log message
ES_BLOCKCHAIN: an object containing all the El Proxy’s calculated values. They are:
- fields: fields of the original log used by the signature process
- hash: the hmac hash calculate as described before
- previous_hash: the hmac hash of the previous log message
- iteration: the iteration number of the signature key
- timestamp_ms: the signature epoch timestamp in milliseconds

For example, given this key:

{
  "key": "d1bf0c925ec44e073f18df0d70857be56578f43f6c150f119e931e85a3ae5cb4",
  "iteration": 11
}

when this log is received:

{
   "value": "A log message",
   "origin": "linux-apache2",
   "EBP_METADATA": {
      "agent": {
         "type": "auditbeat",
         "version": "7.10.1"
      },
      "customer": "neteye",
      "retention": "6_months",
      "blockchain_tag": "0",
      "event": {
         "module": "elproxysigned"
      }
   }
}

then this signed log will be generated:

{
   "value": "A log message",
   "origin": "linux-apache2",
   "EBP_METADATA": {
      "agent": {
         "type": "auditbeat",
         "version": "7.10.1"
      },
      "customer": "neteye",
      "retention": "6_months",
      "blockchain_tag": "0",
      "event": {
         "module": "elproxysigned"
      }
   },
   "ES_BLOCKCHAIN": {
      "fields": {
         "value": "A log message",
         "origin": "linux-apache2"
      },
      "hash": "HASH_OF_THE_CURRENT_LOG",
      "previous_hash": "HASH_OF_THE_PREVIOUS_LOG",
      "iteration": 11,
      "timestamp_ms": 123456789
    }
}

The diagram shown in Fig. 205 offers a detailed view on how El Proxy uses the Signature Keys to sign a batch of Logs.

../_images/log-signing.jpg — Fig. 205 El Proxy Log Signing flowchart¶

Below you can find the meaning of the variables used in the flowchart.

SIG_KEY(b)
The key used to sign the logs of a given Blockchain b.

ES_LAST_LOG(b)
The log with greatest iteration saved in Elasticsearch for the Blockchain b.

How the index name is determined¶

The name of the Elasticsearch index for the signed logs is determined by the content of the EBP_METADATA field of the incoming Log.

The index name has the following structure:

{EBP_METADATA.agent.type}-{EBP_METADATA.agent.version}-{EBP_METADATA.event.module}-{EBP_METADATA.customer}-{EBP_METADATA.retention}-{EBP_METADATA.blockchain_tag}-YYYY.MM.DD

The following rules and constraints are valid:

All of these fields are mandatory:
- EBP_METADATA.agent.type
- EBP_METADATA.customer
- EBP_METADATA.retention
- EBP_METADATA.blockchain_tag
The YYYY.MM.DD part of the index name is based on the epoch timestamp of the signature
If the {EBP_METADATA.event.module} is not present, El Proxy will use by default elproxysigned

For example, given this log is received on 23 March, 2021:

{
   "value": "A log message",
   "origin": "linux-apache2",
   "EBP_METADATA": {
      "agent": {
         "type": "auditbeat",
         "version": "7.10.1"
      },
      "customer": "neteye",
      "retention": "6_months",
      "blockchain_tag": "0",
      "event": {
         "module": "elproxysigned"
      }
   }
}

Then the inferred index name is: auditbeat-7.10.1-elproxysigned-neteye-6_months-0-2021.03.23

As a consequence of the default values and of the default Logstash configuration, most of the indexes created by El Proxy will have elproxysigned in the name. Consequently, special care should be applied when manipulating those indexes and documents; in particular, the user must not delete or rename *-elproxysigned-* indices manually nor alter the content of ES_BLOCKCHAIN or EBP_METADATA fields as any change could lead to a broken blockchain.

How Elasticsearch Ingest Pipeline are defined for each log¶

A common use case is that logs going through El Proxy need to be enriched by some Elasticsearch Ingest Pipeline when they are indexed in Elasticsearch.

El Proxy supports this use case and allows its callers to specify, for each log, the ID of Ingest Pipeline that needs to enrich the log. The ID of the Ingest Pipeline is defined by the field EBP_METADATA.pipeline of the incoming log. If EBP_METADATA.pipeline is left empty, the log will not be preprocessed by any specific Ingest Pipeline.

Error Handling¶

El Proxy implements two different recovery strategies in case of errors. The first one is a simple retry strategy used in case of unrecoverable communication errors with Elasticsearch; the second one, instead, is used in case Elasticsearch has issues with processing some of the sent logs and aims at addressing a widely known issue named backpressure.

Figure Fig. 206 shows how El Proxy integrates the two aforementioned strategies, which will be thoroughly explained in the next paragraphs.

../_images/error-handling.jpg — Fig. 206 Error Handling in El Proxy¶

Backpressure explained

In the context of data pipelines, backpressure represents the situation where a resistance prevents a fast processing of the input data. In the long term this resistance causes the overload of the system, because inside a certain component of the system data arrive at a higher rate with respect to the output rate.

In the context of the El Proxy, the resistance is represented by Elasticsearch refusing to index some specific logs because, for example, some unexpected fields inside a log lead to a failure in the Elasticsearch pipelines. This would cause El Proxy and Logstash to keep trying to index these logs without ever succeeding.

The way El Proxy overcomes the backpressure is by:

trying to modify the failing logs to avoid the indexing error
writing these logs into a Dead Letter Queue, as explained in the Strategy two section

By performing these actions El Proxy can quickly process also the logs which cannot be indexed in Elasticsearch and so it guarantees that these logs will not cause the backpressure.

Strategy One: Bulk retry¶

El Proxy implements an optional retry strategy to handle communication errors with Elasticsearch; when enabled (see the Configuration section), whenever a generic error is returned by Elasticsearch, El Proxy will retry for a fixed amount of times to resubmit the same signed_logs to Elasticsearch.

This strategy permits to deal, for example, with temporary networking issues without forcing Logstash to resubmit the logs for a new from-scratch processing.

Nevertheless, while this can be useful in dealing with a large set of use cases, it should also be used very carefully. In fact, due to the completely sequential nature of the blockchain processing, a too high number of retries could lead to an ever growing queue of logs waiting to be processed while El Proxy is busy with processing over and over again the same failed logs.

In conclusion, whether it is better to let El Proxy fail fast or retry more times is a decision that needs a careful, case-by-case analysis.

Strategy Two: Single Log Reprocessing and Dead Letter Queue¶

In some cases, Elasticsearch can correctly process the log bulk request but it can fail to index some of the contained logs. In this situation, El Proxy reprocesses only those failed logs and follows a procedure aimed at ensuring that each signed log will be indexed in Elasticsearch.

The procedure is the following:

If the log indexing error is caused by the fact that the Elasticsearch Ingest Pipeline specified for the log does not exist in Elasticsearch, then:
1. El Proxy tries to index again the log without specifying any Ingest Pipeline. When doing this, El Proxy will also add the tag _ebp_remove_pipeline to the Elasticsearch document, so that once the log is indexed, it will be visible that the Ingest Pipeline was bypassed during the document indexing.
2. If the indexing fails again, El Proxy proceeds with step 2 of this procedure.
El Proxy removes from the failed log all the fields not required by the signature process and sends the modified document to Elasticsearch. Note that also all the document tags (including the _ebp_remove_pipeline tag) will be removed from the document.

This strategy is based on the assumption that the indexing of the specific log fails due to incompatible operations requested by a pipeline. For example, a pipeline could attempt to lowercase a non-string field causing the failure. Consequently, resubmitting the log without the problematic fields could lead to a successful indexing.

If after following this procedure some logs are still not indexed in Elasticsearch, then El Proxy will dump the failed logs to a Dead Letter Queue on the filesystem and it will send an ok response to Logstash. Usually, when this happens, the blockchain will be in an incoherent state caused by holes in the numeric progression iteration. The administrator is in charge of investigating the issue and, if possible, recover the non-indexed logs with the dlq recover command or manually acknowledge the problem.

As mentioned before, the Dead Letter Queue is saved on the filesystem inside the {dlq_dir} folder as set in the configuration file. The Dead Letter Queue consists of a set of text files in Newline delimited JSON format with each row in a file representing a failed log. These files are grouped by customer and index name following the naming convention:

{dlq_dir}/{customer}/{index_name}.ndjson

where {index_name} is the name of the Elasticsearch index where the failed logs were supposed to be written.

If the dlq recover command is used to recover the logs, the tag _ebp_dlq_recovered is added to each log at the moment of indexing in Elasticsearch. The successfully recovered logs are moved inside the {dlq_recovered} folder as set in the configuration file. The logs that failed to be recovered remain unchanged in the {dlq_dir} folder.

If the administrator wants to move the logs from the {dlq_dir} manually, the convention is to move them inside a folder called {archive_dlq}. The parent folder of each NDJSON file should not be changed. So the naming convention for a manually moved file should be:

{dlq_recovered}/{customer}/{index_name}.ndjson

Each entry in the Dead Letter Queue file contains the original log whose indexation failed and, if present, the Elasticsearch error that caused the failure. For example:

{
   "document":  {
      "value": "A log message",
      "origin": "linux-apache2",
      "EBP_METADATA": {
         "agent": {
            "type": "auditbeat",
            "version": "7.10.1"
         },
         "customer": "neteye",
         "retention": "6_months",
         "blockchain_tag": "0",
         "event": {
            "module": "elproxysigned"
         }
      },
      "ES_BLOCKCHAIN": {
         "fields": {
            "value": "A log message",
            "origin": "linux-apache2"
         },
         "hash": "HASH_OF_THE_CURRENT_LOG",
         "previous_hash": "HASH_OF_THE_PREVIOUS_LOG",
         "iteration": 11,
         "timestamp_ms": 123456789
      }
   },
   "el_error": {
         "reason": "field [string_field] of type [java.lang.Integer] cannot be cast to [java.lang.String]",
         "type": "illegal_argument_exception"
   }
}

Health Checks¶

To make sure that the blockchain is sound and no errors occurred during the collection of the logs, NetEye defines different Health Checks to alert the users of any irregularities that may occur.

logmanager-blockchain-creation-status : Makes sure that all logs are written correctly into the blockchain.

If this Health Check is CRITICAL, it means that some logs could not be written successfully to Elasticsearch. The logs in question are then written to the Dead Letter Queue, where you can also find the reason for which they could not be indexed in Elasticsearch.

To resolve this issue visit section Handling Logs in Dead Letter Queue

logmanager-blockchain-keys : Verifies that no backup of the Signature Key is present on the NetEye installation, to prevent any tampering on the blockchains from a compromised system.

Background and solution for this issue can be found in Log Signature Flow.

logmanager-blockchain-missing-elasticsearch-pipeline : Checks if any Elasticsearch ingest pipeline used to enrich logs is missing.

If a log was sent through Logstash with a non-existing pipeline, Elasticsearch will refuse to persist the log and return with an error. As seen in Error Handling, the Health Check then periodically queries Elasticsearch for logs with a corresponding tag and if found, set the status to CRITICAL.

To resolve this, remove the tags from the document in Elasticsearch.

Error Tags¶

If an error occurs, El Proxy may add a tag to mark the log that caused the problem.

Currently, EL Proxy uses three tags to indicate such errors:

_ebp_remove_pipeline : Will be added as seen in this figure
_ebp_skip_previous_hash : Will be added to a log if the previous log hash was not available. You can have a look at Fig. 205 to understand the circumstances under which this can happen.
_ebp_dlq_recovered: Will be added to a log if it was recovered from the Dead Letter Queue with the dlq recover command.

Blockchain Verification¶

In order to ensure the underlying blockchain was not altered or corrupted, you can use verify subcommand provided by El Proxy. Prior to running the command, you need to perform the verification setup.

The command retrieves signature data of each log stored in the Elasticsearch blockchain, recalculates the hash of each log and compares it with the one stored in the signature, reporting a CorruptionId for all non-matching logs of the blockchain. The CorruptionId can then also be used with the acknowledge command, as explained in section Acknowledging Corruptions of a Blockchain.

By default, the command verifies always two batches in parallel, however the argument --concurrent-batches allows the user to increase the amount of parallel workers.

The flowchart depicted in figure Fig. 207 provides a detailed view of the operations performed during the verification process and the reporting of discovered corruptions.

../_images/blockchain-verification.svg — Fig. 207 El Proxy verification process Overview¶

For more information on the verify subcommand and its parameters, please consult the associated configuration section.

Verification and Retention Policies

The blockchain stored in Elasticsearch is subject to a specific retention policy which defines how long the logs will be kept. When the logs reach their maximum age, Elasticsearch deletes them. If the logs inside the blockchain get deleted as an effect of the retention policy, it is impossible to verify the blockchain from start to end. For this reason, El Proxy must consider the retention policy and thus start verifying the blockchain from the first present log.

El Proxy uses a so-called Blockchain State History (BSH) file to retrieve the first present log inside the blockchain. This file is updated on every successful verification and contains one entry for each day that specifies the first and last iteration of the indexed logs for that day.

When the verify command is executed, El Proxy fetches the BSH file searching for the entry corresponding to the first day that still contains all the logs at the moment of the verification. If the entry is present, El Proxy can retrieve the first iteration for that day and start the verification from the iteration specified in the entry. If the entry is not present or El Proxy cannot get the first iteration from the BSH, then El Proxy directly queries Elasticsearch to retrieve the first present log.

Note that although querying Elasticsearch is an option, the only way for El Proxy to verify the blockchain’s integrity is to retrieve the first iteration from the BSH. For this reason, if the first log is retrieved by querying Elasticsearch, the verify command will throw a warning after the successful verification. For more information about warnings and errors that could appear during the verification, please consult associated section.

Verification with Retries

The elasticsearch-indexing-delay parameter of the verify command can help when the Blockchain subject to verification contains recently created logs. The reason is that Elasticsearch might take some time to index a document; therefore, El Proxy could be trying to verify a batch in which some logs are missing. In order to avoid the erroneous failure of the verification, if El Proxy detects that some logs could be missing because of an indexing delay of Elasticsearch, it repeats the whole batch verification. The parameter elasticsearch-indexing-delay defines the maximum allowed time in seconds for Elasticsearch to index a document. If Elasticsearch takes more than elasticsearch-indexing-delay seconds to index a log, the verification will fail (see El Proxy Configuration). If we name timestamp_next_log the timestamp of the next present log and timestamp_verification the timestamp of the log verification, El Proxy considers a log as missing due to Elasticsearch indexing delay whenever timestamp_next_log > timestamp_verification - elasticsearch-indexing-delay. It is worth noticing that El Proxy will always retry the batch verification for a finite amount of time, corresponding to elasticsearch-indexing-delay seconds in the worst case.

El Proxy Verification¶

In El Proxy, the verify command can be used to understand if the underlying blockchain has been corrupted. In the remainder, some advanced aspects of the verification are outlined.

First Iteration Retrieval¶

During the verification process, one of the first steps performed by El Proxy is the retrieval of the first expected iteration. This aspect is particularly important when considering the retention policies, since the first expected iteration needs to be calculated based on the logs that we still expect to find in Elasticsearch. The following diagram outlines the first steps of the verification process which involve the retrieval of the first iteration and the various errors or warnings raised.

../_images/blockchain-verification-first-iteration.png — Fig. 208 El Proxy verification process - First iteration retrieval.¶

(*) for more information please refer to the following Must Exists vs May Exist section

Must Exist vs May Exist

When retrieving the first expected iteration, we distinguish two possible cases:

El Proxy must find the first expected iteration in the blockchain to complete the verification successfully (Must Exist)

El Proxy may find the first expected iteration in the blockchain to complete the verification successfully (May Exist)

The difference between these two alternatives resides in how El Proxy is able to calculate the first expected iteration, taking into account that not all days have indexed logs. Given the first date within the retention period, El Proxy calculates the first expected iteration as follows:

It extracts the iteration of the first log indexed on the first date within the retention period. If this is not present because on that particular day no logs were indexed, El Proxy checks the following days, until the end of the blockchain, to understand which is the first indexed log within the retention and expects it to be present (Must Exist)

If no logs are found from the first date within the retention, El Proxy checks the previous days and it considers the first expected iteration to be the one of the last index log plus 1. In this case, El Proxy cannot be sure that a log with this specific iteration will be found, since no more logs could have been indexed after a specific date past to the first date within the retention (May Exist).

Warnings and Error Codes¶

The verification process may complete successfully, emit an error or a warning. Errors thrown by El Proxy lead to a failed verification, while warnings may appear also on successful completion.

The following table reports the warnings that could be reported by El Proxy during the verification process, along with the possible causes and actions that can be taken to address the issue.

Warning Code	Description	Actions
Untrusted First Iteration No State History	The first iteration found on the blockchain cannot be trusted because it differs from iteration zero, and the blockchain state history file was not found. Thus, El Proxy cannot confirm that the iterations preceding the one that was found are missing. After the first successful blockchain verification, the blockchain state history file will be created, and the warning will disappear. For more info on this topic, refer to the Verification and Retention Policies section.	Manually investigate the blockchain to confirm that all previous iterations are expected to be missing and re-run the verification.

Untrusted First Iteration Insufficient Info

The first iteration found on the blockchain cannot be trusted because it differs from iteration zero, and the blockchain state history file does not contain enough information to assess the expected first iteration of the blockchain. This warning is thrown when the time between two consecutive verifications exceeds the retention period of the Elasticsearch indices where the blockchain is stored, thus leading to an outdated blockchain state history file.

Reduce the time between consecutive verifications by increasing the verification frequency. For more info on how to modify the verification frequency, refer to the NetEye Setup section.

Unknown First Iteration

The blockchain is empty, and the blockchain state history file does not contain any information about the expected first iteration. This warning is thrown on two occasions:

the blockchain is new and no logs have been indexed yet
all logs have been deleted from the blockchain before any successful verification

Ensure that the blockchain is empty because it is new, and wait for the first log to be indexed before running the verification again.

Empty Blockchain

The first expected iteration and all the subsequent iterations were not found in the blockchain. This warning differs from the Unknown First Iteration warning. In this case, the first expected iteration was retrieved from the blockchain state history file meaning that the blockchain could contain some logs with iterations preceding the retrieved first expected iteration. This warning can be thrown for two reasons:

no logs have been indexed on the blockchain for at least the amount of time corresponding to the retention period of the blockchain
all the logs that were indexed after the last successful verification have been deleted

Ensure that the absence of newly indexed logs in the blockchain is expected, and wait for some logs to be indexed before running the verification again.

Retention Policy Not Applied

The blockchain contains logs that are older than the maximum expected age based on the retention policy of the blockchain. This can happen if the retention policy was not correctly applied to the blockchain or if some logs have been re-indexed after their deletion by the retention policy.

Ensure that no logs have been re-indexed after their deletion by the retention policy.

Similarly, the following table reports the errors that could be reported by El Proxy during the verification process, along with the possible causes and actions that can be taken to address the issue.

Error Code	Description	Actions
Empty Blockchain Corruption	The blockchain is empty, and the first expected iteration was not found, leading to a failed verification. El Proxy expected the first iteration based on the blockchain state history file written during the last successful blockchain verification. This error can be thrown if the blockchain state history file is corrupted or if all the logs have been deleted from the blockchain.	Ensure that the blockchain state history file has not been corrupted, then manually investigate the cause of the missing logs.

Log Verification Bad Iteration

The iteration of a log is wrong. This error could be thrown in case one or several consecutive logs are missing, or if one or several logs iterations have been modified.

Ensure to investigate the problem’s cause and acknowledge the error’s corruption id with the elastic-blockchain-proxy acknowledge command. Then run the verification again.

Log Verification Wrong Previous Hash

The previous hash of a log does not match the hash of the previous log. This error can be thrown if the log’s previous hash was modified.

Same as above.

Log Verification Wrong Hash

The hash of a log does not match the hash corresponding to its content. This error can be thrown if the log’s content is modified.

Same as above.

Acknowledgement Verification Wrong Previous Hash

The previous hash of the acknowledgement for a corruption ID does not match the hash of the log right before the corrupted log. This error can be thrown if the previous hash of the acknowledgement has been modified.

Delete the acknowledgement and run the verification again.

Acknowledgement Verification Wrong Hash

The hash of the acknowledgement for a corruption ID does not match the expected hash. This error can be thrown if the acknowledgement content has been modified.

Same as above.