Backup & Restore¶

Elasticsearch Backup and Restore¶

Elasticsearch provides snapshot functionality which is great for backups because they can be restored relatively quickly.

The main features of Elasticsearch snapshots are:

They are incremental
They can store either individual indices or an entire cluster
They can be stored in a remote repository such as a shared file system

The destination for snapshots must be a shared file system mounted on each Elasticsearch node.

Deleting a snapshot only changes those files that are associated with the deleted snapshot and are not used by any other snapshots. If the deleted snapshot operation is executed while the snapshot is being created, the snapshot process will be aborted and all files created as part of the snapshot process will be removed.

For further details see the Official Elasticsearch snapshot documentation.

Requirements¶

The snapshot module requires the initialization of a repository which contains a reference to a repository path contained in the Elasticsearch configuration file:

/neteye/local/elasticsearch/conf/elasticsearch.yml

This repository, and consequently the destination path for the snapshot, must be initialized manually.

A shared folder must be mounted on each Elasticsearch node at the following path:

/data/backup/elasticsearch

Note

In a cluster environment, all nodes running Elasticsearch must have the same shared folder mounted.

Backup strategy¶

The standard behavior of the Elasticsearch snapshot module is to create incremental backups. You may however want to have full backups in addition to incremental backups. Considering that a full backup is not natively supported, the recommended procedure is to create a new repository for each full backup you need.

Note that in an Elasticsearch cluster installation, all commands must be executed on the Elasticsearch master node. The master node can be retrieved with the following command:

/usr/share/neteye/backup/elasticsearch/elasticsearch-backup -M

The following subsections describe the common operations needed to initialize a repository, and to execute, delete and restore snapshots.

Initialize the default repository¶

The initialization uses the following default mount path:

/data/backup/elasticsearch

In a cluster environment it is mandatory to mount the path on a shared file system for each node:

# The default neteye_log_backup repository will be used
/usr/share/neteye/backup/elasticsearch/elasticsearch-backup -I

Initialize a new repository, or one different from the default¶

If the new repository uses a custom folder, its path must be added to the Elasticsearch configuration file. In particular, the option “path.repo” in the configuration file:

/neteye/local/elasticsearch/conf/elasticsearch.yml

must be an array containing all destination paths for the snapshot. For instance:

path.repo: ["/data/backup/elasticsearch", "/data/full_backup/"]

Note that if you change the Elasticsearch configuration file, you must restart it:

systemctl restart elasticsearch

You can create a new repository with the name “my_repo” and a custom backup path with this script (if the -r option is not specified, the default neteye_log_backup will be used):

/usr/share/neteye/backup/elasticsearch/elasticsearch-backup -r "my_repo" -i /data/full_backup/

Take a snapshot¶

When using a default name: snapshot-Year-Month-Day-Hour:Minute:Second

/usr/share/neteye/backup/elasticsearch/elasticsearch-backup -s

When using a custom name (in this example, “test-snapshot”):

/usr/share/neteye/backup/elasticsearch/elasticsearch-backup -S test-snapshot

Delete a snapshot¶

You can delete one or more snapshots with a regex. In the example here, only the snapshot with the name “test-snapshot” will be removed:

/usr/share/neteye/backup/elasticsearch/elasticsearch-backup -d "test-snapshot"

You can also delete any snapshots that are older than the specified period using the format YY.MM.DD.HH.MM (e.g., 0.1.1.0.0 means 31 days). For more details, see the description of unit.

/usr/share/neteye/backup/elasticsearch/elasticsearch-backup -c 0.1.1.0.0

These two options can be combined, for instance to delete all snapshots that contain “test” in the name and that are older than 1 minute:

/usr/share/neteye/backup/elasticsearch/elasticsearch-backup -d test -C 0.0.0.0.1

Create a full snapshot¶

A new repository (see the previous section) or an empty repository must be used.

/usr/share/neteye/backup/elasticsearch/elasticsearch-backup -r "my_full_backup" -s

Restoring a snapshot¶

Restoring a snapshot requires a configuration file that describes the process. Please see the official guide for more details. We have provided three example configurations in the following folder::

/usr/share/neteye/backup/elasticsearch/conf

These can be invoked with the following script:

/usr/share/neteye/backup/elasticsearch/elasticsearch-restore -c <absolute-config-file-path>

Restore the last snapshot¶

Restore all indices in the most recent snapshot.

actions:
  1:
    action: restore
    options:
      # May be changed according to your setup
      repository: neteye_log_backup
      # If the name is blank, the most recent snapshot by age will be selected
      name:
      # If the indices are blank, all indices in the snapshot will be restored
      indices:
      include_aliases: False
      ignore_unavailable: False
      include_global_state: False
      partial: False
      wait_for_completion: True
      filters:
      filtertype: none

Restore some indices¶

Restore indices with the name provided in indices in the most recent snapshot with state SUCCESS. The indices option supports multiple indices syntax.

In the following example, all the indices starting with “test-” will be restored.

actions:
  1:
    action: restore
    description:
    options:
      # May be changed according to your setup
      repository: neteye_log_backup
      # If the name is blank, the most recent snapshot by age will be selected
      name:
      indices: [test-*]
      include_aliases: False
      ignore_unavailable: False
      include_global_state: False
      partial: False
      filters:
      filtertype: state
      state: SUCCESS

Restore Snapshot Renaming¶

Restore all indices in the most recent snapshot by: - Finding any indices being restored that match the rename_pattern. - Changing the name as described in rename_replacement.

The following example will restore all indices which start with “index_”, but rename it to “restored_index_”. E.g., If you have “index_1”, this will restore “index_1”, but rename it to “restored_index_1”. For additional information, see the documentation.

actions:
  1:
    action: restore
    options:
      # May be changed according to your setup
      repository: neteye_log_backup
      # If the name is blank, the most recent snapshot by age will be selected
      name:
      # If the indices are blank, all indices in the snapshot will be restored
      indices:
      include_aliases: False
      ignore_unavailable: False
      include_global_state: False
      partial: False
      "rename_pattern": "index_(.*)"
      "rename_replacement": "restored_index_$1"
      extra_settings:
      wait_for_completion: True
    filters:
      filtertype: none