operator migrate

The operator migrate command copies data between storage backends to facilitate migrating Vault between configurations. It operates directly at the storage level, with no decryption involved. Keys in the destination storage backend will be overwritten, and the destination should not be initialized prior to the migrate operation. The source data is not modified, with the exception of a small lock key added during migration.

This is intended to be an offline operation to ensure data consistency, and Vault will not allow starting the server if a migration is in progress.

Examples

Migrate all keys:

$ vault operator migrate -config migrate.hcl 2018-09-20T14:23:23.656-0700 [INFO ] copied key: data/core/seal-config2018-09-20T14:23:23.657-0700 [INFO ] copied key: data/core/wrapping/jwtkey2018-09-20T14:23:23.658-0700 [INFO ] copied key: data/logical/fd1bed89-ffc4-d631-00dd-0696c9f930c6/31c8e6d9-2a17-d98f-bdf1-aa868afa1291/archive/metadata2018-09-20T14:23:23.660-0700 [INFO ] copied key: data/logical/fd1bed89-ffc4-d631-00dd-0696c9f930c6/31c8e6d9-2a17-d98f-bdf1-aa868afa1291/metadata/5kKFZ4YnzgNfy9UcWOzxxzOMpqlp61rYuq6laqpLQDnB3RawKpqi7yBTrawj1P...

Migration is done in a consistent, sorted order. If the migration is halted or exits before completion (e.g. due to a connection error with a storage backend), it may be resumed from an arbitrary key prefix:

$ vault operator migrate -config migrate.hcl -start "data/logical/fd"

Configuration

The operator migrate command uses a dedicated configuration file to specify the source and destination storage backends. The format of the storage stanzas is identical to that used to configure Vault, with the only difference being that two stanzas are required: storage_source and storage_destination.

storage_source "mysql" {  username = "user1234"  password = "secret123!"  database = "vault"} storage_destination "consul" {  address = "127.0.0.1:8500"  path    = "vault/"}

Migrating to integrated raft storage

Example configuration

The below configuration will migrate away from Consul storage to integrated raft storage. The raft data will be stored on the local filesystem in the defined path. node_id can optionally be set to identify this node. cluster_addr must be set to the cluster hostname of this node. For more configuration options see the raft storage configuration documentation.

If the original configuration uses "raft" for ha_storage a different path needs to be declared for the path in storage_destination and the new configuration for the node post-migration.

storage_source "consul" {  address = "127.0.0.1:8500"  path    = "vault"} storage_destination "raft" {  path = "/path/to/raft/data"  node_id = "raft_node_1"}cluster_addr = "http://127.0.0.1:8201"

Run the migration

Vault will need to be offline during the migration process. First, stop Vault. Then, run the migration on the server you wish to become a the new Vault node.

$ vault operator migrate -config migrate.hcl 2018-09-20T14:23:23.656-0700 [INFO ] copied key: data/core/seal-config2018-09-20T14:23:23.657-0700 [INFO ] copied key: data/core/wrapping/jwtkey2018-09-20T14:23:23.658-0700 [INFO ] copied key: data/logical/fd1bed89-ffc4-d631-00dd-0696c9f930c6/31c8e6d9-2a17-d98f-bdf1-aa868afa1291/archive/metadata2018-09-20T14:23:23.660-0700 [INFO ] copied key: data/logical/fd1bed89-ffc4-d631-00dd-0696c9f930c6/31c8e6d9-2a17-d98f-bdf1-aa868afa1291/metadata/5kKFZ4YnzgNfy9UcWOzxxzOMpqlp61rYuq6laqpLQDnB3RawKpqi7yBTrawj1P...

After migration has completed, the data is stored on the local file system. To use the new storage backend with Vault, update Vault's configuration file as described in the raft storage configuration documentation. Then start and unseal the vault server.

Join additional nodes

After migration the raft cluster will only have a single node. Additional peers should be joined to this node.

If the cluster was previously HA-enabled using "raft" as the ha_storage, the nodes will have to re-join to the migrated node before unsealing.

Usage

The following flags are available for the operator migrate command.

• -config (string: <required>) - Path to the migration configuration file.

• -start (string: "") - Migration starting key prefix. Only keys at or after this value will be copied.

• -reset - Reset the migration lock. A lock file is added during migration to prevent starting the Vault server or another migration. The -reset option can be used to remove a stale lock file if present.

• -max-parallel int: 10 - Allows the operator to specify the maximum number of lightweight threads (goroutines) which may be used to migrate data in parallel. This can potentially speed up migration on slower backends at the cost of more resources (e.g. CPU, memory). Permitted values range from 1 (synchronous) to the maximum value for an integer. If not supplied, a default of 10 parallel goroutines will be used.

  Note: The maximum number of concurrent requests handled by a storage backend is ultimately governed by the storage backend configuration setting, which enforces a maximum number of concurrent requests (max_parallel).