Auto-unseal using Azure Key Vault

When a Vault server is started, it starts in a sealed state and it does not know how to decrypt data. Before any operation can be performed on the Vault, it must be unsealed. Unsealing is the process of constructing the master key necessary to decrypt the data encryption key.

Challenge

Vault unseal operation requires a quorum of existing unseal keys split by Shamir's Secret sharing algorithm. This is done so that the "keys to the kingdom" won't fall into one person's hand. However, this process is manual and can become painful when you have many Vault clusters as there are now many different key holders with many different keys.

Solution

Vault supports opt-in automatic unsealing via cloud technologies: AliCloud KMS, AWS KMS, Azure Key Vault, Google Cloud KMS, and OCI KMS. This feature enables operators to delegate the unsealing process to trusted cloud providers to ease operations in the event of partial failure and to aid in the creation of new or ephemeral clusters.

This tutorial demonstrates an example of using Azure Key Vault to auto-unseal a Vault server.

Prerequisites

This tutorial assumes the following:

• Terraform installed and configured
• You have a Microsoft Azure account
  • Tenant ID
  • Client ID
  • Client Secret
  • Subscription ID

Azure permissions

This tutorial uses Terraform to provision a Vault environment to demonstrate the auto-unseal. The following Azure resources will be created:

• Azure Key Vault
• Virtual network with CIDR of 10.0.0.0/16
  • Subnet with CIDR, 10.0.1.0/24
• Public IP
• Security group
• A virtual machine to host a Vault server and accessible via public IP

Therefore, the service principal you use to perform the steps in this tutorial must have enough permission to provision all these resources.

Create an Azure service principal for Terraform

To run the provided Terraform configuration, you need an Azure service principal with appropriate permissions to deploy and configure Vault. The following demonstrates the creation of a service principal.

1. Launch the Microsoft Azure Portal and sign in.

2. Select Azure Active Directory and select Properties.

3. Copy the Tenant ID.

4. In a terminal, set the variable TENANT_ID to the Tenant ID.

   $ TENANT_ID=<Tenant ID>

5. From the side navigation, select App registrations.

6. Select New registrations.

7. Enter education in the Name field.

8. Click Register.

9. Copy the Application (client) ID.

10. In a terminal, set the variable CLIENT_ID to the Application (client) ID.

    $ CLIENT_ID=<Client ID>

11. From the side navigation, select Certificate & secrets.

12. Under the Client secrets, click New client secret.

13. Enter a description in the Description field.

14. Click Add.

15. Copy the client secret value.

16. In a terminal, set the variable CLIENT_SECRET to the client secret value.

    $ CLIENT_SECRET=<Client secret>

17. From the side navigation, click API permissions.

18. Under Configured permissions, click Add a permission.

19. Under Supported legacy APIs, click Azure Active Directory Graph.

20. Click Delegated permissions.

21. Expand User, select the check-box for User.Read.

22. Click Application permissions.

23. Expand Application, select the check-box for Application.ReadWrite.All.

24. Expand Directory, select the check-box for Directory.ReadWrite.All.

25. Click API permissions.

26. Click Grant admin consent for azure to grant the permissions.

27. Navigate to the Subscriptions blade.

28. Copy the Subscription ID.

29. In a terminal, set the variable SUBSCRIPTION_ID to the Subscription ID.

    $ SUBSCRIPTION_ID=<Subscription ID>

30. Click the name of the subscription.

31. From the side navigation, click Access control (IAM).

32. Click Add to expand the menu.

33. Click Add a role assignment.

34. Choose Owner from the Role select field.

35. Choose User, group, or service principal from the Assign Access To select field.

36. Enter the application name or application id in the Select field.

37. Click the application when it is displayed.

38. Click Save.

    The application is created with the correct permissions and you have these identifiers and credentials:

    • Tenant ID
    • Client ID
    • Client Secret
    • Subscription ID

39. Display the created credentials.

    $ echo -e \  "tenant_id=$TENANT_ID  subscription_id=$SUBSCRIPTION_ID  client_id=$CLIENT_ID  client_secret=$CLIENT_SECRET"

Download demo assets

Retrieve the additional configuration by cloning the hashicorp-education/learn-vault-autounseal-azure repository from GitHub.

$ git clone https://github.com/hashicorp-education/learn-vault-autounseal-azure.git

This repository contains supporting content for all of the Vault learn guides. The content specific to this tutorial can be found within a sub-directory.

Go into the learn-vault-autounseal-azure directory.

$ cd learn-vault-autounseal-azure

Step 1: Provision the cloud resources

The terraform files reads the credentials from the terraform.tfvars and generates the resources defined in the main.tf.

Modify terraform.tfvars.example with a public SSH key, public_key, and Azure credentials: tenant_id; client_id; client_secret and subscription_id.

tenant_id="0000000-0000-0000-0000000000"public_key = "ssh-rsa AAAA..."client_id="0000000-000000-0000000000"client_secret="AABBBCCCDDDDEEEFFF"subscription_id="0000000-0000-0000-0000-0000000000"

Save the file and create a copy of the file named terraform.tfvars.

$ cp terraform.tfvars.example terraform.tfvars

The main.tf file generates a new resource group with:

• Virtual machine with Vault already installed
• Azure Vault Key (Test-vault-xxxx)
• A key (generated-key)

Initialize the Azure provider plugins.

$ terraform init

Create an execution plan.

$ terraform plan...Plan: 12 to add, 0 to change, 0 to destroy.

Apply the changes.

$ terraform apply -auto-approve...Outputs: ip = 13.82.62.56key_vault_name = Test-vault-1e5a88dessh-addr =    Connect to your virtual machine via SSH:     $ ssh azureuser@13.82.62.56

Step 2: Test the auto-unseal feature

Create a variable named AZURE_VAULT_IP to store the IP address of the virtual machine.

$ AZURE_VAULT_IP=$(terraform output -raw ip)

SSH into the virtual machine with the azureuser user.

$ ssh azureuser@$AZURE_VAULT_IP

Within this SSH session, check the status of the Vault server.

$ vault status Key                      Value---                      -----Recovery Seal Type       azurekeyvaultInitialized              falseSealed                   trueTotal Recovery Shares    0Threshold                0Unseal Progress          0/0Unseal Nonce             n/aVersion                  n/aHA Enabled               true

The output displays that the Vault server is not initialized (Initialized is false).

Initialize the Vault server.

$ vault operator init

Check the status of the Vault server.

$ vault status Key                      Value---                      -----Recovery Seal Type       shamirInitialized              trueSealed                   falseTotal Recovery Shares    5Threshold                3Version                  1.3.0Cluster Name             vault-cluster-092ba5deCluster ID               8b173565-7d74-fe5b-a199-a2b56b7019eeHA Enabled               false

The output displays that the Vault server is unsealed (Sealed is false).

Vault started as a service. The service writes its output to a log.

Display the Vault server log.

$ sudo journalctl --no-pager -u vault ...==> Vault server configuration:        Azure Environment: AzurePublicCloud           Azure Key Name: generated-key         Azure Vault Name: Test-vault-a414d041                Seal Type: azurekeyvault                      Cgo: disabled               Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")                Log Level: (not set)                    Mlock: supported: true, enabled: false                  Storage: file                  Version: Vault v1.3.0              Version Sha: 37a1dc9c477c1c68c022d2084550f25bf20cac33 ==> Vault server started! Log data will stream in below: [WARN]  no `api_addr` value specified in config or in VAULT_API_ADDR; falling back to detection if possible, but this value should be manually set [INFO]  core: stored unseal keys supported, attempting fetch [INFO]  core: vault is unsealed...

The logs displays that the Azure Vault key is being fetched to unseal the Vault server. The Vault server automatically unseals every time Vault is started.

Restart the Vault server.

$ sudo systemctl restart vault

Check the status of the Vault server.

$ vault status

The output displays that the Vault server is automatically unsealed.

Display the Vault service configuration file.

$ cat /usr/lib/systemd/system/vault.service [Unit]Description=Vault AgentRequires=network-online.targetAfter=network-online.target[Service]Restart=on-failurePermissionsStartOnly=trueExecStartPre=/sbin/setcap 'cap_ipc_lock=+ep' /usr/local/bin/vaultExecStart=/usr/local/bin/vault server -config /etc/vault.d/config.hclExecReload=/bin/kill -HUPKillSignal=SIGTERMUser=azureuserGroup=azureuser[Install]WantedBy=multi-user.target

Review the Vault server configuration file.

$ cat /etc/vault.d/vault.hcl storage "file" {  path = "/opt/vault"}listener "tcp" {  address     = "0.0.0.0:8200"  tls_disable = "true"}seal "azurekeyvault" {  client_id      = "YOUR-APP-ID"  client_secret  = "YOUR-APP-PASSWORD"  tenant_id      = "YOUR-AZURE-TENANT-ID"  vault_name     = "Test-vault-XXXXXX"  key_name       = "generated-key"}ui=truedisable_mlock = true

Notice the Vault configuration file defines the azurekeyvault stanza with all parameter values properly populated: client ID, client secret, tenant ID, vault name (generated by Terraform), and Azure Key Vault key name.

Key rotation

With auto-unseal enabled, set up Azure Key Vault with key rotation using the Azure Automation Account and Vault will recognize newly rotated keys since the key metadata is stored with the encrypted data to ensure the correct key is used during decryption operations.

Step 3: Clean up

When you are done experimenting this demo follow these steps to clean up.

Destroy the resources that created by Terraform.

$ terraform destroy -auto-approve

Remove the Terraform state files.

$ rm -rf .terraform terraform.tfstate*

Help and reference

• Seal Migration
• Seal/Unseal
• Configuration: azurekeyvault Seal