News HashiCorp to join IBM to accelerate multi-cloud automation. Read the blog →
Vault
Vault

Install Vault to Kubernetes with TLS enabled

  • 13min
  • |
  • Vault

This tutorial walks through setting up end-to-end TLS on a HA Vault cluster in Kubernetes. You will create a private key and a wildcard certificate using the Kubernetes CA. You will store the certificate and the key in the Kubernetes secrets store. Finally you will configure the Helm chart to use the Kubernetes secret.

Prerequisites

This tutorial requires the Kubernetes command-line interface (CLI) and the Helm CLI installed, minikube, the Vault Helm chart, and the additional configuration to bring it all together.

First, follow the directions to install minikube, including VirtualBox or similar.

Next, install kubectl CLI and Helm CLI.

NOTE: This tutorial was last tested in November 2022 on a macOS 12.6.1 using this configuration.

Docker version.

$ docker versionClient: Cloud integration: v1.0.29 Version:           20.10.20  ## ...

Display minikube version.

$ minikube versionminikube version: v1.27.1commit: fe869b5d4da11ba318eb84a3ac00f336411de7ba

Helm version.

$ helm versionversion.BuildInfo{Version:"v3.10.1", GitCommit:"9f88ccb6aee40b9a0535fcc7efea6055e1ef72c9", GitTreeState:"clean", GoVersion:"go1.19.2"}

These are recommended software versions and the output displayed may vary depending on your environment and the software versions you use.

Install kubectl with Homebrew.

$ brew install kubernetes-cli

Install helm with Homebrew.

$ brew install helm

Start minikube

Minikube is a CLI tool that provisions and manages the lifecycle of single-node Kubernetes clusters running inside Virtual Machines (VM) on your local system.

  1. Start a Kubernetes cluster.

    $ minikube start😄  minikube v1.27.1 on Darwin 12.6.1 (arm64)✨  Automatically selected the docker driver📌  Using Docker Desktop driver with root privileges👍  Starting control plane node minikube in cluster minikube🚜  Pulling base image ...💾  Downloading Kubernetes v1.25.2 preload ... > preloaded-images-k8s-v18-v1...:  320.84 MiB / 320.84 MiB  100.00% 23.57 M > gcr.io/k8s-minikube/kicbase:  348.47 MiB / 348.47 MiB  100.00% 16.91 MiB  > gcr.io/k8s-minikube/kicbase:  0 B [________________________] ?% ? p/s 21s🔥  Creating docker container (CPUs=2, Memory=4000MB) ...🐳  Preparing Kubernetes v1.25.2 on Docker 20.10.18 ... ▪ Generating certificates and keys ... ▪ Booting up control plane ... ▪ Configuring RBAC rules ...🔎  Verifying Kubernetes components... ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5🌟  Enabled addons: storage-provisioner, default-storageclass🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

    The initialization process takes several minutes as it retrieves any necessary dependencies and executes various container images.

  2. Verify the status of the minikube cluster.

    $ minikube statusminikubetype: Control Plane host: Runningkubelet: Runningapiserver: Runningkubeconfig: Configured

Install the Vault Helm chart

Vault manages the secrets that are written to these mountable volumes. To provide these secrets a single Vault server is required. For this demonstration Vault can be run in development mode to automatically handle initialization, unsealing, and setup of a KV secrets engine.

  1. Add the HashiCorp Helm repository.

    $ helm repo add hashicorp https://helm.releases.hashicorp.com"hashicorp" has been added to your repositories

  2. Update all the repositories to ensure helm is aware of the latest versions.

    $ helm repo updateHang tight while we grab the latest from your chart repositories......Successfully got an update from the "hashicorp" chart repositoryUpdate Complete. ⎈Happy Helming!⎈

  3. To verify, search repositories for vault in charts.

    $ helm search repo hashicorp/vaultNAME            CHART VERSION   APP VERSION     DESCRIPTIONhashicorp/vault 0.22.1          1.12.0          Official HashiCorp Vault Chart

Create the certificate

  1. create a working directory

    $ mkdir /tmp/vault

  2. Export the working directory location and the naming variables. 

    $ export VAULT_K8S_NAMESPACE="vault" \export VAULT_HELM_RELEASE_NAME="vault" \export VAULT_SERVICE_NAME="vault-internal" \export K8S_CLUSTER_NAME="cluster.local" \export WORKDIR=/tmp/vault

  3. Generate the private key

    $ openssl genrsa -out ${WORKDIR}/vault.key 2048Generating RSA private key, 2048 bit long modulus....................+++....+++

Create the Certificate Signing Request (CSR).

  1. Create the CSR configuration file

    $ cat > ${WORKDIR}/vault-csr.conf <<EOF[req]default_bits = 2048prompt = noencrypt_key = yesdefault_md = sha256distinguished_name = kubelet_servingreq_extensions = v3_req[ kubelet_serving ]O = system:nodesCN = system:node:*.${VAULT_K8S_NAMESPACE}.svc.${K8S_CLUSTER_NAME}[ v3_req ]basicConstraints = CA:FALSEkeyUsage = nonRepudiation, digitalSignature, keyEncipherment, dataEnciphermentextendedKeyUsage = serverAuth, clientAuthsubjectAltName = @alt_names[alt_names]DNS.1 = *.${VAULT_SERVICE_NAME}DNS.2 = *.${VAULT_SERVICE_NAME}.${VAULT_K8S_NAMESPACE}.svc.${K8S_CLUSTER_NAME}DNS.3 = *.${VAULT_K8S_NAMESPACE}IP.1 = 127.0.0.1EOF

  2. Generate the CSR

    $ openssl req -new -key ${WORKDIR}/vault.key -out ${WORKDIR}/vault.csr -config ${WORKDIR}/vault-csr.conf

Issue the Certificate.

  1. Create the csr yaml file to send it to Kubernetes.

    $ cat > ${WORKDIR}/csr.yaml <<EOFapiVersion: certificates.k8s.io/v1kind: CertificateSigningRequestmetadata:   name: vault.svcspec:   signerName: kubernetes.io/kubelet-serving   expirationSeconds: 8640000   request: $(cat ${WORKDIR}/vault.csr|base64|tr -d '\n')   usages:   - digital signature   - key encipherment   - server authEOF

  2. Send the CSR to Kubernetes

    $ kubectl create -f ${WORKDIR}/csr.yamlcertificatesigningrequest.certificates.k8s.io/vault.svc created

  3. Approve the CSR in Kubernetes.

    $ kubectl certificate approve vault.svccertificatesigningrequest.certificates.k8s.io/vault.svc approved

  4. Confirm the certificate was issued

    $ kubectl get csr vault.svcNAME        AGE   SIGNERNAME                      REQUESTOR       REQUESTEDDURATION   CONDITIONvault.svc   16s   kubernetes.io/kubelet-serving   minikube-user   100d                Approved,Issued

Store the certificates and Key in the Kubernetes secrets store

  1. Retrieve the certificate

    $ kubectl get csr vault.svc -o jsonpath='{.status.certificate}' | openssl base64 -d -A -out ${WORKDIR}/vault.crt

  2. Retrieve Kubernetes CA certificate

    $ kubectl config view \--raw \--minify \--flatten \-o jsonpath='{.clusters[].cluster.certificate-authority-data}' \| base64 -d > ${WORKDIR}/vault.ca

  3. Create the Kubernetes namespace

    $ kubectl create namespace $VAULT_K8S_NAMESPACEnamespace/vault created

  4. Create the TLS secret

    $ kubectl create secret generic vault-ha-tls \   -n $VAULT_K8S_NAMESPACE \   --from-file=vault.key=${WORKDIR}/vault.key \   --from-file=vault.crt=${WORKDIR}/vault.crt \   --from-file=vault.ca=${WORKDIR}/vault.ca

    Output:

    # secret/vault-ha-tls created

Deploy the vault cluster via Helm with overrides

  1. Create the overrides.yaml file.

    $ cat > ${WORKDIR}/overrides.yaml <<EOFglobal:   enabled: true   tlsDisable: falseinjector:   enabled: trueserver:   extraEnvironmentVars:      VAULT_CACERT: /vault/userconfig/vault-ha-tls/vault.ca      VAULT_TLSCERT: /vault/userconfig/vault-ha-tls/vault.crt      VAULT_TLSKEY: /vault/userconfig/vault-ha-tls/vault.key   volumes:      - name: userconfig-vault-ha-tls        secret:         defaultMode: 420         secretName: vault-ha-tls   volumeMounts:      - mountPath: /vault/userconfig/vault-ha-tls        name: userconfig-vault-ha-tls        readOnly: true   standalone:      enabled: false   affinity: ""   ha:      enabled: true      replicas: 3      raft:         enabled: true         setNodeId: true         config: |            cluster_name = "vault-integrated-storage"            ui = true            listener "tcp" {               tls_disable = 0               address = "[::]:8200"               cluster_address = "[::]:8201"               tls_cert_file = "/vault/userconfig/vault-ha-tls/vault.crt"               tls_key_file  = "/vault/userconfig/vault-ha-tls/vault.key"               tls_client_ca_file = "/vault/userconfig/vault-ha-tls/vault.ca"            }            storage "raft" {               path = "/vault/data"            }            disable_mlock = true            service_registration "kubernetes" {}EOF

    Recommendation

    If you are using Prometheus for monitoring and alerting, we recommend to set the cluster_name in the HCL configuration. With the Vault Helm chart, this is accomplished with the config parameter.

  2. Deploy the Cluster

    $ helm install -n $VAULT_K8S_NAMESPACE $VAULT_HELM_RELEASE_NAME hashicorp/vault -f ${WORKDIR}/overrides.yaml

    Example output: 

    NAME: vaultLAST DEPLOYED: Thu Nov  3 19:47:36 2022NAMESPACE: vaultSTATUS: deployedREVISION: 1NOTES:Thank you for installing HashiCorp Vault! Now that you have deployed Vault, you should look over the docs on usingVault with Kubernetes available here: https://www.vaultproject.io/docs/ Your release is named vault. To learn more about the release, try:    $ helm status vault   $ helm get manifest vault

  3. Display the pods in the namespace that you created for vault

    $ kubectl -n $VAULT_K8S_NAMESPACE get podsNAME                                    READY   STATUS    RESTARTS   AGEvault-0                                 0/1     Running   0          22svault-1                                 0/1     Running   0          22svault-2                                 0/1     Running   0          22svault-agent-injector-6679b665fd-92n5q   1/1     Running   0          23s

  4. Initialize vault-0 with one key share and one key threshold.

    $ kubectl exec -n $VAULT_K8S_NAMESPACE vault-0 -- vault operator init \    -key-shares=1 \    -key-threshold=1 \    -format=json > ${WORKDIR}/cluster-keys.json

    The operator init command generates a root key that it disassembles into key shares -key-shares=1 and then sets the number of key shares required to unseal Vault -key-threshold=1. These key shares are written to the output as unseal keys in JSON format -format=json. Here the output is redirected to a file named cluster-keys.json.

  5. Display the unseal key found in cluster-keys.json.

    $ jq -r ".unseal_keys_b64[]" ${WORKDIR}/cluster-keys.jsonezMnsvK9j6aArz1sP0kQNBnTW5PH5u15iFtwPp4Jdzg=

    Insecure operation

    Do not run an unsealed Vault in production with a single key share and a single key threshold. This approach is only used here to simplify the unsealing process for this demonstration.

  6. Create a variable named VAULT_UNSEAL_KEY to capture the Vault unseal key.

    $ VAULT_UNSEAL_KEY=$(jq -r ".unseal_keys_b64[]" ${WORKDIR}/cluster-keys.json)

    After initialization, Vault is configured to know where and how to access the storage, but does not know how to decrypt any of it. Unsealing is the process of constructing the root key necessary to read the decryption key to decrypt the data, allowing access to the Vault.

  7. Unseal Vault running on the vault-0 pod.

    $ kubectl exec -n $VAULT_K8S_NAMESPACE vault-0 -- vault operator unseal $VAULT_UNSEAL_KEY

    Insecure operation

    Providing the unseal key with the command writes the key to your shell's history. This approach is only used here to simplify the unsealing process for this demonstration.

    The operator unseal command reports that Vault is initialized and unsealed.

    Example output: 

    Key                     Value---                     -----Seal Type               shamirInitialized             trueSealed                  falseTotal Shares            1Threshold               1Version                 1.12.0Build Date              2022-10-10T18:14:33ZStorage Type            raftCluster Name            vault-cluster-5b1b8251Cluster ID              2e2f214b-c4e7-39e7-52b5-39bffba1aa12HA Enabled              trueHA Cluster              https://vault-0.vault-internal:8201HA Mode                 activeActive Since            2022-11-04T13:08:48.505370004ZRaft Committed Index    36Raft Applied Index      36

    The Vault server is initialized and unsealed.

Join vault-1 and vault2 pods to the Raft cluster

  1. Start an interactive shell session on the vault-1 pod. 

    $ kubectl exec -n $VAULT_K8S_NAMESPACE -it vault-1 -- /bin/sh/ $

    Your system prompt is replaced with a new prompt / $.

  2. Join the vault-1 pod to the Raft cluster. 

    $ vault operator raft join -address=https://vault-1.vault-internal:8200 -leader-ca-cert="$(cat /vault/userconfig/vault-ha-tls/vault.ca)" -leader-client-cert="$(cat /vault/userconfig/vault-ha-tls/vault.crt)" -leader-client-key="$(cat /vault/userconfig/vault-ha-tls/vault.key)" https://vault-0.vault-internal:8200

    Example output: 

    Key       Value---       -----Joined    true

  3. Exit the vault-1 pod.

    $ exit

  4. Unseal vault-1.

    $ kubectl exec -n $VAULT_K8S_NAMESPACE -ti vault-1 -- vault operator unseal $VAULT_UNSEAL_KEY Key                Value---                -----Seal Type          shamirInitialized        trueSealed             trueTotal Shares       1Threshold          1Unseal Progress    0/1Unseal Nonce       n/aVersion            1.12.0Build Date         2022-10-10T18:14:33ZStorage Type       raftHA Enabled         true

  5. Start an interactive shell session on the vault-2 pod. 

    $ kubectl exec -n $VAULT_K8S_NAMESPACE -it vault-2 -- /bin/sh/ $

    Your system prompt is replaced with a new prompt / $.

  6. Join the vault-2 pod to the Raft cluster. 

    $ vault operator raft join -address=https://vault-2.vault-internal:8200 -leader-ca-cert="$(cat /vault/userconfig/vault-ha-tls/vault.ca)" -leader-client-cert="$(cat /vault/userconfig/vault-ha-tls/vault.crt)" -leader-client-key="$(cat /vault/userconfig/vault-ha-tls/vault.key)" https://vault-0.vault-internal:8200

    Example output: 

    Key       Value---       -----Joined    true

  7. Exit the vault-2 pod.

    $ exit

  8. Unseal vault-2.

    $ kubectl exec -n $VAULT_K8S_NAMESPACE -ti vault-2 -- vault operator unseal $VAULT_UNSEAL_KEY Key                Value---                -----Seal Type          shamirInitialized        trueSealed             trueTotal Shares       1Threshold          1Unseal Progress    0/1Unseal Nonce       n/aVersion            1.12.0Build Date         2022-10-10T18:14:33ZStorage Type       raftHA Enabled         true

    Login to vault and confirm everything is working

  9. Export the cluster root token

    $ export CLUSTER_ROOT_TOKEN=$(cat ${WORKDIR}/cluster-keys.json | jq -r ".root_token")

  10. Login to vault-0 with the root token

    $ kubectl exec -n $VAULT_K8S_NAMESPACE vault-0 -- vault login $CLUSTER_ROOT_TOKEN Success! You are now authenticated. The token information displayed belowis already stored in the token helper. You do NOT need to run "vault login"again. Future Vault requests will automatically use this token. Key                  Value---                  -----token                hvs.lF3yPwVSCl6AwMI7XdrsirdHtoken_accessor       JaQ0MUL3R1zjbTSkUMCKARmktoken_duration       ∞token_renewable      falsetoken_policies       ["root"]identity_policies    []policies             ["root"]

  11. List the raft peers.

    $ kubectl exec -n $VAULT_K8S_NAMESPACE vault-0 -- vault operator raft list-peers Node       Address                        State       Voter----       -------                        -----       -----vault-0    vault-0.vault-internal:8201    leader      truevault-1    vault-1.vault-internal:8201    follower    truevault-2    vault-2.vault-internal:8201    follower    true

  12. Print the HA status

    $ kubectl exec -n $VAULT_K8S_NAMESPACE vault-0 -- vault status Key                     Value---                     -----Seal Type               shamirInitialized             trueSealed                  falseTotal Shares            1Threshold               1Version                 1.12.0Build Date              2022-10-10T18:14:33ZStorage Type            raftCluster Name            vault-cluster-0a1e62efCluster ID              95972f8a-c38e-8200-9a67-0a910b34b691HA Enabled              trueHA Cluster              https://vault-0.vault-internal:8201HA Mode                 activeActive Since            2022-11-04T14:21:05.668816678ZRaft Committed Index    42Raft Applied Index      42

    You now have a working 3 node cluster with TLS enabled at the pod level. Next you will create a secret and retrieve it via and API call to confirm TLS is working as expected.

Create a secret

  1. Start an interactive shell session on the vault-0 pod.

    $ kubectl exec -n $VAULT_K8S_NAMESPACE -it vault-0 -- /bin/sh/ $

    Note

    Your system prompt is replaced with a new prompt / $.

  2. Enable the kv-v2 secrets engine

    $ vault secrets enable -path=secret kv-v2Success! Enabled the kv-v2 secrets engine at: secret/

  3. Create a secret at the path secret/tls/apitest with a username and a password

    $ vault kv put secret/tls/apitest username="apiuser" password="supersecret" ===== Secret Path =====secret/data/tls/apitest ======= Metadata =======Key                Value---                -----created_time       2022-11-07T17:00:23.335480752Zcustom_metadata    <nil>deletion_time      n/adestroyed          falseversion            1

  4. Verify that the secret is defined at the path secret/tls/apitest

    $ vault kv get secret/tls/apitest ===== Secret Path =====secret/data/tls/apitest ======= Metadata =======Key                Value---                -----created_time       2022-11-07T17:00:23.335480752Zcustom_metadata    <nil>deletion_time      n/adestroyed          falseversion            1 ====== Data ======Key         Value---         -----password    supersecretusername    apiuser

  5. Exit the vault-0 pod.

    $ exit

Expose the vault service and retrieve the secret via the API

The Helm chart defined a Kubernetes service named vault that forwards requests to its endpoints (i.e. The pods named vault-0, vault-1, and vault-2).

  1. Confirm the Vault service configuration

    $ kubectl -n $VAULT_K8S_NAMESPACE get service vault NAME    TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)             AGEvault   ClusterIP   10.110.24.17   <none>        8200/TCP,8201/TCP   68m

  2. In another terminal, port forward the vault service.

    $ kubectl -n vault port-forward service/vault 8200:8200 Forwarding from 127.0.0.1:8200 -> 8200Forwarding from [::1]:8200 -> 8200

  3. In the original terminal, perform a HTTPS curl request to retrieve the secret you created in the previous section.

    $ curl --cacert $WORKDIR/vault.ca \   --header "X-Vault-Token: $CLUSTER_ROOT_TOKEN" \   https://127.0.0.1:8200/v1/secret/data/tls/apitest | jq .data.data

    Example output:

    {"password": "supersecret","username": "apiuser"}

    The secret you created earlier is displayed back to us.

Clean up

  1. Stop the running local Kubernetes cluster.

    $ minikube stop ✋  Stopping node "minikube"  ...🛑  Powering off "minikube" via SSH ...🛑  1 node stopped.

    This deactivates minikube, and all pods still exist at this point.

  2. Delete the local Kubernetes cluster.

    Be aware that minikube delete removes the minikube deployment including all pods. Be sure you want everything removed before continuing. 

    $ minikube delete 🔥  Deleting "minikube" in docker ...🔥  Deleting container "minikube" ...🔥  Removing /Users/username/.minikube/machines/minikube ...💀  Removed all traces of the "minikube" cluster.

  3. Remove the files and the working directory you created.

    $ rm -r $WORKDIR

Next steps

You launched Vault in high-availability via a Helm chart with TLS enabled. Learn more about the Vault Helm chart by reading the documentation or exploring the project source code.

Then you created and API call and requested a secret directly from Vault. Explore how pods can retrieve secrets through the Vault Injector service via annotations or secrets mounted on ephemeral volumes.

Previous

Vault installation to minikube with Consul

 Next

Vault installation to Amazon EKS via Helm