Deploy Vault on Amazon EKS Anywhere

This tutorial provides guidance on deploying Vault in Amazon Elastic Kubernetes Service (EKS) Anywhere. Amazon EKS Anywhere is a new deployment option for Amazon EKS that allows customers to create and operate Kubernetes clusters on customer-managed infrastructure, supported by AWS. There are several layers in this stack and this tutorial will help you set up your infrastructure end-to-end.

The bottom-most layer of the stack consists of Bare Metal with VMware's vSphere as the first layer of abstraction. Amazon EKS runs on top of that as the Kubernetes distribution with Vault running as the secrets manager for it.

Prerequisites

The following prerequisites are required to follow the steps in this tutorial:

• AWS account with permissions for S3 and EKS services

  • AWS CLI

• One of the following:

  • Amazon EKS Anywhere cluster already deployed
  • Access to VMware vCenter cluster
  • Equinix bare metal, Terraform and VMware account with access to vCenter Server Appliance (VCSA) binary

• Vault Enterprise license (if using Enterprise version of Vault)

• Administrative machine prerequisites:

  • Docker 20.x.x
  • Kubernetes command-line interface (CLI)
  • Helm CLI installed
  • Mac OS (10.15) or Ubuntu (20.04.2 LTS)
    • 4 CPU cores
    • 16GB memory
    • 30GB free disk space

Download required files

1. Launch the vSphere downloads page. Under Enterprise, select VMware vCenter Server 7.0.3 (or the latest version). Select the GO TO DOWNLOAD link, and then click the DOWNLOAD NOW button.

2. Launch the vSan Management SDK for Python page. Click the DOWNLOAD button to download v7.0.3 (or the corresponding latest version to the previous step).

3. Upload the VMware-VCSA-all-<version>.iso into Amazon S3 bucket to a folder path of your choice, using vcsa-iso-image.

   List S3 objects to discover the vcsa-iso-image path.

   $ aws s3 ls 

   Upload the VMware-VCSA-all-<version>.iso into Amazon S3 bucket.

   $ aws s3 cp /local/path/to/VMware-VCSA-all-7.0.3-<version>.iso s3://vcsa-storage-bucket

4. Upload the Python scripts you downloaded earlier which is located at bindings > vsanmgmtObjects.py to the Amazon S3 folder.

   $ aws s3 cp /<file_path>/vsan-sdk-python/bindings/vsanmgmtObjects.py s3://vcsa-storage-bucket

5. Upload the Python scripts located at samplecode > vsanapiutils.py to the Amazon S3 bucket.

   $ aws s3 cp /<file_path>/vsan-sdk-python/samplecode/vsanapiutils.py s3://vcsa-storage-bucket 

6. The Amazon S3 bucket should look simiar to following

   Object Store Root:     |     |__ Bucket_Name         |         |__ VMware-VCSA-all-7.0.3-18700403.iso        |         |__ vsanapiutils.py        |         |__ vsanmgmtObjects.py

   Note

   For more information, refer the Equinix vSphere GitHub repository.

Deploy vCenter

1. To setup vCenter cluster using Equinix Bare Metal services follow these steps, for a more detailed setup follow the instructions on their website.

2. Create a eksa folder and change the working directory to eksa.

   $ mkdir eksa && cd eksa

3. Clone the terraform repository.

   $ git clone https://github.com/equinix/terraform-metal-vsphere

4. Set the working directory to terraform-metal-vsphere.

   $ cd terraform-metal-vsphere

5. Launch the Equinix console, and select Personal settings > Personal API keys > create. Make a note of the API key value.

6. From the Equinix console, click top-right on your name and select settings > org settings > General > Account Id. Make a note of the org ID.

7. From the Equinix console, select Personal settings > Project Id and copy the project ID.

8. Use your preferred text editor, create a file named, terraform.tfvars as follow. Change the facility value matches to your preference.

   terraform.tfvars

   # Equinix varsauth_token      = "<API_TOKEN_FROM_STEP_12.a>"organization_id = "<ORG_TOKEN_FROM_STEP_12.b>"project_id      = "<PROJECT_ID_FROM_STEP_12.c>"create_project  = falsefacility        = "da11" # AWS varsObject_store_tool        = "s3"Object_store_bucket_name = "vcsa-storage-bucket"s3_url                   = "https://s3.us_west_1.amazonaws.com"vcenter_iso_name         = "VMware-VCSA-all-<version>.iso"s3_access_key            = "<AWS_ACCESS_KEY>"s3_secret_key            = "<AWS_SECRET_KEY>"s3_version               = "S3v4"

9. Perform a terraform init to pull down the necessary provider resources.

   $ terraform init 

10. Run terraform apply and review the planned actions. Your terminal output should indicate the plan is running and what resources will be created.

    $ terraform apply ...snip... Do you want to perform these actions?  Terraform will perform the actions described above.  Only 'yes' will be accepted to approve.   Enter a value: yes

    When prompted, enter yes to confirm and resume.

    Warning

    If you get the “folder not present” error at any step, destroy the terraform resources using `terraform destroy` and edit the `main.tf`. Replace all locations of `$HOME` to `/root`. The output includes the following.

    bastion_host = "<ip_address_of_bastion>"ssh_key_path = "/Users/<username>/.ssh/vmware-on-metal-1-<>-key"vcenter_fqdn = "vcva.metal.local"vcenter_ip = "<ip_address_of_vcenter>"vcenter_password = <vcenter_alphanumeric_password>vcenter_root_password = <vcenter_root_alphanumeric_password>vcenter_username = "Administrator@vsphere.local"vpn_endpoint = "<ip_address_of_vpn>"vpn_password = <vpn_alphanumeric_password>vpn_psk = <vpn_psk_alphanumeric_password>vpn_user = "vm_admin"

Setup VPN to connect to vCenter cluster via CLI (Optioal)

There is an L2TP IPsec VPN client for every platform. You need to reference your operating system's documentation on how to connect to an L2TP IPsec VPN. Make sure to enable all traffic to use the VPN (aka do not enable split tunneling) on your L2TP client.

Deploy Amazon EKS Anywhere on vCenter

If you’re using a self deployed vCenter cluster, refer the Amazon EKS Anywhere create production cluster guide.

1. Login to vCenter and create resource pool under Metal (Datacenter) > Metal1 (Cluster) > EKSA (New resource Pool).

2. Install eksctl and eksctl-anywhere with homebrew.

   Note

   Refer to the Install EKS Anywhere documentation for more details.

   $ brew install aws/tap/eks-anywhere

3. Verify the installed version.

   $ eksctl anywhere version 

4. Generate a yaml file to deploy Amazon EKS Anywhere.

   $ CLUSTER_NAME=mgmt eksctl anywhere generate clusterconfig $CLUSTER_NAME \   --provider vsphere > eksa-mgmt-cluster.yaml

5. Edit the control plane configuration in eksa-mgmt-cluster.yaml.

   eksa-mgmt-cluster.yaml

   ……controlPlaneConfiguration:  count: 2  endpoint:    host: "172.16.0.150" # required unique IP outside DHCP block  machineGroupRef:……

   The CIDR block is defined, and the first 100 IPs are reserved for DHCP, so a good one would be 172.16.0.150.

6. Install govc CLI from the vmware/govmomi repository.

7. Set the GOVC_INSECURE environment variable value to 1 to disable certificate verification.

   $ export GOVC_INSECURE=1

8. Set the GOVC_URL environment variable to the URL of ESXi or vCenter instance to connect to.

   $ export GOVC_URL=145.40.103.170

9. Set the GOVC_USERNAME environment variable to the usename to use if not specified in GOVC_URL.

   $ export GOVC_USERNAME="Administrator@vsphere.local"

10. Set the GOVC_PASSWORD environment variable value to the password to use if not specified in GOVC_URL.

    $ export GOVC_PASSWORD="<VCENTER_PASSWORD>"

11. Use govc CLI to retrieve info about vCenter cluster.

    $ govc datacenter.info Name:                Metal  Path:              /Metal  Hosts:             3  Clusters:          1  Virtual Machines:  11  Networks:          2  Datastores:        4

12. Use the private network for VMs in configuration: vCenter console > Network.

13. Generate thumbprint for vSphere. This is required when the connection is not insecure.

    $ govc about.cert -thumbprint -k 145.40.103.170 61:B3:02:85:48:3F:A7:44:0E:3B:A0:AF:E8:38:11:62:79:16:E9:21

14. Edit the vSphere Data center configuration with the information gathered from the previous step.

    eksa-mgmt-cluster.yaml

    ……apiVersion: anywhere.eks.amazonaws.com/v1alpha1kind: VSphereDatacenterConfigmetadata:  name: eks-mgmtspec:  datacenter: "Metal" #required   insecure: false #setting it to true might throw error  network: "VM Private Net 1" #required   server: "145.40.103.170" #required - vSphere server  thumbprint: "61:B3:02:85:48:3F:A7:44:0E:3B:A0:AF:E8:38:11:62:79:16:E9:21"……

    Warning

    There are some issues when setting the connection as insecure (as of Feb 2022), the Amazon EKS Anywhere team is working on resolving it.

15. Edit vSphere machine configs to use the appropriate datastore and resource pool.

    eksa-mgmt-cluster.yaml

    ……kind: VSphereMachineConfigmetadata:  name: mgmt-cpspec:  datastore: vsanDatastore #required ……  resourcePool: /Metal/host/Metal-1/Resources/EKSA #required……kind: VSphereMachineConfigmetadata:  name: mgmtspec:  datastore: vsanDatastore #required ……  resourcePool: /Metal/host/Metal-1/Resources/EKSA #required……kind: VSphereMachineConfigmetadata:  name: mgmt-etcdspec:  datastore: vsanDatastore #required ……  resourcePool: /Metal/host/Metal-1/Resources/EKSA #required……

16. Set the EKSA_VSPHERE_USERNAME environment variable.

    $ export EKSA_VSPHERE_USERNAME='Administrator@vsphere.local'

17. Set the EKSA_VSPHERE_PASSWORD environment variable.

    $ export EKSA_VSPHERE_PASSWORD='<VCENTER_PASSWORD>'

18. Set the EKSA_LICENSE environment variable. This is not required if you already have one.

    $ export EKSA_LICENSE='my-license-here' 

19. Deploy Amazon EKS Anywhere.

    $ eksctl anywhere create cluster -f eksa-mgmt-cluster.yaml

    

20. Store the kubeconfig file. You need to store kubeconfig in bastion or use VPN from the optional step.

    $ export KUBECONFIG=${PWD}/${CLUSTER_NAME}/${CLUSTER_NAME}-Amazon EKS Anywhere-cluster.kubeconfig

21. Display the nodes of the cluster.

    $ kubectl get nodes NAME           STATUS   ROLES                  AGE   VERSION172.16.0.134   Ready    <none>                 16d   v1.21.6172.16.0.53    Ready    <none>                 16d   v1.21.6172.16.0.63    Ready    control-plane,master   16d   v1.21.6172.16.0.97    Ready    control-plane,master   16d   v1.21.6

Deploy Vault on Amazon EKS Anywhere

Vault installation on Amazon EKS Anywhere is same as any kubernetes installation. For a step-by-step intruction, refer to the Vault on Kubernetes Deployment Guide.

1. Create a vault directory and change the working directory to the vault directory.

   $ mkdir vault && cd vault

2. Create a configuration file to configure Vault installation.

   $ vi config.yamlserver:  standalone:    enabled: true    config: |      ui = true       listener "tcp" {        tls_disable = 1        address = "[::]:8200"        cluster_address = "[::]:8201"      }      storage "file" {        path = "/vault/data"      }  service:    enabled: trueui:  enabled: true  serviceType: NodePort

3. Create a new namespace for the Vault installation.

   $ kubectl create ns vault

4. To access the Vault Helm chart, add the Hashicorp Helm repository.

   $ helm repo add hashicorp https://helm.releases.hashicorp.com

5. Install the Vault helm chart.

   $ helm install vault hashicorp/vault --namespace vault -f config.yaml

6. Initialize and unseal Vault.

   $ kubectl exec --stdin=true --tty=true vault-0 -- vault operator init Unseal Key 1: MBFSDepD9E6whREc6Dj+k3pMaKJ6cCnCUWcySJQymObbUnseal Key 2: zQj4v22k9ixegS+94HJwmIaWLBL3nZHe1i+b/wHz25frUnseal Key 3: 7dbPPeeGGW3SmeBFFo04peCKkXFuuyKc8b2DuntA4VU5Unseal Key 4: tLt+ME7Z7hYUATfWnuQdfCEgnKA2L173dptAwfmenCdfUnseal Key 5: vYt9bxLr0+OzJ8m7c7cNMFj7nvdLljj0xWRbpLezFAI9 Initial Root Token: s.zJNwZlRrqISjyBHFMiEca6GF

   The output displays the key shares and initial root key generated.

   Note

   These keys are critical to both the security and the operation of Vault and should be treated as per your company's sensitive data policy.

7. Unseal the Vault server using the unseal keys until the key threshold is met.

   $ kubectl exec --stdin=true --tty=true vault-0 -- vault operator unseal Unseal Key (will be hidden):

   When prompted, enter the Unseal Key 1 value.

   $ kubectl exec --stdin=true --tty=true vault-0 -- vault operator unseal Unseal Key (will be hidden):

   When prompted, enter the Unseal Key 2 value.

   $ kubectl exec --stdin=true --tty=true vault-0 -- vault operator unseal Unseal Key (will be hidden):

   When prompted, enter the Unseal Key 3 value.

8. Validate that Vault is up and running.

   $ kubectl get pods --selector='app.kubernetes.io/name=vault' NAME                                    READY   STATUS    RESTARTS   AGEvault-0                                 1/1     Running   0          1m49svault-1                                 1/1     Running   0          1m49s

9. Display all Vault services.

   $ kubectl get services -n vault --selector='app.kubernetes.io/name=vault-ui' NAME       TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGEvault-ui   NodePort   10.97.113.241   <none>        8200:30096/TCP   16d

10. Display the nodes of the cluster.

    $ kubectl get nodes NAME           STATUS   ROLES                  AGE   VERSION172.16.0.134   Ready    <none>                 16d   v1.21.6172.16.0.53    Ready    <none>                 16d   v1.21.6172.16.0.63    Ready    control-plane,master   16d   v1.21.6172.16.0.97    Ready    control-plane,master   16d   v1.21.6

11. Install the HashiCorp tap, a repository of all our Homebrew packages.

    $ brew tap hashicorp/tap

12. Install Vault with hashicorp/tap/vault.

    $ brew install hashicorp/tap/vault

13. Set the VAULT_ADDR environment variable. Since we exposed Vault using NodePort, Vault will be available at 172.16.0.97:8200. Access it from your bastion host or VPN from the optional step.

    $ export VAULT_ADDR='http://172.16.0.97:30096'

14. Set the VAULT_TOKEN environment variable value to the initial root token value generated during the Vault initialization.

    $ export VAULT_TOKEN="s.zJNwZlRrqISjyBHFMiEca6GF"

15. Enable the kv secrets engine.

    $ vault secrets enable -path=kv kv Success! Enabled the kv secrets engine at: kv/

16. Store some test data at kv/hello.

    $ vault kv put kv/hello target=world Key                Value---                -----created_time       2022-03-21T21:23:00.540998543Zcustom_metadata    <nil>deletion_time      n/adestroyed          falseversion            1

17. Read the stored data to verify.

    $ vault kv get kv/hello ======= Metadata =======Key                Value---                -----created_time       2022-03-21T21:23:00.540998543Zcustom_metadata    <nil>deletion_time      n/adestroyed          falseversion            1 ===== Data =====Key       Value---       -----target    world

Configure Kubernetes auth method

1. Retrieve the additional configuration by cloning the hashicorp/learn-vault-agent repository from GitHub.

   $ git clone https://github.com/hashicorp/learn-vault-agent.git

2. Change the working directory to learn-vault-agent/vault-agent-k8s-demo.

   $ cd learn-vault-agent/vault-agent-k8s-demo

3. Update the vault-auth service account.

   $ kubectl apply --filename vault-auth-service-account.yaml

4. Create a read-only policy, myapp-kv-ro in Vault.

   $ vault policy write myapp-kv-ro - <<EOFpath "secret/data/myapp/*" {    capabilities = ["read", "list"]}EOF

5. Create some test data at the secret/myapp path.

   $ vault kv put secret/myapp/config \      username='appuser' \      password='suP3rsec(et!' \      ttl='30s'

6. Set the K8S_HOST environment variable value to minikube IP address.

   $ export K8S_HOST=$(kubectl config view --raw --minify --flatten \    --output 'jsonpath={.clusters[].cluster.server}')

7. Enable the Kubernetes auth method at the default path.

   $ vault auth enable kubernetes Success! Enabled kubernetes auth method at: kubernetes/

8. Configure the kubernetes auth method.

   Version compatibility

   Starting in v1.24, Kubernetes will no longer auto-generate the Secret object. So, for the best compatibility with recent Kubernetes versions, ensure you are using Vault v1.9.3 or greater.

   $ vault write auth/kubernetes/config \     kubernetes_host="$K8S_HOST"

   Output:

   Success! Data written to: auth/kubernetes/config

9. Create a role named, example, that maps the Kubernetes Service Account to Vault policies and default token TTL.

   $ vault write auth/kubernetes/role/example \     bound_service_account_names=vault-auth \     bound_service_account_namespaces=default \     token_policies=myapp-kv-ro \     ttl=24h

   Output:

   Success! Data written to: auth/kubernetes/role/example

Verify the Kubernetes auth method configuration

1. Create a variable named EXTERNAL_VAULT_ADDR.

   $ export EXTERNAL_VAULT_ADDR="172.16.0.97:30096"

2. Define a Pod with a container.

   $ cat > devwebapp.yaml <<EOFapiVersion: v1kind: Podmetadata:  name: devwebapp  labels:    app: devwebappspec:  serviceAccountName: vault-auth  containers:    - name: devwebapp      image: burtlo/devwebapp-ruby:k8s      env:        - name: VAULT_ADDR          value: "http://$EXTERNAL_VAULT_ADDR:8200"EOF

   The Pod is named devwebapp and runs with the vault-auth service account.

3. Create the devwebapp pod in the default namespace

   $ kubectl apply --filename devwebapp.yaml --namespace default

4. Display all the pods in the default namespace.

   $ kubectl get pods NAME        READY   STATUS    RESTARTS   AGEdevwebapp   1/1     Running   0          77s

   Wait until the devwebapp pod is running and ready (1/1).

5. Start an interactive shell session on the devwebapp pod.

   $ kubectl exec --stdin=true --tty=true devwebapp -- /bin/sh#

   Your system prompt is replaced with a new prompt #.

6. Set KUBE_TOKEN to the service account token.

   $ export KUBE_TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)

7. Authenticate with Vault through the example role with the KUBE_TOKEN.

   $ curl --request POST \    --data '{"jwt": "'"$KUBE_TOKEN"'", "role": "example"}' \    $VAULT_ADDR/v1/auth/kubernetes/login

   Example output:

   {  "request_id": "2febc920-6feb-182a-19cd-c95c0cd70bf7",  "lease_id": "",  "renewable": false,  "lease_duration": 0,  "data": null,  "wrap_info": null,  "warnings": null,  "auth": {    "client_token": "s.ZOZk5OkIW1rbdMEqGmwJW7vj",    "accessor": "yPwZcQG6LP721LryKzkgA4eA",    "policies": [      "default",      "myapp-kv-ro"    ],    "token_policies": [      "default",      "myapp-kv-ro"    ],    "metadata": {      "role": "example",      "service_account_name": "vault-auth",      "service_account_namespace": "default",      "service_account_secret_name": "",      "service_account_uid": "649f0652-42ef-44ba-a416-76862bc1b6c3"    },    "lease_duration": 86400,    "renewable": true,    "entity_id": "3f684285-bc1c-6b9e-e580-4ce310c956b2",    "token_type": "service",    "orphan": true  }}

Next steps

In this tutorial, you deployed Vault on an Amazon EKS Anywhere cluster. Also, you enabled Kubernetes auth method so that Vault clients can authenticate with Vault using the trusted service account.

Help and reference

• Injecting Secrets into Kubernetes Pods via Vault Agent Containers
• Vault Installation to Amazon Elastic Kubernetes Service via Helm
• Vault documentation