Codify Vault Enterprise management with Terraform

16min
|
Enterprise
Terraform
Vault

Personas

The scenario described in this tutorial introduces the following personas:

admin is the organization-level administrator
student is a user allowed to write data to a path in vault

Challenge

A manual system administration can become a challenge as the scale of infrastructure increases. Often, an organization must manage multiple Vault environments (development, testing, staging, production, etc.). Keeping up with the increasing management demand soon becomes a challenge without some sort of automation.

Solution

One of the pillars behind the Tao of Hashicorp is automation through codification.

HashiCorp Terraform is an infrastructure as code which enables the operation team to codify the Vault configuration tasks such as the creation of policies. Automation through codification allows operators to increase their productivity, move quicker, promote repeatable processes, and reduce human error.

This tutorial demonstrates techniques for creating Vault policies and configurations using Terraform Vault Provider.

Prerequisites

Terraform installed
Vault Enterprise 1.4 or later

Enterprise Only

This tutorial creates namespaces which require Vault Enterprise Standard license.

If you are running open-source Vault, see the Codify Management of Vault Using Terraform tutorial.

Scenario introduction

Vault administrators must manage multiple Vault environments. The test servers get destroyed at the end of each test cycle and a new set of servers must be provisioned for the next test cycle. To automate the Vault server configuration, you are going to use Terraform to provision the following Vault resources.

Type	Name	Description
namespace	finance	A namespace dedicated to the finance organization
namespace	engineering	A namespace dedicated to the engineering organization
namespace	education	A namespace dedicated to the education organization
namespace	training	A child-namespace under `education` dedicated to the training team
namespace	vault_cloud	A child-namespace under `education/training` dedicated to the vault_cloud team
namespace	engineering	A child-namespace under `education/training` dedicated to the boundary team
ACL Policy	admins	Sets policies for the admin team
ACL Policy	fpe-client	Sets policies for clients to encode/decode data through transform secrets engine
auth method	userpass	Enable and create a user, "student" with `admins` and `fpe-client` policies
auth method	approle	Enable approle auth method in the `education/training` and create a `test-role` role
secrets engine	kv-v2	Enable kv-v2 secrets engine in the `finance` namespace
secrets engine	transform	Enable transform secrets engine at `transform`
transformation	ccn-fpe	Transformation to perform format preserving encryption (FPE) transformation on credit card numbers
transformation template	ccn	Define the data format structure for credit card numbers
alphabet	numerics	Set of allowed characters

The admins policy must be created in all namespaces: root, finance, and engineering. The expected admin tasks are the same across the namespaces.

Note

Transform secrets engine requires Vault Enterprise Advanced Data Protection (ADP) license.

The following steps are demonstrated:

Examine the Terraform files

Clone or download the demo assets from the hashicorp/learn-vault-codify GitHub repository to perform the steps described in this tutorial.
```
$ git clone https://github.com/hashicorp/learn-vault-codify.git
```

Change the working directory to learn-vault-codify/enterprise.

$ cd learn-vault-codify/enterprise

The directory contains Terraform files to configure Vault.

$ tree.├── auth.tf├── main.tf├── policies│   ├── admin-policy.hcl│   └── fpe-client-policy.hcl├── policies.tf└── secrets.tf

Review main.tf

Open the main.tf file in your preferred text editor to examine its content.

#------------------------------------------------------------------------------# The best practice is to use remote state file and encrypt it since your# state files may contains sensitive data (secrets).#------------------------------------------------------------------------------# terraform {#       backend "s3" {#             bucket = "remote-terraform-state-dev"#             encrypt = true#             key = "terraform.tfstate"#             region = "us-east-1"#       }# } #-----------------------------------------------------------------------------------# To configure Transform secrets engine, you need vault provider v2.12.0 or later#-----------------------------------------------------------------------------------terraform {  required_providers {    vault = "~> 3.7.0"  }} #------------------------------------------------------------------------------# To leverage more than one namespace, define a vault provider per namespace#------------------------------------------------------------------------------ provider "vault" {} #------------------------------------------------------------------------------# Create namespaces: finance, and engineering#------------------------------------------------------------------------------resource "vault_namespace" "finance" {  path = "finance"} resource "vault_namespace" "engineering" {  path = "engineering"} #---------------------------------------------------------------# Create nested namespaces#   education has childnamespace, 'training'#       training has childnamespace, 'secure'#           secure has childnamespace, 'vault_cloud' and 'boundary'#---------------------------------------------------------------resource "vault_namespace" "education" {  path = "education"}  # Create a childnamespace, 'training' under 'education'resource "vault_namespace" "training" {  namespace = vault_namespace.education.path  path = "training"} # Create a childnamespace, 'vault_cloud' and 'boundary' under 'education/training'resource "vault_namespace" "vault_cloud" {  namespace = vault_namespace.training.path_fq  path = "vault_cloud"} # Create 'education/training/boundary' namespaceresource "vault_namespace" "boundary" {  namespace = vault_namespace.training.path_fq  path = "boundary"}

To create a education/training namespace, use the namespace parameter to point the vault_namespace.education.path parent namespace.

resource "vault_namespace" "training" {  namespace = vault_namespace.education.path  path = "training"}

It is strongly recommended to specify the target server specific information using environment variables (e.g. VAULT_ADDR, VAULT_TOKEN); that is what you are going to do in this tutorial.

Note

If you are not familiar with Vault Enterprise namespace, refer to the Secure Multi-Tenancy with Namespaces tutorial.

Review policies.tf

Open the policies.tf file and examine the vault_policy resources. It uses the provider parameter to specify the target namespace to create the policies.

#---------------------# Create policies#---------------------# Create fpe-client policy in the root namespaceresource "vault_policy" "fpe_client_policy" {  name   = "fpe-client"  policy = file("policies/fpe-client-policy.hcl")} # Create admin policy in the root namespaceresource "vault_policy" "admin_policy" {  name   = "admins"  policy = file("policies/admin-policy.hcl")} # Create admin policy in the finance namespaceresource "vault_policy" "admin_policy_finance" {  namespace = vault_namespace.finance.path  name   = "admins"  policy = file("policies/admin-policy.hcl")} # Create admin policy in the engineering namespaceresource "vault_policy" "admin_policy_engineering" {  namespace = vault_namespace.engineering.path  name   = "admins"  policy = file("policies/admin-policy.hcl")} # Create admin policy in the education namespaceresource "vault_policy" "admin_policy_education" {  namespace = vault_namespace.education.path  name   = "admins"  policy = file("policies/admin-policy.hcl")} # Create admin policy in the 'education/training' namespaceresource "vault_policy" "admin_policy_training" {  namespace = vault_namespace.training.path_fq  name   = "admins"  policy = file("policies/admin-policy.hcl")} # Create admin policy in the 'education/training/vault_cloud' namespaceresource "vault_policy" "admin_policy_vault_cloud" {  namespace = vault_namespace.vault_cloud.path_fq  name   = "admins"  policy = file("policies/admin-policy.hcl")} # Create admin policy in the 'education/training/boundary' namespaceresource "vault_policy" "admin_policy_boundary" {  namespace = vault_namespace.boundary.path_fq  name   = "admins"  policy = file("policies/admin-policy.hcl")}

Review auth.tf

Open the auth.tf file.

Line 4 through 6 enables userpass auth method.
Line 9 through 20 creates a user, "student" with admins and fpe-client policies attached. The password is set to "changeme".
Line 25 through 29 enables approle auth method.
Line 32 through 38 creates a test-role role.

#--------------------------------# Enable userpass auth method#--------------------------------resource "vault_auth_backend" "userpass" {  type = "userpass"} # Create a user named, "student"resource "vault_generic_endpoint" "student" {  depends_on           = [vault_auth_backend.userpass]  path                 = "auth/userpass/users/student"  ignore_absent_fields = true   data_json = <<EOT{  "policies": ["fpe-client", "admins"],  "password": "changeme"}EOT} #--------------------------------------------------------------------# Enable approle auth method in the 'education/training' namespace#--------------------------------------------------------------------resource "vault_auth_backend" "approle" {  depends_on = [vault_namespace.training]  namespace = vault_namespace.training.path_fq  type       = "approle"} # Create a role named, "test-role"resource "vault_approle_auth_backend_role" "test-role" {  depends_on     = [vault_auth_backend.approle]  backend        = vault_auth_backend.approle.path  namespace = vault_namespace.training.path_fq  role_name      = "test-role"  token_policies = ["default", "admins"]}

Review secrets.tf

Open the secrets.tf file.

Line 5 through 10 enables kv-v2 secretes engine in the finance namespace.
Line 19 through 53 defines a new alphabet, template, transformation, and a role.
Line 59 through 69 tests the FPE transformation configured.

#----------------------------------------------------------# Enable secrets engines#----------------------------------------------------------# Enable kv-v2 secrets engine in the finance namespaceresource "vault_mount" "kv-v2" {  # depends_on = [vault_namespace.finance]  # provider = vault.finance  namespace = vault_namespace.finance.path  path = "kv-v2"  type = "kv-v2"} # Transform secrets engine at rootresource "vault_mount" "mount_transform" {  path = "transform"  type = "transform"} # Create an alphabetresource "vault_transform_alphabet" "numerics" {  path = vault_mount.mount_transform.path  name = "numerics"  alphabet = "0123456789"} # Create a transformation templateresource "vault_transform_template" "ccn" {  path = vault_mount.mount_transform.path  name = "ccn"  type = "regex"  pattern = "(\\d{4})-(\\d{4})-(\\d{4})-(\\d{4})"  alphabet = vault_transform_alphabet.numerics.name} # Create a transformation named ccn-fperesource "vault_transform_transformation" "ccn-fpe" {  path = vault_mount.mount_transform.path  name = "ccn-fpe"  type = "fpe"  template = vault_transform_template.ccn.name  tweak_source = "internal"  # payments here listed by name and not reference to avoid circular dependency.  # Vault does not require dependencies like these to exist prior to creating  # other things that reference them, but they may simply not work until they do  # exist.  allowed_roles = ["payments"]} # Create a role named 'payments'resource "vault_transform_role" "payments" {  path = vault_mount.mount_transform.path  name = "payments"  transformations = [vault_transform_transformation.ccn-fpe.name]}  #-------------------------------------------------------------------# Test the transformation#-------------------------------------------------------------------data "vault_transform_encode" "encoded" {  path = vault_transform_role.payments.path  role_name = "payments"  value = "1111-2222-3333-4444"   depends_on = [vault_transform_role.payments]} output "encoded" {  value = data.vault_transform_encode.encoded.encoded_value}

Note

The details about the transformation, template, alphabet, and role are out of scope for this tutorial. If you are not familiar with Transform secrets engine, read the Transform Secrets Engine tutorial.

Run Terraform to configure Vault

Optional: Start a Vault server in development mode with root as the root token if you don't have one running already.
```
$ vault server -dev -dev-root-token-id root
```
Set the client token in the VAULT_TOKEN environment variable.
```
$ export VAULT_TOKEN="root"
```
If the token is different, be sure to set it to the correct token value that has permissions to create policies, enable secrets engines, and enable auth methods.
Set the target Vault server address in the VAULT_ADDR environment variable if it's not done so already.
```
$ export VAULT_ADDR="http://127.0.0.1:8200"
```
If you are connecting to a remote Vault server, be sure to set the VAULT_ADDR value to the correct target Vault API address.
Initialize Terraform to pull Vault provider plugin.
```
$ terraform init
```
This downloads the Vault plugin. When it completes, it displays a message, Terraform has been successfully initialized!

Execute the apply command to configure Vault.

$ terraform apply

This displays the actions to be performed by Terraform.

When prompted, enter yes to accept the plan and proceed with Vault configuration.

Plan: 24 to add, 0 to change, 0 to destroy.Do you want to perform these actions?  Terraform will perform the actions described above.  Only 'yes' will be accepted to approve.  Enter a value: yes

Once completed, the output similar to the following displays.

Apply complete! Resources: 24 added, 0 changed, 0 destroyed.Outputs:encoded = "4079-1004-8601-9194"

Verify the configuration

List the existing namespaces.

$ vault namespace list Keys----education/engineering/finance/

List the nested namespaces.

$ vault namespace list -namespace=education Keys----training/

List namespaces under training.

$ vault namespace list -namespace=education/training Keys----boundary/vault_cloud/

Verify that policies were created.
```
$ vault policy listadminsdefaultfpe-clientroot
```
Verify that admins policy was created under the finance namespace.
```
$ vault policy list -namespace=financeadminsdefault
```
Similarly, verify that admins policy was created under the engineering namespace.
```
$ vault policy list -namespace=engineeringadminsdefault
```
Verify that admins policy was created under the education namespace.
```
$ vault policy list -namespace=educationadminsdefault
```
Verify that admins policy was created under the education/training namespace.
```
$ vault policy list -namespace=education/trainingadminsdefault
```
Verify that admins policy was created under the education/training/vault_cloud namespace.
```
$ vault policy list -namespace=education/training/vault_cloudadminsdefault
```
Verify that admins policy was created under the education/training/boundary namespace.
```
$ vault policy list -namespace=education/training/boundaryadminsdefault
```

Verify that kv-v2 secrets engine is enabled in the finance namespace.

$ vault secrets list -namespace=finance Path          Type            Accessor                 Description ----          ----            --------                 ----------- cubbyhole/    ns_cubbyhole    ns_cubbyhole_f615f334    per-token private secret storage identity/     ns_identity     ns_identity_1ae45907     identity store kv-v2/        kv              kv_abd3188e              n/a sys/          ns_system       ns_system_e5b43cff       system endpoints used for control, policy and debugging

Verify the transformation secrets engine configuration for credit card numbers.

List existing transformations.

$ vault list transform/transformationKeys----ccn-fpe

Read the ccn-fpe transformation details.

$ vault read transform/transformation/ccn-fpeKey              Value---              -----allowed_roles    [payments]templates        [ccn]tweak_source     internaltype             fpe

List existing transformation templates.

$ vault list transform/templateKeys----builtin/creditcardnumberbuiltin/socialsecuritynumberccn

Read the ccn transformation template definition.

$ vault read transform/template/ccnKey               Value---               -----alphabet          numericsdecode_formats    map[]encode_format     n/apattern           (\d{4})-(\d{4})-(\d{4})-(\d{4})type              regex

Now, verify that you can log in with userpass auth method using the username, "student" and password, "changeme".

$ vault login -method=userpass username=studentPassword (will be hidden): Key                    Value---                    -----token                  s.5MadPX3UQgBRuukSHjf9yPJdtoken_accessor         AKvdeYEJspzN4fdOq0OcSSpstoken_duration         768htoken_renewable        truetoken_policies         ["admins" "default" "fpe-client"]identity_policies      []policies               ["admins" "default" "fpe-client"]token_meta_username    student

The generated token has admins and fpe-client policies attached. Now, take a look at the fpe-client policy definition.

$ cat policies/fpe-client-policy.hcl

The fpe-client policy permits update operation against the transform/encode/* and transform/decode/* paths.

# To request data encoding using any of the roles# Specify the role name in the path to narrow down the scopepath "transform/encode/*" {   capabilities = [ "update" ]} # To request data decoding using any of the roles# Specify the role name in the path to narrow down the scopepath "transform/decode/*" {   capabilities = [ "update" ]}

The student user should be able to perform format-preserving encryption (FPE) transformation.

$ vault write transform/encode/payments value=1111-2222-3333-4444 Key              Value---              -----encoded_value    6754-4542-1216-8423

Verify that you can decode the encoded value.

$ vault write transform/decode/payments value="6754-4542-1216-8423" Key              Value---              -----decoded_value    1111-2222-3333-4444

Successfully returns the original credit card number.

Verify that approle auth method is enabled in the education/training namespace.

$  vault auth list -namespace=education/training Path        Type        Accessor                  Description----        ----        --------                  -----------approle/    approle     auth_approle_077406b6     n/atoken/      ns_token    auth_ns_token_99292818    token based credentials

Verify that test-role exists.

$ vault list -namespace=education/training auth/approle/role Keys----test-role

Open a web browser and launch the Vault UI (e.g. http://127.0.0.1:8200/ui) and then login.
Verify that finance, engineering and education namespaces exist. The green check-mark indicates that you are currently logged into the root namespace.
Select the education namespace and verify that training namespace exists.
Select the training namespace and verify that vault_cloud and boundary namespaces exist.
Switch to the engineering namespace and verify that admins policy exists.
Switch to the finance namespace and verify that kv-v2 secrets engine is enabled.
Click the Policies tab to verify that admin policy exists.
Now, return to the root namespace and verify that transform secrets engine is enabled.
Click the Policies tab. Verify that admins and fpe-client policies exist.
Click the Vault CLI shell icon (>_) to open a command shell. Execute vault list transform/transformation in the CLI shell to list existing transformations.
Similarly, execute vault read transform/transformation/ccn-fpe in the CLI shell to read the ccn-fpe transformation details.
Execute vault list transform/template to list existing transformation templates. The output displays ccn in addition to the two built-in templates.
Execute vault read transform/template/ccn to read the ccn transformation template detail.
Sign out of the Vault UI.
In the sign in page, select Username auth method, enter student in the Username text field, and changeme in the Password text field.
Click the Vault CLI shell icon (>_) to open a command shell. Execute vault write transform/encode/payments value=1111-2222-3333-4444 to verify that the user, student can encode a credit card number using the payment role.
Now, execute vault write transform/decode/payments value="<encoded_value>" to verify that the user, student can decode the data.
Navigate to the education/training/ namespace and ensure that approle auth method is enabled.

Clean up

When you are done exploring, you can undo the configuration made by Terraform.

Make sure that VAULT_TOKEN and VAULT_ADDR environment variables are set.
```
$ echo $VAULT_ADDR; echo $VAULT_TOKEN
```

Destroy the Vault resources created by Terraform.

$ terraform destroy -auto-approve ...snip... Destroy complete! Resources: 24 destroyed.

Remove the terraform state files.
```
$ rm *.tfstate.*
```
Unset the VAULT_TOKEN and VAULT_ADDR environment variables.
```
$ unset VAULT_TOKEN VAULT_ADDR
```

Note

To learn more about Terraform, visit Learn Terraform.

Next steps

Treat your Terraform files like any other code and manage them through a version control system such as GitHub. You may integrate it with your favorite CI/CD tool (Jenkins, Travis CI, Circle CI, etc.), always review and test the configuration.

Travis CI example:

You can test your Terraform files against a development server that runs locally, or use a Docker image of Vault.

sudo: falseenv:  - VAULT_ADDR=<target_vault_address>  VAULT_TOKEN=<token>before_install:  - scripts/install.shbefore_script:  - vault server -dev -dev-root-token-id="$VAULT_TOKEN"script:  - terraform init  - terraform apply -auto-approve

Summary

In this guide you learned a technique for creating Vault policies and configurations using the Terraform Vault Provider. For more information, see the help and reference section.

Help and Reference

Terraform Vault Provider documentation page
Terraform Provider GitHub repository
Learn Terraform
Multi-tenancy with Namespaces

Tip

Terraform users can leverage the Vault's dynamic secrets engine to generate short-live cloud credentials when provisioning cloud resources. Inject secrets into Terraform using the Vault provider tutorial demonstrates the use of AWS secrets engine to manage AWS IAM credentials used by Terraform.

Codify management with Terraform

Deploy Consul and Vault on Kubernetes

This tutorial also appears in:

11 tutorials

Operations fundamentals
Learn basic Vault operations that are common to both Vault Community Edition and Vault Enterprise users.
- Vault
25 tutorials

Vault Enterprise
Learn features that are only available to Vault Enterprise.
- Vault
13 tutorials

Use Cases for Terraform
Use Terraform to perform common operations with other technologies, including Consul, Vault, Packer, and Kubernetes.
- Terraform
13 tutorials

HashiCorp Products - Better Together
Use Terraform with other Hashicorp products including Vault, Boundary, Consul, Packer, and Hashicorp Cloud Platform.
- Terraform