PKI Unified CRL and OCSP with cross cluster revocation

A PKI secrets engine in Vault servers at versions before 1.13.0 writes certificate revocation data to the cluster that issues a certificate. This means revocations which require information from the issued certificate must also occur on that same local cluster.

The challenge for end-users in Performance Replication environments becomes tracking which cluster issued a given leaf certificate to verify validity or revoke the certificate by its serial number.

Solution

Vault Enterprise version 1.13.0 introduces new opt-in features which help resolve this challenge:

• Unified CRL and OCSP functionality across clusters. Enabling this feature results in an eventually consistent, globally unified CRL based on the revocation entries of all clusters. With this feature enabled, Vault writes revocation entries from each cluster locally to the cluster that issued the certificate. It then also writes a revocation entry to the primary cluster that replicates across all Vault clusters.

• Asynchronous cross cluster revocation. When you opt in to this feature, you gain the ability to revoke certificates from clusters other than the issuing cluster.

Scenario introduction

In this scenario, you will configure and start 2 Vault Enterprise servers, which will act as the Performance Replication primary and secondary clusters.

You will use the Vault CLI to enable and configure 4 PKI Secrets Engine instances along with a username and password auth method.

You will also use the Vault CLI to enable Performance Replication between the 2 servers and issue leaf certificates from each of the 4 secrets engine roles.

To wrap up, you will query and revoke the certificates to learn more about the unified CRL and OCSP, and cross cluster revocation features.

Personas

The steps described in this tutorial are typically performed by a security engineer.

Prerequisites

To perform the tasks described in this tutorial, you need:

• A Vault Enterprise binary v1.13.0 or later installed and in your system PATH.
  • Vault token with policies to manage Performance Replication and secrets engines (this tutorial uses the initial root token).
• jq binary installed and in system PATH for parsing JSON in terminal sessions.
• A text editor that you can use in terminal sessions to edit scripts.
• openssl binary installed and in system PATH for validating certificates. If your OS distribution provides an openssl binary, you can use it for this tutorial.
• Familiarity with the Vault Enterprise Performance Replication feature; if you are not familiar, you can learn more in Setting up Performance Replication.
• Familiarity with the Vault PKI Secrets Engine; if you are not familiar, you can learn more in Build Your Own Certificate Authority (CA).

Lab setup

The lab for this scenario is a simplified Performance Replication setup made up of 2 Vault Enterprise servers, which represent the primary and secondary clusters.

You will prepare the environment by making directories and writing configuration files.

You will then initialize, and unseal the Vault server for each cluster.

For the sake of simplicity in this lab, you will use the initial root token for all steps in this scenario.

Create lab home

You can create a temporary directory to hold all the content needed for this scenario, and then assign its path to an environment variable for later reference.

Open a terminal, and create the directory /tmp/learn-vault-pki.

$ mkdir /tmp/learn-vault-pki

Export the directory path as the value to the HC_LEARN_LAB environment variable.

$ export HC_LEARN_LAB=/tmp/learn-vault-pki

Set up primary cluster server

The primary cluster server uses Integrated Storage and listens on 127.0.0.1:8200 for client communications.

1. Export your Vault Enterprise license string as the value of the VAULT_LICENSE environment variable.

   $ export VAULT_LICENSE=02MV4UU43BK5....

2. Create the primary server configuration and data directories.

   $ mkdir -p $HC_LEARN_LAB/primary-cluster/{config,data}

3. Write the primary server configuration file.

   $ cat > $HC_LEARN_LAB/primary-cluster/config/server.hcl << EOFdisable_mlock = trueui            = trueapi_addr      = "http://127.0.0.1:8200"cluster_addr  = "http://127.0.0.1:8201" listener "tcp" {  address            = "0.0.0.0:8200"  tls_disable        = "true"} storage "raft" {  path    = "$HC_LEARN_LAB/primary-cluster/data"  node_id = "primary-cluster"}EOF

4. Start the primary cluster Vault server.

   $ vault server -config $HC_LEARN_LAB/primary-cluster/config/server.hcl

5. Open another terminal session, and use it to export an environment variable for the vault CLI to address the primary server.

   $ export VAULT_ADDR=http://127.0.0.1:8200

6. For the purposes of this scenario, you can initialize the server with just 1 key share and a key threshold of 1. Redirect the initialization information to the file $HC_LEARN_LAB/primary-cluster/.init.

   $ vault operator init -key-shares=1 -key-threshold=1 \    > $HC_LEARN_LAB/primary-cluster/.init

7. Export the primary cluster server unseal key as the value to the PRIMARY_UNSEAL_KEY environment variable.

   $ export PRIMARY_UNSEAL_KEY="$(grep 'Unseal Key 1' $HC_LEARN_LAB/primary-cluster/.init | awk '{print $NF}')"

8. Unseal the primary cluster server.

   $ vault operator unseal $PRIMARY_UNSEAL_KEY

   Example expected output:

   Key                     Value---                     -----Seal Type               shamirInitialized             trueSealed                  falseTotal Shares            1Threshold               1Version                 1.13.0+entBuild Date              2023-02-14T14:25:23ZStorage Type            raftCluster Name            vault-cluster-ab95b64aCluster ID              d91ceaa5-859d-f053-a9ac-12f13e89c644HA Enabled              trueHA Cluster              n/aHA Mode                 standbyActive Node Address     <none>Raft Committed Index    64Raft Applied Index      64

   The primary cluster server is ready.

Set up secondary cluster server

The secondary cluster server uses Integrated Storage and listens on 127.0.0.1:8210 for client communications.

1. Export your Vault Enterprise license string as the value of the VAULT_LICENSE environment variable.

   $ export VAULT_LICENSE=02MV4UU43BK5....

2. Create the secondary server configuration and data directories.

   $ mkdir -p $HC_LEARN_LAB/secondary-cluster/{config,data}

3. Write the secondary server configuration file.

   $ cat > $HC_LEARN_LAB/secondary-cluster/config/server.hcl << EOFdisable_mlock = trueui            = trueapi_addr      = "http://127.0.0.1:8210"cluster_addr  = "http://127.0.0.1:8211" listener "tcp" {  address            = "0.0.0.0:8210"  tls_disable        = "true"} storage "raft" {  path    = "$HC_LEARN_LAB/secondary-cluster/data"  node_id = "secondary-cluster"}EOF

4. Start the secondary cluster server.

   $ vault server -config $HC_LEARN_LAB/secondary-cluster/config/server.hcl

5. Open another terminal session, and in that one, export the directory path as the value to the HC_LEARN_LAB environment variable.

   $ export HC_LEARN_LAB=/tmp/learn-vault-pki

6. Change into the lab directory.

   $ cd $HC_LEARN_LAB

7. Export an environment variable for the vault CLI to address the secondary cluster server.

   $ export VAULT_ADDR=http://127.0.0.1:8210

8. For the purposes of this scenario, you can initialize the server with just 1 key share and a key threshold of 1. Redirect the initialization information to the file $HC_LEARN_LAB/secondary-cluster/.init.

   $ vault operator init -key-shares=1 -key-threshold=1 \    > $HC_LEARN_LAB/secondary-cluster/.init

9. Export the secondary cluster server unseal key as the value to the SECONDARY_UNSEAL_KEY environment variable.

   $ export SECONDARY_UNSEAL_KEY="$(grep 'Unseal Key 1' $HC_LEARN_LAB/secondary-cluster/.init | awk '{print $NF}')"

10. Unseal the secondary cluster server.

    $ vault operator unseal $SECONDARY_UNSEAL_KEY

    Example expected output:

    Key                     Value---                     -----Seal Type               shamirInitialized             trueSealed                  falseTotal Shares            1Threshold               1Version                 1.13.0+entBuild Date              2023-02-14T14:25:23ZStorage Type            raftCluster Name            vault-cluster-c272fe50Cluster ID              62b78af3-9fad-eaac-17d4-3d3f0b0a64caHA Enabled              trueHA Cluster              n/aHA Mode                 standbyActive Node Address     <none>Raft Committed Index    64Raft Applied Index      64

    The secondary cluster server is ready.

Enable PKI secrets engines on primary cluster server

The goal of this section is for you to enable and configure 4 instances of the PKI secrets engine, each with a different set of features available.

This way, you can learn about revocation behaviors in each case.

You will use a script to enable the following named secrets engine engines on the primary server:

• pki-local: No unified revocation features enabled.
• pki-unified: You will enable the unified CRL feature.
• pki-cross: You will enable both unified CRL and cross cluster revocation features.

1. Export an environment variable for the vault CLI to address the primary server.

   $ export VAULT_ADDR=http://127.0.0.1:8200

2. Export the primary cluster server initial root token as the value to the VAULT_TOKEN environment variable.

   $ export VAULT_TOKEN="$(grep 'Initial Root Token' $HC_LEARN_LAB/primary-cluster/.init | awk '{print $NF}')"

3. To save time and typing, you will use a script to enable and configure secrets engines based on steps like those in Build Your Own Certificate Authority (CA).

   Use a text editor to write and save the example shell script content to the file enable_engines.sh.

   enable_engines.sh

   #!/bin/shfor engine in local unified cross bothdo   vault secrets enable -path "pki-$engine" pkidonefor engine in local unified cross bothdo   vault secrets tune -max-lease-ttl=87600h "pki-$engine"donefor engine in local unified cross bothdo   vault write -field=certificate "pki-$engine/root/generate/internal" \      common_name="$engine.example.com" \      issuer_name="$engine-root-2023" \      ttl=87600h > "$engine-local-root-2023-ca.crt"   donefor engine in local unified cross bothdo   vault write "pki-$engine/roles/$engine-2023-servers" allow_any_name=true no_store=falsedonefor engine in local unified cross bothdo   vault write pki-$engine/config/urls \      issuing_certificates="$VAULT_ADDR/v1/pki-$engine/ca" \      crl_distribution_points="$VAULT_ADDR/v1/pki-$engine/crl"donefor engine in local unified cross bothdo   vault secrets enable -path=pki-int-$engine pkidonefor engine in local unified cross bothdo   vault secrets tune -max-lease-ttl=43800h "pki-int-$engine"donefor engine in local unified cross bothdo   vault write -field=csr "pki-int-$engine/intermediate/generate/internal" \      common_name="$engine.example.com Intermediate Authority" \      issuer_name="$engine-example-dot-com-intermediate" \      > "pki-int-$engine.csr"donefor engine in local unified cross bothdo   vault write -field=certificate pki-$engine/root/sign-intermediate \      issuer_ref="$engine-root-2023" \      csr=@pki-int-$engine.csr \      format=pem_bundle ttl="43800h" \      > "pki-int-$engine.cert.pem"donefor engine in local unified cross bothdo   vault write "pki-int-$engine/intermediate/set-signed" \      certificate=@"pki-int-$engine.cert.pem"donefor engine in local unified cross bothdo   vault write pki-int-$engine/roles/$engine-example-dot-com \      issuer_ref="$(vault read -field=default pki-int-$engine/config/issuers)" \      allowed_domains="$engine.example.com" \      allow_subdomains=true \      max_ttl="720h" \      no_store=falsedone

4. Make the script executable.

   $ chmod +x enable_engines.sh

5. Execute the script to enable and configure the secrets engines.

   $ ./enable_engines.sh

   Abbreviated expected output:

   Success! Enabled the pki secrets engine at: pki-local/Success! Enabled the pki secrets engine at: pki-unified/Success! Enabled the pki secrets engine at: pki-cross/Success! Enabled the pki secrets engine at: pki-both/Success! Tuned the secrets engine at: pki-local/Success! Tuned the secrets engine at: pki-unified/Success! Tuned the secrets engine at: pki-cross/Success! Tuned the secrets engine at: pki-both/...snip...signature_bits                        256street_address                        []ttl                                   0suse_csr_common_name                   trueuse_csr_sans                          trueuse_pss                               false

6. List enabled secrets engines.

   $ vault secrets list

   Example expected output:

   Path                Type         Accessor              Description----                ----         --------              -----------cubbyhole/          cubbyhole    cubbyhole_f84fa2f4    per-token private secret storageidentity/           identity     identity_67f57adf     identity storepki-both/           pki          pki_c8c83cd0          n/apki-cross/          pki          pki_a9cf7255          n/apki-int-both/       pki          pki_99e7f751          n/apki-int-cross/      pki          pki_18b227b7          n/apki-int-local/      pki          pki_187f3e0d          n/apki-int-unified/    pki          pki_db7cd9d8          n/apki-local/          pki          pki_c49dd19e          n/apki-unified/        pki          pki_2a596159          n/asys/                system       system_8f6f7690       system endpoints used for control, policy and debugging

Configure PKI secrets engines

The unified CRL and OCSP, and cross cluster revocation features are opt-in.

When you wrote and ran the shell script that enables and configures the PKI secrets engines, it did not configure the secrets engines with the new features. This means that 3 of the 4 PKI secrets engines require configuration updates. You can update them through the Set Revocation Configuration API endpoint.

The goal of this section is to learn about these configuration parameters, and apply them to the 3 secrets engines which require configuration updates.

1. Check the current CRL configuration for the pki-int-unified secrets engine.

   $ vault read pki-int-unified/config/crl

   Example expected output:

   Key                              Value---                              -----auto_rebuild                     falseauto_rebuild_grace_period        12hcross_cluster_revocation         falsedelta_rebuild_interval           15mdisable                          falseenable_delta                     falseexpiry                           72hocsp_disable                     falseocsp_expiry                      12hunified_crl                      falseunified_crl_on_existing_paths    false

   Recall that the features covered in this tutorial are opt-in, so unified_crl is false as expected.

2. The pki-int-unified secrets engine needs configuration to use just the unified CRL and OCSP feature and allow it for existing paths. Enabling this feature requires that you also set the auto_rebuild parameter to true, as unified CRLs cannot be rebuilt on every revocation.

   $ vault write pki-int-unified/config/crl \    auto_rebuild=true \    unified_crl=true \    unified_crl_on_existing_paths=true

   Example expected output:

   Key                              Value---                              -----auto_rebuild                     trueauto_rebuild_grace_period        12hcross_cluster_revocation         falsedelta_rebuild_interval           15mdisable                          falseenable_delta                     falseexpiry                           72hocsp_disable                     falseocsp_expiry                      12hunified_crl                      trueunified_crl_on_existing_paths    true

3. The pki-int-cross secrets engine needs configuration to use just the cross cluster revocation feature. Enabling this feature requires that you also set the auto_rebuild parameter to true, as the local cluster may not have the certificate entry when auto_rebuild is not enabled.

   $ vault write pki-int-cross/config/crl \       auto_rebuild=true \       cross_cluster_revocation=true

   Example expected output:

   Key                              Value---                              -----auto_rebuild                     trueauto_rebuild_grace_period        12hcross_cluster_revocation         truedelta_rebuild_interval           15mdisable                          falseenable_delta                     falseexpiry                           72hocsp_disable                     falseocsp_expiry                      12hunified_crl                      falseunified_crl_on_existing_paths    false

4. The pki-int-both secrets engine needs configuration to use both the unified CRL and OCSP, and cross cluster revocation features. Since this secrets engine combines both features, you must also set the auto_rebuild parameter to true.

   $ vault write pki-int-both/config/crl \    auto_rebuild=true \    unified_crl=true \    unified_crl_on_existing_paths=true \    cross_cluster_revocation=true

   Example expected output:

   Key                              Value---                              -----auto_rebuild                     trueauto_rebuild_grace_period        12hcross_cluster_revocation         truedelta_rebuild_interval           15mdisable                          falseenable_delta                     falseexpiry                           72hocsp_disable                     falseocsp_expiry                      12hunified_crl                      trueunified_crl_on_existing_paths    true

You have completed the PKI secrets engine configuration, and the PKI secrets engines are ready for certificate issuance.

Enable Performance Replication

The goal of this section is for you to enable a userpass auth method instance, a user and password, and Performance Replication between the primary and secondary cluster servers.

Primary server

Before enabling performance replication, create a superuser policy and enable userpass auth method with tester user on the primary cluster.

1. Create a superuser ACL policy.

   $ vault policy write superuser - <<EOFpath "*" {capabilities = ["create", "read", "update", "delete", "list", "sudo"]}EOF

2. Enable the userpass auth method.

   $ vault auth enable userpass

3. Create a new user named tester in userpass with the password changeme, and attach the superuser policy.

   $ vault write auth/userpass/users/tester password="changeme" policies="superuser"

Later, you can authenticate with Vault using the userpass auth method.

Use the CLI to enable Performance Replication on the primary and secondary servers.

1. Enable replication on the primary cluster server.

   $ vault write -f sys/replication/performance/primary/enable

   Example expected output:

   WARNING! The following warnings were returned from Vault:* This cluster is being enabled as a primary for replication. Vault will beunavailable for a brief period and will resume service shortly.

2. Generate a secondary activation token and assign its value to the SECONDARY_ACTIVATION_TOKEN environment variable.

   $ export SECONDARY_ACTIVATION_TOKEN="$(vault write -field wrapping_token sys/replication/performance/primary/secondary-token id=secondary)"

   This command is not expected to produce output, but you can verify the environment variable if you like.

   $ echo $SECONDARY_ACTIVATION_TOKEN

   Expected example output:

   eyJhbGciOiJFUzUxMiIsInR5cCI6IkpXVCJ9.eyJhY2Nlc3NvciI6IiIsImFkZHIiOiJodHRwOi8vMTI3LjAuMC4xOjgyMDAiLCJleHAiOjE2NzY5OTgyNTYsImlhdCI6MTY3Njk5NjQ1NiwianRpIjoiaHZzLlJMYmVvbFlsY0Y0MUQwc0hJQjdZc2ZhciIsIm5iZiI6MTY3Njk5NjQ1MiwidHlwZSI6IndyYXBwaW5nIn0.AfnDspZjJ6MRHACJ6_lpAqoPajSJTtpr5421a2pA6x_C96Fj2TPfLrVmojgyiMkJuV-XEyT_yj02qUW3bIKy4WQiAHXXM8FkpvFD0uqKYcsGQuwrzJNcD1X6caDKZRApZGstsuUb3yws87Qjcd9R1dNWOsmTekDV-rPU2cnQqAAX2JmA

Secondary server

1. Export an environment variable for the vault CLI to address the secondary server.

   $ export VAULT_ADDR=http://127.0.0.1:8210

2. Export the secondary cluster server initial root token as the value to the VAULT_TOKEN environment variable.

   $ export VAULT_TOKEN="$(grep 'Initial Root Token' $HC_LEARN_LAB/secondary-cluster/.init | awk '{print $NF}')"

3. Enable replication on the secondary server with the secondary activation token.

   Warning

   This will clear all data in the secondary cluster!

   $ vault write sys/replication/performance/secondary/enable \     token=$SECONDARY_ACTIVATION_TOKEN

   Expected example output:

   WARNING! The following warnings were returned from Vault: * Vault has successfully found secondary information; it may take a while to perform setup tasks. Vault will be unavailable until these tasks and initial sync complete.

Issue certificates

The goal of this section is for you to issue a leaf certificate from each of the PKI secrets engine roles on the secondary cluster server.

You can then try revoking the certificates and observing the different behaviors across each PKI secrets engine instance.

1. Unset the VAULT_TOKEN environment variable.

   $ unset VAULT_TOKEN

2. Before issuing the certificates, you must first use the userpass auth method to authenticate with the secondary cluster server.

   $ vault login -no-print -method userpass username=tester password=changeme

   This command produces no output, but you can confirm that the token helper cached the token value by displaying part of it.

   $ cat ~/.vault-token | cut -d '.' -f1

   Example expected output:

   hvs

   Tip

   If you meet with an error message containing * local node not active but active cluster node not found, when you try to login with the userpass auth method this means that the replication hasn't finished syncing. Wait a moment and try the command again.

3. Issue a certificate from the pki-int-local PKI secrets engine local-example-dot-com role.

   $ vault write \    pki-int-local/issue/local-example-dot-com \    common_name="test.local.example.com" \    ttl="1h" -format=json > test.local.example.com.json

   Store the serial number in a test.local.example.com.serial.txt file.

   $ cat test.local.example.com.json | jq -r '.data.serial_number' > test.local.example.com.serial.txt

   Store the certificate in the test.local.example.com.crt file.

   $ cat test.local.example.com.json | jq -r '.data.certificate' > test.local.example.com.crt

4. Issue a certificate from the pki-int-unified PKI secrets engine unified-example-dot-com role.

   $ vault write \    pki-int-unified/issue/unified-example-dot-com \    common_name="test.unified.example.com" \    ttl="1h" -format=json > test.unified.example.com.json

   Store its serial number in the test.unified.example.com.serial.txt file.

   $ cat test.unified.example.com.json | jq -r '.data.serial_number' > test.unified.example.com.serial.txt

   Store the certificate in the test.unified.example.com.crt file.

   $ cat test.unified.example.com.json | jq -r '.data.certificate' > test.unified.example.com.crt'

5. Issue a certificate from the pki-int-cross PKI secrets engine cross-example-dot-com role.

   $ vault write \    pki-int-cross/issue/cross-example-dot-com \    common_name="test.cross.example.com" \    ttl="1h" -format=json > test.cross.example.com.json

   Store its serial number in the test.cross.example.com.serial.txt file.

   $ cat test.cross.example.com.json | jq -r '.data.serial_number' > test.cross.example.com.serial.txt

   Store the certificate in the test.cross.example.com.crt file.

   $ cat test.cross.example.com.json | jq -r '.data.certificate' > test.cross.example.com.crt

6. Issue a certificate from the pki-int-both PKI secrets engine both-example-dot-com role.

   $ vault write \    pki-int-both/issue/both-example-dot-com \    common_name="test.both.example.com" \    ttl="1h" -format=json > test.both.example.com.json

   Store its serial number in the test.both.example.com.serial.txt file.

   $ cat test.both.example.com.json | jq -r '.data.serial_number' > test.both.example.com.serial.txt

   Store the certificate in the test.both.example.com.crt file.

   $ cat test.both.example.com.json | jq -r '.data.certificate' > test.both.example.com.crt

Explore behavior without features

The goal of this section is for you to query and revoke the leaf certificate from the pki-int-local PKI secrets engine. In doing so, you learn more about the PKI secrets engine behavior without the unified CRL and OCSP feature.

1. Use openssl to query the secondary cluster server OCSP about the certificate that you issued from the pki-int-local PKI secrets engine.

   $ openssl ocsp \    -noverify \    -no_nonce \    -issuer pki-int-local.cert.pem \    -cert test.local.example.com.crt \    -url http://127.0.0.1:8210/v1/pki-int-local/ocsp

   Example expected output:

   test.local.example.com.crt: good    This Update: Feb 26 17:16:09 2023 GMT    Next Update: Feb 27 05:16:09 2023 GMT

   According to the secondary cluster OCSP, the certificate is valid.

2. Try revoking the certificate by its serial number through the primary cluster server. For convenience, you can set the environment variables for the primary cluster server and token as part of the command.

   $ VAULT_ADDR=http://127.0.0.1:8200 \  VAULT_TOKEN="$(grep 'Initial Root Token' $HC_LEARN_LAB/primary-cluster/.init | awk '{print $NF}')" \      vault write pki-int-local/revoke \      serial_number=$(cat test.local.example.com.serial.txt)

   Instead of success, Vault outputs an error:

   Error writing data to pki-int-local/revoke: Error making API request.URL: PUT http://127.0.0.1:8200/v1/pki-int-local/revokeCode: 400. Errors:* certificate with serial 09:82:7f:46:5e:92:d1:aa:7e:c3:3f:3e:88:bb:d6:84:e8:ea:97:d7 not found.

   The primary cluster is not aware of the certificate issued from the secondary cluster. This demonstrates correct and expected behavior from the pki-int-local secrets engine, since it isn't configured with support for unified CRL and OCSP.

3. Try revoking the certificate by its serial number through the secondary cluster server. For convenience, you can set the environment variables for the secondary cluster server and token as part of the command.

   $ VAULT_ADDR=http://127.0.0.1:8210 \  VAULT_TOKEN="$(cat ~/.vault-token)" \      vault write pki-int-local/revoke \      serial_number=$(cat test.local.example.com.serial.txt)

   The certificate state shows a value of revoked.

   Key                        Value---                        -----revocation_time            1677431870revocation_time_rfc3339    2023-02-26T17:17:50.560556Zstate                      revoked

   You revoked the certificate.

Explore unified CRL & OCSP behavior

The goal of this section is for you to revoke the leaf certificate from the pki-int-unified PKI secrets engine. Then, you will check the list of revoked certificates on the secondary cluster server and primary cluster server. In doing so, you learn more about the PKI secrets engine behavior with the unified CRL and OCSP feature enabled.

1. Use openssl to query the secondary cluster server OCSP about the certificate that you issued from the pki-int-unified PKI secrets engine.

   $ openssl ocsp \    -noverify \    -no_nonce \    -issuer pki-int-unified.cert.pem \    -cert test.unified.example.com.crt \    -url http://127.0.0.1:8210/v1/pki-int-unified/ocsp

   Example expected output:

   test.unified.example.com.crt: good    This Update: Feb 26 17:18:58 2023 GMT    Next Update: Feb 27 05:18:58 2023 GMT

   According to the secondary cluster OCSP, the certificate is valid.

2. Revoke the certificate by its serial number through the secondary cluster server. For convenience, you can set the environment variables for the secondary cluster server and token as part of the command.

   $ VAULT_ADDR=http://127.0.0.1:8210 \  VAULT_TOKEN="$(cat ~/.vault-token)" \      vault write pki-int-unified/revoke \      serial_number=$(cat test.unified.example.com.serial.txt)
   Key                        Value---                        -----revocation_time            1677431976revocation_time_rfc3339    2023-02-26T17:19:36.438238Zstate                      revoked

   You revoked the certificate.

3. List revoked certificates on the secondary cluster server.

   $ VAULT_ADDR=http://127.0.0.1:8210 \  VAULT_TOKEN="$(cat ~/.vault-token)" \      vault list /pki-int-unified/certs/revoked

   Example expected output:

   Keys----52:21:bd:4b:3c:12:b0:93:1b:0f:6e:c7:e9:9e:90:79:54:58:8e:fb

   You revoked the certificate.

   Without the unified CRL and OCSP feature, you cannot expect the primary cluster server to have knowledge of the certificate. With it enabled, the primary cluster server should also show the certificate as revoked.

4. Use openssl to query the primary cluster server OCSP about the certificate.

   $ openssl ocsp \     -noverify \     -no_nonce \     -issuer pki-int-unified.cert.pem \     -cert test.unified.example.com.crt \     -url http://127.0.0.1:8200/v1/pki-int-unified/ocsp

   Example expected output:

   test.unified.example.com.crt: revoked    This Update: Feb 26 17:50:25 2023 GMT    Next Update: Feb 26 17:52:25 2023 GMT    Revocation Time: Feb 26 17:19:36 2023 GMT

   With unified CRL and OCSP, the secondary cluster server now also shows the certificate revocation.

Explore cross cluster revocation behavior

The goal of this section is for you to revoke the leaf certificate from the pki-int-cross PKI secrets engine. In doing so, you learn more about the PKI secrets engine behavior with the cross cluster revocation feature enabled.

1. Use openssl to query the secondary cluster server OCSP about the certificate that you issued from the pki-int-cross PKI secrets engine.

   $ openssl ocsp \    -noverify \    -no_nonce \    -issuer pki-int-cross.cert.pem \    -cert test.cross.example.com.crt \    -url http://127.0.0.1:8210/v1/pki-int-cross/ocsp

   Example expected output:

   test.cross.example.com.crt: good    This Update: Feb 26 17:41:41 2023 GMT    Next Update: Feb 27 05:41:41 2023 GMT

   According to the secondary cluster OCSP, the certificate is valid.

2. Try to revoke the certificate by its serial number through the primary cluster server. For convenience, you can set the environment variables for the primary cluster server and token as part of the command.

   $ VAULT_ADDR=http://127.0.0.1:8200 \  VAULT_TOKEN="$(grep 'Initial Root Token' $HC_LEARN_LAB/primary-cluster/.init | awk '{print $NF}')" \      vault write pki-int-cross/revoke \      serial_number=$(cat test.cross.example.com.serial.txt)

   Example expected output:

   WARNING! The following warnings were returned from Vault:* Revocation request was not found on this present node. This request willbe in a pending state until the PR cluster which issued this certificatesees the request and revokes the certificate. If no online cluster has thiscertificate, the request will eventually be removed without revoking anycertificates.Key      Value---      -----state    pending

   Vault warns that it did not find a revocation request on the primary cluster server, and that the request is now pending. This means that with cross cluster revocation enabled, the secondary will learn of the request and revoke the certificate.

3. Try querying the secondary cluster server for the certificate from pki-int-cross.

   $ openssl ocsp \    -noverify \    -no_nonce \    -issuer pki-int-cross.cert.pem \    -cert test.cross.example.com.crt \    -url http://127.0.0.1:8210/v1/pki-int-cross/ocsp

   Example expected output:

   test.cross.example.com.crt: revoked    This Update: Feb 26 17:42:54 2023 GMT    Next Update: Feb 27 05:42:54 2023 GMT    Revocation Time: Feb 26 17:42:48 2023 GMT

   The cross cluster revocation functionality revoked the certificate.

Explore unified CRL & OCSP with cross cluster revocation behavior

The goal of this section is for you to revoke the leaf certificate from the pki-int-both PKI secrets engine from the secondary cluster server. You will then query the certificate with OCSP from the primary cluster server. In doing so, you learn more about the PKI secrets engine behavior with both the unified CRL and OCSP, with cross cluster revocation features enabled.

1. Use openssl to query the secondary cluster server OCSP about the certificate that you issued from the pki-int-both PKI secrets engine.

   $ openssl ocsp \    -noverify \    -no_nonce \    -issuer pki-int-both.cert.pem \    -cert test.both.example.com.crt \    -url http://127.0.0.1:8210/v1/pki-int-both/ocsp

   Example expected output:

   test.both.example.com.crt: good    This Update: Feb 26 17:41:41 2023 GMT    Next Update: Feb 27 05:41:41 2023 GMT

   According to the secondary cluster OCSP, the certificate is valid.

2. Try to revoke the certificate by its serial number through the primary cluster server. For convenience, you can set the environment variables for the primary cluster server and token as part of the command.

   $ VAULT_ADDR=http://127.0.0.1:8200 \  VAULT_TOKEN="$(grep 'Initial Root Token' $HC_LEARN_LAB/primary-cluster/.init | awk '{print $NF}')" \      vault write pki-int-both/revoke \      serial_number=$(cat test.both.example.com.serial.txt)

   Example expected output:

   WARNING! The following warnings were returned from Vault:* Revocation request was not found on this present node. This request willbe in a pending state until the PR cluster which issued this certificatesees the request and revokes the certificate. If no online cluster has thiscertificate, the request will eventually be removed without revoking anycertificates.Key      Value---      -----state    pending

   Vault warns that it did not find a revocation request on the primary cluster server, and that the request is now pending. This means that with cross cluster revocation enabled, the secondary will learn of the request and revoke the certificate.

3. Try querying the secondary cluster server for the certificate from pki-int-both.

   $ openssl ocsp \    -noverify \    -no_nonce \    -issuer pki-int-both.cert.pem \    -cert test.both.example.com.crt \    -url http://127.0.0.1:8210/v1/pki-int-both/ocsp

   Example expected output:

   test.both.example.com.crt: revoked    This Update: Feb 26 17:42:54 2023 GMT    Next Update: Feb 27 05:42:54 2023 GMT    Revocation Time: Feb 26 17:42:48 2023 GMT

4. Use openssl to query the primary cluster server OCSP about the certificate.

   $ openssl ocsp \     -noverify \     -no_nonce \     -issuer pki-int-both.cert.pem \     -cert test.both.example.com.crt \     -url http://127.0.0.1:8200/v1/pki-int-both/ocsp

   Example expected output:

   test.both.example.com.crt: revoked    This Update: Feb 26 17:42:54 2023 GMT    Next Update: Feb 27 05:42:54 2023 GMT    Revocation Time: Feb 26 17:42:48 2023 GMT

   The cross cluster revocation functionality revoked the certificate. Both the primary and secondary server clusters are aware of the revocation.

Summary

You established a lab environment with PKI secrets engines, the userpass auth method, and Performance Replication enabled.

You also learned about the PKI secrets engine unified CRL and OCSP, and cross cluster revocation features and their capabilities by issuing, querying, and revoking certificates.

Usage caveats

Size limitations exist for individual entries within Vault depending on the chosen storage backend.

If your use cases accumulate a large quantities of revoked certificates across various clusters, the unified CRL could hit limits and become unusable.

If a large amount of revocations are occurring, the traffic from the additional writes across all the clusters and/or the background/periodic threads within Vault could overload resources.

Consult Vault Limits and Maximums for more details.

Clean up

1. Stop the Vault server processes.

   $ pkill vault

2. Change into your home directory.

   $ cd

3. Remove the lab directory.

   $ rm -rf $HC_LEARN_LAB

4. Unset the environment variables.

   $ unset HC_LEARN_LAB \    PRIMARY_UNSEAL_KEY \    SECONDARY_ACTIVATION_TOKEN \    SECONDARY_UNSEAL_KEY \    VAULT_ADDR \    VAULT_LICENSE \    VAULT_TOKEN \

5. Close any unneeded terminal sessions.

Next steps

To continue learning about the Vault PKI secrets engine or Performance Replication, check out the following tutorials:

• Build Certificate Authority (CA) in Vault with an offline Root
• PKI Secrets Engine with Managed Keys
• HCP Vault Dedicated Performance Replication

Help and reference

• Build Your Own Certificate Authority (CA)
• Setting up Performance Replication