Protect against resource exhaustion

29min
|
Vault
Interactive

Vault is an API driven system that all communication between the clients and Vault are done through Vault API.

As the number of client applications increases, rogue applications can degrade Vault performance. The issue is often caused by applications that are:

Generating an unbounded number of leases and/or loading too much data into Vault leading to exhausting Consul storage backend's available memory
Consuming an erroneously large amount of bandwidth leading to internal throttling of the Vault cluster as a whole

Solution

Set Vault resource quotas to protect your Vault environment's stability and network, as well as storage resource consumption from runaway application behavior and distributed denial of service (DDoS) attack.

The Vault operators can control how applications request resources from Vault, and Vault's storage and network infrastructure by setting the following:

Feature	Description	Vault Community Edition	Vault Enterprise
Rate Limit Quotas	Limit maximum amount of requests per second (RPS) to a system or mount to protect network bandwidth	✔️	✔️
Lease Count Quotas	Cap number of leases generated in a system or mount to protect system stability and storage performance at scale		✔️

Note

Lease Count Quotas requires Vault Enterprise Standard license.

To set the rate limit quotas and lease count quotas, use the sys/quotas/<type> endpoint. Each resource quota has a name to identify the quota rule. To manage an individual quota rule, the endpoint becomes sys/quotas/<type>/<name>.

The <type> can be:

rate-limit: rate limit quota
lease-count: lease count quota

Note

Batch tokens do not count towards the lease count quota. Creation of new batch tokens will, however, be blocked if the lease count quota is exceeded.

Prerequisites

To perform the tasks described in this tutorial, you need Vault v1.5 or later. Refer to the Getting Started tutorial to install Vault. Make sure that your Vault server has been initialized and unsealed.

Launch Terminal

This tutorial includes a free interactive command-line lab that lets you follow along on actual cloud infrastructure.

Lab setup

Open a terminal and start a Vault dev server with root as the root token.
```
$ vault server -dev -dev-root-token-id root
```
The Vault dev server defaults to running at 127.0.0.1:8200. The server is also initialized and unsealed.
Insecure operation
Do not run a Vault dev server in production. This approach is only used here to simplify the unsealing process for this demonstration.
Export an environment variable for the vault CLI to address the Vault server.
```
$ export VAULT_ADDR=http://127.0.0.1:8200
```
Export an environment variable for the vault CLI to authenticate with the Vault server.
```
$ export VAULT_TOKEN=root
```

The Vault server is ready.

Resource quota configuration

To configure the resource quota, use the sys/quotas/config endpoint.

Parameter	Type	Description
`enable_rate_limit_audit_logging`	boolean	Enable or disable audit logging when requests get rejected due to rate limit quota violations. By default, audit logging is disabled (`false`).

By default, the requests rejected due to rate limit quota violations are not written to the audit log. Therefore, if you wish to log the rejected requests for traceability, you must set the enable_rate_limit_audit_logging to true. The requests rejected due to reaching the lease count quotas are always logged that you do not need to set any parameter.

Note

Enabling the rate limit audit logging may have an impact on the Vault performance if the volume of rejected requests is large.

Enable a file audit device which outputs to /var/log/vault-audit.log (or your desired file location).
```
$ vault audit enable file file_path="/var/log/vault-audit.log"
```
To enable the audit logging for rate limit quotas, execute the following command.
```
$ vault write sys/quotas/config enable_rate_limit_audit_logging=true
```

Read the quota configuration to verify.

$ vault read sys/quotas/config Key                                   Value---                                   -----enable_rate_limit_audit_logging       trueenable_rate_limit_response_headers    falserate_limit_exempt_paths               []

Enable file audit device.

First, create the HTTP request payload specifying the audit log path to be /var/log/vault-audit.log (or your desired file location).

$ tee audit-payload.json <<EOF{  "type": "file",  "options": {      "file_path": "/var/log/vault-audit.log"  }}EOF

Use the sys/audit endpoint to enable file audit log.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \    --request POST \    --data @audit-payload.json \    $VAULT_ADDR/v1/sys/audit/file

Set the target file_path to your desired location.

Create the HTTP request payload to enable the rate limit quota audit logging.
```
$ tee payload.json <<EOF{   "enable_rate_limit_audit_logging": true}EOF
```

Invoke the sys/quotas/config endpoint.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \    --request POST \    --data @payload.json \    $VAULT_ADDR/v1/sys/quotas/config

Read the quota configuration to verify.

$ curl -s --header "X-Vault-Token: $VAULT_TOKEN" \    $VAULT_ADDR/v1/sys/quotas/config | jq -r ".data"

Example output:

{  "enable_rate_limit_audit_logging": true,  "enable_rate_limit_response_headers": false,  "rate_limit_exempt_paths": []}

Rate limit quotas

Rate limit quotas are designed to protect Vault against external distributed denial of service (DDoS) attacks and are fundamental to Vault's security model. Therefore, it is a part of Vault's core feature set available in both Community and Enterprise editions.

To set rate limit quotas, use the sys/quotas/rate-limit/<name> endpoint.

Parameters

Parameter	Type	Description
`name`	string	Name of the quota rule
`path`	string	Target path or namespace to apply the quota rule. A blank path configures a global rate limit quota
`rate`	float	Rate for the number of allowed requests per second (RPS)
`role`	string	Vault 1.12 or later: Login role to apply this quota to. When this parameter is set, the path must be configured to a valid auth method with a concept of roles.
`interval`	second	The duration to enforce rate limiting for (default is 1 second)
`block_interval`	string	If set, when a client reaches a rate limit threshold, the client is prohibited from any further requests until after the 'block_interval' has elapsed.
`inheritable`	boolean	Vault Enterprise 1.15 or later: Determine whether to apply the quota rule to child namespaces (Refer to the Vault Enterprise namespaces section of this tutorial)

Note

If you are running Vault 1.12 or later, the path can be a fully qualified path, and it can end with * (e.g., auth/token/create*).

Create a rate limit quota named, "global-rate" which limits inbound workload to 1500 requests per second.

$ vault write sys/quotas/rate-limit/global-rate rate=1500Success! Data written to: sys/quotas/rate-limit/global-rate

Read the global-rate rule to verify its configuration.

$ vault read sys/quotas/rate-limit/global-rate Key               Value---               -----block_interval    0inheritable       trueinterval          1name              global-ratepath              n/arate              1500role              n/atype              rate-limit

Note

In absence of path, this quota rule applies to the global level instead of a specific mount or namespace.

Create a rate limit quota named, "transit-limit" which limits the access to the Transit secrets engine to be 1000 requests per minute (60 seconds).

First, enable Transit secrets engine at transit.

$ vault secrets enable transitSuccess! Enabled the pki secrets engine at: transit/

Now, create a rate limit quota.

$ vault write sys/quotas/rate-limit/transit-limit \   path="transit" \   rate=1000 \   interval=60

Output:

Success! Data written to: sys/quotas/rate-limit/transit-limit

Read the transit-limit rule to verify its configuration.

$ vault read sys/quotas/rate-limit/transit-limit

Output:

Key               Value---               -----block_interval    0inheritable       trueinterval          60name              transit-limitpath              transit/rate              1000role              n/atype              rate-limit

Rate limit is set to 1000 requests per 60 seconds.

Path granularity

If you are running Vault 1.12 or later, you can set the path to be deeper than the mount point (in this example, transit/).

Create a rate limit quota named, "transit-order" to limit the data encryption requests using orders key to be 500 per second.

First, create an encryption key named, "orders".

$ vault write -f transit/keys/orders Key                       Value---                       -----allow_plaintext_backup    falseauto_rotate_period        0sdeletion_allowed          falsederived                   falseexportable                falseimported_key              falsekeys                      map[1:1695147293]latest_version            1min_available_version     0min_decryption_version    1min_encryption_version    0name                      orderssupports_decryption       truesupports_derivation       truesupports_encryption       truesupports_signing          falsetype                      aes256-gcm96

Now, create the "transit-order" rate limit quota.

$ vault write sys/quotas/rate-limit/transit-order \   path="transit/encrypt/orders" \   rate=500

Output:

Success! Data written to: sys/quotas/rate-limit/transit-order

Verify the rate limit quota configuration.

$ vault read sys/quotas/rate-limit/transit-order

Output:

Key               Value---               -----block_interval    0inheritable       trueinterval          1name              transit-orderpath              transit/encrypt/ordersrate              500role              n/atype              rate-limit

Create a rate limit quota named, "global-rate" which limits inbound workload to 1500 requests per second.

Invoke the sys/quotas/rate-limit endpoint.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \    --request POST \    --data '{ "rate": 1500 }' \    $VAULT_ADDR/v1/sys/quotas/rate-limit/global-rate

Read the newly created global-rate quota rule.

$ curl -s --header "X-Vault-Token: $VAULT_TOKEN" \    $VAULT_ADDR/v1/sys/quotas/rate-limit/global-rate | jq -r ".data"

Output:

{   "block_interval": 0,   "inheritable": true,   "interval": 1,   "name": "global-rate",   "path": "",   "rate": 1500,   "role": "",   "type": "rate-limit"}

Note

In absence of path, this quota rule applies to the root namespace instead of a specific mount or namespace.

Create a rate limit quota named, "transit-limit" which limits the access to the Transit secrets engine to be 1000 requests per minute (60 seconds).

First, enable Transit secrets engine at transit.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \    --request POST \    --data '{"type":"transit"}' \    $VAULT_ADDR/v1/sys/mounts/transit

Now, create a "transit-limit" rate limit quota.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \   --request POST \   --data '{"path": "transit", "rate": 1000, "interval": 60 }' \   $VAULT_ADDR/v1/sys/quotas/rate-limit/transit-limit

Read the transit-limit rule to verify its configuration.

$ curl -s --header "X-Vault-Token: $VAULT_TOKEN" \   $VAULT_ADDR/v1/sys/quotas/rate-limit/transit-limit | jq -r ".data"

Output:

{   "block_interval": 0,   "inheritable": true,   "interval": 60,   "name": "transit-limit",   "path": "transit/",   "rate": 1000,   "role": "",   "type": "rate-limit"}

Rate limit is set to 1000 requests per 60 seconds.

Path granularity

If you are running Vault 1.12 or later, you can set the path to be deeper than the mount point (in this example, transit/).

Create a rate limit quota named, "transit-order" to limit the data encryption requests using orders key to be 500 per second.

First, create an encryption key named, "orders".

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \    --request POST \    $VAULT_ADDR/v1/transit/keys/orders

Now, create the "transit-order" rate limit quota.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \   --request POST \   --data '{ "path": "transit/encrypt/orders", "rate": 500 }' \   $VAULT_ADDR/v1/sys/quotas/rate-limit/transit-order

Verify the rate limit quota configuration.

$ curl -s --header "X-Vault-Token: $VAULT_TOKEN" \   $VAULT_ADDR/v1/sys/quotas/rate-limit/transit-order | jq -r ".data"

Output:

{   "block_interval": 0,   "inheritable": true,   "interval": 1,   "name": "transit-order",   "path": "transit/encrypt/orders",   "rate": 500,   "role": "",   "type": "rate-limit"}

Next step

Be sure to visit the Test to understand the resource quota section to see how resource quotas enforcement works.

If you are running Vault Enterprise, read the Vault Enterprise namepaces section as well.

Lease count quotas

Vault Enterprise

Lease count quotas is a part of Vault Enterprise features.

Lease count quota is designed to protect Vault from a large volume of leases and tokens persisted in Vault, which can pressure its storage backend. This acts as a guard rail for system stability in large-scale Vault Enterprise deployments where Vault is used as a service.

Parameters

Parameter	Type	Description
`name`	string	Name of the quota rule
`path`	string	Target path or namespace to apply the quota rule. A blank path configures a global lease count quota. The `path` can be a fully qualified path, and it can end with `` (e.g., `auth/token/create`).
`max_leases`	int	Maximum number of leases allowed by the quota rule.
`role`	string	Vault 1.12 or later: Login role to apply this quota to. When this parameter is set, the path must be configured to a valid auth method with a concept of roles.
`inheritable`	boolean	Vault Enterprise 1.15 or later: Determine whether to apply the quota rule to child namespaces (Refer to the Vault Enterprise namespaces section of this tutorial).

The following examples set lease count quotas on a speific scope. Vault Enterprise 1.16 introduced a default lease count quota that you can set on a global level. The global default is available for new Vault installations. To learn more, go through the global default lease count quota section.

Create a namespace, "us-west".

$ vault namespace create us-west Key                Value---                -----custom_metadata    map[]id                 AdrqMpath               us-west/

To demonstrate this feature, enable database secrets engine at "postgres" in the us-west namespace.

$ vault secrets enable -ns=us-west -path=postgres databaseSuccess! Enabled the database secrets engine at: postgres/

Create a lease count quota named, "db-creds" which limits the incoming requests for a new set of DB credentials to 100 concurrent, valid leases maximum.
```
$ vault write sys/quotas/lease-count/db-creds \   max_leases=100 \   path="us-west/postgres"
```
Output:
```
Success! Enabled the database secrets engine at: postgres/
```
Note
Similar to rate limit quotas, the quota rules apply globally in the absence of path. In the example, you created a quota rule against the us-west/postgres path. If a global quota rule exists on the root namespace, the quota rule defined on a specific path takes precedence.

Verify the configuration.

$ vault read sys/quotas/lease-count/db-creds Key            Value---            -----counter        0inheritable    falsemax_leases     100name           db-credspath           us-west/postgres/role           n/atype           lease-count

Lease count on a role

If you are running Vault 1.12 or later, you can set lease count on the role if the target auth method has the concept of roles.

Create a role named "webapp" for approle auth method.

First, enable the approle auth method.

$ vault auth enable approleSuccess! Enabled approle auth method at: approle/

Create "webapp" role.

$ vault write auth/approle/role/webapp \   token_policies="webapp-policy" \   token_ttl=4h \   token_max_ttl=12h

Output:

Success! Data written to: auth/approle/role/webapp

Create a lease count quota named "webapp-tokens" which limits the creation of token for the webapp role to maximum of 100.
```
$ vault write sys/quotas/lease-count/webapp-tokens \   max_leases=100 \   path="auth/approle" \   role="webapp"
```
Output:
```
Success! Data written to: sys/quotas/lease-count/webapp-tokens
```
If you want to set it for the approle auth method enable in the us-west namespace, the path should be us-west/auth/approle.

Verify the configuration.

$ vault read sys/quotas/lease-count/webapp-tokens

Output:

Key            Value---            -----counter        0inheritable    truemax_leases     100name           webapp-tokenspath           auth/approle/role           webapptype           lease-count

Create a namespace, "us-west".

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \     --request POST \     $VAULT_ADDR/v1/sys/namespaces/us-west

To learn about this feature, enable database secrets engine at "postgres" in the us-west namespace.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \   --header "X-Vault-Namespace: us-west" \   --request POST \   --data '{"type":"database"}' \   $VAULT_ADDR/v1/sys/mounts/postgres

Create a lease count quota named, "db-creds" which limits the incoming requests for a new set of DB credentials to 100 concurrent, valid leases maximum.
First, create the HTTP request payload defining the parameters for db-creds quota rule.
```
$ tee payload.json <<EOF{   "max_leases": 100,   "path": "us-west/postgres"}EOF
```
Now, invoke the sys/quotas/lease-count endpoint to create the quota rule.
```
$ curl --header "X-Vault-Token: $VAULT_TOKEN" \   --request POST \   --data @payload.json \   $VAULT_ADDR/v1/sys/quotas/lease-count/db-creds
```
Note
Similar to rate limit quotas, the quota rules apply globally in the absence of path. In the example, you created a quota rule against the us-west/postgres path. If a global quota rule exists on the root namespace, the quota rule defined on a specific path takes precedence.

Verify the configuration.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \   $VAULT_ADDR/v1/sys/quotas/lease-count/db-creds | jq -r ".data"

Output:

{   "counter": 0,   "max_leases": 100,   "name": "db-creds",   "path": "us-west/postgres/",   "role": "",   "type": "lease-count"}

Lease count on a role

If you are running Vault 1.12 or later, you can set lease count on the role if the target auth method has the concept of roles.

To demonstrate this feature, you are going to use the approle auth method. Enable the approle auth method.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \   --request POST \   --data '{"type": "approle"}' \   $VAULT_ADDR/v1/sys/auth/approle

Create a role named "webapp" for approle auth method.

First, create the API request payload.

$ tee payload.json <<EOF{   "token_policies": "webapp-policy",   "token_ttl": "4h",   "token_max_ttl": "12h"}EOF

Now, create a "webapp" role.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \   --request POST \   --data @payload.json \   $VAULT_ADDR/v1/auth/approle/role/webapp

Create a lease count quota named "webapp-tokens" which limits the creation of token for the webapp role to maximum of 100.
First, create the request payload.
```
$ tee payload_lcq.json <<EOF{   "max_leases": 100,   "path": "auth/approle",   "role": "webapp"}EOF
```
If you want to set it for the approle auth method enabled in the us-west namespace, the path should be us-west/auth/approle.
Now, invoke the sys/quotas/lease-count endpoint to create the quota rule.
```
$ curl --header "X-Vault-Token: $VAULT_TOKEN" \   --request POST \   --data @payload_lcq.json \   $VAULT_ADDR/v1/sys/quotas/lease-count/webapp-tokens
```

Verify the configuration.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \   $VAULT_ADDR/v1/sys/quotas/lease-count/webapp-tokens | jq -r ".data"

Output:

{   "counter": 0,   "max_leases": 100,   "name": "webapp-tokens",   "path": "auth/approle/",   "role": "webapp",   "type": "lease-count"}

Test

Be sure to visit the Test to understand the resource quota section to see how resource quotas enforcement works.

Global default lease count quota

If you are running Vault Enterprise 1.16 or later, consider setting a global lease count quota with a reasonably low default. This would provide an earlier signal to properly tune the lease limits so that unexpected lease explosion issue can be prevented. This approach would reduce the likelihood of customers accidentally imposing a later untenable load.

New installation only

The global default lease count quota is only applicable to new installations of Vault Enterprise. This is to ensure that Vault is not imposing new requirements on the clients' resources. Upgrade of any kind will not be affected by the default.

Read the current settings.

$ vault read sys/quotas/lease-count/default Key            Value---            -----counter        0inheritable    truemax_leases     300000name           defaultpath           n/arole           n/atype           lease-count

Use the sys/quotas/lease-count/default endpoint to adjust the default lease count quota.
```
$ vault write sys/quotas/lease-count/default \   max_leases=10000
```
Output:
```
Success! Data written to: sys/quotas/lease-count/default
```
You can read the default settings again to verify the changes.

Read the current settings.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \   $VAULT_ADDR/v1/sys/quotas/lease-count/default | jq -r ".data"

Output:

{   "counter": 0,   "inheritable": true,   "max_leases": 300000,   "name": "default",   "path": "",   "role": "",   "type": "lease-count"}

Use the sys/quotas/lease-count/default endpoint to adjust the default lease

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \   --request POST \   --data '{ "max_leases": 10000 }' \   $VAULT_ADDR/v1/sys/quotas/lease-count/default

You can read the default settings again to verify the changes.

Vault Enterprise namespaces

When you are working with Vault Enterprise namespaces, you can use the inheritable parameter to apply the resource quota to subsequent child namespaces.

Requirements

To leverage the inheritable resource quotas, you need Vault version 1.15 or later.
Vault namespaces is an Vault Enterprise feature. If you are new to namespaces, go through the Secure multi-tenancy with namespaces tutorial.

Think of the following namespace hierarchy:

root   └── parent      └── child         └── grand-child

Under the root namespace, you have a parent namespace, and then parent/child and parent/child/grand-child namespaces.

You can set the resource quota on the parent namespace which gets applied to its child namespaces inheritably by setting the inheritable parameter to true. By default, it is set to false.

Remember that you created a global-rate quota rule. This quota rule applies to the global level instead of a specific mount or namespace.

$ vault read sys/quotas/rate-limit/global-rate Key               Value---               -----block_interval    0inheritable       trueinterval          1name              global-ratepath              n/arate              1500role              n/atype              rate-limit

Create a quota rule on the us-west namespace which is inherited by its child namespaces. The rate limit is 500 requests per minute.

$ vault write sys/quotas/rate-limit/us-west \   path="us-west" \   rate=500 \   interval=1m \   inheritable=true

Output:

Success! Data written to: sys/quotas/rate-limit/us-west

Verify the quota rule.

$ vault read sys/quotas/rate-limit/us-west Key               Value---               -----block_interval    0inheritable       trueinterval          60name              us-westpath              us-west/rate              500role              n/atype              rate-limit

Remember that you created a global-rate quota rule. This quota rule applies to the global level instead of a specific mount or namespace.

$ curl -s --header "X-Vault-Token: $VAULT_TOKEN" \    $VAULT_ADDR/v1/sys/quotas/rate-limit/global-rate | jq -r ".data"

Output:

{   "block_interval": 0,   "inheritable": true,   "interval": 1,   "name": "global-rate",   "path": "",   "rate": 1500,   "role": "",   "type": "rate-limit"}

Create a quota rule on the us-west namespace which is inherited by its child namespaces. The rate limit is 500 requests per minute.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \   --request POST \   --data '{"path": "us-west", "rate": 500, "interval": 60, "inheritable": true }' \   $VAULT_ADDR/v1/sys/quotas/rate-limit/us-west

Verify the quota rule.

$ curl -s --header "X-Vault-Token: $VAULT_TOKEN" \   $VAULT_ADDR/v1/sys/quotas/rate-limit/us-west | jq -r ".data"

Output:

{   "block_interval": 0,   "inheritable": true,   "interval": 60,   "name": "us-west",   "path": "us-west/",   "rate": 500,   "role": "",   "type": "rate-limit"}

The us-west namespace and its children are now restricted to 500 requests per minute instead of what is set by the global-quota rule.

To disable the inheritance, set the inheritable parameter to false.

Test to understand the resource quotas

Now that you learned the basic commands, test to see how it behaves.

Rate limit quota test

Enable transit secrets engine if it is not enabled.
```
$ vault secrets enable transit
```
Create a "test" encryption key.
```
$ vault write -f transit/keys/test
```
For the purpose of demonstration, create a "test-transit" rate limit quota such that you can only make 1 requests per minute.
```
$ vault write sys/quotas/rate-limit/rate-test \   path=transit \   rate=1 \   interval=1m
```

Create a shortcut script, test-encryption.sh which makes request to encrypt data using "test" key.

$ tee test-encryption.sh <<EOFecho "Request 1"vault write transit/encrypt/test plaintext=$(base64 <<< "4111 1111 1111 1111")echo "\nRequest 2"vault write transit/encrypt/test plaintext=$(base64 <<< "4222 2222 2222 2222")echo "\nRequest 3"vault write transit/encrypt/test plaintext=$(base64 <<< "4333 3333 3333 3333")EOF

Ensure that the script is executable.
```
$ chmod +x test-encryption.sh
```

Run the script to see how the quota rule behaves.

$ ./test-encryption.sh

Output:

Request 1Key            Value---            -----ciphertext     vault:v1:6zY2JVtuw3r11ST/JjMkmoNbbcCpQwGbJJ0BJfByk//GLQof8mcxAoerq5Y++BZKkey_version    1Request 2Error writing data to transit/encrypt/test: Error making API request.URL: PUT http://127.0.0.1:8200/v1/transit/encrypt/testCode: 429. Errors:* request path "transit/encrypt/test": rate limit quota exceededRequest 3Error writing data to transit/encrypt/test: Error making API request.URL: PUT http://127.0.0.1:8200/v1/transit/encrypt/testCode: 429. Errors:* request path "transit/encrypt/test": rate limit quota exceeded

The second and third requests failed because of the limit.

Earlier, you configured the resource quotas to enable audit logging of requests which are rejected due to rate limit quota rule violation. Inspect your audit log for its entry.

$ more /var/log/vault-audit.log | jq

If your audit log path is not /var/log/vault-audit.log, be sure to set it to the correct path.

...snip...   "request": {      "id": "4170a181-6a18-589b-f026-a62ad08ae644",      "operation": "update",      "namespace": {         "id": "root"      },      "path": "transit/encrypt/test",      "data": {         "plaintext": "hmac-sha256:50978c7cb3f0f463491a26152353f485b579d1e8759ec7b7b7c28f3c5b9232cd"      },      "remote_address": "127.0.0.1",      "remote_port": 61028   },   "error": "request path \"transit/encrypt/test\": rate limit quota exceeded"}

You should find an error message indicating that rate limit quota was exceeded. You can trace the audit log to learn the number of requests which are rejected due to the rate limit quota. It may be working as expected or you may find suspicious activities against a specific path.

If you are running Vault 1.12.0 or later, create another rate limit quota to specify the path to transit/encrypt/test where allowed rate is 2 with interval of 1 minute.
```
$ vault write sys/quotas/rate-limit/encryption-limit \   path="transit/encrypt/test" \   rate=2 \   interval=1m
```

Run the script again to see how the quota rule behaves.

$ ./test-encryption.sh

Output:

Request 1Key            Value---            -----ciphertext     vault:v1:44sOnzePg8c0XaxzGeGzwJ94UZ1uuif8p0nGuze8utbvWVhqNHDiIwo8VZmPrclikey_version    1Request 2Key            Value---            -----ciphertext     vault:v1:w9tyXqoXH6uXuOWUktVWP/yrzOOZDv+TrRbbRU5GZEVappASIRnh3+6KwkJkjWKEkey_version    1Request 3Error writing data to transit/encrypt/test: Error making API request.URL: PUT http://127.0.0.1:8200/v1/transit/encrypt/testCode: 429. Errors:* request path "transit/encrypt/test": rate limit quota exceeded

Note

This time, only the last request failed. When a more granular path is set, the rate limit quota rule takes precedence against the path.

If you have another key or try to decript (transit/decrypt/test), the rate-test rate limit quota will be applied.

Lease count quota test

Similarly, create a limited lease count quota named, "lease-test" which applies on the root level. It only allows 3 tokens and leases to be stored.
```
$ vault write sys/quotas/lease-count/lease-test \   path="auth/token" \   max_leases=3
```

Create a shortcut script, test-token-create.sh which attempts to create tokens 4 times.

$ tee test-token-create.sh <<EOFecho "Request #1"vault token create -policy=defaultecho "\nRequest #2"vault token create -policy=defaultecho "\nRequest #3"vault token create -policy=defaultecho "\nRequest #4"vault token create -policy=defaultEOF

Make sure that the script is executable.
```
$ chmod +x test-token-create.sh
```

Run the script to see how the quota rule behaves.

$ ./test-token-create.sh

Three tokens were created successfully; however, the fourth request failed due to the lease count quota. Your output should look similar to follow.

Request #1Key                  Value---                  -----token                hvs.CAESIAtHe_XP_2UmXVfRaR_zQRalVg_TWSzrsqEy6PmgInDqGh4KHGh2cy5COTNKQ2pzYVN2WlRncjF3akhPMVJ1ZHktoken_accessor       rhmyha9JjSxh9hkjT2i2i8Antoken_duration       768htoken_renewable      truetoken_policies       ["default"]identity_policies    []policies             ["default"]Request #2Key                  Value---                  -----token                hvs.CAESIE0L_nNkTLI3cLayb0NpyEw2gc3x6-pjVIpoYhfyyOBEGh4KHGh2cy5xSjhhNWRaVnp5bzF6V2FqUE44emZUZk8token_accessor       oKKZViZet2tTl8TKqnbYMplXtoken_duration       768htoken_renewable      truetoken_policies       ["default"]identity_policies    []policies             ["default"]Request #3Key                  Value---                  -----token                hvs.CAESIP53xCUOcY0aJplpxagixzxZskdry6z1YUrYx5Urzk6eGh4KHGh2cy5UWGl0RTR2WDhLTUFFMGtLRnJJTjlBcHctoken_accessor       gtM8SoUEsaPkf9TvBLk5dswUtoken_duration       768htoken_renewable      truetoken_policies       ["default"]identity_policies    []policies             ["default"]Request #4Error creating token: Error making API request.URL: POST http://127.0.0.1:8200/v1/auth/token/createCode: 429. Errors:* 1 error occurred:   * request path "auth/token/create": lease count quota exceeded

Note

If your Vault server already has client tokens, the lease count quota may be exceeded sooner.

Also, you can find the lease count quota exceeded error in the audit log. (Be sure to set it to the correct audit log path for your environment.)

$ tail -f /var/log/vault-audit.log | jq ...snip...  "response": {    "mount_type": "token"  },  "error": "1 error occurred:\n\t* request path \"auth/token/create\": lease count quota exceeded\n\n"}

If you revoke one of the tokens, you should be able to request a new one.
Example:
```
$ vault token revoke hvs.CAESIAtHe_XP_2UmXVfRaR_zQRalVg_TWSzrsqEy6PmgInDqGh4KHGh2cy5COTNKQ2pzYVN2WlRncjF3akhPMVJ1ZHkSuccess! Revoked token (if it existed)
```
Now, request a new one.
```
$ vault token create -policy=default
```
The best practice is to set the tokens and leases' time-to-live (TTL) to be short and don't let them hang around longer than necessary. The lease count quotas allow you to set the upper limit to protect your Vault environment from running into an issue due to a lack of token and lease governance.

Delete the lease-test quota rule.

$ vault delete sys/quotas/lease-count/lease-test

Vault 1.12.0 or later

If you are running Vault 1.12.0 or later, you can set more granular lease count quota.

To demonstrate, enable approle auth method if it is not already enabled.
```
$ vault auth enable approle
```

Create a role named "test-role".

$ vault write auth/approle/role/test-role \   token_policies="default" \   token_ttl=1h \   token_max_ttl=4h

Create a lease count quota to limit the max number of leases for the test-role to 2.

$ vault write sys/quotas/lease-count/test-role-limit \   max_leases=2 \   path="auth/approle" \   role="test-role"

Retrieve the role ID of the test-role and store it in role_id.txt.

$ vault read -field=role_id auth/approle/role/test-role/role-id > role_id.txt

Generate a secret ID of the test-role and store it in secret_id.txt.

$ vault write -force -field=secret_id auth/approle/role/test-role/secret-id > secret_id.txt

Create a test script.

$ tee test-approle-login.sh <<EOFecho "Login attempt #1"vault write auth/approle/login role_id=$(cat role_id.txt) secret_id=$(cat secret_id.txt)echo "\nLogin attempt #2"vault write auth/approle/login role_id=$(cat role_id.txt) secret_id=$(cat secret_id.txt)echo "\nLogin attempt #3"vault write auth/approle/login role_id=$(cat role_id.txt) secret_id=$(cat secret_id.txt)EOF

Ensure that the script is executable.
```
$ chmod +x test-approle-login.sh
```

Run the script to see how the quota rule behaves.

$ ./test-approle-login.sh

Output:

Login attempt #1Key                     Value---                     -----token                   hvs.CAESICjr1nqxXo4UDJyh9MkMwvecOyEFlE69Pgs7ftdmUvF4Gh4KHGh2cy4yN3Q1WXRWU0tpSlBsQWpkZGJOdXpMZHYtoken_accessor          1P7KhrLmdrWRFIPR6IJnR6Lstoken_duration          1htoken_renewable         truetoken_policies          ["default"]identity_policies       []policies                ["default"]token_meta_role_name    test-roleLogin attempt #2Key                     Value---                     -----token                   hvs.CAESIGm6g-GUUPCJq3zvs_L1f1RIB4-7RD7a-yQxUaphFi0qGh4KHGh2cy5IVHluNVlEclZrSHNXb3lOZGVraWdmN20token_accessor          ThLLvtE42QE0wMbAkTEvOA8Ntoken_duration          1htoken_renewable         truetoken_policies          ["default"]identity_policies       []policies                ["default"]token_meta_role_name    test-roleLogin attempt #3Error writing data to auth/approle/login: Error making API request.URL: PUT http://127.0.0.1:8200/v1/auth/approle/loginCode: 429. Errors:* 1 error occurred:   * request path "auth/approle/login": lease count quota exceeded

The last attempt failed. The test-role-limit lease count quota should not affect other AppRole roles.

Vault Enterprise namespaces test

To test, create namespace hierarchy: us-west, us-west/california, and us-west/california/san-francisco.

root└── us-west   └── california      └── san-francisco

Create california namespace as a child of the us-west namespace.

$ vault namespace create -namespace="us-west" californiaKey                Value---                -----custom_metadata    map[]id                 dUR8ypath               us-west/california/

Create san-francisco namespace as a child of the us-west/california namespace.

$ vault namespace create -namespace="us-west/california" "san-francisco"Key                Value---                -----custom_metadata    map[]id                 InIeRpath               us-west/california/san-francisco/

Create a rate limit quota named, "us-west" which limits 3 requests per minute on the us-west namespace and make it inheritable.
```
$ vault write sys/quotas/rate-limit/us-west \   path="us-west" \   rate=3 \   interval=1m \   inheritable=true
```

Set the target namespace to us-west/california.

$ export VAULT_NAMESPACE="us-west/california"

Run the test-token-create.sh script to test the quota.

$ ./test-token-create.sh

The rate limit placed onto the us-west namespace is inherited by its child namespaces; therefore, you can make 3 requests per minute on the us-west/california namespace as well.

Request #1Key                  Value---                  -----token                hvs.CAESIOw0mxz1DkoQvEdb77rUkEXFvlaFELTZ8WoEJognD9IxGiQKImh2cy5MRFZWa2N3azB0bTVDZVJTVXd0Z1EzTEEuV2xFZ3Itoken_accessor       CTomum1fi0XaQgaXDXA91aKF.WlEgrtoken_duration       768htoken_renewable      truetoken_policies       ["default"]identity_policies    []policies             ["default"]Request #2Key                  Value---                  -----token                hvs.CAESIENZqKVA8TEE6wA_Vr9e6I-vlR-lBFDtMSL7zrOmyrqwGiQKImh2cy5WQmxaUkdZbnB0STJYQVUzUzFJMHJQT3ouV2xFZ3Itoken_accessor       O8WnsqrLwnDumWndb0VUKyQs.WlEgrtoken_duration       768htoken_renewable      truetoken_policies       ["default"]identity_policies    []policies             ["default"]Request #3Key                  Value---                  -----token                hvs.CAESIFH-IjHLpeM6FT_FgVkxDZXJqocGSBhYoJRSboUSaT-xGiQKImh2cy5OWjlodGMwRUZRa3NOcW04OEJZMTd4TXAuV2xFZ3Itoken_accessor       b9s5wrRNuvKQa6FhOmcENOuA.WlEgrtoken_duration       768htoken_renewable      truetoken_policies       ["default"]identity_policies    []policies             ["default"]Request #4Error creating token: Error making API request.Namespace: us-west/california/URL: POST http://127.0.0.1:8200/v1/auth/token/createCode: 429. Errors:* request path "auth/token/create": rate limit quota exceeded

To test against the us-west/california/san-francisco namespace, update the VAULT_NAMESPACE environment variable.
```
$ export VAULT_NAMESPACE="us-west/california/san-francisco"
```

Run the script to verify that the quota rule is inherited.

$ ./test-token-create.sh

The forth request fails due to quota.

...snip...Request #4Error creating token: Error making API request.Namespace: us-west/california/san-francisco/URL: POST http://127.0.0.1:8200/v1/auth/token/createCode: 429. Errors:* request path "auth/token/create": rate limit quota exceeded

Clean up

If you wish to clean up your environment after completing the tutorial, follow the steps in this section.

Unset the VAULT_TOKEN, VAULT_ADDR, and VAULT_NAMESPACE environment variables.
```
$ unset VAULT_TOKEN VAULT_ADDR VAULT_NAMESPACE
```

Delete files created during the test.

$ rm test-encryption.sh test-token-create.sh test-approle-login.sh role_id.txt secret_id.txt

If you are running Vault locally in -dev mode, you can stop the Vault dev server by pressing Ctrl+C where the server is running. Or, execute the following command.
```
$ pgrep -f vault | xargs kill
```

Next steps

In this tutorial, you learned the basic commands to set resource quotas to protect your Vault environment. To leverage this feature, you need Vault 1.5 or later.

Rate limit quotas allow Vault operators to set inbound request rate limits which can be set on the root level or a specific path. This is available in both Vault Community Edition and Vault Enterprise.

Lease count quotas require Vault Enterprise Platform and allow operators to set the maximum number of tokens and leases to be persisted at any given time. This can prevent Vault from exhausting the resource on the storage backend.

You also learned that audit logging can be enabled to trace the number of requests that were rejected due to the rate limit quota.

Help and Reference

Troubleshoot and tune replication

Codify Vault Enterprise with Terraform

This tutorial also appears in:

11 tutorials

Operations fundamentals
Learn basic Vault operations that are common to both Vault Community Edition and Vault Enterprise users.
- Vault