Migrate Legacy ACL Tokens

Consul 1.4.0 introduced a new ACL system with improvements for the security and management of ACL tokens and policies. Since the policy syntax changed to be more precise and flexible to manage, it's necessary to manually translate old tokens (now called "legacy") into new ones to take advantage of the new ACL system features.

This tutorial describes the process for migrating the tokens already present in your pre-1.4.0 Consul deployment after an upgrade to 1.4.0 or later. You will use three different methods for migrating tokens; single token, simple policy mapping, and combine policy updates. Single token updates are a manual process for sensitive tokens. The simple policy mapping and combine policy updates can be used to automate the update of many tokens.

To successfully run the commands in this tutorial, you will need to configure CONSUL_HTTP_TOKEN with a token that has management privileges. You can set the environment variable temporarily with export, so that it will not persist once you've closed the session.

$ export CONSUL_HTTP_TOKEN=<your_token_here>

Post upgrade ACL status

This process assumes that the 1.4.0 datacenter upgrade is complete, including all legacy ACLs having their accessor IDs populated. Accessor IDs are new to the 1.4 ACL system. This process can take up to several minutes after the servers upgrade. Ensure your tokens have an AccessorID before starting the migration. If the command consul acl token list shows any blank AccessorIDs, wait until Consul has finished the upgrade.

$ consul acl token list

...AccessorID:       00000000-0000-0000-0000-000000000002Description:      Anonymous TokenLocal:            falseCreate Time:      0001-01-01 00:00:00 +0000 UTCLegacy:           falseAccessorID:       dd40ea2f-ff15-e7cb-8249-5d5e95b7ee75Description:      foo-service-token_1Local:            falseCreate Time:      0001-01-01 00:00:00 +0000 UTCLegacy:           trueAccessorID:       905755a7-0872-746a-8aee-e9bff2936c11Description:      Agent TokenLocal:            falseCreate Time:      0001-01-01 00:00:00 +0000 UTCLegacy:           trueAccessorID:       5c72a103-d94e-a379-797c-2b418adec88bDescription:      Master TokenLocal:            falseCreate Time:      0001-01-01 00:00:00 +0000 UTCLegacy:           falsePolicies:   00000000-0000-0000-0000-000000000001 - global-management...

From the output, you can notice that two tokens already exist in the new system, the master token and the anonymous token. All the other tokens present in the list are supported as "legacy" as you can verify by the Legacy: true setting.

Consul 1.4.0 retains full support for "legacy" ACL tokens so upgrades from Consul 1.3.0 are safe. Existing tokens will continue to work in the same way for at least two "major" releases (1.5.x, 1.6.x, etc; note HashiCorp does not use SemVer for our products).

Single token upgrade

While "legacy" tokens will continue to work for several major releases, it's advisable to migrate existing tokens as soon as possible. Migrating also enables using the new policy management improvements, stricter policy syntax rules, and other features of the new system without re-issuing all the secrets in use.

You can upgrade a legacy token in three steps:

1. Identify the token that needs to be upgraded
2. Create a new policy or policies that grant the required access
3. Update the existing token to use those policies

Identify token

Pick a legacy token from the output of consul acl token list command.

$ consul acl token list

...AccessorID:       905755a7-0872-746a-8aee-e9bff2936c11Description:      Agent TokenLocal:            falseCreate Time:      0001-01-01 00:00:00 +0000 UTCLegacy:           true...

You can recognize the legacy tokens by the Legacy: true setting in the output.

For this tutorial,the legacy token will be the "Agent Token".

Create a new policy

For the selected token, you can check what the rules will be once the policy is migrated using the acl translate-rules command.

$ consul acl translate-rules -token-accessor "905755a7-0872-746a-8aee-e9bff2936c11"

node_prefix "" {  policy = "write"} service_prefix "" {  policy = "read"}

Notice the new policy has a service_prefix rule and not the original service. Since the old ACL syntax had an implicit prefix match, the _prefix was applied to all rules without needing to be specified. Now that _prefix is no longer implicit, you will need to add it to ensure any clients relying on prefix matching behavior will still work. You can use the Consul CLI to update the policy, without needing to edit the policy file.

Once you have verified the set of rules for the new policy, you will generate the new one.

$ consul acl policy create -name "migrated-Agent-policy" -from-token 905755a7-0872-746a-8aee-e9bff2936c11 -description "Migrated from legacy ACL agent token";

ID:           dcd37998-a564-b26a-dd8a-89414926d015Name:         migrated-Agent-policyDescription:  Migrated from legacy ACL agent tokenDatacenters:Rules:node_prefix "" {  policy = "write"}service_prefix "" {  policy = "read"}

Note, the use of the -from-token option to specify the rules for the new policy will be based on the rules embedded in the legacy token.

To verify the new policy is in place use the consul acl policy list command again.

$ consul acl policy list

migrated-Agent-policy:   ID:           dcd37998-a564-b26a-dd8a-89414926d015   Description:  Migrated from legacy ACL agent token   Datacenters:...

Update the token

Finally, you will need to update the token to use the new policy.

$ consul acl token update -id 905755a7-0872-746a-8aee-e9bff2936c11 -policy-name "migrated-Agent-policy" -upgrade-legacy

Token updated successfully.AccessorID:       905755a7-0872-746a-8aee-e9bff2936c11SecretID:         dbb88e67-817d-7eb7-f03b-af570462ae6fDescription:      Agent TokenLocal:            falseCreate Time:      0001-01-01 00:00:00 +0000 UTCPolicies:   dcd37998-a564-b26a-dd8a-89414926d015 - migrated-Agent-policy

For the upgrade to succeed you need to use the -upgrade-legacy option, which will ensure that legacy rules are removed as well as the new policies added.

If not specified the migration will result in an error:

$ consul acl token update -id 905755a7-0872-746a-8aee-e9bff2936c11 -policy-name "migrated-Agent-policy"

Failed to update token 905755a7-0872-746a-8aee-e9bff2936c11: Unexpected response code: 500 (Rules cannot be specified for this token)

After updating, the token is no longer considered "legacy" and will have all the properties of a new token, however it keeps its SecretID (the secret part of the token used in API calls) so clients already using that token will continue to work. It is assumed that the policies you attach continue to grant the necessary access for existing clients; this is up to the operator to ensure.

Simple policy mapping

If you have multiple tokens migrating them manually might not be a viable option.

One fast alternative to manual migration is to write an automation script to create a different policy with the same rules set, per legacy token and then upgrade them. This approach is easy to automate, but may result in a lot of policies with exactly the same rules and with non-human-readable names which will make managing policies more difficult.

Identify existing legacy tokens

You can get the AccessorID of every legacy token from the API. For example, using curl and jq in bash.

$ LEGACY_IDS=$(curl -sH "X-Consul-Token: $CONSUL_HTTP_TOKEN" \   'localhost:8500/v1/acl/tokens' | jq -r '.[] | select (.Legacy) | .AccessorID')

$ echo "$LEGACY_IDS"

7dff8907-eace-b403-1966-51ba92189ac73d288b1c-ff11-47fa-e010-42d15a117534dd40ea2f-ff15-e7cb-8249-5d5e95b7ee75

Create policies from legacy tokens

To create a policy for each token you can use a loop to iterate the AccessorID list and migrate the policy.

$ for id in $LEGACY_IDS; do \  consul acl policy create -name "migrated-$id" -from-token $id \    -description "Migrated from legacy ACL token"; \done

ID:           95acbf71-9795-174d-d8fb-fef338c780caName:         migrated-7dff8907-eace-b403-1966-51ba92189ac7Description:  Migrated from legacy ACL tokenDatacenters:Rules:service_prefix "bar" {  policy = "write"}ID:           31189810-9503-eae7-9c09-01eb7c1346afName:         migrated-3d288b1c-ff11-47fa-e010-42d15a117534Description:  Migrated from legacy ACL tokenDatacenters:Rules:service_prefix "bar" {  policy = "write"}ID:           06435a7b-59c2-585f-ae8e-b49a576b9c71Name:         migrated-dd40ea2f-ff15-e7cb-8249-5d5e95b7ee75Description:  Migrated from legacy ACL tokenDatacenters:Rules:service_prefix "foo" {  policy = "write"}## ...

Each policy now has an equivalent set of rules to the original token. This means that the newly created policy grants the same effective permissions as the legacy token rules did.

Update tokens

With the policies created as above, you can automatically upgrade all legacy tokens.

$ for id in $LEGACY_IDS; do \  consul acl token update -id $id -policy-name "migrated-$id" -upgrade-legacy; \done

Token updated successfully.AccessorID:       7dff8907-eace-b403-1966-51ba92189ac7SecretID:         1532ab31-f7aa-41f7-4989-5da3d537456aDescription:      bar-service-token_1Local:            falseCreate Time:      0001-01-01 00:00:00 +0000 UTCPolicies:   95acbf71-9795-174d-d8fb-fef338c780ca - migrated-7dff8907-eace-b403-1966-51ba92189ac7Token updated successfully.AccessorID:       3d288b1c-ff11-47fa-e010-42d15a117534SecretID:         87a185c6-f42a-be80-e72e-d5489d68ddc4Description:      bar-service-token_2Local:            falseCreate Time:      0001-01-01 00:00:00 +0000 UTCPolicies:   31189810-9503-eae7-9c09-01eb7c1346af - migrated-3d288b1c-ff11-47fa-e010-42d15a117534Token updated successfully.AccessorID:       dd40ea2f-ff15-e7cb-8249-5d5e95b7ee75SecretID:         c6aedb37-9600-a3b4-1ce1-d70aee04abc1Description:      foo-service-token_1Local:            falseCreate Time:      0001-01-01 00:00:00 +0000 UTCPolicies:   06435a7b-59c2-585f-ae8e-b49a576b9c71 - migrated-dd40ea2f-ff15-e7cb-8249-5d5e95b7ee75...

The update is now complete, all legacy tokens are now new tokens with identical secrets and enforcement rules.

Combine policies

An automatic batch upgrade for legacy tokens is the suggested approach when policies are relatively simple and specific for each token. With the combined policies method, you will use the same policy for multiple tokens.

If policies are complex or you have overlapping rules across multiple tokens, it is worth it to leverage the migration process to re-engineer the ACL rules in your Consul deployment. This permits you to end up with a minimal set of policies and to take advantage of the flexibility of the new system.

There are a variety of options for how to do this, however you will need to consider the trade-offs. Mainly, increased operator involvement will result in better clarity and re-usability of the resulting policies.

In the tutorial examples, you have two tokens associated to identical policies.

$ for id in $LEGACY_IDS; do \   echo "Policy for $id:";  \   consul acl translate-rules -token-accessor "$id"; \done

...Policy for 7dff8907-eace-b403-1966-51ba92189ac7:service_prefix "bar" {  policy = "write"}Policy for 3d288b1c-ff11-47fa-e010-42d15a117534:service_prefix "bar" {  policy = "write"}Policy for dd40ea2f-ff15-e7cb-8249-5d5e95b7ee75:service_prefix "foo" {  policy = "write"}

One possibility, in these cases, would be to generate only one policy and then upgrade both the legacy tokens to use the same policy.

Enforce the use of new ACLs in your environment

In the tutorial it is assumed that all clients that need to create ACL tokens (e.g. Vault's Consul secrets engine) have been updated to use the new ACL APIs.

Specifically if you are using Vault's Consul secrets engine you need to be running Vault 1.0.0 or higher and you must update all roles defined in Vault to specify a list of policy names rather than an inline policy (which causes Vault to use the legacy API).

Next steps

In this tutorial, you learned the different strategies and commands available in Consul 1.4+ to migrate legacy ACL tokens.

You followed those strategies to:

• manually migrate an agent token and get familiar with the CLI commands for ACL;
• automate the token migration for tokens and policies that don’t require tuning;
• identify opportunities for improvement in your ACL configuration thanks to the new ACL system improvements.