Consul ACL

The Consul ACL system protects the cluster from unauthorized access. When enabled, both Consul and Nomad must be properly configured in order for their integrations to work.

Nomad agents must be configured with their own Consul ACL token, and Consul must be configured to accept workload identities from tasks and services running in Nomad.

Nomad Agents

Nomad agents need access to Consul in order to register themselves in the service catalog and discover other Nomad agents via service discovery for automatic clustering. Nomad clients use Consul tokens from Workload Identity to register services and checks but need permissions on their own token to deregister. Nomad servers also create configuration entries for Consul Service Mesh, so the specific permissions vary slightly between Nomad servers and clients. The following Consul ACL policies represent the minimal permissions Nomad servers and clients need.

agent_prefix "" {  policy = "read"} node_prefix "" {  policy = "write"} service_prefix "" {  policy = "write"} acl  = "write"mesh = "write"

agent_prefix "" {  policy = "read"} node_prefix "" {  policy = "write"} service_prefix "" {  policy = "write"}

Nomad Workload Identities

Starting in Nomad 1.7, Nomad clients can use a task or service's Workload Identity to authenticate to Consul and obtain an ACL token specific to the service or task. When using Nomad workload identities, you no longer need to pass in a Consul ACL token to submit a job.

By default, Nomad does not generate workload identities for services, and tasks only receive an identity that can be used to access data from Nomad itself, such as for reading Variables from a template block. To access Consul, jobs must have additional workload identities defined as identity blocks.

To avoid having to add these additional identities to every job, you can configure Nomad servers with the consul.service_identity and consul.task_identity agent configuration. Upon job registration, the Nomad servers update tasks that have a consul or template block and services that use the Consul service provider with these default identities.

You can also specify identities for Consul directly in the job. When provided, they override the Nomad server configuration. Refer to the Workload Identities for Consul section of the identity block documentation for more information.

Configuring Consul Authentication

You must configure Consul so it can receive, validate, and trust workload identities from Nomad. Since they are encoded as JSON Web Tokens (JWTs), you must create a JWT ACL auth method. The auth method is an endpoint that Nomad can use to exchange workload identities for Consul ACL tokens.

Refer to Consul's Auth Methods Overview documentation for more information.

Consul Auth Method

The auth method configuration points to Nomad's JSON Web Key Set (JWKS) URL. Consul servers call this URL to retrieve the public keys Nomad uses to sign workload identities. With these keys, Consul is able to validate their origin and confirm that they were actually created by Nomad.

Nomad cannot recreate Consul tokens that have been deleted. The auth method configuration should never set the MaxTokenTTL field. Consul tokens are local to the Consul datacenter unless you set TokenLocality: "global" in the auth method. We recommend using local tokens, which is the default. Global tokens require that the primary Consul datacenter is available when allocations start.

{  "JWKSURL": "https://nomad.example.com:4646/.well-known/jwks.json",  "JWTSupportedAlgs": ["RS256"],  "BoundAudiences": ["consul.io"],  "ClaimMappings": {    "nomad_namespace": "nomad_namespace",    "nomad_job_id": "nomad_job_id",    "nomad_task": "nomad_task",    "nomad_service": "nomad_service"  }}

The JWKSURL address must be reachable by all Consul servers and should resolve to multiple Nomad agents to avoid a single point of failure. Both Nomad servers and clients are able to handle this request.

Refer to the Important Considerations About the JWKS URL section for additional information on how to configure the JWKSURL value.

When an allocation that needs access to Consul starts, the Nomad client running it exchanges the Nomad workload identities for tasks and services for Consul ACL tokens.

The auth method also defines the list of authorized audience values, which must have at least one match with the values defined in the Nomad workload identity aud parameter. For security reasons, it is recommended to only define a single audience value.

{  "JWKSURL": "http://nomad.example.com:4646/.well-known/jwks.json",  "JWTSupportedAlgs": ["RS256"],  "BoundAudiences": ["consul.io"],  "ClaimMappings": {    "nomad_namespace": "nomad_namespace",    "nomad_job_id": "nomad_job_id",    "nomad_task": "nomad_task",    "nomad_service": "nomad_service"  }}

Nomad workload identities have a set of claims that can be referenced as dynamic values in Consul ACL configuration. The auth method determines which of these claims are made available to the rest of the configuration.

{  "JWKSURL": "http://nomad.example.com:4646/.well-known/jwks.json",  "JWTSupportedAlgs": ["RS256"],  "BoundAudiences": ["consul.io"],  "ClaimMappings": {    "nomad_namespace": "nomad_namespace",    "nomad_job_id": "nomad_job_id",    "nomad_task": "nomad_task",    "nomad_service": "nomad_service"  }}

Consul Binding Rules

Consul auth methods use binding rules to determine the set of policies applied to the generated ACL token. Nomad workload identities can be used in Consul for two main purpose: registering services and retrieving configuration values and service addresses from Consul using template blocks in tasks. Each purpose requires tokens with different permissions so Nomad requires two binding rules.

The first binding rule associates the Consul ACL token with a service identity, allowing the token to register and manage the lifecyle of a given service. This binding rule is only applied to Nomad workload identities for services because they are the only ones that have a nomad_service claim.

$ consul acl binding-rule create \    -method 'nomad-workloads' \    -bind-type 'service' \    -bind-name '${value.nomad_service}' \    -selector '"nomad_service" in value'

The -bind-name flag restricts the token to only be able to modify services with the same name as the one defined in the Nomad workload identity claim. The -selector flag ensures this binding rule only applies to workload identities for services.

The second binding rule associates the Consul ACL token with an ACL role, which is a collection of ACL policies that define what the token is authorized to do. This binding rule is applied to Nomad workload identities for tasks to access information from Consul. The exact ACL policy rules will depend on the level of access required by the tasks (typically, accessing service addresses and KV with template blocks).

$ consul acl binding-rule create \  -method 'nomad-workloads' \  -bind-type 'role' \  -bind-name 'nomad-tasks-${value.nomad_namespace}' \  -selector '"nomad_service" not in value'

The -bind-name flag defines which role is used for the token. It may reference claim values from the Nomad workload identity to apply different roles to different tasks. Similarly to the binding rule for services, the -selector flag ensures this binding rule only applies to workload identities for tasks since they don't have the nomad_service claim.

The overall configuration structure is illustrated in the following diagram.

The consul.service_auth_method and consul.task_auth_method configuration define the auth method used by Nomad to retrieve Consul ACL tokens for services and tasks.

By default, they follow the structure described above and use a single auth method for both, but it is possible to use two different auth methods, provided the resulting Consul ACL token from each have the expected service identity and role applied.

Consul Namespace Rules EnterpriseEnterprise

Consul Enterprise supports multiple namespaces and Nomad Enterprise allows jobs to use the consul.namespace parameter to register services and read KV data from different Consul namespaces.

In a multi-namespace environment, you should create the auth method and binding rules in the default namespace and configure the auth method with a set of NamespaceRules.

$ consul acl auth-method create \  -name 'nomad-workloads' \  -type 'jwt' \  -config '@auth-method.json' \  -namespace-rule-selector '"consul_namespace" in value' \  -namespace-rule-bind-namespace '${value.consul_namespace}'

Similarly to binding rules, namespace rules have a Selector expression to determine when the rule should be applied and a BindNamespace value that defines the namespace used.

In Nomad Enterprise, workload identities for tasks and services placed within the scope of a consul block with a namespace value, have an additional claim called consul_namespace that represents the Consul namespace defined in Nomad for the workload.

job "example" {  group "cache" {    network {      port "db" {        to = 6379      }    }     consul {      namespace = "prod"    }     service {      port     = "db"      name     = "redis"      provider = "consul"    }     task "redis" {      driver = "docker"       config {        image = "redis:7"        ports = ["db"]      }    }  }}

If a consul block is not defined, then the workload identity will not have the consul_namespace claim, since Nomad is not able to determine which Consul namespace will be used.

Refer to the Consul Namespaces section for more information.

Important Considerations About the JWKS URL

The recommended configuration assumes Consul servers are able to connect to Nomad agents (either client or servers) to retrieve the JSON Web Key Set information.

This section covers additional aspects you should consider depending on how your Consul and Nomad clusters are configured and deployed.

Mutual TLS in Nomad

It is highly recommended to use mutual TLS in production deployments of Nomad. With mTLS enabled, the tls.verify_https_client configuration must be set to false since it is not possible to provide client certificates to the Consul auth method.

Alternatively, you may expose Nomad's JWKS URL from a proxy or a load balancer that handles the mutual TLS connection to Nomad and exposes the JWKS URL endpoint over standard TLS.

Consul Servers Not Able to Connect to Nomad

If the Consul servers are not able to reach Nomad's JWKS URL, you may read the public keys from Nomad's /.well-known/jwks.json endpoint and provide them to the auth method directly using the JWTValidationPubKeys parameter. The keys must be converted from JWKS to PEM format.

You may also host the JWKS JSON response from Nomad in an external location that is reachable by the Consul servers, and use that address as the value for JWKSURL.

It is important to remember that the Nomad keys are rotated periodically, so both approaches should be automated and done continually. The rotation frequency is controlled by the server.root_key_rotation_threshold configuration of the Nomad servers. Keys will be prepublished at half the rotation threshold.

Additional References

The Consul ACL with Nomad Workload Identities tutorial provides guided instructions on how to configure Consul and Nomad for workload identities.

The nomad setup consul command and the hashicorp-modules/nomad-setup/consul Terraform module can help you automate the process of applying configuration to a Consul cluster.

Authenticating Without Workload Identity (Legacy)

If Consul ACLs are enabled, the allow_unauthenticated configuration parameter will control whether a Consul token will be required when submitting a job with Consul namespace configured. The provided Consul token must belong to the correct namespace, and must be backed by a Consul ACL Policy with sufficient service:write and kv:read permissions. An example policy might look like the following.

key_prefix "" {  policy = "read"} service_prefix "" {  policy = "write"}

Migrating to Using Workload Identity with Consul

Migrating from the legacy (pre-1.7) workflow where workload use the agent's Consul token requires configuation on your Consul cluster and your Nomad server agents. It does not require updating your running Nomad jobs. To migrate:

• Create the Consul auth method and binding rules on your Consul cluster.
• Enable consul.service_identity blocks in your Nomad server agent configurations.
• Enable consul.task_identity blocks in your Nomad server agent configurations.
• (Optionally) add identity blocks to your jobs if you want to use a different identity because of how your auth method and binding rules are configured.

Note that when using Workload Identity you will no longer need to pass in a Consul token to submit a job.