Consul ACL with Nomad Workload Identities

Nomad and Consul provide several integration points when deployed together. Workloads running in Nomad can register services in Consul's catalog for service discovery or service mesh, and access configuration values from Consul KV. Nomad agents can also use Consul to discover other Nomad agents and automatically form a cluster.

Production deployments of Nomad and Consul must always run with the Access Control List (ACL) system enabled since it protects against unauthorized access to the cluster. When ACLs are enabled, both Nomad and Consul must be properly configured in order for their integrations to work.

Nomad can generate workload identities for tasks and services, which are represented as JSON Web Tokens (JWT) signed by Nomad. These identities can be used as proof to third parties that a workload was actually created and is managed by Nomad. If the third party is configured to trust Nomad, it can automatically grant specific access and permissions to Nomad workloads.

In this tutorial, you will:

• Start a Nomad and Consul agent with ACL enabled.
• Generate ACL tokens to access Consul and Nomad.
• Configure Consul to accept workload identities from Nomad.
• Configure Nomad to automatically generate and sign workload identities for services and tasks that need access to Consul.
• Deploy sample Nomad jobs that interact with Consul.

Launch Terminal

This tutorial includes a free interactive command-line lab that lets you follow along on actual cloud infrastructure.

Start interactive lab

Prerequisites

This tutorial requires you to have basic familiarity with Nomad and Consul. If you are new to these tools, complete the Nomad Get Started and Consul Get Started tutorials before following this one.

You will need the following tools installed:

• Nomad 1.7 or later installed locally
• Consul 1.14 or later installed locally
• Docker installed and running locally

Start the Consul agent

Create a directory for the tutorial on your local machine, change into that directory, and create a file named consul.hcl to store the Consul agent configuration. Add the following contents to it and save the file.

datacenter = "dc1"node_name  = "host01" acl {  enabled        = true  default_policy = "deny"}

Start a Consul dev agent using consul.hcl as the configuration file.

$ consul agent -dev -config-file 'consul.hcl'==> Starting Consul agent...               Version: '1.17.0'            Build Date: '2023-11-03 14:56:56 +0000 UTC'               Node ID: 'f9625116-3884-cd0b-01fa-2bc2e0d9a69d'...

Open another terminal window in the same directory and bootstrap the Consul ACL system. This terminal will act as the main terminal session where you will run commands.

$ consul acl bootstrapAccessorID:       0164c5f0-4186-f24c-5fc9-28e2d4e6902eSecretID:         da0315e9-8000-4c5e-c470-1c60d79314b9Description:      Bootstrap Token (Global Management)Local:            falseCreate Time:      2023-11-15 17:57:48.724704 -0500 ESTPolicies:   00000000-0000-0000-0000-000000000001 - global-management

Copy the value for SecretID and set it as the environment variable CONSUL_HTTP_TOKEN.

$ export CONSUL_HTTP_TOKEN=...

$ export CONSUL_HTTP_TOKEN=...

$ $Env:CONSUL_HTTP_TOKEN = "..."

Create a node ACL token for the Consul agent.

$ consul acl token create -node-identity 'host01:dc1'AccessorID:       18ddc57f-72e6-2b4f-9b20-b8c8d58d41d7SecretID:         e7fc8ebd-8234-2316-15fe-f59d4f7a5463Description:Local:            falseCreate Time:      2023-11-15 18:06:59.674682 -0500 ESTNode Identities:   host01 (Datacenter: dc1)

Copy the value for SecretID.

Apply the ACL token to the local Consul agent. Replace the <SecretID> placeholder with the value from the previous token creation command.

$ consul acl set-agent-token agent <SecretID>ACL token "agent" set successfully

Create a file named consul-policy-nomad-agents.hcl to store the Consul ACL rules that grant the necessary permissions to Nomad agents. Add the following contents to it and save the file.

agent_prefix "" {  policy = "read"} node_prefix "" {  policy = "write"} service_prefix "" {  policy = "write"}

Create a Consul ACL policy named nomad-agents with the rules defined in the consul-policy-nomad-agents.hcl file.

$ consul acl policy create -name 'nomad-agents' -description 'Policy for Nomad agents' -rules '@consul-policy-nomad-agents.hcl'ID:           7a0fe00b-f7e6-809c-2227-bb0638b873bdName:         nomad-agentsDescription:  Policy for Nomad agentsDatacenters:Rules:agent_prefix "" {  policy = "read"} node_prefix "" {  policy = "write"} service_prefix "" {  policy = "write"}

Create a Consul ACL token for the Nomad agent using the nomad-agents ACL policy.

$ consul acl token create -policy-name 'nomad-agents'AccessorID:       3f436657-823a-95e3-4755-79f3e1e43c8eSecretID:         df179fd2-3211-3641-5901-a57331c14611Description:Local:            falseCreate Time:      2023-11-15 18:23:39.572365 -0500 ESTPolicies:   a5ee20ed-7158-89be-9a19-be213d106d24 - nomad-agents

Save the value of SecretID for the Consul ACL token. You will use it in the next section to configure Nomad.

Start the Nomad agent

Create a file named nomad.hcl. Add the following contents to it, replace the placeholder <Consul token SecretID> text with the value of SecretID for the Consul ACL token, and save the file.

acl {  enabled = true} consul {  address = "127.0.0.1:8500"  token   = "<Consul token SecretID>"   service_identity {    aud = ["consul.io"]    ttl = "1h"  }   task_identity {    aud = ["consul.io"]    ttl = "1h"  }}

acl {  enabled = true} consul {  address = "127.0.0.1:8500"  token   = "<Consul token SecretID>"   service_identity {    aud = ["consul.io"]    ttl = "1h"  }   task_identity {    aud = ["consul.io"]    ttl = "1h"  }}

acl {  enabled = true} consul {  address = "127.0.0.1:8500"  token   = "<Consul token SecretID>"   service_identity {    aud = ["consul.io"]    ttl = "1h"  }   task_identity {    aud = ["consul.io"]    ttl = "1h"  }} plugin "docker" {  config {    allow_caps = [      "CHOWN", "DAC_OVERRIDE", "FSETID", "FOWNER", "MKNOD",      "SETGID", "SETUID", "SETFCAP", "SETPCAP", "NET_BIND_SERVICE",      "SYS_CHROOT", "KILL", "AUDIT_WRITE", "NET_RAW",    ]  }}

The consul block in this configuration file provides the information necessary for Nomad to connect to Consul.

It also defines two default workload identities, service_identity and task_identity, that are automatically added to jobs that need access to Consul. Without them, you would need to define an identity block in your jobs for each service and task that needs access to Consul.

Open another terminal window in the same directory and start the Nomad dev agent.

$ sudo nomad agent -dev -config 'nomad.hcl'==> Loaded configuration from nomad.hcl==> Starting Nomad agent...==> Nomad agent configuration:        Advertise Addrs: HTTP: 127.0.0.1:4646; RPC: 127.0.0.1:4647; Serf: 127.0.0.1:4648            Bind Addrs: HTTP: [127.0.0.1:4646]; RPC: 127.0.0.1:4647; Serf: 127.0.0.1:4648                Client: true             Log Level: DEBUG               Node Id: af9bef00-2e83-b704-2d6c-c62bc005431f                Region: global (DC: dc1)                Server: true               Version: 1.7.0 ==> Nomad agent started! Log data will stream in below:...

$ nomad agent -dev -config 'nomad.hcl' -network-interface '{{GetDefaultInterfaces | attr "name"}}'==> Loaded configuration from nomad.hcl==> Starting Nomad agent...==> Nomad agent configuration:        Advertise Addrs: HTTP: 127.0.0.1:4646; RPC: 127.0.0.1:4647; Serf: 127.0.0.1:4648            Bind Addrs: HTTP: [127.0.0.1:4646]; RPC: 127.0.0.1:4647; Serf: 127.0.0.1:4648                Client: true             Log Level: DEBUG               Node Id: af9bef00-2e83-b704-2d6c-c62bc005431f                Region: global (DC: dc1)                Server: true               Version: 1.7.0 ==> Nomad agent started! Log data will stream in below:...

$ nomad agent -dev -config 'nomad.hcl' -network-interface '{{GetDefaultInterfaces | attr \"name\"}}'==> Loaded configuration from nomad.hcl==> Starting Nomad agent...==> Nomad agent configuration:        Advertise Addrs: HTTP: 127.0.0.1:4646; RPC: 127.0.0.1:4647; Serf: 127.0.0.1:4648            Bind Addrs: HTTP: [127.0.0.1:4646]; RPC: 127.0.0.1:4647; Serf: 127.0.0.1:4648                Client: true             Log Level: DEBUG               Node Id: af9bef00-2e83-b704-2d6c-c62bc005431f                Region: global (DC: dc1)                Server: true               Version: 1.7.0 ==> Nomad agent started! Log data will stream in below:...

Return to your main terminal window and bootstrap the Nomad ACL system.

$ nomad acl bootstrapAccessor ID  = d1de8625-8556-0932-a25c-3aa71bfc0134Secret ID    = 7f10099a-936c-3f3a-8783-f0980493e54bName         = Bootstrap TokenType         = managementGlobal       = trueCreate Time  = 2023-11-16 01:09:26.565422 +0000 UTCExpiry Time  = <none>Create Index = 23Modify Index = 23Policies     = n/aRoles        = n/a

Copy the value of Secret ID and set it as the environment variable NOMAD_TOKEN.

$ export NOMAD_TOKEN=...

$ export NOMAD_TOKEN=...

$ $Env:NOMAD_TOKEN = "..."

Verify that the Nomad agent was able to register itself in the Consul catalog.

$ consul catalog servicesconsulnomadnomad-client

Configure Consul for services workload identities

Create a Consul ACL JWT auth method

Create a file named consul-auth-method-nomad-workloads.json. Add the following contents to it and save the file.

{  "JWKSURL": "http://127.0.0.1:4646/.well-known/jwks.json",  "JWTSupportedAlgs": ["RS256"],  "BoundAudiences": ["consul.io"],  "ClaimMappings": {    "nomad_namespace": "nomad_namespace",    "nomad_job_id": "nomad_job_id",    "nomad_task": "nomad_task",    "nomad_service": "nomad_service"  }}

This file contains important information for creating a JWT auth method in Consul.

• JWKSURL is the URL that Consul uses to contact Nomad and retrieve the data necessary to validate Nomad workload identities. In a production environment, this should resolve to multiple Nomad agents via a reverse proxy, load balancer, or DNS entry to prevent a single point of failure.
• BoundAudiences is a list of strings that configures Consul to only accept JWTs that have one of these audience values. The default identities in the nomad.hcl configuration file match the values in consul-auth-method-nomad-workloads.json.
• ClaimMappings are values extracted from the Nomad workload identity. You will reference them in upcoming steps.

Create a Consul JWT auth method named nomad-workloads using the consul-auth-method-nomad-workloads.json file.

$ consul acl auth-method create -name 'nomad-workloads' -type 'jwt' -description 'JWT auth method for Nomad services and workloads' -config '@consul-auth-method-nomad-workloads.json'Name:          nomad-workloadsType:          jwtDescription:   JWT auth method for Nomad services and workloadsConfig:{  "BoundAudiences": [    "consul.io"  ],  "ClaimMappings": {    "nomad_job_id": "nomad_job_id",    "nomad_namespace": "nomad_namespace",    "nomad_service": "nomad_service",    "nomad_task": "nomad_task"  },  "JWKSURL": "http://127.0.0.1:4646/.well-known/jwks.json",  "JWTSupportedAlgs": [    "RS256"  ]}

This auth method creates an endpoint for generating Consul ACL tokens from Nomad workload identities.

Create a Consul ACL binding rule for Nomad services

Create a binding rule to associate Consul ACL tokens generated from the nomad-workloads auth method to a service identity.

$ consul acl binding-rule create -method 'nomad-workloads' -description 'Binding rule for services registered from Nomad' -bind-type 'service' -bind-name '${value.nomad_service}' -selector '"nomad_service" in value'ID:           3a5c1a4b-58b6-e779-dfc0-4e3a434b6763AuthMethod:   nomad-workloadsDescription:  Binding rule for services registered from NomadBindType:     serviceBindName:     ${value.nomad_service}Selector:     "nomad_service" in value

The service identity automatically grants the token the permissions to manage the service itself and to query other services in the Consul cluster.

The -bind-name '${value.nomad_service}' flag restricts the token to only be able to modify services with the same name as defined in the Nomad job.

The value of ${value.nomad_service} is extracted from the workload identity used to register the service, as defined in the ClaimMappings configuration of the nomad-workloads auth method, and is dynamically set to the name of the service being registered by Nomad.

The -selector '"nomad_service" in value' flag ensures this rule only applies to workload identities used for service registration because they are the only ones that have the nomad_service claim.

Run a Nomad job to register a service in Consul

Create a file named httpd.nomad.hcl. Add the following contents to it and save the file.

job "httpd" {  group "httpd" {    network {      port "http" {}    }     service {      provider = "consul"      name     = "httpd"      port     = "http"    }     task "httpd" {      driver = "docker"       config {        image   = "busybox:1.36"        command = "httpd"        args    = ["-f", "-p", "${NOMAD_PORT_http}"]        ports   = ["http"]      }    }  }}

job "httpd" {  group "httpd" {    network {      port "http" {}    }     service {      provider = "consul"      name     = "httpd"      port     = "http"    }     task "httpd" {      driver = "docker"       config {        image   = "busybox:1.36"        command = "httpd"        args    = ["-f", "-p", "${NOMAD_PORT_http}"]        ports   = ["http"]      }    }  }}

job "httpd" {  group "httpd" {    network {      port "http" {}    }     service {      provider = "consul"      name     = "httpd"      port     = "http"    }     task "httpd" {      driver = "docker"       config {        image   = "hashicorpdev/busybox-windows:ltsc2019"        command = "httpd"        args    = ["-f", "-p", "${NOMAD_PORT_http}"]        ports   = ["http"]      }    }  }}

This Nomad job runs an HTTP server and registers it as service in the Consul service catalog.

The job does not specify any identity for Consul, so the service_identity from the nomad.hcl agent configuration file is used.

Run the httpd.nomad.hcl job file and wait for the deployment to complete.

$ nomad job run 'httpd.nomad.hcl'==> 2023-11-15T20:15:11-05:00: Monitoring evaluation "3e32e483"    2023-11-15T20:15:11-05:00: Evaluation triggered by job "httpd"    2023-11-15T20:15:12-05:00: Evaluation within deployment: "3ab75837"    2023-11-15T20:15:12-05:00: Allocation "020dc113" created: node "6b4dc583", group "db"    2023-11-15T20:15:12-05:00: Evaluation status changed: "pending" -> "complete"==> 2023-11-15T20:15:12-05:00: Evaluation "3e32e483" finished with status "complete"==> 2023-11-15T20:15:12-05:00: Monitoring deployment "3ab75837"  ✓ Deployment "3ab75837" successful     2023-11-15T20:15:23-05:00    ID          = 3ab75837    Job ID      = httpd    Job Version = 0    Status      = successful    Description = Deployment completed successfully     Deployed    Task Group  Desired  Placed  Healthy  Unhealthy  Progress Deadline    db          1        1       1        0          2023-11-15T20:25:21-05:00

Verify that the httpd service is in the Consul catalog.

$ consul catalog servicesconsulhttpdnomadnomad-client

Retrieve the job service definition from Nomad.

$ nomad job inspect -t '{{sprig_toPrettyJson (index .TaskGroups 0).Services}}' 'httpd'[  {    "Name": "httpd",    "Tags": null,    "CanaryTags": null,...

Note that it has an Identity for Consul.

[  {    "Name": "httpd",    "Tags": null,    "CanaryTags": null,    "EnableTagOverride": false,    "PortLabel": "db",    "AddressMode": "auto",    "Address": "",    "Checks": null,    "CheckRestart": null,    "Connect": null,    "Meta": null,    "CanaryMeta": null,    "TaggedAddresses": null,    "TaskName": "",    "OnUpdate": "require_healthy",    "Identity": {      "Name": "consul-service_httpd-http",      "Audience": [        "consul.io"      ],      "ChangeMode": "",      "ChangeSignal": "",      "Env": false,      "File": false,      "ServiceName": "httpd",      "TTL": 3600000000000    },    "Provider": "consul",    "Cluster": "default"  }]

This Identity block is the workload identity that was automatically injected by Nomad based on the configuration provided in the service_identity block in the nomad.hcl file.

Nomad uses this identity to retrieve a Consul ACL token that is allowed to register the service in Consul.

Configure Consul for tasks workload identities

Create a Consul ACL binding rule for Nomad tasks

Create a binding rule to map ACL tokens from the nomad-workloads auth method to ACL roles with names that match the pattern nomad-tasks-${value.nomad_namespace}.

$ consul acl binding-rule create -method 'nomad-workloads' -description 'Binding rule for Nomad tasks' -bind-type 'role' -bind-name 'nomad-tasks-${value.nomad_namespace}' -selector '"nomad_service" not in value'ID:           025fba50-e3f2-84c9-163e-f42e415ea478AuthMethod:   nomad-workloadsDescription:  Binding rule for Nomad tasksBindType:     roleBindName:     nomad-tasks-${value.nomad_namespace}Selector:     "nomad_service" not in value

The value of ${value.nomad_namespace} is dynamically set to the Nomad namespace of the workload requesting the Consul token. You can also use other values from the auth method ClaimMappings, such as nomad_job_id and nomad_task for more fine-grained mapping of roles.

The -selector '"nomad_service" not in value' flag ensures this rule only applies to workload identities that are not used for service registration because they do not have the nomad_service claim.

Create a Consul ACL policy for Nomad tasks

Create a file named consul-policy-nomad-tasks.hcl. Add the following contents to it and save the file. This policy allows tokens to access any service and KV path in Consul. In production environments, you should adjust the policy should to meet specific access requirements.

key_prefix "" {  policy = "read"} service_prefix "" {  policy = "read"}

Create a Consul ACL policy named nomad-tasks using the file consul-policy-nomad-tasks.hcl.

$ consul acl policy create -name 'nomad-tasks' -description 'ACL policy used by Nomad tasks' -rules '@consul-policy-nomad-tasks.hcl'ID:           8fc8b321-10d6-0bee-cbe4-9a40812e4bd7Name:         nomad-tasksDescription:  ACL policy used by Nomad tasksDatacenters:Rules:key_prefix "" {  policy = "read"} service_prefix "" {  policy = "read"}

Create a Consul ACL role for Nomad tasks

Create a Consul ACL role named nomad-tasks-default that matches the name pattern for jobs in the default namespace using the rules set in the file consul-policy-nomad-tasks.hcl.

$ consul acl role create -name 'nomad-tasks-default' -description 'ACL role for Nomad tasks in the default Nomad namespace' -policy-name 'nomad-tasks'ID:           13c4c9dc-a808-7503-ab07-2e7ef710d398Name:         nomad-tasks-defaultDescription:  ACL role for Nomad tasks in the default Nomad namespacePolicies:   8fc8b321-10d6-0bee-cbe4-9a40812e4bd7 - nomad-tasks

Run a Nomad job to read data from Consul

Write test data to Consul's KV store.

$ consul kv put 'httpd/config/cache' '50'; consul kv put 'httpd/config/maxconn' '100'; consul kv put 'httpd/config/minconn' '3'Success! Data written to: httpd/config/cacheSuccess! Data written to: httpd/config/maxconnSuccess! Data written to: httpd/config/minconn

Create a file named consul-info.nomad.hcl. Add the following contents to it and save the file.

job "consul-info" {  type = "batch"   group "consul-info" {    task "consul-info" {      driver = "docker"       consul {}       config {        image   = "busybox:1.36"        command = "/bin/sh"        args    = ["-c", "exit 0"]      }       template {        data        = <<EOFConsul Services:{{- range services}}  * {{.Name}}{{end}} Consul KV for "httpd/config":{{- range ls "httpd/config"}}  * {{.Key}}: {{.Value}}{{end}}EOF        destination = "local/consul-info.txt"      }    }  }}

job "consul-info" {  type = "batch"   group "consul-info" {    task "consul-info" {      driver = "docker"       consul {}       config {        image   = "busybox:1.36"        command = "/bin/sh"        args    = ["-c", "exit 0"]      }       template {        data        = <<EOFConsul Services:{{- range services}}  * {{.Name}}{{end}} Consul KV for "httpd/config":{{- range ls "httpd/config"}}  * {{.Key}}: {{.Value}}{{end}}EOF        destination = "local/consul-info.txt"      }    }  }}

job "consul-info" {  type = "batch"   group "consul-info" {    task "consul-info" {      driver = "docker"       consul {}       config {        image   = "hashicorpdev/busybox-windows:ltsc2019"        command = "/bin/sh"        args    = ["-c", "exit 0"]      }       template {        data        = <<EOFConsul Services:{{- range services}}  * {{.Name}}{{end}} Consul KV for "httpd/config":{{- range ls "httpd/config"}}  * {{.Key}}: {{.Value}}{{end}}EOF        destination = "local/consul-info.txt"      }    }  }}

Note the template block that reads the names of all services registered in Consul and all the keys and values stored in the KV path httpd/config. This information is then written to a file.

Also note how the job does not have any identities for Consul. Nomad uses the task_identity defined in the Nomad agent configuration to retrieve a Consul ACL token for the task.

Run the Nomad job in consul-info.nomad.hcl.

$ nomad job run 'consul-info.nomad.hcl'==> 2023-11-15T21:22:03-05:00: Monitoring evaluation "94df1181"    2023-11-15T21:22:03-05:00: Evaluation triggered by job "consul-info"    2023-11-15T21:22:04-05:00: Allocation "d3504ad8" created: node "6b4dc583", group "consul-info"    2023-11-15T21:22:04-05:00: Evaluation status changed: "pending" -> "complete"==> 2023-11-15T21:22:04-05:00: Evaluation "94df1181" finished with status "complete"

Retrieve the allocation information and wait until the status is complete. You may need to run the command a few times before the status changes to complete.

$ nomad job allocs 'consul-info'ID        Node ID   Task Group   Version  Desired  Status    Created    Modifiedd3504ad8  6b4dc583  consul-info  0        run      complete  2m27s ago  2m27s ago

Print the file created by the job.

$ nomad alloc fs "$(nomad job allocs -t '{{with index . 0}}{{.ID}}{{end}}' 'consul-info')" 'consul-info/local/consul-info.txt'Consul Services:  * consul  * httpd  * nomad  * nomad-client Consul KV for "httpd/config":  * cache: 50  * maxconn: 100  * minconn: 3

The job can access the Consul API and retrieve the list of services in the catalog as well as the values in the KV store under the httpd/config path.

Retrieve the task definition from Nomad.

$ nomad job inspect -t '{{sprig_toPrettyJson (index (index .TaskGroups 0).Tasks 0)}}' 'consul-info'{  "Name": "consul-info",  "Driver": "docker",  "User": "",  "Lifecycle": null,  "Config": {...

Note that it has an identity for Consul in its Identities list.

{  "Name": "consul-info",  "Driver": "docker",  "User": "",  "Lifecycle": null,  "Config": {  ...  "Identities": [    {      "Name": "consul_default",      "Audience": [        "consul.io"      ],      "ChangeMode": "",      "ChangeSignal": "",      "Env": false,      "File": false,      "ServiceName": "",      "TTL": 3600000000000    }  ],  "Actions": null}

This is the workload identity that Nomad automatically injects based on the task_identity block in the nomad.hcl file.

Nomad uses this identity to retrieve a Consul ACL token with permissions to read services and KV values as defined by the nomad-tasks ACL policy.

Next steps

In this tutorial, you configured Nomad and Consul to communicate with ACL enabled. You also configured Nomad to automatically add workload identities for services and tasks that need access to Consul.

You then deployed a Nomad job to register a service in Consul and another job to read data from it. Both jobs used their workload identities to receive a Consul ACL token properly scoped to the work they needed to do.

This process required several steps, with certain values having to match each other in order for everything to work properly.

There are two resources available to help you automate these steps.

• The Nomad CLI command nomad setup consul can be useful for a quick setup with default values for a development or test cluster.
• The hashicorp-modules/nomad-setup/consul Terraform module provides a basis for applying these steps with an infrastructure-as-code approach, which is more suitable for a production environment. The hashicorp-guides/nomad-workload-identity-terraform-demo repository demonstrates how this module can be used.

You may continue exploring additional integrations between Nomad and Consul or learn how to similarly integrate Nomad and Vault with ACL enabled.

• Nomad and Consul Integration
• Nomad Workload Identity
• Nomad Agent consul Configuration
• Consul Access Control List (ACL) Overview
  • Consul Auth Methods Overview