Manage permissions with access control lists (ACLs) on VMs
Consul service mesh provides secure communication across your services within and across your infrastructure, including on-premises and cloud environments. In a service mesh scenario, you configure the single service instances to listen on the loopback interface and, using the upstreams in the service definition, Envoy sidecar proxies permit communication with other services by adding listeners locally. These communication are protected via an mTLS layer that prevents unauthorized services, inside or outside the mesh, to access them.
Consul provides one extra layer of security, Access Control Lists (ACLs), to prevent malicious actors to register services and hijack communications in your datacenter. A Consul datacenter with ACLs enabled and with default-deny policy, will require a valid token for every operation, including service registration.
In this tutorial you will learn how to create credentials required to join an existing Consul datacenter and register a service to the Consul catalog. After an introductory section about the different tokens available for Consul agents' configuration, you will generate tokens for the Consul node using the node-identiy template policy, and for a Consul service using the service-identity policy. You will also learn how to reuse an existing policy to generate a token for the Consul DNS interface, to allow service resolution locally.
You will use the generated credentials to create a service configuration for the API service, and to add a new Consul agent to Consul service mesh. After the agent joins the Consul service mesh, you will verify the service is correctly registered in the mesh using all the interfaces provided by Consul: CLI, API, DNS, and web UI.
Tutorial scenario
This tutorial uses HashiCups, a demo coffee shop application made up of several microservices running on VMs.
At the beginning of the tutorial, you will deploy Consul service mesh, with Envoy sidecar proxies running alongside each service and an API Gateway configured to permit access to the NGINX service.
In addition, you will deploy an extra node that hosts another HashiCups API instance that you want to add to your datacenter.
In this tutorial, you will use ACLs to add the API service to the Consul service mesh.
Once you have registered the service in Consul, it will automatically appear in Consul DNS results and Consul will automatically load balance traffic across the different instances of the service.
Note
All operations in this tutorial, after the scenario deployment, will be performed from an additional node, a bastion host, that will be deployed as part of the scenario. The tutorial provides instructions on how to connect to the bastion host at the end of the deployment.
Prerequisites
This tutorial assumes you are already familiar with Consul service mesh and its core functionalities. If you are new to Consul refer to the Consul Getting Started tutorials collection.
For this tutorial, you will need:
Clone GitHub repository
Clone the GitHub repository containing the configuration files and resources.
$ git clone https://github.com/hashicorp-education/learn-consul-acls-vms.gitEnter the directory that contains the configuration files for this tutorial.
$ cd learn-consul-acls-vms/self-managed/infrastructure/awsCreate infrastructure
With these Terraform configuration files, you are ready to deploy your infrastructure.
Issue the terraform init command from your working directory to download the
necessary providers and initialize the backend.
$ terraform initInitializing the backend...Initializing provider plugins......Terraform has been successfully initialized!...Then, deploy the resources. Confirm the run by entering yes.
$ terraform apply -var-file=../../ops/conf/manage_permissions_with_acls.tfvars## ...Do you want to perform these actions? Terraform will perform the actions described above. Only 'yes' will be accepted to approve. Enter a value: yes## ...Apply complete! Resources: 50 added, 0 changed, 0 destroyed.Tip
The Terraform deployment could take up to 15 minutes to complete. Feel free to explore the next sections of this tutorial while waiting for the environment to complete initialization.
After the deployment is complete, Terraform returns a list of outputs you can use to interact with the newly created environment.
Outputs:connection_string = "ssh -i certs/id_rsa.pem admin@`terraform output -raw ip_bastion`"ip_bastion = "<redacted-output>"remote_ops = "export BASTION_HOST=<redacted-output>"retry_join = "provider=aws tag_key=ConsulJoinTag tag_value=auto-join-hcoc"ui_consul = "https://<redacted-output>:8443"ui_grafana = "http://<redacted-output>:3000/d/hashicups/hashicups"ui_hashicups = "http://<redacted-output>"The Terraform outputs provide useful information, including the bastion host IP address. The following is a brief description of the Terraform outputs:
- The
ip_bastionprovides IP address of the bastion host you use to run the rest of the commands in this tutorial. - The
remote_opslists the bastion host IP, which you can use access the bastion host. - The
retry_joinoutput lists Consul'sretry_joinconfiguration parameter. The next tutorial uses this information to generate Consul server and client configuration. - The
ui_consuloutput lists the Consul UI address. The Consul UI is not currently running. You will use the Consul UI in a later tutorial to verify that Consul started correctly. - The
ui_grafanaoutput lists the Grafana UI address. You will use this address in a future tutorial. - The
ui_hashicupsoutput lists the HashiCups UI address. You can open this address in a web browser to verify the HashiCups demo application is running properly.
List AWS instances
The scenario deploys eight virtual machines.
$ terraform state list## ...aws_instance.api[0]aws_instance.api[1]aws_instance.bastionaws_instance.consul_server[0]aws_instance.database[0]aws_instance.frontend[0]aws_instance.gateway-api[0]aws_instance.nginx[0]## ...After deployment six virtual machines, consul_server[0], database[0], frontend[0], api[0], nginx[0], and gateway-api[0] are configured in a Consul datacenter with service mesh enabled.
The two remaining nodes are:
bastion- the bastion host that you will use to perform the tutorial steps.api[1]- a virtual machine to host a second instance of thehashicups-apiservice.
Login into the bastion host VM
Login to the bastion host using ssh.
$ ssh -i certs/id_rsa.pem admin@`terraform output -raw ip_bastion`#...admin@bastion:~$Configure CLI to interact with Consul
Configure your bastion host to communicate with your Consul environment using the two, dynamically generated, environment variable files.
$ source "/home/admin/assets/scenario/env-scenario.env" && \ source "/home/admin/assets/scenario/env-consul.env"After loading the needed variables, verify you can connect to your Consul datacenter.
$ consul membersNode Address Status Type Build Protocol DC Partition Segmentconsul-server-0 192.168.80.7:8301 alive server 1.17.0 2 dc1 default <all>gateway-api-0 192.168.80.4:8301 alive client 1.17.0 2 dc1 default <default>hashicups-api-0 192.168.80.8:8301 alive client 1.17.0 2 dc1 default <default>hashicups-db-0 192.168.80.2:8301 alive client 1.17.0 2 dc1 default <default>hashicups-frontend-0 192.168.80.12:8301 alive client 1.17.0 2 dc1 default <default>hashicups-nginx-0 192.168.80.11:8301 alive client 1.17.0 2 dc1 default <default>Notice that bastion and hashicups-api-1 are not part of the datacenter.
Create ACL token for ACL management
A management token is required to manage Consul's ACLs system. We recommend creating a new Consul management token for this purpose instead of using the ACL bootstrap token.
$ consul acl token create \ -description="SVC HashiCups API token" \ --format json \ -policy-name="global-management" | tee /home/admin/assets/scenario/conf/secrets/acl-token-management.jsonThat will produce an output similar to the following.
{ "CreateIndex": 115, "ModifyIndex": 115, "AccessorID": "9cdf707d-8847-9bb2-606c-d4f0849692d6", "SecretID": "e4d09a88-f3dc-0191-7ce6-f7a35a7097ae", "Description": "SVC HashiCups API token", "Policies": [ { "ID": "00000000-0000-0000-0000-000000000001", "Name": "global-management" } ], "Local": false, "CreateTime": "2023-11-18T13:27:42.735373628Z", "Hash": "UfTdHatx/OxAw8z+/8suSKQrY6MgpC3CutkuZZAzPC4="}Set your newly generated management token as the CONSUL_HTTP_TOKEN environment variable. This will allow you to perform ACL system operations in the next steps using the new token.
$ export CONSUL_HTTP_TOKEN=`cat /home/admin/assets/scenario/conf/secrets/acl-token-management.json | jq -r ".SecretID"`Create ACL configuration for Consul agent
Consul provides a fine grained set of permissions to manage your Consul nodes and services. In this section you will create the ACL tokens required to register a new Consul client node in a Consul datacenter. You will then register a service in the Consul service mesh with these ACL tokens.
The initial tutorial environment includes an empty VM named hashicups-api-1 for you. You will create ACLs token to register this Consul client node to the Consul datacenter, and register the service in Consul.
Consul has a section in the configuration to specify all the ACL tokens to be used by the agent.
agent-acl-tokens.hcl
...acl { tokens { agent = "<Consul agent token>" default = "<Consul default token>" config_file_service_registration = "<Consul service registration token>" }}...The different tokens are used for different operation performed by the Consul agent:
agent- Used for clients and servers to perform internal operations. This token must have write access to the node name it will register as in order to set any of the node-level information in the catalog.default- Used for the default operations whenagenttoken is not provided and for the endpoints that do not permit a token being passed. This will be used for the DNS interface of the Consul agent.config_file_service_registration- Specifies the ACL token the agent uses to register services and checks. This token needs write permission to register all services and checks defined in this agent's configuration.
You will generate the ACL configuration section after these three tokens are generated.
Generate default token
Each token must be associated with a policy providing the desired permissions. The scenario already contains a policy used to generate the default token for the other nodes that is already suitable for the node default token. The policy is named acl-policy-dns.
Read the policy using the consul acl policy command.
$ consul acl policy read -name acl-policy-dnsID: f77b2144-190f-00ee-46b5-cb769d018e8eName: acl-policy-dnsDescription: Policy for DNS endpointsDatacenters: Rules:# -----------------------------+# acl-policy-dns.hcl |# -----------------------------+ node_prefix "" { policy = "read"}service_prefix "" { policy = "read"}# only needed if using prepared queriesquery_prefix "" { policy = "read"}# needed for prometheus metricsagent_prefix ""{ policy = "read"}The acl-policy-dns policy, used to generate the default token, grants read access to all services and nodes so DNS query can work. It also grants read permissions for prepared queries and agent metrics endpoints.
Generate a new token using the acl-policy-dns policy.
$ consul acl token create \ -description="hashicups-api-1 default token" \ --format json \ -policy-name="acl-policy-dns" | tee /home/admin/assets/scenario/conf/secrets/acl-token-default-hashicups-api-1.jsonThat will produce an output similar to the following.
{ "CreateIndex": 116, "ModifyIndex": 116, "AccessorID": "ba4533ad-6e44-7279-41d3-08bca235306f", "SecretID": "391bc1b8-2837-db2c-6466-613e16051050", "Description": "hashicups-api-1 default token", "Policies": [ { "ID": "f77b2144-190f-00ee-46b5-cb769d018e8e", "Name": "acl-policy-dns" } ], "Local": false, "CreateTime": "2023-11-18T13:27:42.871333836Z", "Hash": "xPN3IX5yLnk7SDzHYXIBCPKpQtAWGP/VoLfZBmiK9kA="}Generate agent token
In order for the agent to set any node-level information in Consul catalog, the agent token must have write access to the node name it will register as. One minimal example for such a policy is the following.
node-identity-policy-template.hcl
# Allow the agent to register its own node in the Catalog and update its network coordinatesnode "<node name>" { policy = "write"} # Allows the agent to detect and diff services registered to itself. This is used during# anti-entropy to reconcile difference between the agents knowledge of registered# services and checks in comparison with what is known in the Catalog.service_prefix "" { policy = "read"}You can use node-identity to help you generate agent tokens for Consul nodes. Node identities let you to quickly construct policies for nodes, rather than manually creating identical polices for each node. Generate a new token for hashicups-api-1 using node-identity.
$ consul acl token create \ -description="hashicups-api-1 agent token" \ --format json \ -node-identity="hashicups-api-1:dc1" | tee /home/admin/assets/scenario/conf/secrets/acl-token-agent-hashicups-api-1.jsonThat will produce an output similar to the following.
{ "CreateIndex": 117, "ModifyIndex": 117, "AccessorID": "715cad9a-ebc5-21cd-0919-ac08effdbf3d", "SecretID": "3c470114-1214-d4fe-e3c5-3dd1f9515373", "Description": "hashicups-api-1 agent token", "NodeIdentities": [ { "NodeName": "hashicups-api-1", "Datacenter": "dc1" } ], "Local": false, "CreateTime": "2023-11-18T13:27:42.918717795Z", "Hash": "dn0S8locXJeqcWcmqHURzQDKSO4UZKwV49QO0YCCbv8="}Generate config_file_service_registration token
The config_file_service_registration token needs write permission to register all services and checks defined in this agent's configuration. For Consul service mesh, the token also needs permissions to register the Envoy sidecar proxy for the service as an independent service in Consul catalog. One minimal example for such a policy is the following.
service-identity-policy-template.hcl
# Allow the service and its sidecar proxy to register into the catalog.service "<service name>" { policy = "write"}service "<service name>-sidecar-proxy" { policy = "write"} # Allow for any potential upstreams to be resolved.service_prefix "" { policy = "read"}node_prefix "" { policy = "read"}You can use service-identity to help you generate config_file_service_registration tokens for Consul nodes. Service identities let you to quickly construct policies for services, rather than manually creating identical polices for each service instance and for their companion sidecar proxy. Generate a new token for hashicups-api-1 using service-identity.
$ consul acl token create \ -description="hashicups-api-1 config_file_service_registration token" \ --format json \ -service-identity="hashicups-api:dc1" | tee /home/admin/assets/scenario/conf/secrets/acl-token-svc-hashicups-api-1.jsonThat will produce an output similar to the following.
{ "CreateIndex": 118, "ModifyIndex": 118, "AccessorID": "8cf68272-25ce-0d87-8dec-713890e8a0ba", "SecretID": "75906ad5-2162-efcb-cf95-63a2b6396840", "Description": "hashicups-api-1 config_file_service_registration token", "ServiceIdentities": [ { "ServiceName": "hashicups-api", "Datacenters": [ "dc1" ] } ], "Local": false, "CreateTime": "2023-11-18T13:27:42.963791211Z", "Hash": "WCtk+YLUNIm5VR0zLykM9oo5TG4xdBbwAIBb3fSZ2qA="}Verify you have created the three tokens and the files containing the tokens are present in the assets/ folder.
$ ls ~/assets/scenario/conf/secrets/ | grep hashicups-api-1acl-token-agent-hashicups-api-1.jsonacl-token-default-hashicups-api-1.jsonacl-token-svc-hashicups-api-1.jsonGenerate Consul agent ACL configuration section
With the three tokens you created, generate the Consul agent ACL configuration.
$ tee /home/admin/assets/scenario/conf/hashicups-api-1/agent-acl-tokens.hcl > /dev/null << EOFacl { tokens { agent = "`cat /home/admin/assets/scenario/conf/secrets/acl-token-agent-hashicups-api-1.json | jq -r ".SecretID"`" default = "`cat /home/admin/assets/scenario/conf/secrets/acl-token-default-hashicups-api-1.json | jq -r ".SecretID"`" config_file_service_registration = "`cat /home/admin/assets/scenario/conf/secrets/acl-token-svc-hashicups-api-1.json | jq -r ".SecretID"`" }}EOFGenerate hashicups-api service configuration
Generate hashicups-api service configuration.
$ tee /home/admin/assets/scenario/conf/hashicups-api-1/svc-hashicups-api.hcl > /dev/null << EOF## -----------------------------## svc-hashicups-api.hcl## -----------------------------service { name = "hashicups-api" id = "hashicups-api-1" tags = [ "inst_1" ] port = 8081 token = "`cat /home/admin/assets/scenario/conf/secrets/acl-token-svc-hashicups-api-1.json | jq -r ".SecretID"`" connect { sidecar_service { proxy { upstreams = [ { destination_name = "hashicups-db" local_bind_port = 5432 } ] } } } checks =[ { id = "check-hashicups-api.public.http", name = "hashicups-api.public HTTP status check", service_id = "hashicups-api-1", http = "http://localhost:8081/health", interval = "5s", timeout = "5s" }, { id = "check-hashicups-api.public", name = "hashicups-api.public status check", service_id = "hashicups-api-1", tcp = "localhost:8081", interval = "5s", timeout = "5s" }, { id = "check-hashicups-api.product", name = "hashicups-api.product status check", service_id = "hashicups-api-1", tcp = "localhost:9090", interval = "5s", timeout = "5s" }, { id = "check-hashicups-api.payments", name = "hashicups-api.payments status check", service_id = "hashicups-api-1", tcp = "localhost:8080", interval = "5s", timeout = "5s" }]}EOF The tutorial shows an example of how to configure a service in Consul service mesh. The same concepts apply to a Consul service discovery. If you are trying to register a new service for service discovery, remove the connect section in the service configuration file.
Add new service to Consul service mesh with ACLs
In the previous paragraphs, you created agent-acl-tokens.hcl, containing token configuration for the Consul agent, and svc-hashicups-api.hcl, containing service configuration for the hashicups-api service instance. The bastion host already contains the remaining configuration needed to configure Consul on the hashicups-api-1 node. Review the full configuration.
$ ls /home/admin/assets/scenario/conf/hashicups-api-1agent-acl-tokens.hclagent-gossip-encryption.hclconsul-agent-ca.pemconsul.hclsvc-hashicups-api.hclCopy the configuration on the remote node.
$ scp -r -i /home/admin/certs/id_rsa /home/admin/assets/scenario/conf/hashicups-api-1/* admin@hashicups-api-1:/etc/consul.d/agent-acl-tokens.hcl ...agent-gossip-encryption.hcl ...consul-agent-ca.pem ...consul.hcl ...svc-hashicups-api.hcl ...Start Consul process
Login to hashicups-api-1 from the bastion host.
$ ssh -i certs/id_rsa hashicups-api-1#..admin@hashicups-api-1:~Verify the VM contains the files required for Consul configuration.
$ ls /etc/consul.dagent-acl-tokens.hclagent-gossip-encryption.hclconsul-agent-ca.pemconsul.hclsvc-hashicups-api.hclStart Consul agent.
$ /usr/bin/consul agent \ -retry-join=consul \ -log-file=/tmp/consul-client \ -config-dir=/etc/consul.d > /tmp/consul-client.log 2>&1 &The command starts the Consul server in the background to not lock the terminal.
You can access the Consul server log through the /tmp/consul-server.log file.
Start Envoy sidecar proxy
To make the API service instance join Consul service mesh, your node requires an Envoy proxy acting as a sidecar for the service. You need a valid token with the permission to register the sidecar as a service in Consul catalog to start the proxy process.
Use the token created previously using the service-identity policy.
$ export _agent_token=`cat /etc/consul.d/agent-acl-tokens.hcl | grep -Po "(?<=registration = \")[^\"]+(?=\")"` && echo ${_agent_token}That will produce an output similar to the following.
75906ad5-2162-efcb-cf95-63a2b6396840Start Envoy sidecar proxy for hashicups-api service.
$ /usr/bin/consul connect envoy \ -token=${_agent_token} \ -envoy-binary /usr/bin/envoy \ -sidecar-for hashicups-api-1 > /tmp/sidecar-proxy.log 2>&1 &The command starts the Envoy sidecar proxy in the background to not lock the
terminal. You can access the Envoy log through the /tmp/sidecar-proxy.log file.
Start the API service instance
Once the Consul node joined the Consul datacenter and the Envoy sidecar is configured and ready to route traffic to and from your service, start the hashicups-api service to listen on the local network interface.
$ ~/start_service.sh localThe output of the command will show some Docker output while the Docker images that compose the services are being pulled on the node.
To continue with the tutorial, exit the ssh session to return to the bastion host.
$ exitlogoutConnection to hashicups-api-1 closed.admin@bastion:~$Verify service registration
After you registered the service, it will start appearing among the results when Consul catalog is queried.
Tip
To simplify tests all services already present in the Consul service mesh had a tag assigned, inst_0 while the new instance got registered using a different tag, inst_1.
Verify it using the different interfaces exposed by Consul.
Use the consul catalog command to query the Consul catalog and show all available services.
$ consul catalog services -tagsThe command will produce an output similar to the following.
consul gateway-api hashicups-api inst_0,inst_1hashicups-api-sidecar-proxy inst_0,inst_1hashicups-db inst_0hashicups-db-sidecar-proxy inst_0hashicups-frontend inst_0hashicups-frontend-sidecar-proxy inst_0hashicups-nginx inst_0hashicups-nginx-sidecar-proxy inst_0Notice that the hashicups-api and hashicups-api-sidecar-proxy instances present two different tags, inst_0 and inst_1, that indicate that both instances are registered within Consul.
Destroy the infrastructure
Now that the tutorial is complete, clean up the infrastructure you created.
From the ./self-managed/infrastruture/aws folder of the repository, use terraform to destroy the infrastructure.
$ terraform destroy --auto-approveNext steps
In this tutorial you learned the necessary steps to generate Access Control Lists' tokens to add a new Consul agent to your Consul datacenter and to register a new service to your Consul service mesh.
Specifically you:
- Generated a new management token used the built-in global-management policy.
- Generated a new token from a pre-existing policy
- Generated a new token using the
node-identitypolicy - Generated a new token using the
service-identitypolicy - Verified the successful node and service registration using all the different interfaces provided by Consul: CLI, API, DNS, and UI
For more information about the topics covered in this tutorial, refer to the following resources:
To learn more about other security features provided by Consul service mesh refer to: