Observe Consul service mesh traffic

In the previous tutorial, you learned how to configure and deploy your services using Consul service mesh solution. By using Consul service mesh, you enabled zero trust security in your network by having all service-to-service traffic encrypted and managed by Envoy sidecar proxies.

In this tutorial, you will configure and use Consul to observe traffic within your service mesh. This enables you to quickly understand how services interact with each other and effectively debug your services' traffic.

Using the Grafana suite, you can have your monitoring configuration embedded in your service VM and have the metrics collection configuration automatically scale when you deploy new services without the need to manually add new scraping endpoints for each new service you want to deploy.

In this tutorial, you will use a test environment to:

• Create configuration files for Grafana agent tool
• Start Grafana agent on all Consul nodes
• Visualize metrics in Grafana
• Visualize available metrics in Consul UI

Tutorial scenario

This tutorial uses HashiCups, a demo coffee shop application made up of several microservices running on VMs.

At the beginning of the tutorial, you have a fully deployed Consul service mesh with Envoy sidecar proxies running alongside each service.

By the end of this tutorial, you will have enabled metrics collection on your Consul nodes and will be able to visualize metrics both in Consul UI and in a Grafana dashboard.

Prerequisites

This tutorial uses an interactive lab to guide you through how to enable metrics collection for your Consul service mesh. The lab environment includes all required binaries and sample configurations.

Launch Terminal

This tutorial includes a free interactive command-line lab that lets you follow along on actual cloud infrastructure.

Start interactive lab

Configure Grafana Agent

You can configure Grafana Agent to collect several kinds of data from your VM. In this tutorial, you will use configurations for:

• metrics block, to define a collection of Prometheus-compatible scrape configs to be written in Mimir.
• logs block, to configure how the Agent collects logs and sends them to a Loki push API endpoint.

Generate configuration for Grafana Agent

This tutorial and interactive lab environment uses scripts in the tutorial's GitHub repository to generate the Consul configuration files for your client agents.

The Bastion Host includes the script in the ops/scenarios/99_supporting_scripts folder.

$ tree ./ops/scenarios/99_supporting_scripts/./ops/scenarios/99_supporting_scripts/|-- generate_consul_client_config.sh|-- generate_consul_monitoring_config.sh|-- generate_consul_server_config.sh|-- generate_consul_server_tokens.sh|-- generate_consul_service_config.sh`-- generate_consul_service_intentions.sh 1 directory, 6 files

The script requires a few parameters to work correctly:

• an OUTPUT_FOLDER to place the files generated
• a PROMETHEUS_URI to push metrics to. In this scenario we configured Grafana Mimir for this task listening on the bastion host.
• a LOKI_URI to push logs to. In this scenario we configured Grafana Loki for this task listening on the bastion host.

$ export OUTPUT_FOLDER="./assets/scenario/conf/"; \  export PROMETHEUS_URI=`getent hosts mimir | awk '{print $1}'`; \  export LOKI_URI=`getent hosts loki | awk '{print $1}'`

With these values configured, generate the configuration.

$ bash ops/scenarios/99_supporting_scripts/generate_consul_monitoring_config.sh -- Parameter Check -- Generating Grafana Agent configuration

The script creates the Grafana Agent configuration for all agents.

$ tree ${OUTPUT_FOLDER}monitoring./assets/scenario/conf/monitoring├── grafana-agent-consul-server-0.yaml├── grafana-agent-gateway-api.yaml├── grafana-agent-hashicups-api.yaml├── grafana-agent-hashicups-db.yaml├── grafana-agent-hashicups-frontend.yaml└── grafana-agent-hashicups-nginx.yaml 0 directories, 6 files

Copy configuration on client VMs

After the script generates the configuration files, you will copy these files in each client node.

$ scp -O -i certs/id_rsa ${OUTPUT_FOLDER}monitoring/grafana-agent-consul-server-0.yaml consul-server-0:grafana-agent.yaml; \  scp -O -i certs/id_rsa ${OUTPUT_FOLDER}monitoring/grafana-agent-hashicups-db.yaml hashicups-db:grafana-agent.yaml; \  scp -O -i certs/id_rsa ${OUTPUT_FOLDER}monitoring/grafana-agent-hashicups-api.yaml hashicups-api:grafana-agent.yaml; \  scp -O -i certs/id_rsa ${OUTPUT_FOLDER}monitoring/grafana-agent-hashicups-frontend.yaml hashicups-frontend:grafana-agent.yaml; \  scp -O -i certs/id_rsa ${OUTPUT_FOLDER}monitoring/grafana-agent-hashicups-nginx.yaml hashicups-nginx:grafana-agent.yaml

Start Grafana Agent on VMs

Once you copied the configuration files on the different VMs, login on each Consul client VMs and start the Grafana Agent.

Start Grafana Agent for Consul server

Select the tab that corresponds with the service — in this case, Consul.

Start the Grafana Agent.

$ grafana-agent -config.file grafana-agent.yaml > /tmp/grafana-agent.log 2>&1 &

Start Grafana Agent for Database

Select the tab that corresponds with the service — in this case, Database.

Start the Grafana Agent.

$ grafana-agent -config.file grafana-agent.yaml > /tmp/grafana-agent.log 2>&1 &

Start Grafana Agent for API

Select the tab that corresponds with the service — in this case, API.

Start the Grafana Agent.

$ grafana-agent -config.file grafana-agent.yaml > /tmp/grafana-agent.log 2>&1 &

Start Grafana Agent for Frontend

Select the tab that corresponds with the service — in this case, Frontend.

Start the Grafana Agent.

$ grafana-agent -config.file grafana-agent.yaml > /tmp/grafana-agent.log 2>&1 &

Start Grafana Agent for NGINX

Select the tab that corresponds with the service — in this case, NGINX.

Start the Grafana Agent.

$ grafana-agent -config.file grafana-agent.yaml > /tmp/grafana-agent.log 2>&1 &

Visualize metrics in Grafana

Once started all the Grafana Agents, the metrics will be available to Grafana. Open the Grafana UI to view the metrics.

The interactive lab environment includes some predefined Grafana dashboards.

Navigate to the Grafana UI tab to find the Grafana dashboard.

The HashiCups dashboard shows an overview of the services deployed in the service mesh.

Topology visualization in Consul UI

Consul provides configuration entries that can be used to get a summary of traffic across services as well as some metrics to get a basic overview of service health.

In order to visualize metrics it is necessary to generate some traffic for your application. In the lab, select the tab HashiCups tab and perform a few transactions in the application.

After completing a few purchases select the Consul UI tab and login using the bootstrap token.

In your Consul dashboard, select Services then hashicups-api to find the topology page.

Click on the Open Dashboard link. The link under the service box will open the dashboard with the specific service selected to get more specific information directly.

Next steps

In this tutorial, you learned how to monitor your Consul service mesh and the services deployed in it using the Grafana suite.

You now have a distributed system to monitor your Consul service mesh. Using the Grafana Agent lets you embed the metrics export into your VMs' golden images and have your metrics automatically gathered when you add new services to the mesh without the need to edit your monitoring suite configuration.

For more information about the topics covered in this tutorial, refer to the following resources:

• Service Mesh Observability Overview
• Service Mesh Observability: UI Visualization