Service mesh observability with Docker Compose
In this tutorial, you will create a local Consul service mesh and observability stack using Docker Compose. You will then explore your Consul service mesh with the observability suite.
The complexity of a service mesh environment requires advanced mechanisms to determine where services are running, the traffic patterns of the data flowing, and the health of the connections between them. Having this information makes it easier to not only understand failures occurring in your deployments, but also gives you the granularity needed to identify more subtle failures within your network such as performance degradation, packet loss, memory pressure, etc.
True service mesh observability consist of three core elements: metrics, logs and traces. To explore each of these elements, you will create an observability suite comprised of Prometheus (metrics), Loki (logs), Tempo (traces), and Grafana (visualization). Your Consul service mesh will host an example microservice application that will exhibit the ingress, API, and web layers. As you access the application; metrics, logs, and traces will be sent to your observability stack for you to explore.
This tutorial uses elements that are not suitable for production environments including an unsecured Consul datacenter, relaxed security policies, and container configurations only suited for development. The tutorial will teach you the core concepts for understanding and exploring observability in a Consul service mesh. Refer to the Consul Reference Architecture for Consul best practices and the Docker Documentation for Docker best practices.
Prerequisites
Linux or MacOS
Elements of this tutorial require a Linux or MacOS environment for deployment. See the Docker plugin page for more information about OS requirements.
Docker
You will need a local install of Docker running on your machine for this tutorial. You can find the instructions for installing Docker on your specific operating system here. Docker Engine 19.03.0 was tested in this tutorial.
Docker Compose
Docker Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file to configure your application’s services. Visit the Docker Compose install guide for operating system specific installation instructions. Docker Compose format 3.7 was used in this tutorial.
Loki logging drivers
Install the Loki logging drivers for Docker. See the Loki Docker driver plugin page for more information.
$ docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissionsGitHub repository
Clone the GitHub repository containing the configuration files and resources.
$ git clone https://github.com/hashicorp/learn-consul-docker.gitChange into the directory with the newly cloned repository. This directory contains the complete configuration files.
$ cd learn-consul-docker/datacenter-deploy-observabilityEnvironment overview
The observability suite provides insight into the metrics, logs, and traces that occur throughout the Consul service mesh. This diagram illustrates the components of the observability suite and Consul service mesh that you will create in this tutorial.
Create environment
From within your working directory, run the following Docker Compose command to create your local service mesh environment and observability suite.
$ docker-compose up --detach Creating network "datacenter-deploy-observability_vpcbr" with driver "bridge"Creating node-exporter ... doneCreating tempo ... doneCreating grafana ... doneCreating consul-server ... doneCreating prometheus ... doneCreating loki ... doneCreating ingress ... doneCreating web ... doneCreating api ... doneCreating api_proxy ... doneCreating ingress_proxy ... doneCreating web_proxy ... doneNote
Your first run will take the longest as Docker will pull the respective images from the Docker Hub repository. Extra runs will not need you to download the image again and should only take a few seconds to complete.
Perform testing procedures
Perform the following steps of each section to explore the elements involved in the service mesh and observability suite.
Explore the Consul UI
Navigate to http://localhost:8500/ on your browser to access the Consul UI. Consul will navigate to the services tab by default.
Notice the services monitored by Consul. Each of the applications (ingress, web, and api) includes a sidecar proxy that makes up the service mesh.
Note
If your Consul UI details do not match the image above, restart your container stack environment.
Explore the sample application
Navigate to http://localhost:9090/ui/ and refresh the page to generate traffic.
The example application shows the architecture of your complete application stack. Each refresh of this page will generate extra logs, metrics, and traces that your observability suite will ingest.
Explore Prometheus targets
Navigate to http://localhost:9092/targets.
Notice that Prometheus is pre-configured to scrape metric-related data from four endpoints: Consul, Node-Exporter, Tempo, and itself.
Explore Grafana data sources
Navigate to http://localhost:3000/datasources.
Grafana is pre-configured with Prometheus, Loki, and Tempo as data sources. These sources enable Grafana to visualize metrics, logs, and traces.
Explore Logs and Traces
Navigate to http://localhost:3000/explore.
Execute a search with these parameters: {container_name="web"} |= "trace_id"
Open one of the log lines, locate the TraceID field, then click the nearby Tempo link to jump directly from logs to traces.
Using this functionality, notice how logs and traces provide insight into service mesh performance.
To learn more about how application communication traverses through the service mesh, see the distributed tracing article.
Explore Metrics
Navigate to http://localhost:3000/dashboards.
Explore the metrics-related dashboards: Consul Server Monitoring and Node Exporter.
The Consul Server Monitoring dashboard presents important application-level metrics for Consul while the Node Exporter dashboard presents system-level metrics. Each of these dimensions provides important insight into the health and performance of your service mesh.
To learn more about the various runtime metrics reported by Consul, see the Consul telemetry docs.
Clean up your environment
To clean up your environment, execute the following command.
$ docker-compose down --rmi all Stopping ingress_proxy ... doneStopping api_proxy ... doneStopping web_proxy ... doneStopping web ... doneStopping api ... doneStopping ingress ... doneStopping grafana ... doneStopping node-exporter ... doneStopping prometheus ... doneStopping consul-server ... doneStopping tempo ... doneStopping loki ... doneRemoving ingress_proxy ... doneRemoving api_proxy ... doneRemoving web_proxy ... doneRemoving web ... doneRemoving api ... doneRemoving ingress ... doneRemoving grafana ... doneRemoving node-exporter ... doneRemoving prometheus ... doneRemoving consul-server ... doneRemoving tempo ... doneRemoving loki ... doneRemoving network datacenter-deploy-observability_vpcbrRemoving image hashicorp/consul:1.8.10Removing image nicholasjackson/fake-service:v0.21.0Removing image nicholasjackson/consul-envoy:v1.6.1-v0.10.0Removing image prom/node-exporter:v1.1.2Removing image prom/prometheus:v2.26.0Removing image grafana/tempo:1f1c40b3Removing image grafana/grafana:7.5.3Removing image grafana/loki:2.1.0Next steps
In this tutorial, you learned how to deploy and configure a local Consul service mesh and observability stack using Docker Compose. You also learned how to explore your Consul service mesh with the open source observability suite. Finally, you learned how to clean up your Docker environment.
You can continue learning the concepts of a secure Consul datacenter by completing the deploy a secure Consul datacenter with Docker Compose tutorial. This will teach you the concepts of Access Control Lists (ACLs), gossip encryption, and transport encryption.
You can also extend your Consul skills by exploring the following tutorials:
For additional reference documentation on the technologies and Docker images used in this guide, refer to the following websites: