News HashiCorp to join IBM to accelerate multi-cloud automation. Read the blog →
Consul
Get Started on Kubernetes

Observe Consul service mesh traffic

  • 10min
  • |
  • Consul

Deployment

In this tutorial, you will configure and use Consul to observe traffic within your service mesh. This enables you to quickly understand how services interact with each other and effectively debug your services' traffic.

To enable Consul's observability features, you will:

  • Enable additional Consul features with Terraform and Helm
  • Deploy monitoring suite (Grafana, Loki, Prometheus)
  • Generate traffic in the HashiCups UI
  • Explore the Consul UI metrics visualization
  • Explore the monitoring suite

Prerequisites

The tutorial assumes that you have completed the previous tutorials in this getting started collection.

For this tutorial, you will need:

Enable Consul telemetry features

You will now enable additional Consul features in your Kubernetes cluster using Terraform.

Review the additional configuration details. These configuration items enable metrics features for Consul agents, the Consul UI, and the Envoy proxy sidecars.

helm/values-v3.yaml

# Contains values that affect multiple components of the chart.global: ## … metrics:   enabled: true   # Enables Consul servers and clients metrics.   enableAgentMetrics: true   # Configures the retention time for metrics in Consul servers and clients.   agentMetricsRetentionTime: "1m"## …ui: metrics:   enabled: true   # The metrics provider specification.   provider: "prometheus"   # The URL of the prometheus metrics server.   baseURL: http://prometheus-server.default.svc.cluster.local## ...connectInject: metrics:   defaultEnabled: true   # Enables service metrics and sidecar proxy metrics merging into one endpoint.   defaultEnableMerging: true## …

Update your Consul deployment's Helm chart values.

$ cp helm/values-v3.yaml modules/eks-client/template/consul.tpl

Then, deploy the resources. Confirm the run by entering yes.

$ terraform apply ## ...Do you want to perform these actions? Terraform will perform the actions described above. Only 'yes' will be accepted to approve.  Enter a value: yes ## ... Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

In addition to updating your Consul environment, you will also configure the Envoy proxy sidecars in your service mesh to expose a Prometheus metrics endpoint. This allows Prometheus to scrape metrics from the Envoy sidecar proxies.

The ProxyDefaults configuration entry lets you configure global defaults across all services for Consul service mesh proxy configurations. The proxy/proxy-defaults.yaml file configures envoy_prometheus_bind_addr to expose a Prometheus metrics endpoint.

proxy/proxy-defaults.yaml

apiVersion: consul.hashicorp.com/v1alpha1kind: ProxyDefaultsmetadata:  name: globalspec:  config:    envoy_prometheus_bind_addr: '0.0.0.0:20200'

Modify the proxy defaults.

$ kubectl apply -f proxy/proxy-defaults.yamlproxydefaults.consul.hashicorp.com/global created

Review the Consul Proxy Defaults documentation to learn more.

Restart sidecar proxies

You need to restart your sidecar proxies to retrieve the updated proxy defaults configuration. To do so, redeploy your HashiCups services.

First, delete your existing HashiCups services.

$ kubectl delete --filename hashicups/v1/

Then, redeploy the HashiCups application.

$ kubectl apply --filename hashicups/v1/

Confirm that your proxy defaults updated your Envoy proxy’s configuration. You can view your Envoy sidecar proxy’s configuration by going to port 19000 from a service that has a sidecar proxy.

$ kubectl port-forward deploy/frontend 19000:19000

Open http://localhost:19000/config_dump in your browser to find the Envoy configuration. Search for 20200, the value you defined in the ProxyDefaults. You should find two different stanzas that reference this port. One of them is included next for reference.

{  "name": "envoy_prometheus_metrics_listener",  "address": {  "socket_address": {    "address": "0.0.0.0",    "port_value": 20200  }}

This confirms that Consul has configured Envoy to publish Prometheus metrics.

Deploy observability suite

The monitoring suite you deploy in this tutorial uses Grafana for visualization, Prometheus for metrics, and Loki for logs.

Deploy the observability suite. This adds and installs the respective Helm charts for Grafana, Prometheus, and Loki.

$ ./install-observability-suite.sh

Tip

The observability suite deployment will take about 1-2 minutes to complete.

Check the pods to make sure the monitoring suite is up and running successfully. Your complete output will include Grafana, Prometheus services, Loki services, and HashiCups microservices.

$ kubectl get pods --namespace defaultNAME                                             READY   STATUS    RESTARTS   AGEfrontend-5cd9997867-2kn2k                        2/2     Running   0          4m53sgrafana-679c5c7854-wdpsm                         1/1     Running   0          50sloki-0                                           1/1     Running   0          2m13sloki-promtail-597t7                              1/1     Running   0          2m13snginx-64f569ff6f-fkpln                           2/2     Running   0          4m53spayments-6888957c45-k28jm                        2/2     Running   0          4m52sproduct-api-55c547f984-gfssk                     2/2     Running   0          4m52sproduct-api-db-58cbc7f8bb-cv4cm                  2/2     Running   0          4m53sprometheus-kube-state-metrics-644f869f97-wv8g4   1/1     Running   0          2m37sprometheus-node-exporter-hq7bt                   1/1     Running   0          2m37sprometheus-pushgateway-67cf8576b7-k88c8          1/1     Running   0          2m37sprometheus-server-84dfcc8695-nhtlp               2/2     Running   0          2m37spublic-api-bd744c57b-jnkbf                       2/2     Running   0          4m52s

Generate traffic in the demo application

Since you deployed HashiCups, start generating some traffic so that you will have some metrics to look at in the Consul UI and Grafana.

Output the Consul API Gateway value to your terminal and paste it in your browser to access the HashiCups UI through the API Gateway. Click through some components within the HashiCups UI to generate traffic for visualization in the next steps.

$ echo $CONSUL_APIGW_ADDRhttp://a46cc0cc9dc6841a4b1a0fa1771973ca-1032165333.us-west-2.elb.amazonaws.com:8080

HashiCups UI

Explore the Consul UI metrics visualization

HCP Consul Dedicated does not currently support the Consul UI metrics visualization, however, all Consul metrics information is available for ingestion into your observability suite.

Explore the monitoring suite

Consul service mesh lets you expose metrics from the Envoy proxies and your services for third-party monitoring tools like Prometheus and Grafana to provide visualization for layer 7 application traffic.

Open a separate terminal window and expose the Grafana UI with kubectl port-forward using the grafana service name as the target.

$ kubectl port-forward svc/grafana --namespace default 3000:3000

Navigate to the custom HashiCups dashboard in Grafana at http://localhost:3000/d/hashicups/hashicups. Notice this dashboard includes both the Envoy proxy's metrics for each HashiCups services and the individual HashiCups service logs.

Grafana HashiCups Dashboard

Consul integrates with a wide variety of providers to enhance your complete service mesh observability experience. Check out this page to learn more about Consul and observability.

Clean up

Destroy the Terraform resources to clean up your environment. Confirm the destroy operation by inputting yes.

$ terraform destroy

Note: Due to race conditions with the various cloud resources created in this tutorial, it may be necessary to run the destroy operation twice to ensure all resources have been properly removed.

Next steps

In this tutorial, you enabled Consul monitoring features and deployed a monitoring suite into your Kubernetes cluster. After deploying the monitoring suite, you used the monitoring suite to explore the metrics and logs of the demo application, HashiCups, in your service mesh.

You have now completed the getting started collection!

Check out the following resources to learn more about running Consul on Kubernetes:

Previous

Enable external traffic ingress

 Next Collection

Get Started on VMs

Deployment