News HashiCorp to join IBM to accelerate multi-cloud automation. Read the blog →
Vault
Archive

Monitoring Vault with Datadog

  • 6min
  • |
  • Enterprise
  • Vault

Challenge

Monitoring is a critical part of administration of any software system. Proactively collecting, visualizing, and analyzing data on Vault is no different, as insight into the details of how Vault instance(s) are important to decision making, quality assurance and troubleshooting.

Solution

In order to get aggregated data about the instance(s) of Vault, you can use Datadog. It is a mature monitoring solution that will report common resource metrics like CPU, memory and network metrics out of the box.

This tutorial covers set-up and configuration of Datadog Agent to monitor an instance of Vault Enterprise. Then you will look over the metrics available, and finally clean up the Datadog Agent and the local Vault installation.

Prerequisites

To enable the Datadog Agent to gather metrics from Vault Enterprise, you will need to have:

  • Free Tier Datadog account. Sign up for a free account at Datadog Pricing.
  • A Vault Enterprise environment. Refer to the Getting Started tutorial to install Vault.
  • Mac Workstation with MacOS 10.12 or higher, though much of this will be the same for other platforms.

Set up the Datadog agent locally

  1. You will need an API key from Datadog. From the Datadog dashboard, select your user name at the bottom of the left navigation. Datadog UI

  2. Select Organizational Settings, and then API Keys which lists existing API keys.

  3. Select an API key created for your username. Datadog API Key

  4. Copy the API Key value, and then store it as a DATADOG_API_KEY environment variable. 

    $ export DATADOG_API_KEY=<DATADOG_API_KEY>

  5. Open a terminal and run the following command to install Datadog Agent. 

    $ DD_AGENT_MAJOR_VERSION=7 DD_API_KEY=$DATADOG_API_KEY \    DD_SITE="datadoghq.com" \    bash -c "$(curl -L https://s3.amazonaws.com/dd-agent/scripts/install_mac_os.sh)"

    Output: 

    ...Your Agent is running properly. It will continue to run in the background and submit metrics to Datadog. You can check the agent status using the "datadog-agent status" command or by opening the webui using the "datadog-agent launch-gui" command. If you ever want to stop the Agent, please use the Datadog Agent App or the launchctl command. It will start automatically at login.

  6. Verify that the agent is running. 

    $ datadog-agent status

    Example output: 

    2022-07-29 15:59:23 PDT | CORE | WARN | (pkg/util/log/log.go:591 in func1) | Deactivating Autoconfig will disable most components. It's recommended to use autoconfig_exclude_features and autoconfig_include_features to activate/deactivate features selectively2022-07-29 15:59:23 PDT | CORE | INFO | (cmd/system-probe/config/config.go:118 in Merge) | no config exists at system-probe.yaml, ignoring...2022-07-29 15:59:23 PDT | CORE | ERROR | (cmd/system-probe/config/config.go:179 in load) | Could not parse system_probe_config.sysprobe_socket: system-probe unsupported2022-07-29 15:59:23 PDT | CORE | INFO | (cmd/agent/app/status.go:125 in requestStatus) | Getting the status from the agent.2022-07-29 15:59:28 PDT | CORE | INFO | (cmd/agent/app/status.go:163 in requestStatus) |===============Agent (v7.38.0)===============  Status date: 2022-07-29 15:59:23.776 PDT / 2022-07-29 22:59:23.776 UTC (1659135563776)  Agent start: 2022-07-29 15:59:02.482 PDT / 2022-07-29 22:59:02.482 UTC (1659135542482)  Pid: 34339  Go Version: go1.17.11  Python Version: 3.8.13  Build arch: amd64  Agent flavor: agent  Check Runners: 6  Log Level: info  Paths  =====    Config File: /opt/datadog-agent/etc/datadog.yaml    conf.d: /opt/datadog-agent/etc/conf.d    checks.d: /opt/datadog-agent/etc/checks.d    ...snip...

    If you encountered an error, try executing datadog-agent stop and then datadog-agent run again.

Set up the Vault integration

In order for Vault to send metrics, you need to set up access for the agent to connect to Datadog. For this tutorial, use unauthenticated access.

  1. Open another terminal and start a Vault dev server with root as the root token.

    $ vault server -dev -dev-root-token-id root

  2. Navigate to /opt/datadog-agent/etc/conf.d/vault.d.

    $ cd /opt/datadog-agent/etc/conf.d/vault.d

  3. In the /vault.d directory, make a copy of conf.yaml.example.

    $ cp conf.yaml.example conf.yaml

  4. Open the conf.yaml file with an editor of your choice. Find the instances: section, and notice that the api_url parameter points to the Vault address to pull metrics from which is set to the locally running Vault (http://localhost:8200/v1).

    Locate the no_token parameter and set it to true for the convenience of this tutorial.

    conf.yaml

    ...snip... init_config: instances:   ## @param api_url - string - required   ## URL of the Vault to query. - api_url: http://localhost:8200/v1    ## @param no_token - boolean - optional - default: false   ## Attempt metric collection without a token.   no_token: true

  5. Restart the Datadog Agent to apply the configuration changes.

    Stop the Datadog Agent. 

    $ datadog-agent stopAgent successfully stopped

    Run the agent again. 

    $ datadog-agent run2022-07-21 10:20:15 CDT | CORE | INFO | (pkg/util/log/log.go:571 in func1) | runtime: final GOMAXPROCS value is: 102022-07-21 10:20:15 CDT | CORE | INFO | (pkg/util/log/log.go:571 in func1) | Features detected from environment:2022-07-21 10:20:15 CDT | CORE | INFO | (cmd/agent/app/run.go:252 in StartAgent) | Starting Datadog Agent v7.37.12022-07-21 10:20:16 CDT | CORE | INFO | (cmd/agent/app/run.go:310 in StartAgent)...

Monitor the Vault metrics

Now that the agent is installed and running, you need to validate that the agent is correctly sending data to Datadog. Metrics about both your workstation and your Vault Enterprise Instance should be streaming to Datadog.

  1. In your Datadog dashboard, select Metrics > Explorer. Metrics Explorer

  2. The Explorer page shows the default metrics of system.cpu.user. Metrics->Explorer

  3. With Metrics selected, start typing in vault. and you will see a number of available Vault metrics. Metrics Explorer

    Explorer the available metrics collected by the Datadog.

Clean up

When you are done exploring, clean up both the Datadog Agent and the Vault environment.

  1. Find your platform Uninstall the Agent for instructions on how to uninstall the agent for your platform of choice.

  2. If you are running Vault locally in dev mode, stop the Vault dev server by pressing Ctrl+C where the server is running. Or, execute the following command.

    $ pgrep -f vault | xargs kill

References

Collection Overview

Archive

 Next

Monitor telemetry with Prometheus & Grafana