Collect resource utilization metrics

Understanding the resource utilization of an application is important, and Nomad supports reporting detailed statistics in many of its drivers. The main interface for outputting resource utilization is the alloc status command with the -stats flag.

This section will use the job named "docs" from the previous sections, but these operations and command largely apply to all jobs in Nomad.

As a reminder, here is the output of the run command from the previous example:

$ nomad job run docs.nomad.hcl==> Monitoring evaluation "42d788a3"    Evaluation triggered by job "docs"    Allocation "04d9627d" created: node "a1f934c9", group "example"    Allocation "e7b8d4f5" created: node "012ea79b", group "example"    Allocation "5cbf23a1" modified: node "1e1aa1e0", group "example"    Evaluation status changed: "pending" -> "complete"==> Evaluation "42d788a3" finished with status "complete"

To fetch the detailed usage statistics, issue the following command. Your allocation id will be different; replace 04d9627d with the allocation id from your running "docs" job:

$ nomad alloc status -stats 04d9627dID            = 04d9627dEval ID       = 42d788a3Name          = docs.example[2]Node ID       = a1f934c9Job ID        = docsClient Status = running Task "server" is "running"Task ResourcesCPU        Memory          Disk      Addresses75/100 MHz  784 KiB/10 MiB  300 MiB  http: 10.1.1.196:5678 Memory StatsCache   Max Usage  RSS      Swap56 KiB  1.3 MiB    784 KiB  0 B CPU StatsPercent  Throttled Periods  Throttled Time0.00%    0                  0 Recent Events:Time         Type      Description<timestamp>  Started   Task started by client<timestamp>  Received  Task received by client

The output indicates that the job is running near the limit of configured CPU but has plenty of memory headroom. You can use this information to alter the job's resources to better reflect its actual needs:

resources {  cpu    = 200  memory = 10}

Adjusting resources is very important for a variety of reasons:

• Ensuring your application does not get OOM killed if it hits its memory limit.
• Ensuring the application performs well by ensuring it has some CPU allowance.
• Optimizing cluster density by reserving what you need and not over-allocating.

While single point in time resource usage measurements are useful, it is often more useful to graph resource usage over time to better understand and estimate resource usage. Nomad supports outputting resource data to statsite and statsd and is the recommended way of monitoring resources. For more information about outputting telemetry, consult the Telemetry Guide.

For more advanced use cases, the resource usage data is also accessible via the client's HTTP API. Learn more about it in the allocation API documentation.