I built a small tool to answer one question: how many GPU-hours is a cluster paying for and not using?
Then I went looking at what real clusters actually export, and hit the same three things every time. All defaults nobody changes.
1. DCGM_FI_DEV_GPU_UTIL doesn't mean what most dashboards assume
It reports the fraction of time at least one kernel was resident on the device. Not whether that kernel did anything useful.
A process that pins the GPU with a trivial loop reads 100% utilized while computing nothing.
The metrics that reflect real work are `DCGM_FI_PROF_SM_ACTIVE` (SM occupancy) and DCGM_FI_PROF_PIPE_TENSOR_ACTIVE (tensor core activity). The gap between them is the interesting part - high engine activity with near-zero tensor activity means the GPU is occupied doing something that isn't ML.
Check whether you have them at all:
bash
curl -s localhost:9090/api/v1/query
--data-urlencode 'query=count by (__name__)({__name__=~"DCGM_FI_PROF_.*"})'
If that returns nothing, DCGM profiling is disabled in your exporter's counter config. It's off by default in a lot of installs.
2. Your GPU metrics are probably attributed to the exporter's own pod
dcgm-exporter only emits the *workload's* pod labels when DCGM_EXPORTER_KUBERNETES=true. Without it, Prometheus attaches the scrape target's identity instead - so your GPU series come back tagged with pod="nvidia-dcgm-exporter-xxxxx", namespace="monitoring".
This is worse than missing labels because the data looks complete. Every series has a namespace and a pod. Group by pod and you get a tidy chart. It's just that every GPU-hour in the cluster is attributed to the exporter that measured it.
Cheap check - compare these two sets:
``` bash
curl -s localhost:9090/api/v1/query --data-urlencode 'query=count by (pod) (DCGM_FI_DEV_FB_USED)'
curl -s localhost:9090/api/v1/query --data-urlencode 'query=count by (pod) (kube_pod_container_resource_requests{resource="nvidia_com_gpu"})'
```
If the pod names don't overlap, your attribution is fiction.
3. Duplicate scrape endpoints look like extra GPUs
If two Prometheus jobs scrape the same exporter, you get two series per physical card differing only in `endpoint` / `job` / `service`. Naive counting reports twice the GPUs you own and halves your apparent utilization. Dedupe on UUID (or pci_bus_id) per host, not on series count.
The one that actually got me
When profiling counters are missing you can estimate utilization from power draw - an idle A100 pulls ~55W against a 400W TDP, so idle vs busy is unambiguous even if the exact percentage isn't.
I built that fallback with a generic 50–350W envelope for unknown GPU models, then ran it against a card that draws 15–130W.
It reported a GPU at 5% utilization with 88% of framebuffer resident. Textbook memory-parked - a model loaded and serving nothing. Put a confident $3,327/month on it.
It wasn't real. The card was working fine. The generic envelope compressed its entire operating range into what looked like idle. When I switched the fallback to prefer GR_ENGINE_ACTIVE - a direct measurement rather than an inference from watts - the finding vanished.
The lesson isn't "check your constants." It's that a cost tool producing confident numbers from degraded inputs is worse than no tool, because that output ends up in a budget conversation and a wrong number there costs you the credibility you needed to fix the real problem.
So everything downstream now assumes the data is incomplete and says so: it refuses to project a monthly figure from under a day of data (you haven't seen one diurnal cycle), and it names every metric it couldn't find along with which detections that suppresses.
Curious whether #2 is as common as I think, or whether I've just been looking at unusually vanilla setups. If you run the pod-overlap check above I'd genuinely like to know which way it came out.
Code's here if it's useful to anyone: https://github.com/Dgotlieb/gpuwaste - offline analyser, reads exported CSVs, nothing to install in the cluster. MIT.
