trace_gpu_starvation

The trace_gpu_starvation gadget detects GPU starvation: a process holds GPU memory, but its GPU sits idle while one of its threads keeps burning CPU. This is the signature of a pipeline that is CPU-bound in the code path that is supposed to be feeding the GPU (slow preprocessing, a Python for loop between kernel launches, a blocking dependency between threads, …). The gadget attributes the wasted CPU to the exact thread and call stack keeping the GPU starved.

It joins two signals:

• the GPU metrics published by the gpu-ebpf-bridge daemon — per device, whether the GPU is currently busy (SM utilization from NVML hardware counters); and per process, the VRAM it holds and which device that memory lives on — and
• a finish_task_switch scheduler kprobe that measures each thread's on-CPU time and captures its user stack.

Because GPU idleness is read from NVML hardware counters rather than inferred from CPU behaviour, the gadget is not fooled by asynchronous CUDA: a thread that launches a kernel and immediately returns to CPU work is only flagged if the GPU is genuinely idle.

Requirements

• Minimum Kernel Version: 6.2 (for bpf_ktime_get_boot_ns in tracing programs and the user-stack map helpers).
• The gpu-ebpf-bridge daemon must be running on the host. It publishes the bpffs-pinned gpu_per_pid and gpu_meta maps that this gadget reads. Without it, the gadget starts but never emits events.
• finish_task_switch must be attachable as a kprobe. On most kernels the symbol is emitted with a compiler suffix (e.g. finish_task_switch.isra.0); the gadget's WASM module resolves the right name at startup via /proc/kallsyms.

Getting started

kubectl gadget

$ kubectl gadget run ghcr.io/inspektor-gadget/gadget/trace_gpu_starvation:latest [flags]

ig

$ sudo ig run ghcr.io/inspektor-gadget/gadget/trace_gpu_starvation:latest [flags]

Guide

With the bridge running and a GPU workload that stalls its GPU between frames:

ig

$ sudo ig run trace_gpu_starvation:latest --collect-ustack
RUNTIME.CONTAINERNAME  COMM     PID      TID      CPU_TIME_NS       IDLE_NS  HIT_COUNT
inference-server-2     python  12346    12360      412583910     317884012        188
inference-server-2     python  12346    12360      398117204     684551221        174
^C

Each row means: thread 12360 of process 12346 spent CPU_TIME_NS on-CPU during the last one-second window while the process's GPU had been idle for IDLE_NS. HIT_COUNT is the number of context-switch samples that contributed to the window.

Pass --collect-ustack to capture the user stack of the starving thread, and add --collect-otel-stack --symbolizers=otel-ebpf-profiler (with hostPid=true) to resolve Python source lines. The raw kernel stack is available with --collect-kstack but is rarely useful (it is mostly scheduler frames).

Parameters

--min-idle-ms

Minimum GPU idle duration (ms) before a thread's CPU time counts as starvation. Must be at least 2× the bridge poll interval (see Limitations).

Default value: "1000"

--min-gpu-mem-bytes

Only consider processes holding at least this many bytes of GPU memory. Filters out processes with trivial or leftover allocations.

Default value: "1"

--stale-threshold-ms

Ignore bridge data older than this (ms), so a crashed or paused bridge does not produce stale verdicts. Should be about 3× the bridge poll interval.

Default value: "300"

--collect-kstack

Collect kernel stack traces. Mostly scheduler frames; the user stack is the useful one.

Default value: "false"

--collect-ustack

Collect the user stack of the starving thread.

Default value: "false"

Architecture

The gpu-ebpf-bridge daemon polls NVML and publishes two things the gadget joins:

• per device (gpu_device map): the current SM utilization and the wall-clock timestamp of that sample, and
• per process (gpu_per_pid map): the GPU memory held and which device that memory primarily lives on (gpu_device_primary).

Device-level SM utilization is used deliberately in place of per-process utilization. NVML's per-process activity API only samples at ~1 s and intermittently drops a busy process entirely, so it cannot resolve sub-second GPU-idle stalls. "Is the GPU idle" is really a device property; the gadget attributes it to a process via the device that process's GPU memory lives on.

To keep the "GPU is active" signal fresh, the gadget maintains a device-activity latch in a separate sched_switch tracepoint program: for each device it records the most recent time the device was seen busy (SM utilization > 0). Running on every context switch system-wide, the latch stays current even while a GPU-holding thread runs a long CPU slice without ever being scheduled out. The GPU-idle duration for a device is then simply now - last_active[device]. (The latch is a separate program from the finish_task_switch kprobe below so that its per-device loop gets its own verifier instruction budget.)

The gadget hooks finish_task_switch, which runs just after a context switch in the context of the incoming task and receives the outgoing task as its argument:

• schedule-in: for a thread whose process holds GPU memory (with fresh bridge data) the gadget records the start time and captures the user stack. It captures at schedule-in because Inspektor Gadget's OTel eBPF-profiler symbolizes the current task, which at finish_task_switch is the incoming thread — so the unwind is valid.
• schedule-out: when that same thread is switched off the CPU, the gadget computes the on-CPU slice and, if its device has been idle for at least --min-idle-ms, accumulates the slice into a per-thread window and emits at most one event per second per thread.

The device sample timestamp is NVML's CLOCK_REALTIME value; it is converted to the kernel's CLOCK_BOOTTIME using clock_offset_ns (published by the bridge and recalibrated every poll tick), so it can be compared against bpf_ktime_get_boot_ns() inside eBPF.

Why a kprobe (not the sched_switch tracepoint)

OTel-based user-stack symbolization is reached through a bpf_tail_call into a BPF_PROG_TYPE_KPROBE program, and bpf_tail_call requires the caller and callee to share the same program type. Only kprobe-family programs can therefore obtain OTel stacks; tracepoint/tp_btf programs cannot. finish_task_switch is a plain (non-NOKPROBE) kernel function, so a kprobe on it gives both scheduler visibility and OTel-capable stacks. (The device-activity latch does not need stacks, so it uses a lightweight sched_switch tracepoint instead.)

Limitations

1. Minimum detectable stall is bounded by the device SM sampling resolution (~167 ms) and the bridge poll interval. Fine-grained pipelines (e.g. 10 ms CPU + 10 ms GPU per frame) are not detectable — the GPU never stays idle long enough for the NVML device SM counters to register a zero, regardless of --min-idle-ms. This is a fundamental NVML hardware limit.

2. --min-idle-ms must be ≥ 2× the bridge poll interval, and large enough to clear the device SM sampling granularity. With the default 100 ms bridge poll the hard lower bound is 200 ms. The default is 1000 ms, high enough to suppress sampling noise while still catching real multi-hundred-millisecond stalls. Lower it (e.g. --min-idle-ms 200) only if the bridge polls fast and you accept the extra false positives.

3. Python stacks require --collect-ustack --collect-otel-stack --symbolizers=otel-ebpf-profiler and hostPid=true. The OTel eBPF profiler resolves Python source lines using DWARF and interpreter introspection (no frame pointers required). Without these flags, stacks show CPython C internals. C extensions (numpy, torch) are still recognisable without OTel.

4. Blocking-dependency scenarios produce two stacks. When Thread A holds the GPU but waits on Thread B to prepare data, both threads generate events: Thread A shows the blocking call (futex, queue.get), Thread B shows the actual CPU work. Together they tell the complete story; in a flamegraph Thread B dominates (more cpu_time_ns weight), correctly identifying the bottleneck.

5. Bridge required. The gadget emits nothing without the gpu-ebpf-bridge daemon running and its maps pinned under /sys/fs/bpf/.

6. Host processes need --host. As with other Inspektor Gadget gadgets, non-containerized (host) processes are only traced when --host is passed.