Skip to main content
Version: v0.50.0

bpfstats

The bpfstats Gadget provides CPU and memory usage for Gadgets and eBPF programs.

Getting started​

Running the gadget:

$ kubectl gadget run ghcr.io/inspektor-gadget/gadget/bpfstats:v0.50.0 [flags]

Flags​

--gadgets-only​

Show statistics for only gadget-related eBPF programs

Default value: "false"

Guide​

Start the server part of IG:

$ kubectl gadget deploy

Then, let's run the gadget:

$ kubectl gadget run bpfstats:v0.50.0
NODENAME           GADGETID  GADGETNAME         GADGETIMAGE        PROGID PROGNAME               RUNTIME     RUNCOUNT  %CPU   %RELCPU   TIME+        MAPMEMORY MAPC…
minikube                                                                2 hid_tail_call                0            0  0.00   0.00      0:00.000         8512     1
minikube                                                               38 sched_process_e       89770034        15277  0.05   0.01      0:00.089      4391808     4
minikube                                                               39 task_newtask         172413186        91790  0.10   0.02      0:00.172      4391808     4
minikube                                                               40 sched_process_e      144593647        91635  0.08   0.02      0:00.144      4391808     4
minikube                                                               41 __x64_sys_renam        7185786         2173  0.01   0.00      0:00.007      1083840     4

You can run the gadget with --help to get the meaning of the different columns:

  • comms List of processes using the eBPF program
  • gadgetID Unique ID assigned to each Gadget instance
  • gadgetImage Name of the Gadget image (like trace_open, trace_exec, etc.)
  • gadgetName Name of the Gadget instance
  • mapCount Number of maps used by the eBPF program or Gadget
  • mapMemory Memory used by maps in bytes
  • pids List of PIDs using the eBPF program
  • progID eBPF program ID assigned by the Linux kernel
  • progName Name of the eBPF program
  • progType Type of the eBPF program
  • cpuTimeStr Total CPU time formatted as duration
  • cpuUsage CPU usage percentage of the eBPF program
  • cpuUsageRelative CPU usage relative to the number of available CPUs
  • runcount Number of times the eBPF program or Gadget has run
  • runtime Time that the eBPF program or Gadget has run in nanoseconds

By default, it will print statistics for all eBPF programs running on the system. You can restrict the view to only Inspektor Gadget programs by using the --gadgets-only flag. Let's create a couple of gadgets:

$ kubectl gadget run --name mytraceopen trace_open:v0.50.0 --detach
$ kubectl gadget run --name mytraceexec trace_exec:v0.50.0 --detach

The bpfstats will print the statistics for these two gadgets:

$ kubectl gadget run bpfstats:v0.50.0 --gadgets-only
NODENAME           GADGETID  GADGETNAME         GADGETIMAGE        PROGID PROGNAME               RUNTIME     RUNCOUNT  %CPU   %RELCPU   TIME+        MAPMEMORY MAPC…
minikube           6bcfcca7a mytraceopen        trace_open:v0.50.0     0                        2537846        10798  0.12   0.03      0:02.537     1710240   6
minikube           040e3fc1d mytraceexec        trace_exec:v0.50.0     0                           8265           12  0.00   0.00      0:00.008     8742176   7

In this mode, the gadget shows consolidated information for the running Gadgets. Here, PROGID is 0 and PROGNAME is empty, as a Gadget can have multiple programs. Let's view the per-program statistics by removing the --gadgets-only flag.

$ kubectl gadget run bpfstats:v0.50.0
NODENAME           GADGETID  GADGETNAME         GADGETIMAGE        PROGID PROGNAME               RUNTIME     RUNCOUNT  %CPU   %RELCPU   TIME+        MAPMEMORY MAPC…
minikube                                                                2 hid_tail_call                0            0  0.00   0.00      0:00.000         8512     1
minikube                                                               38 sched_process_e       89770034        15277  0.05   0.01      0:00.089      4391808     4
minikube                                                               39 task_newtask         172413186        91790  0.10   0.02      0:00.172      4391808     4
minikube                                                               40 sched_process_e      144593647        91635  0.08   0.02      0:00.144      4391808     4
minikube                                                               41 __x64_sys_renam        7185786         2173  0.01   0.00      0:00.007      1083840     4
minikube                                                               42 __x64_sys_renam       23368056         2184  0.02   0.00      0:00.023      6467272     7
minikube                                                               43 __x64_sys_renam        4649957         2069  0.01   0.00      0:00.004      1083840     4
minikube                                                               44 __x64_sys_renam       18957942         2072  0.02   0.00      0:00.018      6467272     7
minikube                                                               45 __x64_sys_renam              0            0  0.00   0.00      0:00.000      1083840     4
minikube                                                               46 __x64_sys_renam              0            0  0.00   0.00      0:00.000      6467272     7
minikube                                                               47 __x64_sys_unlin       37799602        14147  0.03   0.01      0:00.037      1083840     4
minikube                                                               48 __x64_sys_unlin      120719371        14370  0.07   0.02      0:00.120      6467272     7
minikube                                                               49 __x64_sys_unlin       17037065         9561  0.02   0.00      0:00.017      1083840     4
minikube                                                               50 __x64_sys_unlin       56861068         9561  0.04   0.01      0:00.056      6467272     7
minikube                                                             1244 ig_execve_e           15430709         3239  0.02   0.00      0:00.015       849792     2
minikube                                                             1245 ig_execve_x            1792662         3655  0.01   0.00      0:00.001       837504     1
minikube                                                             1246 ig_fa_pick_e           2899018         4581  0.01   0.00      0:00.002        21632     2
minikube                                                             1247 ig_fa_pick_x           4478788         4581  0.01   0.00      0:00.004      1194336     4
minikube                                                             1248 ig_sched_exec          2759885         2728  0.01   0.00      0:00.002       837504     1
minikube           6bcfcca7a mytraceopen        trace_open:v0.50.0  1259 ig_open_x                    0            0  0.00   0.00      0:00.000      1615904     4
minikube           6bcfcca7a mytraceopen        trace_open:v0.50.0  1260 ig_openat_x           14944051        83160  0.02   0.00      0:00.014      1615904     4
minikube           6bcfcca7a mytraceopen        trace_open:v0.50.0  1261 ig_open_e                    0            0  0.00   0.00      0:00.000      1098240     3
minikube           6bcfcca7a mytraceopen        trace_open:v0.50.0  1262 ig_openat_e           33097520        83005  0.03   0.01      0:00.033      1098240     3
minikube           040e3fc1d mytraceexec        trace_exec:v0.50.0  1263 ig_execve_x             200186          570  0.00   0.00      0:00.000      8647840     5
minikube           040e3fc1d mytraceexec        trace_exec:v0.50.0  1264 ig_execveat_x                0            0  0.00   0.00      0:00.000      8647840     5
minikube           040e3fc1d mytraceexec        trace_exec:v0.50.0  1265 security_bprm_c         225868          362  0.00   0.00      0:00.000      8372040     4
minikube           040e3fc1d mytraceexec        trace_exec:v0.50.0  1266 ig_execve_e             340822          518  0.00   0.00      0:00.000      8372040     4
minikube           040e3fc1d mytraceexec        trace_exec:v0.50.0  1267 ig_execveat_e            37108           52  0.00   0.00      0:00.000      8372040     4
minikube           040e3fc1d mytraceexec        trace_exec:v0.50.0  1268 ig_sched_exec           226692          362  0.00   0.00      0:00.000      8647840     1

Finally, clean the system:

$ kubectl gadget remove mytraceexec mytraceopen

Exporting metrics​

The bpfstats Gadget provides the following metrics for mapCount, mapMemory, runcount and runtime. To enable the metrics listener, check the Exporting Metrics documentation. To enable the collector for this gadget, run the following command:

WIP: Headless mode for kubectl gadget is under development
WIP: Headless mode for kubectl gadget is under development

Finally, stop metrics collection:

WIP: Headless mode for kubectl gadget is under development

Limitations​

Memory usage of maps​

The shown value for MapMemory is read from /proc/<pid>/fdinfo/<map_id>. This is the maximum size the map can have, but it doesn't necessarily reflect its current memory allocation. Additionally, maps can be used by more than one program and would account towards the MapMemory of all those programs.

Also note:

  • BPF_MAP_TYPE_PERF_EVENT_ARRAY: value_size is not counting the ring buffers, but only their file descriptors (i.e. sizeof(int) = 4 bytes)
  • BPF_MAP_TYPE_{HASH,ARRAY}_OF_MAPS: value_size is not counting the inner maps, but only their file descriptors (i.e. sizeof(int) = 4 bytes)