Skip to main content
Version: latest

bpfstats

The bpfstats Gadget provides CPU and memory usage for Gadgets and eBPF programs.

Getting started​

Running the gadget:

$ kubectl gadget run ghcr.io/inspektor-gadget/gadget/bpfstats:latest [flags]

Flags​

--all​

Collect statistics for all eBPF programs

Default value: "false"

Guide​

In order for this gadget to work, the server needs to be running with the --enable-bpfstats flag:

$ kubectl gadget deploy --enable-bpfstats

Then, let's run the gadget:

$ kubectl gadget run bpfstats:latest
NODENAME           GADGETID  GADGETNAME         GADGETIMAGE        PROGID PROGNAME               RUNTIME     RUNCOUNT    MAPMEMORY MAPC…

You can run the gadget with --help to get the meaning of the different columns:

  • gadgetID Unique ID assigned to each Gadget instance
  • gadgetImage Name of the Gadget image (like trace_open, trace_exec, etc.)
  • gadgetName Name of the Gadget instance
  • mapCount Number of maps used by the eBPF program or Gadget
  • mapMemory Memory used by maps in bytes
  • progID eBPF program ID assigned by the Linux kernel
  • progName Name of the eBPF program
  • runcount Number of times the eBPF program or Gadget has run
  • runtime Time that the eBPF program or Gadget has run in nanoseconds

It won't print anything as there are not gadgets running on the system. Let's create a couple of gadgets:

$ kubectl gadget run --name mytraceopen trace_open:latest --detach
$ kubectl gadget run --name mytraceexec trace_exec:latest --detach

The bpfstats will print the statistics for these two gadgets:

$ kubectl gadget run bpfstats:latest
NODENAME           GADGETID  GADGETNAME         GADGETIMAGE        PROGID PROGNAME               RUNTIME     RUNCOUNT    MAPMEMORY MAPC…
minikube           6bcfcca7a mytraceopen        trace_open:latest     0                        2537846        10798      1710240     6
minikube           040e3fc1d mytraceexec        trace_exec:latest     0                           8265           12      8742176     7

By default the gadget prints consolidated information for the Gadgets. In this case PROGID is 0 and PROGNAME is empty, as a Gadget can have multiple programs. It's possible to get per-program statistics by using the --all flag. In this mode, the statistics for eBPF programs not handled by Inspektor Gadget are shown as well.

$ kubectl gadget run bpfstats:latest --all
NODENAME           GADGETID  GADGETNAME         GADGETIMAGE        PROGID PROGNAME               RUNTIME     RUNCOUNT    MAPMEMORY MAPC…
minikube                                                                2 hid_tail_call                0            0         8512     1
minikube                                                               38 sched_process_e       89770034        15277      4391808     4
minikube                                                               39 task_newtask         172413186        91790      4391808     4
minikube                                                               40 sched_process_e      144593647        91635      4391808     4
minikube                                                               41 __x64_sys_renam        7185786         2173      1083840     4
minikube                                                               42 __x64_sys_renam       23368056         2184      6467272     7
minikube                                                               43 __x64_sys_renam        4649957         2069      1083840     4
minikube                                                               44 __x64_sys_renam       18957942         2072      6467272     7
minikube                                                               45 __x64_sys_renam              0            0      1083840     4
minikube                                                               46 __x64_sys_renam              0            0      6467272     7
minikube                                                               47 __x64_sys_unlin       37799602        14147      1083840     4
minikube                                                               48 __x64_sys_unlin      120719371        14370      6467272     7
minikube                                                               49 __x64_sys_unlin       17037065         9561      1083840     4
minikube                                                               50 __x64_sys_unlin       56861068         9561      6467272     7
minikube                                                             1244 ig_execve_e           15430709         3239       849792     2
minikube                                                             1245 ig_execve_x            1792662         3655       837504     1
minikube                                                             1246 ig_fa_pick_e           2899018         4581        21632     2
minikube                                                             1247 ig_fa_pick_x           4478788         4581      1194336     4
minikube                                                             1248 ig_sched_exec          2759885         2728       837504     1
minikube           6bcfcca7a mytraceopen        trace_open:latest  1259 ig_open_x                    0            0      1615904     4
minikube           6bcfcca7a mytraceopen        trace_open:latest  1260 ig_openat_x           14944051        83160      1615904     4
minikube           6bcfcca7a mytraceopen        trace_open:latest  1261 ig_open_e                    0            0      1098240     3
minikube           6bcfcca7a mytraceopen        trace_open:latest  1262 ig_openat_e           33097520        83005      1098240     3
minikube           040e3fc1d mytraceexec        trace_exec:latest  1263 ig_execve_x             200186          570      8647840     5
minikube           040e3fc1d mytraceexec        trace_exec:latest  1264 ig_execveat_x                0            0      8647840     5
minikube           040e3fc1d mytraceexec        trace_exec:latest  1265 security_bprm_c         225868          362      8372040     4
minikube           040e3fc1d mytraceexec        trace_exec:latest  1266 ig_execve_e             340822          518      8372040     4
minikube           040e3fc1d mytraceexec        trace_exec:latest  1267 ig_execveat_e            37108           52      8372040     4
minikube           040e3fc1d mytraceexec        trace_exec:latest  1268 ig_sched_exec           226692          362      8647840     1

Finally, clean the system:

$ kubectl gadget remove mytraceexec mytraceopen

Exporting metrics​

The bpfstats Gadget provides the following metrics for mapCount, mapMemory, runcount and runtime. To enable the metrics listener, check the Exporting Metrics documentation. To enable the collector for this gadget, run the following command:

WIP: Headless mode for kubectl gadget is under development
WIP: Headless mode for kubectl gadget is under development

Finally, stop metrics collection:

WIP: Headless mode for kubectl gadget is under development

Limitations​

Memory usage of maps​

The shown value for MapMemory is read from /proc/<pid>/fdinfo/<map_id>. This is the maximum size the map can have, but it doesn't necessarily reflect its current memory allocation. Additionally, maps can be used by more than one program and would account towards the MapMemory of all those programs.

Also note:

  • BPF_MAP_TYPE_PERF_EVENT_ARRAY: value_size is not counting the ring buffers, but only their file descriptors (i.e. sizeof(int) = 4 bytes)
  • BPF_MAP_TYPE_{HASH,ARRAY}_OF_MAPS: value_size is not counting the inner maps, but only their file descriptors (i.e. sizeof(int) = 4 bytes)

bpfstats not disabled for Kernels < 5.8​

The bpfstats Gadgets enables collection of eBPF program stats by trying to use BPF_ENABLE_STATS first (which requires Linux >= 5.8). If that fails, it will fall back to trying to enable via sysctl (/proc/sys/kernel/bpf_stats_enabled), on this case, the collection of stats will remain active even after stopping Inspektor Gadget.

It is a temporary limitation that we're working to fix, in the meanwhile you can use sysctl kernel.bpf_stats_enabled=0 to disable it.