Skip to main content
Version: latest

Running Gadgets

Once you have installed Inspektor Gadget either on Kubernetes or on the local host, you can run Gadgets with the run command:

$ kubectl gadget run trace_exec:latest
K8S.NAMESPACE    K8S.PODNAME      K8S.CONTAINERNA… COMM           PID      TID PCOMM        PPID ARGS     K8S.NODE E… TIMESTAMP         USER     LOGINUS… GROUP
default          mypod2           mypod2           ping        425494   425494 sh         343042 /bin/pi… minikube    2024-07-31T18:08… root     uid:429… root
default          mypod2           mypod2           wget        425719   425719 sh         343042 /bin/wg… minikube    2024-07-31T18:09… root     uid:429… root
^C

$ kubectl gadget run snapshot_process:latest -A
K8S.NAMESPACE            K8S.PODNAME              K8S.CONTAINERNAME        COMM                    PID           TID           UID           GID         PPID K8S.NODE
kube-system              kube-control…er-minikube kube-controller-manager  kube-control…        336554        336554             0             0       336505 minikube
kube-system              kube-apiserver-minikube  kube-apiserver           kube-apiserv…        336561        336561             0             0       336532 minikube
kube-system              etcd-minikube            etcd                     etcd                 336644        336644             0             0       336577 minikube
kube-system              kube-scheduler-minikube  kube-scheduler           kube-schedul…        336737        336737             0             0       336713 minikube
kube-system              kube-proxy-vgp6m         kube-proxy               kube-proxy           337401        337401             0             0       337339 minikube
kube-system              coredns-7db6d8ff4d-xskcm coredns                  coredns              337598        337598         65532         65532       337566 minikube
kube-system              storage-provisioner      storage-provisioner      storage-prov…        338431        338431             0             0       338410 minikube
gadget                   gadget-ptnct             gadget                   gadgettracer…        341345        341345             0             0       341324 minikube
default                  mypod2                   mypod2                   sh                   343042        343042             0             0       343021 minikube
^C

The run command runs a gadget from an OCI image. The run command will use the following defaults to refer an OCI image:

  • ghcr.io as the registry
  • inspektor-gadget/gadget as the repository prefix
  • latest as the tag

You can find a growing collection of Gadgets on Artifact HUB. This includes both in-tree Gadgets (hosted in this git repository in the /gadgets directory and third-party Gadgets).

Event Filtering​

Inspektor Gadget supports filtering the events that are gathered by a Gadget. It's useful to (1) reduce the performance overhead of the tracing and (2) remove the noise to focus on relevant information.

The filtering is supported in two different ways:

In Kernel Filtering​

Inspektor Gadget supports to efficiently filter events directly on eBPF for some common fields:

  • --node string, show only data from pods running in that node
  • -n string, --namespace string, show data from pods in that namespace
  • -A, --all-namespaces, show data from pods in all namespaces
  • -p string, --podname string, show only data from pods with that name
  • -c string, --containername string, show only data from containers with that name
  • -l string, --selector string: show only data that matches the given label or selector. Only = is currently supported (e.g. key1=value1,key2=value2).

We can use one or more of these parameters to choose which pods or containers will be inspected by our gadgets. For example:

$ kubectl gadget run trace_exec:latest -n demo -l app=myapp

Runs the trace_exec gadget filtering events generated by for all pods in the demo namespace that have the app=myapp label.

Filtering in User Space​

The --filter flag allows you to filter events based on specific field values provided by the gadget. This is particularly useful for narrowing down the output to entries that meet certain criteria.

The filter syntax supports the following operations:

- `field==value`: Matches if the content of `field` equals exactly `value`.
- `field!=value`: Matches if the content of `field` does not equal exactly `value`.
- `field>=value`: Matches if the content of `field` is greater than or equal to `value`.
- `field>value`: Matches if the content of `field` is greater than `value`.
- `field<=value`: Matches if the content of `field` is less than or equal to `value`.
- `field<value`: Matches if the content of `field` is less than `value`.
- `field~value`: Matches if the content of `field` matches the regular expression `value`. See [RE2 Syntax](https://github.com/google/re2/wiki/Syntax) for more details.

Examples​

Equal filter: To filter events where the comm field equals cat, use:

--filter comm==cat

This filters for events related to the cat command.

Not equal filter: To filter out events where the uid is not 0 (root user), use:

--filter uid!=0

Regular expression filter: To match a regular expression in a specific column, use:

--filter comm~'^c.*'

This filters for commands starting with the letter "c".

Multiple filters: You can combine multiple filters to narrow down results further. For example, to find processes executed by the root user (uid==0) where the command is cat, use:

--filter comm==cat,uid==0

Output Format​

The -o or --output flag lets us decide the output format. The default columns output shows some of the information gathered, arranged in text columns on the console.

This can be overridden with:

  • json
  • jsonpretty
  • yaml
  • columns

JSON Output​

Passing -o json will print all the information gathered in JSON format. Each entry is printed on a single line, so the output can be easily parsed line by line.

$ kubectl gadget run trace_tcp:latest -A -o json
{"comm":"wget","dst":{"addr":"1.1.1.1","port":80,"proto":6,"version":4},"gid":0,"k8s":{"containerName":"mypod2","hostnetwork":false,"namespace":"default","node":"minikube","podName":"mypod2"},"mntns_id":4026536533,"netns_id":4026535383,"pid":446916,"runtime":{"containerId":"53225e24386f30ac68a58cd57c1689590875e4904f9449d0c19835f68da0c0b6","containerImageDigest":"","containerImageName":"","containerName":"","runtimeName":""},"src":{"addr":"10.244.0.8","port":41464,"proto":6,"version":4},"tid":446916,"timestamp":"2024-07-31T18:38:31.977700392Z","timestamp_raw":1722451111977700392,"type":"connect","type_raw":0,"uid":0}

JSON Pretty Output​

Passing -o jsonpretty will print all the information gathered in JSON format but with indentation making it easier to read.

$ kubectl gadget run trace_tcp:latest -o jsonpretty
{
  "comm": "wget",
  "dst": {
    "addr": "1.1.1.1",
    "port": 80,
    "proto": 6,
    "version": 4
  },
  "gid": 0,
  "k8s": {
    "containerName": "mypod2",
    "hostnetwork": false,
    "namespace": "default",
    "node": "minikube",
    "podName": "mypod2"
  },
  "mntns_id": 4026536533,
  "netns_id": 4026535383,
  "pid": 450698,
  "runtime": {
    "containerId": "53225e24386f30ac68a58cd57c1689590875e4904f9449d0c19835f68da0c0b6",
    "containerImageDigest": "",
    "containerImageName": "",
    "containerName": "",
    "runtimeName": ""
  },
  "src": {
    "addr": "10.244.0.8",
    "port": 51080,
    "proto": 6,
    "version": 4
  },
  "tid": 450698,
  "timestamp": "2024-07-31T18:42:09.094958020Z",
  "timestamp_raw": 1722451329094958020,
  "type": "connect",
  "type_raw": 0,
  "uid": 0
}

YAML Output​

Passing -o yaml will print all the information gathered in YAML format. Each entry is preceded by the end of directives markers (---).

$ kubectl gadget run trace_tcp:latest -o yaml
---
comm: wget
dst:
  addr: 1.1.1.1
  port: 80
  proto: 6
  version: 4
gid: 0
k8s:
  containerName: mypod2
  hostnetwork: false
  namespace: default
  node: minikube
  podName: mypod2
mntns_id: 4026536533
netns_id: 4026535383
pid: 452381
runtime:
  containerId: 53225e24386f30ac68a58cd57c1689590875e4904f9449d0c19835f68da0c0b6
  containerImageDigest: ""
  containerImageName: ""
  containerName: ""
  runtimeName: ""
src:
  addr: 10.244.0.8
  port: 55282
  proto: 6
  version: 4
tid: 452381
timestamp: "2024-07-31T18:43:51.457963290Z"
timestamp_raw: 1722451431457963290
type: connect
type_raw: 0
uid: 0
---

Selecting Specific Fields​

The --fields flag allows to choose which columns to print. You can use run [gadget] -h to see which columns are available:

$ kubectl gadget run trace_exec:latest -h
...
  --fields string               Available data sources / fields
                                  "exec" (data source):
                                    args
                                    args_count
                                    args_size
                                    comm
                                      Process name
                                    cwd
                                      The current working directory of the process (require --paths flag)
                                    error
                                    error_raw
                                    gid
                                      Group ID
                                    group
                                    k8s.containerName
                                    k8s.hostnetwork
                                    k8s.namespace
                                    k8s.node
                                    k8s.podName
                                    loginuid
                                    loginuser
                                    mntns_id
                                      Mount namespace inode id
                                    pcomm
                                      The process name of the parent process
                                    pid
                                      Process ID
                                    ppid
                                    pupper_layer
                                      Whether the executable's parent in the process hierarchy is in the upper layer of the overlay filesystem. Only initialized when the execution succeeded.
                                    runtime.containerId
                                    runtime.containerImageDigest
                                    runtime.containerImageName
                                    runtime.containerName
                                    runtime.runtimeName
                                    sessionid
                                    tid
                                      Thread ID
                                    timestamp
                                    timestamp_raw
                                    uid
                                      User ID
                                    upper_layer
                                      Whether the executable is in the upper layer of the overlay filesystem
                                    user

For example, we can only show the containername, comm, pid and ppid of new executed processes:

$ kubectl gadget run trace_exec:latest --fields=k8s.containername,comm,pid,ppid
K8S.CONTAINERNAME                COMM              PID              PPID
mypod2                           wget           478644            44638

It's possible to hide and show some fields by prefixing them with +/-, for instance

$ kubectl gadget run trace_exec:latest --fields=-error,-timestamp
COMM  PID  TID  PCOMM  PPID  ARGS  K8S.NODE  K8S.NAMESPACE  K8S.PODNAME  K8S.CONTAINERNAME  USER  LOGINUSER  GROUP

$ kubectl gadget run trace_exec:latest --fields=+uid,+pid
COMM  PID  TID  PCOMM  PPID  ARGS  K8S.NODE  K8S.NAMESPACE  K8S.PODNAME  K8S.CONTAINERNAME  ERROR  USER  LOGINUSER  GROUP  UID

Run for a specific amount of time​

Many gadgets will run forever, printing the gathered output until we press Ctrl-C to stop them. If we want to run a gadget only for a window of time, we can use the --timeout int flag, passing the number of seconds during which we want to run the gadget.

$ kubectl gadget run trace_exec:latest --timeout 5
K8S.NAMESPACE  K8S.PODNAME    K8S.CONTAINER… COMM        PID     TID PCOMM      PPID ARGS    K8S.NO… E… TIMESTAMP       USER   LOGIN… GROUP
default        mypod2         mypod2         wget     509661  509661 sh       446383 /bin/w… miniku…    2024-07-31T19:… root   uid:4… root
default        mypod2         mypod2         wget     509732  509732 sh       446383 /bin/w… miniku…    2024-07-31T19:… root   uid:4… root