Skip to main content

Using Inspektor Gadget from Golang Applications

Mauricio Vásquez Bernal
· 16 min read

In our previous Rewriting the control plane of BCC tools in Golang blog post we explained why and how we implemented the control plane of BCC tools in Golang. In the middle of that process, we realized that the Golang packages we were implementing were generic enough to be used in other applications besides Inspektor Gadget. In particular, three common features are provided:

  1. Tracing different events on the host
  2. Filtering events by container
  3. Pretty printing the output

This blog post provides some examples showing how the Inspektor Gadget Golang packages can be used. Specifically, we'll cover the following:

  • Tracers: Packages to collect events from the host, like process creation, file access, etc.
  • Container Collection: Package to trace the creation and removal of containers in a host.
  • Trace Collection: Package used to filter events by containers.
  • Columns: Package to create a "pretty" representation of the events generated by the tracers.

We try to keep these examples as simple as possible to focus on the package itself, a real-world usecase is left to the reader.

An application must create an instance of a tracer to get events on the host. This tracer is configured with a set of filtering options to determinate the events to take into consideration. We'll cover this in detail later. The tracer uses eBPF to capture the events on the kernel and provide them to the application using a callback.

Picture showing the different components. On the top we have the application that passes the configuration to the tracer. Then the tracer returns events. The tracer uses eBPF programs and maps to get information from the kernel.

"Architecture Diagram"

In the next sections we show how to configure and use those tracers. For each example we provide the full code, so readers can compile and try it out.

Using the Tracers without Filtering​

Let's start with the simplest case, a tracer without any filtering. We'll use the exec trace gadget to create an application that prints a message to the terminal each time a new process is created on the host.

First of all, we have to remove the memlock to be able to load programs in some kernel versions:

// In some kernel versions it's needed to bump the rlimits to
// use run BPF programs.
if err := rlimit.RemoveMemlock(); err != nil {
	return
}

We need to define a callback that will be invoked by the tracer each time an event is produced, i.e. a new process is created in this case. This callback receives as its argument an event. In this example, we only print the name and the pid of the new process:

eventCallback := func(event types.Event) {
	fmt.Printf("A new %q process with pid %d was executed\n",
			   event.Comm, event.Pid)
}

Then, we have to create the tracer. In this case we're not interested in any filtering nor Kubernetes data enrichment, hence we pass an empty Config struct and a nil enricher.

tracer, err := tracer.NewTracer(&tracer.Config{}, nil, eventCallback)
if err != nil {
	fmt.Printf("error creating tracer: %s\n", err)
	return
}
defer tracer.Stop()

We need to prevent the application from closing, in this case by waiting for SIGINT

exit := make(chan os.Signal, 1)
signal.Notify(exit, syscall.SIGINT, syscall.SIGTERM)
<-exit

And that's pretty much it. This is the full code of the example:

package main

import (
"fmt"
"os"
"os/signal"
"syscall"

    "github.com/cilium/ebpf/rlimit"

    "github.com/inspektor-gadget/inspektor-gadget/pkg/gadgets/trace/exec/tracer"
    "github.com/inspektor-gadget/inspektor-gadget/pkg/gadgets/trace/exec/types"

)

func main() {
// In some kernel versions it's needed to bump the rlimits to
// use run BPF programs.
if err := rlimit.RemoveMemlock(); err != nil {
return
}

    // Define a callback to be called each time there is an event.
    eventCallback := func(event types.Event) {
    	fmt.Printf("A new %q process with pid %d was executed\n",
    		event.Comm, event.Pid)
    }

    // Create the tracer. An empty configuration is passed as we are
    // not interesting on filtering by any container. For the same
    // reason, no enricher is passed.
    tracer, err := tracer.NewTracer(&tracer.Config{}, nil, eventCallback)
    if err != nil {
    	fmt.Printf("error creating tracer: %s\n", err)
    	return
    }
    defer tracer.Stop()

    // Graceful shutdown
    exit := make(chan os.Signal, 1)
    signal.Notify(exit, syscall.SIGINT, syscall.SIGTERM)
    <-exit

}

That's all, so let's compile and execute it:

$ go build -o exec .
$ sudo ./exec
A new "calico" process with pid 118594 was executed
A new "portmap" process with pid 118606 was executed
A new "bandwidth" process with pid 118611 was executed
A new "runc" process with pid 118616 was executed
A new "docker-init" process with pid 118623 was executed
^C

The example was run in a Kubernetes node; hence a lot of new processes were being created. You can try to run some tools like ping, cat, etc. to create new processes if you don't see any output.

This example traces all the new processes created on the host, but what about if we are only interested in processes created in some containers? Let's do that next.

Container Collection Tools​

Before adding filtering to our previous example, we need to study the container collection package which is used to keep track of containers on a host. This package can track creation and removal of containers, and uses different approaches to "enrich" an event providing extra information about a container, like its name, Linux namespaces, cgroups, etc.

For the sake of simplicity in this example, we'll create a container collection instance that only gets notifications from containers created through runc and uses docker or containerd to get their name.

The first thing is to define a callback that is invoked each time a container is created or removed:

callback := func(event containercollection.PubSubEvent) {
	switch event.Type {
	case containercollection.EventTypeAddContainer:
		fmt.Printf("Container added: %q pid %d\n",
				event.Container.Name, event.Container.Pid)
	case containercollection.EventTypeRemoveContainer:
		fmt.Printf("Container removed: %q pid %d\n",
				event.Container.Name, event.Container.Pid)
	}
}

Then, we have to define the options for the container collection. In this case we'll use RuncFanotify: a mechanism that uses fanotify to get notifications about containers started with runc and we'll use docker and containerd to get more information about the containers, like the name.

// Define the different options for the container collection instance
opts := []containercollection.ContainerCollectionOption{
	// Indicate the callback that will be invoked each time
	// there is an event
	containercollection.WithPubSub(callback),

	// Get containers created with runc
	containercollection.WithRuncFanotify(),

	// Enrich those containers with data from the container
	// runtime. docker and containerd in this case.
	// (It's needed to have the name of the container in this example).
	containercollection.WithMultipleContainerRuntimesEnrichment(
		[]*containerutils.RuntimeConfig{
			{Name: docker.Name},
			{Name: containerd.Name},
		}),
}

Then, we only need to create and initialize the instance:

// Create and initialize the container collection
containerCollection := &containercollection.ContainerCollection{}
err := containerCollection.Initialize(opts...)
if err != nil {
	fmt.Printf("failed to initialize container collection: %s\n", err)
	return
}
defer containerCollection.Close()

The full code is the following one:

package main

import (
"fmt"
"os"
"os/signal"
"syscall"

    containercollection "github.com/inspektor-gadget/inspektor-gadget/pkg/container-collection"
    containerutils "github.com/inspektor-gadget/inspektor-gadget/pkg/container-utils"
    "github.com/inspektor-gadget/inspektor-gadget/pkg/container-utils/containerd"
    "github.com/inspektor-gadget/inspektor-gadget/pkg/container-utils/docker"

)

func main() {
// Function that will be called for events. event contains
// information about the kind of event (added, removed) and an
// instance of the container.
callback := func(event containercollection.PubSubEvent) {
switch event.Type {
case containercollection.EventTypeAddContainer:
fmt.Printf("Container added: %q pid %d\n",
event.Container.Name, event.Container.Pid)
case containercollection.EventTypeRemoveContainer:
fmt.Printf("Container removed: %q pid %d\n",
event.Container.Name, event.Container.Pid)
}
}

    // Define the different options for the container collection instance
    opts := []containercollection.ContainerCollectionOption{
    	// Indicate the callback that will be invoked each time
    	// there is an event
    	containercollection.WithPubSub(callback),

    	// Get containers created with runc
    	containercollection.WithRuncFanotify(),

    	// Enrich those containers with data from the container
    	// runtime. docker and containerd in this case.
    	// (It's needed to have the name of the container in this example).
    	containercollection.WithMultipleContainerRuntimesEnrichment(
    		[]*containerutils.RuntimeConfig{
    			{Name: docker.Name},
    			{Name: containerd.Name},
    		}),
    }

    // Create and initialize the container collection
    containerCollection := &containercollection.ContainerCollection{}
    err := containerCollection.Initialize(opts...)
    if err != nil {
    	fmt.Printf("failed to initialize container collection: %s\n", err)
    	return
    }
    defer containerCollection.Close()

    // Graceful shutdown
    exit := make(chan os.Signal, 1)
    signal.Notify(exit, syscall.SIGINT, syscall.SIGTERM)
    <-exit

}

Let's compile and start it:

$ go build -o container-collection .
$ sudo ./container-collection

It'll print the list of existing containers. Let's run a container and see how it prints a message when it is created:

$ docker run --name hi --rm ubuntu /bin/bash -c "echo hi && sleep 2"
hi

$ sudo ./container-collection
Container added: "hi" pid 130258
Container removed: "hi" pid 130258

Filtering Events by Containers​

In many cases we're not interested in all the events that happen in a host, but we want to filter them by container, PID, UID, etc. Even if it is possible to implement this logic in the application, it won't be very efficient as we'll be carrying the events all the way up from the kernel just to discard them. The tracers we implemented provide a mechanism to filter events by container directly in the eBPF code. This reduces the performance overhead as we don't collect events we're not interested in. Filtering by other parameters like PID and UID is not yet supported by our tracers.

The mechanism to filter by containers is based on the mount namespace inode id. The tracer expects an eBPF map with the inode ids of the containers of interest. Getting the inode id of the mount namespace of the container to populate the map is not trivial. Hence, we'll use the tracer collection package that simplifies this for us.

The first thing we need is to create a tracer collection. This is the structure that "glues" the container collection and the tracer instance.

tracerCollection, err := tracercollection.NewTracerCollection(containerCollection)
if err != nil {
	fmt.Printf("failed to create trace-collection: %s\n", err)
	return
}

In the options of the container collection, we then need to pass the TraceMapsUpdater() as a callback. It's a function that will update the filtering map each time a container is created or removed.

containercollection.WithPubSub(tracerCollection.TracerMapsUpdater()),

We then need to define a container selector with the containers we want to match:

containerSelector := containercollection.ContainerSelector{
	Name: containerName,
}

After that, we need to create a tracer. It's the glue code that allows us to filter the events.

if err := tracerCollection.AddTracer(traceName, containerSelector); err != nil {
	fmt.Printf("error adding tracer: %s\n", err)
	return
}
defer tracerCollection.RemoveTracer(traceName)

Finally, we need to get the mount namespace map for this tracer...

// Get mount namespace map to filter by containers
mountnsmap, err := tracerCollection.TracerMountNsMap(traceName)
if err != nil {
	fmt.Printf("failed to get mountnsmap: %s\n", err)
	return
}

...and pass it when creating the exec tracer

// Create the tracer
tracer, err := tracer.NewTracer(&tracer.Config{MountnsMap: mountnsmap}, containerCollection, eventCallback)
if err != nil {
	fmt.Printf("error creating tracer: %s\n", err)
	return
}
defer tracer.Stop()
package main

import (
"flag"
"fmt"
"os"
"os/signal"
"syscall"

    "github.com/cilium/ebpf/rlimit"

    containercollection "github.com/inspektor-gadget/inspektor-gadget/pkg/container-collection"
    containerutils "github.com/inspektor-gadget/inspektor-gadget/pkg/container-utils"
    "github.com/inspektor-gadget/inspektor-gadget/pkg/container-utils/containerd"
    "github.com/inspektor-gadget/inspektor-gadget/pkg/container-utils/docker"
    "github.com/inspektor-gadget/inspektor-gadget/pkg/gadgets/trace/exec/tracer"
    "github.com/inspektor-gadget/inspektor-gadget/pkg/gadgets/trace/exec/types"
    tracercollection "github.com/inspektor-gadget/inspektor-gadget/pkg/tracer-collection"

)

const traceName: "trace_exec"

func main() {
var containerName string
flag.StringVar(&containerName, "containername", "", "Show only data from containers with that name")
flag.Parse()

    if containerName == "" {
    	fmt.Printf("you must provide --containername\n")
    	return
    }

    // In some kernel versions it's needed to bump the rlimits to
    // use run BPF programs.
    if err := rlimit.RemoveMemlock(); err != nil {
    	return
    }

    // Create and initialize the container collection
    containerCollection := &containercollection.ContainerCollection{}

    tracerCollection, err := tracercollection.NewTracerCollection(containerCollection)
    if err != nil {
    	fmt.Printf("failed to create trace-collection: %s\n", err)
    	return
    }
    defer tracerCollection.Close()

    // Define the different options for the container collection instance
    opts := []containercollection.ContainerCollectionOption{
    	// Indicate the callback that will be invoked each time
    	// there is an event
    	containercollection.WithPubSub(tracerCollection.TracerMapsUpdater()),

    	// Get containers created with runc
    	containercollection.WithRuncFanotify(),

    	// Enrich events with Linux namespaces information
    	// It's needed to be able to filter by containers in this example.
    	containercollection.WithLinuxNamespaceEnrichment(),

    	// Enrich those containers with data from the container
    	// runtime. docker and containerd in this case.
    	containercollection.WithMultipleContainerRuntimesEnrichment(
    		[]*containerutils.RuntimeConfig{
    			{Name: docker.Name},
    			{Name: containerd.Name},
    		}),
    }

    if err := containerCollection.Initialize(opts...); err != nil {
    	fmt.Printf("failed to initialize container collection: %s\n", err)
    	return
    }
    defer containerCollection.Close()

    // Define a callback to be called each time there is an event.
    eventCallback := func(event types.Event) {
    	fmt.Printf("A new %q process with pid %d was executed in container %q\n",
    		event.Comm, event.Pid, event.Container)
    }

    // Create a tracer instance. This is the glue piece that allows
    // this example to filter events by containers.
    containerSelector := containercollection.ContainerSelector{
    	Name: containerName,
    }

    if err := tracerCollection.AddTracer(traceName, containerSelector); err != nil {
    	fmt.Printf("error adding tracer: %s\n", err)
    	return
    }
    defer tracerCollection.RemoveTracer(traceName)

    // Get mount namespace map to filter by containers
    mountnsmap, err := tracerCollection.TracerMountNsMap(traceName)
    if err != nil {
    	fmt.Printf("failed to get mountnsmap: %s\n", err)
    	return
    }

    // Create the tracer
    tracer, err := tracer.NewTracer(&tracer.Config{MountnsMap: mountnsmap}, containerCollection, eventCallback)
    if err != nil {
    	fmt.Printf("error creating tracer: %s\n", err)
    	return
    }
    defer tracer.Stop()

    // Graceful shutdown
    exit := make(chan os.Signal, 1)
    signal.Notify(exit, syscall.SIGINT, syscall.SIGTERM)
    <-exit

}

It can be compiled again with go build .. This time it needs a --containername parameter:

$ sudo ./exec --containername foo

Create a foo container that executes some commands in another terminal

$ sudo docker run --rm --name foo ubuntu bash -c "cat /dev/null && sleep 2"

The first terminal should show the processes created in this container:

$ sudo ./exec --containername foo
A new "bash" process with pid 445451 was executed in container "foo"
A new "cat" process with pid 445512 was executed in container "foo"
A new "sleep" process with pid 445451 was executed in container "foo"

Formatting the events​

In the previous events, we just used fmt.Printf() to print a message to the user's terminal when an event happened. But what if we want to print a "formatted" version of the events? We implemented a columns package for this.

First, we need to create a formatter. In this case we specify the columns we want to print:

// Create a formatter. It's the component that converts events to columns.
colNames := []string{"container", "pid", "ppid", "comm", "ret", "args"}
formatter := textcolumns.NewFormatter(
	types.GetColumns().GetColumnMap(),
	textcolumns.WithDefaultColumns(colNames),
)

And then, we use the FormatEntry() method to get a string representation of an event:

// Define a callback to be called each time there is an event.
eventCallback := func(event types.Event) {
	// Convert the event to columns and print to the terminal.
	fmt.Println(formatter.FormatEntry(&event))
}

And that's it. This is the full code of the example:

package main

import (
"flag"
"fmt"
"os"
"os/signal"
"syscall"

    "github.com/cilium/ebpf/rlimit"

    "github.com/inspektor-gadget/inspektor-gadget/pkg/columns/formatter/textcolumns"
    containercollection "github.com/inspektor-gadget/inspektor-gadget/pkg/container-collection"
    containerutils "github.com/inspektor-gadget/inspektor-gadget/pkg/container-utils"
    "github.com/inspektor-gadget/inspektor-gadget/pkg/container-utils/containerd"
    "github.com/inspektor-gadget/inspektor-gadget/pkg/container-utils/docker"
    "github.com/inspektor-gadget/inspektor-gadget/pkg/gadgets/trace/exec/tracer"
    "github.com/inspektor-gadget/inspektor-gadget/pkg/gadgets/trace/exec/types"
    tracercollection "github.com/inspektor-gadget/inspektor-gadget/pkg/tracer-collection"

)

const traceName: "trace_exec"

func main() {
var containerName string
flag.StringVar(&containerName, "containername", "", "Show only data from containers with that name")
flag.Parse()

    if containerName == "" {
    	fmt.Printf("you must provide --containername\n")
    	return
    }

    // In some kernel versions it's needed to bump the rlimits to
    // use run BPF programs.
    if err := rlimit.RemoveMemlock(); err != nil {
    	return
    }

    // Create and initialize the container collection
    containerCollection := &containercollection.ContainerCollection{}

    tracerCollection, err := tracercollection.NewTracerCollection(containerCollection)
    if err != nil {
    	fmt.Printf("failed to create trace-collection: %s\n", err)
    	return
    }
    defer tracerCollection.Close()

    // Define the different options for the container collection instance
    opts := []containercollection.ContainerCollectionOption{
    	// Indicate the callback that will be invoked each time
    	// there is an event
    	containercollection.WithPubSub(tracerCollection.TracerMapsUpdater()),

    	// Get containers created with runc
    	containercollection.WithRuncFanotify(),

    	// Enrich events with Linux namespaces information
    	// It's needed to be able to filter by containers in this example.
    	containercollection.WithLinuxNamespaceEnrichment(),

    	// Enrich those containers with data from the container
    	// runtime. docker and containerd in this case.
    	containercollection.WithMultipleContainerRuntimesEnrichment(
    		[]*containerutils.RuntimeConfig{
    			{Name: docker.Name},
    			{Name: containerd.Name},
    		}),
    }

    if err := containerCollection.Initialize(opts...); err != nil {
    	fmt.Printf("failed to initialize container collection: %s\n", err)
    	return
    }
    defer containerCollection.Close()

    // Create a formatter. It's the component that converts events to columns.
    colNames := []string{"container", "pid", "ppid", "comm", "ret", "args"}
    formatter := textcolumns.NewFormatter(
    	types.GetColumns().GetColumnMap(),
    	textcolumns.WithDefaultColumns(colNames),
    )

    // Define a callback to be called each time there is an event.
    eventCallback := func(event types.Event) {
    	// Convert the event to columns and print to the terminal.
    	fmt.Println(formatter.FormatEntry(&event))
    }

    fmt.Println(formatter.FormatHeader())

    // Create a tracer instance. This is the glue piece that allows
    // this example to filter events by containers.
    containerSelector := containercollection.ContainerSelector{
    	Name: containerName,
    }

    if err := tracerCollection.AddTracer(traceName, containerSelector); err != nil {
    	fmt.Printf("error adding tracer: %s\n", err)
    	return
    }
    defer tracerCollection.RemoveTracer(traceName)

    // Get mount namespace map to filter by containers
    mountnsmap, err := tracerCollection.TracerMountNsMap(traceName)
    if err != nil {
    	fmt.Printf("failed to get mountnsmap: %s\n", err)
    	return
    }

    // Create the tracer
    tracer, err := tracer.NewTracer(&tracer.Config{MountnsMap: mountnsmap}, containerCollection, eventCallback)
    if err != nil {
    	fmt.Printf("error creating tracer: %s\n", err)
    	return
    }
    defer tracer.Stop()

    // Graceful shutdown
    exit := make(chan os.Signal, 1)
    signal.Notify(exit, os.Interrupt, syscall.SIGTERM)
    <-exit

}

This can be compiled with the same command as above:

$ go build .
$ sudo ./exec --container name foo

Let's run a container to generate some events:

$ sudo docker run --rm --name foo ubuntu bash -c "cat /dev/null && sleep 2"

And this is how the events should be printed:

$ sudo ./exec --containername foo
CONTAINER                        PID        PPID       RET ARGS
foo                              103752     103724     0   /usr/bin/bash -c cat /dev/null && sleep 2
foo                              103796     103752     0   /usr/bin/cat /dev/null
foo                              103752     103724     0   /usr/bin/sleep 2
^C

Using IG packages for security applications

Inspektor Gadget packages are designed for getting insights on Kubernetes but not as security mechanism to block unwanted events. There are several reasons why using those packages to implement security applications might be difficult or not suitable:

  • By the time the Golang tracer detects an unwanted event, it is already too late to stop it.
  • There might be ways for malicious applications to alter their behavior to hide some generated events.

Conclusion

The packages provided by Inspektor Gadget allow tracing different events on the host, to filter by container and to pretty print them. Check out the examples folder where existing and future examples are published. If you think there is something we're missing or something that is not working, please reach out. We're always happy to help.