Empowering Observability: The Advent of Image-Based Gadgets

Today, with the release of v0.31.0, we announce the general availability of image-based Gadgets, the core building block of Inspektor Gadget’s newly defined data collection and systems inspection framework; enabling an ecosystem of observability tooling to be built with Inspektor Gadget.

Image-based Gadgets are OCI-images containing eBPF programs with some additional artifacts we’ll discuss below. Image-based Gadgets have been tucked behind an experimental flag for the past year as the team built out the complementary features that have expanded the project from a tool for interactive debugging sessions to a comprehensive framework for observability.

In this post, we provide some background, a high-level overview of Inspektor Gadget’s new functionality, project goals, core-concepts, the current state of the project and ongoing work to be introduced in subsequent releases. A series of posts will follow that will go into detail about the various aspects of the project discussed here.

Some background

We started the Inspektor Gadget project as an interactive tool to debug and inspect apps running in Kubernetes using the unique features and characteristics of eBPF, e.g., safety, efficiency, performance. Gadgets were simply eBPF programs embedded into the Inspektor Gadget binary that was deployed to Kubernetes nodes all in one go.

Note: These embedded eBPF programs are still available and referred to as built-in Gadgets. We do not plan to add new built-in gadgets or give them new features. In time, they will be deprecated and eventually removed. Image-based Gadgets should be used going forward.

Adding more Gadgets to the project necessitated generalizing how we worked with various types of eBPF programs and handled diverse types of behavior and output. This process of generalization enabled us to reduce the boilerplate code and quickly add new Gadgets.

This and feedback from users started a few topics of conversation.

• How to define a consistent interface to the distinct types of Gadgets
• How to support long-running Gadgets for data collection (e.g., delivering metrics to Prometheus)
• How to enable building gadgets decoupled from the project release process
• How to export data from various types of Gadgets in various formats and to various destinations.
• How to allow users to deactivate or modify certain stages of the data collection process
• How to support running Gadgets directly on Linux hosts independent of Kubernetes

Inspektor Gadget as a framework

These topics led us to start thinking of the project not as a single tool but rather a collection of related components, and the data being collected and output as flowing through successive stages of a system. In effect, we started viewing Inspektor Gadget in a general way as a framework for data collection that could not only be leveraged by our project but useful to anyone interested in observability or debugging.

With this perspective, we took a step back to assess how we could make Inspektor Gadget as useful as possible. A framework is most useful when it provides outsized value to both those building with the framework and end-users of the framework.

For example, Docker provides value to application developers by enabling them to package their apps and easily distribute them to users; lowering the barrier to getting started. The end-users experience this value by being able to take these containerized apps and easily deploy them locally or in production in a myriad environments.

So, we set out on an iterative process to figure how we could provide a similarly outsized value proposition with Inspektor Gadget.

Image-based Gadgets

The first step was to pull Gadgets, the eBPF programs, out of the core of the binary. This provides several benefits. It

• allows developers to create new Gadgets independent of the project release cycle,
• allows for the creation of Gadgets too specific to be considered for the core project,
• enables easily sharing Gadgets,
• allows administrators to control which Gadgets are installed,
• ensures that third-party developers have the same experience as we have as project maintainers.

In deciding how to package Gadgets, the logical choice was to use OCI images, the same format used to package Docker containers. This has the advantages of

• being a well-defined, standard, and popular format,
• making use of the robust infrastructure and tools built for containers; registries, libraries, etc.,
• make use of our team’s knowledge of working with and building container runtimes,
• being easily understandable by anyone familiar with cloud-native infrastructure or containers.

Gadgets not only contain eBPF programs, but they also contain information and modules that are used by the framework to provide additional functionality. Specifically, it contains

• eBPF program(s)
• a metadata file (gadget.yaml) that includes
  • author information,
  • license,
  • URI to source code,
  • A description of the datasource; names, display hints and descriptions of fields,
  • eBPF parameters that can be used for filtering, configuration etc.,
• optional WASM modules used by Inspektor Gadget operators (more on that below) that allow for post-processing data and overriding or adding to default functionality,
• btfgen: generated BTF files to run eBPF programs on kernels that do not have BTF (BPF Type Format) enabled.

Thus, image-based Gadgets provide the core unit upon which Inspektor Gadget’s framework operates and once built can be used in a variety of ways, which we will look at now.

Supporting more environments

As mentioned, originally Inspektor Gadget only supported running on Kubernetes. But with the goal of providing the most value from the framework for both end-users and developers of Gadgets, we needed to expand beyond Kubernetes; it turned out we already had the tool for that.

ig: run gadgets directly on Linux hosts

We had been having conversations with users about using Inspektor Gadget to debug Kubernetes itself, meaning Inspektor Gadget needed to run directly on the Linux host but still with an understand of containers and Kubernetes resources. Luckily, we had a tool called local-gadget used to test Gadgets in isolation without having to spin up a Kubernetes cluster. It turned out this was easy to break out into its own top-level tool to serve the aforementioned purpose. This tool was later renamed to ig and is the tool for not only running Gadgets directly on Linux hosts, but also for building and managing image-based Gadgets.

gadgetctl: client for connecting to remote ig instances

In addition, we wanted to support working with Inspektor Gadget on non-Linux environments; MacOS, Windows, etc. For Kubernetes we already had kubectl-gadget which can be used from any platform. But ig is only supported on Linux, as eBPF is a Linux kernel technology and Gadgets and must operate on Linux hosts. So, we needed a client that could connect to a daemon running on a Linux host, like how the docker client connects to a Docker daemon. So, we added a --daemon flag to ig and built gadgetctl^† as a client for working with non-Kubernetes environments.

kubectl debug: support

Recently, kubectl got the debug subcommand which allows users to debug issues on a specific Kubernetes node using a transient environment. The team quickly added the ability to use Inspektor Gadget with kubectl-debug. This feature is convenient because it allows you to quickly investigate issues without installing Inspektor Gadget into the cluster.

With these additions, Gadgets can be used across many environments.

Programmatic modes of operation

The typical user interacts with Inspektor Gadget via the command-line interface using either kubectl-gadget, ig or gadgetctl. But we have also introduced a couple other modes of operations to programmatically work with Inspektor Gadget.

gRPC API

Applications can interface with Inspektor Gadget using its gRPC API. This is the same API used by our client tools, kubectl-gadget and gadgetctl. This allows, for example, interacting with Inspektor Gadget from a Web UI. We are currently working with the Headlamp project, to build a plugin that will provide a Web UI for Inspektor Gadget which is using this API.

Golang library

Another means of interacting with Inspektor Gadget is via its Golang library. This allows for embedding Gadget functionality into Go applications, taking advantage of features Inspektor Gadget provides out of the box; enrichment, filtering, etc. This is used, for example, by Microsoft Defender for Containers and Kubescape from ARMO.

All the above features provide a wealth of ways for users to work with Gadgets in a variety of environments and modes of operation.

A word on architectural changes

We will leave going into the details of the Inspektor Gadget architecture for a dedicated follow-up post. But let us take a quick look at a high-level diagram.

This diagram depicts the modes of operation covered above. But you will also notice a sequence of blocks central to Inspektor Gadget.

Inspektor Gadget Operators

One key goal was to provide the ability for developers and users to modify various aspects of the framework. This drove the team to encapsulate key functionality into what we refer to as operators, a modular component that is integrated into the workflow of a gadget.

Thus, we have encapsulated each stage of the Inspektor Gadget framework into an operator. For example, fetching Gadgets and loading eBPF programs are each an operator. For data processing, each stage (i.e., enrichment, post-processing, export) is an operator, opening the possibility for customization. Additionally, filtering, sorting and formatting are also done by operators allowing preparing data to specific needs.

We are excited about the introduction of operators as it empowers developers to fully leverage Inspektor Gadget as a framework, and end-users to customize the output of Gadgets to meet their exact needs.

Note: Operators are currently under-documented. We aim to rectify this over the next releases.

Security mechanisms

The ability to fetch Gadgets from remote servers means new security controls needed to be put in place. For example, an administrator should be able to

• restrict which, and from where, Gadgets can be loaded onto the system
• preinstall and lockdown which Gadgets are available on a system
• define the above in a global configuration

For example, an administrator wanting to make debug tools available to users can preinstall a set of Gadgets for the users of a system and disable fetching new Gadgets. Or that same administrator could allow fetching new Gadgets from a specified repository which they control.

In any case, these security mechanisms are an essential tool that allows Inspektor Gadget to support more use cases.

Current state

The team’s major effort over the last year was to build out the functionality and flexibility of the framework around image-based Gadgets. This work is not finished but by taking image-based Gadgets out from behind the experimental flag, we feel image-based Gadgets have the core supporting functionality it needs and are ready for daily use.

What this means concretely, is

• almost all built-in Gadgets have been ported to image-based Gadgets,
• the core image-based Gadget format is defined,
• security mechanisms for managing image-based Gadgets are available,
• all existing features work, or have been expanded, with image-based Gadgets,
• WASM modules are supported for extending functionality.

While this is a major milestone for the project, it is only a first step.

Coming iterations

While the core functionality has landed, some of the most exciting features will land in subsequent releases. The features aim to enhance the usability of Inspektor Gadget both as a framework and a tool. Some examples of what is planned are as follows:

• Declarative runtime configuration to support running one or more gadgets using ig run -f=my-runtime-config.yaml
• Expand support for OpenTelemetry tracing
• Full support for long-run metrics collection
• Plugin for Headlamp, a Kubernetes web UI, that automatically generates a web UI for controlling Gadgets and visualizing their output.
• Port the few remaining unported built-in Gadgets

In addition, we will be focusing on making improvements to the documentation.

Conclusion

We hope this post has provided an understanding of the Inspektor Gadget project as it stands today and where it is headed. Image-based Gadgets have opened a bevy of new possibilities for the eBPF and observability communities.

We look forward to how you will make use of Inspektor Gadget and are eager to hear your feedback. Please reach out to us on Slack and/or let us know if you find an issue or have a feature request using via Github issues.

And do stay tuned for follow posts that will demonstrate features and provide examples of how you can use and build with Inspektor Gadget. And as always, please refer to the project documentation for any

^† gadgetctl shares code with kubectl-gadget and we are exploring the possibility of fully merging them.