Skip to main content
Version: latest

Exporting Metrics (Prometheus)

Inspektor Gadget supports exporting metrics to Prometheus using the metrics exporter. In order to do so, you need to enable the metrics listener and configure each gadget to actively expose its metrics.

Enabling the metrics listener​

In order to enable the metrics listener, you need to set the otel-metrics-listen parameter to true:

WIP: Headless mode for kubectl gadget is under development

Enabling export for gadgets​

Some gadgets have the ability to export metrics. The Exporting metrics section of the gadget's documentation will tell you whether or not the gadget exports metrics and, in case, what metrics it exports. The profile_blockio gadget is an example of a gadget that exports metrics.

note

If you're developing your own gadget, you can add metrics to it by following the instructions in the Adding Metrics to your Gadget guide.

Now, in order to enable the export of metrics for a gadget, you need to do the following:

  • Annotate the data source with metrics.collect=true: This tells the gadget to collect and export metrics for this data source.
  • Specify a unique metrics name for the data source using the --otel-metrics-name datasource:metricsname flag: This is required even if datasource and metricsname are the same. This makes sure that you don't export metrics by accident and thereby skew existing data as the metricsname will be the used as the otel-scope.
  • [Optional] If you are only interested in exporting metrics and not displaying them in the CLI, you can use the --detach flag to run the gadget in headless mode.

Following is an example of how to enable metrics export for the profile_blockio gadget:

WIP: Headless mode for kubectl gadget is under development

Viewing metrics​

Once you have enabled the metrics listener and configured the gadgets to export metrics, you can query the metrics using Prometheus. By default, the metrics will be available at http://0.0.0.0:2224/metrics. You can change this address by setting the otel-metrics-listen-address parameter.

Metrics will be available under the metricsname you specified when enabling the metrics export for the gadget. For example, under blockio-metrics for the profile_blockio gadget we ran above:

WIP: Headless mode for kubectl gadget is under development

Stopping metrics collection​

You can stop the metrics collection at any time by deleting the gadget instance:

WIP: Headless mode for kubectl gadget is under development

Other Exporters​

If you have more complex requirements for metric exporting, you can setup multiple exporters using the config file. In this section we're exploring other exporters.

OTLP-gRPC​

In order to setup an exporter using OTLP over gRPC, you can add the following section to your config file:

operator:
  otel-metrics:
    exporters:
      myexporter:
        exporter: otlp-grpc
        endpoint: "localhost:4317"
        insecure: true
        temporality: delta
        interval: 30s

This will add a new exporter named "myexporter" that can be selected by using the flag --otel-metric-exporter myexporter when running a gadget. exporter needs to be set to otlp-grpc and you at least need to configure an endpoint.

Insecure​

This boolean flag determines whether to use encryption when communicating with the server.

Temporality​

Can be cumulative (default) or delta. See the official documentation for more information.

Interval​

Interval in which to report metrics to the server.