trace_dns
The trace_dns gadget is used to trace DNS queries and responses.
Getting started
Running the gadget:
- kubectl gadget
- ig
$ kubectl gadget run ghcr.io/inspektor-gadget/gadget/trace_dns:latest [flags]
$ sudo ig run ghcr.io/inspektor-gadget/gadget/trace_dns:latest [flags]
Guide
- kubectl gadget
- ig
DNS is a centerpiece of communication in Kubernetes, it allows pods to discover other applications and communicate with the outside world. So, it is important to be able monitor DNS requests and responses to debug DNS issues.
In this guide, we will use the trace_dns
gadget to trace DNS requests and responses in Kubernetes (single node minikube cluster). The gadget can be used to trace DNS requests at different stages e.g. from application pods to kube-dns
service, from kube-dns
service to CoreDNS pod, from CoreDNS pod to upstream DNS server, etc.
The gadget can also be used to trace DNS requests based on different filters e.g. namespace, pod name, container name, domain name, etc.
Check out video, Demystifying DNS: A Guide to Understanding and Debugging Request Flows in Kubernetes Clusters to learn troubleshooting DNS issues with Inspektor Gadgets.
Flow of DNS request in Kubernetes:
Before we jump to scenarios, it is important to understand DNS in Kubernetes is based on multiple components:
- Application Pods: These are the pods that make DNS requests.
kube-dns
Service: The Kubernetes service to which application pods send requests (dnsPolicy=ClusterFirst
).- CoreDNS: The DNS server that resolves DNS requests in Kubernetes.
- Upstream DNS: The DNS server that CoreDNS forwards DNS requests to.
You might have other components in your DNS setup e.g. nodelocaldns
, etc. but for the purpose of this guide, we will focus on the above components.
The overall flow of DNS request is as follows:
Tracing an Application Pod:
Let's try to understand the flow at different stages. We will start by tracing DNS requests for specific pod. First, create a namespace:
kubectl create namespace demo
with expected output:
namespace/demo created
Run the gadget in a terminal, in this case we are only focused on DNS requests/response in demo
namespace for pod mypod
:
kubectl gadget run trace_dns:latest -n demo -p mypod
Run a pod in a different terminal:
kubectl -n demo run mypod -it --image=wbitt/network-multitool -- /bin/sh
Inside the pod, perform a DNS request:
nslookup -query=a inspektor-gadget.io.
The request/response will be logged by the DNS gadget:
K8S.NODE K8S.NAMESPACE K8S.PODNAME K8S.CONTAINERNAME SRC DST COMM PID QR QTYPE NAME RCODE ADDRESSES
minikube-docker demo mypod mypod p/demo/mypod:56093 s/kube-system/kube-dns:53 isc-net-0… 422204 Q A inspektor-gadget.io.
minikube-docker demo mypod mypod s/kube-system/kube-dns:53 p/demo/mypod:56093 isc-net-0… 422204 R A inspektor-gadget.io. Success 172.67.16…
The first line shows the DNS request from the mypod
pod to kube-dns
service and the second line shows the response from kube-dns
service back to the mypod
pod as confirmed by SRC
and DST
fields.
Tracing Application Pod and CoreDNS Pod:
Let's try to expand the scope and see what happens to the DNS request at CoreDNS pod. Restart the gadget as:
kubectl gadget run trace_dns:latest -n demo,kube-system -F "k8s.podName~mypod|coredns-.*" -F "name==inspektor-gadget.io." --fields=k8s.node,k8s.namespace,k8s.podname,id,src,dst,qr,name,rcode,timestamp
This will filter the DNS request/response for pods mypod
and CoreDNS pods. See how are able to choose different fields via --fields
.
On performing the DNS request again, you will see the following output from the gadget:
K8S.NODE K8S.NAMESPACE K8S.PODNAME ID SRC DST QR NAME RCODE TIMESTAMP
minikube-docker demo mypod c3b5 p/demo/mypod:50037 s/kube-system/kube-dns:53 Q inspektor-gadget.io. 2024-08-30T15:15:21.13712…
minikube-docker kube-system coredns-7db…8ff4d-r7hwl c3b5 p/demo/mypod:50037 p/kube-system/coredns-7db6d8ff4d-r7hwl:… Q inspektor-gadget.io. 2024-08-30T15:15:21.13720…
minikube-docker kube-system coredns-7db…8ff4d-r7hwl 0e01 p/kube-system/coredns-7db6d8ff4d-r7hwl:… 192.168.49.1:53 Q inspektor-gadget.io. 2024-08-30T15:15:21.13748…
minikube-docker kube-system coredns-7db…8ff4d-r7hwl 0e01 192.168.49.1:53 p/kube-system/coredns-7db6d8ff4d-r7hwl:… R inspektor-gadget.io. Success 2024-08-30T15:15:21.16513…
minikube-docker kube-system coredns-7db…8ff4d-r7hwl c3b5 p/kube-system/coredns-7db6d8ff4d-r7hwl:… p/demo/mypod:50037 R inspektor-gadget.io. Success 2024-08-30T15:15:21.16524…
minikube-docker demo mypod c3b5 s/kube-system/kube-dns:53 p/demo/mypod:50037 R inspektor-gadget.io. Success 2024-08-30T15:15:21.16526…
In this trace, we can see the request/response involving both the CoreDNS pod and mypod
. The first two lines show the request reaching CoreDNS pod. Since we are trying to resolve an external domain name inspektor-gadget.io.
, CoreDNS forwards the request to the upstream DNS server (confirmed by new ID=0e01
) in our case the upstream server is 192.168.49.1:53
. The remaining lines show the response (ID=0e01
) of the upstream DNS server to CoreDNS pod and the response (ID=c3b5
) of CoreDNS pod to the pod.
You might not see the upstream DNS request / response if you are using CoreDNS cache plugin since CoreDNS will cache the upstream response. Also, the output isn't sorted based on timestamp
but you can use it to understand the sequence of events.
This showcases how trace_dns
gadget can be used to trace DNS request at different stages. This is helpful to understand if a request/response is being dropped/altered at a specific stage. Again the output can vary based on the DNS/CNI setup in your cluster but the idea here is to provide a starting point to debug DNS issues.
Start the gadget in a terminal:
$ sudo ig run trace_dns:latest --containername test-trace-dns
RUNTIME.CONTAINERNAME SRC DST COMM PID QR QTYPE NAME RCODE ADDRESSES
Launch a container that makes DNS requests:
$ docker run --name test-trace-dns -it --rm wbitt/network-multitool /bin/sh
/ # nslookup -querytype=a inspektor-gadget.io.
The requests will be logged by the DNS gadget:
RUNTIME.CONTAINERNAME SRC DST COMM PID QR QTYPE NAME RCODE ADDRESSES
test-trace-dns 172.17.0.2:36282 192.168.0.1:53 isc-net-0000 675195 Q A inspektor-gadget.io.
test-trace-dns 192.168.0.1:53 172.17.0.2:36282 isc-net-0000 675195 R A inspektor-gadget.io. Success 104.21.11.16…
Finally, clean the system:
- kubectl gadget
- ig
$ kubectl delete namespace demo
$ docker rm -f test-trace-dns
Limitations
DNS over TCP support is limited since it doesn't support TCP stream reassembly. This means that the gadget will not be able to trace DNS requests that are sent over multiple TCP packets.