Skip to content

Network debug with nsenter

Introduction

The nsenter command executes a program in the namespaces specified in the command-line options. If a program is not specified, ${SHELL} is run. The default shell is /bin/sh.

The supported namespaces are as follows:

  • IPC namespace: IPC namespaces isolate certain IPC resources, such as System IPC identifiers and their own POSIX message queue file system. Objects created in one IPC namespace are visible to all other processes within the same namespace but are invisible to processes in other IPC namespaces.
  • UTS namespace: UTS namespaces isolate two system identifiers: the hostname and the NIS domain name. Changes made to these identifiers in one UTS namespace are visible to all other processes in the same namespace but are invisible to processes in other UTS namespaces.
  • Network namespace: Network namespaces isolate system resources associated with networking, including network devices, IPv4 and IPv6 protocol stacks, IP routing tables, firewall rules, and other networking-related resources. Processes in one network namespace have independent networking resources compared to processes in other network namespaces.
  • PID namespace: PID namespaces isolate the process ID number space, meaning processes in different PID namespaces can have the same PID. PID namespaces allow containers to suspend or resume the set of processes in the container and migrate the container to a new host while maintaining the same PIDs.
  • Mount namespace: Mount namespaces isolate the list of mount points seen by processes in each namespace instance. Processes in each mount namespace instance will see distinct single-directory hierarchies.

Situation

Kubernetes is a widely-used platform for managing containerized applications. As networking is a critical component of any Kubernetes cluster, issues may arise that necessitate troubleshooting. nsenter, a Linux utility that allows entry into the namespaces of other processes, can be a helpful tool in these situations.

In a Kubernetes cluster, each pod has its own network namespace. Thus, nsenter can be used to enter a pod’s network namespace and troubleshoot networking issues from the host node of the pod. This is particularly useful in scenarios where pods lack a shell to exec into or in environments that don’t provide access to a network utility pod for troubleshooting.

Resolution

Issue: Unable to Connect to a Service

If you’re unable to connect to a service running in a Kubernetes cluster, use nsenter to troubleshoot the issue from the pod’s host node. Here’s how:

  1. Identify the pod running the service using the kubectl get pods command. This lists all the pods in your cluster and their current status.
  2. Find the PID of the container running the pod using the ps aux command.
  3. After identifying the PID, enter the container’s network namespace using the nsenter command. The network namespace is located at /proc/{PID}/ns/net. For example, nsenter -t {PID} -n.
  4. Inside the container’s network namespace, use standard networking tools (like ping, curl, or telnet) to test connectivity to the service.

Issue: Pod Cannot Reach the Internet

If a pod cannot reach the internet, follow the same steps to use nsenter for troubleshooting. However, in the final step, use standard networking tools to test connectivity to external hosts instead of a service.

By using nsenter, you can easily troubleshoot networking issues in your Kubernetes cluster without having to exec into the pod or install additional utilities.

Examples

nsenter -t 7172 -p -r top
nsenter -t 7172 -n ip a s
nsenter -t 7172 -n ip route
nsenter -t 7172 -p -r ps -ef
nsenter -t 7172 -u hostname
nsenter -t 7172 -a

Reference

  • https://www.suse.com/support/kb/doc/?id=000021060
  • https://man7.org/linux/man-pages/man1/nsenter.1.html
  • https://www.redhat.com/sysadmin/container-namespaces-nsenter
  • https://www.ibm.com/docs/en/zos/3.1.0?topic=descriptions-nsenter-execute-programs-in-different-namespaces
  • https://www.suse.com/support/kb/doc/?id=000021060
Feedback