How to debug if No Shell for Your Container

Challenges of Minimal Container Images

While working on building secure container images, you may have come across recommendations to use one of the following minimal base images:

• Scratch (https://dev.to/iblancasa/use-scratch-images-is-it-a-good-idea-32lp)
• Distroless (https://github.com/GoogleContainerTools/distroless)

After overcoming some obstacles, you were able to make your application work with one of these base images and successfully deploy your new application container to production.

However, one day you encounter an issue. Your application is not functioning properly, and you are facing difficulties because you have not yet implemented full instrumentation for your application using OpenTelemetry. Desperate to debug your application, you attempt to access a shell on your container. Unfortunately, you receive an error and are unable to access a shell.

> podman exec -it helloworld sh
Error: crun: executable file `sh` not found in $PATH: No such file or directory: OCI runtime attempted to invoke a command that was not found

This is a problem. Hopefully, we still have some room for error this month.

Building a Minimal Container Image

Before we learn how to debug the application container, let’s build an example application container using distroless as the base image.

The example application will be an HTTP server written in Go that responds to any HTTP request with Hello, World!.

First, initialize a new Go module:

go mod init helloworld

Then, create a main.go file with the following content:

package main

import (
    "log"
    "net/http"
    "os"
)

const defaultAddr = ":8080"

func main() {
 // Support a configurable listen address.
 addr := os.Getenv("HELLOWORLD_ADDR")
 if addr == "" {
  addr = defaultAddr
 }

 // An HTTP handler that logs each request and responds with `Hello, World!`.
 handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
  log.Printf("connection received: path=%s method=%s", r.URL.Path, r.Method)
  w.WriteHeader(http.StatusOK)
  w.Write([]byte("Hello, World!"))
 })

 // Start the HTTP server and wait for requests.
 log.Printf("http server started: addr=%s", addr)
 if err := http.ListenAndServe(addr, handler); err != nil {
  log.Fatalln(err)
 }
}

Next, create a Dockerfile (or Containerfile for Podman users) with the following content:

# Use distroless as the base image.
FROM gcr.io/distroless/base-debian10

# Copy the built binary into the container.
COPY helloworld /

# Run the binary.
CMD ["/helloworld"]

Build the application container image:

docker build -t helloworld .

Run the application container:

docker run -d -p 8080:8080 helloworld

In another terminal session, make an HTTP request to the application to verify that it is running correctly.

In the new terminal session, try to get a shell in the application container. You will receive an error and be unable to get a shell.

Perfect! Now we’re ready to debug.

Debugging with Sidecar Containers

Even though we cannot access a shell in the application container, we can still debug it using a sidecar container. A sidecar container shares access to the same resources as another container, specifically PID and network namespaces.

In the Kubernetes world, a sidecar container runs in the same pod as another container. While we are not using Kubernetes in this example, we can still create a sidecar container and attach it to the same resources as the application container.

Let’s create an Ubuntu container and attach it to the same PID and network namespaces as the application container.

podman run --rm -it \
  --pid container:helloworld \
  --network container:helloworld \
  ubuntu:latest

Next, install some debugging tools inside the new container.

root@5a0f61f5a4d8:/# apt update && apt install -y iproute2 file

Now, we can use these debugging tools to gather information about the application container.

For instance, we can list processes and verify that the helloworld process is running successfully.

We can also list information about network sockets and see that the helloworld process is listening on TCP port 8080.

root@5a0f61f5a4d8:/# ps
    PID TTY          TIME CMD
      1 pts/0    00:00:00 helloworld
      8 pts/0    00:00:00 bash
    581 pts/0    00:00:00 ps

Furthermore, we can even browse files on the filesystem of the application container.

The best part is that all of this access is enabled without modifying the application container image. Once we finish debugging, we can remove the sidecar container without leaving any trace or side effects.

Reference

• https://matthewsanabria.dev/posts/no-shell-for-you-container/
• https://kubernetes.io/docs/concepts/workloads/pods/sidecar-containers/

Leave a message

Name

Email



Send