Enabling and Using Ephemeral Containers on Kubernetes 1.16

Ephemeral Containers are an early-state alpha feature in Kubernetes 1.16 and offer some interesting new dynamics when it comes to tooling that we can use in day-to-day Kubernetes operations.

To see this feature live, in action, check out the demo shell session below:

Before we look at Ephemeral Containers, let’s go over what a Pod is in the Kubernetes world.

Remember that a Pod in Kubernetes is a group of one or more containers (e.g. Docker containers).

With that basic tidbit of information out of the way, we’ll look at some characteristics that Pods and their containers have always had in the past:

• They’re meant to be disposable and easily replaced in a controlled manner with Deployments.
• You could not add containers to pods at runtime.
• Containers in pods can have ports assigned for network access and use things like liveness probes.
• Troubleshooting containers in pods usually meant looking at logs or using kubectl exec to get into the running container and poke around. The latter of course being useless if your container had already crashed.

So here is where I see one of the best use cases for the new Ephemeral Containers feature – troubleshooting.

Ephemeral Containers can be inserted into a live, running pod at runtime.

This means they are great for live troubleshooting of your applications. How many times have you wished your base docker image you’ve built your application image on top of has curl, dig, or even ping in some cases…

If we’ve been following best practices, we’ve kept our docker images as slim as possible, and removed as much attack surface area as possible. This usually means all the useful diagnostic tools are missing.

Ephemeral Containers are great. We can now keep a diagnostic Docker image handy with all the tools we need and live insert a diagnostic container into a running pod to troubleshoot when the time arises.

When an Ephemeral Container runs, it executes within the namespace of the target pod. So you’ll be able to access, for example, the filesystems and other resources that containers in the the pods have.

Demonstration

In order to follow along with this demo, you’ll need Kubernetes 1.16 or higher, and you’ll need to use two pod related features:

• EphemeralContainers (of course) – disabled by default in 1.16 as it’s alpha.
• PodShareProcessNamespace – for sharing the process namespace in a pod (enabled by default in 1.16 as it’s a beta feature).

To enable the Ephemeral Containers feature, edit the following configuration files on your Kubernetes master nodes and restart each master:

Enable the EphemeralContainers alpha feature gate in the following places

• /etc/kubernetes/manifests/kube-apiserver.yaml
• /etc/kubernetes/manifests/kube-scheduler.yaml

by adding the following line inside the command section:

--feature-gates=EphemeralContainers=true

Create a new pod (the one I’m using is Rabbit MQ and specific to ARM architecture as I’m using a Raspberry Pi cluster here), but replace this image with anything you like as its just for testing:

Save this as pod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: rabbit
  labels:
    role: myrole
spec:
  shareProcessNamespace: true
  containers:
    - name: rabbitmq
      image: arm32v7/rabbitmq
      ports:
        - name: rabbit
          containerPort: 5672
          protocol: TCP

Create it with kubectl apply -f ./pod.yaml

Next, create an EphemeralContainer resource saving it as ephemeral-diagnostic-container.json

(Note that I’m using a Docker image I created, shoganator/rpi-alpine-tools with a bunch of diagnostic tools added, and that this image is specific to ARM architecture only). Replace the image in this file with anything else you like, e.g. busybox.

{
    "apiVersion": "v1",
    "kind": "EphemeralContainers",
    "metadata": {
            "name": "rabbit"
    },
    "ephemeralContainers": [{
        "command": [
            "bash"
        ],
        "image": "shoganator/rpi-alpine-tools",
        "imagePullPolicy": "Always",
        "name": "diagtools",
        "stdin": true,
        "tty": true,
        "terminationMessagePolicy": "File"
    }]
}

Apply this to your existing pod:

kubectl -n default replace --raw /api/v1/namespaces/default/pods/rabbit/ephemeralcontainers -f ./ephemeral-diagnostic-container.json

Describe your rabbit pod with kubectl describe pod rabbit

Ephemeral Containers:
   diagtools:
     Container ID:  docker://eb55c71f102ce3d56221934f6ebcabfd2da76204df718bd8d2573da24aecc8e9
     Image:         shoganator/rpi-alpine-tools
     Image ID:      docker-pullable://shoganator/rpi-alpine-tools@sha256:bb00f943d511c29cc2367183630940e797f5e9552683b672613bf4cb602a1c4c
     Port:          
     Host Port:     
     Command:
       bash
     State:          Running
       Started:      Sat, 16 Nov 2019 14:49:58 +0000
     Ready:          False
     Restart Count:  0
     Environment:    
     Mounts:         
 Conditions:
   Type              Status
   Initialized       True
   Ready             True
   ContainersReady   True
   PodScheduled      True

You can see now that the Ephemeral Containers section is populated with the newly running Ephemeral Container that we added into the rabbit pod.

The next step is to actually use this container to do some diagnosis / probing inside the pod. Attach to the diagtools ephemeral container inside the pod:

kubectl attach -it rabbit -c diagtools

Now you can list processes, ping items in the network, or maybe test another web service in your cluster from the context of this pod. E.g.

ps auxww
ping 192.168.192.13
curl http://hello-node1:8081
htop

Remember that with Ephemeral Containers:

• Cannot have ports, so fields such as ports, livenessProbe, readinessProbe are not able to be used.
• Setting resources is disallowed as pod resources are immutable.
  They will disappear if a pod is deleted/re-scheduled.