Ephemeral Containers are an early-state alpha feature in Kubernetes 1.16 and offer some interesting new dynamics when it comes to tooling that we can use in day-to-day Kubernetes operations.
To see this feature live, in action, check out the demo shell session below:
Before we look at Ephemeral Containers, let’s go over what a Pod is in the Kubernetes world.
Remember that a Pod in Kubernetes is a group of one or more containers (e.g. Docker containers).
With that basic tidbit of information out of the way, we’ll look at some characteristics that Pods and their containers have always had in the past:
- They’re meant to be disposable and easily replaced in a controlled manner with Deployments.
- You could not add containers to pods at runtime.
- Containers in pods can have ports assigned for network access and use things like liveness probes.
- Troubleshooting containers in pods usually meant looking at logs or using kubectl exec to get into the running container and poke around. The latter of course being useless if your container had already crashed.
So here is where I see one of the best use cases for the new Ephemeral Containers feature – troubleshooting.
Ephemeral Containers can be inserted into a live, running pod at runtime.
This means they are great for live troubleshooting of your applications. How many times have you wished your base docker image you’ve built your application image on top of has curl, dig, or even ping in some cases…
If we’ve been following best practices, we’ve kept our docker images as slim as possible, and removed as much attack surface area as possible. This usually means all the useful diagnostic tools are missing.
Ephemeral Containers are great. We can now keep a diagnostic Docker image handy with all the tools we need and live insert a diagnostic container into a running pod to troubleshoot when the time arises.
When an Ephemeral Container runs, it executes within the namespace of the target pod. So you’ll be able to access, for example, the filesystems and other resources that containers in the the pods have.
Demonstration
In order to follow along with this demo, you’ll need Kubernetes 1.16 or higher, and you’ll need to use two pod related features:
- EphemeralContainers (of course) – disabled by default in 1.16 as it’s alpha.
- PodShareProcessNamespace – for sharing the process namespace in a pod (enabled by default in 1.16 as it’s a beta feature).
To enable the Ephemeral Containers feature, edit the following configuration files on your Kubernetes master nodes and restart each master:
Enable the EphemeralContainers alpha feature gate in the following places
- /etc/kubernetes/manifests/kube-apiserver.yaml
- /etc/kubernetes/manifests/kube-scheduler.yaml
by adding the following line inside the command section:
--feature-gates=EphemeralContainers=true
Create a new pod (the one I’m using is Rabbit MQ and specific to ARM architecture as I’m using a Raspberry Pi cluster here), but replace this image with anything you like as its just for testing:
Save this as pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: rabbit
labels:
role: myrole
spec:
shareProcessNamespace: true
containers:
- name: rabbitmq
image: arm32v7/rabbitmq
ports:
- name: rabbit
containerPort: 5672
protocol: TCP
Create it with kubectl apply -f ./pod.yaml
Next, create an EphemeralContainer resource saving it as ephemeral-diagnostic-container.json
(Note that I’m using a Docker image I created, shoganator/rpi-alpine-tools with a bunch of diagnostic tools added, and that this image is specific to ARM architecture only). Replace the image in this file with anything else you like, e.g. busybox.
{
"apiVersion": "v1",
"kind": "EphemeralContainers",
"metadata": {
"name": "rabbit"
},
"ephemeralContainers": [{
"command": [
"bash"
],
"image": "shoganator/rpi-alpine-tools",
"imagePullPolicy": "Always",
"name": "diagtools",
"stdin": true,
"tty": true,
"terminationMessagePolicy": "File"
}]
}
Apply this to your existing pod:
kubectl -n default replace --raw /api/v1/namespaces/default/pods/rabbit/ephemeralcontainers -f ./ephemeral-diagnostic-container.json
Describe your rabbit pod with kubectl describe pod rabbit
Ephemeral Containers:
diagtools:
Container ID: docker://eb55c71f102ce3d56221934f6ebcabfd2da76204df718bd8d2573da24aecc8e9
Image: shoganator/rpi-alpine-tools
Image ID: docker-pullable://shoganator/rpi-alpine-tools@sha256:bb00f943d511c29cc2367183630940e797f5e9552683b672613bf4cb602a1c4c
Port:
Host Port:
Command:
bash
State: Running
Started: Sat, 16 Nov 2019 14:49:58 +0000
Ready: False
Restart Count: 0
Environment:
Mounts:
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
You can see now that the Ephemeral Containers section is populated with the newly running Ephemeral Container that we added into the rabbit pod.
The next step is to actually use this container to do some diagnosis / probing inside the pod. Attach to the diagtools ephemeral container inside the pod:
kubectl attach -it rabbit -c diagtools
Now you can list processes, ping items in the network, or maybe test another web service in your cluster from the context of this pod. E.g.
ps auxww ping 192.168.192.13 curl http://hello-node1:8081 htop
Remember that with Ephemeral Containers:
- Cannot have ports, so fields such as ports, livenessProbe, readinessProbe are not able to be used.
- Setting resources is disallowed as pod resources are immutable.
They will disappear if a pod is deleted/re-scheduled.
