x509

Regain Cluster Access After Expired Kubernetes Certificates

by

Recently I had to renew expired kubernetes certificates on my home lab cluster after getting locked out from managing it. I wasn’t tracking their age and all of a sudden I found them expired. Kubeadm has a feature to auto-renew certificates during control plane upgrades. Unfortunately I had not done an upgrade on my home cluster in the last year. I realised when issuing a kubectl command to the cluster and receiving an error along the lines of x509: certificate has expired or is not yet valid.

Preparation

To regain access I needed to SSH onto a master node in the cluster and do the following:

Move / backup old certificate and kubeadm config files:

sudo mv /etc/kubernetes/pki/apiserver.key /etc/kubernetes/pki/apiserver.key.old
sudo mv /etc/kubernetes/pki/apiserver.crt /etc/kubernetes/pki/apiserver.crt.old
sudo mv /etc/kubernetes/pki/apiserver-kubelet-client.crt /etc/kubernetes/pki/apiserver-kubelet-client.crt.old
sudo mv /etc/kubernetes/pki/apiserver-kubelet-client.key /etc/kubernetes/pki/apiserver-kubelet-client.key.old
sudo mv /etc/kubernetes/pki/front-proxy-client.crt /etc/kubernetes/pki/front-proxy-client.crt.old
sudo mv /etc/kubernetes/pki/front-proxy-client.key /etc/kubernetes/pki/front-proxy-client.key.old

sudo mv /etc/kubernetes/admin.conf /etc/kubernetes/admin.conf.old
sudo mv /etc/kubernetes/kubelet.conf /etc/kubernetes/kubelet.conf.old
sudo mv /etc/kubernetes/controller-manager.conf /etc/kubernetes/controller-manager.conf.old
sudo mv /etc/kubernetes/scheduler.conf /etc/kubernetes/scheduler.conf.old

Renew Expired Kubernetes Certificates

Now use kubeadm to renew all certificates.

sudo kubeadm alpha certs renew all

Inspect the generated certificates under /etc/kubernetes/pki to make sure they are generated correctly. E.g. expiry dates a year in the future, etc… For example, checking the new kube api server certificate details:

sudo openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text -certopt no_subject,no_version,no_serial,no_signame,no_issuer,no_pubkey,no_sigdump,no_header,no_aux
Renew Expired Kubernetes Certificates - checking new certificate details

Regenerate Kubeconfig Files

Now you can regenerate the kubeconfig files for the kubelet, controller-manager, scheduler, etc…

sudo kubeadm alpha phase kubeconfig all --apiserver-advertise-address 10.0.0.50

Note: you can leave the --apiserver-advertise-address option off, but the value used will default to your system’s default network interface.

Check that all the .conf files under /etc/kubernetes are re-created. i.e.

  • admin.conf
  • kubelet.conf
  • controller-manager.conf
  • scheduler.conf

You can also update your kubectl config with the admin.conf file that was generated if that is the config/context you were using from the master node.

mv .kube/config .kube/config.old
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
sudo chmod 777 $HOME/.kube/config
export KUBECONFIG=.kube/config

If your kube services have stopped on the master (kubelet etc) as a result of expired certificates and having tried to restart services recently (as was my case), reboot the master node at this point. All services that were failing to start because of expired certificates should now start correctly.

Reconnect and check everything is working as expected:

kubectl get nodes
kubeadm token list