Customising your EKS cluster DNS and the CoreDNS vs KubeDNS configuration differences

by

In the past I’ve used the excellent kops to build out Kubernetes clusters. The standard builds always made use of the kube-dns cluster addon. With EKS and CoreDNS things are a little different.

With kube-dns, I got used to using configMaps to customise DNS upstream servers and stub domains using the standard kube-dns configuration format which looks something like this:

apiVersion: v1
kind: ConfigMap
metadata:
  name: kube-dns
  namespace: kube-system
data:
  stubDomains: |
    {"ec2.internal": ["10.0.0.2"], "shogan.co.uk": ["10.20.0.200"]}
  upstreamNameservers: |
    ["8.8.8.8", "8.8.4.4"]

Amazon EKS and CoreDNS

However recently I’ve started doing a fair bit of Kubernetes cluster setups and configurations using Amazon EKS. I found that EKS and CoreDNS is now the standard and requires a different kind of configuration format which looks something like this:

apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        health
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          upstream
          fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        proxy . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }
    shogan.co.uk:53 {
        errors
        cache 30
        forward . 10.20.0.200
    }
    ec2.internal:53 {
        errors
        cache 30
        forward . 10.0.0.2
    }
kind: ConfigMap
metadata:
  labels:
    eks.amazonaws.com/component: coredns
    k8s-app: kube-dns
  name: coredns
  namespace: kube-system

To add your own custom stub domain nameservers with CoreDNS, the task becomes a case of editing the CoreDNS ConfigMap called coredns in the kube-system namespace.

Add your stub domain configuration blocks after the default .:53 section, with the forward property pointing to your custom DNS nameserver.

Once you’re done adding the new configuration, restart your CoreDNS containers. You can do this gracefully by executing the following in your CoreDNS containers:

kubectl exec -n kube-system coredns-pod-name-x -- kill -SIGUSR1 1

Alternatively, roll your CoreDNS pods one at a time.

Last of all, you’ll want to test the name resolution in a test container using a tool like dig. Your container /etc/resolv.conf files should usually be pointing at the IP address of your CoreDNS Cluster Service. So they’ll talk to the CoreDNS service for their usual look up queries, and CoreDNS should now be able to resolve your custom stub domain records but referring to your custom forwarded nameservers.

Apart from the different configuration format, there seem to be some fairly significant differences between CoreDNS and kube-dns. In my opinion, it would seem that overall CoreDNS is the better, more modern choice. Some of the benefits it enjoys over kube-dns are:

  • CoreDNS has multi-threaded design (leveraging Go)
  • CoreDNS uses negative caching whereas kube-dns does not (this means CoreDNS can cache failed DNS queries as well as successful ones, which overall should equal better speed in name resolution). It also helps with external lookups.
  • CoreDNS has a lower memory requirement, which is great for clusters with smaller worker nodes

Hopefully this helps when it comes to configuring EKS and CoreDNS. For more information, there is a great article that goes into the details of the differences of CoreDNS and kube-dns here.

Leave a Comment