kubectl "x509: certificate signed by unknown authority": your kubeconfig doesn't trust this cluster
Fix kubectl "x509: certificate signed by unknown authority": get a fresh kubeconfig for a recreated cluster, compare the CA, and handle proxies that inspect TLS.
by OhSudo ··
Not sure this is your cause?Find it step by step: Pod not running →
Part of a learning path:Kubernetes: debugging pods
Unable to connect to the server: tls: failed to verify certificate: x509: certificate signed by unknown authorityOlder versions of kubectl leave out the tls: failed to verify certificate part. Either way, kubectl reached the API server, and the server showed a certificate that isn’t signed by the authority your kubeconfig trusts. Each kubeconfig carries its cluster’s certificate authority (CA), and here they don’t match.
The usual reason is mundane: the cluster was recreated, and you still have the old kubeconfig.
- kubectltrusts the CA in your kubeconfigconnects
- API serverpresents its certificatechecked against the CA(fails)
- Signed by another CAx509: unknown authority(breaks here)
TL;DR
Get a fresh kubeconfig from wherever the cluster lives:
aws eks update-kubeconfig --name my-cluster --region eu-central-1az aks get-credentials --resource-group my-rg --name my-clustergcloud container clusters get-credentials my-cluster --region europe-west1kind export kubeconfig --name my-clusterOn k3s, the file is /etc/rancher/k3s/k3s.yaml, and on kubeadm, /etc/kubernetes/admin.conf.
1. Which cluster, and which CA?
kubectl config current-contextkubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}'Check that’s the cluster you meant: a stale context pointing at an address now used by another cluster gives exactly this error.
Then compare the CA in your kubeconfig with the certificate the server shows:
kubectl config view --minify --raw -o jsonpath='{.clusters[0].cluster.certificate-authority-data}' | base64 -d | openssl x509 -noout -subject -enddateecho | openssl s_client -connect 203.0.113.10:6443 2>/dev/null | openssl x509 -noout -issuer -enddateThe server’s issuer should match your CA’s subject. If it doesn’t, your kubeconfig is for another cluster, or an older one. The Base64 decoder also reads certificate-authority-data, if you’d rather paste it.
2. A recreated cluster
Every new cluster gets a new CA, even with the same name and address. So after recreating it, in Terraform, with kind delete and kind create, or with a reinstall of k3s, the old kubeconfig fails. Get a fresh one, with the commands in the TL;DR.
Copying a kubeconfig from a server, check the server address in it too. k3s writes https://127.0.0.1:6443, which you need to change to the server’s real address on your laptop. And the address you use must be one the certificate covers, or you get a different message, described in section 5.
3. A proxy that inspects TLS
Company networks often use a proxy that opens TLS traffic, and signs it again with its own CA. kubectl sees that CA, which isn’t the cluster’s. Check:
env | grep -i proxyecho | openssl s_client -connect 203.0.113.10:6443 -proxy proxy.example.com:3128 2>/dev/null | openssl x509 -noout -issuerAn issuer naming your company’s security product confirms it. Exclude the cluster from the proxy:
export NO_PROXY=$NO_PROXY,203.0.113.10,.eks.amazonaws.comOr ask for the cluster’s address to be excluded from inspection on the proxy itself. Adding the proxy’s CA to the kubeconfig would work, and would also hide every future problem of this kind.
4. In CI and in pods
A pipeline that uses a kubeconfig stored as a secret has the same problem after a cluster is recreated: update the secret. In GitHub Actions or GitLab CI, prefer the cloud provider’s login step, which builds a fresh kubeconfig on each run.
Inside a pod, the service account’s CA is mounted at /var/run/secrets/kubernetes.io/serviceaccount/ca.crt, and client libraries use it automatically. An image that bundles its own kubeconfig can be stale.
5. Similar messages from the same place
x509: certificate is valid for 10.96.0.1, 192.168.1.10, not 203.0.113.10The CA is right, and the address you used isn’t in the certificate. Use an address or name it lists, or add yours to the certificate: with kubeadm, apiServer.certSANs, and with k3s, --tls-san.
x509: certificate has expired or is not yet validCheck your clock first, then the cluster’s certificates: kubeadm certs check-expiration on kubeadm. See also certificate has expired.
Warning
Don’t use insecure-skip-tls-verify: true or --insecure-skip-tls-verify. With it, kubectl sends your credentials to whoever answers, and these credentials often have full control of the cluster. Fix the CA, which is one command in almost every case.
Shell, we have a problem!
Similar, but different: The connection to the server localhost:8080 was refused means kubectl found no kubeconfig at all. error: You must be logged in to the server (Unauthorized) means TLS worked, and your credentials didn’t. Forbidden means you’re logged in, without the rights. Outside Kubernetes, see unable to get local issuer certificate.
$ ./practice.sh
Practice this in the terminal
Open the Terminal Playground with this article’s scenario loaded. It is safe and runs in your browser.
Found this useful? Share it.
Found a mistake? Let us know at contact@ohsudo.com.