Choosing whether to use the client-side load balancing built into the kube-apiserver’s etcd client or to use an external LB/proxy is not as simple as one might expect. I’ve discussed this with a lot of people and it seems there are no well defended opinions. I know there is one out there, just haven’t found it yet.
The etcd client used by kube-apiserver has client-side load balancing logic builtin. So it can be pointed to multiple etcd cluster nodes and handle endpoint disruption.
The gRPC docs list the following pros/cons of either option:
Client-Side LB
Pros:
- High performance because elimination of extra hop
Cons:
- Complex client
- Client keeps track of server load and health
- Client implements load balancing algorithm
- Per-language implementation and maintenance burden
- Client needs to be trusted, or the trust boundary needs to be handled by a lookaside LB.
External/Proxy LB
Pros:
- Simple client
- No client-side awareness of backend
- Works with untrusted clients
Cons:
- LB is in the data path
- Higher latency
- LB throughput may limit scalability
Knowing the Kubernetes etcd client addresses all of the cons listed for the client-side method, I am hard pressed to think it would be better to use an external LB for kube-apiserver to etcd. That said, I see a single pro in this specific case.
We will have likely chosen external etcd to scale etcd and the K8s control plane independently. If we are using client-side LB with K8s, and we scale the etcd cluster, we will need to update the kube-apiserver config for each controller.
That single pro is the only thing I can come up with for external LB. I’m certain there are facets I’m not seeing, and I will update this post if I become aware of any additional. But this single benefit does not warrant sacrificing performance in my opinion.
etcd docs refer to an L7 gRPC Proxy service with this description: The proxy is designed to reduce the total processing load on the core etcd cluster. For horizontal scalability, it coalesces watch and lease API requests. To protect the cluster against abusive clients, it caches key range requests. I have not been successful with getting it to work.
With that out of the way, let’s do it anyways 🙂 (Because I searched for an example and found none. I suspect because it makes no sense, or I’m just terrible at Googling stuff).
I’ll walk through configuring an external LB for etcd and then pointing kube-apiserver to it. I’ll use an HA L4 LB via HAProxy along with a virtual IP managed by Keepalived.
As always, the steps are in a git repo located here.