In this post, I’m taking on the task of getting cluster object properties to a container at initialization. Specifically, values not exposed by the downward api. I am addressing a very specific need to add region and az specific values to a replicated database instance. But this pattern can be used for many other cases.

I am working with a replicated YugaByte redis database, in a K8s cluster spanned across AWS and GCP, with a micro-service app leveraging it for the shopping cart service. If you’ve worked with YugaByte, or other replicated databases, you’ll likely be familiar with  conveying the az, rack, host. that the workload is running in via metadata. In this case, I wanted to dynamically add the metadata to each pod at scheduling to convey the CSP, region, and availability zone they were scheduled to. Continue reading →

A couple of years back, I endeavored to create a proof of concept for predictive auto-scaling of K8s clusters. At the time (k8s 1.10), there really wasn’t a way to easily and ‘automatedly’ scale a cluster out and back in. Flash forward to today, and I can now leverage the work of the Cluster API project to solve that problem.

Cluster API (CAPI) delivers the capability of a K8s cluster managing the life cycle of other K8s clusters. Bit of a chicken and egg at first glance, but if you utilize a low friction cluster bootstrapper (e.g. Kubeadm) for the first cluster, you can then deploy and manage the rest of your clusters from this first instance. Better yet, we could leverage a cluster that was pre-built for us by a K8s as a service provider. Continue reading →

In this post, I was going to cover two controller kinds (Deployment and StatefulSet), the resource kinds PersistentVolume, PersistentVolumeClaim, and StorageClass., and go over examples of non-persistent and persistent service delivery. Then I realized it was going to be way too long. So I’ll cover the basics of stateless/stateful and storage in this post. Next post, I’ll cover Deployment and StatefulSet controllers and provide some examples of use. Continue reading →

In the previous post, I covered the concept of a K8s Service and the clusterIP  service. ClusterIP is  a method to create a stable IP address with a  DNS A record for the service, load balance requests to endpoint pod replicas (endpoints), and is only exposed within the cluster. In this post, I’ll cover the remaining service kinds: of headless, nodePort, and LoadBalancer. I’ll also cover non-service methods of exposing a pod via Ingress Controller and hostNetwork,\hostPort.

The Headless service, like ClusterIP,  is an in-cluster only addressing component. I’ll go into this in more detail when I cover the use of StatefulSet in a future post. A headless service is created by defining clusterIP field as ‘none’ in a Service spec. Continue reading →

In the previous two posts, I went over the basic networking components of K8s, and how a packet flows within the cluster. I discussed the node and pod CIDRs, but purposely avoided getting deeper into the ClusterIP/Service CIDR ( –service-cluster-ip-range). Reason being, we need to cover the concept of endpoints and services first.

While pods in a cluster can all communicate with each other directly via their unique pod IP addresses, it isn’t an  ideal practice to initiate communication directly to a pod IP. As we know,  a pod is ephemeral and potentially short lived. There is no guarantee the IP address will remain the same across recreation. In fact, K8s will always attempt change it.We also wouldn’t achieve the benefits of load balancing across pod replicas if we initiated communication by specific pod IP. Continue reading →

In my previous post, I covered the basics of Kubernetes networking. I thought it would be useful to follow up with a visual walk through of a packet’s path from pod to a remote node. In this example, I’m using K8s deployed with one master, three worker nodes, and nsx-t networking at both the node and pods level.

You can click the images in this post to open a larger view in a new tab. As you can see, I have three worker nodes with IP addrs 172.15.0.3 – 172.15.0.5. The node IP addressing for this cluster is configured using the 172.15.0.0/24 CIDR.

Next, let’s look at one of the pods I have running. This is the kube-dns pod. As you can see, it’s running in the kube-system namespace, on worker node 172.15.0.3 (As seen in the list of worker nodes above), has an IP addr assigned of 172.16.1.2, and hosts a container called kubedns. Continue reading →

I’ve been procrastinating with preparing for my CNCF Certified Kubernetes Administrator certification. Figure it was time to get to it and thought a series of blog posts on the various topics I’m digging into would be of interest to others.

Beginning with K8s networking, I’ll go into the details of the various layers of networking, how they come together, and how k8s leverages them to provide us with functioning container services.

This is a big topic, so I’ve decided to take a multi-post approach with it. I’ll start with basic networking of nodes and pods. I’ll cover network policies in another post. And then I’ll cover services, deployments, service discovery, etc. in a final post. Continue reading →

If you work with VMware tech at all, it’s unlikely you’ve not heard the buzz related to the K8s announcements by now. And I’ve been waiting a long time to be able to openly discuss them. Gladly, the day has finally arrived.

Tanzu (pronounced tahn-zu) is a brand name announcement. Not too exciting but conveys a direction to bring the various cloud native product lines together. Moving forward, we’ll see all products related to cloud native branded as Tanzu ‘something’. Continue reading →

In building toward a k8s predictive auto-scale capability, I’ve built some simple constructs with virtual machines running on vSphere with kubeadm installed k8s. If you’ve followed the series, you know I reached the anticipated point of manually installed k8s becoming too inefficient to operate and had decided to implement VMware PKS to address that pain point.

Over the past year, many blogs have posted instructions on how to install and configure PKS. I suggest following the VMware and/or Pivotal online documentation. PKS is rapidly developed and released, and it doesn’t take long for a blog post on installation to become out of date. So I won’t post installation instructions here. Continue reading →

In the previous posts, I detailed the two main functions for performing the auto-scaling procedure. One to scale the K8s VM backed cluster across additional physical hosts, and one to scale the K8s pod deployment across the added hosts/nodes. The predictive trigger of these functions was to be the focus of this post.

As time passed and work took me away from this project, I am now without my former lab setup to test it. I could rebuild the lab and continue  the final bit of code that predicts a pattern in past CPU demand and calls the functions. But for my sanity’s sake, I’m going to pass on that for now and move on to the next logical progression in this series. Continue reading →