Containers, VMs, and VMware – Containers in Virtual Machines – The Best of Both Worlds

Containers in Virtual Machines – The Best of Both Worlds

My plan was to follow the previous post on Harbor with a rundown of vSphere Integrated Containers. But as I began to write, I realized just the topic of the relationship between containers and VMs was fairly lengthy. I’ve decided to cover just that in this post. In the next post, I will cover VIC.

So here we go, this is where I think things get interesting.

From Session 1 of this series, we know containers are not a matter of virtualization. Containers leverage a single OS kernel with potentially multiple isolated user space applications.

The reason it is important to draw that distinction is to debunk a commonly repeated misunderstanding with regard to running containers in a VM. Running a container in a VM is not nested virtualization (running a VM in a VM). Nested virtualization is generally frowned upon because it gets us into a Matrix type construct, where management and resource allocation becomes fuzzy. It’s not necessarily a bad thing to do from a resource utilization perspective. That is a matter of how the virtualization is nested. Irrespective of that debate, a container in a VM is not nested virtualization. Containers are not VMs.

A container engine on a Linux instance only runs one kernel on the hardware. It manages the various containers’ access to the kernel through process isolation and scheduling. So, whether we run a single container or multiple on a Linux OS, the containers are sharing the same 100% of the I/O offered by the single kernel (a small amount of the resources will be consumed by the container engine). Containers are essentially applications with a small or large chunk of OS files, running on a Linux host that can’t see each other from an I/O perspective.

As we run Linux kernels in VMs today, with well established and accepted benefits, there is no reason not to run a Linux kernel with containers in a VM. We benefit from hardware abstraction, we benefit further from operating system abstraction. Make sense? Hopefully; because running containers in VMs maintains all of the benefits we have derived from virtualization already, as well as addresses some current complexity in the evolving technology.

I think confusion over the benefit of running containers in virtual machines comes from the Dev side. Developers want infrastructure in place and always available. They don’t care how this happens, just that it does. Operations is responsible for caring about how this happens. And it’s Operations that should more clearly see the benefit.

The other common point of resistance I’ve encountered is that spinning up a VM for a container takes longer. That’s a bit of a stretch to concern oneself with, but I’ll address it. As you’ll read further down, we can run one container per VM or multiple. If we run multiple, no problems, the VM is already running. If we run one per, that’s where Photon OS comes in. Photon OS is another VMware open source project and is a micro Linux kernel that when combined with VMware Instant Clone, allows us to spin a container host up blazingly fast (I know, not exactly a measurable metric, but it keeps me out of trouble, it’s fast). One of our customers spins up 30,0000 VMs in 30 minutes as part of their DR testing. This concern is not worth losing sleep over.

We wouldn’t likely run a single container per physical Linux host for production use. It makes little sense as the code base in that container would have nothing else to conflict with and it would likely be a terrible waste of capacity. There are some exceptions to this. For example, when it comes to monster workloads that can consume 100% of usable capacity.

But, we might run a single container per VM. In this model, we are still able to run multiple containers per physical host and get the benefits of container portability and tooling provided by the container technology. By running in a VM, we get all the benefits of the hypersior and the mature ecosystem of tooling that comes with it (Think back to the DevOps session in this series).

Other constructs that have needed to be created to address multi-container applications (e.g. Networking between containers) are now replaced with mature constructs that connect and orchestrate VMs. For example, firewall rules at the VM level allow us to provide firewall rules between containers.

This approach of “one VM/one container” was the initial strategy VMware went to market with when vSphere Integrated Containers was released. And it is still a model available today in VIC 1.2.

The other option is run multiple containers per VM. This approach fits better with orchestration tools like Kebrnetes and Swarm, because this is what they expect to see and have been developed to work with. So VIC 1.2 also provides this option. In this case, we still get the benefit of VM level management, but we do lose some benefit. For example, the benefit of leveraging VM networking. VMware has developed its multi-hypervisor virtual networking NSX-T to address this. Bosh (Covered more in a coming post) also fills in here.

Either way, your developers have all the Docker orchestration they love and your operations team have all of the tools they’re familiar with to manage with. Containers and “legacy” workload VMs can run side-by-side on the same capacity.

In short, you get the best of both worlds and you don’t compromise on performance when you run containers in a VM.

The above two approaches to running containers in a VM are competing to a degree. In my opinion, running one container per VM provides the greatest benefit for operations. But, it does not fit well with container specific orchestration and management tools. So we see some recent shift toward the multiple container per VM model. In this case, VMware Pivotal Container Service (VMware PKS) appears to be the current lead horse.