I teased this post earlier last year, then got distracted with life. I figure it’s time to knock it out. In this one, I’ll provide an overview for setting up a truly multi-tenant Crossplane platform with automation provided by Crossplane.
Using 100% open source software; including: Crossplane, Argo CD, vCluster, Dex, OpenLDAP,, Pinniped, and Velero. The result is a base architecture with GitOps , OIDC auth, Velero DR, and vCluster based tenanting. You could use this recipe for just about any K8s based service you wanted to automate for multiple teams.
I start with a base HA K8s cluster deployed by Kubeadm. On that cluster, I create a vcluster for ArgoCD / Crossplane, and a vCluster for Dex / Pinniped / OpenLDAP. Then use Argo CD and Crossplane to deploy additional vClusters to the base cluster. Finally, I configure those vClusters with the OIDC config pointing to the Dex / Pinniped / OpenLDAP for auth and Velero for DR.
After starting this post, I decided I was putting too much into a single post. I chose to spend some time covering OIDC and OIDC with K8s before walking through it You can read my three part series on OIDC, my posts on vCluster with OIDC, and my previous info on Crossplane and Argo CD to get the required info.
You’ll start with a primary K8s cluster running Crossplane, vCluster, and Argo CD. Configure an OIDC provider. Configure a Composition to deploy vCluster instances configured with OIDC and proper RBAC, install and configure Crossplane, Velero, and Argo CD. Then you can submit a Claim to your Git repo that kicks it all off.
These posts break down the steps to get there:
https://vrelevant.net/oidc-how-it-works/
https://vrelevant.net/oidc-devops-and-sre-level-part-2/
https://vrelevant.net/oidc-devops-and-sre-level-part-3/
https://vrelevant.net/vcluster-with-oidc/
https://vrelevant.net/vcluster-with-automated-ingress-config/
There are a few pieces that still need to be fixed in Crossplane itself. The biggest issues I’m aware of are the performance problems associated with Crossplane providers loading thousands of CRDs (I see some activity on finally addressing this). Another is a deletion order issue that leaves cloud resources dangling and requiring manual intervention to correct (This is also being worked on).
Finally, Upbound announced today that it has relaunched its SaaS Crossplane offering. They have put a lot of thought and effort into it. It is SaaS only as of now. But if you want an easy button for all of the things above, and are able to use SaaS model, that is an option. From a quick look at the announcement, there is at least one cool/valuable thing in it that isn’t available in upstream Crossplane. There is a Provider Identity config option that is apparently similar to CSP workload identity. This promises to eliminate the need to store CSP credentials in the Crossplane cluster.
The new Upbound Cloud offering doesn’t fix the deletion order or CRD scale issue. When those are fixed, it will be in upstream Crossplane. So either way you go, you’ll get those benefits if/when they are fixed.
It’s been a great ride with Crossplane, but it’s time to move on to other things.