THIS POST IS OUT OF DATE BASED ON ENHANCEMENTS WITHIN THE PROJECTS. SEE THIS POST FOR UPDATED GUIDANCE.
TL;DR – Use XRC and configure Argo CD to ignore XRs created by them. A GitOps platform should only track resources in the repo matching the same resources to the cluster, anything created beyond that should be the responsibility of an operator to determine sync.
Edit note: This post has undergone a fairly significant revision based on further thought and collaboration. I believe this revision best covers the topic and issue.
This topic remains somewhat unresolved from my previous post. In that post, I had to make some implementation opinions on cluster vs. namespace scoped Crossplane resources when used with Argo CD. In this post I’ll dig a little deeper into the issue and then share my current opinions on the matter.
GitOps is all about storing and deriving your infrastructure and application configurations in/from a Git repository. GitOps synchronizes a Git repo with a Kubernetes cluster, ensuring what is running matches what is defined in the Git repo.
As it stands today, Argo CD accomplishes this by labeling anything it creates in the cluster (from the repo) with the Argo CD Application name. It does this by adding a label called app.kubernetes.io/instance, with the Argo CD application name as the value. The operator created resources that show owner ref chain relationship to the top level resource will be displayed, but are not part of the sync’d evaluation. Given the challenges of rationalizing relationships via owner ref and those represented resources having no part of the sync eval, I think this is something that could be removed from the UI altogether.
Owner reference is one of two prescribed ways to define relationships between Kubernetes resources (The other being through label selection). Interestingly enough, the Deployment example above uses both. I say ‘interestingly’ because it conveys there is a reason to use one, the other, or both. Let’s take a closer look at owner references, the docs say this about it::
In Kubernetes, some objects are owners of other objects. For example, a ReplicaSet is the owner of a set of Pods. These owned objects are dependents of their owner.
…
Ownership is different from the labels and selectors mechanism that some resources also use.
…
Owner references help different parts of Kubernetes avoid interfering with objects they don’t control.
That last part, “avoid interfering with objects they don’t control” gets to the center of owner reference’s intended purpose. Specifically, it aids Kubernetes garbage collection.
Garbage collection is a collective term for the various mechanisms Kubernetes uses to clean up cluster resources.
…
Many objects in Kubernetes link to each other through owner references. Owner references tell the control plane which objects are dependent on others. Kubernetes uses owner references to give the control plane, and other API clients, the opportunity to clean up related resources before deleting an object. In most cases, Kubernetes manages owner references automatically.
Because owner reference was implemented for garbage collection to prevent unwanted deletion, there are some enforced rules governing where the relationship is permitted.
Cross-namespace owner references are disallowed by design. Namespaced dependents can specify cluster-scoped or namespaced owners. A namespaced owner must exist in the same namespace as the dependent. If it does not, the owner reference is treated as absent, and the dependent is subject to deletion once all owners are verified absent. Cluster-scoped dependents can only specify cluster-scoped owners.
Ok, the important things to take away:
- Owner reference conveys relationships between Kubernetes resources, for the purpose of garbage collection.
- Because its purpose is to protect unrelated resources during garbage collection, namepsaced resources cannot own cluster scoped resources or resources in other namespaces.
This means Argo CD is bound by the rules imposed on owner reference and therefore cannot rationalize a relationship with a resource that begins in a namespace and then creates resources at the cluster level or within another namespace.
In the case of Crossplane’s Composite Resource Claim, we create a namespaced resource that results in a cluster scoped resource called Composite Resource. From that single Composite Resource, we can have many additional Composite Resources, all of which go through a Composition, generating Managed Resources that create Provider Config Usages. That relationship cannot be rationalized in Argo CD so it doesn’t show any downstream resources from XRC. No big deal, that’s the way it could be for every resource type by just removing that feature from the UI.
Another issue comes in with how Argo CD labels resources from the repo and how Crossplane replicates labels between XRC and XR. When Crossplane replicates the labels from an XRC created by Argo CD, it replicates the label that tells Argo CD it tracks it from repo. The XR doesn’t exist in the repo so Argo CD considers the Application out of sync and wants to prune/delete the XR.
Until an enhancement is made to Argo CD for determining how it tracks something from a repo, and/or Crossplane enhances how labels are propagated from XRC to XR, we need to manually configure Argo CD to ignore the XRs created from XRC.
Fortunately, Argo CD and Crossplane are open source projects and we can all participate in the direction and evolution. Pertinent to this topic is issue #5082. I won’t speculate to what the solution should be, I don’t have nearly enough context of what has been considered and ruled out to date, nor the why. I will say that at face value, it seems relationships defined by some other method than owner reference makes a lot of sense and align better with Kubernetes in this case.
For now, My opinion is to configure Argo CD to ignore downstream CompositeResources (XRs) and start with ConpositeResourceClaim (XRC). I do still hope to see Argo CD improve its understanding of relationships between resources so we don’t have to manually configure it to ignore certain things.
If you’d like to experiment with this topic on your own, I have created two resources to make it very simple. For MacOS, a shell script that will install and configure Argo CD, Crossplane, and Gatekeeper, with a sample Azure AKS application here argo-cp-all-in-one.That script leverages the configs in this repo, where you can find more context in the README file.