This post is a published draft. I will be adding to it as I can. If you have items you think should be added, I’ve enabled the comments section for it. Happy to add your troubleshooting scenarios.

This post will cover basic troubleshooting of Crossplane. I have other posts that describe the components of Crossplane, so I won’t rehash all of that here. This is certainly not all encompassing. It is simply a list of common issues and how to resolve them.

Additional tips can be found here: link to Crossplane.io troubleshooting tips.

Everything that happens before the Managed Resource is created is controlled by the Crossplane core. If you are experiencing issues with Claims, XRs, or Compositions being created, then troubleshoot at the Crossplane core.

Issues that involve MRs not being created are almost always due to an issue with Composition. MRs not provisioning to the remote API are almost always going to be Provider related or due to the Provider not getting the MR data in the format it expects. In these cases, focus on the Composition and Provider.

The majority of Crossplane issues can be investigated through K8s events. kubectl describe and kubectl get events will satisfy 90% of the troubleshooting needs. For cluster scoped resources, Crossplane streams events to the default namespace. In cases where you absolutely can’t get the hints you need from events, you can hope for something from the logs.

For Provider log review, you can enable more verbosity for providers with a Controller Config (link to docs). Crossplane typically sends more to events than logs, but this may help in some niche situations. You can also kubectl execute into the provider and core crossplane containers to evaluate whats happening at a more granular level.

Before getting into the basics, if you see the following with a ‘jet’ based provider: the object has been modified; please apply your changes to the latest version and try again, ignore it.

Issue 1: Crossplane package won’t build (or it does build), there are little to no errors emitted. It won’t load and become healthy.

Resolution: kubectl describe lock. Look for errors. Then kubectl create each XRD and Composition manually until you find the one with the error.

Issue 2: Creating Claim or XR does not result in expected provisioning

Resolution: Begin with checking the XRD (kubectl get xrd), verify that is ready and offered if a Claim. Then kubectl describe the Claim, and then the XR. Look for errors in the events that might lead to resolution. There are often errors that are meaningless, like can’t update, try again. You will have to use Crossplane enough to figure out which you can ignore and which are more meaningful (until those errors are suppressed). Next use kubectl describe your Composition, and look for meaningful errors there. Finally, kubectl describe your Managed Resources to look for errors. If all of those are good, troubleshoot the Provider and ProviderConfig (see Issue 3).

Issue 3: Provider not provisioning Managed Resources

Resolution: kubectl get providers to check your provider is installed and healthy. kubectl get providerconfigs.<your provider name> to check your ProviderConfig is present. kubectl get po -n <your crossplane system namespace> to ensure the provider pod is running. kubectl describe <provider instance>, look for errors. kubectl describe providerconfigs.<your provider> to look for errors. Ensure your ProviderConfig credentials are configured correctly.

Issue 4: The Composition Resource Template isn’t provisioning the MR

Check to make sure your Resource Template has a name field if all others do. kubectl describe <managed resource> and evaluate errors. It’s possible that you or the api dpcs for the resource are incorrect. Check the api docs again and if that doesn’t help, kubectl get <provider crd in question> -o yaml and make sure you aren’t using a list where you should be using an object (this will require you to understand openapi v3 schema).

Catchall:

Use event emits to track from Claim through MR. Understand the way Crossplane works in that path. If you’re following the docs to a tee, and it’s not working, dig a bit deeper on CRD definitions and investigate the remote side of your provisioning (e.g. look in the cloud provider console to see what’s happening).

If none of that resolves the issue, post it on Crossplane slack and/or open an issue in the appropriate Provider repo.