Kube-Prometheus-Stack And ArgoCD 2.3 - How to remove a workaround
"prometheuses.monitoring.coreos.com" is invalid: metadata.annotations: Too long: must have at most 262144 bytes

!!Update!!
See my latest article on using `Server-Side apply``

This is an update of my previous article:
Because with the release of ArgoCD 2.3, we now have a better way to solve this issue.

This involves two easy steps in our recipe, with splitting the deployment in two sepereate ArgoCD Applications:
Application I: skipCrds
With the pull request #8012, the team around ArgoCD introduced following cool feature: "feat: add skipCrds flag for helm charts". This means we can skip the installation of CRDs.
What are CRS, short refresher from the official kubernetes documentation:
Custom Resources
Custom resources are extensions of the Kubernetes API. This page discusses when to add a custom resource to your Kubernetes cluster and when to use a standalone service. It describes the two methods for adding custom resources and how to choose between them.
Helm installs custom resource definitions in the crds folder by default if they are not existing and with skipCrds we can now control this behaviour.
To get kube-prometheus-stack working, we will set this flag to false. The first application defintions looks like this:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: kube-prometheus-stack
annotations:
argocd.argoproj.io/sync-wave: "3"
spec:
destination:
name: in-cluster
namespace: monitoring
project: default
source:
repoURL: 'https://prometheus-community.github.io/helm-charts'
targetRevision: 34.1.1
helm:
skipCrds: true
values: |-
#Snip the values
chart: kube-prometheus-stack
syncPolicy:
automated:
prune: true
selfHeal: true
Application II: Sync options Replace
The second part, is to use Replace then by default, ArgoCD executes kubectl apply operation to apply the configuration stored in Git. This is now not suitable as the resource spec might is too big and won't fit into kubectl.kubernetes.io/last-applied-configuration.
If the Replace=true sync option is set, ArgoCD will use kubectl replace or kubectl create command to apply changes.
The second application defintion looks like this:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: kube-prometheus-stack-crds
annotations:
argocd.argoproj.io/sync-wave: "2"
spec:
destination:
name: in-cluster
namespace: monitoring
project: default
source:
repoURL: https://github.com/prometheus-community/helm-charts.git
path: charts/kube-prometheus-stack/crds/
targetRevision: kube-prometheus-stack-34.1.1
directory:
recurse: true
syncPolicy:
syncOptions:
- CreateNamespace=true
- Replace=true
automated:
prune: true
selfHeal: true
See the documentation for more details on sync options.

The End
No external code this time

About the author
Cloud Native Pilgrim | Kubernetes Enthusiast | Serverless Believer | Customer Experience Architect @ Pulumi | (he/him) | CK{A,AD} |
Comments
Join the discussionEngin Diri Hi, I would like to ask about this approach. If I am not using helm, will this 2 Applications.yaml deployment work? Thanks!
Yes, or you can use the SSA feature in the latest version of ArgoCD -> https://argo-cd.readthedocs.io/en/latest/proposals/server-side-apply/
Thanks for sharing. I came up with a similar solution, and it is nice to see I'm not alone on this approach.
Engin Diri Any reason you choose such a large number difference for the sync-wave (ex. 2 and 10)? I'm doing things slightly different where I have the CRD's create the namespace, which prevents the other argocd application from running.
venkata mutyala Just random to show that CRD app has to run before and the create namespace flag went lost here :D Will add it, thanks pointing out!




