# Kube-Prometheus-Stack And ArgoCD - How Workarounds Are Born

## !!Update!!

Please check this new article, if you use ArgoCD > 2.3 to not use the described workaround anymore!

%[https://blog.ediri.io/kube-prometheus-stack-and-argocd-23-how-to-remove-a-workaround]

## Situation

Currently, I am part of a very huge and interesting project. While I worked on the installation of the `kube-prometheus-stack` in run full-speed into following strange error message:

```bash
one or more objects failed to apply, reason: 
CustomResourceDefinition.apiextensions.k8s.io "prometheuses.monitoring.coreos.com"
is invalid: metadata.annotations: Too long: must have at most 262144 bytes,unable to
recognize "/dev/shm/987607118": 
no matches for kind "Prometheus" in version "monitoring.coreos.com/v1"
```

![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1642706179448/1nx6iPu_5m.png)

After some very fruitless attempts to figure it, I asked Dr. Google for help.

![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1642705844106/Br3oI1KgE.png)

I did found some GitHub issues, regarding this topic. Let me share, the one I took some of my ideas to create this workaround.

https://github.com/prometheus-community/helm-charts/issues/1500

## Workaround

The problem is, that `ArgoCD` use `kubectl apply` to deploy the manifests, which stores the entire manifest in an annotation. As a result you get big CustomResourceDefinition and voilà, it breaks!

![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1642706161913/bvAb_w48F.png)

The way I created my workaround now is:  I create a `kustomization` file with following content:

```yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - https://raw.githubusercontent.com/prometheus-community/helm-charts/kube-prometheus-stack-30.1.0/charts/kube-prometheus-stack/crds/crd-alertmanagerconfigs.yaml
  - https://raw.githubusercontent.com/prometheus-community/helm-charts/kube-prometheus-stack-30.1.0/charts/kube-prometheus-stack/crds/crd-alertmanagers.yaml
  - https://raw.githubusercontent.com/prometheus-community/helm-charts/kube-prometheus-stack-30.1.0/charts/kube-prometheus-stack/crds/crd-podmonitors.yaml
  - https://raw.githubusercontent.com/prometheus-community/helm-charts/kube-prometheus-stack-30.1.0/charts/kube-prometheus-stack/crds/crd-probes.yaml
  - https://raw.githubusercontent.com/prometheus-community/helm-charts/kube-prometheus-stack-30.1.0/charts/kube-prometheus-stack/crds/crd-prometheuses.yaml
  - https://raw.githubusercontent.com/prometheus-community/helm-charts/kube-prometheus-stack-30.1.0/charts/kube-prometheus-stack/crds/crd-prometheusrules.yaml
  - https://raw.githubusercontent.com/prometheus-community/helm-charts/kube-prometheus-stack-30.1.0/charts/kube-prometheus-stack/crds/crd-servicemonitors.yaml
  - https://raw.githubusercontent.com/prometheus-community/helm-charts/kube-prometheus-stack-30.1.0/charts/kube-prometheus-stack/crds/crd-thanosrulers.yaml
```
Using the remote locations to not need to download them. Take care to pin the version, to the currently used version of the `kube_prometheus_stack` helm chart.

Then I called the `kustomize` script, via a dedicated pipeline `kube-prometheus-stack-workaround.yaml`.

```
kubectl create -k controlplane/service/monitoring/kube-prometheus-stack/workaround/kube-prometheus-stack-crds
``` 

After this, I could then deploy the Helm Chart without a problem.

## Le end

I hope this will be fixed soon, as I don't want to keep this workaround to long in my productive environment.

![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1642706821236/i1VFfDk2h.png)