Search for a command to run...
"prometheuses.monitoring.coreos.com" is invalid: metadata.annotations: Too long: must have at most 262144 bytes
Cloud Native Pilgrim | Kubernetes Enthusiast | Serverless Believer | Customer Experience Architect @ Pulumi | (he/him) | CK{A,AD} |
No comments yet. Be the first to comment.
GPU scheduling in Kubernetes has always felt like buying a mansion when you need a studio apartment. A small inference workload that needs 2GB of GPU memory gets scheduled on an entire 80GB A100, and there's nothing you can do about it. The device pl...
TL;DR: The code https://github.com/dirien/quick-bites Nothing is more controversial in the Kubernetes community than whether to use Helm or Kustomize. I always advocate the philosophy of using the right tool for the right job. It avoids the problem...
TL;DR Le code https://github.com/dirien/quick-bites Introduction This article is part three of my series on secret management on Kubernetes with the help of Pulumi. In my first article, we talked about the Sealed Secrets controller. The second arti...
With AWS Lambda
Disaster Recovery, Data Migration made easy!
Please check this new article, if you use ArgoCD > 2.3 to not use the described workaround anymore!
Currently, I am part of a very huge and interesting project. While I worked on the installation of the kube-prometheus-stack in run full-speed into following strange error message:
one or more objects failed to apply, reason:
CustomResourceDefinition.apiextensions.k8s.io "prometheuses.monitoring.coreos.com"
is invalid: metadata.annotations: Too long: must have at most 262144 bytes,unable to
recognize "/dev/shm/987607118":
no matches for kind "Prometheus" in version "monitoring.coreos.com/v1"
After some very fruitless attempts to figure it, I asked Dr. Google for help.
I did found some GitHub issues, regarding this topic. Let me share, the one I took some of my ideas to create this workaround.
https://github.com/prometheus-community/helm-charts/issues/1500
The problem is, that ArgoCD use kubectl apply to deploy the manifests, which stores the entire manifest in an annotation. As a result you get big CustomResourceDefinition and voilà, it breaks!
The way I created my workaround now is: I create a kustomization file with following content:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- https://raw.githubusercontent.com/prometheus-community/helm-charts/kube-prometheus-stack-30.1.0/charts/kube-prometheus-stack/crds/crd-alertmanagerconfigs.yaml
- https://raw.githubusercontent.com/prometheus-community/helm-charts/kube-prometheus-stack-30.1.0/charts/kube-prometheus-stack/crds/crd-alertmanagers.yaml
- https://raw.githubusercontent.com/prometheus-community/helm-charts/kube-prometheus-stack-30.1.0/charts/kube-prometheus-stack/crds/crd-podmonitors.yaml
- https://raw.githubusercontent.com/prometheus-community/helm-charts/kube-prometheus-stack-30.1.0/charts/kube-prometheus-stack/crds/crd-probes.yaml
- https://raw.githubusercontent.com/prometheus-community/helm-charts/kube-prometheus-stack-30.1.0/charts/kube-prometheus-stack/crds/crd-prometheuses.yaml
- https://raw.githubusercontent.com/prometheus-community/helm-charts/kube-prometheus-stack-30.1.0/charts/kube-prometheus-stack/crds/crd-prometheusrules.yaml
- https://raw.githubusercontent.com/prometheus-community/helm-charts/kube-prometheus-stack-30.1.0/charts/kube-prometheus-stack/crds/crd-servicemonitors.yaml
- https://raw.githubusercontent.com/prometheus-community/helm-charts/kube-prometheus-stack-30.1.0/charts/kube-prometheus-stack/crds/crd-thanosrulers.yaml
Using the remote locations to not need to download them. Take care to pin the version, to the currently used version of the kube_prometheus_stack helm chart.
Then I called the kustomize script, via a dedicated pipeline kube-prometheus-stack-workaround.yaml.
kubectl create -k controlplane/service/monitoring/kube-prometheus-stack/workaround/kube-prometheus-stack-crds
After this, I could then deploy the Helm Chart without a problem.
I hope this will be fixed soon, as I don't want to keep this workaround to long in my productive environment.
