-
Notifications
You must be signed in to change notification settings - Fork 885
Description
PodMonitor 缺少自定义标签支持
问题概述
Higress Helm chart 创建的 PodMonitor 资源不支持添加自定义标签,这导致当 Prometheus 配置了 podMonitorSelector 时无法发现该 PodMonitor。
如果用户想让 Prometheus 发现 Higress PodMonitor,必须将标签添加到 gateway.labels,这会影响所有 Higress 资源,而不仅仅是 PodMonitor。
环境信息
- Higress 版本: 2.1.8
- Helm Chart: higress-core 2.1.8
- Kubernetes 版本:1.31.8
- 监控栈: Prometheus Operator (kube-prometheus-stack)
问题描述
当部署 Higress 并启用 gateway.metrics.enabled: true 时,创建的 PodMonitor 资源不包含 Prometheus podMonitorSelector 所需的自定义标签。
当前行为
PodMonitor 模板位于 charts/higress-core/templates/podmonitor.yaml,仅包含来自 gateway.labels 的标签:
metadata:
name: {{ printf "%s-metrics" (include "gateway.name" .) | trunc 63 | trimSuffix "-" }}
namespace: {{ .Release.Namespace }}
labels:
{{- include "gateway.labels" . | nindent 4}}
annotations:
{{- .Values.gateway.annotations | toYaml | nindent 4 }}期望行为
PodMonitor 应该支持通过 gateway.metrics.labels 配置自定义标签,类似于其他监控资源的工作方式。
实际影响
许多 Prometheus 部署使用 podMonitorSelector 来过滤要发现的 PodMonitor。例如:
# Prometheus 配置
podMonitorSelector:
matchLabels:
release: kube-prometheus-stack如果无法为 PodMonitor 添加自定义标签,用户将无法:
- 让 Prometheus 发现 Higress PodMonitor
- 收集 Higress Gateway Pod 的监控指标
- 使用 Higress 监控面板
当前解决方法
用户必须采用以下任一方法:
- 将标签添加到
gateway.labels(会将标签应用到所有 Higress 资源,而不仅仅是 PodMonitor) - 部署后手动修补 PodMonitor
- Fork 并修改 Helm chart
所有解决方法都不是最优的。
建议的解决方案
在 PodMonitor 模板中添加对 gateway.metrics.labels 的支持:
--- a/helm/core/templates/podmonitor.yaml
+++ b/helm/core/templates/podmonitor.yaml
@@ -5,6 +5,9 @@ metadata:
namespace: {{ .Release.Namespace }}
labels:
{{- include "gateway.labels" . | nindent 4}}
+ {{- with .Values.gateway.metrics.PodMonitorSelector }}
+ {{- toYaml . | nindent 4 }}
+ {{- end }}
annotations:
{{- .Values.gateway.annotations | toYaml | nindent 4 }}
spec:
diff --git a/helm/core/values.yaml b/helm/core/values.yaml使用示例
--- a/helm/core/values.yaml
+++ b/helm/core/values.yaml
@@ -528,6 +528,11 @@ gateway:
metrics:
# -- If true, create PodMonitor or VMPodScrape for gateway
enabled: false
+ # -- Selector for PodMonitor
+ # When using monitoring.coreos.com/v1.PodMonitor, the selector must match
+ # the label "release: kube-prome" is the default for kube-prometheus-stack
+ PodMonitorSelector:
+ release: kube-prome
# -- provider group name for CustomResourceDefinition, can be monitoring.coreos.com or operator.victoriametrics.com
provider: monitoring.coreos.com
interval: ""优势
- ✅ 用户可以专门为 PodMonitor 添加标签,而不影响其他 Higress 资源
- ✅ 符合 Kubernetes 和 Prometheus Operator 最佳实践
- ✅ 向后兼容(可选配置)
- ✅ 只需最小的代码更改
- ✅ 与
gateway.metrics.relabelings等其他 metrics 配置的工作方式保持一致
附加信息
此问题影响与以下系统的集成:
- Prometheus Operator
- VictoriaMetrics Operator
- 任何使用标签选择器进行服务发现的监控解决方案
相同的模式应该可能也应用于:
gateway.metrics.annotations用于 PodMonitor 注解- VMPodScrape 模板(用于 VictoriaMetrics 用户)
PodMonitor lacks custom tag support
Problem overview
The PodMonitor resource created by the Higress Helm chart does not support adding custom labels, which results in the PodMonitor not being discovered when Prometheus is configured with podMonitorSelector.
If the user wants Prometheus to discover the Higress PodMonitor, labels must be added to gateway.labels, which affects all Higress resources, not just the PodMonitor.
Environment information
- Higress version: 2.1.8
- Helm Chart: higress-core 2.1.8
- Kubernetes version: 1.31.8
- Monitoring stack: Prometheus Operator (kube-prometheus-stack)
Problem description
When deploying Higress with gateway.metrics.enabled: true enabled, the PodMonitor resource created does not contain the custom labels required by Prometheus podMonitorSelector.
Current behavior
The PodMonitor template is located in charts/higress-core/templates/podmonitor.yaml and contains only labels from gateway.labels:
metadata:
name: {{ printf "%s-metrics" (include "gateway.name" .) | trunc 63 | trimSuffix "-" }}
namespace: {{ .Release.Namespace }}
labels:
{{- include "gateway.labels" . | nindent 4}}
annotations:
{{- .Values.gateway.annotations | toYaml | nindent 4 }}Expected behavior
PodMonitor should support configuring custom labels via gateway.metrics.labels, similar to how other monitoring resources work.
Practical impact
Many Prometheus deployments use podMonitorSelector to filter which PodMonitors are discovered. For example:
# Prometheus configuration
podMonitorSelector:
matchLabels:
release: kube-prometheus-stackWithout the ability to add custom labels to PodMonitor, users will not be able to:
- Let Prometheus discover Higress PodMonitor
- Collect monitoring indicators of Higress Gateway Pod
- Use Higress monitoring panel
Current solution
Users must use one of the following methods:
- Add labels to
gateway.labels(will apply labels to all Higress resources, not just PodMonitor) - Manually patch PodMonitor after deployment
- Fork and modify the Helm chart
All solutions are not optimal.
Suggested solution
Add support for gateway.metrics.labels in PodMonitor template:
--- a/helm/core/templates/podmonitor.yaml
+++ b/helm/core/templates/podmonitor.yaml
@@ -5,6 +5,9 @@ metadata:
namespace: {{ .Release.Namespace }}
labels:
{{- include "gateway.labels" . | nindent 4}}
+ {{- with .Values.gateway.metrics.PodMonitorSelector }}
+ {{- toYaml . | nindent 4 }}
+ {{- end }}
annotations:
{{- .Values.gateway.annotations | toYaml | nindent 4 }}
spec:
diff --git a/helm/core/values.yaml b/helm/core/values.yamlUsage examples
--- a/helm/core/values.yaml
+++ b/helm/core/values.yaml
@@ -528,6 +528,11 @@ gateway:
metrics:
# -- If true, create PodMonitor or VMPodScrape for gateway
enabled: false
+ # -- Selector for PodMonitor
+ # When using monitoring.coreos.com/v1.PodMonitor, the selector must match
+ # the label "release: kube-prome" is the default for kube-prometheus-stack
+ PodMonitorSelector:
+ release: kube-prome
# -- provider group name for CustomResourceDefinition, can be monitoring.coreos.com or operator.victoriametrics.com
provider: monitoring.coreos.com
interval: ""Advantages
- ✅ Users can add tags specifically to PodMonitor without affecting other Higress resources
- ✅ Comply with Kubernetes and Prometheus Operator best practices
- ✅ Backward compatibility (optional configuration)
- ✅ Requires minimal code changes
- ✅ Consistent with the way other metrics configurations such as
gateway.metrics.relabelingswork
Additional Information
This issue affects integration with the following systems:
- Prometheus Operator
- VictoriaMetrics Operator
- Any monitoring solution that uses tag selectors for service discovery
The same pattern should probably also apply to:
gateway.metrics.annotationsfor PodMonitor annotations- VMPodScrape template (for VictoriaMetrics users)