Skip to content

【Bug】: PodMonitor 缺少自定义标签支持 || [Bug]: PodMonitor lacks custom tag support #3021

@lwpk110

Description

@lwpk110

PodMonitor 缺少自定义标签支持

问题概述

Higress Helm chart 创建的 PodMonitor 资源不支持添加自定义标签,这导致当 Prometheus 配置了 podMonitorSelector 时无法发现该 PodMonitor。
如果用户想让 Prometheus 发现 Higress PodMonitor,必须将标签添加到 gateway.labels,这会影响所有 Higress 资源,而不仅仅是 PodMonitor。

环境信息

  • Higress 版本: 2.1.8
  • Helm Chart: higress-core 2.1.8
  • Kubernetes 版本:1.31.8
  • 监控栈: Prometheus Operator (kube-prometheus-stack)

问题描述

当部署 Higress 并启用 gateway.metrics.enabled: true 时,创建的 PodMonitor 资源不包含 Prometheus podMonitorSelector 所需的自定义标签。

当前行为

PodMonitor 模板位于 charts/higress-core/templates/podmonitor.yaml,仅包含来自 gateway.labels 的标签:

metadata:
  name: {{ printf "%s-metrics" (include "gateway.name" .) | trunc 63 | trimSuffix "-" }}
  namespace: {{ .Release.Namespace }}
  labels:
    {{- include "gateway.labels" . | nindent 4}}
  annotations:
    {{- .Values.gateway.annotations | toYaml | nindent 4 }}

期望行为

PodMonitor 应该支持通过 gateway.metrics.labels 配置自定义标签,类似于其他监控资源的工作方式。

实际影响

许多 Prometheus 部署使用 podMonitorSelector 来过滤要发现的 PodMonitor。例如:

# Prometheus 配置
podMonitorSelector:
  matchLabels:
    release: kube-prometheus-stack

如果无法为 PodMonitor 添加自定义标签,用户将无法:

  1. 让 Prometheus 发现 Higress PodMonitor
  2. 收集 Higress Gateway Pod 的监控指标
  3. 使用 Higress 监控面板

当前解决方法

用户必须采用以下任一方法:

  1. 将标签添加到 gateway.labels(会将标签应用到所有 Higress 资源,而不仅仅是 PodMonitor)
  2. 部署后手动修补 PodMonitor
  3. Fork 并修改 Helm chart

所有解决方法都不是最优的。

建议的解决方案

在 PodMonitor 模板中添加对 gateway.metrics.labels 的支持:

--- a/helm/core/templates/podmonitor.yaml
+++ b/helm/core/templates/podmonitor.yaml
@@ -5,6 +5,9 @@ metadata:
   namespace: {{ .Release.Namespace }}
   labels:
     {{- include "gateway.labels" . | nindent 4}}
+    {{- with .Values.gateway.metrics.PodMonitorSelector }}
+    {{- toYaml . | nindent 4 }}
+    {{- end }}
   annotations:
     {{- .Values.gateway.annotations | toYaml | nindent 4 }}
 spec:
diff --git a/helm/core/values.yaml b/helm/core/values.yaml

使用示例

--- a/helm/core/values.yaml
+++ b/helm/core/values.yaml
@@ -528,6 +528,11 @@ gateway:
   metrics:
     # -- If true, create PodMonitor or VMPodScrape for gateway
     enabled: false
+    # -- Selector for PodMonitor
+    # When using monitoring.coreos.com/v1.PodMonitor, the selector must match
+    # the label "release: kube-prome" is the default for kube-prometheus-stack
+    PodMonitorSelector:
+      release: kube-prome
     # -- provider group name for CustomResourceDefinition, can be monitoring.coreos.com or operator.victoriametrics.com
     provider: monitoring.coreos.com
     interval: ""

优势

  1. ✅ 用户可以专门为 PodMonitor 添加标签,而不影响其他 Higress 资源
  2. ✅ 符合 Kubernetes 和 Prometheus Operator 最佳实践
  3. ✅ 向后兼容(可选配置)
  4. ✅ 只需最小的代码更改
  5. ✅ 与 gateway.metrics.relabelings 等其他 metrics 配置的工作方式保持一致

附加信息

此问题影响与以下系统的集成:

  • Prometheus Operator
  • VictoriaMetrics Operator
  • 任何使用标签选择器进行服务发现的监控解决方案

相同的模式应该可能也应用于:

  • gateway.metrics.annotations 用于 PodMonitor 注解
  • VMPodScrape 模板(用于 VictoriaMetrics 用户)

PodMonitor lacks custom tag support

Problem overview

The PodMonitor resource created by the Higress Helm chart does not support adding custom labels, which results in the PodMonitor not being discovered when Prometheus is configured with podMonitorSelector.
If the user wants Prometheus to discover the Higress PodMonitor, labels must be added to gateway.labels, which affects all Higress resources, not just the PodMonitor.

Environment information

  • Higress version: 2.1.8
  • Helm Chart: higress-core 2.1.8
  • Kubernetes version: 1.31.8
  • Monitoring stack: Prometheus Operator (kube-prometheus-stack)

Problem description

When deploying Higress with gateway.metrics.enabled: true enabled, the PodMonitor resource created does not contain the custom labels required by Prometheus podMonitorSelector.

Current behavior

The PodMonitor template is located in charts/higress-core/templates/podmonitor.yaml and contains only labels from gateway.labels:

metadata:
  name: {{ printf "%s-metrics" (include "gateway.name" .) | trunc 63 | trimSuffix "-" }}
  namespace: {{ .Release.Namespace }}
  labels:
    {{- include "gateway.labels" . | nindent 4}}
  annotations:
    {{- .Values.gateway.annotations | toYaml | nindent 4 }}

Expected behavior

PodMonitor should support configuring custom labels via gateway.metrics.labels, similar to how other monitoring resources work.

Practical impact

Many Prometheus deployments use podMonitorSelector to filter which PodMonitors are discovered. For example:

# Prometheus configuration
podMonitorSelector:
  matchLabels:
    release: kube-prometheus-stack

Without the ability to add custom labels to PodMonitor, users will not be able to:

  1. Let Prometheus discover Higress PodMonitor
  2. Collect monitoring indicators of Higress Gateway Pod
  3. Use Higress monitoring panel

Current solution

Users must use one of the following methods:

  1. Add labels to gateway.labels (will apply labels to all Higress resources, not just PodMonitor)
  2. Manually patch PodMonitor after deployment
  3. Fork and modify the Helm chart

All solutions are not optimal.

Suggested solution

Add support for gateway.metrics.labels in PodMonitor template:

--- a/helm/core/templates/podmonitor.yaml
+++ b/helm/core/templates/podmonitor.yaml
@@ -5,6 +5,9 @@ metadata:
   namespace: {{ .Release.Namespace }}
   labels:
     {{- include "gateway.labels" . | nindent 4}}
+ {{- with .Values.gateway.metrics.PodMonitorSelector }}
+ {{- toYaml . | nindent 4 }}
+ {{- end }}
   annotations:
     {{- .Values.gateway.annotations | toYaml | nindent 4 }}
 spec:
diff --git a/helm/core/values.yaml b/helm/core/values.yaml

Usage examples

--- a/helm/core/values.yaml
+++ b/helm/core/values.yaml
@@ -528,6 +528,11 @@ gateway:
   metrics:
     # -- If true, create PodMonitor or VMPodScrape for gateway
     enabled: false
+ # -- Selector for PodMonitor
+ # When using monitoring.coreos.com/v1.PodMonitor, the selector must match
+ # the label "release: kube-prome" is the default for kube-prometheus-stack
+ PodMonitorSelector:
+ release: kube-prome
     # -- provider group name for CustomResourceDefinition, can be monitoring.coreos.com or operator.victoriametrics.com
     provider: monitoring.coreos.com
     interval: ""

Advantages

  1. ✅ Users can add tags specifically to PodMonitor without affecting other Higress resources
  2. ✅ Comply with Kubernetes and Prometheus Operator best practices
  3. ✅ Backward compatibility (optional configuration)
  4. ✅ Requires minimal code changes
  5. ✅ Consistent with the way other metrics configurations such as gateway.metrics.relabelings work

Additional Information

This issue affects integration with the following systems:

  • Prometheus Operator
  • VictoriaMetrics Operator
  • Any monitoring solution that uses tag selectors for service discovery

The same pattern should probably also apply to:

  • gateway.metrics.annotations for PodMonitor annotations
  • VMPodScrape template (for VictoriaMetrics users)

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions