GCP-447: inject token-minter as native sidecar init container by cristianoveiga · Pull Request #7965 · openshift/hypershift

cristianoveiga · 2026-03-13T21:50:51Z

What this PR does / why we need it:

Changes InjectTokenMinterContainer in the cpov2 framework to inject the token-minter as a https://kubernetes.io/docs/concepts/workloads/pods/sidecar-containers/ (RestartPolicy: Always) with a StartupProbe that blocks main containers until the token file exists. This eliminates a race condition where the main container starts before the token is written, causing fatal crashes (observed in GCP CCM CI).

For backwards compatibility, the management cluster K8s version is detected at startup via ServerVersion(). On K8s >= 1.29 (where the SidecarContainers feature gate is beta and enabled by default), the native sidecar pattern is used. On older clusters, the current regular sidecar injection is preserved.

Which issue(s) this PR fixes:

Fixes GCP-447

Special notes for your reviewer:

The race condition has only been observed in GCP CI so far, but the underlying issue exists for all providers using InjectTokenMinterContainer and could surface for other cloud providers in the future.

The second commit removes the GCP CCM crash toleration from podCrashTolerations in the e2e test suite, as it is no longer needed.

Checklist:

Subject and description added to both, commit and PR.
Relevant issues have been referenced.
This change includes docs.
This change includes unit tests.

Summary by CodeRabbit

Release Notes

New Features
- Added support for native Kubernetes sidecar containers on management clusters running Kubernetes 1.29+, enabling more efficient container initialization.
- Enhanced token minter container injection to intelligently adapt between native sidecars, init containers, and regular sidecars based on platform capabilities.
Changes
- Updated GCP cloud controller manager deployment labels for consistency.
Tests
- Added comprehensive test coverage for native sidecar container injection across multiple scenarios and Kubernetes versions.

openshift-ci-robot · 2026-03-13T21:50:55Z

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

openshift-ci · 2026-03-13T21:50:56Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

coderabbitai · 2026-03-13T21:51:03Z

Important

Review skipped

Auto reviews are limited based on label configuration.

🚫 Review skipped — only excluded labels are configured. (1)

do-not-merge/work-in-progress

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Pro

Run ID: e8564532-a4e8-4649-adcd-4390a41d438d

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

@coderabbitai resume to resume automatic reviews.
@coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

▶️ Resume reviews
🔍 Trigger review

📝 Walkthrough

Walkthrough

This change introduces support for Kubernetes native sidecar containers (available in Kubernetes 1.29+) across the control plane operator. It detects native sidecar capability through management cluster version discovery, propagates this flag through the reconciler and control plane context, and uses it in token minter container injection logic to conditionally inject containers as native sidecars with startup probes or as regular sidecar containers. The feature also includes test coverage for various injection scenarios and removes a toleration entry for the GCP cloud controller manager.

Sequence Diagram(s)

sequenceDiagram
    participant main as main.go
    participant cap as Capability Detection
    participant reconciler as HostedControlPlaneReconciler
    participant context as ControlPlaneContext
    participant injector as TokenMinterContainerInjector
    participant pod as PodSpec

    main->>cap: DetectManagementClusterCapabilities(discoveryClient)
    cap->>cap: supportsNativeSidecarContainers(k8s >= 1.29)
    cap-->>main: CapabilityNativeSidecarContainers flag
    
    main->>reconciler: Initialize with NativeSidecarContainersEnabled flag
    
    reconciler->>context: Create ControlPlaneContext with NativeSidecarContainersEnabled
    
    context->>injector: injectContainer(nativeSidecarsEnabled, podSpec, container, ...)
    
    alt nativeSidecarsEnabled
        injector->>pod: Add container to InitContainers with RestartPolicy: Always
        injector->>pod: Add StartupProbe (check token file existence)
    else not nativeSidecarsEnabled
        injector->>pod: Add container to Containers as regular sidecar
    end
    
    injector->>pod: Mount volume on first container in PodSpec

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

openshift-ci-robot · 2026-03-13T21:51:10Z

@cristianoveiga: An error was encountered searching for bug GCP-447 on the Jira server at https://issues.redhat.com. No known errors were detected, please see the full error message for details.

Full error message.


No response returned: Get "https://issues.redhat.com/rest/api/2/issue/GCP-447": GET https://issues.redhat.com/rest/api/2/issue/GCP-447 giving up after 5 attempt(s)

Please contact an administrator to resolve this issue, then request a bug refresh with /jira refresh.

Details

In response to this:

What this PR does / why we need it:

Changes InjectTokenMinterContainer in the cpov2 framework to inject the token-minter as a https://kubernetes.io/docs/concepts/workloads/pods/sidecar-containers/ (RestartPolicy: Always) with a StartupProbe that blocks main containers until the token file exists. This eliminates a race condition where the main container starts before the token is written, causing fatal crashes (observed in GCP CCM CI).

For backwards compatibility, the management cluster K8s version is detected at startup via ServerVersion(). On K8s >= 1.29 (where the SidecarContainers feature gate is beta and enabled by default), the native sidecar pattern is used. On older clusters, the current regular sidecar injection is preserved.

Which issue(s) this PR fixes:

Fixes GCP-447

Special notes for your reviewer:

The race condition has only been observed in GCP CI so far, but the underlying issue exists for all providers using InjectTokenMinterContainer and could surface for other cloud providers in the future.

The second commit removes the GCP CCM crash toleration from podCrashTolerations in the e2e test suite, as it is no longer needed.

Checklist:

Subject and description added to both, commit and PR.

Relevant issues have been referenced.

This change includes docs.

This change includes unit tests.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

cristianoveiga · 2026-03-13T21:51:20Z

@coderabbitai review

coderabbitai · 2026-03-13T21:51:32Z

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

openshift-ci-robot · 2026-03-13T21:58:10Z

@cristianoveiga: An error was encountered searching for bug GCP-447 on the Jira server at https://issues.redhat.com. No known errors were detected, please see the full error message for details.

Full error message.


No response returned: Get "https://issues.redhat.com/rest/api/2/issue/GCP-447": GET https://issues.redhat.com/rest/api/2/issue/GCP-447 giving up after 5 attempt(s)

Please contact an administrator to resolve this issue, then request a bug refresh with /jira refresh.

Details

In response to this:

What this PR does / why we need it:

Changes InjectTokenMinterContainer in the cpov2 framework to inject the token-minter as a https://kubernetes.io/docs/concepts/workloads/pods/sidecar-containers/ (RestartPolicy: Always) with a StartupProbe that blocks main containers until the token file exists. This eliminates a race condition where the main container starts before the token is written, causing fatal crashes (observed in GCP CCM CI).

For backwards compatibility, the management cluster K8s version is detected at startup via ServerVersion(). On K8s >= 1.29 (where the SidecarContainers feature gate is beta and enabled by default), the native sidecar pattern is used. On older clusters, the current regular sidecar injection is preserved.

Which issue(s) this PR fixes:

Fixes GCP-447

Special notes for your reviewer:

The race condition has only been observed in GCP CI so far, but the underlying issue exists for all providers using InjectTokenMinterContainer and could surface for other cloud providers in the future.

The second commit removes the GCP CCM crash toleration from podCrashTolerations in the e2e test suite, as it is no longer needed.

Checklist:

Subject and description added to both, commit and PR.

Relevant issues have been referenced.

This change includes docs.

This change includes unit tests.

Summary by CodeRabbit

New Features

Added automatic detection of native sidecar container support based on management cluster version (Kubernetes 1.29.0 and later).

Enabled conditional injection of sidecar containers as native init containers when supported, with appropriate startup probes and volume mounting.

Tests

Added comprehensive test coverage for native sidecar detection and injection logic across multiple Kubernetes versions and deployment scenarios.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

cristianoveiga · 2026-03-15T14:55:46Z

/test ?

cristianoveiga · 2026-03-15T14:57:15Z

/test e2e-aks
/test e2e-aws

cwbotbot · 2026-03-15T16:35:03Z

Test Results

e2e-aks

Status: ✅ PASS
Started: 2026-03-15T14:57:44Z
View Job
View Job History

e2e-aws

Status: ✅ PASS
Started: 2026-03-15T14:57:45Z
View Job
View Job History

cristianoveiga · 2026-03-16T21:49:51Z

/retest

bryan-cox · 2026-03-17T11:29:56Z

/uncc @bryan-cox

cristianoveiga · 2026-03-17T15:24:56Z

/test ?

cristianoveiga · 2026-03-17T15:27:07Z

/test e2e-gke

enxebre · 2026-03-25T17:25:34Z

+		MetricsSet:                     r.MetricsSet,
+		EnableCIDebugOutput:            r.EnableCIDebugOutput,
+		ImageMetadataProvider:          r.ImageMetadataProvider,
+		NativeSidecarContainersEnabled: r.ManagementClusterCapabilities.Has(capabilities.CapabilityNativeSidecarContainers),


can we pass this to karpenter-operator/controllers/karpenter/karpenter_controller.go as well

karpenter_controller.go creates the karpenter component (v2/karpenter/component.go) which doesn't call InjectTokenMinterContainer, so setting NativeSidecarContainersEnabled in its ControlPlaneContext would have no effect.

The karpenter-operator component (v2/karpenteroperator/component.go) does use InjectTokenMinterContainer, but it's reconciled by the HCP controller which already sets the flag.

Happy to add it proactively if you prefer. I just want to confirm since it appears to be a no-op currently.

enxebre · 2026-03-25T17:27:11Z

  revisionHistoryLimit: 2
  selector:
    matchLabels:
-      app: gcp-cloud-controller-manager


this will break on any already existing cluster

We don't have any production GCP clusters at the moment - and we can re-create any clusters in integration, if needed.

Please note that this reverts the label change in #7926.

This was only needed for the crash toleration exception, which is no longer necessary with the native sidecar fix.

We prefer reverting to keep GCP CCM consistent with how other CCMs are labeled (app: cloud-controller-manager).

enxebre · 2026-03-25T17:27:36Z

dropped some feedback. This looks great to me overall.

Change InjectTokenMinterContainer in the cpov2 framework to inject the token-minter as a native sidecar init container (RestartPolicy=Always) with a StartupProbe that blocks main containers until the token file exists. This eliminates the race condition where the main container starts before the token is written, causing fatal crashes (observed in GCP CCM CI). For backwards compatibility, the management cluster K8s version is detected at startup via DetectManagementClusterCapabilities. On K8s >= 1.29 (where the SidecarContainers feature gate is beta and enabled by default), the native sidecar pattern is used. On older clusters, the current regular sidecar container injection is preserved. OneShot token-minters are injected as regular init containers, which run to completion before main containers start. Ref: https://issues.redhat.com/browse/GCP-447 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

cristianoveiga · 2026-03-26T01:38:08Z

/test verify

cristianoveiga · 2026-03-26T02:05:22Z

/retest-required

cristianoveiga · 2026-03-26T02:05:58Z

/test e2e-gke

cristianoveiga · 2026-03-26T12:36:57Z

/test e2e-gke

enxebre · 2026-03-26T15:46:49Z

/approve

openshift-ci · 2026-03-26T15:47:34Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cristianoveiga, enxebre

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [enxebre]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

cristianoveiga · 2026-03-26T16:04:58Z

/verfified by @cristianoveiga

The e2e-gke validated these changes - the failure is unrelated to this PR.

PR #7887 recently added the controlPlaneVersion status check to the e2e validation, which requires all ControlPlaneComponents to report RolloutComplete: True.

The cluster-network-operator never completes because cloud-network-config-controller is stuck — the secret cloud-network-config-controller-creds doesn't exist for GCP yet (tracked in PR #7824 / GCP-431).

This was previously invisible and is now surfaced by the new check.

cc: @apahim

muraee · 2026-03-26T16:36:47Z

/lgtm

openshift-ci-robot · 2026-03-26T16:38:00Z

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aks-4-21
/test e2e-aws-4-21
/test e2e-aks
/test e2e-aws
/test e2e-aws-upgrade-hypershift-operator
/test e2e-kubevirt-aws-ovn-reduced
/test e2e-v2-aws

cristianoveiga · 2026-03-26T16:45:27Z

/verified by @cristianoveiga

openshift-ci-robot · 2026-03-26T16:45:40Z

@cristianoveiga: This PR has been marked as verified by @cristianoveiga.

Details

In response to this:

/verified by @cristianoveiga

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot · 2026-03-26T18:04:35Z

/retest-required

Remaining retests: 0 against base HEAD f0b245f and 2 for PR HEAD 9dc48ce in total

openshift-ci-robot · 2026-03-26T19:04:30Z

/retest-required

Remaining retests: 0 against base HEAD a12b316 and 1 for PR HEAD 9dc48ce in total

openshift-ci-robot · 2026-03-27T01:04:14Z

/retest-required

Remaining retests: 0 against base HEAD ee66457 and 0 for PR HEAD 9dc48ce in total

openshift-ci · 2026-03-27T02:15:36Z

@cristianoveiga: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-aws-techpreview	`3c7f914`	link	false	`/test e2e-aws-techpreview`
ci/prow/verify	`77737f8`	link	true	`/test verify`
ci/prow/e2e-gke	`9dc48ce`	link	false	`/test e2e-gke`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Mar 13, 2026

openshift-ci Bot added the do-not-merge/needs-area label Mar 13, 2026

cristianoveiga force-pushed the GCP-447 branch from 4962be8 to fb0e18e Compare March 13, 2026 22:00

openshift-ci Bot added the area/testing Indicates the PR includes changes for e2e testing label Mar 13, 2026

openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 15, 2026

cristianoveiga force-pushed the GCP-447 branch 2 times, most recently from efc1c40 to b9226a3 Compare March 15, 2026 14:53

openshift-merge-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 15, 2026

cristianoveiga marked this pull request as ready for review March 16, 2026 20:37

openshift-ci Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Mar 16, 2026

openshift-ci Bot requested review from bryan-cox and csrwng March 16, 2026 20:42

openshift-ci Bot removed the request for review from bryan-cox March 17, 2026 11:30

enxebre reviewed Mar 25, 2026

View reviewed changes

openshift-ci-robot removed the verified Signifies that the PR passed pre-merge verification criteria label Mar 25, 2026

openshift-ci Bot added area/platform/aws PR/issue for AWS (AWSPlatform) platform area/platform/azure PR/issue for Azure (AzurePlatform) platform and removed lgtm Indicates that a PR is ready to be merged. labels Mar 25, 2026

cristianoveiga requested a review from enxebre March 25, 2026 19:34

cristianoveiga force-pushed the GCP-447 branch from ecfc449 to 77737f8 Compare March 25, 2026 20:23

cristianoveiga force-pushed the GCP-447 branch from 77737f8 to 9dc48ce Compare March 26, 2026 00:36

openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 26, 2026

openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Mar 26, 2026

openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Mar 26, 2026

openshift-merge-bot Bot merged commit 37e3785 into openshift:main Mar 27, 2026
29 of 30 checks passed

Conversation

cristianoveiga commented Mar 13, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Checklist:

Summary by CodeRabbit

Release Notes

Uh oh!

openshift-ci-robot commented Mar 13, 2026

Uh oh!

openshift-ci Bot commented Mar 13, 2026

Uh oh!

coderabbitai Bot commented Mar 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Reviews paused

Walkthrough

Sequence Diagram(s)

Uh oh!

openshift-ci-robot commented Mar 13, 2026

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Checklist:

Uh oh!

cristianoveiga commented Mar 13, 2026

Uh oh!

coderabbitai Bot commented Mar 13, 2026

Uh oh!

openshift-ci-robot commented Mar 13, 2026

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Checklist:

Summary by CodeRabbit

Uh oh!

cristianoveiga commented Mar 15, 2026

Uh oh!

cristianoveiga commented Mar 15, 2026

Uh oh!

cwbotbot commented Mar 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results

e2e-aks

e2e-aws

Uh oh!

cristianoveiga commented Mar 16, 2026

Uh oh!

bryan-cox commented Mar 17, 2026

Uh oh!

cristianoveiga commented Mar 17, 2026

Uh oh!

cristianoveiga commented Mar 17, 2026

Uh oh!

enxebre Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

cristianoveiga Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

enxebre Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

cristianoveiga Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

cristianoveiga Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

enxebre commented Mar 25, 2026

Uh oh!

cristianoveiga commented Mar 26, 2026

Uh oh!

cristianoveiga commented Mar 26, 2026

Uh oh!

cristianoveiga commented Mar 26, 2026

Uh oh!

cristianoveiga commented Mar 26, 2026

cristianoveiga commented Mar 13, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Mar 13, 2026 •

edited

Loading

cwbotbot commented Mar 15, 2026 •

edited

Loading

cristianoveiga commented Mar 26, 2026 •

edited by openshift-ci Bot

Loading