SPLAT-2830: Add CI changes and periodic PROW changes for CI-Cluster C…

[nischawl]

…apacity monitoring

Summary by CodeRabbit

• Adds a six-hourly capacity-monitor job for the OpenShift Splat CI cluster.
• Runs the job on build01 with Nutanix credentials and the configured monitoring endpoint.
• Builds the project before monitoring cluster capacity.
• Reports monitoring failures and errors to the Splat alerts channel.
• Hardens vm-orphan-monitor with strict shell handling, restricted credential permissions, cleanup on exit, and a 30-second grace period.

[openshift-ci-robot]

@nischawl: This pull request references SPLAT-2830 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the spike to target the "5.0.0" version, but no target version was set.

Details

In response to this:

…apacity monitoring

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Pro Plus

Run ID: b5cd4e9b-52c1-4004-8789-ac2eadbc837b

📥 Commits

Reviewing files that changed from the base of the PR and between dce02c1 and 251b948.

📒 Files selected for processing (1)

• ci-operator/config/openshift-splat-team/ci-cluster-monitor/openshift-splat-team-ci-cluster-monitor-main__periodics.yaml

---

Walkthrough

Updates the cluster monitor configuration. It hardens Nutanix credential handling, adds resource settings to the VM orphan monitor, and schedules a six-hourly capacity-monitor job on build01 with Splat failure and error reporting.

Changes

Cluster monitoring

Layer / File(s)	Summary
Credential and resource hardening ci-operator/config/openshift-splat-team/ci-cluster-monitor/openshift-splat-team-ci-cluster-monitor-main__periodics.yaml	The VM orphan monitor enables strict shell handling, restricts temporary credential-file permissions, removes credentials on exit, adds a 30-second grace period, and sets CPU and memory requests.
Capacity monitor job ci-operator/config/openshift-splat-team/ci-cluster-monitor/openshift-splat-team-ci-cluster-monitor-main__periodics.yaml	Adds a six-hourly capacity-monitor job on build01 with Nutanix credentials, project build steps, endpoint and port settings, Splat failure and error reporting, and a 30-second grace period.

Estimated code review effort: 2 (Simple) | ~10 minutes

Suggested reviewers: annazivkovic, mtulio

---

Important

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

❌ Failed checks (1 error, 1 inconclusive)

Check name	Status	Explanation	Resolution
No-Sensitive-Data-In-Logs	❌ Error	Both monitor commands enable xtrace before invoking the binary, so the expanded prism_central_host endpoint can appear in Prow logs.	Keep xtrace disabled while invoking the monitors, or mask the endpoint before tracing is enabled.
Container-Privileges	❓ Inconclusive	The Prow configuration does not declare privilege fields, but the runtime user is inherited from an unspecified image and cannot be verified from this manifest.	Inspect the resolved builder and source image securityContext or Dockerfile to confirm whether either job runs as root.

✅ Passed checks (13 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly identifies the SPLAT issue and the CI and periodic PROW changes for CI-Cluster capacity monitoring.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	The commit changes only a Prow periodic YAML file; it adds no Ginkgo tests or It/Describe/Context/When titles, so no unstable test name exists.
Test Structure And Quality	✅ Passed	The PR changes only a periodic-job YAML file; it adds no Ginkgo It blocks, cluster-test setup, waits, or assertions to review.
Microshift Test Compatibility	✅ Passed	The PR patch changes only a periodic YAML file and adds no Ginkgo tests or test declarations, so the MicroShift compatibility check is not applicable.
Single Node Openshift (Sno) Test Compatibility	✅ Passed	The commit changes only a periodic YAML configuration; it adds no Ginkgo e2e tests or test constructs requiring SNO compatibility review.
Topology-Aware Scheduling Compatibility	✅ Passed	The change only adds Prow periodic job configuration and shell commands; it has no pod anti-affinity, topology spread, node selectors, tolerations, replica, or PDB constraints.
Ote Binary Stdout Contract	✅ Passed	The PR changes only one Prow YAML file; no OTE binary source or process-level Go code is added or modified, so the stdout contract check is not applicable.
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	The pull request changes only periodic-job YAML; it adds no Ginkgo tests or test networking, so this compatibility check is not applicable.
No-Weak-Crypto	✅ Passed	The PR adds only shell/YAML job controls and cleanup; added lines contain no MD5, SHA1, DES, RC4, Blowfish, ECB, crypto implementation, or secret comparison.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[nischawl]

/pj-rehearse periodic-ci-openshift-splat-team-ci-cluster-monitor-main-periodics-capacity-monitor

[openshift-merge-bot]

@nischawl: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

[coderabbitai]

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@ci-operator/config/openshift-splat-team/ci-cluster-monitor/openshift-splat-team-ci-cluster-monitor-main__periodics.yaml`:
- Around line 79-85: Update the report_template in reporter_config to use
failure-neutral wording for all failure and error states, such as identifying
the Nutanix capacity-monitor job failure and retaining the status URL link. Do
not claim that a capacity threshold was exceeded unless threshold-specific
reporting is separated from execution failures.
- Around line 93-95: Update the credential setup commands in the Nutanix
authentication block to set umask 077 before creating /tmp/nutanix-auth,
ensuring the directory and username/password files are owner-only. Add cleanup
on shell exit that removes the temporary credential directory, while preserving
the existing credential contents and paths.
- Around line 90-101: Add errexit, nounset, and pipefail to the shell options at
the start of the commands script, before the existing set +x in the CI monitor
command block. Preserve the credential setup, build, and capacity-monitor
invocation unchanged.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Pro Plus

Run ID: 8b04fff8-defb-4b44-8f8a-ee541cc03d13

📥 Commits

Reviewing files that changed from the base of the PR and between f9f3744 and 3ca880d.

⛔ Files ignored due to path filters (1)

• ci-operator/jobs/openshift-splat-team/ci-cluster-monitor/openshift-splat-team-ci-cluster-monitor-main-periodics.yaml is excluded by !ci-operator/jobs/**

📒 Files selected for processing (1)

• ci-operator/config/openshift-splat-team/ci-cluster-monitor/openshift-splat-team-ci-cluster-monitor-main__periodics.yaml

[nischawl]

/pj-rehearse periodic-ci-openshift-splat-team-ci-cluster-monitor-main-periodics-capacity-monitor

[openshift-merge-bot]

@nischawl: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

[openshift-merge-bot]

[REHEARSALNOTIFIER]
@nischawl: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name	Repo	Type	Reason
periodic-ci-openshift-splat-team-ci-cluster-monitor-main-periodics-vm-orphan-monitor	N/A	periodic	Ci-operator config changed
periodic-ci-openshift-splat-team-ci-cluster-monitor-main-periodics-capacity-monitor	N/A	periodic	Periodic changed

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

[jcpowermac]

/lgtm

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jcpowermac, nischawl

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• ci-operator/config/openshift-splat-team/ci-cluster-monitor/OWNERS [jcpowermac,nischawl]
• ci-operator/jobs/openshift-splat-team/ci-cluster-monitor/OWNERS [jcpowermac,nischawl]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[nischawl]

/pj-rehearse ack

[openshift-merge-bot]

@nischawl: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

[openshift-ci]

@nischawl: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.