Skip to content

Add event verification helper and FAR lifecycle event assertions - #71

Merged
ugreener merged 1 commit into
medik8s:mainfrom
ugreener:feat/far-event-verification
Aug 10, 2026
Merged

Add event verification helper and FAR lifecycle event assertions#71
ugreener merged 1 commit into
medik8s:mainfrom
ugreener:feat/far-event-verification

Conversation

@ugreener

@ugreener ugreener commented Aug 6, 2026

Copy link
Copy Markdown

Problem

The FAR destructive tests validate remediation outcomes (node reboot, taint application, CR status conditions, workload eviction, controller handover) but do not verify the Kubernetes Events emitted by the FAR operator during the remediation lifecycle. These events are part of the operator's stable contract and are used by monitoring tools, NHC, and cluster administrators to track remediation progress.

Summary

Add a shared event verification helper and use it to assert FAR lifecycle events in two destructive tests that exercise distinct code paths.

Changes

  • Add tests/internal/helpers/events.go with a reusable WaitForEvents helper that polls the API server for expected Kubernetes Events using field selectors. The helper is operator-agnostic and designed for reuse across all medik8s operator test suites (NHC, SNR, SBR)
  • Add event assertions to the standalone FAR remediation test (OCP-61229): verify RemediationStarted, FenceAgentSucceeded, and RemediationFinished events on the FAR CR, and NodeRemediationCompleted on the target Node
  • Add event assertions to the leader failover test (OCP-70638): verify the same lifecycle events survive controller failover when the active leader node is rebooted. This exercises a distinct code path where the emitting controller dies mid-remediation and a new leader must complete the event sequence
  • Event checks run after waitForRemediation() completes (reboot + Ready), so all events are expected to be present by that point

Jira: RHWA-1330

Summary by CodeRabbit

  • Tests

    • Expanded remediation coverage for standalone worker and leader-node remediation after controller failover.
    • Added verification of FAR lifecycle events, node remediation completion, workload pod eviction, and controller lease transfer.
    • Added reusable Kubernetes event validation supporting event reasons, types, and message matching.
  • Bug Fixes

    • Improved cleanup of temporary SSH key files when setup encounters permission errors.

@openshift-ci
openshift-ci Bot requested review from eemcmullan and mshitrit August 6, 2026 08:50
@openshift-ci openshift-ci Bot added the approved label Aug 6, 2026
@coderabbitai

coderabbitai Bot commented Aug 6, 2026

Copy link
Copy Markdown

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 5b1312da-906f-4394-9503-4eed9396caab

📥 Commits

Reviewing files that changed from the base of the PR and between d952659 and 6563ab2.

📒 Files selected for processing (5)
  • tests/far-operator/README.md
  • tests/far-operator/internal/farparams/const.go
  • tests/far-operator/tests/far_destructive.go
  • tests/internal/helpers/events.go
  • tests/internal/helpers/node_ops.go
🚧 Files skipped from review as they are similar to previous changes (5)
  • tests/far-operator/README.md
  • tests/far-operator/tests/far_destructive.go
  • tests/far-operator/internal/farparams/const.go
  • tests/internal/helpers/events.go
  • tests/internal/helpers/node_ops.go

📝 Walkthrough

Walkthrough

The PR adds Kubernetes event helpers, extends FAR destructive tests with remediation event checks, updates documented pass criteria, and removes temporary SSH keys when permission changes fail.

Changes

FAR event verification

Layer / File(s) Summary
Kubernetes event polling helpers
tests/internal/helpers/events.go
Adds event expectation types, involved-object references, event listing and matching, and timeout-bounded polling.
FAR remediation event assertions
tests/far-operator/internal/farparams/const.go, tests/far-operator/tests/far_destructive.go, tests/far-operator/README.md
Checks FAR lifecycle events and NodeRemediationCompleted during standalone and leader-node remediation. Updates the documented pass criteria.

SSH key cleanup

Layer / File(s) Summary
Temporary SSH key cleanup
tests/internal/helpers/node_ops.go
Removes the temporary SSH key file when changing its permissions fails.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related PRs

Suggested labels: ok-to-test

Suggested reviewers: eemcmullan, mshitrit, razo7

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 40.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely summarizes the new event verification helper and FAR lifecycle event assertions.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@qodo-2-for-medik8s

Copy link
Copy Markdown

PR Summary by Qodo

Verify FAR operator Kubernetes Events in destructive tests

🧪 Tests 🕐 20-40 Minutes

Grey Divider

AI Description

• Add a reusable helper to wait for expected Kubernetes Events on an involved object.
• Assert FAR remediation lifecycle events on the FAR CR in the primary destructive test.
• Assert remediation completion event on the remediated Node after remediation finishes.
Diagram

graph TD
Test["FAR destructive test"] --> Helper["WaitForEvents helper"] --> APIServer["K8s API server"] --> Events[("Kubernetes Events")]
FAR["FAR operator"] --> Events --> CR["FAR CR"]
Events --> Node["Target Node"]
Loading
High-Level Assessment

The following are alternative approaches to this PR:

1. Watch Events instead of polling
  • ➕ Lower API-server load than repeated list calls
  • ➕ Can assert ordering/timing if needed
  • ➖ More moving parts (watch setup, reconnect handling)
  • ➖ Harder to keep operator-agnostic in tests
2. Filter by involvedObject.uid (in addition to kind/name)
  • ➕ Avoids false positives when objects are recreated with the same name
  • ➕ More precise when tests run repeatedly against the same cluster
  • ➖ Requires reliably obtaining UID for each involved object before waiting
  • ➖ Some operators may emit early Events before UID lookup depending on flow
3. Use events.k8s.io/v1 API
  • ➕ Aligns with the newer Events API
  • ➕ Potentially better forward-compatibility
  • ➖ Adds extra API/version branching depending on cluster capabilities
  • ➖ Current core/v1 Events are widely available and simpler for test suites

Recommendation: The current list+fieldSelector polling approach is appropriate for destructive tests: it is simple, robust against transient failures, and keeps the helper reusable across operator suites. If flakiness appears (object recreation/name reuse), consider extending the selector to include involvedObject.uid and improving timeout errors to print the last missing expectations.

Files changed (2) +144 / -0

Tests (1) +26 / -0
far_destructive.goAssert FAR remediation lifecycle Events on CR and Node +26/-0

Assert FAR remediation lifecycle Events on CR and Node

• Adds post-remediation assertions that required FAR lifecycle Events exist on the FenceAgentsRemediation CR (RemediationStarted, FenceAgentSucceeded, RemediationFinished). Also asserts a NodeRemediationCompleted Event is emitted for the target Node.

tests/far-operator/tests/far_destructive.go

Other (1) +118 / -0
events.goAdd shared WaitForEvents helper for Kubernetes Event verification +118/-0

Add shared WaitForEvents helper for Kubernetes Event verification

• Introduces an operator-agnostic helper to list Events using server-side field selectors and poll until a set of expected (reason/type/message substring) Events are present. Exposes small data structures to describe the involved object and expected Event filters.

tests/internal/helpers/events.go

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/far-operator/tests/far_destructive.go`:
- Around line 425-447: Expose a one-attempt event-check helper in events.go,
then replace the two WaitForEvents assertions in the FAR lifecycle and Node
completion checks with Eventually using farparams.FARConditionTimeout and
farparams.DefaultPollInterval. Keep WaitForEvents unchanged and do not wrap it
in Eventually; assert the new helper’s single-attempt result directly within the
polling callback.

In `@tests/internal/helpers/events.go`:
- Around line 26-30: The event helpers at tests/internal/helpers/events.go:26-30
and tests/internal/helpers/events.go:78-81 must bind assertions to the current
remediation: extend InvolvedObjectRef with the UID and occurrence-time boundary,
add involvedObject.uid to the selector, and client-side filter using EventTime,
Series.LastObservedTime, or legacy LastTimestamp. In
tests/far-operator/tests/far_destructive.go:425-447, update createFARCR to
return the API-created object and fail on AlreadyExists, then pass its UID and
CreationTimestamp, along with the target Node UID, to both event checks.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 08a58b1c-ce3f-43fb-b5b7-4f6efc76eb89

📥 Commits

Reviewing files that changed from the base of the PR and between d952659 and b7be8b2.

📒 Files selected for processing (2)
  • tests/far-operator/tests/far_destructive.go
  • tests/internal/helpers/events.go

Comment thread tests/far-operator/tests/far_destructive.go
Comment thread tests/internal/helpers/events.go
@qodo-2-for-medik8s

qodo-2-for-medik8s Bot commented Aug 6, 2026

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📜 Skill insights (0)

Grey Divider


Remediation recommended

1. Swallowed event list errors ✓ Resolved 🐞 Bug ☼ Reliability
Description
helpers.WaitForEvents drops ListEventsForObject errors and continues polling, so persistent
API/RBAC/transport failures surface as a generic timeout instead of the real error. This makes the
new event assertions harder to debug and can add unnecessary delay before failing.
Code

tests/internal/helpers/events.go[R48-51]

+			events, err := ListEventsForObject(ctx, clientset, involved)
+			if err != nil {
+				return false, nil
+			}
Relevance

●●● Strong

Team previously accepted propagating API errors instead of swallowing them to avoid opaque timeouts
(same pattern).

PR-#59

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The poll loop explicitly converts any list error into a non-error retry, which means the caller
won’t see the underlying failure and will instead get a timeout/cancellation error from the polling
context.

tests/internal/helpers/events.go[46-51]
PR-#59

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`WaitForEvents` currently ignores errors from `ListEventsForObject` and keeps polling until the timeout, which hides the real root cause and turns errors into opaque context timeouts.

### Issue Context
This helper is intended to be reused across operator test suites. When the event list call fails (e.g., RBAC misconfiguration or transient apiserver issues), we should fail with the underlying error (or at least include it in the final error) rather than silently retrying until timeout.

### Fix Focus Areas
- tests/internal/helpers/events.go[46-69]
- tests/internal/helpers/events.go[73-92]

### Suggested fix
- In `WaitForEvents`, change the error handling from `return false, nil` to `return false, err`.
- Optionally: capture `lastErr` and, if the poll exits due to timeout/cancel, wrap the returned error to include the last list error and/or the last missing expectations for easier debugging.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


2. Event selector missing UID ✓ Resolved 🐞 Bug ≡ Correctness
Description
ListEventsForObject filters only on involvedObject.kind and involvedObject.name, so events from a
prior object instance with the same name (after delete/recreate) can satisfy expectations. This
weakens the new event assertions by not guaranteeing the events belong to the specific remediation
object instance under test.
Code

tests/internal/helpers/events.go[R78-81]

+	selector := fields.SelectorFromSet(fields.Set{
+		"involvedObject.kind": involved.Kind,
+		"involvedObject.name": involved.Name,
+	})
Relevance

●● Moderate

Team likes identity-disambiguation (UID/CreationTimestamp) but no direct precedent for event UID
selectors; may be seen as overkill.

PR-#52
PR-#53

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The helper’s selector omits UID, while client-go’s canonical event selection includes it to uniquely
identify an involved object. The FAR test uses the helper without any UID input, so it cannot
disambiguate events if the object name is reused.

tests/internal/helpers/events.go[73-86]
tests/far-operator/tests/far_destructive.go[423-447]
vendor/k8s.io/client-go/kubernetes/typed/core/v1/event_expansion.go[161-203]
tests/far-operator/tests/far_destructive.go[780-803]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`ListEventsForObject` only selects by involved object `kind` and `name`. Kubernetes object names can be reused after deletion, and Events can outlive the object; without selecting by `involvedObject.uid`, the helper can match stale events.

### Issue Context
This PR’s main purpose is to verify the FAR operator’s lifecycle Events. To ensure we verify events emitted for the current object instance, the selector should include `involvedObject.uid` (and optionally `involvedObject.namespace`), and call sites should pass the UID.

### Fix Focus Areas
- tests/internal/helpers/events.go[24-31]
- tests/internal/helpers/events.go[71-92]
- tests/far-operator/tests/far_destructive.go[423-447]

### Suggested fix
- Extend `InvolvedObjectRef` to include a `UID` field (e.g., `types.UID` or `string`).
- Add `involvedObject.uid` (and optionally `involvedObject.namespace`) to the field selector when provided.
- Update FAR destructive test call sites to fetch and pass the UID:
 - For FAR CR: `Get` the created CR (or rely on `Create`-populated UID) and pass its UID.
 - For Node: `Get` the Node and pass its UID.
- Alternative: use client-go’s Events `GetFieldSelector(...)` behavior as guidance (it includes UID when available).

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Grey Divider

Context used
⚠️ Tickets: not configured — ticket URL found in PR but could not be fetched — check ticket provider credentials
✅ Compliance rules (platform): 13 rules

Grey Divider

Tip of the day
💡 Did you know, you can reply 'qodo' on any finding to push back, ask questions, or dig deeper

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Grey Divider

Qodo Logo

Comment thread tests/internal/helpers/events.go Outdated
Comment thread tests/internal/helpers/events.go Outdated
@ugreener
ugreener force-pushed the feat/far-event-verification branch from b7be8b2 to 5b723d8 Compare August 6, 2026 09:15
@coderabbitai

coderabbitai Bot commented Aug 6, 2026

Copy link
Copy Markdown

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/internal/helpers/node_ops.go`:
- Around line 180-185: Update the chmod and write error branches in findSSHKey
to close tmpFile before calling os.Remove, matching the existing close-error
branch; preserve the current error propagation while ensuring cleanup occurs
before removal.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 6619125b-1f35-49de-8865-94a0b2418da2

📥 Commits

Reviewing files that changed from the base of the PR and between d952659 and 5b723d8.

📒 Files selected for processing (4)
  • tests/far-operator/README.md
  • tests/far-operator/tests/far_destructive.go
  • tests/internal/helpers/events.go
  • tests/internal/helpers/node_ops.go
🚧 Files skipped from review as they are similar to previous changes (2)
  • tests/far-operator/tests/far_destructive.go
  • tests/internal/helpers/events.go

Comment thread tests/internal/helpers/node_ops.go
@ugreener
ugreener force-pushed the feat/far-event-verification branch from 5b723d8 to 2ff2c58 Compare August 6, 2026 09:28
@coderabbitai

coderabbitai Bot commented Aug 6, 2026

Copy link
Copy Markdown

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

@ugreener

ugreener commented Aug 6, 2026

Copy link
Copy Markdown
Author

/test 4.22-konflux-e2e-far-aws

@ugreener
ugreener force-pushed the feat/far-event-verification branch from 2ff2c58 to 43ecf70 Compare August 6, 2026 12:08
@ugreener ugreener changed the title Add Kubernetes Event verification to FAR destructive tests Add event verification helper and FAR lifecycle event assertions Aug 6, 2026
@coderabbitai

coderabbitai Bot commented Aug 6, 2026

Copy link
Copy Markdown

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

@ugreener

ugreener commented Aug 6, 2026

Copy link
Copy Markdown
Author

/test 4.22-konflux-e2e-far-aws

Add a shared WaitForEvents helper in tests/internal/helpers/events.go
that polls the API server for expected Kubernetes Events on a given
involved object (CR or Node). The helper uses field selectors for
efficient server-side filtering and wait.PollUntilContextCancel for
retry logic.

Add event assertions to two FAR destructive tests:
- OCP-61229 (standalone worker remediation): verify RemediationStarted,
  FenceAgentSucceeded, RemediationFinished events on the FAR CR and
  NodeRemediationCompleted on the target Node
- OCP-70638 (leader node remediation): verify the same lifecycle events
  survive controller failover when the active leader is rebooted

The helper is operator-agnostic and can be reused by other operator
test suites (NHC, SNR, SBR) that emit events via medik8s/common.

Jira: RHWA-1330

Co-Authored-By: Claude <noreply@anthropic.com>
@ugreener
ugreener force-pushed the feat/far-event-verification branch from 43ecf70 to 6563ab2 Compare August 6, 2026 12:17
@coderabbitai

coderabbitai Bot commented Aug 6, 2026

Copy link
Copy Markdown

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

@ugreener

ugreener commented Aug 6, 2026

Copy link
Copy Markdown
Author

/test 4.22-konflux-e2e-far-aws

@JonahSussman JonahSussman left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure why we need an additional spaces here, but I won't block on that

@openshift-ci

openshift-ci Bot commented Aug 9, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: eemcmullan, JonahSussman, razo7, ugreener

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:
  • OWNERS [eemcmullan,razo7,ugreener]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ugreener
ugreener merged commit 58b984a into medik8s:main Aug 10, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants