tests/snr: add 5 destructive remediation tests (worker + master)

tests/snr-operator/README.md

@@ -1,18 +1,29 @@
 # SNR Operator Tests
 
 Automated tests validating the Self Node Remediation (SNR) operator
-deployment, configuration, OLM metadata, CRD validation, and config lifecycle.
+deployment, configuration, OLM metadata, CRD validation, config lifecycle,
+and destructive remediation (kubelet stop, node reboot via NHC detection).
 
 ## Prerequisites
 
 - OpenShift cluster with SNR operator installed via OLM
 - `KUBECONFIG` set with cluster-admin access
 - SNR installed in `openshift-workload-availability` namespace
+- For destructive worker tests (15-17): NHC operator installed, 2+ worker nodes
+- For destructive master tests (18-19): NHC operator installed, 1+ worker nodes,
+  3+ master nodes for etcd quorum safety
 
 ## Running
 
 ```bash
+# All SNR tests (non-destructive + destructive)
 ginkgo --label-filter="snr" ./tests/snr-operator/...
+
+# Non-destructive tests only
+ginkgo --label-filter="snr && disruption:nondestructive" ./tests/snr-operator/...
+
+# Destructive remediation tests only
+ginkgo --label-filter="snr && disruption:destructive" ./tests/snr-operator/...
 ```
 
 Or via the test runner:
@@ -191,3 +202,70 @@ and the Disabled condition disappears.
 - **Environment**: Connected or disconnected
 - **Standalone**: `ginkgo --label-filter="snr" --focus="SNRC deletion disables" ./tests/snr-operator/...`
 - **Pass criteria**: DS pods deleted after SNRC removal; SNR CR shows Disabled/ConfigurationNotFound; after SNRC recreation DS pods return and Disabled condition is absent
+
+### Destructive Tests
+
+Tests that stop kubelet on nodes, triggering NHC-based health detection
+and SNR remediation with node reboots. Require NHC operator installed,
+2+ worker nodes, and 3+ master nodes. Run time: ~5-10 minutes per test.
+
+### 15. Verify Worker Node Remediation After Kubelet Stop ([OCP-52416](https://polarion.engineering.redhat.com/polarion/#/project/OSE/workitem?id=OCP-52416))
+
+Stops kubelet on a worker node, NHC detects the unhealthy node and
+creates an SNR CR, SNR remediates by rebooting the node, then verifies
+the node recovers. Also validates that the OutOfServiceTaint strategy
+was auto-selected (OCP 4.15+) via controller-manager logs.
+
+- **Operators**: SNR v0.13.0+, NHC v0.12.0+
+- **Cluster**: Multi-node with 2+ workers
+- **Environment**: Connected or disconnected
+- **Standalone**: `ginkgo --label-filter="snr" --focus="worker node after kubelet stop" ./tests/snr-operator/...`
+- **Pass criteria**: Node rebooted (boot ID changed), creation timestamp unchanged (not deleted/recreated), OutOfServiceTaint auto-selected log message found
+
+### 16. Verify ResourceDeletion Strategy Evicts Workload Pod ([OCP-50772](https://polarion.engineering.redhat.com/polarion/#/project/OSE/workitem?id=OCP-50772))
+
+Creates a ResourceDeletion SNRT, deploys a test workload pod on the
+target worker, stops kubelet, and verifies the pod is evicted from the
+remediated node after SNR completes the remediation cycle.
+
+- **Operators**: SNR v0.13.0+, NHC v0.12.0+
+- **Cluster**: Multi-node with 2+ workers (skips if insufficient)
+- **Environment**: Connected (workload pod uses `registry.k8s.io/pause:3.9`)
+- **Standalone**: `ginkgo --label-filter="snr" --focus="ResourceDeletion" ./tests/snr-operator/...`
+- **Pass criteria**: Node rebooted, creation timestamp unchanged, workload pod evicted (deleted or moved off remediated node)
+
+### 17. Verify OutOfServiceTaint Strategy Evicts Workload Pod ([OCP-61594](https://polarion.engineering.redhat.com/polarion/#/project/OSE/workitem?id=OCP-61594))
+
+Creates an OutOfServiceTaint SNRT, deploys a test workload pod on the
+target worker, stops kubelet, and verifies the pod is evicted from the
+remediated node after SNR completes the remediation cycle.
+
+- **Operators**: SNR v0.13.0+, NHC v0.12.0+
+- **Cluster**: Multi-node with 2+ workers (skips if insufficient)
+- **Environment**: Connected (workload pod uses `registry.k8s.io/pause:3.9`)
+- **Standalone**: `ginkgo --label-filter="snr" --focus="OutOfServiceTaint" ./tests/snr-operator/...`
+- **Pass criteria**: Node rebooted, creation timestamp unchanged, workload pod evicted (deleted or moved off remediated node)
+
+### 18. Verify Master Node Remediation After Kubelet Stop ([OCP-55059](https://polarion.engineering.redhat.com/polarion/#/project/OSE/workitem?id=OCP-55059))
+
+Stops kubelet on a master/control-plane node, NHC detects the unhealthy
+node and creates an SNR CR, SNR remediates by rebooting the node, then
+verifies the node recovers and was not deleted/recreated.
+
+- **Operators**: SNR v0.13.0+, NHC v0.12.0+
+- **Cluster**: Multi-node with 3+ masters (etcd quorum safety)
+- **Environment**: Connected or disconnected
+- **Standalone**: `ginkgo --label-filter="snr" --focus="master node after kubelet stop" ./tests/snr-operator/...`
+- **Pass criteria**: Master rebooted (boot ID changed), creation timestamp unchanged (not deleted/recreated), node returns to Ready
+
+### 19. Verify Simultaneous Master and Worker Remediation ([OCP-56069](https://polarion.engineering.redhat.com/polarion/#/project/OSE/workitem?id=OCP-56069))
+
+Stops kubelet on both a master and a worker node simultaneously, creates
+separate NHC CRs for each role, and verifies both nodes are remediated
+concurrently -- both reboot and recover independently.
+
+- **Operators**: SNR v0.13.0+, NHC v0.12.0+
+- **Cluster**: Multi-node with 3+ masters and 1+ workers
+- **Environment**: Connected or disconnected
+- **Standalone**: `ginkgo --label-filter="snr" --focus="simultaneously" ./tests/snr-operator/...`
+- **Pass criteria**: Both nodes rebooted (boot IDs changed), creation timestamps unchanged (not deleted/recreated), both recovered to Ready state

tests/snr-operator/internal/snrparams/const.go

@@ -11,6 +11,9 @@ const (
 	// ExpectedReplicas defines the expected number of replicas for SNR controller manager.
 	ExpectedReplicas = int32(2)
 
+	// ManagerContainerName is the name of the main controller container in the SNR pod.
+	ManagerContainerName = "manager"
+
 	// CRDGroup is the Kubernetes API group for all SNR custom resources.
 	CRDGroup = "self-node-remediation.medik8s.io"
 
@@ -61,4 +64,65 @@ const (
 
 	// SNRMessageConfigNotFound is the status condition message when SNRC is missing.
 	SNRMessageConfigNotFound = "SelfNodeRemediation is disabled because configuration does not exist"
+
+	// --- Destructive (remediation) test constants ---
+
+	// OcDebugTimeout is the timeout for oc debug node/ commands.
+	// 5 minutes to allow for slow debug pod scheduling on ARM64/nested virt.
+	OcDebugTimeout = 5 * time.Minute
+
+	// SNRDeletionTimeout is how long to wait for the SNR CR to be deleted
+	// after remediation completes. This is the longest wait in the
+	// remediation cycle -- the Python tests use 800s.
+	SNRDeletionTimeout = 15 * time.Minute
+
+	// NodeReadyTimeout is how long to wait for a node to return to Ready
+	// after reboot.
+	NodeReadyTimeout = 15 * time.Minute
+
+	// RemediationCRDeletionTimeout is how long to wait for a CR to be
+	// fully deleted during cleanup (retry-safe deletion). NHC CRs can
+	// take several minutes to delete due to webhook finalizer processing.
+	RemediationCRDeletionTimeout = 5 * time.Minute
+
+	// WorkloadPodReadyTimeout is how long to wait for a test workload pod
+	// to reach Running phase.
+	WorkloadPodReadyTimeout = 2 * time.Minute
+
+	// WorkloadEvictionTimeout is how long to wait for workload pods to be
+	// evicted or rescheduled after remediation.
+	WorkloadEvictionTimeout = 5 * time.Minute
+
+	// NHCCRDName is the CRD name for NodeHealthCheck, used to detect if
+	// NHC is installed.
+	NHCCRDName = "nodehealthchecks.remediation.medik8s.io"
+
+	// NHCAPIGroup is the API group for NodeHealthCheck CRs.
+	NHCAPIGroup = "remediation.medik8s.io"
+
+	// NHCAPIVersion is the API version for NodeHealthCheck CRs.
+	NHCAPIVersion = "v1alpha1"
+
+	// NHCTestName is the name used for test NHC CRs targeting workers.
+	NHCTestName = "snr-test-nhc-workers"
+
+	// NHCMasterTestName is the name used for test NHC CRs targeting masters.
+	NHCMasterTestName = "snr-test-nhc-masters"
+
+	// SNRTResourceDeletionName is the name for the ResourceDeletion strategy SNRT.
+	SNRTResourceDeletionName = "snr-test-resource-deletion-template"
+
+	// SNRTOutOfServiceTaintName is the name for the OutOfServiceTaint strategy SNRT.
+	SNRTOutOfServiceTaintName = "snr-test-out-of-service-taint-template"
+
+	// OutOfServiceTaintKey is the taint key applied by SNR when using the
+	// OutOfServiceTaint remediation strategy (standard K8s taint).
+	OutOfServiceTaintKey = "node.kubernetes.io/out-of-service"
+
+	// PauseImage is the container image used for test workload pods.
+	PauseImage = "registry.k8s.io/pause:3.9"
+
+	// MinReadyMasterNodes is the minimum number of Ready master nodes
+	// required for master remediation tests (etcd quorum safety).
+	MinReadyMasterNodes = 3
 )

tests/snr-operator/tests/config_lifecycle.go

@@ -49,7 +49,7 @@ var _ = Describe(
 					By("Reading current SNRC to preserve original watchdog path")
 
 					snrc := &unstructured.Unstructured{}
-					snrc.SetGroupVersionKind(snrcGVK())
+					snrc.SetGroupVersionKind(snrcGVK)
 
 					err := APIClient.Get(context.TODO(),
 						client.ObjectKey{
@@ -136,7 +136,7 @@ var _ = Describe(
 					By("Saving current SNRC spec for later recreation")
 
 					snrc := &unstructured.Unstructured{}
-					snrc.SetGroupVersionKind(snrcGVK())
+					snrc.SetGroupVersionKind(snrcGVK)
 
 					err := APIClient.Get(context.TODO(),
 						client.ObjectKey{
@@ -158,7 +158,7 @@ var _ = Describe(
 						By("DeferCleanup: ensuring SNRC exists")
 
 						checkSNRC := &unstructured.Unstructured{}
-						checkSNRC.SetGroupVersionKind(snrcGVK())
+						checkSNRC.SetGroupVersionKind(snrcGVK)
 
 						getErr := APIClient.Get(context.TODO(),
 							client.ObjectKey{
@@ -228,7 +228,7 @@ var _ = Describe(
 					By("Verifying SNR status shows configuration not found")
 
 					liveSNR := &unstructured.Unstructured{}
-					liveSNR.SetGroupVersionKind(snrGVK())
+					liveSNR.SetGroupVersionKind(snrGVK)
 
 					Eventually(func() error {
 						getErr := APIClient.Get(context.TODO(),
@@ -289,7 +289,7 @@ var _ = Describe(
 						"Failed to create verification SNR CR for node %q", testNodeName)
 
 					verifySNR := &unstructured.Unstructured{}
-					verifySNR.SetGroupVersionKind(snrGVK())
+					verifySNR.SetGroupVersionKind(snrGVK)
 
 					Eventually(func() error {
 						getErr := APIClient.Get(context.TODO(),