WIP: pkg/agent: wait for all volumes to be detached before rebooting

This commit provides PoC version of implementing agent waiting for all
volumtes attached to the node to be detached as a step after draining
the node, as shutting down the Pod does not mean the volume has been
detached, as usually CSI agent will be running as a DaemonSet on the
node and will take care of detaching the volume from the node when the
pod shuts down.

This commit improves rebooting experience, as right now if there is not
enough time for CSI agent to detach the volumes from the node, node gets
rebooted and pods using attached volumes have no way to be attached to
other nodes, which effectively increases the downtime caused for
stateful workloads.

This commit still requires tests and better interface for the users.

If someone wants to try this feature on their own cluster, I've
published the following image I've been testing with:

quay.io/invidian/flatcar-linux-update-operator:97c0dee50c807dbba7d2debc59b369f84002797e

Closes #30

Signed-off-by: Mateusz Gozdek <[email protected]>

Author: invidian

examples/deploy/rbac/cluster-role.yaml

@@ -55,6 +55,12 @@ rules:
       - daemonsets
     verbs:
       - get
+  - apiGroups:
+      - storage.k8s.io
+    resources:
+      - volumeattachment
+    verbs:
+      - list
   - apiGroups:
       - policy
     resourceNames:

pkg/agent/agent.go

@@ -291,6 +291,30 @@ func (k *klocksmith) process(ctx context.Context) error {
 
 	klog.Info("Node drained, rebooting")
 
+	for {
+		attachments, err := k.clientset.StorageV1().VolumeAttachments().List(ctx, metav1.ListOptions{})
+		if err != nil {
+			klog.Errorf("Listing volume attachments: %v", err)
+			continue
+		}
+
+		anyVolumeAttached := false
+
+		for _, attachment := range attachments.Items {
+			if attachment.Status.Attached && attachment.Spec.NodeName == k.nodeName {
+				anyVolumeAttached = true
+				klog.Infof("Volume %q is still attached, waiting for detach", attachment.Name)
+			}
+		}
+
+		if !anyVolumeAttached {
+			klog.Info("All volumes are detached from node, rebooting.")
+			break
+		}
+
+		time.Sleep(5 * time.Second)
+	}
+
 	// Reboot.
 	k.lc.Reboot(false)