drop the default CPU limit on the host shield

[mnacharov]

Greetings, sysdiglabs team 👋🏽

I'm not 100% sure, but it looks like the default limits.cpu option beat me on one of the k8s clusters I operate.

I'm running sysdig/shield across several clusters and have never encountered any CPU issues with it, but I recently started running it in the same cluster as my ARC (actions runner controller), which has pretty high pod churn.
I noticed that on that cluster, my nodes started having PSI "some" CPU pressure reported by node-exporter, and also a huge communication disruption with containerd, causing CreateContainerError for any pod scheduled on that node.

I've noticed that the shield's pods were not ready on nodes experiencing the error, and a simple restart helps resolve the issue. At that moment, I also found that shield's pods experienced huge throttling (8-12s). Removing the limits.cpu seems to fully solve the issue on my side (I've had 1 day without the issue so far). Also, the CPU consumption has not increased

Quickly studying a topic led me to conclude that CPU limits on node-level infrastructure in the syscall path — CNI, CSI, security agents — cloud cause the issue I've experienced:

• https://docs.sysdig.com/en/sysdig-secure/classic-agent-drivers/
• https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#requests-and-limits

What do you think? Does the default field for the limits.cpu of the sysdig/shield DaemonSet really need to be set?

[mavimo]

@mnacharov thanks for reaching us!

We will like to retain the limit as save value for most of the use cases (as you mention no issues in most of the scenario) and let user to tune specific values for edge cases.

I suppose that the issue you face is related to the QoS where there is contention in the CPU usage, one option can be to increase the QOS for the Host Shield by setting the request = limit in this case automatically kubernetes will assign a higher QoS

If you prefer to not allocate 1000m for CPU request and you're fine to do not set the CPU limit you can set the value to null, as example:

helm template --set host.resources.shield.limits.cpu=null --set sysdig_endpoint.access_key=aaaaaaa --set sysdig_endpoint.region=us1 --set cluster_config.name=mycluster sysdig/shield | yq 'select(.kind == "DaemonSet") | .spec.template.spec.containers[0].resources'

will not set the CPU limit without the need to change the default values (the important part here is --set host.resources.shield.limits.cpu=null)

Let us know if this is a viable option for you.

[mnacharov]

Thanks for the quick response, @mavimo!
Yeah, I've already solved the problem on my end by setting the host.resources.shield.limits.cpu parameter to null. And I did it not only for the cluster with ARC but also for others.

The reason for raising this issue is that the default CPU resources of the shield component currently do not stick to any of the valid options:

1. Have Guaranteed QoS: request = limit, so CFS groups are not applied
2. Do not set limits, so CFS groups are not applied

For me, it looks like deploying with defaults definitely degrades the cluster's performance, just not enough to notice.