fix(shield): drop the default CPU limit on the host shield - #2717
Open
mnacharov wants to merge 1 commit into
Open
fix(shield): drop the default CPU limit on the host shield#2717mnacharov wants to merge 1 commit into
mnacharov wants to merge 1 commit into
Conversation
solves sysdiglabs#2716 Signed-off-by: Mikhail Nacharov <1246847+mnacharov@users.noreply.github.com>
Contributor
|
Hi @mnacharov. Thanks for your PR. After inspecting your changes someone with write access to this repo needs |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
solves #2716
What this PR does / why we need it:
Removes
host.resources.shield.limits.cpu: 1000mfrom the chart defaults. Requests (250mCPU /384Mi) and the memory limit (1Gi) are unchanged.The host shield's CPU demand is set by two things the chart cannot see: the node's vCPU count — its userspace side is a single consumer draining one ring buffer per CPU — and the container creation rate. A flat
1000mthat is comfortable on an 8-core node is structurally undersized on a 64-core one: same quota, 8× the buffers to drain and 8× the aggregate syscall volume.The limit also doesn't bound the cost it appears to bound. The eBPF program or kernel-module hook runs in the context of whichever process issued the syscall, so that CPU time is charged to that container's cgroup; capping the shield container constrains only the drain side. And CFS throttling is all-or-nothing within a period — the shield freezes for tens of milliseconds rather than running proportionally slower.
On large, high-churn nodes this propagates off the container. Sustained throttling produces event drops, drops leave the thread and fd tables inconsistent and force
/procrescans that are CPU- and IO-heavy, which drives more throttling. Meanwhile per-container runtime metadata queries and their retries add latency to the containerd socket that kubelet's PLEG relist shares. The observed end state isPLEG is not healthyandNodeNotReadyon a node whose other 63 cores are idle.Changes
values.yaml(remove the key, add a comment on the trade-off and how to set one back) ·tests/host/daemonset_test.yaml(no CPU limit by default; an explicit limit is honoured)·
Chart.yamlversion bumpNo template change is needed —
host.resourcesalready passes straight throughtoYaml.values.schema.jsonis unaffected ($defs.HosthasadditionalProperties: true).For reviewers
Out of scope:
host.resources.kmodule,host_windows.resources, and the1000mmirrored inrh-shield-operatorsamples and the CSV. Happy to follow up on any of them — I only have evidence for the Linux host shield.Checklist
feat(agent,node-analyzer,sysdig-deploy):)