Repository navigation
feat: register container runtime factories after startup - #3953
Open
karaolidis wants to merge 12 commits into
Open
karaolidis wants to merge 12 commits into
karaolidis wants to merge 12 commits into
Conversation
Factories make runtime API calls. Holding the read lock across them made RegisterContainerHandlerFactory wait for a slow runtime, and every lookup queued behind the waiting writer. Signed-off-by: Nikolaos Karaolidis <nick@karaolidis.com>
A docker daemon that accepted connections but never answered blocked container discovery forever. Signed-off-by: Nikolaos Karaolidis <nick@karaolidis.com>
Register and handler construction had no deadline, so a runtime that accepted connections but never answered blocked them forever. CRI-O's --crio_client_timeout defaults to none. Each TaskPid retry gets its own deadline, so the backoff sleeps don't use it up. CrioClient has no context parameters, so the bounded CRI-O calls are reached through a type assertion; other clients fall back to the interface methods. Steady-state calls are unchanged. Signed-off-by: Nikolaos Karaolidis <nick@karaolidis.com>
The client was dialed in a sync.Once that also cached errors, so a containerd that started after cAdvisor was never picked up. Signed-off-by: Nikolaos Karaolidis <nick@karaolidis.com>
The default namespace was fixed on the shared connection by whichever caller dialed first. When docker's containerd snapshotter path dialed first with "moby", the containerd plugin looked for k8s.io containers in the moby namespace and never claimed them. Fixes google#3951. Signed-off-by: Nikolaos Karaolidis <nick@karaolidis.com>
RootDir() caches its first answer. For docker that is the --docker_root fallback if docker was down when the FS context was built; for podman it is "" if its own request fails. Either way a later Register used the wrong root. podman's /info always sets the root, so an empty one now fails Register, before any watcher starts. Signed-off-by: Nikolaos Karaolidis <nick@karaolidis.com>
Retry is opt-in so that Plugin's contract doesn't change: plugins that don't implement RetryablePlugin are still called once. Nothing calls InitializePlugins more than once yet. Signed-off-by: Nikolaos Karaolidis <nick@karaolidis.com>
The previous commits bound every call on their Register paths and make a failed Register leave nothing behind. systemd's Register can't fail. Signed-off-by: Nikolaos Karaolidis <nick@karaolidis.com>
A runtime that was down or slow when cAdvisor started was never registered until a restart. Its containers were tracked as raw, or with --docker_only not at all. Each global housekeeping pass now retries pending plugins before discovery, so a factory that registers claims new containers in the same pass. Watchers can only be started from Start, so any returned on a retry are dropped. Absent runtimes now fail every pass, so only their first failure is logged at the default level. Signed-off-by: Nikolaos Karaolidis <nick@karaolidis.com>
The raw factory accepts every cgroup by default, so a runtime container seen before its factory could claim it stayed raw forever. The new containerData is built before the raw entry is destroyed, so a failure keeps the raw entry. No deletion event is sent, because the container didn't exit. The re-check stops after half the interval and resumes where it stopped, so a slow runtime can't starve discovery. newContainerData also leaked its CPU load reader when updateSpec failed. Signed-off-by: Nikolaos Karaolidis <nick@karaolidis.com>
With registration retried, a restart no longer fixes a missing runtime, so operators need a signal to alert on. Signed-off-by: Nikolaos Karaolidis <nick@karaolidis.com>
Socket-activated podman exits when idle, so every registration retry pays its cold start. A host where that takes over 10s would never register. The defaults keep today's 10s. Signed-off-by: Nikolaos Karaolidis <nick@karaolidis.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
cAdvisor registers each runtime factory (docker, podman, containerd, CRI-O) once, in
Start. If the runtime is down or slow at that moment, the factory never registers until cAdvisor is restarted. Its containers are tracked as raw cgroups with no runtime labels, or not at all with--docker_only. Nothing reports the failure beyond one Info log line.The usual trigger is boot ordering, e.g. rootless podman whose socket comes up after cAdvisor. This was asked for in #953, and #1968's workaround was "restart cAdvisor".
Changes
RetryablePlugininterface, so thePlugincontract is unchanged./validateand exported ascadvisor_container_plugin_registered{plugin}.--docker_client_timeoutand--podman_client_timeout, defaulting to the current 10s, for socket-activated runtimes with slow cold starts.A runtime that becomes reachable is picked up within one global housekeeping interval (1m by default). Behaviour when every runtime is up at startup is unchanged, apart from "; will retry" on the first failure log of plugins whose runtime isn't installed.
The commits are meant to be reviewed one at a time.