ateom-gvisor: terminate a workload whose runsc containers are already gone - #30
Merged
Merged
Conversation
teemow
added a commit
that referenced
this pull request
Sep 13, 2026
…are already gone (#30, 65867b3; giantswarm/giantswarm#37773, #37742 row 43) Signed-off-by: Timo Derstappen <teemow@gmail.com>
… gone TerminateWorkload cleans up the sandbox's runsc containers by checking each one's state and then deleting it. `runsc state` is fatal on a container runsc does not know (`loading container: file does not exist`, exit status 128), so once a container was gone the terminate could never succeed: every retry failed at the state check, atelet never reset the actor's directories, ateapi never released the worker, and the actor stayed DELETING with its worker assigned until the worker pod was replaced. A container gets into that state on the ordinary path. When a golden boot crashes, the Run failure cleanup detaches the bundle's rootfs overlay; the first terminate then runs `runsc delete -force` on the stopped sandbox, which destroys the container's state and fails afterwards on the filestore file that lived on the detached overlay (`destroying container: failed to delete filestore file ... no such file or directory`). The delete workflow retries, and from the second attempt on `runsc state pause` fails for good. Observed on a four-worker pool where two crashed golden actors of superseded templates pinned two workers, the control plane retrying the delete every minute. List the containers runsc knows under the actor's state root first (`runsc list -quiet`; a root without container state lists nothing) and check and delete only those, application containers first and the pause container last. A container that is already gone is logged and skipped: it is already cleaned up, and the terminate goes on to release everything else. A delete that fails once its container is gone -- the filestore case above -- has done its job and is logged instead of failing the terminate, so a crashed boot is torn down in the first attempt. Tests: cleanupContainers with a runsc stand-in -- every container gone, the application container gone with the sandbox present, every container present, a delete that fails after removing its container, a delete that fails with the container still there, and a failing list (cmd/ateom-gvisor). Signed-off-by: Timo Derstappen <teemow@gmail.com>
…are already gone (#30, c3e452c; giantswarm/giantswarm#37773, #37742 row 43) Signed-off-by: Timo Derstappen <teemow@gmail.com>
teemow
force-pushed
the
fork/ateom-terminate-gone-containers
branch
from
September 13, 2026 20:53
0a45ac6 to
4455582
Compare
This was referenced Sep 13, 2026
teemow
added a commit
to giantswarm/kagent-upstream
that referenced
this pull request
Sep 13, 2026
… gVisor worker's terminate collects a crashed golden actor (giantswarm/giantswarm#37773) (#32) SUBSTRATE_VERSION 0.0.27-gs.7 -> 0.0.27-gs.9: v0.0.27-gs.9 = gs.8 + giantswarm/substrate#30, the worker (ateom-gvisor) whose TerminateWorkload skips runsc containers that are already gone and accepts a runsc delete that fails after removing its container. Before it a superseded AgentTemplate revision whose golden boot had crashed was never collected and its golden actor pinned one worker of the pool for good (gazelle 2026-09-13, two of four workers). The published chart stamps substrateWorkerPool.workerImage with the new worker; the platform's WorkerPool rolls once. FORK.md's pin row follows. Signed-off-by: Timo Derstappen <teemow@gmail.com>
teemow
added a commit
that referenced
this pull request
Sep 14, 2026
…are already gone (#30, c3e452c; giantswarm/giantswarm#37773, #37742 row 43) Signed-off-by: Timo Derstappen <teemow@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Carried patch for the line (giantswarm/giantswarm#37773, sub-issue of the kagent API v2 epic giantswarm/giantswarm#37705). Rebase-merge, one commit per patch; the second commit is the FORK.md ledger row.
Problem
TerminateWorkloadchecks each runsc container's state before deleting it.runsc stateis fatal on a container runsc does not know (loading container: file does not exist, exit 128), so once a container is gone the terminate can never succeed. On gazelle (2026-09-13, Substrate0.0.27-gs.8, worker image0.0.27-gs.7) two crashed golden actors of supersededAgentTemplaterevisions sat inDELETINGwith their worker assignment intact —kubectl ate get workersshowed 2 of 4 workersASSIGNED(1/1)— while the kagent controller loggedfailed to collect runtime revision … runsc state: exit status 128every minute per template.How the container gets lost: the golden boot crashes → the Run failure cleanup detaches the bundle's rootfs overlay (its
runsc delete -force pausewas killed by the 30 s cleanup budget) → the first terminate'srunsc delete -force pausedestroys the container state and then fails on the filestore file that lived on the detached overlay (destroying container: failed to delete filestore file … no such file or directory, exit 128) → every retry fails atrunsc state pause.Change
cleanupContainerslists the containers runsc knows under the actor's state root (runsc list -quiet; a root without state lists nothing) and checks and deletes only those; a container that is gone is logged and skipped. Arunsc deletethat fails once its container is gone (the filestore case) counts as done, so a crashed boot is torn down in the first attempt. Application containers first, the pause container last, as before.Tests
TestCleanupContainerswith a runsc stand-in (every container gone / application container gone, pause present / every container present / delete fails after removing the container / delete fails with the container still there / listing fails) andTestListArgs, incmd/ateom-gvisor.Lab proof (agentlab, 2026-09-13)
An
AgentTemplatewhose one skill carries a top-levelversion:(giantswarm/agent-skills@cb1fb768, the same crash as gazelle'svera) crash-loops its golden boot; the template is then changed. On0.0.27-gs.7the superseded revision was never collected in 56 minutes — the sameAborted→runsc delete: exit status 128→runsc state: exit status 128sequence, actorDELETING, workerASSIGNED. On this branch's first dev build (skip only) the sweep after the failed first attempt loggedrunsc container is already gone, nothing to clean up,Actor terminated, and the worker wentFREE; the delete tolerance in the final commit removes that first failed attempt.Upstream
No upstream issue or pull request on this path (kagent-dev/substrate and agent-substrate/substrate searched 2026-09-13); agent-substrate#1518 is the sibling symptom for a
runsc restorefailure. The upstream-facing branch isupstream/ateom-terminate-gone-containerson the mirrormain; listed in giantswarm/giantswarm#37742 (row 43).