Skip to content

Side-loaded images are imported as unprefixed import-<date>@sha256 refs; after a container restart kubelet's checkpoint-image check fails with CreateContainerError until the ref is removed on the node #150

Description

@teemow

Problem

agentlab platform side-loads platform and Substrate images into the kind node (the image-archive import). containerd records each imported image as import-<date>@sha256:<manifest digest> — a reference without a registry prefix. The CRI plugin normalizes that name to docker.io/library/import-<date>@sha256:… for its image record, and kubelet's container-create path ("is this a checkpoint image?") asks containerd for the normalized name, which containerd does not have.

As long as a pod's image ID also carries a proper reference the CRI resolves first, nothing shows. After a container restart (an OOM-killed pod on agentlab-dev2, 2026-09-11 00:22Z) the CRI resolved the image ID of rustfs to the import ref first and every new pod of it failed with

CreateContainerError: failed to check if this is a checkpoint image: failed to get image from containerd "sha256:c5102f…": image "docker.io/library/import-2026-09-11@sha256:e3266f…": not found

crictl rmi <id> and crictl pull do not clear it (the pull says "Image is up to date"), and every agentlab platform run re-imports the images, re-creating the record. The Substrate helm upgrade --install then fails (Deployment/ate-system/rustfs not ready), so platform never reaches the meta chart — the lab cannot be restored by agentlab platform alone. Recovery that worked: ctr -n k8s.io images rm import-<date>@sha256:<digest> for the image's import ref (the exact unprefixed name from crictl inspecti <id>), keep the registry ref (docker.io/rustfs/rustfs@sha256:…), systemctl restart containerd on the node to flush the CRI cache, recreate the pod. Every side-loaded image (13 import-2026-09-11 records on that node) has the same exposure.

Proposed solution

The side-load leaves no unprefixed reference behind: import under the image's real name (ctr images import --index-name <registry ref> / tag the imported image to the reference the pod spec uses and delete the import-* ref), or side-load through the CRI (crictl pull from the local registry / kind load docker-image, which tags properly). Existing labs get a one-time cleanup of import-* refs on platform.

Acceptance criteria

  • After agentlab platform, ctr -n k8s.io images ls on the node shows no import-* reference; every side-loaded image is referenced by the name the pod specs use.
  • Deleting a pod of a side-loaded image (e.g. kubectl -n ate-system delete pod -l app=rustfs) recreates it Running, no CreateContainerError.
  • agentlab platform on a lab whose side-loaded pod was restarted converges without manual containerd surgery.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions