Skip to content

Stop using podman for qm - #993

Open
alexlarsson wants to merge 31 commits into
mainfrom
qm-next
Open

alexlarsson wants to merge 31 commits into
mainfrom
qm-next

Conversation

@alexlarsson

Copy link
Copy Markdown
Collaborator

This is a major change in how we start the qm container. Instead of using podman + quadlet, we're using a standard systemd qm.service file which starts "qmc launch". qmc is a small C program that generates the required OCI json file (similar to what podman would) in /run/qm/ and starts the container using crun. This lets qm start much faster, earlier and in ways more integrated with the system. The qmc app can be used in the installed image to e.g. run programs in the container (qmc exec bash) and inspect the current status (qmc inspect). This app replaces the old qmctl, even though its not an exact 1-to-1 replacement.

Interesting differences:

  • qm.service starts earlier (before basic.target)
  • qm.service starts before network-online.target, but still updates the resolv.conf in the qm container when it is later updated on the host side.
  • systemd in the qm container reports the sd_notify to the host qm.service when the qm systemd reached the default target (i.e. complete boot) which makes it possible on the host side to depend on a "fully started qm".
  • We run network setup (which is kinda slow) in parallel with the qm container start. Also, there is preparatory work that will allow the host network-online.target to signal the qm container and have its network-online.target follow that.
  • There is no dependency on netavark. Also, we're not using a bridge for the virtual network for the container, just a peer-to-peer veth.
  • The qmc config system support a very useful drop-in system that allows you to easily tweak the things we require in qm without having to change any file. These drop-ins are much more flexible that what we used to have, including things like optional device adding using globs or automatic use of seats. This allows the current oci hook + annotation hacks that we use for things like sound/video/tty to be replaced with simple qmc.conf.d drop ins. Also this should start much faster as we don't have to launch a bunch of processes in these hooks.
  • We now forward the host machine-id to the qm container, which is good now that we fixed the host machine-id to not be transient
  • qm systemd stdout output is visible as the host qm.service journal output, which makes things easier to debug

Note: while I tested and reviewed the main qmc work in another repo pretty carefully, the creation of this qm PR was mostly done by AI, so I marked this as draft, as I need to carefully review it. However, I want this in a PR early so people can look at it, and I can test out the CI changes.

@dougsland

dougsland commented Oct 17, 2026 •

Copy link
Copy Markdown
Member

The code looks good to me, even though this is a fairly large change. I couldn't find anything that caught my attention, particularly from a safety or potential side effects perspective.

We will need time to investigate why the CI/CD pipeline is failing and to perform additional testing locally. These would be good candidates for follow-up Jira tickets.

As a start, I would ask AI to fix the flake8 issues and shellcheck warnings here:
https://github.com/containers/qm/actions/runs/38773586065/job/116555407581?pr=993

Add your DCO too

@dougsland

dougsland commented Oct 17, 2026 •

Copy link
Copy Markdown
Member

The bandit CI/CD failure seems false positive https://github.com/containers/qm/actions/runs/38773585735
One possible solution is ignore the tests dir as it's not a production code for check. Or go file by file and ignore it using # nosec B108 like.

 .github/workflows/bandit
exclude_dirs: ['oci-hooks/tests','tools/tests','tests/qmc']

From:
https://github.com/containers/qm/actions/runs/38773585735

@alexlarsson

Copy link
Copy Markdown
Collaborator Author

I'll have a look at the failures later this week. But, the missing DCO is intentional, as I want to review things more before I sign off.

@dougsland

dougsland commented Oct 19, 2026 •

Copy link
Copy Markdown
Member

I believe worth speak with @pypingou regarding if we want to mimic qmctl commands into qmc as we had a RFE jira about it too.

This is added next to the existing Makefile, but the intent is to
replace it later, as we will eventually use it to build C code.

Assisted-by: AI
Signed-off-by: Alexander Larsson <alexl@redhat.com>
Use Meson for RPM builds and keep Makefile as a small developer
wrapper.

We move all the tests to CentOS 10, as that is the current target for
QM. All backwards compat features for CentOS 9 are dropped.

Assisted-by: AI
Signed-off-by: Alexander Larsson <alexl@redhat.com>
This adds the new qmc binary, a C-based replacement for using podman
to spawn the container. This binary replaces podman and netavark, and
the primary use of it is to do the qm network setup and and to create
the OCI container json config for crun. The actual running of the
container is still done by crun, and the json file is similar.

Assisted-by: AI
Signed-off-by: Alexander Larsson <alexl@redhat.com>
Register the existing unittest and C fixtures with Meson and RPM
checks, and build them in CentOS 10 CI.

Assisted-by: AI
Signed-off-by: Alexander Larsson <alexl@redhat.com>
Install qm.service and matching DNS/hosts watchers in place of the
outer Quadlet. Also adds the needed sysctl config files.

Assisted-by: AI
Signed-off-by: Alexander Larsson <alexl@redhat.com>
Assisted-by: AI
Signed-off-by: Alexander Larsson <alexl@redhat.com>
Move KVM/tun, sound, camera and TTY selections to qmc config
drop-ins, instead of using quadlets with oci-hooks. This is
fine now as the new config is much more expressible.

Assisted-by: AI
Signed-off-by: Alexander Larsson <alexl@redhat.com>
Pass qm-sysusers.conf explicitly to systemd-sysusers so enabling QM
user namespaces does not process unrelated host account definitions.

Assisted-by: AI
Set ping_group_range while preparing the host-owned network namespace,
before crun enters the payload user namespace. Resolve payload root
through the OCI GID mapping, retaining GID 0 without a user namespace.
Remove the OCI sysctl that crun cannot set from the child namespace.

Assisted-by: AI
Translate qmc-owned runtime files and host signal state into the
payload namespace, including the private /run/secrets directory.
Leave rootfs and writable etc/var mounts alone: their ownership is
already shifted on disk. Check both cases in bundle preparation.

Assisted-by: AI
Label qmc as a container runtime and relay non-terminal output through
runtime-owned pipes that qm_t can inherit. Add -i/--interactive for
stdin forwarding and -t/--tty for crun PTYs, with -t implying -i.
Default stdin to /dev/null while preserving output and exit status.

Use bounded nonblocking buffers so full output pipes or journal
sockets cannot prevent signal forwarding. Cover stream integrity,
file flags, backpressure, arguments and terminal selection in tests.

Assisted-by: AI
Build the controller and QM nodes from CentOS Stream 10 and select
the matching COPR repositories, as required by the native launcher.

Assisted-by: AI
Move outer test nodes to 10.91.0.0/16 so their routes do not overlap
QM at 10.90.0.0/24 or nested Podman at 10.89.0.0/16.

Assisted-by: AI
Replace outer Podman exec, inspect and Quadlet configuration with qmc
commands, native service state and JSON drop-ins across the existing
integration suites. Preserve Podman for the nested workloads.

Use the prepared writable trees for file copies, scope cleanup to
test-owned drop-ins, and bound waits for nested service readiness.

Assisted-by: AI
Add delegated fixtures for real crun startup, network/NAT behavior,
cleanup and recovery, exec streams and signals, and pre-shifted
filesystem ownership with mapped runtime mounts and ping sockets.
Check systemd machine IDs, readiness and host signal path activation.

Keep namespace and networking changes inside disposable test contexts;
document the wrapper and retain explicit SELinux coverage limits.

Assisted-by: AI
Add a bounded integration helper for a disposable machine with
enforcing SELinux, prepared storage and a local nested image. Check
service startup, exec, mounts, resolver updates, networking and nested
Quadlets using one userns mode matching the fixture ownership.

Assisted-by: AI
Replace the generated outer service directly, preserve activation
during upgrades, and reject leftover Quadlet configuration at startup
with migration instructions. Remove the host Podman requirement and
obsolete CLI packages, enable native QM during setup, and preserve
prepared filesystems on package removal.

Assisted-by: AI
Preserve README topics while updating service, capability and resource
examples. Add user-facing command, configuration and migration
references, build qmc(1) and qmc.conf(5) from the same MkDocs sources,
and port the existing usage, IPC, networking, subsystem and demo
instructions.

Assisted-by: AI
Signed-off-by: Alexander Larsson <alexl@redhat.com>
This is basically a rewrite, so lets make that clear in the version.

Signed-off-by: Alexander Larsson <alexl@redhat.com>
Write the controller socket address before restarting QM so a promptly
starting agent reads the final configuration. Collect its journal
through qmc exec on failure, alongside the host runtime journal.

Signed-off-by: Alexander Larsson <alexl@redhat.com>
The range used to uncomment additionalimagestores did not stop at a
closing bracket on its starting line. With the CentOS single-line
default it also uncommented the next section description, leaving
storage.conf invalid TOML and preventing nested Podman from starting.

Stop on the current line for inline arrays and remove leading comment
whitespace so the existing store insertion recognizes the
setting.

Signed-off-by: Alexander Larsson <alexl@redhat.com>
Remove the hook implementations, package, build jobs, and sanity
checks. These are not needed anymore with the new podman-less
qm.

Signed-off-by: Alexander Larsson <alexl@redhat.com>
Drop the unused qm-is-ostree helper, the obsolete KVM tar exclusion
list, and the empty host containers.conf.d installation.

Remove parted from the runtime dependencies. The disk test helper
installs it when partprobe is missing, without relying on the qm
package to provide it.

Signed-off-by: Alexander Larsson <alexl@redhat.com>
Define qm and qm-wayland as separate Packit packages, each with its
own spec and source preparation action. Explicitly select the package
on build, release, and test jobs, including the Wayland job inheriting
the main Copr template.

Keep all current test jobs attached to qm so a Wayland build cannot
launch them with the wrong artifacts.

Signed-off-by: Alexander Larsson <alexl@redhat.com>
Ignore the AIB test job until automotive-image-builder supports the
qmc launcher.

Signed-off-by: Alexander Larsson <alexl@redhat.com>
Non-terminal qmc exec keeps stderr separate from stdout. Capture both
streams when checking setenforce and require a failing exit status as
well as the expected permission denial, so command errors and actual
success cannot be mistaken for a pass.

Signed-off-by: Alexander Larsson <alexl@redhat.com>
The native launcher provides a PTY console for systemd to allow qm console
output (from e.g. systemd) to go into the host journal. This means that
the check that /dev/console doesn't exist breaks.

So, accept a /dev/console, but only if it is a /dev/pts, and that its
private (i.e. not the same as host /dev/console (in case that is a
pts).

Signed-off-by: Alexander Larsson <alexl@redhat.com>
Enable TMT's AVC check for the message-queue isolation test. Preserve
the denial details when queue creation fails, rather than reporting
only Permission denied.

Signed-off-by: Alexander Larsson <alexl@redhat.com>
Give the libkrun Copr priority 90 so its libkrun and firmware are
selected together instead of mixing newer AutoSD libkrun with older
Copr firmware. Otherwise I'm, getting a conflict where it picks
a newer libkrun from autosd than the one in the libkrun copr.

Signed-off-by: Alexander Larsson <alexl@redhat.com>
This is currently broken with the latest crun due to:
containers/crun#2487

Once that is fixed in cs10 we can re-enable this.

Signed-off-by: Alexander Larsson <alexl@redhat.com>
@alexlarsson
alexlarsson marked this pull request as ready for review November 20, 2026 16:02

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry @alexlarsson, this pull request is too large to review

It changes 25,127 lines. Sourcery can review pull requests with up to 20,000 changed lines.

We have some intermittent failures in check_container_kvm_run_in_qm
and check_qm_podman_quadlet_is_ok. This adds some more logging in case
they fails so we can debug these better.

Signed-off-by: Alexander Larsson <alexl@redhat.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants