Add snapsafety features to VMClock device #5564

bchalios · 2025-12-05T11:15:41Z

Changes

Implement snapshot safety features for VMClock device as described in this Linux patchset: https://lore.kernel.org/lkml/[email protected]/.

Expose the vm_generation_counter field in the vmclock_abi. Also, implement the notification feature. This simply means that during snapshot resume we bump vm_gneration_counter and inject an interrupt to the guest before resuming vCPUs.

Reason

Provide a means for the guest userspace to know that the microVM has been loaded from a snpahost

License Acceptance

By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache 2.0 license. For more information on following Developer
Certificate of Origin and signing off your commits, please check
CONTRIBUTING.md.

PR Checklist

This functionality cannot be added in rust-vmm.

codecov · 2025-12-05T11:20:55Z

Codecov Report

❌ Patch coverage is 91.04478% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.24%. Comparing base (5baa588) to head (e419450).

Files with missing lines	Patch %	Lines
src/vmm/src/devices/acpi/vmclock.rs	87.09%	4 Missing ⚠️
src/vmm/src/arch/aarch64/fdt.rs	91.66%	1 Missing ⚠️
src/vmm/src/device_manager/persist.rs	80.00%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #5564   +/-   ##
=======================================
  Coverage   83.23%   83.24%           
=======================================
  Files         277      277           
  Lines       29265    29310   +45     
=======================================
+ Hits        24359    24399   +40     
- Misses       4906     4911    +5

Flag	Coverage Δ
5.10-m5n.metal	`83.58% <90.90%> (+<0.01%)`	⬆️
5.10-m6a.metal	`82.91% <90.90%> (+<0.01%)`	⬆️
5.10-m6g.metal	`80.28% <88.46%> (+0.09%)`	⬆️
5.10-m6i.metal	`83.58% <90.90%> (+<0.01%)`	⬆️
5.10-m7a.metal-48xl	`82.90% <90.90%> (+<0.01%)`	⬆️
5.10-m7g.metal	`80.28% <88.46%> (+0.09%)`	⬆️
5.10-m7i.metal-24xl	`83.54% <90.90%> (+<0.01%)`	⬆️
5.10-m7i.metal-48xl	`83.54% <90.90%> (+<0.01%)`	⬆️
5.10-m8g.metal-24xl	`80.28% <88.46%> (+0.09%)`	⬆️
5.10-m8g.metal-48xl	`80.28% <88.46%> (+0.09%)`	⬆️
6.1-m5n.metal	`83.60% <90.90%> (+<0.01%)`	⬆️
6.1-m6a.metal	`82.94% <90.90%> (+<0.01%)`	⬆️
6.1-m6g.metal	`80.28% <88.46%> (+0.09%)`	⬆️
6.1-m6i.metal	`83.59% <90.90%> (-0.01%)`	⬇️
6.1-m7a.metal-48xl	`82.93% <90.90%> (+<0.01%)`	⬆️
6.1-m7g.metal	`80.28% <88.46%> (+0.09%)`	⬆️
6.1-m7i.metal-24xl	`83.61% <90.90%> (+<0.01%)`	⬆️
6.1-m7i.metal-48xl	`83.61% <90.90%> (+<0.01%)`	⬆️
6.1-m8g.metal-24xl	`80.28% <88.46%> (+0.09%)`	⬆️
6.1-m8g.metal-48xl	`80.28% <88.46%> (+0.09%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Add support for vm_generation_counter and notifications. Keep this separately for now, since currently upstream bindings don't include it. We will recreate this for Linux headers once they are released. Signed-off-by: Babis Chalios <[email protected]>

Add support for `vm_generation_counter` field in VMClock ABI. This field is similar to `disruption_marker` but it's only updated on snapshot loading events (not in live migration). It is meant to provide the guest with snapshot safety notifications. Moreover, add support for the notification capability. This capability require us to send an ACPI notification every time we change the seq_count field to a new even value. This essentially means that we need to send a notification upon resuming from a snapshot just before resuming vCPUs. Signed-off-by: Babis Chalios <[email protected]>

Extend VMClock integration tests to also account for the vm_generation_counter field and notification support flag. Signed-off-by: Babis Chalios <[email protected]>

Expose VMClock device to guest via DT and enable compiling the vmclock for ARM architectures. Keep VMClock tests only on x86 until we get support from the guest kernel. Signed-off-by: Babis Chalios <[email protected]>

Add support to apply patches while build kernels for our CI. For the time being we use this to apply patches for snapshot safety VMClock extensions [1] (backported to 5.10 and 6.1 AL kernels). [1] https://lkml.org/lkml/2025/12/3/653 Signed-off-by: Babis Chalios <[email protected]>

Enable CONFIG_PTP_1588_CLOCK_VMCLOCK for both 5.10 and 6.1 kernels Signed-off-by: Babis Chalios <[email protected]>

We now apply backported patches that add support for VMClock on Aarch64 systems via DT bindings, so enable tests for Aarch64 as well. Moreover, backported patches add support for the poll() system calls family, so enable the relevant test. Signed-off-by: Babis Chalios <[email protected]>

Manciukic · 2025-12-10T17:24:31Z

src/vmm/src/devices/acpi/generated/vmclock_abi.rs

 pub const VMCLOCK_FLAG_TIME_ESTERROR_VALID: u64 = 32;
 pub const VMCLOCK_FLAG_TIME_MAXERROR_VALID: u64 = 64;
 pub const VMCLOCK_FLAG_TIME_MONOTONIC: u64 = 128;
+pub const VMCLOCK_FLAG_VM_GEN_COUNTER_PRESENT: u64 = 256;


should we add this to our bindgen patches?

So, at the moment we should be doing that, in theory. In the future, when the ABI changes are merged in Linux mainline we won't need to. And this change will disappear. So, IOW, I think it doesn't matter how we apply this change now, because in the future we will just recreate the bindings and that's it. If we have this in our patches, the patches won't apply. Which isn't a big deal, we can re-create the patch, so either way works for me.

yeah, my concern is that if we need to regenerate the bindings we may inadvertently lose these manual changes as there's no patch to apply this.

if we do that, the Firecracker won't compile so we'll find out. If I add it as bindgen patches, we won't be able to recreate the bindings next time. Choose your poison :P (either way works for me)

fair, but I'd rather keep the generated files fully auto-generated. But it's no strong opinion, hopefully this is all temporary anyways.

Manciukic · 2025-12-10T17:38:51Z

resources/rebuild.sh

+    # Undo any patches previsouly applied, so that we can build the same kernel with different
+    # configs, e.g. no-acpi
+    git reset --hard $(get_tag $KERNEL_VERSION)
+    git clean -f -d
+


should we just create a temporary branch for the build? iow replace

- git checkout $(get_tag $KERNEL_VERSION) + git checkout -B tmp $(get_tag $KERNEL_VERSION)

That should work as well and it's much cleaner. I'll give it a try

Manciukic · 2025-12-10T17:43:01Z

resources/guest_configs/microvm-kernel-ci-aarch64-5.10.config

 # Enable PHYLIB and NETWORK_PHY_TIMESTAMPING to see the additional clocks.
 #
 CONFIG_PTP_1588_CLOCK_KVM=y
+CONFIG_PTP_1588_CLOCK_VMCLOCK=y


I think we should have this in our patches on top of this base config, otherwise we risk removing these when we rebase the config.

Problem is that our patches apply in the Linux tree and this config doesn't exist inside it. We would need a separate set of patches which apply in the configs, right?

I am not following. We're adding patches to the linux tree to support this device on aarch64. Why wouldn't the configs apply to it?
We do this after checking out the tree and applying the patches:

# Concatenate all config files into one. olddefconfig will then resolve # as needed. Later values override earlier ones. cat "$@" >.config make olddefconfig

So having it here or in vmclock.config should be equivalent, right?

oh right. I had forgotten about the extra configs we're keeping. I though you meant having it as part of the linux kernel source code pathces (the one I just added). Sure, I'll move it there.

We created a CI bucket that includes VMClock kernels with backported patches for the snapshot safety features. Signed-off-by: Babis Chalios <[email protected]>

bchalios marked this pull request as draft December 5, 2025 11:15

bchalios force-pushed the vmclock-snapsafe branch from 6818148 to 81dbeb2 Compare December 5, 2025 11:16

bchalios force-pushed the vmclock-snapsafe branch 8 times, most recently from 5487e22 to 115a1d6 Compare December 10, 2025 15:26

bchalios added 7 commits December 10, 2025 17:06

vmclock: test snapshot safety features

734f841

Extend VMClock integration tests to also account for the vm_generation_counter field and notification support flag. Signed-off-by: Babis Chalios <[email protected]>

vmclock: add support for Aarch64

5b6f3c8

Expose VMClock device to guest via DT and enable compiling the vmclock for ARM architectures. Keep VMClock tests only on x86 until we get support from the guest kernel. Signed-off-by: Babis Chalios <[email protected]>

test(vmclock): enable VMClock for Aarch64 kernels

7903cca

Enable CONFIG_PTP_1588_CLOCK_VMCLOCK for both 5.10 and 6.1 kernels Signed-off-by: Babis Chalios <[email protected]>

bchalios force-pushed the vmclock-snapsafe branch from 115a1d6 to 99d68b0 Compare December 10, 2025 16:07

Manciukic reviewed Dec 10, 2025

View reviewed changes

temp: point to CI bucket with VMClock kernels

e419450

We created a CI bucket that includes VMClock kernels with backported patches for the snapshot safety features. Signed-off-by: Babis Chalios <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add snapsafety features to VMClock device #5564

Add snapsafety features to VMClock device #5564

bchalios commented Dec 5, 2025

Uh oh!

codecov bot commented Dec 5, 2025 •

edited

Loading

Uh oh!

Manciukic Dec 10, 2025

Uh oh!

bchalios Dec 11, 2025

Uh oh!

Manciukic Dec 11, 2025

Uh oh!

bchalios Dec 11, 2025

Uh oh!

Manciukic Dec 11, 2025

Uh oh!

Manciukic Dec 10, 2025

Uh oh!

bchalios Dec 11, 2025

Uh oh!

Manciukic Dec 10, 2025

Uh oh!

bchalios Dec 11, 2025

Uh oh!

Manciukic Dec 11, 2025

Uh oh!

bchalios Dec 11, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Add snapsafety features to VMClock device #5564

Are you sure you want to change the base?

Add snapsafety features to VMClock device #5564

Conversation

bchalios commented Dec 5, 2025

Changes

Reason

License Acceptance

PR Checklist

Uh oh!

codecov bot commented Dec 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codecov bot commented Dec 5, 2025 •

edited

Loading