Skip to content

RUNTIME.md

Latest commit

 

History

History
282 lines (248 loc) · 15.7 KB

RUNTIME.md

File metadata and controls

282 lines (248 loc) · 15.7 KB
Raw

Runtime reference

Run a VM

# boot VM 0 (default) — creates TAP fc0, guest gets 172.16.0.2
./start-vm.sh

# ssh in
ssh -i guest.id_rsa -o UserKnownHostsFile=.known_hosts root@172.16.0.2

# see what's running (or stale)
./list-vms.sh

# stop it
./stop-vm.sh

Clean shutdown

stop-vm.sh runs systemctl poweroff over SSH, waits for the guest to actually halt, then reaps the firecracker process and tears down host networking — so ext4 unmounts properly instead of replaying the journal on next boot.

Two non-obvious facts drive the design:

  • No ctrl-alt-del. The API's SendCtrlAltDel is inert here: start-vm.sh boot args pass i8042.noaux i8042.nomux i8042.nopnp i8042.dumbkbd, the controller probe fails (error -22), so the injected scancode reaches no driver (the call still returns 204). It is intentionally not used.
  • A clean poweroff does NOT exit firecracker. x86 Firecracker has no power device, so the kernel prints reboot: Power off not available: System halted instead and parks the VCPUs while the parent process keeps running. "Process exited" is therefore not the halt signal; stop-vm.sh always reaps.

Halt is confirmed by two independent signals, checked each poll:

  1. Serial log (authoritative). The boot log ends with that final reboot: line — proof poweroff.target completed and filesystems were unmounted. The log is rm -f'd every boot, so a match can't be stale.
  2. Fallback: guest answers no ping (a live-but-idle VM always answers ICMP; parked VCPUs kill virtio-net RX) and its CPU time stays frozen across two consecutive polls. Ping is the gate that stops an idle live guest from being misread as halted.

Consequences:

  • ssh must be present on the host, or you silently fall back to hard kills.
  • Every outcome prints how it was decided (serial log confirms… vs guest unreachable + CPU frozen…); a path that ends in a kill without halt evidence says so on stderr. Read that line rather than assuming a clean stop.
  • Typical clean stop is ~12s (matches the image's systemd teardown); the wait budget is 45s to absorb slow NFS unmounts from shared workspaces.

Multiple concurrent VMs

Each VM id gets its own API socket, TAP device, and /30 subnet:

./start-vm.sh 0   # guest 172.16.0.2  (TAP fc0)
./start-vm.sh 1   # guest 172.16.0.6  (TAP fc1)
./start-vm.sh 2   # guest 172.16.0.10 (TAP fc2)

./list-vms.sh     # status of all of them (ping, PID, vCPU/mem)
./list-vms.sh 0 2 # or check specific ids ("absent" = really stopped)

How the guest IP is assigned (no DHCP needed)

The CI rootfs ships with a fcnet-setup.sh that derives the guest IP from the interface MAC: 06:00:ac:10:00:GG → 172.16.0.GG. start-vm.sh sets the MAC accordingly, so the guest auto-configures its address on boot with no DHCP server.

Environment overrides

All paths are env-overridable for testing or non-default layouts:

FC_DIR=/some/other/dir ./start-vm.sh
BIN_INSTALL_DIR=~/.local/bin ./update-firecracker.sh binary
FC_SOCKET_DIR=/run ./start-vm.sh           # put API sockets in /run
API_SOCKET=/run/fc-vm0.sock ./start-vm.sh   # fully override one socket's path
VM_ID=3 FC_DIR=/some/other/dir ./start-vm.sh   # VM_ID env wins over the positional arg
KERNEL=/path/to/vmlinux ROOTFS=/path/to/rootfs.ext4 SSH_KEY=/path/to/key ./start-vm.sh
SHARE_DIR=~/project ./start-vm.sh          # live-mount ~/project at /workspace in the guest
SHARE_DIR=~/project SHARE_MNT=/work ./start-vm.sh  # ...or at a custom guest path
VCPU_COUNT=4 MEM_SIZE_MIB=8192 ./start-vm.sh   # override the machine profile (default: 2 vCPU / 2048 MiB)
AGENT_PROFILE=fc-agents ./auth-login.sh    # auth profile name (default fc-agents)
ANTHROPIC_CONFIG_DIR=... ./auth-login.sh   # ant's config dir (default ~/.config/anthropic-fc,
                                           #   or an existing <repo>/anthropic-config)

# rootfs isolation (start-vm.sh) — each VM gets its own writable bytes
ROOTFS_MODE=overlay ./start-vm.sh          # read-only shared base + per-VM overlay layer
ROOTFS_MODE=copy ./start-vm.sh             # per-VM copy of the base (no initrd needed)
ROOTFS_MODE=shared ./start-vm.sh           # old behavior: write the shared image (unsafe)
OVERLAY_SIZE_MIB=4096 ./start-vm.sh        # size of this VM's writable layer (default 2048)
RESET_LAYER=1 ./start-vm.sh                # discard this VM's writable bytes first
INITRD=/path/to/initrd-overlay.img ./start-vm.sh
PER_VM_KEY=0 ./start-vm.sh                 # reuse guest.id_rsa instead of this VM's own key

# network policy (start-vm.sh; applied to the whole fc-nat table)
GUEST_LAN_ACCESS=1 ./start-vm.sh           # let guests reach RFC1918 (default: blocked)
GUEST_HOST_PORTS=2049,8080 ./start-vm.sh   # host ports guests may reach (default: 2049)
GUEST_HOST_PORTS=2049,111 ./start-vm.sh    # ...add rpcbind if an NFS mount ever needs it
GUEST_HOST_FILTER=0 ./start-vm.sh          # disable guest->host filtering entirely
GUEST_EGRESS_ALLOW=api.anthropic.com,registry.npmjs.org ./start-vm.sh
                                           # allowlist guest egress, drop the rest
                                           #   (default: unset = unrestricted)
GUEST_EGRESS_DNS=1.1.1.1 ./start-vm.sh     # resolvers reachable while the allowlist is on
                                           #   (default 1.1.1.1,8.8.8.8)

# download integrity (update-firecracker.sh)
STRICT_PINS=1 ./update-firecracker.sh agent      # any pin change is fatal, incl. the installer
ALLOW_UNVERIFIED=1 ./update-firecracker.sh binary # proceed if upstream ships no checksum

FORCE_POWEROFF=1 ./stop-vm.sh 3            # ssh-poweroff the guest IP even with no TAP present

Notes:

  • start-vm.sh, stop-vm.sh, and list-vms.sh all understand FC_SOCKET_DIR and VM_ID — pass the same values you started the VM with when stopping or listing it.
  • API sockets default to $XDG_RUNTIME_DIR/firecracker (falling back to /tmp/firecracker-$UID), created 0700, and firecracker is launched under umask 077. The socket is a full control channel — anything that can connect to it can attach arbitrary host files as guest drives and read guest memory — and its mode otherwise comes from whatever umask you happen to have. stop-vm.sh and list-vms.sh still find sockets left in /tmp by an older start-vm.sh, so an already-running VM is not orphaned by the change.
  • A fully-renamed API_SOCKET is only seen by ./list-vms.sh <id> (single-id mode), not by the socket scan.
  • stop-vm.sh will not ssh-poweroff a guest address when its TAP is absent — that address may belong to something that is not ours. FORCE_POWEROFF=1 overrides.
  • VM_ID must be an integer in 0..63 — each id consumes one /30 out of the 172.16.0.0/24 range.
  • VCPU_COUNT (>= 1) and MEM_SIZE_MIB (>= 128) are validated and rejected if not integers; Firecracker has no memory or vCPU hot-plug, so a running VM keeps the profile it booted with (./list-vms.sh shows it).
  • ROOTFS_MODE defaults to auto: overlay when initrd-overlay.img exists, otherwise copy. Either way a VM never shares writable bytes with another — booting off an image some other VM already has attached is refused, and in overlay mode an unclean base is refused too, because a read-only mount cannot replay an ext4 journal (e2fsck -fy it on the host).
  • SSH_KEY still overrides the key, and in overlay/copy mode its public half ($SSH_KEY.pub) is installed in that VM instead of a generated one. Otherwise each VM gets guest-vm<id>.id_rsa, created on first boot; stop-vm.sh and share-dir.sh resolve the same key.
  • GUEST_EGRESS_ALLOW resolves hostnames on the host, when the ruleset is built. For CDN-fronted names the address set rotates, so it bounds where a guest can dial rather than guaranteeing an allowed name keeps resolving.
  • There is one fc-nat table for all VMs, so the allowlist is host-wide and remembered in .fc-egress-policy: stop-vm.sh rebuilds the same table, and without the remembered policy stopping one VM would silently restore open egress for every VM still running. It is forgotten when the last VM stops, or immediately if you pass an empty GUEST_EGRESS_ALLOW=. Pass it again on a later start-vm.sh to change it.
  • An allowlisted address inside RFC1918/CGNAT is dropped by the no guest->LAN rule before the allowlist is consulted; start-vm.sh warns rather than letting it look effective. GUEST_LAN_ACCESS=1 if you mean it.
  • ./list-vms.sh has a ROOTFS column: overlay:<layer> means a read-only base plus that VM's own layer; rw:<image> means a writable root.

Layout

firecracker-scripts/
├── prereqs.sh
├── update-firecracker.sh
├── start-vm.sh
├── list-vms.sh
├── stop-vm.sh
├── share-dir.sh
├── auth-login.sh
├── lib-fcnet.sh           # sourced by the four VM scripts (not executable on its own)
├── image-pins.lock        # sha256 of artifacts that publish no checksum (committed)
├── README.md
├── docs/THREAT-MODEL.md
└── .gitignore

After ./update-firecracker.sh (plus agent), the repo directory also contains (all gitignored):

vmlinux-<version>          # guest kernel
ubuntu-<version>.ext4      # guest rootfs (agent step: 2 GiB, claude, nfs-common, apt state stripped)
ubuntu-<version>.squashfs.upstream
squashfs-root/             # extracted rootfs tree the ext4 is built from
guest.id_rsa / .pub        # dedicated SSH keypair (gitignored)
anthropic-config/          # ant profile + credentials for the guest (gitignored, SECRET)
claude-sessions/           # Claude Code transcripts shared out of /root/.claude (gitignored, optional)
vmlinux-latest             # -> vmlinux-<version>
ubuntu-latest.ext4         # -> ubuntu-<version>.ext4
ubuntu-latest.id_rsa       # -> guest.id_rsa
fc-vm*.log                 # per-VM serial console logs (mode 0600)
.known_hosts               # TOFU guest host keys; deleted on an images rebuild
.fw-forward-added          # marker: we enabled firewalld intra-zone forwarding
.fc-egress-policy          # the egress allowlist in force (survives stop-vm.sh; see
                           #   GUEST_EGRESS_ALLOW). Removed when the last VM stops.

Architecture notes

  • The firecracker binary is installed to /usr/local/bin/firecracker-<tag>-<arch> with a stable firecracker symlink pointing at it.
  • Firecracker's serial console goes to fc-vm<ID>.log — tail -f it to watch boot. For interactive access, use SSH (the scripts launch firecracker detached with stdin from /dev/null, so the serial console is read-only by design).
  • Networking uses a hardcoded 172.16.0.0/24 range. Each VM's /30 subnet and TAP name are derived from VM_ID in lib-fcnet.sh (GUEST_IP, HOST_IP, TAP, and the MAC are computed, not env-overridable). If the range collides with another network on your host, change FC_SUBNET in lib-fcnet.sh and the matching derivation in start-vm.sh.
  • The host firewall treats every VM as its own trust domain. fc_nft_apply in lib-fcnet.sh rebuilds the whole fc-nat table atomically on every start and stop (validated with nft -c first, so a bad ruleset is reported instead of half-applied), from the TAPs that exist at that moment. The policy: guests reach the internet masqueraded; guest-to-guest is dropped; guests may not reach RFC1918/CGNAT/link-local destinations, i.e. your LAN (GUEST_LAN_ACCESS=1 opts out); guest-to-host is limited to NFS and ping (GUEST_HOST_PORTS adds ports, GUEST_HOST_FILTER=0 disables it); and a packet arriving on a TAP with a source outside the VM range is dropped, so a guest cannot spoof past any of it. Services on the host itself are unaffected by the LAN block — those are input, not forward. The guest→host allowance is 2049 only, on the basis that NFSv4 needs no portmapper; if a mount ever fails against a host whose nfs-utils disagrees, GUEST_HOST_PORTS=2049,111 is the escape hatch.
  • IPv6 is disabled on the TAP. The fc-nat rules are ip-family only, so a link-local address on the host end of the TAP would be an unfiltered path from the guest to any host service bound to :: — around the guest→host filtering entirely. start-vm.sh sets net.ipv6.conf.<tap>.disable_ipv6=1 before bringing the device up, and warns if a link-local address survives anyway. Nothing here uses IPv6: the /30, the NAT and the NFS mount are all IPv4.
  • Guest SSH host keys are verified. The scripts use a repo-local .known_hosts with StrictHostKeyChecking=accept-new: an unseen key is recorded, a changed one is refused. Host keys are baked into the image, so they are stable across boots; update-firecracker.sh images deletes .known_hosts when it rebuilds, since the rebuild regenerates them.
  • Guest internet egress needs four things, all handled by start-vm.sh (each was a real failure mode on Fedora): the host routes (net.ipv4.ip_forward=1, persisted to /etc/sysctl.d/99-fc-agents.conf), masquerade by source subnet leaving via the uplink (nft), the TAP bound to the same firewalld zone as the uplink interface + intra-zone forwarding (--add-forward) — firewalld rejects cross-zone forwarding even when our own nft chains accept — and a default route in the guest (the CI fcnet-setup.sh ships none; the agent step patches it in).
  • ./update-firecracker.sh agent operates on the extracted squashfs-root/ tree and rebuilds the ext4 from it — any state accumulated in the previous ext4 (e.g. a newer claude pulled by the auto-updater) is discarded by design, keeping rebuilds reproducible. The apt/dpkg state stripped from the shipped image lives on in squashfs-root/ (the strip runs on a hardlink staging copy), so re-running agent stays a fast idempotent no-op for the apt part. Re-run agent after every images rebuild. The flip side still holds: agent never removes anything the previous run put into squashfs-root/ — converting a tree built by an older script version (e.g. one with git) requires ./update-firecracker.sh images --force (fresh extract; plain images skips if versions match) followed by agent.
  • The guest ext4 is persistent across guest reboots (e.g. claude's self-updates survive), but not an images/agent rebuild. It ships no working package manager: the image is built appliance-style — packages are installed in the build chroot only, via AGENT_APT_PKGS in update-firecracker.sh.
  • The CI rootfs ships an empty /etc/resolv.conf; the agent step bakes working nameservers (1.1.1.1, 8.8.8.8) into the image. Without it nothing resolves in the guest.
  • The agent step hardens the guest image. The upstream CI rootfs allows SSH password authentication and ships root with an empty password field in /etc/shadow, guarded only by PermitEmptyPasswords no; the build turns password auth off (sshd_config.d/10-fc-agents.conf) and locks the account. Pubkey login is unaffected by a locked password. It also masks rpcbind, which nfs-common pulls in and which otherwise listens on 0.0.0.0:111 — NFSv4 only ever talks to 2049.
  • Downloads are checksum-verified. The firecracker tarball is checked against the .sha256 file upstream publishes, and a mismatch is fatal. The CI kernel/rootfs and claude.ai/install.sh publish no checksums, so their hashes are recorded in image-pins.lock (committed) on first fetch and checked afterwards — for the CI artifacts a later change is fatal, since a dated CI key is immutable; for the installer it is reported and re-pinned, since it legitimately changes. STRICT_PINS=1 makes everything fatal, ALLOW_UNVERIFIED=1 relaxes the first case. This does not protect a first fetch; see THREAT-MODEL.md.