[Bug]: apiserver never finishes startup when com.apple.pfd is unresponsive — every container command hangs forever (macOS 27.0)

[afischer-zedas]

Summary

On macOS 27.0, every container command hangs forever — system start, system status, all of them. There is no error, no timeout, and no way to diagnose it
with the CLI itself, because the CLI is the thing that blocks.

The underlying fault is outside container: /usr/libexec/pfd never answers an
XPC request from /usr/libexec/InternetSharing. But container turns an
unresponsive OS daemon into an indefinite hang rather than an error, which is
the part I think is actionable here.

Environment

• container 1.4.1 (Homebrew, brew install container)
• macOS 27.0 (26A428), Apple silicon
• Reproduced across reboots; persists indefinitely once triggered

What happens

$ container system start
Launching container-apiserver...
Testing access to container-apiserver...
<hangs forever>

$ container system status
<hangs forever, no output at all>

Both had to be killed manually. This also hangs any automation that shells out
to container — in my case an unattended update run sat blocked for hours
overnight, which is how I found it.

Where it blocks

container-apiserver starts, registers plugins, and stops during startup while
bringing up the network plugin. Its last log line, and then silence:

container-apiserver [com.apple.container:APIServer] initializing networks service
container-apiserver [com.apple.container:APIServer] Registering plugin [id=com.apple.container.container-network-vmnet.default]
container-apiserver [com.apple.xpc:connection] activating connection: mach=true listener=false peer=false name=com.apple.container.network.container-network-vmnet.default

Because the apiserver never finishes startup, it never activates its own
listener, so every client connecting to com.apple.container.apiserver blocks
forever. That is why system status hangs exactly like system start.

sample of container-network-vmnet, parked in the same place for 3+ hours:

NetworkVmnetHelper.Start.run()
 └─ protocol witness for Network.start() in conformance ReservedVmnetNetwork
     └─ ReservedVmnetNetwork.startNetwork(configuration:log:)
         └─ vmnet_network_create  (in vmnet)
             └─ _NETRBCreateNetwork  (in Netrb)
                 └─ _NETRBClientCreateInternal
                     └─ NETRBXPCSetupAndSend
                         └─ xpc_connection_send_message_with_reply_sync
                             └─ mach_msg2_trap

xpc_connection_send_message_with_reply_sync has no timeout, so this never
returns. The chain below it:

container-network-vmnet
 └─ com.apple.NetworkSharing → /usr/libexec/InternetSharing
     last log: "[com.apple.pf] user xpc send -> delete", then connects to com.apple.pfd, then silence
     └─ /usr/libexec/pfd  ✗ exits 3 immediately: "no pf starter references held"
        launchd re-spawns it every 10s forever (runs counter climbs by 1 per 10s)

Why this is not a misconfigured host

pf's own state is clean, which rules out a stale anchor from a VPN or security
product (this machine has third-party network and endpoint-security system
extensions installed; the output below is what exonerates them):

$ sudo pfctl -s info
Status: Disabled

$ sudo pfctl -s References
No pf starter references held

$ sudo pfctl -s Anchors
  com.apple

So pfd is reporting the true state, not a corrupted one — it just exits on
that condition without replying to the in-flight request. That is an Apple bug
and I am filing it with Apple separately.

Things that do not help, for the record:

• Uninstalling or fully stopping container. Kill every container process and
  pfd still re-spawns every 10s while InternetSharing stays wedged.
• Rebooting. With container completely stopped and no container process
  present, com.apple.Virtualization.VirtualMachine (a Virtualization.framework
  VM starting at login — Colima, in my case) connects to
  com.apple.NetworkSharing about a minute after boot, InternetSharing reaches
  the same dead end, and the pfd loop resumes. Any VM asking for a shared/NAT
  network re-arms it.
• sudo launchctl kickstart -k system/com.apple.NetworkSharing — refused:
  150: Operation not permitted while System Integrity Protection is engaged.

What I'd like container to do

I don't expect container to fix pf. But right now an unresponsive system
daemon is indistinguishable from slow work, for an unbounded time:

1. Bound the vmnet network bring-up. vmnet_network_create is reached
   through a synchronous XPC send with no reply timeout. A bounded wait that
   fails with something like "timed out creating vmnet network; the
   com.apple.NetworkSharing / pfd services are not responding" would turn hours
   of silence into an actionable message.
2. Bound plugin registration during apiserver startup, so one unhealthy
   plugin cannot prevent the apiserver from ever activating its listener. Even
   starting degraded, with the network plugin reported as failed, would leave
   container system status usable — which matters because right now the
   diagnostic command is broken by the same fault it would diagnose.
3. Give the CLI a client-side timeout on the apiserver connection, so
   container <anything> cannot hang indefinitely against a half-started
   apiserver.

Happy to supply full sample output or unified-log extracts if useful.

Reproduction

I can't give a clean recipe for wedging pfd on demand — it appeared with the
macOS 27.0 upgrade on this machine. But the container-side behaviour is
reproducible from any state where com.apple.pfd doesn't answer: the apiserver
hangs at network-plugin registration and every CLI call blocks forever. The
diagnostics above (launchctl print system/com.apple.pfd showing a climbing
runs count with last exit code = 3, plus sample of
container-network-vmnet) identify it in a few seconds once you know to look.

[afischer-zedas]

Follow-up with a more precise diagnosis of the blocking call, which changes what
the right mitigation is.

A sample of the blocked /usr/libexec/InternetSharing names both the call and
the queue it runs on:

DispatchQueue_36: com.apple.framework.pf          <- serial
  -> PFUserDeleteRules  (in PacketFilter)
    -> _dispatch_lane_barrier_sync_invoke_and_complete
      -> PFXPCSetupAndSend
        -> xpc_connection_send_message_with_reply_sync
          -> mach_msg2_trap

2508 of 2508 samples, 32 minutes after the daemon launched.

com.apple.framework.pf is a serial queue entered through a barrier sync, so the
missing reply does not merely fail one request — it occupies that queue
permanently. Every later pf operation in the daemon queues behind it with no
route back, so the daemon stays unusable for the life of the boot, and SIP blocks
launchctl kickstart -k system/com.apple.NetworkSharing as a recovery.

The consequence for container: a retry would not help. Once this state is
reached, nothing downstream of com.apple.NetworkSharing can succeed until
reboot, so a bounded wait plus an actionable error — ideally naming the
unresponsive service — is the only useful behaviour, rather than a retry loop.

The underlying defect is reported to Apple through Feedback Assistant.

[afischer-zedas]

Resolved on my side, and the trigger turns out to be simple and deterministic —
which corrects my earlier "host-specific, don't expect a clean install to show
it" framing. It reproduces on a default-configured Mac.

The condition is that no process holds a pf starter reference.

macOS's packet filter is reference-counted. When pfctl -s References reports
none held, /usr/libexec/pfd exits 3 with no pf starter references held
without replying to the request in flight. /usr/libexec/InternetSharing then
blocks forever in PFUserDeleteRules on its serial com.apple.framework.pf
queue, and container-network-vmnet blocks behind it in vmnet_network_create,
so the apiserver never finishes startup.

The usual reference holder is the macOS Application Firewall — but it installs pf
rules, and so takes a reference, only when stealth mode or block-all-incoming is
enabled. Stealth mode is off by default, so a stock Mac with no other pf user
holds zero references and is in the failing state.

Reproduction on macOS 27.0 (26A428), container 1.4.1:

sudo /usr/libexec/ApplicationFirewall/socketfilterfw --getstealthmode   # off
sudo pfctl -s References                                               # No pf starter references held
# start any VM requesting a shared/NAT network, then:
container system start                                                 # hangs indefinitely

The inverse, which fixes it:

sudo /usr/libexec/ApplicationFirewall/socketfilterfw --setstealthmode on
sudo pfctl -s References        # socketfilterfw now appears under TOKENS
sudo pfctl -s info              # Status: Enabled
# reboot (the already-blocked InternetSharing cannot recover in place), then:
time container system start     # completes in ~3.5s

I verified both directions, and a second Mac on the same macOS 27.0 with the same
container 1.4.1 and stealth mode on never showed the problem. The pfd respawn
loop stopped the instant the token appeared.

This does not change what this issue asks for. A user has no way to guess that
their firewall's stealth-mode setting determines whether container works, and
right now the failure mode is an unbounded silent hang in every subcommand,
including container system status — the one you would reach for to diagnose it.
A bounded wait with an error naming the unresponsive service would have made this
self-diagnosable in seconds instead of hours.

The pfd behaviour itself is Apple's to fix and is reported to them separately.