docs/feat(connectivity): Substrate actors on Cilium socket-LB cannot resolve names over UDP — the actor's query leaves the worker with the kube-dns ClusterIP untranslated (`world`, denied); document `socketLB.hostNamespaceOnly: true` as a prerequisite and ship the worker resolver knob

[teemow]

Problem

On a Cilium installation with the socket load balancer on (socketLB.enabled: true, kube-proxy-replacement: true — Cilium's default and the shape of every Giant Swarm cluster), an actor's UDP DNS never gets an answer. A skill agent whose skill has a git source never reaches Ready: its golden actor logs

fatal: unable to access 'https://github.com/giantswarm/agent-skills/': Could not resolve host: github.com
ERROR failed to materialize Agent Plugins: materialize skill "agent-self-awareness": exit status 128

and atelet re-runs the golden boot every minute (ACTOR_STATE_RESUMING forever, the worker held). A plain agent boots only because its Go resolver honours the worker pod's options use-vc (see below) — every other resolver in the actor (git/libcurl, getaddrinfo in a musl or glibc image, Python, Node) asks over UDP and fails.

Hubble on the worker's node shows why:

kagent/kagent-default-…:37166 (ID:5728) <> kube-system/kube-dns:53 (world) policy-verdict:none EGRESS DENIED (UDP)
kagent/kagent-default-…:37166 (ID:5728) <> kube-system/kube-dns:53 (world) Policy denied DROPPED (UDP)

The destination is the kube-dns ClusterIP (10.96.0.10), untranslated, which Cilium classifies as world; the worker's egress policy allows k8s-app: kube-dns endpoints, not world, so it drops. Without the policy the packet would leave the node with the ClusterIP still in it and never be answered (to-network FORWARDED, conntrack [UNREPLIED]) — the same failure one hop later.

Seen with Substrate 0.0.27-gs.2, kagent 0.11.0-gs.3, chart 4.7.0, Cilium 1.20.1. The agentlab (kind, no Cilium) completes the same skill agent — the lab's proofs cannot catch this class of gap.

Root cause

Cilium's socket LB translates a ClusterIP in the connect()/sendmsg() cgroup hook of the socket's own network namespace. An actor's traffic is not socketed by the worker pod: ateom runs the sandbox in a nested netns (169.254.17.2 behind the ateom0 veth) and, per internal/ateomnet/net.go, REDIRECTs the actor's TCP to the atunnel (:15001, whose own connect() in the pod netns is translated and tunnels to atenet-egress) but only masquerades UDP. The masqueraded UDP query is a forwarded packet: it never passes a socket hook in the pod netns, the ClusterIP stays, and with socket LB on, the tc datapath does not do service translation for pod traffic. This is the documented Cilium limitation for nested-netns runtimes ("Istio sidecar, KubeVirt, Kata Containers, gVisor"), and the documented remedy is socketLB.hostNamespaceOnly: true, which keeps socket LB for the host namespace and re-enables tc-based service translation for pod traffic — the forwarded UDP query is then DNAT'd to a kube-dns pod at the worker's veth and the egress policy sees the kube-dns identity.

Proposed solution

1. Document the prerequisite (README / the prerequisites page of Docs for a self-service install: examples per cluster shape, a prerequisites page, the identity-provider guide with the redirect URIs, UPGRADE.md's CRD story per install path, NOTES.txt's next steps #309, next to the Kubernetes 1.35 feature gates): on Cilium with kube-proxy replacement, socketLB.hostNamespaceOnly: true is required for Substrate actors to resolve names over UDP; without it only Go-resolver actors work, and only with options use-vc on the worker pods.
2. Offer the worker-pod resolver knob in the chart: kagent.substrateWorkerPool.template exposes no dnsConfig; the installation today injects spec.dnsConfig.options: [{name: use-vc}] into the worker pods with a Kyverno mutation. If the WorkerPool template can carry it (or the Substrate chart can), ship it as a value so the Go-resolver path does not need a per-installation mutation.
3. Consider carrying upstream (substrate line): ateom masquerades actor UDP and assumes the node translates ClusterIPs for forwarded packets; a REDIRECT of UDP/53 into the atunnel (as it does for TCP) would make actor DNS independent of the CNI's socket-LB mode.

Acceptance criteria

• The chart's docs name socketLB.hostNamespaceOnly: true as a prerequisite for Substrate on Cilium (kube-proxy replacement / socket LB) with the reason (nested-netns actors, forwarded UDP DNS).
• A skill agent with a git-sourced skill reaches Ready on a Cilium installation configured per the docs (golden actor resolves github.com).
• Decision recorded on (2) and (3) — values knob and/or upstream patch prepared in the fork.