Pod Security forbids creating Flux pod when set `privileged` #473

p-null · 2025-02-03T00:34:12Z

p-null
Feb 3, 2025

When I tried to apply flux-aio.cue following the flux-aio doc, i got flux stuck at waiting for 19 resource(s) to become ready...

Then I inspected it:, it's due to Pod Security violation:

❯ kubectl -n flux-system get replicaset
NAME              DESIRED   CURRENT   READY   AGE
flux-865fc67484   1         0         0       8m56s
❯ kubectl -n flux-system describe replicaset flux-865fc67484
...
Conditions:
  Type             Status  Reason
  ----             ------  ------
  ReplicaFailure   True    FailedCreate
Events:
  Type     Reason        Age                   From                   Message
  ----     ------        ----                  ----                   -------
  Warning  FailedCreate  9m7s                  replicaset-controller  Error creating: pods "flux-865fc67484-qff9g" is forbidden: violates PodSecurity "baseline:latest": host namespaces (hostNetwork=true), hostPort (containers "source-controller", "kustomize-controller", "helm-controller", "notification-controller" use hostPorts 9690, 9790, 9791, 9792, 9793, 9794, 9795, 9796, 9797, 9798, 9799)
...

But I did set securityProfile in flux-aio.cue:

        "flux": {
            module: url: "oci://ghcr.io/stefanprodan/modules/flux-aio"
            namespace: "flux-system"
            values: {
                hostNetwork:     true
                securityProfile: "privileged"
            }
        }

When I apply, I see:

❯ timoni bundle apply -f flux-aio.cue
6:34PM INF b:flux-aio > applying 1 instance(s)
6:34PM INF b:flux-aio > i:flux > applying module timoni.sh/flux-aio version 2.4.0-2
6:34PM INF b:flux-aio > i:flux > installing flux in namespace flux-system
6:34PM INF b:flux-aio > i:flux > Namespace/flux-system created
6:35PM INF would violate PodSecurity "restricted:latest": host namespaces (hostNetwork=true) logger=KubeAPIWarningLogger

Checking labels:

❯ kubectl get ns flux-system -o jsonpath='{.metadata.labels}'

{"app.kubernetes.io/managed-by":"timoni","app.kubernetes.io/name":"flux","app.kubernetes.io/part-of":"flux","app.kubernetes.io/version":"v2.4.0","instance.timoni.sh/name":"flux","instance.timoni.sh/namespace":"flux-system","kubernetes.io/metadata.name":"flux-system"}

Could anyone advise how to fix this? Thank you in advance!

Answered by stefanprodan

Feb 3, 2025

You need to set podSecurityProfile: "privileged", see stefanprodan/flux-aio#99

View full answer

stefanprodan · 2025-02-03T07:47:59Z

stefanprodan
Feb 3, 2025
Maintainer

You need to set podSecurityProfile: "privileged", see stefanprodan/flux-aio#99

5 replies

p-null Feb 3, 2025
Author

Thanks for your help!

I updated that but still stuck at waiting for 19 resource(s) to become ready...

Here is my config:

bundle: {
    apiVersion: "v1alpha1"
    name:       "flux-aio"
    instances: {
        "flux": {
            module: url: "oci://ghcr.io/stefanprodan/modules/flux-aio"
            namespace: "flux-system"
            values: {
                hostNetwork:     true
                podSecurityProfile: "privileged"
            }
        }
        "cilium": {
            module: url: "oci://ghcr.io/stefanprodan/modules/flux-helm-release"
            namespace: "flux-system"
            values: {
                repository: url: "https://helm.cilium.io"
                chart: {
                    name:    "cilium"
                    version: "*"
                }
                helmValues: {
                    operator: replicas: 1
                    ipam: mode:         "kubernetes"
                    hubble: {
                        relay: enabled: true
                        ui: enabled: true
                    }
                    kubeProxyReplacement: true
                    cgroup: {
                        autoMount: enabled: false
                        hostRoot: "sys/fs/cgroup"
                    }
                    env: {
                    "KUBERNETES_SERVICE_HOST": "localhost"
                    "KUBERNETES_SERVICE_PORT": "7445"
                    }
                }
                sync: targetNamespace: "kube-system"
            }
        }
    }
}

Here is my debug process:

❯ kubectl -n flux-system get pods
NAME                    READY   STATUS             RESTARTS       AGE
flux-865fc67484-jtlwc   0/4     CrashLoopBackOff   20 (52s ago)   7m8s

❯ kubectl -n flux-system describe deployment flux
...
OldReplicaSets:  <none>
NewReplicaSet:   flux-865fc67484 (1/1 replicas created)
Events:
  Type    Reason             Age    From                   Message
  ----    ------             ----   ----                   -------
  Normal  ScalingReplicaSet  7m21s  deployment-controller  Scaled up replica set flux-865fc67484 from 0 to 1
...

The pod is in CrashLoopBackOff status:

❯ kubectl -n flux-system get pods

NAME                    READY   STATUS             RESTARTS       AGE
flux-865fc67484-jtlwc   0/4     CrashLoopBackOff   24 (12s ago)   8m53s

event logs:

❯ kubectl -n flux-system get events --sort-by=.lastTimestamp
...
8m49s       Normal    Created             pod/flux-865fc67484-jtlwc    Created container: helm-controller
8m49s       Normal    Started             pod/flux-865fc67484-jtlwc    Started container helm-controller
8m49s       Normal    Started             pod/flux-865fc67484-jtlwc    Started container source-controller
8m37s       Warning   Unhealthy           pod/flux-865fc67484-jtlwc    Readiness probe failed: Get "http://136.243.111.196:9790/": dial tcp 136.243.111.196:9790: connect: connection refused
8m37s       Warning   Unhealthy           pod/flux-865fc67484-jtlwc    Readiness probe failed: Get "http://136.243.111.196:9799/readyz": dial tcp 136.243.111.196:9799: connect: connection refused
8m31s       Warning   Unhealthy           pod/flux-865fc67484-jtlwc    Liveness probe failed: Get "http://136.243.111.196:9794/healthz": dial tcp 136.243.111.196:9794: connect: connection refused
8m31s       Warning   Unhealthy           pod/flux-865fc67484-jtlwc    Liveness probe failed: Get "http://136.243.111.196:9799/healthz": dial tcp 136.243.111.196:9799: connect: connection refused
8m31s       Warning   Unhealthy           pod/flux-865fc67484-jtlwc    Liveness probe failed: Get "http://136.243.111.196:9792/healthz": dial tcp 136.243.111.196:9792: connect: connection refused
8m31s       Warning   Unhealthy           pod/flux-865fc67484-jtlwc    Liveness probe failed: Get "http://136.243.111.196:9796/healthz": dial tcp 136.243.111.196:9796: connect: connection refused
8m27s       Warning   Unhealthy           pod/flux-865fc67484-jtlwc    Readiness probe failed: Get "http://136.243.111.196:9796/readyz": dial tcp 136.243.111.196:9796: connect: connection refused
8m21s       Normal    Killing             pod/flux-865fc67484-jtlwc    Container notification-controller failed liveness probe, will be restarted
8m21s       Normal    Killing             pod/flux-865fc67484-jtlwc    Container source-controller failed liveness probe, will be restarted
8m21s       Normal    Killing             pod/flux-865fc67484-jtlwc    Container kustomize-controller failed liveness probe, will be restarted
7m49s       Normal    Pulled              pod/flux-865fc67484-jtlwc    Container image "ghcr.io/fluxcd/source-controller:v1.4.1" already present on machine
7m49s       Normal    Created             pod/flux-865fc67484-jtlwc    Created container: source-controller
3m51s       Warning   Unhealthy           pod/flux-865fc67484-jtlwc    Readiness probe failed: Get "http://136.243.111.196:9794/readyz": dial tcp 136.243.111.196:9794: connect: connection refused
3m47s       Normal    Killing             pod/flux-865fc67484-jtlwc    Container helm-controller failed liveness probe, will be restarted
...

Could you please help take another look? Thanks!

stefanprodan Feb 3, 2025
Maintainer

You didn't set the the env in the flux-aio values

p-null Feb 3, 2025
Author

Got it, thanks! But after updating it, I got a new error

❯ kubectl -n kube-system get pods
NAME                                    READY   STATUS                  RESTARTS        AGE
cilium-5w85q                            0/1     Init:CrashLoopBackOff   4 (41s ago)     2m25s
cilium-envoy-ds7br                      1/1     Running                 0               2m25s
cilium-envoy-k6k5w                      1/1     Running                 0               2m25s
cilium-envoy-k7kxv                      1/1     Running                 0               2m25s
cilium-operator-7dd7d5fc7d-vvst6        1/1     Running                 0               2m25s
cilium-qsr29                            0/1     Init:CrashLoopBackOff   4 (35s ago)     2m25s
cilium-st7s4                            0/1     Init:CrashLoopBackOff   4 (19s ago)     2m25s
coredns-578d4f8ffc-b4m9r                0/1     Pending                 0               5m55s
coredns-578d4f8ffc-ztbx2                0/1     Pending                 0               5m55s
hubble-relay-59b94b6665-85pdm           0/1     Pending                 0               2m25s
hubble-ui-69d69b64cf-8b6hs              0/2     Pending                 0               2m25s
kube-apiserver-talos-qph-9a2            1/1     Running                 0               5m36s
kube-controller-manager-talos-qph-9a2   1/1     Running                 1 (6m12s ago)   5m36s
kube-scheduler-talos-qph-9a2            1/1     Running                 1 (6m13s ago)   5m36s
omni-kube-service-exposer-5jgvr         1/1     Running                 2 (91s ago)     5m45s
omni-kube-service-exposer-cqwpb         1/1     Running                 1 (118s ago)    5m7s
omni-kube-service-exposer-jzg7s         1/1     Running                 1 (111s ago)    5m11s

It says: failed to fulfil mount request: open /run/containerd/io.containerd.runtime.v2.task/k8s.io/clean-cilium-state/sys/fs/cgroup: no such file or directory

❯ kubectl describe pod cilium-qsr29 -n kube-system
...
  Normal   Created    39s (x5 over 2m11s)  kubelet            Created container: clean-cilium-state
  Warning  Failed     39s (x5 over 2m11s)  kubelet            Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: failed to fulfil mount request: open /run/containerd/io.containerd.runtime.v2.task/k8s.io/clean-cilium-state/sys/fs/cgroup: no such file or directory
  Warning  BackOff    11s (x10 over 2m9s)  kubelet            Back-off restarting failed container clean-cilium-state in pod cilium-qsr29_kube-system(bffe7d9d-c5b5-427f-9a4d-5c535ea57cf9)

I mirrored the same config I used in helm install:

bundle: {
    apiVersion: "v1alpha1"
    name:       "flux-aio"
    instances: {
        "flux": {
            module: url: "oci://ghcr.io/stefanprodan/modules/flux-aio"
            namespace: "flux-system"
            values: {
                hostNetwork:     true
                podSecurityProfile: "privileged"
                env: {
                    "KUBERNETES_SERVICE_HOST": "localhost"
                    "KUBERNETES_SERVICE_PORT": "7445"
                }
            }
        }
        "cilium": {
            module: url: "oci://ghcr.io/stefanprodan/modules/flux-helm-release"
            namespace: "flux-system"
            values: {
                repository: url: "https://helm.cilium.io"
                chart: {
                    name:    "cilium"
                    version: "*"
                }
                helmValues: {
                    operator: replicas: 1
                    ipam: mode:         "kubernetes"
                    hubble: {
                        relay: enabled: true
                        ui: enabled: true
                    }
                    kubeProxyReplacement: true

                    cgroup: {
                        autoMount: enabled: false
                        hostRoot: "sys/fs/cgroup"
                    }
                    k8sServiceHost: "localhost"
                    k8sServicePort: "7445"

                    securityContext: {
                        capabilities: {
                            ciliumAgent: ["CHOWN", "KILL", "NET_ADMIN", "NET_RAW", "IPC_LOCK", "SYS_ADMIN", "SYS_RESOURCE", "DAC_OVERRIDE", "FOWNER", "SETGID", "SETUID"]
                            cleanCiliumState: ["NET_ADMIN", "SYS_ADMIN", "SYS_RESOURCE"]
                        }
                    }
                }
                sync: targetNamespace: "kube-system"
            }
        }
    }
}

Here is the successful Helm Install config I tried to mirror:

helm install \
    cilium \
    cilium/cilium \
    --version 1.16.6 \
    --namespace kube-system \
    --set ipam.mode=kubernetes \
    --set hubble.relay.enabled=true \
    --set hubble.ui.enabled=true \
    --set kubeProxyReplacement=true \
    --set securityContext.capabilities.ciliumAgent="{CHOWN,KILL,NET_ADMIN,NET_RAW,IPC_LOCK,SYS_ADMIN,SYS_RESOURCE,DAC_OVERRIDE,FOWNER,SETGID,SETUID}" \
    --set securityContext.capabilities.cleanCiliumState="{NET_ADMIN,SYS_ADMIN,SYS_RESOURCE}" \
    --set cgroup.autoMount.enabled=false \
    --set cgroup.hostRoot=/sys/fs/cgroup \
    --set k8sServiceHost=localhost \
    --set k8sServicePort=7445

All config values are the same.

I tried both cgroup: { autoMount: enabled: false } and cgroup: { autoMount: enabled: true }, both having the same error.

What could result in such difference when values passed are the same, and how can I further debug? Thanks!!

I am using Omni Talos if that's relevant.

p-null Feb 3, 2025
Author

Removing the cgroup entry will allow Cilium to install successfully, while I still couldn't figure the discrepancy of providing cgroup using Tinomi vs. Helm install... 🤔

stefanprodan Feb 3, 2025
Maintainer

On both clusters you could run helm get values and see if there are any differences.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Pod Security forbids creating Flux pod when set `privileged` #473

Uh oh!

{{title}}

Uh oh!

Replies: 1 comment 5 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Uh oh!

Pod Security forbids creating Flux pod when set privileged #473

Uh oh!

p-null Feb 3, 2025

Replies: 1 comment · 5 replies

Uh oh!

stefanprodan Feb 3, 2025 Maintainer

Uh oh!

p-null Feb 3, 2025 Author

Uh oh!

stefanprodan Feb 3, 2025 Maintainer

Uh oh!

Uh oh!

p-null Feb 3, 2025 Author

Uh oh!

p-null Feb 3, 2025 Author

Uh oh!

stefanprodan Feb 3, 2025 Maintainer

Pod Security forbids creating Flux pod when set `privileged` #473

p-null
Feb 3, 2025

Replies: 1 comment 5 replies

stefanprodan
Feb 3, 2025
Maintainer

p-null Feb 3, 2025
Author

stefanprodan Feb 3, 2025
Maintainer

p-null Feb 3, 2025
Author

p-null Feb 3, 2025
Author

stefanprodan Feb 3, 2025
Maintainer